How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev · ‎08-23-2020

I use the example code in Create a FeatureLayer from a shapefile to add zipped shapefile to my 3D scene.

But this code uses ArcGIS REST API Generate and as stated there - Generate's json output is limiyed to 4000 records.

But I need to insert shapefile with more than 4000 records...

I will greatly appreciate help how to do it.

JoshuaBixby · ‎08-23-2020

The documentation you link to for Generate states that GeoJSON is limited to 4000, but you state you are working with shape files. Are you sure you even have an issue?

MichaelLev · ‎08-24-2020

Dear Joshua Bixby‌, (Dear Robert Scheitlin, GISP‌)

I apologize on not understanding your remark...

When I want to use Generate in order to insert zipped shapefile (please see the example code I pointed to), Generate returns json, and it can't return json with more than 4,000 records. It's written in its documentation, and I also got an error message if my zipped shapefile has more than 4,000 records).

But I get the zipped shapefile by transforming AutoCad DWG of a user, into shapefile, and that could have more than 4,000 records.

If I have not suceeded to clearly explain my problem, please let me know.

I wish I could insert by my esri Web AppBuilder (Developer Edition) custom widget, a zipped shapefile with more than 4,000 records, into the 3D Scene.

With appreciation,

Michael

RobertScheitlin__GISP · ‎08-24-2020

Michael,

When using the REST API's generate method there is always going to be a limit to what esri allows. It is great that you are getting 4000 records I remember when the limit has been 1000 records. WAB and esri's JS API were never meant to be a replacement for ArcGIS Desktop, so there will always be limits to what you can do online.

MichaelLev · ‎08-24-2020

Dear Robert,

Thank you very much for the immediate and informative answer.

However, since, as you can see, the usage of ESRI WebAppBuilder (developer Edition) is expanding, I wish to pass a request to ESRI REST API development team, to enlarge the limit a second time. I wish to enlarge the limit to at least 10,000 records, since nowadays regular Autocad DWG files that I convert to zipped shapefiles (by my FME cloud) contain near 10,000 reocrds. I am not an expert in all aspects of enlarging the limit, but as the limit has been multiplied by 4, from 1000 to 4000, I hope that nowadays it could be expanded even more. (If it could be enlarged to more then 10,000, it's even better, but I fear to request too much...)

I understand that there is no convenient way for widget in the AppBuilder to "bypass" the Convert REST API, so it makes sense to expand second time this limit, since it will expand the usage of ESRI Web AppBuilder and maybe more web application using ESRI REST API.

Can you please promote this request, or instruct me what can I do in order to promote it?

With appreciation,

Michael

RobertScheitlin__GISP · ‎08-24-2020

Michael,

The further increase would be like esri shooting themselves in the foot. Why will people need to buy desktop software is they can do everything online without purchasing any software to do it. Oh course this is just my opinion.

MichaelLev · ‎08-24-2020

Dear Robert,

I'll try to present my point of view -

Indeed I'm newbie to whole GIS area - I'm computer programmer for many many years. I've learned ArcGis Pro for a semester in University, in really good course and good instructor. Now I'm working for 3 months developing custom widgets, as you know.

What I understand, is that ArcGis Pro has its "area" of usage, and Web AppBuilder (Developer Edition) has another area.

Since many many companies who need GIS information, know nothing about GIS, so they use Web Applications (or Web AppBuilder (Developer Edition) applications, developed by companies who "know" GIS. And as you know, the widgets and custom-widgets are tailor made, in order to enable the users in companies who know nothing on GIS, to benefit GIS information, though they are simple users who know nothing.

More than that, the Web Applications and at least the Web AppBuilder (Developer Edition), supply specific gadgets and tools that the users need, and that could not be contained in ArcGIS Pro, since they are for specific needs of the users.

You can't expect that each user of GIS information will need to use ArcGIS Pro, which is a heavy tool.

Indeed, when a company's "main business" is GIS, this company "must" use ArcGIS Pro, but if it's business is something else, they wish to have an application which will help them to do specific tasks simple and easy, where the GIS processing is "under the hood" and they don't want to know about it.

For example, the company I'm working at, does many many aerial photography bu planes, and it's business is selling the option to view and inquire 3D Scenes, and our customers are simple people in companies that need many specific widgets to help them get the information they are interested in, from these 3D Scenes.

So they can't use ArcGIS Pro.

Our company pays for portal and online (and also ArcGIS Pro), and we use esri products, including the REST API, to give the users what they need.

I hope I made clear my opinion, that each product has different task and different "niche".

Indeed, esri products are excellent and I want to use them, and I am not an expert to evaluate what is better for esri.

I only presented my point of view.

I think esri should endeavour to the goal that users in thw whole world will prefer to use its products, and it will be if each product will "do its best" to supply users needs. ArcGIS Pro is a product, and REST API is a product. By answering users needs, both would thrive and prosperous.

I assume that enlarging REST API's Generate method maximum recods limit will answer current users needs, since the world is going forward and the needs grow constantly.

With appreciation,

Michael

MichaelLev · ‎08-26-2020

Dear Robert Scheitlin, GISP‌,

Can you please direct me where to request for increase of generate records limit?
Do I understand correctly from Henry's answer "you have to step trough the data E.G query record 1-4000 then 4001 to 8000 etc. etc. you can achieve this in REST as well" that this is a possible solution? But I don't know where can I find documentation/code_examples how to do this. Is it possible at all, to create chunks of less than 400 each (maybe per layer in the zipped shapefile) and then combine them to get the full featureCollection? I suspect that it's possible only on an already existing shape but not on zipped shape...

Thanks,

Michael

RobertScheitlin__GISP · ‎08-26-2020

Michael,

Unfortunately you guys are talking about two different things here. You are wanting to use the generate method on the API and that has a limit that is not configurable by you the end user. What Henry is talking about is a query of a layer on AGOL or ArcGIS Server (nothing to do with using a zipped shapefile).

MichaelLev · ‎08-26-2020

Dear Robert Scheitlin, GISP‌,

Thank you very much for clarifying what I have already suspected, that it’s impossible to create from the zip “by chunks”.

Only my intention in “1. ... where to request” was not “how to do by API”, but - “How to ask from ESRI company, to increase that limit a little more”.

Can you please tell me how to send such request?

You said that in the past that limit was 1000 records, and then ESRI raised the limit to 4000, so I hope that they will handle such request positively.

With appreciation to your immediate responses and explanations,

Michael