topic Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000 in Web AppBuilder Custom Widgets Questions

How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Sun, 23 Aug 2020 15:00:26 GMT

I use the example code in Create a FeatureLayer from a shapefile to add zipped shapefile to my 3D scene.

But this code uses ArcGIS REST API Generate and as stated there - Generate's json output is limiyed to 4000 records.

But I need to insert shapefile with more than 4000 records...

I will greatly appreciate help how to do it.

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

JoshuaBixby — Mon, 24 Aug 2020 00:38:03 GMT

The documentation you link to for Generate states that GeoJSON is limited to 4000, but you state you are working with shape files. Are you sure you even have an issue?

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Mon, 24 Aug 2020 08:27:58 GMT

Dear Joshua Bixby‌, (Dear Robert Scheitlin, GISP‌)

I apologize on not understanding your remark...

When I want to use Generate in order to insert zipped shapefile (please see the example code I pointed to), Generate returns json, and it can't return json with more than 4,000 records. It's written in its documentation, and I also got an error message if my zipped shapefile has more than 4,000 records).

But I get the zipped shapefile by transforming AutoCad DWG of a user, into shapefile, and that could have more than 4,000 records.

If I have not suceeded to clearly explain my problem, please let me know.

I wish I could insert by my esri Web AppBuilder (Developer Edition) custom widget, a zipped shapefile with more than 4,000 records, into the 3D Scene.

With appreciation,

Michael

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

RobertScheitlin__GISP — Mon, 24 Aug 2020 12:21:43 GMT

Michael,

When using the REST API's generate method there is always going to be a limit to what esri allows. It is great that you are getting 4000 records I remember when the limit has been 1000 records. WAB and esri's JS API were never meant to be a replacement for ArcGIS Desktop, so there will always be limits to what you can do online.

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Mon, 24 Aug 2020 13:14:42 GMT

Dear Robert,

Thank you very much for the immediate and informative answer.

However, since, as you can see, the usage of ESRI WebAppBuilder (developer Edition) is expanding, I wish to pass a request to ESRI REST API development team, to enlarge the limit a second time. I wish to enlarge the limit to at least 10,000 records, since nowadays regular Autocad DWG files that I convert to zipped shapefiles (by my FME cloud) contain near 10,000 reocrds. I am not an expert in all aspects of enlarging the limit, but as the limit has been multiplied by 4, from 1000 to 4000, I hope that nowadays it could be expanded even more. (If it could be enlarged to more then 10,000, it's even better, but I fear to request too much...)

I understand that there is no convenient way for widget in the AppBuilder to "bypass" the Convert REST API, so it makes sense to expand second time this limit, since it will expand the usage of ESRI Web AppBuilder and maybe more web application using ESRI REST API.

Can you please promote this request, or instruct me what can I do in order to promote it?

With appreciation,

Michael

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

RobertScheitlin__GISP — Mon, 24 Aug 2020 13:21:51 GMT

Michael,

The further increase would be like esri shooting themselves in the foot. Why will people need to buy desktop software is they can do everything online without purchasing any software to do it. Oh course this is just my opinion.

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Mon, 24 Aug 2020 14:09:46 GMT

Dear Robert,

I'll try to present my point of view -

Indeed I'm newbie to whole GIS area - I'm computer programmer for many many years. I've learned ArcGis Pro for a semester in University, in really good course and good instructor. Now I'm working for 3 months developing custom widgets, as you know.

What I understand, is that ArcGis Pro has its "area" of usage, and Web AppBuilder (Developer Edition) has another area.

Since many many companies who need GIS information, know nothing about GIS, so they use Web Applications (or Web AppBuilder (Developer Edition) applications, developed by companies who "know" GIS. And as you know, the widgets and custom-widgets are tailor made, in order to enable the users in companies who know nothing on GIS, to benefit GIS information, though they are simple users who know nothing.

More than that, the Web Applications and at least the Web AppBuilder (Developer Edition), supply specific gadgets and tools that the users need, and that could not be contained in ArcGIS Pro, since they are for specific needs of the users.

You can't expect that each user of GIS information will need to use ArcGIS Pro, which is a heavy tool.

Indeed, when a company's "main business" is GIS, this company "must" use ArcGIS Pro, but if it's business is something else, they wish to have an application which will help them to do specific tasks simple and easy, where the GIS processing is "under the hood" and they don't want to know about it.

For example, the company I'm working at, does many many aerial photography bu planes, and it's business is selling the option to view and inquire 3D Scenes, and our customers are simple people in companies that need many specific widgets to help them get the information they are interested in, from these 3D Scenes.

So they can't use ArcGIS Pro.

Our company pays for portal and online (and also ArcGIS Pro), and we use esri products, including the REST API, to give the users what they need.

I hope I made clear my opinion, that each product has different task and different "niche".

Indeed, esri products are excellent and I want to use them, and I am not an expert to evaluate what is better for esri.

I only presented my point of view.

I think esri should endeavour to the goal that users in thw whole world will prefer to use its products, and it will be if each product will "do its best" to supply users needs. ArcGIS Pro is a product, and REST API is a product. By answering users needs, both would thrive and prosperous.

I assume that enlarging REST API's Generate method maximum recods limit will answer current users needs, since the world is going forward and the needs grow constantly.

With appreciation,

Michael

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

HenryLindemann — Tue, 25 Aug 2020 05:33:56 GMT

Good Day Michael,

To answer your question why are you not getting all the records? This is because there is a limit on the records that you can query at one time, the reason behind this is simple because it is quite heavy to query all the records in a table at one time.

Then using the Rest API's is quite complex if possible try using the Python API for ArcGIS it is free

index | ArcGIS for Developers

To Answer your question you have to step trough the data E.G query record 1-4000 then 4001 to 8000

you can achieve this in REST as well

in python you can do this as follows

the below code will iterate trough a feature service and print out all the lines

from arcgis.features import FeatureLayer
from arcgis.gis import GIS

class BuildPacks:
    def __init__(self, list_of_items, pack_size=500😞
        self._list_of_items = list_of_items
        self._length = self._list_of_items.__len__()
        self._pack_size = pack_size
        self._packs = None

    def build_packs(self😞
        if self._length > self._pack_size:
            self._packs = int(self._length / self._pack_size)
            for pack in range(self._packs):
                idx_start = pack * self._pack_size
                idx_end = idx_start + self._pack_size
                yield self._list_of_items[idx_start:idx_end]
            idx_start = self._pack_size * self._packs
            remainder = self._length - idx_start
            if remainder > 0:
                yield self._list_of_items[idx_start:self._length]
        else:
            yield self._list_of_items

gis = GIS("https://www.arcgis.com", "username", "password")
con = FeatureLayer("https://serviceurl/server/rest/services/Hosted/myservice/FeatureServer/0", gis)
rows = con.query(return_ids_only=True) # gets all the ids in a table
row_ids = rows['objecIds']
packs = BuildPacks(row_ids) # builds packs of 500

# Quries 500 records at a time
for pack in packs.build_packs():

id_strings = [str(x) for x in pack]
records = con.query(where=f'ObjectId IN ({id_strings})') #e.g "ObjectId IN (01234,01235)" # this is standard SQL

for record in records.features:

print(record.__dict__) #prints the records

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Tue, 25 Aug 2020 11:43:21 GMT

Dear Henry Lindemann‌,

Thank you very much for showing ways to do it.

Since I'm not yet familiar with Python, I currently prefer the other way you noted, as you said "you can achieve this in REST as well", that is to activate generate on 4000 records chunks each, till I get all.

However, I don't know how to call generate the exact needed times and collect all results to create correctly the featureCollection that has to be added to the map.

I will appreciate it very much if you show/explain/point_to_examples how to do it., as I don't see any documentation how to do it.

Michael

P.S.

In other words - how to "upgrade" the "sandbox" of the code example Create a FeatureLayer from a shapefile (mentioned on my question, and my code is based on it) so that it will work also on zipped shapefiles with total records num greater than 4000.

Below is my current relevant code, though I assume the easier way is to modify the mentioned code example and test it.

var params = {
name: filename, // that is the ziped shpefile
targetSR: this.sceneView.spatialReference,
maxRecordCount: 1000000,
enforceInputFileSizeLimit: false,
enforceOutputJsonSizeLimit: false
};
params.generalize = true;
  params.maxAllowableOffset = 10;
  params.reducePrecision = true;
  params.numberOfDigitsAfterDecimal = 0;

  var myContent = {
      filetype: "shapefile",
      publishParameters: JSON.stringify(params),
      f: "json"
  };

  // use "generate" REST API to generate a feature collection from the zipped shapefile
  request(this.portalUrl + "/sharing/rest/content/features/generate", {
      query: myContent,
      body: this.uploadForm,
      responseType: "json"
  })
      .then(function (response) {
          var layerName =
              response.data.featureCollection.layers[0].layerDefinition.name;
          that._addShapefileToMap(response.data.featureCollection);
      })
      .catch(that._errorHandler);

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

HenryLindemann — Wed, 26 Aug 2020 11:03:40 GMT

Hi Michael,

Unfortunately I am not a Java Script programmer so I am not going to be of much help here, but let me try to give you some help any way.

you can go to Query a feature layer | ArcGIS API for JavaScript 4.16 to find the documentation for Java Script as well as examples

ArcGIS Online is basically just a newer version of the ArcGIS Enterprise system so this is how it looks when you query a layer.

What you want to research is the Result Offset which basically means from 0 how may records to skip and Result Record Count which means how many records to return E.G if the offset is 10 start at record 10 and return 5 records.

Info:

Query (Map Service/Layer)—ArcGIS REST API | ArcGIS for Developers

Example:

https://some_server.esri-southafrica.com/server/rest/services/Hosted/SubDist16T_Layer_WFL1/FeatureServer/0/query?where=1%3D1&resultOffset=10&resultRecordCount=1&f=json

Returns

{
"exceededTransferLimit": true,
"features": [{
"attributes": {
"municname": "City of Tshwane"
},
"geometry": {
"rings": [[...]]
}
}
],
"fields": [{
"name": "municname",
"type": "esriFieldTypeString",
"alias": "MUNICNAME",
"length": 100,
"defaultValue": "",
"modelName": "MUNICNAME"
}
],
"geometryType": "esriGeometryPolygon",
"spatialReference": {
"wkid": 102100,
"latestWkid": 3857
},
"objectIdFieldName": "objectid_12",
"hasZ": false,
"hasM": false
}

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Wed, 26 Aug 2020 16:07:37 GMT

Dear Robert Scheitlin, GISP‌,

Can you please direct me where to request for increase of generate records limit?
Do I understand correctly from Henry's answer "you have to step trough the data E.G query record 1-4000 then 4001 to 8000 etc. etc. you can achieve this in REST as well" that this is a possible solution? But I don't know where can I find documentation/code_examples how to do this. Is it possible at all, to create chunks of less than 400 each (maybe per layer in the zipped shapefile) and then combine them to get the full featureCollection? I suspect that it's possible only on an already existing shape but not on zipped shape...

Thanks,

Michael

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

RobertScheitlin__GISP — Wed, 26 Aug 2020 16:14:56 GMT

Michael,

Unfortunately you guys are talking about two different things here. You are wanting to use the generate method on the API and that has a limit that is not configurable by you the end user. What Henry is talking about is a query of a layer on AGOL or ArcGIS Server (nothing to do with using a zipped shapefile).

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

BillFox — Wed, 26 Aug 2020 16:28:08 GMT

this too:

https://community.esri.com/thread/139590

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Wed, 26 Aug 2020 18:50:20 GMT

Dear Robert Scheitlin, GISP‌,

Thank you very much for clarifying what I have already suspected, that it’s impossible to create from the zip “by chunks”.

Only my intention in “1. ... where to request” was not “how to do by API”, but - “How to ask from ESRI company, to increase that limit a little more”.

Can you please tell me how to send such request?

You said that in the past that limit was 1000 records, and then ESRI raised the limit to 4000, so I hope that they will handle such request positively.

With appreciation to your immediate responses and explanations,

Michael

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

RobertScheitlin__GISP — Wed, 26 Aug 2020 19:00:48 GMT

By submitting a "Idea" to ArcGIS Ideas‌ and or contacting esri tech support.

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Thu, 27 Aug 2020 08:01:00 GMT

All I know is that in web application, if I want to add zipped shapefile to map/scene by page javascript, I have to use ESRI REST API "generate", and ESRI has imposed an arbitrary artificial limit on records count. Once the limit has been that if the zipped shapefile contains more than 1000 records, it will NOT be handled, nad now this arbitrary artificial limit is 4000.

I think I will address ESRI either by submitting a "Idea" to ArcGIS Ideas or by contacting esri tech support,

requesting them to raise more this artificial limit.

Michael

Re: How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

MichaelLev — Thu, 10 Sep 2020 07:46:38 GMT

Robert Scheitlin, GISP‌, Bill Fox‌, Henry Lindemann‌, Joshua Bixby‌, Shay Lavi‌, Carlos Augusto Nantes de Castilho Rivilino‌, Majdoleen Awadallah‌, Harish Palaniappan‌, Ken Buja‌

I followed Robert's advise and submitted to "ArcGIS Ideas" - here,

a request to increase the arbitrary limit of 4,000 records for ArcGIS REST API - "Generate",

in order that web application based on esri REST API will be able to import zipped shapefiles with more than 4,000 records.

I will greatly appreciate it if you "upvote" it.

With appreciation,

Michael