How to add shapefile with more than 4000 records? since REST API Generate is limited to 4000

[MichaelLev]

I use the example code in Create a FeatureLayer from a shapefile to add zipped shapefile to my 3D scene.

But this code uses ArcGIS REST API Generate and as stated there - Generate's json output is limiyed to 4000 records.

But I need to insert shapefile with more than 4000 records...

I will greatly appreciate help how to do it.

[RobertScheitlin__GISP]

By submitting a "Idea" to ArcGIS Ideas‌ and or contacting esri tech support.

[HenryLindemann]

Good Day Michael,

To answer your question why are you not getting all the records? This is because there is a limit on the records that you can query at one time, the reason behind this is simple because it is quite heavy to query all the records in a table at one time.

Then using the Rest API's is quite complex if possible try using the Python API for ArcGIS it is free

index | ArcGIS for Developers 

To Answer your question you have to step trough the data E.G query record 1-4000 then 4001 to 8000

you can achieve this in REST as well

in python you can do this as follows 

the below code will iterate trough a feature service and print out all the lines

from arcgis.features import FeatureLayer
from arcgis.gis import GIS

class BuildPacks:
    def __init__(self, list_of_items, pack_size=500😞
        self._list_of_items = list_of_items
        self._length = self._list_of_items.__len__()
        self._pack_size = pack_size
        self._packs = None

    def build_packs(self😞
        if self._length > self._pack_size:
            self._packs = int(self._length / self._pack_size)
            for pack in range(self._packs):
                idx_start = pack * self._pack_size
                idx_end = idx_start + self._pack_size
                yield self._list_of_items[idx_start:idx_end]
            idx_start = self._pack_size * self._packs
            remainder = self._length - idx_start
            if remainder > 0:
                yield self._list_of_items[idx_start:self._length]
        else:
            yield self._list_of_items

gis = GIS("https://www.arcgis.com", "username", "password")
con = FeatureLayer("https://serviceurl/server/rest/services/Hosted/myservice/FeatureServer/0", gis)
rows = con.query(return_ids_only=True) # gets all the ids in a table
row_ids = rows['objecIds']
packs = BuildPacks(row_ids) # builds packs of 500

# Quries 500 records at a time
for pack in packs.build_packs():

   id_strings = [str(x) for x in pack]
   records = con.query(where=f'ObjectId IN ({id_strings})')  #e.g "ObjectId IN (01234,01235)" # this is standard SQL

   for record in records.features:

      print(record.__dict__) #prints the records

[MichaelLev]

Dear Henry Lindemann‌,

Thank you very much for showing ways to do it.

Since I'm not yet familiar with Python, I currently prefer the other way you noted, as you said "you can achieve this in REST as well", that is to activate generate on 4000 records chunks each, till I get all.

However, I don't know how to call generate the exact needed times and collect all results to create correctly the featureCollection that has to be added to the map.

I will appreciate it very much if you show/explain/point_to_examples how to do it., as I don't see any documentation how to do it.

Michael

P.S.

In other words - how to "upgrade" the "sandbox" of the code example Create a FeatureLayer from a shapefile (mentioned on my question, and my code is based on it) so that it will work also on zipped shapefiles with total records num greater than 4000.

Below is my current relevant code, though I assume the easier way is to modify the mentioned code example and test it.

    var params = {
        name: filename, // that is the ziped shpefile
        targetSR: this.sceneView.spatialReference,
        maxRecordCount: 1000000,
        enforceInputFileSizeLimit: false,
        enforceOutputJsonSizeLimit: false
    };
    params.generalize = true;
    params.maxAllowableOffset = 10;
    params.reducePrecision = true;
    params.numberOfDigitsAfterDecimal = 0;

    var myContent = {
        filetype: "shapefile",
        publishParameters: JSON.stringify(params),
        f: "json"
    };

    // use "generate" REST API to generate a feature collection from the zipped shapefile
    request(this.portalUrl + "/sharing/rest/content/features/generate", {
        query: myContent,
        body: this.uploadForm,
        responseType: "json"
    })
        .then(function (response) {
            var layerName =
                response.data.featureCollection.layers[0].layerDefinition.name;
            that._addShapefileToMap(response.data.featureCollection);
        })
        .catch(that._errorHandler);

[HenryLindemann]

Hi Michael,

Unfortunately I am not a Java Script programmer so I am not going to be of much help here,  but let me try to give you some help any way.

you can go to Query a feature layer | ArcGIS API for JavaScript 4.16  to find the documentation for Java Script as well as examples

ArcGIS Online is basically just a newer version of the ArcGIS Enterprise system  so this is how it looks when you query a layer.

What you want to research is the Result Offset which basically means from 0 how may records to skip and Result Record Count which means how many records to return E.G if the offset is 10  start at record 10 and return 5 records.

Info:

Query (Map Service/Layer)—ArcGIS REST API | ArcGIS for Developers 

Example:

https://some_server.esri-southafrica.com/server/rest/services/Hosted/SubDist16T_Layer_WFL1/FeatureSe...

Returns

{
"exceededTransferLimit": true,
"features": [{
"attributes": {
"municname": "City of Tshwane"
},
"geometry": {
"rings": [[...]]
}
}
],
"fields": [{
"name": "municname",
"type": "esriFieldTypeString",
"alias": "MUNICNAME",
"length": 100,
"defaultValue": "",
"modelName": "MUNICNAME"
}
],
"geometryType": "esriGeometryPolygon",
"spatialReference": {
"wkid": 102100,
"latestWkid": 3857
},
"objectIdFieldName": "objectid_12",
"hasZ": false,
"hasM": false
}

[BillFox]

this too:

https://community.esri.com/thread/139590 

[MichaelLev]

All I know is that in web application, if I want to add zipped shapefile to map/scene by page javascript, I have to use ESRI REST API "generate", and ESRI has imposed an arbitrary artificial limit on records count. Once the limit has been that if the zipped shapefile contains more than 1000 records, it will NOT be handled, nad now this arbitrary artificial limit is 4000.

I think I will address ESRI either by submitting a "Idea" to ArcGIS Ideas or by contacting esri tech support,

requesting them to raise more this artificial limit.

Michael

[MichaelLev]

Robert Scheitlin, GISP‌, Bill Fox‌, Henry Lindemann‌, Joshua Bixby‌, Shay Lavi‌, Carlos Augusto Nantes de Castilho Rivilino‌, Majdoleen Awadallah‌, Harish Palaniappan‌, Ken Buja‌

I followed Robert's advise and submitted to "ArcGIS Ideas" - here,

a request to increase the arbitrary limit of 4,000 records for ArcGIS REST API - "Generate",

in order that web application based on esri REST API will be able to import zipped shapefiles with more than 4,000 records.

I will greatly appreciate it if you "upvote" it.

With appreciation,

Michael