Hi,
I have a fairly large layer (about 2gb) that I am propagating through Databricks using ArcGIS API for Python.
There seems to be a limit on these to 10mb, but also, that you can also do this via the REST API as parts. I can't find anything anywhere that documents this.
Does anyone have any suggestions how to do this?
Much appreciated.
I won't publish 2GB data in one go from the client (ArcGIS Pro / ArcMap).
Instead, publish an empty service first, then populate the service. there are two options:
1. Use edit_features function to add data in batches (batch size depends on the data, use smaller size if complicated polygon)
# get the features you are going to add
features = fset_add.features
# divide the features into batches, 100 for example. The batch size depends
# on the size of the record (paritcularly the geometry)
num = int(len(features)/100)
remain = len(features)%100
print(f'num:{num} remain:{remain}')
for i in range(num):
pos_start = i*100
part = features[pos_start:pos_start+100]
result = feature_layer.edit_features(adds=part)
print(f"start at:{pos_start}, and the length of the list is:{len(part)}, {len(result['addResults'])} records added!")
if (remain>0):
pos_start = num*100
part = features[num*100:]
result = feature_layer.edit_features(adds=part)
print(f"start at:{pos_start}, and the length of the list is:{len(part)}, {len(result['addResults'])} records added!")
2. You can try if you can create a FGDB and upload it to your portal, and use append the service. I had success to append near 1 million records (points)
itemId = "xxxxxb4cxxxx2eb7xxxxxxxx"
FL_Property_pnt.append(item_id=itemId,upload_format="filegdb",source_table_name="Property_Pnt")
Thanks for this; I did think about creating an empty map service then adding data; the polygons themselves are quite simple, it's just that there are lots of them.
I am working out of databricks and can't access ArcPy from it, so I can't create a FileGDB unfortunately.
The other option was going to try, was to create a SEDF, create a Shape file from that, zip it up, then publish a featureclass from that.
Thanks for the ideas, will have a go at this today.
Cheers
Paul