Creating hosted FeatureLayer over 10mb - REST or PYTHON API?

Question

Hi,

I have a fairly large layer (about 2gb) that I am propagating through Databricks using ArcGIS API for Python.

There seems to be a limit on these to 10mb, but also, that you can also do this via the REST API as parts.  I can't find anything anywhere that documents this.

Does anyone have any suggestions how to do this?

Much appreciated.

PaulHallett1305 · Answer

Thanks for this; I did think about creating an empty map service then adding data; the polygons themselves are quite simple, it's just that there are lots of them.

I am working out of databricks and can't access ArcPy from it, so I can't create a FileGDB unfortunately.

The other option was going to try, was to create a SEDF, create a Shape file from that, zip it up, then publish a featureclass from that.

Thanks for the ideas, will have a go at this today.

Cheers

Paul

simoxu · Answer

I won't publish 2GB data in one go from the client (ArcGIS Pro / ArcMap).

Instead, publish an empty service first, then populate the service. there are two options:

1. Use edit_features function to add data in batches (batch size depends on the data, use smaller size if complicated polygon)

# get the features you are going to add
features = fset_add.features

# divide the features into batches, 100 for example. The batch size depends 
# on the size of the record (paritcularly the geometry)
num = int(len(features)/100)
remain = len(features)%100
print(f'num:{num} remain:{remain}')

for i in range(num):
    pos_start = i*100
    part = features[pos_start:pos_start+100]
    result = feature_layer.edit_features(adds=part)
    print(f"start at:{pos_start}, and the length of the list is:{len(part)}, {len(result['addResults'])} records added!")
if (remain>0):
    pos_start = num*100
    part = features[num*100:]
    result = feature_layer.edit_features(adds=part)
    print(f"start at:{pos_start}, and the length of the list is:{len(part)}, {len(result['addResults'])} records added!")

2. You can try if you can create a FGDB and upload it to your portal, and use append the service. I had success to append near 1 million records (points)

itemId = "xxxxxb4cxxxx2eb7xxxxxxxx"
FL_Property_pnt.append(item_id=itemId,upload_format="filegdb",source_table_name="Property_Pnt")