topic Re: Creating hosted FeatureLayer over 10mb - REST or PYTHON API? in ArcGIS API for Python Questions

Creating hosted FeatureLayer over 10mb - REST or PYTHON API?

PaulHallett1305 — Tue, 23 Mar 2021 08:16:08 GMT

Hi,

I have a fairly large layer (about 2gb) that I am propagating through Databricks using ArcGIS API for Python.

There seems to be a limit on these to 10mb, but also, that you can also do this via the REST API as parts. I can't find anything anywhere that documents this.

Does anyone have any suggestions how to do this?

Much appreciated.

Re: Creating hosted FeatureLayer over 10mb - REST or PYTHON API?

simoxu — Wed, 24 Mar 2021 05:11:34 GMT

I won't publish 2GB data in one go from the client (ArcGIS Pro / ArcMap).

Instead, publish an empty service first, then populate the service. there are two options:

1. Use edit_features function to add data in batches (batch size depends on the data, use smaller size if complicated polygon)

# get the features you are going to add features = fset_add.features # divide the features into batches, 100 for example. The batch size depends # on the size of the record (paritcularly the geometry) num = int(len(features)/100) remain = len(features)%100 print(f'num:{num} remain:{remain}') for i in range(num): pos_start = i*100 part = features[pos_start:pos_start+100] result = feature_layer.edit_features(adds=part) print(f"start at:{pos_start}, and the length of the list is:{len(part)}, {len(result['addResults'])} records added!") if (remain>0): pos_start = num*100 part = features[num*100:] result = feature_layer.edit_features(adds=part) print(f"start at:{pos_start}, and the length of the list is:{len(part)}, {len(result['addResults'])} records added!")

2. You can try if you can create a FGDB and upload it to your portal, and use append the service. I had success to append near 1 million records (points)

itemId = "xxxxxb4cxxxx2eb7xxxxxxxx" FL_Property_pnt.append(item_id=itemId,upload_format="filegdb",source_table_name="Property_Pnt")

Re: Creating hosted FeatureLayer over 10mb - REST or PYTHON API?

PaulHallett1305 — Wed, 24 Mar 2021 08:11:15 GMT

Thanks for this; I did think about creating an empty map service then adding data; the polygons themselves are quite simple, it's just that there are lots of them.

I am working out of databricks and can't access ArcPy from it, so I can't create a FileGDB unfortunately.

The other option was going to try, was to create a SEDF, create a Shape file from that, zip it up, then publish a featureclass from that.

Thanks for the ideas, will have a go at this today.

Cheers

Paul