Automating upload of json to Portal

[ellipsol]

I need to automate uploading a ~35,000 record json file to portal. The method I'm using shows the points and record count on the portal, but the attribute table is empty. I'm using the

fl.edit_features(adds="")

 method. I started trying to chunk the json into smaller parts b/c I thought that would be useful, but it's the same result, an empty table.  In the code below I'm using a chunk size of 100 and limiting to 500 rows for quicker testing. Does anyone know what I can try to get the table to display its data?

    localJSON = "E:/batch/AddressRematch/data.json"
    with open(localJSON, "r") as jsonfile:
        all_data = jsonfile.read()
    
    # load json data to a dict
    entire_featureset_dict = json.loads(all_data)
    
    # convert to a list
    features_list = entire_featureset_dict["features"]
    base_featureset_dict = entire_featureset_dict.copy()
   
    # create a chunk
    start = 0
    chunk_size = 100
    end = start + chunk_size

    #breakpoint()
    while start < 500: #len(features_list):
        # grab a chunk
        # copy a slice of the feature set
        base_featureset_dict["features"] = features_list[start:end]
        chunk_fs = arcgis.features.FeatureSet.from_dict(base_featureset_dict)
        #print(chunk_fs)
        
        # upload it to the layer
        fl.edit_features(adds=chunk_fs)
        #print(arcpy.GetMessages(0))
        start += chunk_size
        end += chunk_size

[jcarlson]

Can you print off a subset of the features list? There really shouldn't be a problem with doing this this way, but the FeatureSet.from_dict may be expecting a specific format. Or try printing chunk_fs, does that register as a valid FeatureSet object?

Another approach might be to read the file into a pandas dataframe, then use df.spatial.to_featureset() to create your FeatureSet object. But I don't think it's necessary, we have scripts that build and apply edits using JSON and the requests module.

[ellipsol]

Ok, I'll try and work with a df. Printing the chunk_fs returns valid records, but that might not be what edit_features() is looking for, you're right.

[jcarlson]

The fact that the attribute table is empty suggests the issue is how the fields are being given. It should follow the REST API spec:

[
  {
    "attributes": {
      "field1": "value",
      "field2": 12
    },
    "geometry": {
      …
    }
  },
  {
    // and so on
  }
] 

Make sure your non-spatial attributes end up in their own dict, rather than being at the same level as the geometry key.

[ellipsol · Accepted solution]

I figured something out. Thanks Josh, that was a good idea looking at the json format.

    # prepare json for portal
    with open(localJSON) as f:
        data = json.load(f)
    keys = list(data.keys())

    if 'features' in keys:
        needed_data = data['features']
        with open("E:/batch/AddressRematch/code/new.json", "w") as f:
            json.dump(needed_data, f)

    with open("E:/batch/AddressRematch/code/new.json", "r") as jsonfile:
        all_data = jsonfile.read()
    
    entire_featureset_list = json.loads(all_data)