topic Re: JSON.Load python file -- Add Features to Feature Service in ArcGIS API for Python Questions

JSON.Load python file -- Add Features to Feature Service

kapalczynski — Wed, 04 Oct 2023 13:47:17 GMT

I have a service that I am scrapping via Python that I am scrapping in Chucks do to the size of the dataset... about 12,000 records

During this scrape it writes the values to individual JSON files 1000 records at a time. I can open these JSON files and see there is an OBJECTID =1 and then OBJECTID=1000 and then the next one is OBJECTID=1001 and OBJECTID=2000 etc etc...

I can Copy that contents of that JSON file and put it in the REST Services and update from there with NO issues.. so I know the format is correct.

I read somewhere that there is a limit on characters of the JSON.Load which I am trying in the second def below. My files are about 830 KB in size...it appears not all of them are getting in... as I only get about 4300 of the 12k records in the Append Process...

Good news is that it does NOT Error out. But simply does not append all the records.

If there is a limitation on the JSON.Load how do I get around it... I tried to decrease the 1000 record count in my code to make more files but that does not seem to work... Im puzzled...

Thoughts?

def scrapeData(out_json_path, out_fc_path): try: service_props = ds.getServiceProperties(ds.service_url) max_record_count = 1000 layer_ids = ds.getLayerIds(ds.service_url) id_groups = list(ds.chunks(layer_ids, max_record_count)) for i, id_group in enumerate(id_groups): print(' group {} of {}'.format(i+1, len(id_groups))) layer_data = ds.getLayerDataByIds(ds.service_url, id_group) level = str(i) outjsonpath = outputVariable + level + ".json" if i==0: # If first iteration of id_groups layer_data_final = layer_data with open(outjsonpath, 'w') as out_json_file: json.dump(layer_data_final, out_json_file) else: layer_data_final2 = layer_data with open(outjsonpath, 'w') as out_json_file: json.dump(layer_data_final2, out_json_file)

I then loop through the JSON files and Append them to the Feature Service

def addDEQData(): for x in os.listdir(path): if x.startswith("output"): filetoImport = path + x f = open(filetoImport) data = json.load(f) featureAddingAdd = data['features'] gis_payloadAdd = { 'token': currenttoken, 'f': 'json', 'features': f'''{featureAddingAdd}''' } urlAdd = urlADDFeatures files=[] headers = {} response = requests.request("POST", urlAdd, headers=headers, data=gis_payloadAdd, files=files)

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Wed, 04 Oct 2023 13:49:49 GMT

Note I am doing it this way at REST Service due to issues with Servers and .sde connection files... I am not looking to hit the Oracle server and use sde connection files with a Truncate and Append... Its slower anyways...

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Wed, 04 Oct 2023 14:58:12 GMT

I think I found it... testing now.

EDIT..... Nope still only 4500 or so out of the 12k

Crazy thing is .... the JSON files have to correct number of records in them only 200 as I specified and I can see a ton more JSON files being created... but for some reason no more records are being added with the Append.... very puzzling

Re: JSON.Load python file -- Add Features to Feature Service

EarlMedina — Wed, 04 Oct 2023 15:17:52 GMT

Are you certain the problem is on the json.loads side? When you load from the files, does the count match what you expect? I suspect the problem is actually on the append side.

I've encountered similar issues in the past when attempting to add several thousand records all at once. My solution was to apply the edits in batches (of 500 or so).

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Wed, 04 Oct 2023 15:33:44 GMT

Yea Im very confused...

So when I scrape the Data and write to JSON files I get about 224 of them... This is set to CHUNKS of 100 records...

I have set the CHUNKS to about 100 records per JSON File.. As you can see below I am Loading each JSON file -- Grabbing the 'features' key and then using a POST to past this data.

As you can see from the 2nd image below which is showing the 224th JSON file you can see the OBJECTID of 24,401

If I manually copy the JSON and paste it into the AddFeatures of Rest -- 3rd Image below it adds the 100 records with no issues... I can even do this with 1000 records ... the only reason I went down to 100 was to test if this was an issue... the JSON being to large.

So right up until I Append the Records the JSON file has the 100 records + 224 JSON files which is the 24,000+ records I am trying to add... so EVERYTHING seems good to go...

BUT only a 1/6 or so of the records make it....

f = open(filetoImport) data = json.load(f) featureAddingAdd = data['features'] print(featureAddingAdd) gis_payloadAdd = { 'token': currenttoken, 'f': 'json', 'features': f'''{featureAddingAdd}''' } urlAdd = urlADDFeatures files=[] headers = {} response = requests.request("POST", urlAdd, headers=headers, data=gis_payloadAdd, files=files)

Re: JSON.Load python file -- Add Features to Feature Service

EarlMedina — Wed, 04 Oct 2023 16:41:57 GMT

Just an observation - I don't know if it's significant - but one difference between the two methods is the header. Maybe looking into the differences between the two requests could yield some useful information.

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Wed, 04 Oct 2023 18:45:33 GMT

OK so the JSON Files seem to be being created ok as they seem to only have 100 records in them like expected.

When I load the Json files I put 2 prints there and when I look it appears that there are also 100 features there in the Features KEY. So that looks good as well.

I stripped all the JSON files except one and then re ran and ONLY 19 records were added to the Feature Class. hmmmmm

SO its leaning towards the Requests.request it appears....something is not right there...

Crazy is that I can grab the Parameter values from "featureAddingAdd" from the code below Copy and Paste the Json into Rest Services Add Feature and it adds all 100 records no issues

But the request below that uses the same variable json values ONLY adds 19 records.... what gives.

print("Appending: " + x) f = open(filetoImport) data = json.load(f) featureAddingAdd = data['features'] print(featureAddingAdd) gis_payloadAdd = { 'token': currenttoken, 'f': 'json', 'features': f'''{featureAddingAdd}''' } print("-------") print(gis_payloadAdd) urlAdd = urlADDFeatures files=[] headers = {} response = requests.request("POST", urlAdd, headers=headers, data=gis_payloadAdd, files=files)

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Wed, 04 Oct 2023 19:01:08 GMT

I tried this way as well and still only get 19 of the 100 records from the JSON file being added... ugggg

resp = requests.post(urlAdd, gis_payloadAdd)

As I can return the value of the variable "featureAddingAdd" and it has all 100 records in it ... its not the json.load that seems to be the issue... it looks like the request post Append that is going on...

It only adds 19 records but I can copy the json from the variable above and go to Rest Services Add Feature and post the json code and it adds all 100 records...

ugggggggggggggggg

Re: JSON.Load python file -- Add Features to Feature Service

EarlMedina — Thu, 05 Oct 2023 01:12:10 GMT

Out of curiosity have you attempted the equivalent workflow using the ArcGIS API for Python? I realize you may have your reasons for using the REST API directly, but thought I would ask in case there's a possibility that might work better. The Python API will be a little different through the use of requests.Session (I'm pretty sure) and headers. I'm certain there is some configuration that is missing here that is giving you these odd results.

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Thu, 05 Oct 2023 14:55:06 GMT

I need to do the Append data via REST API where I do not have to have a .sde connection file... if there is another way to do this without an SDE connection file / DB permissions then that would be great...

So puzzled why only 20% of the features get through

If I build the JSON files where they have 1000 records in them it appends 200
If I build the JSON files where they have 100 records in them it appends 19

Does not seem to be the REST itself as I can copy the JSON into the REST webpage and it appends all 1000 or 100 with no issues..

It appears to be the request statement where for some reason it only appends about 20% of the records in the JSON file

I will paste all my code below in a few if anyone can see anything....

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Thu, 05 Oct 2023 15:00:51 GMT

import arcpy import arcgis from arcgis import GIS import requests import json import os targetURL = "https://apps.deq.virginia.gov/arcgis/rest/services/public/EDMA/MapServer/102" gis = GIS("https://xxxxxxx/portal", "username", "password") currenttoken = gis._con.token urlADDFeatures = r"https://xxxxxx/env/rest/services/DEV/VDEQ_Petro_Tank_Facility_DEV/FeatureServer/0/addFeatures" urlDeleteFeatures = r"https://xxxxxx/env/rest/services/DEV/VDEQ_Petro_Tank_Facility_DEV/FeatureServer/0/deleteFeatures" osPath = os.path.dirname(os.path.abspath(__file__)) outputVariable = osPath + "\\PetroleumTankFacilities\\output" path = osPath + "\\PetroleumTankFacilities\\" arcpy.env.overwriteOutput = True class DataScraper(): def __init__(self): # URL to map service you want to extract data from self.service_url = targetURL def getServiceProperties(self, url): URL = url PARAMS = {'f' : 'json'} r = requests.get(url = URL, params = PARAMS) service_props = r.json() return service_props def getLayerIds(self, url, query=None): URL = url + '/query' PARAMS = {'f':'json', 'returnIdsOnly': True, 'where' : '1=1'} if query: PARAMS['where'] = "ST = '{}'".format(query) r = requests.get(url = URL, params = PARAMS) data = r.json() return data['objectIds'] def getLayerDataByIds(self, url, ids): # ids parameter should be a list of object ids URL = url + '/query' field = 'OBJECTID' value = ', '.join([str(i) for i in ids]) PARAMS = {'f': 'json', 'where': '{} IN ({})'.format(field, value), 'returnIdsOnly': False, 'returnCountOnly': False,'returnGeometry': True, 'outFields': '*'} r = requests.post(url=URL, data=PARAMS) layer_data = r.json() return layer_data def chunks(self, lst, n): # Yield successive n-sized chunks from list for i in range(0, len(lst), n): yield lst[i:i + n] #def scrapeData(out_json_path, out_fc_path): def scrapeData(): try: service_props = ds.getServiceProperties(ds.service_url) max_record_count = service_props['maxRecordCount'] layer_ids = ds.getLayerIds(ds.service_url) # RETURN 1000 Records id_groups = list(ds.chunks(layer_ids, max_record_count)) # RETURN 100 Records #id_groups = list(ds.chunks(layer_ids, 100)) for i, id_group in enumerate(id_groups): print(' group {} of {}'.format(i+1, len(id_groups))) layer_data = ds.getLayerDataByIds(ds.service_url, id_group) level = str(i) outjsonpath = outputVariable + level + ".json" if i==0: # If first iteration of id_groups print(outjsonpath) print(outshapefilepath) layer_data_final = layer_data print('Writing JSON file...') with open(outjsonpath, 'w') as out_json_file: json.dump(layer_data_final, out_json_file) else: print(outjsonpath) print(outshapefilepath) layer_data_final2 = layer_data print('Writing JSON file...') with open(outjsonpath, 'w') as out_json_file: json.dump(layer_data_final2, out_json_file) except Exception: # Handle errors accordingly...this is generic tb = sys.exc_info()[2] tb_info = traceback.format_tb(tb)[0] pymsg = 'PYTHON ERRORS:\n\tTraceback info:\t{tb_info}\n\tError Info:\t{str(sys.exc_info()[1])}\n' msgs = 'ArcPy ERRORS:\t{arcpy.GetMessages(2)}\n' print(pymsg) print(msgs) def deleteDEQData(): print("Starting to Delete") url = urlDeleteFeatures whereclause='1=1' gis_payload = { 'token': currenttoken, 'f': 'json', 'where': f'''{whereclause}''' } response = requests.request("POST", url=url, data=gis_payload) print("Done Deleting") def addDEQData(): print("Starting to Append") for x in os.listdir(path): if x.startswith("output"): filetoImport = path + x print("Appending: " + x) f = open(filetoImport) data = json.load(f) featureAddingAdd = data['features'] print(featureAddingAdd) gis_payloadAdd = { 'token': currenttoken, 'f': 'json', 'features': f'''{featureAddingAdd}''' } print("-------") print(gis_payloadAdd) urlAdd = urlADDFeatures files=[] headers = {} #response = requests.request("POST", urlAdd, headers=headers, data=gis_payloadAdd, files=files) #response = requests.request("POST", urlAdd, data=gis_payloadAdd) #print(response) resp = requests.post(urlAdd, gis_payloadAdd) print(resp) if __name__ == '__main__': # Instantiate DataScraper class ds = DataScraper() # Run the function to get your data scrapeData() deleteDEQData() addDEQData() print("Done Appending")

Re: JSON.Load python file -- Add Features to Feature Service

EarlMedina — Thu, 05 Oct 2023 15:50:33 GMT

Okay, I think maybe I understand. You might be confusing ArcPy with the ArcGIS API for Python? For ArcPy you would most likely edit data directly using an sde connection file.

The ArcGIS API for Python, on the other hand, is just a wrapper for the REST API that makes it easier to use. I guess the only way you wouldn't be able to use it for this purpose is if you are working with a standalone ArcGIS Server. Otherwise, you would simply log into your portal, load the target Feature Layer, and use the FeatureLayer object's "edit_features" method to accomplish a similar result.

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Thu, 05 Oct 2023 16:58:24 GMT

Not sure what you are getting at... is my approach doable? I mean why does it add 19 of 100 records and 200 of 1000 records and just stop appending?

Re: JSON.Load python file -- Add Features to Feature Service

EarlMedina — Thu, 05 Oct 2023 17:16:21 GMT

From your sample, you're already using the api for login purposes. Why don't you give the edit_features method a shot with your json and see if that produces different results? I don't observe the same behavior using just the API, so I think at least part of the problem may be missing headers.

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Fri, 06 Oct 2023 00:25:18 GMT

Not sure how to do this... Why don't you give the edit_features method a shot with your json

Do you have an example... does this work without an SDE connection file?

What would be missing in the headers? Parameters? Not sure there.

Re: JSON.Load python file -- Add Features to Feature Service

EarlMedina — Fri, 06 Oct 2023 01:04:31 GMT

There's some examples here: Editing Features | ArcGIS API for Python.

No SDE connection required. Your code would change to something like this

import arcgis from arcgis.features import FeatureLayer gis = GIS("https://xxxxxxx/portal", "username", "password") fl_url = "https://xxxxxx/env/rest/services/DEV/VDEQ_Petro_Tank_Facility_DEV/FeatureServer/0" fl = FeatureLayer(fl_url, gis) ...other logic... fl.edit_features(adds=YOUR_ADDS, deletes=YOUR_DELETES, updates=YOUR_UPDATES)

Where the adds, deletes, updates are a FeatureSet object, or a list of features in json form (I think you basically have this already with your JSON).

Re: JSON.Load python file -- Add Features to Feature Service

kapalczynski — Fri, 06 Oct 2023 17:05:57 GMT

WORKED FANTASTIC... thank you very much @EarlMedina with steering me in the correct direction... and its FAST!!!

portal_item = gis.content.get('6e91cfxxxxx1fc7294493dddd') ports_layer = portal_item.layers[0] for x in os.listdir(path): if x.startswith("output"): filetoImport = path + x f = open(filetoImport) data = json.load(f) featureAddingAdd = data['features'] add_result = ports_layer.edit_features(adds = featureAddingAdd)

Re: JSON.Load python file -- Add Features to Feature Service

EarlMedina — Fri, 06 Oct 2023 17:03:55 GMT

Awesome glad to help streamline things!