JSON.Load python file -- Add Features to Feature Service
I have a service that I am scrapping via Python that I am scrapping in Chucks do to the size of the dataset... about 12,000 records
During this scrape it writes the values to individual JSON files 1000 records at a time. I can open these JSON files and see there is an OBJECTID =1 and then OBJECTID=1000 and then the next one is OBJECTID=1001 and OBJECTID=2000 etc etc...
I can Copy that contents of that JSON file and put it in the REST Services and update from there with NO issues.. so I know the format is correct.
I read somewhere that there is a limit on characters of the JSON.Load which I am trying in the second def below. My files are about 830 KB in size...it appears not all of them are getting in... as I only get about 4300 of the 12k records in the Append Process...
Good news is that it does NOT Error out. But simply does not append all the records.
If there is a limitation on the JSON.Load how do I get around it... I tried to decrease the 1000 record count in my code to make more files but that does not seem to work... Im puzzled...
Thoughts?
def scrapeData(out_json_path, out_fc_path):
try:
service_props = ds.getServiceProperties(ds.service_url)
max_record_count = 1000
layer_ids = ds.getLayerIds(ds.service_url)
id_groups = list(ds.chunks(layer_ids, max_record_count))
for i, id_group in enumerate(id_groups):
print(' group {} of {}'.format(i+1, len(id_groups)))
layer_data = ds.getLayerDataByIds(ds.service_url, id_group)
level = str(i)
outjsonpath = outputVariable + level + ".json"
if i==0: # If first iteration of id_groups
layer_data_final = layer_data
with open(outjsonpath, 'w') as out_json_file:
json.dump(layer_data_final, out_json_file)
else:
layer_data_final2 = layer_data
with open(outjsonpath, 'w') as out_json_file:
json.dump(layer_data_final2, out_json_file)
I then loop through the JSON files and Append them to the Feature Service
def addDEQData():
for x in os.listdir(path):
if x.startswith("output"):
filetoImport = path + x
f = open(filetoImport)
data = json.load(f)
featureAddingAdd = data['features']
gis_payloadAdd = {
'token': currenttoken,
'f': 'json',
'features': f'''{featureAddingAdd}'''
}
urlAdd = urlADDFeatures
files=[]
headers = {}
response = requests.request("POST", urlAdd, headers=headers, data=gis_payloadAdd, files=files)
Accepted Solutions
Yea Im very confused...
So when I scrape the Data and write to JSON files I get about 224 of them... This is set to CHUNKS of 100 records...
I have set the CHUNKS to about 100 records per JSON File.. As you can see below I am Loading each JSON file -- Grabbing the 'features' key and then using a POST to past this data.
As you can see from the 2nd image below which is showing the 224th JSON file you can see the OBJECTID of 24,401
If I manually copy the JSON and paste it into the AddFeatures of Rest -- 3rd Image below it adds the 100 records with no issues... I can even do this with 1000 records ... the only reason I went down to 100 was to test if this was an issue... the JSON being to large.
So right up until I Append the Records the JSON file has the 100 records + 224 JSON files which is the 24,000+ records I am trying to add... so EVERYTHING seems good to go...
BUT only a 1/6 or so of the records make it....
f = open(filetoImport)
data = json.load(f)
featureAddingAdd = data['features']
print(featureAddingAdd)
gis_payloadAdd = {
'token': currenttoken,
'f': 'json',
'features': f'''{featureAddingAdd}'''
}
urlAdd = urlADDFeatures
files=[]
headers = {}
response = requests.request("POST", urlAdd, headers=headers, data=gis_payloadAdd, files=files)
