topic Add Features at REST in Chunks in ArcGIS API for Python Questions

Add Features at REST in Chunks

kapalczynski — Sun, 21 Jan 2024 00:50:37 GMT

I am trying to add Features to a Feature Service via REST... previous to this I was trying to delete them and have this post... Thanks @JRhodes
https://community.esri.com/t5/arcgis-api-for-python-questions/grab-sections-of-service-to-process/m-p/1372556#M9552
NOTE: I am writing from my AGOL Enterprise Hosted Feature Layer to an SDE Oracle Feature Class via the Rest Endpoint Feature Service...

Any ideas why Im erroring out?

Note im reading a dictionary for the AGOL address and then the second key is the REST feature service I want to write to....(sourceFC and targetFC)

Any help would be greatly appreciated.

layers_to_append2 = { "https://services.arcgis.com/p5x3l7/arcgis/rest/services/Inventory/FeatureServer/8/query?sublayer=8" : "https://xxx.gov/env/rest/services/DEV/DEV/FeatureServer/8" } def appendLayers2(): for layer in layers_to_append2.keys(): sourceFC = layer targetFC = layers_to_append2[layer] where_clause = "1=1" batch_size = 2000 arcpy.env.preserveGlobalIds = True query_url = f"{sourceFC}&where={where_clause}&returnIdsOnly=true&f=json&token={agoltoken}" query_result = requests.get(query_url).json() object_ids = query_result.get("objectIds", []) print(object_ids) if not object_ids: break # No more features to delete for i in range(0, len(object_ids), batch_size): batch_ids = object_ids[i:i + batch_size] where_clause_with_ids = f"{where_clause} AND OBJECTID >= {min(batch_ids)} AND OBJECTID <= {max(batch_ids)}" print(where_clause_with_ids) add_params = { 'where': where_clause_with_ids, 'f': 'json', 'rollbackOnFailure': True, 'token': portaltoken } print(add_params) print("") response = requests.post(f"{targetFC}/addFeatures", data=add_params) add_result = response.json() if 'addResults' in add_result: added_features = add_result['addResults'] for feature in added_features: if 'success' in feature and feature['success']: print(f"Add feature with objectId {feature['objectId']}") else: print(f"Failed to add feature with objectId {feature['objectId']}") else: print(f"Error adding features. Response: {add_result}")

For some reason it wont write... Do I need to add a "Features" json string to the 'add_params' data load...Looks like the query of the AGOL Feature Layer works as there are only 5 records..BUT the post to add them to the Rest Endpoint errors????

ERROR:

Starting to append: https://services.arcgis.com/xxx/arcgis/rest/services/Inventory/FeatureServer/8
https://services.arcgis.com/xxx/arcgis/rest/services/Inventory/FeatureServer/8/query?where=1=1&sublayer=0&returnIdsOnly=true&f=json&token=wErLJwbDhStxHwstlxyDjKkRSgze-g91Yh9R9PPgzv9GrZfuWuxxxZlKCedGYufGzHZEkl6CAJ5X9Q..
[1, 2, 3, 4, 5]
1=1 AND OBJECTID >= 1 AND OBJECTID <= 5
{'where': '1=1 AND OBJECTID >= 1 AND OBJECTID <= 5', 'f': 'json', 'rollbackOnFailure': True, 'token': 'K0rj4angx4nn3xF0-W0ppQVWdj2rHb27FfPzMKOr5lYCMhslxQ'}

Error adding features. Response: {'error': {'code': 500, 'message': 'Unable to complete operation.', 'details': ['Parser error: Some parameters could not be recognized.']}}

Re: Add Features at REST in Chunks

JRhodes — Sun, 21 Jan 2024 22:00:42 GMT

Hi, I see two issues right away:

There are no features in the addFeatures payload. You must pass an array of features to be added.
addFeatures does not accept a 'where' parameter.

You need to query your hosted (ArcGIS Online) feature layer, then pass those features to the 'features' parameter of the addFeatures payload. If you're wanting to do this in chunks, the chunks would be best handled on the query; in other words, query 2000 records and add them, then query the next 2000 and add those, etc.

If you have attachments, those will need to be handled separately. But this should move your features over in 2000-feature chunks:

import requests import json source_service_url = "" destination_service_url = "" batch_size = 2000 offset = 0 while True: query_url = f"{source_service_url}/query" query_params = { "f": "json", "where": "1=1", # Change if you want to specify which features to query from source "outFields": "*", "resultOffset": offset, "resultRecordCount": batch_size, "token": agoltoken } query_response = requests.get(query_url, params=query_params) features = query_response.json().get("features") if not features: break # no more features to process add_url = f"{destination_service_url}/addFeatures" add_params = { "f": "json", "features": json.dumps(features), "token": portaltoken } add_response = requests.post(add_url, data=add_params) offset += batch_size

Re: Add Features at REST in Chunks

kapalczynski — Tue, 23 Jan 2024 14:20:31 GMT

OK great thanks... Yea I have attachments ... I think that is what is making this all so difficult and a slow process... I assume is a queryAttachents on the URL and then something similar to the above addFeatures? or addAttachements?

Re: Add Features at REST in Chunks

kapalczynski — Tue, 23 Jan 2024 16:36:58 GMT

OK I ran this with some mods to fit my code and data... Basically just changing the original dataset and then target dataset...I put a print on the resulting features from the initial query of the in data... I got something weird...

There are only like 5 records in the original data... not sure why it is looping two times...thought it was supposed to get the first 2000 records.

And the second one has {'f': 'json', 'features': '[{"attributes": at the begining.. and the token at the end...

As such nothing was added to the FC.

[{'attributes': {'OBJECTID': 1, 'rel_globalid': '7fc73b9d-b347-4464-92a4-dfa8217c30f5', 'globalid': '517988eb-06b9-4c80-b3d6-771d1aa096a7', 'created_user': 'emial', 'created_date': 1635644468483, 'last_edited_user': 'emial', 'last_edited_date': 1635644468483, 'SWMID': '800311', 'DISTRICT': 'Salem', 'GENBMP': 'Infiltration', 'SPECBMP': 'Permeable Pavement 1 CH - IIB', 'INSPNAME': 'Patrick.Wood@VDOT.Virginia.gov', 'INSPTYPE': 'Annual Inspection', 'INSPDATE': 1582045200000, 'NEXTINSPDATE': 1613586948000, 'ACGA_LQ_BMPAC': 'Yes', 'ACGA_EASENEED': None, 'ACGA_EASEEXIST': None, 'ACGA_MINOREROS': None, 'ACGA_ACCESBLKVEG': None, 'ACGA_ACCESDMGBLK': None, 'ACGA_RMVNEVRCONSTR': None, 'ACGC_NOTES': None, 'ACSN_LQ_BMPIDSNCON': 'No', 'ACSN_BMPIDSNREP': 'Yes', 'ACSN_NOTES': 'No ID SIGN. ', 'ACMCIA_LQ_MCIGCON': 'N/A', 'ACMCIA_MCIGMNDMG': None, 'ACMCIA_MCIGNOTSEAT': None, 'ACMCIA_MCIGBRKMS': None, 'ACMCIA_NOTES': None, 'ACGF_LQ_GFABST': 'Yes', 'ACGF_GFOBSACPT': None, 'ACGF_GFOBSNACPT': None, 'ACGF_NOTES': None, 'CDASCTD_LQ_CDAACPT': 'Yes', 'CDASCTD_CDAVGCVLES75': None, 'CDASCTD_MNDB': None, 'CDASCTD_EXDB': None, 'CDASCTD_NOTES': None, 'CDAIDDE_LQ_NOBSIDDE': 'Yes', 'CDAIDDE_OBSFLWON2PRPT': None, 'CDAIDDE_OBSFLWFRMOTH': None, 'CDAIDDE_NOTES': None, 'CDA_LQ_OUTFLPODPRSNT': None, 'IAICESTD_LQ_BMPCON': 'Yes', 'IAICESTD_BMPSFMNER': None, 'IAICESTD_BMPEXER': None, 'IAICESTD_CHNLMNER': None, 'IAICESTD_CHNLSIGER': None, 'IAICESTD_CHNLEXER': None, 'IAICESTD_CHDMEXSED': None, 'IAICESTD_NOTES': None, 'IAICIASC_LQ_VEGCVGR75': 'Yes', 'IAICIASC_VEGCVREQLES10SY': None, 'IAICIASC_VEGCVREQMOR10SY': None, 'IAICIASC_LQ_LADPLAGR75': 'Yes', 'IAICIASC_LADPLAGR50': None, 'IAICIASC_LADPLALES50': None, 'IAICIASC_NOTES': None, 'IAICIBU_LQ_BMPSFCHNLCON': 'Yes', 'IAICIBU_CHNLMNDMG': None, 'IAICIBU_CHNLEXDMG': None, 'IAICIBU_LQ_PIPECON': 'Yes', 'IAICIBU_PIPEJTMNSEP': None, 'IAICIBU_PIPEJTEXSEP': None, 'IAICIBU_PIPEJTSIGSEP': None, 'IAICIBU_PIPEMJDMG': None, 'IAICIBU_EWMNDMG': None, 'IAICIBU_EWERLES25': None, 'IAICIBU_EWERGR25': None, 'IAICIBU_SIDEMNSCERLES1': None, 'IAICIBU_SIDESIGSCERGR1': None, 'IAICIBU_INFLWSCERLES1': None, 'IAICIBU_INFLWSIGSCERGR1': None, 'IAICIBU_NOTES': None, 'FPPTPL_LQ_PTDVPRES': 'N/A', 'FPPTPL_NOTPRES': None, 'FPPTPL_NOTES': None, 'FPESTD_LQ_DVNOESTD': 'Yes', 'FPESTD_MNESTD': None, 'FPESTD_EXESDT': None, 'FPESTD_STNDISPLC': None, 'FPESTD_LQ_AREANOER': 'Yes', 'FPESTD_AREAMNER': None, 'FPESTD_AREAEXER': None, 'FPESTD_NOTES': None, 'IPTAPPMS_LQ_PVMSFNOMS': 'Yes', 'IPTAPPMS_PVMSFMSOBS': None, 'IPTAPPMS_NOTES': None, 'IPTAPPSS_LQ_PVMSFNOSED': 'Yes', 'IPTAPPSS_PVMSFMNSED': None, 'IPTAPPSS_PVMSFSIGSED': None, 'IPTAPPSS_PVMSFSINK': None, 'IPTAPPSS_NOTES': None, 'IPTAPPDVR_LQ_PVMNOVGRTDMG': 'Yes', 'IPTAPPDVR_UNSPECVG': None, 'IPTAPPDVR_PVMDEADVG': None, 'IPTAPPDVR_PVMDMGRT': None, 'IPTAPPDVR_PVMEXPOSOIL': None, 'IPTAPPDVR_NOTES': None, 'IPTAPPSC_LQ_PMPVMNOCRAK': 'Yes', 'IPTAPPSC_PVMSFMNCRAK': None, 'IPTAPPSC_PVMSFEXCRAK': None, 'IPTAPPSC_PVMSFSEAL': None, 'IPTAPPSC_NOTES': None, 'IPTAFTES_LQ_CLOWCON': 'Yes', 'IPTAFTES_CLOWCAPMSDMG': None, 'IPTAFTES_CLOWPIPEDMG': None, 'IPTAFTES_CLOWOBSTR': None, 'IPTAPPSW_LQ_PMPVMSFNOSW': 'Yes', 'IPTAPPSW_PVMSFSWOBS': None, 'IPTAPPSW_NOTES': 'Monthly cleaning and testing scheduled for night of February 18th. ', 'IPTAPPDS_LQ_DRNGSYSCON': 'Yes', 'IPTAPPDS_OBSWELLDMG': None, 'IPTAPPDS_WELLWATROBS': None, 'IPTAPPDS_DRNGNOTFUNC': None, 'IPTAPPDS_DRNGSEDOBS': None, 'IPTAPPDS_DRNGCORSN': None, 'IPTAPPDS_DRNGDMG': None, 'IPTAPPDS_NOTES': None, 'OSDCRC_LQ_RECVCHNLCON': 'Yes', 'OSDCRC_RECVCHNLERNODMG': None, 'OSDCRC_RECVCHNLERDMG': None, 'OSDCRC_RECVSEWERER': None, 'OSDCRC_RECVSEWERBLK': None, 'OSDCRC_NOTES': None, 'INSP_OVERALL_RATING': 2, 'EE_ER': 'N/A', 'MAINT_COMPLETION_DATE': None, 'FPPTPL_DVCONFMPLAN': None, 'SPEC_BMP_NOTES': None, 'FPPTPL_COLLAPSED': None, 'UNAUTH_DISCH': None}}, {'attributes': {'OBJECTID': 2, 'rel_globalid': '7fc73b9d-b347-4464-92a4-dfa8217c30f5', 'globalid': 'da644253-ca67-4269-8164-3ace092ce9be', 'created_user': 'emial', 'created_date': 1635644468483, 'last_edited_user': SNIP ======================================== 'IPTAPPDS_NOTES': None, 'OSDCRC_LQ_RECVCHNLCON': 'Yes', 'OSDCRC_RECVCHNLERNODMG': None, 'OSDCRC_RECVCHNLERDMG': None, 'OSDCRC_RECVSEWERER': None, 'OSDCRC_RECVSEWERBLK': None, 'OSDCRC_NOTES': 'Storm sewer system. ', 'INSP_OVERALL_RATING': 1, 'EE_ER': None, 'MAINT_COMPLETION_DATE': None, 'FPPTPL_DVCONFMPLAN': None, 'SPEC_BMP_NOTES': None, 'FPPTPL_COLLAPSED': None, 'UNAUTH_DISCH': None}}]

On the second pass I got this for the features JSON:

{'f': 'json', 'features': '[{"attributes": {"OBJECTID": 1, "rel_globalid": "7fc73b9d-b347-4464-92a4-dfa8217c30f5", "globalid": "517988eb-06b9-4c80-b3d6-771d1aa096a7", "created_user": "email", "created_date": 1635644468483, "last_edited_user": "emial", "last_edited_date": 1635644468483, "SWMID": "800311", "DISTRICT": "Salem", "GENBMP": "Infiltration", "SPECBMP": "Permeable Pavement 1 CH - IIB", "INSPNAME": "Patrick.Wood@VDOT.Virginia.gov", "INSPTYPE": "Annual Inspection", "INSPDATE": 1582045200000, "NEXTINSPDATE": 1613586948000, "ACGA_LQ_BMPAC": "Yes", "ACGA_EASENEED": null, "ACGA_EASEEXIST": null, "ACGA_MINOREROS": null, "ACGA_ACCESBLKVEG": null, "ACGA_ACCESDMGBLK": null, "ACGA_RMVNEVRCONSTR": null, "ACGC_NOTES": null, "ACSN_LQ_BMPIDSNCON": "No", "ACSN_BMPIDSNREP": "Yes", "ACSN_NOTES": "No ID SIGN. ", "ACMCIA_LQ_MCIGCON": "N/A", "ACMCIA_MCIGMNDMG": null, "ACMCIA_MCIGNOTSEAT": null, "ACMCIA_MCIGBRKMS": null, "ACMCIA_NOTES": null, "ACGF_LQ_GFABST": "Yes", "ACGF_GFOBSACPT": null, "ACGF_GFOBSNACPT": null, "ACGF_NOTES": null, "CDASCTD_LQ_CDAACPT": "Yes", "CDASCTD_CDAVGCVLES75": null, "CDASCTD_MNDB": null, "CDASCTD_EXDB": null, "CDASCTD_NOTES": null, "CDAIDDE_LQ_NOBSIDDE": "Yes", "CDAIDDE_OBSFLWON2PRPT": null, "CDAIDDE_OBSFLWFRMOTH": null, "CDAIDDE_NOTES": null, "CDA_LQ_OUTFLPODPRSNT": null, "IAICESTD_LQ_BMPCON": "Yes", "IAICESTD_BMPSFMNER": null, "IAICESTD_BMPEXER": null, "IAICESTD_CHNLMNER": null, "IAICESTD_CHNLSIGER": null, "IAICESTD_CHNLEXER": null, "IAICESTD_CHDMEXSED": null, "IAICESTD_NOTES": null, "IAICIASC_LQ_VEGCVGR75": "Yes", "IAICIASC_VEGCVREQLES10SY": null, "IAICIASC_VEGCVREQMOR10SY": null, "IAICIASC_LQ_LADPLAGR75": "Yes", "IAICIASC_LADPLAGR50": null, "IAICIASC_LADPLALES50": null, "IAICIASC_NOTES": null, "IAICIBU_LQ_BMPSFCHNLCON": "Yes", "IAICIBU_CHNLMNDMG": null, "IAICIBU_CHNLEXDMG": null, "IAICIBU_LQ_PIPECON": "Yes", "IAICIBU_PIPEJTMNSEP": null, "IAICIBU_PIPEJTEXSEP": null, "IAICIBU_PIPEJTSIGSEP": null, "IAICIBU_PIPEMJDMG": null, "IAICIBU_EWMNDMG": null, "IAICIBU_EWERLES25": null, "IAICIBU_EWERGR25": null, "IAICIBU_SIDEMNSCERLES1": null, "IAICIBU_SIDESIGSCERGR1": null, "IAICIBU_INFLWSCERLES1": null, "IAICIBU_INFLWSIGSCERGR1": null, "IAICIBU_NOTES": null, "FPPTPL_LQ_PTDVPRES": "N/A", "FPPTPL_NOTPRES": null, "FPPTPL_NOTES": null, "FPESTD_LQ_DVNOESTD": "Yes", "FPESTD_MNESTD": null, "FPESTD_EXESDT": null, "FPESTD_STNDISPLC": null, "FPESTD_LQ_AREANOER": "Yes", "FPESTD_AREAMNER": null, "FPESTD_AREAEXER": null, "FPESTD_NOTES": null, "IPTAPPMS_LQ_PVMSFNOMS": "Yes", "IPTAPPMS_PVMSFMSOBS": null, "IPTAPPMS_NOTES": null, "IPTAPPSS_LQ_PVMSFNOSED": "Yes", "IPTAPPSS_PVMSFMNSED": null, "IPTAPPSS_PVMSFSIGSED": null, "IPTAPPSS_PVMSFSINK": null, "IPTAPPSS_NOTES": null, "IPTAPPDVR_LQ_PVMNOVGRTDMG": "Yes", "IPTAPPDVR_UNSPECVG": null, "IPTAPPDVR_PVMDEADVG": null, "IPTAPPDVR_PVMDMGRT": null, "IPTAPPDVR_PVMEXPOSOIL": null, "IPTAPPDVR_NOTES": null, "IPTAPPSC_LQ_PMPVMNOCRAK": "Yes", "IPTAPPSC_PVMSFMNCRAK": null, "IPTAPPSC_PVMSFEXCRAK": null, "IPTAPPSC_PVMSFSEAL": null, "IPTAPPSC_NOTES": null, "IPTAFTES_LQ_CLOWCON": "Yes", "IPTAFTES_CLOWCAPMSDMG": null, "IPTAFTES_CLOWPIPEDMG": null, "IPTAFTES_CLOWOBSTR": null, "IPTAPPSW_LQ_PMPVMSFNOSW": "Yes", "IPTAPPSW_PVMSFSWOBS": null, "IPTAPPSW_NOTES": "Monthly cleaning and testing scheduled for night of February 18th. ", "IPTAPPDS_LQ_DRNGSYSCON": "Yes", "IPTAPPDS_OBSWELLDMG": null, "IPTAPPDS_WELLWATROBS": null, "IPTAPPDS_DRNGNOTFUNC": null, "IPTAPPDS_DRNGSEDOBS": null, "IPTAPPDS_DRNGCORSN": null, "IPTAPPDS_DRNGDMG": null, "IPTAPPDS_NOTES": null, "OSDCRC_LQ_RECVCHNLCON": "Yes", "OSDCRC_RECVCHNLERNODMG": null, "OSDCRC_RECVCHNLERDMG": null, "OSDCRC_RECVSEWERER": null, "OSDCRC_RECVSEWERBLK": null, "OSDCRC_NOTES": null, "INSP_OVERALL_RATING": 2, "EE_ER": "N/A", "MAINT_COMPLETION_DATE": null, "FPPTPL_DVCONFMPLAN": null, "SPEC_BMP_NOTES": null, "FPPTPL_COLLAPSED": null, "UNAUTH_DISCH": null}}, {"attributes": {"OBJECTID": 2, "rel_globalid": "7fc73b9d-b347-4464-92a4-dfa8217c30f5", "globalid": "da644253-ca67-4269-8164-3ace092ce9be", SNIP ============================================ "IPTAPPSW_NOTES": null, "IPTAPPDS_LQ_DRNGSYSCON": "Yes", "IPTAPPDS_OBSWELLDMG": null, "IPTAPPDS_WELLWATROBS": null, "IPTAPPDS_DRNGNOTFUNC": null, "IPTAPPDS_DRNGSEDOBS": null, "IPTAPPDS_DRNGCORSN": null, "IPTAPPDS_DRNGDMG": null, "IPTAPPDS_NOTES": null, "OSDCRC_LQ_RECVCHNLCON": "Yes", "OSDCRC_RECVCHNLERNODMG": null, "OSDCRC_RECVCHNLERDMG": null, "OSDCRC_RECVSEWERER": null, "OSDCRC_RECVSEWERBLK": null, "OSDCRC_NOTES": "Storm sewer system. ", "INSP_OVERALL_RATING": 1, "EE_ER": null, "MAINT_COMPLETION_DATE": null, "FPPTPL_DVCONFMPLAN": null, "SPEC_BMP_NOTES": null, "FPPTPL_COLLAPSED": null, "UNAUTH_DISCH": null}}]', 'token': 'KaFO048yLUbKB8E064Pl27vREp7l4IJ5NhAv22wGJ6nGs-dPE-tXcoiukW53SllE74sUioDhyZn4zGcrSSyIEIoYvQsaqEcnKFoLgWm0qwANKfl9Q8P63Q5Ckl0QeFTm'}

Re: Add Features at REST in Chunks

kapalczynski — Tue, 23 Jan 2024 17:03:40 GMT

I tried to grab that Features JSON return from the query and do the ADDFeature right at REST....
Any idea what this error could be about?

Both GUID fields just difference in Case for the field name

This is the Portal Feature Service for REL_GLOBALID

This is the AGOL Feature Layer for REL_GLOBALID

Re: Add Features at REST in Chunks

kapalczynski — Tue, 23 Jan 2024 17:23:18 GMT

OK so I hardcoded these in there because its a GUID field { }

It ran through now I get this error?

Re: Add Features at REST in Chunks

kapalczynski — Tue, 23 Jan 2024 17:30:01 GMT

Think I might have found it... there are a bunch of fields that have a Domain of Yes and No... I am trying to pass in NONE... its breaking...