topic Re: Issue When adding a list of 90K dictionaries using ArcGIS API Python in ArcGIS API for Python Questions

Issue When adding a list of 90K dictionaries using ArcGIS API Python

Manager_HMWSSBCDC — Mon, 26 Feb 2024 11:50:41 GMT

I'm fetching the my data from an API and passing the list into my feature layer in ArcGIS Portal. I configured the json data from API to a dictionary like below:

dict = {"attributes":

{"latitude": 33.75, "longitude": -118.25, "country": "US", "harborsize": "L", "label_position": "SW", "port_name": "LOS ANGELES", "short_form": "LAX"},

"geometry": {"x": -13044788.958999995, "y": 3857756.351200014}

}

Then using

feature_layer.edit_features(adds = [dict])

Its worked for features that are below 6k whereas the same query raising an error for 90K records. The error message is following:

Exception: Error performing apply edits operation
(Error Code: 500)

Please let me know if there any way to address this issue.

Re: Issue When adding a list of 90K dictionaries using ArcGIS API Python

HenryLindemann — Mon, 26 Feb 2024 12:15:59 GMT

Hi @Manager_HMWSSBCDC, yes that is just way to much you must break it up into smaller chunks, here is a piece of code that I wrote to do just that.

class BuildPacks: def __init__(self, list_of_items, pack_size=500): self._list_of_items = list_of_items self._length = self._list_of_items.__len__() self._pack_size = pack_size self._packs = None def build_packs(self): if self._length > self._pack_size: self._packs = int(self._length / self._pack_size) for pack in range(self._packs): idx_start = pack * self._pack_size idx_end = idx_start + self._pack_size yield self._list_of_items[idx_start:idx_end] idx_start = self._pack_size * self._packs remainder = self._length - idx_start if remainder > 0: yield self._list_of_items[idx_start:self._length] else: yield self._list_of_items

how to use it

import build_packs import arcigs # GIS Login con_src=arcgis.gis.GIS('portalurl','username','password') layer = arcgis.features.FeatureLayer('service url', gis=con_src) # your list of features item_ids_txt = [list of features] packs = build_packs.BuildPacks(item_ids_txt, pack_size=2000) for item_id in packs.build_packs(): layer.edit_features(adds=item_id)

hope it helps

Regards

Henry

Re: Issue When adding a list of 90K dictionaries using ArcGIS API Python

Manager_HMWSSBCDC — Thu, 29 Feb 2024 12:49:19 GMT

Thanks you very much. Its worked...

Re: Issue When adding a list of 90K dictionaries using ArcGIS API Python

Manager_HMWSSBCDC — Mon, 18 Mar 2024 05:58:42 GMT

Again the same issue, the error message is

Exception: Error performing apply edits operation
(Error Code: 500)

I don't know where the error was whether it is with data or from ArcGIS. Please help me in debugging.

Re: Issue When adding a list of 90K dictionaries using ArcGIS API Python

HenryLindemann — Mon, 18 Mar 2024 07:02:39 GMT

Hi @Manager_HMWSSBCDC you can change the script as below to get more details on what is going on, if all the edit are failing reduce the size, if it still fails then did something change? Schema changes might brake the script as they are not automatically pulled through to portal for ArcGIS.

import build_packs import arcigs import logging logging.basicConfig(level=logging.DEBUG) # GIS Login con_src=arcgis.gis.GIS('portalurl','username','password') layer = arcgis.features.FeatureLayer('service url', gis=con_src) # your list of features item_ids_txt = [list of features] packs = build_packs.BuildPacks(item_ids_txt, pack_size=2000) for item_id in packs.build_packs(): result = layer.edit_features(adds=item_id) logging.info(result)

Regards

Henry

Re: Issue When adding a list of 90K dictionaries using ArcGIS API Python

Manager_HMWSSBCDC — Mon, 18 Mar 2024 07:18:35 GMT

Thanks for the reply Henry,

After running the query, I got the following debug information but I"m not able to understand the message. Could you please let me know what exactly is the error message :

DEBUG:requests_oauthlib.oauth2_session:Generated new state Xn3syc0nx7aGDFWll87SGz0buc6bRt.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "GET /portal/sharing/rest/oauth2/authorize?response_type=token&client_id=pythonapi&redirect_uri=https%3A%2F%2Fgis.hyderabadwater.gov.in&state=Xn3syc0nx7aGDFWll87SGz0buc6bRt&expiration=600&allow_verification=false&style=dark&locale=en-US HTTP/1.1" 400 925
DEBUG:requests_oauthlib.oauth2_session:Generated new state 1tmVaPMN14lj0IuU8nHMTzZL9qiFuL.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "GET /portal/sharing/rest/oauth2/authorize?response_type=code&client_id=pythonapi&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&state=1tmVaPMN14lj0IuU8nHMTzZL9qiFuL&expiration=20160&allow_verification=false&style=dark&locale=en-US HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "POST /portal/sharing/oauth2/signin HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "GET /portal/sharing/oauth2/approval?code=Wz_cCtHUb3VQTGHMhD7g8yFnxwVy_63R-FdJg-Hhv_bHG5q0lBO1SSsuHUDAL8jvtAifw7JOQVUHN9es-FlKTpFgu4M75lJt0DbjUosgTSQ5y94Y8CFeYFA1sWODhjkpdeH53go0dhwLD6N6WpH2iG-s8BedQtH7UXMvdT1DnxV8fEAvWQYkK9giPY2NwNnErqD7mNq2YLbX4_0CtYgKpA..&state=1tmVaPMN14lj0IuU8nHMTzZL9qiFuL HTTP/1.1" 200 None
DEBUG:requests_oauthlib.oauth2_session:Requesting url https://gis.hyderabadwater.gov.in/portal/sharing/rest/oauth2/token using method POST.
DEBUG:requests_oauthlib.oauth2_session:Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'authorization_code', 'client_id': 'pythonapi', 'code': 'Wz_cCtHUb3VQTGHMhD7g8yFnxwVy_63R-FdJg-Hhv_bHG5q0lBO1SSsuHUDAL8jvtAifw7JOQVUHN9es-FlKTpFgu4M75lJt0DbjUosgTSQ5y94Y8CFeYFA1sWODhjkpdeH53go0dhwLD6N6WpH2iG-s8BedQtH7UXMvdT1DnxV8fEAvWQYkK9giPY2NwNnErqD7mNq2YLbX4_0CtYgKpA..', 'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob', 'expiration': '20160'}
DEBUG:requests_oauthlib.oauth2_session:Passing through key word arguments {'timeout': None, 'auth': None, 'verify': True, 'proxies': None}.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "POST /portal/sharing/rest/oauth2/token HTTP/1.1" 200 831
DEBUG:requests_oauthlib.oauth2_session:Request to fetch token completed with status 200.
DEBUG:requests_oauthlib.oauth2_session:Request url was https://gis.hyderabadwater.gov.in/portal/sharing/rest/oauth2/token
DEBUG:requests_oauthlib.oauth2_session:Request headers were {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8', 'Content-Length': '337'}
DEBUG:requests_oauthlib.oauth2_session:Request body was grant_type=authorization_code&client_id=pythonapi&code=Wz_cCtHUb3VQTGHMhD7g8yFnxwVy_63R-FdJg-Hhv_bHG5q0lBO1SSsuHUDAL8jvtAifw7JOQVUHN9es-FlKTpFgu4M75lJt0DbjUosgTSQ5y94Y8CFeYFA1sWODhjkpdeH53go0dhwLD6N6WpH2iG-s8BedQtH7UXMvdT1DnxV8fEAvWQYkK9giPY2NwNnErqD7mNq2YLbX4_0CtYgKpA..&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&expiration=20160
DEBUG:requests_oauthlib.oauth2_session:Response headers were {'Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '831', 'Content-Type': 'application/json; charset=utf-8', 'Expires': '0', 'Vary': 'Origin', 'X-XSS-Protection': '1; mode=block', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000', 'Date': 'Mon, 18 Mar 2024 07:14:48 GMT'} and content {"access_token":"-nLMNi4_illILlvxNbnYc_Bff4EfXz-y2tq-Q1ZMv2M6gH4Eb8IudlGoCicvnb3-ZVYG6EcEwiB08VVdUMU5Vm9xqhKSv9dwNoKWCcGaeeXlehxvolcaOMVYxYUAsFgknhiH2KWWvY8nKH_1x7QF0CwXbdUI1k_VlDzVnwGDGGDUlAPGHscg0VXN81kvN01zyfUkXJ6S8qwypIQdmAYsegWwPHgk8Wen3I6a_SMxB0vM7_LDSPp7afM5cboUhYBZ","expires_in":1800,"username":"portaladmin","ssl":true,"refresh_token":"cFuNxVaXcUz2S6104aS6IIhi-gPHIM3YRi6NmIaHirKEKlOQHI42V3asmHcD0HAgn-F45IU2V1PA3USZhistn2GmU3hfivQxaNPeuJSICvtOblGfMlbxsEbqm1N715QJa0TaUkE2OZm8mGRaPSpyWuULE3pS3Jof5EfmG_Qt_EtfOe2z9qERRmrWk6fKD2Bgwd9jI7EWSA_AUSlT4nsrIqBPTNjjQZAZU9Lf7E9wXf8LaUF0vp6ZpctKh-ROA3_bmByAU2IfGFW7DWDLyJiyLgz4QRn6qRVqRMqdHE8TFMdJRAtuCH44RwTgPI8IryH7jgC9LkOXb19gJQEvohL4v-8Vf7cRbZ8LjnTzQBLLFutkm8hQBiz_ULu0BP8g5sBvaZozq3HZgSN5pjv84YNrydWOmuiMqyHuQYSkZgc69eXmEfbYNSzVNBUvs47Q6nLW","refresh_token_expires_in":1209593}.
DEBUG:requests_oauthlib.oauth2_session:Invoking 0 token response hooks.
DEBUG:requests_oauthlib.oauth2_session:Obtained token {'access_token': '-nLMNi4_illILlvxNbnYc_Bff4EfXz-y2tq-Q1ZMv2M6gH4Eb8IudlGoCicvnb3-ZVYG6EcEwiB08VVdUMU5Vm9xqhKSv9dwNoKWCcGaeeXlehxvolcaOMVYxYUAsFgknhiH2KWWvY8nKH_1x7QF0CwXbdUI1k_VlDzVnwGDGGDUlAPGHscg0VXN81kvN01zyfUkXJ6S8qwypIQdmAYsegWwPHgk8Wen3I6a_SMxB0vM7_LDSPp7afM5cboUhYBZ', 'expires_in': 1800, 'username': 'portaladmin', 'ssl': True, 'refresh_token': 'cFuNxVaXcUz2S6104aS6IIhi-gPHIM3YRi6NmIaHirKEKlOQHI42V3asmHcD0HAgn-F45IU2V1PA3USZhistn2GmU3hfivQxaNPeuJSICvtOblGfMlbxsEbqm1N715QJa0TaUkE2OZm8mGRaPSpyWuULE3pS3Jof5EfmG_Qt_EtfOe2z9qERRmrWk6fKD2Bgwd9jI7EWSA_AUSlT4nsrIqBPTNjjQZAZU9Lf7E9wXf8LaUF0vp6ZpctKh-ROA3_bmByAU2IfGFW7DWDLyJiyLgz4QRn6qRVqRMqdHE8TFMdJRAtuCH44RwTgPI8IryH7jgC9LkOXb19gJQEvohL4v-8Vf7cRbZ8LjnTzQBLLFutkm8hQBiz_ULu0BP8g5sBvaZozq3HZgSN5pjv84YNrydWOmuiMqyHuQYSkZgc69eXmEfbYNSzVNBUvs47Q6nLW', 'refresh_token_expires_in': 1209593, 'expires_at': 1710747888.4648676}.
DEBUG:urllib3.connectionpool:Resetting dropped connection: gis.hyderabadwater.gov.in
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "POST /server/rest/services/tankers/FeatureServer/4/applyEdits HTTP/1.1" 200 100

Re: Issue When adding a list of 90K dictionaries using ArcGIS API Python

Manager_HMWSSBCDC — Mon, 18 Mar 2024 12:36:49 GMT

I found the problem is with date value.

If I add my data value milliseconds format like '17345735723453', then its adding into my layer otherwise if I use date of any format, its showing error like unable to applyedits function.