Issue When adding a list of 90K dictionaries using ArcGIS API Python

[Manager_HMWSSBCDC]

I'm fetching the my data from an API and passing the list into my feature layer in ArcGIS Portal. I configured the json data from API to a dictionary like below:

dict = {"attributes":

                                 {"latitude": 33.75, "longitude": -118.25, "country": "US", "harborsize": "L", "label_position":                                       "SW", "port_name": "LOS ANGELES", "short_form": "LAX"},

            "geometry": {"x": -13044788.958999995, "y": 3857756.351200014}

}

Then using

feature_layer.edit_features(adds = [dict])

Its worked for features that are below 6k whereas the same query raising an error for 90K records. The error message is following:

Exception: Error performing apply edits operation
(Error Code: 500)

Please let me know if there any way to address this issue.

[HenryLindemann · Accepted solution]

Hi @Manager_HMWSSBCDC, yes that is just way to much you must break it up into smaller chunks, here is a piece of code that I wrote to do just that.

class BuildPacks:
    def __init__(self, list_of_items, pack_size=500):
        self._list_of_items = list_of_items
        self._length = self._list_of_items.__len__()
        self._pack_size = pack_size
        self._packs = None

    def build_packs(self):
        if self._length > self._pack_size:
            self._packs = int(self._length / self._pack_size)
            for pack in range(self._packs):
                idx_start = pack * self._pack_size
                idx_end = idx_start + self._pack_size
                yield self._list_of_items[idx_start:idx_end]
            idx_start = self._pack_size * self._packs
            remainder = self._length - idx_start
            if remainder > 0:
                yield self._list_of_items[idx_start:self._length]
        else:
            yield self._list_of_items

how to use it 

import build_packs
import arcigs
# GIS Login
con_src=arcgis.gis.GIS('portalurl','username','password')
layer = arcgis.features.FeatureLayer('service url', gis=con_src)

# your list of features
item_ids_txt = [list of features]
packs = build_packs.BuildPacks(item_ids_txt, pack_size=2000)
for item_id in packs.build_packs():
    layer.edit_features(adds=item_id)

hope it helps

Regards

Henry

[Manager_HMWSSBCDC · Accepted solution]

Thanks you very much. Its worked...

[Manager_HMWSSBCDC]

Hi

Again the same issue, the error message is 

Exception: Error performing apply edits operation
(Error Code: 500)

I don't know where the error was whether it is with data or from ArcGIS. Please help me in debugging. 

[HenryLindemann]

Hi @Manager_HMWSSBCDC you can change the script as below to get more details on what is going on, if all the edit are failing reduce the size, if it still fails then did something change? Schema changes might brake the script as they are not automatically pulled through to portal for ArcGIS.

import build_packs
import arcigs
import logging
logging.basicConfig(level=logging.DEBUG)
# GIS Login
con_src=arcgis.gis.GIS('portalurl','username','password')
layer = arcgis.features.FeatureLayer('service url', gis=con_src)

# your list of features
item_ids_txt = [list of features]
packs = build_packs.BuildPacks(item_ids_txt, pack_size=2000)
for item_id in packs.build_packs():
    result = layer.edit_features(adds=item_id)
    logging.info(result)

Regards

Henry

[Manager_HMWSSBCDC]

Thanks for the reply Henry,

After running the query, I got the following debug information but I"m not able to understand the message. Could you please let me know what exactly is the error message :

DEBUG:requests_oauthlib.oauth2_session:Generated new state Xn3syc0nx7aGDFWll87SGz0buc6bRt.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "GET /portal/sharing/rest/oauth2/authorize?response_type=token&client_id=pythonapi&redirect_uri=https%3A%2F%2Fgis.hyderabadwater.gov.in&state=Xn3syc0nx7aGDFWll87SGz0buc6bRt&expiration=600&allow_verification=false&style=dark&locale=en-US HTTP/1.1" 400 925
DEBUG:requests_oauthlib.oauth2_session:Generated new state 1tmVaPMN14lj0IuU8nHMTzZL9qiFuL.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "GET /portal/sharing/rest/oauth2/authorize?response_type=code&client_id=pythonapi&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&state=1tmVaPMN14lj0IuU8nHMTzZL9qiFuL&expiration=20160&allow_verification=false&style=dark&locale=en-US HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "POST /portal/sharing/oauth2/signin HTTP/1.1" 302 0
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "GET /portal/sharing/oauth2/approval?code=Wz_cCtHUb3VQTGHMhD7g8yFnxwVy_63R-FdJg-Hhv_bHG5q0lBO1SSsuHUDAL8jvtAifw7JOQVUHN9es-FlKTpFgu4M75lJt0DbjUosgTSQ5y94Y8CFeYFA1sWODhjkpdeH53go0dhwLD6N6WpH2iG-s8BedQtH7UXMvdT1DnxV8fEAvWQYkK9giPY2NwNnErqD7mNq2YLbX4_0CtYgKpA..&state=1tmVaPMN14lj0IuU8nHMTzZL9qiFuL HTTP/1.1" 200 None
DEBUG:requests_oauthlib.oauth2_session:Requesting url https://gis.hyderabadwater.gov.in/portal/sharing/rest/oauth2/token using method POST.
DEBUG:requests_oauthlib.oauth2_session:Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'authorization_code', 'client_id': 'pythonapi', 'code': 'Wz_cCtHUb3VQTGHMhD7g8yFnxwVy_63R-FdJg-Hhv_bHG5q0lBO1SSsuHUDAL8jvtAifw7JOQVUHN9es-FlKTpFgu4M75lJt0DbjUosgTSQ5y94Y8CFeYFA1sWODhjkpdeH53go0dhwLD6N6WpH2iG-s8BedQtH7UXMvdT1DnxV8fEAvWQYkK9giPY2NwNnErqD7mNq2YLbX4_0CtYgKpA..', 'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob', 'expiration': '20160'}
DEBUG:requests_oauthlib.oauth2_session:Passing through key word arguments {'timeout': None, 'auth': None, 'verify': True, 'proxies': None}.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gis.hyderabadwater.gov.in:443
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "POST /portal/sharing/rest/oauth2/token HTTP/1.1" 200 831
DEBUG:requests_oauthlib.oauth2_session:Request to fetch token completed with status 200.
DEBUG:requests_oauthlib.oauth2_session:Request url was https://gis.hyderabadwater.gov.in/portal/sharing/rest/oauth2/token
DEBUG:requests_oauthlib.oauth2_session:Request headers were {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8', 'Content-Length': '337'}
DEBUG:requests_oauthlib.oauth2_session:Request body was grant_type=authorization_code&client_id=pythonapi&code=Wz_cCtHUb3VQTGHMhD7g8yFnxwVy_63R-FdJg-Hhv_bHG5q0lBO1SSsuHUDAL8jvtAifw7JOQVUHN9es-FlKTpFgu4M75lJt0DbjUosgTSQ5y94Y8CFeYFA1sWODhjkpdeH53go0dhwLD6N6WpH2iG-s8BedQtH7UXMvdT1DnxV8fEAvWQYkK9giPY2NwNnErqD7mNq2YLbX4_0CtYgKpA..&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&expiration=20160
DEBUG:requests_oauthlib.oauth2_session:Response headers were {'Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '831', 'Content-Type': 'application/json; charset=utf-8', 'Expires': '0', 'Vary': 'Origin', 'X-XSS-Protection': '1; mode=block', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000', 'Date': 'Mon, 18 Mar 2024 07:14:48 GMT'} and content {"access_token":"-nLMNi4_illILlvxNbnYc_Bff4EfXz-y2tq-Q1ZMv2M6gH4Eb8IudlGoCicvnb3-ZVYG6EcEwiB08VVdUMU5Vm9xqhKSv9dwNoKWCcGaeeXlehxvolcaOMVYxYUAsFgknhiH2KWWvY8nKH_1x7QF0CwXbdUI1k_VlDzVnwGDGGDUlAPGHscg0VXN81kvN01zyfUkXJ6S8qwypIQdmAYsegWwPHgk8Wen3I6a_SMxB0vM7_LDSPp7afM5cboUhYBZ","expires_in":1800,"username":"portaladmin","ssl":true,"refresh_token":"cFuNxVaXcUz2S6104aS6IIhi-gPHIM3YRi6NmIaHirKEKlOQHI42V3asmHcD0HAgn-F45IU2V1PA3USZhistn2GmU3hfivQxaNPeuJSICvtOblGfMlbxsEbqm1N715QJa0TaUkE2OZm8mGRaPSpyWuULE3pS3Jof5EfmG_Qt_EtfOe2z9qERRmrWk6fKD2Bgwd9jI7EWSA_AUSlT4nsrIqBPTNjjQZAZU9Lf7E9wXf8LaUF0vp6ZpctKh-ROA3_bmByAU2IfGFW7DWDLyJiyLgz4QRn6qRVqRMqdHE8TFMdJRAtuCH44RwTgPI8IryH7jgC9LkOXb19gJQEvohL4v-8Vf7cRbZ8LjnTzQBLLFutkm8hQBiz_ULu0BP8g5sBvaZozq3HZgSN5pjv84YNrydWOmuiMqyHuQYSkZgc69eXmEfbYNSzVNBUvs47Q6nLW","refresh_token_expires_in":1209593}.
DEBUG:requests_oauthlib.oauth2_session:Invoking 0 token response hooks.
DEBUG:requests_oauthlib.oauth2_session:Obtained token {'access_token': '-nLMNi4_illILlvxNbnYc_Bff4EfXz-y2tq-Q1ZMv2M6gH4Eb8IudlGoCicvnb3-ZVYG6EcEwiB08VVdUMU5Vm9xqhKSv9dwNoKWCcGaeeXlehxvolcaOMVYxYUAsFgknhiH2KWWvY8nKH_1x7QF0CwXbdUI1k_VlDzVnwGDGGDUlAPGHscg0VXN81kvN01zyfUkXJ6S8qwypIQdmAYsegWwPHgk8Wen3I6a_SMxB0vM7_LDSPp7afM5cboUhYBZ', 'expires_in': 1800, 'username': 'portaladmin', 'ssl': True, 'refresh_token': 'cFuNxVaXcUz2S6104aS6IIhi-gPHIM3YRi6NmIaHirKEKlOQHI42V3asmHcD0HAgn-F45IU2V1PA3USZhistn2GmU3hfivQxaNPeuJSICvtOblGfMlbxsEbqm1N715QJa0TaUkE2OZm8mGRaPSpyWuULE3pS3Jof5EfmG_Qt_EtfOe2z9qERRmrWk6fKD2Bgwd9jI7EWSA_AUSlT4nsrIqBPTNjjQZAZU9Lf7E9wXf8LaUF0vp6ZpctKh-ROA3_bmByAU2IfGFW7DWDLyJiyLgz4QRn6qRVqRMqdHE8TFMdJRAtuCH44RwTgPI8IryH7jgC9LkOXb19gJQEvohL4v-8Vf7cRbZ8LjnTzQBLLFutkm8hQBiz_ULu0BP8g5sBvaZozq3HZgSN5pjv84YNrydWOmuiMqyHuQYSkZgc69eXmEfbYNSzVNBUvs47Q6nLW', 'refresh_token_expires_in': 1209593, 'expires_at': 1710747888.4648676}.
DEBUG:urllib3.connectionpool:Resetting dropped connection: gis.hyderabadwater.gov.in
DEBUG:urllib3.connectionpool:https://gis.hyderabadwater.gov.in:443 "POST /server/rest/services/tankers/FeatureServer/4/applyEdits HTTP/1.1" 200 100

[Manager_HMWSSBCDC]

I found the problem is with date value.

If I add my data value milliseconds format like '17345735723453', then its adding into my layer otherwise if  I use date of any format, its showing error like unable to applyedits function.