topic Re: Overwrite Hosted Feature Layer Overflow error in ArcGIS API for Python Questions

Overwrite Hosted Feature Layer Overflow error

AaronKoelker — Mon, 30 May 2022 16:29:01 GMT

Hey all,

I'm trying to script out an overwrite workflow using the method found here:

*Relevant code from above link (Overwrite Feature Layer section near the bottom):

from arcgis.features import FeatureLayerCollection
cities_flayer_collection = FeatureLayerCollection.fromitem(cities_item)

#call the overwrite() method which can be accessed using the manager property
cities_flayer_collection.manager.overwrite(os.path.join('data', 'updating_gis_content', 'updated_capitals_csv', 'capitals_1.csv'))

I've used this in the past for other scripts and it works swell. However, I now have a need to create a single "master" feature layer comprised of multiple datasets that have been merged together, and it comes out as a pretty big file size when zipped together in a fgdb for upload (2.1 GB). When I manually execute the Overwrite using the menu buttons on the item detail page in ArcGIS Online, it uploads and overwrites the feature layer from my zipped fgdb with no problem.

When I try to replicate this via the scripting method above, I get the following error:

Traceback (most recent call last):
File "C:/Users/user_user/.PyCharmCE2019.3/config/scratches/DRP_Update_SNAPP_Master.py", line 146, in <module>
    flayer.manager.overwrite(newdata)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\managers.py", line 1268, in overwrite
    if related_data_item.update(item_properties=params, data=data_file):
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 8165, in update
    large_thumbnail)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\portalpy.py", line 2228, in update_item
    resp = self.con.post(path, postdata, files)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 1106, in post
    resp = opener.open(req)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 544, in _open
    '_open', req)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1075, in _send_output
    self.send(chunk)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 996, in send
    self.sock.sendall(data)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py", line 975, in sendall
    v = self.send(byte_view[count:])
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py", line 944, in send
    return self._sslobj.write(data)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py", line 642, in write
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

I haven't found any sort of note within the API reference stating that the overwrite uploads are limited to 2GB -- and it works when I do it manually -- so I'm confused what I'm running into here, and unsure how to work around it. Is it a limitation of the urllib or ssl packages that the .overwrite function uses?

Re: Overwrite Hosted Feature Layer Overflow error

AaronKoelker — Mon, 16 Mar 2020 21:03:04 GMT

Since it's been a couple of weeks without an answer, I'm probably going to end up using .sd files to publish the data from within an ArcGIS Pro project as a workaround... but if anyone down the road sees this question and knows of a solution I would still love to hear it, because I much prefer the workflow above using the ArcGIS API for Python.

EDIT: Welp, I can't get anywhere with .sd files from Pro, either. I get the obscure ERROR 999999 (The base table definition string "0" is invalid?) randomly when trying to use the Copy Features tool one by one for each layer in my hosted feature collection. Apparently, my hosted feature is too big to do anything with now, because I can no longer grab it programmatically or download it manually from the item details page. I'm at a complete loss. If anyone has any idea how to resolve either of these issues, please give me a shout.

-Aaron

Re: Overwrite Hosted Feature Layer Overflow error

AlexRexroad2 — Sun, 12 Dec 2021 09:14:46 GMT

I haven't seen a solution yet, but wanted to add that I am getting the same error when using the arcgis.gis.Item.update() function arcgis.gis module — arcgis 1.8.0 documentation.

itemProps = {}
success = item.update(item_properties=itemProps,data=src_file)‍‍‍
‍‍‍‍‍

item is an existing Tile Package item on ArcGIS Online.

src_file is a path to a Tile Package (.lpkx) locally, size 2.1 GB.

Error Returned:

File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 8165, in update
 large_thumbnail)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\portalpy.py", line 2228, in update_item
 resp = self.con.post(path, postdata, files)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 1106, in post
 resp = opener.open(req)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 526, in open
 response = self._open(req, data)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 544, in _open
 '_open', req)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 504, in _call_chain
 result = func(*args)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 1361, in https_open
 context=self._context, check_hostname=self._check_hostname)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 1318, in do_open
 encode_chunked=req.has_header('Transfer-encoding'))
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1254, in request
 self._send_request(method, url, body, headers, encode_chunked)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1300, in _send_request
 self.endheaders(body, encode_chunked=encode_chunked)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1249, in endheaders
 self._send_output(message_body, encode_chunked=encode_chunked)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 1075, in _send_output
 self.send(chunk)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\http\client.py", line 996, in send
 self.sock.sendall(data)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py", line 975, in sendall
 v = self.send(byte_view[count:])
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py", line 944, in send
 return self._sslobj.write(data)
 File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py", line 642, in write
 return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

The Upload function from the AGOL Item page seems to work without issue. I've also made sure the filename being uploaded is the same as the original file.

Re: Overwrite Hosted Feature Layer Overflow error

Sean-Sweeney — Mon, 14 Jun 2021 16:03:22 GMT

I know it's been 15 months, but for reference I'm getting the same exact error in ArcGIS API for Python version 1.8.5. The zip file I'm failing on is an FGDB with a layer that has attachments (photos), so it's quite large - 8GB. The size may have something to do with the issue because smaller (< 1GB) files work, but that's completely circumstantial evidence at this point. I don't currently have any better metrics.

Re: Overwrite Hosted Feature Layer Overflow error

MaryamBarzegar — Mon, 28 Feb 2022 02:35:16 GMT

Hi, I have the same issue with overwriting a large SD file (2.15 GB). Seems there is no solution for this, right? :((

Definitely size is the issue because I can overwrite other SD files with smaller size.

Re: Overwrite Hosted Feature Layer Overflow error

DanielWebb — Wed, 06 Apr 2022 18:46:21 GMT

Me too. I'm using Python to update a Portal file geodatabase item. It seems to be because of the size (3.2GB). When I delete the photos attachment tables to reduce size (753MB) then it updates fine.

fgdbItem = gis.content.get('itemId')
fgdbItem .update(data=r"link to zip file")

I was also able to use gis.content.add to successfully upload the 3.2GB zipped fgdb. So adding a large file works, but updating does not.

I can also manually update item through Portal. So file size doesn't matter there, either. It just seems to be Item.update that throws an issue.

Re: Overwrite Hosted Feature Layer Overflow error

StanislausCounty — Thu, 26 May 2022 23:39:23 GMT

I'm also getting this error; does Esri plan on fixing this?

Re: Overwrite Hosted Feature Layer Overflow error

KentWilliams — Wed, 15 Jun 2022 03:55:09 GMT

I am encountering the same issue uploading a zipped FGDB to overwrite a hosted feature layer. "Exception: A general error occurred: string longer than 2147483647 bytes"

When I limited my test data to reduce the number of features, the size was kept the size down and the script worked flawlessly. The zipped FGDB with the entire dataset is 3.6GB and the exact same identical code fails. This is just plain unacceptable behavior for a problem reported over 2 years ago.