topic Re: In a Notebook, download shapefile from FTP and update existing feature service in ArcGIS API for Python Questions

In a Notebook, download shapefile from FTP and update existing feature service

rdbutger — Tue, 12 Sep 2023 04:31:19 GMT

I have a stand-alone Python script to download a zipped shapefile from an FTP site and upload to ArcGIS Online update an existing hosted feature service. Here's the simplified version --

#Set folder to download to
folder = 'C:\\myftpdownloads\\'
out_path = os.path.join(folder)

#FTP logon
ftp = FTP('ftp.myftp.com')
ftp.login('FTPusername', 'FTPpassword')

#Get a list of all files in the folder - for simplicity, let's pretend there is only ever 1 zip file there
files = ftp.nlst()

#Loop through the files to download
for zipfile in files:
with open(os.path.join(out_path, zipfile), 'wb') as local_file:
ftp.retrbinary('RETR '+ zipfile, local_file.write)

#Upload and update existing hosted feature service in ArcGIS Online
gis = GIS("https://arcgis.com/", "AGOUser", "AGOpassword")
item = gis.content.get('123456789mycontentid987654321')
# Call the update method to replace/overwrite it with the zip file from disk
item.update({}, zipfile)
# Update hosted feature layer
# Make sure to uncheck editing in Settings
item.publish(overwrite=True,file_type='shapefile')

I want to migrate this to an ArcGIS Online Notebook so that it can be scheduled and without having a dedicated machine running ArcGIS Pro and always on.

I see good articles on Uploading datasets to use with ArcPy, Using ArcPy in ArcGIS Notebooks, and Attempting to Upload Files to ArcGIS Notebooks Programmatically -- but I can't find the exact right tutorial or post that puts it all together...

I need to programmatically download from an FTP site and upload directly ArcGIS Notebook.

Would anyone have ideas? Is this even possible?

Re: In a Notebook, download shapefile from FTP and update existing feature service

EarlMedina — Wed, 13 Sep 2023 13:30:07 GMT

I don't use ArcGIS Notebooks much these days, so my assumption is there's some problem you're running into writing the zips to disk. I think you may be able to accomplish this in memory using a BytesIO object. I don't have immediate access to an FTP so I'm going to approximate this workflow using a web request, but the logic is pretty much the same.

Here is a working example of how to do the initial publish:

import requests import io from arcgis import GIS zip_url = "https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_cd116_20m.zip" r = requests.get(zip_url) f = io.BytesIO() f.write(r.content) gis = GIS("https://arcgis.com/", "username", "pass") item_properties = { "type": "Shapefile", "title": "test", "fileName": "virtual_zip.zip" } shpfile = gis.content.add(item_properties, f) published_service = shpfile.publish()

You can overwrite like this:

import requests import io from arcgis import GIS from arcgis.features import FeatureLayerCollection zip_url = "https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_cd116_20m.zip" r = requests.get(zip_url) f = io.BytesIO() f.write(r.content) gis = GIS("https://arcgis.com/", "username", "pass") item_id = "d44e3b2892a54f25a11def293724acbc" item = gis.content.get(item_id) flc = FeatureLayerCollection.fromitem(item) flc.manager.overwrite(f)

In your case, I think the for loop would change to something like this:

f = io.BytesIO() ftp.retrbinary("RETR "+ zipfile, f.write) f.seek(0)

Re: In a Notebook, download shapefile from FTP and update existing feature service

RichardButgereit — Thu, 14 Sep 2023 21:01:11 GMT

Thanks, that looks promising. I will get it a shot and let you know how it goes.

Re: In a Notebook, download shapefile from FTP and update existing feature service

RichardButgereit — Tue, 19 Sep 2023 20:22:14 GMT

I submitted a case with Esri and got this back --
------------------------------------------------------------------------------------------------------------
Case #03441759
The behavior you've described, where FTP connections appear to be blocked when running a script in ArcGIS Online Notebooks, is likely a result of the security and network access restrictions enforced within the ArcGIS Online environment. ArcGIS Online is hosted in a secure cloud-based platform, and these measures are in place to ensure the security of the platform and its users.

If you're encountering a "ConnectionRefusedError: [Errno 111] Connection refused" error, this is indicative of the FTP connection not being allowed. To work around this limitation, I recommend using an alternative approach: consider utilizing another server where the ArcGIS API for Python can be employed to upload data to ArcGIS Online. Unfortunately, AGOL Notebooks cannot use FTP to download data due to these security restrictions.
------------------------------------------------------------------------------------------------------------

So yeah -- FTP connections are blocked with AGOL and AGOL Notebooks. So don't bother going down this path.

Re: In a Notebook, download shapefile from FTP and update existing feature service

dzahsh — Wed, 06 Dec 2023 18:02:10 GMT

Just to add to this: I tried to connect to a public FTP server in an ArcGIS Online Notebook using the python module ftplib and received the "ConnectionRefusedError: [Errno 111] Connection refused" error