In a Notebook, download shapefile from FTP and update existing feature service

rdbutger

I have a stand-alone Python script to download a zipped shapefile from an FTP site and upload to ArcGIS Online update an existing hosted feature service. Here's the simplified version -- 

#Set folder to download to
folder = 'C:\\myftpdownloads\\'
out_path = os.path.join(folder)

#FTP logon
ftp = FTP('ftp.myftp.com')
ftp.login('FTPusername', 'FTPpassword')

#Get a list of all files in the folder - for simplicity, let's pretend there is only ever 1 zip file there
files = ftp.nlst()

#Loop through the files to download
for zipfile in files:
     with open(os.path.join(out_path, zipfile), 'wb') as local_file:
     ftp.retrbinary('RETR '+ zipfile, local_file.write)

#Upload and update existing hosted feature service in ArcGIS Online
gis = GIS("https://arcgis.com/", "AGOUser", "AGOpassword")
item = gis.content.get('123456789mycontentid987654321')
# Call the update method to replace/overwrite it with the zip file from disk
item.update({}, zipfile)
# Update hosted feature layer
# Make sure to uncheck editing in Settings
item.publish(overwrite=True,file_type='shapefile')

I want to migrate this to an ArcGIS Online Notebook so that it can be scheduled and without having a dedicated machine running ArcGIS Pro and always on. 

I see good articles on Uploading datasets to use with ArcPy, Using ArcPy in ArcGIS Notebooks, and Attempting to Upload Files to ArcGIS Notebooks Programmatically -- but I can't find the exact right tutorial or post that puts it all together...

I need to programmatically download from an FTP site and upload directly ArcGIS Notebook. 

Would anyone have ideas? Is this even possible? 

EarlMedina

I don't use ArcGIS Notebooks much these days, so my assumption is there's some problem you're running into writing the zips to disk. I think you may be able to accomplish this in memory using a BytesIO object. I don't have immediate access to an FTP so I'm going to approximate this workflow using a web request, but the logic is pretty much the same.

 

Here is a working example of how to do the initial publish:

 

import requests
import io
from arcgis import GIS

zip_url = "https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_cd116_20m.zip"
r = requests.get(zip_url)
f = io.BytesIO()
f.write(r.content)

gis = GIS("https://arcgis.com/", "username", "pass")
item_properties = {
    "type": "Shapefile", 
    "title": "test", 
    "fileName": "virtual_zip.zip"
}
shpfile = gis.content.add(item_properties, f)
published_service = shpfile.publish()

 

 

 

You can overwrite like this:

 

import requests
import io
from arcgis import GIS
from arcgis.features import FeatureLayerCollection

zip_url = "https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_cd116_20m.zip"
r = requests.get(zip_url)
f = io.BytesIO()
f.write(r.content)

gis = GIS("https://arcgis.com/", "username", "pass")
item_id = "d44e3b2892a54f25a11def293724acbc"
item = gis.content.get(item_id)
flc = FeatureLayerCollection.fromitem(item)
flc.manager.overwrite(f)

 

 

 

In your case, I think the for loop would change to something like this:

 

f = io.BytesIO()
ftp.retrbinary("RETR "+ zipfile, f.write)
f.seek(0)