Here is an example python script that can be run to specifically backup hosted feature layers from ArcGIS Online to a local drive or network drive on your computer or network. Although not ideal to code in credentials, if it is stored and ran from a secure machine, it can be scheduled to run automatically with Windows Task Scheduler. Alternatively, you can run this script manually and alter it to prompt you for credentials. Thanks to Adam Koelker for presenting the core of this script via YouTube. I adapted to query and loop through a specific set, and added some print statements.
from time import strftime
print (strftime("%c"))
from arcgis.gis import GIS
###Authenticate to ArcGIS Online
gis=GIS("https://yourorganization.maps.arcgis.com","yourUsername","yourPassword")
###Query to for all items to be downloaded. In this case, it's searching for all feature layers marked as authoritative
myFeatureCollections=gis.content.search(query="contentstatus:org_authoritative",item_type="Feature Layer Collection",max_items=1000)
###Output location for downloaded backups
output=r"C:/path/to/output/location"
###Initiate cycle to export, download, and delete backups
for item in myFeatureCollections:
try:
print('Exporting '+str(item.title))
currentItemID=item.itemid
dataitem=gis.content.get(currentItemID)
### Create Backup
tempfile=strftime(dataitem.title+"_backup_%Y%m%d")
dataitem.export(title=tempfile,export_format="File Geodatabase",parameters=None,wait=True)
except:
print("Could not create backup for "+str(item.title)+' ('+str(item.itemid)+')')
pass
try:
### Find and download export
myexport=gis.content.search(tempfile,item_type="File Geodatabase")
fgdb=gis.content.get(myexport[0].itemid)
fgdb.download(save_path=output)
print("Downloaded "+str(fgdb.title)+" to "+output)
except:
print("Could not download export for "+str(myexport[0]))
pass
try:
###Delete export
fgdb.delete()
except:
print("Could not delete export for "+str(fgdb))
pass
print("Script completed at {}".format(strftime("%c")))
EDIT: This query no longer retrieves explicitly hosted feature layers, but also stored web layers. See https://developers.arcgis.com/rest/users-groups-and-items/items-and-item-types.htm for updated item types that can be queried
Hi @NorthSouthGIS, this is exactly what I have been looking for, I currently run a similar script that I've picked up somewhere online in which I need to enter in my credentials, the location to save the data to and the number of hosted feature layers to backup. So this is great.
I have added the script to a notebook in AGOL so that I can schedule it to run when I am away. I've done a test run and everything seems to complete without an issue. However, when I go to check my backup location (D:\Northumberland_PP\Back_ups\Automated_backups) there is nothing there.
You don't happen to have any thoughts on why this might be do you?
Hi Andrew,
Did you ever figure this out? Having the same issue. Tried reversing the backslashes, using a simple Temp folder on my C: drive, using a UNC file path, etc.
Teri
Hi @TeriL, I didn't find a way to get it to work unfortunately and then it dropped off my radar. Perhaps I'll contact ESRI to see if they can advise
Hi @Teri , I contacted ESRI abouth this and they suggested I post on here. See the below.
I have always had issues running it in AGOL, it downloads the files but they actually don't make it to your computers directory, they actually stay in AGOL they fall into your Notebook's Files so it is best to run this on your computer locally through ArcGIS Pro (which uses task scheduler) or just task scheduler. Computer just needs to be on, ArcGIS Pro doest need to be open.
Hey @Nick_Creedon, thanks for sharing this! I am using this script to download Hosted Services from ArcGIS Enterprise and it works great. However, the download for some services seems to be very slow, maybe I think its due to the size of the hosted data? Do you have any thoughts on optimizing this script for faster downloads?
Also, it failed to download some of the hosted services and also ignored some of the hosted service while the script was running. Do you have any idea why that's happening?
Hi Harun,
Happy to hear other people are using this!
I use this for ArcGIS Online Hosted Feature Layers, and I come across some layers that are slow to download.
I do not have a right answer for this, but large data sets and or layers with lots of attachments can definitely slow down the process. If you manually export a layer from AGOL or Portal as a GDB a large dataset will take longer than a small one. So these symptoms are across the board, its not only in the script.
To optimize faster downloads, you could break down group layers, relocate attachments to a web service and embed them as links in your layers to take the heavy lifting off your system. We do this for a few layers or else they would be over 10gb easy.
Since I posted this script I have definitely made some changes because have also had issues with some layers timing out when they are attempting to export. I have added a "Retry" action into the script to try at least 3 times if it were to fail.
import os
import datetime as dt
from arcgis.gis import GIS
import certifi
import urllib3
import time
# Suppress only the single InsecureRequestWarning from urllib3 needed for unverified HTTPS requests
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# Connect to ArcGIS Online or Portal for ArcGIS with certificate verification
gis = GIS("home", verify_cert=certifi.where())
# Set the parent folder path for backups
parent_folder_path = r"FILE LOCATION HERE"
# Create a new folder for today's date (e.g., "2025-02-11 Backup")
today_date = dt.datetime.now().strftime("%Y-%m-%d Full Backup")
folder_path = os.path.join(parent_folder_path, today_date)
# Create the folder if it doesn't exist
if not os.path.exists(folder_path):
os.makedirs(folder_path)
# Define the number of items to fetch
num_items = 1000
# Search for the most recently modified feature services
query_string = "type:Feature Service"
items = gis.content.search(query=query_string, max_items=num_items, sort_field='modified', sort_order='desc')
# Print the number of items to be backed up and their details
print(str(len(items)) + " items will be backed up to " + folder_path + ". See the list below:")
for item in items:
print(item.title)
# Function to download items as file geodatabases with retry mechanism
def download_as_fgdb(item_list, backup_location, max_retries=3):
for item in item_list:
retries = 0
while retries < max_retries:
try:
if 'View Service' in item.typeKeywords:
print(item.title + " is a view, not downloading")
break
else:
print("Downloading " + item.title)
version = dt.datetime.now().strftime("%d_%b_%Y")
result = item.export(item.title + "_" + version, "File Geodatabase")
result.download(backup_location)
result.delete()
print("Successfully downloaded " + item.title)
break
except Exception as e:
retries += 1
print(f"An error occurred downloading {item.title}: {str(e)} (Retry {retries}/{max_retries})")
if retries < max_retries:
time.sleep(5) # Wait for 5 seconds before retrying
else:
print(f"Failed to download {item.title} after {max_retries} retries")
print("The function has completed")
# Call the function to download items
download_as_fgdb(items, folder_path)
Thank you so much @Nick_Creedon for the prompt response! I will give it a shot and let you know.😊