topic Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections) in ArcGIS Online Questions

Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

NorthSouthGIS — Fri, 18 Feb 2022 19:19:08 GMT

Here is an example python script that can be run to specifically backup hosted feature layers from ArcGIS Online to a local drive or network drive on your computer or network. Although not ideal to code in credentials, if it is stored and ran from a secure machine, it can be scheduled to run automatically with Windows Task Scheduler. Alternatively, you can run this script manually and alter it to prompt you for credentials. Thanks to Adam Koelker for presenting the core of this script via YouTube. I adapted to query and loop through a specific set, and added some print statements.

from time import strftime print (strftime("%c")) from arcgis.gis import GIS ###Authenticate to ArcGIS Online gis=GIS("https://yourorganization.maps.arcgis.com","yourUsername","yourPassword") ###Query to for all items to be downloaded. In this case, it's searching for all feature layers marked as authoritative myFeatureCollections=gis.content.search(query="contentstatus:org_authoritative",item_type="Feature Layer Collection",max_items=1000) ###Output location for downloaded backups output=r"C:/path/to/output/location" ###Initiate cycle to export, download, and delete backups for item in myFeatureCollections: try: print('Exporting '+str(item.title)) currentItemID=item.itemid dataitem=gis.content.get(currentItemID) ### Create Backup tempfile=strftime(dataitem.title+"_backup_%Y%m%d") dataitem.export(title=tempfile,export_format="File Geodatabase",parameters=None,wait=True) except: print("Could not create backup for "+str(item.title)+' ('+str(item.itemid)+')') pass try: ### Find and download export myexport=gis.content.search(tempfile,item_type="File Geodatabase") fgdb=gis.content.get(myexport[0].itemid) fgdb.download(save_path=output) print("Downloaded "+str(fgdb.title)+" to "+output) except: print("Could not download export for "+str(myexport[0])) pass try: ###Delete export fgdb.delete() except: print("Could not delete export for "+str(fgdb)) pass print("Script completed at {}".format(strftime("%c")))

EDIT: This query no longer retrieves explicitly hosted feature layers, but also stored web layers. See https://developers.arcgis.com/rest/users-groups-and-items/items-and-item-types.htm for updated item types that can be queried

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

AndrewHankinson — Tue, 10 Oct 2023 05:54:32 GMT

Hi @NorthSouthGIS, this is exactly what I have been looking for, I currently run a similar script that I've picked up somewhere online in which I need to enter in my credentials, the location to save the data to and the number of hosted feature layers to backup. So this is great.
I have added the script to a notebook in AGOL so that I can schedule it to run when I am away. I've done a test run and everything seems to complete without an issue. However, when I go to check my backup location (D:\Northumberland_PP\Back_ups\Automated_backups) there is nothing there.
You don't happen to have any thoughts on why this might be do you?

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

TeriL — Thu, 08 Feb 2024 17:01:56 GMT

Hi Andrew,

Did you ever figure this out? Having the same issue. Tried reversing the backslashes, using a simple Temp folder on my C: drive, using a UNC file path, etc.

Teri

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

AndrewHankinson — Fri, 09 Feb 2024 00:37:30 GMT

Hi @TeriL, I didn't find a way to get it to work unfortunately and then it dropped off my radar. Perhaps I'll contact ESRI to see if they can advise

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

AndrewHankinson — Fri, 23 Feb 2024 03:38:29 GMT

Hi @Teri , I contacted ESRI abouth this and they suggested I post on here. See the below.

I have had a further look into your case, and as you are using ArcGIS Api for Python you require a developer support subscription for further support on this.

I would recommend that you post a thread on the page that you found the notebook script on to find a solution.

As this is out of scope of regular support I will move to close the case

You haven't happened to find a solution have you?

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

Nick_Creedon — Mon, 19 Aug 2024 12:04:22 GMT

I have always had issues running it in AGOL, it downloads the files but they actually don't make it to your computers directory, they actually stay in AGOL they fall into your Notebook's Files so it is best to run this on your computer locally through ArcGIS Pro (which uses task scheduler) or just task scheduler. Computer just needs to be on, ArcGIS Pro doest need to be open.

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

Nick_Creedon — Mon, 19 Aug 2024 12:05:58 GMT

I'm attaching the script I compiled and use.

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

AndrewHankinson — Mon, 16 Sep 2024 00:53:10 GMT

@Nick_Creedon, thanks for this works great.

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

HarunJoeVellikkara — Tue, 08 Apr 2025 06:55:36 GMT

Hey @Nick_Creedon, thanks for sharing this! I am using this script to download Hosted Services from ArcGIS Enterprise and it works great. However, the download for some services seems to be very slow, maybe I think its due to the size of the hosted data? Do you have any thoughts on optimizing this script for faster downloads?

Also, it failed to download some of the hosted services and also ignored some of the hosted service while the script was running. Do you have any idea why that's happening?

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

Nick_Creedon — Tue, 08 Apr 2025 11:24:21 GMT

Hi Harun,

Happy to hear other people are using this!

I use this for ArcGIS Online Hosted Feature Layers, and I come across some layers that are slow to download.

I do not have a right answer for this, but large data sets and or layers with lots of attachments can definitely slow down the process. If you manually export a layer from AGOL or Portal as a GDB a large dataset will take longer than a small one. So these symptoms are across the board, its not only in the script.

To optimize faster downloads, you could break down group layers, relocate attachments to a web service and embed them as links in your layers to take the heavy lifting off your system. We do this for a few layers or else they would be over 10gb easy.

Since I posted this script I have definitely made some changes because have also had issues with some layers timing out when they are attempting to export. I have added a "Retry" action into the script to try at least 3 times if it were to fail.

import os import datetime as dt from arcgis.gis import GIS import certifi import urllib3 import time # Suppress only the single InsecureRequestWarning from urllib3 needed for unverified HTTPS requests urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) # Connect to ArcGIS Online or Portal for ArcGIS with certificate verification gis = GIS("home", verify_cert=certifi.where()) # Set the parent folder path for backups parent_folder_path = r"FILE LOCATION HERE" # Create a new folder for today's date (e.g., "2025-02-11 Backup") today_date = dt.datetime.now().strftime("%Y-%m-%d Full Backup") folder_path = os.path.join(parent_folder_path, today_date) # Create the folder if it doesn't exist if not os.path.exists(folder_path): os.makedirs(folder_path) # Define the number of items to fetch num_items = 1000 # Search for the most recently modified feature services query_string = "type:Feature Service" items = gis.content.search(query=query_string, max_items=num_items, sort_field='modified', sort_order='desc') # Print the number of items to be backed up and their details print(str(len(items)) + " items will be backed up to " + folder_path + ". See the list below:") for item in items: print(item.title) # Function to download items as file geodatabases with retry mechanism def download_as_fgdb(item_list, backup_location, max_retries=3): for item in item_list: retries = 0 while retries < max_retries: try: if 'View Service' in item.typeKeywords: print(item.title + " is a view, not downloading") break else: print("Downloading " + item.title) version = dt.datetime.now().strftime("%d_%b_%Y") result = item.export(item.title + "_" + version, "File Geodatabase") result.download(backup_location) result.delete() print("Successfully downloaded " + item.title) break except Exception as e: retries += 1 print(f"An error occurred downloading {item.title}: {str(e)} (Retry {retries}/{max_retries})") if retries < max_retries: time.sleep(5) # Wait for 5 seconds before retrying else: print(f"Failed to download {item.title} after {max_retries} retries") print("The function has completed") # Call the function to download items download_as_fgdb(items, folder_path)

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

HarunJoeVellikkara — Wed, 09 Apr 2025 06:56:11 GMT

Thank you so much @Nick_Creedon for the prompt response! I will give it a shot and let you know.😊

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

HarunJoeVellikkara — Wed, 09 Apr 2025 11:49:39 GMT

Hey @Nick_Creedon sorry to bother you! You have been very helpful.

I had a query since I am running this script for ArcGIS Enterprise is there a way we can tweak this script to get only Hosted Feature services. Since, query_string = "type:Feature Service" includes both Hosted and Referenced services right?

Also, do you think taking a GeoJson backup would be as helpful as taking a File Geodatabase backup? For example if one of my hosted service is corrupted/deleted which one would be more feasible to restore (JSON Vs Fgdb).

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

Nick_Creedon — Wed, 09 Apr 2025 12:59:08 GMT

These are good questions,

Referenced services like versions or "View Layers"? To my understanding it will try to export the versioned layer but it will fail and only export the parent. If this is what you are referring too. I have lots of view layers and they don't make it into my backups.

I left this out of my previous messages, but you could specify "Tags" for your backups so you only backup specific layers and not your whole organization every time. I do weekly backups of specific tagged layers and quarterly backups of the whole organization.

I have never worked with JSON, I cannot answer that question. I would love to get to understand its uses.

If you deploy a layer from ArcGIS Pro to Portal or AGOL, you create a Service Definition. I have pulled a service definition back to use the original schema and used data from a FGDB to fix a layer.

Maybe this is the JSON? I don't know.

To download Service Definitions the section would look something like this:

I need to update this script to Retry if one fails to download.

Re: Backup AGOL Hosted Feature Layers (aka Hosted Feature Services, aka Hosted Feature Collections)

HarunJoeVellikkara — Thu, 10 Apr 2025 08:48:06 GMT

Thanks @Nick_Creedon for the detailed explanation! Appreciate all your help so far. Cheers!!!🥂