How to automate AGOL backups from a specific online folder

Question

Hello folks,

I need to do batch backups of my online data.  I used this page as a guide to create a script in a notebook:

https://support.esri.com/en/technical-article/000022524

The script works well, it exports an FGDB for each feature layer and downloads it to a local folder.  But I don't necessarily want to back up everything I have online every time I run the script.  I would like to be able to specify just this or that folder to be backed up.  Can anyone suggest how I would adapt the script to do that?  I am totally new to Python so any help will be appreciated.  Thank you!

AndreaF-CO · Answer

Thank you! This script worked like a charm for me! Kudos!

Philip_Ohlinger · Answer

Here is a script I've been using to backup AGOL data. I used GOVAI to help me write this, and it works like a charm. This script creates a log file to keep track of the backups each day, and has some basic error handling.

You need to replace the following in the script with your own variables:

#BASEFOLDERPATH
#PORTAL

#AGOL USERNAME

#AGOL PASSWORD

I wrote this script so it would only look for feature services that I have tagged with "DailyBackup" in my AGOL account. You can change this to look for a different tag by editing the section of code that defiles the tag.

I used Windows Task Scheduler to have this run every day. When you create the task, for the action choose "Start a program", and for the program/script choose the location of your ArcGIS Pro python installation (similar to this: "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe") and then for the "add arguments" put the path to this python script, like this: "C:\Users\username\GIS\Scripts\Python Scripts\AGOL_Daily_Backup.py"

from time import strftime  
from arcgis.gis import GIS  
import logging  
import os

# Initialize logging  
log_file = r"#BASEFOLDERPATH\backup_log.txt"  
logging.basicConfig(filename=log_file, level=logging.INFO)

def log_message(message):  
    logging.info(f"{strftime('%Y-%m-%d %H:%M:%S')} - {message}")

# Input Variables  
base_output_dir = r"#BASEFOLDERPATH"  
date_str = strftime('%Y_%m_%d')

# Create Backup Function  
def backup_feature_services():  
    try:  
        # Log into AGOL  
        gis = GIS('https://#PORTAL.maps.arcgis.com/', '#AGOL USERNAME', '#AGOL PASSWORD')

# Search for items with the tag 'DailyBackup'  
        search_query = "tags:DailyBackup"  
        items = gis.content.search(query=search_query, item_type='Feature Service')

if not items:  
            log_message("No items found with the tag 'DailyBackup'.")
            return

# Create a folder for today's date  
        date_folder_path = os.path.join(base_output_dir, date_str)  
        if not os.path.exists(date_folder_path):  
            os.makedirs(date_folder_path)

for item in items:  
            try:

log_message(f"Processing item: {item.title} (ID: {item.id})")

# Export the item to File Geodatabase  
                tempfile = f"{item.title}_{date_str}"  
                export_item = item.export(tempfile, 'File Geodatabase', parameters=None, wait=True)

# Search and download the exported File Geodatabase  
                myexport = gis.content.search(tempfile, item_type='File Geodatabase')  
                if myexport:  
                    fgdb = gis.content.get(myexport[0].id)  
                    fgdb.download(save_path=date_folder_path)  
                    fgdb.delete()  
                    log_message(f"Successfully backed up: {item.title} to {date_folder_path}")  
                else:  
                    log_message(f"Error: Export for {item.title} not found.")

except Exception as e:  
                log_message(f"Error processing item {item.title}: {str(e)}")

log_message("Script completed successfully.")

except Exception as e:  
        log_message(f"General error: {str(e)}")

# Execute the script  
if __name__ == "__main__":  
    if not os.path.exists(base_output_dir):  
        os.makedirs(base_output_dir)  
    log_message("Backup process started.")  
    backup_feature_services()

Hatsocks · Answer

Ok, thanks for trying! 🙂  I might try generating an Item Report and see if I can identify just the items in a certain folder, and get the ItemIDs from that, but if anyone else has any ideas I'd love to hear them!  Thanks.