Python script to export to FGDB only works for some online layers

[CiaraMcGlade]

I have the following script which I use to export online layers to File Geodatabases in a folder on my computer.

import arcgis
from arcgis.gis import GIS
from datetime import datetime
from time import strftime
import os
import itertools

# Backup Script
for (a, b) in zip(item_ids, folder_list):
    folderlocation = mainfolder + "/" + b
    itemid = a
    print(itemid, folderlocation)
    
    #create a backup online
    dataitem = gis.content.get(itemid) #fetch the item to be backed-up
    backupname = f"{dataitem.title}_Backup_"+ date_time #create a name for the backup using the existing item name
    dataitem.export(backupname, itemtype, parameters=None, wait=True, tags=itemtags, snippet = itemdesc) #export the data to a file geodatabase within the user's content

    #create a backup online
    dataitem = gis.content.get(itemid) #fetch the item to be backed-up
    backupname = f"{dataitem.title}_Backup_"+ date_time #create a name for the backup using the existing item name
    dataitem.export(backupname, itemtype, parameters=None, wait=True, tags=itemtags, snippet = itemdesc) #export the data to a file geodatabase within the user's content

    ## Move the backup to designated folder, if not root
    searchquery = f"title:{backupname} AND owner:{username}" # search query using item title and owner
    searchresult = gis.content.search(backupname, item_type=itemtype) #find the backup that was just created
    backupitem = gis.content.get(searchresult[0].itemid) #get the itemid of that new item
    print(backupitem)

    # download backup and delete online file
    backupitem.download(save_path=folderlocation, file_name= backupname + ".zip")
    backupitem.delete()

    # extract zipped folders
    zippath = folderlocation + "/"+ backupname + ".zip"
    extractpath = folderlocation + "/"+ backupname
    if not os.path.exists(extractpath):
        os.makedirs(extractpath)
    import zipfile
    with zipfile.ZipFile(zippath, 'r') as zip_ref:
        zip_ref.extractall(extractpath)
    os.remove(zippath)

This script works fine for 3 feature layers that I have online, but not for another 4. It seems that the problem lies with the data.export part of the script, which for these 4 layers exports an empty file geodatabase. I've attached a screenshot of one of the attribute table of one of these layers to show the issue - the data schema is exported but none of the data actually within the layer.

Basically, I am just wondering what could prevent the data from being exported - I have checked the settings on the layers and can't see anything different from the ones that successfully exported.

Thank you

[MobiusSnake]

Are the four services that fail larger than the other three?  I've had issues with the export() call once a service contains too much data.  What I've seen is a bit different though, typically I end up with a very small, corrupted FGDB that can't be opened in Pro, rather than one with empty tables.

[MichaelVolz]

Ciara:

At my org, we have found some polyline feature layers in AGOL do not get successfully exported to a file gdb.  The problematic polyline feature layers do not have any attachments.

I would try to simplify the feature layers as much as possible to see if you can get the export to work and then go step by step make the feature layer more complicated until you find the root cause.

[CiaraMcGlade]

No, they are quite mixed in terms of size and also whether there are attachments. The smallest one fails to export, as does the largest. I think there must be something in terms of how the data was created initially, I might try republishing some of the non-working ones to see if this helps, although it isn't obvious from any of the settings what the root cause is.

[JakeSkinner]

@CiaraMcGlade are you able to export the problematic feature services manually to a File Geodatabase using AGOL?

[CiaraMcGlade]

Yes, I was doing the export manually, and decided to try and wrap it all in a script so I have exported successfully using the button on the overview page.

[CiaraMcGlade]

Apologies, I actually was doing the manual download under a different account - however the account I use to run the script has admin access, could it be something to do with ownership of the data?

[Nick_Creedon]

Yes, ownership does take into effect. The layers must have export enabled in their setting, if you are not the owner and not a admin. As Administrator your role overrules the need to have export enabled.

[Nick_Creedon]

This isn't the solution for everyone, but I use AI when I am not finding the answers I am looking for. When doing this, read the script carefully and fine tune it. You can also work with AI to talk through what it is doing vs what you want it to do. I copied your code, and this is what it spit out with notes:

import arcgis
from arcgis.gis import GIS
from datetime import datetime
import os
import zipfile

# Assuming these variables are defined elsewhere in your script or environment
# mainfolder, item_ids, folder_list, itemtype, itemtags, itemdesc, username

# Backup Script
gis = GIS("home")  # Connect to the ArcGIS Online or Portal
date_time = datetime.now().strftime("%Y%m%d_%H%M%S")  # Format current date and time

for item_id, folder_name in zip(item_ids, folder_list):
    folder_location = os.path.join(mainfolder, folder_name)
    print(f"Processing item: {item_id} into folder: {folder_location}")

    try:
        # Fetch the item to be backed up
        data_item = gis.content.get(item_id)
        backup_name = f"{data_item.title}_Backup_{date_time}"

        # Export the data to a file geodatabase within the user's content
        backup_item = data_item.export(backup_name, itemtype, parameters=None, wait=True, tags=itemtags, snippet=itemdesc)

        # Search for the newly created backup item
        search_query = f"title:{backup_name} AND owner:{username}"
        search_result = gis.content.search(search_query, item_type=itemtype)
        
        if search_result:
            backup_item = search_result[0]
            print(f"Backup item found: {backup_item.title}")

            # Download backup
            zip_file_path = os.path.join(folder_location, f"{backup_name}.zip")
            backup_item.download(save_path=folder_location, file_name=f"{backup_name}.zip")

            # Extract the zip file
            extract_path = os.path.join(folder_location, backup_name)
            if not os.path.exists(extract_path):
                os.makedirs(extract_path)
            with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
                zip_ref.extractall(extract_path)
            
            # Remove the zip file after extraction
            os.remove(zip_file_path)

            # Delete the online backup item after downloading
            backup_item.delete()
            print(f"Backup completed and cleaned up for {data_item.title}")

        else:
            print(f"Backup item not found for {data_item.title}")

    except Exception as e:
        print(f"An error occurred while backing up {item_id}: {str(e)}")

[Nick_Creedon]

Here is what I currently use for backups (just ran it this morning), I use a specific Tag for the layers I want to back up, GDBWeeklyBackUp is my current tag. This runs through all of the hosted feature layers in our organization and exports the tagged layers. I am an administrator, so I have access to export any layer, that is also something to keep in mind. If you're not an Admin, you will have to make sure that the layers you do not own have export enabled in their layer settings.

import time
from arcgis.gis import GIS
import os
import datetime
from tqdm import tqdm
import logging

# Setup logging
logging.basicConfig(filename='backup_error.log', level=logging.ERROR)

# Constants for rate limiting
DOWNLOAD_DELAY = 1  # seconds between each download

def authenticate():
    return GIS("home")

def setup_directory():
    now = datetime.datetime.now()
    folderName = now.strftime("%Y-%m-%d")
    parent_dir = os.getenv('BACKUP_PATH', r"YOUR FILE PATH HERE")
    path = os.path.join(parent_dir, folderName)
    if not os.path.isdir(path):
        os.mkdir(path)
        print(f"Directory '{folderName}' created")
    return path

def download_with_retry(gis, item, path, downloadFormat='File Geodatabase', max_retries=3, delay=5):
    for attempt in range(max_retries):
        try:
            result = item.export(f'{item.title}', downloadFormat)
            result.download(path)
            result.delete()
            print(f"Successfully downloaded: {item.title}")
            return True
        except Exception as e:
            logging.error(f"Attempt {attempt + 1} failed for {item.title}: {e}")
            if attempt < max_retries - 1:  # Not the last attempt
                print(f"Retrying download for {item.title} in {delay} seconds...")
                time.sleep(delay)
            else:
                print(f"Failed to download {item.title} after {max_retries} attempts.")
                return False

def download_items(gis, path, downloadFormat='File Geodatabase'):
    download_items = gis.content.search(query="tags: 'GDBWeeklyBackUp'", item_type='Feature Layer', max_items=-1)
    if not download_items:
        print("No items found with the specified tag.")
        return

    for item in tqdm(download_items, desc="Downloading Items"):
        if item.type == 'Feature Service':
            if not download_with_retry(gis, item, path, downloadFormat):
                logging.error(f"Download permanently failed for {item.title}")
            time.sleep(DOWNLOAD_DELAY)  # Wait before the next download

# Main execution
gis = authenticate()
download_path = setup_directory()
download_items(gis, download_path)