Issues Updating Feature Layer from CSV using ArcGIS API for Python

drewcab — Thu, 31 Oct 2024 19:39:08 GMT

I'm using a Python script with the ArcGIS API to update feature layers from a CSV file stored on a shared drive. The script works as expected for some feature layers, but not others. The strange part is that, regardless of whether the update actually succeeds, it always indicates that the data is updated in ArcGIS Online after running the script. Does anyone have any ideas on how to fix this issue or a more robust solution to complete the same task? Here is the template of the script:

from dotenv import load_dotenv

import os

from arcgis.gis import GIS

from arcgis.features import FeatureLayer

import socket

import pandas as pd

load_dotenv()

def get_internal_network_prefixes(csv_file😞

"""Load internal network prefixes from a CSV file."""

df = pd.read_csv(csv_file)

prefixes = df.iloc[:, 1].astype(str).tolist() # Assuming prefixes are in the second column

return prefixes

def get_current_ip():

"""Retrieve the current machine's IP address."""

hostname = socket.gethostname()

ip_address = socket.gethostbyname(hostname)

print(f"Current IP address: {ip_address}")

return ip_address

def is_internal_network(ip_address, prefixes😞

"""Check if the current IP address belongs to the internal network."""

for prefix in prefixes:

if ip_address.startswith(prefix😞

return True

return False

def overwrite_csv(agol_item, source_file_path😞

"""Overwrite the CSV data for the specified AGOL item."""

try:

agol_item.update({}, data=source_file_path)

print(f"File successfully overwritten for item '{agol_item['title']}' in AGOL.")

except Exception as e:

print(f"Error overwriting item '{agol_item['title']}' with file '{source_file_path}': {e}")

def main(ip_address, prefixes😞

username = os.getenv('GIS_USER')

password = os.getenv('GIS_PASS')

# Check if we're connecting internally or externally

if is_internal_network(ip_address, prefixes😞

print("Internal network detected. Connecting via proxy...")

try:

gis = GIS("https://www.arcgis.com", username, password)

print("Connected to AGOL using the internal proxy.")

except Exception as e:

print(f"Failed to connect via internal proxy: {e}")

return

else:

print("External network detected. Connecting without proxy...")

try:

gis = GIS("https://www.arcgis.com", username, password)

print("Connected to AGOL externally.")

except Exception as e:

print(f"Failed to connect to AGOL externally: {e}")

return

# Define the source directory where CSV files are located

source_directory = 'SHARED/FILE/PATH'

# Define a dictionary mapping AGOL item IDs to corresponding CSV filenames

agol_mapping = {

'AGOL_ITEM_ID':'FILE_NAME.csv'

}

# Process each AGOL item

for agol_item_id, csv_file_name in agol_mapping.items():

print(f"Processing AGOL item ID: {agol_item_id}...")

# Retrieve the AGOL item using its ID

try:

agol_item = gis.content.get(agol_item_id)

if not agol_item:

print(f"AGOL Item with ID {agol_item_id} not found. Skipping.")

continue

except Exception as e:

print(f"Error retrieving AGOL item with ID {agol_item_id}: {e}")

continue

# Verify if the CSV file exists in the source directory

source_file_path = os.path.join(source_directory, csv_file_name)

if not os.path.exists(source_file_path😞

print(f"Source file '{csv_file_name}' not found for item '{agol_item.title}'. Skipping.")

continue

print(f"Found matching file '{csv_file_name}' for AGOL item '{agol_item.title}'.")

print(f"Attempting to overwrite the item '{agol_item.title}' with the file '{csv_file_name}'...")

# Overwrite the AGOL item with the new CSV data

overwrite_csv(agol_item, source_file_path)

# Paths

csv_file = 'PREFIX/FILE/PATH'

# Get internal network prefixes and current IP

internal_network_prefixes = get_internal_network_prefixes(csv_file)

current_ip = get_current_ip()

# Run the main function

if __name__ == "__main__":

print("Starting script...")

main(current_ip, internal_network_prefixes)

print("Script completed.")

Re: Issues Updating Feature Layer from CSV using ArcGIS API for Python

EarlMedina — Mon, 04 Nov 2024 14:05:23 GMT

From past experience, it is far easier and more reliable to do a truncate/add vs overwrite from a CSV. If you search truncate/append you will find numerous examples. Here's one: Solved: What is the recommended method to overwrite featur... - Esri Community

I suspect the problem you are encountering has to do with the format of the CSV data. In many cases, I've observed this behavior when a string exceeds the maximum length of a field - this is enough to derail the entire add operation. To isolate the problem with truncate/add, simply submit edits in small batches until you can narrow down the source.

topic Re: Issues Updating Feature Layer from CSV using ArcGIS API for Python in ArcGIS API for Python Questions

Issues Updating Feature Layer from CSV using ArcGIS API for Python

Re: Issues Updating Feature Layer from CSV using ArcGIS API for Python