topic Script runs manually but fails in automated task due to directory error in ArcGIS Notebooks Questions

Script runs manually but fails in automated task due to directory error

JiříKomínek — Tue, 29 Apr 2025 06:21:40 GMT

I have a Python script in an ArcGIS Online Jupyter Notebook that processes GeoJSON data and saves a CSV file. It works perfectly when I run it cell-by-cell manually. However, when scheduling it as an automated task (e.g., every 15 minutes), it fails with this error:

OSError: Cannot save file into a non-existent directory: '/arcgis/home/nextbike'

import pandas as pd # Load GeoJSON data features = geojson_data['features'] # Extract desired columns including coordinates data = [] for feature in features: properties = feature['properties'] id = properties.get('id') typ_stojanu = properties.get('typ_stojanu') coordinates = feature['geometry']['coordinates'] x = coordinates[0] y = coordinates[1] data.append({'id': id, 'typ_stojanu': typ_stojanu, 'x': x, 'y': y}) # Convert to DataFrame df = pd.DataFrame(data) # Save to CSV csv_file = r"/arcgis/home/nextbike/filtered_data.csv" df.to_csv(csv_file, index=False)

Problem:

The folder /arcgis/home/nextbike exists when I run the notebook manually.
The automated task fails at df.to_csv() with the directory error, even though the folder appears to exist.

What I’ve Tried:

Confirmed the folder exists manually.
Another script with explicit directory creation works (see below).

Working Example:

import os folder_path = r"/arcgis/home/nabijecky" os.makedirs(folder_path, exist_ok=True) df.to_csv(os.path.join(folder_path, "file.csv"))

Question:
Why does the automated task fail to recognize the directory, and how can I fix this?

Re: Script runs manually but fails in automated task due to directory error

AustinAverill — Tue, 29 Apr 2025 10:44:26 GMT

AGOL Notebooks have always been iffy for me. A couple things you could try -

1. Navigate to the directory using os.chdir() then call the to_csv() method using a relative path. This may work since you have been able to successfully interact with the directory structure using OS.

import pandas as pd import os # Load GeoJSON data features = geojson_data['features'] # Extract desired columns including coordinates data = [] for feature in features: properties = feature['properties'] id = properties.get('id') typ_stojanu = properties.get('typ_stojanu') coordinates = feature['geometry']['coordinates'] x = coordinates[0] y = coordinates[1] data.append({'id': id, 'typ_stojanu': typ_stojanu, 'x': x, 'y': y}) # Convert to DataFrame df = pd.DataFrame(data) # Save to CSV path = "/arcgis/home/nextbike/" os.chdir(path) csv_file = "filtered_data.csv" df.to_csv(csv_file, index=False)

2. Construct the full path using os.path.join(). This should not be functionally any different from how you have hardcoded the full path, but may prevent failure during compiling since it will handle the system preference on path construction.

import pandas as pd import os # Load GeoJSON data features = geojson_data['features'] # Extract desired columns including coordinates data = [] for feature in features: properties = feature['properties'] id = properties.get('id') typ_stojanu = properties.get('typ_stojanu') coordinates = feature['geometry']['coordinates'] x = coordinates[0] y = coordinates[1] data.append({'id': id, 'typ_stojanu': typ_stojanu, 'x': x, 'y': y}) # Convert to DataFrame df = pd.DataFrame(data) # Save to CSV path = "/arcgis/home/nextbike csv_file = "filtered_data.csv" save_path = os.path.join(path, csv_file) df.to_csv(save_path, index=False)

Re: Script runs manually but fails in automated task due to directory error

JakeSkinner — Wed, 30 Apr 2025 12:25:14 GMT

Hi @JiříKomínek when configuring the scheduled task, check on the Use workspace option: