Script runs manually but fails in automated task due to directory error

JiříKomínek

I have a Python script in an ArcGIS Online Jupyter Notebook that processes GeoJSON data and saves a CSV file. It works perfectly when I run it cell-by-cell manually. However, when scheduling it as an automated task (e.g., every 15 minutes), it fails with this error:

 

OSError: Cannot save file into a non-existent directory: '/arcgis/home/nextbike'

import pandas as pd

# Load GeoJSON data
features = geojson_data['features']

# Extract desired columns including coordinates
data = []
for feature in features:
    properties = feature['properties']
    id = properties.get('id')
    typ_stojanu = properties.get('typ_stojanu')
    coordinates = feature['geometry']['coordinates']
    x = coordinates[0]
    y = coordinates[1]
    data.append({'id': id, 'typ_stojanu': typ_stojanu, 'x': x, 'y': y})

# Convert to DataFrame
df = pd.DataFrame(data)

# Save to CSV
csv_file = r"/arcgis/home/nextbike/filtered_data.csv"
df.to_csv(csv_file, index=False)

Problem:

• The folder /arcgis/home/nextbike exists when I run the notebook manually.

• The automated task fails at df.to_csv() with the directory error, even though the folder appears to exist.

What I’ve Tried:

• Confirmed the folder exists manually.

• Another script with explicit directory creation works (see below).

Working Example:

import os

folder_path = r"/arcgis/home/nabijecky"
os.makedirs(folder_path, exist_ok=True)
df.to_csv(os.path.join(folder_path, "file.csv"))

 

Question:
Why does the automated task fail to recognize the directory, and how can I fix this?

AustinAverill

AGOL Notebooks have always been iffy for me. A couple things you could try - 

1. Navigate to the directory using os.chdir() then call the to_csv() method using a relative path. This may work since you have been able to successfully interact with the directory structure using OS. 

import pandas as pd
import os

# Load GeoJSON data
features = geojson_data['features']

# Extract desired columns including coordinates
data = []
for feature in features:
    properties = feature['properties']
    id = properties.get('id')
    typ_stojanu = properties.get('typ_stojanu')
    coordinates = feature['geometry']['coordinates']
    x = coordinates[0]
    y = coordinates[1]
    data.append({'id': id, 'typ_stojanu': typ_stojanu, 'x': x, 'y': y})

# Convert to DataFrame
df = pd.DataFrame(data)

# Save to CSV
path = "/arcgis/home/nextbike/"
os.chdir(path)
csv_file = "filtered_data.csv"
df.to_csv(csv_file, index=False)

2. Construct the full path using os.path.join(). This should not be functionally any different from how you have hardcoded the full path, but may prevent failure during compiling since it will handle the system preference on path construction.

import pandas as pd
import os

# Load GeoJSON data
features = geojson_data['features']

# Extract desired columns including coordinates
data = []
for feature in features:
    properties = feature['properties']
    id = properties.get('id')
    typ_stojanu = properties.get('typ_stojanu')
    coordinates = feature['geometry']['coordinates']
    x = coordinates[0]
    y = coordinates[1]
    data.append({'id': id, 'typ_stojanu': typ_stojanu, 'x': x, 'y': y})

# Convert to DataFrame
df = pd.DataFrame(data)

# Save to CSV
path = "/arcgis/home/nextbike
csv_file = "filtered_data.csv"
save_path = os.path.join(path, csv_file)
df.to_csv(save_path, index=False)