Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

[MichaelVolz]

I have been running scheduled tasks using Windows Task Scheduler for many years with the ArcMap version of python.  Now that my org is migrating to ArcGIS Pro, I am finding that the same bat file is now randomly skipping the ArcGIS Pro python scripts.  The bat file runs fine outside of Windows Task Scheduler, so both the python scripts and bat file itself are fine.  Sometimes even the first python script in the bat file gets skipped with Windows Task Scheduler, so associating each python script with a bat file will not guarantee success with that setup.

Has anyone encountered an issue like this with a similar setup where you call multiple python scripts sequentially so once one python script has finished running the next python script is then executed?

Does anyone use a different job schedule software other than Windows Task Scheduler to run GIS jobs?

Any help or feedback is greatly appreciated.

[jcarlson]

What does the bat file look like? I never used ArcMap, but with Pro, you need to activate the conda environment first, then call the python file, so your bat file should include another line for that.

Esri's docs suggest using a built-in propy.bat script rather than calling python directly.

https://pro.arcgis.com/en/pro-app/latest/arcpy/get-started/using-conda-with-arcgis-pro.htm

[MichaelVolz]

I have already changed the bat file to use propy.bat, but the python scripts still get randomly skipped.

[jcarlson]

Curious! So, are you certain they're being skipped, as opposed to erroring out? And the script(s) being skipped aren't consistent? I have had scripts that worked fine as a bat file and separate python files, but would fail in the Task Scheduler. Sometimes the Task Scheduler showed an error, sometimes it claimed the script exited successfully. It wasn't always clear why, unfortunately.

The data involved in our scripts changed daily, and sometimes it was an unexpected datatype conflict between source and destination databases, or the ArcGIS Python API and Pandas implementing datetime values in conflicting ways. May not be the case for your scripts, though.

Do you have any logging going on? I would try starting each script with something like

import logging

logging.basicConfig(
    filename = 'some-log-file.log',
    level = logging.DEBUG
)

logging.debug('Starting some script')

That would at least confirm if the scripts are starting at all.

To answer your other question about alternatives: I got really tired of issues like these, and reworked my scripts so that they would run on a lightweight Linux machine, then added the scripts to the machine's crontab to run at set times.

I don't know what your scripts look like and how feasible such a thing would be, but it was absolutely worth the effort. Haven't had a problem since.

[MichaelVolz]

Josh:

Are you sure you need to need to activate the conda environment if you are just using python modules that are packaged with ArcGIS Pro by default?

What modifications would need to be done to the script to run on a Linux machine?  Did you need to install ArcGIS Pro onto the Linux machine?

[jcarlson]

For me, absolutely, but that's because I'm using python for lots of other things, and I've got half a dozen different python envs. If I don't specify which, it just uses the base env, which won't have the modules I need.

It's worth nothing that I primarily use the ArcGIS Python API, not ArcPy, so it's a little easier to use in Linux. Been a while since I tried ArcPy in Linux, but you can download arcpy through the Esri conda channel. Authentication is different w/o Pro installed.

EarthData has a neat approach using Wine: https://www.earthdata.nasa.gov/learn/gis/gis-how-tos/run-arcpy-on-arcgis-linux

[MichaelVolz]

Josh:

At what interval(s) do you need your various GIS scheduled tasks to run at?

I ask because tasks that need to get run once a day seem to be fine, but when a task needs to be run multiple times per day (e.g. hourly) that is when the skipping of python scripts occurs.

[anonymous_geographer]

"I have already changed the bat file to use propy.bat, but the python scripts still get randomly skipped."

This has been an issue for a long time with ArcGIS Pro's python.exe and propy.bat, at least for me. Call the pythonw.exe file instead and you should be okay. It's stable.

python.exe and propy.bat will fail to launch my scripts 60-80 percent of the time. I have had zero issues with calling pythonw.exe though. It's been my workaround for a few years now.

Edit: Just know that pythonw.exe will hide the Command Prompt window. So if you need user interaction with that window or will rely on output messages within it, you'll have problems with this approach. More info here.

[GavinMcDade]

We've tried pythonw.exe, and no dice (I'm Michael Volz's co-conspirator). The forums are replete w/ people complaining about the undue time needed to import the arcpy module (most of it due to needing to make the call to the license server to authenticate the license outside of having Pro opened). The issue is squarely from the Windows Task Scheduler context... that is, something in the way that Task Scheduler executes (even as it's technically going through a .bat file) seems to be evaluating the wait time for the arcpy module to import, and just kills the script if that time value exceeds some threshold. If you manually run import arcpy in a Python window, you'll see varying delay times for it to complete. This would seem to explain why the Task runs sometimes, and skips other times. 

I don't know of any setting inside of Task Scheduler that would force it to wait for a "long" process to finish. 

[anonymous_geographer]

@GavinMcDade- What's your environment setup like? We primarily execute on virtual environments using Windows Server 2016 and 2022. Finally decommissioned Windows Server 2012 because too many oddities were occurring once support got dropped.

For troubleshooting, have you already incorporated PAUSE into any of the .bat files (temporarily) to see if any Command Prompt specific messages are being generated for you before the window closes? Sometimes that has helped me diagnose if those .bat files are even being reached during a workflow.