topic Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File in ArcGIS Pro Questions

Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

MichaelVolz — Mon, 22 Apr 2024 16:34:58 GMT

I have been running scheduled tasks using Windows Task Scheduler for many years with the ArcMap version of python. Now that my org is migrating to ArcGIS Pro, I am finding that the same bat file is now randomly skipping the ArcGIS Pro python scripts. The bat file runs fine outside of Windows Task Scheduler, so both the python scripts and bat file itself are fine. Sometimes even the first python script in the bat file gets skipped with Windows Task Scheduler, so associating each python script with a bat file will not guarantee success with that setup.

Has anyone encountered an issue like this with a similar setup where you call multiple python scripts sequentially so once one python script has finished running the next python script is then executed?

Does anyone use a different job schedule software other than Windows Task Scheduler to run GIS jobs?

Any help or feedback is greatly appreciated.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

jcarlson — Mon, 22 Apr 2024 17:14:50 GMT

What does the bat file look like? I never used ArcMap, but with Pro, you need to activate the conda environment first, then call the python file, so your bat file should include another line for that.

Esri's docs suggest using a built-in propy.bat script rather than calling python directly.

https://pro.arcgis.com/en/pro-app/latest/arcpy/get-started/using-conda-with-arcgis-pro.htm

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

MichaelVolz — Mon, 22 Apr 2024 17:18:43 GMT

I have already changed the bat file to use propy.bat, but the python scripts still get randomly skipped.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

jcarlson — Mon, 22 Apr 2024 18:48:12 GMT

Curious! So, are you certain they're being skipped, as opposed to erroring out? And the script(s) being skipped aren't consistent? I have had scripts that worked fine as a bat file and separate python files, but would fail in the Task Scheduler. Sometimes the Task Scheduler showed an error, sometimes it claimed the script exited successfully. It wasn't always clear why, unfortunately.

The data involved in our scripts changed daily, and sometimes it was an unexpected datatype conflict between source and destination databases, or the ArcGIS Python API and Pandas implementing datetime values in conflicting ways. May not be the case for your scripts, though.

Do you have any logging going on? I would try starting each script with something like

import logging logging.basicConfig( filename = 'some-log-file.log', level = logging.DEBUG ) logging.debug('Starting some script')

That would at least confirm if the scripts are starting at all.

To answer your other question about alternatives: I got really tired of issues like these, and reworked my scripts so that they would run on a lightweight Linux machine, then added the scripts to the machine's crontab to run at set times.

I don't know what your scripts look like and how feasible such a thing would be, but it was absolutely worth the effort. Haven't had a problem since.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

MichaelVolz — Mon, 22 Apr 2024 19:20:09 GMT

Josh:

Are you sure you need to need to activate the conda environment if you are just using python modules that are packaged with ArcGIS Pro by default?

What modifications would need to be done to the script to run on a Linux machine? Did you need to install ArcGIS Pro onto the Linux machine?

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

David_Lindsey — Mon, 22 Apr 2024 21:10:44 GMT

"I have already changed the bat file to use propy.bat, but the python scripts still get randomly skipped."

This has been an issue for a long time with ArcGIS Pro's python.exe and propy.bat, at least for me. Call the pythonw.exe file instead and you should be okay. It's stable.

python.exe and propy.bat will fail to launch my scripts 60-80 percent of the time. I have had zero issues with calling pythonw.exe though. It's been my workaround for a few years now.

Edit: Just know that pythonw.exe will hide the Command Prompt window. So if you need user interaction with that window or will rely on output messages within it, you'll have problems with this approach. More info here.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

jcarlson — Mon, 22 Apr 2024 20:45:15 GMT

For me, absolutely, but that's because I'm using python for lots of other things, and I've got half a dozen different python envs. If I don't specify which, it just uses the base env, which won't have the modules I need.

It's worth nothing that I primarily use the ArcGIS Python API, not ArcPy, so it's a little easier to use in Linux. Been a while since I tried ArcPy in Linux, but you can download arcpy through the Esri conda channel. Authentication is different w/o Pro installed.

EarthData has a neat approach using Wine: https://www.earthdata.nasa.gov/learn/gis/gis-how-tos/run-arcpy-on-arcgis-linux

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

GavinMcDade — Fri, 26 Apr 2024 15:50:22 GMT

We've tried pythonw.exe, and no dice (I'm Michael Volz's co-conspirator). The forums are replete w/ people complaining about the undue time needed to import the arcpy module (most of it due to needing to make the call to the license server to authenticate the license outside of having Pro opened). The issue is squarely from the Windows Task Scheduler context... that is, something in the way that Task Scheduler executes (even as it's technically going through a .bat file) seems to be evaluating the wait time for the arcpy module to import, and just kills the script if that time value exceeds some threshold. If you manually run import arcpy in a Python window, you'll see varying delay times for it to complete. This would seem to explain why the Task runs sometimes, and skips other times.

I don't know of any setting inside of Task Scheduler that would force it to wait for a "long" process to finish.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

MichaelVolz — Thu, 02 May 2024 16:09:37 GMT

Josh:

At what interval(s) do you need your various GIS scheduled tasks to run at?

I ask because tasks that need to get run once a day seem to be fine, but when a task needs to be run multiple times per day (e.g. hourly) that is when the skipping of python scripts occurs.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

David_Lindsey — Thu, 02 May 2024 17:00:31 GMT

@GavinMcDade- What's your environment setup like? We primarily execute on virtual environments using Windows Server 2016 and 2022. Finally decommissioned Windows Server 2012 because too many oddities were occurring once support got dropped.

For troubleshooting, have you already incorporated PAUSE into any of the .bat files (temporarily) to see if any Command Prompt specific messages are being generated for you before the window closes? Sometimes that has helped me diagnose if those .bat files are even being reached during a workflow.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

jcarlson — Mon, 06 May 2024 14:36:04 GMT

Our scripts run daily at most. Is it possible that a prior run is hung up on something, preventing later runs from kicking off properly? I don't know for sure, but I could imagine that a script utilizing ArcPy might have some constraints with checking out a license?

Did you try adding a log statement at the very beginning of the script?

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

MichaelVolz — Wed, 08 May 2024 20:53:45 GMT

Hi Josh:

When we put a print statement in before import arcpy we get that to be logged, but then the script just exits.

When you clone your python environments, do you use ESRI's interface or do you perform this using Conda command line (A GIS consultant my org works with says that they have encountered issues with ESRI's interface for cloning and now only use Conda command line)?

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

jcarlson — Wed, 08 May 2024 21:01:14 GMT

So the script isn't skipping, but something about importing arcpy throws it off… For debugging purposes, you could try wrapping sections of your code in try / except blocks, maybe? I'm not quite sure how to further diagnose why importing arcpy crashes it out.

Similar to your consultant, we use the conda CLI directly for managing our envs.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

MichaelVolz — Wed, 08 May 2024 21:04:01 GMT

Wrapping in try/except did not capture any faults, as that has already been tested.

Do you have any reasons for not using ESRI's interface to clone your python environment?

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

jcarlson — Wed, 08 May 2024 21:18:18 GMT

Familiarity, for one. I was using python and conda prior to doing any kind of GIS python, so I already knew how to get what I needed.

I also find the CLI to be a lot faster than any conda GUI, especially using mamba.

Third, we've got some python scripts that are totally separate from our Esri / GIS workflows, and there's simply no need to manage the envs for those scripts in Pro. I like managing everything in one place if I can.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

ArcGISDev — Fri, 04 Apr 2025 12:56:09 GMT

I have python 2.7 and 3.x scripts in batch files that run nightly and have had the same issue of sometimes not completing or failing to start.
Most of my issues ended up being locks on the data but the script just exited.

To catch errors in a batch file you can pipe the stderr to a file like this:
rem Execute the Python 3.x script using a command like and catch error:
"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "C:\ArcGisScripts\python3_script.py" 2>> python_log.txt
"C:\Python27\ArcGIS10.8\python.exe" "C:\ArcGisScripts\python27_script.py" 2>> python_log.txt

The "2>>" only pipes out the stderr but if you just want to log whether if was successful or failed then do this:
Add a timestamp to the log to make it easy to see when it started and finished.

echo %date% %time% : Task START Update Python3_Script >> python_log.txt
"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "C:\ArcGisScripts\python3_script.py" && echo Successfully updated! >> python_log.txt || echo ERROR, Did not update! >> python_log.txt
echo %date% %time% : Task END Update Python3_Script >> python_log.txt

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

MikeVolz — Fri, 04 Apr 2025 18:36:22 GMT

The scripts that appeared to be skipping were actually being blocked by anti-virus/anti-malware software on the machine which was rather difficult to isolate as the reason. This is mostly outside of ESRI tech support's scope unfortunately and they were not even able to replicate the issue.

Re: Task Scheduler for GIS Tasks - Skipping Python Scripts Called from Bat File

David_Lindsey — Fri, 18 Apr 2025 03:15:19 GMT

Thanks for that head's up, Mike. I'll test this with my own issues soon. I hadn't considered anti-virus software contributing to that.