Last run result message 0xF6 when trying to run a python script using Windows Task Scheduler.

I ran into the same problem using Pro 2.7.3. Switching from python.exe to pythonw.exe seems to have resolved the issue.

All comments

MDB_GIS

Did you confirm if it was the anti-virus that was causing it to fail?

MatthewStull1

I had a similar issue to this and this is what solved it for me:

My python script runs by itself, not through the Pro scheduler thing, in Windows Task Scheduler. It syncs updates from an enterprise geodatabase to a replica file Geodatabase on another server. My script kept failing with error 000732 (can't find the geodatabase). I fixed it by adding the folder location on the other server where my replica file geodatabase lives to the ArcGIS Server data store. I am running my script on one of my ArcGIS Server servers using the ArcGIS Server python (not Pro). Hope that helps anyone who might be having this or a similar issue.

John:

Is there a chance you could try asking as I have a bunch of exceptions in place in Cisco AMP based on ESRI tech support and forums, but so far failures are still occurring for me?

JohnNergeBrooklynPark

Hi Michael,

I'm the third person on the telephone line (I told my IT, who told our vendor, who made the change and let IT know, who let me know), so I don't have the most comprehensive answer. According to my IT they had to put the server that I run my Python scripts on under the right policy. I assume that policy includes exceptions for what can/can't run on the server, but I don't know for sure. I also would be surprised if my IT were to allow an entire server not to run Cisco AMP, so my guess would be the policy provides more specific exceptions, but again I don't know. Sorry I can't be more helpful.

John:

Your ultimate solution was having a vendor completely disable Cisco Exploit Prevention/Script Protection Engines on your GIS server?

My IT department would never allow that so I was wondering if you were ever able to keep Cisco Exploit Prevention/Script Protection Engines running with some exceptions. I have gathered a bunch of exceptions both from ESRI support and the ESRI forums, none of which have resolved the problem so far.

So this might have solved the problem in Pro 2.7.3 to use pythonw.exe (Not entirely sure it did), but the problem exists again at Pro 2.9.x thru 3.3.x where using pythonw.exe does not resolve the problem. I currently have a Windows 2019 Server where AMP anti-virus software is disabled and the python script is not being skipped. I'm hoping my org can figure how to configure AMP exceptions to get it to stick work without completely turning off AMP anti-virus software which would be a severe security risk.

JohnNergeBrooklynPark

Okay, I finally have some success to report! After playing around with python vs pythonw, tweaking the propy.bat file, and setting up exceptions on them, things still weren't working.

What did finally end up working was disabling Cisco's Exploit Prevention/Script Protection Engines. We have a vendor who manages this for us, so they did this by setting up a custom policy specifically for the server running the Python scheduled tasks.

Been watching our scripts for the last week or so now, and they've been succeeding 100% again.

JakeKnowlesDMM

We were also having these sporadic failed arcpy imports resulting in a 246 error code using a stock arcgispro-py3 environment. Though the pythonw.exe and import looping workarounds did work, we found a permanent solution in adding an exception policy for the environment's python.exe in our server's antivirus software. This resolved all problems with importing arcpy. This might also explain why pythonw.exe worked - since it doesn't spawn a shell, it didn't trigger the same behaviour with our antivirus.

JohnNergeBrooklynPark

I tried my luck with Esri tech support too. Similar response, but at least I got an additional link to share: https://support.esri.com/en/Technical-Article/000012657

My IT team is suspicious that Cisco AMP could be a potential cause, so after a little more troubleshooting I think we're gonna try adding exceptions for the propy.bat file and/or python.exe to run. Will report back findings!

Currently we are at v2.8.8. My recollection is we tried 2.6, 2.7 and 2.8 and they all exhibited the same issue.

Hi Greg,
Would you please let me know what version of Pro you were using?
Tests we just ran were very convincing to our Network guy. He is asking us to upgrade to 2.9.5 (we are at 2.9.1). We don't want to move to 3.x yet.
I am curious which version you are at.
I am also keeping Esri Tech Support informed of the progress because it is truly something to do with "import arcpy" and CiscoAMp/Secure Endpoint (whichever is the real name).
But yes, the intermittency is super baffling to me as well. We have been struggling with this for about 3 months as well.
Thanks,
Carmen

I spent 6~8 weeks trying to figure out what was going on. I never imagined it was AntiVirus in that sometimes it worked, sometimes it didn't. I'm not a security person but if an AntiVirus says this exe is bad, why wouldn't it always be bad?

From Task Scheduler we are executing batch files (.bat) that contain something like this:

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "c:\my_source_code\main.py"

Thus, if one clones environments then the cloned python.exe will be located elsewhere and I presume will need to be whitelisted too. Carmen, I hope it turns out well.

Thank you Greg.

I sent this information to IT and they didn't think this was the issue (those executables being whitelisted).

IT has agreed to do a test disable of CiscoAmp (not sure this is totally correct terminology) for a couple hours while I run my test scripts every 5 minutes during that time. If this is NOT the issue, we should see a failure pretty quickly based on our experience.

I have a call in to Esri tech support as well but we have gotten to the point where they are saying it is out of their hands.

It is so bizarre how it will run fine some times, but not on others for no obvious reason.

Thanks,

Carmen

We have determined that Cisco Amp was interfering with our Python / arcpy applications that were scheduled through Task Scheduler on our Windows Server 2019.
Whitelisting C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe in Cisco Amp corrected the issue.

Continued python script sadness. The above security policy was already set for our service account we use. ☹️

JamesHood

Thanks Carmen,

I'll add that since that change was made and since my post above, our server has run all of its tasks successfully, without fail, for 30 days so. I am satisfied that we resolved the cause of our issue.

James,

Thank you for posting this. I see you added this since I first looked at this post. We have been battling this same exact thing since converting MANY python scripts to 3. The intermittent-ness is driving us nuts. I have submitted to Esri and they are basically saying it is out of their wheelhouse. I ran tests with very basic scripts with nothing but import commands. The one without "arcpy" ran constantly without fail. The one with "import arcpy" ran 2/3 of the time.

I have yet to call IT and have them check the policy, but I want to thank you for updating this post. You have given me one more thing to try...... This has been driving me bonkers. We went from never having to monitor these tasks when using 2.7 to having to access the servers EVERYDAY in task scheduler to determine what didn't run. When you have 20-30 scripts running nightly on a few servers, that is a lot more overhead for supposedly converting to something "better".

I'll repost if this is the solution for us. I'm just happy to have something else to try!

JamesHood

Sorry for the long story here but this is everything I did to trouble shoot and resolve this issue over the course of a long time and I wanted to share in case anybody else is stuck as I was.

Whenever I schedule scripts to run using Windows task Scheduler, sometimes they run and sometimes they do not. It seems intermittent and random for when they would choose to fail. 0xF6 is the “Last Run Result” we would receive in windows task scheduler.

Note that this problem only started after I converted my scripts from python 2.7 to 3.x and started using the pro interpreter. 2.7 never had any problems but I need to switch to 3.x

Note also that the scripts always run no matter what when I manually run them. This is only a scheduled task problem

We are using VM server with windows datacenter 2022 but we were experiencing the same problems on an older VM running an older OS, maybe Windows Server 2016.

We tried switching from python.exe to pythonw.exe did not resolve the issue.

We tried setting up a batch file (.bat) to execute the scripts and then have task scheduler execute the batch file and this had the same result.

I tried making lots of changes in task scheduler like Run with Highest Privileges, run whether user is logged on or not, run only when user is logged in, making sure user is logged in, etc., no combination of these had any effect.

Speaking with Esri support they recommended the following:
1. Import script into .tbx as script tool then schedule to run from within Pro, this will add it to task scheduler anyway but it creates its own task with error logging.

2. Use a .tbx rather than .atbx (since .atbx is new and might have kinks to iron out.

3. Open the task in task scheduler to find the path to the new task.py and open that to see the path to where the logs are created. Should be something like this: "C:\Users\me\AppData\Local\ESRI\ArcGISPro\Geoprocessing\ScheduledTools\NameOfYourProject\task.py"

4. Then you can open the logs to see meaningful errors not just the meaningless 0xf6 in Task Scheduler. With meaningful errors you can work to resolve the issue.

So I did that and it still failed with 0xF6 and I found that in fact the logs were not being created at all which suggested that the pythonw.exe was not even being allowed to start executing since not even an empty log was created.

So we asked IT to check the anti-virus and other security systems on the server that might be preventing it from executing. They found no issues.

Finally, IT checked some local security policies. There’s a local security policy on each server that they set to prevent bad guys from mining bitcoins I guess. They adjusted these to add the “logon as a batch job” permission for my user on that server. Finally the 0xf6 errors stopped and everything runs as it should for the past week.

So hopefully I’m not speaking too soon but this looks resolved for me. I hope this helps somebody else.

FlorianSteiner

We just recently found out that it seems to be related to security features in our business environment. Our IT department uses "Cisco AMP" als malware protection. Temporarily deactivating this application solved the issue so far.

Before that, we used a workaround using batch files to check the error code and running the script again if it failed. Source: https://community.esri.com/t5/arcgis-api-for-python-questions/error-code-246-when-attempting-to-import-arcgis/m-p/1096014/highlight/true#M6598

Hope this helps!

We have run into Task Scheduler issues where sometimes the "import arcpy" statement is crashing Python.exe. This is with Python3. It is not an issue with Python2. I have confirmed the issue on 2 Windows Server 2019 and 4 different Windows 10 computers.

The scripts run just fine from VS Code and the command line. In fact these are not particularly new scripts.

I have one script that runs every 5 minutes. In an hour it might crash 3~4 times on the import arcpy statement.

JeffHaupt

Thanks for that update. I was thinking about opening a ticket, but I'm not sure I will if this is the response that I can likely expect. The interesting thing is that I've noticed this "crash" a few times when I am testing scripts by simply running the script in the command line (no task scheduler), nothing happens for a few seconds and then a new command line appears without the python script running. I have yet to experience this issue running a python script with any standard python library.

FlorianSteiner

I just had a ticket with ESRI regarding this issue. I have the problem even if using pythonw.exe or the scheduling using ArcGIS Pro. Their response was: No support, since a Windows tool ist concerned. My objection, that ArcGIS Pro uses this tool didn't change their optinion. I have to add, that this was a ticket not directly with ESRI Inc. but with "ESRI Deutschland". They were stilll trying to be helpful, which is why I've sent them a small test-script which they were able to run with no problems in ArcGIS Pro and Windows Task Scheduler. There was sadly nothing more they could do.

SStopyak_BruceHarris

I have the same problem. I had mine set to run a bat file that calls propy.bat and the script. I'm going to try the solution of running pythonw instead. I have also noticed that either arcpy or the api fails to import intermittently when running scripts in PyCharm. I think it may be related. One last observation is that scripts seem like they fail when the user gets signed out of Pro. Soon, ArcDesktop's 2.7 will no longer be viable for scripted processes. I'm extremely uneasy about this because using Pro for automation has so many issues right now. I've had automated tasks running in 2.7 without failing for months on end. With Pro, I'm lucky to go a week or two without a failure.

JeffHaupt

I notice this topic keeps getting updated with different users having the same issue and trying different approaches to the problem with mixed results. Has anyone from ESRI ever acknowledged this issue and recommended a certain approach that entities should take to handle the issue of Scheduled Tasks that rely on Arcpy to run?

JaredPilbeam2

@MichaelVolz

My script wouldn't run via Windows Task Scheduler with any of the above suggestions, until I tried running it from Pro's scheduler. I loosely followed this Pro help page: https://pro.arcgis.com/en/pro-app/latest/help/analysis/geoprocessing/basics/schedule-geoprocessing-tools.htm

Basically, I saved my stand-alone script as a script tool with no parameters. Before you hit Run, set up the schedule. I added a bunch of arcpy.AddMessage() print statements so I could see how it was progressing.

As MichaelTorbett said, Pro actually uses Windows Task Scheduler to run the geoprocessing tool. In the Task Scheduler, under the Actions tab, it used the pythonw.exe file as the Program/script and the argument was stuffed away here: "C:\Users\me\AppData\Local\ESRI\ArcGISPro\Geoprocessing\ScheduledTools\WasteRecyclingReport_02012022_0325PM\task.py"

ISP_graynic

I was having this issue and getting this same error code in Task Scheduler but was properly running pythonw.exe as suggested. I found changing the setting back to "Run only when user is logged in", caused it to work. Before I had it set to run even when the user is logged out. It must be some sort of a permissions issue.

I ran into the same problem using Pro 2.7.3. Switching from python.exe to pythonw.exe seems to have resolved the issue.

davehorsey1

Hi all. I also ran into this issue recently, and we couldn't put our finger on why it was failing. We ended up going down the same route as Michael Torbett and noticed that the scheduled task in Pro was using the pythonw.exe, not python.exe. We reverted our original task scheduler task to run pythonw.exe and since then it's been working without any problems.

Hope this helps.