Last run result message 0xF6 when trying to run a python script using Windows Task Scheduler.

[MichaelTorbett]

I am running several python scripts on a schedule on a virtual machine that has ArcGIS Pro 2.6.1 installed. I am using the python.exe from my Pro python environment to run all of the scripts in task scheduler. However ever since ArcGIS Pro version 2.5.2, I have been getting the message 0xF6 in the last run results. The crazy thing is that this happens sporadically. Sometimes the tasks run fine and other times they don't. All of my python scripts run fine by themselves. 

Has anyone else had this issue and know of a fix? 

Thanks,

Michael 

[CarmenDurham]

Continued python script sadness.  The above security policy was already set for our service account we use.  ☹️

[GregHorne]

We have determined that Cisco Amp was interfering with our Python / arcpy applications that were scheduled through Task Scheduler on our Windows Server 2019.
Whitelisting C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe in Cisco Amp corrected the issue.

[CarmenDurham]

Thank you Greg.

I sent this information to IT and they didn't think this was the issue (those executables being whitelisted).

IT has agreed to do a test disable of CiscoAmp (not sure this is totally correct terminology) for a couple hours while I run my test scripts every 5 minutes during that time.  If this is NOT the issue, we should see a failure pretty quickly based on our experience.

I have a call in to Esri tech support as well but we have gotten to the point where they are saying it is out of their hands.

It is so bizarre how it will run fine some times, but not on others for no obvious reason.

Thanks,

Carmen

[GregHorne]

I spent 6~8 weeks trying to figure out what was going on.  I never imagined it was AntiVirus in that sometimes it worked, sometimes it didn't.  I'm not a security person but if an AntiVirus says this exe is bad, why wouldn't it always be bad?  

From Task Scheduler we are executing batch files (.bat) that contain something like this:

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "c:\my_source_code\main.py"

Thus, if one clones environments then the cloned python.exe will be located elsewhere and I presume will need to be whitelisted too.  Carmen, I hope it turns out well.

[CarmenDurham]

Hi Greg,
Would you please let me know what version of Pro you were using?
Tests we just ran were very convincing to our Network guy. He is asking us to upgrade to 2.9.5 (we are at 2.9.1). We don't want to move to 3.x yet.
I am curious which version you are at.
I am also keeping Esri Tech Support informed of the progress because it is truly something to do with "import arcpy" and CiscoAMp/Secure Endpoint (whichever is the real name).
But yes, the intermittency is super baffling to me as well. We have been struggling with this for about 3 months as well.
Thanks,
Carmen

[GregHorne]

Currently we are at v2.8.8.  My recollection is we tried 2.6, 2.7 and 2.8 and they all exhibited the same issue.

[JohnNergeBrooklynPark]

I tried my luck with Esri tech support too. Similar response, but at least I got an additional link to share: https://support.esri.com/en/Technical-Article/000012657

My IT team is suspicious that Cisco AMP could be a potential cause, so after a little more troubleshooting I think we're gonna try adding exceptions for the propy.bat file and/or python.exe to run. Will report back findings!

[JakeKnowlesDMM]

We were also having these sporadic failed arcpy imports resulting in a 246 error code using a stock arcgispro-py3 environment. Though the pythonw.exe and import looping workarounds did work, we found a permanent solution in adding an exception policy for the environment's python.exe in our server's antivirus software. This resolved all problems with importing arcpy. This might also explain why pythonw.exe worked - since it doesn't spawn a shell, it didn't trigger the same behaviour with our antivirus.

[JohnNergeBrooklynPark]

Okay, I finally have some success to report! After playing around with python vs pythonw, tweaking the propy.bat file, and setting up exceptions on them, things still weren't working.

What did finally end up working was disabling Cisco's Exploit Prevention/Script Protection Engines. We have a vendor who manages this for us, so they did this by setting up a custom policy specifically for the server running the Python scheduled tasks.

Been watching our scripts for the last week or so now, and they've been succeeding 100% again.

[MichaelVolz]

John:

Your ultimate solution was having a vendor completely disable Cisco Exploit Prevention/Script Protection Engines on your GIS server?

My IT department would never allow that so I was wondering if you were ever able to keep Cisco Exploit Prevention/Script Protection Engines running with some exceptions.  I have gathered a bunch of exceptions both from ESRI support and the ESRI forums, none of which have resolved the problem so far.