Danger Will Robinson! Your Pro Python Code is Wrecking the Spaceship!
For years, I have with great success been updating my AGOL feature services with on-prem data using https://www.esri.com/arcgis-blog/products/analytics/analytics/updating-your-hosted-feature-services-... as a nightly scheduled task. Emphasis on years. Great tool Kevin Hibma ! In my work flow, for folks needing to make a "Map" (not edit), we push them to use AGOL hosted feature services which allow me to dial down the number of service instances I need to maintain on my GIS server farm, which I limit to just "Editors". In this scenario, any edits the editors make will show up in the mappers maps the next day. We have a few scenarios where that doesn't work obviously, so I spawn more instances by default to support the mappers. I control all of this with AD groups driving the permissions on the GIS servers, as well as the Portal (no federation!). This allows me to not need to deploy about 20 GIS servers, as we have discovered there is a finite number of service instances a single server can push, despite memory and CPU.
Until this past weekend. The server this was running on, as a scheduled task using a domain service account, went on a protest strike. 2 days of troubleshooting narrowed it down to an issue with the local profile of the service account, which we could no longer troubleshoot, as we couldn't even get access to the profile, not even with Powershell. Long story short, here's what was happening.
Python runs at 1910:
Success is written to a log
But....
A massive plethora of junk ends up in C:\Users\[domainsvcaccount]\AppData\Local\Temp!
8 items to be exact. Times 100-ish hosted feature services being updated daily. Times 2 years.
What had happened is the local\temp folder filled up with several hundred million of these things, several hundred gb worth. It simply blew up the OS. I was able to clear it out using Snake Oil.
It's well known that clearing local\temp solves many ESRI problems, and we do so frequently with desktop. But this is the first time I've seen the problem using Python.
So the warning here is, if you're a heavy user of https://www.esri.com/arcgis-blog/products/analytics/analytics/updating-your-hosted-feature-services-... , make sure to automate cleaning C:\Users\[domainsvcaccount]\AppData\Local\Temp.
As a side note, I'm curious (concerned) as to why Py is throwing that much stuff in the temp folder, but like I said, automated clean up in the solution here.
and thanks for the advice!