Am wondering if anyone else has come across a situation like this.
We have a python script published as a geoprocessing service. Basically, it takes an id as input, opens an mxd, uses the id to update a definition query, refreshes, exports and returns a pdf. For about a year or so, using ArcGIS server 10.2.2 it worked fairly consistently at about 30 seconds to complete.
Recently, we upgraded the server to 10.3.1. It took us a week or so to realise, but now each time it runs, the script takes longer and longer to complete. Today I inserted some time stamping into the script to save some benchmarking - it seems that every time it runs it takes about 2-3 seconds longer. We started noticing it of course when it was starting to take 5-8 minutes to run!
If we restart the GP service, it goes back to taking 20-30 seconds to run, then starts slowing down from there.
If I run the python script directly in ArcMap on the server it takes about 15-20 seconds consistently and it didn't slow down even after running repeatedly.
I have tried publishing a second GP service from the same script, then run the first one a bunch of times until it slows right down, then run the second one. The second one takes the initial 20 seconds, then slows down over time too.
All this suggests that a) there is nothing wrong with the script and b) that the issue is contained within the GP service and not related to the underlying datasets or database.
If anyone could help shed any light on this it would be much appreciated.