Intermittent failure of Custom Print service

[GinoMellino]

Hi Guys, 

My team administers a number of ArcGIS Server sites with a variety of service types. One particular custom print GP service is showing intermittent failures, seemingly at random; we cannot discern concrete steps to force it to fail every time. This failure occurs when using the service through an OOTB Web AppBuilder widget, or when executing via the rest endpoint. Approximately 50% of the time the job will work, the other half it will fail, with the same input and conditions and in direct succession. There are no messages produced and polling the service endlessly returns esriJobExecuting; i.e., the job doesn't complete, however it is most certainly dead.

The manager UI doesn't really give much useful information, returning only the following messages:

• Execution of job j9f6c3e5919bd45b4bbbe8b640ce510b0 failed. The service was unable to change the status of the job.
• Error executing tool. Export Web Map Job ID: j9f6c3e5919bd45b4bbbe8b640ce510b0

The only other bread crumb trail I have is the messages.xml file in the job folder on the server indicates that the server believes the user cancelled the job (which isn't the case). Message below:

        <GPMessage xsi:type='typens:GPMessage'>

            <MessageType>200</MessageType>

            <MessageCode>0</MessageCode>

            <MessageDesc>Cancelled function</MessageDesc>

        </GPMessage>

        <GPMessage xsi:type='typens:GPMessage'>

            <MessageType>200</MessageType>

            <MessageCode>0</MessageCode>

            <MessageDesc>(ExportWebMap) aborted by User.</MessageDesc>

        </GPMessage>

I have set the server logging level to debug and forced the print job to fail but I cannot see anything in the logs that are of interest (although someone with more knowledge than I may see something more, I can supply logs if need be). I cannot see anything of interest in the Windows Event Logs either. This failure occurs only in the production instance of this service, the instance on the test server works fine. 

Has anyone seen this kind of intermittent failure before? To me it looks like the job has died but the server is not aware of its death status and treats it as executing. I would be very appreciative of any advice on how to fix this or where to go to troubleshoot further for more information on what is going on here? Please let me know if you need any further details, happy to provide what I can. 

Many thanks in advance. 

Gino

Windows Server 2012 R2

ArcGIS Server 10.7.1

Published using ArcGIS Pro 2.8.0 (I note that this is not the most direct version equivalent to Server 10.7.1, however there are numerous other print services which have been deployed with this version and are functioning fine).

[DanielBrumm1]

We actually were running into a similar issue with our print services. I spent weeks troubleshooting and the fix ended up being really strange

In the Print Service log folder : <AGS Host machine> C:\arcgis server\logs\<machine name>\services\utilities\PrintingTools.GPServer

Each print job would create 2 log files which I could open and delete.  However, there was one log file in this directory that was owned by some other account. It was 2kb and I could not open or delete it. I would always get an error saying I was not the owner.  . 

Working with my System Admin he was able to login as this file owner and delete it. The file would come back sometimes and he would have to go in and clear it out. But eventually it stopped appearing. 

Our print services started working 99% of the time following that file removal. 

Not sure how much this helps but I thought I would share. 

Dan 

[GinoMellino · Accepted solution]

Hi Daniel. 

Thanks for sharing and reminding me to update this thread. That is a strange issue.

For us the issue was that one of the servers in the site (this site runs on 2 machines) had a short hostname. We changed it to a fully qualified hostname and the print service hasn't failed since to my knowledge. Unsure why this affected this one particular service and none of the other ones on the site however. 

Cheers, 

Gino