Job stuck in running state

[MargoHuang]

Hi all,

We have a workflow manager item deployed on our Enterprise 11.3 environment back in May and it has been running fine. Starting on 11/4, we have had 2 jobs stucked in running mode on 2 different steps. The job has a spinning icon right next to it, and there is no button listed under Actions that are clickable. Since it is running, I can't do anything to it. 

Has anyone experienced similar issue? Is there a way to get a job out of a running state?

I have also noticed some error messages in our server log that might or might not be related to this issue. 

On our WFM server, I can see errors like this - 

And at the same time, on our ArcGIS Server Manager, it fires dozens of warnings like these within a second - 

I saw these warnings when a person clicked on 'proceed' and it got stuck. But most of the time, the operation runs through even though these warnings are genereated.

Another thing I noticed is - when I check on History in the Detail pane for a specific job, there are steps that say running even though they ran successfully. This happens in our closed jobs as well. 

Any insights are appreciated. 

Thanks!

Margo

[CodyPatterson]

Hey @MargoHuang 

This  looks like some sort of token or user permissions that could have corrupted the job while it was running. Would it be possible to log onto the server where Workflow Manager is installed and restart the Workflow Manager process? It may clear up the job queue, you could also consider restart the ArcGIS Server during a maintenance window or downtime.

Cody

[MargoHuang]

Hi Cody,

Thanks for the quick response! I just tried restarting ArcGIS Server service and ArcGIS Workflow Manager Server service on the WFM server machine. And the two jobs are still hanging. I will let you know if restarting ArcGIS Server fixes it.

Thanks!

Margo

[CodyPatterson]

Hey @MargoHuang 

Thanks for the update! You could also try checking the Jobs folder of the ArcGIS Server itself, personally once I start digging around in the file system of the server, I usually loop Esri Support in just incase I may damage a part of the environment!

Cody

[anayoung]

We are experiencing the same issue and plan to restart the all services this weekend during maintenance including WMX. Fingers crossed!

[anayoung]

Restarting the service did not help - the job is still stuck.

[MargoHuang]

Hey Ana,

Are you able to submit a tech support ticket? I was able to work with an ESRI person to get my job out of the running state. It evolved deleting some records in the backend, which has to be done extremely carefully. I would recommend working with someone from ESRI and preferrably  a person who is famillar with WFM on that.

Margo

[anayoung]

Hi Margo,

Yes, we might have to do that since our tech guy says the API trick would require us to install a tool that we'll be deprecating soon anyway. Thank you for posting this issue.

[anayoung]

Our tech guy did find a way to get the job unstuck:

I found the brute force method.

On the above:

1. Note the Workflow manager hash. This is the hash used in the hosted feature layer for the Workflow.
2. Find your Job Id and select it.
3. Use the hamburger menu and select “Show Job ID”
4. Note the Job ID for the next step.

Find the hosted feature layer for the workflow using the Workflow Manager hash from the URL [Not Job ID]

1. Click on the “Data” Tab.
2. Change the layer to “jobProgress”
3. Sort the Table [I used Descending]
4. Locate the Job ID and find the bottom-most one listed [Upper-most if you Sorted Ascending!!!!!!]
5. Change the “stepAssignedTo” to you or whoever will pause and change steps.
6. Have that person go into their Jobs as they are now the step owner.