Workflow Manager Pane - Create Version is stuck on Running

RichardOxales · ‎10-18-2023

I'm deploying Workflow Manager onto a QA environment and encountering issues. I used some Esri-provided scripts to migrate my Workflow items in a dev environment to a QA environment and apart from a couple of manual updates to things like Job Name format, everything else seemed to copy over just fine.

On the QA environment, whenever a user creates a job through the Workflow Manager Pane in ArcGIS Pro, the Workflow Pane shows the job as Running, but it never gets out of this state. It's supposed to transition to an Open Pro Project Item to open a Map and point the version to the newly created version.

I can correct this by clicking on 'Switch active Workflow Item', the Cloud button in the bottom left of the panel, and re-entering the Workflow Item. It will have shown to properly enter the next step, opens the created version and it runs as normal. However, having to back out of the Workflow item and re-open it for every job isn't ideal for the users, as it should automatically transition to the next step (which is set to run Automatically when entered).

I have not encountered this issue on the dev environment, and verified the Workflow Job Diagrams and settings in WFM are the same.

My Dev environment has ArcGIS Pro 3.1.2 and my QA environment has ArcGIS Pro 3.1.1. Both use WFM 11.1.

Any ideas on where the problem might be?

In searching around, I'm guessing maybe this patch could work but I'm not sure otherwise what the issue could be.

https://support.esri.com/en-us/patches-updates/2023/arcgis-workflow-manager-server-11-1-patch-1

RichardOxales · ‎10-24-2023

Quick update. The environment I'm working in has been updated to ArcGIS Pro 3.1.3 and has implemented the linked WFM Server patch.

The patch didn't seem to have a meaningful effect. The best way I found to make the system run a bit more smoothly was to remove the "Manage Job - Basics" default permission set from the users with the most problems, leaving them with "Manage Job - Advanced" permissions. This gets the jobs I have running a bit better when starting up, but the card list in the Workflow Manager Pane still noticeably doesn't update sometimes to reflect the new status. I have to force a refresh by either changing the filters at the top of the pane (e.g. move from Custom, to My Jobs, back to Custom) or clicking the 'Switch active Workflow item'. Doing this however results in the job entering a state where the 'Resume' and 'Pause' button appears and also breaks the flow I have with multiple steps set to run Automatically.

JFarmer · ‎10-25-2023

Hi @RichardOxales

Thanks for the post and update.

I just want to confirm the core issue you are seeing here. From your screenshot, you have a Create Version step and that step seems to stay in this running state and not finish. If you exit the item and return, the step then shows as finished and the job card displays the next step? Is it only this step? Do you see the same thing in the web application in this QA env. or just in Pro? I'm also a little surprised changing roles would impact this.

The patch you referenced addressed an issue where things out of sync behind the scenes and jobs would get stuck in a running state. This would happen both in Pro and the web application. However, in that case, there was no way out. Exiting the workflow item and returning would not have fixed that issue. So I think your issue is something different (but still good to run the latest patches anyway).

Lastly, the issue you mentioned where the job card shows Resume and Pause in Pro on steps set to run automatically. This is a known issue and we will work on addressing it soon, though I can't provide a Pro release where this will be addressed just yet. It's a high priority issue for us though.

Jonathan

RichardOxales · ‎11-29-2023

@JFarmer Hi Jonathan,

Been busy with other things on my project and only just getting back to this. I've attached a log file where I opened ArcGIS Pro, started a few new jobs, and had the last one in my list (IGIS As-Built Services 48) get stuck in a "Starting" loop (same as my image but replace the word "Running" with "Starting". I created 46 and 47 just before and did not have this issue. Changing between the Workflow Pane's filters (e.g. click My Jobs then click Custom) refreshes the card to instead show the 'Resume' button.

The effect seems to be mostly visual in the Pane since it corrects itself when the cards are forced to update, but it's still an annoyance.

RichardOxales · ‎11-30-2023

Here's another log I have with a more straightforward, repeatable behavior I've found. I go to Create a Job, select my HGIS Platting workflow, which should transition automatically from Start -> Update Job Properties, both of which are set to automatically run. Instead of opening Update Job Properties and allowing me to start typing in my job properties, the card wants me to first click 'Resume', though I can proceed normally from there.

This currently happens on all jobs I make in my dev environment.

JFarmer · ‎11-30-2023

Hi @RichardOxales

OK, based on the behavior you're mentioning here, this sounds like a bug we're looking into now. I'd anticipate we'd have this work done for the Pro 3.3 release but it's still on-going. Unfortunately, we don't have a workaround for this outside of the steps you are currently taking to move forward on your jobs now.

Jonathan

RichardOxales · ‎10-25-2023

@JFarmer Hi Jonathan,

Thanks for the response. Initially, the Create Version step presented the most problems as it was the starting point on many of my job diagrams. I'm not sure why updating permissions helped to fix its perpetual running state but it noticeably ran better and I just rolled with it after spending a day working through it. Create Version wasn't the only step that was hanging for me. I attached an image showing one of my job diagrams and highlighted the steps which sometimes don't properly start. These are using the Step templates like Update Job Properties and Open Pro Project Items (opens a map in Pro).

The users for WFM will be working primarily in Pro but the job statuses do update properly on the WFM web application.

From your answer, it sounds like my best course of action is to make certain steps manual instead of automatic? Virtually all of my steps in my job diagrams are automatic at the moment.

JFarmer · ‎10-25-2023

Thanks @RichardOxales

To your last point, you can leave them automatic. The Resume/Pause issue isn't consistent and you'll see steps run automatically as expected. Even if they don't, choosing the Resume option would be the same as manually starting the step from an end user standpoint so it makes sense to leave them as is.

For the running issue, sounds like maybe this is Pro specific. In which case, some logs would help. This article goes into how to set them up: https://support.esri.com/en-us/knowledge-base/how-to-set-up-logging-for-arcgis-workflow-manager-in-a.... You can attach them here, though, it's possible they don't provide a ton of insight and then next steps would be creating a case with Tech Support for some one-on-one help. If you'd rather do that instead of continuing here, that's fine as well.

Jonathan

RichardOxales · ‎10-25-2023

@JFarmer I'm looking at the link to set up logging for WFM in Pro, but at Step 4, it doesn't mention what to rename the string entry to for the log file location. What should it be renamed to?

JFarmer · ‎10-25-2023

@RichardOxales

Hmm, you're right. Thanks for mentioning it, I'll make sure the right people know so we can get that fixed. You should name it LogsPath. Should look like my entry attached when you're done.