topic Re: Does the new Workflow Manager database support a direct database connection at all? in ArcGIS Workflow Manager Questions

Does the new Workflow Manager database support a direct database connection at all?

Phota — Fri, 12 Feb 2021 22:12:13 GMT

Hi all, workflow manager users and the workflow manager team.

Does anyone know if the new workflow manager database (designed for service driven workflows) supports a direct database connection?

I am not meaning connecting to workflow manager, but rather connecting to the database as an eGDB in ArcGIS Pro, ArcMap or any other application that supports direct database connections.

Thank you

Ally

Re: Does the new Workflow Manager database support a direct database connection at all?

JFarmer — Fri, 12 Feb 2021 22:42:28 GMT

Hi Ally,

Can you expand some more on what your end goal is here? Are you trying to setup a workflow where users can open ArcGIS Pro and then work with data from an enterprise geodatabase? Or something different?

Thanks!

Jonathan

Re: Does the new Workflow Manager database support a direct database connection at all?

Phota — Fri, 12 Feb 2021 23:02:47 GMT

Hi John,

My question is around access to the workflow manager database, access to the information within the workflow manager database, treating it like any other RDBMS.

For example,

As the workflow manger classic database is built on an eGDB / RDBMS, I can create a database connection in ArcGIS Pro, for creating query layers across jtx and business tables providing insights, reporting.

Re: Does the new Workflow Manager database support a direct database connection at all?

MJames — Tue, 16 Feb 2021 01:23:42 GMT

I would also like to see a direct connection capability similar to what exists in Workflow Manager (Classic). The Classic version of workflow manager allows us to interrogate our data using an almost endless amount of user interfaces which we have found extremely useful. One specific example is that we have been able to make direct connections to our database using Power BI so our staff can visualise job metrics and create reports in a format that is familiar to them.

Re: Does the new Workflow Manager database support a direct database connection at all?

JFarmer — Tue, 16 Feb 2021 20:35:37 GMT

Thank you both for your comments! A couple of things below and some questions.

To answer your initial question, no we do not support directly connecting to the underlying database in the new services driven architecture. This product utilizes ArcGIS Data Store which we do not support direct connections to like we do for enterprise geodatabases.
However, you can access the data in a couple of ways:
- Each workflow item created creates 3 separate files in your Portal. One of those files is a hosted feature layer that you can use to access all of the underlying tables and data (go to the item page in Portal and click on the Data tab).
- Or, and this would be the recommended way, you can use the Python API to pull data programmatically. More information on the arcgis.gis.workflowmanager module can be found here: https://developers.arcgis.com/python/api-reference/arcgis.gis.workflowmanager.html
We are considering supporting alternate data storage options outside of ArcGIS Data Store that would allow the schema to be created on an enterprise geodatabase but don't have a timeline for this right now.

It would be useful for us to know if the methods of accessing the data mentioned above would work for your use cases? Also, we'd like to learn more about how you are using Power BI as well if you could expand more on that topic.

Thanks!

Jonathan

Re: Does the new Workflow Manager database support a direct database connection at all?

Phota — Tue, 16 Feb 2021 22:06:56 GMT

Thankyou Jonathan,

This response does answer my original question.

However,

I am not too familiar working with feature layers within ArcGIS Pro, I will have to test this to see if I can get the same result.

Working with the python API, I would be able to pull the data out into another database (replicate into an eGDB) resulting in data duplication and static data.

Re: Does the new Workflow Manager database support a direct database connection at all?

JHart — Tue, 16 Feb 2021 22:51:10 GMT

Hi all,

Hearing that the service driven architecture does not currently support eGDBs and is built on the ArcGIS Data Store is a bit disappointing, not allowing nana in the kitchen (ref-data storage strategies).

"We are considering supporting alternate data storage options outside of ArcGIS Data Store that would allow the schema to be created on an enterprise geodatabase but don't have a timeline for this right now."

I think this would definitely be a good idea.

I have included some PowerBI screen shots, as an example only, the information is coming straight from the workflow manager database (using a direct connection to an enterprise geodatabase) and is updated when operators utilise workflow manager.

Re: Does the new Workflow Manager database support a direct database connection at all?

Anonymous User — Wed, 17 Feb 2021 04:31:27 GMT

Hi @JHart,

What's your use case for moving to the new service-driven architecture and timeline for transitioning to it?

There are a few technical reasons why we're currently only supporting the ArcGIS Datastore and a couple of practical reasons as well. It is on our roadmap to support alternate databases, but we haven't heard the request too often. If you just need it for reporting, then PowerBi can use REST calls as data sources as well. You could author a custom search using our API and likely tie that in as your report's datasource.

Thanks for contributing!

Michael

Re: Does the new Workflow Manager database support a direct database connection at all?

JHart — Wed, 17 Feb 2021 07:15:33 GMT

HI Michael thanks for the response and sharing your current reasons.

Please keep us posted on any change in your road map around this, as this maybe a requirement, however it is hard to say at this stage.

At the moment the use case to move to the new service-driven architecture is for non-GIS/Arc Desktop users.

However, the majority of our users are GIS professional and use ArcGIS Pro or ArcMap on a daily basis, making this use case a very low priority.

Future use case is to maintain an ArcGIS Workflow Manager capability, if the current (classic) Workflow Manager is to be retired.

The timeline to transition is not set but I think it will have two driving factors;

The life cycle of the current (classic) workflow manager.
New capabilities/hype of the service-driven architecture creating a demand within our non-GIS users and potentially our GIS users as well.

The move to ArcGIS Datastore would require a large assessment for the business which may affect certain activities such as reporting through PowerBi, SQL (views) queries in the database and custom steps using python.

Acknowledge that PowerBi can use REST calls and there is a Python API for the new architecture, both maybe worthwhile substitutes, there could well be a large body of work ahead to transition.

Finally, I think it is worth noting the current (classic) Workflow Manager is working well for us; the majority of the workforce are using Workflow Manger in ArcGIS Pro for workflow manager activates, and get their insights and reports through ArcGIS Pro, PowerBi, ArcGIS Dashboards with additionally send gdb cuts (derived feature classes) from the workflow manager database. The workforce are using the workflow manager more and more as time goes on requiring numerus extended and related job properties with additional business tables in the database.

Josh

Re: Does the new Workflow Manager database support a direct database connection at all?

Anonymous User — Wed, 17 Feb 2021 18:27:55 GMT

@JHart If you have existing Pro Workflow Manager licenses, you will start to receive user type extension licenses for the new service driven architecture automatically for each Pro license. This is to facilitate conversion of a Classic system to the new service-oriented, but you could use them in the short-term to flesh out the implementation and requirements for these non-GIS use cases and use it as a spring board to learn the technology and its advantages.

I'm glad to hear you're enjoying Workflow Manager (classic) and that it is so successful in your organization. If you were interested we'd love to have a case study recorded on your experiences integrating PowerBi etc.

Take care,

Michael

Re: Does the new Workflow Manager database support a direct database connection at all?

SU00861536 — Sun, 07 Aug 2022 06:55:19 GMT

Hi @JFarmer

As per my requirement want to get the job sequence id before creating the job, because it helps integration of workflow manager job with other modules of applications. I tried the available rest api but not able to get the next job sequence.

Also I want to know is any way to reset the sequence...

Re: Does the new Workflow Manager database support a direct database connection at all?

JFarmer — Mon, 08 Aug 2022 20:49:34 GMT

@SU00861536

Thanks for the information on your requirements for getting the next Job ID in the sequence. We don't expose a way to get the next Job ID in our APIs currently.

There is also not a way to reset the sequence. This is because we keep track of all job activity, including deleted jobs. So we need to keep the sequence going to properly track that history. The only option here currently would be to create a new Workflow Item.

But we'll consider these things as we move forward so thanks again for mentioning them.

Jonathan