Approval step for online edits

[ChrisMcLean2]

The online editing (feature access) web service is a great tool for users that do not have a knowledge of the ArcGIS software, but are required to update features on a regular basis. The question is how does one maintain the spatial data quality of the edits, before rolling the data out to a wider audience? 

The editor tracking functionality is great to see who create or modified elements, however there is no step for the approval of those edits. The edits are immediately available for users with view access to make decisions. In order to maintain the integrity of the data, it would be great if an approval step could be integrated with the layer. Where the owner of the data is required to approve the changes before the rest of the organization is able to view the changes.  

A potential solution for this would be to only update the Web Map Service (WMS) using the Feature Service when the changes to the Feature Service have been approved by the data owner. At the moment the WMS is updated as soon as the Feature Service. It should be possible to disconnect these to service and only update manually when approved.

[ThomasEdghill]

Thank you for this idea!

Addressing this idea will be different depending on the type of Feature Service being leveraged. For example, if your Feature Service is based on data coming from a registered Enterprise Geodatabase, it is possible to utilize versioning to create a version for online edits, and share a Feature Service which is referencing this version. This allows for some level of data validation at the database level, where the edits sent to the aforementioned version can undergo QA/QC before being synced back to a parent version.

While the versioning option could be seen as a workaround to this, it does not address Feature Services which use nonversioned archiving; nor does it address Feature Services which are shared to ArcGIS Online or Portal for ArcGIS via a Hosting Server.

While we investigate this idea further, I'd like to understand your workflows here - are you typically using a Feature Service based in an Enterprise Geodatabase? If so, have you explored the option of using versioned data to see if it may meet your needs, or are there other limitations which you are facing?

[ChrisMcLean2]

Hi Thomas,

Thank you for the reply. 

Yes, we are using a Feature service hosted in an Enterprise Geodatabase. The Portal is federated with the server, and the same server acts as the Hosting Server. The editing of features is online, so I'm not sure if the versioning is built for this. 

The overview:

Everything takes place within the Portal Web App environment. Our users are not able to use ArcGIS Desktop to view or approve data. The reason for this model is essentially to mitigate the change management and training required. We do not have a GIS department - there are two of us implementing the Enterprise GIS solution to 3000 employees. We have integrated all the current source systems that they use (mostly CAD and Excel) to the Server database and from there push and pull data. The main purpose for the GIS solution is to allow all the users to have access to view the information. With that comes issues, whereas previously maps were printed and signed as an approval process, now currently the information is available for all users over the web (Portal) to see and make business decisions. The viewing users need to be confident that the data they see is the most up-to-date and has been approved by the responsible person. 

The business requirements:

1) The editor of the data (either in the Source system or Portal online) may not be the approver of the data. 

               - Current solution: AD groups to control access to editor and approver web maps, some SQL

2) The approver requires functionality to see the previous version and the changes made quickly

              - Current solution: Editor tracking, time slider and comparison screen, colour coding

3) The approver must be notified and a system in place to track edits

             - Current solution: SQL back-end, SharePoint front-end

After this Idea post, I worked a little more on the problem and came up with the following work around:

The current solution uses the Editor Tracking functionality to track the edits, there is a field called "Approval" that the editor does not have access to change. The default for this field is "Pending", so any new edits are automatically assigned as "Pending". The approver only have access to the "Pending" field and can not update geometry or change other fields, he/she can change the "Pending" to "Approved" (this is a domain) - either one feature at a time or as a batch (web app tool). Then to display the feature I used some colour changes to make everything that is "Pending" grey for the approver and all the other viewers. Once this changes to "Approved" it adopts the correct colour and line styles. 

The issues:

If the feature is "Approved", and the editor makes changes is does not change the field back to "Pending". This is where some SQL comes in and checks to see if the last_editor field (editor tracking) is equal to the approver user name. If it is then those features are set to "Approved", if it is not then the feature was edited and the field is set to "Pending". At the same time the information is posted to SharePoint for the approver user to get a notification. 

The unfixable (at the moment):

I can not seem to figure out a way to publish a feature service that is symbolized using two fields. For example, the symbology field (what ever that might be) and the Approval field. This would make it a lot easier to manage this process. And I know that I can concatenate the fields together to make and Symbology-Approval field, but then the editor user have the option to immediately draw a feature that will be approved. 

Next steps:

Offline Editing on Web? Offline Editing - COP     GitHub - Esri/offline-editor-js: ArcGIS JavaScript library for handling offline editing and tiling. - but this is not something I'm familiar with implementing

Looking forward to your response,

Chris

[ThomasEdghill]

Hi Chris,

In reading through your overview and current way of tackling these workflows, it sounds like you may still be in a position where versioning could be implemented at the Enterprise Geodatabase level and may be beneficial. As long as the data is still stored in a registered Enterprise Geodatabase, and is not copied into a managed database (such as the ArcGIS Data Store), then the workflow linked in my comment may still be applicable.

Here is an additional resource which outlines the ways that versioned data can be implemented in workflows involving offline edits. The core principle of maintaining a version for field edits which can be posted and reconciled with a parent version following data validation is still applicable here, even if the maps won't be going offline.

If it works for you, I'll reach out via a private message so that we can discuss some of the finer details about your project there and hopefully help to understand your workflow further.

Aside from this, the workflow which you have detailed above is a clever solution, and the idea posting does raise some interesting use-cases where approval processes for online edits are needed. The current "unfixable" which you have referred to is an idea which currently has a corresponding posting on the Ideas site, so you are certainly not alone in this request:  

Publish service that is symbolized on multiple attributes‌

I recommend that you upvote this idea if you have not already done so, so that it gains some more traction on this site. 

Best,

Thomas

[ChrisMcLean2]

Update - and review:

Thanks Thomas, I got the versioning sorted out and it does work very well – there should be a topic on the help desk to be able to do this. It was quite different compared to collector type versioning setup.

The technical setup:

1. Versioning the database with all the bells and whistles – GUID, Editor Tracking

2. Created a child version of the DEFAULT – I called it “Editing”

3. Registered the Enterprise Geodatabase while connected to the “DEFAULT” version

4. Registered it again, but this time connected to the “Editing” version

5. Published a feature service of the DEFUALT version (with sync disabled)

6. Published another feature service while connected to the “Editing” version (also with sync disabled)

7. Published a Geoprocessing Service to reconcile the “Editing” version to the “DEFAULT” version

The results:

1. The editors can now go into the feature service published under the Editing version and make changes

2. The approvers see those changes immediately – and on their Web App have both the DEFAULT and the Editing versions with a swiper and slightly different colours to see what changed

3. In the approver web app I added the GP service to reconcile, so when they click on that they “approve” the changes

4. The viewers see the map service of the DEFAULT version – so they can’t edit and only see what has been approved

5. It works very well, but brings with it its own set of challenges

The challenges:

1. The entire Enterprise Geodatabase is reconciled 
   • But this can be fixed by adding multiple versions for each feature class and publishing GP services to onoly reconcile those versions - so just a lot of admin
2. You will be registering many Enterprise Geodatabase instances, one for each version
3. Additional load on the server to spool the extra services
4. The approver must wither approve all changes, or reject all. They are not able to select certain entities and only approve those.
5. Unknowns... I'm not sure how conflicts will be handled, or if strange things will start to happen.

The summary:

• Three different techniques identified
  1. Versioning
     • The method described above 
  2. Approval Field
     • Colour changes to show which feature entities are still Pending
  3. Query Layers based on the Approval field
     • Not discussed in this post, but possible if you don't want the normal viewer users to see the "Pending" changes at all before they are approved
• Each have their own pro's and con's, and by having all at your disposal it's possible to meet the many approval requirements.

Please add your comments and your ideas. Hopefully, this will be a built in function one day without the complexities of the technical setup. 

All the best and good luck,

Chris

[Kevin_MacLeod]

Versioning could be a great QA system.  As it stands, it's dangerous, unless you really know what you are doing. Particularly for places that collect and change data often in a field data version. 

Perhaps some of the ideas that follows here could be in Data Reviewer.  But that as it stands is more for checking data against rules that users develop and enforce, like topology or data ranges in fields, etc.  But Versioning should really be simple.  And more user-friendly to avoid errors, perhaps a way of previewing what will happen after a post. (Differences pane... isn't the best. I want to see the attribute table, before/after, as a preview)

I should be able to pick what layers sync and do not sync.  I should be able to then pick which features sync or do not sync, based on either spatial and/or SQL selection.  Thirdly, third level of inception, is I should be able to pick what fields in a particular record of a particular layer sync.

And finally, roll back. It should have a roll back feature, at least if Archiving is enabled. I don't see why it couldn't.  Yea, we all know, SQL backup is the best method.  Some devs (myself) do not have access unfortunately to the SQL Server.  For a variety of reasons, it would make sense to have a simple roll-back. Frankly, a simple roll back to any particular point in time for the entire database; or again, per layer, if Archiving is enabled.  

Data Management team.... thoughts?  I have only used Traditional versioning, note.  Particularly with something this mission critical, I do not want to start using a new thing (like branched versioning) until it's been out a decade. That is often how long bugs live.  There are some advantages I see with Branch, particularly no more darn compressing, and some point soon I will look at migrating to it, maybe upon the next full stack upgrade cycle (SDE, Enterprise, Portal, OS) but not yet.

Chris, that's a very, very cool idea to offload the data QA and even the mechanics of post and rec, to users.  Frankly, we in GIS often get roped into certifying the data itself is correct. Which is dangerous, since we are (not always) not engineers, surveyors, biologists, or whatever it may be, who actually generated the data.  But again, the if users could approve and post and rec just particular layers, or even features / fields, that would be powerful.  Basically, versioning could be used for QA QC.   I am well aware of Differences and Conflicts panels.  However, with the changes above, they would be come far more powerful and useful.  I think most people starting to use versions just assume any difference is a conflict.  The way conflict panel works is nice, but it only works for "conflicts". I would want it to (optionally) work for differences too, to choose which to check in and approve, which to reject and which to modify.