Community
Select to view content in your preferred language

Lessons learnt overwriting many feature services in ArcGIS Online

205
0
a month ago
LindsayRaabe_FPCWA
by MVP Regular Contributor
MVP Regular Contributor
5 0 205
a month ago

Preamble

In our organisation, we've been using ArcGIS Online since the early days after it's initial release. This means we have many layers that have been around for quite some time. One thing we never really understood early on, and have found to be problematic in the last couple of years, is not having well considered and consistent spatial references for all our feature services. 

Whilst the default spatial reference in ArcGIS Online WebMaps is WGS 1984 Web Mercator (auxilary sphere), feature services can be published using any coordinate system ArcGIS Online then projects them on-the-fly in webmaps. This usually is not an issue. We have found it though to be problematic when using the layers in ArcGIS Pro, and when there are a mix of spatial references that use both GDA94 and GDA2020. This mix was introducing data shifts that could not be dealt with through transformation selections.

We have since decided the most robust way forward would be to republish everything in ArcGIS Online to use the same projection: WGS 1984 Web Mercator. This way, when loaded into a ArcGIS Pro project, we don't have to worry about incompatibility between GDA94 and 2020 being an issue any more. 

NOTE: We still have many non- AGOL datasets that need to be reprojected as well, but thats a much simpler task. 

Considerations

As I'm sure many could appreciate, making significant changes to many feature services is no small feat. This meant scheduling a freeze period where users would not be able to capture or edit data without risking that data being lost. It also meant that all offline areas in Field Maps would need to be recreated (which is quite a lot in our org)! We estimated it would take our team 3 days to republish our feature services , check them then update impacted WebMaps & ArcGIS Pro templates.

We had 102 feature services in total that needed their spatial reference updated. These then had another 120 associated hosted views that needed to be considered. We also wanted to preserve the Editor Tracking history in all our layers (something easily lost without proper consideration). 

You may have seen recently issues around Editor Tracking fields and new default high-precision date fields in ArcGIS Online (Solved: Editor Tracking – High Precision Date Fields issue... - Esri Community). To mitigate these issues and speed up the process as much as possible, I built a model builder tool that would download each sub-layer/table in a feature service and recreate the Editor Tracking fields, preserving the creation/editor history and enabling editor tracking on the local (temp) data prior to publishing. This speeds up 's Solution in the above post. These tools are attached for download (one for feature services with and without tracking and one for tables with tracking). 

NOTE: It looks for the default Creator/CreationDate/Editor/EditDate fields created in ArcGIS Online. If your fields vary from this, the model will need to be modified to recognise these. 

Model to download feature service and maintain editor trackingModel to download feature service and maintain editor tracking

 

 

Process

ArcGIS Pro Setup

  1. Open new ArcGIS Pro project and insert a new map
  2. Check the “Enable assignment of Unique ID’s” option in the Map Properties > General window
  3. Ensure map projection is set to WGS 1984 Web Mercator (auxiliary sphere)
  • Consider using the setting Project > Options > Map and Scene > Spatial Reference > “Choose spatial reference” instead of “Use spatial reference of first operational layer” to prevent the map changing to a different spatial reference when adding data

LindsayRaabe_FPCWA_1-1739512029941.png

Layer Setup

  1. Add hosted feature service to ArcGIS Pro from Portal
  2. Run “Download FS with Editor Tracking” tool (W:\Mapping.atbx)

    NOTE: Select the input feature service by browsing through Portal – don’t select the layer in the map. Ensure you select the sub-layer so that the URL has the layer ID number at the end (e.g. /0)

    LindsayRaabe_FPCWA_4-1739512668023.png

    • If the tool fails, it is due to the layer not having matching Editor Tracking fields. Run the version of the tool called “Download FS without Editor Tracking” or update the tool to match your editor tracking field names. 
    • If a feature service has more than one sub-layer, run this process for each sublayer.
  3. Right click feature service layers you added to the map in Step 1 > Copy then right click the corresponding layer/sublayer output from Step 2 > Paste Properties. Repeat for each downloaded (temp) sublayer until all are are done. 

    LindsayRaabe_FPCWA_6-1739513733704.png
  4. Rename the layers to match the existing feature service sublayers. 
  5. Update the new layer properties to match the feature services ID (number) at the end of the service URL (data source) – ensure you do this last as “Paste Properties” will change this

    image.png

     

  6. Remove the layer added in Step 1 (existing feature service) leaving only your temporary data
  7. Confirm your map projection is still the desired spatial reference

Updating Feature Service

  1. Go to ArcGIS Online and disable Sync on any Hosted Views (and the base if applicable) – all replicas need to be deregistered or the next step won’t be able to proceed
  2. Go to the Share ribbon > Web Layer > Overwrite Web Layer

    LindsayRaabe_FPCWA_7-1739514150978.png
  3. Select the original feature service and click OK
  4. In the Configuration section, ensure you check the “Preserve editor tracking info” (unless no editor tracking was found when downloading the feature service)
  5. Update the Time Zones (if required)

    LindsayRaabe_FPCWA_8-1739514378553.png

     

  6. Analyse the layer for any errors and rectify as required
  7. Publish the feature service

    NOTE: If the republish fails, check that you didn't miss any Sync options on Views. If you can't get it to publish successfully, you may need to publish the layer as a V2 feature service and then change the data source of any associated Views to the new feature service. See "Learnings" below for more details. 
  8. Check feature service and all hosted views to ensure that they work (check the Data tab in ArcGIS Online).
    • If you find any views where the Data tab shows “Data Error”, try republishing the layer again (this will likely fix the issue).
    • Also ensure you check that you can select a feature in your feature service View attribute tables and zoom to the features. We found occasionally a view appeared to be working (showed records) but no features were visible in the map. See "Learnings" below for more details. 
  9. Reenable Sync on any Views

Learnings

  • When overwriting a feature service, you get a warning window that states you may lose all of it's associated data and popups. Data should be retained through the download process undertaken at the beginning. Popups and other styling (symbology, labels, etc) should also be retained through the Copy > Paste Properties steps undertaken earlier. This warning does NOT affect any views of the feature service. They will remain as they were before the layer was overwritten. 

LindsayRaabe_FPCWA_9-1739515105640.png

 

  • As mentioned in Step 8 above, sometimes we successfully overwrote a feature service, but a view may fail to reflect the updates, returning a Data Error on the Data tab of the items page on ArcGIS Online Item page. 

    LindsayRaabe_FPCWA_12-1739515780036.png

    Sometimes clicking Publish in ArcGIS Pro again would fix this. If you tried the Publish step again and you still can’t get all Views working, you may need to recreate the view manually and replace it in all WebMaps and ArcGIS Pro templates that it exists in. We found this was easiest done with the old one and new one side-by-side on screen so you can step through the defined view settings (Settings > Update View), Visualisation and Settings pages.

     LindsayRaabe_FPCWA_13-1739516177789.png

  • We found that some of our feature services simply refused to be overwritten. These ones we had to publish again from scratch (with "V2" appended to the name - see screenshots below). It's important to remember that you can't change the source of a hosted view to another feature service unless they have the same spatial reference as the original, and all the required fields are present (amongst other things - further reading here: Swap the source of a hosted feature layer view—ArcGIS Online Help | Documentation). Because we were going through this process in order to update spatial references, this meant publishing the V2 layer with the original spatial reference (e.g. GDA94 MGA Zone 50), changing the sources of all the views to the V2 feature service, and then we would go through the above republish (overwrite) process again on the V2 layer to update it's spatial reference (to WGS 1984 Web Mercator) with much more success. 

    LindsayRaabe_FPCWA_10-1739515325059.png 
    LindsayRaabe_FPCWA_11-1739515402405.png

 

Summary

Despite a slow start as we ironed out the kinks in our process and ran at 2/3 team capacity for a while, we soon got on a roll and were churning through layers at a good pace. It certainly helps having someone involved who is intimately knowledgeable about the data being worked with as you can soon land in big mess if you misunderstand the relationships between hosted feature services, their views and the map/app products they exist in (not to mention the potential complications that related tables and other such things bring to the party - thankfully we only had 1 of those). 

Overall, it took us 3 days to:

  • Republish (or recreate if necessary) all 100 of the target feature services
  • Check that they (and their views) were functional
  • Check WebMaps and ArcGIS Pro templates for broken data sources and update them
  • Ensure that all offline enabled WebMaps were still ready to work offline (and fix if required)

Beyond the 3 day "freeze" period, a few small oversights were found and were fixed (thankfully nothing major) and Experiences were updated to reflect changed data sources (resulting in broken expressions). 

FeatureServiceDownloadToolbox.zip
About the Author
After leaving TAFE as a Mining & Engineering Surveyor, I got my first real taste of GIS at the Forest Products Commission. After a year, an economic downturn resulted in me getting a job doing FIFO at a BHP mine site as a Mining Surveyor. 3.5 years of that was enough and a spot opened up at the FPC again and I've been there ever since (10+ years now). I enjoy my work managing and developing FPC's GIS, and being a part of a great team where we create tools and workflows that make the lives of our office based and field staff easier. Using ArcGIS Online, ArcGIS Pro, Field Maps, Survey123, GeoMaster and ArcMap, we aim to improve data flows to and from the field by improving the way data is collected, displayed, used and shared within the organisation and with our peers in government and private enterprise.