Best Practices for working with Utility Network: Publishing Utility Network Service
Authors: @MichaelHirschheimer , @Saranya-Kesavan
Over the past few years, Mike and I have published multiple Utility Network services in an ArcGIS Enterprise environment. Over the course, we have made silly mistakes as we relied too much on muscle memory and felt the pain of having to redo work. We’ve also learned about some simple tweaks to the publishing map that resulted in overall performance improvements. In this blog we will share our wisdom gained from those experiences and provide you with some preventative strategies to publish a Utility Network service quickly.
Utility Network Foundational solutions
One of the key benefits of Utility Network is the industry standard data model that comes with it. Starting with Esri’s Utility Network Foundation solution, organizations do not have to spend their time and implementation budget on building custom data models and information products for their network management in GIS (but are welcome to configure the data model as needed, to fit their specific needs).
Industry specific Utility Network Foundations can be readily downloaded from the ArcGIS Solutions website.
These foundation solutions include a data model that configures ArcGIS Enterprise and the ArcGIS Utility Network to represent <Industry> networks. It deploys foundational services, simplifying the creation of additional web and mobile apps and provides a desktop data editing environment. A collection of tasks, with step-by-step directions, are also provided to help deploy the solution to an Enterprise environment.
Pertaining to those tasks, we will be sharing our wisdom in the following topics
- Importance of the publishing map
- Feature layers in the publishing map
- Optimal visible scale considerations for feature layers
- Linked Web Maps
- Maximum number of coded domains
- Max Heap Size
- Troubleshooting version management related issues
- Troubleshooting register data source issues
- Backing up ArcGIS Pro project
Importance of the publishing map
Update: Please read this related article for a more comprehensive discussion of the best practices for managing the typical set of maps and services used to support a Utility Network implementation.
It will take multiple maps to support your Utility Network implementation. There will be a map specific to publishing the Utility Network (called Electric Utility Network in the Foundation project. See #1 in screenshot below). Each layer’s data source in the publishing map will come from a database connection and there are specific requirements that must be met. Once published, you can create editing and viewing maps using subtype group layers (see #2 and #3 in the screenshot below). Their data source will be from a featureservice which will allow you to access the Utility Network in mobile and web applications. More information to come on editing and viewing maps in future blogs.
Feature layers in the publishing map
At Esri, we recognized that the number of visible layers with visible content can have an impact on performance. Querying each layer in the map resulted in multiple requests being sent and responses being received from the database or service. In the case of Utility Network feature dataset, the dilemma of whether to add all the objects to a publishing map is understandable, considering the sheer number of it. But our experts vouch for better performance if all the objects in the Utility Network feature dataset are added to the publishing map.
Here’s why -- When a user is editing data from an ArcGIS Server published service, each update is a separate edit operation. This means when you change the attribute of a single feature, that update is a single edit transaction, which requires locking all the objects associated with the object being edited. So, if you update a single attribute on 10 different features, one at the time, that is 10 edit transactions. On top of it, if an object in the feature dataset is not in the map, then additional database queries need to be made. This will result in a performance hit in ArcGIS Pro.
Another tip here -- if a feature class participates in a relationship class, add the related object to the map. Because, when the feature class is queried/edited, ArcGIS Pro will try to open the related object. If it is not on the map, then additional database queries need to be made. To simplify this querying process and thereby to optimize performance it is advised to add the related object to the publishing map.
While adding multiple layers in the map helps to optimize overall performance with feature edits, one might experience slowness when the map is being rendered with all the layers. This slowness can be minimized by turning off the visibility for those layers that need not be seen on the map. This will come in handy when you pan and zoom in your publishing map. You can also choose to pause map rendering altogether if you do not want to visualize your data in the publishing map.
Finally, if you use landbase along with Utility Network, think about end-user workflows. If landbase edits need to be seen in the same version where facilities are being edited (for example, in a new Underground Residential Development, new roads are being added), then the landbase layers should be in the same service. So, add the landbase to your Utility Network feature service publishing map. On the other hand, if landbase and Utility Network are in separate feature services, landbase edits would be in its own version and the facility edits would be in a different version.
Linked Web Maps
In ArcGIS Pro, there are no capabilities to overwrite an existing Web Map in Portal. However, when you add a map to your project from Portal it becomes a linked web map which you can then use to overwrite the existing web map.
This is an important workflow for when you have built applications that reference that web map, since this saves you from having to reconfigure any downstream applications to point at the new map. However, you should exercise caution when making changes to existing web maps, especially adding or removing layers, since they may impact configurations in downstream applications.
To add a linked web map to your project, use the following steps:
- In ArcGIS Pro, open the Catalog pane
- Navigate to the Portal tab on the Catalog pane
- Select the Web Map in your portal
- Right click on Web Map > and select Add and Open
- Once the map is loaded, notice the “link” icon on the web map. This indicates it’s linked from your Portal
- Make your desired changes
- Go to the Share tab on the Ribbon
- Click the Save Web Map button in the Manage group
Note: The Save Web Map button is only enabled when a "linked" Web Map is the active map
The Web Map in your portal will now reflect the changes you made in ArcGIS Pro.
If you save this project you can continue to make and republish changes using this linked Web Map
Note: Your local copy will not reflect changes made by other users, and they will be lost if you overwrite the Web Map with your local copy. The Save Web Map has an option to create a backup of the web map before overwriting, you can refer back to this copy if you need to revert changes.
Optimal visible scale range considerations for feature layers
It is a best practice to determine the scale range at which individual feature layers will be rendered. Failing to do so will result in performance issues when the map is being rendered.
For example, if the Line feature layer is set to be visible at the county level, there could be millions of objects that are rendered and that takes a considerable amount of processing and time. Despite that time and effort, end users cannot perceive any meaningful information. Similarly, a point layer that is visible at all scales will not mean anything when the map is zoomed in at the city level. So, pay attention to your workflows and determine the appropriate rendering scales for each layer. This will dramatically reduce the load when the map is rendering.
Setting a visible scale range at the feature layer level will ensure that these visibility scale ranges are passed to all the maps downstream that will be using the feature service. Depending on the feature service clients, you could fine tune the scale ranges as needed.
Maximum number of coded domains
Another setting that is sometimes forgotten is the maximum number of coded domains. The default value of 25,000 may not be large enough for a Utility Network. Change the value to 1 million: 1000000
But why? Let’s do the math. There are at least 12 database objects in your Utility Network. As an example, the Electric Utility Network model contains an average of 38 fields and 13 subtypes in each object. Suddenly the number creeps to a range closer to 25,000. If you choose to enhance the data model, the number of coded domains can exceed more than the default.
Remember, with the default setting, 25,000 is the maximum number of domain codes that can be returned from all fields, subtypes, layers & tables. When the number of domain codes returned by the MaxDomainCodeCount property exceeds the default value, the service may continue to run normally, but you will see the domain descriptions not displaying as expected for some attributes. In addition, errors will be captured in your ArcGIS Server logs.
So, remember to increase the maximum number of coded domains in the configuration tab while sharing the web layer. Share Tab > Web Layer > Publish Web Layer > Configuration tab > Click the Feature’s pencil icon > Configuration Parameters tab > tip open Advanced
You can also edit it with the ArcGIS Server manager, as shown below.
Max Heap Size
Customers who have very large subnetworks may encounter errors when trying to trace or update them. If you encounter one of the following errors when running Trace or Update Subnetwork: “Error 160213 or Error 003642: A networking error occurred,” you should consider adjusting the max heap size of your utility network feature service (for synchronous operations) and the utility network tools geoprocessing service (for asynchronous operations).
Please refer to this KB article: Determine the proper heap size setting for large trace results in an enterprise deployment of the ut.... While it is possible to set the Max Heap Size at a server level, this will increase the memory usage of all services in that size. Because of this, it is recommended that you take a more targeted approach and only increase the Max Heap Size for service that need it, and that you only make it as large as it needs to be. Increasing the Max Heap Size increases the memory consumption of each SOC used by the server, so setting the Max Heap Size to be 512mb will cause each SOC to consume an additional 512mb of memory even when idle.
When updating a feature service, you must place the javaHeapSize in the properties element of the Feature Service's service properties. This property is not exposed by default in the service, so if this is your first time adjusting this value you will need to add a new property.
When updating the UtilityNetworkTools GP service you must place this javaHeapSize in the frameworkProperties element of the GP Service's service properties. This property is not exposed by default in the service, so if this is your first time adjusting this value you will need to add a new property.
Troubleshooting version management related issues
To publish a Utility Network Service, all feature layers and tables that are added to the publishing map must be registered as a branched version. An efficient way to find out the publishing map’s conformance to this is by inspecting the Configuration tab: Share Tab > Web Layer > Publish Web Layer > Configuration tab
If you see an option to enable “Version Management” capability in the Share As Web Layer pane, you are all set. This confirms that all the feature layers are registered as branched versions in your publishing map.
CAUTION: Do not spend any time changing symbology, renaming layer names or other map changes, before checking that version management capability is present in the configuration tab. This could be all for nothing if you can’t enable Version Management.
Sometimes, things do not go as planned/expected. Without the Version Management capability, Utility Network functionality won’t be available upon publishing. Troubleshooting steps include:
- Make sure the layers added to the map come from a Geodatabase connection file whose versioning type is Branch. You can check it here: Catalog > Right click on Geodatabase connection file > Geodatabase Connection Properties
- Inspect each layer in the map and confirm its versioning type is Branch. (Right click on a feature layer > Properties > Source)
- If the layer is not flagged as branched in the properties dialog, remove it from the map and add it again using the branched geodatabase connection file.
Troubleshooting register data source issues
When you analyze the map to publish your Utility Network Service for the first time, you are most likely to encounter an error indicating that the “Layer’s data source must be registered with the server.”
An effortless way to navigate this error is to right click on error message for one of those layers and choose to register data source with server. You will be prompted to add your data store to your server.
Upon successfully resolving the error for one layer, the other layers will be automatically updated. Click Analyze in the Share as Web Layer pane to make sure all the errors are cleared out.
Backing up ArcGIS Pro project
Upon successful publishing of the Utility Network service, it is always a good idea to back up the ArcGIS Pro project so that you have a copy of all the configurations & the database connection files. This will come in handy when you need to make changes to the existing service, have multiple administrators managing the Utility Network environment, when upgrading the enterprise version or when gearing up for creating another service.
There are different ways you could back up your ArcGIS Pro project: saving the project from your local machine to a network drive or uploading it to your portal. If you choose to upload to portal, you can share it with others for collaboration. Also, you can use groups to ensure that only a certain member of staff can access it. There is also an option to mark that project as Authoritative which indicates it’s a trustworthy project to use.
Conclusion
The purpose of this blog is to share the knowledge we’ve gained from our firsthand experiences on publishing a performance optimized Utility Network service. We hope our insights save you time by avoiding common pitfalls. The items discussed can be considered as best practices to follow along with the tasks you will receive as part of Utility Network Foundation solution.
We look forward to sharing more of our findings in our upcoming blogs. Stay tuned!
About the Authors
Mike & Saranya are Solution Engineers on the Esri Infrastructure Team with a focus on the Utility Network. Between the two of us, we have 10+ years of applying ArcGIS to utilities, and we are continuously excited to work with customers to improve their experience.
While building a 10.8.1 environment and exercising Utility Network workflows, we’ve accumulated a wealth of knowledge that will aid in maintaining your System of Record. We look forward to sharing what we’ve learned with you in a series of blog posts. These blogs could be considered as best practices for Utility Network.
Mike Hirschheimer is a solution engineer on the Utilities Team with a focus on the Utility Network. He is a big sports fan who enjoys watching NFL & NBA games and playing disc golf.
Saranya Kesavan
Saranya Kesavan is a solution engineer on the Utilities Team with a focus on the Utility Network. Outside of work, she is an avid reader and a big-time motorcycle enthusiast.