Solution tools to help with data modeling and data migration
- Creating Utility Network Copy Workbook
- Apply Utility Network Copy Workbook
- Summarize Utility Network Errors
- Sync C_Tables Asset Group/Type
- Assign Terminal Connections
- Conclusion
Are you looking to do a utility network implementation, and want to know some of the tricks of the trade to make the process easier? Or maybe you have previously implemented a utility network, and are looking for ways to streamline your processes? Either way this article will show you the five most important tools provided by the ArcGIS Solutions team to help with all the configuration and quality assurance that goes into each data migration, most of which you’ve probably never heard of!
All the tools in this article are either part of the Utility Network Package toolbox or the Utility Data Management Support tools. The UN Package tools are included with the utility network data models provided by the ArcGIS Solutions team. The Utility Data Management Support tools can be downloaded, along with their online help document, through their GitHub page.
You can watch the records of the ArcGIS Utility Network: Utility Network Foundations presentation from the 2024 Esri User Conference for a discussion of how to use many of these tools.
The tools we will be discussing in this article are:
- Create Utility Network Copy Workbook
- Apply Utility Network Copy Workbook
- Summarize Utility Network Errors
- Sync C_Tables
- Assign Terminal Connections
Creating Utility Network Copy Workbook
Most data migration projects make use of a Utility Network Foundation. These are data models for different industries that come pre-configured for use with ArcGIS Utility Network. Most of these foundations have several variations. There is an Expanded model, that includes all the different configurations for many different workflows, but there may be other Essential models that include a subset of the model tailored for different companies in the industry (distribution, transmission, etc) or even a simpler that is designed to be expanded on.
This raises the natural question, what do you do if you start with an Essentials model and want to borrow some of the elements from an Expanded model? The answer is you use the Create/Apply Utility Network Copy Workbook tools. The first tool, Create Utility Network Copy Workbook, lets you pick two utility network foundations. The source asset package should be the model that contains all the configuration, the Expanded model. The target asset package should be the model you want to import the configuration into, your Essentials model.
Once you run the tool it will create a workbook that shows all the asset types in the source model that are available or you to import into your target model. In this example we want to import the service meters from the Expanded model into our Essentials model.
The screenshot below shows the asset types for service meters from the expanded model:
The following screenshot shows the asset types for service meters in the essentials model:
To do this we open the Utility Network Copy Workbook created by the tool and look for rows that define the Service Meters.
Once you know which asset types you want to copy, you populate the To Source/AssetGroupName/AssetTypeName columns in the spreadsheet. This will define which asset group(s) the asset types will be added to, and what they will be named in your model after copying. You only need to populate the To AssetGroupCode/AssetTypeCode in the spreadsheet if there are specific codes you want to use in your target Asset Package. Otherwise, Asset Package.
When these asset types are copied into your target model, all the configuration and rules associated with them will also be copied. If you only want to copy some of the configuration, you can use the right most columns to control what you do and do not want to copy with each asset type.
In this example we want to add the detailed Service Meter asset types to our model, so we would populate the spreadsheet as follows:
We will discuss what happens when you run the tool in the next section.
Apply Utility Network Copy Workbook
Once you’ve created and populated a Utility Network Copy Workbook, the next thing you need to do is apply it to your target Asset Package. To do this you run the Apply Utility Network Copy Workbook tool.
In this section we will continue our example from the previous section, where we are copying the detailed Service Connection asset types from the Expanded Model into the Essentials model. After running the tool against the populated Utility Network Copy Workbook, we can verify the changes in our asset package.
The Asset Type domain has been updated to include the new asset types. We can see that the existing asset type (originally Service Meter) is still in place, and new asset types have been added to the domain.
We can also verify the other configuration was copied as well by looking at the various B_* tables in the asset package. In the example below we can see that new rules have been added to the model for the added asset types.
If we look closely at the rules, we will see that no junction-edge rules have been imported for the new Service Meters. This is because the Expanded model has more detailed asset types for Service Lines that we didn’t import.
There are two ways we can address this:
First, if we want to keep the single asset type in the Essentials model, we will need to manually configure rules for the new Service Meters.
Otherwise, if we want to have more granular asset types for Service lines in our target asset model, we can add those to our model as well. This will then allow the Service Meters to connect to the Service Lines. However, we will now have a problem that the imported Service Lines won’t have rules for all the Service Valves, Fittings, etc in the simpler model.
This shows an important consideration when copying information from an Expanded model into an Essentials model. Because the tool will only import configuration for the asset types that exist in both models, you will always need to manually adjust some of the configuration to incorporate it into your model.
Summarize Utility Network Errors
The quality assurance provided by Network Rules and Feature Restrictions of the utility network allow you to quickly find data quality issues that are meaningful for your industry. While the utility network includes tools designed to help you identify and resolve tools on a row-by-row basis, using pop-ups or the error inspector, they often fall short when it comes to dealing with the thousands of errors that can occur during an initial data migration. The Migration toolset released with ArcGIS Pro 3.5 includes tools to help with the most common types of errors, and I recommend you read the Analyzing Topology Errors and Resolving Topology Errors articles if you haven't already. However, if you are on an older release of ArcGIS Pro or have errors not handled by those tools you should look at using the Summarize Utility Network Errors tool.
Once you have deployed a utility network and enabled the network topology, run the Summarize Utility Network Errors tool to create a mobile geodatabase and report. By default, the tool summarizes all the topology and subnetwork errors in the Dirty Areas table of your utility network.
The tool will add two layers to your map:
- Error Summary – This table summarizes all the types of errors discovered in the database.
- Errors By Type – This table breaks down the errors by table, asset group, and asset type.
There are many ways to use this information, but a common way to develop an action plan is to look at the Errors By Type table and try to identify a corrective action for each row in the table. Some of these errors will require configuration changes, others will require manual or automated data cleanup. There are a few tricks that can help make this process easier.
The first is to create a chart using the Errors By Type table to help visualize the different problems. This can help you prioritize which errors to fix first by identifying the error code with the most errors.
The Summary geodatabase created by the tool also includes several additional classes that aren’t added to the map by default that are useful. One such class is the Errors feature class. This polygon feature class shows the location and details of every error the tool identified. It is best used when you want to look at examples of errors in your data, as it lets you quickly navigate to the locations where specific errors were discovered.
Sync C_Tables Asset Group/Type
Most data migration projects make use of Asset Packages to stage and apply migrated data between systems. Each asset package contains a set of tables that represent the schema, data, and configuration of a utility network. Tables in the asset package that are prefixed with a “C_” represent information that will be loaded into system tables using special tools. Keeping these tables in sync with the rest of the data in the asset package can be tedious, but this is where the Sync C_ Tables tool can help.
Some of the fields in these tables, like feature Global IDs, are easy to populate during a data migration while others, like Asset Group/Asset Type names, can be difficult or inefficient to populate during a data migration.
Because of this, the ArcGIS Solutions team created the Sync C_ Tables tool that can be run on an asset package that will synchronize the Asset Group and Asset Type information in the C_ tables of an asset package with the Asset Group/Asset Type information on the corresponding rows in the Asset Package. This means you can populate each C_table with a minimal amount of information shown below.
After running the tool, the asset group and asset type fields will be populated.
If this information is not synchronized before the Asset Package is deployed this will result in validation errors that must be corrected before you can proceed.
Assign Terminal Connections
Terminals are a new way of representing connectivity used by Utility Networks. When data is migrated into a utility network, features that connect to a device with terminals must know which terminal they are connected. Failure to populate this information can result in ambiguous connectivity errors if the two features are allowed to connect using different terminals, but the terminal being used isn’t specified.
Fortunately, the ArcGIS Solutions team developed the Assign Terminal Connections tool that can analyze utility network data and specify which terminals on a device each line is connected to by making some reasonable assumptions. Running this tool will look at your devices with terminals, their terminal configurations, rules, and the lines connected to them and use this information to connect each line to the terminal the tool thinks is most appropriate.
Once the tool has finished running you should validate the changes to ensure that all ambiguous connectivity errors are resolved. This process works well for bi-directional terminal configurations, because the terminals on these devices are typically interchangeable. However, you should carefully review any locations where the tool populated terminal connectivity on a device with a directional terminal configuration, like pumps or step transformers. In these situations, the terminal that each feature is connected to is important because it controls what is upstream and downstream, so if the tool assigns a feature to the wrong feature this will affect tracing.
An easy way to perform this is to use the Layers tab of the Attributes pane to review all the features with directional terminal configuration, then use the Modify Terminal Connections pane to correct any features that are improperly connected.
Conclusion
If you’re interested to discovering more tools to help with utility network project, check out the Utility Data Management Support Tools GitHub page for the full list. You can also learn about more tools to help with configuring maps as well as reporting.
If you want to watch a video discussing many of these tools, watch the ArcGIS Utility Network: Utility Network Foundations presentation from the 2024 Esri User Conference for a discussion of how to use many of these tools.
If you have specific questions about how to solve different workflows, please visit the ArcGIS Utility Network channel on the Esri Community site. This is a large community with thousands of active members.
