Migrating data to the utility network
Update: Comments are now locked on this thread. If you have questions, please ask in them in the ArcGIS Utility Network Questions community.
With the release of the new Migration toolset for the utility network this provides a good opportunity to review how these new tools can be incorporated into existing best practices for implementing a utility network. In this article we will be discussing some of the best practices for using the tools, along with links to resources where you can learn additional information.
Before we talk about the new tools, let’s do a quick recap of the existing best practices for implementation. This is an abbreviated version of the best practices outlined in the Utility Network Data Migration: Best Practices article.
Historical tools and models
Prior to the release of the utility network migration toolset, migrating to the utility network meant you needed to use the Utility Network Package toolbox along with a pre-defined data model (asset package), such as those provided with Esri’s Utility Network Foundations. Data could then be migrated into the asset package using the Data Loading toolset, or any other data migration tool or process you wanted to build.
As more customers began using the data loading tools, the ArcGIS Solutions team released the Create Simple Data Mapping and Create Migration Workspace tools to make the process of creating a data loading workspace easier. These tools act as a utility network specific front end to the Data Loading toolset.
If you’re interested in learning more about these existing tools and models, check out the following resources:
- Migrating to the ArcGIS Utility Network (learning series)
- Introducing the utility network foundations (article)
- Simplifying ArcGIS Utility Network migrations (article)
- Load data into a utility network (learn tutorial)
- Electric transmission data loading (under development)
- Electric unbalanced distribution data loading (under development)
- Sewer data loading (solution tutorial)
- Stormwater data loading (under development)
- Water distribution data loading (under development)
Historically, quality assurance has been performed on your source data using ArcGIS Data Reviewer., Error analysis and error remediation workflows relied on manually reviewing errors created by the utility network, requiring a combination of manual edits and creating data cleanup scripts to resolve. The ArcGIS Solutions team released the Utility Data Management Support toolbox which includes several tools to make the process of reviewing (Summarize Utility Network Errors) or resolving some errors (Assign Terminals) easier, but it was still a largely manual process.
With history out of the way, let’s now discuss how the tools available in the new Migration toolset can supplement or replace some of these processes.
Migration toolset
The new migration toolset for the utility network includes tools focused on automating the migration to the utility network by creating a utility network that is based on your existing layers and fields. You can learn more about how this tool works by reading the Introducing the Migration toolset for the utility network article and the Building a utility network article.
To summarize the previous articles, the approach of the migration tools is quite different from what has come before and has several important implications. The biggest positive is that you maintain the subtypes, fields, and domains of your existing GIS features. The downside to this approach is that, because you are not implementing a known model that comes pre-configured for a specific set of industry workflows, you bear the responsibility of configuring your utility network model to behave the way you want.
Some customers may prefer this approach; however, it does mean that, instead of time spent focused on the translation of your data to conform to an industry standard model, you will be spending time focusing on configuring these same behaviors in your organization’s model.
Every time the Migrate To Utility Network tool runs, it creates a new mobile database. This begs the question of what to do when you’ve spent time configuring your utility network and need to refresh the data? You can use the Data Loading toolset to remigrate your data without losing any configuration or schema changes you’ve made. Let’s look at how the Migration toolset and the Data Loading toolset interact with each other.
Data Loading workspace
In addition to creating a mobile database containing a utility network, the migrate tool also creates a data loading workspace. A data loading workspace is a collection of files used by the Data Loading tools to migrate data from one schema to another.
This allows you to re-run the data migration of your source data into the data model that was originally created by the tool.
The main benefit of this approach is that it allows you to keep the configuration changes you’ve made to your model, while still being able to refresh your data. However, this approach does have some limitations.
When you run the Load Data Using Workspace tool it must remove any existing data from the database. If you have enabled subnetwork controllers or created associations in your data, then these must be removed before the tool can delete the corresponding data. There are two ways around this. We will discuss the first approach in this section.
The first step to creating a repeatable migration is to run the migration tool with the Load data parameter unchecked. (it is checked by default). This will create a data model (referred to in this article as a template) and a data loading workspace containing data mappings that can be used to remigrate your data. You can then keep a copy of this empty database to use for data loading purposes.
Once you’ve loaded data and made configuration changes these can be incorporated into your data migration process. Associations and subnetwork controllers can be exported or imported using the Export Associations, Import Associations, Export Subnetwork Controllers, and Import Subnetwork Controllers geoprocessing tools. Any configuration changes you’ve made during your data migration can be applied to your template geodatabase as well.
If you make any other schema changes to your geodatabase you will also need to apply them to your template. Depending on what has changed, you may need to adjust the field values or translations of your data loading workspace. You can learn more about this in the data loading workspace concepts topic in the online help.
This approach can take you quite far; however, note that if you decide to push the limits of the data model and its configuration, you will find certain types of changes are not allowed. The most common example is you are not allowed to remove or rename asset groups or asset types. So how can you deal with this limitation?
This is where the second approach comes in. To discuss the second approach, we must first discuss the history of data migration projects for the utility network and specifically discuss asset packages.
Asset Packages
The first data migrations built for the utility network required you to create and configure your network from scratch. Because this was a time-consuming process Esri developed a set of tools in the Utility Network Package toolbox to help streamline this process. These tools allow you to create and manage a specially structured file geodatabase called an asset package. An asset package is a specially formatted file geodatabase that contains the geodatabase schema (tables, subtypes, fields, domains) and tables that define the schema and configuration of the utility network being created. Another set of tools then reads this specially formatted database, using it as a template to create and configure a geodatabase and utility network matching that schema.
However, an asset package isn’t just a schema, it also contains data. Any data loaded into the geodatabase tables is appended to the corresponding tables of the newly created geodatabase. The asset package also allows you to populate the utility network system tables for subnetwork controllers and associations by populating several specialized tables in the asset package.
This is a powerful approach, but how does it apply to the migration toolset? The Utility Network Package toolbox contains a tool that allows you to export an existing utility network with all its configurations to a new asset package. This means that, once you reach the end of a prototype/pilot phase using a model that was created with the Migrate To Utility Network tool, you can use the Export Asset Package tool to turn it into an asset package for re-use in subsequent migrations.
Once you’ve exported your utility network configuration to an asset package, you then need to update the data loading workspace to point to the new asset package. This approach allows you to adjust the model, reload data to the asset package using the Data Loading toolset, and use the Asset Package to Geodatabase tool to deploy the asset package to a new utility network.
Smaller and simpler projects may not need to take this approach, but for larger, multi-month projects this allows you to iteratively develop and refine your data migration over time while still using a model produced by the migration toolset.
Conclusion
To recap, the primary purpose of the Migrate To Utility Network tool and the Migration toolset as a whole is to provide customers with a straightforward path to the utility network from their current geodatabase. However, as we all know, what can start as a simple project can sometimes evolve over time into something more involved, which is why these tools can be integrated into existing best practices for utility network migration to suit your needs.
In this article you learned how you can overcome the most common migration challenges using existing Esri tools. This knowledge should prepare you to handle any challenges that may arise in your journey and, even if you don’t need to employ any of these techniques right now, it’s good to know they exist should they be needed in the future. Check out the Get Started with the Migration toolset article for more information on how to access these tools and find additional resources covering topics like configuration, data cleanup, and data migration.