We need to migrate approximately 8 billion records into a PostgreSQL Enterprise Geodatabase containing an ArcGIS Utility Network. Due to data volume limitations, the data will be loaded incrementally in multiple batches into the same Utility Network. What is the ESRI-recommended approach for: * Incremental batch loading * Merging batches into an existing Utility Network * Managing topology during loads * Handling associations and dirty areas * Maintaining performance at billion-record scale Are there any documented best practices, scalability recommendations, or reference architectures for Utility Networks of this size? Thanks in advance for any guidance.

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

ESRI Recommendation for Incremental Loading of ~8 Billion Records into an Existing Utility Network

JayaramBT

We need to migrate approximately 8 billion records into a PostgreSQL Enterprise Geodatabase containing an ArcGIS Utility Network. Due to data volume limitations, the data will be loaded incrementally in multiple batches into the same Utility Network.

What is the ESRI-recommended approach for:

Incremental batch loading
Merging batches into an existing Utility Network
Managing topology during loads
Handling associations and dirty areas
Maintaining performance at billion-record scale

Are there any documented best practices, scalability recommendations, or reference architectures for Utility Networks of this size?

Thanks in advance for any guidance.

Find more posts tagged with

Accepted answers

RobertKrisher

It is my understanding you are already in touch with support about the performance issues you are experiencing. You are already aware that enabling the network topology is a resource intensive process, so faster CPU, plenty of RAM, lots of storage for processing the transaction (locally and on the DBMS), and also a low-latency connection between all clients and components (client performing build, and DBMS).

Best practices are practices that can be generally applied across all situations and said to be the best. You are dealing with exceptional circumstances that most customers are not likely to experience, so you are likely going to have some extraordinary hardware requirements as well as processes for managing this data load. If you need help working through these questions I recommend you reach out to an experienced business partner or Esri Professional Services to help your organization.

If you are loading data in batches, it would be best if each batch was error free before loading data and that the master database is error free as well. Creating errors requires additional CPU and Memory during the build process, so reducing the number of errors before you enable the network topology will make the process more efficient. You can do think like Enabling the network topology in the batches, or use tools like Analyze Network Data to identify topology errors before you enable.

All comments

RobertKrisher

@JayaramBT

You can find reference architecture in the ArcGIS Architecture Center.

The best practice is to load data into a unversioned geodatabase, with the network topology disabled. This includes our associations.

The best practice for handling topology errors is to resolve as many of them as possible during the migration, and to resolve the remainder of them before you register the database as versioned and enable the network topology. This includes ensuring you are not importing any invalid or duplication associations.

The best practice for subnetworks is to ensure your data is topologically clean and all data corrections to have valid, fully connected subnetworks occur before you import your subnetwork controllers and perform the initial update subnetwork.

If you decide to load data incrementally into a versioned environment, this will cause additional considerations, but the biggest risk you are adding is that you may introduce duplicate global IDs or associations during your loads. We keep incremental loads faster by relaxing some of our validation checks, but this also puts the burden on the data loader to ensure they are not loading bad data. There are many other concerns and considerations when it comes to loading data into a versioned environment, especially if you have an enabled topology and subnetworks, which is why it's not considered best practice. Its supported, but requires careful thought, planning, and testing of the tools and processes you plan on using.

JayaramBT

Thank you Robert for your quick replay. If the Utility Network ultimately contains approximately 8 billion records, is Esri's recommendation still to wait until all batches are loaded before enabling topology, or would Esri recommend periodically enabling topology and validating portions of the network to reduce the risk of discovering data issues at the end of the migration?

Are there any tested reference architectures, database sizing recommendations, or customer implementations where Utility Network datasets exceeded 1 billion records, and if so, what loading and topology-enablement strategy was used.

Real time example below:
During our testing, enabling network topology on approximately 300 million records required around 60 hours to complete.
Our target dataset is expected to grow to approximately 8 billion records. Based on this experience, we are concerned that topology enablement could become operationally challenging at the final migration stage.
Does Esri have any guidance, best practices, or reference implementations for topology enablement at this scale?
Specifically:
Are there any documented Utility Network deployments exceeding 1 billion records?Does Esri recommend a phased topology-enablement strategy for extremely large datasets?Are there known scaling characteristics or sizing recommendations for topology enablement at multi-billion-record volumes?What infrastructure (CPU, RAM, storage architecture) would Esri recommend for datasets of this magnitude?Would Esri recommend periodic validation during migration rather than waiting until all 8 billion records are loaded?

"300 million took 60 hours, therefore 8 billion will take 1,600 hours."???

RobertKrisher

JayaramBT

Thank you Robert for your suggestions. My concern is regarding the final consolidation of the data. If each batch is loaded separately, what is the recommended Esri approach to merge multiple Utility Network datasets into a single Utility Network?

Specifically:

Can the Append tool be used to merge Utility Network datasets from different batches?
If Append is not recommended, what are the supported or best-practice approaches for consolidating multiple Utility Network loads into a single Enterprise Geodatabase?
How should Utility Network associations (containment, connectivity, structural attachment), Global IDs, and topology integrity be managed during the merge process?
Would Esri recommend loading all batches directly into the same Utility Network dataset, or creating separate Utility Networks for each batch and merging them later?
Are there any performance, scalability, or topology-related considerations for large-scale migrations when using a batch-loading strategy?

We would appreciate Esri's guidance on the recommended approach for handling batch-based Utility Network migrations at this scale.

If I go with batch processing, how can I merge the Utility Network datasets? For example, if each batch contains 100 exchanges, how can I merge the data from one Utility Network dataset with another? Can I use the Append tool, or will that not work? What are the available approaches for merging Utility Network datasets? Please clarify and suggest the best approach for merging Utility Network data.

RobertKrisher

In my earlier post I was assuming you would be using Append, since it is the most straightforward way of loading data and preserving Global IDs, which are required for associations.

The questions you are asking are beyond the scope of a simple community post and are along the lines of a consulting or services engagement. There needs to be an in-depth discussion of your data migration process, quality assurance processes, architecture, and timeline before anyone should consider answering those questions.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.