Build an huge Network dataset

[AldoBertone]

Hi all,

I'm creating a Network dataset for an entire country using HERE data. I'm using a python script working as follow: first, I create a Network dataset from an xml file template, second the records from HERE dataset are appended (Append tool) to the Network dataset and third I build the network (Build Network tool).

Since HERE data includes about 60 million of records, I use a loop that append data and build the Network dataset using blocks of 5000 records at a time (i.e., the first 5000 records are appended to the Network dataset and  the Network dataset is build, then other 5000 records are appended to the Network dataset and  the Network dataset is build, etc). Every 100000 records added, I recalculate the spatial index of Network dataset and compact the geodatabase.

Here's my problem: by increasing the size of the network dataset, the execution time increases exponentially (at the beginning 5000 records were processed in about 2 minutes, with 30 million records added after a week of work the execution time is 4 times greater).

What is the best practice for creating and add lots of records (60 million) to  Network dataset?

Thank you very much in advance

[MelindaMorang · Accepted solution]

Hello.  I spoke with the developer responsible for the Build Network tool, and he suggests that the best practice is to add all your data up front and build the network once.  Even though that build will probably take quite a while, it should be faster than building it incrementally after adding chunks of records.

It's also worth noting that we improved the performance of incrementally rebuilding the network in the Pro 3.2 release that just came out, but it's still more efficient to build the whole thing at once.

[AldoBertone]

Thank you very much!

I followed your suggestions, the initial script adding chunks of records took more than 4 weeks, now with your suggestion to add all the data up front and build the network once it completed the execution in just 3 days (network with 35 million of streets).

[MelindaMorang]

Well, that's certainly better!  It's still pretty long, though.  Our team is continuing to work on performance improvements in Build Network, so hopefully we can improve upon this in future releases.