Build an huge Network dataset

AldoBertone

Hi all,

I'm creating a Network dataset for an entire country using HERE data. I'm using a python script working as follow: first, I create a Network dataset from an xml file template, second the records from HERE dataset are appended (Append tool) to the Network dataset and third I build the network (Build Network tool).

Since HERE data includes about 60 million of records, I use a loop that append data and build the Network dataset using blocks of 5000 records at a time (i.e., the first 5000 records are appended to the Network dataset and  the Network dataset is build, then other 5000 records are appended to the Network dataset and  the Network dataset is build, etc). Every 100000 records added, I recalculate the spatial index of Network dataset and compact the geodatabase.

Here's my problem: by increasing the size of the network dataset, the execution time increases exponentially (at the beginning 5000 records were processed in about 2 minutes, with 30 million records added after a week of work the execution time is 4 times greater).

What is the best practice for creating and add lots of records (60 million) to  Network dataset?

Thank you very much in advance

Melinda Morang

Hello.  I spoke with the developer responsible for the Build Network tool, and he suggests that the best practice is to add all your data up front and build the network once.  Even though that build will probably take quite a while, it should be faster than building it incrementally after adding chunks of records.

It's also worth noting that we improved the performance of incrementally rebuilding the network in the Pro 3.2 release that just came out, but it's still more efficient to build the whole thing at once.