Build Network performance

266
8
07-30-2019 11:31 AM
Highlighted
New Contributor III

Hello

I am working on automating a process which imports data from local and regional transit authorities, builds a network dataset, and then publishes a Network Analysis service to our ArcGIS Server to use later to solve routing problems.

The process works fine, but the 'build network' step takes quite a long time (8+ hrs) and I was wondering if there are any suggestions of ways I can speed it up.

Network dataset info:

-Global turns only

-No elevation

-End point connectivity

-Roughly 70000 edge features

-Two network attributes added in addition to default Length attribute: one Cost, one Restriction

I am using ArcMap 10.7. Building the network takes a long time both via ArcMap GUI as well as via ArcPy.

Things I have tried:

-changed network attributes from python to vb script

-hardcoded cost and restriction values into the underlying feature class and changed the network dataset attributes to simply read those fields rather than running any script to derive them

Reply
0 Kudos
8 Replies
Highlighted
Esri Regular Contributor

In general, I wouldn't expect building a network with 70,000 edges to take more than a few minutes.

You don't specify, but since this is an automated procedure you're developing, I presume your network dataset is in a file geodatabase and not a shapefile?  File geodatabase will likely be faster than shapefile.

Check your network's source feature classes and make sure they have a spatial index.  If they don't, that's probably why everything is slow.

If that's not it...can you describe your procedure in a bit more detail?  For example, are you using the Create Network Dataset From Template tool, or have you developed some other method (via ArcObjects, for example) for creating the network dataset?

Highlighted
New Contributor III

You are correct that my network dataset (and the source feature class) are in file geodatabase on a local drive.

My source feature class does indeed have a spatial index so unfortunately that is not the issue.

To create the network dataset, in the Catalog pane in ArcMap I right click on the Feature Dataset containing my source feature class, and choose New > Network Dataset. I then follow the wizard to choose the source features, set turns, set elevation, create attributes from evaluators, and set the driving directions name attribute.

My plan is to then in a batch process truncate and re-populate the source feature class using FME, and then rebuild the network using ArcPy.  

Reply
0 Kudos
Highlighted
Esri Regular Contributor

Okay, so you don't have an automated procedure in place yet.  Understood.  When you get to that point, check out the Create Network Dataset From Template tool: https://desktop.arcgis.com/en/arcmap/latest/tools/network-analyst-toolbox/create-network-dataset-fro...  I think this might be more efficient and easier to do than what you're suggesting with truncating and re-populating tables.

Taking another wild guess here, which is kind of a long shot: When you make edits to the features of a network dataset, the network will create an internal table of "dirty areas" indicating which parts of the network have been changed.  When you build the network, it only builds the dirty areas instead of the whole thing.  This is intended to make the procedure more efficient because if you have, say a nationwide network dataset but you only edited one neighborhood, it can just leave everything except that neighborhood alone.

However, if you've made wide-sweeping edits, such as replacing all the streets with different streets in a different geography, you might have built up a sizable dirty area file.  Possibly the Build process is actually slower if it's trying to sort through all these dirty areas and take care of them rather than just rebuilding the entire network as if from scratch.

But, that's just a guess which may or may not apply to your situation.

Do you get the slow behavior if you put your streets in a fresh geodatabase with a fresh feature dataset and create the network dataset completely from scratch?  If not, then the above might describe what's going on.  If it's still slow in this case, then something else is probably weird with your data.  I would suggest you call Support so they can look at the data for you and diagnose the problem.

Highlighted
New Contributor III

Thank you for the suggestions!


I'm thinking I may attempt to just recreate the network dataset using xml templates via script.  I did it once and it appears to build fast the first time, so I think your theory about the dirty areas sounds correct.

I do have one more question - can DeleteFeatures_management be used to delete a network dataset? If I want to recreate the same dataset from template repeatedly via script, I believe I need to delete the original dataset each time as CreateNetworkDatasetFromTemplate will not overwrite an existing network dataset with the same name as specified in the template.

However, I when I try to use DeleteFeatures_management and point it my network dataset in my gdb I get an error 000732.

Reply
0 Kudos
Highlighted
Esri Regular Contributor

You'd have to use the regular Delete tool, not DeleteFeatures.  That's the short answer to your question.

The longer answer is that your automated process (script or model) should probably be doing this:

- Create a new file geodatabase

- Create a new feature dataset in that file geodatabase

- Copy the new street data into that feature dataset

- Run Create Network Dataset From Template using that feature dataset

- Build the new network

You don't need to delete any old networks to make this work.   You can create an entirely new network from a template as long as the schema of the input data matches the schema used when creating the network dataset.  So, if you had data from multiple municipalities that all use the same data model, you can create a separate network for each one using the same template.

Highlighted
New Contributor III

Thanks so much for your help Melinda!

I've done some testing and a variation of the process you describe works fine for me.

-> Data_template.gdb contains feature datasets and source feature classses, but no network datasets

-> Via FME I truncate my source feature classes in Data_template and reload them with fresh data

-> Via python I:

----> copy Data_template.gdb to Data_working.gdb

----> use na.CreateNetworkDatasetFromTemplate to create the network in Data_working.gdb from my template

----> use na.BuildNetwork to build the new network

It performs quite well, the entire process described above now takes less than 10 minutes.

Reply
0 Kudos
Highlighted
Esri Regular Contributor

Glad that's working.  Keep an eye on it to make sure it doesn't slow down over time.  If it does, then it probably has to do with the truncation and re-population of those tables.

Reply
0 Kudos
Highlighted
Esri Regular Contributor

When the build network process is running, it writes to a log file called "build.txt". This file can be found in a sub folder for current arc process in the system %TEMP% folder.

Examine the contents of that to see where it is spending the time. This might help find the culprit of the slow build times.

Also, even if a spatial index exists on your feature class, you may want to re-build it in case it has become stale for some reason.

Jay Sandhu