Computing OD Matrix using Network Analyst

[TDGoswami]

Our current situation is that we have around 40,000 geolocated settlements across Africa, and each is located on a detailed road network shapefile. We would like to calculate the shortest distances by road from each settlement to every other settlement (i.e. creating a huge 40,000 x 40,000 Origin-Destination matrix of distances) - we have a methodology to do this for smaller datasets within GIS software, but are held back by memory limitations for such a large operation. Hence, we were interested to explore your methods before we consider developing our own solutions. Do you have any advice or suggestions on proceeding with this please ?

[JaySandhu]

Cedric,
I think TD here is not generating all nodes to all nodes OD. I believe he has discrete locations that he wants to use as input. So comuting distances from any location to any location is not as straight-forward as all nodes to all nodes type of algorithms you see in CS books. Also, as the Network Analyst already has this solver implemented why re-invent the wheel but concentrate on other things.  Your point on clustering is well taken as 40k by 40 k matrix will be very large and MATLAB will not be able to take the whole thing in. But then TD has not specified if he wants to use the whole matrix or some small part of it for further data mining.
Jay Sandhu

[TDGoswami]

Thanks for all you help guys.

I am planning to use the contents of the entire matrix. That is, every element in that matrix is equally important for my further work.

I am trying to do a mechanical job of running 80 simulations in subsets of 500 sources covering the entire 40000 points. It is taking a lot of time. Moreover, it is taking even longer time to export the results of the OD Cost Matrix layer into a text file (since it has to write between 9 million to 12 million points). Is there a way I could make this Export faster?

Thanks,
TD

[TDGoswami]

Thanks to everyone for their help. I am faced with a peculiar problem now. I find that the distance computed by the OD Cost Matrix Layer is not symmetric. That is, for example, the distance between settlement 1 and settlement 500 is not the same as that between settlement 500 and settlement 1.

Do you guys have a reason for this behavior? Whats the solution?

Thanks
TD

[JaySandhu]

OD Matrix does not have to be symmetric. A road network may have many one-way streets, turn restrictions or penalties and the fact that it is a directed network, so travel to and travel from impedances may be different for the same edge. All of these can cause the travel time from location 1 to location 500 be different. If you want symmetric values, then either create a network without the above mentioned things (i.e., no turn penalties, no one-way information, no restrictions, and same travel-to/from impedances) OR use one half of the matrix and fill the other side to make it symmetric after the fact.
Regards,
Jay Sandhu

[TDGoswami]

I am now faced with an extremely peculiar problem.

I first created the network and then I was running the OD Cost Layer matrix on it. I find that some of the settlements have merged into one, although, these settlements were originally very far away from each other and had no similarities. Is there a reason for this behavior?

[ToreBorvik]

have you geocoded the points? Adresses with inaccurate quality might risk being placed in the centerpoint of a region (for instance a county, if county name is the only match of the address). In this occation, several villages might be placed at the same location.

[bogdanpalade1]

Hello,
Sorry to jump in to the discussion with my own problem but its very similar to the original post and I thought its not necessary to open a new thread. In my case,  I do not have to worry about memory problems because I have only 131 cities.
I thought that OD matrix would solve the problem by putting the 131 cities as origins and the same 131 cities as destinations and solve it. But the output is always 131 routes with  0 value because it computes only the closest distance between the points that will be given by the distance between the same city (once in origins and once in destination)

How can I do it to compute all the possible 131X131 distances?

I'm not expert in python but I can manage basic stuff

Thank you,

Bogdan

[JaySandhu]

Bogdon,
What network dataset are you using to computer the OD? If the 131 by 131 is returning only 1 closest the most likely there is no path from where those locations are being snapped on the network to rest of the points. Have you tried running a simple shortest path (Route) on a pair of cities and does that work. Possible reasons for not finding paths could be related to the way the network was set up. And of course make sure the OD is not set to compute only 1 closest, by default it is set to find "<All>".
Regards,
Jay Sandhu

[JonathanWeyman]

Hi Everyone,

I know this thread is quite old, but I am faced with a similar problem as the original poster. I am trying to compute an OD Cost Matrix with accumulated impedance values for distance from approx. 50,000 origins to ALL of approx. 34,000 destinations.

The "python loop" approach described above sounds promising, and I will certainly experiment with creating an initial model in modelbuilder and exporting to python in order to edit further. But, as I have little experience with Python, I'm wondering if anyone might be able to point me on the right track as far as coding the "loop" described above to select 100 or 200 records, run the OD cost matrix, export results, select the next batch of records, and so on?

This would be extremely helpful to me.

Thank you,

Tosh

[jp]

Hi Jay,

Can we restrict the network analysis results to filter calculation between origin and destination with same name or ID? Or can we make the results symmetric so distance between A to B is calculated but not B to A?

I couldn't find any such property in ArcMap.

Thanks.

OD Matrix does not have to be symmetric. A road network may have many one-way streets, turn restrictions or penalties and the fact that it is a directed network, so travel to and travel from impedances may be different for the same edge. All of these can cause the travel time from location 1 to location 500 be different. If you want symmetric values, then either create a network without the above mentioned things (i.e., no turn penalties, no one-way information, no restrictions, and same travel-to/from impedances) OR use one half of the matrix and fill the other side to make it symmetric after the fact.
Regards,
Jay Sandhu