Hi Joseph.
First off, WOW! This appears to be a HUGE problem. Frankly, I'm amazed that Pro and your machine managed to survive for 630 hours 27 minutes 29 seconds (over 26 days!) of processing. I can honestly say I've never witnessed any tool run that long, let alone one of mine.
This, of course, helps you not at all since the tool died before finishing. The traceback unfortunately doesn't tell me much. Basically the OD Cost Matrix calculation must have crashed or died for some reason (reason not apparent from the log), and the parallel process caught the crash and stopped the tool. Given the size of the problem and the lengthy run time, I would guess some kind of resource limit (your computer ran out of space, ran out of CPU or memory, got tired, etc.) or some kind of process interruption (your computer tried to update, it lost a connection to an output folder if it was on a network, your virus scan did something, etc.). Unfortunately, I really just don't know what happened.
I was actually just talking with someone yesterday about having some kind of retry logic for processes that fail like this, but that's not something currently implemented in the tool's logic. Right now the entire tool will stop and fail as soon as one of the processes errors out. Possibly I can consider enhancing this at some point.
Regardless, I also note that the OD Cost Matrix calculation was only a quarter of the way through. That means that even if the process had not failed, you would be looking at running this tool for 3-4 months before you get a solution, and that really just doesn't seem tractable. Also, it has to do some post-processing in the end once the OD Cost Matrix calculations are done, and if the problem is truly this gigantic, you may run out of memory at that point as well.
I think you will have to break this problem down into smaller parts or it just isn't going to be tractable. Here are some ideas, data points, and suggestions to help you consider how to approach it:
- The size of the network dataset doesn't really matter. You can include all your transit agencies in one network and use it for each subset of inputs without substantially impacting the performance.
- You're right that you probably need to include all or most of the destinations since your job market is regional.
- Your best bet is to break up the origins by county or geographic area so you have manageable chunks that solve in a reasonable amount of time.
- Precalculate the network locations in advance to save some time.
- I noticed from your log that you're using only 5 parallel processes. If you have the ability to use more, this will definitely help a lot. Do you have access to a more powerful machine with more logical cores?
- Do you have access to more than one machine? If you could distribute the problem onto a couple of machines that can each run a chunk of origins, this would help you get a result in a reasonable amount of time.
- Consider spinning up some cloud machines. This problem honestly might just be intractably large for solving on one machine, and if you don't have the resources in house, maybe it would be worth the cost to rent some temporary processing capacity from the cloud.
- What version of ArcGIS Pro are you using? We made some substantial performance improvements to the OD Cost Matrix in the 2.9 release, so if you're using older software, updating will definitely help a lot. ODs may be 20-70% faster, depending on the analysis settings. If my math is right, it seems like each 1000x1000 OD chunk is taking about 14 seconds to run, which honestly seems rather slow to me.
How many origins and destinations do you actually have?
