How to deal with ERROR 030096: The maximum records for Origins limit of 1000 has been exceeded In ArcGIS Pro?

6164
15
10-11-2017 02:18 PM
SomayehMafi
New Contributor II

Hello

I have run "Generate Origin Destination Cost Matrix" in "ArcGIS Pro". But I have got ERROR 030096 telling that the maximum records for Origins limit of 1,000 has been exceeded. The number of origins for my project is nearly 67,000 points, and I don't want to split them into 67 groups; hence I am looking for a way to increase the limit from 1,000 to say 67,000 in ArcGIS pro. 

Thank you in advance

15 Replies
JaySandhu
Esri Regular Contributor

If your local network dataset has been built with support for live traffic and you are subscribing to a traffic feed then, Yes, you can use it to compute the OD based on a start time. But remember, that travel time (based on real-time traffic) is only valid for that start time.

Jay Sandhu

AmberKeller2
New Contributor II

If you use ArcMap Network Analysis there is no cost and no limit. I wrote this tutorial for ArcMap GIS/NA.md at master · akell47/GIS · GitHub 

0 Kudos
JACooper
New Contributor III

The solutions presented here and in other resources and discussions on this topic (here and here) all rely on using local network layers and do not address the core inquiry of this question: 


I am looking for a way to increase the limit from 1,000

Local network layers may work fine, but there is substantial convenience to using ESRI's built-in network dataset to circumvent having to build your own network dataset. This is the core principle of a lot of the credit-consuming functions ESRI offers, is it not? Offering users a for-cost option for executing workflows and geoprocessing that would be laborious for the user to scale on their own. I would like to be able to tap into ESRI's network dataset because it is exactly what I'm looking for and better than what I could create independently in a short amount of time.

I understand wanting to caution the user against running large datasets through the OD Cost Matrix function. However, the artificial, immovable 1,000 record limit on both origins and destinations doesn't make sense to me. I have purchased credits and am willing to spend a good number of them on using ESRI's built-in network dataset. If I understand the risks and costs of executing the ODCM function with large datasets, why am I not allowed to do so? 

My suggestion would be to add a user warning checkbox in the GUI and argument in the python functions with the default set to unchecked and "False", respectively. If the default remains in effect as it does now, the user would get a warning flag that running the OCDM would take a long time and consume a high volume of credits. Then, if the user understands and wants to continue, they check the box or set the warning flag to "True" and re-execute.

MelindaMorang
Esri Regular Contributor

The limits are not solely in place to prevent users from accidentally consuming huge amounts of credits.  Rather, they are in place to ensure that the services continue to function adequately and remain performant using the back-end architecture they run on.  If a single user submitted an enormous OD Cost Matrix job, the service could become unresponsive for other users, and you may also run into limits in the amount of data you can send via a JSON response.

If you need to solve a large OD Cost Matrix problem and want to use the AGOL service (indeed, a reasonable desire), you can download this toolbox to help solve large network analysis problems: https://github.com/Esri/large-network-analysis-tools.  This tool will automatically chunk up the problem into and solve the chunks in parallel using whatever data source you want.  For AGOL, the chunk size is limited to 1000 and the number of parallel processes is limited to 4, but you should get your answer eventually.

Another option is to purchase Streetmap Premium so you can have your own Esri-provided high quality network dataset that you can use for unlimited analysis.

JACooper
New Contributor III

Melinda, thank you for your explanation and suggestion; both are very helpful. This is a really valuable tool, and I appreciate your hard work on it. I have tested it in my context of using over 1K origins and destinations, and it works well for me. In my mind, your response answers the original question. 

Any plans to include the large network analysis tools by default within a future release of ArcGIS Pro or arcpy?

I'll also follow up with my ESRI rep to explore Streetmap Premium; thanks.

MelindaMorang
Esri Regular Contributor

Great, I'm glad I was able to solve your problem.

I don't think we will be able to incorporate the Large Network Analysis Tools into the core software for a variety of technical reasons, but my team continues to maintain them and expand them based on user feedback.  We are always working toward improving performance of the Network Analyst solvers, and you should be able to solve problems much larger than 1000x1000 with local data even without chunking.

A lot of users ask us about the OD Cost Matrix analysis limits in the ArcGIS Online services, and while I can't promise any changes to those limits, I can send your feedback to our product manager.