nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How to deal with ERROR 030096: The maximum records for Origins limit of 1000 has been exceeded In ArcGIS Pro?

SomayehMafi

Hello

I have run "Generate Origin Destination Cost Matrix" in "ArcGIS Pro". But I have got ERROR 030096 telling that the maximum records for Origins limit of 1,000 has been exceeded. The number of origins for my project is nearly 67,000 points, and I don't want to split them into 67 groups; hence I am looking for a way to increase the limit from 1,000 to say 67,000 in ArcGIS pro.

Thank you in advance

Find more posts tagged with

Accepted answers

All comments

DanPatterson_Retired

You will need a field to use SplitByAttributes ....

You can load this function in the Field Calculator and calculate the grouping in a numeric (long) field.

Python parser!!!

The key is the seq_group(!OBJECTID!, xyz) line. In the example below, if the objected mod 10 equals 0, the sequential 'cnt' value increases, thereby starting a new group of values.

Give it a shot, then split by attributes on that field

cnt <SPAN class="operator token">=</SPAN> <SPAN class="number token">0</SPAN>
<SPAN class="keyword token">def</SPAN> <SPAN class="token function">seq_group</SPAN><SPAN class="punctuation token">(</SPAN>id<SPAN class="punctuation token">,</SPAN> val<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">global</SPAN> cnt
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> id <SPAN class="operator token">%</SPAN> val <SPAN class="operator token">==</SPAN> <SPAN class="number token">0</SPAN> <SPAN class="punctuation token">:</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cnt <SPAN class="operator token">+=</SPAN> <SPAN class="number token">1</SPAN>
&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">return</SPAN> cnt
__esri_field_calculator_splitter__
seq_group<SPAN class="punctuation token">(</SPAN>!OBJECTID!<SPAN class="punctuation token">,</SPAN> <SPAN class="number token">10</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Tested in ArcGIS Pro, should work in ArcMap

SomayehMafi

Mr Patterson

Thanks for your prompt response. Your solution could work, but I prefer to increase the limit from 1,000 to 67,000 if there is any way to do that in ArcGIS pro.

DanPatterson_Retired

it will break into groups of 1000, if you want groups of 67,000, then change the 10 to 67000

seq_group(!OBJECTID!, 67000)  # 67000 is the number of objects to form one group, could be 10, 100, 1000 whatever

</CODE><CODE>Which means you didn't try it did you?  Once you have it numbered, which you can confirm rapidly, then you splitby attribute on that field you created

SomayehMafi

How can I load seq_group function into my Field Calculator? I don't have this function in my Field Calculator .

BTW, I still prefer NOT to split my origins into groups, and instead I prefer to increase the limit of "network analysis tool" from 1,000 to 67,000 (if there is any way) in order to not get ERROR 030096 anymore.

JaySandhu

In ArcGIS Pro, the OD Cost Matrix itself does not have a 1000 origin limit. If you have your own local network dataset then you can solve for larger number of origins.

The 1000 Origins limit is imposed when you are using ArcGIS.com to solve this problem. This is a limit imposed on the OD service. How many destinations do you have? If you have a large number of destinations then the size of the computed OD become quite large and hard to transfer that data back. So try to use the splitting it as Dan outlined.

Jay Sandhu

SomayehMafi

Thank you very much Mr. Sandhu.

The number of my destinations is around 4,000. Based on your explanation, it seems that there is no way to increase origin limit when using ArcGIS.com for solving the problem.

In case I use my own local network dataset, can I calculate time taken from origin to destination based on existing traffic on the roadway (NOT distance from origin to destination)?

JaySandhu

If your local network dataset has been built with support for live traffic and you are subscribing to a traffic feed then, Yes, you can use it to compute the OD based on a start time. But remember, that travel time (based on real-time traffic) is only valid for that start time.

Jay Sandhu

AmberKeller2

If you use ArcMap Network Analysis there is no cost and no limit. I wrote this tutorial for ArcMap GIS/NA.md at master · akell47/GIS · GitHub

DavidDLong

Hi Jay,

I'm pointing to a local network data source, but I'm still getting the 1,000 record limit error. Any thoughts on how to change that limit?

Melinda Morang

The Generate Origin Destination Cost Matrix Ready-To-Use tool always uses the ArcGIS Online service, and you cannot change it to use local data. If you are indeed using this tool, that's why you're still running into the limit.

This tool has been deprecated in the last couple of releases of ArcGIS Pro, and confusion over the network data source was one of the reasons. If you want to do a network analysis, particularly using local data, it's better to use the standard workflow of creating an OD Cost Matrix layer, adding locations, and solving, as described in this tutorial: https://pro.arcgis.com/en/pro-app/latest/help/analysis/networks/od-cost-matrix-tutorial.htm

(And if possible, update to the latest version of Pro.)

JACooper

The solutions presented here and in other resources and discussions on this topic (here and here) all rely on using local network layers and do not address the core inquiry of this question:

I am looking for a way to increase the limit from 1,000

Local network layers may work fine, but there is substantial convenience to using ESRI's built-in network dataset to circumvent having to build your own network dataset. This is the core principle of a lot of the credit-consuming functions ESRI offers, is it not? Offering users a for-cost option for executing workflows and geoprocessing that would be laborious for the user to scale on their own. I would like to be able to tap into ESRI's network dataset because it is exactly what I'm looking for and better than what I could create independently in a short amount of time.

I understand wanting to caution the user against running large datasets through the OD Cost Matrix function. However, the artificial, immovable 1,000 record limit on both origins and destinations doesn't make sense to me. I have purchased credits and am willing to spend a good number of them on using ESRI's built-in network dataset. If I understand the risks and costs of executing the ODCM function with large datasets, why am I not allowed to do so?

My suggestion would be to add a user warning checkbox in the GUI and argument in the python functions with the default set to unchecked and "False", respectively. If the default remains in effect as it does now, the user would get a warning flag that running the OCDM would take a long time and consume a high volume of credits. Then, if the user understands and wants to continue, they check the box or set the warning flag to "True" and re-execute.

Melinda Morang

The limits are not solely in place to prevent users from accidentally consuming huge amounts of credits. Rather, they are in place to ensure that the services continue to function adequately and remain performant using the back-end architecture they run on. If a single user submitted an enormous OD Cost Matrix job, the service could become unresponsive for other users, and you may also run into limits in the amount of data you can send via a JSON response.

If you need to solve a large OD Cost Matrix problem and want to use the AGOL service (indeed, a reasonable desire), you can download this toolbox to help solve large network analysis problems: https://github.com/Esri/large-network-analysis-tools. This tool will automatically chunk up the problem into and solve the chunks in parallel using whatever data source you want. For AGOL, the chunk size is limited to 1000 and the number of parallel processes is limited to 4, but you should get your answer eventually.

Another option is to purchase Streetmap Premium so you can have your own Esri-provided high quality network dataset that you can use for unlimited analysis.

JACooper

Melinda, thank you for your explanation and suggestion; both are very helpful. This is a really valuable tool, and I appreciate your hard work on it. I have tested it in my context of using over 1K origins and destinations, and it works well for me. In my mind, your response answers the original question.

Any plans to include the large network analysis tools by default within a future release of ArcGIS Pro or arcpy?

I'll also follow up with my ESRI rep to explore Streetmap Premium; thanks.

Melinda Morang

Great, I'm glad I was able to solve your problem.

I don't think we will be able to incorporate the Large Network Analysis Tools into the core software for a variety of technical reasons, but my team continues to maintain them and expand them based on user feedback. We are always working toward improving performance of the Network Analyst solvers, and you should be able to solve problems much larger than 1000x1000 with local data even without chunking.

A lot of users ask us about the OD Cost Matrix analysis limits in the ArcGIS Online services, and while I can't promise any changes to those limits, I can send your feedback to our product manager.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.