Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

3808
21
Jump to solution
01-26-2018 06:34 AM
chiranjeevijanjanam
New Contributor II

Hi,

I am doing my Master thesis with GIS Data where I have to churn a large amount of data, I have more than 2 million rows, which I must intersect with another map to get the common attribute table which I use in other software to get the required results, but processing itself is taking hours, yesterday I started it on ArcMap and it ran for 18 hours but just said "drawing features 36,000" at which point I thought it would take months to complete the operation and installed pro as i saw this thread.Dissolving Large Data 

But still the problem is persisting and I ran into a problem again, I waited for 3 hours at which point Pro returned an error and started processing again, can you guys say how to go through this large data intersect.

0 Kudos
1 Solution

Accepted Solutions
DanPatterson_Retired
MVP Esteemed Contributor

Some suggestions.

create a featureclass from the shapefile will improve things (assume it should be in the same geodatabase)

prior to doing any intersections, make sure that you have made the  featureclass with the join permanent (ie a new featureclass)

limit your area of interest prior to doing any intersection.  In other words are there areas in both files that you know won't intersect? you want to remove having to check geometries against geometries that won't participate in an intersection at all.  This could be simplified by doing a select by spatial location to limit to only the overlapping areas.

View solution in original post

21 Replies
DanPatterson_Retired
MVP Esteemed Contributor

What are your computer specs?  Pro would be the first choice, but your ram is going to be a factor.

Can you break the data into manageable parts (or tiles) to subset further?

chiranjeevijanjanam
New Contributor II

Hi

I don't think its a problem my Ram is 8gb and I am using the i7 processor.

0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

kill edge and chrome and other  processes to regain memory would be a start

JoshuaBixby
MVP Esteemed Contributor

What file formats are you working with, and what are the arguments you are passing to Pairwise Intersect?

0 Kudos
MervynLotter
Regular Contributor

For large geoprocessing tasks like this I usually: (1) use Dice split up polygons with more than 20k vertices; (2) run Repair Geometry to find and correct any geometry errors that may have crept in; and (3) I save output to a geodatabse.

0 Kudos
KenHartling
Esri Contributor

Dice should only be used on features that your system cannot handle at all (can't edit, can't draw, etc. See Dicing Godzillas (features with too many vertices) | ArcGIS Blog  ). Chopping up features like this could actually slow some processes down and cause other issues and require additional post processing.

If you have a specific case where this is required to get through the data efficiently could I see it? If you can, I'll send you a private message on how to get it to me efficiently.

Thanks. Ken

MervynLotter
Regular Contributor

Hi Ken

There is nothing recent to send as an example. Several years ago I was

unable to run several intersect tasks involving land cover and vegetation

types. The datasets were massive, and I thought it was just too big a task

for AcrMap. But after running Dice, I was able to easily complete my

analyses running the GP tools (I think I used* Erase *too). Since then I

almost always run Dice on complex polygons before a large GP task and the

tools always work. I have continued with this "best practice" even with

ArcGIS Pro.

Good to know that Dice should probably be reserved until after a tool

fails, then only try and run *Dice *and see if that helps.

Thanks

Mervyn

KenHartling
Esri Contributor

Over the past several years there have been many advancements in processing large data on the Desktop.  In ArcMap we've added 64bit background, ArcGIS Pro is a native 64bit application.  We've been modifying our products to take more and more advantage of the 64bit world and the additional resources it allows us to use when processing data.  

I would highly encouraging everyone that needs to process any data of size or complexity to give these 64bit offerings a try... and at the same time I would discourage dicing up features prior to processing if at all possible. Running Dice on your data should be a last resort.

Mervyn, if you do run into a process that won't complete using a 64bit offering (lots of stuff 'should' fail when run using 32bit applications) please let me know. Feel free to contact me with the data and case personally.

Thanks! Ken

KenHartling
Esri Contributor

8GB of RAM isn't a whole lot if your data contains large complex features with complex interactions... number of features may also be a factor depending on the amount of resources your machine has. You can get a better understanding of this by reading this old blog: Be successful overlaying large, complex datasets in Geoprocessing | ArcGIS Blog 

That said, can you share the data and the exact GP tool call (with all parameters) that is taking a long time?

If you can, let me know here and I'll send you a private message on how you can get it to me efficiently.

Thanks. Ken (ESRI GP Product Engineer)