Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

[chiranjeevijanjanam]

Hi,

I am doing my Master thesis with GIS Data where I have to churn a large amount of data, I have more than 2 million rows, which I must intersect with another map to get the common attribute table which I use in other software to get the required results, but processing itself is taking hours, yesterday I started it on ArcMap and it ran for 18 hours but just said "drawing features 36,000" at which point I thought it would take months to complete the operation and installed pro as i saw this thread.Dissolving Large Data 

But still the problem is persisting and I ran into a problem again, I waited for 3 hours at which point Pro returned an error and started processing again, can you guys say how to go through this large data intersect.

[DanPatterson_Retired]

What are your computer specs?  Pro would be the first choice, but your ram is going to be a factor.

Can you break the data into manageable parts (or tiles) to subset further?

[chiranjeevijanjanam]

Hi

I don't think its a problem my Ram is 8gb and I am using the i7 processor.

[DanPatterson_Retired]

kill edge and chrome and other  processes to regain memory would be a start

[JoshuaBixby]

What file formats are you working with, and what are the arguments you are passing to Pairwise Intersect?

[MervynLotter]

For large geoprocessing tasks like this I usually: (1) use Dice split up polygons with more than 20k vertices; (2) run Repair Geometry to find and correct any geometry errors that may have crept in; and (3) I save output to a geodatabse.

[KenHartling]

Dice should only be used on features that your system cannot handle at all (can't edit, can't draw, etc. See Dicing Godzillas (features with too many vertices) | ArcGIS Blog  ). Chopping up features like this could actually slow some processes down and cause other issues and require additional post processing.

If you have a specific case where this is required to get through the data efficiently could I see it? If you can, I'll send you a private message on how to get it to me efficiently.

Thanks. Ken

[MervynLotter]

Hi Ken

There is nothing recent to send as an example. Several years ago I was

unable to run several intersect tasks involving land cover and vegetation

types. The datasets were massive, and I thought it was just too big a task

for AcrMap. But after running Dice, I was able to easily complete my

analyses running the GP tools (I think I used* Erase *too). Since then I

almost always run Dice on complex polygons before a large GP task and the

tools always work. I have continued with this "best practice" even with

ArcGIS Pro.

Good to know that Dice should probably be reserved until after a tool

fails, then only try and run *Dice *and see if that helps.

Thanks

Mervyn

[KenHartling]

Over the past several years there have been many advancements in processing large data on the Desktop.  In ArcMap we've added 64bit background, ArcGIS Pro is a native 64bit application.  We've been modifying our products to take more and more advantage of the 64bit world and the additional resources it allows us to use when processing data.  

I would highly encouraging everyone that needs to process any data of size or complexity to give these 64bit offerings a try... and at the same time I would discourage dicing up features prior to processing if at all possible. Running Dice on your data should be a last resort.

Mervyn, if you do run into a process that won't complete using a 64bit offering (lots of stuff 'should' fail when run using 32bit applications) please let me know. Feel free to contact me with the data and case personally.

Thanks! Ken

[KenHartling]

8GB of RAM isn't a whole lot if your data contains large complex features with complex interactions... number of features may also be a factor depending on the amount of resources your machine has. You can get a better understanding of this by reading this old blog: Be successful overlaying large, complex datasets in Geoprocessing | ArcGIS Blog 

That said, can you share the data and the exact GP tool call (with all parameters) that is taking a long time?

If you can, let me know here and I'll send you a private message on how you can get it to me efficiently.

Thanks. Ken (ESRI GP Product Engineer)