topic Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap. in Geoprocessing Questions

Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

chiranjeevijanjanam — Fri, 26 Jan 2018 14:34:07 GMT

Hi,

I am doing my Master thesis with GIS Data where I have to churn a large amount of data, I have more than 2 million rows, which I must intersect with another map to get the common attribute table which I use in other software to get the required results, but processing itself is taking hours, yesterday I started it on ArcMap and it ran for 18 hours but just said "drawing features 36,000" at which point I thought it would take months to complete the operation and installed pro as i saw this thread.Dissolving Large Data

But still the problem is persisting and I ran into a problem again, I waited for 3 hours at which point Pro returned an error and started processing again, can you guys say how to go through this large data intersect.

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

DanPatterson_Retired — Fri, 26 Jan 2018 14:41:59 GMT

What are your computer specs? Pro would be the first choice, but your ram is going to be a factor.

Can you break the data into manageable parts (or tiles) to subset further?

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

chiranjeevijanjanam — Fri, 26 Jan 2018 15:53:41 GMT

I don't think its a problem my Ram is 8gb and I am using the i7 processor.

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

DanPatterson_Retired — Fri, 26 Jan 2018 16:32:52 GMT

kill edge and chrome and other processes to regain memory would be a start

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

JoshuaBixby — Fri, 26 Jan 2018 17:22:49 GMT

What file formats are you working with, and what are the arguments you are passing to Pairwise Intersect?

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

MervynLotter — Fri, 26 Jan 2018 17:52:36 GMT

For large geoprocessing tasks like this I usually: (1) use Dice split up polygons with more than 20k vertices; (2) run Repair Geometry to find and correct any geometry errors that may have crept in; and (3) I save output to a geodatabse.

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

KenHartling — Fri, 26 Jan 2018 18:11:42 GMT

8GB of RAM isn't a whole lot if your data contains large complex features with complex interactions... number of features may also be a factor depending on the amount of resources your machine has. You can get a better understanding of this by reading this old blog: Be successful overlaying large, complex datasets in Geoprocessing | ArcGIS Blog

That said, can you share the data and the exact GP tool call (with all parameters) that is taking a long time?

If you can, let me know here and I'll send you a private message on how you can get it to me efficiently.

Thanks. Ken (ESRI GP Product Engineer)

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

KenHartling — Fri, 26 Jan 2018 18:21:06 GMT

Dice should only be used on features that your system cannot handle at all (can't edit, can't draw, etc. See Dicing Godzillas (features with too many vertices) | ArcGIS Blog ). Chopping up features like this could actually slow some processes down and cause other issues and require additional post processing.

If you have a specific case where this is required to get through the data efficiently could I see it? If you can, I'll send you a private message on how to get it to me efficiently.

Thanks. Ken

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

MervynLotter — Fri, 26 Jan 2018 18:42:01 GMT

Hi Ken

There is nothing recent to send as an example. Several years ago I was

unable to run several intersect tasks involving land cover and vegetation

types. The datasets were massive, and I thought it was just too big a task

for AcrMap. But after running Dice, I was able to easily complete my

analyses running the GP tools (I think I used* Erase *too). Since then I

almost always run Dice on complex polygons before a large GP task and the

tools always work. I have continued with this "best practice" even with

ArcGIS Pro.

Good to know that Dice should probably be reserved until after a tool

fails, then only try and run *Dice *and see if that helps.

Thanks

Mervyn

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

KenHartling — Fri, 26 Jan 2018 18:58:37 GMT

Over the past several years there have been many advancements in processing large data on the Desktop. In ArcMap we've added 64bit background, ArcGIS Pro is a native 64bit application. We've been modifying our products to take more and more advantage of the 64bit world and the additional resources it allows us to use when processing data.

I would highly encouraging everyone that needs to process any data of size or complexity to give these 64bit offerings a try... and at the same time I would discourage dicing up features prior to processing if at all possible. Running Dice on your data should be a last resort.

Mervyn, if you do run into a process that won't complete using a 64bit offering (lots of stuff 'should' fail when run using 32bit applications) please let me know. Feel free to contact me with the data and case personally.

Thanks! Ken

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

chiranjeevijanjanam — Fri, 26 Jan 2018 20:57:43 GMT

Thank you, Ken.

.....data removed....

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

DanPatterson_Retired — Fri, 26 Jan 2018 21:12:53 GMT

Some suggestions.

create a featureclass from the shapefile will improve things (assume it should be in the same geodatabase)

prior to doing any intersections, make sure that you have made the featureclass with the join permanent (ie a new featureclass)

limit your area of interest prior to doing any intersection. In other words are there areas in both files that you know won't intersect? you want to remove having to check geometries against geometries that won't participate in an intersection at all. This could be simplified by doing a select by spatial location to limit to only the overlapping areas.

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

VinceAngelo — Fri, 26 Jan 2018 21:43:17 GMT

Two million rows isn't all that much with modern computers. I regularly process 20m x 60k Intersect operations, and they rarely run more than an hour. 8Gb RAM isn't all that much -- 16Gb is a modern low-end RAM allocation.

But more important than RAM is the type and speed of your disk. A laptop with a clunky old 40ms HDD seek disk would take a compute-year or more to do what a hot new <1ms SSD can do in minutes.

- V

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

JoshuaBixby — Fri, 26 Jan 2018 23:12:08 GMT

I agree with Dan, work in the native file formats for the best performance. Working with shape files, CSV files, Excel files, and doing joins and geoprocessing is a recipe for disaster. We have come across several geoprocessing operations in Pro that take 10x longer working with shape files than native file formats.

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

DanPatterson_Retired — Sat, 27 Jan 2018 03:24:24 GMT

Also from the PRO docs

(Tiled) Processing of Large Datasets

Summary of recommendations, from there and others from above... not all needed, but here they are

Use a beefy machine machine.
Read the docs in the link to see which tools exploit adaptive subdivision (like quadtrees for vectors). If your process doesn't, then you may have to slice and dice into smaller parcels yourself
Don't run other applications while running large dataset processes (ie. skip email and Netflix)
Use Dice (even if subdivision will be going on)
Use Enterprise and file geodatabases

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

chiranjeevijanjanam — Sat, 27 Jan 2018 18:50:28 GMT

Thank you worked for me now, it worked in Arcpro and pairwise intersect took just less than 2 minutes, and normal intersect took around 15 minutes, after following all your steps carefully it worked pretty good, checked 3-4 times all your suggestions and followed them.

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

chiranjeevijanjanam — Sat, 27 Jan 2018 18:51:15 GMT

your answer is also right, same as above...it helped me a lot to do work efficiently.

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

ColeAndrews — Wed, 27 Feb 2019 02:41:32 GMT

Vince Angelo‌ you're accomplishing that in what? Surely not in Desktop...

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

VinceAngelo — Wed, 27 Feb 2019 16:44:58 GMT

Yes, ArcMap 10.4.1 (32-bit ArcPy), in fact (and don't call me Shirley )

- V

Re: Problem while Geoprocessing (Intersecting) large amount on data in Arcpro and Arcmap.

ColeAndrews — Fri, 05 Apr 2019 16:46:09 GMT

That's solid...I want the insiders secret

I'm running intersects in Pro using NVMe SSD, the best there is, and Pro would 'Shirley' crash before it ever completed 20m points to 60k polygon intersects.

Alteryx is the fastest analytics software with a GUI I've worked with, and it took 2 days for us to intersect 30 million points with 1k drive time polygons.