I am doing my Master thesis with GIS Data where I have to churn a large amount of data, I have more than 2 million rows, which I must intersect with another map to get the common attribute table which I use in other software to get the required results, but processing itself is taking hours, yesterday I started it on ArcMap and it ran for 18 hours but just said "drawing features 36,000" at which point I thought it would take months to complete the operation and installed pro as i saw this thread.Dissolving Large Data
But still the problem is persisting and I ran into a problem again, I waited for 3 hours at which point Pro returned an error and started processing again, can you guys say how to go through this large data intersect.
Solved! Go to Solution.
create a featureclass from the shapefile will improve things (assume it should be in the same geodatabase)
prior to doing any intersections, make sure that you have made the featureclass with the join permanent (ie a new featureclass)
limit your area of interest prior to doing any intersection. In other words are there areas in both files that you know won't intersect? you want to remove having to check geometries against geometries that won't participate in an intersection at all. This could be simplified by doing a select by spatial location to limit to only the overlapping areas.
What are your computer specs? Pro would be the first choice, but your ram is going to be a factor.
Can you break the data into manageable parts (or tiles) to subset further?
For large geoprocessing tasks like this I usually: (1) use Dice split up polygons with more than 20k vertices; (2) run Repair Geometry to find and correct any geometry errors that may have crept in; and (3) I save output to a geodatabse.
Dice should only be used on features that your system cannot handle at all (can't edit, can't draw, etc. See Dicing Godzillas (features with too many vertices) | ArcGIS Blog ). Chopping up features like this could actually slow some processes down and cause other issues and require additional post processing.
If you have a specific case where this is required to get through the data efficiently could I see it? If you can, I'll send you a private message on how to get it to me efficiently.
There is nothing recent to send as an example. Several years ago I was
unable to run several intersect tasks involving land cover and vegetation
types. The datasets were massive, and I thought it was just too big a task
for AcrMap. But after running Dice, I was able to easily complete my
analyses running the GP tools (I think I used* Erase *too). Since then I
almost always run Dice on complex polygons before a large GP task and the
tools always work. I have continued with this "best practice" even with
Good to know that Dice should probably be reserved until after a tool
fails, then only try and run *Dice *and see if that helps.
Over the past several years there have been many advancements in processing large data on the Desktop. In ArcMap we've added 64bit background, ArcGIS Pro is a native 64bit application. We've been modifying our products to take more and more advantage of the 64bit world and the additional resources it allows us to use when processing data.
I would highly encouraging everyone that needs to process any data of size or complexity to give these 64bit offerings a try... and at the same time I would discourage dicing up features prior to processing if at all possible. Running Dice on your data should be a last resort.
Mervyn, if you do run into a process that won't complete using a 64bit offering (lots of stuff 'should' fail when run using 32bit applications) please let me know. Feel free to contact me with the data and case personally.
8GB of RAM isn't a whole lot if your data contains large complex features with complex interactions... number of features may also be a factor depending on the amount of resources your machine has. You can get a better understanding of this by reading this old blog: Be successful overlaying large, complex datasets in Geoprocessing | ArcGIS Blog
That said, can you share the data and the exact GP tool call (with all parameters) that is taking a long time?
If you can, let me know here and I'll send you a private message on how you can get it to me efficiently.
Thanks. Ken (ESRI GP Product Engineer)