Dissolve issues with very large merged landcover dataset

[JosephWalderman1]

I have taken a large mosaic land cover dataset for the James River Watershed and manipulated it through iterate, clip, and raster to polygon then merging all the resulting shapefiles into a file geodatabase. The resulting merged file is quite large, 8GB and 22 million features. I tried to run the dissolve with ArcMap 10.4.1 but received invalid topology or topoengine errors, sometimes with (out of memory), even though I am running a pc with 12GB RAM. I am trying to run it on ArcGIS Pro since that automatically utilizes my computers 64-bit geoprocessing capability and it worked well for a while, getting up to 87% in about 12 hours but it has been stuck at 87 for the past 8 hours. I really do not want to cancel and have all this time wasted. Any recommendations?

[JosephWalderman1]

Only other thing I can think of it to try multi part but I feel like that would mess with the lines especially with such high-res data

[ChrisDonohue__GISP]

Some ideas:

• Maybe take an approach of merging the data into say 8 "chunks", dissolving each, then merging all 8 and dissolving again.
• Shapefiles do not store parametric curves, File Geodatabases do, so going from Shapefiles to File Geodatabase can lead to many slivers generated, which can then lead to quite a bit of topology cleanup.
• ESRI Help article - Tiled processing of large datasets  ArcGIS Help (10.2, 10.2.1, and 10.2.2)

Chris Donohue, GISP 

[JosephWalderman1]

I am not exactly sure how to partition data like that, how would that work?

[DanPatterson_Retired]

do the merge in Pr in a geodatabase, a shapefie can't be > 2 GB so you are fighting a losing battle there if you have to have everything in one layer and can't partition.  speaking of partition, you could spatially tile your data and run it on the bits, but I would consider either partition spatially or separate your attributes and rejoin llater..  I don't know if woring on non-locally saved data would improve things.  What sort of spatial resolution are you using with the raster to polygon? and is it needed, given the data were in raster format to begin with.  I assume that you didn't try to make a finer resolution than existed in the raster.

[DarrenWiens2]

Actuuuually, the .shp + .dbf can exceed 2GB (but neither individually). Just a fun fact: FAQ: Are there file size limitations for shapefiles?

Anyhow, since some of your errors seem to indicate geometry issues, you may want to run Check Geometry or Repair Geometry on your feature class.

[DanPatterson_Retired]

I always forget, since I rarely have attributes other than FID, Shape and ID ... my bad for those in attribute world  and not in geometry world

[JosephWalderman1]

So does it seem like it would be best to just abandon the dissolve at 87%?

[DarrenWiens2]

I suppose you've considered staying in raster-world until you absolutely have to switch to vector? "Raster is faster" (although vector is correcter...)

[JosephWalderman1]

I'm not exactly sure how I would be able to stay in raster world. I am trying to calculate core habitat so I need to dissolve by land cover type at some point and I am not sure I can get there by raster.