Dissolve issues with very large merged landcover dataset

[JosephWalderman1]

I have taken a large mosaic land cover dataset for the James River Watershed and manipulated it through iterate, clip, and raster to polygon then merging all the resulting shapefiles into a file geodatabase. The resulting merged file is quite large, 8GB and 22 million features. I tried to run the dissolve with ArcMap 10.4.1 but received invalid topology or topoengine errors, sometimes with (out of memory), even though I am running a pc with 12GB RAM. I am trying to run it on ArcGIS Pro since that automatically utilizes my computers 64-bit geoprocessing capability and it worked well for a while, getting up to 87% in about 12 hours but it has been stuck at 87 for the past 8 hours. I really do not want to cancel and have all this time wasted. Any recommendations?

[DarrenWiens2]

What are you getting by dissolving on land cover type that you can't get from a raster? It sounds like maybe you've thought this through, but many people are just more comfortable with vector so they try to force the data into that format, unnecessarily.

Also, do you need all land cover types or only those relevant to core habitat? Con or Reclassify may be able to simplify your raster before converting to vector. The point is to have the fewest huge polygons. It takes a long time to convert a million square pixels, but a short time to convert just a few classes.

[JosephWalderman]

All I need is the forest classification which is classified by a gridcode. I am not exactly familiar with either of those tools though, how would that work?

[DarrenWiens2]

In Map Algebra -> Raster Calculator, you can use an expression like:

Con("landcover"==5,1)

Translation: for every pixel in the land cover raster, if land cover value is 5 (or whatever your forest gridcode is), make the output value 1 (or whatever value you want), otherwise make the value NoData. Now when you convert to polygon, it will only consider 1 vs NoData, so many less separate conversions.

[JosephWalderman]

Thank you so much for this help. If you will note, though, my model below, I needed to overlay the landcover mosaic raster with the Catchment layer, which is what has the classifications. So I do not believe I can do it just from the raster unfortunately.

[DarrenWiens2]

If your catchment layer has classes (landcover classes?) then what does your landcover raster have? I'm clearly missing something here.

[ChrisDonohue__GISP]

Keep in mind that even if your Catchment layer is in vector, you can convert it to raster and do the overlay in raster (using Map Algebra), then convert back.

Chris Donohue, GISP

[ChrisDonohue__GISP]

Darren Wiens has a good idea - keep it all raster until the end.  His suggestion of using Con or Reclassify accomplishes a similar effect to Dissolve, but on the raster side.

To use Con or Reclassify, one would use Map Algebra, which is available with the Spatial Analyst Extension.

A quick tour of using Map Algebra

ArcGIS Help (10.2, 10.2.1, and 10.2.2)

Chris Donohue, GISP

[JosephWalderman1]

I can not do any tiling unfortunately my boss says that it will will create way too much overlap between polygons as it tries to smooth the edges. with this type of work, I need to keep the hard boundaries on the pixels so they don't overlap. The model that I used to get the shapefiles in attatched here as an image. When I did the merge the output was a feature class in a file geodatabase so that should not be an issue. I am not entirely sure how to go about doing this partitioning. Should I just give up on the dissolve at 87% and not waste my time?

[JosephWalderman1]

Additionally I am on a bit of a time deadline and running almost any geoprocessing tool, even just a check geometry, will likely take an hour at the very least, so a solution with the minimal number of steps would really be optimal, but I have been working on this so long that I will try anything at this point really.

[DanPatterson_Retired]

Can you clarify why you need to use modelbuilder?  This appears to be a workflow that will be done very infrequently.  I don't understand the boundaries/edges and smoothing.  You are largely working with rasters.  boundaries, edges etc are handled  by specifying cell size, extent and snap raster, so this is a non issue.  I still think tiling or some other spatial selection/arrangement will speed things up since the whole can overwhelm what can be done in parts.