topic Re: Dissolve issues with very large merged landcover dataset in ArcGIS Pro Questions

Dissolve issues with very large merged landcover dataset

JosephWalderman1 — Mon, 18 Jul 2016 20:29:14 GMT

I have taken a large mosaic land cover dataset for the James River Watershed and manipulated it through iterate, clip, and raster to polygon then merging all the resulting shapefiles into a file geodatabase. The resulting merged file is quite large, 8GB and 22 million features. I tried to run the dissolve with ArcMap 10.4.1 but received invalid topology or topoengine errors, sometimes with (out of memory), even though I am running a pc with 12GB RAM. I am trying to run it on ArcGIS Pro since that automatically utilizes my computers 64-bit geoprocessing capability and it worked well for a while, getting up to 87% in about 12 hours but it has been stuck at 87 for the past 8 hours. I really do not want to cancel and have all this time wasted. Any recommendations?

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman1 — Mon, 18 Jul 2016 20:59:14 GMT

Only other thing I can think of it to try multi part but I feel like that would mess with the lines especially with such high-res data

Re: Dissolve issues with very large merged landcover dataset

ChrisDonohue__GISP — Mon, 18 Jul 2016 21:10:36 GMT

Some ideas:

Maybe take an approach of merging the data into say 8 "chunks", dissolving each, then merging all 8 and dissolving again.
Shapefiles do not store parametric curves, File Geodatabases do, so going from Shapefiles to File Geodatabase can lead to many slivers generated, which can then lead to quite a bit of topology cleanup.
ESRI Help article - Tiled processing of large datasets ArcGIS Help (10.2, 10.2.1, and 10.2.2)

Chris Donohue, GISP

Re: Dissolve issues with very large merged landcover dataset

DanPatterson_Retired — Mon, 18 Jul 2016 21:11:03 GMT

do the merge in Pr in a geodatabase, a shapefie can't be > 2 GB so you are fighting a losing battle there if you have to have everything in one layer and can't partition. speaking of partition, you could spatially tile your data and run it on the bits, but I would consider either partition spatially or separate your attributes and rejoin llater.. I don't know if woring on non-locally saved data would improve things. What sort of spatial resolution are you using with the raster to polygon? and is it needed, given the data were in raster format to begin with. I assume that you didn't try to make a finer resolution than existed in the raster.

Re: Dissolve issues with very large merged landcover dataset

DarrenWiens2 — Mon, 18 Jul 2016 21:17:04 GMT

Actuuuually, the .shp + .dbf can exceed 2GB (but neither individually). Just a fun fact: FAQ: Are there file size limitations for shapefiles?

Anyhow, since some of your errors seem to indicate geometry issues, you may want to run Check Geometry or Repair Geometry on your feature class.

Re: Dissolve issues with very large merged landcover dataset

DanPatterson_Retired — Mon, 18 Jul 2016 21:31:20 GMT

I always forget, since I rarely have attributes other than FID, Shape and ID ... my bad for those in attribute world and not in geometry world

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman1 — Mon, 18 Jul 2016 21:35:40 GMT

I can not do any tiling unfortunately my boss says that it will will create way too much overlap between polygons as it tries to smooth the edges. with this type of work, I need to keep the hard boundaries on the pixels so they don't overlap. The model that I used to get the shapefiles in attatched here as an image. When I did the merge the output was a feature class in a file geodatabase so that should not be an issue. I am not entirely sure how to go about doing this partitioning. Should I just give up on the dissolve at 87% and not waste my time?

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman1 — Mon, 18 Jul 2016 21:40:48 GMT

Additionally I am on a bit of a time deadline and running almost any geoprocessing tool, even just a check geometry, will likely take an hour at the very least, so a solution with the minimal number of steps would really be optimal, but I have been working on this so long that I will try anything at this point really.

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman1 — Mon, 18 Jul 2016 21:47:41 GMT

So does it seem like it would be best to just abandon the dissolve at 87%?

Re: Dissolve issues with very large merged landcover dataset

DarrenWiens2 — Mon, 18 Jul 2016 22:00:36 GMT

I suppose you've considered staying in raster-world until you absolutely have to switch to vector? "Raster is faster" (although vector is correcter...)

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman1 — Mon, 18 Jul 2016 22:09:40 GMT

I'm not exactly sure how I would be able to stay in raster world. I am trying to calculate core habitat so I need to dissolve by land cover type at some point and I am not sure I can get there by raster.

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman1 — Mon, 18 Jul 2016 22:11:54 GMT

I am not exactly sure how to partition data like that, how would that work?

Re: Dissolve issues with very large merged landcover dataset

DarrenWiens2 — Mon, 18 Jul 2016 22:31:32 GMT

What are you getting by dissolving on land cover type that you can't get from a raster? It sounds like maybe you've thought this through, but many people are just more comfortable with vector so they try to force the data into that format, unnecessarily.

Also, do you need all land cover types or only those relevant to core habitat? Con or Reclassify may be able to simplify your raster before converting to vector. The point is to have the fewest huge polygons. It takes a long time to convert a million square pixels, but a short time to convert just a few classes.

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman — Mon, 18 Jul 2016 22:50:55 GMT

All I need is the forest classification which is classified by a gridcode. I am not exactly familiar with either of those tools though, how would that work?

Re: Dissolve issues with very large merged landcover dataset

DarrenWiens2 — Mon, 18 Jul 2016 22:57:06 GMT

In Map Algebra -> Raster Calculator, you can use an expression like:

Con("landcover"==5,1)

Translation: for every pixel in the land cover raster, if land cover value is 5 (or whatever your forest gridcode is), make the output value 1 (or whatever value you want), otherwise make the value NoData. Now when you convert to polygon, it will only consider 1 vs NoData, so many less separate conversions.

Re: Dissolve issues with very large merged landcover dataset

ChrisDonohue__GISP — Mon, 18 Jul 2016 23:01:24 GMT

Darren Wiens has a good idea - keep it all raster until the end. His suggestion of using Con or Reclassify accomplishes a similar effect to Dissolve, but on the raster side.

To use Con or Reclassify, one would use Map Algebra, which is available with the Spatial Analyst Extension.

A quick tour of using Map Algebra

ArcGIS Help (10.2, 10.2.1, and 10.2.2)

Chris Donohue, GISP

Re: Dissolve issues with very large merged landcover dataset

JosephWalderman — Mon, 18 Jul 2016 23:58:40 GMT

Thank you so much for this help. If you will note, though, my model below, I needed to overlay the landcover mosaic raster with the Catchment layer, which is what has the classifications. So I do not believe I can do it just from the raster unfortunately.

Re: Dissolve issues with very large merged landcover dataset

DarrenWiens2 — Tue, 19 Jul 2016 03:59:29 GMT

If your catchment layer has classes (landcover classes?) then what does your landcover raster have? I'm clearly missing something here.

Re: Dissolve issues with very large merged landcover dataset

DanPatterson_Retired — Tue, 19 Jul 2016 06:06:37 GMT

Can you clarify why you need to use modelbuilder? This appears to be a workflow that will be done very infrequently. I don't understand the boundaries/edges and smoothing. You are largely working with rasters. boundaries, edges etc are handled by specifying cell size, extent and snap raster, so this is a non issue. I still think tiling or some other spatial selection/arrangement will speed things up since the whole can overwhelm what can be done in parts.

Re: Dissolve issues with very large merged landcover dataset

ChrisDonohue__GISP — Tue, 19 Jul 2016 06:11:13 GMT

Keep in mind that even if your Catchment layer is in vector, you can convert it to raster and do the overlay in raster (using Map Algebra), then convert back.

Chris Donohue, GISP