Need workflow for dissolving very large dataset

[OvidioRivero]

I have a large feature class (22 million polygons) that I need to dissolve based on a series of attributes.   The process crashes after several hours. I have run repair geometry and exported to a new geodatabase to create new spatial indexes but it still fails.   I am using a relatively powerful computer, 2.8 GHZ Quad Core with 4 GB of RAM with Win XP and ArcGIS 10.   The dissolve tool crashes with   "ERROR 999999: Error executing function. Invalid Topology [Topoengine error.] Failed to execute (Dissolve)."  Looking at the old forums I found that it seems it is a system resources issue, the computer is not powerful enough for the operation.  I  have also tried doing the same operation on FME  but it fails with an "insufficient memory for the operation" error. 

Does any one have any suggestions on how to break the operation into smaller operations?

For example I was thinking about taking the largest features and dicing them into smaller ones. Or breaking the entire dataset into tiles, process each tile and merge them together at the end but I am afraid that as I get closer to the final output I will start running into problems again.

[JamesBoyden]

H i there, Just wondering if there has been any feedback for addressing this error.  I am using ArcGIS Desktop 9.3.1 and am getting exactly the same error when atempting to dissolve a large dataset: 

Invalid Topology [Topoengine error.]. 

Cheers
James

[SarahGreenlaw]

Hi James!

I checked out this older post: http://forums.esri.com/thread.asp?c=93&f=983&t=289514
and followed the last person's directions. 

I used the "Repair Geometry Tool" (it took 18 minutes as my dataset was 122 MB)
and then I used the "Dissolve" tool again, and it worked!

Hope it works for you too!

Sarah

[ChuckStelzner]

I've actually had some success with moving my own work from 32-bit Win7 (didn't work) to 64-bit Win7 (worked better). By "better", I mean that I was able to dissolve by a field, combining some of the features but not all of them together thus far. This may be the better option in the short-term.

[OvidioRivero]

Sorry I never gave an update on this. I was able to solve the problem by breaking the work in chunks.  For example if working with the entire US break it in to four parts dissolve each then put all results into one layer and dissolve gain. Also before running the dissolve it is good practice to check or repair geometry.

[WeifengHe]

As a general guideline, when running into this ERROR 999999, it is always a good practice to run the geoprocessing tool Check Geometry and Repair Geometry.

The dissolving tool can create features with too many vertices (what we call Godzillas, since they can cause problem with your computer resources) and impact editing, rendering and geoprocessing.  A new tool called Dicing has been developed to deal with Godillas in ArcGIS 10.0.

In this particular situation, you may also need to consider do you really need so many vertices to describe your geometry.  You may use Simplfy Polygon or Simplify Polyline tools to get rid off the unnecessay vertices.  Also for performance reasons mentioned above, you may also want to limit the number of features in a feature class.