How can I speed up a simple dissolve?

KimOllivier

I have a singlepart featureclass of polygons (2.5M) with an indexed key that I want to convert to multipart polygons. There are less than 0.5% (12,480) that are actually multipart, the rest are single parts. They are all reasonable multiparts, nothing silly that spans the whole dataset, no universal polygon or unreasonably large polygon such as a huge road casing.

Since there are only a few to process I thought: "Aha, select the polygons with multiple keys, process these and merge them back with the remaining singles". That dissolve of the subset does complete in human time.

But I can't complete the process because merging single and multipart features is not permitted. Who knew that? I thought multiparts were to be avoided for analysis, now I know.

Is there a way of merging multiparts without rewriting the entire tool in Python? I didn't think that multipart was a different feature type from single part. I had thought that a single part feature was just a multipart with one part. Is that not so?

The simple way (Dissolve_management) on the whole featureclass takes hours for sample sets (and fails to complete when it runs out of memory on the full set), giving me time to write this... ah here it is: 'unable to read memory, crashed'.

I should expect this by now, nearly any process on a large dataset seems to run out of memory far too often. Just a simple relate with a selection (keyfile select) takes hours just to select and fails to write a subset, compared to Workstation which takes seconds. I have tried using a Python set and creating an SQL query as a layer but this is also very slow when it comes to writing out the records.

Maybe I should just write out the records when running a cursor to test the key against the set and write them out manually? But that is rewriting the tool which I hoped to avoid. I am using local file geodatabases and 9.3, Duo Quad (8 processors) and 4 GB of memory,  TB disk.

Any other ideas?

ChrisSnyder

merging single and multipart features is not permitted

Kim, I don't seem to have any issues creating a FC with mixed states of single and multipart (made a singlepart dissolve, and then a multipart dissolve, and merge together - no error). What error do you get? I think maybe I'm not grasping your issue?

My understanding is that the stupid ArcGIS Dissolve tool ALWAYS  makes a multipart FC, and then depending on what you want, will break it into singlepart as a post process. Unfortunately I've found that that doesn't even work sometimes, and that you need to run the MultipartToSinglepart tool after you run the Dissolve (singlepart) tool to fully break it up. There is some logged bug about this... Coverages were easier as everything was always singlepart - no surprises!

Relates: I found a pretty slick way of doing large relates in a FGDB... The cool thing is you can pass EXTREEMLY large SQL statements to a FGDB - like millions of characters long... I never did find a limit... So instead of relying on the ArcGIS Relate "tool". I just gather the "key" field values in a search cursor, and use that to build a SQL expression to select the relate table with. It's even more efficient when using the OBJECTID fields cause you can use the .fidset describe property to gather the selected OIDs.