Large dataset failures using intersect or spatia join.

639
2
05-09-2012 10:45 AM
NathanielRoth
New Contributor III
I receive a topology error notification when intersecting a point dataset (with ~5 million points), with a polygon dataset created by buffering those points by a half mile. The goal is to create a table containing the intersection of the two datasets such that I have a list of all points within that half mile radius of each starting point. I can generate effectively identical results using either an intersection or a spatial join.

My prototype of this process works fine when I work on a small subset of each dataset. When I scale up to the full dataset, the intersect operation fails with a topology error, and the spatial join fails with an out of memory error (which is plausible given the dataset size, and memory addressing limitations of a 32bit application). Is this

Much of the time I do these operations in PostGIS (successfully and easily), but on this project I'm constrained to working in ArcGIS, with the assumption that my users will have only the ArcView level of licensing. I've also done these operations in spatiallite. I'd really rather not have to pull in OGR2OGR to move the datasets to spatiallite for the processing, but can if I must.

Machine specs: Intel Core2 Quad (Q9550), Windows 7 (64bit), 8GB of ram, plenty of hard drive space
0 Kudos
2 Replies
MathewCoyle
Frequent Contributor
Try a repair geometry on your datasets.
0 Kudos
NathanielRoth
New Contributor III
The geometries show as good based on the repair geometry tool.

On gis.stackexchange.com a responder suggested that a 4GB limit on scratchspace filegeodatabases may be at fault.
0 Kudos