topic Re: Check Geometry not finding invalid self-touching polygons in Geoprocessing Questions

Check Geometry not finding invalid self-touching polygons

MartinDavis — Tue, 02 Aug 2011 21:17:53 GMT

I have a dataset with over 24M polygons in it, which I need to load into SDE. I am using FME to perform the load. The load fails due to some polygons having invalid structure according to ESRI rules. The invalid polygons consist of an outer shell and holes which touch the shell at a single point. This is invalid according to the ESRI polygon structure rules (which I have seen documentation for, but can't find a link for right now). Instead, according to the ESRI rules these kinds of area should be formed as "inverted polygons".

The attached screenshot shows an example of an invalid polygon.

Because of the large size of the dataset, I need an automated way to detect and fix these invalid polygons. The obvious tool to do this is Check Geometry and Repair Geometry.The problem is that Check Geometry does not identify these polygons. Also, Repair Geometry does not repair them.

Is this a bug in Check Geometry?
Is Check Geometry expected to work on FGDBs?
Is there another way of detecting and fixing these polygons?

I'm running ArcGIS 10.

Re: Check Geometry not finding invalid self-touching polygons

ChrisSnyder — Tue, 02 Aug 2011 22:53:53 GMT

What is the native format of these polygons? Could it be that FME is causing the problem? Maybe load the datinto a FGDB 1st, clean it up (may not be neccessary), and then load it into SDE?

Re: Check Geometry not finding invalid self-touching polygons

MartinDavis — Wed, 03 Aug 2011 15:25:28 GMT

Thanks for the suggestions...

The native format that we're dealing with is FGDB. I'm pretty sure that FME is not the cause of the problem, since I can inspect the structure of the polygons (using Edit/Sketch Properties) and verify that it contains a "self-touching" hole, rather than being in the ESRI-valid "inverted polygon" structure.

But I agree that fixing it in Arc and then retrying the load is a good idea, to verify the cause of the problem. I don't have immediate access to the load environment, so this will take a little while.

As for fixing the polygons to have correct structure, running them through the Union tool corrects the structure to be ESRI-valid. So this is a potential fix. However, I still need a way to identify all invalid polygons in the 24M-feature FGDB. If Check Geometry won't do this, I'll need to think of some other way of detecting them (possibly via some custom Python code).

Re: Check Geometry not finding invalid self-touching polygons

ChrisSnyder — Wed, 03 Aug 2011 15:31:16 GMT

Dumb question maybe, but why are you using FME to load FGDB data into SDE? Why not just use the build in ESRI tools? Seems the ESRI tools would be more bullet proof for ESRI formats...

Re: Check Geometry not finding invalid self-touching polygons

MartinDavis — Wed, 03 Aug 2011 15:35:33 GMT

Fair question. We're working in the environment of a large corporate spatial data warehouse (the BC government), and the standard pipeline for loading SDE uses FME.

But it would be interesting to know if using ESRI tools would work where FME fails. I don't think that's a long-term solution, however.

Re: Check Geometry not finding invalid self-touching polygons

ChrisSnyder — Wed, 03 Aug 2011 17:00:18 GMT

Don't know about the wisdom of using a 3rd party software to convert an ESRI format into another ESRI format, but rules are rules, and FME is a Canadian product I believe.

After cleaning the data in FGDB format (Union or the RepairGeometry tool) you could run the FeatureCompare tool to identify what features changed. RepairGeom tool should give you a specific list of OBJECTIDs that were "fixed". I have had some bad experiences with the RepairGeometry tool... Be sure to do some random sampling of the features that are "fixed" to make sure it comes out the way it should.

Note that if your data contains overlapping polygons, the union tool will intersect your data (split the geometry) at the overlap bounadries...

Re: Check Geometry not finding invalid self-touching polygons

MartinDavis — Wed, 03 Aug 2011 17:30:04 GMT

Yes, FME is pretty much the home team, so we're all kinda partial to using it where it fits. And generally we've had good experiences with the software and the support (not to mention that the licensing is cheaper!)

We've done some more testing, and found out that:

* indeed, using a pure ESRI pipeline to load the FGDB into SDE (via ArcCatalog) SOLVES the polygon problem. The polygon structure is reshaped into an ESRI-valid structure (an inverted polygon).

* It turns out the the latest FME Beta (2012?) still exhibits the load failure on the invalid data.

* That let us verify that re-shaping the polygon using Union did actually fix the problem, since teh updated dataset loaded into SDE via FME 2012 Beta with no problems.

Thanks for the warning about Union. We realize that it's a pretty blunt tool, so we want to be very careful about what we update using it. Since Check Geometry and Repair Geometry are non-starters about fixing these invalid polygons, I"m hoping that we can write some custom Python code to identify just the polygons in error, and then fix each one individually, to cause the least amount of change in the dataset.

Re: Check Geometry not finding invalid self-touching polygons

KenHartling — Thu, 04 Aug 2011 14:44:06 GMT

Martin,

Could I get a copy of some of these problem polygons? I'm pretty sure I know why the check/repair geometry tools didn't catch it, but I'd like to confirm.

Thanks,
Ken
ESRI
Geoprocessing Product Engineer

Re: Check Geometry not finding invalid self-touching polygons

MartinDavis — Thu, 04 Aug 2011 14:52:04 GMT

Sure. I'm attaching a ZIP file containing an FGDB with 3 polygons. One of them has the invalid structure (the one with 3 holes all touching).

Any assistance you can provide will be very welcome.

Re: Check Geometry not finding invalid self-touching polygons

KenHartling — Thu, 04 Aug 2011 15:35:00 GMT

Thanks. I'll get back to you once I've got some info.
Ken

Re: Check Geometry not finding invalid self-touching polygons

SandyLyle — Thu, 04 Aug 2011 15:43:55 GMT

Thought I'd throw another factor into the pot of thngs to think of.
Database precision can be a problem. We use FME (so much faster and flexible for so many things) but had some trouble with the repair geometry tool and using FME to do the fixes brought some problems with DB precision.

Re: Check Geometry not finding invalid self-touching polygons

MitaHajirakar — Fri, 12 Aug 2011 19:48:31 GMT

Hi Martin,

Thanks for also notifying Safe Software of the problem. We did some testing and found that sending that feature through an AreaBuilder make it load successfully to ArcSDE. Another option for you anyways.

Kind regards,
Mita Hajirakar
Safe Software Inc.

Re: Check Geometry not finding invalid self-touching polygons

KenHartling — Fri, 19 Aug 2011 14:54:43 GMT

We've taken a look at the "offending" features that will not load to ArcSDE using FME, here's what we have found.

1. The features are simple (meaning valid).
2. Using all ArcGIS data loading tools these features successfully load to ArcSDE.
3. ArcSDE's geometry validation routines are slightly different than GDB validation. See ArcSDE validation rules: http://edndoc.esri.com/arcsde/9.2/concepts/geometry/shapes/verificationrules.htm
4. The recommended workflow that should be used to take care of this difference is for 3rd party developers to use SE_generate_from_binary or SgShapeFromBinary to import a simple polygon and make it valid for SDE.

In summary, it appears that FME is not using SE_generate_from_binary or SgShapeFromBinary when loading features to ArcSDE.

Ken

Re: Check Geometry not finding invalid self-touching polygons

JeffKruys — Mon, 29 Oct 2012 16:41:39 GMT

In following up on this self-same problem, I've just stumbled on something. Inverted polygons (ie. polygons with a hole that shares a vertex with the outer shell) that are stored in the FGDB with the SDE-invalid vertex ordering seem to have their vertices rearranged and made SDE-valid by the following:

In an FGDB, create a feature dataset and import the feature class into it

Create a topology that acts on the feature class -- no topology rules are necessary

Validate the topology

See the attached FGDB containing a few polygons before and after the process.

Is this behaviour expected? Are there any potential dangers, eg. could polygon geometries be modified in ways I'm not expecting? I haven't run my entire polygon feature class through this process yet, but I expect this will work on all of the problem polygons I have, and thus solve the problem by fixing the polygons in the FGDB before FME has a chance to choke on them while transferring them to SDE.

Re: Check Geometry not finding invalid self-touching polygons

KenHartling — Mon, 29 Oct 2012 20:05:22 GMT

From the ValidateTopology tool doc:
Validate Topology performs the following operations:
    Cracking and clustering of feature vertices to find features that share geometry (have common coordinates)
    Inserting common coordinate vertices into features that share geometry
    Running a set of integrity checks to identify any violations of the rules that have been defined for the topology

So there are possible modifications to your features. I'd suggest reading the Topology topic at the bottom of the ValidateTopology tool documentation to determine if the changes made to the features are acceptable for your workflow.

Ken