Error: ???two donuts or two outer shells overlap???,

[Anonymous User]

Original User: Jamal432@gmail.com

Error: �??two donuts or two outer shells overlap�?�,

I wanted to copy enterprise geodatabase layer to file geodatabase but I got the error below (�??two donuts or two outer shells overlap�?�):

[ATTACH=CONFIG]33329[/ATTACH], [ATTACH=CONFIG]33330[/ATTACH], [ATTACH=CONFIG]33331[/ATTACH]

What might be the issue here?

Thank you

Best

Jamal

[MarcoBoeringa]

�?� Correct. I didn�??t apply the �??Geometry validator�?� for the Roads layer and Boundary layer. My final layer is neither the Roads nor the Boundary but one layer that combines both Land use and Roads. This is why I didn�??t apply the �??Geometry validator�?� Roads or Boundary but I did before the �??LanduseAndRoadsPolygons_FME�?� Layer. This is the layer that is collected in the enterprise layer as it represents one urban master plan for a particular community.

OK, misunderstood that part.

�?� Topology\repair geometry\integrated could be solutions. What is horrible is that the enterprise layer accepts features with errors, and that is in turn, generates big problems such as �??two donuts or two outer shells overlap�?� and leads to deadlock.

I think that the enterprise layer (SQL or Oracle) should be designed not to accept features with errors as this will lead to destroy the data.

Yes and No. ESRI already applies one of the strictest models for valid geometries in enterprise geodatabases in the industry. I am still not sure how to judge this "loop hole" you found with this layer, whether it is a bug or "by design"... I am still eager to hear more from Vince or other ESRI staff regarding this particular issue, especially since the ESRI enterprise geodatabases, not allowing Check Geometry and Repair Geometry to be run, are "supposed" to contain valid shapes. As the ArcGIS Help says in the Check Geometry page:

"SDE Geodatabases automatically check the validity of each geometry when they are uploaded; therefore the Check Geometry and Repair Geometry tools are not for use with SDE."

On the other hand, databases will always be kind of "open". The geometry validation as implemented by ArcSDE, is a layer "on top" of the enterprise database. You could always circumvent this by, for example, using a SQL client (e.g. SQL Server Management Studio, Oracle SQL*Plus) to write out invalid shapes and thus still cause issues.

This also highlights the need (and you are already working on this!), to validate the data before loading in any conceivable way, especially if suspected "dirty" data from a third party system needs to be ingested.

[Anonymous User]

Original User: Jamal432@gmail.com

OK, misunderstood that part.



Yes and No. ESRI already applies one of the strictest models for valid geometries in enterprise geodatabases in the industry. I am still not sure how to judge this "loop hole" you found with this layer, whether it is a bug or "by design"... I am still eager to hear more from Vince or other ESRI staff regarding this particular issue, especially since the ESRI enterprise geodatabases, not allowing Check Geometry and Repair Geometry to be run, are "supposed" to contain valid shapes. As the ArcGIS Help says in the Check Geometry page:

"SDE Geodatabases automatically check the validity of each geometry when they are uploaded; therefore the Check Geometry and Repair Geometry tools are not for use with SDE."

On the other hand, databases will always be kind of "open". The geometry validation as implemented by ArcSDE, is a layer "on top" of the enterprise database. You could always circumvent this by, for example, using a SQL client (e.g. SQL Server Management Studio, Oracle SQL*Plus) to write out invalid shapes and thus still cause issues.

This also highlights the need (and you are already working on this!), to validate the data before loading in any conceivable way, especially if suspected "dirty" data from a third party system needs to be ingested.

Many thanks Marco for the valuable remarks.

I do agree. The only option that I have to be on safe side is to validate the geometry before loading it in the enterprise layer.

Best

Jamal

[MarcoBoeringa]

Back to my issue:
Please, consider the attached file geodatabase layer. It has �??self intersections�?� error. Nevertheless, Copy\Paste from file geodatabse to enterprise works fine without fixing the error (repair geometry).

[ATTACH=CONFIG]33421[/ATTACH], [ATTACH=CONFIG]33422[/ATTACH], [ATTACH=CONFIG]33423[/ATTACH]

My question here: how the Enterprise Geodatabase Layer (SQL) accepts features that have error?

Jamal,

I now had a more thorough look at the sample data you provided as a File Geodatabase in above post. I can confirm:

- That the Feature Class it contains (LanduseAndRoadsPolygons_FME) has an error in the last geometry with OBJECTID 51, as detected by Check Geometry and reported as "self intersections".

- Using Copy/Paste to a Desktop Geodatabase in SQL Server 2012 Express doesn't lead to automatic fixing of the geometry in question (contrary to what the Check Geometry Help topic suggests). If I copy the Feature Class back to the FGDB and run Check Geometry again, it is still reported as having self intersections.

- Using Import/Export (which uses the Feature Class to Feature Class tool) instead of Copy/Paste gives a similar result: the invalid geometry is happily transferred from the Desktop Geodatabase to the FGDB and back, without any automated fixes applied. So I can't reproduce the "difference" Vince described in this post in the thread.

- I can not reproduce the "two donuts or outer shells overlap" issue you described, but since you provided another small test dataset with a detected error, and not the large one that initiated this thread, this is no real surprise.

- I also tried to load the data with the erroneous geometry in the Open Source QGIS (2.2.0 Valmiera). QGIS has an option for checking geometries as well. I opened both straight the FGDB Feature Class (which is possible using the Add Vector Layer option and choosing "folder" instead of "file"), and a converted shapefile version of the layer, but QGIS is unable to detect any geometric error in both of these... so it considers them valid (may have to do with the type of validation checks).

It starts to look like a bug / loop hole that may need fixing. Sorry to see no one else of ESRI staff responded yet and did some tests with this data, although an official support call is the way to go here.

[Anonymous User]

Original User: Jamal432@gmail.com

Jamal,

I now had a more thorough look at the sample data you provided as a File Geodatabase in above post. I can confirm:

- That the Feature Class it contains (LanduseAndRoadsPolygons_FME) has an error in the last geometry with OBJECTID 51, as detected by Check Geometry and reported as "self intersections".

- Using Copy/Paste to a Desktop Geodatabase in SQL Server 2012 Express doesn't lead to automatic fixing of the geometry in question (contrary to what the Check Geometry Help topic suggests). If I copy the Feature Class back to the FGDB and run Check Geometry again, it is still reported as having self intersections.

- Using Import/Export (which uses the Feature Class to Feature Class tool) instead of Copy/Paste gives a similar result: the invalid geometry is happily transferred from the Desktop Geodatabase to the FGDB and back, without any automated fixes applied. So I can't reproduce the "difference" Vince described in this post in the thread.

- I can not reproduce the "two donuts or outer shells overlap" issue you described, but since you provided another small test dataset with a detected error, and not the large one that initiated this thread, this is no real surprise.

- I also tried to load the data with the erroneous geometry in the Open Source QGIS (2.2.0 Valmiera). QGIS has an option for checking geometries as well. I opened both straight the FGDB Feature Class (which is possible using the Add Vector Layer option and choosing "folder" instead of "file"), and a converted shapefile version of the layer, but QGIS is unable to detect any geometric error in both of these... so it considers them valid (may have to do with the type of validation checks).

It starts to look like a bug / loop hole that may need fixing. Sorry to see no one else of ESRI staff responded yet and did some tests with this data, although an official support call is the way to go here.

Thank you very much Macro for the massive work.

Then our issue is clear now. The enterprise Geodatabase (SQL, Oracle, etc.) doesn�??t reject features with errors as they are loaded to a particular enterprise layer despite the fact that accepting features with errors in an enterprise layer leads to destroy the data.


I hope ESRI experts can provide their inputs in this regard. This issue is critical as it leads to data loss.

[VinceAngelo]

I hope ESRI experts can provide their inputs in this regard. This issue is critical as it leads to data loss.

I think this description mischaracterizes the problem. You have bad data on input, and insufficient processing to correct the errors. The data loss occurred long before it entered the geodatabase. Sometimes you need to code your own algorithms for data recovery, and sometimes data recovery isn't fully possible. In over 25 years as a data conversion specialist, I've run into quite a few sources that were too corrupt to recover, even if we were to spend a million dollars on the effort. At that point, you need a client/manager who isn't willing to send any more good money after bad, no matter how promising the next recovery attempt looks. - V

[Anonymous User]

Original User: mboeringa2010

I think this description mischaracterizes the problem. You have bad data on input, and insufficient processing to correct the errors. The data loss occurred long before it entered the geodatabase. Sometimes you need to code your own algorithms for data recovery, and sometimes data recovery isn't fully possible. In over 25 years as a data conversion specialist, I've run into quite a few sources that were too corrupt to recover, even if we were to spend a million dollars on the effort. At that point, you need a client/manager who isn't willing to send any more good money after bad, no matter how promising the next recovery attempt looks. - V

Although I basically agree with most of what you write (which can be exemplified by the well known "garbage in - garbage out"), I don't think it is fair towards Jamal to suggest he isn't doing his best to get this right.

I have seen my share of data conversion / quality issues in the past 20 years, not likely to be anywhere as broad as your experiences though, but enough to well know that at some point you have to fix things in production itself, and possibly significantly adjust workflows including the introduction of new software like for example AutoCAD Map 3D, to allow more stringent quality control at the data production sites via geometry and topology checks in the native software, instead of by trying to "fix" things afterwards in GIS.

That said, considering as I understood it, Jamal needs to work with a range of data producers in his country Palestine (different municipalities and such), I think this task is a very tough one, even for way better positioned countries like the Netherlands and the US (much longer history of GIS / CAD usage, way more budget and expertise: there isn't even a local distributor for ESRI in Palestine if I have to believe the ESRI website...).

Here in the Netherlands, they are now going into a huge multi million project in an attempt to build a uniform large scale (1:100-1:1000) object oriented map of the entire country (point, line, polygon). This will require the cooperation of literally hundreds of parties that already maintain the base data (e.g. country, state and municipality level). Data will be stored in one central database. Statistics for this are daunting: expected 100 million(!) geometries, with over 1 billion vertices, with a need to store history as well.

Although much focus has been put on automated geometric integrity and topology checks and automated assembly prioritizing high precision data, only the near future will tell what plethora of data conversion and data quality issues they will run into... It is likely to open Pandora's box for some organizations... while others probably fare well.

To get back to the real unanswered question: it would still be nice to hear some comments by ESRI as to why the particular geometry is not rejected / auto-fixed. I realize this is a "Technical Support" matter, but my ArcGIS for Home Use license doesn't allow me to start a support call. Jamal may, but is also not in the best position considering no local support in his country.

Well, it is probably going to remain unanswered for now...

[JamalNUMAN]

Although I basically agree with most of what you write (which can be exemplified by the well known "garbage in - garbage out"), I don't think it is fair towards Jamal to suggest he isn't doing his best to get this right.

I have seen my share of data conversion / quality issues in the past 20 years, not likely to be anywhere as broad as your experiences though, but enough to well know that at some point you have to fix things in production itself, and possibly significantly adjust workflows including the introduction of new software like for example AutoCAD Map 3D, to allow more stringent quality control at the data production sites via geometry and topology checks in the native software, instead of by trying to "fix" things afterwards in GIS.

That said, considering as I understood it, Jamal needs to work with a range of data producers in his country Palestine (different municipalities and such), I think this task is a very tough one, even for way better positioned countries like the Netherlands and the US (much longer history of GIS / CAD usage, way more budget and expertise: there isn't even a local distributor for ESRI in Palestine if I have to believe the ESRI website...).

Here in the Netherlands, they are now going into a huge multi million project in a attempt to build a uniform large scale (1:100-1:1000) object oriented map of the entire country (point, line, polygon). This will require the cooperation of literally hundreds of parties that already maintain the base data (e.g. country, state and municipality level). Data will be stored in one central database. Statistics for this are daunting: expected 100 million(!) geometries, with over 1 billion vertices, with a need to store history as well.

Although much focus has been put on automated geometric integrity and topology checks and automated assembly prioritizing high precision data, only the near future will tell what plethora of data conversion and data quality issues they will run into... It is likely to open Pandora's box for some organizations... while others probably fare well.

To get back to the real unanswered question: it would still be nice to hear some comments by ESRI as to why the particular geometry is not rejected / auto-fixed. I realize this is a "Technical Support" matter, but my ArcGIS for Home Use license doesn't allow me to start a support call. Jamal may, but is also not in the best position considering no local support in his country.

Well, it is probably going to remain unanswered for now...

Many thanks Vince and Macro for the contribution.

Precisely Macro. You have perfectly explained how the situation looks like is in developing countries such as Palestine when it comes to GIS.

However, the challenge is still there: I do believe in �??garbage in-garbage out�?� but it is the time not to let garbage to be in.

At the moment, all what we do is converting TENS of urban master plans from CAD format to GIS using the FME. The end product by FME is feature class (stored in geodatabase). These feature classes (that represents urban master plans) are collected in one enterprise feature class. At this point, we don�??t need this enterprise layer to accept features with errors. THAT�??S ALL.

How to solve the issue of �??two donuts or two outer shells overlap�?� in the enterprise geodatabase (SQL, Oracle, etc.)?

[Anonymous User]

Original User: mboeringa2010

Many thanks Vince and Macro for the contribution.

It is Marco, macros is what you run to fix your data 😉

How to solve the issue of �??two donuts or two outer shells overlap�?� in the enterprise geodatabase (SQL, Oracle, etc.)?

The specific problem of the geometry with "self intersections" as discussed above, can be repaired by Repair Geometry.

I don't have an answer for the "donuts/outer shells" issue, nor can probably provide one, but could you potentially post this entire layer, maybe in parts if to big to post in one ZIP, that is causing these issues? You could drop unnecessary attribute fields to reduce the size somewhat. Without that, it is impossible to say more.

[JamalNUMAN]

It is Marco, macros is what you run to fix your data 😉



The specific problem of the geometry with "self intersections" as discusses above, can be repaired by Repair Geometry.

I don't have an answer for the "donuts/outer shells" issue, nor can probably provide one, but could you potentially post this entire layer, maybe in parts if to big to post in one ZIP, that is causing these issues? You could drop unnecessary attribute fields to reduce the size somewhat. Without that, it is impossible to say more.

Very much appreciated Marco. Sorry for the typo.

At the moment I don�??t have this problem (�??two donuts or two outer shells overlap�?�). As this problem appeared, I delete the entire enterprise layer and replace it with the correct one from the backup.

Now, this problem is like a nightmare for me. From now on, I�??ll never load data to the enterprise layer without repairing it.

Best

Jamal

[Anonymous User]

Original User: mboeringa2010

Now, this problem is like a nightmare for me. From now on, I�??ll never load data to the enterprise layer without repairing it.

That is very sensible, and good practice, especially considering the source of the data (not ArcGIS).