Detect and remove overlaps resulting from polygons with crossing edges

[Aline-Wbg]

I have a polygon layer with overlapping polygons (drawn on top of each other not adjacent and only contiguous for two edges (polygon 1 and 4 on the outer rim)) that should not have any overlap that results from crossing polygon edges (the overlap between polygon 1 and 5 and the one between 2 and 3, see below, should be eliminated and being merged with the bigger of the involved overlapping polygon). Overlaps between polygons 1 and 2,  1 and 3, 1 and 4, and 1 and 6 are to be kept as no edge crossing is taking place. 

For context: the polygons are meant to be in a print product where edges are set to overprint, so polygons overlapping are not spared out and edges crossing another feature will be visible.

When I run the Remove Overlap (multiple) (Analysis) tool with the Thiessen method I end up with 2 valid features (below), while 4 have no geometry anymore, as overlaps combines features that are on top of each other. This tool only produces a target result for overlapping features with crossing edges that are not fully contained within another feature in the layer.

When I create a topology where I don't allow overlaps I end up with the following polygon errors. 

But only the two highlighted in yellow are the ones I would be interested fixing.

Manually and with few polygons this is all doable, but I have layers with many polygons and would like to automatize that, without double-checking if the overlap is an overlap with crossing edges or just a polygon fully contained in another one.

The Intersect (Analysis) tool results in many duplicate geometries and removes the biggest polygon.

If there's only a handful of polygons some manual handling (to treat only the overlaps with crossing edges) with the help of topology is doable, but I'm looking at polygon layers with many polygons and I would like to automate this.

If there was a way to do the intersection that keeps all the polygons, then detect polygons that have vertices in another polygon (so not a detached polygon) and if they do NOT all of them (which would imply crossing edges) I could then use Eliminate (Data Management) tool and merge all the selected polygons with the neighboring polygon with the largest area for instance.

I would be happy for an input in order to solve that specific problem.

[BrennanSmith1 · Accepted solution]

I think you can identify those specific overlapping conditions using the Polygon geometry and arcpy. 

from itertools import combinations

#define input polygon layer
poly_fc = r"input_polygon"

#initialize an empty overlap layer to insert into
arcpy.management.CreateFeatureclass(arcpy.env.workspace, 'overlap_polygon', "POLYGON")
arcpy.management.AddField('overlap_polygon', 'ID', 'DOUBLE')
counter = 1

# Get all geometries
geo_list = [row[0] for row in arcpy.da.SearchCursor(poly_fc,'SHAPE@')]

# pairwise compare all geometries
for poly, compare in combinations(geo_list, 2):
    # test for disjoint (if they don't touch at all, stop here)
    if not poly.disjoint(compare):
        # test for contain/within (if fully enclosed by another, stop here)
        if not (poly.contains(compare) or poly.within(compare)):
            # write the intersecting geometry to our overlap_polygon feature class.
            intersect_geo = poly.intersect(compare,4)
            # check for zero area
            if intersect_geo.getArea() > 0:
                # Need at least one non-shape field for Insert Cursor which is why we're using a dummy counter
                with arcpy.da.InsertCursor('overlap_polygon', ['ID', 'SHAPE@']) as cursor:
                    cursor.insertRow((counter,intersect_geo))
                    counter+=1

UPDATE:

Actually, I think we can edit the geometries directly with this approach.  Code is sloppy but worked on my demo dataset.

from itertools import combinations

#define input polygon layer
poly_fc = r"input_polygon"

# make a copy because we're going to be editing it in place
arcpy.management.CopyFeatures(poly_fc, 'edited_polygon')

# Get a dictionary of OIDs and geometries
geo_dict = {row[0]:row[1] for row in arcpy.da.SearchCursor('edited_polygon',['OID@','SHAPE@'])}

# pairwise compare all OIDs
for oid1, oid2 in list(combinations(geo_dict, 2)):
    # get geometries for those IDs
    geo1 = geo_dict[oid1]
    geo2 = geo_dict[oid2]
    # test for disjoint (if they don't touch at all, stop here)
    if not geo1.disjoint(geo2):
        # test for contain/within (if fully enclosed by another, stop here)
        if not (geo1.contains(geo2) or geo1.within(geo2)):
            # check for intersecting geometry
            intersect_geo = geo1.intersect(geo2,4)
            if intersect_geo.getArea() > 0:
                # determine which geometry is larger
                if geo1.getArea() > geo2.getArea():
                    # and erase the overlap
                    with arcpy.da.UpdateCursor('edited_polygon', 'SHAPE@','OBJECTID = '+str(oid2)) as cursor:
                        for row in cursor:
                            row[0] = geo2.difference(geo1)
                            cursor.updateRow(row) 
                else:
                    with arcpy.da.UpdateCursor('edited_polygon', 'SHAPE@','OBJECTID = '+str(oid1)) as cursor:
                        for row in cursor:
                            row[0] = geo1.difference(geo2)
                            cursor.updateRow(row) 

[Aline-Wbg]

Dear Brennan thank you so much for your help and putting in your time! Your updated script works perfectly!