Insert missing features from one layer to another

[CCWeedcontrol]

I need to insert missing features from one layer to another. I don't want to use the append tool because I don't want a new dataset, I just want to insert the missing features. Both are polygons and have pretty much the same fields except for 3 fields ('Field1','Field2', 'Field3'). I have the following but I get no error when I run it.

import arcpy

fc1 = "lyTB" 
targetFC = "lyA_2"

dsc = arcpy.Describe(targetFC)
fields = dsc.fields
out_fields = [dsc.OIDFieldName, dsc.lengthFieldName, dsc.areaFieldName, 'Field1','Field2', 'Field3']
fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields]

keepList = []
with arcpy.da.SearchCursor(fc1, fieldnames) as cursor:
    for row in cursor:
        keepList.append(row[0])
        #print(keepList)
del cursor

#list of values from fc1 to inject into targetFC
ids = []

with arcpy.da.SearchCursor(fc1, fieldnames) as cursor:
    for row in cursor:
        #print (row)
        if row[0] not in keepList:
            ids.append(row)
del cursor

#import all fields
#create insert cursor variable
with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor:
    for rows in ids:
        insertCursor.insertRow(rows)

print ('Done')

[AlfredBaldenweck]

What happens if you remove GlobalID from the list of fields? Necessarily, it won't be the same between each copy.

[CCWeedcontrol]

I wasn't sure if you meant, if in the script or the layers, I removed the GlobalID from both feature classes and I am still getting doubles. Would you like me to share some of the data?

[AlfredBaldenweck]

I meant in the script, placing it in out_fields

Sure, if you can share a few records (in a gdb of some sort), I'd be happy to take a look at it.

[CCWeedcontrol]

I have these in a file geodatabase, and I removed all the none important fields.

Thanks I really appreciate your help.

[AlfredBaldenweck]

So, I took a look at your data, and I think I've found the problem?

1) Placing "GlobalID" into the out_fields immediately helped and cut back the number of records added. 

2) You have a few issues in your data that were causing the rows to be separate:

• Not all of the records in your target feature class have geometry (89 records)
  This causes row[0] to be None, rather than a polygon object, making the two tables mismatch.
• You have some mismatched acreages between the two tables (There are 35 of these where their shapes are the same.)
• Not sure if there's a difference in Parcel#, but there could be

Code below to tell you which ones have problems.

Spoiler

aprx = arcpy.mp.ArcGISProject("CURRENT")
mp = aprx.activeMap
fc1 = mp.listLayers("sourceFC")[0] # Copy in the Edit GDB, contains 25096 records
targetFC = mp.listLayers("targetFC")[0] #Copy in the Published GDB, contains 11396 records

dsc = arcpy.Describe(targetFC)
fields = dsc.fields

out_fields = ["FID", "Shape_Leng", "Shape_Area", "Duplicate", "GlobalID"]
fieldnames = [field.name if field.name != 'Shape'  else 'SHAPE@' for field in fields if field.name not in out_fields]

print(out_fields)
print(fieldnames)

keepList = []
# Switched from fc1 to targetFC
# You want to check the final product first
with arcpy.da.SearchCursor(targetFC, fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor:
    for row in cursor:
        if row[0] is None:
            print(row)
            keepList.append(row) #Appended row, not row[0]. 
                             #Otherwise they might not check for the same thing
        #print(row)
del cursor
#print(keepList)

print(len(keepList))
print("Break \n\n")

#list of values from fc1 to inject into targetFC
ids = []
#Check for missing values comparing by row in total.
with arcpy.da.SearchCursor(fc1, fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor: 
    for row in cursor:
        for k in keepList:
            if row[0] == k[0]:
                if row[3] != k[3]:
                    print(row, k)
        #if row[1] not in keepList:
                    ids.append(row)
            #print(row)
del cursor

print(len(ids))

[CCWeedcontrol]

Oh wow. I looked back at my original data and there no empty geometry. I think I might have shared one (targetFC),that must have corrupted after testing, my apologies. 

I have attached a copy of the original data(targetFC), with the unnecessary fields removed. I verified there were no empty geometry, again my apologies.

After verifying that both dataset had no empty geometry and verified with the code you attached.

0
Break 
0

I ran the code below, but it created empty geometry on the target feature class. It appears to that the empty geometry ones are the missing ones that are being inserted into the targetFC.

dsc = arcpy.Describe(targetFC)
fields = dsc.fields
out_fields = ['OID@', 'Shape_Leng', 'Shape_Area', 'Duplicate', 'GlobalID']
fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields]
#print(out_fields)
#print(fieldnames)

keepList = []
# Switched from fc1 to targetFC
# You want to check the final product first
with arcpy.da.SearchCursor(targetFC, fieldnames) as cursor:
    for row in cursor:
        keepList.append(row) #Appended row, not row[0]. 
                             #Otherwise they might not check for the same thing
print("Keeplist length: ", len(keepList))
del cursor

#list of values from fc1 to inject into targetFC
ids = []
#Check for missing values comparing by row in total.
with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: 
    for row in cursor:
        if row not in keepList:
            ids.append(row)
print("ids: ", ids, len(ids))
del cursor

#import all fields
#create insert cursor variable
with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor:
    for rows in ids:
        insCursor.insertRow(rows)



print ('Done', arcpy.management.GetCount(targetFC))

[CCWeedcontrol]

I was able to solve the empty geometry, by adding ['Shape@'] in the field names. I also included the sql_clause you had.

Based on I am seeing and the print out, the script is just adding all the features from sourcFC and not acutally comparing and inserting the missing ones.

Would it be best to compare two or three fields, if so how?

dsc = arcpy.Describe(targetFC)
fields = dsc.fields
out_fields = ['OID@',  'Duplicate', 'GlobalID']
fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields]
#print(out_fields)
#print(fieldnames)

keepList = []
# Switched from fc1 to targetFC
# You want to check the final product first
with arcpy.da.SearchCursor(targetFC, ['SHAPE@'] + fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor:
    for row in cursor:
        keepList.append(row) #Appended row, not row[0]. 
                             #Otherwise they might not check for the same thing
print("Keeplist length: ", len(keepList))
del cursor

#list of values from fc1 to inject into targetFC
ids = []
#Check for missing values comparing by row in total.
with arcpy.da.SearchCursor(fc1, ['SHAPE@'] + fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor: 
    for row in cursor:
        if row not in keepList:
            ids.append(row)
print("ids: ", ids, len(ids))
del cursor

#import all fields
#create insert cursor variable
with arcpy.da.InsertCursor(targetFC, ['SHAPE@'] + fieldnames) as insCursor:
    for rows in ids:
        insCursor.insertRow(rows)

[AlfredBaldenweck]

Sorry for the delay in getting back to you. 

Don't bother to add SHAPE@ to the fields list like that; it's already the first entry in the fieldslist.

Unfortunately, I don't really have a good answer for you.

If it works for you, your original approach of just comparing shape will work fine. You can do this either with a cursor, or select by location where they're identical and invert the spatial relationship. (There is a difference of one record between comparing via cursor and select by location. I'm not sure which one it is, and I'm not sure why).

I don't know your data as well as you do, so it isn't clear to me what differences are acceptable. It might be most time-efficient to add the missing records based on shape and then manually reviewing (FindIndentical is a lifesaver)

[CCWeedcontrol]

No worries, I really appreciate you trying to help.