topic Re: Insert missing features from one layer to another in Python Questions

Insert missing features from one layer to another

CCWeedcontrol — Wed, 29 Mar 2023 21:00:10 GMT

I need to insert missing features from one layer to another. I don't want to use the append tool because I don't want a new dataset, I just want to insert the missing features. Both are polygons and have pretty much the same fields except for 3 fields ('Field1','Field2', 'Field3'). I have the following but I get no error when I run it.

import arcpy fc1 = "lyTB" targetFC = "lyA_2" dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = [dsc.OIDFieldName, dsc.lengthFieldName, dsc.areaFieldName, 'Field1','Field2', 'Field3'] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] keepList = [] with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: keepList.append(row[0]) #print(keepList) del cursor #list of values from fc1 to inject into targetFC ids = [] with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: #print (row) if row[0] not in keepList: ids.append(row) del cursor #import all fields #create insert cursor variable with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor: for rows in ids: insertCursor.insertRow(rows) print ('Done')

Re: Insert missing features from one layer to another

DanPatterson — Wed, 29 Mar 2023 21:06:12 GMT

Append doesn't create a new output featureclass... merge does

Append (Data Management)—ArcGIS Pro | Documentation

Re: Insert missing features from one layer to another

CCWeedcontrol — Wed, 29 Mar 2023 21:11:20 GMT

I guess I mixed up append and merge. Append will append all feature in one layer to the other making duplicates, which is what I am trying to avoid.

Re: Insert missing features from one layer to another

AlfredBaldenweck — Wed, 29 Mar 2023 21:12:45 GMT

Not if you have the records you want selected, and drag the layer into the Append dialog

Re: Insert missing features from one layer to another

AlfredBaldenweck — Wed, 29 Mar 2023 21:15:28 GMT

It looks like you're checking fc1 twice with the search cursor.

The first round adds all of the objectIDs to the list, then the second round checks the same table to see if it's somehow missing the objectIDs you just added to it.

I'd check using something other than object ID if I were you, probably by checking the contents of the row, reason being that if one of the missing records is in the middle, objectIDs will not match up.

Re: Insert missing features from one layer to another

CCWeedcontrol — Wed, 29 Mar 2023 21:36:08 GMT

I made the following changes.

line 24 was

if row[0] not in keepList:

should be;

if row not in keepList:

Line 32 was

InsertCursor .insertRow(rows)

Should have been

insCursor.insertRow(rows)

fc1 = "lyTB" targetFC = "lyA_2" dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = [dsc.OIDFieldName, dsc.lengthFieldName, dsc.areaFieldName, 'Field1','Field2', 'Field3'] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] keepList = [] with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: keepList.append(row[0]) #print(keepList) del cursor #list of values from fc1 to inject into targetFC ids = [] with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: #print (row) if row not in keepList: ids.append(row) del cursor #import all fields #create insert cursor variable with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor: for rows in ids: insCursor.insertRow(rows) print ('Done')

How can I check using two other fields other than just the OID?

Re: Insert missing features from one layer to another

AlfredBaldenweck — Thu, 30 Mar 2023 13:58:14 GMT

So, first off, I owe you an apology. I made some assumptions about your code without fully understanding your fields variables or testing it myself. As it turns out, you weren't checking ObjectID at all.

In the code below, I've made keepList from the targetFC, and ids from the difference between it and fc1, checking by the entire row. Since those rows don't actually include ObjectID, if row[0] is, in fact, a unique field like a site number or something (and you know it's unique), I think that'd be fine?

I only was able to test through the second search cursor, but the sum of keepList and ids came to the total of records in fc1, so I think this is at least a good start.

aprx = arcpy.mp.ArcGISProject("CURRENT") mp = aprx.activeMap fc1 = mp.listLayers()[0] # Copy in the Edit GDB, contains 25096 records targetFC = mp.listLayers()[1] #Copy in the Published GDB, contains 11396 records dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = [dsc.OIDFieldName, dsc.lengthFieldName, dsc.areaFieldName, 'Field1','Field2', 'Field3'] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] #print(out_fields) #print(fieldnames) keepList = [] # Switched from fc1 to targetFC # You want to check the final product first with arcpy.da.SearchCursor(targetFC, fieldnames) as cursor: for row in cursor: keepList.append(row) #Appended row, not row[0]. #Otherwise they might not check for the same thing del cursor #list of values from fc1 to inject into targetFC ids = [] #Check for missing values comparing by row in total. with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: if row not in keepList: ids.append(row) del cursor #import all fields #create insert cursor variable with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor: for rows in ids: insCursor.insertRow(rows) print ('Done')

Also maybe check out Feature Compare (Data Management)—ArcGIS Pro | Documentation and see if that's helpful?

Re: Insert missing features from one layer to another

CCWeedcontrol — Thu, 30 Mar 2023 16:10:56 GMT

The current codes insert all the features in fc1 into the targetFc, at least for me. It doesn't appear to only insert the missing ones. If I run the code again, it just adds the total number of features in fc1again. I appreciate your input.

Re: Insert missing features from one layer to another

AlfredBaldenweck — Thu, 30 Mar 2023 17:14:01 GMT

I'm confused; I just tested it with no problems?

I have a my raw (target) table, which is purple and has 5196 records.

My edited table (fc1) is brown and has 5198 records. (I don't have these two records selected when I run the code)

Running the code:

aprx = arcpy.mp.ArcGISProject("CURRENT") mp = aprx.activeMap fc1 = mp.listLayers("Edited")[0] # Copy in the Edit GDB, contains 25096 records targetFC = mp.listLayers("Raw")[0] #Copy in the Published GDB, contains 11396 records dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = [dsc.OIDFieldName, dsc.lengthFieldName, dsc.areaFieldName, 'Field1','Field2', 'Field3'] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] #print(out_fields) #print(fieldnames) keepList = [] # Switched from fc1 to targetFC # You want to check the final product first with arcpy.da.SearchCursor(targetFC, fieldnames) as cursor: for row in cursor: keepList.append(row) #Appended row, not row[0]. #Otherwise they might not check for the same thing print("Keeplist length: ", len(keepList)) del cursor #list of values from fc1 to inject into targetFC ids = [] #Check for missing values comparing by row in total. with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: if row not in keepList: ids.append(row) print("ids: ", ids, len(ids)) del cursor #import all fields #create insert cursor variable with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor: for rows in ids: insCursor.insertRow(rows) print ('Done', arcpy.management.GetCount(targetFC))

My two new records are added to the target (ignore the OBJECTIDs; I've run this a few times already).

Re: Insert missing features from one layer to another

CCWeedcontrol — Thu, 30 Mar 2023 20:28:34 GMT

I was running this outside Pro and thought maybe that was why I was getting different results but no. I put the info in Pro and ran the code.

Here are my results

I thought maybe I had something wrong in the code, so I copied the code you posted and ran it.

aprx = arcpy.mp.ArcGISProject("CURRENT") mp = aprx.activeMap fc1 = mp.listLayers("Source")[0] # Copy in the Edit GDB, contains 25096 records targetFC = mp.listLayers("target")[0] #Copy in the Published GDB, contains 11396 records dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = [dsc.OIDFieldName, dsc.lengthFieldName, dsc.areaFieldName, 'Field1','Field2', 'Field3'] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] #print(out_fields) #print(fieldnames) keepList = [] # Switched from fc1 to targetFC # You want to check the final product first with arcpy.da.SearchCursor(targetFC, fieldnames) as cursor: for row in cursor: keepList.append(row) #Appended row, not row[0]. #Otherwise they might not check for the same thing print("Keeplist length: ", len(keepList)) del cursor #list of values from fc1 to inject into targetFC ids = [] #Check for missing values comparing by row in total. with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: if row not in keepList: ids.append(row) print("ids: ", ids, len(ids)) del cursor #import all fields #create insert cursor variable with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor: for rows in ids: insCursor.insertRow(rows) print ('Done', arcpy.management.GetCount(targetFC))

Re: Insert missing features from one layer to another

AlfredBaldenweck — Thu, 30 Mar 2023 20:33:31 GMT

What happens if you remove GlobalID from the list of fields? Necessarily, it won't be the same between each copy.

Re: Insert missing features from one layer to another

CCWeedcontrol — Fri, 31 Mar 2023 15:23:41 GMT

I wasn't sure if you meant, if in the script or the layers, I removed the GlobalID from both feature classes and I am still getting doubles. Would you like me to share some of the data?

Re: Insert missing features from one layer to another

AlfredBaldenweck — Fri, 31 Mar 2023 23:21:44 GMT

I meant in the script, placing it in out_fields

Sure, if you can share a few records (in a gdb of some sort), I'd be happy to take a look at it.

Re: Insert missing features from one layer to another

CCWeedcontrol — Mon, 03 Apr 2023 17:08:31 GMT

I have these in a file geodatabase, and I removed all the none important fields.

Thanks I really appreciate your help.

Re: Insert missing features from one layer to another

AlfredBaldenweck — Mon, 03 Apr 2023 18:37:25 GMT

So, I took a look at your data, and I think I've found the problem?

1) Placing "GlobalID" into the out_fields immediately helped and cut back the number of records added.

2) You have a few issues in your data that were causing the rows to be separate:

Not all of the records in your target feature class have geometry (89 records)
This causes row[0] to be None, rather than a polygon object, making the two tables mismatch.
You have some mismatched acreages between the two tables (There are 35 of these where their shapes are the same.)
Not sure if there's a difference in Parcel#, but there could be

Code below to tell you which ones have problems.

aprx = arcpy.mp.ArcGISProject("CURRENT") mp = aprx.activeMap fc1 = mp.listLayers("sourceFC")[0] # Copy in the Edit GDB, contains 25096 records targetFC = mp.listLayers("targetFC")[0] #Copy in the Published GDB, contains 11396 records dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = ["FID", "Shape_Leng", "Shape_Area", "Duplicate", "GlobalID"] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] print(out_fields) print(fieldnames) keepList = [] # Switched from fc1 to targetFC # You want to check the final product first with arcpy.da.SearchCursor(targetFC, fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor: for row in cursor: if row[0] is None: print(row) keepList.append(row) #Appended row, not row[0]. #Otherwise they might not check for the same thing #print(row) del cursor #print(keepList) print(len(keepList)) print("Break \n\n") #list of values from fc1 to inject into targetFC ids = [] #Check for missing values comparing by row in total. with arcpy.da.SearchCursor(fc1, fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor: for row in cursor: for k in keepList: if row[0] == k[0]: if row[3] != k[3]: print(row, k) #if row[1] not in keepList: ids.append(row) #print(row) del cursor print(len(ids))

Re: Insert missing features from one layer to another

CCWeedcontrol — Mon, 03 Apr 2023 21:49:27 GMT

Oh wow. I looked back at my original data and there no empty geometry. I think I might have shared one (targetFC),that must have corrupted after testing, my apologies.

I have attached a copy of the original data(targetFC), with the unnecessary fields removed. I verified there were no empty geometry, again my apologies.

After verifying that both dataset had no empty geometry and verified with the code you attached.

0 Break 0

I ran the code below, but it created empty geometry on the target feature class. It appears to that the empty geometry ones are the missing ones that are being inserted into the targetFC.

dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = ['OID@', 'Shape_Leng', 'Shape_Area', 'Duplicate', 'GlobalID'] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] #print(out_fields) #print(fieldnames) keepList = [] # Switched from fc1 to targetFC # You want to check the final product first with arcpy.da.SearchCursor(targetFC, fieldnames) as cursor: for row in cursor: keepList.append(row) #Appended row, not row[0]. #Otherwise they might not check for the same thing print("Keeplist length: ", len(keepList)) del cursor #list of values from fc1 to inject into targetFC ids = [] #Check for missing values comparing by row in total. with arcpy.da.SearchCursor(fc1, fieldnames) as cursor: for row in cursor: if row not in keepList: ids.append(row) print("ids: ", ids, len(ids)) del cursor #import all fields #create insert cursor variable with arcpy.da.InsertCursor(targetFC,fieldnames) as insCursor: for rows in ids: insCursor.insertRow(rows) print ('Done', arcpy.management.GetCount(targetFC))

Re: Insert missing features from one layer to another

CCWeedcontrol — Mon, 03 Apr 2023 22:24:33 GMT

I was able to solve the empty geometry, by adding ['Shape@'] in the field names. I also included the sql_clause you had.

Based on I am seeing and the print out, the script is just adding all the features from sourcFC and not acutally comparing and inserting the missing ones.

Would it be best to compare two or three fields, if so how?

dsc = arcpy.Describe(targetFC) fields = dsc.fields out_fields = ['OID@', 'Duplicate', 'GlobalID'] fieldnames = [field.name if field.name != 'Shape' else 'SHAPE@' for field in fields if field.name not in out_fields] #print(out_fields) #print(fieldnames) keepList = [] # Switched from fc1 to targetFC # You want to check the final product first with arcpy.da.SearchCursor(targetFC, ['SHAPE@'] + fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor: for row in cursor: keepList.append(row) #Appended row, not row[0]. #Otherwise they might not check for the same thing print("Keeplist length: ", len(keepList)) del cursor #list of values from fc1 to inject into targetFC ids = [] #Check for missing values comparing by row in total. with arcpy.da.SearchCursor(fc1, ['SHAPE@'] + fieldnames, sql_clause = (None, "ORDER BY PermitNum")) as cursor: for row in cursor: if row not in keepList: ids.append(row) print("ids: ", ids, len(ids)) del cursor #import all fields #create insert cursor variable with arcpy.da.InsertCursor(targetFC, ['SHAPE@'] + fieldnames) as insCursor: for rows in ids: insCursor.insertRow(rows)

Re: Insert missing features from one layer to another

AlfredBaldenweck — Mon, 10 Apr 2023 18:12:08 GMT

Sorry for the delay in getting back to you.

Don't bother to add SHAPE@ to the fields list like that; it's already the first entry in the fieldslist.

Unfortunately, I don't really have a good answer for you.

If it works for you, your original approach of just comparing shape will work fine. You can do this either with a cursor, or select by location where they're identical and invert the spatial relationship. (There is a difference of one record between comparing via cursor and select by location. I'm not sure which one it is, and I'm not sure why).

I don't know your data as well as you do, so it isn't clear to me what differences are acceptable. It might be most time-efficient to add the missing records based on shape and then manually reviewing (FindIndentical is a lifesaver)

Re: Insert missing features from one layer to another

CCWeedcontrol — Tue, 11 Apr 2023 19:07:27 GMT

No worries, I really appreciate you trying to help.