topic Re: Iterate selection and calculate field not working in Python Questions

Iterate selection and calculate field not working

ThomasColson — Sat, 11 Dec 2021 22:41:37 GMT

With the following, I am trying to select each polygon by its name, select points from another feature class that are within the selected polygon, then calculate a field using the same name that was used to select the polygon (by attributes). A spatial intersect, or any other method that results in creating a new FC as output and having to do joins won't work here, as this is (hopefully) to be an unattended nightly update script. It iterates through the polygons fine, but the result is that ALL points get the name of the last polygon.

import os
import errno
import arcpy
import sys  
import traceback  
from arcpy import env
update_feature_class = r'C:\Users\tcolson\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\GRSM_RESEARCH_LOCATIONS'
county_feature_class = r'C:\Users\tcolson\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\CountyBoundaries'

with arcpy.da.SearchCursor(county_feature_class,['SHAPE@','NAME']) as cursor:
    for row in cursor:
        print('Selecting '+row[1])
        expression = "NAME = '"+row[1]+"'"
        arcpy.SelectLayerByAttribute_management (county_feature_class, "NEW_SELECTION", expression)
        arcpy.SelectLayerByLocation_management(update_feature_class, "INTERSECT", county_feature_class)
        arcpy.CalculateField_management(update_feature_class, "COUNTY","'"+row[1]+"'", "PYTHON3")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Sat, 22 Sep 2018 19:28:00 GMT

Does it work? You indentation is off, or that is a copy-paste-format problem.

or are you looking for alternatives

If you don't want to use the existing arctoolbox tools, you could do it with the polygon geometry class's

'contains' method

Polygon—ArcPy classes | ArcGIS Desktop

it returns a boolean, so if the feature is within the bounds, you can assign the ID to the feature, then move on to the next polygon

Indicates if the base geometry contains the comparison geometry. contains is the opposite of within.

Re: Iterate selection and calculate field not working

ThomasColson — Sat, 11 Dec 2021 22:41:38 GMT

No, it doesn't work, it calculates all points, regardless of selection, to the the last "Name" in the loop. How would

contains (second_geometry, {relation}) work any different? You still have to iterate through a select-by-attribute loop?

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Sat, 22 Sep 2018 22:32:27 GMT

It is not a select by attribute loop. You have your points and your polygons. You loop through the points and when it is found in a polygon you 'pop' it out, grab the polygon id and move on to the next polygon

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Sat, 11 Dec 2021 22:41:41 GMT

import numpy as np
import arcpy

def id_geom(in_fc):
    """The main code segment which gets the id and shape information and
    explodes the geometry to individual points
    """
    with arcpy.da.SearchCursor(in_fc, ['OBJECTID', 'SHAPE@']) as cursor:
        a = [[row[0], row[1]] for row in cursor]
    return a


def pip_arc(pnt_fc, poly_fc):
    """Naive point-in-polygon using arcpy
    """
    pnts = id_geom(pnt_fc)
    polys = id_geom(poly_fc)
    out = []
    for pnt in pnts:
        pid, p = pnt
        for poly in polys:
            plid, pol = poly
            if p.within(pol):
                out.append([pid, plid])  #[pid, p, plid, pol])
                break
        continue
    return out

# ----
pnt_fc = r"path_to\your.gdb\pnts_2000"      # points featureclass
poly_fc = "path_to\your.gdb\SamplingGrids"  # polygon featureclass

out = pip_arc(pnt_fc, poly_fc)  # ---- run pip_arc and convert to array
out = np.array(out)
dt = [('PntID', '<i4'), ('PolyID', '<i4')]  # ---- make an output array
z = np.ndarray((len(out), ), dtype=dt)      # to be used with ExtendTable
z['PntID'] = out[:, 0] 
z['PolyID'] = out[:, 1]

arcpy.da.ExtendTable(pnt_fc, "OBJECTID", z, "PntID")  # --- make sure Pro is closed‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Iterate selection and calculate field not working

ThomasColson — Sun, 23 Sep 2018 13:54:41 GMT

Traceback (most recent call last):
File "C:\Users\bigfoot\Documents\ArcGIS\Projects\MyProject\PIP_TEST.py", line 33, in <module>
out = pip_arc(pnt_fc, poly_fc) # ---- run pip_arc and convert to array
File "C:\Users\bigfoot\Documents\ArcGIS\Projects\MyProject\PIP_TEST.py", line 23, in pip_arc
if p.within(pol):
AttributeError: 'NoneType' object has no attribute 'within'

Both FC"s are GCS NAD 83

Re: Iterate selection and calculate field not working

JoshuaBixby — Sun, 23 Sep 2018 15:15:00 GMT

A spatial intersect, or any other method that results in creating a new FC as output and having to do joins won't work here, as this is (hopefully) to be an unattended nightly update script.

As much as I like ArcPy, there are times using geoprocessing tools are both the fastest and best way to get a task done. What it is about this being an unattended nightly update script that you think disallows the use of geoprocessing tools that results in new feature classes?

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Sun, 23 Sep 2018 17:37:33 GMT

null point? or not a point featureclass? it is a point in polygon demo, assume your are working local, adjust if otherwise

Re: Iterate selection and calculate field not working

ThomasColson — Sun, 23 Sep 2018 17:53:38 GMT

yeah, checked that. 700+ points, each one of them is constrained by a polyogn (7 of them). Data is local FGDB created by Pro

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Sat, 11 Dec 2021 22:41:43 GMT

no clue, check again, 2000 pnts in 200 polygons... prior to arcpy.da.ExtendTable

out2
array([[   1,   30],
       [   2,   34],
       [   3,   30],
       [   4,   84],
       [   5,   78],
       [   6,   90],
       [   7,   18],
       [   8,   84],
       [   9,   16],
       [  10,   67],
       ...,
       [1991,   24],
       [1992,   96],
       [1993,   70],
       [1994,   41],
       [1995,   81],
       [1996,    8],
       [1997,   61],
       [1998,   29],
       [1999,   26],
       [2000,   67]])‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Iterate selection and calculate field not working

ThomasColson — Sun, 23 Sep 2018 20:31:40 GMT

Ok I got it now, sort of. Turns out there WAS bad geometry, the OGC check in Pro found it. Now I got past that barrier, how does one use arcpy.da.ExtendTable to append the polyId to an existing field in the point table, as opposed to adding a new field?

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Sun, 23 Sep 2018 20:46:06 GMT

Simple.... delete the old field first (using code of course) and 'extend' the table with the new data.

It is a limitation of esri's implementation that they don't allow direct replacement of the old field...

But it doesn't matter anyway.

because... I never trust 'updating' a column in any event, just in case things go wrong... I would rather add a new field, compare to the old field, then delete the old field.

BUT … on the upside, imagine having an incremental record of your updates! Poly_id01, Poly_id02, …. Poly_id07... and at the end of the week you can review, summarize and consolidate

This was just a demo though, so I am sure that a temporary field could be added using extend table, then a CalculateField to update and existing field, then a Delete of the extended field could be quite readily added and be transparent to the user since there is no 'join'

Re: Iterate selection and calculate field not working

MichaelVolz — Mon, 24 Sep 2018 15:22:13 GMT

Thomas:

I am curious about the statement "Ok I got it now, sort of. Turns out there WAS bad geometry, the OGC check in Pro found it."

I have some SDE polygon data that I initially thought only had broken links when added to Pro 2.2.x as it works fine in ArcMap 10.5.1 (Just like you the OGC check in Pro detected the issue but not the ESRI check). I have started to investigate upgrading to ArcMap 10.6.1 as Pro does not support several important workflows at this time. The same data also does not load in ArcMap 10.6.1. So would you know if your data that needed to be fixed could be used in 10.6.x before it was fixed (Although I'm not sure if your org is even using 10.6.1 at this time)?

It seems that Pro 2.2.x as well as ArcMap 10.6.1 has higher standards for data that worked in earlier versions of ArcMap.

Re: Iterate selection and calculate field not working

RichardFairhurst — Sat, 11 Dec 2021 22:41:46 GMT

You original code fails because you have not created layers for the two feature classes and are using the feature classes directly. I am surprised the select actions did not cause an error, because they only work with layers, not feature classes. The Calculate Field will always calculate every record if the input is a feature class and not a layer.

Your code would work if you create layers from the feature classes and use the layers:

import arcpy
update_feature_class = r'C:\Users\tcolson\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\GRSM_RESEARCH_LOCATIONS'
county_feature_class = r'C:\Users\tcolson\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\CountyBoundaries'

arcpy.MakeFeatureLayer_management(update_feature_class,"update_lyr")
arcpy.MakeFeatureLayer_management(county_feature_class,"county_lyr")
with arcpy.da.SearchCursor(county_feature_class,['NAME']) as cursor:
    for row in cursor:
        county_name = row[0]
        print('Selecting '+county_name)
        expression = "NAME = '"+county_name+"'"
        arcpy.SelectLayerByAttribute_management ("county_lyr", "NEW_SELECTION", expression)
        arcpy.SelectLayerByLocation_management("update_lyr", "INTERSECT", "county_lyr")
        arcpy.CalculateField_management("update_lyr", "COUNTY","'"+county_name+"'", "PYTHON3")‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Tue, 25 Sep 2018 20:05:59 GMT

Thomas... still ruled out ExtendTable with a Delete field, if needed, as an option? It certainly gets around that cursor stuff and could be added to to check for bad geometries if needed

Re: Iterate selection and calculate field not working

ThomasColson — Wed, 26 Sep 2018 11:46:17 GMT

Dan, it's been a few days and things are going sideways at work but: I got your code to run, but the problem(s) I'm running into are twofold: I need a string (county name) to be added to an existing string field in the points layer. The numpy solution seems to be limited to only adding an integer to a new field. I was reading the numpy documentation and fiddling around with string arrays, and getting nothing done other than throw errors. Another limitation here is temporarily adding fields, doing joins, etc...as these feature classes are pushing feature services, which put a schema lock on the feature classes, so adding fields is not possible.

Re: Iterate selection and calculate field not working

DanPatterson_Retired — Wed, 26 Sep 2018 13:29:31 GMT

I can fix that, extendtable is like a join, but permanent, you have a common id field, then you can join anything, string, integer float, a single field or multiple fields

I use it all the time, with or without field deletion.

But it sounds like you aren't doing this stuff when the data can be worked with or it wouldn't have locks

I will post a blog, scripts and a toolbox so you can read it at your leisure

Re: Iterate selection and calculate field not working

ThomasColson — Wed, 26 Sep 2018 16:08:52 GMT

I think an example like that would have pretty broad application beyond my use. Lots of folks needing to make spatial type updates to feature services that are driving feature services, e.g a field survey of stormwater drains using collector and the python updates what zoning district or whatever, without requiring alterations to the schema or service disruption.

Re: Iterate selection and calculate field not working

RichardFairhurst — Sat, 11 Dec 2021 22:41:49 GMT

Try the code below. It does not alter the schema at all and updates values in an existing field. It is also more efficient than your original approach and will minimize the number of polygon to point comparisons required to match all of the county names to the points.

import arcpy

update_feature_class = r'C:\Users\tcolson\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\GRSM_RESEARCH_LOCATIONS'
county_feature_class = r'C:\Users\tcolson\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\CountyBoundaries'

county_count = 0
relateDict = {}
# Set up search cursor for county fc
with arcpy.da.SearchCursor(county_feature_class,['NAME','SHAPE@']) as search_cursor:

    # Iterate through counties
    for polygon in search_cursor:
        if not polygon[0] in relateDict:
            relateDict[polygon[0]] = polygon[1]
        else:
            relateDict[polygon[0]] = relateDict[polygon[0]].union(polygon[1])

for name in relateDict:
    county_count += 1

point_count = 0
comparison_count = 0
# Set up update cursor for point fc
with arcpy.da.UpdateCursor(update_feature_class, ['SHAPE@', 'COUNTY']) as update_cursor:

    # Iterate through points
    for point in update_cursor:
        point_count += 1
        for name in relateDict:
            comparison_count += 1
            county = relateDict[name]
            # Check if the current point is within the current polygon
            if point[0].within(county):  # Returns True or False
                # Set the value of the field as the county name
                point[1] = name
                # Update the cursor with the new value
                update_cursor.updateRow(point)
                break
print("Processed {0} County/Point comparisons for {1} Counties and {2} Points".format(comparison_count, county_count, point_count))
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Iterate selection and calculate field not working

MichaelVolz — Wed, 26 Sep 2018 16:25:41 GMT

If you use intersect to capture points on the border, wouldn't you then have to deal with 2 names from the 2 polygons that intersect the point that is on the border of each polygon?