topic Re: Issue with geometry not updating properly within multiple cursors in Python Questions

Issue with geometry not updating properly within multiple cursors

RebeccaStrauch__GISP — Sat, 11 Dec 2021 16:10:53 GMT

I'm pretty sure I just have a syntax or indentation issue, but I've been trying to track this down a good chunk of the day, so time to "rubber duck". The full program is a bit more than anyone will be able to deal with, but if anyone really wants to look at the full code, I can upload it. Typing is out is sometime helpful for me too....so here's hoping.

Basic (and simplified) description: given a random point with a "transtype" Riparian or Contour,

if riparian, grabs river, creates route, snaps point to route.
- if river < desired length, grabs full river
- if river >= desired length, tries to create segment with length/2 in each direction
  - if first half < length/2, shifts both points, and vice versa (i.e., if snaped point is too close to end)
- write to array...with a bunch of other stuff
if contour, creates contour as route, snaps point to route (only grabs segments > 5000m)
- if contour < desire length, grabs full contour
- if contour >= desired length, tries to create segment
- with length/2 in each direction
  - if first half < length/2, shifts both points, and vice versa (i.e., if snaped point is too close to end)
- write to array...with a bunch of other stuff
take the array and populates a FC with all the info.

For 2000 random points, it finished with no error, but for some reason about 380 points (or 279 of those 380 points) are duplicating the transect above it. I do have multiple cursors, which for the most part seem to be fine, but one of the cursors has the "for aRoute2 in route2:" being bypassed. I believe this is the issue.

So, with that all said, here is the part that I think is the issue. Not sure that this will be of any used, but in case someone sees something obvious... I'm on a very short deadline....i.e., here tonight until I get this fixed, most likely. So any help is appreciated, but may not respond right away.

Dan Patterson Curtis Price Darren Wiens Jake Skinner Joshua Bixby any of you seeing anything obvious? I know cursors within cursors is n necessarily a good idea, and have no time to change it in a major way (another time), I'm in triage mode right now. Thanks for any help you might offer. I'll by you a free drink if I ever see you at the dev summit or UC.

NOTE: trying to fix the indentation in geonet is messing it up more....the first else: and everything else until the final else is indented....

# snippet...too much needed to send all and expect it to work
else:  # transtype == 'contour'
    # ##############
    #   Contours
    # ##############
    print(" -Create contour for elev {0} m".format(elevm))
    ContourList(inDEM, fullContour, elevm)
    #print(" -Clipping full contour by study area")
    arcpy.Clip_analysis(fullContour, inStudy, clipContour)
    #print("       making clipped contour FL")
    arcpy.MakeFeatureLayer_management(clipContour, "tmpClipContour")
    print(" -Selecting contour segs > 5000 m")
    #arcpy.SelectLayerByAttribute_management("tmpClipContour", "NEW_SELECTION", where_clause="Shape_Length >= 5000")  
    arcpy.SelectLayerByAttribute_management("tmpClipContour", "NEW_SELECTION", "Shape_Length >= 5000")

    # copies selection to new FC
    #print(" - Copying selection to new FC")  # here 4/4/2016
    arcpy.CopyFeatures_management("tmpClipContour", tmpContourFC)
    arcpy.MakeFeatureLayer_management(tmpContourFC, "tmpContourFL") 
    # # wrong? tmpContourFL = arcpy.mapping.Layer("tmpClipContour") 
    #print("\n**New selection of nearest contour segment to pt")
    #print(" - Creating new feature layer from origPtFL")
    arcpy.SelectLayerByLocation_management("tmpContourFL", nearType, "origPtFL", nearDist)    
    #print(" - Copy selection to new tmp FC contourSel")
    arcpy.CopyFeatures_management("tmpContourFL", contourSel)

    # Creates route of the selected contour segment
    print(" -Create temp route; create and process cursor")
    arcpy.CreateRoutes_lr(contourSel, contourField, tmpRoute)
    #print(" - Create SearchCursor from route")

    routeSCur2 = arcpy.da.SearchCursor(tmpRoute, ["SHAPE@", "Contour"])            
    print("getting here")

    with routeSCur2 as route2:
        print("getting inside cursor")   # will get to here
        for aRoute2 in route2:
            print("getting here2")     # #######   will not always get to here   #######
            geo = aRoute2[0]          # get all contour elevation for the route
            theElev = aRoute2[1]      # get elevation, actual contour as a route
            #snapPt = (geo.snapToLine(ptGeom)).centroid # shp_fld
            arcpy.LocateFeaturesAlongRoutes_lr(ptGeom, tmpRoute, contourField, nearDist, tmpRoutePtLoc, rid)
            lenTrans = geo.length  
            print("  **Contour elev {0}:  {1} m".format(theElev, RoundUp(lenTrans)))
            flyFullContour = lenTrans <= int(targetLength)
            if flyFullContour:
                print(" **Taking full contour: {0}m ".format(lenTrans))
                startx  = geo.firstPoint.X
                starty  = geo.firstPoint.Y
                endx = geo.lastPoint.X
                endy = geo.lastPoint.Y

                startPtdd = arcpy.PointGeometry(arcpy.Point(startx, starty) , initialSR).projectAs(geoSR)
                startxdd = startPtdd.centroid.X
                startydd = startPtdd.centroid.Y

                #print(" -Start point {0} {1}".format(startxdd, startydd))
                endPtdd = arcpy.PointGeometry(arcpy.Point(endx, endy) , initialSR).projectAs(geoSR)
                endxdd = endPtdd.centroid.X
                endydd = endPtdd.centroid.Y

                #print(" -End point {0} {1}".format(endxdd, endydd))
                tmpTrans = geo # taking full length.segmentAlongLine(startDist, endDist)
                #tlength = tmpTrans.length
                #ripNotes = "short"
                #print("{0} transect #  {1}".format(transtype, transID))
                print("{0} transect #  {1}  length: {2}  Notes: {3}".format(transtype, transID, tlength, ripNotes))
            elif lenTrans >= int(targetLength):    #else:

                ptSCue = arcpy.SearchCursor(tmpRoutePtLoc)
                field_name = 'MEAS'
                for routeMeas in ptSCue:
                    snapPtDist =  routeMeas.getValue(field_name)
                    print("  ->Measurement along route: {0}".format(snapPtDist))
                    startDist = snapPtDist - halfTrans
                    endDist = snapPtDist + halfTrans
                    print("\n -Initial startDist: {0}  endDist: {1}".format(startDist, endDist))
                    initialSegment = geo.segmentAlongLine(endDist, startDist)
                    initialSegmentLength = initialSegment.length
                    print(" -->Initial length: {0}".format(initialSegmentLength))
                    if RoundUp(initialSegmentLength) < int(targetLength):
                        # Check each side to make sure distance is = halfTrans
                        #  if not, shift as needed

                        # check length of the segment from the snapPtDist to startDist
                        firstSegment = geo.segmentAlongLine(snapPtDist, startDist)
                        firstSegmentLength = firstSegment.length
                        #print(" -First Length: {0}".format(firstSegmentLength))

                        #check first
                        if RoundUp(firstSegmentLength) < (halfTrans):
                            print(" -First half is short: {0}m".format(firstSegmentLength))
                            # calculate the distance it is short
                            shiftDist = halfTrans - firstSegmentLength
                            print("   ..shift distance for first segment: {0}m".format(shiftDist))
                            #startDist = startDist - shiftDist
                            # short from start to mid, so extend end distance
                            endDist = endDist + shiftDist
                            print("  ..new endDist: {0}m".format(endDist))
                        else:  # first segment ok, must be second part
                            endSegment = geo.segmentAlongLine(snapPtDist, endDist)
                            endSegmentLength = endSegment.length
                            print(" -Second half is short: {0}m".format(endSegmentLength))
                            # calculate the distance it is short
                            shiftDist = halfTrans - endSegmentLength
                            print("  ..shift distance for first segment: {0}m".format(shiftDist))
                            # short from end to mid, so start earlier on the line, it a minus measure
                            startDist = startDist - shiftDist
                            #endDist = endDist - shiftDist
                            print("  ..new startDist: {0}m".format(startDist))                  

                    # use the startDist and endDist to create the point geometries
                    #startPt = geo.positionAlongLine(startDist)
                    #endPt = geo.positionAlongLine(endDist)

                    startPt = geo.positionAlongLine(startDist)
                    startx = startPt.centroid.X    #need DD
                    starty = startPt.centroid.Y            
                    #print("Start Dist: {0}".format(startDist))
                    # midPt = geo.positionAlongLine(snapPtDist) #not using mid points
                    endPt = geo.positionAlongLine(endDist)
                    endx = endPt.centroid.X
                    endy = endPt.centroid.Y    
                    tmpTrans = geo.segmentAlongLine(startDist, endDist)
                    tlength = tmpTrans.length

            del ptSCue
            # Processing pts to geographic
            startPtdd = arcpy.PointGeometry(arcpy.Point(startx, starty) , initialSR).projectAs(geoSR)
            startxdd = startPtdd.centroid.X
            startydd = startPtdd.centroid.Y                 
            #print("start point {0} {1}".format(startxdd, startydd))
            #midPtdd = arcpy.PointGeometry(arcpy.Point(midPt.centroid.X, midPt.centroid.Y) , initialSR).projectAs(geoSR)   #not using mid points
            endPtdd = arcpy.PointGeometry(arcpy.Point(endx, endy) , initialSR).projectAs(geoSR)
            endxdd = endPtdd.centroid.X
            endydd = endPtdd.centroid.Y                            

            #tmpTrans = geo.segmentAlongLine(startDist, endDist)

            #tlength = tmpTrans.length
            shortLength = int(targetLength) - 500
            if tlength < shortLength:   # add A to ID for short trans...add in field.
                transID = "{0}A".format(transID)

            if tmpTrans.isMultipart:
                ripNotes = "contourParts"
                print("  ** Transect in multiple parts **")
            else:
                ripNotes = "contour"
            #print("{0} trans # {1}".format(transtype, transID))
            print("{0} transect #  {1}  length: {2}  Notes: {3}".format(transtype, transID, tlength, ripNotes))

else:
    break

Edit: added not above the python script..reformatting was not good.

Re: Issue with geometry not updating properly within multiple cursors

DanPatterson_Retired — Wed, 27 Apr 2016 00:53:53 GMT

can you confirm that the else statements on line 2 and 153 are supposed to be indented at the same level?

Re: Issue with geometry not updating properly within multiple cursors

RebeccaStrauch__GISP — Wed, 27 Apr 2016 01:05:44 GMT

Good catch on the indentation Dan...I'll fix it. Such is copying and pasting snippets. Actual code have the first else indented four levels.

I should also add that, running an individual point manually works and does create the correct transect, so I know it is not the input random points, DEM, or riparian layers that are an issue. I also should mention it does not always skip that missing that one cursor. There is some combination from the previous point that causes the issue, but looking at the data and output, their is no obvious pattern. I just isolated 3 of the "problem" points and ran them thru the script and they worked fine. That is why I think it might be an issue with the cursors not being lined up correctly, or the actual geometry field "tmpTrans" not being updated to over write the previous values....which then of course make all the coordinates, lengths, etc calcd from it off.

just fyi, thie is what I am appending

transList.append([tmpTrans, ptid, ptX, ptY, transtype, tlength, tlengthKm, transID, elevft, elevm, startxdd, startydd, endxdd, endydd, startxdms, startydms, endxdms, endydms, ripNotes, offset, displayID])

Re: Issue with geometry not updating properly within multiple cursors

DarrenWiens2 — Wed, 27 Apr 2016 01:06:57 GMT

ptScue is an old-style cursor, which behave unpredictably (or predictably bad) when nested. Does it help to switch it to a da cursor? In general, I also suggest using ' with ' syntax on all cursors to avoid them hanging in the void if your run ends in an error.

Re: Issue with geometry not updating properly within multiple cursors

RebeccaStrauch__GISP — Wed, 27 Apr 2016 01:12:03 GMT

Good catch Darren. I usually always use da cursors. Not sure what I did there. I had noticed I wasn't using the "with" for that one, and it didn't work last week when I was trying...I was "cursor" blind on that one and hadn't noticed. I'll give that a try and be back...

Re: Issue with geometry not updating properly within multiple cursors

DanPatterson_Retired — Wed, 27 Apr 2016 01:16:50 GMT

I am kindof with Darren on this I am finding it hard to figure out the cursor stuff in lines 32 to 38. But I am not a cursor person. Darren and Xander have posted some good examples but I don't know how much reshaping would be needed.

Is there anything you can 'def' to consolidate the code now? or is this for a non-deadline time?

Re: Issue with geometry not updating properly within multiple cursors

DanPatterson_Retired — Wed, 27 Apr 2016 01:34:09 GMT

line 70 and 170 ... is the deletion of the cursor lined up correctly? maybe I will wait until the while stuff is done

Re: Issue with geometry not updating properly within multiple cursors

RebeccaStrauch__GISP — Wed, 27 Apr 2016 01:44:51 GMT

Change those to da cursors helped me get "into" that cursor now, so that is good. I actually rolled back to a version before I made the numerous changes I did this afternoon, made those the changes to with .da, and renamed a few of my cursors, so I wasn't repeating names. Also got rid of my "del cursor" statements, which I had added when trouble shooting.

Anyway, still having issues, but may be able to track things for a bit again. I'll update as I have more.

Thanks again for your feedback Dan and Darren.

Re: Issue with geometry not updating properly within multiple cursors

RebeccaStrauch__GISP — Sat, 11 Dec 2021 16:10:56 GMT

Still having issues, but figured out partially what the issue is, but not why.

I did switch the one cursor to a .da version. It took me quite a while to figure out what I was trying to do. The table (not FC) actually only has one record (or is supposed to have 1), and I was just trying to pull a the value from the "MEAS" field. It is coming down to the input to my cursor occasionally had zero records. What is strange is, if I do it manually, I get my one record, even on the problem points, but my program is skipping it for some points for some reason.

The command I am running (sorry about all the variables)

arcpy.LocateFeaturesAlongRoutes_lr(ptGeom, tmpRoute, contourField, nearDist, tmpRoutePtLoc, rid)

Although manually (i.e. in the python window), I do get one record with my nearDist value, and in most cases it works in the script, for some records it doesn't return anything. So now I am testing for the number of records, and although I haven't fixed my duplication problem, I'm at least setting a flag in my notes field for my output line segments...that will help me pull those out for further testing and to fix them.

myCount = int(arcpy.GetCount_management(tmpRoutePtLoc).getOutput(0))
print("the count from Locate Along Route {0}".format(myCount))
if myCount == 0:
    ripNotes = "ERRORcon"

Giving up for the night, going to run it overnight, maybe with a new set of points, and deal with the problem points tomorrow...which is in about 40m. (sigh)

Re: Issue with geometry not updating properly within multiple cursors

RichardFairhurst — Wed, 27 Apr 2016 13:22:15 GMT

Rebecca:

It appears you have created code that uses embedded cursors. The code must be rewritten to eliminate that configuration if you want to make it process efficiently, since embedded cursors are the absolute worst way to approach any problem. Instead one of the cursors should load the data to a dictionary for any look up of data and only one update cursor should be processed at a time. The code revision can result in a 500 fold improvement in speed if you are processing 10,000 records against 10,000 records, since my approach would only have to process 20,000 record reads while an embedded cursor would process up to 10,000,000 record reads..

Re: Issue with geometry not updating properly within multiple cursors

RebeccaStrauch__GISP — Wed, 27 Apr 2016 16:23:22 GMT

Hi Richard,

Thanks for the comments, and I know there may be a more efficient ways to do this, but the embedded cursors in-and-of-themselves are not the issue, unless I have an indent off somewhere, i.e. the updating of my "tmpTrans" variable in the incorrect place. The issue seem to be that with some points, because I eliminate contour-parts that are < 5000m before making it into a route, it is no longer finding a "location along the route" within my search distance, which should be adequate. (BTW the contour is created on the fly from the point's elevm value),

Now that I fixed the two SearchCursor to the da.SearchCursor that I had missed, I was able to see that it was still crashing at the "for row in cursor:", but was making into the "with..." for the cursor. That lead me to check what was being created in the table feeding the cursor, and then setting a flag if the count=0. I need to back track further to see why it is not getting any records, since stepping thru it manually works, but I'm sure I am just missing something. At that point, I will need to set a "fix" by maybe dropping the elevm value a couple meters and trying again. Our DEM is not as accurate as lower-48, and these are traversed via plane (SuperCub) so a couple meters will not matter in the scheme of things. Also, the missed points typically are near the top of a hill/mountain.

BTW - we used to have many more of these issues in river valleys, so I added a "riparian" option that grabs the stream and gives and offset distance and direction from the stream...that is working. We labelled these as "skipped points" and dealt with them in the field.

After the crunch/field time is over, and for my own satisfaction, I will look into using dictionaries etc., and to resolve this issue fully. But since I was working on my logic from Avenue and AML programs, and the lead was changing/adding new requirements often, I did what I understood, and what I think is important, what the next person might be able to read and understand, if they are new to python. With that said, dictionaries and other solutions may be the way to go on a rewrite....but that will be all on my time....so a fun project.

Just an FYI - I ran this on 1250 points last night...74 came up with the flag. At this point, these would be similar to the older "skipped points" and I can put them and deal/test them directly.

I know this is more than anyone cares about fo this project, but it is good for me to think it out by typing.

Re: Issue with geometry not updating properly within multiple cursors

RebeccaStrauch__GISP — Wed, 27 Apr 2016 16:23:35 GMT

I'm giving Darren credit for the answer at this point in time. Even though the da.SearchCursor did not resolve the issue entirely, and is actually working the same, it did help me find where it was having the issue and bandage it for now. (i.e. by adding the "with.." AND a "for..", found it was the "for..." that had the problem....i.e. zero records in cursor). So thanks for finding that Darren.