Select to view content in your preferred language

Issue with geometry not updating properly within multiple cursors

4430
11
Jump to solution
04-26-2016 05:45 PM
RebeccaStrauch__GISP
MVP Emeritus

I'm pretty sure I just have a syntax or indentation issue, but I've been trying to track this down a good chunk of the day, so time to "rubber duck".  The full program is a bit more than anyone will be able to deal with, but if anyone really wants to look at the full code, I can upload it.    Typing is out is sometime helpful for me too....so here's hoping.

Basic (and simplified) description: given a random point with a "transtype" Riparian or Contour,

  • if riparian, grabs river, creates route, snaps point to route.
    • if river < desired length, grabs full river
    • if river >= desired length, tries to create segment with length/2 in each direction
      • if first half < length/2, shifts both points, and vice versa (i.e., if snaped point is too close to end)
    • write to array...with a bunch of other stuff
  • if contour, creates contour as route, snaps point to route (only grabs segments > 5000m)
    • if contour < desire length, grabs full contour
    • if contour >= desired length, tries to create segment
    • with length/2 in each direction
      • if first half < length/2, shifts both points, and vice versa (i.e., if snaped point is too close to end)
    • write to array...with a bunch of other stuff
  • take the array and populates a FC with all the info.

For 2000 random points, it finished with no error, but for some reason about 380 points (or 279 of those 380 points) are duplicating the transect above it.  I do have multiple cursors, which for the most part seem to be fine, but one of the cursors has the "for aRoute2 in route2:" being bypassed.  I believe this is the issue.

So, with that all said, here is the part that I think is the issue.  Not sure that this will be of any used, but in case someone sees something obvious...    I'm on a very short deadline....i.e., here tonight until I get this fixed, most likely.  So any help is appreciated, but may not respond right away.

Dan PattersonCurtis PriceDarren WiensJake SkinnerJoshua Bixby​   any of you seeing anything obvious?  I know cursors within cursors is n necessarily a good idea, and have no time to change it in a major way (another time), I'm in triage mode right now.  Thanks for any help you might offer.  I'll by you a free drink if I ever see you at the dev summit or UC. 

NOTE: trying to fix the indentation in geonet is messing it up more....the first else: and everything else until the final else is indented....

# snippet...too much needed to send all and expect it to work
else:  # transtype == 'contour'
    # ##############
    #   Contours
    # ##############
    print(" -Create contour for elev {0} m".format(elevm))
    ContourList(inDEM, fullContour, elevm)
    #print(" -Clipping full contour by study area")
    arcpy.Clip_analysis(fullContour, inStudy, clipContour)
    #print("       making clipped contour FL")
    arcpy.MakeFeatureLayer_management(clipContour, "tmpClipContour")
    print(" -Selecting contour segs > 5000 m")
    #arcpy.SelectLayerByAttribute_management("tmpClipContour", "NEW_SELECTION", where_clause="Shape_Length >= 5000")  
    arcpy.SelectLayerByAttribute_management("tmpClipContour", "NEW_SELECTION", "Shape_Length >= 5000")

    # copies selection to new FC
    #print(" - Copying selection to new FC")  # here 4/4/2016
    arcpy.CopyFeatures_management("tmpClipContour", tmpContourFC)
    arcpy.MakeFeatureLayer_management(tmpContourFC, "tmpContourFL") 
    # # wrong? tmpContourFL = arcpy.mapping.Layer("tmpClipContour") 
    #print("\n**New selection of nearest contour segment to pt")
    #print(" - Creating new feature layer from origPtFL")
    arcpy.SelectLayerByLocation_management("tmpContourFL", nearType, "origPtFL", nearDist)    
    #print(" - Copy selection to new tmp FC contourSel")
    arcpy.CopyFeatures_management("tmpContourFL", contourSel)

    # Creates route of the selected contour segment
    print(" -Create temp route; create and process cursor")
    arcpy.CreateRoutes_lr(contourSel, contourField, tmpRoute)
    #print(" - Create SearchCursor from route")

    routeSCur2 = arcpy.da.SearchCursor(tmpRoute, ["SHAPE@", "Contour"])            
    print("getting here")

    with routeSCur2 as route2:
        print("getting inside cursor")   # will get to here
        for aRoute2 in route2:
            print("getting here2")     # #######   will not always get to here   #######
            geo = aRoute2[0]          # get all contour elevation for the route
            theElev = aRoute2[1]      # get elevation, actual contour as a route
            #snapPt = (geo.snapToLine(ptGeom)).centroid # shp_fld
            arcpy.LocateFeaturesAlongRoutes_lr(ptGeom, tmpRoute, contourField, nearDist, tmpRoutePtLoc, rid)
            lenTrans = geo.length  
            print("  **Contour elev {0}:  {1} m".format(theElev, RoundUp(lenTrans)))
            flyFullContour = lenTrans <= int(targetLength)
            if flyFullContour:
                print(" **Taking full contour: {0}m ".format(lenTrans))
                startx  = geo.firstPoint.X
                starty  = geo.firstPoint.Y
                endx = geo.lastPoint.X
                endy = geo.lastPoint.Y

                startPtdd = arcpy.PointGeometry(arcpy.Point(startx, starty) , initialSR).projectAs(geoSR)
                startxdd = startPtdd.centroid.X
                startydd = startPtdd.centroid.Y

                #print(" -Start point {0} {1}".format(startxdd, startydd))
                endPtdd = arcpy.PointGeometry(arcpy.Point(endx, endy) , initialSR).projectAs(geoSR)
                endxdd = endPtdd.centroid.X
                endydd = endPtdd.centroid.Y

                #print(" -End point {0} {1}".format(endxdd, endydd))
                tmpTrans = geo # taking full length.segmentAlongLine(startDist, endDist)
                #tlength = tmpTrans.length
                #ripNotes = "short"
                #print("{0} transect #  {1}".format(transtype, transID))
                print("{0} transect #  {1}  length: {2}  Notes: {3}".format(transtype, transID, tlength, ripNotes))
            elif lenTrans >= int(targetLength):    #else:

                ptSCue = arcpy.SearchCursor(tmpRoutePtLoc)
                field_name = 'MEAS'
                for routeMeas in ptSCue:
                    snapPtDist =  routeMeas.getValue(field_name)
                    print("  ->Measurement along route: {0}".format(snapPtDist))
                    startDist = snapPtDist - halfTrans
                    endDist = snapPtDist + halfTrans
                    print("\n -Initial startDist: {0}  endDist: {1}".format(startDist, endDist))
                    initialSegment = geo.segmentAlongLine(endDist, startDist)
                    initialSegmentLength = initialSegment.length
                    print(" -->Initial length: {0}".format(initialSegmentLength))
                    if RoundUp(initialSegmentLength) < int(targetLength):
                        # Check each side to make sure distance is = halfTrans
                        #  if not, shift as needed

                        # check length of the segment from the snapPtDist to startDist
                        firstSegment = geo.segmentAlongLine(snapPtDist, startDist)
                        firstSegmentLength = firstSegment.length
                        #print(" -First Length: {0}".format(firstSegmentLength))

                        #check first
                        if RoundUp(firstSegmentLength) < (halfTrans):
                            print(" -First half is short: {0}m".format(firstSegmentLength))
                            # calculate the distance it is short
                            shiftDist = halfTrans - firstSegmentLength
                            print("   ..shift distance for first segment: {0}m".format(shiftDist))
                            #startDist = startDist - shiftDist
                            # short from start to mid, so extend end distance
                            endDist = endDist + shiftDist
                            print("  ..new endDist: {0}m".format(endDist))
                        else:  # first segment ok, must be second part
                            endSegment = geo.segmentAlongLine(snapPtDist, endDist)
                            endSegmentLength = endSegment.length
                            print(" -Second half is short: {0}m".format(endSegmentLength))
                            # calculate the distance it is short
                            shiftDist = halfTrans - endSegmentLength
                            print("  ..shift distance for first segment: {0}m".format(shiftDist))
                            # short from end to mid, so start earlier on the line, it a minus measure
                            startDist = startDist - shiftDist
                            #endDist = endDist - shiftDist
                            print("  ..new startDist: {0}m".format(startDist))                  

                    # use the startDist and endDist to create the point geometries
                    #startPt = geo.positionAlongLine(startDist)
                    #endPt = geo.positionAlongLine(endDist)

                    startPt = geo.positionAlongLine(startDist)
                    startx = startPt.centroid.X    #need DD
                    starty = startPt.centroid.Y            
                    #print("Start Dist: {0}".format(startDist))
                    # midPt = geo.positionAlongLine(snapPtDist) #not using mid points
                    endPt = geo.positionAlongLine(endDist)
                    endx = endPt.centroid.X
                    endy = endPt.centroid.Y    
                    tmpTrans = geo.segmentAlongLine(startDist, endDist)
                    tlength = tmpTrans.length

            del ptSCue
            # Processing pts to geographic
            startPtdd = arcpy.PointGeometry(arcpy.Point(startx, starty) , initialSR).projectAs(geoSR)
            startxdd = startPtdd.centroid.X
            startydd = startPtdd.centroid.Y                 
            #print("start point {0} {1}".format(startxdd, startydd))
            #midPtdd = arcpy.PointGeometry(arcpy.Point(midPt.centroid.X, midPt.centroid.Y) , initialSR).projectAs(geoSR)   #not using mid points
            endPtdd = arcpy.PointGeometry(arcpy.Point(endx, endy) , initialSR).projectAs(geoSR)
            endxdd = endPtdd.centroid.X
            endydd = endPtdd.centroid.Y                            

            #tmpTrans = geo.segmentAlongLine(startDist, endDist)

            #tlength = tmpTrans.length
            shortLength = int(targetLength) - 500
            if tlength < shortLength:   # add A to ID for short trans...add in field.
                transID = "{0}A".format(transID)

            if tmpTrans.isMultipart:
                ripNotes = "contourParts"
                print("  ** Transect in multiple parts **")
            else:
                ripNotes = "contour"
            #print("{0} trans # {1}".format(transtype, transID))
            print("{0} transect #  {1}  length: {2}  Notes: {3}".format(transtype, transID, tlength, ripNotes))

else:
    break 

Edit:  added not above the python script..reformatting was not good.

0 Kudos
11 Replies
RichardFairhurst
MVP Alum

Rebecca:

It appears you have created code that uses embedded cursors.  The code must be rewritten to eliminate that configuration if you want to make it process efficiently, since embedded cursors are the absolute worst way to approach any problem.  Instead one of the cursors should load the data to a dictionary for any look up of data and only one update cursor should be processed at a time.  The code revision can result in a 500 fold improvement in speed if you are processing 10,000 records against 10,000 records, since my approach would only have to process 20,000 record reads while an embedded cursor would process up to 10,000,000 record reads..

0 Kudos
RebeccaStrauch__GISP
MVP Emeritus

Hi Richard,

Thanks for the comments, and I know there may be a more efficient ways to do this, but the embedded cursors in-and-of-themselves are not the issue, unless I have an indent off somewhere, i.e. the updating of my "tmpTrans" variable in the incorrect place.  The issue seem to be that with some points, because I eliminate contour-parts that are < 5000m before making it into a route, it is no longer finding a "location along the route" within my search distance, which should be adequate. (BTW the contour is created on the fly from the point's elevm value),  

Now that I fixed the two SearchCursor to the da.SearchCursor that I had missed, I was able to see that it was still crashing at the "for row in cursor:", but was making into the "with..." for the cursor.  That lead me to check what was being created in the table feeding the cursor, and then setting a flag if the count=0.  I need to back track further to see why it is not getting any records, since stepping thru it manually works, but I'm sure I am just missing something.   At that point, I will need to set a "fix" by maybe dropping the elevm value a couple meters and trying again.  Our DEM is not as accurate as lower-48, and these are traversed via plane (SuperCub) so a couple meters will not matter in the scheme of things.  Also, the missed points typically are near the top of a hill/mountain.

BTW - we used to have many more of these issues in river valleys, so I added a "riparian" option that grabs the stream and gives and offset distance and direction from the stream...that is working.  We labelled these as "skipped points" and dealt with them in the field.

After the crunch/field time is over, and for my own satisfaction, I will look into using dictionaries etc., and to resolve this issue fully.  But since I was working on my logic from Avenue and AML programs, and the lead was changing/adding new requirements often, I did what I understood, and what I think is important, what the next person might be able to read and understand, if they are new to python.  With that said, dictionaries and other solutions may be the way to go on a rewrite....but that will be all on my time....so a fun project.

Just an FYI - I ran this on 1250 points last night...74 came up with the flag. At this point, these would be similar to the older "skipped points" and I can put them and deal/test them directly.

I know this is more than anyone cares about fo this project, but it is good for me to think it out by typing.

0 Kudos