Latest Contributions by RichardFairhurst

First I have edited the code to stop and start an edit operation before moving from the insert cursor to the update cursor.  Also, I had to add statements to delete the cursors.  You should stop Python entirely and reboot to clear all of the InsertCursors your code runs may have left open. What version of Desktop are you using.  Do you have attribute indexes on the Insert fc?  Is this Oracle, SQL Server or what?  Are the tables versioned or unversioned?  I have seen very slow sde performance before, but it usually is something about the fc or the editor session, not the da insertCursor directly. It only happens with sde.  A file geodatabase would be done in under a minute (I did 121K inserts into a fgdb fc in 19 minutes even with the print statements left in the insert loop when I tested my code.  Without the print statements the code finished 121K inserts in about 13 seconds).  Get the difference in time for running just the ic.insertRow(scRow) command.  Your print statements should have done that.  Also, try changing the edit.startEditing(True,True) to edit.startEditing(False,True) or edit.startEditing(False,False).  You do not need the undo stack since you don't include cancel operation logic. ... View more

A note on performance.  The majority of time in the loop should be used by the AddMessage, print, and log file processing, not the actual reading of the dictionary, or the insertion or update of rows by the cursor.  To really see what the performance difference is from eliminating the embedded cursor and OrderBy processing of the cursor and how fast da cursors really are, the AddMessage, print and log statements should be commented out within the two cursor loops. ... View more

Line 36 is fixed.  I should have used brackets, not parentheses, to access the item out of the tuple.  As i said, it is untested code and little details like that get me nearly every time. ... View more

All source data should be read into a dictionary and then the insert cursor and update cursor should be populated from that.  Never do embedded cursors that iterate through where clauses or use cursors that include the OrderBy clause.  Totally a waste of time.  Add the sort field as the first part of a tuple dictionary key and sort the dictionary keys.  Performance will be incredibly fast.  The untested code below should work and be done in under 1 minute for the cursor portions. The update cursor error occurred because you were trying to update the search cursor row, not the update cursor row.  InsertRow and updateRow do not have the same syntax. See my blog about Turbo Charging Data Manipulation with Python Cursors and Dictionaries. import arcpy, os, time  import Utils  import MM    logFileName = "C:/GISData/Utilities/Logs/ValveInspections.log"  mmapp = MM.InitializeLicense()  previousAUMode = MM.DisableAUs()    workspace = 'Database Connections/OSAuth.sde'  sourceFC = 'Database Connections/OSAuth.sde/Inspection/giValveInspection'  targetTable = 'Database Connections/OSAuth.sde/GVALVEINSPECTION'  targetFC = 'Database Connections/OSAuth.sde/Gas/gValve'    fieldsToInsert = ['FACILITYID','REPAIRSMADE','CONDITIONFOUND','CONDITIONLEFT','MAINTENANCENOTES','TIMEWORKED','LUBRICATEDVALVE','WORKEDBY','INSPECTOR','INSPECTIOID','INSPECTIONDATE','INSPECTIONSTATUS','PAINTED','VACUUMED','TAGGED','WORKEDBY','FLAGGED','MARKER','INSECTICIDE','CGIBEG','CGIEND','MAPCORRECT','EXERCISED','GASVALVEOID']  fieldsToUpdate = ['FACILITYID','VALVEBOX','INSPECTIONDATE','INSPECTIONSTATUS','TAGGED','FLAGGED','MARKER']   logFile = Utils.Log("<PROCESS name='ValveInspections.py'><TIMESTAMP position='start'>" + time.strftime("%Y%m%d.%H%M%S") + "</TIMESTAMP>\n", None, logFileName)  print("PROCESS ValveInspections.py start " + time.strftime("%Y%m%d.%H%M%S"))  try:      Utils.Log("<COMMENT>Starting edit operation</COMMENT>\n", logFile)      print("Starting edit operation")      edit = arcpy.da.Editor(workspace)      edit.startEditing(False,True)      edit.startOperation()  except Exception as err:      raise err      print err.message      Utils.Log("<ERROR>Failed to start edit operation</ERROR>\n", logFile)  try:      Utils.Log("<COMMENT>Create dictionary from sourceFC to insert rows into targetTable start<TIMESTAMP position='start'>" + time.strftime("%Y%m%d.%H%M%S") + "</TIMESTAMP></COMMENT>\n", logFile)      print("Create dictionary from sourceFC to insert rows into targetTable start " + str(time.strftime("%Y%m%d.%H%M%S")))      valueDict = {(r[10],r[0]):(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, fieldsToInsert)}      ic = arcpy.da.InsertCursor(targetTable, fieldsToInsert)      for keyValue in sorted(valueDict.keys()):            scRow = list(valueDict[keyValue])         scRow.insert(0,keyValue[1])         arcpy.AddMessage("Inserting inspection for valve number " + (scRow[0]) + " with inspection date " + str((scRow[10])))          print("Inserting inspection for valve number " + (scRow[0]) + " with inspection date " + str((scRow[10])))          Utils.Log("<ACTION>Inserting inspection for " + (scRow[0]) + " with inspection date " + str((scRow[10])) + "</ACTION>\n", logFile)          ic.insertRow(scRow)     del ic     edit.stopOperation()     Utils.Log("<COMMENT>Stopped searching sourceFC to insert rows into targetTable<TIMESTAMP position='end'>" + time.strftime("%Y%m%d.%H%M%S") + "</TIMESTAMP></COMMENT>\n", logFile)      print("Stopped searching sourceFC to insert rows into targetTable end " + str(time.strftime("%Y%m%d.%H%M%S")))  except Exception as err:     raise err     print err.message     Utils.Log("<ERROR>Failed Insert cursor operation<ERROR>\n", logFile) try:     edit.startOperation()     Utils.Log("<COMMENT>Create dictionary from sourceFC to update rows in targetFC<TIMESTAMP position='start'>" + time.strftime("%Y%m%d.%H%M%S") + "</TIMESTAMP></COMMENT>\n", logFile)      print("Create dictionary from sourceFC to update rows in targetFC start " + str(time.strftime("%Y%m%d.%H%M%S")))      valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, fieldsToUpdate)}      uc = arcpy.da.UpdateCursor(targetFC, fieldsToUpdate)      for updateRow in uc:         keyValue = updateRow[0]         if keyValue in valueDict:             for n in range (1,len(fieldsToUpdate)):                  updateRow = valueDict[keyValue][n-1]             arcpy.AddMessage("Updating gGasValve " + (keyValue) #+ " ValveBox to " + (sc2Row[1]))              print("Updating gGasValve " + (keyValue) #+ " ValveBox to " + (sc2Row[1]))              Utils.Log("<ACTION>Updating gGasValve " + (keyValue) + "</ACTION>\n", logFile)              try:                  uc.updateRow(updateRow)              except Exception as err:                  raise err                  print err.message                  Utils.Log("<ERROR>Failed updating gGasValve FacilityID " + valveNumber + "</ERROR>\n", logFile)      del uc     Utils.Log("<COMMENT>Stopped searching sourceFC to update rows in targetFC<TIMESTAMP position='end'>" + time.strftime("%Y%m%d.%H%M%S") + "</TIMESTAMP></COMMENT>\n", logFile)      print("Stopped searching sourceFC to update rows in targetFC end " + str(time.strftime("%Y%m%d.%H%M%S")))  except Exception as err:      raise err      print err.message  finally:      edit.stopOperation()      edit.stopEditing(True)      mmapp.Shutdown      del mmapp      Utils.Log("<PROCESS name='ValveInspections.py'><TIMESTAMP position='end'>" + time.strftime("%Y%m%d.%H%M%S") + "</TIMESTAMP>\n", logFile)      print("ValveInspections.py end '>" + str(time.strftime("%Y%m%d.%H%M%S")))  ... View more

Did you read this help file?‌  It specifically mentions the Dissolve tool and road casings as an example of what causes the kinds of failures you are experiencing.  The Dice tool‌ may be a more effective way of removing the errors you are seeing than the partitioning techniques you are using, since it accounts for vertices counts that create the issues you are seeing. Also, given that this is a buffer, use of the Cartography generalization tools should be considered acceptable, since these buffers are probably not exact boundaries that represent real legal boundaries or any other real world precise information.  Buffering is already a form of generalization, and some additional cartographic generalization may be useful specifically for avoiding the errors you are receiving without necessarily making radical changes to the final output. For example, Dissolving the lines on name attributes and then using Simplify Line at the start of the process prior to creating buffers could have a significant effect on the vertice count of the buffer output, which is the issue you are running into.  At least it should eliminate vertices that contribute no real geometry angles if you use a small tolerance value, but retaining those extraneous vertices increases the likelihood of tool failure.  If you try the Simplify Line tool review this help on that topic as well.  In addition, or as an alternative, you could also try the Simplify Polygons tool prior to using the Dissolve tool to reduce the vertice count of the buffers.  That may eliminate large numbers of very small true curve segments, which may cause tolerance errors when they are being split during the Dissolve process. In addition to the issues mentioned in the help about shapefiles and personal geodatabase size limitations, if you are outputting to shapefile or personal geodatabase, the vertice counts will explode to convert true curves at end caps into nodes only to dissolve them away, so I hope you are only doing this with a file geodatabase.  Reducing vertice counts with the Simplify Lines or Simplify Polygon tools also probably helps eliminate many very short true curve segments created by the Dissolve which may also contribute to the geometry errors being reported if you are using a file geodatabase already.  Retaining such very small curves will contribute very little to your buffer data analysis in the end and make any other geoprocessing you may do based on the output problematic. ... View more

Overlaps interior to a feature class will affect your final result.  I was conceiving of your interests as more like 5 sets of parcel ownerships or 5 different land use categories, where edge overlaps between polygons in that set do not exist.  However, if that is not the case and these interests have many different overlaps, then first merging them into one feature class will result in the correct answer in terms of which cutblocks they intersect, but the overlaps will result in more redundant records listing the same interest and cutblock combination due to any overlap intersections between separate interests.  A Dissolve would be necessary to remove these redundancies to get all of the unique combinations of interests and cutblocks in the minimum number of records and shapes. So if FN, Trappers, Guides, Private Land and Range have the same set of attributes then Merge or Append will keep those fields to the minimum for each shape.  Merge or append will not change the shapes.  However, during the intersect if this combination of 5 shapefiles into 1 shapefile contains many overlapping polygons within the shapefile, each of those overlapping shapes will first be cut wherever they overlap (so for example 2 polygons that overlap on one shared edge would first turn into 3 polygons), but none would be eliminated due to intersecting another polygon in the same feature class and the number of attributes fields would remain the same in that one merged feature class.  So, if a lot of overlaps in your interests exist in the set of combined polygons, then the result would not be exactly the same as the individual intersects, because more records would be created to reflect the breakup of those overlaps.  The same is true in the cutblocks as well.  Overlaps of 2 cutblocks will first be cut up into at least 3 shapes first and then those 3 shapes will intersect with the interests. Union would do the same with two shapefile inputs, cutting shapes at every overlap within either of the two feature classes and at every overlap between the two feature classes, but the output would include every combination at the end and could also generate gap shapes where none of the inputs exist that are completely interior to the outermost boundary.  So if an interest was not in any cutblock it would be included in the output and if any cutblock was not included in any interest, it would be included in the output. ... View more

In what way doesn't intersecting all 5 interests against the cutblocks "work"?  You have given very little information that would let us replicate your data or outputs.  Intersect rarely needs to be done one polygon set at a time if the attributes are evaluated correctly from the full set.  So I see nothing in your post that lets me understand what kind of problem you are having.  I would appreciate a little more insight into what your inputs look like (are there just two polygon feature classes, many feature classes, etc) and what your output should look like (separate tables for each interest or one table with all interests listed and all applicable cutblocks). If you have 5 separate interest polygon feature classes, then they would have to be merged first (with appropriate attributes to keep track of what polygons came from what sources) before intersecting them with the cutblocks if you want to do everything in a single intersect analysis. ... View more

Self-Relates or Self-Joins are not possible for feature classes or tables directly to themselves.  However, for a standalone table an Event layer (XY or Route Event Layer) created from the table can be related or joined back to the table that created it, since the Event layer exists in memory with a different schema (at least an additional shape field).  So even if a table had no XY coordinates in it, any numeric field could be used as fake XY values to generate an XY Event layer for the purposes of a self-relate. You cannot relate a feature class to itself even when you add the feature class in multiple layers with different layer names and settings and you cannot use the Event layer tools that work with tables.  However, for a feature class (and a standalone table) you can use the Create Query Table tool to generate an in memory layer from the feature class and perform a self-relate to the Query Table output.  I prefer the XY and Route Event layers over a Query Table, because XY and Route Event layers are editable and respond to edits in the source table, while a Query Table cannot be edited and will not respond to edits in the source feature class or table.  For SDE databases some other tools (Create Query Layer) may have the same effect. ... View more

I agree with Daniel that the code Xander posted (using tuples or lists) works with Python 2.7.8 and ArcMap 10.3.  I get the same results he showed.  The coordinates work even with my default state plane coordinate system which is not really designed for this data. Sounds like a reboot is in order if it is not working. ... View more

The missing part of your code that most likely is part of the problem is that you have not shown what the output variable is assigned.  Maybe it is writing somewhere else than where you think it is writing, depending on the workspace settings, fc name, etc.  Since it is not throwing an error, it probably is a valid output, but I have wasted plenty of time checking what I thought I told my code to do when I actually told my code to do something different. ... View more

Then go back and make the code more like the InsertCursor help.  Also, never create an InsertCursor inside a loop without also deleting it inside the loop.  For this code and most code an InsertCursor should only be created once outside the loop and only the insertRow operations should be inside the loop.  Then be sure to delete the InsertCursor, since there is no other correct way to stop it and release its locks.  Failure to delete the cursor in your previous code may be the reason why the cursor never completes and refuses to show the records it created, since it still thinks it is waiting to create more records and you have opened many cursors on the same source. cursor = arcpy.da.InsertCursor(output, ["SHAPE@"]) for j in xys:     print j     array = arcpy.Array([arcpy.Point(j[1], j[2]), arcpy.Point(j[3], j[4])])     polyline = arcpy.Polyline(array)     cursor.insertRow([polyline])     del array     del polyline     print 'inserted row' del cursor ... View more

Why is there a comma in the line that reads: cursor.insertRow((polyline,))?  Also, shouldn't the field list and polygon array be lists?  So try: for j in xys:     print j     myList = []     myList.append(arcpy.Point(j[1], j[2]))     myList.append(arcpy.Point(j[3], j[4]))     array = arcpy.Array(myList)     polyline = arcpy.Polyline(array)     cursor = arcpy.da.InsertCursor(output, ["SHAPE@"])     cursor.insertRow([polyline])     print 'inserted row' ... View more