Latest Contributions by RichardFairhurst

At the time I originally wrote this blog I had not really mastered the techniques I have described in my Turbo Charging Data Manipulation Using Python Cursors and Dictionaries.  I have now written a script using that technique that can perform the steps described in the Creating Address Ranges on the Centerlines section of this blog without the need for creating any intermediate geoprocessiing output tables.  The script is able to create ranges for all 29 of the jurisdictions in my County from over 200,000 address points for 50,000 centerline segments in under 5 minutes.  I also wrote a script for creating the Locate Features Along Routes outputs for all of my jurisdictions and am currently working on scripts that can automate several aspects of the QC analysis processes described in blog.  While I don't think I can completely eliminate the time someone would have to spend to manually correct data errors that impact the address range outputs, I can dramatically reduce the number of separate geoprocessing steps the user would perform to discover them and analyze them. I would like to hear from the users on this forum that are interested in my Address Range scripts before I publish a new blog on this topic, so please post a response to this blog if you are interested.  I also plan on asking that anyone that uses or adapts the scripts I will be publishing in the new blog to retain a credit for my work in their copies of the scripts. ... View more

Executing 20,000 separate selections against 10 million records should take the amount of time you are seeing.  This means you have to read through the 10 million records 20,000 times.  You asked for my help based on my Turbo Charging Data Manipulation with Python Cursors and Dictionaries.  That technique solves this problem by reading the 20,000 records into memory once and then running through each of the 10 million records once updating the record from a lookup dictionary.  This eliminates 19,999 reads through the 10 million records.  So instead of iterating the 20,000 records you need to do an update iteration through the 10 million records after loading the 20,000 records in the dictionary. I don't really know how to set up the update iteration of your 10 million records, so I can't suggest how to do that.  However, reading the 20,000 records into a dictionary and iterating through a da.updateCursor would be done with this code: inputFC = arcpy.GetParameter(0) dbName = arcpy.GetParameterAsText(1) tblName = arcpy.GetParameterAsText(2) fldStn = arcpy.GetParameterAsText(3) fldX = arcpy.GetParameterAsText(4) fldY = arcpy.GetParameterAsText(5) # Check FC is point type, has ID field. inDesc = arcpy.Describe(inputFC)# Get description of FC if inDesc.shapeType.lower() != "point": raise customErr, "Data must be point type" if not inDesc.hasOID: raise customErr, "Data must have an ID field" # Get reference to active data frame mxd = arcpy.mapping.MapDocument("CURRENT") df = mxd.activeDataFrame arcpy.AddWarning("Updating Database...") cnGP = sqlite3.connect(dbName) fields = ['Station_Value', 'SHAPE@X','SHAPE@Y'] valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(inputFC, fields)}‍‍ updateFC = dbName updateFieldsList = [fldStn, fldX, fldY] with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows: for updateRow in updateRows: # store the Join value of the row being updated in a keyValue variable keyValue = updateRow[0] # verify that the keyValue is in the Dictionary if keyValue in valueDict: # transfer the values stored under the keyValue from the dictionary to the updated fields. for n in range (1,len(sourceFieldsList)): updateRow[n] = valueDict[keyValue][n-1] updateRows.updateRow(updateRow) del valueDict ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ ... View more

Glad to hear it.  You can experiment with the formatted label version now if you want. ... View more

Please show me the properties of the LINKID fields in both the feature class and the table.  I want to confirm they are both Long fields.  If one is Long and the other is Double, that would cause a problem, since Python would store the Long as 53861 and the double as 53861.0, and these two values do not match exactly. To make sure both are interpreted as Long values change Line 18 and Line 35 to: 18        relateKey = int(relateRow[0]) 35  labelKey = int([LINKID]‍‍‍‍‍‍)‍‍‍ ... View more

How big are the Dictionary Record sets being reported for the different field lists you have tried?  The record count of the dictionary should be less than the record count of the related table itself, because all records are being collapsed into just the count of unique LINKID values in your table.  Probably the record count is higher when you use , because this would be summarized on the OBJECTID values, which are unique for every record, not the LINKIDs, which can be duplicated on multiple records.  Perform a summary on your LINKID field of your related table to find out if the count of the dictionary matches the count or records summarized on the LINKID field.  If they match the dictionary was loaded correctly.  If the record count of the summary is less than the number of features in your feature class, than some features have no related records in the table, assuming that the LINKID values of the features are unique for each feature. Probably the label should avoid any special formatting or attempts to make it look like a table until you get a result.  Try the simple label code below. # Initialize a global dictionary for a related feature class/table relateDict = {} def FindLabel ( [LINKID] ):   # declare the dictionary global so it can be built once and used for all labels   global relateDict   # only populate the dictionary if it has no keys   if len(relateDict) == 0:     # Provide the path to the relate feature class/table     relateFC = r"D:\delbay_v3_3.gdb\Envr_hab_ls_v3_3_delbay_sp_02"     # create a field list with the relate field first (ROUTE_NAME),     # followed by sort field(s) (COMNAME), then label field(s) (FEAT_LABEL)     relateFieldsList = ["LINKID", "COMNAME", "FEAT_LABEL"]     # process a da search cursor to transfer the data to the dictionary     with arcpy.da.SearchCursor(relateFC, relateFieldsList) as relateRows:       for relateRow in relateRows:         # store the key value in a variable so the relate value         # is only read from the row once, improving speed         relateKey = relateRow[0]         # if the relate key of the current row isn't found         # create the key and make it's value a list of a list of field values         if not relateKey in relateDict:           # [searchRow[1:]] is a list containing           # a list of the field values after the key.           relateDict[relateKey] = [relateRow[1:]]         else:           # if the relate key is already in the dictionary           # append the next list of field values to the           # existing list associated with the key           relateDict[relateKey].append(relateRow[1:])     # delete the cursor, and row to make sure all locks release     del relateRows, relateRow   # store the current label feature's relate key field value   # so that it is only read once, improving speed   labelKey = [LINKID]   # start building a label expression.   # My label has a bold key value header in a larger font   expression = '<FNT name="Arial" size="12"><BOL>{}</BOL></FNT>'.format(labelKey)   # determine if the label key is in the dictionary   if labelKey in relateDict:     # sort the list of the list of fields     sortedList = sorted(relateDict[labelKey])     # add a record count to the label header in bold regular font     expression += '\n<FNT name="Arial" size="10"><BOL>Species Count = {}</BOL></FNT>'.format(len(sortedList))     # process the sorted list     for fieldValues in sortedList:        # append related data to the label expression        # my label shows a list of related        # cross streets and measures sorted in driving order        expression += '\n{} - {}'.format(fieldValues[0], fieldValues[1])     # clean up the list variables after completing the for loop     del sortedList, fieldValues   else:     expression += '\n<FNT name="Arial" size="10"><BOL>Species Count = 0</BOL></FNT>'     expression += 'Not in {} dictionary IDs'.format(len(relateDict))   # return the label expression to display   return expression‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Please screen shot a record in the table view from a selected feature you are trying to label in your map view with the LINKID field value showing and also the selected set of records that match it in the related table's table view with the 3 field values showing. ... View more

Try the code below.  I have adjusted the label expressions to fit your field values, which are both string, and to sort and print the Comname then the Feat_Label.  The feature dataset of the layer should not matter.  I also have changed the label for the case where no related record is found to report how many records are in the dictionary to help diagnose the problem. # Initialize a global dictionary for a related feature class/table relateDict = {} def FindLabel ( [LINKID] ):   # declare the dictionary global so it can be built once and used for all labels   global relateDict   # only populate the dictionary if it has no keys   if len(relateDict) == 0:     # Provide the path to the relate feature class/table     relateFC = r"D:\delbay_v3_3.gdb\Envr_hab_ls_v3_3_delbay_sp_02"     # create a field list with the relate field first (ROUTE_NAME),     # followed by sort field(s) (COMNAME), then label field(s) (FEAT_LABEL)     relateFieldsList = ["LINKID", "COMNAME", "FEAT_LABEL"]     # process a da search cursor to transfer the data to the dictionary     with arcpy.da.SearchCursor(relateFC, relateFieldsList) as relateRows:       for relateRow in relateRows:         # store the key value in a variable so the relate value         # is only read from the row once, improving speed         relateKey = relateRow[0]         # if the relate key of the current row isn't found         # create the key and make it's value a list of a list of field values         if not relateKey in relateDict:           # [searchRow[1:]] is a list containing           # a list of the field values after the key.           relateDict[relateKey] = [relateRow[1:]]         else:           # if the relate key is already in the dictionary           # append the next list of field values to the           # existing list associated with the key           relateDict[relateKey].append(relateRow[1:])     # delete the cursor, and row to make sure all locks release     del relateRows, relateRow   # store the current label feature's relate key field value   # so that it is only read once, improving speed   labelKey = [LINKID]   # variables to adjust table cell sizes   iMaxLbl1Sz = 0    iMaxLbl2Sz = 0    iSpace = 5    # determine if the label key is in the dictionary   if labelKey in relateDict:     # sort the list of the list of fields     sortedList = sorted(relateDict[labelKey])     # process the sorted list to determine cell spacing     for fieldValues in sortedList:       strLabel1 = fieldValues[0]        strLabel2 = fieldValues[1]        if (len(strLabel1) > iMaxLbl1Sz):         iMaxLbl1Sz = len(strLabel1)        if (len(strLabel2) > iMaxLbl2Sz):         iMaxLbl2Sz = len(strLabel2)      # clean up the fieldValues variable once the for loop is done     del fieldValues     # My label has a key value header followed by a record count     expression = labelKey     expression += '\n<UND>Species Count = {}</UND>'.format(len(sortedList)) + '_' * (iMaxLbl1Sz + iMaxLbl2Sz + iSpace + 1 - len('Species Count = {}'.format(len(sortedList))))     # process the sorted list     for fieldValues in sortedList:       strLabel1 = fieldValues[0]        strLabel2 = fieldValues[1]       k1 = (iMaxLbl1Sz - len(strLabel1)) + 2       k2 = iSpace + (iMaxLbl2Sz - len(strLabel2)) - 3       # append related data to the label expression       # my label shows a list of related       # cross streets and measures sorted in driving order       expression += '\n' + strLabel1 + "." * k1       expression += "|"       expression += "." * k2 + strLabel2 + "|"     # clean up all list variables after completing the for loops     del sortedList, fieldValues   else:     # My label has a key value header followed by a record count     expression = labelKey     expression += '\n<UND>Species Count = 0</UND>'     expression += 'Not in {} dictionary IDs'.format(len(relateDict))   # return the label expression to display   return expression‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ ... View more

I can't interpret the code, since the indentation is gone.  Please indent the code properly. I can say that line 47 and 60 need to change to just the field value, since your data in the field you are referencing is not a float value. ... View more

Jason: Can you please post your code.  I may be able to see what modifications are causing the output to not count any features.  It is critical that the LINKID field values in the related table and the field you used for the labeled features contain matching values.  Using the wrong field from the feature class as your relate field that does not actually match any of the values in the LINKID field would likely not cause an error, but it would never find any related records in the related table and would produce labels without any related values listed and a related record count of 0. ... View more

I wanted to mention that the bearing conversion Python formula I gave works both to convert from Geographic bearings to Arithmetic bearings and vice versa.  (%LOC_ANGLE% is the angle field or value you want to convert) (450 - %LOC_ANGLE%) % 360 Also, you can get the complimentary angle of any angle in either bearing system with the Python formula: (180 +  %LOC_ANGLE%) % 360 ... View more

When you use the Make Route Event Layer, you should actually not use the Offset field option (I mistakenly used it in the screenshot).  You want the points to have no offset so that they will fall on the midpoint of the line directly.  The offset_dist field was created to be used with the Bearing Distance to Line tool to create lines that extend from the centerline outward across the casing boundaries. ... View more

The angles created by the Linear Referencing layer are Arithmetic Bearings, not Geographic Bearings.  Geographic Bearings start with 0 degrees pointing North and the degrees increase in a clockwise direction, but Arithmetic Bearings start with 0 degrees pointing East and the degrees increase in a counterclockwise direction.  You have your layer set to show Geographic Bearings and need to change it to show Arithmetic Bearings.  Change the setting on the Symbology tab by pressing the Advanced button and choosing the Rotation... option.  Then choose your angle field and the Arithmetic Rotation Style and press OK for all the dialogs. \ Anyway, I just noticed that the Bearing Distance to Line tool expects degrees to be Geographic Bearings.  To convert Arithmetic Bearings to Geographic Bearings create a new double field and use this Python calculation (where %LOC_ANGLE% is the Arithmetic angle field you want to convert): (450 - %LOC_ANGLE%) % 360 I am sorry for the numerous steps needed just to convert the outputs of one tool for use as an input for another tool.  The Geoprocessing Tool approach tends to run into these kinds of issues where the tools don't easily integrate with each other.  A Python script could streamline the process and eliminate several steps, but my instructions are just meant to get the job done.  Once the process is proven to work you can invest time in learning ways to optimizing the steps if you will need to do this process repeatedly. ... View more

The output of step 10 should be a table, because the Bearing Distance to Line tool in Step 11 expects a table as its main input. ... View more

Is the table an Excel file or csv file or any table that lacks an ObjectID?  If it is then relates and joined record selections are prohibited.  See the help on the Fundamentals of Objectid fields.  You have to convert Excel files and csv files to either a geodatabase table that has ObjectIDs or a dbf table that has FIDs to be able to create a relate or do joined record selections.  The table conversion probably should occur before you use the XY Table to Point Layer tool for all tables involved in your relates or joins or for the point layer table you could export the point layer created by the tool to transform the in-memory layer the tool creates (which may lack a proper ObjectID if the table source is a csv file) into a permanent on-disk geodatabase point feature class (which will automatically generate an ObjectID field).  The Table to Table tool can be used to do the conversion of a csv file into a geodatabase table. ... View more

Here is how I would set it up in ArcMap (I won't use Pro).  The Route Identifier Field for both the Input Route Features and the Event Table Properties is the "FID_segment_original" field.  The "halfway" field is the Measure field.  The "offset_dist" field is the Offset Field.  It appears that the compliment angle option changes the Angle field output rather than creating a second field with the compliment angle, so you may have to create a new field for the compliment angle and calculate it.  I would need to see examples of the output you get to give you the field calculator formula but basically it would involve adding or subtracting 180 degrees and correcting values that are lower than 0 degrees or higher than 360 degrees. ... View more

The cell size is 0.5 feet height and width per pixel.  That is the resolution of my aerial.  The maximum area of my aerial I can work with from my Image Service is 15,000x15,000 pixel, because it has a restriction on the number of columns and rows I can download in order to change the NODATA values of my aerials.  I was using 500x500 pixels for my chips, because it exactly divides my downloaded aerials into whole chips and at the size the chip will normally contain one or more whole buildings and bigger portions of large buildings. Smaller size chips like 375x375 pixels (56.25% of a 500x500 pixel chip) or 300x300 pixels (36% of a 500x500 pixel chip) would also exactly divide my downloaded aerials into whole chips, but of course the number of whole buildings in each chip will be reduced and more chips will only contain partial buildings.  However, if I create a fishnet I can also do an Identity with my building polygons.  Then I should be able to statistically analyze those results so that the majority of chips I use for training contain whole buildings or all portions of large buildings that won't fit within a single chip. I don't really understand what the graph of learning rates means or how different learning rate numbers will affect the model training.  Can you provide a little more explanation of what adjustments to the learning rate does and what aspects of the learning rate graph I really need to focus on? ... View more