Latest Contributions by RichardFairhurst

IN case you ever need to update SDE here is some code I am using to do that kind of editing.  I don't want to risk editing all 120k records at once, since SDE is touchy about bulk updates, so I have made the updateFC into a layer with a definition query to process fewer records.  The code below also illustrates that the field lists can be different between the source and the update feature class without major modifications to my code set up.  I am processing my records right now and it is updating my target FC extremely fast. import arcpy import os sourceFC = r"L:\rfairhur\Layers\Local_Government\Local_Government.gdb\ReferenceData\RoadCenterline" updateFC = r"C:\Users\RFAIRHUR\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Trans Connection to SQL Server.sde\GDB_TRANS.TRANS.TRANSPORTATION_MAINT\CENTERLINE" arcpy.MakeFeatureLayer_management(updateFC, "CENTERLINE", "CL_ID > 10000 AND CL_ID < 20001") updateFC = "CENTERLINE" workspace = r"C:\Users\RFAIRHUR\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\Trans Connection to SQL Server.sde" edit = arcpy.da.Editor(workspace)  edit.startEditing(False, True)   edit.startOperation() sourceFieldsList = ["CL_ID", "ROUTE_ORIENTED", "ONEWAYDIR", "CALTRANS_FUNCTIONAL_CLASS", "FEDROUTE", "FEDRTETYPE", "STROUTE", "STRTETYPE", "CTYROUTE", "MUNILEFT", "MUNIRIGHT", "ZIPLEFT", "ZIPRIGHT"]    updateFieldsList = ["CL_ID", "ROUTE_ORIENTED", "TRAVEL_DIRECTIONS", "CA_FUNCTIONAL_CLASS", "FEDERAL_ROUTE", "FEDERAL_ROUTE_TYPE", "STATE_ROUTE", "STATE_ROUTE_TYPE", "COUNTY_ROUTE", "CITY_LEFT", "CITY_RIGHT", "ZIP_LEFT", "ZIP_RIGHT"] print "Reading Records"   valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}  print "Records are read"   with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows:      for updateRow in updateRows:          keyValue = updateRow[0]          if keyValue in valueDict:              for n in range (1,len(sourceFieldsList)):                    updateRow = valueDict[keyValue][n-1]              updateRows.updateRow(updateRow)    del valueDict   edit.stopOperation()  edit.stopEditing(True) ... View more

I understand that you are not seeing the updated values, but I am not clear whether the print statements are showing you the values you expect or not.   I will assume that they are showing the correct values unless you tell me otherwise. You have indented line 32 one level too far.  it should be outside of the loop that reads the fields, since it only needs to post the update of the row after all the fields of the row are updated.  Updating the row one field at a time will slow down the process needlessly. Do not del the updateRows variable.  The "with" cursor creation syntax automatically cleans up the cursor at the end.  I seem to recall that if you del that variable after the loop closes it messes things up with this syntax.  The variable Row becomes nothing by the time the cursor is fully read, so it does not need a del statement.  The other variables are simple strings, small lists and numbers and don't normally get del statements.  I only del the valueDict and never had any problems with memory leaks.  I would get rid of all the del statements except for the del valueDict. ... View more

Luke: The table you want could have been produced 4 days ago if you would just have used the Summary Statistics tool.  The "category" field would be the input to the case_field parameter.  A frequency field will automatically be generated that gives the counts you want, so you don't really need a summary to get the output you want, but the tool requires at least one summary parameter input.  Since any field can be used for the summary, just use the "category" field and get its Count to duplicate the values in the Frequency field. The output of the Summary Statistics tool will be a table that can be joined back to your original data for SQL selections and/or field calculations, or any of other uses that have been discussed so far.  Do not tell me that the tool cannot do what you want, because it definitely can.  As an example, here is the Summary Statistics tool output counting the codes that define my recorded map feature class categories: OBJECTID * RECORDERS_TYPE FREQUENCY COUNT_RECORDERS_TYPE 1 MB 11977 11977 2 PM 12159 12159 3 RS 4141 4141 4 SB 96 96 5 SD 134 134 ... View more

Ian Murray wrote: Richard, I don't believe he was questioning its availability, more its effectiveness with his current task.  Since he is working with text based fields and not numeric summary statistics doesn't seem to be what he is needing. The tool does everything he described.  The tool works in Modelbuilder and will automatically count every unique text category within a point of interest text field if he makes his point-of-interest category field the unique case field.  Summary Statistics remains one of the solutions to this problem as long as the output is defined as a table with every unique text value from a categorizing field with an accompanying count of the instances of that value contained within the source table.  That is exactly what the tool is designed to do. Other solutions are possible, but none will be simpler. ... View more

Luke Newman wrote: summary statistics doesn't work for what i'm doing because the values in the table that i am trying to categorize are not numeric, the field in which i need the unique values contain different types of points of interest in Central London such as automotive, eating/drinking, tourism etc and I need to create a new table which displays a tally of each point of interest category. Thanks anyway. The tool does exactly what you need, based on what you have described.  A Frequency field counting all instances of a unique case value in the source table is always included in every record of the output and all field types (numeric, text or date) can be used as unique case fields.  Text fields also can be used as summary fields if you want the Min, Max, First, Last or Count statistics for the text values.  I use text fields in summaries all the time to get unique street name lists or summaries of text maintenance categories. The toolbox version is much better than the Table View context menu version you are probably thinking of, but even that version works to generate a count field for each unique text field value in a table list. If you need this field combined with another field you can used the Summary Statistics tool to create a multi-field unique key value list with counts.  This normally applies where you have broad categories in one field with more detailed subcategories in another field that could be repeated under more than one broad category.  Multi-field keys cannot be joined back to another table using a Standard Join unless you concatenate the multi-field values into a new field in both the source and output.  However, the Make Table Query tool can create multi-field joins and a Python cursor routine can also do multi-field matching. ... View more

The Summary Statistics tool help link is for 10.1 and says it is available for all license levels.  I have personally used that tool under 10.1 at all license levels.  Are you able to open the Analysis Tools toolbox and the Statistics toolset and see it?  Does it not open when you double-click it? The Frequency and Tabulate Intersection tools that are also found under that toolset are not available with any license lower than an Advanced license.  Only the Summary Statistics tool will work under a Basic license. EDIT:  This new forum format makes it easy to miss the person each reply is being directed to.  I see the issue with opening a toolbox was directed to Dan and not me.  I wish the reply would include quotes from the post being replied to like the old forum did so that it was more obvious what the context of each reply was. ... View more

Use this code to populate a dictionary with your fields and transfer them to an Update Cursor.  The look up of the dictionary will be at least 100 times faster than any embedded cursor code, since it places the look up in memory rather than hitting the hard disk like a cursor does.  So there are only two cursor reads straight through 2 tables/feature classes using this code, rather than an exponential growth curve of loops through the dual cursors. Substitute you table/feature class to be read, your table/feature class to be updated. you field list for the table to be read with the matching keyfield listed first, and a second field list for the table to be updated if the field names are different.  This assumes a 1 to 1 match in the fields being read and the fields being updated.  If every field in the table is matched by a field of the same name and type in the feature class you could extract a field list and just make sure the key matching field is the first in the list.  Sorting the field list is not necessary. import arcpy arcpy.env.workspace = "path to file geodatatbase" sourceFC = "table" updateFC = "points_fc" sourceFieldsList = ["ID", "FIELD1", "FIELD2"] updateFieldsList = sourceFieldsList valueDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)} with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows:     for updateRow in updateRows:         keyValue = updateRow[0]         if keyValue in valueDict:              for n in range (1,len(sourceFieldsList)):                    updateRow = valueDict[keyValue][n-1]              updateRows.updateRow(updateRow) del valueDict ... View more

Well I have been doing this for many years without the Roads and Highways Extension, and still have not used the extension myself, although I am interested in it.  I do have the ability to create LR events based on human readable limits without it and a weekly script that supports the derivative layers needed to do that.  The LR events can be managed with side offsets as I have mentioned and do not cut up my Centerlines.  I calculate values that detect geometry changes to my routes so that I can correct affected measures of existing events. However, I do not keep historic network changes, and do not deal with date stamped versions of my data.  If I did it would be fixed features in separately archived historic datatsets outside of my current data.  So far history has not been important to my organization and they are only interested in current and going forward.  The text history is in the Street Saver database, but the geometry to display it on a map would only exist if no changes to that part of my road network have occurred. It does seem that you should look at time enabled layers for tracking network changes over time if you intend to centralize them all in a single feature class, although the extension makes all of this considerably easier with its toolsets and event listening behaviors.  But you are right that the extension is not free, which is why I said it is something to consider rather than download. ... View more

My own LR system has evolved over several years.  I only maintain a single Centerline layer for all geometry.  These are simple polylines with no LR capabilities. To extend my options to use LR I started by adding fields to my Centerlines to define Route_Names for my Centerlines.  If you have a numbering pattern in your state you can use that for the route names.  I did not so my Route_Names are derived from human readable strings, starting with the street name. My jurisdiction is a County, where the same street name may occur in widely separated parts of the County that have no potential operational relationship to each other.  So initally the Route_Name was the street name combined with an area plan abbreviation from my general plan.  In some cases I found more than 1 operational segment with this name pattern so I added a three character string that could randomly distinguish them, favoring my own jurisdiction to get the version without a random number.  This random number would be used to break off small branching centerlines that messed up the main route stationing geometry into their own routes. I used the Create Routes tool to build these routes (I recommend using the Lower Left corner as a default priority to create stationing from west to east or south to north like engineers use).  I found I had to build my entire route network in phases and then append it all together to avoid overtaxing the Create Route tool.  I also found that some routes could not be built from the Lower Left corner, since they were diagonal in the wrong direction and the behavior of that setting becomes unpredictable in those cases.  So I added a Build_Priority field and assigned LL, UL, LR, or UR values so that these Centerlines would be selected in separate groups for use wit the Create Route tool and appended together. Roads that have their DOTS split and combine in different parts have to be built in a different process.  I have two more fields that separate North or West DOTs from South or East DOTs.    Combined portions get both fields filled in and split portions get one field filled in.  The names include the DOT and a single digit number appended to the Route_Name I described above, so for example I could have a Centerline segment that combines both DOTs with both "I-10 RIVER 001 WB0" and "I-10 RIVER 001 EB0" as the two DOT based Route names that would use the geometry.  These are built with at least 2 separate runs of the Create Route tool.  They have a different schema from my other routes so I use the Merge tool to integrate them. All of this is scripted and runs weekly to recreate my entire Route Network from my underlying Centerlines.  These routes are the LR skeleton.  If I switch over to Roads and Highways that process will change.  I would upload my current routes and then have to develop a route maintenance process would integrate with the Centerline change process. ... View more

We use Street Saver too track all of our Pavement Management data history.  It is designed to deal with Pavement Management rankings and contains algorithms you won't find in any ArcMap application I have seen.  I download it and geocode it from Segment Descriptions using LR event overlays.  I do not encode it into the underlying Centerline and never will.  Since segments can change frequently, I am considering the Roads and Highways extension for improvement history tracking, but our Pavement Management group will not change to ArcGIS as core of its data maintenance tracking at any time in the foreseeable future.  GIS is not responsive enough and not good enough at handling one-to-many relationships for me to recommend it to anyone.  ArcMap reporting is also not equal to the task of outputting the data reports our Pavement Management group needs, or any reporting application as far as I am concerned.  These are ArcGIS's main deficiencies as a database in my view. ... View more

Actually, I consider Summary Statistics the more powerful tool than Frequency, despite the fact that it is available for all license levels.  You can choose one field for the unique case values or choose many fields to generate multi-field unique case values, or no unique case field at all (Frequency requires at least one field for unique case values).  The tool lets you actually reorder fields in the unique case fields or the summary fields (Frequency does not) and is the only tool in all of ArcGIS I have found that does that. You can actually get most summary type values beyond First and Last for strings like Min, Max and Count (unlike the Summary you can do from the context menu in a table view), and all summary types for dates (if you type the date field name into the summary input text box and ignore the warning beep).  Dates are converted to numeric values, which will convert back to date values when you calculate them back into a standard date field.  Frequency does not let you get summary values for strings and dates at all.  The Frequency tool only supports Sum operations on numeric fields and no other summary types. ... View more

If you are going the Linear Referencing approach you should also look at the Roads and Highways Extension.  If places a pair of date stamps on your Centerline network, which lets you retire historic alignments, but still maintain the historic events tied to the old alignments.  You use the time slider to expose events over different time ranges and you can set rules for how realignments, retirements, extensions, etc affect the route measures while the extension maintains measure synchronization for all registered event tables.  To cover a complete history, this is the approach I would recommend. If you have ArcMap 10.2.2 they have added support for creating road segment events based on limit descriptions derived from any features that cross your roadways, including cross-street names, city boundaries, railroads, bridges, etc.  Either limit can include an offset distance along the road and a compass direction that defines the offset direction Your address ranges do not need to change with LR when events simply divide an existing Centerline segment, since LR uses measure positions to segment the road rather than line splits.  All LR events can be combined together with LR tools without the problems of other overlay techniques, because they only overlay on the route they are connected to and the events will not overbound their limits.  LR events are tied directly to the route Centerline geometry, but can be displayed with a side offset to the correct side of the road for each DOT of a speed limit if they are created as separate Roads and Highways event records.  Alternatively you can create a single event for the speed limit with separate fields for Left and Right speeds and offsets, and after extracting the data to a non-Roads and Highways standalone table you can create two event layers (one layer for each DOT) to display the one record with offsets for each DOT. If you are going to divide your actual Centerlines with address ranges I recommend you look at the Address Management Add-In and Attribute Assistant Add-In that can be downloaded from here.  This add-in adds a Create Road and Split Intersecting Road construction tool that will divide address ranges proportionally where a newly created line crosses an existing Centerline with address range data.  For splits that do not involve a new centerline I create a dummy centerline to create the split and then delete that dummy centerline.  You can modify a configuration file to use your own Centerline schema rather than the Local Government schema if you want to. Attribute Assistant can be modified to perform many other automations that you can configure for maintaining any of your data.  It can automatically perform autonumbering with unique values, look ups, validations, field calculations, spatial joins, etc. as you edit your features after you have created the correct rules in the DynamicValues table. ... View more

If you had ArcGIS 10.1 a standard join and an export of the data will multiply the rows to convert a one-to-many or many-to-many relationship to a one-to-one relationship.  That might make some charting options work for this problem.  Unfortunately, at 10.0 this is not an option and there is no tool within ArcGIS that does what the Pivot Table tool does for license levels lower than Advanced. Excel can transpose columns and rows if you follow this procedure.  That is your best bet. If you find that you actully need multiple rows like 10.1 can produce through a Join and Export, the Make Query Table tool is the best option for converting a one-to-many or many-to-many relationship to a one-to-one relationship at 10.0 or earlier. ... View more