Latest Contributions by RichardFairhurst

@LydiaRill I offer this blog and the transformations it can bring anyone's work flow completely free of charge.  Our time is the most valuable thing we have and I'm just glad you took the time to share with me Your gratitude.  With the time this has Blog has helped you regain, please share it with others that may benefit. If you ever go to the Esri UC conference (when it's no longer just virtual) it would be great to meet you.  I'd love hear more about what this Blog has helped you to do and perhaps share more of what I've learned from the numerous ways I've used this technique in the years since I wrote this Blog.  The conference is always such a great place for all of us to draw inspiration from others who love GIS. ... View more

As proof of the relevance of my comments, see this post: attribute-transfer-mapping-in-spatial-adjustment  The esri community is the primary source of help on the real world requirements of the Attribute Transfer tool.  That should be the job of the tool designers and the help authors. ... View more

Both layers have to be Selectable under the List By Selection button at the top of the Contents pane for the tool to work. ... View more

Edit:  I originally wrote that I withdrew my comments, but I am reversing that comment.  I see this is about the Attribute Transfer Setup in Pro.  I know my comments are relevant to ArcMap Desktop, and to the extent Pro behaves the same my comments apply. The tool won't work unless both layers are selectable in Desktop and the same is true in Pro.  At least the Desktop help explains this relationship between the selectable state of the layers and the tool, but I just read the Pro help and there is no mention about it https://pro.arcgis.com/en/pro-app/latest/help/editing/transfer-attributes-between-features.htm.  Since the tool simply won't work in Pro if either of the layers mapped are not selectable, the omission of this information in the Pro help for this tool is astounding to me. The restriction to only using selectable layers is frustrating to me, but I understand the behavior and can troubleshoot it.  However, no one I have trained understands it and they just think the tool doesn't work as advertised and give up on it when they try to transfer feature attributes and the tool randomly (to them) works or doesn't work when they click on the source or target feature.  At no point did the set up or the use of the tool explain that their layer selection settings are central to the way this tool behaves, and unless I am there to fix it for them, they give up. At minimum an additional comment at the bottom of the setup screen should be included that reads something like: "The Layers mapped should be selectable under the List By Selection Contents button"  This would give my users a clue about why the tool doesn't work, since if they do try to do any troubleshooting, they usually check their field mapping settings and would see this essential comment about the tool's behavior and where they should look to fix it. ... View more

I reexamined your methodology and think there my be an alternative explanation of this behavior that still involves the difference in the behavior of a layer attribute join and the export of that join.  The Attribute Join of a layer in Arcmap is in reality a one-to-one join that only considers the first record in the one-to-many relationship when it applies the definition query.  So if the first record it encounters does not meet your query criteria the record is filtered out, even if the second record of the join would have met your criteria.  However, an export examines all records in the joined table and will include matches that fail to meet your criteria on the first record, but succeed on the second record or any other records than the first.  The export may also create duplicates of the Census Blocks if more than one record in the join table meets your query definition (if the inputs of the join came from the same file geodatabase this duplication of records would definitely occur).  Both methods can produce incorrect results for what you want to achieve, which makes the use of ?-to many joins unreliable and unsuitable for this kind of analysis. The proper way to do this analysis to ensure all records that meet your criteria are included and no duplication happens is to do the following steps: Create a definition query in the broadband shapefile to show only Census blocks at 25/3 or greater. This reduces the size of the attribute table from 982,015 records to 654,582 records.  Perform a Summary on the Census Blocks field of that shapefile to get a table that contains only one record for each unique Census Block value.  You could include the first and last or min and max values in the 24/3 fields in the output, but that is optional. Then perform an attribute join of the Census Block shapefile as the target table to the Summary output as the join table, keeping only joined records.  Both the Attribute Join and the export will show the same number of records that the Summary table contains (provided all of the Census Block shapefile features are in fact unique) since both will be the output of a true one-to-one join.  This methodology will reliably and consistantly give the sum of households for all Census Blocks that actually meet your criteria.  MS Access would also require you to do the same step of creating a Summary query first, and then using that summary output for creating a one-to-one relationship with the Census Blocks in order to get the sum of households you want, so this is the standard  database methodology for solving this problem.  MS Access is incapable of providing you with the sum of households you want directly from any ?-to-many relationship. ... View more

  Your Census Block feature are not unique in the shapefile or the dbf table that you say has only one feature/record per Census Block value.  The only way I will accept that you have proven me wrong is if you do the Dissolve I have shown you or you do a Summary of the Census Blocks and find that the count of every Census Block value is 1.  If you refuse to do that your on your own to figure this out.  But I can assure you that the problem is not with the software, it is with your unproven assumptions about your data. A difference in the presence or absence of an FID field in the shapefile and table is another factor that can affect the export behavior, but it would be unusual for either if those file types to lack an FID field. ... View more

Use the Dissolve tool (Data Management Tools, Generalization Toolbox) on the shapefile that is only supposed to have one shape for each Census Block.  Only use the Census Block field as a Dissolve field in the tool and add the Census Block field as a Statistics field with a Statistics Type of Count   The output will have a single shape for each Census Block and export the results you are are wanting.  Anything in the output Count field with a value greater that 1 was split into more than one piece in your original Census Block shapefile.  Below is how I would set up the tool for my Census Block feature class and the output with all Census Blocks that collapsed from many features for those Census Block values to a single feature.   ... View more

@SamMontoia1Your suggestion doesn't help very much in my opinion.  It would show me what won't work, but only if I take action to use the new options you are adding.  I most likely won't ever use those options, because, as I mentioned, I rarely consider the selectable state of my layers while I setup a transfer.  Your proposal continues to make me responsible to think about the selectable state of my layers and that responsibility should not fall on me, it should fall on the designers of the tool.  Additionally, I don't want the tool to show me that it currently won't allow me to do what I want, I want the tool to assist me with getting to the point where it will do what I want. My goals is not to prevent myself from setting up the transfer I want.  My goal is to make the transfer I set up actually work with as little thought about the selectable state of my layers as possible.  I want it to be the tool's responsibility to think about and help me correctly setup the selectable state of my layers for every transfer I set up by the time I have exited the tool, regardless of whatever selectable state my layers were in before I entered the tool.. ... View more

As a frequent user of the Attribute Transfer tool I know that the tool only operates when the source and the target layers are both selectable.  More often than not, I fail to think about the selectable state of my layers when I set up the Attribute Transfer Mapping and use an unselectable layer that will disable the tool until I change that layer to selectable,  There is no message that ever appears saying that either layer is unselectable at any point, and the user is completely on their own to figure out why the tool won't work when they try it and troubleshoot it. It would be helpful if the Attribute Transfer Mapping tool at least gave a warning when I close the tool that one or more of the layers set up for the transfer is unselectable so that I would immediately make the connection between the tool and the cause of it not working.  Ideally the warning would also list the unselectable layers that have been used in my setups.  Even better, if the message would give me the option to make all of the layers set up for transfer selectable, I would almost always use that option.  I don't really want the proposal offered by this idea, because putting the selectable state of my layers as the control of the transfer setups I can do is backwards to me.  To me that just moves the confusion about why I can't setup the tool to do what I want inside the tool and makes me have to exit the tool to change my selectable layers before I can even create a setup with the Attribute Transfer tool.  When I use the Attribute Transfer setup I want to be able to set up any transfer that I want and be able to expect it to work immediately upon exiting the tool or warned that the tool won't work until I fix the selectable state of the layers I set up. ... View more

Attribute Joins do not display the full set of records that actually result from a one-to-many or many-to-many join.  The table shown by ArcMap is functionally a one-to-one or many-to-one set of records, meaning that new records are not created in the parent table to match all of the records in the joined table.  For all intents and purposes it is hiding from you the true number of denormalized records that would result from the full output of that relational join for performance reasons.  However, when you do an export the true denormalized record set of the one-to-many or many-to-many records is generated, which inserts records that you would not see in a layer attribute join.  This is a desired behavior, since this is the best way to make ArcMap behave like Access when outputting a result from a one-to-many or many-to-many relationship.  The export increases the number of records to convert these relationships to a true one-to-one output.  Creating a one-to-one representation of the data in the ?-to-many relationships is often useful for a variety of analysis purposes, and I have made use of this export behavior many times. The difference in your results is exposing an erroneous assumption on your part that the Census Block shape file has only unique values for each Census Block field value.  If you do a Summary of your Census Block field in the Census Block table you almost certainly will discover their are duplicate values for one or more of your Census Blocks.  If even one Census Block value is duplicated in the shape file that should only have unique values it will double the number of records in the export for that value, and if it is duplicated on 3 features it will triple the values in the output of the export for that value.  So if the shapefile that allows duplicate values has 100,000 features with that Census Block value you will end up with 100,000 more records than you expect when you do an export if that value is duplicated and not actually unique.  You need to clean up your features to merge together all parts of the Census Block into a single multipart feature to actually have a many-to-one relationship like the Attribute Join shows you.  The Dissolve tool can do the merge and output a new feature class or you can Edit the existing FC by using the Merge option under the Editor button when you have selected all features for a single Census Block value.  That will make that shapefile conform to your relationship assumption and it will behave the same as the Attribute Join when you do an export, since then it will create a many-to-one join and not a many-to-many join with denormalized records hidden by the Attribute Join. ... View more

Using Embedded cursors is not a good idea for any data that will continue to grow into more than 1,000 records that match.  Each cursor reopens the table on disk and processes all records in the table during each loop to isolate the records specified by the query filter, so the Embedded cursors result in an exponential performance hit as the record sets grows.  Using an attribute index on all filter fields gives a performance boost, but that boost gets overwhelmed when the number of unique values it needs to filter grows beyond a certain point. Dictionaries give a huge performance enhancement over query filters when there are a huge number of unique values to access.  Dictionaries are random access with no real performance delay for accessing the 1st record or the 1 millionth record.  The Maximum time it has taken me to load 40 fields from 1 million records into a dictionary (including the shape field) is 10 minutes and that allows me to match them to 1 million records in an update table in a single pass.  While I haven't done this in my blog, I frequently gain another performance enhancement by adding a python expression that compares the 40 field values so I only write updates to the actual records that changed (typically 1,000 records or less).  With that enhancement I can complete the updates in under another 10 minutes.  The comparison expression is up to 20 times faster than a cursor writing to a record that isn't actually changing, especially in large tables.  The base code in my blog and the code I provided in my latest response can be revised so that it dramatically speeds up again if your record set is very large and the number of records actually getting updated is small by only writing to records that actually changed.  It only takes a revision of one or two lines of code so I will go back and add that. ... View more

@ChaseSchieffer, The last example in my blog covering using a dictionary to replace a summary table applies to your situation, with a little adaptation to your specific needs.  If your main feature class had a LatestStatusDate field you could also update that to go along with your LatestStatus value for the record.   November 12, 2021:  I enhanced the code performance by adding an if condition that only updates the master table record when the related table date in the dictionary is the more recent than the one in the LatestStatusDate field or the status value of the dictionary is different from the one held in the LatestStatus field.  This gives a significant performance boost if you have two large sets of records to process and match, but in reality only a few records actually need to be updated due to value changes.     from time import strftime print( "Start script: " + strftime("%Y-%m-%d %H:%M:%S")) import arcpy sourceFC = r"C:\Path\RelateTable" sourceFieldsList = ["HydrantID", "CreateDate", "LastEditDate", "Status"] # Build a summary dictionary from a da SearchCursor with unique key values # of a field storing a list of the latest date and status. valueDict = {} with arcpy.da.SearchCursor(sourceFC, sourceFieldsList) as searchRows: for searchRow in searchRows: keyValue = searchRow[0] if not keyValue in valueDict: # assign a new keyValue entry to the dictionary with the #latest date of the Created or Last Edit Date field. lastDate = searchRow[1] if searchRow[2] > lastDate: lastDate = searchRow[2] valueDict[keyValue] = [lastDate, searchRow[3]] else: # update the date and status of an existing keyValue entry in # the dictionary if the current record's # Created or Last Edit Date is the Latest date. newDate = searchRow[1] if newDate < searchRow[2]: newDate = searchRow[2] if valueDict[keyValue][0] < newDate: valueDict[keyValue] = [newDate, searchRow[3]] updateFC = r"C:\Path\UpdateFeatureClass" updateFieldsList = ["FacilityID", "LatestStatusDate", "LatestStatus"] with arcpy.da.UpdateCursor(updateFC, updateFieldsList) as updateRows: for updateRow in updateRows: # store the Join value of the row being updated in a keyValue variable keyValue = updateRow[0] # verify that the keyValue is in the Dictionary if keyValue in valueDict: if updateRow[1] < valueDict[keyValue][0] or updateRow[2] != valueDict[keyValue][1]: # Transfer the data updateRow[1] = valueDict[keyValue][0] # Latest Status Date updateRow[2] = valueDict[keyValue][1] # Latest Status updateRows.updateRow(updateRow) del valueDict print( "Finished script: " + strftime("%Y-%m-%d %H:%M:%S"))     ... View more

I am using ArcGIS Pro 2.7, and out of the box it behaves the way I described, but it does have the Option Angular Units setting so I tried that.  That resolved the issue.  But I am glad to hear that ArcGIS Pro 2.8 has fully fixed this issue for those like me who didn't think to look at the Angular Units Option.  I tried installing 2.8 yesterday, but it failed.  I will contact Esri support to get that resolved.   I look forward to the version where the default line symbol can be set for the initial course of each new traverse, since my Idea for that is in the product plan. ... View more

No.  I want to choose the template before ever creating any traverse and before the drop down is available.  I am currently having to use the drop down every time for thousands of new traverses that only have one course.  My traverses always start with the top item in my domain and ignore the field default value. If I am creating side-yard traverses. I want a way to set the starting template for the traverse tool to use the template below once and have it stay that way as long as I am working on side-yards.  It should take me a total of 1 mouse move and 3 mouse clicks to do this.  Then I may do 300 separate traverses in a row with this template for an average tract.  Currently to do that I have to add 1,495 more mouse moves and clicks using the drop down for 299 of the side yard traverses.  I want to avoid this wasteful and tedious set of steps that almost doubles the time I spend working on these traverses. If I am creating centerline traverses. I want a way to set the starting template for the traverse tool to use the template below once and have it stay that way as long as I am working on centerlines.  It should take me a total of 1 mouse move and 3 mouse clicks to do this.  Then I may do 40 separate traverses in a row with this template for an average tract.  Currently to do that I have to add 195 more mouse moves and clicks using the drop down for 39 of the Centerline traverses.  I want to avoid this tedious set of steps that almost doubles the time I spend working on these traverses. A potential way to solve this is by adding another drop down to the main body of the tool under the Layer drop down for the Initial Traverse Template.  The picture below shows what the tool should look like immediately after pressing the Set Starting Location button and choosing a position on the map.  This would make the Traverse tool act like the Create Features tool, where I only have to choose an editing template once to create as many features as I want with that template until I change the template in the Create Features tool window.   ... View more