Adding data from two different layers into another layer, stormwater pipes feature IDs

[MaureenSpiessl]

Hello,

This is a going to be a long shot, but I am looking for advice or website material anything to help with an issue I have.  This is a stormwater ArcGIS Pro issue.

The issue:

• I need to create two new fields (Up_stream and Down_stream nodes) in our established inventory list of stormwater pipe infrastructure. The two new fields (which I have created) in our Pipe layer,  needs to be populated to indicate which stormwater infrastructure the pipes are flowing to and from.
  • EX. Inlet X (up steam flows to) ------Pipe-----> Manhole Y or Inlet Z (down stream flows from)
• I need to find a way to populate into the two new fields the feature IDs for the To (downstream) and From (Upstream) stormwater infrastructures.

Research question:

• Does anyone have any insight and how to tackle this situation or have ideas where I can look up tutorials etc.   
• Has anyone had experience with this? I’m sure we are not the only one out to encounter this type of issue.

Worst case I manually input the feature ID for each pipe segment, but I am sure there must be something to help jumpstart the process.

Thank you an advance 🙂 

[MErikReedAugusta]

I think we're talking slightly tangential things, here:

1. What linkage IDs are worth having
2. How to update linkage IDs for new features, as they're created
3. How to populate linkage IDs for existing features, right now

1. What linkage IDs are worth having

At minimum, I'd generally recommend what you already have: Upstream ID and Downstream ID for your Line Features (i.e., Pipes).  For probably 90% of use-cases, this is probably all you need.  You might also consider an Outlet ID from your Polygon Features, like a detention pond.  That's not handled at all by the code I provided below, though.

In our organization, we've also experimented with a suite of "helper" fields on our Point Feature, because it helps some of our older-school field people when they're locating new Assets.  The goal was to not make them digitize a line feature from the field, so we needed a place to put the most critical datapoints for a newly-discovered pipe, like Material & Diameter.  That evolved outward into back-populating the ID of the Pipe into one of those "helper" fields, once it's created.  Downside is that it massively bloats your Point Feature's schema.  It's a headache that I don't recommend doing lightly.

2. How to update linkage IDs for new features, as they're created:

Go with the code @RhettZufelt linked elsewhere in this thread, and build it as an Attribute Rule.  It might or might not need a little retooling, depending on what linkage IDs you end up using.  At a glance, it will work pretty much as-is, if all you have is Upstream ID / Downstream ID for a line feature.

(Sidebar: Thanks for finding that, Rhett—that step was in my near future and it's nice to not have to reinvent the wheel!)

3. How to update linkage IDs for existing features, right now:

A few assumptions/caveats/notes on this code, before I get to it:

1. This assumes your line features are snapped to your point features.
   • With the data we inherited in our organization, this wasn't originally the case.
   • You should be able to automate the snapping with existing ArcGIS tools, if memory serves, but it's been a while.  Just be careful of the effect on Point 2 concerning duplicates, below.
   • You could also adapt the code to look for nodes within a certain radius, but that's quite a bit more memory-intensive, and we didn't need it to do so, so I also didn't write that in, here.
2. This assumes that you don't have any coincident duplicates in your point features.
   • Be careful if you have to do snapping from Point 1, because you might end up with duplicates afterwards, depending on your snap tolerances.
   • This tool could be modified relatively easily to handle duplicates if they exist, but I didn't write it this way, this time.
3. Lines 15-28 are your inputs, and they are not pre-populated.
   • If you want to run this standalone, just replace None on lines 19-20 with the full path to the relevant Feature Class.  Then replace None on lines 23-28 with the name of the relevant field, as a string.
   • If you want to run this as a tool, replace None on all of the above lines with arcpy.GetParameterAsText(#), where # is the applicable index of the tool input for the tool you've created.
4. If you run this as a tool, it should automatically trigger progress bars.  If you run it standalone, it'll have little/no reporting messages to provide you.
5. This is less-complex than what our organization uses, but slightly more-complex than what your OP demands.
   • Our code covers a bunch of other related circumstances and is currently in a bit of a rebuild, anyway, so I essentially rewrote a heavily-trimmed down stub.
   • I'm also pretty debug-happy with the stuff we actually run internally, because if/when something breaks, I like to know exactly where it broke and why, but I stripped all those steps out for the snippet below.
   • While it's a little more complex than what your OP necessitates, in my eyes it's a slightly better platform to build from if you do end up expanding or elaborating it in the future.
6. I can pretty much guarantee that this isn't the most-efficient possible way for this to run.
   • For one, I think the Spatial Join functions run more in C and/or in ArcGIS's back-end stuff, so I'm pretty sure they'd be more efficient.  I didn't go that route with our code for a bunch of complicated reasons, though, and I decided not to rebuild it that way, here.
   • Our dataset is about 50k each for points and lines, at the moment, and it runs quickly enough on that set for our purposes.  Plus, "There's nothing so permanent as a 'temporary' solution."

See the SPOILER below for the code, and happy automating!

Spoiler

"""Stormwater Upstream/Downstream Locator Tool
    Author: MErikReedAugusta (ESRI Community)
    Python: 3.9
    ArcGIS Pro: 3.1.3

    Scans the provided Storm Structure feature to generate a list of nodes, then matches those
    nodes to the up/downstream ends of the provided Storm Pipe feature, and logs the relevant
    Storm Structure ID.
"""
# IMPORTS Standard Library

# IMPORTS 3rd Party
import arcpy

##############################################################################################################################
# INPUTS
##############################################################################################################################
# The Feature Classes you intend to target with this tool
structure_Feature = None
pipe_Feature      = None

# These are the ID Fields for each Asset ON THEIR RESPECTIVE FEATURE (This is their "self" ID)
structure_IDField = None
pipe_IDField      = None

# These are the Upstream/Downstream ID Fields on the Pipe Feature
upstream_IDField    = None
downstream_IDField  = None

##############################################################################################################################
# CLASSES & EXCEPTIONS
##############################################################################################################################
class NodeNotFoundError(KeyError):
  """Exception raised when a specified Node couldn't be found in the NodeMap."""
  pass


class DuplicateNodeError(KeyError):
  """Exception raised when a specified Node already exists in the NodeMap."""
  pass


class NodeObj():
  """Defines a geographic point of significance for the geometry of an Asset in 2-dimensional space."""
  def __init__(self, x, y):
    self.X = x
    self.Y = y

    self.ID_Structure = None


class NodeMap():
  """A searchable list-like database of all Nodes encountered."""
  def __init__(self):
    self._data = []

  #################################################################################################
  # NodeMap Navigation
  #################################################################################################
  def HasNode(self, nodeX, nodeY):
    """Returns True if the provided X/Y Coordinate is found in the NodeMap."""
    # Get a shortened list of just the nodes that match your coordinates.
    matchingNodes = [node for node in self._data if node.X == nodeX and node.Y == nodeY]

    # If the list is 0, then those coordinates aren't here.  If it's bigger, then they are.
    return len(matchingNodes) > 0

  def GetNode(self, nodeX, nodeY):
    """Given a provided coordinate pair, this finds and returns the first matching NodeObj in the NodeMap."""
    for node in self._data:
      if node.X == nodeX and node.Y == nodeY:
        return node

    # If you're still here, then no nodes have matched.  Probably should've checked first!  Have an Exception, instead.
    raise NodeNotFoundError()

  def AddNode(self, nodeObj):
    """Adds the provided Node Object to the NodeMap."""
    if self.HasNode(nodeObj.X, nodeObj.Y):
      raise DuplicateNodeError()

    self._data.append(nodeObj)

  #################################################################################################
  # Read/Write Assets
  #################################################################################################
  def ReadPoints(self):
    """Reads the Point Feature Class provided above and converts it into Node Objects."""
    # Sets up an ArcGIS Progressor, if you're running this in a Tool
    totalRows = int(arcpy.management.GetCount(structure_Feature)[0])
    stepFrag = 10 ** (len(str(totalRows)) - 2)
    arcpy.SetProgressor('step',
                        message='Populating Nodes from Structures...',
                        min_range=0,
                        max_range=totalRows,
                        step_value=stepFrag,
                       )
    progCount = 0

    # Iterates through the Structure Table, reading Nodes
    with arcpy.da.SearchCursor(structure_Feature, ['OID@', structure_IDField, 'SHAPE@']) as cursor:
      for oid, assetID, geo in cursor:
        # Update the Progressor as we go, if this is in a Tool
        progCount += 1
        if progCount % stepFrag == 0:
          arcpy.SetProgressorPosition()
          arcpy.SetProgressorLabel(f'Reading Feature Class for Assets: {progCount: >{len(str(totalRows))}}/{totalRows}...')

        # Do the actual work of pulling the Node
        node = NodeObj(geo.firstPoint.X, geo.firstPoint.Y)
        node.ID_Structure = assetID

        self.AddNode(node)
        # Clean up the node object(s) for safety & paranoia, so nothing gets forwarded where it shouldn't
        del node

  def WriteLines(self):
    """Reads the endpoints of the Line Feature Class provided above and retrieves the specified Node, if applicable."""
    # Sets up an ArcGIS Progressor, if you're running this in a Tool
    totalRows = int(arcpy.management.GetCount(pipe_Feature)[0])
    stepFrag = 10 ** (len(str(totalRows)) - 2)
    arcpy.SetProgressor('step',
                        message='Populating Nodes from Structures...',
                        min_range=0,
                        max_range=totalRows,
                        step_value=stepFrag,
                       )
    progCount = 0

    # Iterates through the Structure Table, reading Nodes
    with arcpy.da.UpdateCursor(pipe_Feature, ['OID@', pipe_IDField, upstream_IDField, downstream_IDField, 'SHAPE@']) as cursor:
      for oid, assetID, frID, toID, geo in cursor:
        # Update the Progressor as we go, if this is in a Tool
        progCount += 1
        if progCount % stepFrag == 0:
          arcpy.SetProgressorPosition()
          arcpy.SetProgressorLabel(f'Reading Feature Class for Assets: {progCount: >{len(str(totalRows))}}/{totalRows}...')


        # Look for the Upstream Node and read the ID there; write a sentinel value if it fails
        try:
          nodeObj_UP = self.GetNode(geo.firstPoint.X, geo.firstPoint.Y)
          frID = nodeObj_UP.ID_Structure
        except NodeNotFoundError():
          frID = 'ID_Not_Found'

        # Look for the Downstream Node and read the ID there; write a sentinel value if it fails
        try:
          nodeObj_DN = self.GetNode(geo.lastPoint.X, geo.lastPoint.Y)
          toID = nodeObj_DN.ID_Structure
        except NodeNotFoundError():
          toID = 'ID_Not_Found'

        cursor.updateRow((oid, assetID, frID, toID, geo))
        # Clean up the node object(s) for safety & paranoia, so nothing gets forwarded where it shouldn't
        del nodeObj_UP, nodeObj_DN


##############################################################################################################################
# MAIN
##############################################################################################################################
def Main():
  arcpy.AddMessage('Begin.')
  nodeMap = NodeMap()

  arcpy.AddMessage('Populating Nodes from Structures...')
  nodeMap.ReadPoints()

  arcpy.AddMessage('Updating Linkage Fields in Pipes...')
  nodeMap.WriteLines()

if __name__ == '__main__':
  Main()

[RhettZufelt]

I have done this in the past with using a searchcursor to get the SHAPE@ field.  With the SHAPE@ object you can get firstpoint/lastpoint of the line.  ( line[1].lastPoint.X,line[1].lastPoint.Y )

I then iterate through each line and grab the first point (from the line) and use that point to make a feature layer so that I can use it to Select by Location the appropriate manhole ad get it's ID.  Then I use an update cursor to assign this MH id to the UpstreamMH field.

Then, Select by Location using the last point, grab that MH id and update cursor to the DownstreamMH field.

I would think you could do this fairly easy with Arcade as well using the paths of the line, but have not looked into it.  The main benefit of using Arcade is that you could also use it to make an attribute rule that will populate these values automagically in the future when editing/adding new features.

R_

[RhettZufelt]

To add a little more, my data has separate featureclass for mahnoles or cleanouts on the sewer, and manholes, cleanouts, inlets, outlets, discharge, etc. on the storm.

So, to make the script easier so I only have to intersect with one layer, I run an append operation that copies over ALL the respective point features to a single temporary featureclass with the feature ID. 

R_

[Bud]

I was going to suggest the same thing! Glad I’m on the right track.

[RhettZufelt]

It looks like this post shows how to do that with Arcade.  Is for an attribute rule, but shows the syntax that could be modified run in field calculator (or use as a rule 🙂

R_

[Bud]

Possibly related: Calculate From and To Endpoint IDs (from one of several point feature classes)

[Bud]

Related: Pull data from sewer line feature to manhole point feature