topic Re: Adding data from two different layers into another layer, stormwater pipes feature IDs in ArcGIS Pro Questions

Adding data from two different layers into another layer, stormwater pipes feature IDs

MaureenSpiessl — Tue, 27 Feb 2024 15:43:48 GMT

Hello,

This is a going to be a long shot, but I am looking for advice or website material anything to help with an issue I have. This is a stormwater ArcGIS Pro issue.

The issue:

I need to create two new fields (Up_stream and Down_stream nodes) in our established inventory list of stormwater pipe infrastructure. The two new fields (which I have created) in our Pipe layer, needs to be populated to indicate which stormwater infrastructure the pipes are flowing to and from.
- EX. Inlet X (up steam flows to) ------Pipe-----> Manhole Y or Inlet Z (down stream flows from)
I need to find a way to populate into the two new fields the feature IDs for the To (downstream) and From (Upstream) stormwater infrastructures.

Research question:

Does anyone have any insight and how to tackle this situation or have ideas where I can look up tutorials etc.
Has anyone had experience with this? I’m sure we are not the only one out to encounter this type of issue.

Worst case I manually input the feature ID for each pipe segment, but I am sure there must be something to help jumpstart the process.

Thank you an advance 🙂

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

RobChasan — Wed, 28 Feb 2024 23:57:42 GMT

Is the Feature ID field you are referencing the built in non-editable field that comes automatically with shape file feature classes, aka ObjectID in GDBs? If it is, you shouldn't use it and you should define a separate field in the feature class.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

Bud — Fri, 01 Mar 2024 03:16:49 GMT

Can you provide a few screenshots? With labelled features?
And can you show us the attribute table? And manually populate a few values to indicate what you're looking for?
Does a tool like Spatial Join help you?
1. ~~Also, an idea implemented in 3.2: Spatial Join by Attribute (Match Fields)~~
Do you want to batch-populate the values one time, then manually update them going forward? Or do you want to also find a way to automatically populate the fields every time a feature is created or modified, using an attribute rule?
What version of ArcGIS Pro?
What kind of geodatabase? File geodatabase? Oracle enterprise geodatabase? Etc.
1. If it's an enterprise geodatabase, then what spatial type doe the feature classes use? ST_GEOMETRY?
Are the pipe features digitized in the direction of flow?
Do the inlet and manhole features have invert elevations? I.e., a field that indicates the elevation that the pipe connects to the inlet or manhole?
Is Up_stream always either an inlet or a manhole?
Is Down_stream always a manhole, never an inlet?
1. What about outfalls?
It would be relatively straightforward to determine what points (inlets and manholes) would be associated with a given pipe -- due to proximity. But it's not clear to me how you would know what inlet/manhole is to be the Up_stream value (in the pipe layer). And what inlet/manhole is to be the Down_stream value.

For example, for Pipe1, how do you know what manhole is upstream vs. downstream?

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

Bud — Fri, 01 Mar 2024 16:46:21 GMT

I asked a colleague, and they said they use the direction of the line to determine what manhole is upstream vs. downstream.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

MaureenSpiessl — Mon, 04 Mar 2024 14:20:46 GMT

Hello, thank you for your list of questions. I will answer them as best as I can. I will state that I am fairly new to this position as well as this Stormwater GIS analyst position is new. So I am dealing with a database that may or may not have been maintained well. I have a lot of data cleanup in my near future 😊

1) The screen shots of the application.

We do have arrows that are used to visually identify water flow of the pips.
The manholes, inlets, outfalls, etc. are all given a separate FeatureID separate from the ObjectID.
Visually speaking we can tell the direction of water flow, however we need the upstream and downstream fields filled in with infrastructure to which the pipes are flowing from and to.
- EX.
- We need to populate the two fields to fill in the data relating to either a manhole to manhole or manhole to inlet or inlet to outfall etc. Note: I manually added the data here to show what I am trying to do.
- The reason we need this data is for one of our camera truck vendors need these fields filled in for their work flow to begin.

2) Attribute table.

3) Spatial Join, I am unsure if that will be useful but I can look into it.

4) I would like to batch-populate all current fields and then manually update as we move forward and add new infrastructure. Although that might be nice to find a way to automatically populate the fields every time a feature is created or modified, using an attribute rule, but to start I just need to populate the fields so that our vendors can start work.

5) ArcGIS Pro 3.2.0

6) I am pretty sure it is just a file geodatabase.

7) Yes the pipe features are created to indicate the flow of water.

😎 Inlet and manhole invert elevations--- not really. That data is unfortunately not the best maintained. The feature layers for inlets and manholes do have some invert elevation data but I would not rely on it. The pipe feature layer does have the invert elevation data.

Inlet feature layer attribute table

Pipe feature layer attribute table

9) I believe so that the up_stream is always going to be either a manhole or an inlet.

10) Yes, once again I believe so. Some downstream may empty via a outfall or a ditch.

11) I hear ya, this is one of the biggest issues I am dealing with.

Your screen shot is what I need to accomplish and the only way I know which manhole or inlet is up or down stream is by visually looking at the map with the labels turned on and filling in the fields manually

12) I was unaware with the state of the data that I am working with, so even if I can figure out a way to populate the two new field with as much data as I can and then I can manually add the rest. There are over 900,000 pipe segments in this database and our elevation data is not the best kept. So even if I can get a portion of that filled in that would be great. I have a lot of work to do cleaning up years’ worth of data.

Thank you.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

MaureenSpiessl — Mon, 04 Mar 2024 14:23:58 GMT

That is what we generally use as well but we have vendor that requires the up stream and down stream fields filled in to complete work orders.

Our water and sewer department have these fields filled in but they are all manhole to manhole and it was easier for them to populate the fields then stormwater.

Thank you.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

RhettZufelt — Mon, 04 Mar 2024 17:14:49 GMT

I have done this in the past with using a searchcursor to get the SHAPE@ field. With the SHAPE@ object you can get firstpoint/lastpoint of the line. ( line[1].lastPoint.X,line[1].lastPoint.Y )

I then iterate through each line and grab the first point (from the line) and use that point to make a feature layer so that I can use it to Select by Location the appropriate manhole ad get it's ID. Then I use an update cursor to assign this MH id to the UpstreamMH field.

Then, Select by Location using the last point, grab that MH id and update cursor to the DownstreamMH field.

I would think you could do this fairly easy with Arcade as well using the paths of the line, but have not looked into it. The main benefit of using Arcade is that you could also use it to make an attribute rule that will populate these values automagically in the future when editing/adding new features.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

Bud — Tue, 05 Mar 2024 08:26:45 GMT

Similar to what @RhettZufelt mentioned:

If your pipes are digitized in the right direction (arrow symbol points the right way) then you could use the point (manhole, inlet, or outfall) that is closest to the start point of the line to populate the Up_stream field of the pipe. And use the point (manhole, inlet, or outfall) that is closest to the end point of the line to populate the Down_stream field of the pipe.

That should be possible using OOTB geoprocessing tools.

@MErikReedAugusta might find this interesting.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

MErikReedAugusta — Tue, 05 Mar 2024 15:19:33 GMT

Thanks for the tag, @Bud . I'm skimming this, now, and I'm pretty sure we have this exact setup, and I already wrote a Python script to fill in those Up/Down IDs. I'll drop it here in a bit, once I reformat it.

@MaureenSpiessl , do you also have any linking IDs on the Manholes? For example, on our Point feature, we have OutPipe and InPipe fields where I can drop an ID, in addition to the Up/Down ID fields on the Pipes. That's a bit more complicated, but it's also doable.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

RhettZufelt — Tue, 05 Mar 2024 15:28:48 GMT

To add a little more, my data has separate featureclass for mahnoles or cleanouts on the sewer, and manholes, cleanouts, inlets, outlets, discharge, etc. on the storm.

So, to make the script easier so I only have to intersect with one layer, I run an append operation that copies over ALL the respective point features to a single temporary featureclass with the feature ID.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

Bud — Tue, 05 Mar 2024 15:33:24 GMT

I was going to suggest the same thing! Glad I’m on the right track.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

RhettZufelt — Tue, 05 Mar 2024 15:46:38 GMT

It looks like this post shows how to do that with Arcade. Is for an attribute rule, but shows the syntax that could be modified run in field calculator (or use as a rule 🙂

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

MaureenSpiessl — Tue, 05 Mar 2024 16:08:16 GMT

Hello,

We do not have any linking IDs connecting the manhole or inlet with a pipe, but that does seem useful for the future. I believe before I joined the team here, they relied on visual maps etc. to see the connections of pipes to manholes and inlets.

Thank you.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

RhettZufelt — Tue, 05 Mar 2024 16:17:37 GMT

that code doesn't rely on any common field/IDs. It just intersects the beginning and end point of the line with the points (MH's) and transfers the MHID and ELEV values (you can just drop the elevation part) from the point to the respective end of the line.

This code is for an attribute rule to do exactly what you requested in the original post. You should be able to modify it to iterate through all the lines in calculate field tool.

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

MErikReedAugusta — Tue, 05 Mar 2024 23:10:59 GMT

I think we're talking slightly tangential things, here:

What linkage IDs are worth having
How to update linkage IDs for new features, as they're created
How to populate linkage IDs for existing features, right now

1. What linkage IDs are worth having

At minimum, I'd generally recommend what you already have: Upstream ID and Downstream ID for your Line Features (i.e., Pipes). For probably 90% of use-cases, this is probably all you need. You might also consider an Outlet ID from your Polygon Features, like a detention pond. That's not handled at all by the code I provided below, though.

In our organization, we've also experimented with a suite of "helper" fields on our Point Feature, because it helps some of our older-school field people when they're locating new Assets. The goal was to not make them digitize a line feature from the field, so we needed a place to put the most critical datapoints for a newly-discovered pipe, like Material & Diameter. That evolved outward into back-populating the ID of the Pipe into one of those "helper" fields, once it's created. Downside is that it massively bloats your Point Feature's schema. It's a headache that I don't recommend doing lightly.

2. How to update linkage IDs for new features, as they're created:

Go with the code @RhettZufelt linked elsewhere in this thread, and build it as an Attribute Rule. It might or might not need a little retooling, depending on what linkage IDs you end up using. At a glance, it will work pretty much as-is, if all you have is Upstream ID / Downstream ID for a line feature.

(Sidebar: Thanks for finding that, Rhett—that step was in my near future and it's nice to not have to reinvent the wheel!)

3. How to update linkage IDs for existing features, right now:

A few assumptions/caveats/notes on this code, before I get to it:

This assumes your line features are snapped to your point features.
- With the data we inherited in our organization, this wasn't originally the case.
- You should be able to automate the snapping with existing ArcGIS tools, if memory serves, but it's been a while. Just be careful of the effect on Point 2 concerning duplicates, below.
- You could also adapt the code to look for nodes within a certain radius, but that's quite a bit more memory-intensive, and we didn't need it to do so, so I also didn't write that in, here.
This assumes that you don't have any coincident duplicates in your point features.
- Be careful if you have to do snapping from Point 1, because you might end up with duplicates afterwards, depending on your snap tolerances.
- This tool could be modified relatively easily to handle duplicates if they exist, but I didn't write it this way, this time.
Lines 15-28 are your inputs, and they are not pre-populated.
- If you want to run this standalone, just replace None on lines 19-20 with the full path to the relevant Feature Class. Then replace None on lines 23-28 with the name of the relevant field, as a string.
- If you want to run this as a tool, replace None on all of the above lines with arcpy.GetParameterAsText(#), where # is the applicable index of the tool input for the tool you've created.
If you run this as a tool, it should automatically trigger progress bars. If you run it standalone, it'll have little/no reporting messages to provide you.
This is less-complex than what our organization uses, but slightly more-complex than what your OP demands.
- Our code covers a bunch of other related circumstances and is currently in a bit of a rebuild, anyway, so I essentially rewrote a heavily-trimmed down stub.
- I'm also pretty debug-happy with the stuff we actually run internally, because if/when something breaks, I like to know exactly where it broke and why, but I stripped all those steps out for the snippet below.
- While it's a little more complex than what your OP necessitates, in my eyes it's a slightly better platform to build from if you do end up expanding or elaborating it in the future.
I can pretty much guarantee that this isn't the most-efficient possible way for this to run.
- For one, I think the Spatial Join functions run more in C and/or in ArcGIS's back-end stuff, so I'm pretty sure they'd be more efficient. I didn't go that route with our code for a bunch of complicated reasons, though, and I decided not to rebuild it that way, here.
- Our dataset is about 50k each for points and lines, at the moment, and it runs quickly enough on that set for our purposes. Plus, "There's nothing so permanent as a 'temporary' solution."

See the SPOILER below for the code, and happy automating!

"""Stormwater Upstream/Downstream Locator Tool Author: MErikReedAugusta (ESRI Community) Python: 3.9 ArcGIS Pro: 3.1.3 Scans the provided Storm Structure feature to generate a list of nodes, then matches those nodes to the up/downstream ends of the provided Storm Pipe feature, and logs the relevant Storm Structure ID. """ # IMPORTS Standard Library # IMPORTS 3rd Party import arcpy ############################################################################################################################## # INPUTS ############################################################################################################################## # The Feature Classes you intend to target with this tool structure_Feature = None pipe_Feature = None # These are the ID Fields for each Asset ON THEIR RESPECTIVE FEATURE (This is their "self" ID) structure_IDField = None pipe_IDField = None # These are the Upstream/Downstream ID Fields on the Pipe Feature upstream_IDField = None downstream_IDField = None ############################################################################################################################## # CLASSES & EXCEPTIONS ############################################################################################################################## class NodeNotFoundError(KeyError): """Exception raised when a specified Node couldn't be found in the NodeMap.""" pass class DuplicateNodeError(KeyError): """Exception raised when a specified Node already exists in the NodeMap.""" pass class NodeObj(): """Defines a geographic point of significance for the geometry of an Asset in 2-dimensional space.""" def __init__(self, x, y): self.X = x self.Y = y self.ID_Structure = None class NodeMap(): """A searchable list-like database of all Nodes encountered.""" def __init__(self): self._data = [] ################################################################################################# # NodeMap Navigation ################################################################################################# def HasNode(self, nodeX, nodeY): """Returns True if the provided X/Y Coordinate is found in the NodeMap.""" # Get a shortened list of just the nodes that match your coordinates. matchingNodes = [node for node in self._data if node.X == nodeX and node.Y == nodeY] # If the list is 0, then those coordinates aren't here. If it's bigger, then they are. return len(matchingNodes) > 0 def GetNode(self, nodeX, nodeY): """Given a provided coordinate pair, this finds and returns the first matching NodeObj in the NodeMap.""" for node in self._data: if node.X == nodeX and node.Y == nodeY: return node # If you're still here, then no nodes have matched. Probably should've checked first! Have an Exception, instead. raise NodeNotFoundError() def AddNode(self, nodeObj): """Adds the provided Node Object to the NodeMap.""" if self.HasNode(nodeObj.X, nodeObj.Y): raise DuplicateNodeError() self._data.append(nodeObj) ################################################################################################# # Read/Write Assets ################################################################################################# def ReadPoints(self): """Reads the Point Feature Class provided above and converts it into Node Objects.""" # Sets up an ArcGIS Progressor, if you're running this in a Tool totalRows = int(arcpy.management.GetCount(structure_Feature)[0]) stepFrag = 10 ** (len(str(totalRows)) - 2) arcpy.SetProgressor('step', message='Populating Nodes from Structures...', min_range=0, max_range=totalRows, step_value=stepFrag, ) progCount = 0 # Iterates through the Structure Table, reading Nodes with arcpy.da.SearchCursor(structure_Feature, ['OID@', structure_IDField, 'SHAPE@']) as cursor: for oid, assetID, geo in cursor: # Update the Progressor as we go, if this is in a Tool progCount += 1 if progCount % stepFrag == 0: arcpy.SetProgressorPosition() arcpy.SetProgressorLabel(f'Reading Feature Class for Assets: {progCount: >{len(str(totalRows))}}/{totalRows}...') # Do the actual work of pulling the Node node = NodeObj(geo.firstPoint.X, geo.firstPoint.Y) node.ID_Structure = assetID self.AddNode(node) # Clean up the node object(s) for safety & paranoia, so nothing gets forwarded where it shouldn't del node def WriteLines(self): """Reads the endpoints of the Line Feature Class provided above and retrieves the specified Node, if applicable.""" # Sets up an ArcGIS Progressor, if you're running this in a Tool totalRows = int(arcpy.management.GetCount(pipe_Feature)[0]) stepFrag = 10 ** (len(str(totalRows)) - 2) arcpy.SetProgressor('step', message='Populating Nodes from Structures...', min_range=0, max_range=totalRows, step_value=stepFrag, ) progCount = 0 # Iterates through the Structure Table, reading Nodes with arcpy.da.UpdateCursor(pipe_Feature, ['OID@', pipe_IDField, upstream_IDField, downstream_IDField, 'SHAPE@']) as cursor: for oid, assetID, frID, toID, geo in cursor: # Update the Progressor as we go, if this is in a Tool progCount += 1 if progCount % stepFrag == 0: arcpy.SetProgressorPosition() arcpy.SetProgressorLabel(f'Reading Feature Class for Assets: {progCount: >{len(str(totalRows))}}/{totalRows}...') # Look for the Upstream Node and read the ID there; write a sentinel value if it fails try: nodeObj_UP = self.GetNode(geo.firstPoint.X, geo.firstPoint.Y) frID = nodeObj_UP.ID_Structure except NodeNotFoundError(): frID = 'ID_Not_Found' # Look for the Downstream Node and read the ID there; write a sentinel value if it fails try: nodeObj_DN = self.GetNode(geo.lastPoint.X, geo.lastPoint.Y) toID = nodeObj_DN.ID_Structure except NodeNotFoundError(): toID = 'ID_Not_Found' cursor.updateRow((oid, assetID, frID, toID, geo)) # Clean up the node object(s) for safety & paranoia, so nothing gets forwarded where it shouldn't del nodeObj_UP, nodeObj_DN ############################################################################################################################## # MAIN ############################################################################################################################## def Main(): arcpy.AddMessage('Begin.') nodeMap = NodeMap() arcpy.AddMessage('Populating Nodes from Structures...') nodeMap.ReadPoints() arcpy.AddMessage('Updating Linkage Fields in Pipes...') nodeMap.WriteLines() if __name__ == '__main__': Main()

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

Bud — Sat, 16 Mar 2024 03:15:32 GMT

Re: Adding data from two different layers into another layer, stormwater pipes feature IDs

Bud — Thu, 18 Apr 2024 17:06:53 GMT