Trace Network - how to select subnetworks based on feature count

[JasperRomero]

Hello folks,

I have some beginner questions about what is possible using a Trace Network:

I have created a Trace Network with storm drain nodes (points) as junctions and storm drain pipes (lines) as edges. Because of isolated components of the storm drain infrastructure, there are numerous small "subnetworks" (preferred term?) in the layer - e.g. a single pipe with a node at each end representing a culvert passing under a road, disconnected from anything else in the network.

Is it possible to interact with/select these subnetworks, and/or their constituent underlying features, based on criteria like number of junctions or edges per subnetwork? E.g., I want to be able to select the line for every subnetwork with only one or two pipes (so only two or three nodes) and output those lines to a new feature class.

I have found the "aggregated_lines/points" and "result_type" parameters for the Trace GP tool, but haven't found examples of how to perform the selection/isolate specific subnetworks based on criteria like "made up of </= 2 edge features".

I am a, let's say, novice intermediate arcpy scripter, so happy for any suggestions that involve scripts, but grateful for specific code examples to illustrate the concepts at work. Alternatively, if this can be done purely using GP tools I haven't figured out yet, that's great too!

Thanks in advance for any suggestions!

[RobertKrisher · Accepted solution]

If you consider the main part of your stormwater system to be 'connected', then there are several ways you could describe these features. You could say they are disconnected (from the main system), they are islands of connectivity, or they are disjoint sets of features.

The term subnetwork is used by the utility network to mean something very specific; a subnetwork is the collection of features that are traversable to a subnetwork controller. In the case of stormwater systems, its every feature that is traversable to an outfall or outlet.

To identify these features in a trace network, you could set starting points on all your outfalls/terminal discharge locations, then run an upstream trace (Trace (Trace Network)—ArcGIS Pro | Documentation). Switch the selection set on all your network layers (Select Layer By Attribute (Data Management)—ArcGIS Pro | Documentation with the SWITCH_SELECTION option). This would give you the collection of all the features that have no outlet, and is a good place to start for a reasonable amount of effort.

If you wanted to find all the distinct islands, you could do a connected trace for every line (or device or junction) in the network, output the JSON, and parse the elements to see how many features are connected to it. As you iterate through the features and results, keep track of which features you've already discovered so you don't trace the same feature more than once.

[JasperRomero]

Thanks for your reply, @RobertKrisher - and for sharing some clearer terminology.

Am I understanding correctly that I could accurately call my islands of connectivity subnetworks, in every case where they have an outlet (e.g., each individual culvert with one inlet, one pipe, and one outlet is its own subnetwork because it is traversable to an outlet feature)? As I am working only with Trace Networks (not Utility Networks - maybe this was the wrong community for this post, but I could not find a Trace Network-specific community), I am not sure at the moment if the idea of a subnetwork controller applies to my scenario.

Your response also helped me to understand that there is so much more about trace networks I don't understand yet. Briefly:

1) When applying GP tools to a trace network, what do the tools treat as a discrete feature? Are they operating on each disjoint unit of connectivity, or on each individual feature (points/lines) that comprise the disjoin units?

2) When setting starting points for traces - are these defined by locations, or by features? e.g., if I have a distinct layer of points that are spatially coincident with my outlet nodes in the network, can I pass that layer as my starting points for trace operations or do I need to actively select the points in the network itself?

3) I have worked solely with arcpy/python, but have heard about JSON enough that I probably need to start learning how to use it. Beyond what comes up in a quick search through Esri documentation, are there any self-teaching resources for using JSON with GIS that you could recommend?

Thanks again - I know this thread is getting disparate, please let me know if there is a better place/way to discuss these topics.

[RobertKrisher]

I would avoid using the word subnetwork to describe your situation with trace networks, because it is a very specific word used to describe a feature of utility networks so it will often lead others to believe you are using a stormwater utility network. If you were using a stormwater utility network, these situations would be easier to discover but you would also have to adopt that data model.

1 - A starting location can be placed on each point feature, or at any point along a line.

2 - Look at the schema of the starting points layer to answer this question. You'll need to uniquely identify the feature within the network, and if it is a line feature you will need to say what percentage along the length of the line to start at. You must pass in the trace locations layer, which has a very specific schema. The utility network has a GP tool call "Add Trace Locations" that makes this process easier, but unfortunately there is no corresponding tool for trace networks.

3 - I wrote an article about extracting JSON files using trace and parsing JSON files for the utility network. It's not a 1:1 for trace networks, but many of the same concepts apply. The parsing article includes some code samples to get you started (for the utility network), but again you should be able to adapt them.

The community site is usually the best place to have these discussions, if we get too far off-topic we can create a new thread.

[JasperRomero]

Thanks again for the prompt and helpful response - will give those articles a read!

I am able to use a feature layer of points as the input starting point layer, and I am starting to better grasp how the Trace (Trace Network) tool works. One remaining question I have there: 

Is there some way to configure the Functions parameter with the Count function to limit the output to features comprising disjoint connected units with fewer than some threshold number of features? Or does that Function parameter only have the ability to operate on actual attribute values, rather than a count of features traversed? I have tried the Count function for this without success, so I'm wondering if I may not understand how to apply it correctly - otherwise, I will dive into the JSON suggestion you outlined.

Thanks again!

[RobertKrisher]

A function barrier determines the criteria for when the trace should stop. It is as you've described, you can stop tracing when you've traversed 10 features.

For what you're trying to do you'll need to either use selections or parse the JSON results to answer your questions. Brute forcing the problem will be quite slow (24 hours to analyze a network of 100k features), but with some clever tricks you can get the process down to being <30 minutes for a similarly sized dataset.

How much time you invest in creating a solution to the problem depends on how often you need to answer this question. Rhetorical question time: Is this a one-off thing you're doing now? or part of a larger analysis/quality assurance/regulatory requirement?

[JasperRomero]

Thanks for bearing with me - 

I had initially been looking at the Functions parameter, rather than Function Barrier, but I think Function Barrier is closer to what I am looking for. However, I am still not quite understanding what I need to do differently to get what I want.

Parameters for Trace:

My expectation from this would be an output of all point and line features where the count of traversed features moving upstream from each outlet point is </= 5, but instead it outputs a selection of every top (i.e. inlet, i.e. terminal in reversed flow direction) node in the entire network.

Apologies if I am not understanding something about your instructions, this is still very new to me.

For context:
I am trying to select the line features for all culverts* in the storm drain system so I can use them to hydrocondition a DEM. In the short term, I just need a workable solution as soon as possible - which is why I'm trying to do things I don't fully understand in a hurry (not ideal). In the long term, I will be working to learn the ins and outs of network tracing, possibly transition to a utility network model, and want to develop more versatile workflows for the network applications of the stormwater drainage system.

*(which tend to be disjoint pipes/lines with one inlet and one outlet node, though some are two pipes with inlet, outlet, and one intermediate note, thus the 5 feature limit)

Thanks again for any suggestions.

[RobertKrisher]

There is no one trace you can do to solve this problem; you will need to construct a workflow (outlined above) to help answer these questions. Because this is a very specific kind of analysis, I recommend you post your questions on the water resources community site: Water Resources - Esri Community

It sounds like you're trying to areas of the network where there are less than 5 features upstream from your starting location. You are not going to be able to do this with any kind of function or function barrier, since those can only calculate statistics or cause a trace to stop prematurely. If these culverts are connected to an outlet, then you would need to analyze all your outlets to find them. If these culverts are disjoint, then you could first need to run a disjoint analysis and identify all the island in your network to determine which ones meet your criteria. This is a very advanced analysis that will take time to develop and understand, which is why I suggest you reach out to the water resources community to see if someone has a model they can share with you.

Here is an article that discusses the trace framework more in-depth: Exploring the ArcGIS Utility Network Trace Framework (esri.com) . It was written about the UN, but most of the concepts in this article are shared between the TN and UN framework.

You can also find a presentation about the trace framework here: ArcGIS Utility Network: Advanced Network Management - Esri Videos: GIS, Events, ArcGIS Products & In...

[JasperRomero]

Thanks for all your help, Robert. I initially started out in Water Resources, but was not having much luck getting questions answered - maybe I will try re-framing my questions and asking again.

I really appreciate all the resources you have suggested! For now I will try to find some solution for my short-term needs, but I'm very interested in trying to develop the JSON-based workflow you suggested so I may follow up here when I am able.