Select to view content in your preferred language

Water Utility Network Editing Tools in ArcGIS Pro and Utility Network

1179
17
11-19-2024 08:51 AM
shildebrand
Frequent Contributor

Does anyone know if the water geometric network editing and analysis tools have been replicated in ArcGIS Pro for use in a utility network?  In arcmap, we used this tool to automate water lateral creation between a meter location and an adjacent water/sewer main.  I have thousands of water meter locations (points) and need to create laterals from the point to the main. 

If this isn't possible, is there a download of the attribute assistant and water utility network editing tools addins available for use within ArcGIS 10.5.1?  We still have a copy of arcmap and could theoretically replicate this process outside of the utility network.

0 Kudos
17 Replies
MikeMillerGIS
Esri Frequent Contributor

We prototyped an attribute rule to replace this function

https://esri.github.io/arcade-expressions/docs/3.3/CreateAlongPointToFeature.html

Tool to create the AR is here:

https://github.com/Esri/arcade-expressions/blob/master/attribute-rules/attribute_assistant/Attribute...

The help provides some details, but for this specific workflow, you need to do the following:

  • Create a new group template on Service Connection
    • The attributes of this template determine determine if the attribute rule fires during insert/update
  • Change Service Connection to Line Tool
  • Add the lateral to the template
  • Add the tap to the temple
  • Add the service valve to the template
  • Add anything else you want
  • Run the  Create Along Point To Feature tool
    • Select the Service Connection layer
    • Select the Group template you just create
  • Create a template on Service Connection that matches the attributes you used when defining the group template
    • Also set the DynamicValue field(Think it is  Create Along Point To Feature or something like that)
  • Place a new point
    • Should fire the AR and create the laterals etc
  • Now it is working, test it on existing rows
    • Select those rows and set the dynamic value attribute

You can contact Support to get a copy of the old arcmap solution if you need it for arcmap.

 

0 Kudos
shildebrand
Frequent Contributor

Thanks Mike,

It looks like this tool was developed for version 3.3 and we're running 3.0.3, so that probably explains why when I open the "Create Along Point to Feature" tool, it fails to open.  In the meantime, i'm having a hard time following the workflow.  Can you send a screenshot or two to illustrate what the group template should look like?  

Is this tool creating laterals from the main to the point locations?  Ideally this tool would create laterals from selected points (service connections/meter locations) to the nearest water/sewer main.

0 Kudos
MikeMillerGIS
Esri Frequent Contributor

Yes, it is only for 3.3. 

 

The GP tool is used to just create the attribute rule

MikeMillerGIS_0-1732125255716.png

 

Here is the AR rule that GP tool creates

Expects($feature, 'assetgroup', 'lifecyclestatus', 'assettype', 'terminalconfiguration', 'constructionstatus');
/*
Description:  This rule finds the closet line and draws a line to the closest point on it.  This can be used in a water or sewer network to draw the service line.
*/
var debug = [];
var laterals_rule_bit = 1;
if (($feature.dynamicvalue & laterals_rule_bit) != laterals_rule_bit) {
    return;
}
var feature_template_value_lookup = [
    {
        "assettype": 3,
        "assetgroup": 7,
        "terminalconfiguration": "Default",
        "lifecyclestatus": 2,
        "constructionstatus": 6
    },
    {
        "assettype": 1,
        "assetgroup": 7,
        "terminalconfiguration": "Default",
        "lifecyclestatus": 2,
        "constructionstatus": 6
    }
];

var feature_to_template = [
    {
        "search_settings": {
            "search_distance": 30.480060960121925,
            "extend_to_class": "SewerLine",
            "extend_to_sql": "ASSETGROUP = 1",
            "extend_to_oid_field": "OBJECTID"
        },
        "start_at_feature": true,
        "utility_network_settings": {
            "use_associations": true
        },
        "points_at_start": [],
        "points_at_end": [
            {
                "class_name": "SewerDevice",
                "attributes": {
                    "assettype": 1,
                    "assetgroup": 21,
                    "terminalconfiguration": "Default",
                    "lifecyclestatus": 2,
                    "constructionstatus": 6
                },
                "is_proportional": 0,
                "offset_distance": 0,
                "side": 0,
                "vertex_before_offset": true,
                "distance": 3.048
            },
            {
                "class_name": "SewerJunction",
                "attributes": {
                    "assettype": 54,
                    "assetgroup": 20,
                    "lifecyclestatus": 2,
                    "constructionstatus": 6
                },
                "is_proportional": 0,
                "offset_distance": 0,
                "side": 0,
                "vertex_before_offset": true,
                "distance": 0
            }
        ],
        "lines": [
            {
                "class_name": "SewerLine",
                "attributes": {
                    "assettype": 1,
                    "assetgroup": 3,
                    "lifecyclestatus": 2,
                    "material": "XXX",
                    "constructionstatus": 6
                },
                "two_point": true
            }
        ]
    },
    {
        "search_settings": {
            "search_distance": 30.480060960121925,
            "extend_to_class": "SewerLine",
            "extend_to_sql": "ASSETGROUP = 1",
            "extend_to_oid_field": "OBJECTID"
        },
        "start_at_feature": true,
        "utility_network_settings": {
            "use_associations": true
        },
        "points_at_start": [],
        "points_at_end": [
            {
                "class_name": "SewerDevice",
                "attributes": {
                    "assettype": 1,
                    "assetgroup": 30,
                    "terminalconfiguration": "Default",
                    "lifecyclestatus": 2,
                    "constructionstatus": 6
                },
                "is_proportional": 0,
                "offset_distance": 0,
                "side": 0,
                "vertex_before_offset": true,
                "distance": 3.048
            },
            {
                "class_name": "SewerJunction",
                "attributes": {
                    "assettype": 54,
                    "assetgroup": 20,
                    "lifecyclestatus": 2,
                    "constructionstatus": 6
                },
                "is_proportional": 0,
                "offset_distance": 0,
                "side": 0,
                "vertex_before_offset": true,
                "distance": 0
            }
        ],
        "lines": [
            {
                "class_name": "SewerLine",
                "attributes": {
                    "assettype": 1,
                    "assetgroup": 3,
                    "lifecyclestatus": 2,
                    "material": "XXX",
                    "constructionstatus": 6
                },
                "offset_distance": 0,
                "side": 0,
                "vertex_before_offset": true,
                "two_point": false
            }
        ]
    }
];

var round_Z_value_factor = 6;
// Lines less than this length, will not be processed, value is in meters
var min_line_length = 0;

function find_template_index() {
    for (var i in feature_template_value_lookup) {
        var valid_template = True;
        for (var k in feature_template_value_lookup[i]) {
            if ($feature[k] != feature_template_value_lookup[i][k]) {
                valid_template = False;
                break;
            }
        }
        if (valid_template){
          return i;
        }
    }
    return -1;
}

function featureset_by_name_switchyard(class_name) {
    if (class_name == 'SewerLine'){
        return FeatureSetByName($datastore, 'main.SewerLine', ['OBJECTID'], true);
    }
}

function get_point_along_location(pull_back, is_proportional, from_start) {
    //This only works on 2 point lines

    // Calculate the location of the distance, using percentage.  Return the point and the distance from the start
    var percent_along;
    if (is_proportional == true || is_proportional == 1) {
        percent_along = pull_back / 100;
    } else {
        percent_along = Min([1, pull_back / lateral_line_length]);
    }
    var a = start_point;
    var b = end_point;

    var x1 = a.X;
    var y1 = a.Y;
    var z1 = a.Z;
    var x2 = b.X;
    var y2 = b.Y;
    var z2 = b.z;
    var dx = (x2 - x1) * percent_along;
    var dy = (y2 - y1) * percent_along;
    var dz = (z2 - z1) * percent_along;
    if (from_start) {
        return [Point({
            x: x1 + dx,
            y: y1 + dy,
            z: z1 + dz,
            spatialReference: spat_ref
        }), lateral_line_length * percent_along];
    } else {
        return [Point({
            x: x2 - dx,
            y: y2 - dy,
            z: z2 - dz,
            spatialReference: spat_ref
        }), lateral_line_length - (lateral_line_length * percent_along)];
    }
}

function find_closest_line(fs, search_distance) {
    var candidates = Intersects(fs, Buffer($feature, search_distance, 'meters'));

    var shortest = [Infinity, null];
    for (var line in candidates) {
        var d = Distance($feature, line);
        if (d < shortest[0]) {
            shortest = [d, line];
        }
    }
    return shortest[-1];
}

function create_line(point_list) {
    var path_coords = [];
    for (var i in point_list) {
        var pnt = point_list[i];
        Push(path_coords, [pnt.X, pnt.Y, pnt.Z]);
    }
    return Polyline({
        "paths": [path_coords],
        "spatialReference": spat_ref
    });

}

// Used to check different empty null states, override of core IsEmpty
function IsEmptyButBetter(data) {
    if (IsEmpty(data)) return true;
    for (var x in data) return false;
    return true;
}

function pDistance_with_ZM(x, y, x1, y1, z1, m1, x2, y2, z2, m2) {
    // adopted from https://stackoverflow.com/a/6853926
    var A = x - x1;
    var B = y - y1;
    var C = x2 - x1;
    var D = y2 - y1;

    var dot = A * C + B * D;
    var len_sq = C * C + D * D;
    var param = -1;
    if (len_sq != 0) //in case of 0 length line
        param = dot / len_sq;

    var line_length = Sqrt(len_sq);
    var xx, yy, zz, mm;
    var is_vertex = true;
    mm = null;

    if (param < 0) {
        //Start of the line
        xx = x1;
        yy = y1;
        zz = z1;
        mm = m1;
    } else if (param > 1) {
        //End  of the line
        xx = x2;
        yy = y2;
        zz = z2;
        mm = m2;
    } else {
        xx = x1 + param * C;
        yy = y1 + param * D;
        dx = xx - x1;
        dy = yy - y1;
        var distance_to_coord = Sqrt(Abs(dx * dx + dy * dy));
        zz = Round((z2 - z1) * (distance_to_coord / line_length), round_Z_value_factor);
        if (m1 == null || IsNan(m1) || m2 == null || IsNan(m2)) {
            mm = null;
        } else {
            mm = (m2 - m1) * (distance_to_coord / line_length);
        }
        is_vertex = false;
    }

    var dx = x - xx;
    var dy = y - yy;
    // Note, this distance is the distance the point away from the line
    return [Sqrt(dx * dx + dy * dy), [xx, yy, zz, mm], is_vertex];
}

function coordinate_to_xyzm(coordinate, hasZ, hasM) {
    var x, y;
    var z = 0;
    var m = null;

    if (TypeOf(coordinate) == 'Point') {
        x = coordinate.x;
        y = coordinate.y;
        if (hasZ) {
            z = coordinate.z;
        }
        if (hasM) {
            m = coordinate.m;
        }
    } else {
        x = coordinate[0];
        y = coordinate[1];
        if (hasZ) {
            z = coordinate[2];
        }
        if (hasM && hasZ) {
            m = coordinate[3];
        } else if (hasM) {
            m = coordinate[2];
        }
    }
    return [x, y, z, m];
}

function get_location_info(line_geometry, X, Y) {
    // Check if the line has already been converted to a dict
    var line_shape = null;
    var hasZ, hasM;
    if (TypeOf(line_geometry) == 'Dictionary') {
        line_shape = line_geometry;
    } else {
        // As of 3.2, can remove the call to Text
        line_shape = Dictionary(Text(line_geometry));
    }
    hasZ = line_geometry['hasZ'];
    hasM = line_geometry['hasM'];
    var min_distance = Infinity;
    var segment_id = [];
    var line_path = line_shape['paths'];

    for (var i in line_path) {
        var current_path = line_path[i];
        // Loop over vertex, exit when at last vertex
        for (var j = 0; j < Count(current_path) - 1; j++) {
            var from_coord = coordinate_to_xyzm(current_path[j], hasZ, hasM);
            var to_coord = coordinate_to_xyzm(current_path[j + 1], hasZ, hasM);

            var shortest = pDistance_with_ZM(X, Y, from_coord[0], from_coord[1], from_coord[2], from_coord[3], to_coord[0], to_coord[1], to_coord[2], to_coord[3]);
            var dist = shortest[0];
            var coordinates = shortest[1];
            var isVertex = shortest[2];
            if (dist <= min_distance) {
                segment_id = [i, j, coordinates, isVertex, dist];
                min_distance = dist;
            }
        }
    }
    if (IsEmptyButBetter(segment_id)) {
        return [];
    }
    // return just the new coordinate
    return segment_id;
}

function process_points(template_list, from_start) {

    for (var i in template_list) {
        var edit_template = template_list[i];
        var dist = 0;
        var is_proportional = false;
        var feature_geo;
        var distance_down;
        var results;
        if (HasValue(edit_template, "distance")) {
            dist = edit_template.distance;
        }
        if (HasValue(edit_template, "is_proportional")) {
            is_proportional = edit_template.is_proportional;
        }
        if (dist != 0) {
            results = get_point_along_location(dist, is_proportional, from_start);
            feature_geo = results[0];
            distance_down = results[1];
        } else {
            feature_geo = iif(from_start == true, start_point, end_point);
            distance_down = iif(from_start == true, 0, lateral_line_length);
        }
        // Convert distance to text to use as key of dict, round at text to number can change precision
        distance_down = Text(Round(distance_down, 4));
        if (!HasKey(new_feature_at_distance, distance_down)) {
            new_feature_at_distance[distance_down] = [];
        }
        Push(new_feature_at_distance[distance_down],
            {
                "class_name":edit_template.class_name,
                "geometry": feature_geo,
                'attributes': edit_template.attributes
            });
    }
}

function add_vertex_to_extend_to_class(line_geometry, oid) {
    // Add a vertex to the closets line if needed
    if (at_vertex) {
        return;
    }
    // At 3.2, we can remove the call to text
    var line_geo_dict = Dictionary(Text(line_geometry));
    Insert(line_geo_dict['paths'][path_index], segment_index + 1, snapped_loc);
    if (!HasKey(updated_features, edit_template_info.search_settings.extend_to_class)) {
        updated_features[edit_template_info.search_settings.extend_to_class] = [];
    }
    Push(updated_features[edit_template_info.search_settings.extend_to_class],
        {
            'objectID': oid,
            'geometry': Polyline(line_geo_dict)
        });
}

function process_lines(template_list, line_geo) {
    var edit_template;
    for (var i in template_list) {
        edit_template = template_list[i];
        if (!HasKey(new_features, edit_template.class_name)) {
            new_features[edit_template.class_name] = [];
        }
        if (edit_template.two_point == true || edit_template.two_point == 1) {

            for (var j = 0; j < Count(line_geo.paths[0]) - 1; j++) {
                var two_point_geo = create_line(Slice(line_geo.paths[0], j, j + 2));
                Push(new_features[edit_template.class_name],
                    {
                        "geometry": two_point_geo,
                        'attributes': edit_template.attributes
                    });
            }
        } else {
            Push(new_features[edit_template.class_name],
                {
                    "geometry": line_geo,
                    'attributes': edit_template.attributes
                });
        }
    }
}

function sort_new_features(feat_dict, from_start) {

    var keys = [];

    for (var v in feat_dict) {
        Push(keys, Number(v));
    }
    keys = Sort(keys);
    if (!from_start) {
        keys = Reverse(keys);
    }
    var sorted_features = {};
    var vertices_list = [];
    var tag_list = [];
    for (var i in keys) {

        var new_feat_list = feat_dict[Text(keys[i])];
        // Loop over all the features at that location and add them to the new features dict.  tag them so we can make associations later if needed
        for(var j in new_feat_list) {
            var new_feat = new_feat_list[j];
            var assoc_tag =  i + "::" + j;
            if (!HasKey(sorted_features, new_feat.class_name)) {
                sorted_features[new_feat.class_name] = [];
            }
            var n_d = {
                    "geometry": new_feat.geometry,
                    'attributes': new_feat.attributes,
                    'tag': assoc_tag
                };
            if (i == 0 && j ==0 && create_assoc){
                 n_d.associationType = 'connected';
            }
            Push(sorted_features[new_feat.class_name], n_d);
            Push(tag_list, [assoc_tag, new_feat.class_name]);
        }
        if (i != 0 && i != lateral_line_length){
            // store only the first feat if there are more than one at a location
            Push(vertices_list, new_feat_list[0].geometry)
        }
    }
    return [sorted_features, vertices_list, tag_list];
}

var template_index = find_template_index();
if (template_index == -1) {
    return;
}

var edit_template_info = feature_to_template[template_index];
if (IsEmptyButBetter(edit_template_info)) {
    return;
}

var create_assoc = edit_template_info.utility_network_settings.use_associations;

// Convert the input point to xyzm
var point_geo = Geometry($feature);
var point_coords = coordinate_to_xyzm(point_geo, point_geo.hasZ, point_geo.hasM);
var spat_ref = point_geo.spatialReference;
point_geo = Point({
    x: point_coords[0],
    y: point_coords[1],
    z: point_coords[2],
    m: point_coords[3],
    spatialReference: spat_ref,
    hasM: true,
    hasZ: true
});

var fs = featureset_by_name_switchyard(edit_template_info.search_settings.extend_to_class);
if (edit_template_info.search_settings.extend_to_sql != null) {
    fs = Filter(fs, edit_template_info.search_settings.extend_to_sql);
}
var closest = find_closest_line(fs, edit_template_info.search_settings.search_distance);
if (closest == null) {
    return;
}

var closet_line_geo = Geometry(closest);
var snapped_info = get_location_info(closet_line_geo, point_geo.X, point_geo.Y);

if (IsEmptyButBetter(snapped_info)) {
    return;
}
var path_index = snapped_info[0];
var segment_index = snapped_info[1];
var snapped_loc = snapped_info[2];
var at_vertex = snapped_info[3];

// Generate the line connecting the point to the line
var snapped_point = Point({
    x: snapped_loc[0],
    y: snapped_loc[1],
    z: snapped_loc[2],
    spatialReference: spat_ref
});

// Create the line from $feature to the snap point in the orientation specified in the edit template
var start_point;
var end_point;
var from_start = edit_template_info['start_at_feature'];
if (from_start) {
    start_point = point_geo;
    end_point = snapped_point;
} else {
    start_point = snapped_point;
    end_point = point_geo;
}
if (Equals(start_point, end_point)) {
    return;
}

// Create the line, this is used to determine the point locations along
var lateral = create_line([start_point, end_point]);
var lateral_line_length = Length(lateral, 'meters');

// If line is less than min required length, exit
if (lateral_line_length < min_line_length) {
    return;
}
// Create dicts to hold new and updated features
var new_feature_at_distance = {};

// Process all the points along, these are the points from end and beginning
process_points(edit_template_info.points_at_start, true);
process_points(edit_template_info.points_at_end, false);

var updated_features = {};

// Sort the new points along by their distance
var sorted_features_vertices = sort_new_features(new_feature_at_distance, from_start);
var new_features = sorted_features_vertices[0];
var vertices = sorted_features_vertices[1];
var assoc_tags = sorted_features_vertices[2];

// Recreate the line geometry
// Insert the start and end points
Insert(vertices, 0, start_point);
Push(vertices, end_point);
lateral = create_line(vertices);
// Process all line templates
process_lines(edit_template_info.lines, lateral);

// To connect the new features to the exiting, a vertex is require, add one if needed.
add_vertex_to_extend_to_class(closet_line_geo, closest[edit_template_info.search_settings.extend_to_oid_field]);

// create the return edit payload
var return_edits = [];
for (var k in new_features) {
    Push(return_edits, {'className': k, 'adds': new_features[k]});
}
for (var k in updated_features) {
    Push(return_edits, {'className': k, 'updates': updated_features[k]});
}

if (create_assoc){
    var assoc_rows = []
    for (var i =0; i< Count(assoc_tags) -1; i++){
        var from_assoc_tag = assoc_tags[i][0];
        var from_assoc_cls = assoc_tags[i][1];

        var to_assoc_tag = assoc_tags[i+1][0];
        var to_assoc_cls = assoc_tags[i+1][1];
        Push(assoc_rows,{
            "fromClass":from_assoc_cls,
            "fromGlobalId":from_assoc_tag,
            "toClass": to_assoc_cls,
            "toGlobalId": to_assoc_tag,
            "associationType": "connectivity"
        })
    }
     Push(return_edits, {'className': '^UN_Association', 'adds': assoc_rows});
}

//console(debug);

return {
    'result': {
        'attributes':
            {
                'dynamicvalue': $feature.dynamicvalue ^ laterals_rule_bit
            }
    },
    "edit": return_edits
}

 

0 Kudos
TSmith
by
Regular Contributor

I have a question about this tool. 

I am pointing it to the Water Junction layer (tool won't take inputs from the UN feature service) and the group template I have is identical to the one in the editing map that has the service valve and meter. 

 

I am getting this error upon execution: 

[2024-12-05 12:07:05] iar 0.3.4
[2024-12-05 12:07:06]
 line 143, in get_data_path_from_layer
TypeError: unsupported operand type(s) for /: 'NoneType' and 'str'

Failed script Create Along Point To Feature...
Failed to execute (CreateAlongPointToFeature).

 

TSmith_0-1733418822641.png

Does this need to be a subtype group layer pointing to the DB (not the feature service)? Or can it be a FeatureLayer referencing the FC without the subtype. I have also seen some attribute rules on the github for creating laterals. Any help is much appreciated! Thanks as always!

0 Kudos
MikeMillerGIS
Esri Frequent Contributor

My guess is we never coded for services.  As this results in a schema change(add attribute rule), which requires a database connection, we probably should just better block service inputs.  Try using your database connection

0 Kudos
TSmith
by
Regular Contributor

Sorry- should have specified, I am pointing the tool to the water junction layer in a Map, and that layer is coming from the database connection

0 Kudos
MikeMillerGIS
Esri Frequent Contributor

ahh ok,  let me set up an enterprise database and see if I can repo then.  Might be something specific to EGDB.  You could try on a FGDB version of your UN and if it works, just apply the AR to your Enterprise layer.  

0 Kudos
TSmith
by
Regular Contributor

Was able to get it to work- think the fact I ran it on a VDI had something to do with it (the appdata folder doesnt exist on there or is blocked), I'm guessing the only difference between the FGDB and EGDB variant is adding the database owner to the class variables? 

 

(i.e. UNOwner.WaterLine, instead of WaterLine) 

 

I attached the CSV it generated. Thanks so much for helping me through this process. 

0 Kudos
MikeMillerGIS
Esri Frequent Contributor

I'm guessing the only difference between the FGDB and EGDB variant is adding the database owner to the class variables?  - Yes

If you are trying to create a lateral when you place a service connection, then you should create the group template on the service connection(and assign the attribute rule there) and remember the service connection templates attribute control the filter in the attribute rule to see if it should process and create the lateral(in addition to the value of the dynamic value field)

0 Kudos