Skip navigation
All Places > Electric and Gas > Blog
1 2 3 Previous Next

Electric and Gas

33 posts

By Tom DeWitte and Tom Coolidge

Installing the correct components when constructing a new pipe system or replacing an existing portion of a pipe system is critical to the safety and reliability of the overall pipe system.  When dealing with buried pipe utilities such as natural gas, water, district heating, district cooling, and hazardous liquids, this is a real issue.  Every year field crews inadvertently make the following mistakes:

   -install polyethylene assets that have been sitting in the service yard for too long


   -contractor installed a component for a utility company that is not on the utility companies’ approved manufacturer list


   -field crew installed a pipe system component which is no longer compatible with company standards. 


Without real-time field validation, these honest mistakes typically do not get identified until after the construction is complete and the pipe components have been covered over.  This latency in identification leads to expensive post-construction repairs. 


Real-time Validation

If the field construction crews could be notified that a specific pipe component about to be installed does not meet the requirements for valid installation, the previously listed issues could be eliminated. What field crews need is real-time validation.


Configuring Collector for Real-time Validation

In early 2019 Collector for ArcGIS was enhanced to support arcade scripting in the web maps which provide the configuration of Collector’s behavior.  As noted in previous blog articles, this opened the capability for real-time decoding of a pipe component’s barcode.


-Tracking and Traceability 2019: Part 1

-Tracking and Traceability 2019: Part 2


The ability to add arcade scripts to the web map pop-up provides an advanced configuration ability to provide field crews with real-time validation.


What’s a field person to do?

A field person can easily use this real-time validation capability.  Since Collector runs on Apple, Android and Windows mobile devices, they could check the validity of pipe segments, plastic device and plastic fittings while unloading them from the delivery truck.  All the field person would have to do is to use their smart phone running the Collector to scan the barcode using the device’s camera.

Screenshot of portion of Collector pop-up


Collector will automatically decode the barcode information and open a pop-up window with the validation results.  Invalid pipe segments, devices and fittings never reach the installation trench.         

Keeping invalid pipe components out of installation trenches improves safety, system reliability, and eliminates unwanted costs. No one wants to have to re-dig the construction location to remove the invalid pipe components.


How is this possible?

Esri makes real-time validation possible by allowing arcade scripts to be added to the web map configuration file.  More specifically the arcade script is added to the pop-up configuration in the web map of the pipe, device or fitting layer.

Screenshot of portion of pop-up layer configuration


With the arcade script added to the desired layer pop-ups, the web map is now ready for real-time validation.  For the field user, initiation of the validation process occurs automatically when the field user presses the “Submit” button in the upper right corner of the Collector display.

Screenshot of top portion of Collector application


The pressing of the “Submit” button after collecting some information such as scanning of a barcode also automatically opens the pop-up to show the validation results.  It really is that easy to deploy and that seamless an experience for the field user.            

What is the script doing?

The logic in the arcade script is the key to enabling Collector to perform real-time validation.  What must the script do? 

The simple answer is that it must be able to acquire the information needed to answer a question.  For example, a core validation for plastic pipe construction is whether the polyethylene plastic material is too old.  Polyethylene plastic is susceptible to the suns UV rays.  Let a roll of medium density polyethylene pipe site in the service yard for over 3 years and the sun’s UV rays will have degraded the material to the point where it should not be installed. The information needed to assess whether the role of pipe is too old is the date of manufacture and the current date.  The date of manufacture is acquired form the scanning and decoding of the ASTM F2897 barcode.  The current date is acquired from the mobile device itself.  Subtract the manufacture date from the current date and you have a time difference.  If the time difference exceeds the industry recommended shelf life then that roll of pipe is invalid and should not be installed. 

Here is a snippet of the arcade script to determine whether the polyethylene plastic pipe or component has exceeded the recommended shelf life.

Portion of arcade script to determine material shelf life


Where can I get these scripts?

Many people have told me that they find it easier to modify someone else’s script than to write one from scratch. With that statement in mind we have written arcade scripts against a UPDM 2019 data model and the ASTM F2897 barcode standard to address three validation scenarios.

  • Scenario 1: Material for HDPE and MDPE has exceeded its shelf life
  • Scenario 2: The manufacturer of the pipe system component is not on the utilities approved list.
  • Scenario 3: The specific size and model of the component is not part of the utilities set of codes and standards.


These arcade scripts are available for download from the following location on geonet. In addition to the scripts are detailed instructions on how to configure and deploy the scripts into your ArcGIS Enterprise or Online organization.  That’s right, web map based arcade scripts not only work for ArcGIS Enterprise environments they also work for ArcGIS Online organizations.


What else can Collector real-time validations do

In addition to the real-time validation scenarios previously listed, there are other opportunities for applying real-time validation.  For example, you could create custom barcodes for welding and plastic fusion operators.  The custom barcodes could embed the worker’s operator qualifications. A Collector web map embedded arcade script could decode that scanned operator’s badge barcode and immediately determine whether the operator is qualified and whether the qualifications are still valid.

The advanced configuration capabilities of web maps with arcade scripting open capabilities that previously required complex and expensive customization.  The universal use of web maps in web applications and mobile applications such as Collector allow this configuration to be done once and utilized across Windows mobile devices, Android mobile devices, Apple mobile devices, and web applications. 


And did I mention that these real-time validations work even when the device is disconnected from the network?


PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.



By Tom Coolidge and Tom DeWitte


Part 3 of 3

This the third and final blog in a series that explains how the ArcGIS platform with the ArcGIS Utility Network Management extension and the Utility and Pipeline Data Model (UPDM) can be utilized to model a cathodic protection system.


What is a cathodic protection zone and why does a pipe organization need to understand it?  


Cathodic Protection Zones

What is a CP zone? In the second blog of this series we described the components which comprise a cathodic protection zone and how UPDM 2019 provides a template for organizing the information about those components.  But, a cathodic protection zone is more than its components.

Cathodic Protection System

A cathodic protection zone is really an electrical circuit.  Electricity flows through it to protect the connected components from corrosion. So, to understand what a cathodic protection zone is, we need an understanding of the connectivity between the components.  But even that is not enough. In addition to understanding connectivity we need to understand what connected components have characteristics which will cause the flow of electricity to stop. 


This means the GIS model representing the cathodic protection zone needs to know that plastic pipe is non-conducting and will therefore stop the flow of electricity.  The GIS system needs to understand that devices and fittings can be insulated, and this will also stop the flow of electricity. 


The ArcGIS Utility Network Management extension provides this higher level of understanding within ArcGIS.


Defining the Cathodic Protection Zone

To create a cathodic protection zone within the utility network, all PipelineLine, PipelineDevice and PipelineJunction features must have their CPTraceability populated. Additionally, the test points must be configured as terminals and designated as a subnetwork controller.

The logic that defines how the utility network discovers a cathodic protection zone is as follows:

  1. Start the trace from the sources (Test Point(s))
  2. Use the utility network connectivity to begin traversing the system.
  3. Stop traversing the network when the trace encounters a feature with a CPTraceability = Not Traceable.

The tool within the utility network which performs this task is the “Update Subnetwork” geoprocessing tool.


When the “Update Subnetwork” is run, it aggregates the following PipelineLine features to create the subnetwork geometry.

  • Distribution lines
  • Transmission lines
  • Gathering lines


Additionally, the “Update Subnetwork” is preconfigured in UPDM 2019 to summarize the following information and write it to the subnetwork feature record.

  • Number of Anodes
  • Number of Rectifiers
  • Number of Test Points
  • Total Length
  • Total Surface Area


Defining Flow for Cathodic Protection

In the digital world of flow analysis, there are two types of flow networks; source, and sink.

  • SOURCE — A source is an origin of the resource delivered. For example, for a natural gas distribution system, sources of natural gas are the utility transfer meters within town border stations.


  • SINK —A sink is the destination of the gathered resource. For example, when modeling the Mississippi river basin, the sink of the pipe network is the outflow into the Gulf of Mexico, just south of the city of New Orleans.


A pressure system is another example of a source flow system. The source of gas to the gas pressure zone is the regulator device. A single gas pressure zone will typically have multiple regulators feeding gas into the pressure zone.

Diagram of Pressure Zone

Within the utility network, a single domain may only have one type of subnetwork controller (Source or Sink). The gas pipe system tiers (System, Pressure, Isolation) are modeled as sources.  In UPDM 2019, the Pipeline domain models the subnetwork controller type as a “Source” to support the pipe system tiers.


The cathodic protection system of a pipe system is not as consistent a flow model as the pressurized pipe system. For the impressed current system, the rectifier would be the logical source and the anode would be an intermediate device.  For the galvanically protected system, the anode would be the logical source. Because of this inconsistency, it was decided that the best option was to make the test point the source as it is typically a part of both the galvanically protected system and the impressed current protection system.


Tracing Across a Cathodic Protection Zone

Now that the cathodic protection zones have been defined with the “Update Subnetwork” geoprocessing tool users can begin to perform traces across the cathodic protection system.  Some common questions to ask the utility network via a trace are:

  • Where is are the Test Points?
  • Where is the nearest test point?
  • Which pipe system components participate in the zone?

Outside of the trace tools simple attribute queries can be run to understand the following:

  • Which pipe system components are bonded?
  • Which pipe system components are cathodic protection insulators

With the cathodic protection zones defined in the utility network, these questions can be easily answered.



Data management and analysis of cathodic protection systems was a challenge in legacy geospatial systems.  Entering the information has always been a straight forward process.  Maintaining an intelligent representation of the cathodic protection system has historically been the challenge. With the utility network combined with the UPDM 2019 configuration, maintaining and analyzing a cathodic protection system is now an intuitive process.


If you missed the first two blogs in this series, we encourage you to check them out. The first blog provided an overview of how cathodic protection systems works to provide GIS professionals and IT administrators with enough knowledge to be able to correctly create a digital representation of a cathodic protection system utilizing UPDM 2019 and the utility network . The second blog went into detail on the use of UPDM 2019 to organize the digital presentation of the cathodic protection system.


PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions

By Tom Coolidge and Tom DeWitte

Part 2 of 3


Our first blog in this series provided an overview of how cathodic protection systems works to provide GIS professionals and IT administrators with enough knowledge to be able to correctly create a digital representation of a cathodic protection system utilizing Utility and Pipeline Data Model (UPDM) 2019 and the utility network.


This second blog goes into detail on the configuration of UPDM to manage the components which make up the cathodic protection system.


Many, many years ago, being new to the natural gas and hazardous liquid industries, the management of cathodic protection was a mystery.  The data about the cathodic protection system was not being stored in the GIS along with the assets of the pipe system.  When I asked the GIS staff about this, the common answer was that the cathodic protection group maintained their data separately. This leads to the next question. What system were they using?  The most common answer I got was paper and colored pencils. That’s right the cathodic protection data was being manually maintained on a set of paper maps with colored pencils.  And every winter the cathodic protection group would manually transpose the data from last year’s paper maps to the current year’s paper maps.

Over time, the cathodic protection data started to show up in more gas GIS systems.  Most often it was an incomplete representation of the cathodic protection system.  You might see some test points and anode beds, but they usually were not connected to the pipe system.  Additionally, other important information such as insulators and rectifiers were commonly not mapped.


Some natural gas or hazardous liquid companies did map the entire cathodic protection system. But they needed special tools to manage and maintain this information.


With the release of UPDM 2019 and the utility network, it is now possible to maintain the entire cathodic protection system with the standard data management and editing tools provided by Esri.


No colored pencils required!


UPDM 2019

The 2019 edition of UPDM provides a template for organizing natural gas and hazardous liquid pipe system information. This data model is an Esri-structured geodatabase.  It is written to be able to be used and managed with the standard data management tools provided by Esri’s ArcGIS products.


UPDM 2019 and Modeling Cathodic Protection Data

The release of UPDM 2019 introduces a new, simpler, and more complete data model for managing cathodic protection data in an ArcGIS geodatabase.  These changes are intended to be used with the ArcGIS Utility Network Management Extension to allow for the modeling of the cathodic protection system.

Cathodic Protection Components in UPDM

The discrete components of a cathodic protection system modeled in UPDM 2019 are anodes, rectifiers, test points, wire junctions, and insulation junctions.  The anodes, rectifiers, and test points are point features stored as asset groups within PipelineDevice featureclass.

These PipelineDevice features are not inline features of the pipe system.  Instead they physically sit adjacent to the pipe system.  These anodes, rectifiers, and test points are connected to the pipe system assets by wires and cables. The location where the test lead wires connect to the pipe system can be identified with the PipelineJunction AssetGroup type of Wire Junction.  The modeling of test junctions is not required, as the UPDM default rulebase for the utility network also allows the wires and cables to connect directly to the PipelineLine pipe segments.


The location of insulators can be specified with the PipelineJunction AssetGroup type of Insulator Junction.

The wires and cables are classified as bonding lines, rectifier cables, and test lead wires. Within UPDM they are stored in the PipelineLine featureclass.

Modeling Insulating Components

Within UPDM 2019, management of insulating pipe components is key to successfully modeling cathodic protection systems. From the perspective of modeling cathodic protection systems, the management of insulators is the defining of whether a pipe system component can be electrically traversed.

  • Pipe system component is insulating       = Not traversable
  • Pipe system component is not insulating = Traversable


In ArcGIS and the utility network, we simulate traverseability with tracing.  This means that if a pipe system component is not insulated, it is traversable which means it is traceable when defining a cathodic protection system.

  • Pipe system component is insulated       = Not traversable             = Not traceable
  • Pipe system component is not insulated = traversable                    = Traceable


In UPDM 2019, determination of whether a pipe system component is traceable is defined with the attribute: CPTraceability.  The following UPDM featureclasses which participate in the utility network have the CPTraceability attribute:

  • PIpelineLine
  • PipelineDevice
  • PipelineJunction


This attribute is assigned a coded value domain called: CP_Traceability.  This coded domain has the following values:







Not Traceable

Coded Value Domains for CP_Traceability

Within the utility network properties predefined in UPDM 2019, this attribute has been associated to the network attribute: cathodic protection traceability. This allows the value to be utilized within the trace definition which is used to define the cathodic protection zone.


Within UPDM 2019, a pipe system asset is defined as being insulated by setting the BondedInsulated attribute to a value of “Insulated”. The following UPDM featureclasses which participate in the utility network have the BondedInsulated attribute:

  • PipelineLine
  • PipelineDevice
  • PipelineJunction


The attribute BondedInsulated has been assigned the coded value domain: Bonded_Insulated.  This coded value domain has the following values:








Coded Value Domain for Bonded_Insulated


Management of Bonding Lines

Bonding lines are the wires which are used to extend the electrical connection of the cathodic protection system.  They are used to span pipeline assets which are non-conductive.

Example of Binding Wire Spanning Plastic Pipe Segment


In some legacy GIS systems, the management of bonding lines was tedious. Data editors were required to draw in the bonding line and insure that is was connected to the metallic pipe system components on each end of the line.  In the UPDM 2019 configuration, the need for geometry feature creation has been minimized by allowing an attribute on the non-conductive pipe system asset which is being spanned to indicate that the asset has been bonded.  Instead of drawing the spanning bonding line, a user simply needs to change the attribute value of the attribute: BondedInsulated to a value of “Bonded”. This means that within the Utility Network, the spanned feature can be considered traceable.


Automating Cathodic Protection Data Management

The previously described attributes, Material, BondedInsulated and CPTraceability are the PipelineDevice and PipelineJunction attributes which UPDM 2019 and the utility network use to define a cathodic protection zone. The attributes AssetType, BondedInsulated and CPTraceability are used with PipelineLine.


Attribute Purpose


PipelineDevice/ PipelineJunction

Determine material type



Determine whether bonded or insulated



Determine CP traceability




To provide automation and improve data quality, attribute rules were written to auto-populate the CPTraceability attribute based on the values of the AssetType, Material, and BondedInsulated attributes.


To explain the logic embedded within the CPTraceability attribute rules here are three scenarios:

  • Scenario 1: Metallic Pipe Segment
    • Asset Type           = Coated Steel
    • Bonded Insulated = null


  • Scenario 2: Insulated Gas Valve
    • Material                = Steel
    • Bonded Insulated = Insulated


  • Scenario 3: Plastic Pipe Spanned by Bonding Line
    • Asset Type           = Plastic PE
    • Bonded Insulated = Bonded


In each of these scenarios the CPTraceability attribute is automatically populated by the UPDM 2019-provided attribute rules.

  • Scenario 1: Metallic Pipe Segment
    • Asset Type           = Coated Steel
    • Bonded Insulated = null
    • CP Traceability   = Traceable


  • Scenario 2: Insulated Gas Valve
    • Material                = Steel
    • Bonded Insulated = Insulated
    • CP Traceability   = Not Traceable


  • Scenario 3: Plastic Pipe Spanned by Bonding Line
    • Asset Type           = Plastic PE
    • Bonded Insulated = Bonded
    • CP Traceability   = Traceable


To have the CP Traceability attribute correctly set, all the editor must do is insure that the Material/AssetType and the BondedInsulated attributes are correctly set.



The new enhanced representation of cathodic protection data in UPDM 2019 makes managing a digital representation of your cathodic protection data easier. This enhanced presentation can be created and maintained with the standard tools provided by ArcGIS Pro and the standard capabilities provided by the utility network. 


In the third and final blog of this series, we will dive into how the utility network enables organizations to understand cathodic protection zones, discover when an insulating fitting or device stops the electric circuit of the cathodic protection zone, and which pipe materials are non-conducting. 


All of this is done without colored pencils.


PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions

On January 28, 2020 over 13,500 utility professionals gathered in San Antonio TX for the annual Distributech conference to learn about the latest innovations in the electric utility industry. This year Distributech was huge and the action never let up.


The Esri booth flooded each day with visitors learning ways to geo-enable the modern utility – using the complete ArcGIS platform to accomplish digital transformation. Two themes rose to the top – Grid Modernization and Field Mobility.


Visitors enjoyed demonstrations in the following areas: Asset management, Safety first, Customer engagement, Grid Mod, Network management, Analytics, Field operations, Innovation, Real-time/IoT, and Emergency management.


The demonstration theater seemed to run almost non-stop drawing crowds and often filling the adjacent isle addressing such topics as:

  • Esri - Maps and Data for Utilities, ArcGIS Utility Network,  Seeing your Business Holistically and in Real-Time, Enabling your Field Workforce with Apps, Leveraging drone Imagery for Mapping Inspection, Utility of the Future
  • SAP/Critigen – Integration of Spatial Data using SAP HANA and ArcGIS
  • UDC – Moving Utilities from a Reactive to Proactive Reliability Approach, Utility Network Migration – Getting Down to the Details
  • EPOCH Solutions – EpochField: Field Work Management Made Simple
  • DataCapable – How Dominion has Transformed Safety and Reliability, How Central Hudson Gas and Electric has Transformed Safety and Reliability with a New Platform
  • 3GIS – Avoiding Fiber Deployment Roadblocks, Accelerating Speed to Activation
  • Critigen – EAM, ADMS, OMS, Design, Esri’s Utility Network, Mobility, What should we do first?

Business Partners Bring Advanced Solutions

Esri had a very large business partner presence. Critegen, Cyclomedia, DataCapable, EOS Positioning Systems, Epoch Solutions, SAP, UDC, and 3GIS all presented solutions in the Esri booth. In total, 44 Esri business partners exhibited this year demonstrating the heightened interest in real-world solutions. Numerous companies expressed a desire to form new partner relationships to leverage with wide-spread adoption of ArcGIS in utilities worldwide.


A formal press release announced an exciting new partnership. Electric, gas, and water utilities will now be able to leverage both ArcGIS Utility Networks and the Open Systems International, Inc.(OSI) monarch operational technology (OT) platform as they become more tightly integrated.


"Many of our utility customers are adopting new Esri technology, such as ArcGIS Utility Network Management, which provides advanced network modeling capability," said Bahman Hoveida, president and CEO of OSI. "We are very excited about our partnership with Esri, as it will enable us to provide the best technical solutions to our joint customers, leveraging the latest functionality ArcGIS Utility Network Management provides."



Esri’s Bill Meehan presented to an engaged audience on why Field Mobility is more than just giving maps to field workers! Bill discussed ways to improve entire workflows with accurate data, and awareness/ access for everyone. Leading utilities are using ArcGIS mobile solutions to improve KPIs in every corner of the business.


Remi Myers shared about Analyzing Lightning Events to Improve Electric System Reliability. Remi hit on some very popular themes of Network Management, Big Data, and Analytics. He processed over 600,000 lightning strike data points in a live demo that identified broken grounds on a utility’s transmission system – impressive!


Make Plans to Join Esri Next Year

Make plans to join us next year when Distributech will return to sunny San Diego on February, 9-11, 2021.


Continue the Conversation

Got information overload? The Esri Industry Solutions Team curates the best material for our users - Follow @EsriElectricGas on Twitter for the latest! Sign up for our newsletter.

By Tom Coolidge and Tom DeWitte


We were struck recently in reading NACE International’s estimate of the money spent each year on corrosion-related costs for monitoring, replacing, and maintaining U.S. metallic pipe networks. The estimated annual tab is $7 billion for gathering and transmission pipelines and another $5 billion for gas distribution pipelines. That’s $12 billion each year!


Metallic pipe has been around for a long time. It has been used by the gas utility and pipeline industries since the 1800s when cast iron pipe first replaced wooden pipe. Advances in metallurgy through the years have steadily resulted in different types and better quality of metal for pipe networks. Today there is a lot of metallic pipe of one kind or the other in the ground. In fact, even after much cast iron and other metallic distribution pipe have been replaced by plastic pipe, there remains today several hundred thousand miles of in-service metallic pipe in America’s gas and hazardous liquids transmission and distribution networks. Much of it is old, and all of it is subject to corrosion.


Most people understand that if you put iron or steel in contact with moisture and oxygen, the metal will begin to rust or corrode. What most people do not understand is that this basic electro-chemical process can be slowed or even halted.


Gas utilities and pipelines understand that, though. That’s why today they dedicate considerable human and financial resources to the cause of cathodic protection. They do it because they are committed to safe operations, and they do it for regulatory compliance as cathodic protection has been required for much of America’s pipe networks since 1971.


This is the first blog of a series that explores how ArcGIS provides capabilities for the management of cathodic protection networks.


Protecting the Pipe from Corrosion

There are several methods to protect metallic pipe buried in the ground. One method is to apply a coating to the pipe to form a barrier between the metal pipe and the corrosion-causing mixture of water and air.

Coated Metallic Pipe


This is very common for natural gas and hazardous liquid carrying pipelines.  But it is not perfect, as a single scratch through the coating layer diminishes the protection.  A second method is to manipulate the same electro-chemical process which causes corrosion to instead protect the pipe from corrosion.  This method is called cathodic protection. Two common forms of cathodic protection are galvanically-protected and impressed-current protection.

Galvanically Protected


We Need a Sacrifice

A galvanically-protected cathodic protection system is also called a passive-cathodic protection system.  It is passive in that no foreign electrical energy is needed.  Galvanic protection works by connecting a more electrochemically active metal into the system than the pipe system which is being protected.  This electrochemically active metal is simply a hunk of metal buried in the ground near the pipe system. This component is called an anode. Common materials for anodes are zinc and magnesium. In a galvanic protection system, the anode gives up electrons to the pipe system.  This sacrifice of electrons results in the anode corroding instead of the pipe system.

Impressed Current Cathodic Protection

Charge It

Impressed-current cathodic protection systems are typically used to protect large pipe systems such as transmission pipelines.  The rectifier inserts direct current (DC) voltage into the cathodic protection system.  A rectifier cable connects the rectifier’s positive terminal to the anodes within the anode bed.  A second rectifier cable connects the rectifier’s negative terminal to the pipe system.


The Electric Circuit

The foundational concept to keep in mind when trying to understand cathodic protection is that the components of a cathodic protection system are connected to form an electric circuit.  If the circuit is broken, then the metallic pipe system components will lose their protection and the rate of corrosion will accelerate. If not corrected, the pipe system components will weaken and eventually fail.


Soil Is A Conductor

With cathodic protection, it is important to remember that the soil between the anode and the metallic pipe acts as a conductor.  The soil as a conductor of electricity completes the electric circuit connecting the anode to the metallic pipe.


Material Type Matters

The material of the pipe system components is critical to a cathodic protection system.  Some materials such as polyethylene (Plastic PE) are non-conducting and act as insulators. These insulating materials break the electric circuit.

Cathodic Protection System With Insulating Plastic Pipe


In addition to plastic pipes and plastic components acting as insulators and breaking the electric circuit, metallic components can be manufactured so that they, too, can be insulating devices or junctions.

Cathodic Protection Systems Separated By An Insulating Valve


Managing Cathodic Protection Data with UPDM

Management of the cathodic protection components in a Geodatabase is not difficult.  The anodes, rectifiers, and test points are typically modeled as point features.  The test lead wires; bonding lines, and rectifier cables are modeled as line features. Utility and Pipeline Data Model (UPDM) 2019 provides a template data model for managing these cathodic protection components.


Where data management of the cathodic protection systems gets challenging is the defining and maintaining of the cathodic protection zone.  The cathodic protection zone is the combination of pipeline, pipe devices, pipe junctions, cathodic protection devices, and cathodic protection lines, which together form an electric circuit.

Cathodic Protection System


Data management and analysis of cathodic protection systems was a challenge in legacy geospatial systems.  Entering the information has always been a straight forward process.  Maintaining an intelligent representation of the cathodic protection system has historically been the challenge.

With the utility network combined with the UPDM 2019 configuration, maintaining and analyzing a cathodic protection system is now an intuitive process. 


About This Blog Series

This blog article is the first of a three-part series explaining how the Esri ArcGIS platform with the Utility Network Management Extension and the Utility and Pipeline Data Model (UPDM) can be utilized to manage a digital representation of a cathodic protection system.  It is intended to provide GIS professionals and IT administrators with enough knowledge of how a cathodic protection system works to be able to correctly configure and deploy UPDM and the utility network.


The second blog article will go into detail on the configuration of UPDM to manage the components which makes up the cathodic protection system.


The third blog article will explain how the utility network uses its capabilities to model the cathodic protection system.


PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.

By Tom Coolidge and Tom DeWitte


Gas utility and pipeline GIS data management is increasingly important.  With a pipe network typically geographically widespread, topologically complex, and buried underground, the performance of many tasks and workflows, in a wide range of functional areas and roles, necessarily involves application software operating on a digital model of the pipe network and the surroundings through which it passes.


These models are only as good as the data available to them.  Today’s pipe network GIS typically contains extensive and detailed information about each and every component of the physical network, what is going on within it, the natural and man-made surroundings through which the pipe network passes, and activity occurring around it.


Models most often are built from that data in one of two ways – depending upon whether the objective being examined is around “where is it located” or “how is it connected.”  Linear referencing is the model building method for the first, connectivity modeling for the second.  While both methods create a network model, they do it in different ways.


Before arrival of the shared centerline feature class with ArcGIS 10.8/Pro 2.5, pipe network modelers to satisfy both modeling needs had to create and maintain multiple digital mirror representations of their real pipe network.  One of these was defined by linear referencing.  Linear referencing is a language that expresses pipeline attribute and event locations in terms of measurements along a pipeline, from a defined starting point.  The network model in Pipeline Referencing is established by the sequence of strictly increasing or decreasing measures on a continuous, unbroken non-branching run of physical pipe.


Another was defined by connectivity.  Connectivity describes the state where two or more features either share a connectivity association, or the collection of features are geometrically coincident at an endpoint (or midspan at a vertex), and a connectivity rule exists that supports the relationship.  For those to whom connectivity associations is a new term, they are used to model connectivity between two point features (Device or Junction) that are not necessarily geometrically coincident. An example of this in a pipe system is a ****** bolted to a valve.  There is no pipe component between the ****** and the valve in the physical world.  Now with connectivity associations in the utility network, this point to point connectivity can be correctly modeled in the digital world.


Traditionally, each of these ways was enabled by a separate set of data – one for linear referencing and another for connectivity modeling.


Multiple types of operators manage natural gas or hazardous liquids pipe networks and face the challenge of needing to create and maintain multiple models.  One type is vertically-integrated gas companies.  They span all or part of the way from the wellhead to the customer meter and typically operate an integrated pipe network that includes multiple subsystems – for example, transmission and distribution subsystems.  Historically, these subsystems have been modeled separately.


Transmission pipelines also face the same challenge, not because they operate multiple subsystems, but because the range of application software their GIS needs to support requires access to both kinds of models.


Moreover, all types of operators are searching for better interoperability among software systems at the enterprise level.  They also are experiencing the convergence of information technology and operations technology systems.


For all these reasons, a better solution to the need to create and maintain multiple digital models of the real pipe network is needed.


The Solution: Unified Pipe Data Management

Esri’s vision for pipe network operators is to create a single representation of the entire pipe network that mirrors the real network and can support both types of model building.  This removes the traditional barriers between industry subsystems – for example, between transmission and distribution subsystems – that result in data silos.  A single representation also enables users to work with that digital network just as they do with the real network.  Linear referencing and connectivity modeling now can be performed on the same single network representation.  We call this new data management capability: Unified Pipe Data Management.

The solution for vertically-integrated gas companies also is the solution for standalone transmission pipeline operators that, while they don’t operate multiple industry subsystems, have a need for both types of models to satisfy the data input requirements of the range of application software being supported by their GIS.


A single representation of the pipe network requires a unique data organization approach to store the entire pipe system—from wellhead to meter—and support the information model requirements of the ArcGIS Utility Network Management extension and Pipeline Referencing. Esri’s Utility & Pipeline Data Model (UPDM) 2019 is a data model template that provides this data organization.


Benefits Of Both Extensions Working On the Same Geodatabase

The ability for Pipeline Referencing and the ArcGIS Utility Network Management extensions to work on not just the same geodatabase but the same feature classes within the enterprise geodatabase, provides important benefits to pipe network operators.  First, the two extensions bring important advancements in essential industry-specific data management into Esri’s core technology.  This relieves the need for Esri business partners to fill capability gaps and frees them to extend the capabilities further and focus on adding value to uses of the data.  At the same time, it gives pipe network operators the opportunity to mix and match application software built on ArcGIS from multiple Esri business partners.  In addition, the ability for both extensions to work on the same geodatabase simplifies staff training, provides better management of high-pressure distribution pipe, and improves scalability and performance for operators of larger pipe networks.



For decades pipe organizations have had to either implement multiple models stored in separate data repositories or had to settle for one data management method over the other.  With the release of ArcGIS 10.8/Pro 2.5, a single digital representation of the physical pipe system can be created and maintained.  This reduces IT administration and support costs by allowing server systems and database licenses to be consolidated.  For data editors, the process is simplified by providing a single editing experience regardless of where the edit occurs across the vertically-integrated pipe system.  For end users, using the pipe system data is simpler because there is only one representation of the pipe system to work from.


One is better than more.


PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.

By Tom Coolidge and Tom DeWitte


Pipe network modelers have several important enhancements available to them at ArcGIS Pro 2.5 with ArcGIS Enterprise 10.8.


ArcGIS Pipeline Referencing and Utility Network Integration Enhanced

In an earlier blog post, we discussed the introduction of the shared centerline feature class as a key that enables the ArcGIS Pipeline Referencing and ArcGIS Utility Network Management extensions to work on the same geodatabase.  A shared centerline feature allows users to utilize those features to create and maintain LRS networks, with routes that are linear referenced, while also continuing to have the centerlines participate in and take advantage of utility network capabilities.  In Pipeline Referencing, an LRS network is a collection of routes measured to a specific location referencing method.  Availability of the shared centerline feature class is the first step in integrating Pipeline Referencing and the utility network.


With the release of ArcGIS Pro 2.5/ArcGIS Enterprise 10.8, support for measures on centerlines has been added to enhance integration with a utility network.  Start and end measures now are stored on centerline pipeline segments.  Also, calibration points are added at these centerline endpoints when a user utilizes centerlines in the create, extend, and realign route tools.


In addition, at ArcGIS Pro 2.5/ArcGIS Enterprise 10.8, two new geoprocessing tools are added to support ArcGIS Pipeline Referencing and utility network integration.  One is the Configure Utility Network Feature Class tool.  This tool configures a utility network pipeline feature class for use with a linear referencing system.  Another is the Update Measures From LRS tool.  This tool populates or updates the measures and route ID on utility network features such as pipes, devices, and junctions.


Taken together, these capabilities further strengthen the data management integration essential to the new pipe network modeling era ahead.


Benefits Of Both Extensions Working On The Same Data

The ability for the ArcGIS Pipeline Referencing and the ArcGIS Utility Network Management extensions to work on not just the same geodatabase, but the same feature classes within the same enterprise geodatabase, provides important benefits to all transmission pipe network operators. This is the case regardless of whether the transmission pipe is part of a standalone or vertically-integrated operator. Those benefits are amplified as the ability is enabled by industry-specific capabilities in Esri’s core technology.  


The year ahead promises to be a rewarding one as pipe network modelers take advantage of these new capabilities!


PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions.

The 2019 GeoConX meetup held at the Cobb Galleria Centre in Atlanta, Georgia, saw the largest number of utility and telecom GIS professionals ever gathered to share their work, collaborate on new projects, and discuss new ways of leveraging GIS and location intelligence to support utilities.  The theme was: Geo-Enabling the Intelligent Enterprise.


GeoConX 2019 had its highest registration ever with 1,192 attendees from 414 different companies,  a 25% increase over last year. This year’s conference also included the AEC Summit to kick things off and concluded with the Water Summit.


The event kicked off with a half-day opening plenary session featuring geospatial thought leadership from Jack Dangermond, CEO of Esri, along with ArcGIS user presentations and ArcGIS technology updates.


Highlights from the plenary included our local “host” utility – Southern Company’s use of GIS across gas and electric business units with over 4,800 named users. They easily share information and increase operational efficiencies while also bringing new business to Georgia. Energy Queensland made a fascinating presentation on their Look Up And Live application which is routinely reducing accidents and saving lives in Australia.


Undoubtedly the most unique part of the plenary was a fun “magic show” by Esri’s Bill Meehan and Brian Baldwin demonstrating the capability of real-time IoT sensor integration in ArcGIS (no actual magic required).  The opening session really set the energy for the rest of the week and there was a lot of buzz around improving common utility workflows. Here are a few highlights from the week.


Peer Connects 

Connecting with peers is what GeoConX is all about. Each of the various peer-connects sessions were well attended with excellent discussions of timely topics.

Esri Technical Sessions

Esri staff engaged attendees with technical presentations on a long list of interesting topics. These included: Utility Network, Machine Learning, Understanding Customers with Business Analyst, Gas, Electric, Administration Tips & Tricks, System of Engagement, and System of Insight.


User Paper Sessions

Throughout the week, many users of Esri’s ArcGIS shared how they are Geo-Enabling their Intelligent Enterprises. There were so many good presentations it was often difficult to decide which one to attend. Here is just a sampling of the Sessions:


Gas – Integrating Enterprise Systems, Improving Data Quality, Risk and safety, Improving Field Facility Data, Field Operations, Asset Management, Improving Data Quality,


Electric – Emergency Management, Utility Network Migration, Utility Networks in Production, Grid Modernization, System Operations, Field Mobility, Asset Management, Field Operations and Analytics.


Tech Updates, Hands-on Learning Lab, and Data Health Check

Numerous new updates to Esri technology were shown at GeoConX and following the positive feedback of the hands-on learning lab last year, the lab was brought back this year and even more Esri products were available for attendees to try out and play with, and training courses were available for attendees to work through while at the event. The Data Health Check-up team took appointments to review and analyze customer GIS data, focusing on features and attributes and made specific recommendations.



New Tech Highlights:

  • Machine Learning Tools An update to the machine learning tools in ArcGIS was shared in a session that focused on spatial tools for classification, clustering, and prediction. Some of tools shown were Random Trees, Density-based Clustering, and Geographically Weighted Regression. Also, show was the integration of ArcGIS with external machine learning frameworks like TensorFlow and Scikit Learn. Image detection for detecting features in imagery, such as poles and sidewalks, gained a lot of interest from fiber planners.
  • Field Apps– The demonstrated Esri field apps showed how you can coordinate field activities using Workforce, how to efficiently get to the location of work using Navigator, how to gain spatial awareness and mark up maps using Explorer, how to accurately locate, capture and inspect assets using CollectorSurvey123, and QuickCapture, and how you can improve accountability and enhance situational awareness using Tracker and Operations Dashboard.
  • Sensors, Big Data, and Analytics– Highlighted in this session was the ability to track field personnel as sensors, consuming their location with GeoEvent Server for visualization, geofencing, and storage for improved field operations and increased safety. GeoAnalytics Server was highlighted to help with the analysis of large collections of sensor data. Finally, a new Esri product in development was introduced: ArcGIS Analytics for IoT. This is a SaaS product that combines capabilities of GeoEvent Server and GeoAnalytics Server into a scalable, cloud-based product.
  • Business Analytics– New updates to ArcGIS Business Analyst were shown in a session that highlighted ways to improve customer engagement leveraging Esri Demographics . A crowd favorite was the improved dynamic infographics that can be configured and generated from apps across ArcGIS.


GeoConX Expo

Throughout the week, attendees had the opportunity to meet with Esri teams, including solutions engineers and product managers in the GeoConX Expo. Esri staff and representatives from over 60 Esri Business Partners presented solutions and answered questions. The floor was very active and fun this year with great snacks and a conversational tone that many really enjoyed.



Join GeoConX Next Year!

This year’s GeoConX was another great meetup for utility GIS professionals, and we look forward to keeping the conversation going throughout the year, and seeing everyone at GeoConx 2020 in Denver, Colorado.  Be sure to stay engaged with the community on GeoNet and follow us on Twitter @EsriElectricGas and on LinkedIn.

By Tom Coolidge and Tom DeWitte

Earlier this year, I was walking down a street in mid-Manhattan. During the walk I noticed a surprising amount of steel plates covering active excavation projects under the streets and sidewalks. There were also many open trenches which exposed an amazing number of pipes and wires.  Not being from New York City, I was surprised at the shear volume and complexity of buried infrastructure. This got me to pondering about recent gas events and the questions an organization asks itself.

-Could the damage to the pipe system have been prevented?

-What organization processes could have performed better to prevent the damage?


This leads to an organization self-examination of processes to determine where improvements need to be made. These self-exams are important and valuable, but they are also reactive. 

Typical reactive questions which are commonly asked include: 

  • Did the locator group correctly mark the location?
  • Was the locator provided with accurate and timely information to assist with the locate?
  • Were the correct engineering and construction documents provided?


These reactive self-exams often overlook a root cause: inaccurate data gathering methods.


The American Gas Association recently published a white paper titled “Implementing Damage Prevention in Field Operations.” The white paper can be accessed at:


Included in the white paper is a section on “Mapping/As-Builts” that resonates with important aspects of the role of geographic information systems (GIS) in gas utilities. This blog touches on those important aspects.


Inaccurate data gathering with relative offsets

The AGA whitepaper allocates two of the nine sections to address the importance of data gathering techniques around the construction and mapping/as-built processes for documenting “where” the pipe components were installed. Inaccurate data gathering methods have persisted in the gas industry since its founding. In early gas company days, locating a buried pipe was based on “relative offsets”.  A relative offset is a measurement from an identifiable above ground feature to the location of the buried pipe asset.  It was not uncommon to see measurements, such as 45ft NNE of the Old Oak Tree. But what happens when the tree falls over after a storm and is removed?


Later relative offset techniques switched to using street curb edges or house corners, such as 12ft South of the North curb.  This too is subject to change over time.  How accurate is this measurement when an additional lane is added to the north side of the street?

As pointed out in the previously mentioned AGA white paper, these methods are inaccurate over time.  Yet, many in the industry are still using these centuries-old methods to document the location of newly installed pipe.


How to proactively prevent damage

Now there is an opportunity to be proactive.  To use current cost-effective data gathering tools such as Global Positioning System (GPS), lightweight digital collection tools such as mobile phones and tablets, and data management systems such as GIS.  This allows the location of buried assets to be based on its absolute location instead of its relative location.  Using absolute location eliminates the historical issues of relative features changing over time.


An opportunity to improve

Today’s gas organizations are undergoing a massive capital improvement process to replace or abandon many old and inaccurately mapped gas pipe components. This increase in capital projects provides the opportunity to accurately locate new infrastructure and decrease future damages.  Gas organizations can update data gathering methods to cost-effectively collect new construction with a sub-foot absolute accuracy. They can implement a geospatial system that is capable of electronically collecting and storing this information. Future engineering, construction, and locate organizations will have the confidence in knowing that their mapped pipe components are spatially accurate and reliable.


If you haven’t already read the recently published AGA white paper, we encourage you to do so. It’s well worth your time. There are many other points worth noting.


The next time I visit a major city and again see the steel plates and excavation pits, I hope to have the confidence in knowing that the local gas organization has been proactive and is using current data gathering tools and techniques to prevent future damage.

PLEASE NOTE: The postings on this site are my own

and don’t necessarily represent Esri’s position, strategies, or opinions.

By Tom Coolidge and Tom DeWitte


It’s always a joy for us to see the amazing work our customers are doing with ArcGIS for the betterment of their organizations and those they serve. It’s even sweeter when a customer is in a position where they can freely share that work with others so they, too, can put it to work. That’s the case with GTI and the work they have been doing to create Survey123 templates for the natural gas industry.


For those of you who may not know, GTI is a leading non-profit research, development, and training organization addressing global energy and environmental challenges. Among their hundreds of initiatives across the energy value chain, they develop and implement tools, methodologies, and technologies for maintaining a safe and intelligent natural gas infrastructure. The GIS department within GTI has been working to encourage electronic field data collection with GIS to optimize the entire data management process for utility and pipeline operations, significantly reducing the cost and complexity of capturing real-time high-accuracy information.


With these common goals in supporting the natural gas industry, Esri and GTI collaborate to help natural gas utilities be successful in their efforts to implement and leverage geospatial solutions to address industry business challenges. More specifically, recently, we have been looking at ways to make it easier for the natural gas industry to use the robust geospatial tools that are available today.


Through this collaboration, GTI now is publishing Survey123 form templates that they have created. The first survey form to be published is Indoor Gas Meter Set Risk Assessment. This form assigns a risk score to indoor meter set evaluations in real time based on user input.


The capabilities embedded in this first template are impressive. It includes examples for:

                -Using HTML tags to set text color

                -Performing calculations based on the user’s response to the questions.

                -Hidden fields

                -Context driven logic

                -pick lists


You can review this survey123 template yourself following these steps:

  1. Open Survey123 Connect

   If you do not already have Survey123 Connect, you can download it from this location:


   2. Click on the “New Survey” option


3. Within the New Survey dialogue, select the “Community” radio button


4. Scroll through the list of posted surveys to find the survey named: GTI/OTD – Indoor Gas Meter Set Risk Assessment


  5. Select GTI survey and click on the Create Survey button


  6. Review the newly created survey


The indoor meter set risk assessment is the first of many surveys to be posted by the GIS team at GTI.  These survey templates will help natural gas organizations of all sizes to deploy this powerful mobile data collection application.  Take a look at these surveys and see if they can help your organization improve its data collection activities.


PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions

By Tom Coolidge and Tom DeWitte


Today’s Collector and ArcGIS Enterprise provide new enhancements and capabilities. These enhancements include; improved user interface, better GPS antenna support, direct capture of barcode via the mobile device camera, and allow for a more streamlined workflow for field users.  In addition to those important enhancements, the enterprise geodatabase capability of attribute rules allows for the automatic decoding of the barcode and the derived barcode data to be automatically written to the appropriate attributes.  This automatic decoding and attribute population provides significant productivity gains for field users and allows for a simpler deployment pattern for administrators. In this blog we will take a deeper dive into how to configure and deploy the ArcGIS platform and collector to address the industry need of Tracking and Traceability.


For an introductory explanation of how the ArcGIS platform addresses Tracking and Traceabiliy, please read the first blog of this 2 blog series: Tracking & Traceability – Part 1


Like any good recipe for success, we need to know the required ingredients.  The Tracking and Traceability solution requires the following software:

  • Collector for ArcGIS
  • ArcGIS Enterprise 10.6.1 or higher


Additionally, we will need arcade scripts which provide the logic of how to decode the ASTM F2897 barcode 16-character string and use the derived data to automatically populate the appropriate attributes.


Though not required, most deployments also include a GPS Antenna to improve spatial accuracy.  


The Basic deployment steps

Deploying the ArcGIS Platform to meet the needs of Tracking and Traceability can be broken down into 5 steps.  These steps are:

  1. Preparing the enterprise geodatabase
  2. Creation of staging geodatabase layers
  3. Application of attribute rules
  4. Publication of staging geodatabase layers as a feature service
  5. Creation of web map for Collector


The overall data flow process for Tracking and Traceability is to have Collector post the field collected features directly to the staging geodatabase.  There is NO translation or conversion of the field collected data.  Once the field collected data is submitted to the staging geodatabase a GIS mapping technician can review the new features and append them into the enterprise geodatabase.

Preparing the Enterprise Geodatabase

The first step to setting up this workflow is ensuring your Enterprise Geodatabase has the required feature classes, feature class attributes and coded value domains to store the information collected in the field.


If you are starting a new enterprise geodatabase, it is recommended that you use the Esri provided pipe system data model called Utility and Pipeline Data Model (UPDM). The 2019 edition of UPDM includes everything needed to store the information collected in the field. You can download this data model with this link:

 UPDM 2019 Edition download


 If you have an existing enterprise geodatabase, then you need to make sure the asset feature classes have the correct attributes to store the field collected data. Examples of assets captured by field staff include, fittings, valves, and pipe segments. Here is a specific listing of the minimally required attributes:


Point Asset Featureclasses

Field Name

Field Definition

Coded Value Domain








Long Integer







Pipeline_ASTM_Manufacture Component

















Line Asset Featureclasses

Field Name

Field Definition

Coded Value Domain








Long Integer







Pipeline_ASTM_Pipe_Manufacture Component









Pipeline_Pipe_Wall Thickness


After your Enterprise Geodatabase is ready to accept the decoded barcode values and the appropriate ASTM F2897 coded value domains have been assigned, you are ready to create the staging geodatabase.


Creating the Staging GDB

This step involves setting up your staging geodatabase layers.  These layers should be a schema duplicate of the enterprise geodatabase asset layers. Being a schema duplicate will simplify the appending of data from the staging geodatabase to the enterprise geodatabase.


The simplest approach to setting up the staging geodatabase is to create schema duplicate feature classes in the enterprise geodatabase.  I recommend creating a new feature dataset to store these duplicate layers.  If using the UPDM 2019 edition data model the feature classes to duplicate are:

  • PipelineDevice
  • PipelineJunction
  • PipelineLine

To help keep the staging layers uniquely separate from the production layers I like to rename the layers as follows:

  • StagingDevice
  • StagingJunction
  • StagingLine

These duplicate layers should not have any features/records.


To properly support disconnected field capabilities, you should use the “Add GlobalID” tool to add a GlobalID field to every staging feature class.


Additionally, though not required, it is recommended that you enable “Editor Tracking” to allow all edits to have a date/time stamp and the ArcGIS platform user ID of who created and last updated the feature/record.


A final step not to be overlooked is to decide whether you want to include photos as part of the new construction data collection process.  It the answer is “yes” then remember to “Enable Attachments” for each of the layers you want to have field staff capturing photos.


With the staging geodatabase layers now created it is time for attribute rules.


Application of attribute rules

With ArcGIS Enterprise 10.6.1 the attribute rule capability has evolved to provide a robust automation capability for managing attributes. For Tracking and Traceability, attribute rules provide the ability to automatically read the barcode value, decode the barcode and automatically populate the derived attribute fields (manufacturer, manufacture lot #, manufacture component type, manufacture date, material, diameter, and wall thickness). When this capability is applied to the staging geodatabase layers, the auto-population occurs when Collector submits the new feature.  This means a connected mobile device running Collector to capture new construction will be able to see the decoded information while they are documenting the new assets in the field.


The following link provides the arcade attribute rule scripts and detailed documentation on how to apply them.

ASTM F2897 barcode decode attribute rules 


The way attribute rules work is to assign them to a single attribute field. This means the decoding of the barcode is broken out into 9 separate arcade scripts.  Here is a breakdown of how the arcade scripts are applied to the staging geodatabase layers. 


StagingDevice Featureclass

Attribute Fields

Arcade attribute rule script




















StagingJunction Featureclass

Attribute Fields

Arcade attribute rule script




















StagingLine Featureclass

Attribute Fields

Arcade attribute rule script
















Once the attribute rules are successfully applied to your enterprise geodatabase staging layers you are ready to publish the staging layers as a feature service.


Publication of staging geodatabase layers as a feature service

Publishing the staging layers from ArcGIS Pro is a very straight forward process. The steps are as follows:

  1. Create a new Map
  2. Add staging gdb layers to map
  3. Symbolize layers as desired
  4. Publish map as a feature service

After the map is created and the staging geodatabase layers are added to you map you will have a ArcGIS Pro map which looks like the following:

I find using ArcPro for defining the symbology to be easier and quicker than using the ArcGIS Enterprise Portal map viewer tools.  Additionally, I can use more advanced symbology such as the UPDM2019_Symbols style set that is included in the UPDM 2019 Edition download.  When the layers are symbolized as desired, remove the basemaps and prepare to publish.

To publish the staging layers as a feature service, use the sharing ribbons’ web layer – Publish Web Layer tool to create the feature service.

With the feature service now published your staging geodatabase layers are ready for the final step which is to create the web map for Collector.


Creation of web map for Collector

Creating a web map for Collector is the opportunity to fine tune the interface your field staff will use for documenting the new construction.  Items to think about when creating the web map are:

  • Scale Constraints of layers
  • Which data fields will be exposed to the field staff
    • Which fields will be exposed during editing
    • Which field will be exposed during viewing

Both the ArcGIS Enterprise portal map viewer or the ArcGIS Pro desktop tool can be used to accomplish this task.


When the web map is defined and saved you are now ready to take Collector to the field to being collecting your new gas pipe construction.



With the latest enhancements to Collector and the new attribute rule capability for enterprise geodatabases. Deploying the ArcGIS platform to address the needs of tracking and traceability is easier than ever. Five basic steps  are all that it takes to enable your field staff to efficiently capture new construction digitally and retire the time consuming and inefficient historical paper based process.

  1. Preparing the enterprise geodatabase
  2. Creation of staging geodatabase layers
  3. Application of attribute rules
  4. Publication of staging geodatabase layers as a feature service
  5. Creation of web map for Collector


PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions

By Tom Coolidge and Tom DeWitte

Tracking and Traceability is now a well-established practice in the natural gas distribution industry supported by ArcGIS®.


ArcGIS mobile app advances over the last three years have helped adoption of Tracking and Traceability activity grow. Collector for ArcGIS has evolved to now include the ability to use a mobile device’s camera to read the ASTM F2897 barcode. Collector also now includes the capability to run arcade scripts in the pop-up window while the device is disconnected from the network.  Not to be overlooked, Esri also released a new enterprise geodatabase capability called attribute rules.


Those three new capabilities have enabled many gas utilities, and increasingly gas pipe installation contractors; to use Collector to capture the location, barcode, and other information about the newly-installed pipe and its related components. These new capabilities and lessons learned from the many organizations actively using Collector for the digital as-builting portion of the Tracking and Traceability workflow have resulted in a more efficient and streamlined process for performing these tasks.


The purpose of this blog is to give an overview of how the current version of Collector, when combined with an ArcGIS 10.7 or higher enterprise geodatabase, can result in a simpler and more efficient Tracking and Traceability workflow. A second blog article will follow with a detailed explanation of the new attribute rule arcade scripts which completely automate the decoding of the ASTM F2897 barcode and the automatic population of the derived attributes.


A quick review of Tracking and Traceability

PHMSA proposed rules in May of 2015 to 49 CFR part 192 to address the need for operators to better ‘track’ the details and location of assets after their delivery from the manufacturer or supplier.  The rule also speaks to the need for better ‘traceability’ of assets; meaning the ability to locate assets by material, size, manufacturer, model, or other attribute.


The ASTM F2897 standard, developed collaboratively by the natural gas industry and its leading suppliers, specifies a 16-digit alphanumeric barcode format that embodies identification of a pipeline component’s manufacturer, lot number, production date, model, material, diameter, and wall thickness.  This barcode standard is now a common piece of the manufacturer provided information for plastic pipe and its plastic components.  Additional efforts spearheaded by the Gas Technology Institute are currently underway to define a more advanced barcode standard which can be applied to both steel and plastic pipe and their components.  This barcode “thing” is not going away.  Just the opposite, it is going to expand significantly in the years to come.


Pattern Overview

The ArcGIS deployment pattern for Tracking and Traceability is comprised of four steps, as illustrated here:



Step 1: Digital as-builting

The recent improvements to Collector have made this process easier than it was just a few years ago.  The first enhancement was the revamping of the interface to simplify data entry. The second enhancement was to increase the certification of GPS vendors and their devices. Here is a link to the list of GPS receivers which can be used with Collector:

The third enhancement is the native ability of Collector to use the mobile device’s camera to capture the ASTM F2897 barcode.


With these enhancements, field staff can go into the field and capture the as-built information of the new construction using a smart device running Collector. The smart device is Bluetooth-connected to a high precision GPS antenna.  The field staff use the high accuracy GPS antenna to capture the location of the newly installed assets. The collected location data is directly streamed into Collector as native ArcGIS features.  No translation or conversation is required.  The field staff then manually input into Collector a minimal amount of information, such as Installation Date, and installation method.  The field staff then uses the device’s camera to capture the barcode and automatically populate the BARCODE attribute of the GIS feature.  The BARCODE value contains information about the asset, such as size, material, manufacturer and manufacture date.  Once the BARCODE value is captured, the field staff no longer need to manually enter this information.


The recent enhancement to Collector supporting the ability to run arcade scripts in the pop-up window, provides the ability to immediately display the decoded data to the field staff even when the device is disconnected.


An additional capability of an Esri mobile app on a smart device or tablet is the ability to capture photos of the newly installed assets.  These photos are automatically associated to the GIS feature.


When the field staff have completed the collection of the newly installed assets, the GIS features are submitted to the staging geodatabase.


Step 2: Contractor/crew assessable storage

A fundamental challenge of Tracking and Traceability is how to correctly integrate high precision GPS geospatial data, with less accurate legacy geospatial data.  A key component to overcoming this challenge is the staging geodatabase.  A staging geodatabase can be either hosted in ArcGIS Online as hosted feature layers or stored on premise with a local ArcGIS Enterprise implementation. The key purpose of the staging geodatabase is to provide an easily accessible data repository for the field crews to submit their collected construction information too.  The staging geodatabase only holds the newly collected construction information.  The construction data sits in the staging geodatabase until a mapping professional using ArcGIS Desktop accesses and downloads it to the enterprise geodatabase.


With the new enterprise geodatabase capability of attribute rules, it is possible to have the captured barcode value automatically read and used to auto-populate the derived attributes manufacturer, lot number, production date, model, material, diameter, and wall thickness.  If the digital as-builting described in step 1 happens while the device is connected to the enterprise geodatabase, then Collector will automatically decode the barcode, auto-populate the derived attributes and display the decoded information immediately after the new/updated GIS feature is submitted by Collector. In the second blog, we will provide links to these arcade scripts and describe how to apply them to an enterprise geodatabase.


Step 3: Append to enterprise geodatabase

One of the time saving capabilities of ArcGIS Desktop is the ability to interact with data from both the staging geodatabase and the enterprise geodatabase at the same time.  This allows the mapping professional to easily select the staging geodatabase features and append them into the final enterprise geodatabase feature classes. 


If the staging geodatabase layers are stored in ArcGIS Online, the previously described attribute rule arcade scripts can be applied to enterprise geodatabase layers. 


NOTE: Attribute rules only work with ArcGIS Enterprise 10.7 or higher. Additionally, ArcGIS Pro is the only desktop tool to understand attribute rules.  If using ArcMap and a geometric network, it is important that the staging geodatabase layers be stored in an enterprise geodatabase and the attribute rules are applied to the staging geodatabase layers.


The standard arctoolbox geoprocessing append tool can be used to copy the newly collected GIS features from the staging geodatabase layers to the final enterprise geodatabase feature classes.


Step 4: Mappers connect digital as-built with gas system

With the new construction data now appended from the staging geodatabase into the enterprise geodatabase and the barcode value decoded, the mapping professional now needs to determine how to connect the high precision geospatial features with the less accurate geospatial features. The outcome of this process needs to honor two data requirements:

  • Connecting the new features with the legacy features to create a single topologically connected gas pipe system.
  • Preserving the high precision GPS collected geospatial coordinate data.


The recommended best practice for accomplishing this seemingly disparate set of requirements is for the enterprise geodatabase point features such as Meters, Excess Flow Valves, and Non-Controllable Fittings to have the following attributes added: SPATIALACCURACY, GPSX, GPSY, GPSZ.  Here is another example where attribute rules can streamline the population of these GPS fields.  If using ArcMap and the geometric network, then a configuration of Esri’s Attribute Assistant tool or ArcFM’s AutoUpdater capability can be used to automatically populate these fields.  This will preserve the original GPS location values, which can be used later to rubbersheet all features (legacy and GPS) to the more accurate GPS location preserved in the GPSX, GPSY, and GPSZ attributes.  With the GPS location preserved, the mapper can adjust the new construction features as required to connect to the legacy gas pipe system.


Business value of using ArcGIS platform

This approach to Tracking and Traceability provides an opportunity for the GIS department to once again show the greater gas organization that not only can the GIS Department provide a solution which addresses this new common industry practice, but it can do so in a manner that improves the operational efficiency of the gas organization.  This pattern improves the operational efficiency of the gas organization and their contractors as follows:

  • Using Collector to collect construction data improves location accuracy and attribute quality by eliminating translation to paper and interpretation of paper based information.
  • Bluetooth integration with high precision GPS antennas improves the speed at which data is collected.
  • Capturing the barcode value reduces the amount of information the field staff manually collects. Material, diameter, manufacturer, manufacture model, manufacture data, manufacture lot number are all automatically populated by the decoding of the barcode.
  • Digitally collected data is immediately available for GIS department to process into enterprise geodatabase. This eliminates the historical latency problem of the GIS department waiting for the inter office mail transmittal of the construction packet.
  • The GIS department mapping professional task of updating the as-built representation of the gas pipe system is simplified. The mapper is no longer manually transposing paper based red-line drawings, but instead appending field collected geospatial features.  This improves the speed at which a mapper can complete the task of updating the as-built representation of the gas pipe system.
  • Safety of field operations staff is improved by providing the new construction data in a timelier manner.


This deployment pattern not only provides the ability to improve the efficiency of the field data collection, it improves the productivity of the mapping professional, and provides new construction updates to locators and field operations staff in a timely manner.


Next blog

In our next blog, we will dig into how to configure and deploy the arcade scripts for this solution to Tracking and Traceability.


PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions.

***This workflow works with etlsolutions v0.5.2***

Click Here to learn more about benefits and how to get started with the Data Translation Tools


We have had requests for additional documentation and samples at the UC. A zip pro package has been attached to the thread.


We want to bring data from one file geodatabase to another file geodatabase. This is also known as Extract, Transfer, and Load (ETL). Let me set the stage.


We have a file geodatabase or fgdb for short. Let's call this fgdb, our source.  Inside this geodatabase, we have a feature class named Cars.  We have three subtypes in this feature class: Red, Blue and Green. There are domains assigned to each of these subtypes. I want all my data in this feature class to shuttle over to my target file geodatabase. This target fgdb has a feature class called Trucks with different schema. Schema is a term that includes attributes, data types, domains, and other data management concepts. The Trucks feature class has three subtypes as well, but they are different than those found in Cars. They are One, Two, and Three rather than Red, Blue and Green.


There are three ways to translate data with the workbook. The first method does a straight field mapping without any domains. The second method uses a sheet as a lookup table for domains. Finally, the last method uses a hard coded value to ‘burn in’ values. I will cover each method in this blog.




The first tool in our Data Translation toolbox is Create Mapping Workbooks. 

  • Open the GP Tool Create Mapping Workbooks tool within Data Translation toolbox.
  • Point each one of your source feature classes to its corresponding target feature class.
  • Specify the output folder for your mapping workbooks.
  • (Optional) Check the box to Calculate feature count statistics to generate information on what fields have populated data.




Once the tool has run, you will see a Points folder that contains a mapping workbook called Cars and a mapping.xlsx file which will be used later when using the Load Data from Workbook tool. Let's go into how to populate information into these excel workbooks.


Method One: Straight Field Mapping


Step 1:

Open the Cars workbook and navigate to the Mapping sheet.  We have columns for targetField, sourceField, and fieldType which are all system derived. The columns, expression, sheet, sheetKeys, and sheetValue are used for methods two and three. This default workbook is configured to transfer data to your new file geodatabase without any translations.




What this means is the domains will be mapped using their codes. For example, Red will not be mapped as anything since there isn't a 0 code in the domain "Type" of our target feature layer. Blue will be mapped as One since they share the domain code of 1. Green will be mapped as Two since they share the domain code of 2.


Step 2:

Before we use the Load Data from Workbook tool to Extract, Transfer and Load our data, it's time to inspect our mapping.xlsx workbook.


Each row in this workbook directs to the tool to: set the source database, set the target database, and set the lookup workbook. This first row was created when we pointed our Cars feature class to our Trucks feature class during the Create Mapping Workbooks section at the beginning of this workflow.



Step 3:

Now that I have inspected everything, it's time to run Load Data from Workbook tool.

  • Open the Load Data from Workbook tool in Data Translation toolbox.
  • Specify the location of the mapping workbook created from Create Mapping Workbooks tool.
  • (Optional) Truncate the target geodatabase before loading.




When we run this tool, we return values of 0, One, and Two.



Method Two: Field Mapping with Lookup


Step 1:

Open the Cars workbook and navigate to the Mapping sheet. In this sheet, we will define what sheets and columns to use as a lookup table. Since I am not using the sourceField for a straight mapping, I will remove Type from the sourceField column In this sheet, the values I have entered are highlighted. In the sheet column, "Type", highlighted in yellow, is used by the Load Data from Workbook tool to locate the correct excel sheet when mapping to Type.  In the sheetKeys column, "Type", highlighted in yellow, is used by the Load Data from Workbook tool to locate the correct column(s) on the Type tab when mapping to Type. In the sheetValue column, "NewType", highlighted in orange, is used by the Load Data from Workbook tool to locate which column to use as a lookup table on the Type tab.



Navigating to the Type tab, I have highlighted the Type column in yellow to show the relationship to the user-defined column in keys on the mapping tab previously mentioned.  I have also highlighted the user-defined column and values in orange. I entered the domain values 1, 2, and 3 in column C to map to Red, Blue, and Green, respectively.  I have also included their domain descriptions in column D. What I have accomplished in this workbook is creating a lookup table for old domain values to new domain values. Where Red Car was using a domain value of 0 to describe Red, Truck One uses a domain value of 1 to describe One.



Repeat steps 2 and 3 above from Method One


Here is our final output.




Method Three: Hard Code Burn-in


Step 1.

Open the Cars workbook and navigate to the Mapping sheet. In this sheet, remove all the values you entered in Method 2. We will simply enter in a hard-coded value of "3" into the expression column. This will burn-in the value of 3 for Red, Blue, and Green.



Repeat steps 2 and 3 above from Method One


Here is our final output.


In his book The Road Ahead, Bill Gates said, "We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten. Don't let yourself be lulled into inaction." We are already seeing increased complexity in electric distribution circuits. If Gates is correct, we will see much more in the coming years.

Electric utilities historically operate their distribution systems in sections called circuits. This blog series looks at some important characteristics of the circuits of the future and how they may differ from those of the past and the present. These differences are acting to convert our familiar circuits into a network that relies on electronics and data for routine operation. Are you setting your system up for success in the face of these changes?

Part 1 of this blog introduced some fundamental differences between circuits and future networks. Part 2 examined why networks must be capable of being split into smaller parts. This final segment will consider the network's greater complexity and its need to handle rapid changes.

Greater Complexity

The sheer volume of electric system devices is going through the roof—microgrids, distributed generators, smart inverters, sensors, and automatic switches all bring greater complexity. These devices are more sophisticated than most of the circuit devices commonly in use today.

Arguably the most common device on a circuit today is a fuse—a skinny piece of wire that, as its sole operation, burns up! An important insight is to realize that most circuits, on their journey to become a network, are starting from a very low level of sophistication.

Sophisticated devices have more connection points, bypass functions, and test provisions. In addition, they are often configurable, integrate with communication systems, and exchange parameters with other devices and systems. These parameters help govern equipment settings, price signals, and protection from harmful conditions. Much of this complexity is linked to modern electronics that consume data in real time.

Smaller network pieces and sophisticated devices complicate routine operating decisions. Formerly simple manual operations, like opening an overhead pole switch, will be initiated remotely with the use of a new and vastly more sophisticated switch. Instead of simply verifying adequate electrical capacity and switching from one circuit to another, operators will need to understand how each of these changes affects the entire network.

I investigated several high-voltage accidents while working for utilities. Two of the worst injury accidents had their root cause in misidentified energized equipment. In tight spaces, like substations or underground structures, complexity brings the need for more equipment which takes up precious working space. More equipment and less space makes safe operation that much more challenging. To work safely, the data and information systems supporting these new networks must also accommodate their greater complexity and detail.

Rapid Changes

Traditional circuit layouts tend to be static, changing only for specific tasks or between summer and winter configurations. Dispatch and field personnel often have them nearly memorized. A gray-haired supervisor may confidently tell a new apprentice, "That transformer is on circuit number 121—it feeds from Buckingham substation up on the hill," speaking on the assumption that the circuit's characteristics remain constant.

Self-healing capabilities such as reverse power flow and the use of automatic switches and microgrids can all change networks rapidly and without much warning. They can alternate in response to different conditions in a short period of time. Traditionally, to alert employees, such changes are announced over the operation's two-way radios mounted in work trucks. Advanced network changes may occur with little or no human interaction, and without radio announcement. This real-time operating paradigm sparks different work procedures and safety concerns because such rapid changes were not normal in the past.

Staff are not used to their circuits changing quickly. They are accustomed to referencing their relatively static maps. Historically, a period of weeks to apply map updates was acceptable. But now, last month's map products from the Maps and Records division will be simply inadequate to meet the real-time operating needs of new networks. All users, in the office and the field, will need more detailed information in near real time.


Is there a time coming when we won't even think of circuits at all? Probably, but not in the immediate future. For decades, circuits were the only source of power to the distribution system. Today, every rooftop solar installation is another source to consider. The circuit at the substation may not be the only source on the network, but it will certainly remain important for quite some time.

Many of the standards necessary to implement a smarter network are still under development. Given all the forces acting to change circuits into networks, prepare for a continuous evolution of equipment and capability. When you don't know exactly what will be required, flexibility is a key strategy.

New functions will continue to be added, improving our ability to optimize distribution operations for power quality, cost, and reliability. Grid modernization and circuit evolution also mean a great deal of physical work, building networks and systems to support them.

Because networks of the future will be controlled with electronics and data, the underlying information models and systems will be foundational to success. Like Bill Gates said, "Don't let yourself be lulled into inaction." The ArcGIS platform is specifically designed to help utilities model and operate these new complex and rapidly changing networks.

For more information on how the ArcGIS platform helps electric utilities manage advanced networks, visit our site.Advertisement

In electric utilities, we are really attached to our circuits. Get ready- those circuits are going to change, and as we progress, may become unrecognizable! Circuits are so embedded in our culture that a colleague once remarked that we are fixated on them!  He was right. We are fixated on circuits, and with good reason.

Electric utilities typically operate their distribution systems in pieces called circuits or feeders. We map by circuit, patrol facilities, trim trees, and report statistics by their circuit name or number.  As the utility industry changes, circuits will need to morph into more of an interconnected network. “Network” is a much better term than “circuits” to describe the future state. Is your utility preparing to successfully operate the network of the future?  

Part I of this series introduced some fundamental differences between traditional circuits and future networks.  This part II will examine the first difference – smaller pieces.

Smaller Pieces

In the future, electric networks will need to break down into smaller pieces for greater operating flexibility. Utilities established the sections of today’s circuits to reduce the customer impact of power outages, perform maintenance tasks, and supply large blocks of customer load. Utilities now need additional flexibility to accommodate different types of both customer usage and power generation. Distributed generation, including solar and wind, is steeply on the rise.  These variable resources bring constantly shifting power flows to circuits that were only designed for one-way power flow. Networks need smaller pieces with flexibility to handle variable power flows.

Electric vehicles can plug-in anywhere moving their electricity demand around like a big 2-story house on wheels! Yesterday’s large stable blocks of customer load are becoming less consistent and are now driving from one place to another.  Networks must be more configurable than circuits to meet the needs of tomorrow’s customers.

Smart-grid technologies too will drive networks to operate in smaller sections. Self-healing networks use smart switches to sense real-time conditions. In the blink of an eye, they communicate with other devices and compare observations. Together they determine when a power problem occurs and limit customer impact with instant automatic switching.  As utilities implement more self-healing capability, the pieces of the network will get smaller enhancing customer value by improving reliability.

It’s clear the operating pieces of the network will be smaller than those of a typical circuit today, delivering sorely needed operational flexibility.

Wrap up

New devices will split the coming network into smaller pieces. Because networks will be controlled with electronics and data, the backbone of this evolution will be a central data model of the entire system. This feature-rich model may be called a digital twin.  A fully functional digital twin, adequate to support disparate utility roles, is a big step from the straightforward facility mapping models of the past.

A digital twin should be sophisticated enough to represent each device accurately at its precise location. Device location on the network will guide its every operation.Smart utility operations begin and end with this location intelligence.

We are already seeing the writing on the wall as pilot projects adapt existing circuits to accommodate new unconventional devices. Common distribution circuits will evolve into a more robust network.  This flexible network must consist of smaller pieces, include numerous new complexities, and change quickly in response to system and customer needs.

The ArcGIS platform gives all stakeholders the ability to access and share critical network information. Advanced network information will have to become embedded in our “circuit culture” as it evolves.   The ArcGIS Utility Network Management extension is specifically designed to handle the smaller pieces and greater detail of the advanced electric networks now on the horizon.  

For more information on how the ArcGIS platform helps electric utilities manage advanced networks, visit our site.