Within the world of documenting new construction projects for utilities, there is a common practice that is needed but very inefficient. This is the practice of using AutoCAD to document the construction project and ArcGIS to document the current as-built state of the entire utility system. The problem with this dual documentation workflow is that the AutoCAD editor and the ArcGIS mapper perform the same data edits. The ArcGIS for AutoCAD plugin provides a solution to this redundant data entry.

In Roman times, March 15th was the deadline for settling debts. Today in the United States, March 15th is the deadline for submitting annual reports for gas distribution and transmission companies to the U.S. Department of Transportation (U.S. DOT) Pipeline and Hazardous Materials Safety Administration (PHMSA). 

Officially, PHMSA estimates that the time required to gather the data needed to complete these forms is 16 hours for Gas Distribution and 47 hours for the Gas Transmission and Gas Gathering form. This is achievable with a well-attuned geospatial dataset such as an ArcGIS data repository organized with the Utility and Pipeline Data Model (UPDM). If your data is poorly organized, this effort can take several months.

Benefits of Modern Network Management for Gas

By Kevin Ruggiero & Tom DeWitte

PHMSA requires all gas and hazardous liquid system operators to maintain accurate and up-to-date records and/or maps of their pipe system. Records must include tests, inspections, and attributes required by the manufacturing specifications applicable at the time the pipe was manufactured or installed. Managing this information is foundational to pipe organization’s “knowing their systems”.

To truly “know your system” is more than simply managing information about pipes on a map or within a database. The organization needs to understand how the gas or hazardous liquid pipe system is connected and how the commodity flows through the connected pipe components. Every department within the organization is making decisions that require this knowledge. Even departments such as finance and legal, make decisions based on where the pipe is located and how the commodity flows thru it. Finance determines how much and who they owe taxes too based on where those pipe assets are located. Legal needs to understand who is downstream from a specific location to validate that service level agreements are being met.

Historically this information has been segmented and siloed, making it difficult for an organization’s departments and staff to easily access this information. These silos are often given names like distribution, transmission, storage, and gathering. Each represents a subsystem within the connected pipe network. This unnatural segmentation does not accurately represent the real-world pipe systems they are attempting to represent. Real-world pipe systems are fully interconnected. If these subsystems are owned by a single organization, we refer to that organization as being vertically integrated.

Managing digital information about these vertically integrated pipe organizations and how the gas or hazardous liquid commodity flows through them requires special capabilities. Special capabilities that enable an organization to model the entire pipe network from its extraction from a well to the final delivery of the product to the customer. Special capabilities to model this complex pipe network as ONE network. Special capabilities that are provided with ArcGIS.

ArcGIS Modern Network Management

ArcGIS provides the many tools and capabilities required by those responsible for maintaining this information to build these digital representations of these pipe networks as a single Pipe Network.

For managing the connectivity of the assets across a single pipe network, ArcGIS provides a collection of tools and capabilities called Utility Network. Utility Network manages the connectivity between all assets.

When the transmission subsystem is a part of the single pipe network, additional tools, and capabilities to create and manage routes is required. This set of tools and capabilities is called ArcGIS Pipeline Referencing.

Organizing this information into one database where the assets can be modeled as a single pipe network with network connectivity, and linear referencing requires a well-organized data model.  A data model that marries the Utility Network information model, with the ArcGIS Pipeline Referencing information model, and the asset information needed to correctly describe the pipe system’s components. This data model is called: Utility and Pipeline Data Model (UPDM).

UPDM is the gas and hazardous liquid industry data model created, tested, and maintained by Esri.

Bringing together, Utility Network, ArcGIS Pipeline Referencing with UPDM provides a Modern Network Management system designed for today’s enterprise requirements.

Single Source of Truth

Leveraging services-based architecture, this modern network management system provides secure access to this data for the entire organization.  A services-based architecture allows for a single point of access by applications and systems to this single pipe network. This eliminates duplication of data and the need for ETL’s (Extract, Transform, and Load).  Now the entire organization can be confident that they are seeing the same information regardless of application or location.  Everyone working off the same source data, whether you are in Engineering, Dispatch, Project Design, Planning, or operations. Everyone working off the same source of data, whether you are in the office or deployed to the field.  Having the most accurate data available at all times, enables good business decisions and improved safety of your mobile workforce.

Accessing the same information across the organization

Managing Connectivity

Managing network connectivity is accomplished with the tools and capabilities of the Utility Network within ArcGIS. These tools allow natural gas and hazardous liquid organizations to not only connect their assets together, but to also accurately model the flow through those assets.

Utility Network tools enable designers and mappers to define, place, and connect all the assets of a pipe network. These connections can be based on spatial coincidence or a logical association. By creating logical associations, you can show how equipment is connected without having to spatially snap them together via spatial coincidence. Welded assets to be connected without drawing an artificial pipe segment between them. Anodes can be connected to pipes without having to draw the connecting wire.

Correctly assembling a network as complex as a natural gas or hazardous liquid pipe network, requires strong capabilities for ensuring data quality. ArcGIS provides the ability to define business rules is part of the toolset to help data creators to enter the information correctly the first time. Within ArcGIS we use terms like network rules, contingent values, coded value domains, and attribute rules to name the capabilities provided to implement business rules. UPDM provides configurations of these capabilities which are specifically configured for pipe networks.

The Utility Network tools include validation against the rule base. This ensures that editing is consistent within your organization.  Validating your edits becomes part of your workflow.  This ensures that the data being posted to your utility network has passed all the rules. 

Attribute rules allow an almost endless ability to define custom rules for assets and structures within your network.  These practices ensure that the data being entered into the utility network is accurate.

ArcGIS Utility Network is configurable for any utility, municipality, or organization with linear assets. Esri solutions explore the world of electric, gas, and water utilities, architecture, engineering, construction (AEC), and telecommunications. 

Analytics

Pipe networks tend to be looped and interconnecting to improve reliability. Understanding how your commodity traverses your pipe network is challenging. ArcGIS provides tools called Trace tools, which leverage the connectivity to help organizations understand how your commodity traverses your pipe network. Trace tools provide the analytical capability to enable your organization to understand which customers are downstream of a location, where the gas or liquid originates from, and most importantly what valves to close to isolate a section of pipe. These trace tools are part of the geoprocessing framework within ArcGIS. This means the results of a trace can be incorporated into multi-step analytics within ModelBuilder or scripted within a python script. This opens new understandings for risk analysis and main replacement prioritization, such as tabulating for each pipe segment the number of critical facilities and the number of customers impacted by the failure of a pipe segment.

More on these capabilities can be found here: ArcGIS Utility Network | Spatial Information Management System for Utilities (esri.com)

In this image is the result of a Distribution Gas Isolation trace.

Viewing Complex Facilities

When accurately modeling your pipe network, you will find portions of your service area where the map display can be crowded or cluttered. In these locations such as regulator stations, compressor stations, and even multi-meter buildings such as apartments, you need more intelligent feature display controls than most map viewing software can provide. For these feature-dense locations the ArcGIS Utility Network tools and capabilities, provide an additional feature display capability called Containers. Containers allow office and mobile users to initially see a simplified representation of the facility. For most data viewing activities simply knowing where the regulator station, compressor station, or wellhead is located is enough. But there will also be situations where the office or mobile user requires an understanding of the assets within the facility. Situations where the user needs to open the container to see the assets contained within the facility. Containers provides the feature display control capability to hide the assets within the gas or hazards liquid facility until the user decides they need to see what is within the facility.

Access from Anywhere!

Accessing the same data accessed through ArcGIS Field Maps.   Users in the field are looking at the same data as you are in the office, the same data, across the enterprise. 

Pipe organizations require the ability to access their pipe network data from anywhere, anytime and on any device.  The services-based architecture of the Utility Network provides the infrastructure to enable pipe organizations to meet this need.  This single source of the truth allows office users of web applications to be able to answer their questions about the pipe network and how the commodity flows thru it. Using a web application, engineers planning maintenance which will require disruption of service can run traces to understand what valves to close and which customers are impacted. Mobile workers also have access to these same web services and capabilities on their mobile phones, tablets, and laptops. Using a mobile application such as FieldMaps an operations supervisor can run the same traces as the office staff to understand the impact and required response to a system outage.

Having access to the latest accurate data protects lives and safety for anyone who access or uses this data to perform their jobs.  From the field to the office, everyone is using the single source of truth.

Integrated with Industry Solutions

Outage Management Systems (OMS), SCADA, CSS, Hydraulics, Pipe Calculators, Meter Reading, AMR, just to name a few, can be integrated into one system and accessible to those departments that need this information to perform their jobs.  By integrating these systems together, agencies within your organization can be visualizing the data as it’s being collected and analyzed.

Provide integration of Industry specific applications.  As shown in this image, Integration between ArcGIS Enterprise and Gas/Pipeline Industry Applications and made available to your organization.

This is an image of an ArcGIS Dashboard configured to show Leak Information.  Data is compiled together to show both Enterprise Utility Network data as well as Leak data being captured out in the field.

In Summary

Implementing an ArcGIS Utility Network for your organization provides your office and mobile workers with the ability to understand and explore the assets and connectivity within your pipe system.  Utility network supports the utilities need to deliver their commodity safely and economically with efficient tools and rules that conform to the operations of your organization.  Capturing the data accurately and having that data available to your organization provides the document of record for your organization or the single source of truth, everyone working off the same data.  Working with a high-performing model that is scalable and enforces the integrity of your organization, allows for both small and large utilities to take advantage of the benefits of the utility network.  For more information, please visit our GIS for Gas Utilities website:  GIS for Gas Utilities | Natural Gas Distribution Mapping & Analysis (esri.com)

Esri’s Utility and Pipeline Data Model (UPDM) 2023 is available now. This release continues Esri’s practice of maintaining a template data model ready “out-of-the-box” to manage gas and hazardous liquid pipe system data within an Esri geodatabase. This release includes enhancements to keep up with changes in industry practice and implementation feedback received since the previous release.

Offline Map Areas is the name Esri gives to the capability that enables the ArcGIS system to automatically transmit data changes between a mobile device and the organization’s ArcGIS data repositories for a specified geographic extent. In this final blog article of the series we will explain how bi-directional synchronization works between your ArcGIS data repositories and your mobile devices, and the management tools available within ArcGIS to help you to manage a large fleet of mobile devices.

 Taking Your Maps Offline: Creating Offline Map Areas

By Mike Hirschheimer, Tom DeWitte, Kevin Ruggiero

Part 4 of 5

Your Utility wants to setup a process that enables field workers to have access to your pipes, conductors, and cabling geospatial data when not connected to the communication network. The GIS/IT support team has read the ArcGIS documentation for offline map areas and is ready to begin. For an initial proof of concept (POC) the team created new web maps and then used the Field Maps Designer web app tools to manually define the extent of a single offline map area.  The feedback from the field has been positive and now leadership wants to expand to cover the entire service territory which includes 100 distinct areas.  The GIS/IT team is super excited about the project moving forward but secretly cringes when thinking about the effort and hours it will take to manually repeat the steps from the POC for the remaining 100+ areas. 

A similar scenario happened to our team earlier this year.  Our customer had 100 “operating areas” throughout their service territory and manually repeating the steps executed in the POC for each offline area wasn’t sustainable.  We needed a way to automate the creation of these 100 offline map areas. The ArcGIS API for Python provided the scripting ability that was needed for this automation.

In this blog, we’ll discuss best practices for managing the web maps along with using the ArcGIS API for Python to create and report on offline map areas.

Configure the web maps

The data that the field worker sees on their mobile device is driven by the layers, tables, and configuration in the web map.  From ArcGIS documentation, we know that a single web map can support up to 16 offline map areas.  With 100 areas to define, a minimum of 7 web maps is needed (100 / 16 = 6.25) if the areas were broken up evenly.  In reality, it took 13 web maps as the ease of use for the field worker was a major factor when making assignments.

Managing 13 web maps can become overwhelming pretty quickly when a single change would need to applied 13 times.  The recommendation is to create a “Master” web map using ArcGIS Pro that includes the facilities, landbase features, the offline enabled base map, and other pertinent data feeds.  Once the configurations like scale ranges, pop-ups, labels, locators, etc. are set, publish the web map to Portal using the Share Tab à Save as Web Map button.  Then the “Master” web map can be copied and given a new name as many times as needed to support our offline map areas requirements. 

NOTE: If using ArcGIS Enterprise 10.9.1 or 11.1, don’t include Subtype Group Layers in your web maps.  The offline map area creation tool will fail because it doesn’t understand Subtype Group Layers.

Creating the Offline Map Areas

There are 2 approaches to create the offline map areas.  The Field Maps Designer web application could be used to manually define polygon extents, or the ArcGIS API for Python could be used to programmatically generate the polygon extents based on the operating area polygon layer most utilities already have.

In the Field Maps Designer web app, there are tools to draw a rectangle or a polygon to define the extent and then enter information about the area (name, how often to refresh, levels needed in the basemap).  For a POC, this approach is quick and easy but doesn’t scale well when moving into production mode.

The programmatic approach uses a Python script to automate the creation of offline map areas.   Attached to this blog is a script that creates offline map areas using a polygon feature’s geometry.  The script uses a configuration file that contains your ArcGIS Enterprise credentials, the URL of your polygon layer of operating areas, Web Map IDs and the instructions for creating the offline map areas.

In this sample configuration file, there are 2 web maps that are defined in the offlineAreaConfig section.

When the script runs, offline map areas will be created for the polygon features that satisfy the “polygonWhereClause”.  In both instances, both Virginia and Colorado had less than 16 polygons.  If these states contained more than 16 areas, the where clause would need to be redefined.

Knowing that every utilities’ data model will be slightly different, multiple entries in the configuration file are used.  This gives the script flexibility to name the offline map areas using the fields in your data along with allowing the script to run in your Dev, Test & Production environments without making code changes.

Another benefit of using the Python script is the log file.  This provides documentation as to when each offline map areas was created and the duration it took. 

Reporting on the Offline Map Areas

After setting up the configuration file and running the python script, 100 offline map areas in 13 web maps have been created.  A simple way to report on these offline map areas is now needed.  The log file has good information but wasn’t meant to be a report.  There is another attached python script that generates a CSV file containing details about web maps with offline map areas.  The CSV identifies the contents of each offline map area and the files sizes of every Vector Tile package and SQLite database.  Not only is this report useful for knowing what exists in your Portal but also show the amount of data to be downloaded to a mobile device.

What needs to occur if I need to change the web map?

Your Utility has been in production for a few months now and the Field workers are actively using the offline capabilities.  To be more efficient, they are asking for a couple of changes to the web map.    Specifically, a labeling change on a linear feature, a change to the order of attributes on a specific device and the need to capture a new attribute on a structure.

Making those changes the web map is easy enough but that won’t initiate the change to the offline map areas.  Once an offline map area is created, it’s schema and it’s corresponding web map is locked down.  New web maps need to be created to get these changes available to the field worker.

To apply schema or map changes, the workflow would be something like this:

1. In Portal, existing web maps with offline map areas should be renamed and/or deleted
2. Make the necessary schema changes (ex.  Add a field)
3. Using ArcGIS Pro, update the “Master” web map in Portal with the labeling and pop-up configuration
4. Make the necessary copies of the “Master” web map
5. Re-run the Python script to create offline map areas
6. Notify field users that existing offline map areas should be deleted from their device and that newly created offline map areas are available for download

As you can see, this python script not only helps with the initial creation and deployment of offline map areas, but it also streamlines the ongoing support and propagation of changes.

Simplifying a GIS Administrators Job

The recommendations in this blog are the real-world lessons we learned from assisting a large utility customer with deploying offline map areas. From creating the master web map to using the Python API to automate the creation of the offline map areas.  These learned lessons removed the GIS administrator cringe of doing time-consuming manual processes for the deployment, and instead provided an automated process allowing our servers to continue working while we went home.

If you are interested in using the python scripts described in this blog, they are available on Esri GitHub so that you don’t have to start coding from scratch.

About this Blog Series

This is the fourth blog in our series on offline map areas. In future blog articles we will continue to explain decisions an administrator will need to make during deployment.

The first blog provided and overview of offline map areas.

The second blog provided details on preparing your data for offline usage.

The third blog will provide details and options for publishing the selected data repositories.

The fifth and final blog will provide details on the deployment and management of offline map areas for a large mobile workforce.

PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.

Organizations large and small need to keep their mobile workers informed. This requires a consistent and reliable capability for transmitting new and updated information. Publishing the data within ArcGIS is a critical step in successfully deploying Offline Map Areas to enable a bi-directional synchronization with mobile devices.

In any utility organization the field crew plays the vital role of your eyes and ears in the field. They are the first to identify issues, report emergencies, and ensure the smooth execution of tasks. ArcGIS Field Maps can improve your team's safety and efficiency in the field. 

 Users of ArcGIS Pipeline Referencing and ArcGIS Utility Network have a new best friend in the Unified Pipeline Tools ribbon. This new ribbon for the first time enables users of both extensions to access tools from both on one tab instead of having to move back and forth among multiple individual tabs.

Utilities desire is a mechanism that can automatically and frequently pass changes to the data from the office to the field and from the field to the office. This method of only passing changes is called bi-directional synchronization, though many will simply call it delta syncing. Deploying bi-directional synchronization with ArcGIS requires proper configuration of the data, and the server environment. It also needs a mobile computing device with ArcGIS Field Maps. 

If this capability is of interest to you, you may be wondering, can I take my data offline? What modifications do I need to make to my data to prepare for bi-directional synchronization with my organization’s mobile devices?