Utility and Pipeline Data Model 2021 is Released

[TomDeWitte]

Utility and Pipeline Data Model 2021 is Released

By Tom DeWitte and Tom Coolidge

Esri’s Utility and Pipeline Data Model (UPDM) 2021 is available now. This release continues Esri’s practice of maintaining a template data model ready “out-of-the-box” to manage natural gas and hazardous liquid pipe system data within an Esri geodatabase. This release includes enhancements to keep up with changes in industry practice, regulatory requirements, and previous implementation feedback.

Gas and Pipeline Enterprise Data Management

For many gas utility and pipeline enterprises, deploying the ArcGIS platform is more than simply loading the UPDM 2021 data model into an enterprise geodatabase. That’s because ArcGIS leverages the concepts of a service-oriented web GIS. It requires additional steps, such as creating an ArcGIS Pro map configured for publishing the data model, publishing of the Pro map to create the required map and feature services and, perhaps, configuring a location referencing system. To help simplify these additional steps performed with UPDM 2021, Esri has embedded UPDM 2021 into a newly renamed ArcGIS for Gas solution. The new solution is called Gas and Pipeline Referencing Utility Network Foundation. This solution provides UPDM 2021, sample data, and an ArcGIS Pro project configured with tasks and performance optimized maps. You can access this solution from the Esri ArcGIS for Gas solution site.

A full data dictionary of UPDM 2021 is available online.

A change log documenting the full list of changes incorporated into UPDM 2021 is also available online.

Enhancements for Managing Cathodic Protection

Interest in leveraging ArcGIS to manage Cathodic Protection (CP) data has grown significantly over the last few years. Recent implementations have shown an interest in modeling a more detailed representation of CP systems. With UPDM 2021, our industry data model now includes models for managing the following CP assets:

• Linear Anode
• AC Mitigation Wire
• Decoupler
• Grounding Point
• Cathodic Assembly
• Grounding Mat

Feedback from Pipeline Implementations

Pipeline implementations of UPDM have been many and varied. These previous implementations have resulted in feedback that has resulted in dozens of adjustments to improve how we model pipeline assets. Two specific enhancements based on previous implementations to highlight are the defining of the attributes “odorized” and “piggable” as Utility Network Network Attributes.

Having “odorized” and “piggable” as Network Attributes provides additional capabilities for leveraging tracing. Tracing is the ability to understand how the components of a pipe system are connected, and how the gas and hazardous liquids flow through the pipe system. With this new template configuration, pipeline operators can trace across their pipe system and see where the “piggable” portion stops when it reaches an asset which is tagged as not piggable. Similarly, planners and engineers can trace to see what portion of their pipe system is “odorized.”                   

Feedback from Gas Utility Implementations

Previous implementations of UPDM in Gas Utilities have also resulted in feedback to help fine-tune the gas and pipeline industry data model. Some of these adjustments include improved modeling of relief valves, flanges, and taps to name a few. Other enhancements include adjustments to the Utility Network rulebase.

What is UPDM

UPDM is a geodatabase data model template for operators of pipe networks in the gas and hazardous liquids industries. UPDM is a moderately normalized data model that explicitly represents each physical component of a gas pipe network from the wellhead to the customer meter, or a hazardous liquids pipe network from the wellhead to the terminal or delivery point, in a single database table object.    

UPDM is the only industry model which can manage a single representation of the entire pipe system.  For many companies around the world this single data repository aligns well with enterprise practices to vertically integrate business processes and operations. 

Why UPDM

The goal of the Esri UPDM is to make it easier, quicker, and more cost-effective for pipeline operators and gas utilities to implement the ArcGIS platform. The Esri UPDM accomplishes this by freely providing a data model that takes full advantage of the capabilities of the geodatabase. The data model is created and tested with ArcGIS products to ensure that it works. This significantly reduces the complexity, time, and cost to implement a spatially enabled hazardous liquid or gas pipe system data repository.

Looking Forward to UPDM 2022

A wise man once said “change is the only constant.” This is a great quote when thinking about UPDM going forward. The Esri development team will continue to enhance the capabilities of ArcGIS. Industry will continue to evolve its practices. To continue adjusting to industry practices and incorporating new ArcGIS capabilities, UPDM will continue to evolve.  This evolution will help assure gas utilities and pipeline operators that their GIS industry-specific data model is current with their needs.

PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.

[DiegoGuidi]

download broken I retried the entire procedure and now it worked

[JohnHu]

Tom,

I noted that UPDM 2021 has domain values for attributes such as lifecyclestatus and commodity type.  We are still in UPDM2019.  How can I find the similar domain codes/values for above attributes ?   Please help.   Thanks.

John Hu

[JSchroeder]

Hi @JohnHu,

Both UPDM 2019 and the latest release, UPDM 2021, have domain codes and values for lifecyclestatus (Pipeline_Lifecycle_Combined) and commodity type (Pipeline_Commodity_Type). If you want to see those values/codes, you can search for those domains in the UPDM File geodatabase. Also, you can look at the Data Dictionary for the current solution and browse or search the feature classes, fields, domains, etc... It is located as a link at the Solution page for Gas and Pipeline Utility Network Foundation.

Further, based on which feature class utilizes these domains, they may be applied at the subtype level (View the subtypes in the file GDB for the feature class to see which domains are applied to which subtypes). For example, PipelineLine has subtypes for Bonding Line and Distribution Pipe, but commodity is only relevant for a line transporting a commodity, so the domain is not applied at the field level (across all subtypes), rather it is applied at the subtype level (so it is only applied for specific subtypes). Hope this helps.

[liangxu1234]

I want Data Dictionary  of UPDM2021,please help,thanks.

[JSchroeder]

Hi @liangxu1234,

In this blog, if you look below the 2nd paragraph, there is a direct link to the UPDM 2021 dictionary. Right above that link is another link to the solution product page where the documentation also points to the online Data Dictionary. If you want an HTML version of the Data Dictionary, you can deploy the solution to your Online or Enterprise portal, where you can download the solution as a ZIP file. Within the ZIP file is a link to the online Data Dictionary, as well as an HTML version of the Data Dictionary. You will see a folder named 'Data Dictionary' after you extract the contents of the ZIP file. There is a File Geodatabase named UPDM_UtilityNetwork.gdb within the 'Databases' folder of the ZIP file which allows you to fully interact with the data model and Utility Network settings. Finally, there is a Pro Project included in the ZIP which includes a series of tasks that will walk you through the data model and solution in general.

[ThomasSchuster]

Hi @TomDeWitte, @JSchroeder,

thanks again for continuing the development on the Gas Solution, UPDM with APR. I just ran through the tasks to build a UPDM 2021 UN + LRS. Unfortunately I could not pass the 'Enable APR on UPDM' step in my environment.

The error I'm getting is: 'Failed to find required classes for Creating the LRS'. Within the Python Toolbox 'Utility Data Management Support' I see the script is checking for the existence of the required APR core tables (line 1512 to 1515). When I review the FGDB I've created so far, I can't find the 'P_Centerline_Sequence' table. It is present although in my 'AssetPackage.gdb'.

Do you know what could have gone wrong? These are the versions I'm using: 

• ArcGIS Pro 2.9.2.32739
• untools 2.9.1
• Asset Package schema version: 2.71

After having created the 'P_Centerline_Sequence' table by hand (including the editor tracking and global id option) , the task to 'Enable APR on UPDM' did run further, but is stopping with this error now:

Failed to execute. Parameters are not valid.
ERROR 000728: Field engrouteid does not exist within table
ERROR 000728: Field engfromm does not exist within table
ERROR 000728: Field engtom does not exist within table
Failed to execute (ConfigureUtilityNetworkFeatureClass).

I'd expect the 'ConfigureUtilityNetworkFeatureClass' tool is looking for the PipelineLine feature class? If so (after checking) the above fields are missing. At which point in the installation process should these fields be created?

Cheers, Thomas

[TomDeWitte]

Hi Thomas,

The "Gas and Pipeline Referencing Utility Network Foundation" version 2.1 includes the P_Centerline_Sequence table in the Asset Package (as you noted), and in the sample dataset file GDB. Please verify that you are using the current version of the Gas and Pipeline solution.

Assuming you are using the current version, I would then suggest recreating file geodatabase from the Asset package.  

Final comment. The ArcGIS Pipeline Referencing information schema is included with the UPDM data model embedded in the solution.  Please use the "...from existing DataSet" version of the Pipeline Referencing GP Tools to configure your APR instance. These tools are for this situation where the ArcGIS Pipeline Referencing information schema featureclasses and tables already exist in your geodatabase.

Thanks

Tom DeWitte

Esri Technical Lead - Natural Gas Industry 

[ThomasSchuster]

Thanks @TomDeWitte for your quick response. Recreating the File GDB from the Asset Package did solve my issue.

Cheers, Thomas

[walsher]

I am new to this model so forgive my potentially stupid questions:

1.  Are the domain values in Pipeline_Coating_Type the only way to describe your coating?  No manufacturer, product name, thickness, coating applicator, date applied?  i.e. does "14-16 Mils 3M 6352 with 30 Mils 3M 6353 ARO" simply become "FUSION BONDED EPOXY"?

2. Can a user add domain values?

Thanks

[TomDeWitte]

The Esri provided Utility and Pipeline Data model represents pipe coating with a single coded value domain. The values in this coded value domain can be changed to the examples you describe.  You can also add additional attributes to the PipelineLine featureclass to describe the coating in greater detail.

Hope this helps

Tom DeWitte

[dabercro]

I have downloaded the 2021 UPDM package and the PipelineLine feature class is giving me this error.  I have redownloaded and get the same error.  Any thought on why or what I can do to resolve?  

[AyanPalit]

@dabercro Perhaps You are using ArcMap/ArcCatalog to review the PipelineLine feature class. UPDM and the related applications of ArcGIS Utility Network and ArcGIS Pipeline Referencing are home-based in ArcGIS Pro. Please use ArcGIS Pro for your review.

[ThomasSchuster]

Hi, this data model now supporting APR and the UN sounds like a big step forward. I understand that we're able to model our assets from upstream till downstream - just like in the image above. In the scenario APR + UN I understand that all pipes in Pipeline_Line participate in the UN. But not all pipes are managed using APR. E.g. I only want APR to manage my transmission pipelines in the midstream sector. To do so, I only create engineering routes for my transmission pipelines. And here is where my question starts:

The Pipeline_Line feature class has a field 'Joint Number' - which indicates that I can integrate my transmission lines joint by joint (lots of small line geometries). Creating a route (APR) from some  centerlines (up to 100 features within Pipeline_Line) is not a problem. But with my transmission lines having 10.000+ joints, creating an engineering route is becoming a challenge (performance wise).

I've heard that for the UN, the small (joint based) line geometries don't provide any additional value. One line geometry covering my transmission pipeline would be sufficient (and must faster when running traces in UN or running APR tools).

For my example, how would I best keep the performance high and at the same time keep the joint numbers within the data model?

[TomDeWitte]

Hi Thomas,

Really good question on how to optimally leverage the UN with APR. Our Utility and Pipeline Data Model (UPDM) is designed to support three implementation patterns of ArcGIS:

1) Utility Network only

2) Utility Network with ArcGIS Pipeline Referencing

3) ArcGIS Pipeline Referencing Only

The vast majority of the content of UPDM is used the same regardless of implementation pattern. But, there are some portions of UPDM which are implemented differently across the 3 implementation patterns. The P_Centerline featureclass and the joinnumber attribute on PipelineLine are two such examples. JointNumber attribute and P_Centerline featureclass are intended for the APR only implementation pattern where PipelineLine is configured as a line event layer. This APR only configuration uses P_Centerline as the source for line geometries to build routes. 

Maintaining the JointNumber information in a UN + APR or UN Only implementation pattern could be achieved in manner which would not result in PipelineLine being divided into tens of thousands of segments per engineering route.

-Store JointNumber on the PipelineJunction features (welds, fittings, etc). This does not require splitting of the PipelineLine features.

Hope this helps

Tom

[Krishna_RaoTVB]

Hi Tom, 

What roles will the creator user require for implementing Utility Network with ArcGIS Pipeline Referencing at Enterprise 11 and Pro 3.0.1?

Any work around for the Viewer to do UN tracing?

[TomDeWitte]

Hi Krishna,

There are multiple user roles and licensing requirements for deploying administering, editing, tracing and viewing of a Utility Network and ArcGIS Pipeline Referencing configured pipe dataset.

Here are a few of the key licensing requirements:

1) All Utility Network editors require the Utility Network user extension.

2) All Utility Network tracing capabilities require the Utility Network user extension.

3) All route editors of the ArcGIS Pipeline Referencing managed routes will require ArcGIS Pro and a Location Referencing desktop extension license.

4) To publish the Location Referencing capability into a feature service requires the Location Referencing server extension.

Viewers of the Utility Network and/or ArcGIS Pipeline Referencing data itself do NOT require additional licensing. Viewers with the Utility Network user extension license are able to perform Utility Network traces.

I hope this helps

Tom DeWitte

Esri Technical Lead - Natural Gas and District Energy

[ThomasSchuster]

Hello, the esri solutions team is providing a convenient deployment process for UPDM + UN + APR. This deployment process currently supports ArcGIS Pro 2.9 and Enterprise 10.9.1. Is it planned to provide an updated solution to support current versions of Pro and Enterprise? - @Krishna_RaoTVB already mentioned Enterprise 11 and Pro 3.0.1.

Cheers, Thomas

[RichMosar]

Hi Tom and ESRI Team,

I have a question regarding the DASurveyReadings layer for the Direct Assessment surveys.  What would be the difference between the pson and the dcpotentialvolts field?   In our completed DCVG surveys, our survey contractors provide the signal shift (mv) and overline remote earth (mv) voltage readings.  We are wondering if those values belong in any of the fields in the existing Esri data model, or whether we may need to extend it.  Thanks in advance!

[ThamerTarabzouni]

Greetings Tom

We have developed a comprehisive solution for downstream using UPDM 2019. Did customizations to many features classes. Now we would like to extend the solution to Upstream.

1-With the release of 2021, is there a miogration utility to use it to migrate 2019 Schema tp 2021?

2- I went through the change log and found there are few changes in terms of integrity and survey classes. 

Having said that, will you advice to exert the effort to migrate to 2021 or continue use 2019. If you have any technical advice please share. 

[TomDeWitte]

Hi Thamer,

Glad to hear you have had success with the UPDM 2019 version of our gas and hazardous liquids industry data model. Below are my responses to your questions.

1-With the release of 2021, is there a miogration utility to use it to migrate 2019 Schema tp 2021?

Tom Response: No there is no migration tools to assist migration between data model versions.

2- I went through the change log and found there are few changes in terms of integrity and survey classes.

Tom Response: The UPDM 2021 release included the addition of many cathodic protection components to more fully document CP assets. There was also a focused effort to incorporate great feedback from the midstream industry and improve transmission and storage data management. UPDM 2021 also included some updates to the core information schema of ArcGIS Pipeline Referencing to keep the data model in sync with product enhancements.

Your last question is do I recommend migrating to UPDM 2021. My recommendation is "no". Even though UPDM 2021 contains many great enhancements to better manage gas and hazardous liquid assets, those changes from UPDM 2019 are still very small compared to the overall data model. I would suggest comparing the two data models and via python scripting (gets the job done and documents what your did) update your existing data model with those enhancements which will improve your daily geospatial data management activities.

Tom DeWitte

Esri Technical Lead - Natural Gas and District Energy

[ThamerTarabzouni]

Good Day Tom,  

Thank you for your feedback, and I do appreciate your suggestion about update the  enhancements which will add values.  

[ThamerTarabzouni]

Greetings Tom,

I'm planning to create a comreisive data entry app to update the attributes data for needed tables and feature classes of the UPDM 2019. This wll be provided to number of users to maintain their data up-to-date and even collect data.  UPDM 2019 domain values, however, are accessible using ARCGIS Pro. This whould require some learning curve for Engineers. I tried to exprot these FCs for example to Dbase files, but domains were not exported.

My question where I need your support, is their an approach or a way to export these FCs with their domains and relationship diagrams so for example I can build this data entry forms to guide and validate the users inputs by using the realtive domain values for fields. For simplistiy and clarity, I might use MS Access with combo fields to assesst users data input.

I hope my question is clear , looking forward for your positive feedback.  

[TomDeWitte]

Hi Thamer,

One of the base capabilities of ArcGIS that I really enjoy as a creator of content for others is the automatic inheritance of domains across desktop, web, and mobile esri applications.

For the data entry application you are looking to create for your engineers, giving them ArcGIS Pro, is asking a lot.  Have you looked configuring a web application or configuring a set of smart forms within ArcGIS Field Maps or Survey123. All 3 of these end user applications will automatically create, and assign coded value domains to those data fields which they are assigned within the Geodatabase.  Just like in ArcGIS Pro, where the picklists are auto-generated, the same "creator" experience applies to ArcGIS web applications and to Survey123, and ArcGIS Field Maps. 

Tom DeWitte

[ThamerTarabzouni]

Good Day Tom,

Thank you for your reply.  I agree with you about the ARCGIS Pro, and I as I stated, I was going to Use MS ACCESS to create data entry forms. But as I also stated, when I exported the UPDM FCs as tables, all domians refernce were lost, whihc is expected. 

But your suggestion of using Field Maps is worth exploring.  Thank you for that. 

[jeannl]

Hi,

We are looking to implement UPDM on Enterprise 10.9.1.

I downloaded the Solutions directly from my portal and I got a project file  GasPipelineReferencingUtilityNetworkFoundationV2.

I have :

SQL Server 2019 
ArcGIS Pro 2.9.32739
untools 2.9.0
Executing from ArcGIS Pro, 0 map(s), activeMap = False
Parallel processing enabled: 4
Asset Package schema version: 2.9

I recreated an asset package from the UPDM_UtilityNework.gdb that came with the solution.

I started from tasks Deploy Solution to an Enteprise Environment

I have setup enterprise database

I have created the utility network staging the the Service Territory by using the one in the new AssetPackage just created.

When I get to the Apply Asset Package it seems that LoadData process doesn't want to work giving me all sorts of error.

Are there specific combination of ArcGIS Pro/untools/dltsolutions to use with the solutions package???

Can anybody give me his own combination that work and resulted in a success creation and deployment?

[JSchroeder]

Hi @jeannl, I recommend that you update untools to the latest version (2.9.7) and use the latest version (2.3) of the Gas and Pipeline Referencing Utility Network Foundation which can be acquired from ArcGIS Online or ArcGIS Enterprise 11.2. 

Would you mind creating a new post in ArcGIS Utility Network Questions after trying the above steps and posting the errors in the new post? This will give the issue more visibility with our support team here at Esri versus being posted in comments for a blog posting - it won't see the same eyeballs as it will in Utility Network Questions. Thank you.

[jeannl]

Hi, 

Yes sure I will do so.

But as I mentionned I am on Enterprise 10.9.1, so by default solution is then at version 2...

I will have to look at 11.1 after that also...

[JSchroeder]

@jeannl, the documentation for 2.3 states that it is compatible for 10.9.1, so you can download the solution from ArcGIS Online and use with 10.9.1. I would, however, recommend using 11.1 as that is the most current long term release which follows the Utility Network Management Release. 

Upgrading will not only give you a longer support window, but also give you the advantage of the upgrades since 10.9.1 was released. We understand that some organizations may have business reasons which prevent upgrading to 11.1, so if this is your case, you can stick with 10.9.1.