UPDM 2018 Edition

[#1]

This is the official release of the 2018 Edition of the Utility and Pipeline Data Model (UPDM).

 It is designed to support data management for the natural gas and hazardous liquids industries.  Supporting the diverse needs of these industries means supporting multiple implementation patterns.  Specifically a network topology with the Utility Network implementation pattern, and a linear referencing implementation pattern with the ArcGIS Pipeline Referencing solution..

If you have questions, please post them to geonet, so everyone can see and share the information.

Thank you

Tom DeWitte

Esri Technical Lead – Natural Gas Industry

[WhiteWaterMidstream]

Tom, I was able to download the draft UPDM2018 database from Version 1 of your post. I was reviewing and testing it and have encountered some errors in the database schema that is causing an error when trying to import XML workspace document using Xray or geoprocessing. It keeps throwing an error on domains with split policy of GEOMETRY RATIO. Also some other domains throw up error for invalid code values. Can you all please test import/ export with XML to resolve?

Thanks

Lee

lee@wwm-llc.com

[JuanHerranz_Lama]

Hi Tom, when do you think UPDM 2018 Edition will be ready to download?. Thanks!!

[TomDeWitte]

Juan,

Current schedule is to repost UPDM 2018 edition by the end of February.

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[TomDeWitte]

Lee,

Prior to reposting, this will be retested. 

Thanks

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[JakeJacobs]

Tom - how close is the data dictionary in v1 to what will be released next?  We are assessing the current version of UPDM against a high pressure records review project to see if it fits our data needs better than our current model, and I just want to know if I can get started or need to wait. 

[TomDeWitte]

Erica,

The data dictionary v1, is approximately 90% consistent with what is being worked on for UPDM 2018 Edition Update 1.  Major changes are planned for the tier group definitions, and some minor changes are planned for the PipeSystem Line, Device and Junction featureclasses.  The other featureclasses and tables are not being modified.

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[WhiteWaterMidstream]

How is the schedule for release going? We are anxious to re-build our GIS with UPDM 2018. Thanks,

[PareshPatel2]

Tom,

Any update on the release date? Are there any implementation guides available?

[ConradStanley]

We are stalled!! We cannot move until UPDM is available. Thanks

[TomDeWitte]

To all,

The revisions to the UPDM 2018 edition are almost complete.  Scripts and models have been updated.  A change log has been created.  Working on finishing the updates to the data dictionary.  The plan is to have the full model reposted early next week.

Thanks

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[WhiteWaterMidstream]

Please consider making sample GDB a more common projected coordinate system. Perhaps USA Contiguous Albers or Lambert. Also last time I had alot of errors trying to built the gdb using xray please test. Thanks,

[ConradStanley]

Please also consider creating a sample GDB with correct tolerances in World Mercator:

WGS_1984_Web_Mercator_Auxiliary_Sphere

WKID: 3857 Authority: EPSG

as all the background maps are in that and the measurement issues:

BUG-000088279 Geodesic areas cannot be calculated using the Measure tool in ArcMap if the data frame is set to a projected coordinate system (Web Mercator).<https://my.esri.com/#/support/bugs/BUG-000088279>

Seem to have been fixed in ArcPro.

[TomDeWitte]

UPDM 2018 Edition has been reposted to geonet. 

Thanks

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[JakeJacobs]

I found a typo in a field name in the asset package.  What's the best way to give feedback like that?

StructureJunction.locationdesctiption

[MarkCederholm]

I see some differences comparing it to the current Gas asset package.  I'm assuming these are two different beasts?

[TomDeWitte]

Mark,

In early April the current Gas Asset Package will be updated with the asset package that was included in the UPDM 2018 zip file. The Esri solution site has a set update release schedule, that is why the updated package will not be available on the Esri solution site until April.

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[TomDeWitte]

Thank you Jake!

This site is monitored by Esri staff.  We will get this identified type-o to the correct folks for correction.

Thank you

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[WhiteWaterMidstream]

What is the best way of changing the spatial reference? There is not an option for this using the Import XML Workspace Document tool in ArcGIS Pro.

Also the option in X-ray is not working either.

[MarkCederholm]

untools has functionality for changing the spatial reference of an asset package.  See this link for details:

https://community.esri.com/docs/DOC-11409-assetpackagetoolszip

[WhiteWaterMidstream]

I just manually changed all dataset and feature class spatial reference by hand. It was kind of cumbersome but seems to be working ok. 

Thanks,

[TomDeWitte]

Erica,

The type-o on the Structure Junction attribute field LocationDescription has been corrected and an revised zip file has been reposted.

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

[WhiteWaterMidstream]

I'm finding many other small errors like field domains not properly assigned to all subtypes. This is time consuming to fix because pipe system is paired down to 5 features with numerous subtypes each.

Also wondering why some fields are all upper case and some all lower case? It would be good if that were consistent.

Could you all consider a BETA user group which identifies these issues for a future update/ correction?

Overall I think its a great model, a vast improvement over APDM, PODS.

Thank you,

Lee Davis

WhiteWater Midstream

[JordanMiller4]

Does the UPDM track Atmospheric corrosion points for items such as risers, meters, valves, and 1st cut regulators? We're also wondering about leak survey line feature class too. The only thing we found was the Leak Survey Area polygon feature class. 

[EdwardBlair]

Tried applying the Asset Packge gdb downloaded from UPDM site after staging a new Utility Network.    After running quite a while this failed with a message that a table is already locked.  This is a new database and new utility network.

Any suggestions would be much appreciated.

Thx,

Ed

[EdwardBlair]

BTW... at first I thought the problem was that I had a map open while running the GP tool, which the instructions clearly indicate not to do!   However I deleted the utility network, re-ran the "Stage Utility Network" tool (which worked) and re-ran "Apply Asset Package" with the Pipeline_AssetPackage.gdb with no map open and got the same results.

Ed

[RobertKrisher]

Ed,

 Did you bounce Pro after closing the map?  With the new architecture of pro (64-bit asynchronous) I've often found that locks can be pretty sticky and the only way to clear them is to bounce the whole application.  Also, if anything goes wrong while standing up an asset package I've typically found it to be easier and more reliable to scrap the whole database then to try and recover it.

-Robert

[EdwardBlair]

Did not bounce Pro after closing. Per your suggestion will try re-creating

the database from scratch and try again.

Thanks much for the info!

Ed

[jhd011]

Will there be an update to UPDM 2018 that will include the changes for the April 2018 release of the Gas Utility Network Configuration or are they already included?

Release notes for ArcGIS Pro 2.1—Get Started | ArcGIS for Gas 

[EdwardBlair]

Robert -

That was it! I bounced Pro between staging the utility network and

applying the asset package and it completed successfully. Much appreciated!

Would be good if Esri added this step to their process.

Ed

[MarkCederholm]

One thing that I've noticed: for the Controllable Valve subtype, it looks as though designtype should use the UPDM_ValveControllableType domain, not the UPDM_ValveFlowType domain.

[MarkCederholm]

There's a typo on line 2152 of 01CreateUPDMBaseAdd-On_2018.py that assigns the UPDM_ValveFlowType domain to subtype 3 rather than 4.

[MarkCederholm]

In 03CreateInspectionModule_2018.py line 392, the "Replaced Regulator" coded value for the UPDM_LeakWorkPerformed domain has an incorrect, non-unique description.  Also, on line 1541, Global IDs are added to P_LineValveInspection after enabling attachments, instead of before.

[Anonymous User]

Tom DeWitte‌

In the document that is included with zip file, Implementation Patterns for UPDM 2018.pdf, under the section Network Topology with Utility Network Data Management, Step 9 indicates that ArcGIS Pipeline Referencing attributes are added after running 03CreateAssetInspectionModule_2018.py.

1. Does this mean that going forward I need to purchase licenses(s) for the ArcGIS Pipeline Referencing extension?
2. If I implement the second pattern, Network Topoology with Utility Network Data Management, do I need to  have the ArcGIS Pipeline Referencing extension to use the Assest Inspection module in the UPDM?
3. If we don't have the ArcGIS Pipeline Reference extension, what are the limitations to using the UPDM?

In general I am confused on the necessity of ArcGIS Pipeline Referencing extension. We are a gas distribution operator with a service territory not going beyond the County boundary. We don't have transmission lines that span across county or state lines. We are planning to implement the UPDM with v10.6.1 and migrate to UN.

thanks

[TomDeWitte]

Hi Darris,

UPDM 2018 is a unique data model, in that it is the only data model in the industry that was designed to support the needs of gathering, storage, transmission, and distribution.  Most gas companies will only utilize the portion of the data model which is specific to their needs and deployment pattern.  In answer to your questions, for a distribution only gas utility:

1. Does this mean that going forward I need to purchase licenses(s) for the ArcGIS Pipeline Referencing extension?

Esri Response:  No.

2. If I implement the second pattern, Network Topoology with Utility Network Data Management, do I need to  have the ArcGIS Pipeline Referencing extension to use the Assest Inspection module in the UPDM?

Esri Response: No

3. If we don't have the ArcGIS Pipeline Reference extension, what are the limitations to using the UPDM?

Esri Response: The ArcGIS Pipeline Referencing extension provides the capability to use linear referencing to place and locate your gas facilities.  This is not a method typically used within gas distribution companies.

If you have additional questions, please respond to this thread.  

thanks

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

tdewitte@esri.com

[deleted-user-FrOR7qrGn6Op]

I've found many domains that are duplicated in the database and either not assigned to a field or incorrectly assigned to a field that is not applicable. These are all of the domains that have a prefix of "CP" and already exist with a prefix of "UPDM_CP". There are also fields that were added to the 2018 model that are inconsistent with the upper casing, as someone has already mentioned, but they are also not applicable to anything - for example, PipelineDevice has an added field of [adddetails] that is not being used by any of the subtypes and has been incorrectly assigned the domain of CP Test Point Wire Count which is a duplicate of the domain UPDM_CPTestPointNumWires. This field is in the data dictionary, although only in the list of fields and no where else where it would be relevant or applicable to the subtypes of this feature class. One feature class, ServiceTerritory in the PipeSystem dataset, is mentioned in the data dictionary in the list of feature classes but there is no description of the layer itself and it contains four [globalID] fields. Another example is where the P_Coupon feature class was incorporated into the PipelineJunction feature class as a subtype yet it is still also present as an individual feature class at the root of the database. We're currently reviewing the schema in its entirety for these types of errors and, as you can see, have identified several. Is there a new data dictionary and/or build of the database schema that can be used to help with corrections? Unfortunately we have already imported data, added customization fields for internal use of tracking as-builds, are in a production environment, and it will not be easy to make changes. Perhaps a list of fixes would be best?

[TomDeWitte]

Good morning Jeanetta,

Thank you for identifying the items you listed in your message.  We are always looking for feedback to help us improve the data model and its documentation.  We are about to begin the process of updating the data model for 2019.  On the "fix it" list is the clean-up of the coded value domains and the addition of the ServiceTerritory featureclass to the Data Dictionary.  We will also be looking at ways to further simplify the creation of UPDM and its modules, such as Darris pointed out in his message.

Our goal is to have this update completed and posted by the end of March 2019. 

Thanks

Tom DeWitte

Esri Technical Lead - Natural Gas Industry

tdewitte@esri.com

[WhiteWaterMidstream]

Tom, 

Could you all also consider adding a column for pipeline commodity/product? Most midstream and pipeline transmission operators have lines of different service and need to query systems by commodity. I added a PRODUCT column to most feature classes.

Suggested domain types: Natural Gas, Crude Oil, Natural Gas Liquids, Refined Products, Liquid Petroleum Gas, CO2, Water, Highly Volatile Liquid, Other Gas, Other Liquid

Thanks,

Lee

WhiteWater Midstream

[JoeHershman]

Junctions that could have different inlet and outlet do not seem to be supported to work with the ASTM standard.

Reducer has UPDM_ReducerFittingSize assigned, but there is no real way to get to this value from the D1/D2 or C1/C2 values that the ASTM standard returns.

Tee there is no domain assigned to the diameter field, and I do not see any field such as designtype that even indicates what the diameter(s) of the Tee are.

How does the data model support Junctions that will/could have different inlet/outlet diameters and relate it back to the ASTM standard.  Personally I think we should have fields with attached domains based on the C1, C2 values as this would align with how the ComponentTypes are configured.

Thanks

-Joe

[TomDeWitte]

Hey Joe,

You are correct, the DIAMETER2 and WALLTHICKNESS2 attributes for Devices and Junctions featureclasses was mistakenly left out of the UPDM 2018 model.  It is scheduled to be added to the UPDM 2019 Edition, which is slated for release in Q1 of 2019.

Thank you for pointing this out.  Please let us know if you find any other items which will enhance or correct UPDM.

Tom DeWitte

Technical Lead - Natural Gas Industry

Esri, Inc

[TomDeWitte]

Hi Lee,

The UPDM 2018 Edition already has an attribute named "COMMODITY" on the PIPELINELINES featureclass in the PipeSystem feature dataset.  It has a subtype specific coded domain assigned to the Subtypes: Service Pipe, Distribution Pipe, Transmission Pipe, Gathering Pipe, Station Pipe, Customer Pipe, Riser Pipe.  The values in the coded domain are: Natural Gas, Synthetic Gas, Hydrogen Gas, Propane Gas, Landfill Gas, Petroleum, Gasoline, Diesel, Water, Wastewater, Steam.  

Should this attribute be added to other assets, such as Devices or Junctions?  Or is the pipe feature the only featureclass that needs this descriptor?

When I compared your list in your message, I see you listed the following that I did not have in my list: Crude Oil, Natural Gas Liquids, Refined Products, Liquid Petroleum Gas, CO2, Highly Volatile Liquid, Other Gas, Other Liquid.  Is this a complete list?  Another question.  Some of your items are generic classifications, should be list be restricted to specific types of gas and liquids?

Appreciate hearing your thoughts and the thoughts of anyone else who wished to add input.

Thanks

Tom DeWitte

Technical Lead - Natural Gas Industry

Esri, Inc

[WhiteWaterMidstream]

Tom,

Pipeline commodity is actually a complicated question as different entities use slightly different descriptions. I did a little bit of research on the subject and have come up with a good list of commodity types which would work for UPDM. Please see attached table of recommended attribution. In the database I would propose handling this subject with 2 columns, COMMODITY & COMMODITYTYPE.

I use Pipeline Line feature class to store information on pipe material specification. Having the commodity column there is good but I think also necessary for all the key features such as assembly, device, junction, structure boundary, structure line. In the pipeline model it is very useful to be able to query systems by commodity as they are typically separate and distinct systems. If you would like to discuss in more detail please email me direct at lee@wwm-llc.com.

[EdwardBlair]

Don't know if this is a question/issue specific to UPDM or if its general to the Utility Network, but going to give it a try.   (Tom D - please let me know if I should post elsewhere.)

I've loaded my own sample data into a UPDM 2018-based asset package -- first changing the spatial reference and then appending data to the pipeline feature classes.  Validated the network topology and only a handful of topology errors remain.   Published map/feature services.   Now I can open and edit feature services in the Enterprise "Map Viewer", which I take as a good sign.  Here's the feature layer as I see it in Enterprise:

 

And here is the data as displayed in "Map Viewer":

   

However, when I try to add these services to ArcGIS Pro I get nothing.  Examining the server log shows  a bunch of "operation was attempted on an empty geometry" errors (below).  I've triple checked the source data and there are no features that have an empty geometry.   Also, I checked the service and it *is* enabled for the Utility Network (below). Scratching my head here.  

- Is Pro sensitive to lines with parametric curves?  There was one in the source data.  I deleted it and still get these errors.

- The database is SQL*Server 2017.  Is there something about geometry data storage in this database that Pro doesn't like?

- Some features have no data - StructureJunctions, for example.  Does Pro not like this?

- There is one table in the map service - its part of the Utility Network and called ValveInspection.  Is this an issue?

- Is there something about the coordinate system?  My data is in UTM meters - UTM 1983 Zone 16N.   I usually attempt to add data to an empty ArcGIS Pro map, and the map seems to take on the coordinate system of my data.  I've also tried changing the map coordinate system to the data coordinate system before adding it.  Same results.

- I'm getting my ArcGIS Pro license from ArcGIS Online but my portal in on the local test machine I've set up - would this be an issue?

Any pointers would be appreciated.

Ed

 

 

[PaulLeBlanc1]

Ed,

I cannot see any of your pictures (they appear to be hosted in GMail?). Based on what you described, nothing seems out of the ordinary.

If you create a brand new UN without applying the UPDM asset package and publish the bare bones, do you get the same error?

[EdwardBlair]

Paul -

Thanks for the feedback.  Updated the images in previous post.  Hopefully they show up now.  

If I just publish a brand new UN without the UPDM package the layers would be empty, true?   So you're suggesting to test whether the empty feature service layer would show up?

Ed

[PaulLeBlanc1]

Yes, I can see the images.

Correct, the layers would be empty -- you should be able to add this to a new map without error.

This is probably better suited as a different post since we're beyond UPDM troubleshooting at this point.

[Anonymous User]

I had a similar issue with a new empty feature class published as a feature service. My following notes are from ArcGIS v10.5 and ArcCatalog so adjust as necessary for Pro. In Catalog open the properties for each empty feature class, go to the Spatial extents, it might be 0,0,0,0 or the extents may be smaller than the area you were trying to edit ( your edits may have been outside the extents). You can import an extent from another feature class that is big enough or choose update extents. You will have to stop the service (and possibly disconnect all users) to release the edit locks to perform this step.

I suggest you update all extents even non-empty feature classes while you are in there.

[EdwardBlair]

Darris -

Thank you for the suggestion! Pretty much at the same time Hussein offered

a different set of steps that seem to work for me.

Much appreciated,

Ed

[EdwardBlair]

Hello -

Am trying to modify a subnetwork controller with no luck.  The error message says there is an "expected field [that] was not found or could not be retrieved properly".  Is there a way for me to identify the field in question?

Thx,

Ed

[MarkCederholm]

We maintain an ANSI valve rating field in our data model.  Is that common enough in the industry to warrant a UPDM device field?

[EdwardBlair]

Forgot to update this.   I must have been using an old, outdated version of the UPDM model.  When I re-started from scratch using the Gas Utility Network Service 2.0 and re-loaded my data I was able to set my subnetwork controller.

Ed