Latest Contributions by TomDeWitte

By Tom Coolidge and Tom DeWitte Earlier this year, I was walking down a street in mid-Manhattan. During the walk I noticed a surprising amount of steel plates covering active excavation projects under the streets and sidewalks. There were also many open trenches which exposed an amazing number of pipes and wires.  Not being from New York City, I was surprised at the shear volume and complexity of buried infrastructure. This got me to pondering about recent gas events and the questions an organization asks itself. -Could the damage to the pipe system have been prevented? -What organization processes could have performed better to prevent the damage? This leads to an organization self-examination of processes to determine where improvements need to be made. These self-exams are important and valuable, but they are also reactive.  Typical reactive questions which are commonly asked include:  Did the locator group correctly mark the location? Was the locator provided with accurate and timely information to assist with the locate? Were the correct engineering and construction documents provided? These reactive self-exams often overlook a root cause: inaccurate data gathering methods. The American Gas Association recently published a white paper titled “Implementing Damage Prevention in Field Operations.” The white paper can be accessed at: https://www.aga.org/contentassets/11def1ef52f844b4b06935276b911010/implementing-damage-prevention-in-field-operations-whitepaper--final-for-publish.pdf Included in the white paper is a section on “Mapping/As-Builts” that resonates with important aspects of the role of geographic information systems (GIS) in gas utilities. This blog touches on those important aspects. Inaccurate data gathering with relative offsets The AGA whitepaper allocates two of the nine sections to address the importance of data gathering techniques around the construction and mapping/as-built processes for documenting “where” the pipe components were installed. Inaccurate data gathering methods have persisted in the gas industry since its founding. In early gas company days, locating a buried pipe was based on “relative offsets”.  A relative offset is a measurement from an identifiable above ground feature to the location of the buried pipe asset.  It was not uncommon to see measurements, such as 45ft NNE of the Old Oak Tree. But what happens when the tree falls over after a storm and is removed? Later relative offset techniques switched to using street curb edges or house corners, such as 12ft South of the North curb.  This too is subject to change over time.  How accurate is this measurement when an additional lane is added to the north side of the street? As pointed out in the previously mentioned AGA white paper, these methods are inaccurate over time.  Yet, many in the industry are still using these centuries-old methods to document the location of newly installed pipe. How to proactively prevent damage Now there is an opportunity to be proactive.  To use current cost-effective data gathering tools such as Global Positioning System (GPS), lightweight digital collection tools such as mobile phones and tablets, and data management systems such as GIS.  This allows the location of buried assets to be based on its absolute location instead of its relative location.  Using absolute location eliminates the historical issues of relative features changing over time. An opportunity to improve Today’s gas organizations are undergoing a massive capital improvement process to replace or abandon many old and inaccurately mapped gas pipe components. This increase in capital projects provides the opportunity to accurately locate new infrastructure and decrease future damages.  Gas organizations can update data gathering methods to cost-effectively collect new construction with a sub-foot absolute accuracy. They can implement a geospatial system that is capable of electronically collecting and storing this information. Future engineering, construction, and locate organizations will have the confidence in knowing that their mapped pipe components are spatially accurate and reliable. If you haven’t already read the recently published AGA white paper, we encourage you to do so. It’s well worth your time. There are many other points worth noting. The next time I visit a major city and again see the steel plates and excavation pits, I hope to have the confidence in knowing that the local gas organization has been proactive and is using current data gathering tools and techniques to prevent future damage. PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions. ... View more

By Tom Coolidge and Tom DeWitte   It’s always a joy for us to see the amazing work our customers are doing with ArcGIS for the betterment of their organizations and those they serve. It’s even sweeter when a customer is in a position where they can freely share that work with others so they, too, can put it to work. That’s the case with GTI and the work they have been doing to create Survey123 templates for the natural gas industry.   For those of you who may not know, GTI is a leading non-profit research, development, and training organization addressing global energy and environmental challenges. Among their hundreds of initiatives across the energy value chain, they develop and implement tools, methodologies, and technologies for maintaining a safe and intelligent natural gas infrastructure. The GIS department within GTI has been working to encourage electronic field data collection with GIS to optimize the entire data management process for utility and pipeline operations, significantly reducing the cost and complexity of capturing real-time high-accuracy information.   With these common goals in supporting the natural gas industry, Esri and GTI collaborate to help natural gas utilities be successful in their efforts to implement and leverage geospatial solutions to address industry business challenges. More specifically, recently, we have been looking at ways to make it easier for the natural gas industry to use the robust geospatial tools that are available today.   Through this collaboration, GTI now is publishing Survey123 form templates that they have created. The first survey form to be published is Indoor Gas Meter Set Risk Assessment. This form assigns a risk score to indoor meter set evaluations in real time based on user input. The capabilities embedded in this first template are impressive. It includes examples for:                 -Using HTML tags to set text color                 -Performing calculations based on the user’s response to the questions.                 -Hidden fields                 -Context driven logic                 -pick lists   You can review this survey123 template yourself following these steps: Open Survey123 Connect    If you do not already have Survey123 Connect, you can download it from this location:    https://www.esri.com/en-us/arcgis/products/survey123/resources    2. Click on the “New Survey” option     3. Within the New Survey dialogue, select the “Community” radio button   4. Scroll through the list of posted surveys to find the survey named: GTI/OTD – Indoor Gas Meter Set Risk Assessment   5. Select GTI survey and click on the Create Survey button     6. Review the newly created survey The indoor meter set risk assessment is the first of many surveys to be posted by the GIS team at GTI.  These survey templates will help natural gas organizations of all sizes to deploy this powerful mobile data collection application.  Take a look at these surveys and see if they can help your organization improve its data collection activities.   PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions ... View more

By Tom Coolidge and Tom DeWitte   Today’s Collector and ArcGIS Enterprise provide new enhancements and capabilities. These enhancements include; improved user interface, better GPS antenna support, direct capture of barcode via the mobile device camera, and allow for a more streamlined workflow for field users.  In addition to those important enhancements, the enterprise geodatabase capability of attribute rules allows for the automatic decoding of the barcode and the derived barcode data to be automatically written to the appropriate attributes.  This automatic decoding and attribute population provides significant productivity gains for field users and allows for a simpler deployment pattern for administrators. In this blog we will take a deeper dive into how to configure and deploy the ArcGIS platform and collector to address the industry need of Tracking and Traceability. For an introductory explanation of how the ArcGIS platform addresses Tracking and Traceabiliy, please read the first blog of this 2 blog series: Tracking & Traceability – Part 1   Like any good recipe for success, we need to know the required ingredients.  The Tracking and Traceability solution requires the following software: Collector for ArcGIS ArcGIS Enterprise 10.6.1 or higher Additionally, we will need arcade scripts which provide the logic of how to decode the ASTM F2897 barcode 16-character string and use the derived data to automatically populate the appropriate attributes.   Though not required, most deployments also include a GPS Antenna to improve spatial accuracy.   The Basic deployment steps Deploying the ArcGIS Platform to meet the needs of Tracking and Traceability can be broken down into 5 steps.  These steps are: Preparing the enterprise geodatabase Creation of staging geodatabase layers Application of attribute rules Publication of staging geodatabase layers as a feature service Creation of web map for Collector   The overall data flow process for Tracking and Traceability is to have Collector post the field collected features directly to the staging geodatabase.  There is NO translation or conversion of the field collected data.  Once the field collected data is submitted to the staging geodatabase a GIS mapping technician can review the new features and append them into the enterprise geodatabase. Preparing the Enterprise Geodatabase The first step to setting up this workflow is ensuring your Enterprise Geodatabase has the required feature classes, feature class attributes and coded value domains to store the information collected in the field.   If you are starting a new enterprise geodatabase, it is recommended that you use the Esri provided pipe system data model called Utility and Pipeline Data Model (UPDM). The 2019 edition of UPDM includes everything needed to store the information collected in the field. You can download this data model with this link:  UPDM 2019 Edition download  If you have an existing enterprise geodatabase, then you need to make sure the asset feature classes have the correct attributes to store the field collected data. Examples of assets captured by field staff include, fittings, valves, and pipe segments. Here is a specific listing of the minimally required attributes: Point Asset Featureclasses Field Name Field Definition Coded Value Domain barcode Text(16)   manufacturer Text(2) Pipeline_ASTM_Manufacturer manufacturerlotno Long Integer   manufacturedate Date   manufacturecomponent Text(2) Pipeline_ASTM_Manufacture Component material Text(2) Pipeline_ASTM_Material diameter Double Pipeline_Fitting_Diameter diameter2 Double Pipeline_Fitting_Diameter wallthickness Double   wallthickness2 Double     Line Asset Featureclasses Field Name Field Definition Coded Value Domain barcode Text(16)   manufacturer Text(2) Pipeline_ASTM_Manufacturer manufacturerlotno Long Integer   manufacturedate Date   manufacturecomponent Text(2) Pipeline_ASTM_Pipe_Manufacture Component Material Text(2) Pipeline_ASTM_Material nominaldiameter Double Pipeline_Pipe_Diameter wallthickness Double Pipeline_Pipe_Wall Thickness   After your Enterprise Geodatabase is ready to accept the decoded barcode values and the appropriate ASTM F2897 coded value domains have been assigned, you are ready to create the staging geodatabase. Creating the Staging GDB This step involves setting up your staging geodatabase layers.  These layers should be a schema duplicate of the enterprise geodatabase asset layers. Being a schema duplicate will simplify the appending of data from the staging geodatabase to the enterprise geodatabase.   The simplest approach to setting up the staging geodatabase is to create schema duplicate feature classes in the enterprise geodatabase.  I recommend creating a new feature dataset to store these duplicate layers.  If using the UPDM 2019 edition data model the feature classes to duplicate are: PipelineDevice PipelineJunction PipelineLine To help keep the staging layers uniquely separate from the production layers I like to rename the layers as follows: StagingDevice StagingJunction StagingLine These duplicate layers should not have any features/records.   To properly support disconnected field capabilities, you should use the “Add GlobalID” tool to add a GlobalID field to every staging feature class.   Additionally, though not required, it is recommended that you enable “Editor Tracking” to allow all edits to have a date/time stamp and the ArcGIS platform user ID of who created and last updated the feature/record.   A final step not to be overlooked is to decide whether you want to include photos as part of the new construction data collection process.  It the answer is “yes” then remember to “Enable Attachments” for each of the layers you want to have field staff capturing photos.   With the staging geodatabase layers now created it is time for attribute rules. Application of attribute rules With ArcGIS Enterprise 10.6.1 the attribute rule capability has evolved to provide a robust automation capability for managing attributes. For Tracking and Traceability, attribute rules provide the ability to automatically read the barcode value, decode the barcode and automatically populate the derived attribute fields (manufacturer, manufacture lot #, manufacture component type, manufacture date, material, diameter, and wall thickness). When this capability is applied to the staging geodatabase layers, the auto-population occurs when Collector submits the new feature.  This means a connected mobile device running Collector to capture new construction will be able to see the decoded information while they are documenting the new assets in the field.   The following link provides the arcade attribute rule scripts and detailed documentation on how to apply them. ASTM F2897 barcode decode attribute rules  The way attribute rules work is to assign them to a single attribute field. This means the decoding of the barcode is broken out into 9 separate arcade scripts.  Here is a breakdown of how the arcade scripts are applied to the staging geodatabase layers.    StagingDevice Featureclass Attribute Fields Arcade attribute rule script manufacturer Device_Manufacturer.txt manufacturerlotno Device_Manufacturelotno.txt manufacturedate Device_ManufactureDate.txt manufacturecomponent Device_ManufactureModel.txt material Device_Material.txt diameter Device_Diameter.txt diameter2 Device_Diameter2.txt wallthickness Device_Wallthickness.txt wallthickness2 Device_Wallthickness2.txt    StagingJunction Featureclass Attribute Fields Arcade attribute rule script manufacturer Junction_Manufacturer.txt manufacturerlotno Junction_Manufacturelotno.txt manufacturedate Junction_ManufactureDate.txt manufacturecomponent Junction_ManufactureModel.txt material Junction_Material.txt diameter Junction_Diameter.txt diameter2 Junction_Diameter2.txt wallthickness Junction_Wallthickness.txt wallthickness2 Junction_Wallthickness2.txt   StagingLine Featureclass Attribute Fields Arcade attribute rule script manufacturer Line_Manufacturer.txt manufacturerlotno Line_Manufacturelotno.txt manufacturedate Line_ManufactureDate.txt manufacturecomponent Line_ManufactureComponent.txt material Line_Material.txt nominaldiameter Line_NominalDiameter.txt wallthickness Line_Wallthickness.txt   Once the attribute rules are successfully applied to your enterprise geodatabase staging layers you are ready to publish the staging layers as a feature service. Publication of staging geodatabase layers as a feature service Publishing the staging layers from ArcGIS Pro is a very straight forward process. The steps are as follows: Create a new Map Add staging gdb layers to map Symbolize layers as desired Publish map as a feature service After the map is created and the staging geodatabase layers are added to you map you will have a ArcGIS Pro map which looks like the following: I find using ArcPro for defining the symbology to be easier and quicker than using the ArcGIS Enterprise Portal map viewer tools.  Additionally, I can use more advanced symbology such as the UPDM2019_Symbols style set that is included in the UPDM 2019 Edition download.  When the layers are symbolized as desired, remove the basemaps and prepare to publish. To publish the staging layers as a feature service, use the sharing ribbons’ web layer – Publish Web Layer tool to create the feature service. With the feature service now published your staging geodatabase layers are ready for the final step which is to create the web map for Collector. Creation of web map for Collector Creating a web map for Collector is the opportunity to fine tune the interface your field staff will use for documenting the new construction.  Items to think about when creating the web map are: Scale Constraints of layers Which data fields will be exposed to the field staff Which fields will be exposed during editing Which field will be exposed during viewing Both the ArcGIS Enterprise portal map viewer or the ArcGIS Pro desktop tool can be used to accomplish this task.   When the web map is defined and saved you are now ready to take Collector to the field to being collecting your new gas pipe construction. Summary With the latest enhancements to Collector and the new attribute rule capability for enterprise geodatabases. Deploying the ArcGIS platform to address the needs of tracking and traceability is easier than ever. Five basic steps  are all that it takes to enable your field staff to efficiently capture new construction digitally and retire the time consuming and inefficient historical paper based process. Preparing the enterprise geodatabase Creation of staging geodatabase layers Application of attribute rules Publication of staging geodatabase layers as a feature service Creation of web map for Collector PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions ... View more

This zip file contains the arcade scripts to enable Collector for ArcGIS to decode the ASTM F2897 barcodes used by natural gas industry plastic pipe and component manufacturers.    Additionally, the zip file contains documentation on how to apply these arcade scripts as attribute rules and as expressions to your web maps for Collector.     If you have questions, or suggestions for further improvement of the Collector for ArcGIS digital data collection process, please post them to Geonet, so everyone can see and share the information.   Thank you Tom DeWitte Esri Technical Lead – Natural Gas Industry ... View more

By Tom Coolidge and Tom DeWitte Tracking and Traceability is now a well-established practice in the natural gas distribution industry supported by ArcGIS ® .   ArcGIS mobile app advances over the last three years have helped adoption of Tracking and Traceability activity grow. Collector for ArcGIS has evolved to now include the ability to use a mobile device’s camera to read the ASTM F2897 barcode. Collector also now includes the capability to run arcade scripts in the pop-up window while the device is disconnected from the network.  Not to be overlooked, Esri also released a new enterprise geodatabase capability called attribute rules.   Those three new capabilities have enabled many gas utilities, and increasingly gas pipe installation contractors; to use Collector to capture the location, barcode, and other information about the newly-installed pipe and its related components. These new capabilities and lessons learned from the many organizations actively using Collector for the digital as-builting portion of the Tracking and Traceability workflow have resulted in a more efficient and streamlined process for performing these tasks.   The purpose of this blog is to give an overview of how the current version of Collector, when combined with an ArcGIS 10.7 or higher enterprise geodatabase, can result in a simpler and more efficient Tracking and Traceability workflow. A second blog article will follow with a detailed explanation of the new attribute rule arcade scripts which completely automate the decoding of the ASTM F2897 barcode and the automatic population of the derived attributes. A quick review of Tracking and Traceability PHMSA proposed rules in May of 2015 to 49 CFR part 192 to address the need for operators to better ‘track’ the details and location of assets after their delivery from the manufacturer or supplier.  The rule also speaks to the need for better ‘traceability’ of assets; meaning the ability to locate assets by material, size, manufacturer, model, or other attribute.   The ASTM F2897 standard, developed collaboratively by the natural gas industry and its leading suppliers, specifies a 16-digit alphanumeric barcode format that embodies identification of a pipeline component’s manufacturer, lot number, production date, model, material, diameter, and wall thickness.  This barcode standard is now a common piece of the manufacturer provided information for plastic pipe and its plastic components.  Additional efforts spearheaded by the Gas Technology Institute are currently underway to define a more advanced barcode standard which can be applied to both steel and plastic pipe and their components.  This barcode “thing” is not going away.  Just the opposite, it is going to expand significantly in the years to come. Pattern Overview The ArcGIS deployment pattern for Tracking and Traceability is comprised of four steps, as illustrated here:     Step 1: Digital as-builting The recent improvements to Collector have made this process easier than it was just a few years ago.  The first enhancement was the revamping of the interface to simplify data entry. The second enhancement was to increase the certification of GPS vendors and their devices. Here is a link to the list of GPS receivers which can be used with Collector: https://doc.arcgis.com/en/collector/ipad/help/high-accuracy-prep.htm The third enhancement is the native ability of Collector to use the mobile device’s camera to capture the ASTM F2897 barcode. With these enhancements, field staff can go into the field and capture the as-built information of the new construction using a smart device running Collector. The smart device is Bluetooth-connected to a high precision GPS antenna.  The field staff use the high accuracy GPS antenna to capture the location of the newly installed assets. The collected location data is directly streamed into Collector as native ArcGIS features.  No translation or conversation is required.  The field staff then manually input into Collector a minimal amount of information, such as Installation Date, and installation method.  The field staff then uses the device’s camera to capture the barcode and automatically populate the BARCODE attribute of the GIS feature.  The BARCODE value contains information about the asset, such as size, material, manufacturer and manufacture date.  Once the BARCODE value is captured, the field staff no longer need to manually enter this information.   The recent enhancement to Collector supporting the ability to run arcade scripts in the pop-up window, provides the ability to immediately display the decoded data to the field staff even when the device is disconnected.   An additional capability of an Esri mobile app on a smart device or tablet is the ability to capture photos of the newly installed assets.  These photos are automatically associated to the GIS feature.   When the field staff have completed the collection of the newly installed assets, the GIS features are submitted to the staging geodatabase. Step 2: Contractor/crew assessable storage A fundamental challenge of Tracking and Traceability is how to correctly integrate high precision GPS geospatial data, with less accurate legacy geospatial data.  A key component to overcoming this challenge is the staging geodatabase.  A staging geodatabase can be either hosted in ArcGIS Online as hosted feature layers or stored on premise with a local ArcGIS Enterprise implementation. The key purpose of the staging geodatabase is to provide an easily accessible data repository for the field crews to submit their collected construction information too.  The staging geodatabase only holds the newly collected construction information.  The construction data sits in the staging geodatabase until a mapping professional using ArcGIS Desktop accesses and downloads it to the enterprise geodatabase.   With the new enterprise geodatabase capability of attribute rules, it is possible to have the captured barcode value automatically read and used to auto-populate the derived attributes manufacturer, lot number, production date, model, material, diameter, and wall thickness.  If the digital as-builting described in step 1 happens while the device is connected to the enterprise geodatabase, then Collector will automatically decode the barcode, auto-populate the derived attributes and display the decoded information immediately after the new/updated GIS feature is submitted by Collector. In the second blog, we will provide links to these arcade scripts and describe how to apply them to an enterprise geodatabase. Step 3: Append to enterprise geodatabase One of the time saving capabilities of ArcGIS Desktop is the ability to interact with data from both the staging geodatabase and the enterprise geodatabase at the same time.  This allows the mapping professional to easily select the staging geodatabase features and append them into the final enterprise geodatabase feature classes.    If the staging geodatabase layers are stored in ArcGIS Online, the previously described attribute rule arcade scripts can be applied to enterprise geodatabase layers.    NOTE: Attribute rules only work with ArcGIS Enterprise 10.7 or higher. Additionally, ArcGIS Pro is the only desktop tool to understand attribute rules.  If using ArcMap and a geometric network, it is important that the staging geodatabase layers be stored in an enterprise geodatabase and the attribute rules are applied to the staging geodatabase layers.   The standard arctoolbox geoprocessing append tool can be used to copy the newly collected GIS features from the staging geodatabase layers to the final enterprise geodatabase feature classes. Step 4: Mappers connect digital as-built with gas system With the new construction data now appended from the staging geodatabase into the enterprise geodatabase and the barcode value decoded, the mapping professional now needs to determine how to connect the high precision geospatial features with the less accurate geospatial features. The outcome of this process needs to honor two data requirements: Connecting the new features with the legacy features to create a single topologically connected gas pipe system. Preserving the high precision GPS collected geospatial coordinate data.   The recommended best practice for accomplishing this seemingly disparate set of requirements is for the enterprise geodatabase point features such as Meters, Excess Flow Valves, and Non-Controllable Fittings to have the following attributes added: SPATIALACCURACY, GPSX, GPSY, GPSZ.  Here is another example where attribute rules can streamline the population of these GPS fields.  If using ArcMap and the geometric network, then a configuration of Esri’s Attribute Assistant tool or ArcFM’s AutoUpdater capability can be used to automatically populate these fields.  This will preserve the original GPS location values, which can be used later to rubbersheet all features (legacy and GPS) to the more accurate GPS location preserved in the GPSX, GPSY, and GPSZ attributes.  With the GPS location preserved, the mapper can adjust the new construction features as required to connect to the legacy gas pipe system. Business value of using ArcGIS platform This approach to Tracking and Traceability provides an opportunity for the GIS department to once again show the greater gas organization that not only can the GIS Department provide a solution which addresses this new common industry practice, but it can do so in a manner that improves the operational efficiency of the gas organization.  This pattern improves the operational efficiency of the gas organization and their contractors as follows: Using Collector to collect construction data improves location accuracy and attribute quality by eliminating translation to paper and interpretation of paper based information. Bluetooth integration with high precision GPS antennas improves the speed at which data is collected. Capturing the barcode value reduces the amount of information the field staff manually collects. Material, diameter, manufacturer, manufacture model, manufacture data, manufacture lot number are all automatically populated by the decoding of the barcode. Digitally collected data is immediately available for GIS department to process into enterprise geodatabase. This eliminates the historical latency problem of the GIS department waiting for the inter office mail transmittal of the construction packet. The GIS department mapping professional task of updating the as-built representation of the gas pipe system is simplified. The mapper is no longer manually transposing paper based red-line drawings, but instead appending field collected geospatial features.  This improves the speed at which a mapper can complete the task of updating the as-built representation of the gas pipe system. Safety of field operations staff is improved by providing the new construction data in a timelier manner.   This deployment pattern not only provides the ability to improve the efficiency of the field data collection, it improves the productivity of the mapping professional, and provides new construction updates to locators and field operations staff in a timely manner.   Next blog In our next blog, we will dig into how to configure and deploy the arcade scripts for this solution to Tracking and Traceability. PLEASE NOTE: The postings on this site are my own and don’t necessarily represent Esri’s position, strategies, or opinions. ... View more