Latest Contributions by TomDeWitte

AutoCAD and ArcGIS Working Together By Tom DeWitte and Tom Coolidge Esri released in January of 2024 an updated version of its ArcGIS for AutoCAD plug-in for Autodesk® AutoCAD®, AutoCAD Map 3D®, and Civil 3D®. The Esri plug-in allows AutoCAD users to become editors of ArcGIS-managed data. The January 2024 update (version 430) of ArcGIS for AutoCAD enhances the editing experience by supporting branch versioning within AutoCAD. This branch versioning support further strengthens the incorporation of the AutoCAD editor into the data management workflows within the world of ArcGIS. What do these new tools and capabilities mean for Utilities? Within the world of documenting new construction projects for utilities, there is a common practice that is needed but very inefficient. This is the practice of using AutoCAD to document the construction project and ArcGIS to document the current as-built state of the entire utility system. After construction, AutoCAD drawings are created as part of the project closeout activity to document what was installed, retired, and modified. The AutoCAD drawings are often converted into PDF files and stored in a document management system as a historical project snapshot. At the same time, the ArcGIS mapping team is updating the as-built representation of the entire utility system to reflect the newly constructed changes. This information is shared with the organization through web and mobile applications, giving the whole utility organization a current representation of the entire utility network. The problem with this dual documentation workflow is that the AutoCAD editor and the ArcGIS mapper perform the same data edits. The ArcGIS for AutoCAD plugin provides a solution to this redundant data entry. Eliminate the Duplication The ArcGIS for AutoCAD plugin solves this duplicate data entry issue by allowing the AutoCAD user to update their CAD drawing with the data edits made by the ArcGIS mapping team. Now, a simple “Synchronize” from the ArcGIS ribbon in AutoCAD allows the AutoCAD mapper to update their AutoCAD drawing with the features created by the ArcGIS mapper. Since the ArcGIS data maps an area much larger than the project area of interest to the AutoCAD mapper, a blue project area bounding box is used in AutoCAD to define the geographic extent of data of interest to the AutoCAD user. When the synchronization is complete, the GIS features within the project area are loaded into the AutoCAD drawing. No additional conversion, prep, or translation is required. This synchronization with ArcGIS is more than simply bringing over geometries. It also brings over features’ attributes, including coded value domain descriptions. CAD And GIS Working Together The increasing collaboration between Esri and Autodesk directly benefits Utilities. The ability of AutoCAD to consume ArcGIS layers of data and basemaps simplifies their tasks for creating CAD drawings and improves their productivity. Long gone are the days when the CAD community and GIS community were antagonists to each other. Now, we can all work off the same dataset. PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions. ... View more

   Beware the Ides of March By Tom DeWitte and Tom Coolidge Beware the Ides of March. Stated in less Shakespearean form, beware of March 15th. This statement was true for Julius Caesar and is true today for many GIS and compliance professionals across the United States natural gas industry. In Roman times, March 15th was the deadline for settling debts. Today in the United States, March 15th is the deadline for submitting annual reports for gas distribution and transmission companies to the U.S. Department of Transportation (U.S. DOT) Pipeline and Hazardous Materials Safety Administration (PHMSA). The submittal of the U.S. DOT PHMSA Annual Report for Gas Distribution (Form F 7100.1-1) and its sibling, the U.S. DOT PHMSA Annual Report for Gas Transmission and Gathering Pipeline Systems (F 7100.2-1), is an exercise in data mining. If your data is well organized and attuned to the data summarization requirements of these reports, gathering the information required to complete these forms is manageable. Suppose your data is not attuned to the data query, filtering, and summarization needs of these reports' data query, filtering, and summarization needs. In that case, this annual effort is scarier than having a best friend named Brutus. Officially, PHMSA estimates that the time required to gather the data needed to complete these forms is 16 hours for Gas Distribution and 47 hours for the Gas Transmission and Gas Gathering form. This is achievable with a well-attuned geospatial dataset such as an ArcGIS data repository organized with the Utility and Pipeline Data Model (UPDM). If your data is poorly organized, this effort can take several months. What does a well-attuned geospatial dataset look like? Summarizing by Pipe Material The first indication of a data model well attuned to these reports' data summarization needs is how material is defined. Part B of the Gas Distribution report does not ask for the specific material grade of the pipe (PE2708, Grade X42, etc.). It asks for a generic categorization of material (Plastic, Coated Steel, Bare Steel, Cast Iron, etc.). Part D of the Gas Transmission and Gas Gathering report similarly asked for a generic categorization of the material instead of the specific grade. The UPDM data model explicitly defines these categorizations of material in a data field called; AssetType. This AssetType data field has a set list of values which align with the categorization material values requested in the DOT reports. A simple summarization of the assettype and cptraceability data fields will produce the data values needed for section B1 of the Gas Distribution Report. A summarization of unique combinations of assettype, cptraceability, and regulatorytype will produce the data values needed for Part D of the Gas Transmission and Gas Gathering report. These summarizations can be easily accomplished in ArcGIS with UPDM-organized data in minutes versus hours, days, and even weeks. Summarizing Transmission By SMYS Part K of the Gas Transmission and Gas Gathering Report asks for the miles of transmission pipe based on the operating pressure as a percent of specified minimum yield strength (SMYS) for a given DOT class designation. Storing engineering information about the pipe as descriptors about the pipe greatly simplifies summarizing this information for the Gas Transmission and Gas Gathering Report. When the UPDM data fields for this engineering data are populated, this tabulation can be performed. The specific data fields within the pipelineline layer of UPDM are operatingpressure, SMYS, percentSMYS, dotclass and shape_length. Similar to the mileage by material table, a summarization of the unique combinations of percentSMYS, and dotclass, then summing the total length of pipe produces the information needed to populate this table. Summarizing Leaks by Cause UPDM and its optimization of data organization to support the reporting needs of the annual DOT reports extend beyond pipes. It is also attuned to the reporting requirements of leaks. Part C of the Gas Distribution report asks for the number and severity of leaks on mains and services. Here again, the P_GasLeaks featureclass in UPDM is intentionally organized to accommodate this categorization of leak cause and severity. The data field, leakcause includes a set list of valid values, which correlates to the Cause of Leak categories of the reporting table. The data field, revisedleakclass provides the leak severity distinction, and the leakstatus data field provides the indication of the status of the leak. UPDM Alignment with DOT Reporting was no Accident The fact that UPDM is so well aligned with the reporting needs of these annual reports is not an accident of data modeling. When UPDM was originally created, these DOT reports were studied. The intent way back in 2009 was to organize this information to not only address the daily engineering and operational needs of natural gas organizations but also to simplify this required annual reporting task. In the 15 years since the original release of UPDM, the gas team at Esri has continued to monitor the evolution of these compliance reports and, when needed modify UPDM to accommodate changes in reporting. March 15th was a very bad day for Julius Caesar; it does not have to be a bad day for you.   PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions. ... View more

Utility and Pipeline Data Model 2023 is Released By Tom DeWitte and Tom Coolidge Esri’s Utility and Pipeline Data Model (UPDM) 2023 is available now. This release continues Esri’s practice of maintaining a template data model ready “out-of-the-box” to manage gas and hazardous liquid pipe system data within an Esri geodatabase. This release includes enhancements to keep up with changes in industry practice and implementation feedback received since the previous release. Updates to Industry Barcode Standard In mid-2023, the ASTM technical committee overseeing the F2897 barcode standard published an update. F2897 is the Tracking and Traceability encoding system defining the format of the barcode that manufacturers place on their pipe, fittings, valves, and other assets. This update added new materials, new manufacturer components and new manufacturers to the standard. New materials include PE80, PE100 and Reinforced Epoxy Resin, to name a few of the additions. New manufacturer components include new types of pipe, new types of fittings, and more combination components such as Reducer Socket Fusion with EFV. There were also four new manufacturers added to the standard. These additional manufacturers are:                 -Improved Piping Products                 -Shawcor Composite Production Systems                 -Krah USA LLC                 -Hawkeye Industries Inc All these modifications now have been baked into UPDM 2023 to continue its support of this important industry standard. Updates Based on Implementation Feedback There are now dozens of gas and hazardous liquid organizations in production with UPDM. As each organization goes through its journey of implementing this geospatial system of record, Esri’s industry data model gets tested and retested. These projects are located globally, and our implementation partners continue to provide feedback when a gap is identified. This list of recommendations from implementations is short, as the data model has reached a level of maturity and completeness that limits the need for change. For this release we have: added Storage Facility as a new type of PipelineAssembly. added Olet as a new group to PipelineJunction to cover a type of fitting known as olets.  Weldolet, Threadolet, and Sockolet are types of olets that are now a part of our industry data model. added Service Station as an additional type of PipelineDevice to support the emerging idea of compressed natural gas, and hydrogen as fuels for transportation. Gas and Pipeline Referencing Utility Network Foundation For many gas utility and hazardous liquid pipeline enterprises, deploying ArcGIS is more than simply loading the UPDM 2023 data model into an enterprise geodatabase. That’s because ArcGIS leverages the concepts of a service-oriented web GIS. It requires additional steps, such as creating an ArcGIS Pro map configured for publishing the data model, publishing of the Pro map to create the required map and feature services and, perhaps, configuring a location referencing system. To help simplify these additional steps performed with UPDM 2023, Esri has embedded UPDM 2023 into the Gas and Pipeline Referencing Utility Network Foundation. This solution provides UPDM 2023, sample data, and an ArcGIS Pro project configured with tasks and performance-optimized maps. You can access this solution from the Esri ArcGIS for Gas solution site. A full data dictionary of UPDM 2023 is available online. A change log documenting the full list of changes incorporated into UPDM 2023 is also available online. Esri’s Template Data Model for the Industry Esri first released UPDM in 2015 as a part of a new vision of how a geospatial system of record for pipe systems can be much more than a departmental solution. It can be a foundational enterprise system providing a unified office and mobile workforce with a near real time single source of the truth. This belief that there should be a single source of the truth from which the entire organization can view, query, create and maintain their entire pipe network has driven not only the development of this industry data model, but also the development of our network management and linear referencing capabilities. UPDM is the only industry data model built by Esri in collaboration with Esri’s ArcGIS software development team to support the enterprise needs of the gas and hazardous liquid pipe industries. UPDM is a moderately normalized data model that explicitly represents each physical component of a gas pipe network from the wellhead to the customer meter, or a hazardous liquids pipe network from the wellhead to the terminal or delivery point, in a single database table object. This freely available data model is designed to take full advantage of the capabilities of the geodatabase. The data model is created and tested with ArcGIS products to ensure that it works. This significantly reduces the complexity, time, and cost to implement a spatially enabled hazardous liquid or gas pipe system data repository. Looking ahead to the Future A wise man once said “change is the only constant.” This is a great quote when thinking about UPDM going forward. The Esri development team will continue to enhance the capabilities of ArcGIS. Industry will continue to evolve its practices. To continue adjusting to industry practices and incorporating new ArcGIS capabilities, UPDM will continue to evolve.  This evolution will help assure gas utilities and hazardous liquid pipeline operators that their GIS industry-specific data model is current with their needs. PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions. ... View more

By Tom DeWitte, Kevin Ruggiero, Mike Hirschheimer Part 5 of 5 Maintaining a local cache of data on a mobile device, has historically been a challenge for organizations large and small. The paper medium used for the first 150 years of the utility industry was unable to automatically update the printed copies sitting in a mobile worker’s horse wagon or truck. The first generation of mobile map viewer applications were also static. Updating these digital snapshots of data meant fully replacing the mobile device’s local cache of data. This was time consuming to the mobile worker, extremely taxing of the data server being asked to extract its contents and congested most organizations communication networks. Neither of these options are ideal for providing utility field workers with the up-to-date information they require to safely perform their jobs. What utilities desire is a mechanism that not only maintains the mobile device’s local cache of data without requiring dedicated time from the mobile worker, but also minimizes the query load against the data servers, and minimizes the impact on the communication network. What utilities desire is offline map areas. Offline Map Areas Offline Map Areas is the name Esri gives to the capability that enables the ArcGIS system to automatically transmit data changes between a mobile device and the organization’s ArcGIS data repositories for a specified geographic extent. In the previous articles of this blog series, we described the first three steps required to deploy this capability within an organization.   In this blog article we will explain the fourth and final step, deploying to the mobile device. The process of deploying to the mobile device can be divided into two parts, the initial download, and the updating of the mobile device’s data cache. How is the data initially deployed to the device When an offline map area is initially defined, a package of the requested data, layers and tables within a geographic extent are written to the portal server. This process of packaging an offline map area’s data once and storing it on the portal server addresses the issue of overwhelming the enterprise databases. By allowing mobile devices to be initially provisioned with a pre-packaged set of data, mobile workers, contractors, and mutual aid crews can quickly deploy. Each feature service included within the web map will have its extracted data written as a mobile geodatabase to a unique SQLite database file. Each image service defined within the web map will have its extracted data written to a tile package. Each vector tile package will have its extracted data written to a vector tile package. For example, a web map referencing 2 feature services, 1 hosted feature service, and 1 vector tile service will have a pre-packaged set of 3 SQLite files and 1 vector tile package sitting on the portal server waiting for download.   For a typical installation of ArcGIS Enterprise, these files will be stored in the arcgisportal install directory. NOTE: When an offline map area is created a companion scheduled process will also be created to update the portal server package. By default, this will be a weekly update. Requesting the download of an offline map area is a simple process. Within Field Maps the user will click on the download icon next to the desired offline map area. The simplicity of the download eliminates the historically time-consuming task of manually loading the packaged data onto the device. When the download is initiated, it will automatically sequence thru three steps.  The first step is to copy the portal hosted files to the mobile device. The 2nd step is to register the mobile device with the requested geodatabases. In this step each geodatabase or Data Store (hosted feature layers) will register the requesting mobile device and begin tracking the last successful synchronization date and time. This tracking of last successful synchronization date and time is how all future syncs will know which records to request to update the mobile device with changes made since the last successful sync. The 3rd step is to update the mobile device mobile geodatabases (SQLite files) with all changes made since the portal package’s last synchronization date. In this step all changes made to the enterprise data repositories (enterprise geodatabase and Portal Data Store), since the portal package was last synced, and within the defined geographic extent will be queried and downloaded to the mobile device’s existing mobile geodatabases. This ability to track the date and time of when a mobile device’s synchronization was last successfully completed for each of the local data repositories referenced by the web map is how offline map areas minimizes the network traffic required to keep current the mobile device’s data cache. With an offline map area now successfully downloaded, registered, and updated it is ready to be taken into the field for mobile worker use. Updating the Mobile Device Data Cache The mobile worker with their mobile device local cache of data, can view, and query existing data. The mobile worker can also update existing data, create new data, and delete existing data, if the web map and feature service have been configured to allow these capabilities. With this full set of editing capabilities against the local cache of data, we need to understand how those changes get back to the enterprise data repositories. When we talk about offline map area synchronization, we are talking about only receiving and sending changes to the data. Like step 3 of the initial download, the determination of what needs to be transmitted between the mobile device and the enterprise environment is based on the last successful synchronization datetime and the geographic extent of the offline map area. This datetime value is stored in both the mobile device’s mobile geodatabases, and each unique enterprise data repository providing feature data. If no changes have occurred since the last sync, there is no data to send or receive across the network. ArcGIS Field Maps provides the ability to automate this bi-directional synchronization. When auto-sync is turned on it will automatically initiate a sync when it detects a network connection to the enterprise environment. This auto-sync will occur every 15 minutes if a persistent connection to the network is maintained. For the mobile user this provides a very simple daily workflow. On a typical day the mobile worker would start in the office. While in the office the mobile worker simply opens Field Maps and selects the web map offline map area they intend to use for the day. By opening the map while connected to the office network Field Maps will automatically initiate a sync.  With the beginning of the workday sync completed, the mobile worker is ready to leave the office and head out to the field. During the day the mobile worker can view, query, and edit against their local offline map area. All edits will be stored on the device. At the conclusion of the workday, the mobile worker returns to the office. The mobile worker by simply coming into range of the office wi-fi, initiates the sync. This is due to Field Maps detecting a connection to the network and automatically initiating the sync. The mobile device never needs to leave the mobile worker’s pocket or pouch. This end of day sync will push the day’s edits to the servers and update the mobile data cache with changes posted to the enterprise environment. This automated sync frees the mobile worker from having to dedicate time to managing the local cache of data. Managing Changes to the Data Structure With the offline map areas generated and easily deployed to the mobile workforce, administrators will be asking: what happens when the structure of the data changes? The first type of data structure change to consider is the addition or removal of a layer or table to the web map. For these changes, the Update tool within the Field Maps web application is the tool to use. The update tool is very simple to use, simply click on the Update button and the Portal hosted offline map area package will be updated. When the Update tool is run it will initiate the updating of those mobile geodatabases within the Portal hosted offline map area package, which have had layers/tables added or removed. Additionally, the Update tool process will remove mobile geodatabases whose content is no longer referenced by the web map. When the schema of an existing layer or table is modified the Recreate tool is used. Examples of schema changes which require the Recreate tool are the addition or deletion of data fields, and modifications to coded value domains, range domains, and contingent value lists. Running the recreate offline area tool will delete and recreate all the mobile geodatabases contained within the portal hosted package. Managing Changes to the Mobile Devices Administering a fleet of mobile devices and the data caches stored on them can involve situations not everyone thinks about during initial planning and user acceptance testing (UAT).  For example, how does an organization deactivate a mobile device from syncing when that device has been lost or stolen. Or how do you unregister a mobile device’s offline map area when the device breaks? A different set of administration tools is needed to manage these types of situations. To manage the offline map area registration data stored within the enterprise geodatabase we need ArcGIS Pro. ArcGIS Pro provides the tools to visualize and manage all the offline map areas (feature service replicas), registered to an enterprise geodatabase. Accessing these Manage Replica tools is done via an administrative connection directly to the enterprise geodatabase. Within the Manage Replica tools is a tab for Feature Service Replicas.  This tab will present the list of all mobile device caches registered to the enterprise geodatabase. Searching by the username of the person whose device has failed or is lost, you can find the replicas to that mobile user. From this listing you can select the feature service replica which needs to be removed and unregister.  Unregistering will remove its entries from the enterprise geodatabase and prevent synchronization. Summary Unlike the book of paper maps originally stored under the bench of the horse drawn utility wagon, today’s technology enables utility organization to provide mobile users with a local cache of up-to-date information. Offline Map Areas provides the ease of use your mobile users expect from modern computing systems. These offline map areas are maintained with modern data management and communication methodologies which minimize load onto the enterprise geodatabases, and consumption of network bandwidth. All of this is done with a mature set of administrative tools, to simplify management of this enterprise system. About this Blog Series This is the fifth and final blog in our series on offline map areas. You can access our previous blogs with these links. The first blog provided and overview of offline map areas. The second blog provided details on preparing your data for offline usage. The third blog provided details and options for publishing the selected data repositories. The fourth blog provided details on how to automate the creation of the offline map areas. PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.                 ... View more

 Taking Your Maps Offline: Creating Offline Map Areas By Mike Hirschheimer, Tom DeWitte, Kevin Ruggiero Part 4 of 5 Your Utility wants to setup a process that enables field workers to have access to your pipes, conductors, and cabling geospatial data when not connected to the communication network. The GIS/IT support team has read the ArcGIS documentation for offline map areas and is ready to begin. For an initial proof of concept (POC) the team created new web maps and then used the Field Maps Designer web app tools to manually define the extent of a single offline map area.  The feedback from the field has been positive and now leadership wants to expand to cover the entire service territory which includes 100 distinct areas.  The GIS/IT team is super excited about the project moving forward but secretly cringes when thinking about the effort and hours it will take to manually repeat the steps from the POC for the remaining 100+ areas.  A similar scenario happened to our team earlier this year.  Our customer had 100 “operating areas” throughout their service territory and manually repeating the steps executed in the POC for each offline area wasn’t sustainable.  We needed a way to automate the creation of these 100 offline map areas. The ArcGIS API for Python provided the scripting ability that was needed for this automation. In this blog, we’ll discuss best practices for managing the web maps along with using the ArcGIS API for Python to create and report on offline map areas. Configure the web maps The data that the field worker sees on their mobile device is driven by the layers, tables, and configuration in the web map.  From ArcGIS documentation, we know that a single web map can support up to 16 offline map areas.  With 100 areas to define, a minimum of 7 web maps is needed (100 / 16 = 6.25) if the areas were broken up evenly.  In reality, it took 13 web maps as the ease of use for the field worker was a major factor when making assignments. Managing 13 web maps can become overwhelming pretty quickly when a single change would need to applied 13 times.  The recommendation is to create a “Master” web map using ArcGIS Pro that includes the facilities, landbase features, the offline enabled base map, and other pertinent data feeds.  Once the configurations like scale ranges, pop-ups, labels, locators, etc. are set, publish the web map to Portal using the Share Tab à Save as Web Map button.  Then the “Master” web map can be copied and given a new name as many times as needed to support our offline map areas requirements.  NOTE: If using ArcGIS Enterprise 10.9.1 or 11.1, don’t include Subtype Group Layers in your web maps.  The offline map area creation tool will fail because it doesn’t understand Subtype Group Layers. Creating the Offline Map Areas There are 2 approaches to create the offline map areas.  The Field Maps Designer web application could be used to manually define polygon extents, or the ArcGIS API for Python could be used to programmatically generate the polygon extents based on the operating area polygon layer most utilities already have. In the Field Maps Designer web app, there are tools to draw a rectangle or a polygon to define the extent and then enter information about the area (name, how often to refresh, levels needed in the basemap).  For a POC, this approach is quick and easy but doesn’t scale well when moving into production mode. The programmatic approach uses a Python script to automate the creation of offline map areas.   Attached to this blog is a script that creates offline map areas using a polygon feature’s geometry.  The script uses a configuration file that contains your ArcGIS Enterprise credentials, the URL of your polygon layer of operating areas, Web Map IDs and the instructions for creating the offline map areas. In this sample configuration file, there are 2 web maps that are defined in the offlineAreaConfig section. When the script runs, offline map areas will be created for the polygon features that satisfy the “polygonWhereClause”.  In both instances, both Virginia and Colorado had less than 16 polygons.  If these states contained more than 16 areas, the where clause would need to be redefined. Knowing that every utilities’ data model will be slightly different, multiple entries in the configuration file are used.  This gives the script flexibility to name the offline map areas using the fields in your data along with allowing the script to run in your Dev, Test & Production environments without making code changes. Another benefit of using the Python script is the log file.  This provides documentation as to when each offline map areas was created and the duration it took.  Reporting on the Offline Map Areas After setting up the configuration file and running the python script, 100 offline map areas in 13 web maps have been created.  A simple way to report on these offline map areas is now needed.  The log file has good information but wasn’t meant to be a report.  There is another attached python script that generates a CSV file containing details about web maps with offline map areas.  The CSV identifies the contents of each offline map area and the files sizes of every Vector Tile package and SQLite database.  Not only is this report useful for knowing what exists in your Portal but also show the amount of data to be downloaded to a mobile device. What needs to occur if I need to change the web map? Your Utility has been in production for a few months now and the Field workers are actively using the offline capabilities.  To be more efficient, they are asking for a couple of changes to the web map.    Specifically, a labeling change on a linear feature, a change to the order of attributes on a specific device and the need to capture a new attribute on a structure. Making those changes the web map is easy enough but that won’t initiate the change to the offline map areas.  Once an offline map area is created, it’s schema and it’s corresponding web map is locked down.  New web maps need to be created to get these changes available to the field worker. To apply schema or map changes, the workflow would be something like this: In Portal, existing web maps with offline map areas should be renamed and/or deleted Make the necessary schema changes (ex.  Add a field) Using ArcGIS Pro, update the “Master” web map in Portal with the labeling and pop-up configuration Make the necessary copies of the “Master” web map Re-run the Python script to create offline map areas Notify field users that existing offline map areas should be deleted from their device and that newly created offline map areas are available for download As you can see, this python script not only helps with the initial creation and deployment of offline map areas, but it also streamlines the ongoing support and propagation of changes. Simplifying a GIS Administrators Job The recommendations in this blog are the real-world lessons we learned from assisting a large utility customer with deploying offline map areas. From creating the master web map to using the Python API to automate the creation of the offline map areas.  These learned lessons removed the GIS administrator cringe of doing time-consuming manual processes for the deployment, and instead provided an automated process allowing our servers to continue working while we went home. If you are interested in using the python scripts described in this blog, they are available on Esri GitHub so that you don’t have to start coding from scratch. About this Blog Series This is the fourth blog in our series on offline map areas. In future blog articles we will continue to explain decisions an administrator will need to make during deployment. The first blog provided and overview of offline map areas. The second blog provided details on preparing your data for offline usage. The third blog will provide details and options for publishing the selected data repositories. The fifth and final blog will provide details on the deployment and management of offline map areas for a large mobile workforce. PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.   ... View more

Taking Your Maps Offline: Publishing the Data By Tom DeWitte, Kevin Ruggiero, Mike Hirschheimer Part 3 of 5 Organizations large and small need to keep their mobile workers informed. Whether it is a blue-sky day or storms are brewing overhead, mobile workers need current information on the utility system they spend every day maintaining. This information informs the mobile worker about which conductors are energized, pipes are pressurized, and cables are live. Critical information that helps to keep the mobile worker safe and the utility system reliable. Consistently and reliably transmitting new and updated information to a utilities entire mobile workforce is not easy. The paper maps deployed for the initial 150 years of the utility industry were unable to receive incremental updates. The day after the paper maps were put into the hands of the mobile worker, the information on the map was already out-of-date. The same issue applied to early mobile map viewer applications, like ArcReader. The snapshot of information stored locally on the mobile device was unable to be incrementally updated. It too became more and more outdated with every passing day after the snapshot of information was created. These lessons learned from previous efforts to provide timely and current information to the mobile worker, tell us that a mechanism is needed to update a mobile worker’s information incrementally and consistently. In today’s mobile world, this equates to maintaining a local copy of the geospatial representation of the pipes, conductors and cables on the mobile workers phone, tablet, or laptop. The mechanism of transmitting this information to the mobile device when it is connected to a network is web services. What Are Web Services A web service is a method of communication between your mobile device and your data server. When a web service is created it is an always-on process, that is listening for mobile client applications to send requests. There are many different types of web services. Within ArcGIS Enterprise there are specialized types of web services for geocoding, geoprocessing, sharing image data, and sharing vector data.  The sharing of vector data is how feature and table records are passed between the mobile device and the centralized data repository. ArcGIS Enterprise supports multiple types of vector data web services. These include, KML, WFS, ArcGIS Feature Services and Hosted Feature Layers. ArcGIS Feature Services and Hosted Feature Layers are the only vector data sharing web services which support the synchronization of changes between the mobile device and the centralized data repository. Put another way, to keep our mobile workers informed with data changes being made by office staff and data changes being made by other mobile workers, there must be either a Feature Service or a Hosted Feature Layer to handle the communication. A feature service is how ArcGIS Enterprise Geodatabase managed data such as Utility Network data is shared to mobile devices for synchronization. A hosted feature service is how ArcGIS Portal hosted feature layer data is shared to the mobile devices for synchronization. Organizing Data with Feature Services In the example we have been describing in this blog series, the data that needs to be made available to the mobile device for local caching is coming from four different data repositories. Two of the data repositories are Enterprise Geodatabases, one data repository is the Portal’s hosted feature layers, and the fourth is an ArcGIS Online registered vector tile basemap configured for export. Each Enterprise Geodatabase will contain multiple featureclasses and tables which need to have their records synchronized to the mobile devices. A single feature service can be a grouping of 1 or more featureclasses and tables. When trying to determine how to organize your data into feature services, it is important to know that each individual feature service can only connect to one data repository.  This means that all featureclasses and table must be accessing the same enterprise geodatabase thru the same relational database connection. Publishing data for offline use has some additional constraints on the organization of content in the feature service.                   -A featureclass or table can only be referenced once.                 -Subtype group layers are not a supported layer type        Publishing a Feature Service for Syncing Creating a feature service can also be referred to as publishing the data. Publishing the data is the 2nd of the four primary steps in creating offline map areas. The “Publish the Data” step is typically accomplished with the desktop tool, ArcGIS Pro. The “Share as Web Layer” tool is the specific tool to use to publish the feature service. By default, a feature service does not support synchronization. To activate synchronization, check the box “Enable Sync” within the feature properties configuration panel. Many of the featureclasses in enterprise geodatabases have their geometries configured to support the storage of elevation (Z) and distance along a line (M). Field editors of this Z-enabled and M-enabled data may not have this information available at the time of data capture. To allow field editors to collect this information without a complete geometry definition, default values and behavior need to be defined. For the Z value check the box to turn on the “Apply default to features with Z-values”. Then set the default z-value to the desired value. For the M value check the box to turn on the “Allow geometry updates without m-value”. This will set the M value to NULL when a new feature is created.       Publishing Your Utility Network Key to keeping mobile workers safe is providing current information about their utility assets. When these assets are stored in an enterprise geodatabase and managed with the utility network capabilities, the publishing step is a matter of modifying the properties of an already published feature service. For the office mappers to create and maintain the digital connected representation of your utility assets, requires that the data be configured as branch versioned and published as a feature service. The already published utility network feature service may not be fully configured for syncing with offline map areas. To support syncing to offline map areas requires two configurations of the feature service that will need to be set. The first is to modify the already mentioned feature service properties to enable syncing. The 2nd configuration is to define the role that branch versioning will have during the synchronization process.  There are two options for branch versioning during sync: -None -Create a version for each downloaded map To know which option to set starts with an understanding of what your mobile workers will do with this utility asset data. Version Creation = None If the answer to this question is that mobile workers will only view and query the utility network data, then the default setting of “None” is the correct setting. With this configuration no additional versions are created or required to successfully sync. If you set the Version Creation option to “None” and allow your mobile workers to edit the utility assets, those edits will be posted directly to the default version. Syncing to the default version will share these edits immediately to all other office and field users. Version Creation = Version for each downloaded map This option is recommended when your mobile workers are editing the utility network managed utility assets, and those edits need to be reviewed and quality control checked prior to sharing with the rest of the organization. When this option is chosen, a branch version is created within the enterprise geodatabase when the mobile worker downloads the offline map area for the first time. All field edits will be synchronized to the created branch version. A more detailed explanation is available within the Esri online documentation. Version Creation = Create a version for each user This option is not valid for branch versioned feature services. With these two configurations set, the utility network feature service is ready to support offline map area synchronization. Publishing Your Landbase Unlike the Utility Network, a feature service may not already exist for sharing the landbase information. Office mappers may be using the ArcMap desktop application and its direct connection capability to access the landbase data stored in an enterprise geodatabase. In this example configuration the enterprise geodatabase will be using traditional versioning to track and manage changes to the landbase. Publishing the landbase feature service to support offline map area syncing requires checking feature service configuration option, Enable Sync. With sync enabled you now must decide what versioning strategy to deploy to support the mobile users. If your mobile users are not intended to create or modify the landbase features, then the correct Sync Version Creation option is: None. If you mobile users will be creating new landbase features or modifying existing landbase features then a version needs to be created to manage the flow of landbase edits. Two version management options are available for traditional versioning: “Create a version for each downloaded map” or “Create a version for each user”. A more detailed explanation of these sync version feature service options is available with the Esri online documentation. Scaling your Feature Services At the beginning of each workday there will likely be a spike in the number of mobile devices attempting to sync to the published feature services. This spike is caused by the mobile workers opening Field Maps and the offline map area enabled web map on their mobile device. This will initiate a synchronization. The capacity of a single feature service to scale to accommodate this morning spike in concurrent devices requesting to sync is defined within the pooling portion of a feature service’s properties. The pooling parameters which directly impact capacity are: Minimum number of instances per machine” and “Maximum number of instances per machine”. These parameters are adjustable after the feature service is published. The Minimum number of instances per machine value represents the base level of concurrent users that a single server can support.  As the number of requests for data increases the server will automatically start creating new additional instances.  This will continue until either the concurrent demand for data is met or the Maximum number of instances per machine is reached. If the number of concurrent user requests exceeds the maximum number of instances value, additional users will be put into a queue and must wait for an instance to become available. After the morning spike in synchronization, the number of concurrent users requesting data from the feature service will decline.  This will cause the number of instantiated instances to decline until the minimum number of instances per machine value is reached. Publishing Your Field Collected Data Hosted Feature layers provide another method of publishing data for synchronization to mobile devices. The publication process is integrated into the Hosted Feature layer initial creation process. Once a Hosted Feature layer is created it is automatically published as a feature service. Similar to an Enterprise published feature service, a hosted feature service is not sync enabled when it is initially created. Enabling sync is a task for the hosted feature layer owner, modifying the settings. Checking the box to Enable Sync is all that is required to enable syncing. Registering the Basemap to your Org The final data source to prepare for offline map areas is the basemap. For our example we will use an Esri curated basemap. Most of the Esri provided basemaps are not compatible with offline map area syncing. All ArcGIS Enterprise referenced default basemaps are NOT valid for offline map areas. Only a subset of the ArcGIS Online basemaps is valid for use within offline map areas. A listing of the 19 Esri vector basemaps for offline map areas (for export) can be viewed here. Using these ArcGIS Online basemaps in ArcGIS Enterprise requires that the basemap be registered as a layer in Enterprise, and that a valid ArcGIS Online user account is embedded within the registered vector tile layer. It is recommended that this ArcGIS Online user account is not an employee’s active account, nor should it be an account with administrative rights. This embedded user account should be a non-user account created expressly for access to the basemaps. A Persistent Listener Consistently and reliably transmitting new and updated information to a utilities entire mobile workforce requires web services. These published feature services have specific parameters which need to be configured to support syncing. Once a service is configured for syncing, it becomes a persistent listener, that is ready to respond to a mobile devices’ request for changes in data. This persistent listener is available on stormy days as well as sunny days to ensure that the utilities mobile workers have the current information they require to safely maintain a reliable utility system. About this Blog Series This is the third blog article in our series on offline map areas. In future blog articles we will continue to explain the details of how offline map areas work and the specific decisions an administrator will need to make during deployment. The first blog provided an overview of offline map areas. The second blog provided details on how to prepare the data for offline usage and synchronization. The fourth blog will provide details on the creation of offline map areas, how they are stored and managed in the portal environment. The fifth and final blog will provide details on the deployment and management of offline map areas for a large mobile workforce. PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions. ... View more

Unified Pipeline Tools ribbon: New BFF of Pipeline Referencing and Utility Network Users By Jeff Allen, Tom Coolidge, and Tom DeWitte   Users of ArcGIS Pipeline Referencing and ArcGIS Utility Network have a new best friend in the Unified Pipeline Tools ribbon. That is because the new ribbon for the first time enables users of both extensions to access tools from both on one tab instead of having to move back and forth among multiple individual tabs. So now, not only does ArcGIS enable you to build pipe network models using both linear referencing and connectivity modeling methods on the same geodatabase, but it also greatly simplifies the performance of workflows requiring tools from both extensions.   Here's one example of how this advance can minimize clickiness and increase productivity. Let’s say that I am a mapping technician responsible for maintaining the transmission pipe network. On my desk is the construction packet for a newly replaced section of one of the transmission pipelines. Using the standard ArcGIS Pro configuration of tools and ribbons, I am going to use the following ribbons while updating my transmission pipe network to reflect this newly completed change to the pipe system. First, I am going to use the Map ribbon to explore and pan into my area of interest. Then I will switch to the edit ribbon to create a set of new pipeline features along the replaced section. Next I will use the location referencing ribbon and its route realignment tools to realign the pipeline to the new section of pipe. Lastly, I will use the Utility Network ribbon to update terminal settings and validate my edits. That’s four tabs to complete the task. Now if we have the Unified Pipeline Tools ribbon, we will complete all explore, create, save, route realign, and validation in one ribbon. Benefits Of Accessing Tools from Both Extensions on One Ribbon That is a significant increase in efficiency. The Unified Pipeline Tools ribbon is a step forward for users of ArcGIS Pipeline Referencing and ArcGIS Utility Network. It places at their fingertips a new capability to achieve greater efficiency when performing tasks and workflows that necessarily require accessing tools on multiple tabs. Personalize the New Ribbon The Unified Pipeline Tools ribbon comes pre-configured with what our research has revealed to date to be the most used tools for managing transmission pipe networks. Since this is based in ArcGIS Pro, users can add additional tools to the ribbon to make it even more aligned and complete for their most common workflows and tasks. Get the Unified Pipeline Tools ribbon now Esri users interested in getting the Unified Pipeline Tools ribbon should email Tom Coolidge at tcoolidge@esri.com.   ... View more

By Tom DeWitte, Kevin Ruggiero, Mike Hirschheimer Part 2 of 5 How do I get critical utility pipe, conductor, and cable data into the hands of my utility field workers? Before the advent of mobile computers that were not the size of briefcases, the only practical answer was paper. During a disaster event, the mapping department would have to print hundreds of copies of the same map pages to be handed out to utility field workers and foreign crews. With first-generation mobile computers, a digital copy of the pipe, conductor, and cable data was loaded onto the mobile computers by office staff for use by utility field workers. This snapshot of data was static, and likely did not include updates still in backlog and unable to reflect the transactional updates to the utility’s network as utility field workers were responding to the disaster event. Neither of these options are ideal for providing utility field workers with the up-to-date information they require to safely perform their jobs. What utilities desire is a mechanism that can automatically and frequently pass changes to the data from the office to the field and from the field to the office. This method of only passing changes is called bi-directional synchronization, though many will simply call it delta syncing. Bi-directional synchronization keeps the entire organization viewing and using the same data, whether they are in the office, or in the field. Syncing with ArcGIS Deploying bi-directional synchronization with ArcGIS requires proper configuration of the data, and the server environment. It also needs a mobile computing device with ArcGIS Field Maps. Offline map areas is the capability that provides bi-directional synchronization support to enterprise organizations with hundreds, thousands, or tens of thousands of utility field workers.   If this capability is of interest to you, you may be wondering, can I take my data offline? What modifications do I need to make to my data to prepare for bi-directional synchronization with my organization’s mobile devices? -In this blog we will answer those questions. Preparing the Data for Offline Map Areas Preparing the data for offline map areas is the first step in our four-step process. When preparing data for bi-directional synchronization with offline map areas we need to determine which formats of data are supported. Valid data sources are:                 -Enterprise Geodatabases                 -Hosted Feature Layers File based spatial data formats such as shapefiles, file geodatabases, and CAD files do not support this advanced form of synchronization.   Being a feature class or table within an Enterprise Geodatabase or a hosted feature layer does not guarantee the ability to support bi-directional synchronization. There are additional requirements of the data structure. Prepare to be Published as a Feature Service For enterprise geodatabases, the first set of requirements is the requirement to publish the data as a feature service. This requirement is the following: feature class or table is registered with the geodatabase, each feature class or table must have a Global ID field, and all relationship classes must have Global ID as the primary key. A detailed explanation on preparing data for use in offline feature services can be found in the ArcGIS Pro online help. Hosted feature layers when created are already prepared for offline sync. All you must do is update the Setting properties and check the box to Enable Sync. A detailed explanation for enabling a hosted feature layer for offline can be found in the ArcGIS Enterprise online help. Providing the complete map view that mobile workers require to perform the tasks assigned to them requires a wide range of data. Multiple data repositories can be referenced within a single web map for offline map areas. These different data repositories will often use different methods for tracking changes. Tracking Changes with Enterprise Geodatabases For the software to be able to identify the changes which have occurred since the last successful synchronization requires an ability to track changes within the enterprise geodatabase feature class or table. There are three options within the enterprise geodatabase for tracking changes to a feature class or table. These three options are called: Non-Versioned Archiving, Traditional Versioning, and Branch Versioning. Non-Versioned Archiving Non-Versioned Archiving is the database method for tracking a change to an individual record. This is perfect for workflows such as inspections, damage assessments, and field observation reports. These short transaction activities preserve every modification made to a record. For each iteration of change ArcGIS automatically assigns a “from date” and a “to date” which defines when that iteration of change was the current state of the record. A more detailed explanation of non-versioned archiving can be found in the ArcGIS Pro online help. Traditional Versioning Traditional Versioning was Esri’s original method for tracking a group of changes as a single transaction. This group of changes includes additions, updates, and deletes to many records across many feature classes and tables. This is critical to preserving the integrity of the full utility system for common changes such as documenting the extension of the utility system to a new subdivision.  This method uses a set of database tables to track the sequence in which the changes were made and submitted. A more detailed explanation of traditional versioning can be found in the ArcGIS Pro online help. Branch versioning Branch versioning is Esri’s current and recommended method for tracking a group of changes as a single transaction. This updated long-transaction methodology uses a simpler database structure to track changes. Modifications and additions to records will be marked with a date and time. A more detailed explanation of branch versioning  can be found in the ArcGIS Pro online help. You are not required to select one of these change tracking methods for your offline map areas deployment. Offline map areas have the capability to bi-directionally synchronize against multiple feature services and hosted feature layers; each using a different method of tracking changes. Preparing Utility Network Data The Utility Network is the feature service-based capability to manage networks of pipes, conductors, and cables. The Utility Network has many of the same data preparation requirements as offline map areas. They both require registration with the geodatabase, global IDs, and relationship classes based on the primary key field being global ID. Utility Networks require editor tracking and branch versioning. The branch versioning tracking of date and time will be used by the offline map area bi-directional synchronization to identify what changes have been made across the records in the data that need to be transmitted to the mobile application running ArcGIS Field Maps. For most utilities using ArcGIS, the utility mapping department is the only department allowed to edit the utility network configured data. This data repository, when published as a feature service will therefore be read-only to the utility field workers.   Preparing Landbase Data In larger utility organizations, it is not unusual to have a separate department for landbase mapping and utility mapping. The editing of the landbase is also a long-transaction method. The changes to curb lines, building footprints, and lot lines are commonly tracked as a group of changes which need to be posted together. In this landbase mapping department scenario, traditional versioning has been configured as the method for tracking these changes. The landbase mapping department is the only group allowed to edit the landbase data. This published feature service will be read-only to the utility field workers. Preparing Field Collected Data Field collected data can be any inspection or field observation report that will be documented by the utility field workers. In this configuration the field collected data is being stored as Hosted Feature Layers. Hosted Feature Layers use the Non-Versioned Archiving method of tracking changes. Preparing the Basemap Utilities have specific needs of their basemaps.  First, the basemap must cover a very large geographic area. Second, the basemap needs to provide highly detailed display scales of the basemap. This dual need for both a large geographic extent and highly detailed display scales does not work well with traditional image tile layers.  But vector tile layers are very well suited to these requirements. Esri provides a large set of vector tile basemaps with a beautiful range of cartography styles. But not all vector tile layer basemaps provided by Esri support exporting for offline map areas. There is a separate list of these “For Export” vector tile basemaps. The list of vector tile basemaps configured for exporting can be found here. Putting it all Together The ability of ArcGIS Field Maps and its offline map areas capability to pull data from multiple repositories provides the flexibility utility enterprise GIS administrators need to pull data from across the organization. In our utility scenario a single web map has been configured to use Utility Network data from a branch versioned Enterprise Geodatabase, landbase data from a traditional versioned Enterprise Geodatabase, inspection data from non-versioned archived hosted feature layers, and a vector tiles layer (for export) from Esri managed services. The ability to pull data from multiple repositories provides the flexibility utility enterprise GIS administrators need to pull data from across the organization. This avoids complex back-office duplication or replication processes to prepare the data. With a single web map an organization can put all the pipe, conductor, cable, landbase, inspection, and basemap data a utility field worker needs to safely perform their daily tasks. About this Blog Series This is the second blog article in our series on offline map areas. In future blog articles we will continue to explain the details of how offline map areas work and the specific decisions an administrator will need to make during deployment. The first blog provided an overview of offline map areas. The third blog will provide details and options for publishing the selected data repositories. The fourth blog will provide details on the creation of offline map areas, how they are stored and managed in the portal environment. The fifth and final blog will provide details on the deployment and management of offline map areas for a large mobile workforce. PLEASE NOTE: The postings on this site are our own and don’t necessarily represent Esri’s position, strategies, or opinions.                     ... View more