Loading ILI Data into ArcGIS

[AyanPalit]

Co-authors: @AyanPalit and @Kevin_Ruggiero 

As members of the Esri’s Infrastructure Management group, there never seems to be a meeting or event where we don’t get asked “How to load ILI Data into ArcGIS?”. Pipeline Inline Inspection (ILI) data refers to the information collected from devices, often called smart pigs, that travel inside pipelines to assess their condition and detect defects. These tools are critical for pipeline integrity management, allowing operators to detect potential issues like corrosion, cracks, or other anomalies that could lead to failures.  This blog lists the primary implementation patterns of storing and managing ILI data and then focuses on the most common pattern of ILI as a linear referencing system (LRS) event.

The Challenges

Pipeline integrity management is a required practice for utility owners and pipeline operators. Integrity managers desire and benefit from ILI data being stored and managed in an enterprise database. So why is it so important to be able to spatialize and visualize your ILI data? 

ILI data is the most efficient and reliable method for validation of pipeline integrity.  ILI also delivers valuable information about your pipeline.  Data such as cracks, material defects, mechanical damage, metal loss can all be measured and tracked with ILI data. By looking at this data spatially, we can start to answer the “Where” is this located, “What” is it and “Why” is this occurring at these location questions. 

The following characteristics make the management of ILI data challenging:

• Data Volume & Density: ILI inspections generate large amounts of data, especially in long pipelines. The data density can be extremely high, particularly for tools capable of high-resolution scans.
• Data Format: The ILI survey is recorded by smart pig devices and the readings are exported out by the vendor in tabular formats. The vendor delivers the raw survey typically in Microsoft Excel or CSV files.  Another challenge is the naming convention, attribution varies depending on the vendor and/or pigging technology.
• Location Reference: ILI survey data may include location coordinates that are in latitude-longitude format or Northing-Easting.   
• Accuracy and Tolerance: The accuracy of the ILI survey is dependent on multiple factors including pigging device, ILI vendor technology and location along the pipeline. Pig slippage is a common occurrence where odometer readings are not normally distributed; certain portions may get stretched while others may get compressed.
• Lifecyle Tracking: ILI inspections are repeated at defined intervals to ascertain corrosion growth rates through time-based analysis. ILI data also enables operators to track the progression of individual anomalies (like dents) over time, comparing growth rates and prioritizing repairs.

Implementation Patterns

Pipeline operators use various methods to store and manage ILI survey data. Linear referencing is a widespread modeling technique given the nature of the data. Here are the primary implementation patterns seen across the industry:

1. Relational tables:  store as flat tables in relational database or as geodatabase table
2. Geodatabase feature class: store as simple features in a geodatabase
3. Linear referenced feature class: store as  LRS event feature in a geodatabase
4. Linear referenced external event: store as  LRS external event in relational database or as geodatabase feature class

Each implementation pattern has its own merit and organizations evaluate their business requirements and choose the option that best meets their ILI workflows. ArcGIS Pipeline Referencing offers a comprehensive linear referencing system to manage pipeline assets and integrity data layers.   Utility owners and pipeline operators who manage their pipelines with LRS routes and measures, find it suitable to manage their ILI information as well.   The following section covers the methods to load ILI data in an APR managed, linear referenced event feature class.

Overview of Method

ILI vendors offer multiple ways to deliver the survey data and to calibrate that data to the existing pipeline data.  Pipeline operators develop a comprehensive process to manage the lifecycle of ILI data which often includes the ILI vendors and third-party service providers. 

1. Plan your ILI Runs

Utility owners and pipeline operators have a rigorous, documented process in determining and employing reassessment intervals to ensure pipeline integrity. As part of this program, they schedule the lines that they want to run within a given year.  Integrity departments are usually the department that manages the ILI schedules, runs, ILI data analysis that results in identification of pipe maintenance jobs and follow-up with reassessment schedule.

As part of the planning, pipeline integrity team works with GIS professionals to design and model the LRS routes and ILI events as part of their Pipeline Referencing implementation. The planning process is specific to the organization, their business practices as well as how pipeline data is modelled and configured with ArcGIS.

One of the design decisions is the representation of the pipe centerline as a LRS Route. The ILI data is then calibrated or linear referenced to the LRS route using attributes of route identifier (RouteID) and Measure.

• Engineering Network: an engineering network is modeled as a LRS Line Network. This option is preferred when most of the ILI data spatially matches with the engineering route that is created using as-built surveys, as in examples below. The raw ILI survey typically needs some data processing and alignment to match the engineering route.

      

• ILI Network: alternately, an ILI specific network can be modeled as a LRS Network (Nonline) using the ILI odometer readings. This option is preferred when the extent of the ILI survey does not match the engineering route and spatial coincidence is lacking. There could be various reasons for this discrepancy including how data was captured during as-built survey versus ILI run, such as including the length of the pig trap launcher segment.

          

2. Acquire ILI Data

ILI data can be delivered in a number of formats such as CSV, Excel, dbTable etc. ILI data typically has odometer/measure attribution for linear referencing. In some cases, the location data is provided in terms of latitude and longitude that can be used for spatialization.

The following samples show what raw ILI run would look like in an Excel file:

ArcGIS Pro has various tools and methods to process raw tabular ILI data:  

• Use ArcGIS Pro tool Excel to Table
  • Display XY Data, using LONGITUDE, LATITUDE fields

3. Calibrate the data

As discussed previously, the LRS network choice of engineering route or ILI route, is a key parameter that determines the calibration process of the ILI data. If ILI network is chosen, use the ArcGIS Pro tool Points to Line to first create a centerline using the raw ILI survey data. Using the centerline, create a route, specific to the ILI Run using the Location Referencing Tools in ArcGIS Pro. In this case, a continuous Route called DDFT-R1 is created representing the ILI run. 

 

Whether you use an engineering route or an ILI route, the attributes of the route identifier (RouteID) and Measure on the ILI data points must correlate to the route parameters. Loading the ILI data is achieved by Pipeline Referencing tool Append Events to the ILI Survey event layer. The RouteID and Measure fields must be populated that allows to associate ILI data to the correct route.  The RouteID can be obtained from the LRS network attribute table or by identifying the route feature.

Calculate the  RouteID and other fields required for the ILI Survey event layer as defined in the data model.  You can Copy and Paste them into a calculation that populates the fields as shown. 

Alternately, the ArcGIS Pro tool Locate features along route can be used, if RouteID and Measure are missing in the ILI data. The geoprocessing tool intersects the input point features and route features and writes the route and measure information to a new event table.

4. Appending Events

Once the ILI data is processed and formatted per the LRS event requirements, use the Append Events to load the data to the APR managed ILI Survey event layer.

The ILI data is now loaded into your GIS by referencing an LRS route network of choice. If you had domain codes in your captured data, you could match those domains when you load your data to the ILI Survey event layer.

This is another example of how the Geographic Approach can help to manage your ILI Data for use within your organization. It is noteworthy that all tools used in the workflow are core, out of the box tools in ArcGIS Pro and Pipeline Referencing extension. A developer is not required to customize the tools. However, models and automation scripts can certainly be developed for a repeatable and consistent process.  We hope this helps in understanding the high-level process to load ILI data to your pipeline geodatabase, using one of the implementation patterns mentioned.  Organizations can test this ILI loading workflow as well as evaluate other implementation patterns that best meet their business requirements.