Latest Contributions by AyanPalit

ArcGIS Parcel Fabric, as one of the next generation ArcGIS applications, is based on the branch versioning geodatabase transaction model.  Branch versioning follows a services-based architecture for viewing and editing data.  It facilitates long database transactions using feature services, taking advantage of ArcGIS Server's Version Management service. Direct SQL access to the relational database is not supported or recommended with this geodatabase transaction model.  Database Views are more applicable to traditional versioning and data level joins or integrations. In the absence of multi-versioned views, updates/insert/deletes using SQL should not be applied on feature classes that have been branch versioned. This can cause data corruption and other unexpected behavior within ArcGIS applications. Create Database View geoprocessing tool will accept branch versioned GDB as input workspace. However, the tool is not branch version aware. Design Considerations Users see opportunities in this changed services-based landscape that offer numerous benefits. But fundamental tasks like generating reports and database integration interfaces are key to running business processes. Solution architects and GIS Managers have carefully re-assessed the business needs and worked their way using a mix of the newer design patterns. Refer to Utility Network Journey: Branch Versioning and SQL by @RobertKrisher  for additional insights. REST API: The supported method to query branch versioned data is through the REST API for e.g. Query (Feature Service/Layer), where appropriate and meets the need. Specific methods have been developed in REST and ArcGIS clients to ensure data integrity is in place when accessing the branch versioned data. Python Scripting: The ArcGIS API for Python is branch version aware and scripts can be used to generate reports from branch versioned geodatabase. The scripts can be run as scheduled jobs, often during non-business hours to compile data reports.      ArcGIS Python API - Version Manager ArcGIS REST API - VersionManagementServer ArcGIS REST API - ParcelFabricServer ArcGIS Python API - Parcel Fabric Manager Non-versioned Publication Geodatabase: Required datasets can be replicated to a non-versioned publication geodatabase that can be used for reporting, SQL views, integration etc. Note that there is an overhead of the replication process and maintaining a secondary geodatabase copy. So, a careful design is recommended based on the layer requirements, frequency of replication, delta changes and overall cost benefit analysis. SQL Views: Additional SQL logic may be used for view definition (read-only) access to branch versioned data. The document Branch Version SQL Views presents several samples: Create Materialized Views and refresh Views Create Reporting Table and refresh Tables Note that the SQL Views option, is considered secondary for legacy database level operations, and INSERT, UPDATE and DELETE functionality is not available. REST API based query is the best practice, recommended option moving forward for service-based architecture. Parting thoughts (courtesy xkcd)   ... View more

Branch Versioning ArcGIS Pipeline Referencing (APR) at launch was based on traditional versioning geodatabase transaction model. However, as one of the next generation ArcGIS applications, APR added compatibility with branch versioning at ArcGIS Enterprise 10.6.1 with ArcGIS Pro 2.2. At subsequent releases, newer capabilities like services-based editing, Utility Network integration and conflict prevention require branch versioning. APR with branch versioning is the recommended and preferred implementation pattern for pipeline organizations. Branch versioning follows a services-based architecture for viewing and editing data.  It facilitates long database transactions using feature services, taking advantage of ArcGIS Server's Version Management service. Direct SQL access to the relational database is not supported or recommended with this geodatabase transaction model.  Database Views are more applicable to traditional versioning and data level joins or integrations. In the absence of multi-versioned views, updates/insert/deletes using SQL should not be applied on feature classes that have been branch versioned. This can cause data corruption and other unexpected behavior within ArcGIS applications. Create Database View geoprocessing tool will accept branch versioned GDB as input workspace. However, the tool is not branch version aware. Design Considerations Users see opportunities in this changed services-based landscape that offer numerous benefits. But fundamental tasks like generating reports and database integration interfaces are key to running business processes. Solution architects and GIS Managers have carefully re-assessed the business needs and worked their way using a mix of the newer design patterns. Refer to Utility Network Journey: Branch Versioning and SQL by @RobertKrisher for additional insights. REST API: The supported method to query branch versioned data is through the REST API for e.g. Query (Feature Service/Layer), where appropriate and meets the need. Specific methods have been developed in REST and ArcGIS clients to ensure data integrity is in place when accessing the branch versioned data. Python Scripting: The ArcGIS API for Python is branch version aware and scripts can be used to generate reports from branch versioned geodatabase. The scripts can be run as scheduled jobs, often during non-business hours to compile data reports.      ArcGIS Python API - Version Manager ArcGIS REST API - VersionManagementServer ArcGIS REST API – Linear Referencing Server Non-versioned Publication Geodatabase: Required datasets can be replicated to a non-versioned publication geodatabase that can be used for reporting, SQL views, integration etc. Note that there is an overhead of the replication process and maintaining a secondary geodatabase copy. So, a careful design is recommended based on the layer requirements, frequency of replication, delta changes and overall cost benefit analysis. SQL Views: Additional SQL logic may be used for view definition (read-only) access to branch versioned data. The document Branch Version SQL Views presents several samples: Create Materialized Views and refresh Views Create Reporting Table and refresh Tables Note that the SQL Views option, is considered secondary for legacy database level operations, and INSERT, UPDATE and DELETE functionality is not available. REST API based query is the best practice, recommended option moving forward for service-based architecture. Parting thoughts (courtesy xkcd)   ... View more

Conceptual Architecture Event Editor is a companion web app for linear referenced event data editing. This JavaScript app is powered by linear referencing services published using ArcGIS Pipeline Referencing (APR) Server extension. A conceptual architecture of APR is shown below. The deployment of the solution components needs some careful planning, reviewing various implementation options.       Deployment Considerations The recent ArcGIS Pro 3.3 and ArcGIS Enterprise 11.3 releases of ArcGIS Pipeline Referencing offers multiple options that must be taken into consideration. ArcGIS Enterprise 11.3 release includes Experience Builder LRS widgets. This is the forward looking, long-term web app for event editing. Customers implementing ArcGIS Enterprise 11.3 and above, are recommended to go with this option.   Event Editor (legacy) is downloaded from My Esri and deployed as a standalone web app on a web server. The 11.3 version of Event Editor will be the final release of the app, per the Event Editor deprecation notice. LRS event editing capability was added at ArcGIS Pro 3.0, compatible with ArcGIS Enterprise 11.0.  Editors can perform comprehensive LRS editing in a single desktop environment without leaving ArcGIS Pro. A question often asked is ‘Do I need APR Server Extension and or the Event Editor?’. Another way to ask the same is ‘Does APR for ArcGIS Pro provide me all the functionality needed for pipeline editing?’ Since the initial launch of APR (at ArcGIS Pro 1.4) through ArcGIS Pro 2.9, the Route Network management has been exclusively at the Pro desktop level while event editing has been in Event Editor web app. While advanced linear referencing for pipeline data management can be modeled and piloted in the ArcGIS Pro environment, you need the APR Server for enterprise deployments with versioned geodatabase (branch version) and services-based, multi-user editing. The Location Referencing toolbox in ArcGIS Pro allows bulk load of LRS events and event maintenance following route edits.  Based on user requests, LRS event editing capabilities were added to ArcGIS Pro. This is specially needed by organizations with editors who perform all aspects of a pipeline job as a single workflow. For them, a streamlined experience is to complete all steps in ArcGIS Pro, versus having to perform certain editing tasks in Event Editor. However, map-based interactive editing of events must be done using feature services with linear referencing. So, you do need APR extension at ArcGIS Pro, APR Server extension and Event Editor web app for comprehensive enterprise-wide capabilities.   While this is great news for the user community, this prompts a revisit of the question asked previously.  ‘Do I need APR Server Extension and or the Event Editor?’. You still need APR Server that powers the LRS services – event editing in ArcGIS Pro uses the feature service. As discussed, ArcGIS Enterprise deployments (with APR Server) offers robust and comprehensive capabilities leveraging branch versioned geodatabase and services-based, multi-user editing. The USP for Event Editor still remains: a light-weight web app, that’s easy for non-GIS users, non-LRS users.  ArcGIS Pro will be too complex, perhaps overwhelming for these users – Event Editor provides just the right set of tools, to edit and manage the few data layers they care about. Adding ArcGIS Pro and APR extension is additional licensing costs as well. Event Editor web app (one or multiple) is not a license bearing component and utilizes the existing APR Server software.  Federated Server with Portal As in the schematic above, a full deployment of ArcGIS Enterprise is required for APR. This includes ArcGIS Server with APR Server extension, federated with Portal. To further clarify, you must have an ArcGIS Server site licensed for APR Server extension (no install needed). The APR Server must be explicitly federated with Portal; ArcGIS Online does not support this pattern.    ArcGIS Enterprise at its simplest, can be a single machine deployment. In this case, the Event Editor can go on the same machine. However, in a multi-machine site configuration, Event Editor is typically deployed on the web server alongside ArcGIS Web Adaptor. Event Editor is registered with Portal as a web mapping application and an ‘app ID’ is generated. Event Editor now inherits the users and groups   access control from Portal. Single Instance Deployment Every production implementation of APR has at least a single instance of Event Editor. In this scenario, the LRS Network(s), all the Event(s) and redline layers are published as services. The APR Server extension adds the Linear Referencing capability to the map service. Version management capability becomes available when the LRS layers in the geodatabase are branch versioned. A web map must be created in Portal, using the linear referenced map service. This web map is then used for the Event Editor configuration(legacy), as shown below. Event Editor leverages online content like basemaps from ArcGIS Online. The map service, the web map and the Event Editor is secured and shared with the Portal group of editors.     Multiple Instance Deployment At larger APR implementations, that have multiple teams with various users owning specific event layers, multiple instances of Event Editor have been deployed. Event Editor, being a light-weight web app, is easy for non-GIS users who may not have awareness of advanced linear referencing concepts. Event Editor restricts edits at the event-level and along with versioned editing, there is little chance of new users making errors that have system-wide impacts.    Such organizations, have a long list of LRS events to manage. As part of the planning and design, the events are categorized into groups that will be managed by specific business teams. As in example below, the Integrity team owns the ‘Pipeline Risk’ layer and is responsible for updating risk scores on line-segments based on risk model parameters.   In this scenario, the data ownership is more distributed among business teams, while the GIS team supports the QC and posting of edits to the geodatabase. The business teams feel empowered and control the data layers, they care about most. The editing of the event data layers is also isolated, so in this example, Land Contracts team cannot edit an ‘HCA Segment’. Operations Integrity Land Contracts ROW Clearing Test Pressure Range Pipeline Risk Right of Way Vegetation Management Operating Pressure Range HCA Segment Line Operational Use     In this deployment pattern, publishing of map services with Linear Referencing capability is standard. However, a different web map is created for each business team with their curated layers.  As in the example below, web map X is for Integrity team and web map Y is for Land Contracts team. Copies of the Event Editor web app is deployed on the web server, renaming each web folder instance. For example, in IIS, the default location of the web folder will be ‘C:\inetpub\wwwroot\EventEditorX’ and ‘C:\inetpub\wwwroot\EventEditorY’. At the Event Editor configuration(legacy), reference to the corresponding web map parameter is updated in the config.json file.  Each instance of Event Editor is registered with Portal using a unique URL and the respective portalAppId is updated in the config.json file.  The same pattern is followed for additional instances of Event Editor as needed. Operations Integrity Land Contracts ROW Clearing  Same -  - LRS -  - Map - - Service Web map W Web map X Web map Y Web map Z EventEditor W EventEditor X EventEditor Y EventEditor Z portalAppId W portalAppId X portalAppId Y portalAppId Z   Typically, Portal will have groups for each of the business teams. Leverage the Portal groups and sharing to grant access control of the map service, the web map and the Event Editor app to specific teams. Only users with sufficient privileges will be permitted to view and edit the contents of each Event Editor instance. A variation of this pattern is where the LRS map service is also individualized and isolated for each app instance. Note the use case where, an inspection layer is maintained by third part contractors and must have access to the service and Event Editor app outside the organization’s firewall. In this case, a separate LRS map service is published, added to its own web map and using a proxy is referenced by and Event Editor instance sitting in the DMZ.   Another recommended best practice relevant for this deployment pattern, is for various editors to create public versions during their edit sessions. This allows the GIS team/GIS admins to QC the edits performed by each business team and reconcile & post the versions. Event Editor also provides granular control on versioning parameters, and they can be set to different values for each of the app instances depending on the comfort level of users. Event Editor Versioning parameters: allowReconcileAndPost: true or false allowChangeVersions: true or false allowCreateVersions: true or false allowDeleteVersions: true or false Conclusion The design considerations and configuration parameters discussed, further helps to provide a very defined, targets web app for LRS editors. GIS administrators and business users now have more choice, on how to divvy up the editing process between users as well as desktop versus web applications. We expect the user community to take full advantage of the solution components and devise appropriate workflows as they deem fit for their own organizational needs. ... View more

Configuration of Linear Referencing System The choice of spatial reference and units of measurement is a business decision and guided by geographical region, operational practices, integrity management and compliance reporting functions. The GIS leads must decide and specify the Spatial Reference and X, Y, Z Tolerance and Resolution to  be used for the core information model objects within ArcGIS Pipeline Referencing (APR). The APR toolbox provides various tools to create and modify an LRS that factors in spatial refence. Specifically, the Create LRS tool offers option to set X, Y, Z Tolerance and Resolution. The tool will default to the Tolerance and Resolution settings and units of the Spatial Reference. However, the parameters can be user defined and changed per requirements. It is recommended to use the suggested 1/10th ratio for tolerance/resolution defaults suggested by ArcGIS. Following example shows the default settings using USA Contiguous Albers Equal Area Conic WKID 102003. Note that using Geographic Coordinate System WGS 1984 WKID 4326 will default the XY Tolerance and Resolution to Decimal Degrees. USA Albers                                    GCS WGS 1984          Resulting Spatial Reference settings at the LRS dataset including Projected Coordinate System (PCS), Geographic Coordinate System (GCS) and Vertical Coordinate System (VCS):       Resulting Tolerance and Resolution settings (in Meters per the parameters) at the LRS dataset: Note that the units can be set to supported units including Kilometers, Meters, Centimeters, Miles, Yards, Feet or Inches in the tool parameters for the same spatial reference; the appropriate X, Y, Z Tolerance and Resolution assigned, will be different from the defaults. Configuration of LRS Network Once the LRS is created, the Create LRS Network tool offers option to set the Units of Measure along the Network. The Units can be user defined and changed per requirements (supported units include Kilometers, Meters, Centimeters, Miles, Yards, Feet or Inches). Following example shows the default units of Meters using USA Contiguous Albers Equal Area Conic WKID 102003.      Following example shows creating a second network and defining the units to Feet using the LRS that is using USA Contiguous Albers Equal Area Conic Meters.     APR calculates the required M tolerance and resolution for the network feature class that will align the XY tolerance with that of the M tolerance in the desired unit of measure. The XY and Z tolerance and resolution for the network will come from the centerline of the LRS. As in the example above, the XY tolerance is 0.001 meters and the M tolerance is set as 0.00328083333333333 for the LRS Network measured in Feet. The XY resolution is 0.0001 meters and the M resolution is set as 0.000328083333333333 for the LRS Network measured in Feet. ArcGIS stores the measure values on each vertex of the polyline M-enabled feature class. This is the geometry model that ArcGIS and APR is based on. 0.001 meters ≈ 0.003280833 feet 0.0001 meters ≈ 0.0003280833 feet The APR Create LRS Event tool takes the X, Y, Z, M tolerances and resolutions of the LRS network and propagates the settings to all events that are registered to that particular Network. Note that APR supports multiple networks as illustrated by additional examples below. Following instance shows creating a third network and defining the units to Miles using the LRS that is using USA Contiguous Albers Equal Area Conic Meters.       Following example shows creating a fourth network and defining the units to Centimeters using the LRS that is using USA Contiguous Albers Equal Area Conic Meters.       Pre-defined Geodatabase Schema Organizations may have a pre-determined data model like Esri’s UPDM or PODS. For users with a pre-defined schema, the appropriate Spatial Reference along with X, Y, Z, M Tolerance and Resolution (per conversion calculations) must be set at the time of creation of the empty geodatabase schema. Specifically, the APR information model objects, the LRS Network(s) and related LRS Events need to have compatible X, Y, Z, M Tolerance and Resolution. The configuration options in the APR toolbox along with understanding of Tolerance and resolution settings for the LRS will allow the user to set the desired combination of Spatial Reference and linear units of measurement.  The following table provides the correct tolerance, and resolution settings for key spatial references using Feet or Meters as units of measurement in your LRS. Spatial Ref: GCS WGS84 / NAD83 / NAD27 US FEET GCS WGS84 / NAD83 / NAD27 METERS PCS NAD83 UTM Zone 14N US FEET PCS NAD83 UTM Zone 14N Miles XY Resolution 1e-09 Degree 1e-09 Degree   0.0001 Meter   0.0001 Meter   XY Tolerance 8.98315284119521e-09 Degree 8.98315284119521e-09 Degree   0.001 Meter   0.001 Meter   M Resolution 0.00036481184328240743 0.00011119487222222222   0.0003280833333333333   6.213699494949494e-08 M Tolerance 0.0032771605464840194 0.0009988805323293938   0.003280833333333333   6.213699494949494e-07 Z Resolution 0.0001 0.0001 0.0001 0.0001 Z Tolerance 0.001 0.001 0.001 0.001   If using other spatial reference with different units, ArcGIS Solutions provides tools to calculate tolerances, resolutions and change the geodatabase spatial reference. Any data translation or re-projection of native Spatial Reference and linear units of measurement to different settings will require appropriate transformations. The units of an established linear referencing system should match the proposed system or a unit conversion factored in. Depending on the process and tools used, users may observe spatial shift and/or length mismatches. However, using appropriate transformations are known to minimize such data discrepancies. ... View more

Interested in learning linear referencing concepts and applying them for pipeline data management using APR? Checkout the attached course topics and sign-up for instructor-led APR training. The course was recently updated for ArcGIS Enterprise 11.1 | ArcGIS Pro 3.1 and includes hands-on exercises on the latest functionality.  ... View more

Do you maintain your LRS data in GCS, yet want to maintain accurate linear distances along route centerlines? Read the attached document to find out more. ... View more