New Tools for Evaluating Large Datasets with ArcGIS Data Reviewer

[Jaspar_Saadi-Klein]

IMPORTANT: Please be aware that the tools shown in this blog are undergoing active development and testing. These are a work-in-progress and we welcome any feedback, enhancements, and reports of issues as you use the tools.

Why these tools were created

It is a common practice to evaluate every feature in a dataset to establish a baseline understanding of its quality, especially to determine whether a new dataset is fit for a project or whether a provider’s data deliverable meets contractual quality requirements. The Create Evaluation Grids and Evaluate Rules by Grid tools were developed to address the challenges associated with assessing the quality of very large datasets containing millions of features. In these scenarios, an extent-based approach can help support this type of rigorous, large-scale feature quality evaluation, by breaking the evaluation into more manageable subsets.

What these tools do

These tools break attribute rule evaluation into smaller, more manageable extents. It's conceptually similar to choosing to evaluate rules on the visible extent in ArcGIS Pro. Under the hood, an extent is passed to the Evaluate Rules geoprocessing tool and evaluation runs only on features within that extent. Rather than evaluating an entire dataset at once, these tools divide the dataset into a collection of smaller extents and evaluate each one individually. This allows large datasets to be processed incrementally while still ensuring that all rules are evaluated across the full dataset.

How evaluation grids are created

The Create Evaluation Grids tool generates the extents that will be used during evaluation.

To create these extents, the tool leverages the Create Cartographic Partitions geoprocessing tool. This tool creates output polygons based on the density and distribution of features, allowing the evaluation workload to be divided into more manageable areas.

The Maximum Features Per Grid parameter is used to establish an upper limit for the number of features that can be included within a given evaluation extent. One or more feature classes can be specified as inputs, and the total feature count across all selected feature classes is used when determining the partition boundaries. As a result, areas with higher feature density produce smaller grid polygons, while less dense areas produce larger grid polygons.

When the tool completes, it adds an Evaluation Required field and extent information fields to the generated partitions, applies symbology, and adds the resulting layer to the active map.

The resulting grid layer becomes the input for the Evaluate Rules By Grid tool.

How evaluation is performed

The Evaluate Rules By Grid tool uses the generated evaluation grid to evaluate attribute rules one grid polygon at a time.

For each grid polygon, the tool extracts the extent and passes that extent to the Evaluate Rules geoprocessing tool or Evaluate REST API. Evaluation is then performed against all feature classes and attribute rules on that extent before moving on to the next grid polygon. This process continues until all grid polygons have been processed.

As the tool runs, a geoprocessing message is added after each polygon grid is evaluated, indicating whether the grid was processed or skipped. This makes it easier to track progress during long-running evaluations.

Understanding the Evaluation Required field

The Evaluation Required field exists solely to track grid-processing progress.

After a grid polygon has been successfully evaluated, the tool updates this field to indicate that the grid no longer requires processing. This allows evaluation to resume where it left off if execution is interrupted due to a timeout or an unexpected issue.

When the tool is rerun with the parameter Skip Already Evaluated Grids enabled, grid polygons that have already been processed are skipped and evaluation resumes with the remaining grids.

It is important to note that the Evaluation Required field does not indicate whether features require validation.

Information about whether a feature requires validation is maintained separately through each feature class's Validation Status field. Validation Status is updated when features are modified or when validation attribute rules are added or changed. The Evaluation Required field only tracks whether a specific grid polygon still needs to be processed by the Evaluate Rules By Grid workflow.

How to use the tools

First, run the Create Evaluation Grids tool to create the grid that the Evaluate Rules By Grid tool will use. To do so, complete the following steps:

1. In ArcGIS Pro, add all feature layers that contain validation attribute rules to a map
2. Add the GridEvaluation.pyt toolbox to a project
3. Click on Create Evaluation Grids
4. For Input Features, select the feature layers that contain validation attribute rules
5. Set the Maximum Features Per Grid parameter
6. Set the output location
   1. Outputs can be placed in file, mobile, or enterprise geodatabases.
   2. The output will be in WGS 1984, so if placing the output in a feature dataset, it must be in WGS 1984.
7. Run the tool

Next run the Evaluate Rules By Grid tool

1. Open the tool and for Evaluation Grid, select the grid output from the Create Evaluation Grids run
2. Keep Skip Already Evaluated Grids checked
3. For Workspace to Evaluate, select the file, mobile, or feature service to evaluate attribute rules on
   1. If the workspace is a feature service URL, additional parameters will appear
      1. Fill out the Portal URL, Portal Username, and Portal Password for the enterprise user that will be running the evaluation.
      2. Specify the version to run evaluation on.
         1. Please note that this field will automatically be populated with a dropdown list of available versions once the portal information is specified. This may take a minute.
4. Set the Evaluation Type
5. Run the tool
   1. While the tool is running you can view the messages. This will tell you which grid polygons have been evaluated and which were skipped (if any).

Items of note

1. It is preferred that you evaluate without using these tools when able. If you are able to increase the service timeouts for your feature service and perform database tuning, it will be faster to do one evaluation run rather than many smaller evaluations as these tools do. Only use these tools when evaluating large datasets with many millions of records after you have exhausted all other options.
2. You can reuse the same grid and don’t have to run the Create Evaluation Grids tool again by either unchecking “Skip Already Evaluated Grids” or by manually resetting the Evaluation Required field to No (0).

GridRuleEvaluationV0.1.8.zip