Community

Data Loading Tools - What's New July 2022

1294
0
07-28-2022 02:58 PM
TedHoward2
by Esri Contributor
Esri Contributor
1 0 1,294
‎07-28-2022 02:58 PM

The Data Loading Tools can help you streamline loading data from a source to a target dataset, with the ability to perform data transformations. 

With the July 2022 release of the tools, we have added some new and exciting capabilities. Read along to learn about these changes and how they may be used for your own data loading. 

 

Data Loading Tools and ArcGIS Pro versions align 

 

The Data Loading Tools toolbox versions were updated to align with your version of ArcGIS Pro. Use the 2.9.x version of the tools with ArcGIS Pro 2.9.x and the 3.0.x version of the tools for the corresponding ArcGIS Pro 3.0.x version.

 

Create Data Loading Workspace (updated) 

 

The Create Data Loading Workspace tool allows you to match source and target datasets and creates a Data Loading Workspace that can be used for data loading. With this tool, you can predefine if certain datasets, values, and domains should be matched. You can also enable predictive field matching options based on similarity and determine if subtypes should be matched. 

TedHoward2_0-1659041739106.png

 

What’s New: 

 

  • Map Feature Classes and Tables – It is now possible to map and load data from a feature class to a table.
  • Prevent words or phrases from matching – The Predefined Mapping parameter now allows you to block two words or phrases from being matched even if they are similar or exactly the same. 

TedHoward2_1-1659041739108.png

  • Control Subtype matching – A new parameter to control whether separate DataMapping workbooks will be created for each Subtype. If this option is unchecked, the matching of datasets will only be attempted at the class level. If classes contain subtypes, a subtype sheet will be created in the DataMapping workbook. 
  • New “Info” worksheet – Each mapping workbook will come with an Info worksheet which provides instructions and information about the workbook. 

TedHoward2_0-1659044690756.png

  • New Functions – Several new Python functions can be found in the base.py file in the Scripts folder. These functions are now available in the mapping workbooks from the dropdown in the Expression column, with placeholder parameter examples to help you understand how to call the functions.  

TedHoward2_3-1659041739110.png

For example, if you want to load data to a field based on a condition, use the new conditional() function: 

TedHoward2_4-1659041739111.png

In the example above, the Source field ROTATION is evaluated to detect Null values. If the ROTATION field has a value, then the value will be loaded into the symbolrotation target field. If it is Null, a value of 180 is loaded into the symbolrotation target field instead. 

This is similar to Python expressions in the Calculate Field geoprocessing tool. See Calculate Field Python examples. 

  • Preserve Attachments and GlobalIDs – You can now preserve attachments and Global IDs when loading your data. Columns for MaintainAttachments and PreserveGlobalId were added to the DataReference workbook so they can be controlled row by row. 

 

What’s fixed: 

 

  • If the Attribute Domain Coded Value Description Similarity option is not enabled in the Predictive Field Matching Options, we now allow field name matching for fields with domains.  
  • Predictive matching for Feature Service layer names now uses the proper alias. 

 

Update Data Loading Workspace Schema (new) 

 

An existing Data Loading Workspace only reflects the state of your Source(s) and Target(s) when the Data Loading Workspace was created. What if you added fields to your Source and changed several Domains in your Target workspace?  

This tool is intended to make it easier to continue using your Data Loading Workspace without having to manually update your mapping workbooks to reflect schema changes. The tool creates a copy of your Data Loading Workspace and updates all mapping and domain workbooks in the copy.  

This tool can also be used to upgrade your existing workbooks with some of the latest released features (new Info worksheet, new functions, and columns to preserve attachments and Global IDs)! 

TedHoward2_5-1659041739112.png

 

How it works: 

 

  • Each mapping workbook referenced in the DataReference.xlsx file will be processed.  
  • All Domain workbooks in the Domains folder will be regenerated. 
  • Within the mapping workbooks, the yellow Schema and Subtype worksheets will be regenerated.  
  • The green Mapping worksheet will be updated to reflect any changes made to fields including additions, deletions, and updates (such as field type), in the Source and Target.  
  • The blue Lookup worksheets will be updated to reflect any domain changes in the Source.  

TedHoward2_6-1659041739112.png

  • A new base.py file containing new functions will be added to the Scripts folder. This will overwrite any functions you have added to base.py, while a new user.py file will be provided and is meant for your custom functions. 
  • The Mapping worksheet Expression column will now include all existing and new functions found in the Scripts folder. 

Notes:  

  • The tool generates a copy of your Data Loading Workspace and makes changes to the copy only. It will never make changes to your existing Data Loading Workspace. 
  • Predictive Field Matching will not be affected. Any existing matchings, expressions or lookups found in the mapping workbooks will remain untouched, and no new matching will be performed. 

 

Execute Data Load (updated) 

 

Use this tool with the Data Reference workbook from your Data Loading Workspace to load data from a source to a target database or feature service. 

TedHoward2_7-1659041739113.png

 

What’s New: 

 

  • Preview – Select whether to load data to the Target workspace or to a Preview Geodatabase. You can now provide an output folder to generate a Preview Geodatabase and the data will be loaded to this Preview instead of the Target. 
  • Attachments and GlobalIDs – Preserved and loaded to the Target if allowed in the corresponding column in the DataReference workbook. 
  • Feature Services – When executing the data load, Target Delete Queries are now more performant. 

 

Append with Transformation (updated) 

 

If you are looking for an ad hoc data loading process or processing single datasets, you may only need the Append with Transformation tool. This tool allows you to directly map and load your data from the source to the target, without needing the mapping workbooks. 

This leverages the Append geoprocessing tool from ArcGIS Pro, which you may already be familiar with, but allows for a more detailed field mapping and the use of complex Expressions. 

TedHoward2_8-1659041739114.png

 

What’s New: 

 

  • Preview – Select whether to load data to the Target workspace or to a Preview Geodatabase. You can now provide an output folder to generate a Preview Geodatabase and the data will be loaded to this Preview instead of the Target. 
  • Environment Controls - Added support for Maintaining Attachments and Preserving GlobalIDs. 

 

What’s fixed: 

 

  • Shape manipulation no longer fails if functions in your scripts use imports. 

 

Access the Data Loading Tools documentation here: Data Loading Tools | ArcGIS Solutions