The Data Loading Tools - Video Series

5177
20
05-12-2020 07:18 AM
AlixVezina
Esri Regular Contributor
2 20 5,177

DLT-mosaic-banner.png

 The Data Loading Tools are provided by Esri to help you streamline loading data from a source to a target dataset, with the ability to perform in-flight data transformation. 

A series of videos are provided to help you better understand these tools and how you can get started using them today. 

 

(1) In the first video, we introduce a series of demonstrations on how you can use these tools to consolidate data from various sources and schemas into a new target schema.

Introducing the Data Loading Tools

To get started with the tools, go to www.esriurl.com/dlt for more information.

Configurations of the Data Loading Tools are available for both Water Distribution and Electric Distribution and Transmission systems. Explore these configurations to better understand how to migrate from the Geometric Network to a Utility Network.

 

Configurations of the Data Loading Tools for Water and Electric Utilities data loading.

(2) In the second video, we go through the steps to get started with your first data loading project. You will first learn how to clone your python environment and install the data loading tools package.

 

Then, we will define our input and target datasets to create our data loading workspace, and explore the structure of the data loading workspace before executing the data load.

 

(3) In the third video, you will learn how to map your own source and target datasets using the Data Loading workspace.

 

We step through the process of preparing the data loading workspace, mapping the source fields to target fields, translating hardcoded values, and mapping coded value domains using expressions and lookup columns to improve the accuracy of the data loading.

(4) In the fourth video, you will learn how to use inline expressions and external functions to further streamline some of your data transformation process, and understand the mapping of data models which support subtypes.

Thumbnail-AdvancedMapping.png

20 Comments
SSMIC3038
Occasional Contributor III

Excellent video, looking forward to the rest.

XanderBakker
Esri Esteemed Contributor

Hi avezina-esristaff , great resource and thanks for sharing!

Is there any ETA on when the other parts will be available?

AlixVezina
Esri Regular Contributor

Hi Xander, 
We are planning on releasing the next video in July, and the next ones monthly from there.

Regards

AlixVezina
Esri Regular Contributor

Hi Xander, the second video is now available here: Set the Stage for Data Loading - YouTube 

AlixVezina
Esri Regular Contributor

Check out the third video now available here:  Video Link : 6696 

AllysonReynolds
New Contributor III

avezina-esristaff These are helpful! Do you know when the advanced videos will be avaiable? I'm specifically looking for guidance around using Python scripts in the Expression of the Excel workbooks. 

AlixVezina
Esri Regular Contributor

@AllysonReynolds I'm glad to hear these help! I'm currently working on the advanced mapping video, which will provide some examples for using Python scripts in the Excel workbooks. It should be available in the beginning of January. Also, please feel free to share any questions you may have!

AlixVezina
Esri Regular Contributor

Check out the fourth video now available here: Advanced Mapping for Data Loading

GIS_Solutions
Occasional Contributor II

Hi,

I want to field map the feature class name to the asset group field, can I do this mapping in the field mapping field?

AlixVezina
Esri Regular Contributor

Hi @GIS_Solutions ,

You should be able to handle this mapping in the DataReference workbook. I believe it would be similar to how this is demonstrated in this video: https://youtu.be/rqLwsK2zjck?t=1044

VanessaSimps
Occasional Contributor III

Maybe I am missing this part in the instructions, but what if I have the following: 

VanessaSimps_0-1642528729325.png

can I change the Our Agency value to be "public" so that it matches what we currently have? Can I add in additional values to the asset package tables? or is this a no-no??

AlixVezina
Esri Regular Contributor

Hi @VanessaSimps , you would need to add the value to your target database schema (your asset package), and update the appropriate target workbooks to reflect that change (or recreate all the workbooks).

Then you can assign that value as the new default value for your target features either by hard-coding the value for that field or using the lookup-columns as shown here https://youtu.be/c4x_anYd7UY?t=716

Also note that the domain workbooks (found in the Domains folder) are purely for reference and they do not affect how data is loaded.

VanessaSimps
Occasional Contributor III

Thank you @AlixVezina . I will give this a try. I am assuming the same would be true if I wanted to add in fields that we need to have which aren't present in the new target schema?

AlixVezina
Esri Regular Contributor

I am assuming the same would be true if I wanted to add in fields that we need to have which aren't present in the new target schema?

@VanessaSimps  Yes that is correct.

 

KadinBaird41
New Contributor II

@AlixVezina Thanks for these videos, I am just getting started on my conversion to the UN and I will be utilizing the Data Loading tools to migrate my data. I was wondering if the 5th video mentioned above (Review your data loading) was ever made or was still planned?

AlixVezina
Esri Regular Contributor

Hi @KadinBaird41, the 5th video is not available yet. I am expecting more feedback before we do so. Is there something specific you would be looking for in this video that would be helpful?

VanessaSimps
Occasional Contributor III

@AlixVezina It would be really helpful to better understand what domains/subtypes are required by UN and what we can customize/adjust to fit our existing data schemas.

MikeMillerGIS
Esri Frequent Contributor

@VanessaSimps that is very hard to answer as it depends on the model.  Asset Groups and Asset Types that are used as subnetwork controllers are required if you want to build subnetworks.  Fields used in Network Attributes may be needed depending on the subnetwork definition(for instance Status with open and closed).  

VanessaSimps
Occasional Contributor III

@MikeMillerGIS Thank you and this makes sense. But, what I am getting at is it would be useful to have more information out there for us to better understand this information. I have been trying to follow along with whatever crumb trail I can find out here on migrating from GN to UN. This requires me to making big changes to our existing data and workflows, and if there was more information about this out on the web, I would be less likely to paint the organization in a corner with a decision about a domain  or field values. 

(An example of what I mean is this: We have currently done this in our GN where we have a Active Flag field and a value of "Flagged for Delete" instead of simply allowing staff to delete an asset that is not longer or was accidentally added. This has caused unintended issues in the upstream/downstream trace capabilities as well as with utilizing Data Reviewer. I don't want to repeat things like this as I move us forward!). 

KadinBaird41
New Contributor II

@AlixVezina I don't know that there was anything specific that I was looking for in this video. I was more hoping that there would be good tips/tricks to reviewing the data load once completed. But what does come to mind, at least for my situation, is haw to make this a reiterative process as we are adding new data and acquiring new assets all the time. Once I have migrated the initial network data to the UN it would be good to have a way, possibly with these tools, to get this new data into the system. Thanks.

About the Author
Alix Vézina is a senior product engineer on Esri's ArcGIS Solutions Web Development team, where she has successfully delivered industry web mapping tools since 2019. Her previous work includes assisting in the digital GIS transformation of a property and land-management organization in New Zealand and coauthoring ArcGIS learning resources for the Public Safety sector. In her free time, Alix enjoys hiking with her puppy in the Colorado foothills.