Latest Contributions by BruceHarold

In an earlier post I introduced a technique for capturing map extents from user input and sending these as parameters to a Spatial ETL Tool.  This made the spatial extent of the processing dynamic with user input.  The key was wrapping the ETL tool with ModelBuilder to take advantage of its ability to interact with a map. This post is along similar lines except showing how to capture a user's selection of feature classes to process at run time.  This makes the feature types being processed dynamic with user input. First some background.  The FME Workbench application used for authoring Spatial ETL tools is designed for repeatable workflows with known input feature types, and the work centers around managing output feature characteristics.  In ArcGIS we are used to geoprocessing tools being at the center of data management and needing to handle whatever inputs come along.  We're going to make Spatial ETL a little more flexible like ArcGIS with some modest ModelBuilder effort. Here is some data: In my project database it looks like this (the main point is it is all in one geodatabase): ...and my Project Toolbox has a Spatial ETL Tool and a Model: The Spatial ETL Tool... ...does absolutely nothing!  Well, it reads some default feature types from a default File Geodatabase, then writes them all out to the NULL format (great for demos, it never fails).  The trick here is I made the 'FeatureTypes to Read' input parameter of the File Geodatabase reader a User Parameter (you right click on any parameter to publish it this way). The only other thing to 'know' ahead of the Modelbuilder stuff is that the ArcGIS Pro geoprocessing environment is smart enough to see Spatial ETL tool inputs and outputs that are Workspaces in geoprocessing terms (Geodatabases, Databases, Folders) as the correct variable type in ModelBuilder but that usually other FME Workbench workspace parameters you might expose are seen as String geoprocessing parameter type.  This means in our case if we choose multiple feature classes from my project home Geodatabase, like say 'Adds' and 'Deletes', then the ETL tool wants the value supplied to be a space-separated string like 'Adds Deletes'. Here is the model, DynamicFeatureTypesModel.  Its last process is the Spatial ETL tool DynamicFeatureTypes.  There are three processes ahead of it. On the left is the sole input parameter 'FeaturesToRead', of type Feature Class (Multi Value) (you could use Feature Layer too with a little more work in the model to retrieve source dataset paths): There are three Calculate Value model tools, their properties are: Get GDB: This returns the Geodatabase of the first feature class in the input set. GetFeaturesToRead: This returns the names of the feature classes as a space-separated string. GetGDB and GetFeaturesToRead supply the ETL tool input parameter values. CheckSameWorkspace: This returns a Boolean test that all input feature classes are from the same Geodatabase.  It is used as a precondition on the Spatial ETL Tool as that is designed with a File Geodatabase reader and must receive that format data and only once. That's it!  The DynamicFeatureTypes model can be run like a normal project geoprocessing tool with the ability to select any desired inputs, and the Spatial ETL tool behind the scenes takes what it gets.  If you select inputs from different File Geodatabases the precondition check will prevent the tool from executing. Here is the details view from a run with data from a different Geodatabase. Please do comment in this blog with your comments and experiences.  The project toolbox and ETL source are in the post attachment. ... View more

Earthquakes definitely fall into the 'hard to see' category, but also tricky to get right in your GIS. You can easily find earthquake data, government agencies offer feeds and historic databases from which you can extract data.  This is great for 2D maps, but often the Z (vertical) coordinates are given as positive depth values in kilometers, so 'going the wrong way' for the normal 'positive up' coordinate system.  Another wrinkle is the default Z domain for geodatabases has a Z minimum at -100,000, and the lithosphere extends below this depth in meters, so you can lose features on the way in. I'm not going to do a big post on coordinate systems, I'm just going to throw a couple of things over the fence for you to look at.  Firstly watch the movie file in the blog downloads.  I was involved a few years ago in adjusting GIS data after an earthquake moved the ground (a lot, over 6m in some places).  Just watch the movie to see a year's worth of quakes go by and fly to where a lot of deformation occurred after a severe one; you'll fly past labels of movement values and to a homestead that shifted.  The apparent sudden jump of the property is real, and what you'll see is high resolution orthophotography before and after the adjustment work (it didn't have to be re-flown, just adjusted). The movie was exported from an ArcGIS Pro 3D Scene, but this was only possible with correct 3D points for the quakes, and that data was made from a GeoJSON download and processing with the Spatial ETL tool Quakes2016.fmw that is the second download file. Its a really simple workspace.... ..until you go to the Tool Parameters>Scripting>Startup Script setting and see a bit of fancy footwork making a custom Feature Dataset in the output geodatabase with a Z domain that goes to the center of the earth.  The takeaways are you might not have known about startup scripts and that you can use one to operate on workspace parameters. Please comment on the post with your experiences and ideas. ... View more

Dataset management in ArcGIS has plenty of supporting tools and workflows, but when you don't have control for any reason you may be the person who has to figure out what data changed, and where.   This blog is about a tool published in the ArcGIS Online sample galleries for bulk change detection between pairs of feature classes.   My first example datasets are two parcel feature classes, where one has been revised with survey and subdivision work, but without any edit tracking fields - the data is not managed in ArcGIS.  The maps are named for their content, Original has the old data, Revised has the new data.     The two datasets have about 650,000 features each over a huge area, so visual comparison is impossible, especially as I need to compare attributes too.  The Feature Compare geoprocessing tool is an option if my data has a unique key field to sort on (it does) but its output is a table, I want features.   The Pro Change Detector tool delivers flexible change detection between two feature classes with your choice of attribute and geometry comparison, and outputs feature classes of Adds, Deletes, Updates and NoChanges (Updates are only detectable if the data has a unique key field separate to ObjectID; without a key field updates are output as spatially overlapping deletes and adds).   The tool requires the ArcGIS Data Interoperability extension, but you don't have to learn to drive the Workbench application delivered with Data Interoperability, this sample is just a normal Python script tool.   For my parcel data I chose all the attributes to be considered as well as geometry:     Then 7 1/2minutes later after comparing ~650,000 features per input I had my change sets:     You can compare any geometry type but if you are going to do change detection of multiple pairs of feature classes be sure to change the output objects names as the tool will overwrite its outputs.  Alternatively, keep your data in separate project databases (see below).   For a second example I decided to 'go big' and compare two street address datasets each with about 2 million features and a lot of attributes:     Now its 22 minutes to find a couple of thousand changes to 2 million features:     ...and in the map it is easy to find a locality where subdivision has resulted in new addresses being created - see the extra address points in the Revised map:     To use the tool your data must be in a single File Geodatabase, here is how my Catalog pane looks, note to preserve my change sets I used two separate databases in the Project.     The tool was created with ArcGIS Pro 2.5 beta 2 software (sharp eyed people will see the new style geoprocessing Details view above) but works in Pro 2.4.  You will need ArcGIS Data Interoperability installed and licensed, and you'll need permission to copy a file into the install of your Pro software, please see the README file in the download.   Now go detect some changes and comment in this blog how you get on! Edited 2/8/2021 to replace the .pth file with one suitable for Python 3.7 in Pro 2.7+ ... View more