Latest Contributions by BruceHarold

Earthquakes definitely fall into the 'hard to see' category, but also tricky to get right in your GIS. You can easily find earthquake data, government agencies offer feeds and historic databases from which you can extract data.  This is great for 2D maps, but often the Z (vertical) coordinates are given as positive depth values in kilometers, so 'going the wrong way' for the normal 'positive up' coordinate system.  Another wrinkle is the default Z domain for geodatabases has a Z minimum at -100,000, and the lithosphere extends below this depth in meters, so you can lose features on the way in. I'm not going to do a big post on coordinate systems, I'm just going to throw a couple of things over the fence for you to look at.  Firstly watch the movie file in the blog downloads.  I was involved a few years ago in adjusting GIS data after an earthquake moved the ground (a lot, over 6m in some places).  Just watch the movie to see a year's worth of quakes go by and fly to where a lot of deformation occurred after a severe one; you'll fly past labels of movement values and to a homestead that shifted.  The apparent sudden jump of the property is real, and what you'll see is high resolution orthophotography before and after the adjustment work (it didn't have to be re-flown, just adjusted). The movie was exported from an ArcGIS Pro 3D Scene, but this was only possible with correct 3D points for the quakes, and that data was made from a GeoJSON download and processing with the Spatial ETL tool Quakes2016.fmw that is the second download file. Its a really simple workspace.... ..until you go to the Tool Parameters>Scripting>Startup Script setting and see a bit of fancy footwork making a custom Feature Dataset in the output geodatabase with a Z domain that goes to the center of the earth.  The takeaways are you might not have known about startup scripts and that you can use one to operate on workspace parameters. Please comment on the post with your experiences and ideas. ... View more

Dataset management in ArcGIS has plenty of supporting tools and workflows, but when you don't have control for any reason you may be the person who has to figure out what data changed, and where.   This blog is about a tool published in the ArcGIS Online sample galleries for bulk change detection between pairs of feature classes.   My first example datasets are two parcel feature classes, where one has been revised with survey and subdivision work, but without any edit tracking fields - the data is not managed in ArcGIS.  The maps are named for their content, Original has the old data, Revised has the new data.     The two datasets have about 650,000 features each over a huge area, so visual comparison is impossible, especially as I need to compare attributes too.  The Feature Compare geoprocessing tool is an option if my data has a unique key field to sort on (it does) but its output is a table, I want features.   The Pro Change Detector tool delivers flexible change detection between two feature classes with your choice of attribute and geometry comparison, and outputs feature classes of Adds, Deletes, Updates and NoChanges (Updates are only detectable if the data has a unique key field separate to ObjectID; without a key field updates are output as spatially overlapping deletes and adds).   The tool requires the ArcGIS Data Interoperability extension, but you don't have to learn to drive the Workbench application delivered with Data Interoperability, this sample is just a normal Python script tool.   For my parcel data I chose all the attributes to be considered as well as geometry:     Then 7 1/2minutes later after comparing ~650,000 features per input I had my change sets:     You can compare any geometry type but if you are going to do change detection of multiple pairs of feature classes be sure to change the output objects names as the tool will overwrite its outputs.  Alternatively, keep your data in separate project databases (see below).   For a second example I decided to 'go big' and compare two street address datasets each with about 2 million features and a lot of attributes:     Now its 22 minutes to find a couple of thousand changes to 2 million features:     ...and in the map it is easy to find a locality where subdivision has resulted in new addresses being created - see the extra address points in the Revised map:     To use the tool your data must be in a single File Geodatabase, here is how my Catalog pane looks, note to preserve my change sets I used two separate databases in the Project.     The tool was created with ArcGIS Pro 2.5 beta 2 software (sharp eyed people will see the new style geoprocessing Details view above) but works in Pro 2.4.  You will need ArcGIS Data Interoperability installed and licensed, and you'll need permission to copy a file into the install of your Pro software, please see the README file in the download.   Now go detect some changes and comment in this blog how you get on! Edited 2/8/2021 to replace the .pth file with one suitable for Python 3.7 in Pro 2.7+ ... View more

Many organizations publish OGC WFS services as one option for data supply, either to the general public or to a restricted audience.  Often however these services are intended for large scale mapping, such as within a single municipality, and bulk download at national scale is not supported - either a maximum feature collection size per request is set on the server, or response paging is not supported, so an out-of-the-box client is not going to deliver an entire dataset.   Sometimes, although these restrictions are not present, assembling and delivering a request for a large feature collection is beyond the capability of the server or network settings (by design), or the client app doesn't support paging (full disclosure, WFS 2.0.0 response paging is coming to core ArcGIS Pro in a future release; Data Interoperability extension already supports WFS 2.0.0 paging if the server provides next/previous URLs). This blog is about using ArcGIS Data Interoperability to work around these limitations to achieve repeatable bulk download of WFS data at any scale.  You will need solid Data Interoperability (or FME) skills to implement this workflow, or be willing to learn from the content of the blog download. At this point I need to show you a map or you'll go do something else, so I bring you today's subject matter - Norway! It's necessary to use a real world example, and the people at GeoNorge have excellent public WFS services that let me show the issues, so Norway is it.  Browsing their site I settled on a road network service.  Here is how to get there yourself, while optionally learning a little Norwegian.  Here is GeoNorge, (don't use '/en' if your Norwegian is up to it) click on Go to the map catalogue, then in the selector pane on the left choose Type = Service, Topic = Transportation, Distribution form = WFS Service, then of the available services click on ELF Road Transport Network.  Scroll down and you'll see:  Get Capabilites Url: https://wfs.geonorge.no/skwms1/wfs.inspire-tn-ro?request=GetCapabilities&service=WFS. If you don't know OGC standards, be thankful, that's our job!  The URL above is a typical pattern, the XML document returned advertises what the WFS service can do.  You know I'm going to make you click on the above URL don't you and inspect the response, but before the excitement of XML we'll go off road here and begin to understand the problem a little better. Here is a map of 50 food businesses within 500m walking distance of the Royal Palace in Oslo.  I detect a pattern of having to walk north or south of the palace for lunch, which is interesting, maybe its a function of having to cross a major road bisecting the area, but my main point is downtown Oslo has a lot of roads you can walk alongside, whereas up in the arctic circle - not so many (no map, but trust me).  We're going to need a way to read the WFS road transport service in chunks such that we don't request more than the service response limit in cities and don't make unnecessary requests in areas with few roads.  We're going to design a tiled WFS reading strategy. OK now click on the GetCapabilities URL and look for these things: We cannot request pages: We can only get 10000 features at a time: We can retrieve tn-ro:RoadLink feature types in a wide variety of coordinate systems over a huge area: We can request features within a Bounding Box (BBOX): Now for an exercise.  Open the Workbench app from the Analysis ribbon (Data Interoperability will need to be installed and licensed) and add a WFS Reader using these parameters (GetCapabilities URL, WFS Version 2.0.0, RoadLink feature type, no MaxFeatures).  Connect a logger to the reader, there is no need to write anything. Run the workspace, you will see this URL is generated and you'll get a download containing 10000 features. Now add the URL to your browser then edit the URL to add a parameter 'resultType=hits'.  This is a special request to count the number of features available in the service, run the edited URL in your browser.  You'll get a response like this: See the numberMatched property -  1,976,423 Road Link features are available. Norway has a land area of ~385,000 square kilometers, so on average ~5 road link features per square kilometer, and on average ~2,000 square kilometers will have ~10,000 road links, the WFS service limit, roughly a 45km square.  It is going to be a much larger area in the country's north to contain 10,000 features.  Using the scientific method of picking a convenient number out of thin air that is the right order of magnitude, my starting point for a WFS-reading tiling scheme was a 100km square fishnet, made with the Create Fishnet geoprocessing tool (cells that do not intersect land are deleted, and I went with ETRS 1989 UTM Zone 33N projection, which is EPSG:25833 in the service properties): Notice I added some fields (XMin,YMin,XMax,YMax,RoadCount) to the fishnet and set the initial values for the coordinate bounds fields (using Python snippets - these are in the blog download).  These bounds are going to be used as Bounding Box parameter inputs in WFS requests.  Now I need a workflow to refine the fishnet so cells are subdivided progressively so less than 10,000 road link features will be in each.  First I need to figure out the methodology of reading the WFS service in an extent.... If you open Workbench and drag in BasicGetFeatureWithBBOX.fmw from the blog download you'll see a WFS reader with the properties I needed to inspect a GetFeature URL.  The workspace looks like this: Under the reader you can see how I replicated the GetFeature URL in an HTTPCaller but parameterized the BBOX values.  I used a fishnet cell extent containing the city of Trondheim.  The download format is GML  I used the Quick Import geoprocessing tool (available with Data Interoperability) to translate the GML into a file geodatabase.  Here are 10,000 road links around Trondheim: Now I have the building blocks of a tiled WFS reader.  And here it is!  ReadWFSFeatures.fmw: The Spatial ETL tool reads RoadLink features in fishnet cells selected by a WHERE clause, here is the first pass reading features in all cells: I can see not all 100km cells intersect roads - the ones you can see selected in the fishnet layer - so they can be deleted.  Now the work of refining the fishnet begins. The iterative workflow is this (be very careful!): Run ReadWFSFeatures.fmw with a WHERE clause selecting the smallest cell size (initially Shape_Length = 400000, then 200000 when those cells are made, then 100000 when those are made in a subsequent step below...) Add the output RoadLink feature class to your map Run RoadCount.py in the Python window to populate RoadCount in NO_Fishnet Select NO_Fishnet features with RoadCount >= 9000 (undershooting 10,000 to allow for road construction) If there are no NO_Fishnet features selected then BREAK - you are finished making the fishnet Run MinimumBoundingFishnet to create a separate fishnet with cells half the width/height of the previous minimum; it is important the selection on NO_Fishnet is still active Run Delete Features on the selected NO_Fishnet cells Run Append to add the generated smaller fishnet cells to NO_Fishnet, using the field map option. Run SetExtentAttributes.py in the Python window to recalculate the boundary coordinates Delete the RoadLink feature class Go back to the first step The first subdivision of fishnet cells into 50km square features with MinimumBoundingFishnet looks like this: After looping through the fishnet refinement process until no cells contain more than 9,000 roads, you can run ReadWFSFeatures.fmw with a WHERE clause that selects all fishnet cells and create the complete RoadLink feature class.  Finally run RoadCount.py to populate NO_Fishnet with how many road segments intersect each cell.  See if there are any cells with RoadCount = 0 and if you think roads will never be built there then delete the cells, but you'll have to be Norwegian to make that judgement. Downloading all features took exactly 1hr 0s and exactly 1,976,423 arrived, just as advertised by the WFS service.  Here is how the data looks, with the labels being the final road count: The fishnet can be repurposed to access other WFS features from the GeoNorge agency, and the methodology applied to any WFS service that cannot supply a complete dataset with core approaches. This post was created using ArcGIS Pro 2.5 beta 2 software, but the .fmw files should work in Pro 2.4.  If the MinimumBoundingFishnet tool doesn't work for you, download a fresh copy from here. ... View more