ArcGIS Data Interoperability Blog - Page 7

Everyone likes SQLite. It is a single portable file, performs and scales well, supports enough SQL to be useful and has a DB-API compliant Python module and API access in other languages. It is embedded in many mobile and desktop apps, and is directly usable in ArcGIS Pro.

SQLite as a container has an incarnation - OGC GeoPackage - that supports the encoding of vector and raster features for direct use in ArcGIS Pro.  You can read about the standard on the OGC website.

The GIS format most often compared with GeoPackage is the Esri-defined shapefile.  Shapefile is the most shared GIS format on the planet and its encoding of vector features is published.  Note however the publication date - 1998.  At the time the shapefile was designed, the components available had limitations that can frustrate today's advanced workflows.  These include file size limit, attribute field count and name width limits, dates not supporting time, complexity in handling character encodings and lack of null value support for most field types.  Shapefile has been spectacularly successful for handling simple vector features, but it can be limiting.

I think of GeoPackage as the new shapefile without the old limitations and I encourage you to use it. It is a great format for, well, geo-packaging! However, don't go as far as thinking it is a full-blown GIS workspace, it doesn't have geodatabase behaviors like domains and attribute rules. What it does, it does well.

GeoPackage is extensible, and there are approved OGC extensions for gridded tiles of elevation data and table relationships, and non-approved community extensions such as map styling of features and storing vector tiles. ArcGIS Pro does not yet implement support for any GeoPackage extensions (excepting table functionality adopted in the v1.2 release).

What can you do with a GeoPackage in ArcGIS Pro 2.6?

• Read and write simple features (polygons, polylines, points, multipoints, circular arcs, tables)
• Create feature classes with the Feature Class to Feature Class tool
• Create tables with the Table to Table tool
• Use Copy/Paste in the Catalog pane
• Use the Append tool to add data to an existing feature class or table
• Use the Add Raster To Geopackage tool to store imagery

• Edit features or rows with the ability to undo and redo edits

• Modify the schema
• Geoprocess with any tool that takes a simple feature class or table as input
• Share GeoPackage data with other users as a static item in ArcGIS Online
• Use Geopackage  vector and raster data in map workflows
• Read or write GeoPackage in Data Interoperability ETL workflows
• Use SQL statements in SQLite's native dialect

What can you not do with a GeoPackage in ArcGIS Pro 2.6?

• Publish a GeoPackage item as a hosted web layer
• Store or edit metadata
• Use any geoprocessing tool that requires geodatabase output

Some recommendations:  You can add fields and calculate values with geoprocessing tools or ArcPy, but you may find it slower than native geodatabase operations.  Geometry storage in a GeoPackage is not compressed like a geodatabase, so they can get big.  Do your geoprocessing before creating your GeoPackage, then copy your data into it.  Think of GeoPackage as a sharing format.

Move your data into the GeoPackage like this:

• Create a GeoPackage with the Create SQLite Database tool (using the GeoPackage spatial type)
• Use the Copy tool (Data Management, General toolset) to add vector data, or Copy/Paste in the Catalog pane
• Use the Add Raster to GeoPackage tool (Conversion, To GeoPackage toolset) to add raster mosaics

Your GeoPackage is now ready for use.

Note on sharing:  You can upload a .gpkg file to ArcGIS Online, the file type will be recognized.  You can send a link after sharing the item and others can then download it from the content gallery.

Advanced topic:  Because it is based on SQLite, GeoPackage comes with a database engine and good SQL language support.  There are 3rd party tools for working with SQLite which you may find useful, but to include a spatial component in your work the ArcGIS Data Interoperability or Safe Software FME products support scripting SELECT, CREATE, DROP, DUPLICATE, TRUNCATE and CROSS JOIN statements within Spatial ETL tool transformers like SQLCreator and SQLExecutor.  This approach enables very powerful and performant use of a GeoPackage.

This post is about automating repetitive ETL processes right from your desktop.  No code, no server.

Note:  This post originally discussed only one way to schedule ETL processing, but with the ArcGIS Pro 2.5 release, due out soon, job scheduling is coming to Desktop geoprocessing right from any tool's Run button!  I'll leave the 'legacy' approach details in the post but do read through to the 'new' approach once you're able to deploy Pro 2.5.

The legacy approach:

We're seeing many people using Data Interoperability to periodically synchronize datasets between systems of record.  Typically the source data refresh 'trigger' is driven by a schedule and not some random event, and the frequency of updates is based on multiples of a working day.  If you're on this kind of treadmill this post is for you!

You may have heard of this sort of automation in the context of Windows Task Scheduler with a Python script as the task and the script calling a geoprocessing tool or model.

We're going down the task scheduling path too, but without needing Python.

In the modern era there is a lot of emphasis on service oriented architecture and the ArcGIS stack has comprehensive publication and synchronization capabilities amongst apps, but you're reading this because you're working outside the stack, at least at one end of your synchronization workflow.  You have used Data Interoperability's Workbench app to wrangle services, databases, files and so on to achieve your own private batch 'service'.  You don't have to be the server and click 'Run' too.  Your friend is this guy:

C:\Program Files\ArcGIS\Data Interoperability for ArcGIS Pro\fme.exe

That's right, a big fat executable.  This is the one that does all the work when an ETL tool runs.  You may never have noticed, but when you run an ETL tool while being edited in Workbench, the very first line that appears in the log window is:   Command-line to run this workspace:  followed by the path to our new friend above and the path to the open workbench .fmw file, and any arguments the workspace needs.  Its all there, so lets plug it together.

Lets dispense with some legalities first.  With ArcGIS Pro, Enterprise and OnLine you're living in a world of named user licensing.  Your ETL tool may embed these credentials.  Provided the scheduled task you build automates the ETL tool on the machine you would use to run it interactively there should not be any licensing issues.  If someone else needs to run it they should replace the named user credentials first.

For my example I'm going to recycle an ETL tool example from an earlier post.  I use it to maintain a hosted feature service using data harvested from a Geoserver instance via an extended WFS API.  It has an official refresh rate of once a week, each Saturday local time; I run the ETL tool when I remember to on Monday mornings (hey its only a demo).  Let's automate that.  Mondays are getting problematic for me, I may forget.

The example ETL tool reports the command line I should use to run the workspace is:

"C:\Program Files\ArcGIS\Data Interoperability for ArcGIS Pro\fme.exe" C:\Work\Synchronize\Synchronize.fmw --API_Key "im_not_telling_you_my_api_key" --LDS_Unique_ID "address_id"

Because ETL tools store their parameter values it isn't necessary to supply those arguments if they don't change, so this works too:

"C:\Program Files\ArcGIS\Data Interoperability for ArcGIS Pro\fme.exe" C:\Work\Synchronize\Synchronize.fmw

Now we create the scheduled task.  Open Task Scheduler and fill in the dialogs for a Basic Task:

Adjust the settings how you need:

Tip:  If you configure the task exactly as above a command window like below will pop up, if you don't want this use the setting 'Run whether user is logged on or not'.

While I remember, if you're interested in more ways to batch ETL check out this post.

Now do your bit and come in late Mondays!

Note:  We have had reports from the field that Windows Task Scheduler can be impeded from working by some system security settings.  If you find this and cannot work around them with your IT department, log a support call with Esri and ask the analyst to consult Analyst Knowledge Article 000022373 which has a reference to an alternative scheduling technology.

The new approach:

Please read the Pro 2.5 help topic 'Schedule geoprocessing tools' for details, I'll only show the user interface experience here.  Starting with the same 'Synchronize' ETL tool as in the legacy approach outlined above, I create a scheduled tool from the Run button, here is a screen grab:

Select 'Schedule' and you'll get a configuration dialog:

I set up weekly recurrence like in the first example; to refine the 'Begin On' value the pull-down supplies a handy date-time picker:

I'm done!  How easy is that!  Apart from the obvious ease of setting up the automation you should note that the ETL tool is just a tool, there are no special considerations around handling an ETL tool versus a core geoprocessing tool (or model).  Caveat, if you are using concurrent licensing and scheduling a Python script tool that calls any extensions (Data Interoperability for example) then your code will need a CheckOutExtension() call.
A fine point, don't forget to use appropriate power management (disk shutdown, sleep, hibernate) settings for your scheduling PC, talk this through with your IT folks if you have any doubts, for example it is possible for network administrators to enforce rules for hibernation that override the visible power settings.

Now go ahead and automate stuff!

Let me get you through one paragraph of background before we get to the fun stuff:  In an earlier video I included an example of capturing a spatial constraint from the active ArcGIS Pro map or scene and sending it into an ETL workspace.  The sample happened to be working with a WFS service; these have a bounding box parameter that can constrain the features retrieved.  WFS services also support more complex spatial operators which can be used with arbitrary geometry operands supplied as GML fragments.  However, unless you know how to put all the required XML together for WFS requests then you'll be like me and terrified of attempting it.  ArcGIS Pro 2.3 itself only supports a bounding box constraint on WFS services.

Spatial constraints are a lot easier with feature services.  This blog will show you how easy.

Core geoprocessing has supported feature services as input parameters for several releases now, why bother using Spatial ETL against feature services anyway?  Well, if your feature service is heading out the door as some other format, or you are using some transformations indicating Data Interoperability, or your feature service is very large and you don't want to use selections to subset it.  I just helped one customer who needed to dynamically handle a spatial constraint mid-ETL with a FeatureReader transformer (more on that below).  There are many use cases.

Data Interoperability is all about code-free approaches, but I'll take a wee diversion into feature service REST API query parameters so you understand what goes on.  Below is a screen shot of the HTML view of a feature service Query endpoint.  Note there is an Input Geometry parameter (supplied as JSON) and you can set how it is used, in my case it is a Polygon for which I want only features satisfying the constraint Intersects.

So, the trick with applying spatial constraints to feature services is just supplying the geometry!

In the blog download (Pro 2.3+) you'll find the sample tool used, but the approach is very simple, just apply it yourself in your own models.  Click to enlarge this graphic to see the map I used, the feature set in the map and table of contents and the model run as a tool.  The feature set is driving the analysis geometry automatically.

The tool being used is the Model named SpatiallyConstrainedGP which has an input parameter of type Feature Set.  At run time you supply a value by choosing a layer or feature class or creating a feature manually by editing in the map.

SpatiallyConstrainedGP wraps the ETL tool SpatiallyConstrainedETL like this, there is a model tool Calculate Value between the input feature set and the ETL tool:

All that is happening with Calculate Value is the input feature set is turned into a JSON string with a Python snippet:

The JSON is then supplied to the published ETL tool parameter Input Geometry (remember the Query endpoint!) and...

...the ETL tool does its stuff, considering only features intersecting my feature set...

..which is to make a spreadsheet summarizing some parcel area totals per case of an attribute:

So that's it, just grab JSON from the map when you need to supply a feature service reader with an Input Geometry parameter.  if you are using a FeatureReader transformer to read a feature service the workflow is a little different, you'll need to convert the JSON into an actual FME feature with a GeometryReplacer (the geometry encoding is Esri JSON) and supply it as the initiator Spatial Filter constraint of the FeatureReader, like this:

Now you can apply map-driven spatial constraints to your ETL!

Data Interoperability extension sees Point Cloud data, such as ASPRS LAS and Esri LAS Dataset as their own feature type, just like many other formats.  Here is some on a coastline - surf's up!   (Look above the headland)

Formats are designed to deliver specific capabilities, but all geospatial formats have something in common - a coordinate system - and your GIS needs to be able to manage it.  LAS data is a bit of an outlier here as we expect ArcGIS users to collect their data in the coordinate system they intend to use, and stick with it, but in the case where the 'ground moves' (literally, like plate drift or quakes, or if a new datum or realization is published) then ArcGIS's comprehensive core projection tools don't yet support the format.

A situation we hear about is people have LAS data in ellipsoidal heights (say WGS84) and want to generate DEMs in orthometric heights.  Orthometric heights are gravity-defined and approximate height above mean sea level, so they are important if you need to model coastal or estuarine flooding, for example.  You can always create a DEM and reproject its vertical coordinate system with the geoid grids delivered by the ArcGIS Coordinate Systems Data install, or your own local ones, but that leaves the LAS data behind .

Your LiDAR vendor would be pleased to reprocess your LAS data but you can do it yourself with ArcGIS Data Interoperability extension.  The secret is in this transformer - CsmapReprojector:

In the blog download there is a sample specific to accommodating a new vertical datum for New Zealand, but read between the lines in the document delivered in the download and leverage the vertical grids delivered in the Coordinate Systems Data install, or geoid grids you obtain locally, and reproject your LAS data how you need.

Then when a point says its floating you can trust it (bad I.T. pun).

Note:  The blog download and the Geoprocessing gallery sample here are equivalent.

Agencies around the world publish their data on the web using a great variety of technologies, and while standards exist to make them accessible within ArcGIS, nothing performs within ArcGIS like our own services.  Sometimes it just makes sense to regularly synchronize data from its system of record to ArcGIS Online or Portal.  This blog is about how to do that efficiently.

To see if you should read further, download the blog attachment NZ_Street_Address.lyrx and add it to a new map in Pro, then using the Locate pane and the ArcGIS World Geocoding Service zoom to Wellington, NZL (or your favorite other New Zealand locality).  Zoom in to 1:5000 or larger scale, pan around, turn on the label classes Standard Number, Suffix Number and Range Number and inspect the address point house numbers.  Identify features.  Select features.  You are accessing a feature layer in an ArcGIS Online standard feature data store.  Here are links to the item and service.  If you have a reasonable internet connection you will have a good map exploration experience.  The layer you are looking at has over 2 million features.  You can download the data.  You can use it in geoprocessing.  The data is maintained weekly and the synchronization process averaging thousands of updates each week takes under 2 minutes.  The approach uses no coding.  If you want to do this for data accessible to you then read on (click on images to enlarge them).

Firstly, what data sources are candidates for this treatment?  Anything accessible to ArcGIS Data Interoperability extension, which is all these formats and feeds in many storage repositories.  My specific example happens to use data available by WFS service but that is not critical to the discussion, the approach is generic.

Lets dig a little deeper.  To look at the layer a little more closely, with ArcGIS Online as your active portal, Add Data from All Portal with search tags 'LDS' and 'ETL'.

You'll see the same point features (with default symbology) but also in your table of contents there is a standalone table 'Timestamps' with one row:

The value in UpdatedUTC is refreshed at each synchronization so will differ from the graphic but its the key to synchronization.  It lives within the feature service as a layer.  The UTC time of synchronization is the final step of the process that also writes feature updates.

So what are all the steps?  To follow you'll need ArcGIS Pro 2.3+ with Data Interoperability extension installed and enabled, and to have downloaded the toolbox and ETL tool source .fmw files in the blog download Synchronize.zip.  Add the toolbox to your project, you'll see these ETL tools in it:

Right click each ETL tool and repair the source path to its .fmw file.

My target data is available as a bulk download, which i took as a file geodatabase.  I copied the address point feature class into my project home geodatabase.  In any event get your target data into your project home geodatabase, using ETL processes if necessary.

Next I made the Timestamp table using MakeTimestampTable, which looks like this:

See note below its not a great idea to use the table name 'Timestamps' but we'll let it go for now

Repair the destination file geodatabase path to be the same as your features of interest.  If you run MakeTimestampTable in edit mode you can pick your own initial timestamp value with a useful date picker.  I used UTC time but didn't have to get it exact, if you do and live in Greenwich UK then look at your watch and ignore any current daylight savings adjustment, otherwise use a little Python after making the table with any value:

Then calculate UpdatedUTC to equal DownloadedUTC and you'll have it:

Its at this point in blog writing you find out its a really bad idea to use a table name 'Timestamps' as it is too close to a reserved word in many database technologies including file geodatabase, but as it doesn't affect my goal here I'll leave it, but if you go into production use another name!

Now stand up a feature service.  Add your target data and the timestamp table to a map, then select both objects in the table of contents:

Then right click and choose Share as Web Layer:

Configure the service to be a feature layer in the folder you want and let it load.

Included in Synchronize.tbx is an ETL tool LoadData that creates a feature service too if you want to go that way.

Now for the synchronization stuff in the ETL tool Synchronize:

The design of your version will depend on your target data, but in broad strokes:

• The current UTC time at the beginning of processing is captured
• The timestamps layer (table) is read from the Esri web layer
• Your target data is read from its system of record
• Inserts, Updates and Deletes are derived between the target source and Esri web layer
• Inserts, Updates and Deletes are validated by unique identifier comparison with the Esri layer
• Deletes are committed
• Updates are written
• Inserts are written
• The timestamps layer (table) is updated with the UTC time captured when processing began

For my target data the curator provides a changeset API that let me build from/to times into a WFS call which gave exact insert, update and delete sets.  If your data has timestamps for created, edited and retired values you can do this yourself.  If you have nothing to go on you can derive changesets by reading all data from both sources and doing brute force change detection with the UpdateDetector transformer, although this of course may take time.

In the Synchronize ETL tool there are some less obvious features.  The sequence of feature writing is determined by writer order in the Navigator pane, top down.  Writing the timestamp update is therefore enforced to be last, so if anything fails it will not be falsely updated.  ArcGIS Online and Portal feature writers in Delete and Update mode require the ObjectID value in the Esri service to be sent with the feature, so the values are picked up mid stream with a FeatureReader and joined on a layer unique identifier.  Similarly, the Inserts stream looks for existing unique identifiers before writing, only features known to not exist pass.

In the opening paragraph I said the approach uses no coding.  There is a math function used (floor) to calculate a batch number in modulo 20 chunks to obtain target service ObjectIDs.  That's as close to writing code you need to get, although you are free to use Python if you like.

While I mention coding, in a production environment you would want to run synchronization as a scheduled task.  This begins as a Python script.  I stub one out here that assumes things like ETL web connections are available to the process owner, which is easily done by sharing the connection file in a well known directory. 

Another approach I'll blog about separately is calling the FME engine executable directly in a scheduled task.

Do explore the ETL tools supplied and send questions to this forum.

I hope this gives you confidence to build your own synchronizations.  Enjoy!

We're going on a journey to the bottom of the sea, but the real message here is the ability of ArcGIS Data Interoperability to reach out to the web (or anywhere) and get feature and media data into a geodatabase feature class with attachments without having to code.  Well just a tiny bit, but you don't have to sweat the details like a coder.

A colleague came to me asking if ArcGIS Data Interoperability could bring together CSV position and time data of a submersible's missions and related media content and get it all into geodatabase.  In production all data sources will be on the web.  No problem.  Data Interoperability isn't just about formats and transformations, it is also about integrations, and building them without coding.

Python comes into the picture as a final step that avoids a lot of tricky ETL work.  The combination of an ETL tool and a little ArcPy is a huge productivity multiplier for all you interoperators out there.  Explore the post download for how the CSV and media sources are brought together - very simply - below is the whole 'program':

ArcGIS Data Interoperability has a great 'selling point', namely that you can avoid coded workflows and use the visual programming paradigm of Workbench to get data from A to B and in the shape you want.  I often show colleagues how to efficiently solve integration problems with Data Interoperability and its always pleasing to see them 'get it' that challenges don't have to be tackled with code.

Low level coding is the thing we're avoiding.  ArcGIS geoprocessing tools are accessible as Python functions; using geoprocessing tools this way is just a command-line invocation of what you have access to in the Geoprocessing pane and Analysis ribbon tools gallery and so on.  If this is news to you, take a look and run the Get Count tool first from the system toolbox and then use the result in the Python window.

Here is the tool experience:

Now in the Catalog pane History view, right click the result and send to the Python window:

You'll see the Python expression equivalent of the tool:

Note I haven't written any code...

Where am i going with this?  ArcGIS Data Interoperability concepts differ a little from core geoprocessing in that input and output parameters tend to be parent workspaces and not feature types or tables within them.  You frequently write to geodatabases for example, in which case the output parameter of the ETL tool is the geodatabase, not the feature classes and tables themselves, although these are configured in the ETL tool.

What if you need to do something before, during, or after your ETL process for which there is a powerful ArcGIS geoprocessing tool available but which would be really hard to do in Workbench?

You use high level ArcGIS Python functions to do this work in Workbench.

I'll give a simple, powerful example momentarily, but first some Workbench Python tips.

Workbench allows you to configure your Python environment; to avoid any clash with Pro's package management just go with the default and use these settings:

In Tools>FME Options>Translation check you prefer Pro's Python environment:

In your Workbench, check your Python Compatibility will use the preference.

Now you know the ArcGIS Python environment can be used.

For my use case I'll provide a real example (attached below) where I need to create and load geodatabase attachments.  We cannot do this entirely in Workbench (except by how I'll show you) because it cannot create the attachments relationship class.  You could do that manually ahead of loading data, but then you still have to turn images into blob data, manage linking identifiers and other things that make your head hurt, so lets use ArcPy.   Manual steps also preclude creating new geodatabases with the ETL tool, which I want to support.

The example Workbench writes a feature class destined to have attachments, and a table that can be used to load them.  You can research the processing, but the key element to inspect is the shutdown script run after completion, see Tool Parameters>Scripting>Shutdown Python Script.

Here is the embedded script:

Now this isn't a lot of code, a few imports, accessing the dictionary available in every Workbench session to get output path values used at run-time, then just two lines calling geoprocessing tools as functions to load the attachments.

This is a great way to integrate ArcGIS' powerful Python environment in ETL tools.  Sharp eyed people will notice a scripted parameter in the workspace too, it doesn't use ArcPy so I can't claim it as avoiding low level coding, it was just a way to flexibly create a folder for downloading images at run-time.  There are a number of ways to use Python in Workbench, but I would be detracting from my message here that you can start with the simple and powerful one - use ArcGIS geoprocessing where it saves work.  Enjoy!

Last week was the 2019 Esri Partner Conference followed by Developer Summit, events at which we enjoy being challenged by friends from around the world who are using ArcGIS in their work, and also other apps and formats that ArcGIS does not make or manage.

One partner from Europe asked how to use GML (Geography Markup Language) files in ArcGIS Pro.  This format is really a category of formats; the underlying XML - as a markup language intends - can be extended, usually to push a data schema down into the protocol.  He had in mind however what we know as Simple Feature GML, which makes the task - well - simpler, but that isn't critical to this discussion.

In ArcMap, Data Interoperability extension may be used to both directly read GML files (of the simple feature profile, recognized from a .gml filename extension, even without licensing Data Interoperability extension) or to make an interoperability connection to any supported GML profile, such as the complex INSPIRE themes.  This workflow is not implemented in Pro, partly because WFS services (which are usually GML "in motion") are the most common use case for GML and are natively supported in Pro, and partly because interoperability connections are being re-imagined for a future release of Pro.

In ArcGIS Pro, Data Interoperability extension can also be used to convert GML files just like in ArcMap - with the Quick Import geoprocessing tool, or with a Spatial ETL tool, but the partner thought asking everyone to license an extension would be a hurdle.

I decided to blog about an implementation pattern that does what was asked for - convert GML to geodatabase features within ArcGIS Pro - but that can also be used to convert any of the hundreds of formats and connections accessible to ArcGIS Data Interoperability.  The GML data originator has access to ArcGIS Enterprise with Data Interoperability extension so the pattern leverages that, but end users with GML files only need ArcGIS Pro and authenticated access to a geoprocessing service.  You can use this pattern to stand up any format conversion you wish at any scale - but I hasten to add it must be a free or cost-recovery-only service if you make it public.

Enough talk, how do we do this?  We're going to use ArcMap and Pro in a double act.  Why both?  At time of writing ArcGIS Pro cannot publish web tools containing Spatial ETL tools, so we'll use ArcMap for that step.

In the blog download below you'll find a 10.6.1 version toolbox which contains these tools (click any images to enlarge them):

GML2FGDB is a Spatial ETL tool that converts one or more files of any schema of simple feature GML to a file geodatabase named GML2FGDB.gdb (with overwrite!).  It looks like this if edited:

If you run it as a tool you can see it has a parameter exposed for GML geometry axis order that defaults to 1,2.  If your data is in Y,X order you can set 2,1.  3D data is supported by the 1,2,3 and 2,1,3 values.

GML2FGDBModel is a model tool that incorprates the script tool ZipGDB to compress the file geodatabase to zip file.  The compression step is necessary because geoprocessing services do not support workspaces (i.e. geodatabases) as output parameters.

GML2FGDBModel is shared as a geoprocessing service (make it synchronous) which I called GML2FGDBService:

Lastly, the model tool GML2FGDBService wraps the service and adds the script tool UnZIPGDB for the round trip from a local GML file or files, to the web service that does the translation without requiring Data Interoperability locally, then finally unzips the scratch ZIP file containing a scratch geodatabase into a user-selected destination directory.

Now GML2FGDBService can be freely used in Pro:

GML2FGDBService will always output a file geodatabase named scratch.gdb to your output directory, so be careful not to overwrite prior output!

Now anyone with access to the model tool and service can convert suitable GML files (or any other data if you refactor the Spatial ETL component) to local geodatabase using ArcGIS Pro.  Enjoy!

Hello Interoperators!

If you watched the video ArcGIS Data Interoperability In Action you might have some questions on how some of the demos were built.

I tend to create Spatial ETL tools with external sources - namely .fmw files - as I can easily drag them into Workbench after opening it from the Analysis ribbon or share them with FME users.  They are attached below to give you the pattern, but they will not all work for you without repair and replacement of credentials.  Contact me if you need help with any of them.

To head off one question - how did I create the New York county name pulldown parameter and attribute value mapping in the Simple Powerful ETL demo, see the CountyParameter.fmw workspace.  It scrapes the NY website and writes a CSV file that can be used to import parameter values for the demo tool.  You can see the pattern and recycle it elsewhere.

Several discussions in the old forum referenced the Concave Hull Estimator script tool by esri's Bruce Harold, but during the conversion and website migrations all of the links to that tool were broken.  I had made an enhanced version of the tool which I have attached.  The bulk of the code is by Bruce, but the code that allows a user to select a case field was added by me.  The case field creates selection sets of points for processing based on the case field values without having to manually create each set and run the tool separately for each.

An example output of the tool is shown below (I don't recall the k factor I used).  All of the hulls were created using a case field number as shown.

Anyway, while the real credit for this tool belongs to Bruce, his original post has been referenced recently on several websites with each commenting that it the link has been lost.  I wanted to revive the link to a version of that tool (with my own contribution thrown in).  Interestingly, this tool included the use of a dictionary before I understood anything about what dictionaries did.

Here are pictures I just processed on a set of address points for three Subdivisions. For comparison the original parcels are shown. The starting k factor for this tool run was set to 3, but the tool iterated each point set to at least a k factor of 6 before creating each polygon (a k factor of 9 was used for one of them). The tool created the new hull feature class and all 3 hulls in under 35 seconds.  The presence of somewhat regularly distributed points that fill the interior of the hull helps to create a more accurate hull outline than just using the set of points that ought to define the outline.