Skip navigation
All Places > Open Platform: Standards and Interoperability > Blog > 2019 > March
2019

We're going on a journey to the bottom of the sea, but the real message here is the ability of ArcGIS Data Interoperability to reach out to the web (or anywhere) and get feature and media data into a geodatabase feature class with attachments without having to code.  Well just a tiny bit, but you don't have to sweat the details like a coder.

 

A colleague came to me asking if ArcGIS Data Interoperability could bring together CSV position and time data of a submersible's missions and related media content and get it all into geodatabase.  In production all data sources will be on the web.  No problem.  Data Interoperability isn't just about formats and transformations, it is also about integrations, and building them without coding.

 

 

Python comes into the picture as a final step that avoids a lot of tricky ETL work.  The combination of an ETL tool and a little ArcPy is a huge productivity multiplier for all you interoperators out there.  Explore the post download for how the CSV and media sources are brought together - very simply - below is the whole 'program':

 

 

ArcGIS Data Interoperability has a great 'selling point', namely that you can avoid coded workflows and use the visual programming paradigm of Workbench to get data from A to B and in the shape you want.  I often show colleagues how to efficiently solve integration problems with Data Interoperability and its always pleasing to see them 'get it' that challenges don't have to be tackled with code.

 

Low level coding is the thing we're avoiding.  ArcGIS geoprocessing tools are accessible as Python functions; using geoprocessing tools this way is just a command-line invocation of what you have access to in the Geoprocessing pane and Analysis ribbon tools gallery and so on.  If this is news to you, take a look and run the Get Count tool first from the system toolbox and then use the result in the Python window.

 

Here is the tool experience:

 

 

Now in the Catalog pane History view, right click the result and send to the Python window:

 

 

You'll see the Python expression equivalent of the tool:

 

 

Note I haven't written any code...

 

Where am i going with this?  ArcGIS Data Interoperability concepts differ a little from core geoprocessing in that input and output parameters tend to be parent workspaces and not feature types or tables within them.  You frequently write to geodatabases for example, in which case the output parameter of the ETL tool is the geodatabase, not the feature classes and tables themselves, although these are configured in the ETL tool.

 

What if you need to do something before, during, or after your ETL process for which there is a powerful ArcGIS geoprocessing tool available but which would be really hard to do in Workbench?

 

You use high level ArcGIS Python functions to do this work in Workbench.

 

I'll give a simple, powerful example momentarily, but first some Workbench Python tips.

 

Workbench allows you to configure your Python environment; to avoid any clash with Pro's package management just go with the default and use these settings:

 

In Tools>FME Options>Translation check you prefer Pro's Python environment:

 

 

In your Workbench, check your Python Compatibility will use the preference.

 

 

Now you know the ArcGIS Python environment can be used.

 

For my use case I'll provide a real example (attached below) where I need to create and load geodatabase attachments.  We cannot do this entirely in Workbench (except by how I'll show you) because it cannot create the attachments relationship class.  You could do that manually ahead of loading data, but then you still have to turn images into blob data, manage linking identifiers and other things that make your head hurt, so lets use ArcPy.   Manual steps also preclude creating new geodatabases with the ETL tool, which I want to support.

 

The example Workbench writes a feature class destined to have attachments, and a table that can be used to load them.  You can research the processing, but the key element to inspect is the shutdown script run after completion, see Tool Parameters>Scripting>Shutdown Python Script.

 

Here is the embedded script:

 

 

Now this isn't a lot of code, a few imports, accessing the dictionary available in every Workbench session to get output path values used at run-time, then just two lines calling geoprocessing tools as functions to load the attachments.

 

This is a great way to integrate ArcGIS' powerful Python environment in ETL tools.  Sharp eyed people will notice a scripted parameter in the workspace too, it doesn't use ArcPy so I can't claim it as avoiding low level coding, it was just a way to flexibly create a folder for downloading images at run-time.  There are a number of ways to use Python in Workbench, but I would be detracting from my message here that you can start with the simple and powerful one - use ArcGIS geoprocessing where it saves work.  Enjoy!

Last week was the 2019 Esri Partner Conference followed by Developer Summit, events at which we enjoy being challenged by friends from around the world who are using ArcGIS in their work, and also other apps and formats that ArcGIS does not make or manage.

 

One partner from Europe asked how to use GML (Geography Markup Language) files in ArcGIS Pro.  This format is really a category of formats; the underlying XML - as a markup language intends - can be extended, usually to push a data schema down into the protocol.  He had in mind however what we know as Simple Feature GML, which makes the task - well - simpler, but that isn't critical to this discussion.

 

In ArcMap, Data Interoperability extension may be used to both directly read GML files (of the simple feature profile, recognized from a .gml filename extension, even without licensing Data Interoperability extension) or to make an interoperability connection to any supported GML profile, such as the complex INSPIRE themes.  This workflow is not implemented in Pro, partly because WFS services (which are usually GML "in motion") are the most common use case for GML and are natively supported in Pro, and partly because interoperability connections are being re-imagined for a future release of Pro.

 

In ArcGIS Pro, Data Interoperability extension can also be used to convert GML files just like in ArcMap - with the Quick Import geoprocessing tool, or with a Spatial ETL tool, but the partner thought asking everyone to license an extension would be a hurdle.

 

I decided to blog about an implementation pattern that does what was asked for - convert GML to geodatabase features within ArcGIS Pro - but that can also be used to convert any of the hundreds of formats and connections accessible to ArcGIS Data Interoperability.  The GML data originator has access to ArcGIS Enterprise with Data Interoperability extension so the pattern leverages that, but end users with GML files only need ArcGIS Pro and authenticated access to a geoprocessing service.  You can use this pattern to stand up any format conversion you wish at any scale - but I hasten to add it must be a free or cost-recovery-only service if you make it public.

 

Enough talk, how do we do this?  We're going to use ArcMap and Pro in a double act.  Why both?  At time of writing ArcGIS Pro cannot publish web tools containing Spatial ETL tools, so we'll use ArcMap for that step.

 

In the blog download below you'll find a 10.6.1 version toolbox which contains these tools (click any images to enlarge them):

 

 

GML2FGDB is a Spatial ETL tool that converts one or more files of any schema of simple feature GML to a file geodatabase named GML2FGDB.gdb (with overwrite!).  It looks like this if edited:

If you run it as a tool you can see it has a parameter exposed for GML geometry axis order that defaults to 1,2.  If your data is in Y,X order you can set 2,1.  3D data is supported by the 1,2,3 and 2,1,3 values.

 

 

GML2FGDBModel is a model tool that incorprates the script tool ZipGDB to compress the file geodatabase to zip file.  The compression step is necessary because geoprocessing services do not support workspaces (i.e. geodatabases) as output parameters.

 

 

GML2FGDBModel is shared as a geoprocessing service (make it synchronous) which I called GML2FGDBService:

 

 

Lastly, the model tool GML2FGDBService wraps the service and adds the script tool UnZIPGDB for the round trip from a local GML file or files, to the web service that does the translation without requiring Data Interoperability locally, then finally unzips the scratch ZIP file containing a scratch geodatabase into a user-selected destination directory.

 

 

Now GML2FGDBService can be freely used in Pro:

 

 

GML2FGDBService will always output a file geodatabase named scratch.gdb to your output directory, so be careful not to overwrite prior output!

 

 

Now anyone with access to the model tool and service can convert suitable GML files (or any other data if you refactor the Spatial ETL component) to local geodatabase using ArcGIS Pro.  Enjoy!