Latest Contributions by BruceHarold

Data Interoperability integrates hundreds of data formats into ArcGIS and an old favorite workflow in ArcMap and ArcCatalog (personal favorite right there) was to create Interoperability Connections in the catalog tree and use them in maps and geoprocessing tools.  The default behavior for an interoperability connection is to refresh once per session, this saves a cache of the data to disk. Here is how a connection looks:   This has been a popular workflow, but in Pro there are technical challenges to delivering the same experience.  In Pro 2.6 we added the Data Inspector app to the Data Interop control in the Analysis ribbon, which lets you explore data and optionally export all or some of it to another format, like the home geodatabase for your project.  That is good but Data Inspector is a separate process so the data isn't immediately useful in your project. It turns out a simple way to bring interoperability data sources into a project has been sitting under our noses for a while - creating feature classes and tables in the memory workspace and automatically add them to the active map or scene. The blog download has a toolbox with three model tools in it: Here is Interoperability Connection in edit mode, you can see it includes Interoperability Features and Interoperability Tables as submodels. To bring interoperability data into your map or scene simply run Interoperability Connection as a tool, the submodels will recursively search for feature classes and tables in the output file geodatabase created by the system tool Quick Import and copy them into the memory workspace and add them to your table of contents. Quick Import in the model creates a file geodatabase named InteroperabilityConnection.gdb as part of the process.  I make this intermediate data (and hence be automatically deleted).  Make sure your geoprocessing settings allow for overwrite of existing datasets by checking the setting in Options. Of course memory workspace data will not persist between sessions, but you can easily copy it to a project database, and during a session the data will be very performant. You might like to share a project template with this toolbox in it with your colleagues.  Naturally Data Interoperability extension must be installed and licensed for now, but we have some news coming in a later release about another equivalency issue from ArcMap that might allow some exceptions. To give you confidence, here is a screen shot of a million XY events coming from a Snowflake instance: But wait there's more!  A powerful function in Data Interoperability is to use the web as a file system, not just FTP and HTTP but cloud storage platforms.  For example I uploaded a folder of thousands of Mapinfo TAB files to OneDrive, and using the file browser can choose:   Then in my OneDrive connection select a TAB file:   Have fun interoperating! P.S.  Thanks to my colleague Emily K. for test help. Edited December 23rd 2020 to replace the download - the models in the toolbox had a fieldmap with persisting entries.   ... View more

At the ArcGIS Pro 2.7 release in December the FME engine in Data Interoperability extension is FME 2020.2.  The corresponding release of Data Interoperability for ArcGIS Enterprise with the same engine is Enterprise 10.9 pre-release and available simultaneously with Pro 2.7, but if your organization cannot implement pre-release software and you will be using Enterprise 10.8.1 (which uses FME 2020.0.1) until 10.9 ships then Spatial ETL web tools or Workbenches run on a server may fail if either of these conditions is met: The tool uses a format that is not supported by Enterprise 10.8.1/FME 2020.0.1 The tool uses a transformer which has a new version number at FME 2020.2. There is no workaround for the first situation and the affected formats are: Microsoft Direct Draw CityJSON LDAP/Active Directory Mapbox Vector Tile Mapbox Vector Tileset Mapbox MBTiles Vector Tiles Planet Basemaps Snowflake Spatial 12D Model There is a workaround for the second situation, namely to substitute transformers from the FME 2020.0.1 engine when authoring your Pro 2.7 Spatial ETL workspace.  These are available in the blog download, simply open the containing tool by editing it and copy/paste the transformers you want into your workspace.  The affected transformers are: FeatureReader JSONExtractor JSONFlattener JSONFormatter JSONFragmenter JSONUpdater JSONValidator LocalCoordinateSystemSetter SQLExecutor StatisticsCalculator TCLCaller We apologize for this temporary situation which will be resolved when ArcGIS Enterprise 10.9 ships. ... View more

Here is where it rained today in Australia, a bit around Perth, a bunch in Queensland and for the sharp-eyed a little at Cape York to keep the prawns and crocs happy.  The legend shows mm precipitation. The Australian Bureau of Meteorology publishes downloadable weather data; my scenario is I'm interested in republishing rain gauge observations to a hosted feature service, which is in the map above.  After a little digging around I find the data is available via FTP with a schema described in this user guide.  The data is refreshed daily.  While I'm not redistributing the data in this blog I'll mention it is licensed under Creative Commons terms so you can implement my sample if you wish. While the refresh rate is daily each file can contain observations spanning more than 24 hours and from multiple sensors at a site.  Anyway, what I wanted was the daily observations pushed into a feature service in my portal; I could just as easily send the data to ArcGIS Online. These periodic synchronizations from the web are everywhere in GIS.  Data custodians make it easy to manually obtain data, I'm going to show you how simple it is to automate synchronizations with Data Interoperability.  I usually describe Data Interoperability as Esri's 'no-code' app integration technology.  Full disclosure, in this sample I did resort to some Python in the FME Workbenches I created, so I have to back off the no-code claim, but I can say its low-code.  You can see for yourself in the Workbenches. I started out thinking I would make the scheduled process a web tool and schedule it with Notebook Server.  That might be the most fun to build but I realized it just isn't called for in my use case.  I fell back to a pattern I previously blogged about, namely using Windows Task Scheduler, but this time on a server.  Why not use the desktop software approach?  Well, to take advantage of a machine with likely very high uptime I know can be scheduled outside my normal working hours. Here is the Workbench that does the job of downloading the BoM product files and sending features to my portal's hosted feature service: And I can't resist it, here is the Python, not too scary.  It would be unnecessary if the filenames at the FTP site were stable, I could have used an FTPCaller transformer, but they have datestamps as part of their name, it was just easier to handle that with some Python.  While I was downloading the data I also cleaned it up a little (removing enclosing double quotes and newline characters) then sent all observations out into the stream. As the data sources don't change I made all parameters private, this simplifies the command to schedule.  All that I need to do is get the Workbench onto the server and make sure it runs.  In the post download you'll find three FMW files: MakeRainGauges.fmw RefreshRainGauges.fmw RefreshRainGauges - Server Copy.fmw MakeRainGauges creates a geodatabase feature class I used in Pro to instantiate my hosted feature service.  RefreshRainGauges is built from MakeRainGauges and only differs in that it writes to the feature service, with initial truncation.  That's the Workbench I want to schedule.  RefreshRainGauges - Server Copy only differs from RefreshRainGauges in its Python settings, to use Python 3.6+.  I didn't use that name on the server, just to get it into the post download. On my portal server there was a little setup (I have a single machine with everything on on it, don't forget to install and license Data Interoperability!).  RefreshRainGauges uses a web connection to my portal.  In this blog I describe how to create a portal web connection.  This has to be copied to the server for the arcgis user which will run the scheduled process.  The simplest way is method #2 in this article.  Logged onto the server as arcgis, I first created a desktop shortcut to "C:\Program Files\ESRI\Data Interoperability\Data Interoperability AO11\fmeworkbench.exe", started Workbench, then imported the web connection XML file and tested reading the feature service.  I also copied RefreshRainGauges to a folder and edited it to adjust the Python environment to suit the server (the sample was built with Pro 2.7 Beta but the server is running Enterprise 10.8.1).  Running the workspace interactively, the top of the log reveals the command to be scheduled: Command-line to run this workspace: "C:\Program Files\ESRI\Data Interoperability\Data Interoperability AO11\fme.exe" C:\Users\arcgis\Desktop\RefreshRainGauges.fmw The rest is simple, just create a basic task, the hardest part was figuring out what time to execute the command (I see data changing as late as 1AM UTC so I went with 6PM local time on my server, which is in US West).  Make sure the task will run if arcgis is not logged on, and the arcgis user will need batch job rights (or be an administrator, which I can do on my VM machine but you likely will not be allowed to do). That's it, automated maintenance of data I can share to anyone!  I finish with a screen shot of the processing log from last night's synchronization: ... View more