Latest Contributions by BruceHarold

This post is about leveraging ArcGIS Enterprise 10.8.1 with ArcGIS Data Interoperability extension to share Spatial ETL web tools within your organization.  The 'advanced' aspect is we'll be moving data using a web connection to a web platform enabled by an FME package, plus we'll throw in a webhook notification step for fun.   In my on-demand technical workshop for UC 2020 I demonstrated how to create a web tool that performs a format conversion by web service, a classic use case for Data Interoperability combined with the sharing power of ArcGIS Enterprise.  Data Interoperability however has moved on from being a format foundry to being a generic solution for no-code integration across the web, so in this post I will show how to build a web tool that takes non-native data through to a web platform.  I happen to be reading Microsoft Access table XY event data and writing to ArcGIS Online hosted features, but read between the lines and understand the data could go to any cloud platform or app equally easily.  In some ways moving data to the web is a better fit for Spatial ETL tools as FME readers and writers work with containers or workspaces, for example a geodatabase, which is not a valid geoprocessing service output parameter that can be returned to a client.  It can be done, for example by zipping file geodatabase outputs, but if you have a web platform available, then use it, data is more useful on the web than as local files.   Web tools are a new name for geoprocessing services.  The original workflow for publishing geoprocessing services using a history item and an administrative connection to a standalone server is still available in ArcGIS Pro 2.6, but I'll show the new paradigm of sharing a web tool to a portal.  You share a Spatial ETL tool as a web tool from a history item just like any other tool but with two special considerations:   The Spatial ETL source FME workspace must be embedded, and not external to the tool Input parameters should be defined as simple types like Filename (Existing), Text or Choice   The first condition helps with packaging the tool at publication time, the second helps keep the final published tool parameter behavior simple.  You may have to refactor your ETL tool at authoring time to achieve these simplifications.  For example here I'm showing before and after versions of a Microsoft Access input parameter.  The default configuration allows picking multiple files with a large choice of file type extensions, my desired behavior is a single file with only .mdb and .accdb extensions.   Default Access Database Reader Parameter   Modified Access Database Reader Parameter     Conceptually and graphically my ETL tool is simple.  My scenario is I'm working for Fako Mining who do borehole planning with an Access app and record borehole locations as latitude & longitude in a table.  Click to expand the screen capture below, or edit the tool extracted from the blog attachment.  The data flow is:   Read an Access database table that has XY value fields Write (or replace) a hosted point feature layer in ArcGIS Online Trigger a webhook that notifies a Microsoft Teams channel the new data is available   There are three bookmarks in the workspace, in the blue one any existing feature service is deleted (I'm creating services named after the processing date), in the green one the Access data is read and converted to a point feature class in a zipped file geodatabase, and in the tan one the file geodatabase is published and shared to a group and lastly the webhook sent.  The concept is Online members are members of a Teams channel and will be notified the data is available.  While you're inspecting the workspace have a think about how much code this would take if you scripted it...     The key function in this is the ArcGISOnlineConnector transformer that does the work of publishing and sharing the zipped file geodatabase.  It is an FME Hub download and needs to be provisioned on the server, plus it uses a web connection that also needs to be provisioned on the server.   First, set up your server.  Install and license ArcGIS Data Interoperability for Server, make sure your server is federated to your portal, generate and import certificates for portal and server and so on.  Do not install ArcGIS Data Interoperability for Pro on your server, you will get license and tool failures   Once you have Data Interoperability installed, log onto the server as the ArcGIS Service local account user (by default named 'arcgis') and confirm you can open the Workbench app at this path:   C:\Program Files\ESRI\Data Interoperability\Data Interoperability AO11\fmeworkbench.exe   While logged onto the server as arcgis, create a share with the below path and give full control to the domain user (presumably you) who will be publishing the web tool and owns the web connection used in the ArcGISOnlineConnector.   \\<machinename>\Users\arcgis\AppData\Roaming\Safe Software\FME   From the authoring machine and as the publishing author, open the share.  Here is how it looks on my laptop:     Never mind the files in your share are different to mine, they will soon align.   Now we're going to provision the FME Package for ArcGISOnlineConnector and the web connection used by it to the server.  Download the current release of the package to the share.  Still from the authoring machine, open Workbench and then the  FME Tools dialog Default Paths view.  Note (write down!) the profile directory for Data Path, it is the folder where web connections are stored.  You will temporarily change it next.     One way web and database connections are shared is the owner copies the files fme_connections.data and fme_publicKey.jceks from the Data Path to a shared folder (in our case the share above, which is also the default location where the arcgis user will store web connections), redirects the Data Path to use the new location, opens the Web Connections view of FME Tools and selects web connections to be made public.  Having done that revert the Data Path to the original default for the authoring user.  The arcgis server user must have write access to the copied data.  If your server is not on your network it may be easier to use the export/import methodology for credential sharing.   Now as the arcgis user logged onto the server, open Workbench from the path given above and drag the file safe.esri-agol-1.1.4.fpkg (the version may change) from the share at C:\Users\arcgis\AppData\Roaming\Safe Software\FME into the canvas to install it.  Check FME Tools>Web Connections sees the web connection to ArcGIS Online.  Create a test workspace that makes a web connection and uses an ArcGISOnlineConnector to list the contents of a contents folder - the blog download has an example TestWebConnection.fmw.  If everything works the server is ready for action!   Run your Spatial ETL tool as a tool so that a history item is created.  Then right click on the history item and share it as a web tool.  Here is my experience:     Allow uploads, make it asynchronous, set the Info message level, this will display workspace processing messages.     Don't forget to edit the tool parameter(s) - click the pen icon.  You may have to do this twice to make it 'stick'.     It will analyze with a warning, this is normal for Spatial ETL tools.     After the web tool publishes, open the Catalog pane at the portal contents and run your tool.  Mine throws a warning to do with some coordinate system handling, but I'll take the win.  By design there are no features displaying as output as there is no writer in the workspace, my data went to ArcGIS Online courtesy of the ArcGISOnlineConnector.  The log shows intermediate processing worked as expected.     Teams shows me the webhook triggered, which is the last part of the web tool processing, and it gives me a service URL so something is there....  Take a look at what is in the HTTPCaller transformer to see how I got an image into the webhook payload (its a public image item in Online).     Bingo, my data loads into my map!     So that is the round trip, I have a web tool anyone in my organization can use to publish hosted feature layers from an Access database table, with the added touch of automated notifications. ... View more

The 2020 Esri User Conference has many sessions on the ArcGIS Open Platform. We want to make it easy for you to work successfully in a heterogeneous environment. To create an open and interoperable system, Esri has adopted a multifaceted approach, including support of: standards; direct integrations with non-GIS technology; direct read & write of hundreds of data formats; open developers’ tools; ETL tools; metadata; open source; open data sharing; and SDI. Please visit us at the Open Platform: Standards and Interoperability Esri Expo Area Join Us in our Chat Room! or Schedule a Meeting   Links will require conference registration until made public.  Not registered? Click here to register for the Esri UC 2020, July 13–16, 2020 | The world’s largest, virtual GIS event.   Check out these sessions at the Virtual UC: Live Session  Live Session with Q&A! Tue 2:50 pm – 3:50 pm PDT ArcGIS: An Open Platform UC Expo Technical Workshops on-demand ArcGIS Data Interoperability Extension: An Introduction on-demand Understanding and Using Metadata in ArcGIS on-demand Implementing Next-Generation SDI with ArcGIS on-demand Esri Geoportal Server: An Introduction and What's New on-demand Enterprise System Integrations with ArcGIS: Best Practices UC Expo Demo Theater Presentations on-demand ArcGIS for INSPIRE: An Introduction on-demand Working with OGC WMS and WMTS Services on-demand Working with OGC WFS Services on-demand Working with OGC WCS Services on-demand Maritime Standards - Support in Esri's Nautical Products on-demand Esri and Open Source To learn more about Esri’s Open Platform approach and our next generation SDI work visit: Stay Connected on GeoNet Open Platform: Standards and Interoperability Metadata Spatial Data Infrastructure (SDI) Collaboratives Data Interoperability Product page ArcGIS for INSPIRE product page Ask Geoportal product page and get it on GitHub Other Resources Esri's Open Vision homepage Open Platform Blog Esri Support for Open Geospatial Standards (white paper) Latest OGC Compliancy Certificates All Esri OGC Implementations How to work with GeoPackages in ArcGIS Pro Esri UC 2020 Q&A The Esri UC 2020 Q&A is chalk full of what you want to know. To learn more details on our Open Platform approach, start with ‘Components of ArcGIS – Open/Standards/Interoperability/ Metadata’.   User Conference What are the big ideas I should explore before and at this year’s User Conference? Esri What is geospatial infrastructure? What is Esri’s role in geospatial infrastructure? What is Esri’s geospatial cloud strategy? How is Esri supporting open government initiatives? Open/Standards/Interoperability/Metadata What are the different facets of Esri’s Open Platform approach? What is Esri’s approach to enterprise integration? Can you provide information on Esri’s participation in standards-related initiatives? What areas will see greater product support in the coming year? How is Esri supporting the vision of Spatial Data Infrastructure? How does Esri address the needs of SDI for the INSPIRE Directive? What plans do you have for improved support for metadata and search across the platform? What is a GeoPackage and what is the level of support for GeoPackages within the ArcGIS ecosystem? ArcGIS What security and privacy standards do Esri products comply with today? What steps is Esri taking towards meeting accessibility (WCAG, 508 compliance) requirements? Components of ArcGIS – ArcGIS Pro Will ArcGIS Pro include an Export to Adobe Illustrator? Components of ArcGIS – ArcGIS Enterprise What is ArcGIS Enterprise on Kubernetes? Components of ArcGIS – ArcGIS for Developers What are my options if I want to extend the ArcGIS Platform? What is Esri doing to support the open source community? Capabilities of ArcGIS – Spatial Analysis and Data Science How does my data integrate with GeoAnalytics Server? What’s new with R integration and ArcGIS? How are Conda functionalities exposed and used in ArcGIS? What are Esri’s plans for an interactive Coordinate Conversion tool? Capabilities of ArcGIS – Imagery and Remote Sensing Does ArcGIS support all imagery and raster formats? Can ArcGIS handle scientific and multidimensional datasets?  What is Esri’s strategy for integrating lidar into ArcGIS? Capabilities of ArcGIS – 3D Visualization and Analytics What is I3S? Why is the I3S standard important for 3D? How is KML being support in Esri apps? Capabilities of ArcGIS – Data Management What kind of data does the ArcGIS platform work with? Capabilities of ArcGIS – AEC and Asset Management? Is the parcel fabric LADM compliant? What versions of CAD and BIM data are supported in ArcGIS?   Prepared with the assistance of my colleagues jfoust-esristaff and jsaligoe-esristaff‌ ... View more

Data Interoperability (or FME) users whose organizations secure access to ArcGIS Online using Okta need to do a little work to make FME Workbench web connections to their Online organization. Okta is one of many identity providers available for securing ArcGIS Online sign-on.  You can check Okta and other options at this site.  If you attempt the usual web connection creation experience however you'll get rejected at the very last step with an error from OAuth2, this post is about making it work. The first step is to make an App item to hang the connection off.  In a browser, go to your org's home app and log in.  Lets say my home is https://geo-demo.maps.arcgis.com/home/index.html. Authenticate using the Okta option, then go to your Content pane and add an App item. Choose 'An application'. Make sure you select Application app type and give it a name and some tags: Then go to the item Settings tab, scroll down to the Registered Info details and choose to Update the details, and add a Redirect URI with the value http://localhost.  Your details will look like this (App ID will differ): Copy the App ID value and App Secret values into a text editor.  You need to Show Secret to expose the value. Now start Workbench from the Analysis ribbon in ArcGIS Pro and under the Tools menu select FME Options and activate Web Connections: Click on Manage Services at bottom right and when the dialog appears click the small pull-down in the Add Web Service control and choose Create From > Esri ArcGIS Online. Give the service a meaningful name and description, enter the App ID retrieved earlier into the Client ID value and enter the App Secret retrieved earlier into the Client Secret.  Replace the Redirect URI value with http://localhost. Use the Test control to bring up the authentication dialog and choose the Okta option: My org uses multi-factor authentication so I am invited to send a push (you may not see this): Then save your changes and Close the Manage Web Services dialog: Now back in the FME Options > Web Connections dialog click the Add Connection control (the + sign bottom left)... ...and your shiny new Okta authenticated web service option for ArcGIS Online is available to make a connection: Scroll down to the service name, authenticate and you're done! Here is mine: ... View more

In-app updates support incremental functionality delivery during a software release.  ArcGIS Data Interoperability inherits FME's ability to install FME packages for this purpose, this blog shows you how simple this is for Desktop and Server installations at the Pro 2.6 and Enterprise 10.8.1 releases. FME Hub is the default source for packages.  Workbench supports browsing Hub, or you can use a web browser.  In the screen grab below I have gone to the home page for a package that will provide support for reading and writing Socrata portal technology data. Lets install the package in Data Interoperability for ArcGIS Pro first. Download the package to your machine.  You'll get a file with the extension .fpkg. To install the package, open a session of Workbench from the Analysis ribbon and simply drag the fpkg file from File Explorer into the canvas.  You'll get a warning: Then a progress dialog: Then the Workbench log window will show success: Packages from FME Hub are maintained and therefore have versions.  To check if a new version exists, open the FME Packages view of the FME Options dialog and see if the Update button is enabled. Note the package installs into a user profile directory.  At present FME packages cannot be installed at a location shared by multiple users, each user must install the package(s) they require. There is also a command-line option for listing, installing and uninstalling packages using the fme.exe executable: So that is the desktop experience.  What about Data Interoperability for Server?  If you want to share web ETL tools that use packages then the package(s) need to be on the server(s). At the Enterprise 10.8.1 release there are two folders where Data Interoperability is installed, one for web tools published from ArcGIS Desktop 10.x and one for web tools published from ArcGIS Pro 2.6: Web Tool Publishing Environment Data Interoperability FME_HOME on the server Desktop 10.x C:\Program Files\ESRI\Data Interoperability Pro 2.6 C:\Program Files\ESRI\Data Interoperability\Data Interoperability AO11 To successfully share web tools leveraging packages, each package must be installed into the target environment by the server account user on each server machine.  Otherwise the experience is the same as the command-line option for desktop machines.  Log into each server as the server account user, change directory to the appropriate path from the table above, then install each package.  Here is an example installing the Socrata package for the Pro web tool case (apologies for the image being a little smaller). So that's it, you can install and manage FME packages for ArcGIS Data Interoperability! ... View more