Latest Contributions by BruceHarold

If you are sharing hosted feature services, optionally with child services like Map Tile, WFS and OGC services, and your source data changes regularly, you will want to automate refreshing service data without breaking item identifier or metadata elements so your customers' maps and apps keep working.  This blog shows how - using ArcGIS Data Interoperability. First the obligatory map of our study area and data, street addresses in Norway: Addresses in Oslo Just to be clear I'll restate the scenario. You are sharing a hosted feature service which may be large You may be sharing services published from the hosted feature service The source data might not be managed in ArcGIS The source data changes regularly and you want to apply the changes to the service(s) You don't want Portal or Online item identifiers to change when you refresh the data You want to automate this maintenance with minimal downtime You don't want to write any code It is well understood that Data Interoperability can detect dataset changes and apply them to a publication copy of the data.  This works well and allows for zero downtime if you write changes incrementally to a feature service.  However, when you are dealing with millions of features this can be time consuming, both to read the original and revised datasets and to write the change transactions.  It also risks encountering network issues during very long transactions. We need an option that just replaces service data efficiently and quickly.  This can be done by maintaining a file geodatabase copy of the source data on your Portal or Online and replacing service data using a truncate and append workflow. Here is the Data Interoperability tool that shows the pattern. Workbench The blue bookmark is whatever ETL you need to get your data into final shape, in my case I'm downloading some data, doing some de-duplication and make a few tweaks to fields. The pale green bookmark is where I write the data to a zipped file geodatabase and overwrite a file geodatabase item in Online.  The tan bookmark is where I truncate my target feature layer. The brighter green bookmark is the final step where I call the append function that reads from the file geodatabase item and writes into the target feature layer. All very simple isn't it! For roughly 2.7M street address features and from my home network the whole job, including waiting for the asynchronous append operation to complete, takes about an hour. But wait there's more!  I published Map Tile, WFS and OGC services from my target feature service, here is everything in my Online project folder: Services   The caches for each child service take a few extra minutes to refresh but the process is automatic (Vector Tile services need a manual cache rebuild from the item settings page). If I was doing this for a production environment I would move processing to an Enterprise server like this earlier blog describes and schedule the task to run overnight at an appropriate interval. There you have it, automated, efficient bulk refresh of hosted feature services and their derived products.  The tool I'm describing is in the blog download, have fun! Note for Enterprise users:  Currently ArcGIS Enterprise does not support file geodatabase (filegdb) as an append format, you must use shapefile (or Excel or CSV if working with tables).  To append shapefiles it is likely you will need to specify a fieldMappings dictionary in the layerMappings parameter to map the field names in your shapefile to the target feature service. ... View more

At the 2021 Esri User Conference a very popular 'track' was ArcGIS Field Maps, if you missed the sessions you can catch up here.  Of course there are rich automation capabilities in products like Workflow Manager and solutions from Esri partners but I'm going to show you how to create your own no-code automation that connects external systems with Field Maps using ArcGIS Data Interoperability.  The scenario I have in mind is you have an external app that emits data events and you need to send field jobs for each (or selected) event(s).  My sample takes 311 service calls coming from a Socrata site that updates hourly, but another source I hear of regularly is Salesforce.  There are a large number of web apps that are in scope for Data Interoperability. First the obligatory screen grabs of live action on my phone showing job receipt and field map response: An incoming job in Microsoft Teams Mobile: Teams.png   Field Maps Mobile opened from the map link:   FieldMap.png   Now I won't repeat the details here but using the pattern I outlined in this preceding post you would set up the automation of job orchestration from your Pro or (preferably) Enterprise environment to run on a schedule. There are two prerequisites needed for the automation: A Field Map with an editable target hosted feature service. A Spatial ETL Tool that processes the incoming events and sends jobs. I recycled the Socrata feed from the post I refer to above for my field map data.  The web map having this layer I configured to refresh on a 1 minute interval as we could be working in near real time in the field.  The ETL tool is in the blog download but here is a screen shot: Workbench The tool reads the 311 feed (you might be reading anything) and synchronizes the state of the data with a hosted feature service.  Then inserted (i.e. new) features each trigger a post to a Microsoft Teams channel webhook (you will have your own business logic on which data events trigger which field map response).  I include an email option too, which is disabled in the image above.  You might use a number of messaging options available in Data Interoperability.  Everything hinges on constructing a map link according to the rules in this topic at the field map links entry. Full disclosure, this is my first excursion into ArcGIS Field Maps (this may be apparent to veterans).  I'll let you read the how-to topic on field map construction and configuration.  However, I think I have shown a simple, no-code way to automate field job orchestration which may help you when working with external systems that can be leveraged using ArcGIS Data Interoperability.   ... View more

Distributed collaboration is a feature of Enterprise and Online that enables the sharing of edits to feature services and file items.  You may be like me and took in a recent session at Esri's User Conference, which got me thinking about expanding the net of environments which can share data updates the same way - namely automatically, on a schedule, between Portals and Online plus many more environments.  Data like these 311 service requests in Dallas, Texas which update hourly: Dallas 311.png The same business drivers discussed in the UC session that indicate using ArcGIS' distributed collaboration may also apply for data which isn't in a Portal or Online, particularly if data isn't shared to or from the ArcGIS system.  You can still implement automated collaboration using ArcGIS Data Interoperability.  You get a no-code experience just like distributed collaboration. I'm going to show a pattern using Data Interoperability extension with processes authored in Pro and using Enterprise as the compute and scheduling resource.  If you have a Pro machine with high uptime you could use just the Pro machine. What is in scope for automated collaboration?  Hundreds of formats and systems.  What data velocity is reasonable to automate?  Like core distributed collaboration, this approach isn't suited for strictly real-time or very large edit-feature counts.  Event driven real-time integrations are better handled with webhooks (see my earlier post) and big data editing should be centralized outright in one organizational group.  My example reaches into a Socrata instance once an hour and synchronizes a few thousand features.  I would be comfortable with a synchronizing a million features once a day. The goal here is sharing data without downtime and giving your audience a persistent connection to use in maps and apps, this translates into portal or Online items that retain their item identifier, with the underlying data being efficiently replaced.  Here is a file geodatabase item plus a hosted feature service item that are both simultaneously refreshed each hour.  How?  Here is the processing workspace: 2021-07-21_8-00-31.jpg  The blog download has this workspace and another that I used to create data for the target items.  I'll walk you through this one (RefreshDallas311).  The stuff on the left culminating in the green bookmark reads the 311 feed from the web and writes a zipped file geodatabase which is then used to overwrite the file geodatabase item.  A star of the show here is the ArcGISOnlineConnector (which also works with Portals). BruceHarold_0-1626879996315.png The other star of the show is on the right, the ChangeDetector, which calculates which features are new, changed, unchanged or deleted and sends them to the feature service writer with an appropriate fme_db_operation format attribute to set the transaction type per record (ObjectIDs are joined for updates and deletes, if the feature service is edit tracked or already in a distributed collaboration you would join GlobalID). BruceHarold_1-1626880065876.png Processing is simple!  Now lets cover how I got the process onto my server and scheduled. I'm not publishing a web tool, I scheduled an executable on the server, namely C:\Program Files\ESRI\Data Interoperability\Data Interoperability AO11\fme.exe with RefreshDallas311.fmw as the argument to the exe.  The process will require any credentials used to be copied to the server and any local paths to be correct.  In this case the workspace also requires a reader (Socrata) and transformer (ArcGISOnlineConnector) be installed from FME Hub. Firstly, as the arcgis service account owner (i.e. a user that exists) on the server I created a folder somewhere for this stuff to live, C:\Users\arcgis\Desktop\DistributedUnlimited.  I created a shortcut to "C:\Program Files\ESRI\Data Interoperability\Data Interoperability AO11\fmeworkbench.exe" on the desktop so I could start Workbench conveniently. Credentials I needed to copy are web connections for my accounts at ArcGIS Online, Socrata and GMail.  To copy credentials you open the Workbench app from the Analysis ribbon in Pro and then the Tools>FME Options control, go to the Web Connections tab and access the grid of available connections, mine are: BruceHarold_0-1626892519427.png Right click on each one needed to be migrated and export to an XML file.  I copied them to the folder created on the server.  Then I copied RefreshDallas311.fmw into the server folder.  I opened Workbench (no workspace) and imported the XML file web connections, manually checking they work.  I then opened RefreshDallas311.fmw, the Socrata and ArcGISOnlineConnector packages auto-installed.  I checked the FeatureWriter local path to the zipped file geodatabase is valid, namely in the server folder.  I ran the workspace, I had to change my Python compatibility to Python 3.7+.  At the top of the log file was: Command-line to run this workspace: "C:\Program Files\ESRI\Data Interoperability\Data Interoperability AO11\fme.exe" C:\Users\arcgis\Desktop\DistributedUnlimited\RefreshDallas311.fmw Now, for the arcgis user, I created a scheduled task  using the command and argument, running hourly. BruceHarold_0-1626895957319.png   I'm done, I have automated a distributed, unlimited collaboration! The blog download has my ETL tool (Pro 2.8).  My server is Enterprise 10.9.  Don't forget you need ArcGIS Data Interoperability installed and licensed at both ends! Thanks to Dallas, TX.         ... View more

ArcGIS Online hosted feature services are foundational.  If changes in feature service data require a response you don't have to step outside the ArcGIS system to automate it.  This blog is about using feature service webhooks to trigger geoprocessing service jobs that operate on the change set to do anything you want. It is conceptually simple, here is the Model in the blog download that defines my geoprocessing service: BruceHarold_0-1625765863982.png The geoprocessing service (also known as a 'web tool') receives the webhook JSON payload, extracts the changesUrl object, then gets and operates on the change data using a Spatial ETL tool built with ArcGIS Data Interoperability.  You could use a Python script tool too, but then you'll have to write code, and Data Interoperability is no-code technology, so start from here if you want to get going quickly 😉. I used an 'old friend' data source for my feature service, public transport vehicle positions in Auckland, New Zealand, available on a 30 second interval:   RushHour.png   The data is quite 'hot' - no shortage of trigger events! - which was handy for writing this blog.  Your data might not be so busy. Anyway, I initially defined a webhook on the feature service that sent the payload to webhook.site so I could see what a payload looks like.  Here is one:       [ { "name": "WebhookSite", "layerId": 0, "orgId": "FQD0rKU8X5sAQfh8", "serviceName": "VehiclePosition", "lastUpdatedTime": 1625684942414, "changesUrl": "https%3a%2f%2fservices.arcgis.com%2fFQD0rKU8X5sAQfh8%2fArcGIS%2frest%2fservices%2fVehiclePosition%2fFeatureServer%2fextractChanges%3fserverGens%3d%5b2322232%2c2322500%5d%26async%3dtrue%26returnDeletes%3dfalse%26returnAttachments%3dfalse", "events": [ "FeaturesCreated", "FeaturesUpdated" ] } ]       You'll see in my Model I used a Calculate Value model tool to extract and decode the percent-encoded changesUrl object.  In theory this could easily be done in the Spatial ETL tool and I wouldn't need the Model at all, the Spatial ETL tool could be published standalone, but I ran into issues so fell back to a Python snippet (i.e. Calculate Value).  I'm breaking my no-code paradigm story aren't I, but its the only code in this show and you don't have to write it for yourselves now! I'll get to the saga of payload processing but first a word on the form of webhook you'll need for your feature service.  Here is mine: BruceHarold_0-1625768822289.png Firstly notice the HookURL goes via HTTP.  ArcGIS Online feature services are hosted on arcgis.com while my geoprocessing service is on amazonaws.com.  I'm not an IT administrator and didn't know how to set up a certificate that would support HTTPS trust between the domains, so I enabled HTTP on my server.  Payloads contain no credentials and all downstream processing uses HTTPS to access change data, if you have questions (or advice!) then please comment in the blog. Secondly notice the Content Type is application/x-www-form-urlencoded.  If you use application/json your geoprocessing service will not receive the payload. Now about processing, here is the ProcessPayload tool: BruceHarold_0-1625770748402.png Webhook change data is accessed (by default) in a secure fashion that is asynchronous.  The changesUrl is used to start an extraction job and returns a statusUrl.  The statusUrl is used to request a resultsUrl which can be used (when it has a value) to return a response body that contains the change data.  This requires looping, which is done in the ArcGISOnlineWebhookDataGetter custom transformer (available on FME Hub).  This transformer will download as a linked transformer by default, make sure you embed it into your workspace (a right click option).  I edited my copy to check for job completion on 10 second intervals and give up after 10 retries, my data usually arrived on the second retry so that worked for me. The change data comes out of the ArcGISOnlineWebhookDataGetter as a big ugly JSON object which is hard to unpack, so I avoid the whole issue by reading the change data directly out of the source feature service using queries on ObjectID I get from the change data response.  Sneaky, lazy, but effective, and best of all you can recycle this approach for any feature service! When the change data is read you can perform whatever integration you want to do with it.  In my case I just write OGC GeoPackages of the Adds and Updates features and email them to myself, so not a real integration, but you get the idea. For the record, the 'rules' for publishing Spatial ETL tools as geoprocessing services are that the FME workspace must be entirely embedded, you must export any web connection or database connection credentials used by the tools to the arcgis service owner's account on each processing server (get an XML export file by right clicking on the connection in the FME Options control, as the arcgis user on the server start Workbench from fmeworkbench.exe (AO11 version for portal machines)  then import the XML file) and most importantly before you start make sure Data Interoperability is installed and licensed! For my purposes I used a server that was not federated to a portal, but you can go either way.  All this was created in Pro 2.8.1 and Enterprise 10.9. The blog download has my web tool and Model, let me know how you get on out there.  Have fun! ... View more