Really Code Free Web Integration, With a Touch of ArcGIS REST API
This article is a refresh of a previous blog by my colleague @BruceHarold published a while back. While the workflow and data sources are the same, this solution takes away any code previously added, due to the power of the Workbench’s transformer called the HTTPCaller and the ArcGIS REST API.
With a few changes to the original solution, we will build a web to web integration solution with ArcGIS Online.
I’ll also highlight a few improvements of Data Interoperability since this previous blog was written five years ago. With this blog comes a newer version of the working sample that you can download from the link below.
Can you Tell the Difference?
We replaced the destination from file geodatabase to ArcGIS Online. Using the Compare Workspace feature you can see the visual changes on the canvas to help you modify an existing workspace. The Read Only workspace on the right is the original and the new Editable one is the new workspace. On the Editable Navigator window, cyan colored objects are new objects, yellow are modified.
The magenta colored Observations feature class destination in the original workspace is now changed to a blue colored AGOL feature layer in the new one.
We added two transformers and removed any Python scripts that were part of the original sample.
A FeatureMerger was added to replace the Match table output and this Match Table feature type is replaced by the HTTPCaller.
The python code that add the attachments is now replaced by a configuration of the HTTPCaller calling the AddAttachment rest endpoint of the ArcGIS Server feature service.
Start with a Web Connection
You need to this only once in your Tools ---> FME Options menu. For reference on how to build a web connection please see this article. This article shows how to build a web connection for ArcGIS Enterprise, however, the steps are the same for ArcGIS Online. Once you have the web connection, let's get into the weeds.
Dump the Code and Learn to Configure the HTTPCaller to Add Attachments
Yes, ditch the python scripts. In order to reuse the sample to work with your own AGOL instance, you need to understand how to configure the HTTPCaller, changing those parameters marked by the red arrows, shown below.
Here’s the request URL: https://<root>/<serviceName>/FeatureServer/<layerId>/<featureId>/addAttachment
Below is what’s configured on the Request URL for this downloaded sample:
https://services7.arcgis.com/eeZcUziwi5hzGEGs/ArcGIS/rest/services/r2r/FeatureServer/0/@Value(ID)/AddAttachment
The <featureid> parameter @Value(id) is dynamic but the rest is hardcoded. The question is, where do you get the <root> URL from?
The trick is to go to the AGOL feature item page, click on the Observations feature layer under Layers to open the Observations feature layer item page and scroll down until you see the URL section on the right pane below Details.
When the ArcGIS REST Services Directory page open, scroll to the top and on the upper left corner of the page, right click on Observations layer and click Copy Link Address, if you use Chrome as your browser.
This will give you the <root> URL plus the token to replace what’s configured on the sample HTTPCaller. Since you will build a web connection, remove the token in this copied URL. Paste this URL in the Request URL of the transformer.
To complete configuration of HTTPCaller, there is a hidden parameter in the Multipart Upload section of the HTTPCaller, shown below. This is not in the fine print of the API, but briefly mentioned in the HTTPCaller help page.
In the attachment row, where Upload Type is File Upload, you need to configure the mime-type of the attachment that is exposed when you click the 3-dot menu from the Value cell of the row. This hidden parameter is defaulted as application/octet-stream and if it does not match the file it will cause AddAttachment to fail.
Notice the mime-type value I configured for the MIME-Type parameter is actually a dynamic attribute created in the data stream. This is useful when you have multiple types of attachments (e.g. pdf, jpg, png, and others) to load. So where is this attribute created? Look for it in the AttributeManager before the HTTPCaller. I used the path_extension and renamed it to mime-type and assigned a conditional value based on the extension.
Be sure that you know what extensions you are uploading and their mime-types. For a list of supported file types in AGOL, please check out this link.
Do I need to upgrade my Upgradeable Transformers
Another new feature available is the Upgrade All Transformers feature. The previous version of this feature was to upgrade one transformer at a time. Notice in the Navigator pane below that there are 21 transformers that need updating. With one click of the button and going through some prompts your workspace is upgraded. You can also opt to Backup your workspae or not.
To run this new solution successfully it is not necessary to upgrade these transformers, but it is always good practice to do so. The final sample below will have all these transformers upgraded to the latest and greatest.
Running the workspace
The solution is intentionally designed to run two bookmarks separately. Before you run any bookmark make sure that you Enable cache.
Bookmark One creates the Observations feature service and write its features. You run a bookmark by clicking and hovering on the top of the bookmark until small icons appear on the top. Click on the Run Just Contained play button.
Before we run Bookmark Two, we need to go back to the newly created feature layer in AGOL and Click on the Enable Attachment slider to set it to True. You need to do this because the Enable attachment parameter of the Observations feature layer need to be set to True before we run Bookmark Two. Since the tool creates the feature service in the Bookmark One, Enable attachment will be set to False by default.
With attachments enabled, we can run Bookmark two to add attachments related to the Observations feature layer.
Note, that the next time you run the solution and the Feature Type Handling is set to Truncate Existing, Enable Attachment would still be True.
In the original solution Enable Attachment is executed with the Python shutdown script. You can modify this solution to include automating Enable Attachment by adding the UpdateDefinition rest endpoint, but I'd like to keep this solution simple and deliberate.
I’ll leave the enhancement up to you. If scheduling and a single run fits into your workflow, you'll probably need the enhancement.
Hint: If you want to do the complete automation, the picture below will be a helpful start.
Adding Enable Attachment to your solution
Visualize your work
My app of choice to view the results is the Instant App template Attachment Viewer. Just like magic all my attachments appear on an interactive map and photo viewer.
Final word
Before you start crawling your network shares and hard drives for pictures, please note that attachments have storage costs in AGOL and I would be remiss if I don't inform you of the costs by doing so. Here's the guide to understanding AGOL credits.
While there are established patterns for automation and integration of data with attachments, every solution needs to consider several factors. I have detailed some of these factors in this article. For geodatabases, the original solution with the Python Shutdown script is still be the best approach. The AGOL solution is simpler and works well for visualizing data with attachments. So pick your spice and pick it well.
That's it, you can start crawling your drives and load your attachments to AGOL.