Survey123 Tricks of the Trade: Make (previously known as Integromat)
In this blog post I will explore a few techniques to take your use of Survey123 within Make to the next level. I am going to assume that you already know the basics of working with Survey123 and Make. If not, you may want to start with https://community.esri.com/groups/survey123/blog/2019/06/02/getting-started-with-survey123-and-integ...
Many more posts like this could be written, because what you can do with Survey123 and Make deserves an entire book, but we need to start somewhere. So here are a handful of ideas. I hope you discover some new things:
Working with Survey123 attachments
I will start with this one because the topic of working with photo attachments from Survey123 in Make has come up many times before, and it is not entirely obvious how to deal with them. Say your Survey123 smart form includes a photo and a signature question. Can you work with them in Make? Can you include the photo and signature images as attachments in an e-mail? Can you back-up the images in OneDrive or Box? Yes to all of that and here is how.
The trick is using the Make HTTP module, which includes an action called 'Get a file'. This action takes the URL of an image as an input, which you can get from the payload of a Survey123 record.
- Feed the Survey123 'watch survey' module with a smart form that includes one or more photo/signature questions.
- Use the Get a file action in the HTTP module to download your attachment.
The payload of 'Watch a survey' includes a collection of attachmentInfo objects. They are uniquely identified with the name of the question generating the attachment and include a url property that can you use to invoke 'Get a file'.
Once you have the file, you can do a handful of things with it. Of course, you can now include it as an attachment to an email.... That is a classic one!
But you can do more interesting things too. For example:
- Back-up your photos and attachments in Google Drive, Microsoft OneDrive, DropBox.
- Upload images to Flickr, Instagram, Twitter or other social media.
- Reduce the size of an image before storing it.
- Extract metadata from the photo, including its name, size or format.
- Process the photo with Google Cloud Vision to identify elements within the photo or simply extract text from it.
Automating Survey123 custom reports
The Survey123 module includes an action to help you automate Survey123 feature reports. Not familiar with reports in Survey123? We have a video for that here and an introductory blog post too. The Create Feature Report action is pretty much self-explanatory: it takes an input survey, a report template and the objectId of the survey record you want to include in the report. If the trigger of your scenario is a 'Watch a survey' trigger, then you can use again the payload of your Survey123 web hook to extract the objectId. In this case, it is within the Feature-Result object as shown in the next animation.
The use of the HTTP module to download files as described above also applies to the outputs of the Survey123 report action. That is, after you create a custom report file, you can download it and include it as an attachment in an email, or upload it to cloud storage or FTP site. Since many cloud storage solutions include options to automatically synchronize the contents of local and cloud folders, I like to upload the Survey123 report files I create to the cloud so they show up in my computer right away. I know, I am old fashioned, but that feels like magic.
When configuring the Create Feature Report action you will notice that at the bottom of the dialog you can expand a window with advanced settings. In there, you will be able to configure the UTC-Offset. This setting will be your friend if you plan to include dates and times in your reports. Keep in mind that all data within Survey123 -and ArcGIS- is stored in UTC time. Since Make has no real context as from where in the world the report is being invoked, you will need to adjust the UTC offset yourself.
Back to the HTTP module
Earlier, we looked at the 'Get File' action in the HTTP module. This time we will revisit the HTTP module but this time to describe its 'Make a request' action. I personally find this action extremely useful, because it allows your scenarios to interact with pretty much any external web service you want. Using the 'Make a request' action you can for example invoke the ArcGIS REST API to update an attribute in an existing feature, or to simply query a feature service. Lets get down to very specific scenarios:
- Make a point- in-polygon query to an ArcGIS feature service to get the postal code of the submitted Survey123 record location. Then make another request to add the postal code value as an attribute of the submitted feature.
- Use information from a submitted Survey123 record to automatically create a new assignment in a Workforce for ArcGIS project (make a request to add a new feature into the Workforce assignments feature layer).
- After having invoked Google Cloud Vision to categorize the contents of a photo submitted via Survey123, update the attributes of the Survey123 feature accordingly.
- Update an image_url attribute in the submitted survey so you can later create a popup in ArcGIS that shows your image directly when the popup is open.
The mechanics for using the 'Make a request' action imply good knowledge of the target web service your are invoking. I will use next a simple scenario to illustrate the essence of this technique. Say for example we want first to get the location of a submitted survey record, get the current temperature at that location using a weather service and store that temperature value as a GIS attribute of the feature that was originally submitted.
Here is how you get the current temperature. OK: so far we are not using 'Make a request' anywhere. We are just setting this up.
Next, we are going to 'Make a request' to the survey feature layer using the ArcGIS REST API to add the temperature to the temperature field of the submitted record. Specifically, we are going to make a request to the Update Features REST API operation. This operation, as described in the Updates Features ArcGIS REST API Developer documentation, is located at:
https://<featurelayer-url>/updateFeatures
All information needed to construct the URL above is included in the survey web hook payload. It is a matter of combining the surveyInfo.serviceUrl and the feature.layerInfo.id properties, followed by updateFeatures.
Again according to the doc, this operation only supports POST requests and the encoded parameters include:
- f to define the output format as json or html. This is optional but handy so we can debug, if needed, the output within Make.
- token to specify a valid token we can use to securely access the feature layer. This parameter is not needed if the feature layer is shared publicly.
- features to define the actual updates we want to make in the layer. The content of this parameter is a json object with information about the records and fields we want to change.
Lets do f and token first:
The most error prone step is adding the exact JSON value for the features parameter. I like to copy the sample JSON that comes in the developer documentation, and then modify it for my own purposes. It is very important to note that the sample JSON in the documentation must be enclosed with extra brackets as shown in the example below.
Note that the objectId in green, and the temperature in orange are dynamically populated using outputs from other modules in the scenario. Also note the square [] and curly brackets {} I added
[
{
"attributes": {
"OBJECTID": 14,
"temperature_c": If a number do not use quotes,
"city": "If text, use quotes"
}
}
]
You will see yourself going through some trial and error until you get it right, but since you can easily look at the output of the 'Make a request' operation, it typically does not take very long before you figure things out.
Using the 'Make a request' action you can do a lot of things. Updating attributes is one of them, but you can really do many others.
A nice complement to the 'Make a request' action in the HTTP app are the many out of the box tools included in Make to parse XML, JSON, HTML and the many functions to handle text, dates, arrays and much more. They will help tremendously when you need parse responses from the web services you are invoking.
Handling Errors
This is all getting very interesting, but as you create more sophisticated scenarios you will face situations where you need to handle run-time errors. If you are dealing with attachments, what happens if the end-user does not submit one? Or what happens if you hit an external web service and you get a bad output, or a time-out?
By default, when an unexpected error occurs, the execution of the Make scenario is stopped at that point but you can handle errors in other ways. You can for example ignore the error and continue executing, or you can choose to branch off the execution to do different things. This is all described in the Introduction to error handling tutorial in the Integromat Help Center.
The following animation shows how you can add an error handler in case the 'Get file' action fails downloading a non-existing ArcGIS attachment. In this case I chose to simply invoke the Gmail connector to send an e-mail without attachments. If an ArcGIS attachment is found, it gets added to an e-mail, otherwise the email is sent without attachment.
To learn more about error handling in Make, I recommend reading the Advanced error handling help topic.
Connecting Make to an ArcGIS Enterprise instance
By default, the Survey123 modules in Make will connect to ArcGIS Online, but you can actually make Make work against your own instance of ArcGIS Enterprise.
Before we start, it is important to highlight that Make needs to communicate with your own ArcGIS Enterprise instance, and this will not be possible if ArcGIS Enterprise is running completely behind your firewall. You need to make your ArcGIS Enterprise instance is available to Make over the internet. ArcGIS Enterprise administrators are familiar with this concept, which is often resolved by exposing ArcGIS Enterprise through a reverse proxy in the DMZ, although there are other options.
A second step, involves explicitly registering Make as an app in your ArcGIS Enterprise instance. This step is necessary for Make to be able to securely authenticate against your ArcGIS Enterprise. Follow these steps:
- Log into your portal and go to Content.
- Add a new Item of type Application.
- Type: Application
- Title: Make (or any other name you like)
- Tags: Survey123 (Same here, any tag you want)
- Switch to the Settings tab in the item you created in the previous step
- Data source URL: https://www.integromat.com
- Click on the App Registration section and select Update
- Add the following redirect URI's:
- Click on Update to confirm and Save all changes
All the above is necessary for Make to be trusted before you can initiate a secure connection to your Enterprise instance.
Now, it is time to create a new Connection within Make to your ArcGIS Enterprise instance. While adding a new web hook in the Watch a survey or Create a report modules, choose to create a New Connection:
- In the Create a New Connection dialog, expand the advanced settings.
- Add the https URL of your ArcGIS Enterprise instance, including the web adaptor name.
- Populate the AppID and AppSecret properties using the App Registration information from the Application item you created before in your portal.
- Click on continue to authenticate against your ArcGIS Enterprise instance.
You will be able to re-use this connection again and again when creating new web hooks.
Add and Edit Trigger Events
Initially, we added support for Add trigger events, meaning that the Survey123 web and field apps would be able to trigger a web hook when a new record was submitted from a smart form. We are progressively adding support for edit trigger events as well, so you can launch your Make scenarios when a record is updated too.
Support for edit trigger events was added in the Survey123 web app in our https://community.esri.com/groups/survey123/blog/2019/05/24/comet-halley-update-may-23-2019. The Survey123 field app will catch up starting with version 3.5 (scheduled to be released on June 27, 2019).
If you do not now how to update records using the Survey123 web app, check the https://community.esri.com/groups/survey123/blog/2019/05/24/survey123-tricks-of-the-trade-editing-re... blog post.
While you can configure your Make scenarios to be triggered by Add and Edit events, please be aware that the Survey123 field app will not honor this setting until version 3.5. As of 3.4, only the Survey123 web app supports web hook edit triggers.
Below, you can see how you can configure Make to define which triggers you want to activate in your web hook.
If you happen to activate both of them, you can use the output of the Watch survey module output to tell which event triggered your scenario. This is handy in case you want to build a single scenario and add branching logic to handle adds and edits differently.
The trick here is that the eventType property is only included in the payload once there is an actual submission of data. You will actually not see the eventType property pre-populated in the output, until you send data. When working with eventType, I typically put the scenario in listening mode (Run once) and submit data to populate the property. This helps me build the rest of the scenario more easily. Once the scenario is running for good, this property will change accordingly.
Blueprints
Did you know you can export your scenarios as a JSON file and import them later? You can, but only if using a paid subscription to Make.
I included a blueprint of the weather sample as an attachment to this blog so you can play with it. All you need to make it work is a survey with a geopoint question and an additional field where you can store the temperature. Make sure you adjust the name of your temperature field in the features parameter of the 'Make a request' module. By now I am guessing you do not need that I create another gif. I am completely "giffed-out" for today.
As I said at the very beginning, there is so much to write about Survey123 and Make, so this is just a starting point. If you feel there is a particular topic of your interest that should be covered in more detail, do not hesitate asking.
I hope this blog post will give you some ideas to have more fun with Survey123 and Make. I also wish some of you will share some of your creations. Share a screenshot of your scenario and a brief description of what you accomplished with it, or even better the blueprint. Add them as comments below to inspire others!