Automating Service Applications with Integromat and Dashboards

4221
12
05-15-2020 05:57 AM
TylerBragg2
Occasional Contributor II
7 12 4,221

 

I am writing this blog post because I wanted to document a useful workflow that I developed for using Integromat to automate the creation of Feature Reports for Survey123 after editing the survey data.  As someone who had never used Integromat before, I spent a great deal of time researching the different intricacies of the Integromat modules and how to make them work for what I needed, as I couldn't find any examples that were quite the same as mine.  A lot of this workflow was accomplished by trial and error, so I'm posting this here in the event that someone else may find this example useful.  My particular difficulty in this project came when I attempted to process the automation differently when the survey was initially submitted (addData event type) vs. when it was edited (editData event type.)

 

My scenario is this:

 

I work in local government (city level) and we have continually been making services and applications available online as a convenience to our citizens.  As such, one item on our agenda was to make our Application for New Sewer Service available on the internet.  I created a survey in Survey123 for the purpose of allowing citizens to apply to our Sanitary Board for new sewer service (when you move to a new house and need to put the service in your name, or when you build a new house and need brand new service).  I also needed an easy way for our Sanitary Board staff to process these requests.  Creating an automated workflow with Integromat seemed like it would provide the perfect solution for automation, and paired with the power of Dashboards for ArcGIS, I think it did!

 

The Simple Workflow:

 

1) Citizen submits an application for new sewer service.

2) Sanitary Board staff receives the request.

3) Sanitary Board staff processes the request (they need to add some information to the form, such as new account number, deposit cost, date paid, etc.)

4) Microsoft Word application form is filled out with applicant’s information and filed in archive.

 

The Solution:

 

Workflow Item # 1 – Submitting the Application

 

To allow citizens to submit an application, I created a new survey in Survey123 Connect with all pertinent information included and published to our ArcGIS Online Organization.  This was the easy part.

 

Workflow Item # 2 – Receiving the Application

 

In order for the staff to access the information, I was going to train them on how to use the Survey123 online interface.  The problem with doing this is that they also need to have the ability to edit the data, which at the present time is only available to the survey owner or the individual survey submitter.  This was not going to work.  This is where I found out about embedding surveys in a dashboard https://community.esri.com/groups/survey123/blog/2020/04/27/survey123-tricks-of-the-trade-embedding-....  Even more importantly, embedding that survey into a dashboard in edit mode using URL parameters https://community.esri.com/groups/survey123/blog/2019/02/06/survey123-tricks-of-the-trade-web-form-u... (both great blog posts by Ismael Chivite).   This gave our staff the ability to go into the survey record and add data to the “office use only” fields (or change any other field that may have been entered incorrectly in the initial survey), including account number, deposit cost, date paid, etc.  Below is a screenshot of the dashboard with the editable survey form included:

 

 

I also wanted the staff to receive an email when a new record was submitted to avoid missing one or prevent a delay in processing, and this is the first use case for Integromat (I could have used Microsoft Power Automate (MS Flow), but I needed to run a feature report later, which at this time was not available in MS Power Automate).  I’ll post a screenshot of the Integromat Scenario a litte farther down.  I will also attach the blueprint exported from Integromat for those who might want to take a look. That takes care of workflow item #2 – the actual receipt of the application.  Moving on to processing.

 

Workflow Item #3 – Processing the Application

 

Using the dashboard with the editable survey form embedded, staff can now select the survey record (newest submissions come in on top of the list), review all the submitted information, add the new account number and other “office use only” information, then re-submit the now-complete survey form.  Now we need to get all this information onto the Microsoft Word Template that is the actual application form.  This is done using the “Create Feature Report” module in Integromat.  There was a little problem with this step, which I’ll explain a little further down when you can see the Integromat Scenario screenshot.

 

Workflow Item # 4 – Getting the Information onto the “paper” application

 

This item is accomplished using the Integromat “Create Feature Report” for Survey123 module https://support.integromat.com/hc/en-us/articles/360020842234-Survey123I then used the "Microsoft Office 365" module to send the Sanitary Board staff an email with the completed Feature Report attached. 

 

Integromat Scenario

 

Here is a screenshot of the Integromat Scenario:

 

 

Here’s an explanation of the scenario, step-by-step:

  1. Watch the survey for a submission using the “Survey 123 – Watch Survey” module
  2. The first router will send the workflow one way when a new record is submitted (using the addData condition) and the other way when a record is updated (using the editData condition). The top branch is the new submission and the bottom branch is the update (when the staff makes their edits).
    1. When a new survey record is submitted (again, top branch), the first module sends a confirmation email to the person who submitted the survey based upon the email they provided in the survey.
    2. Then the next router looks at the survey record and sends the workflow to one of four paths, depending on how many images were attached to the survey. This part was a bit tricky at first.  I initially had the “add photo” option set as a repeat in the survey, but it seems that repeats are difficult to handle in every aspect (and not supported by many ESRI tools and applications at this time, based on my research), so I simply removed the image repeat and modified the survey to only allow up to 3 images (3 separate “image” fields in the survey).  This router looks at the three image fields to determine if an image either exists for each field.  Based on what it finds, it continues along the appropriate path.  In my setup, the very top path is for surveys with no images, the second one down is for those with only 1 image, the third one down is for 2 images, and the fourth one down is for 3 images. 
    3. The images are retrieved using the “HTTP/Get a File” module. In the event that there is more than one file to retrieve, you can string multiple “Get a File” modules in line to accomplish this task, as you can see that I’ve done for those situations where more than 1 file is submitted with the survey.
    4. Last in this branch, I’ve attached the image(s) to an email that is sent to the Sanitary Board staff. This email serves 2 purposes – First, this is the initial notification that a citizen has submitted an application, and second, it provides the staff with the image attachments they will need to process the application.  This completes the initial submission of the application and concludes the automation of the first branch.  Now the staff processes the application.
  3. With all the information provided with the survey submission, the staff process the application (most of which is done is another computer system, which is where the account number is generated). Once they get all the information needed to complete the application, they open the dashboard I discussed previously and add the “office use only” information into the survey form.  When they hit “Submit”, the bottom branch of the scenario is triggered by the first router in the scenario (again, using the “editData” event type, which is accessed by clicking on the connector AFTER the router and applying the filter to look for “editData”.)
    1. (*Edit - read the comments below from Ismael Chivite - there is an easier way to set up for the Feature Report.  This is how I did it, but he describes a better way that I had not discovered at the time of writing this post.)  The first module on the bottom branch is the “HTTP/Make a Request” module. Theoretically, the next step would be the “Create Feature Report” module, but presently you cannot run the “Create Feature Report” module directly from the “editData” condition.  This has not been built into the tools yet, and it will not work.  I encountered most of my problems in this phase. Based on the research I conducted, only the edited attribute information will be included in the payload after the editData condition (not ALL of the attribute information for the feature; i.e. – only the field that was edited is available in the payload). In order to make the feature report work after an edit, you must first use the “HTTP/Make a Request” module to look up the hosted feature layer in AGOL, queried for the globalid of the current survey record (I also grabbed a few other fields that I used later in the process to assign a document name to my feature report).  Here is a screenshot of my HTTP request module:
    2. Then I used the “Parse JSON” tool. This effectively separates the data returned from the HTTP Request module into separate field names.   This tutorial helped me a lot:  https://youtu.be/QEUSzMadHdE
    3. Next is the “Create Feature Report” module (finally!!). This uses the globalId field that was parsed out from the JSON in the previous step. Basically, this provides the module the globalID for the feature it will use to generate the Feature Report. 
    4. Next is the “HTTP/Get a file” module to grab the completed Feature Report URL.
    5. Finally, I sent the email to the Sanitary Board staff email with the finalized Feature Report attached.

Now, here is the logical explanation:  When a new survey is submitted, a confirmation email is sent to the submitter, then an email is sent to the office staff to notify them of the new submission, including any image attachments submitted with the survey.  The office staff then processes the application and uses the custom Dashboard to edit the survey record to include additional information.  When they submit their edits, a Feature Report is created from the survey record and it attaches to an email and sends to the office staff.  

 I’m sure there are other ways to go about doing what I have done, but this is the way worked for me.  I could not find a good resource that explained how to run the “Create Feature Report” module after an “editData” condition, so if nothing else, I hope this helps to explain that process. I’m no programming master (by a long shot!).  Shoot, I’m not really even a novice.  But I do have to say that this Integromat integration is pretty easy to use.  Good stuff ESRI!

I've exported the blueprint for the scenario from Integromat and attached it to this post for reference.

You should also check out https://community.esri.com/groups/survey123/blog/2019/06/02/getting-started-with-survey123-and-integ... and https://community.esri.com/groups/survey123/blog/2019/06/04/survey123-tricks-of-the-trade-integromat.  There is great information in both of these links.  Thanks!

12 Comments
IsmaelChivite
Esri Notable Contributor

Tyler Bragg‌ This is a great post. Thanks for sharing! In your article you mention that you had difficulties creating a report when a survey record was edited.  I am not sure I fully understand here the issue, but it seems like you could feed the Create Feature Report module with the objectId you get from the Watch Survey trigger.

I recorded a brief video showing what I mean. The objectId of the feature edited is part of the payload you get in Watch Survey: It is in the result object.  Since the Create Report module uses that objectId to invoke the report service, it does not seem like you need to do additional queries.  I think the animation will show more clearly all of this. You can click on the animation to make it bigger.

It is true that when a record is edited, the Watch Survey module presently only includes the attributes that have changed. However, for this particular use case, that bit of the payload does not seem to be relevant at all. All you need is the objectId so you can pass it to the Create Report module.  The report module will fetch all the necessary data for the resport to include all the info.

The bit where you you presently really need the extra query is the send email module. If you want to include in the email subject or content information that the edit payload does not include, then you certainly need an extra call to fetch that.

Now, this will not be the case for very long because some other users have requested that we include in the payload all attributes, even if they have not changed.  This work is currently being done and we expect to have this ready in early July 2020. If you would like to give that a go, email holistictesting@esri.com and ask them to add you to the Survey123 holistic testing email list. We will share a Beta version of of this Integromat module in our next testing event in June.

In any event, it is really amazing to see the great work you have done here. Thanks again for sharing your work. I am sure many people will find it inspiring!

TylerBragg2
Occasional Contributor II

Ismael Chivite‌ Thank you so much for responding, and for the tip.  What you explained is what I was trying to do from the start, but I see now that I was trying to grab my objectId from the wrong place.  That really makes it much easier!  The issue if I change things around is that I still need to pull attributes from the feature in order to name the file dynamically.  So I'll have to pull info from the feature service either way.  Would you agree?  

Now the problem I'm having is with having another org member edit the survey in the dashboard.  I tried to have the person submit some updates this morning, and they are getting this error:

I'm sure it has something to do with permissions, but I've looked though the settings in survey123, settings of the feature layer, and this user's org permissions and everything seems to be correct.  This user is a "Creator" with the role of Data Editor.  She is a member of the group for which this survey is shared.  Have you guys seen this error or do you have an idea of where I should look?  Is it a setting of the embedded survey in the dashboard itself? Below is a screenshot of the URL parameters included in the dashboard:

I'll keep looking for the problem in the mean time.  Thanks for all the help!

TylerBragg2
Occasional Contributor II

Ok Ismael Chivite, I think I've found the problem with my other org member editing in the dashboard.  In the screenshot above of the URL parameter of the embedded survey form, I had used "&objectID = {objectid}", which was based on this article: https://community.esri.com/groups/survey123/blog/2019/05/24/survey123-tricks-of-the-trade-editing-re....  In that article, under the section entitled "Embedding an edit web form within Operations Dashboard for ArcGIS," it is explained to use the globalID URL parameter, but the animation provided shows using the objectId.  I used the objectid after watching the animation.  I just went back and changed from using the objectId to using globalId and it seems to be working now - the other org member seems to be able to edit the survey form and submit edits successfully.  So there is a bit of conflicting information in that article, primarily the animation.  Below is a screen capture of the revised URL paramter using globalId, which seems to be working.  I gather that the difference between the two is critical to functionality of the edit URL parameter, and is important to note.

TylerBragg2
Occasional Contributor II

I should also document one other issue I resolved:

I have it set up so that Integromat will send a confirmation email to the applicant when the survey is submitted.  I recently encountered an error that stopped the entire automation when a survey was submitted when no email address was provided (I didn't want to make this a required field in the even that someone doesn't have an email address, or just doesn't want to provide one.  The information isn't 100% necessary.).  To get around this, I added another router immediately before the confirmation email module.  I then set up a filter on the connector to identify if an email was provided.  Now if an email address is provided in the survey, it sends the confirmation email to the applicant.  If it is not provided, the router skips the confirmation email and sends the process to the next module.  It looks like this:

And here is a screenshot of the entire process now.  

You may notice that the chart is arranged a little differently than it was in the original post.  This is simply because I used the auto-align tool and let it arrange them as it saw fit.  This is also beneficial for checking to make sure that Integromat is interpreting the branches the way you intended them.  

You may also notice that I added an additional email module at the end of the last 4 rows (after getting the images from the survey).  This is because I was asked if I could send a notification email to our business license clerk when a survey was submitted that indicated that the property was a rental.  So I added a filter to the connector at the end of each of the bottom 4 rows that looks for the presence of different property owner information.  If that information exists (in my survey, the existence of property ownership information would indicate that the property is a rental), an email containing the property owner information is drafted and sent to the business license clerk so that she can check our database to ensure that the landlord has obtained the proper licensing for their rental property.  If property owner information is not present, no email is sent to the business license clerk.  Here's a screen capture of the filter:

And the email details:

I think this covers pretty much everything.  I'll also include the revised blueprint, fixing the issues I encountered, with this post.  If there are any other issues discovered, I'll post them up so that they are documented for the reader's benefit.

TexasMike
New Contributor III

Nice work Tyler Bragg, appreciate you sharing!

TylerBragg2
Occasional Contributor II

As we have moved forward with the implementation of this process, there was one additional need that I addressed, and I think it is worthy of documenting.  This particular modification is to the dashboard used by the staff to process the applications. 

Specifically, it was identified that there is a need for staff to have the ability to take applications over the phone when a customer calls in.  Rather than having to direct the customer to the online application (especially in the event that the customer may not have internet access or feel comfortable filling out a form online), the staff often take information for the application over the phone and submit the form on behalf of the customer.  In an effort to streamline the workflow for the staff, our idea is to have the dashboard be a "one-stop-shop" for processing.  So in lieu of having the staff open a separate link to the survey to submit a new application, I embedded the "new submission" version of the survey into the dashboard.  This is actually the second version of the survey that is embedded - the first version of the survey form is the one on the left side of the screen and is the "edit" version of the survey.  This second version uses the URL of the original survey without being in edit mode.  Here's what it looks like:

The edit version of the survey is linked to the list you see and is filtered so that when an item in the list is selected, the corresponding survey record is displayed.  This is described in more detail in the original post.

I added the "new submission" version of the survey as a second tab where the map frame is located.  This allows the staff to toggle between the "Map" and "New Application Form" tabs.  This can be accomplished by adding a new "Embedded Content" item to the survey and using the survey URL with no additional tags. Below is the screenshot of the Embedded Content configuration screen:

This modification allows the staff to submit new survey records AND edit survey records from the same dashboard without having to open up another browser tab.  

AprilChipman1
New Contributor II

Tyler! What a great post! Thank you so much for sharing!!!

In order to get the editdata workflow, did you create two webhooks in your Survey? One webhook to call on when creating the data, and then another to call when/if the data is edited? Or did you select both options when creating your webhook?

Thanks again - I'm working on my own version this afternoon!

TylerBragg2
Occasional Contributor II

April, thanks!  I'm glad you enjoyed my post!

To have the ability to distinguish between addData and editData you only need to set up one webhook.  In the "Add a hook" window, you have the option to "check the box" for either addData, editData, or BOTH.  You simply choose both in order to have the option to select which action will trigger which branch of your scenario.  

Once you do that, you can add in a router where you want to branch off.  My router comes right after the Watch Survey module.  Then you add filters to the connectors AFTER the router going to either branch.  You are going to filter by Event Type, which is included in the payload of your Watch Survey module, as shown below.  Event Type will either be equal to "editData" or "addData", depending on which branch the connector leads to.  

IsmaelChivite
Esri Notable Contributor
MarvinDavis
New Contributor III

Excellent work, Tyler!

SusanMathai
New Contributor III

Excellent post!!

ArmstKP
Occasional Contributor III

@TylerBragg2 Are you able to watch for edits in a specific field?