Bull Release (3.0)

IsmaelChivite · ‎07-05-2018

The festival of San Fermin is a historically rooted celebration held annually in the city of Pamplona, Spain. Since Ernest Hemingway chose San Fermin to be the plot of The Sun Also Rises, this festival has become the most internationally renowned fiesta in Spain. The festival involves traditional and folkloric events being the encierro, or running of the bulls, the most popular. Since the festival takes place for an entire week, a total of 7 encierros are held. You still can make it to the first one: 7th of July at 8:00am. If you can't, encierros will continue for 6 more days in a row.

Our 3.0 release is loaded with many fixes, enhancements and new features. Here are the highlights for what is new.

Survey123 Website

Through the Survey123 website you can design, manage, secure and analyze the result of your surveys. This update includes a number of important enhancements.

Workflow automation with webhooks: Version 3.0 is the first release of Survey123 with official support for webhooks. It may take some experimentation to understand how to best take advantage of them, but no doubt many of you will find this feature very useful.

Our intent adding support for webhooks is to facilitate a number of common workflows such as:

Instant Notifications (SMS, e-mail, push notifications):
- Notify a team supervisor via e-mail whenever a survey inspection form is submitted.
- Send an SMS message to an emergency team leader when a high priority incident is reported.
Instant Data Transfer (Box, Office 365, Google Drive, Box, SalesForce Pardot)
- Automatically add a new row in an Office 365 Excel worksheet with responses from submitted surveys
- Associate customer satisfaction survey responses with prospects in SalesForce Pardot
Other integrations:
- Create a calendar event when a new Inspection is scheduled from Survey123
- Register a new PayPal payment when a new City Code violation is reported
- Invoke my own web service when a new survey is submitted

Webhooks are configured through a new Survey Settings tab added to the Survey123 website and work with both the Survey123 web app as well as the Survey123 field app. For more details you can check our Webhooks—Survey123 for ArcGIS | ArcGIS help topic. A specific blog post on this topic will be shared with details about how to use webhooks with Survey123.

Group Questions in Web Designer: Groups have always been a popular feature in Survey123 Connect, and now you can also take advantage of them in Web Designer. Groups are useful to break-down long surveys into logical sections, helping end-users navigate your forms more easily. Groups are also very handy when you want to hide or show entire sections of your survey. Instead of setting conditional visibility on every question independently, you can create a group, and then apply the visibility rule to the entire group; this is easier to manage, and faster to execute.

Working with Groups in Web Designer is straight-forward: Add a Group to your survey and then drag questions into it. Just as shown in this animation:

File Upload questions added to Web Designer: The File Upload question type allows end-users to attach documents, images and other supported file types into your survey. As a survey author you can control exactly what type of file end-users will be allowed to upload. Uploaded files will be stored as geodatabase attachments in your ArcGIS Feature Service.

File Upload questions are only supported in the Survey123 web app and file sizes are limited to 10Mb.

Filters in the Survey Data tab: The Data tab in the Survey123 website lets you explore individual records submitted to your survey. In this update, you can now filter survey response easily. The new filtering option allows you to define expressions with one or more criteria. In the animation below, for example, a filter is applied to a Polio Vaccination survey to display only households within the governorate of Babylon where the family type has been classified as refugee.

A good amount of effort went into allowing you to apply filters in different ways. It is worth exploring the filtering options with different surveys so you can see the type of things you can do. For example, if your survey contains photos, you can filter survey responses where photos have been included, or where photos are larger than a particular size. You can also filter records created by a particular contributor, select all records within a specific map extent, or filter records where free text questions include more than 200 characters etc

The new filtering options in the Data tab are definitively a great way to effectively drill down into your survey results. As records in your table and map get narrowed down, you can easily click on them to open the survey individual response dialog and inspect all data submitted with each survey in detail. Your filters will also be honored by the export tools and as described below, you can also feed Feature Reports with selections generated through filters.

Enhancements to Feature Reports. Bulk operations and Conditional Template Syntax (Beta): Survey123 lets you create high quality printable reports for data submitted into your surveys. You can define the content, layout and look & feel of your reports by uploading your own Microsoft Word based templates into Survey123. For more details about this capability, check our Print survey results—Survey123 for ArcGIS | ArcGIS help topic. In this release, we are adding two important new features: Bulk reports and Conditional Template Syntax.

Bulk Reports: You can now select multiple survey records and apply a template to them to create many reports at once. At the moment, we are limiting bulk operations to 30 reports (30 selected records). The following animation illustrates how to access the new bulk report capabilities.

The output of a bulk report operation is a compressed file in .zip format containing one document for every record you selected. The .zip file will be stored by default into your ArcGIS account within a folder of your choice. If you wait until the bulk report finishes, we will automatically download the results automatically. Closing your browser while the job is executing is fair game too, because you can always go back to the Data tab of your survey to have access to the last 10 bulk report jobs you submitted. Your outputs will also be kept in your ArcGIS account.

While in Beta, creating bulk reports is free, although we anticipate that once we move into general release bulk reports will consume ArcGIS credits. We are also working on lifting the limit of 30 records, so you can create as many reports as you want in bulk.

We additionally are looking into supporting various destinations for your output reports such as Box, Google Drive and Office 365 One Drive.

Conditional Syntax for Feature Report Templates: Another important enhancement to Feature Reports is support for conditional statements within your custom templates. Just as you can set rules that control the visibility of questions within a survey, you can also insert expressions within your Microsoft Word Custom Report Template to define if particular sections of your report will be shown or not. Feature Report Templates are essentially Microsoft Word documents including certain keywords matching questions in your survey. Our Feature Report Template service simply replaces your keywords with data (numbers, text, photos, maps, signatures...). The exact Feature Report Template syntax is described in our Print survey results—Survey123 for ArcGIS | ArcGIS help topic. Look for the Conditional Statements section for details.

In short, here what a conditional statement looks like:

${if expression} Section of your report to be shown or hidden ${/}

For example, the following statement will include the photo, map and comments of an incident in your report only if the incident is high priority:

${if priority=="High"}
Comments: ${comments}
Photo:
${image1|size:300:0}
Map:
${Location | mapSettings:"":50000}
${/}

You can add any content in between your opening ${if expression} and the ${/} closing statement. For example, you can add Microsoft Word tables to organize your content, fixed images, paragraphs etc.

With the addition of conditional statements you can now author more professional looking reports.

While we originally intended to launch a General Release of the Feature Report capability with 3.0, we estimate that we still need about two more updates to finish up some remaining work. That would push the release to around November 2018. In the meantime, we would appreciate your feedback so we can evolve this feature to your needs.

Survey123 Web App Version Dialog: You can now control what version of the Survey123 web app will be used when rendering your survey. This is an important consideration: If you configure your survey to use the latest available version of the Survey123 web app, any enhancements and bug fixes in product updates will automatically be applied to your survey. While this sounds awesome at first, using the latest version also comes with its own risks: New versions may introduce undesired changes, and in the worst-case scenario even break your survey. We always aim to maintain backwards compatibility, but this new setting gives you ultimate control.

For production surveys, my recommendation is that you lock the version of the Survey123 web app. This means that your survey will be rendered using the same Survey123 web app version used to publish the survey. An additional benefit of version-locking is that it reduces the initial loading time of the form in the user’s web browser.

If you want to incorporate functionality in newer releases into your survey safely, or simply try the latest version to see if it is worth upgrading, all you need to do is to create a copy (save as) of your survey and publish it to try things out. If all works well with your test survey you can go back to the production survey and hit Publish on it. When you publish a survey, it gets upgraded to the latest version and version-locking is enabled.

Tip: If you authored your survey in Web Designer, you can easily test your surveys in the latest version by simply using the Preview option within Web Designer.

By default, all surveys published prior to version 3.0 are automatically rendered using the latest version of the Survey123 web app. That is, pre-3.0 surveys are not version-locked by default. Any survey published with 3.0 or newer, will be versioned-locked by default. Again, you are free to adjust this setting as you see fit.

Other enhancements and fixes:

BUG-000108450 The Survey123 for ArcGIS website (survey123.arcgis.com) does not restrict content viewing on Analyze tab despite having the proper Editor settings configured in ArcGIS .
BUG-000113476 Questions stacked when 'Other' is selected as a choice
When looking at images through the Analyze tab in the Survey123 website, you can now reference back to the original survey individual response.
The map in the Data tab now includes a basemap switcher
Added support for map coordinate format calculations (more on this later in this blog post).
Various usability and cosmetic enhancements in pages and groups.

Survey123 Field App and Survey123 Connect

You can download the latest version of the Survey123 field app (build number 3.0.132) from the Apple, Google Play and Amazon stores. The Windows, Mac and Ubuntu Linux flavors of the app are available from our Survey123 for ArcGIS | ArcGIS Download page.Through the Download page you can also access build 3.0.127 of Survey123 Connect.

Version 3.0 of the Survey123 field app and Connect comes with some serious architectural changes which cause incompatibilities with some older operating systems, versions and configurations of ArcGIS:

Apple's iOS: After many previous announcements, Apple has moved definitively away from 32-bit apps. To keep Survey123 running well on the latest hardware and operating systems from Apple, and to comply with Apple's requirements for apps in their store, version 3.0 of Survey123 no longer runs on 32-bit. This means that 3.0 will no longer be available in older Apple devices such as iPhone 5C (iPhone 5S is fine because it carries a 64-bit chip) and iPad 4.
Android OS: In preparation for Survey123 to support Mobile Map Packages and Vector Tiles in an upcoming release, our 3.0 architecture is only supported in Android 4.4 or newer.
Portal for ArcGIS: Survey123 3.0 does not run against versions of Portal for ArcGIS older than 10.4. Portal for ArcGIS 10.3.1 is no longer supported. This is because Portal 10.3.1 does not support certain REST API operations (supportsApplyEditsWithGlobalIds in the applyEdits operation) required to support nested repeats and improved data integrity through transaction-based rollback operations.
Feature Layers and supportsApplyEditsWithGlobalIds: Starting with version 3.0 you will no longer be able to submit surveys or publish/republish a survey against a feature service that does not have the supportsApplyEditsWithGlobalIds property set to true. While this requirement is not new, we previously only enforced it for surveys with repeats. It is therefore possible that your existing ArcGIS or ArcGIS Enterprise feature services don’t have this capability– so please check them to ensure that they will work when you update. The easiest way to enable this property is to simply enable the sync capability (Synchronization) on your feature service. Please follow instructions in the Allow offline editing or collaboration section of the Manage hosted feature layers—ArcGIS Help | ArcGIS help topic for details on how to enable the sync capability. If working with ArcGIS Server, remember that you cannot turn on sync if your data is versioned or if your layer lacks an indexed GlobalID field.

Survey123 Classic (aka 2.9): If for any reason you cannot comply with all the requirements above, we have made available a version of Survey123 which we will call Survey123 Classic and tag with a 2.9 build number. It preserves the pre 3.0 architecture and as such works with 32-bit devices, older iOS and Android operating systems and against Portal 10.3.1 and feature services without sync enabled. Survey123 Classic is available for Android users (in the Google Play store) as well as on Windows, Mac and Ubuntu through our Download Page. We did our best to get Survey123 Classic into the Apple store, but Apple has not approved its publication at this time.

Survey123 Classic is a temporary measure. We have no plans to update or apply fixes to Survey123 Classic. From a functional perspective, Survey123 Classic is equivalent to Survey123 version 2.8, so any feature announced here for version 3.0 does not apply to Survey123 Classic.

Ok, so lets describe next what is new in this update:

Enhanced Time Control UX: The Survey123 app now features a much improved time user control. This control is used in dateTime and time questions and provides a better user experience for users in the field. Capturing the time when a business opens and closes, when a hydrant was flushed or an asset was inspected... there are many scenarios where you may want to provide a quick experience for capturing time.

Setting the time can be done easily through a touch experience, but you can also type the time.

The time control supports both AM/M and Military time formats. The configuration of the device is used to determine the time format to be used by the control.

As with any usability enhancement, it is hard to describe in words, but based on long hours of testing and feedback provided though our Early Adopters, I anticipate that you will be much more content with this new time control.

Nested Repeats: As its name implies, nested repeats allow you to include repeats within repeats. Up until version 3.0, you could add multiple repeats to a survey, but they could not be nested. This was somewhat limiting. For example, you could author a survey to capture:

A building with multiple households (each household would be a record in the repeat), or
A household that contains multiple people (each person would be a record in the repeat)

But now at 3.0 your survey can contain both:

A building with multiple households (each household would be a record in the repeat), and
A household that contains multiple people (each person would be a record in the repeat)

Technically, there is no limit to the number of repeats that you can nest, however if the number of nested repeats gets too large then things might get a bit busy from a UI perspective.

We have added a new Nested Repeats sample in Survey123 Connect so you can more easily see how to work with nested repeats. I recommend that you take a look at it. This sample also showcases the use of XLSForm aggregation functions across repeats to sum, average or simply count repeated records.

Smart Sketching on Maps: We introduced smart sketching on top of photos and images some time ago as described in the https://community.esri.com/groups/survey123/blog/2017/10/09/smart-skething-in-survey123-stroke-by-st... blog post. In this update we added the option to sketch on top of a map. Essentially, end users can now take a screenshot of a map, and then sketch on top. Sketching on maps is useful in some workflows where you need to quickly annotate a map to highlight relevant facts. The sketching tools are optimized to add text and arrows, which are about the most common things most sketching workflows require.

As usual, the sketch will be stored as an attachment. There is nothing you need to do to enable smart sketching on maps. If your survey already includes smart sketching questions, the new option to annotate maps will come automatically.

Geosearch: While we have had geosearch in the map control of the Survey123 web app for quite some time, we did not have it in the Survey123 field app. That is history with version 3.0. The geosearch control is only available when the map is shown at full screen. The map preview in your survey will not provide an option to run a geosearch, but if you tap in the map to set its location manually, it will.

The geosearch input accepts coordinates in different formats (Degrees Minutes Seconds, Degrees and Decimal Minutes, USNG and MGRS coordinates....). End users can also type a placename or address and a list of candidates sorted by distance from the device's location will be presented.

By default, geosearch uses the locators that have been configured in the ArcGIS Organization. As is commonplace through other geosearch inputs in ArcGIS (Web AppBuilder, Web Map Viewer etc), the end user can also explicitly define the locator to be used in the event that multiple locators have been configured in the ArcGIS organization.

The use of locators through geosearch is only available while . If the device is offline, geosearch will only accept coordinates

Other enhancements and fixes:

BUG-000114023: In Survey123 for ArcGIS, the contrast between the color of the asterisk used for required fields and the background color does not meet the 508 compliance for all font sizes.
BUG-000113272 Survey does not pass Global and Object IDs when pushing Template Survey from Survey123 to Portal for ArcGIS.
BUG-000113639 In Survey123 for ArcGIS, when a geopoint appears due to a relevant statement within a repeat, the values entered are carried into the rest of the repeats but no data is submitted.
BUG-000112961 In Survey123 Connect, clicking "Manage in Survey123" website launches the Survey123 website that was last accessed in the browser, and does not respect the Portal settings in Survey123 Connect.
BUG-000109845 copy the sent data to a new survey>sends updates from Sent->Repeat instead of Adding it overwrites.
BUG-000112374 The word "administrator" is misspelled in the message that appears when a User signs into the Survey123 App and goes to the "Download Surveys" section but has no surveys shared with them.
BUG-000108893 Using Survey123 for ArcGIS field app, if a survey is taken while Offline, then the device is switched to , pending surveys in the Outbox will fail to send with Code 0.
BUG-000108940 Survey123 for ArcGIS, default values are not honored in Repeats with a set repeat count when a Calculation is also present within the Repeat.
BUG-000106662 Windows 10 Lumia phones do not recognize the Survey123 for ArcGIS app when launching the app with a direct link.
BUG-000106467 The Survey123 Connect for ArcGIS splash screen is in English, when installed on a German OS with German language settings.
BUG-000106468 The Survey123 for ArcGIS field app splash screen is in English, when installed on a German OS with German language settings.
BUG-000107010 When fields are passed from a feature service to Survey123 for ArcGIS survey through custom URL scheme, URL encoding is not decoded in final survey result.
BUG-000105814 Publishing a survey in Survey123 Connect for ArcGIS to Portal for ArcGIS configured to use Integrated Windows Authentication (IWA) will fail to publish by getting hung on the "creating form item" step.
BUG-000105387 Survey123 for ArcGIS and Survey123 for ArcGIS Connect applications crash on Windows Server 2012 R2.
BUG-000104637 When publishing a survey to ArcGIS from Survey123 Connect on a Windows 7 machine, Survey123 Connect crashes, returning a 1000 error pointing to AppStudio-WindowsLocation7.dll in the application log.
BUG-000098349 In Survey123 for ArcGIS, sending a survey fails with code 1003 error if the form contains two repeat blocks and if one of them contain a geopoint question.
BUG-000095830 Default values in Hidden fields.
BUG-000107478 Special characters are displayed incorrectly in the Survey123 for ArcGIS app when supplied from a web map.
BUG-000108785 Opening surveys submitted from Survey123 Connect for ArcGIS to the survey123.arcgis.com website configured with Portal for ArcGIS fail with the error: "Surveys published by Survey123 Connect for ArcGIS are currently not supported". This is only reproducible in a portal configured with Integrated Windows Authentication (IWA).
BUG-000114225 Report generated from the Survey123 for ArcGIS website shows incorrect time.
ENH-000106924 Add more coordinate entry formats for Survey123 (specifically USNG).
ENH-000105785 Allow for repeats within a repeat in Survey123 for ArcGIS app (nested repeats).
ENH-000105167 Add the ability to collect coordinates in UTM or conduct a transformation within Survey123.
ENH-000107426 Encrypt the refresh token used in Survey123 for ArcGIS app.
ENH-000104695 Provide the ability to email an email address which was captured from a survey form when the survey is submitted.
ENH-000105588 Using Survey123 for ArcGIS Connect, applying a calculation to a question that is taking the value of a following question (which contains a default value) will not be retained after modifying the value containing the calculation when sending to draft.

Enhancements to XLSForm support

XLSForm is a form standard created to help simplify the authoring of forms in Excel. Survey123 follows the XLSForm specification and in some cases extends it. In this section I will anticipate some important new XLSForm features for which we have added support, although separate upcoming blogs will cover in detail some of them:

Expressions in required and readonly XLSForm columns: The required and readonly XLSForm columns now accept expressions. Before version 3.0 you could flag questions as required or readonly, now you can write expressions to dynamically turn these two properties based on responses in your form.

Reverse geocoding and map coordinate transformations: The pulldata() function has been significantly enhanced by adding extra properties into its pulldata("@geopoint") variation. Starting with version 3.0, you can perform reverse geocoding on an existing geopoint and also perform map coordinate format transformations.

Reverse geocoding is very handy when you want to precompute the address of a location, its postal code etc. Using reverse geocoding against your own custom asset locator, you can also precompute the unique identifier of an asset based on the user's location. This is a very relevant enhancement that will boost the productivity of field users and help with user input validation. An upcoming blog will describe in more detail how to perform reverse geocoding with Survey123. If you want to get into it right away, you can also try the new Reverse Geocoding sample included in Survey123 Connect for ArcGIS or learn more through the Reverse Geocoding section in our Geopoints—Survey123 for ArcGIS | ArcGIS help topic. As of version 3.0, reverse geocoding is only supported in the Survey123 field app, but our Survey123 web app will work with it in our next update.
Map Coordinate Transformations: A new extension to the pulldata function lets you extract MGRS, USNG and other common map coordinate formats from a geopoint. This is useful when you want to persist the coordinates of an event as a GIS attribute, using different coordinate notations. For example, the Search and Rescue community often use coordinates expressed in USNG format, MGRS is a widely accepted standard for military use and UTM coordinates are used globally. We will describe all of this in more detail in a separate blog, although you also have a great Survey Sample in Survey123 Connect and a complete section called Map Coordinate Formats in our Geopoints—Survey123 for ArcGIS | ArcGIS help topic. Map coordinate transformations are supported in both the Survey123 Web and Field apps and work while and offline.

Updated source code in AppStudio for ArcGIS

The source code of Survey123 version 3.0 and 2.8 is now available with the latest version of AppStudio for ArcGIS. Use the description in the template to figure out which one is which.

Congratulations if you made it this far. Definitively, a lot of work in this update. Enjoy!