Extending Survey123 smart forms with custom JS functions

5233
24
08-06-2020 06:22 PM
Esri Frequent Contributor
6 24 5,233

Starting with ArcGIS Survey123 version 3.10, you can incorporate logic in your forms using custom JavaScript (JS) functions. Custom JavaScript functions complement XLSForm expression syntax, giving you flexibility to build better calculations, data validation rules, constraints etc.

This blog provides some guidance to get you started with custom JavaScript functions. For completeness, check the pulldata JavaScript help topic. It assumes familiarity with Survey123 Connect, XLSForm syntax and JavaScript.

Getting started

Let's start with a simple scenario. In Survey123 Connect, create a new survey and add a couple of questions as shown below. Our goal is to create a custom JavaScript function to calculate the greeting question.

typenamelabelcalculation
text

myname

Your Name:

textgreetingGreeting:

Using regular XLSForm syntax, we could easily calculate the greeting as follows:

concat("Hello ", ${myname})

This already teaches us something: custom JavaScript functions are not always the best approach. If you can solve something easily using pure XLSForm functions, do not use a custom JavaScript function. In our case, we will use this example only because it helps us focus on the basics of setting up a custom JavaScript function.

To invoke a JavaScript function from XLSForm, we use the puldata() function. For example:

pulldata("@javascript","myFunctions.js","HelloMe",${myname})

The parameters for the pulldata() function are as follows. First we pass "@javascript" to indicate that we want to execute a JS function. Then, we pass the name of the JavaScript file that contains our function, which in this example is "myFunctions.js". The next parameter is the name of the function within the file that we want to call: "HelloMe". Lastly, we pass as many parameters as the JS function takes. In our case, we will just pass the name of the survey taker, which is contained in the ${myname} question. If the JS function takes more parameters, we would add them in our pulldata() function call as more parameters separated by commas.

typenamelabelcalculation
text

myname

Your Name:

textgreetingGreeting:pulldata("@javascript","myFunctions.js","HelloMe",${myname})

For the pulldata() function above to work, we need to create a "myFunctions.js" file with its corresponding "HelloMe" JavaScript function. In fact, as you refresh your survey preview in Connect you will get a File not found: myFunctions.js error. That is totally expected.

Custom JavaScript files are stored in the survey directory, within a folder called scripts. In Survey123 Connect, click on Files in the left bar. This will open the survey folder directory. Create a folder called scripts as shown below.

Survey123 Custom JavaScript Functions Connect

While custom JavaScript functions are now an official feature of Survey123, some of you have been using them already for the past couple of years. Please note that the folder name for the JavaScript functions is now called scripts. For backwards compatibility the Survey123 field app will continue working with the old folder name of extensions, but it is best to use scripts as the folder name moving forward. Now, keep in mind that Survey123 versions 3.9 and older do not know about the scripts folder!

We will now place our myFunctions.js file in the scripts folder with its corresponding HelloMe function. You can use whatever IDE you want to create the JavaScript file. You can even use Notepad if you like. For this example, we will go with something like this:

function HelloMe(whosthere) {

    return "Hello, " + whosthere;
}

Once the file is saved in the scripts folder, use the Update button in Survey123 Connect to reload the survey to test your calculation.

Survey123 Custom JavaScript Functions Connect

When working with custom JavaScript functions, you will want to frequently make edits in your code and test in Connect. Luckily, Survey123 Connect will automatically reload your JavaScript functions into the preview as soon as you change them from the IDE. In the animation below, note that as soon as the JavaScript file is saved, the form automatically reflects the changes.

Survey123 Custom JavaScript Functions Connect

Naturally, you will find a few bumps before you get your JavaScript functions working. Here are some of the most common errors that you will encounter:

ErrorDescription
File not found: myFunctions.jsYour pulldata() function is trying to load a JavaScript file that cannot be found in the scripts folder
Error in myFunctions.js : 6:16 Expected token `;'Syntax error in line 6 of your function.
@javascript error:TypeError: Property 'HelloMe' of object [object Object] is not a function in myFunctions.js:HelloMeYour pulldata() function is trying to invoke a function that cannot be found in the JS file you specified.

When writing your own custom JavaScript functions for execution within your Survey123 form, remember that your code will not run within the context of a web browser; you are limited to JavaScript ES6. You can't use DOM objects, or frameworks like JQuery, Ember, Angular etc. You can't access local files or make asynchronous calls either. Despite all these limitations, there is still quite a bit you can do!

Once you have your JavaScript function working, you can publish your survey. Custom JS functions are supported in online surveys as well as in the Survey123 field app. However, keep in mind that JS functions will not execute unless a user is signed in to the Survey123 field app or web app.

Custom JavaScript functions are not supported in public surveys. A user must be signed in before the JS function executes.

Parsing complex data structures

A common use for custom JavaScript functions is to parse complex structures, so you can extract key information from them to calculate questions in your form. From an XLSForm perspective, the syntax in your Survey123 form is really not much different from what you already learned in the Getting started section. The real complexity is handled inside the JavaScript function itself.

As an example, let's take the contents of an AAMVA PDF417 barcode. This type of barcode is used in driver licenses and encodes information such as name, birthday and many other things. Since the Survey123 field app has built-in barcode capabilities, you can scan such a barcode. The contents would look something like this:

AAMVA

This JavaScript function formats the AAMVA string from a driver's license into a JSON object, which can then easily be used within XLSForm to extract the specific information you are looking for:

function DL2JSON (data) {
    var m = data.match(/^@\n\u001e\r(A....)(\d{6})(\d{2})(\d{2})(\d{2})/);
    if (!m) {
        return null;
    }

    var obj = {
        header: {
            IIN: m[2],
            AAMVAVersion: parseInt(m[3]),
            jurisdictionVersion: parseInt(m[4]),
            numberOfEntries: parseInt(m[5])
        }
    };

    for (var i = 0; i < obj.header.numberOfEntries; i++) {
        var offset = 21 + i * 10;
        m = data.substring(offset, offset + 10).match(/(.{2})(\d{4})(\d{4})/);
        var subfileType = m[1];
        var offset = parseInt(m[2]);
        var length = parseInt(m[3]);
        if (i === 0) {
          obj.files = [ subfileType ];
        } else {
          obj.files.push(subfileType);
        }
        obj[subfileType] = data.substring(offset + 2, offset + length - 1).split("\n").reduce(function (p, c) {
            p[c.substring(0,3)] = c.substring(3);
            return p;
        }, { } );
    }

    if (obj.DL) {
        ["DBA", "DBB", "DBD", "DDB", "DDC", "DDH", "DDI", "DDJ"].forEach(function (k) {
            if (!obj.DL) return;
            m = obj.DL.match(/(\d{2})(\d{2})(\d{4})/);
            if (!m) return;
            obj.DL = (new Date(m[3] + "-" + m[1] + "-" + m[2])).getTime();
        } );
    }

    return JSON.stringify(obj);
}

This is what the actual XLSForm would look like. Note that the myjson question uses pulldata("@javascript") to first convert the output from the barcode question into a JSON string. Then the pulldata("@json") function is used to extract specific attributes from the string.

typenamelabelcalculation
barcodeaamvaDL
calculatemyjsonJSONpulldata("@javascript","aamva.js","DL2JSON",${aamva})
textdnameNamepulldata("@json",${myjson},"DL.DAC")
textdlastLast namepulldata("@json",${myjson},"DL.DCS")
decimaldweightWeightpulldata("@json",${myjson},"DL.DAW")

Tip: Set the value of bind::esri:fieldType to null in the myjson question if you do not want to store the aamva raw string in your feature layer, but still be able to process it within your form logic.

Working with repeats

Custom JavaScript functions are ideal for processing data in repeats. As of version 3.10, using repeats with custom JavaScript functions is limited to the Survey123 field app. You can retrieve all values for a question within a repeat, or retrieve all records within a repeat.

As of version 3.10, support for repeats in pulldata("@javascript") is limited to the Survey123 field app.

Passing a question within a repeat to pulldata("@javascript")

When you pass a question within a repeat to pulldata("@javascript"), the JavaScript function receives an array of values for the specified question. For example, lets say we want to display a warning message if the user has introduced duplicate values in a question within a repeat. In this case, we want to create a JavaScript function that takes an array and returns true if duplicate values are found. Something like this:

function HasDups (myArray)
{
 return new Set(myArray).size !== myArray.length;
}

Now all we need to do is to call the function and pass the question within the repeat to it:

typenamelabelconstraintcalculation
begin repeatfruitsFruits
select_onefruitFruit${dups}=false
integerquantityQuantity
end repeat
hiddendupsDupspulldata("@javascript","myFunctions.js","HasDups",${fruit})

When passing a question within a repeat to pulldata("@javascript"), it is important to keep the pulldata("@javascript") outside the repeat. In the example above, note that I first keep the result of the duplicates check outside the repeat, and then I use that value in the constraint expression for the fruit question.

Another common scenario: Using a custom JavaScript function to retrieve the last value of a repeat.

typenamelabelcalculation
select_one conditioncurrent_condCurrent Conditionpulldata("@javascript","myFunctions.js","getLast",${cond})
begin repeatinspectionsInspections
dateinsp_dateInspection Date
select_one conditionconditionCondition
textcommentsComments
end repeat

And here the JavaScript function:

function getLast(questionInRepeat){

    return conditionsArray[questionInRepeat.length-1];

}

Passing a repeat to pulldata("@javascript")

You can also pass an entire repeat to pulldata("@javascript"). In this case, your JavaScript function will receive all records within the repeat as an array. Each item in the array is in turn another array representing the values for that record.

In our fruits example above, let's pretend we want to calculate how many bananas have been entered. If we pass the entire fruits repeat, we can use a JavaScript function to loop through every record. If the fruit in the record is banana, then we get the quantity value and add it to our total.

function totalBananas (fruits)
{
 var totalBananas = 0;
 var i;
 for (i = 0; i < fruits.length; i++) {
  if (fruits.fruit=='banana') {
     totalBananas = totalBananas + fruits.quantity;
  } }
 return totalBananas;
}

Here is the XLSForm:

typenamelabelcalculation
begin repeatfruitsFruits
select_onefruitFruit
integerquantityQuantity
end repeat
integerTotal Bananas

pulldata("@javascript","myFunctions.js","totalBananas",${fruits})

Working with web services

Using a custom JavaScript function, you can invoke an ArcGIS or third party web service. For example, say you want to query an existing ArcGIS feature layer and retrieve some attributes for the survey location. Or say you want to call a non-ArcGIS web service to retrieve some data. All of that is possible, as long as the user is online, of course.

The following JavaScript function takes a geopoint object and returns the first intersecting feature found in a polygon feature layer. The specific layer targeted in the example is a US ZIP code layer, but you can change the URL to use your own. The output of this function is a JSON representation of the feature. We will see shortly how you can use pulldata("@json") to work with such output.

function queryPolygon(location,fields,token,debugmode){

    if (location===""){
        return (debugmode? "Location Object is empty":"");
    }

    var featureLayer = "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_ZIP_Codes_2016/FeatureServer/0";

    var coordsArray = location.split(" ");
    var coords = coordsArray[1] + "," + coordsArray[0]

    var xmlhttp = new XMLHttpRequest();
    var url = featureLayer + "/query?geometry=" + coords + "&geometryType=esriGeometryPoint&inSR=4326&spatialRel=esriSpatialRelIntersects&outFields=" + fields + "&returnGeometry=false&returnCount=1&f=json"


    if (token){
        url = url + "&token=" + token;
    }

    xmlhttp.open("GET",url,false);
        xmlhttp.send();

    if (xmlhttp.status!==200){
        return (debugmode? xmlhttp.status:"");
    } else {
        var responseJSON=JSON.parse(xmlhttp.responseText)
        if (responseJSON.error){
            return (debugmode? JSON.stringify(responseJSON.error):"");
        } else {
            if (responseJSON.features[0]){
                return JSON.stringify(responseJSON.features[0]);
            }
            else{
                return (debugmode? "No Features Found":"");
            }
        }
    }
}

The XLSForm looks like this:

typenamelabelcalculationbind::esri:fieldLength
geopointlocationLocation
hiddenmyjsonJSONpulldata("@javascript","myJSFunctions.js","queryPolygon",string(${location}),"*","",true)10000
textzipZIPpulldata("@json",${myjson},"attributes.ZIP_CODE")
textplacenamePLACEMANEpulldata("@json",${myjson},"attributes.PLACENAME")

Note that a hidden question is first used to get the output from the JavaScript function. In general, it is best practice to keep a call to pulldata() in its own question. It makes your XLSForm easier to read and most importantly ensures that the Survey123 web app will handle it well.

The pulldata("@javascript") call includes multiple parameters this time:

  • First comes the location, which is enclosed by the string() XLSForm function. This is needed because JavaScript does not know how to handle XLSForm geopoint objects, but we can work easily with their string representations.
  • Then comes "*", to define the fields we want to retrieve from the query to the ArcGIS feature layer. 
  • An empty string for the ArcGIS token is passed, since the feature layer we are targeting is public. Using the pulldata("@property","token") function you could get the token from Survey123 and pass it in if needed.
  • The last parameter is used to show or hide debug messages.

The JSON output from a query to an ArcGIS feature layer can get lengthy. To avoid the output being cut off, it is best to use the bind::esri:fieldLength column and set its value to a big number. Say 10000 for example.

Finally, note that the pulldata("@json") function is used to extract the ZIP_CODE and PLACENAME attributes from the output of the JavaScript function.

More Samples

Survey123 Connect includes a new XLSForm sample illustrating how to use custom JavaScript functions in multiple scenarios. As you get hands-on, this is a sample worth checking.

24 Comments
New Contributor

Custom JavaScript functions are not supported in public surveys. A user must be signed in before the JS function executes.

Are there any plans to allow this functionality in the future? It is great that this can be used with a login, but we have several public facing applications that need this to work.

New Contributor II

Is there a way to get intersection information using geotrace for a polygon similar to what you accomplished in the Working with Web Service section with the geopoint?

Esri Esteemed Contributor

Hi Collin, 

Can you add a couple of details to the scenario you are describing?  The user is collecting the geotrace, correct?  Where is/are the polygon(s) that would need to be spatially compared?  If a feature service, you can use the same type of spatial intersection queries as you wold with a feature service.

New Contributor III

Hello,

The same question as Michael had - are there any plans to implement support for public surveys? We have a scenario where customer wants to use public survey on the website and based on input from user, he wants to populate couple of other fields from web service. Pulldata() witch csv file is not an option as database is updated dynamically. Is there any another approach we could use?

New Contributor III

Hello, we need to include a QR code in a survey. Would it be possible to generate a QR code with js and include it as an attachment in an image type question?

Thanks

New Contributor

Is there any way to query the previously submitted survey records? 

Esri Frequent Contributor

Hi Neil. The Working with web services section in the blog above describes how you can query an existing feature layer. The JavaScript XLSForm sample (see below) also contains a Working with a Feature Service section showing how you can query existing features. Both examples show the basic pattern. To learn more about the Feature Service API, check: Feature Service—ArcGIS REST API | ArcGIS for Developers 

New Contributor

The scenario we are trying to resolve is, an inspector is walking through a field in a remote area without cell service. They collected several samples, entering each one as a new survey record into Survey123. We want to write some custom logic that uses the past few records. We cannot make a web service call because the inspector does not have cellular service and also because the inspection records have not yet been synchronized to the server. Is there any API for getting at the data that was collected on the device in order to build some custom validation logic with JavaScript? 

Esri Frequent Contributor

Hi Neil. Not currently possible. That is, a custom JS Function can only access data from the current survey record. It is not possible for a JS Function to work with other records in your device.

New Contributor II

Hi, I've a similar question I think. Can I generate a Qr code with JavaScript and insert it into the report?

New Contributor II

Hi, Ismael Chivitewhat do you think about what  Fco Javier Romero proposes? Is it possible to include as an attachment a QR code generated with Javascript in survey123? . 

I have tried two options, write all the code necessary to generate the QR code in the js script, and use the Google Charts API with de url : "https://chart.googleapis.com/chart?" . 

In both cases the JS code does not give me an error, but I do not know if it is possible to use an image question to incorporate it as an attachment:

Thanks

Esri Frequent Contributor

Hi Juan Luis García González‌  Fco Javier Romero‌  I would not know how to dynamically generate and display a QR code within a survey.  You could definitively use a webhook to generate the QR code and email it or SMS it to the respondent, but including the QR code itself within the survey as it gets completed is not something I would know how to do. If you are using online surveys, another approach is to use the https://community.esri.com/groups/survey123/blog/2020/03/05/introducing-the-survey123-web-app-javasc... . With this API, you can build a custom web app that listens to an emebedded web form, and use data from the webform to create and display the QR code within the web app. 

New Contributor II

Thanks Ismael,  I will keep looking for other options for my workflow.

Regards

New Contributor

Hello, I am trying to accomplish the following using AGOL. Please let me know if anyone has any advice. 

 

We have a public form for residents to schedule inspections conducted by our municipality. We want public users to be able to self schedule. 

 

Workflow:    

   Person A: Opens public form, selects Monday August 28th at 8:00 AM, Submits form

   Person B: Opens public form, (Cannot Select same day as person A), Selects available date, Submits form.

New Contributor II

Hi! I have a requirement to integrate to an ERP,
and to be able to discount inventory,
in survey123 I have a list of products and when
I select a product it automatically connects to
the erp through the API and shows me how much
inventory there is of the product,
with this functionality it is this possible?

Esri Contributor

Hi Andrew,

I don't think JavaScript functions will be applicable here for a public survey

Custom JavaScript functions are not supported in public surveys. A user must be signed in before the JS function executes.

Cheers,

Chris

New Contributor II

I tired the JS with License ID Barcode, but the calculated value is not working. Also, is there a way to store the scan record with more than 256 characters (It shows an error if I increase the length)?

Esri Esteemed Contributor

Hi Yaser,

Could you provide details about the barcode you are attempting to read?  What format is it?  

With regard to increasing the number of characters, yes you should do that - you can set it with the bind::esri:fieldLength column

New Contributor II

Ismael Chivite‌ very cool to see this advanced functionality in S123. One use case I am evaluating it for is preventing duplicate submittals based on a unique key in a parameter URL. This use case would mimic how, say, a Survey Monkey form can only be taken one time. I'd have to pull the previous data and check if a key exists already, then use some kind of constraint to prevent another submittal. Kind of out of box thinking here, but maybe it would work.

Please tell me if you think this is possible:

Background

  • Public-facing web Survey123 form

Typical Survey123 steps

  • User clicks a Survey123 URL with parameters in an email that prepopulates a read-only text field in the form (effectively a foreign key)
  • User fills out the web form. Prepopulated key resides in collapsed group.
  • Submit, done.

Duplication check steps

  • User clicks the URL in the email again.
  • Survey123 pulldata checks submittal FS to see if the prepopulated key is already in the service
  • If the key is there, it prevents submitting the form my making an impossible constraint on a read only field in the survey, or perhaps hiding some group that contains the responses.
New Contributor

Hi everyone,
Ismael Chivite Very excited about this update.


I have a scenario where in each entered record, a customer name is selected from a list, and in this way, through a pulldata, other hidden fields such as email, address, telephone number and others are completed.
However, when there is a new customer, which is common, it is necessary for the IT department to update the survey options and the pulldata list, and that is not always immediate. So my question is, Is it possible through javascript, that a function consults a feature service in arcgis that is kept updated with the client data? It would be great, since then, I could extract the name from there and all the other customer data.

Thanks,

Jeffry J.

Esri Frequent Contributor

Hi Administrador Data Gis‌ 

The "working with web services" section in the blog above describes how you can query a feature service to populate questions in your form. This should help replace your pulldata lookups against the local CSV file.

Now, the larger problem in your case, if I understand correctly, is that you want to dynamically populate a list with the output of a feature layer query. This is something that it is still in our backlog. Please open a Tech Support case so your customer number is attached to ENH-000119959 Add capability to populate a survey's choice list from a dynamic location, such as another feature layer or a database table   I cannot anticipate a date for resolution of this ENH but it is definitively something we would love to have our hands on soon.

Esri Frequent Contributor

Hi Elliott Plack (SHA)‌   The logic you describe above makes sense, but JS functions can only be used with private surveys (they do not run on public surveys). 

New Contributor II

I am trying to read the driver's license barcode, PDF417 code.

Occasional Contributor III

I just created an idea for public surveys to have this functionality. You can check it out here: