Extending Survey123 smart forms with custom JS functions

36101
74
08-06-2020 06:22 PM
IsmaelChivite
Esri Notable Contributor
10 74 36.1K

Starting with ArcGIS Survey123 version 3.10, you can incorporate logic in your forms using custom JavaScript (JS) functions. Custom JavaScript functions complement XLSForm expression syntax, giving you flexibility to build better calculations, data validation rules and constraints.

This blog provides some guidance to get you started with custom JavaScript functions. For completeness, check the pulldata JavaScript help topic. It assumes familiarity with Survey123 Connect, XLSForm syntax and JavaScript.

Getting started

 

Let's start with a simple scenario. In Survey123 Connect, create a new survey and add a couple of questions as shown below. Our goal is to create a custom JavaScript function to calculate the greeting question.

typenamelabelcalculation
text

myname

Your Name:

 
textgreetingGreeting: 

 

Using regular XLSForm syntax, we could easily calculate the greeting as follows:

concat("Hello ", ${myname})

This already teaches us something: custom JavaScript functions are not always the best approach. If you can solve something easily using pure XLSForm functions, do not use a custom JavaScript function. In our case, we will use this example only because it helps us focus on the basics of setting up a custom JavaScript function.

To invoke a JavaScript function from XLSForm, we use the puldata() function. For example:

pulldata("@javascript","myFunctions.js","HelloMe",${myname})

The parameters for the pulldata() function are as follows. First we pass "@javascript" to indicate that we want to execute a JS function. Then, we pass the name of the JavaScript file that contains our function, which in this example is "myFunctions.js". The next parameter is the name of the function within the file that we want to call: "HelloMe". Lastly, we pass as many parameters as the JS function takes. In our case, we will just pass the name of the survey taker, which is contained in the ${myname} question. If the JS function takes more parameters, we would add them in our pulldata() function call as more parameters separated by commas.

typenamelabelcalculation
text

myname

Your Name:

 
textgreetingGreeting:pulldata("@javascript","myFunctions.js","HelloMe",${myname})

 

For the pulldata() function above to work, we need to create a "myFunctions.js" file with its corresponding "HelloMe" JavaScript function. In fact, as you refresh your survey preview in Connect you will get a File not found: myFunctions.js error. That is totally expected.

Custom JavaScript files are stored in the survey directory, within a folder called scripts. In the old days, you had to create the folder manually and add your JavaScript files to it. Starting with version 3.12, you will see a Scripts tab at the bottom of Survey123 Connect that will help you with the process as shown in the next animation:

Scripts Tab.gif

 

Next, you can add your own JavaScript function (or functions) to the file. For example:

function HelloMe(whosthere) {

    return "Hello, " + whosthere;
}

You can test and even copy the pulldata() function to invoke your function right from the Scripts tab.

Writting Function.gif

 

Once the file is saved in the scripts folder, make sure your pulldata() function in the XLSForm is invoking the JavaScript function correctly, and give it a try.

Naturally, you will find a few bumps before you get your JavaScript functions working. Here are some of the most common errors that you will encounter:

File not found: myFunctions.jsYour pulldata() function is trying to load a JavaScript file that cannot be found in the scripts folder
Error in myFunctions.js : 6:16 Expected token `;'Syntax error in line 6 of your function.
@javascript error:TypeError: Property 'HelloMe' of object [object Object] is not a function in myFunctions.js:HelloMeYour pulldata() function is trying to invoke a function that cannot be found in the JS file you specified.

 

When writing your own custom JavaScript functions for execution within your Survey123 form, remember that your code will not run within the context of a web browser; you are limited to JavaScript ES6. You can't use DOM objects, or frameworks like JQuery, Ember, Angular etc. You can't access local files or make asynchronous calls either. Despite all these limitations, there is still quite a bit you can do!

 

Once you have your JavaScript function working, you can publish your survey. Custom JS functions are supported in online surveys as well as in the Survey123 field app. However, keep in mind that JS functions will not execute unless a user is signed in to the Survey123 field app or web app.

 

Parsing complex data structures

 

A common use for custom JavaScript functions is to parse complex structures, so you can extract key information from them to calculate questions in your form. From an XLSForm perspective, the syntax in your Survey123 form is really not much different from what you already learned in the Getting started section. The real complexity is handled inside the JavaScript function itself.

 

As an example, let's take the contents of an AAMVA PDF417 barcode. This type of barcode is used in driver licenses and encodes information such as name, birthday and many other things. Since the Survey123 field app has built-in barcode capabilities, you can scan such a barcode. The contents would look something like this:

 

AAMVA

 

This JavaScript function formats the AAMVA string from a driver's license into a JSON object, which can then easily be used within XLSForm to extract the specific information you are looking for:

 

function DL2JSON (data) {
    var m = data.match(/^@\n\u001e\r(A....)(\d{6})(\d{2})(\d{2})(\d{2})/);
    if (!m) {
        return null;
    }

 

    var obj = {
        header: {
            IIN: m[2],
            AAMVAVersion: parseInt(m[3]),
            jurisdictionVersion: parseInt(m[4]),
            numberOfEntries: parseInt(m[5])
        }
    };

 

    for (var i = 0; i < obj.header.numberOfEntries; i++) {
        var offset = 21 + i * 10;
        m = data.substring(offset, offset + 10).match(/(.{2})(\d{4})(\d{4})/);
        var subfileType = m[1];
        var offset = parseInt(m[2]);
        var length = parseInt(m[3]);
        if (i === 0) {
          obj.files = [ subfileType ];
        } else {
          obj.files.push(subfileType);
        }
        obj[subfileType] = data.substring(offset + 2, offset + length - 1).split("\n").reduce(function (p, c) {
            p[c.substring(0,3)] = c.substring(3);
            return p;
        }, { } );
    }

 

    if (obj.DL) {
        ["DBA", "DBB", "DBD", "DDB", "DDC", "DDH", "DDI", "DDJ"].forEach(function (k) {
            if (!obj.DL) return;
            m = obj.DL.match(/(\d{2})(\d{2})(\d{4})/);
            if (!m) return;
            obj.DL = (new Date(m[3] + "-" + m[1] + "-" + m[2])).getTime();
        } );
    }

 

    return JSON.stringify(obj);
}

 

This is what the actual XLSForm would look like. Note that the myjson question uses pulldata("@javascript") to first convert the output from the barcode question into a JSON string. Then the pulldata("@json") function is used to extract specific attributes from the string.

typenamelabelcalculation
barcodeaamvaDL 
calculatemyjsonJSONpulldata("@javascript","aamva.js","DL2JSON",${aamva})
textdnameNamepulldata("@json",${myjson},"DL.DAC")
textdlastLast namepulldata("@json",${myjson},"DL.DCS")
decimaldweightWeightpulldata("@json",${myjson},"DL.DAW")

 

Tip: Set the value of bind::esri:fieldType to null in the myjson question if you do not want to store the aamva raw string in your feature layer, but still be able to process it within your form logic.

Working with repeats

 

Custom JavaScript functions are ideal for processing data in repeats. As of version 3.10, using repeats with custom JavaScript functions is limited to the Survey123 field app. You can retrieve all values for a question within a repeat, or retrieve all records within a repeat.

As of version 3.10, support for repeats in pulldata("@javascript") is limited to the Survey123 field app.

 

Passing a question within a repeat to pulldata("@javascript")

 

When you pass a question within a repeat to pulldata("@javascript"), the JavaScript function receives an array of values for the specified question. For example, lets say we want to display a warning message if the user has introduced duplicate values in a question within a repeat. In this case, we want to create a JavaScript function that takes an array and returns true if duplicate values are found. Something like this:

function HasDups (myArray)
{
 return new Set(myArray).size !== myArray.length;
}

Now all we need to do is to call the function and pass the question within the repeat to it:

typenamelabelconstraintcalculation
begin repeatfruitsFruits  
select_onefruitFruit${dups}=false 
integerquantityQuantity  
end repeat    
hiddendupsDups pulldata("@javascript","myFunctions.js","HasDups",${fruit})

 

When passing a question within a repeat to pulldata("@javascript"), it is important to keep the pulldata("@javascript") outside the repeat. In the example above, note that I first keep the result of the duplicates check outside the repeat, and then I use that value in the constraint expression for the fruit question.

Another common scenario: Using a custom JavaScript function to retrieve the last value of a repeat.

typenamelabelcalculation
select_one conditioncurrent_condCurrent Conditionpulldata("@javascript","myFunctions.js","getLast",${cond})
begin repeatinspectionsInspections 
dateinsp_dateInspection Date 
select_one conditionconditionCondition 
textcommentsComments 
end repeat   

 

And here the JavaScript function:

function getLast(questionInRepeat){

    return conditionsArray[questionInRepeat.length-1];

}

Passing a repeat to pulldata("@javascript")

 

You can also pass an entire repeat to pulldata("@javascript"). In this case, your JavaScript function will receive all records within the repeat as an array. Each item in the array is in turn another array representing the values for that record.

In our fruits example above, let's pretend we want to calculate how many bananas have been entered. If we pass the entire fruits repeat, we can use a JavaScript function to loop through every record. If the fruit in the record is banana, then we get the quantity value and add it to our total.

function totalBananas (fruitsrepeat)
{
 var totalBananas = 0;
 var i;
 for (i = 0; i < fruitsrepeat.length; i++) {
  if (fruitsrepeat[i].fruit=='banana') {
     totalBananas = totalBananas + fruitsrepeat[i].quantity;
  } }
 return totalBananas;
}

Here is the XLSForm:

typenamelabelcalculation
begin repeatfruitsFruits 
select_onefruitFruit 
integerquantityQuantity 
end repeat   
integer Total Bananas

pulldata("@javascript","myFunctions.js","totalBananas",${fruits})

 

Here is another example. Lets pretend we want to calculate a unique ID for every record in a repeat. Here is what the XLSForm could look like:

typenamelabelcalculation
begin repeatbuildingsBuildings 
integerbidBIDonce(pulldata("@javascript", "functions.js", "uniqueID", ${buildings}))
textcommentComments 
end repeat   

 

Note that in this case, I am again passing the entire repeat into the JavaScript function. The function takes all the existing building IDs (bid field) and calculates a new unique value.  Since the calculation applies to a question within the repeat, I need to enclose the JavaScript function call within a once() function.  If I do not use once() then the calculation would be executed in a loop.

Here is the JavaScript function:

function uniqueID(rows)
{

  if (!Array.isArray(rows)) {

     return 1;

  }

 

  var ids = rows.map(row => Number(row.bid));

  var id = Math.max(...ids) + 1;

 

  return id;


}

We could also use a similar technique in a constraint to make sure every building ID is unique within the repeat.  Here is the XLSForm:

typenamelabelconstraint
begin repeatbuildingsBuildings 
integerbidBIDpulldata("@javascript", "functions.js", "isUnique", ${buildings}, ${bid})
textcommentComments 
end repeat   

 

And here the corresponding JS function:

function isUnique(rows,id){

    var ids = rows.filter(row => row.bid==id);

    return ids.length <= 1;

}

Of course, you could combine the calculation and constraint into a single form too!

Working with web services

 

Using a custom JavaScript function, you can invoke an ArcGIS or third party web service. For example, say you want to query an existing ArcGIS feature layer and retrieve some attributes for the survey location. Or say you want to call a non-ArcGIS web service to retrieve some data. All of that is possible, as long as the user is online, of course.

The following JavaScript function takes a geopoint object and returns the first intersecting feature found in a polygon feature layer. The specific layer targeted in the example is a US ZIP code layer, but you can change the URL to use your own. The output of this function is a JSON representation of the feature. We will see shortly how you can use pulldata("@json") to work with such output.

 

function queryPolygon(location,fields,token,debugmode){

 

    if (location===""){
        return (debugmode? "Location Object is empty":"");
    }

 

    var featureLayer = "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_ZIP_Codes_2016/FeatureServer/0";

 

    var coordsArray = location.split(" ");
    var coords = coordsArray[1] + "," + coordsArray[0]

 

    var xmlhttp = new XMLHttpRequest();
    var url = featureLayer + "/query?geometry=" + coords + "&geometryType=esriGeometryPoint&inSR=4326&spatialRel=esriSpatialRelIntersects&outFields=" + fields + "&returnGeometry=false&returnCount=1&f=json"


    if (token){
        url = url + "&token=" + token;
    }

 

    xmlhttp.open("GET",url,false);
        xmlhttp.send();

 

    if (xmlhttp.status!==200){
        return (debugmode? xmlhttp.status:"");
    } else {
        var responseJSON=JSON.parse(xmlhttp.responseText)
        if (responseJSON.error){
            return (debugmode? JSON.stringify(responseJSON.error):"");
        } else {
            if (responseJSON.features[0]){
                return JSON.stringify(responseJSON.features[0]);
            }
            else{
                return (debugmode? "No Features Found":"");
            }
        }
    }
}

 

The XLSForm looks like this:

typenamelabelcalculationbind::esri:fieldLength
geopointlocationLocation  
hiddenmyjsonJSONpulldata("@javascript","myJSFunctions.js","queryPolygon",string(${location}),"*","",true)10000
textzipZIPpulldata("@json",${myjson},"attributes.ZIP_CODE") 
textplacenamePLACENAMEpulldata("@json",${myjson},"attributes.PLACENAME") 

 

Note that a hidden question is first used to get the output from the JavaScript function. In general, it is best practice to keep a call to pulldata() in its own question. It makes your XLSForm easier to read and most importantly ensures that the Survey123 web app will handle it well.

The pulldata("@javascript") call includes multiple parameters this time:

  • First comes the location, which is enclosed by the string() XLSForm function. This is needed because JavaScript does not know how to handle XLSForm geopoint objects, but we can work easily with their string representations.
  • Then comes "*", to define the fields we want to retrieve from the query to the ArcGIS feature layer. 
  • An empty string for the ArcGIS token is passed, since the feature layer we are targeting is public. Using the pulldata("@property","token") function you could get the token from Survey123 and pass it in if needed. Here is how you will want to do it:
typenamelabelcalculation
geopointlocationLocation 
hiddenmytokenTokenpulldata("@property","token")
hiddenmyjsonJSONpulldata("@javascript","myJSFunctions.js","queryPolygon",string(${location}),"*",${mytoken},true)
textzipZIPpulldata("@json",${myjson},"attributes.ZIP_CODE")
textplacenamePLACENAMEpulldata("@json",${myjson},"attributes.PLACENAME")
  • The last parameter is used to show or hide debug messages.

The JSON output from a query to an ArcGIS feature layer can get lengthy. To avoid the output being cut off, it is best to use the bind::esri:fieldLength column and set its value to a big number. Say 10000 for example.

 

Finally, note that the pulldata("@json") function is used to extract the ZIP_CODE and PLACENAME attributes from the output of the JavaScript function.

I put some more additional custom JS function examples here.

More Samples

 

Survey123 Connect includes a new XLSForm sample illustrating how to use custom JavaScript functions in multiple scenarios. As you get hands-on, this is a sample worth checking.

 

74 Comments