Extending Survey123 smart forms with custom JS functions
Starting with ArcGIS Survey123 version 3.10, you can incorporate logic in your forms using custom JavaScript (JS) functions. Custom JavaScript functions complement XLSForm expression syntax, giving you flexibility to build better calculations, data validation rules and constraints.
This blog provides some guidance to get you started with custom JavaScript functions. For completeness, check the pulldata JavaScript help topic. It assumes familiarity with Survey123 Connect, XLSForm syntax and JavaScript.
Getting started
Let's start with a simple scenario. In Survey123 Connect, create a new survey and add a couple of questions as shown below. Our goal is to create a custom JavaScript function to calculate the greeting question.
| type | name | label | calculation |
| text | myname | Your Name: | |
| text | greeting | Greeting: |
Using regular XLSForm syntax, we could easily calculate the greeting as follows:
concat("Hello ", ${myname})
This already teaches us something: custom JavaScript functions are not always the best approach. If you can solve something easily using pure XLSForm functions, do not use a custom JavaScript function. In our case, we will use this example only because it helps us focus on the basics of setting up a custom JavaScript function.
To invoke a JavaScript function from XLSForm, we use the puldata() function. For example:
pulldata("@javascript","myFunctions.js","HelloMe",${myname})
The parameters for the pulldata() function are as follows. First we pass "@javascript" to indicate that we want to execute a JS function. Then, we pass the name of the JavaScript file that contains our function, which in this example is "myFunctions.js". The next parameter is the name of the function within the file that we want to call: "HelloMe". Lastly, we pass as many parameters as the JS function takes. In our case, we will just pass the name of the survey taker, which is contained in the ${myname} question. If the JS function takes more parameters, we would add them in our pulldata() function call as more parameters separated by commas.
| type | name | label | calculation |
| text | myname | Your Name: | |
| text | greeting | Greeting: | pulldata("@javascript","myFunctions.js","HelloMe",${myname}) |
For the pulldata() function above to work, we need to create a "myFunctions.js" file with its corresponding "HelloMe" JavaScript function. In fact, as you refresh your survey preview in Connect you will get a File not found: myFunctions.js error. That is totally expected.
Custom JavaScript files are stored in the survey directory, within a folder called scripts. In the old days, you had to create the folder manually and add your JavaScript files to it. Starting with version 3.12, you will see a Scripts tab at the bottom of Survey123 Connect that will help you with the process as shown in the next animation:
Next, you can add your own JavaScript function (or functions) to the file. For example:
function HelloMe(whosthere) {
return "Hello, " + whosthere;
}
You can test and even copy the pulldata() function to invoke your function right from the Scripts tab.
Once the file is saved in the scripts folder, make sure your pulldata() function in the XLSForm is invoking the JavaScript function correctly, and give it a try.
Naturally, you will find a few bumps before you get your JavaScript functions working. Here are some of the most common errors that you will encounter:
| File not found: myFunctions.js | Your pulldata() function is trying to load a JavaScript file that cannot be found in the scripts folder |
| Error in myFunctions.js : 6:16 Expected token `;' | Syntax error in line 6 of your function. |
| @javascript error:TypeError: Property 'HelloMe' of object [object Object] is not a function in myFunctions.js:HelloMe | Your pulldata() function is trying to invoke a function that cannot be found in the JS file you specified. |
When writing your own custom JavaScript functions for execution within your Survey123 form, remember that your code will not run within the context of a web browser; you are limited to JavaScript ES6. You can't use DOM objects, or frameworks like JQuery, Ember, Angular etc. You can't access local files or make asynchronous calls either. Despite all these limitations, there is still quite a bit you can do!
Once you have your JavaScript function working, you can publish your survey. Custom JS functions are supported in online surveys as well as in the Survey123 field app. However, keep in mind that JS functions will not execute unless a user is signed in to the Survey123 field app or web app.
Parsing complex data structures
A common use for custom JavaScript functions is to parse complex structures, so you can extract key information from them to calculate questions in your form. From an XLSForm perspective, the syntax in your Survey123 form is really not much different from what you already learned in the Getting started section. The real complexity is handled inside the JavaScript function itself.
As an example, let's take the contents of an AAMVA PDF417 barcode. This type of barcode is used in driver licenses and encodes information such as name, birthday and many other things. Since the Survey123 field app has built-in barcode capabilities, you can scan such a barcode. The contents would look something like this:
This JavaScript function formats the AAMVA string from a driver's license into a JSON object, which can then easily be used within XLSForm to extract the specific information you are looking for:
function DL2JSON (data) {
var m = data.match(/^@\n\u001e\r(A....)(\d{6})(\d{2})(\d{2})(\d{2})/);
if (!m) {
return null;
}
var obj = {
header: {
IIN: m[2],
AAMVAVersion: parseInt(m[3]),
jurisdictionVersion: parseInt(m[4]),
numberOfEntries: parseInt(m[5])
}
};
for (var i = 0; i < obj.header.numberOfEntries; i++) {
var offset = 21 + i * 10;
m = data.substring(offset, offset + 10).match(/(.{2})(\d{4})(\d{4})/);
var subfileType = m[1];
var offset = parseInt(m[2]);
var length = parseInt(m[3]);
if (i === 0) {
obj.files = [ subfileType ];
} else {
obj.files.push(subfileType);
}
obj[subfileType] = data.substring(offset + 2, offset + length - 1).split("\n").reduce(function (p, c) {
p[c.substring(0,3)] = c.substring(3);
return p;
}, { } );
}
if (obj.DL) {
["DBA", "DBB", "DBD", "DDB", "DDC", "DDH", "DDI", "DDJ"].forEach(function (k) {
if (!obj.DL) return;
m = obj.DL.match(/(\d{2})(\d{2})(\d{4})/);
if (!m) return;
obj.DL = (new Date(m[3] + "-" + m[1] + "-" + m[2])).getTime();
} );
}
return JSON.stringify(obj);
}
This is what the actual XLSForm would look like. Note that the myjson question uses pulldata("@javascript") to first convert the output from the barcode question into a JSON string. Then the pulldata("@json") function is used to extract specific attributes from the string.
| type | name | label | calculation |
| barcode | aamva | DL | |
| calculate | myjson | JSON | pulldata("@javascript","aamva.js","DL2JSON",${aamva}) |
| text | dname | Name | pulldata("@json",${myjson},"DL.DAC") |
| text | dlast | Last name | pulldata("@json",${myjson},"DL.DCS") |
| decimal | dweight | Weight | pulldata("@json",${myjson},"DL.DAW") |
Tip: Set the value of bind::esri:fieldType to null in the myjson question if you do not want to store the aamva raw string in your feature layer, but still be able to process it within your form logic.
Working with repeats
Custom JavaScript functions are ideal for processing data in repeats. As of version 3.10, using repeats with custom JavaScript functions is limited to the Survey123 field app. You can retrieve all values for a question within a repeat, or retrieve all records within a repeat.
As of version 3.10, support for repeats in pulldata("@javascript") is limited to the Survey123 field app.
Passing a question within a repeat to pulldata("@javascript")
When you pass a question within a repeat to pulldata("@javascript"), the JavaScript function receives an array of values for the specified question. For example, lets say we want to display a warning message if the user has introduced duplicate values in a question within a repeat. In this case, we want to create a JavaScript function that takes an array and returns true if duplicate values are found. Something like this:
function HasDups (myArray)
{
return new Set(myArray).size !== myArray.length;
}
Now all we need to do is to call the function and pass the question within the repeat to it:
| type | name | label | constraint | calculation |
| begin repeat | fruits | Fruits | ||
| select_one | fruit | Fruit | ${dups}=false | |
| integer | quantity | Quantity | ||
| end repeat | ||||
| hidden | dups | Dups | pulldata("@javascript","myFunctions.js","HasDups",${fruit}) |
When passing a question within a repeat to pulldata("@javascript"), it is important to keep the pulldata("@javascript") outside the repeat. In the example above, note that I first keep the result of the duplicates check outside the repeat, and then I use that value in the constraint expression for the fruit question.
Another common scenario: Using a custom JavaScript function to retrieve the last value of a repeat.
| type | name | label | calculation |
| select_one condition | current_cond | Current Condition | pulldata("@javascript","myFunctions.js","getLast",${cond}) |
| begin repeat | inspections | Inspections | |
| date | insp_date | Inspection Date | |
| select_one condition | condition | Condition | |
| text | comments | Comments | |
| end repeat |
And here the JavaScript function:
function getLast(questionInRepeat){
return conditionsArray[questionInRepeat.length-1];
}
Passing a repeat to pulldata("@javascript")
You can also pass an entire repeat to pulldata("@javascript"). In this case, your JavaScript function will receive all records within the repeat as an array. Each item in the array is in turn another array representing the values for that record.
In our fruits example above, let's pretend we want to calculate how many bananas have been entered. If we pass the entire fruits repeat, we can use a JavaScript function to loop through every record. If the fruit in the record is banana, then we get the quantity value and add it to our total.
function totalBananas (fruits)
{
var totalBananas = 0;
var i;
for (i = 0; i < fruits.length; i++) {
if (fruits.fruit=='banana') {
totalBananas = totalBananas + fruits.quantity;
} }
return totalBananas;
}
Here is the XLSForm:
| type | name | label | calculation |
| begin repeat | fruits | Fruits | |
| select_one | fruit | Fruit | |
| integer | quantity | Quantity | |
| end repeat | |||
| integer | Total Bananas | pulldata("@javascript","myFunctions.js","totalBananas",${fruits}) |
Here is another example. Lets pretend we want to calculate a unique ID for every record in a repeat. Here is what the XLSForm could look like:
| type | name | label | calculation |
| begin repeat | buildings | Buildings | |
| integer | bid | BID | once(pulldata("@javascript", "functions.js", "uniqueID", ${buildings})) |
| text | comment | Comments | |
| end repeat |
Note that in this case, I am again passing the entire repeat into the JavaScript function. The function takes all the existing building IDs (bid field) and calculates a new unique value. Since the calculation applies to a question within the repeat, I need to enclose the JavaScript function call within a once() function. If I do not use once() then the calculation would be executed in a loop.
Here is the JavaScript function:
function uniqueID(rows)
{if (!Array.isArray(rows)) {
return 1;
}
var ids = rows.map(row => Number(row.bid));
var id = Math.max(...ids) + 1;
return id;
}
We could also use a similar technique in a constraint to make sure every building ID is unique within the repeat. Here is the XLSForm:
| type | name | label | constraint |
| begin repeat | buildings | Buildings | |
| integer | bid | BID | pulldata("@javascript", "functions.js", "isUnique", ${buildings}, ${bid}) |
| text | comment | Comments | |
| end repeat |
And here the corresponding JS function:
function isUnique(rows,id){
var ids = rows.filter(row => row.bid==id);
return ids.length <= 1;
}
Of course, you could combine the calculation and constraint into a single form too!
Working with web services
Using a custom JavaScript function, you can invoke an ArcGIS or third party web service. For example, say you want to query an existing ArcGIS feature layer and retrieve some attributes for the survey location. Or say you want to call a non-ArcGIS web service to retrieve some data. All of that is possible, as long as the user is online, of course.
The following JavaScript function takes a geopoint object and returns the first intersecting feature found in a polygon feature layer. The specific layer targeted in the example is a US ZIP code layer, but you can change the URL to use your own. The output of this function is a JSON representation of the feature. We will see shortly how you can use pulldata("@json") to work with such output.
function queryPolygon(location,fields,token,debugmode){
if (location===""){
return (debugmode? "Location Object is empty":"");
}
var featureLayer = "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/USA_ZIP_Codes_2016/FeatureServer/0";
var coordsArray = location.split(" ");
var coords = coordsArray[1] + "," + coordsArray[0]
var xmlhttp = new XMLHttpRequest();
var url = featureLayer + "/query?geometry=" + coords + "&geometryType=esriGeometryPoint&inSR=4326&spatialRel=esriSpatialRelIntersects&outFields=" + fields + "&returnGeometry=false&returnCount=1&f=json"
if (token){
url = url + "&token=" + token;
}
xmlhttp.open("GET",url,false);
xmlhttp.send();
if (xmlhttp.status!==200){
return (debugmode? xmlhttp.status:"");
} else {
var responseJSON=JSON.parse(xmlhttp.responseText)
if (responseJSON.error){
return (debugmode? JSON.stringify(responseJSON.error):"");
} else {
if (responseJSON.features[0]){
return JSON.stringify(responseJSON.features[0]);
}
else{
return (debugmode? "No Features Found":"");
}
}
}
}
The XLSForm looks like this:
| type | name | label | calculation | bind::esri:fieldLength |
| geopoint | location | Location | ||
| hidden | myjson | JSON | pulldata("@javascript","myJSFunctions.js","queryPolygon",string(${location}),"*","",true) | 10000 |
| text | zip | ZIP | pulldata("@json",${myjson},"attributes.ZIP_CODE") | |
| text | placename | PLACEMANE | pulldata("@json",${myjson},"attributes.PLACENAME") |
Note that a hidden question is first used to get the output from the JavaScript function. In general, it is best practice to keep a call to pulldata() in its own question. It makes your XLSForm easier to read and most importantly ensures that the Survey123 web app will handle it well.
The pulldata("@javascript") call includes multiple parameters this time:
- First comes the location, which is enclosed by the string() XLSForm function. This is needed because JavaScript does not know how to handle XLSForm geopoint objects, but we can work easily with their string representations.
- Then comes "*", to define the fields we want to retrieve from the query to the ArcGIS feature layer.
- An empty string for the ArcGIS token is passed, since the feature layer we are targeting is public. Using the pulldata("@property","token") function you could get the token from Survey123 and pass it in if needed.
- The last parameter is used to show or hide debug messages.
The JSON output from a query to an ArcGIS feature layer can get lengthy. To avoid the output being cut off, it is best to use the bind::esri:fieldLength column and set its value to a big number. Say 10000 for example.
Finally, note that the pulldata("@json") function is used to extract the ZIP_CODE and PLACENAME attributes from the output of the JavaScript function.
More Samples
Survey123 Connect includes a new XLSForm sample illustrating how to use custom JavaScript functions in multiple scenarios. As you get hands-on, this is a sample worth checking.
