field value retrieval problem with certain field types/parameters cause blank values then written to source data

[BillSimper]

Hi, writing in to report a pernicious problem that is cropping up in certain situations.

The general pattern is that an existing record in a Hosted Feature Layer with properly populated fields is being retrieved somehow into S123 (via Inbox or via URL scheme in a Dashboard) and, under certain ideosyncratic conditions, some populated values show up in the retrieved record as blank. If the record is re-submitted, these blank values will overwrite the existing correct values in the data layer.

One example are text fields with input masks retrieved in the web browser version of S123. They are being linked to from a Dashboard. The input masked fields come in blank, but when the mask is removed they come in properly (this is the workaround in our case).

Another example is 'time' control fields in a repeat (retrieved into the field app via Inbox). In this case the values on all sub-records (except the very first, which does show up) are treated as blank until the reviewer manually scrolls through all repeat records. If the reviewer does not do that and resubmits the survey, then blank values will be written to the source data. As you know these 'time' values are stored as text, so changing the field type to 'text' solves this problem; however in our case this breaks some other necessary functionality (for an unknown reason-- embedded javascript is involved).

Both of these examples occur in QA/QC workflows where reviewers are retrieving records submitted by other users on other devices. So we just want to bring it to your attention, since the end result if unaddressed is that a normal review will actually corrupt existing data.

Thanks!

[Anonymous User]

Hi @BillSimper,

It sounds as if the input masks, calculations or relevant statements being used could be conflicting and when the survey is opened in edit mode, they null out the value. This can happen for a few reasons but it depends on the configuration of the questions in the form. Are you using "null" question types for any of the calculations or not storing values that woudl be needed to update the form in edit mode?

It seems you are using repeats, but do you use repeat_count also? There are some known issues and limitations with repeat_count, and other issues with editing repeats via Inbox or web app and URL parameters.

Can you please provide a copy of the XLSForm (xslx file and any files in media folder)? Also exactly which questions are having the issues, and any workflows or steps to reproduce the problem?

Was this survey created in Connect or web designer? Has it since been republished with latest 3.13 release, and is version locking enabled in the settings for this survey, do you open it with latest version always?

And lastly, have you raised a support incident with Esri Support, if so what is the case id or BUG number?

Regards,

Phil.

[BillSimper]

Hi Phillip,

here's an XLS form (and associated javascript code in the txt file) that illustrates both problems.

To see the problem populate several Visit repeat records with time values, submit the record and then open it from the Inbox. The values in the summary fields, except for the first, will show as NaN:Nan (until all values are scrolled through), which isn't great, but the real problem is that if the record is submitted without scrolling through all repeat records, then blank values will be written back to the HFL, overwriting the existing data. However the form will not complain that the required fields are blank, leading me to believe that the data has been downloaded but is somehow not being 'loaded' from the form's perspective.

To see the problem with input mask, you have to submit a record, then open that record in the WEB form (from a Dashboard). Then it will be blank and that blank value will overwrite the existing value if the record is submitted.

[Anonymous User]

Hi @BillSimper,

Thanks for sharing your XLSForm, we will take a look and get back to you. 

@ZacharySutherby 

Regards,

Phil.

[BillSimper]

Haven't submitted a bug. The survey was published using Connect (using v 3.13 but also earlier versions) and was not version locked. Null fields and repeat_count don't seem to apply.

[ZacharySutherby]

Hello @BillSimper, 

The repeat NaN issue is expected as repeat values and calculations need to be updated, same as how validation works when navigating through repeats. What you can do to warn the field workers if they haven't navigated through the repeats is to set a constraint on the Visit Summary question. Setting the constraint to not(contains(.,'NaN')) will catch if a repeat hasn't been navigated through and will throw an error message. You can set the constraint message to say something like "Please navigate through all repeats" if you like. 

For the webform issue with our 3.13 release the Survey123 website added support for input masks, I'm thinking this issue may be related to that implementation. I am currently installing the 3.12 local website to test there and confirm the behavior started with 3.13 and get an issue logged on our end.  

[erica_poisson]

Hi @ZacharySutherby @Anonymous User  - 

A follow-up on input masks and Dashboards... has this been resolved? I apologize, but I am unclear from this thread. 

I have a Dashboard configured to edit existing Survey123 submissions - I've linked a list element with an embedded Survey123 form. I have one field in the Survey123 form that leverages an input mask. The data has been entered properly, but the field with the input mask shows up blank in the loaded form. If the form is edited and resubmitted, it appears that the stored value is kept (not nulled in the hosted feature layer). 

Here is an example - the field DCR File Number shows blank in the form on the right, but is included in the list to the left:

This survey was published from Connect version 3.13.234.

Thank you,

[BillSimper]

Hi @ZacharySutherby,

Thank you for the constraint tip- that'll go some distance to mitigating our problem.

I'm still kind of confused about this phenomenon though. For example, the Time Control works fine when used in the form when the record is first submitted-- as expected a 5 character string is created and stored in the hosted feature layer (until it's submitted it looks like the form is treating it as an integer, i.e. Unix Time). So that string exists in the feature layer data, and is being successfully downloaded to the form via the Inbox-- we know this because the value shows up when we navigate the repeat, and even if we dont navigate the repeat, Survey123 knows that there is a value present because it doesn't complain that a required field is lacking a value when re-submitting. Furthermore, if I change the question type from Time Control to just a Text field (in a separate, cloned version of the original form specially made to retrieve, view and edit these records) and refresh the Inbox, then the Summary comes out wrong (which I understand and expect based on your response) *but* re-submitting the form will not overwrite the existing values with blanks. So the question is why is the Time Control acting this way when a simple Text field does not? And can this control be fixed (in the Survey123) so that it doesn't do this? 

This is the first I've heard that this is expected behavior (I'm not following your comment about validation)-- is there someplace I can read up on this mechanism?

I'm also baffled by some differences between how a form behaves in Survey123 Connect versus the Survey123 field app. One way I tried to work around this issue was simply to make the problematic time field a text field (with or without an input mask) so that users could simply type in the appropriate value. When I do this and adjust form's javascript code to simply append these text values together (in a calculated field outside the repeat) it works as expected in Connect (you can see that code commented out in the script). However this breaks in the Survey123 field app for some reason-- the summary won't work. Because our users insist that these summaries are absolutely necessary when filling out and submitting a survey,  I'm then forced to keep the time control and parse out the hours and minutes from the Unix time value.... but this leads to the data deletion problem if that record is ever accessed through the Inbox. So any ideas why something would work as expected in Connect (correct concatenation of string fields in the repeat)  but not in the Field App?

Thanks for your help with this!

[AdamMesser1]

@BillSimper and @ZacharySutherby Thanks for the post and link to the 3.13 release notes. We are experiencing the same issue. We had one field with an input mask that started experiencing issues one week ago, based on our data tracking. The survey, had been published with S123 Connect several months ago, and had been working correctly, UNTIL we republished the survey to integrate some additional functionality. We did not change the field with the input mask. We did update to the most recent version of S123 Connect before publishing. Could be worth holding off on upgrades to web surveys using url parameters with fields with input mask until the issue is resolved.