topic Re: Survey123 'username' field is <Null> in ArcGIS Survey123 Questions

Survey123 'username' field is

TI — Tue, 08 Jan 2019 23:06:07 GMT

I have published a Survey123 with a field of type 'username' by which I was hoping to capture the ArcGIS Online user name of the account being used to submit the survey. Unfortunately, this is not working reliably. While most of the records in the feature service are submitted with the username as expected, a substantial number of them are submitted with <Null> (no value).

We are using Survey123 version 2.8.2 on iOS (and cannot upgrade any further for now due to our old iPads not being able to run a recent version of iOS), with ArcGIS Online as a proxy to a feature service on our local ArcGIS Server (not Portal), so that we can capture the data into our local SDE enterprise geodatabase.

In my limited testing, the <NULL> values are recorded when users are signed out (in Survey123) at the time they capture the survey data. Signing back in (in Survey123) to submit the data does not capture the username.

NB: The users are not deliberately signing out. They just get signed out by the system from time to time, without realising it.

My preference would be to be able to use the editor tracking fields for this (create user, last edit user), but in this case, the feature service is hosted on a local ArcGIS Server, not ArcGIS Online, and Survey123 submits to it via an ArcGIS Online proxy, because it cannot authenticate to our ArcGIS Server directly. So the create/edit user is always recorded as that of the saved credentials for this proxy feature service in ArcGIS Online.

Is there any reliable way to capture the username in this scenario?

UPDATE - I have just found these related topics:

Re: Survey123 'username' field is

Anonymous User — Tue, 08 Jan 2019 23:37:35 GMT

Hi there,

The issue you are encountering with null username values is due to the fact that if there is no signed in user when the survey form is loaded for a new collect, the username name field will be null as there is no user signed in to retrieve the value from. The username field gets populated when the survey first loads, therefore for this field to be updated successfully a valid user needs to be signed in at the time the check for a username value is done.

Whilst you are using an old version of the app, this same behaviour will still exists in the current version of the app.

Regards,

Phil.

Re: Survey123 'username' field is

TI — Wed, 09 Jan 2019 00:34:20 GMT

Thanks. That's what I expected.

So my original question still remains...

Is there any reliable way to capture the username in this scenario?

Re: Survey123 'username' field is

Anonymous User — Wed, 09 Jan 2019 21:28:48 GMT

No, it is not possible using the username type question, you will need to have a manual process to update username field in database based on the created by system field which will reflect who submitted the survey. In the app there needs to be a signed in user when the survey is loaded for username to be updated. As mentioned in the other posts you have been posting regarding the same subject, with the latest beta build and soon to be released 3.2 version of the app the user will remain signed in even when offline which will resolve this problem, although you will need to upgrade to 3.2 to get them improved functionality.

Phil.

Re: Survey123 'username' field is

TI — Sun, 13 Jan 2019 21:08:15 GMT

Thanks Phil. Unfortunately, the created by system field (editor tracking) is no good for us. As explained earlier, this survey submits data via an ArcGIS Online feature service proxy using saved credentials for the actual ArcGIS Server feature service. So all the same username gets logged for every survey in the editor tracking fields.

We are stuck on version 2.8.2 for a while yet, due to our iPads being to old to upgrade any further. But I'll keep trying to persuade management here that we need to upgrade in order to overcome some deficiencies in our current service.

Cheers,

Nik.

Re: Survey123 'username' field is

AlessandroRussodivito — Wed, 06 Mar 2019 15:06:37 GMT

Hi Philip,

our customer have same issue.

Regarding the improvement of the login status, to avoid losing "username" even offline, can you confirm that Survey123 Field App ver 3.2 solved the issue?

I don't see anything about this in the "what's new" section of website at 3.2 or 3.3, and we are still exepriencing the issue.

Do you have a workflow to test if this functionality has been implemented?

Thanks

Alessandro

Re: Survey123 'username' field is

Anonymous User — Wed, 06 Mar 2019 22:09:49 GMT

Hi Alessandro,

The What's New in the Survey123 doc pages includes new features and note worthy bugs or enhancements, but does not list every bug fix and change made with each release. The statement "various bug fixes and improvements" covers this. If we listed everyone it would be 100 items long. What's new in Survey123—Survey123 for ArcGIS | ArcGIS

In the release blog posts we go into more detail and list every support case bug fixed and enhancement implemented. You can see this for 3.2 and 3.3 here: https://community.esri.com/groups/survey123/blog/2019/02/01/national-weatherpersons-day-release-32 and https://community.esri.com/groups/survey123/blog/2019/02/26/new-britain-release-33. You will see some of the bugs listed related to IWA and user sign out issues.

Maybe it is better to understand the way it is suppose to work. What type of devices and OS versions are you using?

When you sign in successfully you get an expiry token from the authenticating server if using IWA or ADFS, and as you are not on a Windows device (which IWA and ADFS are developed for) it can not automatically continue to renew your login based on the device sign in credentials. In Survey123, the app will cache you sign in details and use them when the device is offline or has no access to the authenticating server to display the signed in user in the app, therefore while the app is open you should still see the user signed in and the "username" can be used in the surveys fields.

However, the token that the server provided may expire during this time (depending on the server settings and length of expiry), meaning the user must login in again with valid credentials to obtain a new expiry token, this will happen when online and trying to use a feature that needs to request information from the server (download surveys, submit surveys etc).

Hope this helps, please keep in mind there are cases where users will always have to sign in again using IWA or AFDS due to token expiry, this is expected.

Phil.

Re: Survey123 'username' field is

AlessandroRussodivito — Thu, 07 Mar 2019 18:03:19 GMT

Hi Philip,

thank you for this complete answer.

Customers are using Android / iOS App with id@ account (not built in).

Using previous version (<3.2) we decided to set the property(‘username’) field as required.

A logged out user that was filling the form and realizing that a “required” username was missing, had to:

set the form in Draft
go to App Settings to perform Login
come back to that specific Draft form where he faced the issue: the field stored as null was reloaded with null value
in case of editable field, he had the chance do modify it refreshing the field (reload icon)
in case of read only field, he had no chance to modify it: only way out was survey deletion.

I guess that the previous scenario (in particular the second point) can’t happen anymore, now that the field is always filled in with last username, can you confirm this?

Thanks again,

Alessandro

Re: Survey123 'username' field is

Anonymous User — Fri, 08 Mar 2019 01:20:35 GMT

Hi Alessandro,

What you are seeing is the expected behaviour in the field app. Read only questions can not be refreshed if the calculation changes (no refresh button is displayed) and this is expected as the question is set to read only, meaning it can not be changed by the user. This applies to new Collect, Drafts, Sent and Inbox surveys if survey is opened again and had a previous value or null value. Therefore applying required to read only questions can be problematic if you expect the calculation to change and be updated.

The workaround to the above is to not make that username field read only, that way the user can refresh the calculation if it changes, and you can keep it required, so that when the user signs in before sending, after collecting data offline and saving to drafts, they can open the survey from drafts, refresh the username calculation, and then submit.

Note the property() function, including property('username') is no longer supported in the field app and has been removed from the documentation, as it is not supported in the web app. The property() function may continue to work in the field app however, but there may be some known limitations and it could eventually stop working with updates to the field app in the future, so beware using it.

The other option to capture the current signed in username is using the username question type. This question type does the same function as the property('username'), it is supported in both field app and web app, and is hidden on the form by default. However, the downside of it being hidden always is the question can not be refreshed manually as it is not visible for user interaction. Also, the same problem you have encountered above also occurs with username question type when a user is offline. If you are offline when first open the survey and not signed in, this field will be populated with null, and then never gets updated again. A bug has been raised for this and we have an internal issue to track and try to improve this behaviour so that it can be automatically update when null. The public bug number is BUG-000113164 and I have also updated the internal issue with your comments and use case.

Hope this helps.

Phil.

Re: Survey123 'username' field is

AlessandroRussodivito — Fri, 08 Mar 2019 19:03:10 GMT

Hi Philip,

yes it helped me to understand that I misunderstood the bug fix introduced by 3.2 version. There are a few case still unhandled.

At this point:

any already downloaded survey can be accessed by a logged out user, introducing the null username value issue
last logged username is not limitless stored , it gets cleared in some point in time (mainly by user log-out)

In my opinion, one of the 2 functionality needs to be changed.

For example, Survey123 Field App could stop user from accessing survey list (gridview page), if a user explicitly/manually logged out. This, in order to always have a fresh or cached username available, at survey form opening. Could that be a solution?

I also wasn't aware of the documentation change about property('username') (link), thank you for the insight !

This goes against the backward compatibility of many customers production forms.

Is this a taken decision? In this case any idea about the release version in which forms using property will stop working?

Anyway, if the username question type (not refreshable in case of "Draft" Null issue) will be the only way left, workflows based on username will be hard to use in the future. Maybe an "autorefresh" feature on that question type, could be a solution.

What do you think?

Alessandro

Re: Survey123 'username' field is

Anonymous User — Tue, 12 Mar 2019 00:49:34 GMT

Hi Alessandro,

As I mentioned above, we are looking to improve the null username issue when opening survey when user was initially offline and then re-open from Drafts. A bug has been raised for this and we have an internal issue to track and try to improve this behaviour so that it can be automatically update when null. The public bug number is BUG-000113164 and I have also updated the internal issue with your comments and use case.

Phil.

Re: Survey123 'username' field is

wcozzens — Thu, 08 Aug 2019 13:42:06 GMT

Has there been any improvements made to address this issue? As noted above there is no way to populate other fields in the form based on the username if the survey was started when offline. I've noticed in the latest version that users are required to login before submitting a survey, but the username field is still not being populated when the form is submitted. Any idea when this may be addressed? Thank you.

Re: Survey123 'username' field is

JamesTedrick — Fri, 09 Aug 2019 17:01:36 GMT

Hi Keith,

Currently, this issue has not yet been addressed.

Re: Survey123 'username' field is

JoshHabel — Fri, 16 Aug 2019 14:57:54 GMT

Any idea of when this bug will be addressed? I hope soon because this has completely ruled out using survey123 for editing work if you need to be able to limit editors to only specific work assigned to them. Does anyone have any workaround for such a scenario?

Re: Survey123 'username' field is

RachelS — Mon, 19 Oct 2020 16:01:08 GMT

Over a year on and there is still no fix for this. 25% of my records have the username as blank.

It's a pretty redundant field if you can depend on it to work reliably.

Re: Survey123 'username' field is

wcozzens — Wed, 07 Jul 2021 14:39:43 GMT

This issue is causing major issues now as we have many more Survey123 users and this is happening more often. The username field is also being used to get the submitters email address from the SSO login and then process the submission through Integromat. If the user is not logged in, then no email address is available and the whole system falls apart.

Is there any news when this will be fixed? Hopefully ASAP!

Re: Survey123 'username' field is

Anonymous User — Wed, 07 Jul 2021 23:21:09 GMT

Hi @wcozzens,

Have you logged a support case with Esri Support? If so can you please provide the BUG number so we can take a look into the detailed information.

The details of your enterprise configuration, authentication methods, device etc will be key to resolving this issue.

Just an FYI - login information is cached and the app can be used offline and user is still signed in, provided at time of going offline they are still signed in. Problems can occur if during a check to remain logged in (token refresh) if that fails due to bad network, the user may be logged out unexpectantly, and need to sign in again on a valid network.

Regards,

Phil.

Re: Survey123 'username' field is

wcozzens — Thu, 08 Jul 2021 13:51:30 GMT

Hi Phil,

Thanks for the reply. I have not logged a case with Support. The scenario you explained above is probably exactly what's happening. During a check to remain logged in (token refresh) ...the user is in the field without any valid network available so that check fails and they are logged out. They cannot sign in again since they are still in the field without a valid network connection. It's not feasible to ask the users to abandon their work in the field, drive to a location where they can access a network to sign in again, then return to the field to create a survey. That's not going to happen. I imagine this scenario is fairly common for people working in remote locations - in our case wind and solar farms in the middle of nowhere. If this is the case, how can this be resolved? Seems like a major flaw in the intended (disconnected) functionality of Survey123. If there was a way to control/adjust the token refresh requirement wouldn't that solve the issue?

Re: Survey123 'username' field is

Anonymous User — Thu, 08 Jul 2021 23:39:04 GMT

Hi @wcozzens,

The best way to avoid this unexpected sign out is to put the device into offline mode before going into the field. In this workflow, if you download the required surveys and additional data while online, ensure everything is there and ready to go, and then turn off network connections (WiFi, 4G, mobile data etc.). You can leave the GPS (location services enabled) if you need this for the location in the survey.

If you then go out into field in true offline mode, the user will still be signed in (in a cached mode) and the username field and other user based information will still be available and used in the survey. And becuase the app is offline, no checks will be done to keep the user signed in or to refresh token. Once the user has finished in the field, and is back to a location with a good network, they can enable the network and send all their surveys.

This is the best workaround whilst we continue to resolve this issue and make improvements to the way the user sign in, cached credentials and token refresh works with SSO environments.

Regards,

Phil.

Re: Survey123 'username' field is

wcozzens — Mon, 26 Jul 2021 10:42:10 GMT

Thanks Phil. That would work for some people, but many of them would either forget to do that or not want to do that because they need to be connected for other reasons. The ONLY way to make this work reliably is to take the responsibility out of the users hands and force the issue programatically. We cannot rely on the field people to do this on a regular basis. In the office world it seems like a simple solution, but in the real world with hundreds of non-technical people using this app it's just not going to happen. Hopefully there will be a better solution soon.