External Choice List not Updating when Survey123 Opened from Field Maps/URLs

[Vinzafy]

Bringing this question from 2021 back to life that @RobertAnderson3 posed.

Original thread can be found here:

External Choice List not Updating when Survey123 Opened from Field Maps 

Essentially, when using external choice lists, the only way to have surveys receive updated choice lists is to manually open the affected surveys in the Survey123 app. This was stated as expected behaviour as external choice lists are updated via a check on the info page.

Thus, if a survey is launched via URL from a source like Field Maps or Dashboards which bypass the info page, updated choice lists would not be received by the survey. I've also noticed that if a survey is updated, downloading survey updates do not include downloading updated choice lists.

Though the method of downloading updated choice lists is simple, if users are unaware/skip manual updating, they may not receive the updated choice lists before they are required. If those users are now in an area with no signal, they are now stuck using an outdated choice list.

At the very least, if checking for updating external choice lists could be incorporated in the process of updating surveys, that could help solve the issue.

There were comments in the original thread about this issue being on the radar, but it doesn't seem there have been any updates since 2021. Posting this to see if a fix is still on the radar and bring some awareness to this issue for those affected.

Thanks all.

Update: BUG-000141398 was logged in 2021 and brought up in a thread from 2022 with a similar issue. Unfortunately it seems there hasn't been additional progress since the bug was logged, but it has been elevated and another tech support case related to the bug was placed by @RobertAnderson3.

Update: Although the workaround provided by @fklotz marked as the solution doesn't solve the root issue, until an official update from Esri fixes the issue, this seems to be the best solution available!

[fklotz · Accepted solution]

Dynamic external choice lists have potential, but the limitations when we are working offline or starting a survey from other locations (e.g. ArcGIS Field Maps) are too limiting. It is annoying that Survey123 does not recognize that the external choice list has been updated or even that the survey has been updated. We've run into the same challenge as many of our folks are working far from an internet connection. Our solution is to use python to update the external choice csv in the survey's media folder (typically nightly). The users are still required to update their survey in Survey123, but at least the Survey123 App in this case recognizes that the survey has been updated and prompts the user to get the update (along with the updated csv). The python we use to update the csv started with this excellent sample by @ZacharySutherby Find it here: Survey123-tools/Update_Media_Folder at main · Esri/Survey123-tools · GitHub. 
In our case we needed some other functions in there to pull SQL server data and write the csv, but  in your case it sounds like you could grab the csv from online if it is already being managed there. Some sample code for that:

""" Download a csv hosted in ArcGIS online as a CSV Item 
    using the ArcGIS API for Python in Python 3.x
"""
import tempfile
import arcgis

# REQUIRED PARAMETERS
csv_item_id = ''

# OPTIONAL PARAMETERS
username = ''  # if blank, use Pro's login
password = ''  # required if username not blank
download_folder = ''  # if blank uses temp dir

if username:
    gis = arcgis.gis.GIS(username, password)
else:
    gis = arcgis.gis.GIS("Pro")

if gis.users.me:
    print(f"You are logged in {gis.users.me.username}")
    csv_item = gis.content.get(csv_item_id)
    if csv_item:
        if not download_folder:
            tmpdir = tempfile.TemporaryDirectory()
            download_folder = tmpdir.name
        downloaded_csv = csv_item.download(download_folder)
        print(f"Downloaded csv to {downloaded_csv}")
    else:
        csv_item_location = r'https://maps.arcgis.com/home/item.html?id=' + csv_item_id
        print(f"Failed to find the csv item at {csv_item_id}")
else:
    # note: if you have Pro installed and your active connection is not logged in with your
    #       credentials, you may still be connected to an organization as a public user...
    print("It looks like you are not logged in. Please log in and try again.")

[Vinzafy]

Python and Farley to the rescue! Just like the good ol' days. Thanks so much for your detailed response. I always appreciate how you break down the issue, propose a solution, describe why and how the solution works, and provide resources. It's always such a huge help.

It's great to know that updating the external choice CSV in the survey's media folder does trigger an update to the end-user. Thanks for sharing that. It is unfortunate that updates to linked content aren't at least packaged in with updates to surveys as I feel it's a reasonable assumption that it would be. Nevertheless, it's great to see there is at least some kind of workaround to it and that sample code and Github page are great resources. Go Python!

Appreciate your time as always Farley 😊

[RobertAnderson3]

Thank you @Vinzafy for bringing this back around. The fact this hasn't been addressed when Esri is so happy to promote the integration of their apps is baffling. It needs to be included. 

And thank you @fklotz for that work around, it's nice but definitely seems foolish to have to do it this way publishing it as form updates, really removes the purpose of linked content.

I had recently had issues with 3.17 as well where if I published the survey with a linked CSV in the media folder from testing it would not update from AGOL either. So the entire process truly needs a clean up.

I see you put in a case with Esri Support, if you are able to share an Enhancement Request or Bug number or such if they give you one please share it so others can request having it added to our accounts as well!

[Vinzafy]

My pleasure! It definitely is a shortfall and one that could cause some major issues for field folks if big updates to external choice lists are incorporated, but not manually downloaded.

Good to know as well that updating the CSV in the media folder for linked content doesn't update the AGO item. I haven't tried that method yet (I always updated the CSV in AGO directly), and knowing this, it's not something I'll try moving forward.

I have been in touch with a tech support rep. As that case evolves (and hopefully escalates to an enhancement or bug), I'll be sure to post updates and the corresponding request/bug number 🙂

[RobertAnderson3]

Even just small updates like adding a new employee or asset to the list is a major miss if not updated. Oh yeah, it definitely doesn't update the AGOL which I'm glad for. I was talking about a bug where it was preventing the survey on device from grabbing the updated AGOL item, I think 3.18 may have fixed that though.

And alright sounds good, hopefully it will quickly! 

[RobertAnderson3]

As well there was this idea from @abureaux posted on the forum, but for some reason has had minimal traction.

https://community.esri.com/t5/arcgis-survey123-ideas/survey-checks-for-updates-when-loaded-from-web/...

[Vinzafy]

Thanks for sharing the idea! Commented there as well to hopefully build traction.

[Vinzafy]

Update for everyone

I got an update from the Esri Canada Support team and it was brought up that a bug already exists for this issue. See the page below:

BUG-000141398

The bug was submitted on July 13, 2021 and currently has the status, "Under Consideration".

I put forward a case to escalate the bug which I believe is why the last modified date is today.

Current workaround is the following which is not very helpful:

---

Launch ArcGIS Survey123 without using the URL parameters.

---

Note that there is also this thread from 2022 that addresses the same issue:

Survey123 Issue - external choice list CSVs don't download with survey 

Thanks again @fklotz for the Python workaround! Currently seems to be the best option until this bug gets addressed in some fashion.

@RobertAnderson3, @ChristopherCounsell, @ahargreaves_FW, @abureaux - tagging you all here as you've had some involvement with this in the past.

[Vinzafy]

Hey @ZacharySutherby ! Tagging you here as you were on this thread back in 2022 and mentioned BUG-000141398.

Do you know if there have been any updates or further consideration regarding this bug? Any information would be much appreciated!