Updating the media folder of a Survey123 Form item
We often get requests from people who want to update a file in the media folder of a Survey123 Form item but do not want to have to re-publish their survey every time they make a change to the file. For example, the file could be a large CSV containing address information that is constantly updated from another source, and is part of an external choice list in your survey. Or you may have several image thumbnails or audio files used in questions in your survey that need to be updated regularly. Currently the only way to update these files in your survey is to update (copy and paste) the file into the media folder for that survey on your computer, and re-publish the survey via Connect.
In this blog, I will describe a technique for updating the media folder using a Python script and how to use Windows Task Scheduler to run the Python script at a regular interval. The script will connect to ArcGIS Online or Portal, download the survey Form item, extract the zip file to a folder, copy the updated media folder files from a known location into the extracted folders, zip the Form item back up again, upload it to ArcGIS Online or Portal, resulting in the Form item being updated. Users of the field app will need to delete the survey from their gallery in the field app on the device, re-download the survey from the Download Surveys menu, open the survey and they will have access to the updated survey and associated media files.
Below are instructions so you can configure the Python script with your own survey and media files.
Configuring the Python script
The script requires Python 2.7 or above to run and the 'ArcREST' Python module to be installed. If you have ArcGIS Pro or ArcGIS Desktop installed, then Python 2.7 or above should be installed, however you will likely not have the ‘ArcREST’ module. Either way, you can simply install Python and then the 'ArcREST' module as follows:
Once you have Python and 'ArcREST' installed, you need to configure the script as follows:
- Download the updateMediaFolder.py script from ArcGIS Online, extract from the zip file and save to your computer. Copy the updateMediaFolder.py script to your working directory (download_folder) where you will execute the Python script from; this folder will be where the temporary folders and files will be created to update the media files in the Form item each time the script is run.
- Edit and save the “User Parameters” section in the updateMediaFolder.py script as follows:
itemID = ItemID of the Form item you want to update
download_folder = Local folder where the Form data will be downloaded (include a / at end of path)
updated_file = Updated file name (ie MyList.csv)
source_loc = Folder path of the updated file (include a / at end of path)
portal_url = URL address of your ArcGIS portal
credentials = ArcGIS user credentials to authenticate against the portal (NB: case sensitive)
security_type = ArcGIS portal security type (LDAP, NTLM, OAuth, Portal, PKI)
Once the above user parameters have been updated in the script, save your changes and you are ready to go. Simply execute the script and the Form item of your survey will be updated.
Please note that updating the itemsets.csv file directly in the Form item via this script does not update the values in the external_choices sheet of the associated xls workbook. It only updates the generated CSV file in the media folder. If you open the survey in Connect again, or download the survey to Connect from AGOL after you have manually updated the itemsets.csv with this script, there will be a difference between the external_choices sheet in xls workbook (original values) and the itemsets.csv (new values). If you re-publish the survey via Connect again without making any changes to external_choices sheet, the itemsets.csv will be replaced with the original values that are in the external_choices sheet.
If you want to schedule this script to run regularly to update your Form item, the following may assist in getting that set up.
Scheduling the script
Windows operating systems include a simple utility called Task Scheduler. Once you have determined how often your media files change and at what interval you need to update the media folder, you can repeat its execution using Task Scheduler. For example, every 60 minutes. Obviously, the computer where you have setup the task will need to be running all the time and set to run regardless of who is logged in.
The configuration of tasks in the scheduler are pretty much self-explanatory, but here are some specific instructions that can save you some time:
- General: Check the option to run with highest privileges and set the task to run even if you are not logged-in.
- Triggers: If you want to quickly test your task, you can simply select your task in the gallery and then hit Run in the Selected Item panel on the right. When configuring the task for real, I suggest you select the startup trigger and that you also configure the task to run indefinitely.
- Actions: You will need to be particularly careful with this one. The Program/Script setting needs to point to the Python executable.
- If using Python from ArcGIS Pro, it will be under the Pro installation directory. For example: C: \Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe"
- If using your own copy of Python, refer to the installation directory of Python.
You also need to indicate and pass the location of your updateMediaFolder.py script as an argument. Do not forget to include the .py extension in the path. Lastly, set the 'Start in' property so it points to the directory in which you are storing the updateMediaFolder.py file.
Beyond the basics
The technique described in this blog post is simple but effective and will enable you to update files in the media folder of your survey without having to re-publish via Connect. However, please note that all care should be taken to ensure the files you are updating with this script have the exact name as the files in the originally published media folder. Also, the files should have the same format, with the same field names as the original files (in the case of CSV files). Only rows of data should be updated or additional rows added. If you want to change the format of the files (rename or add columns) you should update the files via Connect in the media folder, update the survey xls form, and then re-publish the survey to ensure the changes do not break anything.
It is recommended to test the script on a backup copy of your survey and ensure the survey can be downloaded and updated in the field app, checking that the external choice lists and/or media items are working as expected. This should be confirmed before running the script and updating the media folder and files on your real survey that is currently in use by other Survey123 field app users.