Updating the media folder of a Survey123 Form item

11876
74
12-07-2017 09:06 PM
Philip-Wilson
Esri Frequent Contributor
15 74 11.9K

We often get requests from people who want to update a file in the media folder of a Survey123 Form item but do not want to have to re-publish their survey every time they make a change to the file. For example, the file could be a large CSV containing address information that is constantly updated from another source, and is part of an external choice list in your survey. Or you may have several image thumbnails or audio files used in questions in your survey that need to be updated regularly. Currently the only way to update these files in your survey is to update (copy and paste) the file into the media folder for that survey on your computer, and re-publish the survey via Connect.

 

 

In this blog, I will describe a technique for updating the media folder using a Python script and how to use Windows Task Scheduler to run the Python script at a regular interval. The script will connect to ArcGIS Online or Portal, download the survey Form item, extract the zip file to a folder, copy the updated media folder files from a known location into the extracted folders, zip the Form item back up again, upload it to ArcGIS Online or Portal, resulting in the Form item being updated. Users of the field app will need to delete the survey from their gallery in the field app on the device, re-download the survey from the Download Surveys menu, open the survey and they will have access to the updated survey and associated media files.

 

Below are instructions so you can configure the Python script with your own survey and media files.

 

Configuring the Python script

 

The script requires Python 2.7 or above to run and the 'ArcREST' Python module to be installed. If you have ArcGIS Pro or ArcGIS Desktop installed, then Python 2.7 or above should be installed, however you will likely not have the ‘ArcREST’ module. Either way, you can simply install Python and then the 'ArcREST' module as follows:

 

 

Once you have Python and 'ArcREST' installed, you need to configure the script as follows:

 

  1. Download the updateMediaFolder.py script from ArcGIS Online, extract from the zip file and save to your computer. Copy the updateMediaFolder.py script to your working directory (download_folder) where you will execute the Python script from; this folder will be where the temporary folders and files will be created to update the media files in the Form item each time the script is run.
  2. Edit and save the “User Parameters” section in the updateMediaFolder.py script as follows:

itemID = ItemID of the Form item you want to update

download_folder = Local folder where the Form data will be downloaded (include a / at end of path)

updated_file = Updated file name (ie MyList.csv)

source_loc = Folder path of the updated file (include a / at end of path)

portal_url = URL address of your ArcGIS portal

credentials = ArcGIS user credentials to authenticate against the portal (NB: case sensitive)

security_type = ArcGIS portal security type (LDAP, NTLM, OAuth, Portal, PKI)

 

Once the above user parameters have been updated in the script, save your changes and you are ready to go. Simply execute the script and the Form item of your survey will be updated.

 

Please note that updating the itemsets.csv file directly in the Form item via this script does not update the values in the external_choices sheet of the associated xls workbook. It only updates the generated CSV file in the media folder. If you open the survey in Connect again, or download the survey to Connect from AGOL after you have manually updated the itemsets.csv with this script, there will be a difference between the external_choices sheet in xls workbook (original values) and the itemsets.csv (new values). If you re-publish the survey via Connect again without making any changes to external_choices sheet, the itemsets.csv will be replaced with the original values that are in the external_choices sheet.

 

If you want to schedule this script to run regularly to update your Form item, the following may assist in getting that set up.

 

Scheduling the script

 

Windows operating systems include a simple utility called Task Scheduler. Once you have determined how often your media files change and at what interval you need to update the media folder, you can repeat its execution using Task Scheduler. For example, every 60 minutes. Obviously, the computer where you have setup the task will need to be running all the time and set to run regardless of who is logged in.

 

The configuration of tasks in the scheduler are pretty much self-explanatory, but here are some specific instructions that can save you some time:

 

  1. General: Check the option to run with highest privileges and set the task to run even if you are not logged-in.
  2. Triggers: If you want to quickly test your task, you can simply select your task in the gallery and then hit Run in the Selected Item panel on the right. When configuring the task for real, I suggest you select the startup trigger and that you also configure the task to run indefinitely.
  3. Actions: You will need to be particularly careful with this one. The Program/Script setting needs to point to the Python executable.
  • If using Python from ArcGIS Pro, it will be under the Pro installation directory. For example: C: \Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe"
  • If using your own copy of Python, refer to the installation directory of Python.

 

You also need to indicate and pass the location of your updateMediaFolder.py script as an argument. Do not forget to include the .py extension in the path. Lastly, set the 'Start in' property so it points to the directory in which you are storing the updateMediaFolder.py file.

 

 

Beyond the basics

 

The technique described in this blog post is simple but effective and will enable you to update files in the media folder of your survey without having to re-publish via Connect. However, please note that all care should be taken to ensure the files you are updating with this script have the exact name as the files in the originally published media folder. Also, the files should have the same format, with the same field names as the original files (in the case of CSV files). Only rows of data should be updated or additional rows added. If you want to change the format of the files (rename or add columns) you should update the files via Connect in the media folder, update the survey xls form, and then re-publish the survey to ensure the changes do not break anything.

 

It is recommended to test the script on a backup copy of your survey and ensure the survey can be downloaded and updated in the field app, checking that the external choice lists and/or media items are working as expected. This should be confirmed before running the script and updating the media folder and files on your real survey that is currently in use by other Survey123 field app users.

74 Comments
CuartaOportunidad_Local_IVC
New Contributor

Hi All!, I have a question

Currently, I am using the following python lines to update a form. What I want to update the most is the media file of it, so I the survey can do the pulldata of the new registers. 

I run this lines:

#Getting the survey Item I will update

org_gis = GIS('my_http', 'my_user', 'my_pswrd')
srv_to_update = org_gis.content.get('srv_to_update_id')

#The path of the survey, with the new media file

path = ".../ArcGIS/My Survey Designs/survey.zip"

#Update
srv_to_update.update(data = path, item_properties={'type' : 'Form', 'title': 'survey', 'tags' : []})

I get 

True as output.

However, when I go to the "updated" survey in my portal the pulldata of the new registers is not working.

Any clue of why is this happening? Should I include another parameter in the update function?

Thank you all in advance!

Regards!

RDIApps
New Contributor

I apologize in advance if this has been covered in this thread already, I did search it and didn't find an answer.. I have a need to use this script and currently have 10.6 installed and the py2.7 version that comes with that. I installed pip and installed the arcrest package, I can import that fine, but can't import arcresthelper.. Tried a pip install arcresthelper and that didn't work. Suggestions? probably something simple I am missing here?

Thanks!

RDIApps
New Contributor

Found this thread: https://community.esri.com/thread/166523 

Seems like my pip install arcrest did not work as intended? This is what my site-packages folder looks like..

JamesTedrick
Esri Esteemed Contributor

Hi Luisa,

The code looks correct.  Can you verify what the name of zip file is?  When updating, it should be the same name as it was initially published as.  You could find this out from the item's 'name' property.

CuartaOportunidad_Local_IVC
New Contributor

Dear James,

I did new proofs and I found that the survey is in fact updated, but the pulldata does not work in Survey123 Navigator. It does work for Survey123 DeviceApp and the DesktopApp.

I wonder why.

Thanks in advance for your answer!

Sofia

JamesTedrick
Esri Esteemed Contributor

Hi Luisa,

You are indicating that the pulldata is not being updated in the Survey123 web app, correct? 

CuartaOportunidad_Local_IVC
New Contributor

Yes, that is what is happening

CuartaOportunidad_Local_IVC
New Contributor

Actually, I do not know if it is not updated or if somehow when using the Survey123 web app the updated media folder is unread. Because using the other apps, Survey123 Connect and Survey123 in a movil device I have the updated pulldata information. 

CoryBolen1
New Contributor III

Philip Wilson‌ or James Tedrick‌:

Apologies if this has been covered.  I skimmed thru the previous threads, but nothing got me any further as my python skills are limited.  I am trying to run the above script specifically to update a media file folder within S123 as intended.  I defined my user parameters, the script is saved in my specified download folder, the updated .csv file has the same name / format as the original and I downloaded the ArcRest module.  I have Python 2.7.16 and ArcGIS 10.7.1 installed.  I tried running the script both within IDLE and Python window within ArcCatalog / ArcMap.  I get the following error:

...Starting

...Authenticating

...Querying 

Traceback (most recent call last):

File "C:\S123_Download\updateMediaFolder.py", line 105, in <module>

   zipFileName = serviceInfo['name']

KeyError: 'name'

>>>

Below are my defined user parameters minus AGO credentials:

 

Any thoughts on where I am going wrong?

JamesTedrick
Esri Esteemed Contributor

Hi Cory,

That's fairly unusual - that error would indicate either the login did not succeed (perhaps check the userName / passWord values, which you did not include above for obvious reasons) or the form item doesn't have a 'name' property (which all items should have).

CoryBolen1
New Contributor III

Hi James - Thank you for your reply.  I confirmed the AGOL credentials and the form item ID are correct in the user parameters.  The form does have a assigned name (Ark_DailyOpsLog_20191106) which I wouldn't think would cause any problems.  Is the 'Name' property something different than what I name the actual form in S1223 Connect?  If so, how would I check this?

JamesTedrick
Esri Esteemed Contributor

The 'name' property should be the same as you have in Survey123 Connect (unless you renamed it after publishing).  One thing you could try is inserting a line above line #105 that is 

print(serviceInfo)

To see that everything is there.

YichiLiu
New Contributor

Thanks for the script! Just curious, can the same process be done in Python API for ArcGIS instead of ArcREST? 

Thank you!

JamesTedrick
Esri Esteemed Contributor

Hi Yichi,

Yes, the ArcREST component deal with authentication and download/upload of the zip file - you can swap those out for equivalent Python API functions.

MarkEastwood
New Contributor III

eibarra_geotec  Were you ever able to figure this out? I am getting the same 'BadZipfile: File is not a zip file' error

MarkEastwood
New Contributor III

I updated line 112 data_url = "http://...." to data_url = "https://..." and the error went away.

YichiLiu
New Contributor

Thank you James. I got it working in Python API. However, it looks like the data updates only shows up in the mobile app, but not on the website. Is that expected? I'm using pulldata() to autopopulate some of the form. The file I'm updating in the media folder is a csv file that the pulldata() function is reading data from.

This maybe a different issue, but I've ran into the same problem with using username type and pulldata(). I wanted to display information based on the user names used to sign in, but it would only work in Survey 123 Connect and mobile app, but not the website version.

JamesTedrick
Esri Esteemed Contributor

Hi Yichi,

The csv file should be updated as well in the webform - could you try clearing browser caches to check if that was the issue?  

You shouldn't need to use pulldata to get the username - you can use a field with the type of 'username' to automatically fill that in based on their signed-in credentials.

YichiLiu
New Contributor

Hi James,

Thanks for the quick reply. Yes I am using the type of 'username' and then use the user name to pull data from a csv file. The purpose is to make the form only showing information customized to the specific user. 

I've tried clearing browser, used different browsers and computers. All of them didn't work. 

However, I tried to download the form to Survey 123 Connect. Without editing anything, published it gain. This time, it worked. But this defeats the purpose of using a script to auto update the csv file, since these was still manual publishing involved. 

I'm not sure if there is anything else I could try. Any advice would be greatly appreciated! 

EvenorIbarra
New Contributor

I still have not been able to solve this problem, I developed an

application with xamarín to generate the itemsets.csv file from the device,

I have a service where I update all the catalog lists that I need in the

mobile application, I update it every 30 seconds and look for changes in

the items, so I update the csv locally.

El lun., 9 dic. 2019 a las 15:45, M E (<geonet@esri.com>) escribió:

GeoNet, The Esri Community | GIS and Geospatial Professional Community

<https://community.esri.com/?et=blogs.comment.created>

Updating the media folder of a Survey123 Form item

new comment by M E

<https://community.esri.com/people/a00847138?et=blogs.comment.created> - View

all comments on this blog post

<https://community.esri.com/groups/survey123/blog/2017/12/08/an-alternative-way-to-update-the-media-folder-of-a-survey123-form-item?commentID=80856&et=blogs.comment.created#comment-80856>

JamesTedrick
Esri Esteemed Contributor

Hi Yichi,

Just to check, after uploading the new file, can you download the zip file again and verify the new CSV file is present?  Also, the files are zipped within the esriinfo folder, correct?  This is important to make sure the resources the web form uses is updated.

BamJam
by
New Contributor III

I ran into this issue and was able to resolve it by removing the request to be returned in json format.  So, change the url in the request:

Original line: data_url = "<portalurl>/<webadaptorname>/sharing/rest/content/items/{0}/data/?f=json&token={1}".format(itemID, agol_helper.token)
New line: data_url = "<portalurl>/<webadaptorname>/sharing/rest/content/items/{0}/data/?token={1}".format(itemID, agol_helper.token)
AlexandraWashburn
New Contributor III

Hi!

I am also one of the folks that had trouble with the "bad ZIP file" error using this script with ArcGIS Enterprise, but after a few modifications, I was able to get it to work! I'll drop the full code into this comment, but to summarize the changes:

* Replace references to arcgis.com with portal_url variable

* Added James Tedrick's alternate zip file write function

* Removed f=json from data_url

* Added exception in uploadFile if security parameters were invalid (currently it just prints a message and tells you it was successful even though it was not)

I also had to make a configuration change to my Portal. In portaladmin/security/config, I had to change "disableServicesDirectory" in the security configuration from true to false. When it was disabled, I was getting the bad ZIP file error like everyone else, but when it's enabled, the process runs successfully. Can anyone tell me why this is happening? I'd like to leave the services directory as a security best practice, but I also need this process to work to accommodate our business needs.

Also, with the new version of Survey123, you get an update notification next time you open the app. Once the script executes, the top of the screen on Survey123 will say "Updates available: 1" and you can tap that to refresh the survey! It is no longer necessary to delete/re-download the survey.

Here's the updated code!

#-------------------------------------------------------------------------------
# Name:        updateMediaFolder
# Purpose:     Update the contents of the Media folder of a Survey123 Form item
# Author:      Philip Wilson, Esri
# Created:     04-Sep-2017
# Copyright:   (c) Esri 2017
# Licence:     Apache 2.0
#-------------------------------------------------------------------------------

import ago
import urllib
import urllib2
import zipfile
import shutil
import os
import json
import arcrest
from arcresthelper import securityhandlerhelper

#-------------------------------------------------------------------------------

# User Parameters

# ItemID of the Form item you want to update
itemID = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
# Local folder where the Form data will be downloaded (include a / at end of path)
download_folder = "C:/Temp/"
# Updated file name (ie MyList.csv)
updated_file = "XXXXXXXX.csv"
# Folder path of the updated file (include a / at end of path)
source_loc = "C:/Temp/"
# Address of your ArcGIS portal (ie https://myportal.domain.com/webadaptor)
portal_url = r"https://www.arcgis.com/"
# ArcGIS user credentials to authenicate against the portal (case sensitive)
credentials = {'userName' : 'XXXXXXXXXXXXX''passWord' : 'XXXXXXX'}
# ArcGIS portal security type (LDAP, NTLM, OAuth, Portal, PKI)
# You may need to change additional parameters in the uploadFile function depending on the security type
security_type = "Portal"

#-------------------------------------------------------------------------------

def getServiceDefinition(tokenitemID😞
    '''Gets the JSON representation of the service definition.'''
    response = urllib.urlopen(portal_url+"/sharing/rest/content/items/{0}?f=json&token={1}".format(itemID, token)).read()
    serviceInfo = json.loads(response)
    return serviceInfo  

def downloadFile(urlfilename😞
    print ("...Downloading")
    req = urllib2.urlopen(url)
    with open(filename, 'wb'as fp:
        fp.write(req.read())

def extractZIP(filename,folder😞
    print ("...Extracting")
    zfile = zipfile.ZipFile(filename)
    zfile.extractall(folder)

def uploadFile():
    proxy_port = None
    proxy_url = None

    securityinfo = {}
    securityinfo['security_type'] = security_type
    securityinfo['username'] = credentials['userName']
    securityinfo['password'] = credentials['passWord']
    securityinfo['org_url'] = portal_url
    securityinfo['proxy_url'] = proxy_url
    securityinfo['proxy_port'] = proxy_port
    securityinfo['referer_url'] = None
    securityinfo['token_url'] = None
    securityinfo['certificatefile'] = None
    securityinfo['keyfile'] = None
    securityinfo['client_id'] = None
    securityinfo['secret_id'] = None

    upload_file = download_folder + zipFileName

    shh = securityhandlerhelper.securityhandlerhelper(securityinfo)
    if shh.valid == False:
        raise Exception(shh.message)
    else:
        portalAdmin = arcrest.manageorg.Administration(securityHandler=shh.securityhandler)
        item = portalAdmin.content.getItem(itemId=itemID).userItem

        itemParams = arcrest.manageorg.ItemParameter()

        itemParams.filename = upload_file
        res = item.updateItem(itemParameters=itemParams, data=upload_file)

print ("...Starting")
# initialize the portal helper class
# ago.py is part of the 10.3 python install
agol_helper = ago.AGOLHelper(portal_url)
print ("...Authenticating")
# login
agol_helper.login(credentials['userName'], credentials['passWord'])

print ("...Querying")
# get the name of the zip file
serviceInfo = getServiceDefinition(agol_helper.token, itemID)
zipFileName = serviceInfo['name']
print ("Zip file name: " + zipFileName)

# download the zip file
data_url = portal_url+"/sharing/rest/content/items/{0}/data/?token={1}".format(itemID, agol_helper.token)
download_file = download_folder + zipFileName
downloadFile(data_url, download_file)
print (zipFileName + " downloaded to: " + download_folder)

# extract the contents of the zip file
extractZIP(download_file, download_folder + "extract/")
print (zipFileName + " extracted to: " + download_folder + "extract/")

print ("...Updating")
# copy the updated file into extracted Media folder, overwite existing
source_file = source_loc + updated_file
dest_file = download_folder + "extract/esriinfo/media/" + updated_file
shutil.copyfile(source_file, dest_file)
print (updated_file + " updated to: " + download_folder + "extract/esriinfo/media/")

print ("...Cleaning")
# delete the downloaded zip file before create new zip file
os.remove(download_file)
print (zipFileName + " deleted from: " + download_folder)

print ("...Compressing")
# compress the contents of esriinfo into a new zip file
zf = zipfile.ZipFile(zipFileName, "w")
os.chdir(download_folder + 'extract')
for dirname, subdirs, files in os.walk('esriinfo'😞
  for filename in files:
    zf.write(os.path.join(dirname, filename))
zf.close()
os.chdir(download_folder)
print (zipFileName + " created in: " + download_folder)

print ("...Uploading")
# upload the zip file to ArcGIS portal
uploadFile()
print (zipFileName + " uploaded")

print ("...Cleaning")
# delete the updated zip file after upload
os.remove(download_file)
print (zipFileName + " deleted from: " + download_folder)
# delete the extracted esriinfo folder
shutil.rmtree(download_folder + "extract/")
print ("/extract/ folder deleted from: " + download_folder)

print ("...Finished")
# final successful print message
print (zipFileName + " successfully updated with " + source_file + " and uploaded to your ArcGIS portal!")
LeeIrminger1
New Contributor II

I am following the method by @CuartaOportunidad_Local_IVC  above- see direct link here

 
I am wanting to update the media folder for three CSVs that are accessed with pulldata(). 
 
The method works and changes are picked up in the Survey123 Field App on the desktop but not in the browser. The browser is only shows updates published from Survey123 Connect. Should I do something to have it update the browser? I cleared cache and ensured that it is updating with the same name as the original name of the survey. I also downloaded the survey as a zip and verified the csv reflected the change I made.
 

shutil.make_archive(r'temp' + 'filename', 'zip', r'surveyfolder')

path = r'zippedsurvey'

surveyitemID = "itemid"

surveytoupdate = my_gis.content.get(surveyitemID)

surveytoupdate.update(data = path, item_properties={'type' : 'Form', 'title': 'filename', 'tags' : []})

thank you!