Updating the media folder of a Survey123 Form item

13650
76
12-07-2017 09:06 PM
Philip-Wilson
Esri Frequent Contributor
15 76 13.6K

We often get requests from people who want to update a file in the media folder of a Survey123 Form item but do not want to have to re-publish their survey every time they make a change to the file. For example, the file could be a large CSV containing address information that is constantly updated from another source, and is part of an external choice list in your survey. Or you may have several image thumbnails or audio files used in questions in your survey that need to be updated regularly. Currently the only way to update these files in your survey is to update (copy and paste) the file into the media folder for that survey on your computer, and re-publish the survey via Connect.

 

 

In this blog, I will describe a technique for updating the media folder using a Python script and how to use Windows Task Scheduler to run the Python script at a regular interval. The script will connect to ArcGIS Online or Portal, download the survey Form item, extract the zip file to a folder, copy the updated media folder files from a known location into the extracted folders, zip the Form item back up again, upload it to ArcGIS Online or Portal, resulting in the Form item being updated. Users of the field app will need to delete the survey from their gallery in the field app on the device, re-download the survey from the Download Surveys menu, open the survey and they will have access to the updated survey and associated media files.

 

Below are instructions so you can configure the Python script with your own survey and media files.

 

Configuring the Python script

 

The script requires Python 2.7 or above to run and the 'ArcREST' Python module to be installed. If you have ArcGIS Pro or ArcGIS Desktop installed, then Python 2.7 or above should be installed, however you will likely not have the ‘ArcREST’ module. Either way, you can simply install Python and then the 'ArcREST' module as follows:

 

 

Once you have Python and 'ArcREST' installed, you need to configure the script as follows:

 

  1. Download the updateMediaFolder.py script from ArcGIS Online, extract from the zip file and save to your computer. Copy the updateMediaFolder.py script to your working directory (download_folder) where you will execute the Python script from; this folder will be where the temporary folders and files will be created to update the media files in the Form item each time the script is run.
  2. Edit and save the “User Parameters” section in the updateMediaFolder.py script as follows:

itemID = ItemID of the Form item you want to update

download_folder = Local folder where the Form data will be downloaded (include a / at end of path)

updated_file = Updated file name (ie MyList.csv)

source_loc = Folder path of the updated file (include a / at end of path)

portal_url = URL address of your ArcGIS portal

credentials = ArcGIS user credentials to authenticate against the portal (NB: case sensitive)

security_type = ArcGIS portal security type (LDAP, NTLM, OAuth, Portal, PKI)

 

Once the above user parameters have been updated in the script, save your changes and you are ready to go. Simply execute the script and the Form item of your survey will be updated.

 

Please note that updating the itemsets.csv file directly in the Form item via this script does not update the values in the external_choices sheet of the associated xls workbook. It only updates the generated CSV file in the media folder. If you open the survey in Connect again, or download the survey to Connect from AGOL after you have manually updated the itemsets.csv with this script, there will be a difference between the external_choices sheet in xls workbook (original values) and the itemsets.csv (new values). If you re-publish the survey via Connect again without making any changes to external_choices sheet, the itemsets.csv will be replaced with the original values that are in the external_choices sheet.

 

If you want to schedule this script to run regularly to update your Form item, the following may assist in getting that set up.

 

Scheduling the script

 

Windows operating systems include a simple utility called Task Scheduler. Once you have determined how often your media files change and at what interval you need to update the media folder, you can repeat its execution using Task Scheduler. For example, every 60 minutes. Obviously, the computer where you have setup the task will need to be running all the time and set to run regardless of who is logged in.

 

The configuration of tasks in the scheduler are pretty much self-explanatory, but here are some specific instructions that can save you some time:

 

  1. General: Check the option to run with highest privileges and set the task to run even if you are not logged-in.
  2. Triggers: If you want to quickly test your task, you can simply select your task in the gallery and then hit Run in the Selected Item panel on the right. When configuring the task for real, I suggest you select the startup trigger and that you also configure the task to run indefinitely.
  3. Actions: You will need to be particularly careful with this one. The Program/Script setting needs to point to the Python executable.
  • If using Python from ArcGIS Pro, it will be under the Pro installation directory. For example: C: \Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe"
  • If using your own copy of Python, refer to the installation directory of Python.

 

You also need to indicate and pass the location of your updateMediaFolder.py script as an argument. Do not forget to include the .py extension in the path. Lastly, set the 'Start in' property so it points to the directory in which you are storing the updateMediaFolder.py file.

 

 

Beyond the basics

 

The technique described in this blog post is simple but effective and will enable you to update files in the media folder of your survey without having to re-publish via Connect. However, please note that all care should be taken to ensure the files you are updating with this script have the exact name as the files in the originally published media folder. Also, the files should have the same format, with the same field names as the original files (in the case of CSV files). Only rows of data should be updated or additional rows added. If you want to change the format of the files (rename or add columns) you should update the files via Connect in the media folder, update the survey xls form, and then re-publish the survey to ensure the changes do not break anything.

 

It is recommended to test the script on a backup copy of your survey and ensure the survey can be downloaded and updated in the field app, checking that the external choice lists and/or media items are working as expected. This should be confirmed before running the script and updating the media folder and files on your real survey that is currently in use by other Survey123 field app users.

76 Comments
TL2
by
Occasional Contributor III

This will be VERY useful. Thank you!

SarahLidster
Esri Contributor

Philip Wilson‌, thank you for sharing this. I believe the script you provided will be very useful. Can you clarify if this approach would work on the csv file that is created using the external_choices tab in the XLS form? From my testing I have not been able to update the generated 'itemsets.csv' file and see those changes reflected in my survey. To see changes, I have to modify the external_choices tab in my survey and then republish. Is the approach you provided intended for csv files used with pulldata()?

by Anonymous User
Not applicable

I cannot wait to try this out on some of my surveys as this would help close the loop on a process I am attempting to automate. The idea is to use another posted script that will download the service, export the information into a csv file, and then use this script to update a series of forms that use the same csv for external selects.

Thank you very much for posting this.

Philip-Wilson
Esri Frequent Contributor

Hi Sarah,

Yes, this script can be used to update the itemsets.csv file which is created when you have an external_choices list in your xls form. It can also be used for standalone CSV files and other media files that exist in media folder that are used in pulldata() calculations.

Please note that updating the itemsets.csv file directly in the Form item via this script does not update the values in the external_choices sheet of the associated xls workbook. It only updates the generated CSV file in the media folder. If you open the survey in Connect again, or download the survey to Connect from AGOL after you have manually updated the itemsets.csv with this script, there will be a difference between the external_choices sheet in xls workbook (original values) and the itemsets.csv (new values). If you re-publish the survey via Connect again without making any changes to external_choices sheet, the itemsets.csv will be replaced with the original values that are in the external_choices sheet.

I believe the reason you are not seeing updated values in the survey could be the following. If you are updating the itemsets.csv manually via Excel or another text editor, you need to ensure the format of the CSV remains the same. I noticed that when using Excel and saving the CSV file, it strips out the quotations which means the CSV file can not be read in the field app. The field app is expecting the quotations, as this is how Connect saves the original file format. See the following example:

Original itemsets.csv file created by Connect:

 

itemsets.csv file modified and saved in Excel (note I modified row 6 (value5) and added an extra row 11 (value10):

In the above example Excel removed all the quotations. If I save the same file ensuring that all the values have quotations, then update the Form item using the script, I am able to see the updated values in the field app. Correct format itemsets.csv file:

After the script is successfully run, you need to delete the survey in the field app, re-download the updated survey, and you should see the updated values. My survey before update:

 

Survey after update:

 

Hope this helps.

Regards,

Phil.

SarahLidster
Esri Contributor

Philip,

Thank you for this detailed feedback. The issue I was experiencing was related to an incorrect csv format when I was editing the itemsets.csv. Now that I am updating the csv with the correct format, the script is working successfully.

Thank you so much for your help,

Sarah

somashekarChandrashekaraiah1
New Contributor

I am trying to implement this solution in our project which a similar requirement to update media file.

I have some questions. I was trying to setup the environment for running the provided script.

1. Do we need ArcGIS Desktop environment to run this piece of script ?

I have problem in below step

Download a geoprocessing package to install ArcREST!

Download the GeoProcessing Package here: https://github.com/Esri/ArcREST/blob/master/ArcGIS%20Desktop%20Installer/installing_arcrest.gpk * Note: ArcMap/Catalog is required to use the gpk format. * Note: this tools can be used to update ArcREST as well 1. Open the GPK in the ArcGIS Desktop Product of your choosing. 2. Run the tool by double clicking on the tool icon. 3. Test the import
How to resolve this package on a virtual environment without arcgis desktop >

2. using pip to restore/install some package like urllib, urllib2, json was not successful.
I am using python 3.6 ?
your guidance will be very helpful to implement this solution instead, me exploring the options.

Thanks,

som

Philip-Wilson
Esri Frequent Contributor

Hi Somashekar,

Yes, you can run the script without ArcGIS Desktop installed provided you have Python installed and the required modules. You should be able to install ArcREST without needed to do it via Desktop.

Phil.

SappiGIS
New Contributor III

Hi Phil,

Sorry to tag onto the last reply here but it's last item I tried to resolve installing the ArcREST without using ArcGIS desktop after failing with process you initially described above.

I'm running ArcGIS 10.6 and downloaded ArcREST Version 3.5.9 Release.

After getting an error I tried the suggestion of changing the PATH as described in the installation guide you linked to:

  • Note: If you have not done so, you may need to add your python install path and scripts folder to your environment variables. In your system PATH environment variable, add both the path to Python and the Python Scripts folder. ex: C:\Python27\ArcGIS10.3;C:\Python27\ArcGIS10.3\Scripts

And I've tried running the scrip via arcgis desktop and then through ArcCatalog with the source setting of "Run Python script in process" unchecked and both give errors (in order below as 1 and 2):

1. Executing: installarcrest false true
Start Time: Thu Mar 15 13:43:58 2018
Running script installarcrest...
Processing C:\Python27\ArcGIS10.6\Lib\site-packages
... Adding the ArcRest Module ...
('line 107', u'C:\\Users\\syspmb-arcgis\\Documents\\ArcGIS\\Packages\\installing_arcrest_5244CB87-E044-400D-A5B6-491F41115565\\v104\\scripts\\install_arcrest.py', 'BadZipfile: File is not a zip file')
The modules has been loaded to C:\Python27\ArcGIS10.6\Lib\site-packages ...

-----------------------------------------------

Processing C:\Python27\ArcGISx6410.6\Lib\site-packages
... Adding the ArcRest Module ...
('line 107', u'C:\\Users\\syspmb-arcgis\\Documents\\ArcGIS\\Packages\\installing_arcrest_5244CB87-E044-400D-A5B6-491F41115565\\v104\\scripts\\install_arcrest.py', 'BadZipfile: File is not a zip file')
The modules has been loaded to C:\Python27\ArcGISx6410.6\Lib\site-packages ...

... Process Complete ...
Completed script installarcrest...
Failed to execute (installarcrest).
Failed at Thu Mar 15 13:43:58 2018 (Elapsed Time: 0.03 seconds)

2. Executing: installarcrest false true
Start Time: Thu Mar 15 15:11:09 2018
Running script installarcrest...
ERROR 000714: Error in script installarcrest.
Error in executing: cmd.exe /C D:\Software\ArcREST-3.5.9\ArcGIS Desktop Installer\Install_ArcRest\Tools\Scripts\install_arcrest.py "false" "true"

Failed to execute (installarcrest).
Failed at Thu Mar 15 15:11:09 2018 (Elapsed Time: 0.10 seconds)

So I then tried the method in your latest reply here and installed via the command line. I then get this message:

C:\Users\syspmb-arcgis>C:\Users\syspmb-arcgis\Documents\ArcGIS\Packages\installing_arcrest_5244CB87-E044-400D-A5B6-491F41115565\v104\scripts\install_arcrest.py
Processing C:\Python27\ArcGIS10.6\lib\site-packages
... Adding the ArcRest Module ...
ERROR: ('line 107', 'C:\\Users\\syspmb-arcgis\\Documents\\ArcGIS\\Packages\\installing_arcrest_5244CB87-E044-400D-A5B6-491F41115565\\v104\\scripts\\install_arcrest.py', 'BadZipfile: File is not a zip file')
('line 107', 'C:\\Users\\syspmb-arcgis\\Documents\\ArcGIS\\Packages\\installing_arcrest_5244CB87-E044-400D-A5B6-491F41115565\\v104\\scripts\\install_arcrest.py', 'BadZipfile: File is not a zip file')
The modules has been loaded to C:\Python27\ArcGIS10.6\lib\site-packages ...

... Process Complete ...

Please advise if this is correct and how to be sure that the module has been correctly installed. Apologies for the wall of text.

Thanks

Greg

LeonelRivero
New Contributor II

Hi Phil,

How should I configure the script for use in Portal for ArcGIS?

I configure the url and the credentials, but it gives me an authentication error

Error:

Traceback (most recent call last):
File "E:\Survey123_Actualizar_Pulldata\ActualizarPulldata.py", line 100, in <module>
agol_helper.login(credentials['userName'], credentials['passWord'])
File "C:\Program Files (x86)\ArcGIS\Desktop10.5\ArcToolBox\Scripts\ago.py", line 295, in login
self.information()
File "C:\Program Files (x86)\ArcGIS\Desktop10.5\ArcToolBox\Scripts\ago.py", line 360, in information
self.portal_name = portal_info['portalName']
KeyError: 'portalName'

JamesTedrick
Esri Esteemed Contributor

Hi Leonel,

Which version of Portal are you using?

LeonelRivero
New Contributor II

Hi James,

The portal version is 10.5

JamesTedrick
Esri Esteemed Contributor

Hi Leonel,

This looks to be an issue with the underlying arcrest library.  Can you go to the portal's self REST endpoint (i.e., http://gis.server.com/portal/sharing/rest/porals/self?f=pjson ) and look for the portalName property?  Does it have non-ascii characters?  That might cause the program to break.

LeonelRivero
New Contributor II

Hi James,

Thank you very much for the information, I managed with the url at least jump the authentication error, however I throw a fault on the line 106

105 serviceInfo = getServiceDefinition(agol_helper.token, itemID)

106 zipFileName = serviceInfo['name']

error:

Traceback (most recent call last):
File "E:\Survey123_Actualizar_Pulldata\ActualizarPulldata.py", line 106, in <module>
zipFileName = serviceInfo['name']
KeyError: 'name'

JamesTedrick
Esri Esteemed Contributor

Hi Leonel,

In order to debug, I would suggest inserting a 

print(str(serviceInfo))

if running via ArcGIS Pro/ArcMap, you would instead:

import arcpy
arcpy.addMessage(str(serviceInfo))

to examine what is being returned. 

LeonelRivero
New Contributor II

Hi James,

 

This paste code in ArcGIS Pro 

error:

import arcpy
arcpy.addMessage(str(serviceInfo))
Traceback (most recent call last):
File "<string>", line 2, in <module>
AttributeError: module 'arcpy' has no attribute 'addMessage'

JamesTedrick
Esri Esteemed Contributor

Hi Leonel,

Sorry, the function is AddMessage, not addMessage - AddMessage—Help | ArcGIS Desktop 

ShanonLoughton
Esri Contributor

Hi

I am struggling with this updating correctly. Most of the script seems to run. But nothing gets updated in the survey media folder.

Is it related to "* - attribute not implemented in Portal class." lines below?

OS: Windows 10 64bit

ArcGIS for Desktop 10.3.1

Python 2.7.8

ArcREST 3.5.9

Updating against ArcGIS Online

Survey created in Survey123 Connect 2.6.4

Testing in Survey123 Windows 2.6.7

C:\Python27>python ./updateMediaFolder.py
...Starting
...Authenticating
...Querying
Zip file name: Form_1.zip
...Downloading
Form_1.zip downloaded to: C:/Users/username/ArcGIS/My Survey Designs/Form 1-Download/
...Extracting
Form_1.zip extracted to: C:/Users/username/ArcGIS/My Survey Designs/Form 1-Download/extract/
...Updating
itemsets.csv updated to: C:/Users/username/ArcGIS/My Survey Designs/Form 1-Download/extract/esriinfo/media/
...Cleaning
Form_1.zip deleted from: C:/Users/username/ArcGIS/My Survey Designs/Form 1-Download/
...Compressing
Form_1.zip created in: C:/Users/username/ArcGIS/My Survey Designs/Form 1-Download/
...Uploading
notificationsEnabled - attribute not implemented in Portal class.
vectorBasemapGalleryGroupQuery - attribute not implemented in Portal class.
canSetCustomBuyLink - attribute not implemented in Portal class.
eueiEnabled - attribute not implemented in Portal class.
canSetQuestionnaire - attribute not implemented in Portal class.
useVectorBasemaps - attribute not implemented in Portal class.
allowedOrigins - attribute not implemented in Portal class.
defaultVectorBasemap - attribute not implemented in Portal class.
contentCategorySetsGroupQuery - attribute not implemented in Portal class.
isVerified - attribute not implemented in Portal class.
allowedRedirectUris - attribute not implemented in Portal class.
notificationsEnabled - attribute not implemented in Portal class.
vectorBasemapGalleryGroupQuery - attribute not implemented in Portal class.
canSetCustomBuyLink - attribute not implemented in Portal class.
eueiEnabled - attribute not implemented in Portal class.
canSetQuestionnaire - attribute not implemented in Portal class.
useVectorBasemaps - attribute not implemented in Portal class.
allowedOrigins - attribute not implemented in Portal class.
defaultVectorBasemap - attribute not implemented in Portal class.
contentCategorySetsGroupQuery - attribute not implemented in Portal class.
isVerified - attribute not implemented in Portal class.
allowedRedirectUris - attribute not implemented in Portal class.
groupDesignations - attribute not implemented in Item class.
scoreCompleteness - attribute not implemented in Item class.
groupDesignations - attribute not implemented in UserItem class.
scoreCompleteness - attribute not implemented in UserItem class.
Form_1.zip uploaded
...Cleaning
Traceback (most recent call last):
File "./updateMediaFolder.py", line 150, in <module>
os.remove(download_file)
WindowsError: [Error 2] The system cannot find the file specified: u'C:/Users/username/ArcGIS/My Survey Designs/Form 1-Download/Form_1.zip'

c:\Python27>

Philip-Wilson
Esri Frequent Contributor

Hi Shanon,

The error appears to be that the zip file can not be found in that location:

WindowsError: [Error 2] The system cannot find the file specified: u'C:/Users/username/ArcGIS/My Survey Designs/Form 1-Download/Form_1.zip'

Can you send me a copy of your modified script, mainly the header section with the variables you have used (minus your user credentials).

Phil.

AaronChaput1
New Contributor

Hi,

So I started out trying to use this feature to add new values to a choice list on the fly without having to republish the survey every time, but I got very confused by this: 

"Please note that updating the itemsets.csv file directly in the Form item via this script does not update the values in the external_choices sheet of the associated xls workbook. It only updates the generated CSV file in the media folder. If you open the survey in Connect again, or download the survey to Connect from AGOL after you have manually updated the itemsets.csv with this script, there will be a difference between the external_choices sheet in xls workbook (original values) and the itemsets.csv (new values). If you re-publish the survey via Connect again without making any changes to external_choices sheet, the itemsets.csv will be replaced with the original values that are in the external_choices sheet."

Does that mean that none of the changes to the choice list in the survey will occur until the external_choices tab of the xlsx has been updated manually?  What is the point of the external file if this is the case?

I feel like I'm not understanding correctly.

 

I followed this post as best I could, but the changes were not propagating until I updated the external_choices tab.

Discovering this, I scrapped the external csv file and instead focussed on just updating the choices tab (using FME) and using python to republish to the AGOL. I managed to do this, but the changes wouldn't appear in Survey123 Web or the Survey123 app.  However, they would appear in Survey123 Connect.

I think this has to do with the .xml that published along with the xlsx.  If I make the changes in the XML, the changes show up in the Survey123 app (but still not Survey123 web).

 

Is there a way to get around this? As I understand it, manually publishing with Survey123 Connect uses some API to convert the xlsx to xml.  I'm guessing publishing with python as detailed above doesn't do this.  The original xml is used which doesn't have the new values.

Is there some way to publish (autonomously with python) an updated xlsx (just the choices tab) to AGOL and have it generate a new .xml so that the new values will correctly display in the Survey123 App?

Thanks,

Aaron

Philip-Wilson
Esri Frequent Contributor

Hi Aaron,

When a survey is published via Connect there are a few things that happen, which if you are trying to replicate this via automated publishing, you would need to ensure all those same things occur and happen in the same order.

When you create an external_choices sheet in your xlsx file and then publish, an itemsets.csv file is created in the media folder when the survey is published. This is required for external_choices to work as per the xlsform spec. If you update the values in the external_choices sheet and re-publish in Connect the itemsets.csv file will also be updated.

However, if you manually update the itemsets.csv file via python (detailed above) or another automated process, there will be a difference between the values in the xslx external_choices sheet and the itemsets.csv. Updating the itemsets.csv manually does not update the original values in the xlsx file. If you go and re-publish your xlsx file you will over-write those changes that were made to itemsets.csv.

When you publish a survey using Connect, the xlsx file is converted into a xml file, this is basically the main purpose of Connect. This is done via the pyxform api. This is an important step in the publishing process. If you simply update the xlsx file and manually update it in the form item and upload to AGO, it will not update the xml file, hence you do not see the changes in the app.

To answer your question: "Is there some way to publish (autonomously with python) an updated xlsx (just the choices tab) to AGOL and have it generate a new .xml so that the new values will correctly display in the Survey123 App?" Yes, it is possible, however you would need to include the pyxform api steps in the python script to conver the xlsx file to xml. Basically you are creating a custom build of Connect that is automated.

Hope this helps.

Phil.

AaronChaput1
New Contributor

Hi Phil,

Ah ok, that makes sense.

What is the point of updating the .csv with python as listed above if the values in the external_choices sheet of the associated xls workbook need to be manually updated as well?

I feel like I'm not fully understanding.

Could you give some examples of how to use pyxform API functions in Python to convert the xlsx to xml?

Thanks,

Aaron

Philip-Wilson
Esri Frequent Contributor

Hi Aaron,

The point of updating the .csv file via python is for those users that do not want to republish their survey, ie they have no other changes to make to the survey, all they want to do is update the choice list in external csv file. This can then be automated via python to get the data from another course (xlsx file, database, web server etc) and continually update the csv file in the Form item.

This is not intended to replace publishing your survey from Connect. If you need to make other changes like change labels, calculations, groups, relevants, defaults etc, you still need to re-publish your survey with Connect. Only once you survey is *final* and you plan to make no further changes, can you then update the csv file choice lists using this method to update via python. Otherwise you need to update each time the same values in the xlsx file as detailed above.

The publishing and re-publishing process is not as simple as just converting from xlsx to xml. There are many other steps that occur, including updates to the .info files and webform files, error checking etc. I would suggest you take a look at AppStudio and the source code available for Survey123 (mainly the publishing qml files) if you want to understand entirely how this process works.

Phil.

AndrewMcClary1
New Contributor II

Hi James,

I was wondering if this problem logging onto Portal ever got resolved. My script is failing at the same point as Leonel's. I added print functions to help debug and I believe the problem is that no token is being returned by the AGOLHelper function.

print(agol_helper.token)  returns 'None' even though it seems to be able to log on to our Portal successfully. Is there a setting I need to change in either our portal or the code to get it to return a token? 

Thanks,

Andrew

vidytumuluru
New Contributor

hi Phil

,

I am trying your script and getting this msg. Can you tell me what I am missing here?

# final successful print message

print (zipFileName + " successfully updated with " + source_file + " and uploaded to your ArcGIS portal!")

...Starting

...Authenticating

...Querying

Runtime error

Traceback (most recent call last):

File "<string>", line 108, in <module>

KeyError: 'name'

>>>

thanks,

vidy

AdminValorInmueble
New Contributor III

Hi, 

I have the same problem, in line 105

E:\Proyectos\Formato Fiscalizaciones\updateMediaFolder>updateMediaFolder.py
...Starting
...Authenticating
...Querying
Traceback (most recent call last):
File "E:\Proyectos\Formato Fiscalizaciones\updateMediaFolder\updateMediaFolder.py", line 105, in <module>
zipFileName = serviceInfo['name']
KeyError: 'name'

when printing serviceinfo, displays the following:
serviceInfo: {u'error ': {u'message': u'Request not made over ssl ', u'code': 403, u'details': []}}

some idea, 

tranks

JamesTedrick
Esri Esteemed Contributor

Hi Jeffry,

please make sure all of the URLs specified for the connection are using https - that should address the second error.

AdminValorInmueble
New Contributor III

Thanks James Tedrick, that was the solution!
Now I have a question, the script is configured with the name of the external list or csv file to replace, but what happens when they are more than one list, where you configure that it replaces all the lists or cvs files for pulldata()

thanks for your help!

JamesTedrick
Esri Esteemed Contributor

Hi Jeffry,

You would need to modify the script to repeat the replacing commands for each resource.

AdminValorInmueble
New Contributor III

Sure!, thank you, 

I'm not very good at python programming, but I'll try it, best regards

JamesCrandall
MVP Frequent Contributor

Does this method work for .tpk's loaded in the media folder? 

Our tpk basemaps will be updated frequently and would like to use this method to update the published survey.  Currently, I'm manually running the Desktop Toolbox on an .mxd with an SDE feature class loaded, copying the output to the media folder of the survey and then publishing from Connect.

It'd be great to just write a .py script to programmatically do the above and setup a task to run on a timed interval.

Thanks!

Philip-Wilson
Esri Frequent Contributor

Hi James,

Yes, the same method should work. I have not tested it myself with tpk files, but provided you download the zip file, extract it, replace the tpk in the media folder with exact named file, zip it back up and the upload it, when users re-download the survey on the device they will get the updated tpk.

Phil.

JamesCrandall
MVP Frequent Contributor

I've got most of it converted to meet our needs.  We don't have ArcREST installed but I think I can still make it work all the same.  I did notice that the mxd with a single polygon feature class with <1000 features takes a long time to generate a tpk.  I know that outside of your implementation but, darn is that a long process!  Last thing I need to get done is the final upload, but I'm able to download, unzip, re-zip, etc... just as your awesome script does!

Thanks again for this!

Also: just an FYI--- I'm loving the ability to open the survey from another app on the device, WITH parameters and values! This means we can open the correct survey from out other app (that has the business data) with fields pre-populated.  This is incredibly valuable!

EvenorIbarra
New Contributor

Hi Philip

I'm using Portal 10.6.1, but apparently when you download the .zip file it gets corrupted, when trying to unzip it generates an error.

def downloadFile(url, filename):

    print ("...Downloading")

    req = urllib2.urlopen(url)

    CHUNK = 16 * 1024

    with open(filename, 'wb') as fp:

        while True:

            chunk = req.read(CHUNK)

            if not chunk: break

            fp.write(chunk)



This is the error shown when trying to unzip (raise BadZipfile, "File is not a zip file")

def extractZIP(filename,folder):

    print ("...Extracting")

    zfile = zipfile.ZipFile(filename)

    zfile.extractall(folder)



Could you give me guidance on this error!

Thank you I hope you have a solution for this.

Philip-Wilson
Esri Frequent Contributor

Hi Evenor,

What OS and version are you running the script on, and also what version of Python are you using in what IDE?

Phil.

EvenorIbarra
New Contributor

Hi Phil.

First of all thank you for your time.

I am using Windows 10 professional 64 bits, the IDE I use is Python

Scripter 2.5.3.3 x86 for Python 2.7.

One observation, if I use ArcGIS Online in the same environment mentioned

above it works perfectly, download the zip well, unzip it and upload the

changes in the media folder without problems.

Only on the portal is that I am experiencing this problem.

Evenor.

El dom., 4 ago. 2019 a las 16:52, Philip Wilson (<geonet@esri.com>)

escribió:

GeoNet, The Esri Community | GIS and Geospatial Professional Community

<https://community.esri.com/?et=blogs.comment.created>

Updating the media folder of a Survey123 Form item

new comment by Philip Wilson

<https://community.esri.com/people/Philip.Wilson-esristaff?et=blogs.comment.created>

- View all comments on this blog post

<https://community.esri.com/groups/survey123/blog/2017/12/08/an-alternative-way-to-update-the-media-folder-of-a-survey123-form-item?commentID=76211&et=blogs.comment.created#comment-76211>

Philip-Wilson
Esri Frequent Contributor

Thanks Evenor,

James Tedrick  any ideas on this and why it wouldn't work with Portal?

Phil.

JamesTedrick
Esri Esteemed Contributor

Hi Evenor Ibarra‌,

I'm not aware of any reason that Portal should cause a bad zip file.  That being said, I'm unsure why the script would break the writing process into 16KB chunks.  You should be able to have the downloadFile function look like:

def downloadFile(url, filename):

    print ("...Downloading")
    req = urllib2.urlopen(url)

    with open(filename, 'wb') as fp:

        fp.write(req.read())

You might want to try that revision to the function...

EvenorIbarra
New Contributor

Hi James Tedrick

I tried the function in this way as suggested, but it generates the same

error with portal, when downloading the zip it is incorrect.

def downloadFile(url, filename):

print ("...Downloading")

req = urllib2.urlopen(url)

with open(filename, 'wb') as fp:

fp.write(req.read())

On this occasion I tried it on portal 10.7.1 and the error is the same

except IOError:

raise BadZipfile("File is not a zip file")

if not endrec:

raise BadZipfile, "File is not a zip file"

Evenor.

El jue., 8 ago. 2019 a las 11:41, James Tedrick (<geonet@esri.com>)

escribió:

GeoNet, The Esri Community | GIS and Geospatial Professional Community

<https://community.esri.com/?et=blogs.comment.created>

Updating the media folder of a Survey123 Form item

new comment by James Tedrick

<https://community.esri.com/people/JTedrick-esristaff?et=blogs.comment.created>

- View all comments on this blog post

<https://community.esri.com/groups/survey123/blog/2017/12/08/an-alternative-way-to-update-the-media-folder-of-a-survey123-form-item?commentID=76499&et=blogs.comment.created#comment-76499>

JamesTedrick
Esri Esteemed Contributor

Hi Evenor,

My apologies, but I don't know why it would be seen as not a valid zip file.  Do you happen to have a second form to try this script on to see if it is just affecting the one form or others as well?

EvenorIbarra
New Contributor

Hello James

I tried running the script in Geoprocessing in ArcGIS Desktop

10.6.1 Advanced and the error is the same.

This is the log of results, I printed some parameters to check if something

is more them.

...Starting

...Authenticating

...Querying

Token:

Pe9Ap__VwLxSHOpksp4ZLgPm1e0y6GASUfwgASnTvbhYFy6m_feMAU8EnruXoKG4uyJgYNkGskEWLI9v_JuPuwcuTgrj9bIHRjWewaF8Cg0vruV8A8Xou_vk6HVqFVX3PtN04lwBFqQds0O41qfFiA..

itemID:c26cd1b10e294f98a53000c70008ca55

serviceInfo-> {u'avgRating': 0, u'proxyFilter': None, u'culture': u'es-cr',

u'owner': u'sistema', u'commentsEnabled': True, u'guid': None,

u'screenshots': [], u'id': u'c26cd1b10e294f98a53000c70008ca55', u'size':

76500, u'appCategories': [], u'access': u'private', u'languages': [],

u'title': u'Cierre de Aver\xeda', u'groupDesignations': None,

u'numRatings': 0, u'numComments': 0, u'snippet': None, u'listed': False,

u'largeThumbnail': None, u'itemControl': u'admin', u'type': u'Form',

u'thumbnail': u'thumbnail/Cierre_de_Aver_C3_ADa.png', u'industries': [],

u'numViews': 10, u'description': None, u'tags': [], u'typeKeywords':

, u'extent': [],

u'scoreCompleteness': 33, u'banner': None, u'properties': None,

u'categories': [], u'name': u'Cierre_de_Aver\xeda.zip', u'licenseInfo':

None, u'created': 1565121027089L, u'url': None, u'documentation': None,

u'modified': 1565293550451L, u'isOrgItem': True, u'spatialReference': None,

u'protected': False, u'ownerFolder': u'30377052949d41239f4ee634d1d798b8',

u'accessInformation': None}

Zip file name: Cierre_de_Avería.zip

https://

organization.com/portal/sharing/rest/content/items/c26cd1b10e294f98a53000c70008ca55/data/?f=json&token=Pe9Ap__VwLxSHOpksp4ZLgPm1e0y6GASUfwgASnTvbhYFy6m_feMAU8EnruXoKG4uyJgYNkGskEWLI9v_JuPuwcuTgrj9bIHRjWewaF8Cg0vruV8A8Xou_vk6HVqFVX3PtN04lwBFqQds0O41qfFiA

..

...Downloading

filename->C:/itemsets/descarga/Form_survey123.zip

Form_survey123.zip downloaded to: C:/itemsets/download/

...Extracting

Runtime error

Traceback (most recent call last):

File ")

escribió:

GeoNet, The Esri Community | GIS and Geospatial Professional Community

<https://community.esri.com/?et=blogs.comment.created>

Updating the media folder of a Survey123 Form item

new comment by James Tedrick

<https://community.esri.com/people/JTedrick-esristaff?et=blogs.comment.created>

- View all comments on this blog post

<https://community.esri.com/groups/survey123/blog/2017/12/08/an-alternative-way-to-update-the-media-folder-of-a-survey123-form-item?commentID=76697&et=blogs.comment.created#comment-76697>

EvenorIbarra
New Contributor

Sorry, I changed some name, site, name of the survey123. This would be the name of the zip file
Zip file name: Form_survey123.zip

Running the script in Geoprocessing  in ArcGIS Desktop

10.6.1 Advanced.

JamesCrandall
MVP Frequent Contributor

Edit: if I completely uninstall Survey123 from the device and then re-download the updated survey, the updated tpk shows up in the survey's map.

Hi Philip,

I've implemented a version of your script that first updates a .tpk with arcpy.CreateMapTilePackage_management(), then updates the media folder and finally uploads the packaged zip.  

Everything runs fine, no errors, an updated tpk is generated and included in the final .zip and the AGOL item last update date is correct.  However, after deleting and downloading the Survey onto the device the updated tpk is not showing the new features/updates.

Any input is appreciated!

Philip-Wilson
Esri Frequent Contributor

Hi James,

What version of Survey123 are you using and on what device/OS version? does the same happen on Windows, Android and iOS?

Phil.

JamesCrandall
MVP Frequent Contributor

Hi Phillip!  Sorry for the cross-post I put the details below on the other thread as well!  Thank you!

It's the latest Survey123 version 3.5.177 just installed yesterday. Device details:

Device: iPad Air2

Software: 12.2

Model: MH2M2LL/A

JamesKelly4
New Contributor III

I am having the same issue that Shanon had. See below output. Any idea what I am doing wrong? Thanks!

...Starting
...Authenticating
...Querying
Zip file name: CWD_Surveillance_2.zip
...Downloading
CWD_Surveillance_2.zip downloaded to: C:/Users/bh05078/Temp/
...Extracting
CWD_Surveillance_2.zip extracted to: C:/Users/bh05078/Temp/extract/
...Updating
confirmations.csv updated to: C:/Users/bh05078/Temp/extract/esriinfo/media/
...Cleaning
CWD_Surveillance_2.zip deleted from: C:/Users/bh05078/Temp/
...Compressing
CWD_Surveillance_2.zip created in: C:/Users/bh05078/Temp/
...Uploading
notificationsEnabled  - attribute not implemented in Portal class.
allowedExternalLinks  - attribute not implemented in Portal class.
vectorBasemapGalleryGroupQuery  - attribute not implemented in Portal class.
canSetCustomBuyLink  - attribute not implemented in Portal class.
rasterFunctionTemplatesGroupQuery  - attribute not implemented in Portal class.
eueiEnabled  - attribute not implemented in Portal class.
canSetQuestionnaire  - attribute not implemented in Portal class.
useVectorBasemaps  - attribute not implemented in Portal class.
canListSolutions  - attribute not implemented in Portal class.
allowedOrigins  - attribute not implemented in Portal class.
defaultVectorBasemap  - attribute not implemented in Portal class.
contentCategorySetsGroupQuery  - attribute not implemented in Portal class.
isVerified  - attribute not implemented in Portal class.
canListConsultingServices  - attribute not implemented in Portal class.
notificationsEnabled  - attribute not implemented in Portal class.
allowedExternalLinks  - attribute not implemented in Portal class.
vectorBasemapGalleryGroupQuery  - attribute not implemented in Portal class.
canSetCustomBuyLink  - attribute not implemented in Portal class.
rasterFunctionTemplatesGroupQuery  - attribute not implemented in Portal class.
eueiEnabled  - attribute not implemented in Portal class.
canSetQuestionnaire  - attribute not implemented in Portal class.
useVectorBasemaps  - attribute not implemented in Portal class.
canListSolutions  - attribute not implemented in Portal class.
allowedOrigins  - attribute not implemented in Portal class.
defaultVectorBasemap  - attribute not implemented in Portal class.
contentCategorySetsGroupQuery  - attribute not implemented in Portal class.
isVerified  - attribute not implemented in Portal class.
canListConsultingServices  - attribute not implemented in Portal class.
notificationsEnabled  - attribute not implemented in Portal class.
allowedExternalLinks  - attribute not implemented in Portal class.
vectorBasemapGalleryGroupQuery  - attribute not implemented in Portal class.
canSetCustomBuyLink  - attribute not implemented in Portal class.
rasterFunctionTemplatesGroupQuery  - attribute not implemented in Portal class.
eueiEnabled  - attribute not implemented in Portal class.
canSetQuestionnaire  - attribute not implemented in Portal class.
useVectorBasemaps  - attribute not implemented in Portal class.
canListSolutions  - attribute not implemented in Portal class.
allowedOrigins  - attribute not implemented in Portal class.
defaultVectorBasemap  - attribute not implemented in Portal class.
contentCategorySetsGroupQuery  - attribute not implemented in Portal class.
isVerified  - attribute not implemented in Portal class.
canListConsultingServices  - attribute not implemented in Portal class.
notificationsEnabled  - attribute not implemented in Portal class.
allowedExternalLinks  - attribute not implemented in Portal class.
vectorBasemapGalleryGroupQuery  - attribute not implemented in Portal class.
canSetCustomBuyLink  - attribute not implemented in Portal class.
rasterFunctionTemplatesGroupQuery  - attribute not implemented in Portal class.
eueiEnabled  - attribute not implemented in Portal class.
canSetQuestionnaire  - attribute not implemented in Portal class.
useVectorBasemaps  - attribute not implemented in Portal class.
canListSolutions  - attribute not implemented in Portal class.
allowedOrigins  - attribute not implemented in Portal class.
defaultVectorBasemap  - attribute not implemented in Portal class.
contentCategorySetsGroupQuery  - attribute not implemented in Portal class.
isVerified  - attribute not implemented in Portal class.
canListConsultingServices  - attribute not implemented in Portal class.
groupDesignations  - attribute not implemented in Item class.
scoreCompleteness  - attribute not implemented in Item class.
isOrgItem  - attribute not implemented in Item class.
groupDesignations  - attribute not implemented in UserItem class.
scoreCompleteness  - attribute not implemented in UserItem class.
isOrgItem  - attribute not implemented in UserItem class.
CWD_Surveillance_2.zip uploaded
...Cleaning
Traceback (most recent call last):
  File "T:\CWD\code\updateMediaFolder.py", line 148, in <module>
    os.remove(download_file)
WindowsError: [Error 2] The system cannot find the file specified: u'C:/Users/bh05078/Temp/CWD_Surveillance_2.zip'
[Finished in 13.8s]
JamesKelly4
New Contributor III

I figured it out. I just needed to save the script in the download_folder. Thanks anyway!

JamesKelly4
New Contributor III

Do users have to delete the survey or can they simply download the survey again to get the update info on the .csv file? Deleting it also deletes everything from the sent folder. My guys like to keep their submissions so they can review for accuracy later. Survey123 for ArcGIS

JamesKelly4
New Contributor III

Figured it out. The answer is no, you don't have to delete the survey; you can simply redownload the survey. 

Philip-Wilson
Esri Frequent Contributor

Hi James,

Yes that is correct, simply re-download the survey and it will work, no need to delete.

Phil.

JamesCrandall
MVP Frequent Contributor

I put this into a premium support case and confirmed that this is a bug (only on iOS).  Also, it was mentioned that this issue would possibly be fixed in v3.6 however it doesn't appear to be the case.

I'll move to join the Beta tester program asap in hopes to find a resolution to this issue.  Just need to get TestFlight installed on the dev device.