topic Re: Generate Survey123 Report in ArcGIS API for Python Questions

Generate Survey123 Report

ChelseaRozek — Wed, 28 Nov 2018 17:56:52 GMT

Really excited about the latest Python API update. Would like to be able to export survey responses as reports with Python. I'm currently stuck at this part. I can see that the reports get generated and I can download them (image below from Survey123 website), but I receive an error in the code: KeyError: 'results' (see attachment). I'm also curious if you can access the generated reports directly with Python after making them.

Re: Generate Survey123 Report

MatthewCarey — Thu, 21 Feb 2019 20:40:29 GMT

Chelsea, I wondered if you or anyone else had made any further progress with this. I encountered the same error (although the outputted Word documents seem OK), and I'm also having a hard time figuring out the syntax to use in place of '1=1' if I'm trying to only generate records for a subset of the features in the feature layer. Any pointers / examples appreciated!

Re: Generate Survey123 Report

ChelseaRozek — Thu, 21 Feb 2019 20:56:24 GMT

Glad to hear someone else is trying to do this. I haven't found a solution yet. Are you saying the output was fine from the Survey123 website? You aren't able to access the output in Python, are you?

Re: Generate Survey123 Report

MatthewCarey — Fri, 22 Feb 2019 14:16:00 GMT

Hi, no I'm not able to access the output in Python. For my workflow I'm OK with having the resulting reports be available in AGOL, but I would like to be able only run it for a subset of the features, not all of them.

Re: Generate Survey123 Report

ChelseaRozek — Fri, 22 Feb 2019 17:10:39 GMT

I think I figured your issue out. By default with 1=1 (aka always true), I'm generating 3 reports because there are 3 completed surveys. The error I posted above shows that we're calling _survey.py to run generate_report(). In that .py, it says if where == "1=1": where = {"where":"1=1"} then it uses it like that. So when I put in my code a where statement that was valid and returned only 1 record when using Query under Supported Operations on the Service URL (link on the item's page in AGO), like this in my code: letslearn.generate_report(choice, {"where":"brandname='Fan'"}), only 1 report was generated. Still got my KeyError: 'results', but on the Survey123 website under Report>Recent Tasks>only 1 record in the last task. Hope that helps.

Re: Generate Survey123 Report

MatthewCarey — Fri, 22 Feb 2019 19:55:41 GMT

Yes!!! Thank you, that worked for me. I'm still getting the KeyError: 'results' but this did the trick for the 'where' part. Thanks for delving deeper into it, much appreciated.

Re: Generate Survey123 Report

ChelseaRozek — Mon, 25 Feb 2019 14:50:47 GMT

Update: I put in a support ticket and there is an existing bug:

BUG-000119057 : The Python API 1.5.2 generate_report method of the arcgis.apps.survey123 module, generates the following error: KeyError: 'results'

I was playing with a workaround, where you basically just search for the generated reports in your organization and download them. I also delete them after downloading them so future runs wouldn't re-download them.

#this script generates the KeyError: 'results' but still works

from arcgis.gis import GIS

import os

from zipfile import ZipFile

from arcgis.apps.survey123._survey import SurveyManager, Survey

gis = GIS("https://YOURORGANIZATION.maps.arcgis.com","USERNAME","PASSWORD") #fill in 3 strings here

survey_mgr = SurveyManager(gis)

letslearn = survey_mgr.get("id for survey form item in AGO") #fill in 1 string here

print(letslearn.report_templates) #see all available print templates for this survey

choice = letslearn.report_templates[0] #the print template you want to use (0 = first); fill in 1 number here

print(choice)

#example syntax for generate report for some features: letslearn.generate_report(choice, {"where":"fieldname='fieldvalueyouwant'"})

letslearn.generate_report(choice, "1=1") #generates report for all features

#print(letslearn.reports)

for x in letslearn.reports: #finds all Microsoft Word doc files in AGO with you as their owner and "Survey 123" in the tags. Could upload a Word document that wasn't a generated report, add that tag, and it would be found with this

#print(x.description) remove "#" if you want to see them listed

if x.description: #if it exists.. (some of my old generated reports had null descriptions)

if 'id for survey form item in AGO' in x.description: #fill in 1 string here

id = x.id #get id of that Word doc

data_item = gis.content.get(id) #get Word doc as an item from AGO

data_item.download(save_path = r'C:\where\ever\you\want\it') #fill in 1 string here; downloads that Word doc

data_item.delete() #deletes that Word doc in AGO

Re: Generate Survey123 Report

JohnStowell — Wed, 06 Mar 2019 22:02:51 GMT

Hi Chelsea,

Firstly thank you for the code sample, this led me in the right direction for automating report creation with Python.

I took your code and added more functionality to it, which now generates reports for submissions from the last 24hrs, downloads the docx files, retrieves email addresses collected via survey, and sends a copy of the relevant docx file to the recipient as an email attachment. It also has a workaround try/except block for the known bug.

I'm sharing it here as others may find it useful:

https://github.com/nzjs/Automated-Survey123-Reports

Cheers,

John

Re: Generate Survey123 Report

DannielaCardenas — Fri, 05 Apr 2019 19:57:48 GMT

Hi! I really love your work, but I have a question

Could you please explain to us how works the block try/except/finally to omit the KeyError?

Thanks

Re: Generate Survey123 Report

JohnStowell — Tue, 09 Apr 2019 20:47:19 GMT

Hello, the try/except/finally block is built into Python as a way of handling exceptions.

In this instance we expect the KeyError exception bug (try/except) and tell the script to continue on after it occurs (finally).

Cheers.

Re: Generate Survey123 Report

KassandraRodriguez — Mon, 13 May 2019 21:45:59 GMT

This is really helpful thank you for sharing I'm trying to hopefully use this for our workflow but I keep running into this error maybe someone can help. I made sure to download the other python libraries but looking at the docx.py I'm not sure what's going on. Any help is much appreciated

Re: Generate Survey123 Report

JohnStowell — Mon, 13 May 2019 22:10:24 GMT

Hi Kassandra, I think this could be related to your version of the docx library?

For Python 3.x, you need the python-docx library rather than docx.

These links have more info: https://pypi.org/project/python-docx/

https://stackoverflow.com/questions/22765313/when-import-docx-in-python3-3-i-have-error-importerror-no-module-named-excepti

Try uninstalling the old one, install python-docx, then see how that goes.

Re: Generate Survey123 Report

KassandraRodriguez — Tue, 14 May 2019 16:07:29 GMT

Hi John, That was it thank you for your help

Re: Generate Survey123 Report

SimonAllard2 — Tue, 12 Nov 2019 22:55:28 GMT

Hi Kassandra,

Could you share how you are importing the python-docx module. I keep getting an error in my script saying.

import python-docx
File "<ipython-input-6-d71ba5c26d0b>", line 1
import python-docx
^
SyntaxError: invalid syntax

Cheers,

Simon

Re: Generate Survey123 Report

KassandraRodriguez — Wed, 13 Nov 2019 17:23:26 GMT

My problem was that I was using the wrong version of python which was 2.7, so I still used:

import docx‍

but I think your asking how to install the library? This video should help with that Install Python with Matplotlib, Numpy+MKL, and SciPy - YouTube you would just replace it with

pip install python-docx

let me know if that answers your question

Re: Generate Survey123 Report

SimonAllard2 — Thu, 14 Nov 2019 22:22:11 GMT

Hi Kassandra,

I am using PyCharms IDE with a python virtual environment. When adding python-docx from pipy repository I am getting a invalid ssl certificate error. I think this is an issue with my internal firewall so I have contacted my it department. I tried adding a ssl certificate to list of those trusted in pycharms but that hasn’t worked either. Strange tho that I was able to upgrade the pip and requests modules but not able to add python-docx?

Thanks,

Simon Allard

GIS Analyst

Bay of Plenty Regional Council Toi Moana

Re: Generate Survey123 Report

SimonAllard2 — Mon, 09 Dec 2019 03:44:21 GMT

Hi Kassandra Rodriguez,

I had tried all sorts of things but in the end it was the firewall on our corporate network! We finally got it sorted though and I am working through the script to generate reports. I am having a bit of trouble with a couple lines of code - one is specifying the "where" filter variable and applying this in the 'generate reports module'

where_filter = '{"where":"CreationDate >= CURRENT_TIMESTAMP - INTERVAL \'1\' DAY"}'

survey.generate_report(template, where_filter, utc_offset, report_title)

The script runs but doesn't return any surveys. I have only succeeded when use 1=1 which returns all reports!

I tested the sql query:

which returned results but only when the sql format was set to standard from none.

Any thoughts?

Thanks,

Simon

Re: Generate Survey123 Report

KassandraRodriguez — Mon, 09 Dec 2019 16:22:33 GMT

Hi Simon sorry I can't help with this, we ended up going with the out of the box feature reports. I'd ask John Stowell for further help

Re: Generate Survey123 Report

MichelleWilliams1 — Fri, 14 Feb 2020 16:37:28 GMT

Hi John,

I'm using your script and I'm getting this error, do you know why?

Re: Generate Survey123 Report

MichelleWilliams1 — Fri, 14 Feb 2020 19:29:41 GMT

I installed smtplib, python-docx, and glob but I'm still getting this error. Is it my firewall giving me problems?

I might need to download to a folder.