topic Re: Download survey from device after upload failed in ArcGIS Survey123 Questions

Download survey from device after upload failed

Nicole_Ueberschär — Mon, 03 Oct 2016 11:39:55 GMT

I thought I had seen this question somewhere before but now I can't find it (neither an answer)...

After the upload of a survey failed with the error "code 1003, Operation rolled back", is there any other option to download the data somehow from the device?

For the error itself I have this thread: https://community.esri.com/message/638532-error-code-1003

Re: Download survey from device after upload failed

zkovacs — Tue, 04 Oct 2016 14:18:21 GMT

Hi Nicole,

When connected to the device from your computer (or with a suitable app on the device), you can simply copy the ArcGIS folder to have a backup. The data is organised into My Surveys and My Survey Attachments folders (at least on Android, iOS should be similar).

It might be a bit tricky to find a particular survey as they are referenced by the Form ID stored in AGOL (so not with the name you gave to your survey). You can find this out if you go to the Form in question in AGOL, click on View Item details and have a look at the URL in the address bar. The part after 'id=' will be the same id used to store the survey on the device.

You can try to recover the data from the runtime database(s), have a look at the below tool available in ArcGIS (10.3+).

https://desktop.arcgis.com/en/arcmap/10.3/tools/conversion-toolbox/copy-runtime-geodatabase-to-file-geodatabase.htm

I hope this helps.

Re: Download survey from device after upload failed

Nicole_Ueberschär — Tue, 04 Oct 2016 14:32:08 GMT

Hi Zoltan,

I was able to copy a database abc.sqlite from the tablet and could access the data somehow but the way it is stored will be a bit tricky to have it exported into a gdb. Unfortunately the tool you mentioned is not working for sqlite databases as it seems.

Re: Download survey from device after upload failed

zkovacs — Wed, 05 Oct 2016 08:12:51 GMT

Ah, shoot... I thought the same runtime gdb format was being used, but clearly it's not the case - I just had a quick look now. Sorry, I was wrong about the extraction part. On the other hand, you can open the SQLite database directly in ArcGIS (used 10.3.1), but it does not give you any friendly data as everything is stored as it would be sent to the service holding the FS (and you would need to filter for the survey in question as it's being used for all surveys with unsent data)...

It might be best to ask Ismael in case they have some workaround or tool to extract/submit the data easily. You may be able to construct a JSON out of the .sqlite content and use the REST API to send the data in, but I haven't done anything like that yet...

Sorry I couldn't have been of more help this time.

Re: Download survey from device after upload failed

Nicole_Ueberschär — Thu, 06 Oct 2016 06:39:54 GMT

Hi Zoltan,

I didn't get the idea that I could open the sqlite database straight in ArcGIS 🙂 Thanks for that.

I think I will contact Ismael directly to see if he has something in his pockets to help me.

Re: Download survey from device after upload failed

zkovacs — Thu, 06 Oct 2016 07:57:39 GMT

Hi Nicole,

You're welcome. (Supported DBMS formats list is here.) 🙂

Sounds like a plan. I'd love to know if there is a solution for it too, in case I need to deal with a situation like this in the future. Thanks.

Re: Download survey from device after upload failed

Nicole_Ueberschär — Fri, 07 Oct 2016 09:10:37 GMT

After changing my search to something similar to "survey123 export from sqlite" I found this post for troubleshooting Survey123:

Troubleshoot—Survey123 for ArcGIS | ArcGIS

The last topic deals with how to recover data from a mobile device.

I tested it with my own copy of the survey, not the original one that my client was using. I adjusted the path to the survey in the database with the sqlite database browser matching the path that was created when I filled a survey on my desktop. Then my survey showed me the four surveys that were not uploaded.

After removing some of the constraints in the survey I opened the failed surveys, saved them without changing anything and was able to upload them. I only had the impression that one value got lost in a repeated field (which is bad enough).

Maybe this helps someone having similar issues. I still can't really tell, what caused the Error Code 1003 but at least the data is online now. It might have been something to do with those constraints.

(no answer from Ismael yet)

Re: Download survey from device after upload failed

DanielMcIlroy — Mon, 10 Oct 2016 09:12:05 GMT

I have a similar question, although we are interested in being able to do a backup of the survey data for teams that are in the field without cellular data connectivity for long periods of time (common in Australia!). I will look into this further and see if the process for recovering data from the mobile device could be followed to make backups.

Re: Download survey from device after upload failed

Nicole_Ueberschär — Mon, 10 Oct 2016 11:06:15 GMT

Hi Dan,

I think it's in fact a good strategy to back up the sqlite database in the field instead of uploading it straight away. The way it is described here Troubleshoot—Survey123 for ArcGIS | ArcGIS should work for you.

Re: Download survey from device after upload failed

DanielMcIlroy — Tue, 11 Oct 2016 00:23:19 GMT

That is fantastic, thanks for that Nicole!

At least we know that it can be done now, and we should be able to work something out. I am thinking about building a python too that can do the copy automatically for the user.

Re: Download survey from device after upload failed

KarissaRamstead — Fri, 17 Aug 2018 17:21:50 GMT

I have been trying to work through the same troubleshoot you did, but with no success. My error is 202. I have been able to work through the recover data from the mobile device down to replacing the path names using DB Browser. From there I have no luck. I have opened the Survey123 Connect on my desktop and I cannot find a place to navigate to these files to upload my data from the desktop. If I work from the Survey123 on my tablet I am having the same error and I cannot resend the surveys. And I don't see an obvious way to upload that .sql file with the changed path to Survey123 online. So I think I am missing a critical step. Any thoughts on what I might be missing?

Re: Download survey from device after upload failed

JamesTedrick — Fri, 17 Aug 2018 17:42:07 GMT

Hi Karissa,

Is there additional error text? It sounds like there is a configuration issue between the feature service and the survey. The recovery procedure is the workflow you attempted via the tablet (the Survey123 app, not Survey123 Connect).

Re: Download survey from device after upload failed

KarissaRamstead — Fri, 17 Aug 2018 18:10:19 GMT

The send error give the additional text:

"Incompatible feature service. Parent to child relationship not found from TCEF_Fire_History (0) to plotRepeat(3)"

Re: Download survey from device after upload failed

JamesTedrick — Fri, 17 Aug 2018 18:22:41 GMT

Hi Karissa,

Did Survey123 Connect create the feature service or is this an existing feature service?

Re: Download survey from device after upload failed

KarissaRamstead — Fri, 17 Aug 2018 18:32:24 GMT

Created by Survey123. We did not have an existing feature service we were trying to build off of.

Re: Download survey from device after upload failed

JamesTedrick — Fri, 17 Aug 2018 22:17:32 GMT

Hi Karissa,

I would like to follow up with you further to resolve why this is occuring. In the meantime, I've revised the script ReadS123DB/readDb.py at master · tedrick/ReadS123DB · GitHub to handle repeats; you should be able to export the data in the outbox into CSV files - copy the database to a Windows computer with Python to run the script.

Re: Download survey from device after upload failed

KarissaRamstead — Mon, 20 Aug 2018 21:37:32 GMT

Hey James,

I am a bit rusty on Python. So I think we might be making progress but when I try to run the Python script I am getting the following error:

I have the folder C:\Users\kramstead\Extract set up similar to what I saw on the gitHub page as well with the readDb.py in the main level of the Extract file and the ArcGIS\My Sureys\Databases and the .sqlite file in this folder.

The only change I made to readDb.py was to copy and past the .sqlite db file name just in case they were a few numbers different. Everything else in the script looked like it was fairly generic and should be relevant for any Survey123 database. Any thoughts on what I am missing, or corrections I may need to make to the readDb.py file?

Re: Download survey from device after upload failed

KarissaRamstead — Mon, 20 Aug 2018 22:02:35 GMT

I am not sure if it makes a difference but the version of python that runs from the command line on my computer is 2.7.8

Re: Download survey from device after upload failed

JamesTedrick — Mon, 20 Aug 2018 22:23:42 GMT

Hi Karissa,

The database is not where the script expects it. Either copy it from ArcGIS/My Surveys/Databases to the same folder as the script or modify line 11 of the script to read:

IN_DB = r'./ArcGIS/My Surveys/Databases/a084e65cd9403d813e8ec667ca329a63.sqlite'

and run the script without parameters

Re: Download survey from device after upload failed

KarissaRamstead — Mon, 20 Aug 2018 23:26:34 GMT

I copied the db into the main folder and I see data! Making more progress...sorry this is taking me so long to work through. I am definitely not familiar on how the Survey123 data is set up in the sqlite db and I have not worked with sqlite databases yet so I am only getting the broad strokes of your code.

What do you mean about running the script with no parameters? Do you mean not to add in the [input sqlite file] and [directory to write the CSV files to] the main line of code I input to the command line

C:\Users\kramstead\Extract> python readDb.py a084e65cd9403d813e8ec667ca329a63.sqlite C:\Users\kramstead\Extract\

I tried both and had the same Traceback notice pop up either way I ran it.