Downloading data captured with Survey123

[IsmaelChivite]

[Updated February 8, 2022 to reference a Python script that downloads survey data with attachments]

In this blog post I will describe how you can download data captured with Survey123. There are three basic scenarios where downloading your survey data is useful:

• Analysis in third party tools:  Stata, SPSS, SAS, Tableau or  Microsoft Excel are just some examples of the many tools out there that people use to analyze data. You can download data captured with Survey123 for ArcGIS in formats that these tools understand such as CSV. You can also download your data in shapefile and FileGeodatabase format in case you want to use GIS tools compatible with these formats.
• Back-up: Download the data you capture to have an extra copy. Just in case!
• Enterprise integration: Download your data so you can load it into your own database (Microsoft SQL Server, Oracle etc) or massage it before you bring it over into your own enterprise system.

I will describe a few different techniques to download your data:

Manually downloading your data from survey123.arcgis.com

The easiest way to download your data is using the http://survey123.arcgis.com web site:

1. Log into http://survey123.arcgis.com with your ArcGIS  credentials
2. In the Survey gallery, click on your Survey and go to the Data tab
3. Scroll down, select the output format you want (CSV, Excel, Shapefile or File Geodatabase) and download your data.

Manually downloading your data from ArcGIS

1. Log into your ArcGIS organization and go to Content.
2. For every survey you have published, you will find a folder and within it, a Feature Layer. Open the details page of the Feature Layer and select Export Data.

[Updated February 8, 2022. Included link to Python script to download data and attachments]

Downloading data manually as described above is fine except when you want to:

• Download data at regular intervals. Scripting is the way to go if you want to regularly download your survey data. A classic example is when you want to pull data from ArcGIS  daily, or hourly, into your own database for further analysis or QA/QC workflows.
• Download attachments in their raw format (as opposed to embedded within a File Geodatabase)

Downloading your data with Python

The Survey123 developer documentation includes a Python script that illustrates how to download data. This script can be scheduled to run at regular intervals using ArcGIS Pro tools.  The script also is useful if you want to download all attachments locally.

If you are new to Python, I recommend you follow the step by step instructions in this blog post.

[RonKnepper]

Do you have any easy tips for dealing with exporting attachments? The only real option I see is to download your data to a FGDB then use a python script to pull this data out.  I think everyone would love to see some kind of tool for exporting attachments (named appropriately)

[DougBrowning]

Try using ArcRest.  It gives you a full replica copy locally in a GDB including attachments.  It has been working great for me until I hit 33,000+ records.  I have a ticket open on it.

I assume it will do tables also but not sure.

Download ArcREST and install it.  Then here is the simple code.

Python code

from arcrest.security import AGOLTokenSecurityHandler

from arcrest.agol import FeatureService

from arcrest.common.filters import LayerDefinitionFilter

import arcpy

import datetime

# Set these based on your FS and credentials

username = ""

password = ""

url = "http://services2.arcgis.com/as3H1x8rqlUMyL6s/ArcGIS/rest/services/ETC..."

outDir = r"C:\temp"

#Get a local replica of the HFS  note set layers properly

proxy_port = None

proxy_url = None

#_defaultTokenExpiration = 60 #Minutes

agolSH = AGOLTokenSecurityHandler(username=username,password=password)

fs = FeatureService(url=url,securityHandler=agolSH,proxy_port=proxy_port,proxy_url=proxy_url,initialize=True)

result = fs.createReplica(replicaName='Demo', layers='0,1,2,3,4', keep_replica=False, layerQueries=None, geometryFilter=None, returnAttachments=True,returnAttachmentDatabyURL=False,returnAsFeatureClass=True, out_path=outDir)

[IsmaelChivite]

Thanks Ronald for pointing out the need for an out of the box geoprocessing tool in ArcGIS to export attachments from a featureclass. Please feel free to vote and comment here too: ArcGIS Idea - Batch attachment export

Our friends from Esri Technical support also pointed me to this KB article:  41763 - Batch export attachments from a feature class

[GeoffLewis]

Does anyone know why I get the following error from the Download Survey123 Data Python script

Traceback (most recent call last):

  File "C:/Users/lewisg/Desktop/DownloadSurvey123Data.py", line 89, in <module>

    exportItemId = export_data['exportItemId']

KeyError: 'exportItemId'

Line 89 is  # retrieve the itemId for the export:

    exportItemId = export_data['exportItemId']

We want to set up a scheduled task and avoid manual downloads

[CarmelConnolly]

Hi Geoff,

What ItemID are you using? I had this error too but I was using the wrong ItemID, if you browse to the item details of the feature service you want to download, the ItemID will be at the end of the URL.

Carmel

[TL2]

Ismael Chivite​

Another alternative I am tinkering with is the GeoEvent processor to bring my Survey123 data into my SDE.  I have successfully implemented this on some test features and surveys but not to my exact needs.  My issue arises with attachments (photos).  The hosted data store does not provide a rest end point for the attachments table.  When I make a repeat, I do get to work with an table which has a rest endpoint but not with photo attachments (which I would think is just another table). 

Is there anyway for the GeoEvent processor to access the attachments from the Hosted Data Store?  I am thinking the answer is no.

This may only work with surveys without photo attachments?

Thanks.

[IsmaelChivite]

Adam Mollenkopf​ Looping Adam in case he can help  with this.

[GeoffLewis]

Thanks to Jamie Till at Eagle in New Zealand who identified what I had done wrong:

1. I should have left the portal URL code as is i.e. portal_url = r"https://www.arcgis.com/" and not changed it to our home URL in ArcGISonline
2. I should have ticked the Export Data and Sync boxes in Feature Layer (Hosted ) Settings

Thanks for the suggestions and advice for what was an ID1ot error 😉

[TL2]

The python script provided always runs once for me on the first go.  Then everytime after I get " Exception: !!! Export job failed. Status "partial" "

Anyone else experiencing this?

[DougBrowning]

Is it maybe trying to overwrite the same file?

[TL2]

The issue had to do with the time.sleep module.  5 seconds was not long enough for the asynchronization.  Increasing the time solved the issue.  

[DougBrowning]

You can also use ArcRest.  Here is my simple script for that.

from arcrest.security import AGOLTokenSecurityHandler
from arcrest.agol import FeatureService
from arcrest.common.filters import LayerDefinitionFilter

username = "put here"
password = "put here"
# Put url to HFS here
url = "put here"
# The dir to store the GeoDB zip file (it is now a zip with a .gdb in it)
outDir = r"put here"
# A list of layer numbers you want to download. Get these numbers of the rest details page using the url above
layerList = [0,1,2,3,4]

#Get a local replica of the HFS
proxy_port = None
proxy_url = None
agolSH = AGOLTokenSecurityHandler(username=username,password=password)
fs = FeatureService(url=url, securityHandler=agolSH, proxy_port=proxy_port, proxy_url=proxy_url, initialize=True)
result = fs.createReplica(replicaName='Backup', layers=layerList, dataFormat='filegdb', out_path=outDir, wait=False)

[TL2]

Thanks Doug, I will try that out.  

My main goal is to replicate my HFS in my local SDE.  I have been tinkering with GeoEvent to pass the features over but that will not pass attachments (images) through.  Do you have any suggestions in automating the process of copying HFS survey123 data into SDE on through a script.  Other than the two provided with some appending.

Thanks.

[DougBrowning]

Yea we have been struggling with that a bit too.  Wipe sde, download data to a gdb, then load into sde is the only way we could think of also.  Unless you get fancy with comparing dates from editor tracking.  Had some luck with giving append tool the HFS directly but not always.  For now I think we will wait for Portal replication that is slated to come out with 10.5 at the end of the year.  It was talked about at the conference.

[JoeFlannery]

Geoff:

I have followed your instructions (thank you) and I am still getting the "exportItemId" error message.  Might you have any other words of wisdom for this issue?

[RobbWright]

Has anybody had success in downloading the Survey123 data *with* the associated photos?  I have been unsuccessful (and am frustrated).  If not with this tool, is there another way to pull down the Survey123 data with photos?

[JoeFlannery]

Robb:

Yes, I have had successfully downloading Survey123 data from AGOL to my on-prem server and am able to see the attached photos.  The attached photos are stored as “Blob” data type in a related table in the FGDB.  The attached photos are “related” to the survey points.  In ArcMap, when I use the Identify tool, I can drill down and see the photo that is attached to the point.  See below.

[JoeFlannery]

Robb:

By the way, I am using James Tedrick‌'s python script to bring Survey123 down from AGOL.  James' script will download to enterprise geodatabase or FGDB.

https://community.esri.com/groups/survey123/blog/2016/12/14/migrating-data-from-the-survey-feature-s... 

[JamesTedrick]

Hi Robb,

If you need to export the images out of the file geodatabase, I'd suggest you look at the following Knowledge Base article: How To: Batch export attachments from a feature class  

[AndrewWatkins]

Hello, Great script and very useful. I am new to Python and am having trouble appending a date stamp to the file name in the script. Any help would be great.

Thanks,

Andrew

[JamesTedrick]

Hi Andrew,

I'd suggest looking at Python strftime reference - that's pretty comprehensive for creating text strings.

A quick example:

import datetime

datetime.datetime.now().strftime("%Y%m%d-%H%M%S") > '20171023-144605'

[BretLian]

I'd like to use the python script that downloads to a local gdb in an automated fashion to import data in to an SDE Database, but I'm curious if there is a modification I could make to the script to make the name of the GDB within the zipped folder a consistent one, like "Staging". I can rename the zip folder, but the contents(the GDB itself) has a different id everytime the script is ran.

If I could get a consistent gdb name, I could then automate the appending of data into the corresponding SDE tables.

I'm somewhat inexperienced with python and any direction would be helpful. James's script is fantastic buy doesn't seem to work with photo attachments the way we need it to(attachments table is empty)

Thanks,
Bret

[DougBrowning]

Bret if you look above I had posted a way to download using ArcRest instead.  It downloads a full gdb with all relationships classes and photos. 

Note you must install ArcRest python package first.

What I left out was the code to rename the file after download.  I also added code here to unzip the file downloaded and then you can rename the gdb (or just ask the zip file\os what it contains and use that - os.listdir - use this as your gdb name).  this unzip code is a little diff due to .gdb file type.

Hope it helps

set the un and pass plus out dir

---download gdb

import arcrest
from arcresthelper import securityhandlerhelper
from arcresthelper import common

import os, time

HFSScope=[]


def trace():
    """
        trace finds the line, the filename
        and error message and returns it
        to the user
    """
    import traceback, inspect, sys
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0]
    filename = inspect.getfile(inspect.currentframe())
    # script name + line number
    line = tbinfo.split(", ")[1]
    # Get Python syntax error
    #
    synerror = traceback.format_exc().splitlines()[-1]
    return line, filename, synerror



if __name__ == "__main__":
    proxy_port = None
    proxy_url = None

    securityinfo = {}
    securityinfo['security_type'] = 'Portal'#LDAP, NTLM, OAuth, Portal, PKI
    securityinfo['username'] = "un"#<UserName>
    securityinfo['password'] = "pass"#<Password>
    securityinfo['org_url'] = "http://www.arcgis.com"
    securityinfo['proxy_url'] = proxy_url
    securityinfo['proxy_port'] = proxy_port
    securityinfo['referer_url'] = None
    securityinfo['token_url'] = None
    securityinfo['certificatefile'] = None
    securityinfo['keyfile'] = None
    securityinfo['client_id'] = None
    securityinfo['secret_id'] = None

    # Assumes a config script that has FriendlyName, ItemID
    path, script    = os.path.split(os.path.realpath(__file__))
    Config          = path.replace('\Scripts','\Config')+'\\'+'BackupAllAGOHFS.txt'.replace("/", "\\")

    # where to save the zip files
    savePath = r"c:\save"


    # read in the config file for list of HFS for Survey123
    BackupAllAGOHFSConfigFile = open(Config, 'r')
    HFSList = BackupAllAGOHFSConfigFile.read().splitlines()
    BackupAllAGOHFSConfigFile.close()

    for HFS in HFSList:
        HFSname = HFS.split(",")[0]
        itemId = HFS.split(",")[1]          # this now comes from the main AGO page with all the settings.
                                            # example itemId = "5a9ecdf30f53466bad14f5dbbedb1014"  http://co.maps.arcgis.com/home/item.html?id=d92ed61e72cb49fd9142fc1817487379
        if len(HFSScope)==0 or HFSname in HFSScope:
            try:
                shh = securityhandlerhelper.securityhandlerhelper(securityinfo=securityinfo)
                if shh.valid == False:
                    print shh.message
                else:
                    print HFSname
                    admin = arcrest.manageorg.Administration(securityHandler=shh.securityhandler)

                    item = admin.content.getItem(itemId)
                    user = admin.content.users.user(username=item.owner)

                    # use to set jsut some layers.  NOT using this
        ##            exportParameters = {"layers":[
        ##                                   {
        ##                                       "id": 0,
        ##                                       "where": "OBJECTID = 1"
        ##                                       }]
        ##                               }

                    # this version waits for the download so no pause needed
                    res  = user.exportItem(title=HFSname,
                                        itemId=itemId,
                                        exportFormat="File Geodatabase",
                                        exportParameters=None,
                                        wait=True)

                    exportItemId = res.id
                    exportItem = admin.content.getItem(exportItemId)
                    itemDataPath = exportItem.itemData(f=None, savePath=savePath)

                    # Delete from AGO list to remove clutter
                    exportItem.userItem.deleteItem()

                    print "Backed up " + HFSname + " to " + itemDataPath

                    # rename file so it makes sense
                    renameFile = os.path.dirname(itemDataPath) + "\\" + HFSname + " Backup " + time.asctime().replace(':', '-') + ".zip"
                    os.rename(itemDataPath, renameFile)

            except (common.ArcRestHelperError),e:
                print "error in function: %s" % e[0]['function']
                print "error on line: %s" % e[0]['line']
                print "error in file name: %s" % e[0]['filename']
                print "with error message: %s" % e[0]['synerror']
                if 'arcpyError' in e[0]:
                    print "with arcpy message: %s" % e[0]['arcpyError']

            except:
                line, filename, synerror = trace()
                print "error on line: %s" % line
                print "error in file name: %s" % filename
                print "with error message: %s" % synerror
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

----download

-----unzip

# use this to unzip a zip file that had a .gdb in it


def unzip(path, zipFile):
    # If the output location does not yet exist, create it
    #
    if not isdir(path):
        os.makedirs(path)

    zip = zipfile.ZipFile(zipFile, 'r')
    for each in zip.namelist():
        print "Attempting unzip"
        arcpy.AddMessage("Extracting " + os.path.basename(each) + " ...")

        # Check to see if the item was written to the zip file with an
        # archive name that includes a parent directory. If it does, create
        # the parent folder in the output workspace and then write the file,
        # otherwise, just write the file to the workspace.
        #
        if not each.endswith('/'):
            print "Splitting files"
            root, name = split(each)
            directory = normpath(join(path, root))
            if not isdir(directory):
                os.makedirs(directory)
            print "Writing files"
            file(join(directory, name), 'wb').write(zip.read(each))
            print "Successful Unzip"


outdir = r"C:\temp"
zipfile = r"C:\temp\file.zip"

unzip(outDir, zipfile)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[Anonymous User]

oscarvega4279‌

[JosephKerski]

Very helpful - thank you.  Exporting attachments (photos) is what many of our educational users want to do ... is there an easy way to do this part that does not require writing and running a script?

[DougBrowning]

Yes just go to the item page for the feature service and click Export Data (this is what the script simulates).

If you do not see this button then go to Setting for the item and check Allow others to Export different formats.

[JosephKerski]

Thanks Doug ... where do the photos end up once you do this export?

--Joseph K 

[DougBrowning]

They are in the geodatabase still. 

I am guessing you want the images exported to the file system as a .jpg file old school style?  If so there is no way to do that except for a script.  See this post  https://community.esri.com/groups/survey123/blog/2017/03/22/working-with-your-survey123-photos

and this one  https://support.esri.com/en/technical-article/000011912

[NicolasRoger]

Can you provide a similar script for .NET (I use C#).  The python script crash at this line: "import ago" so it seems I dont have what it takes.

The documentation on how to extract data with c# is difficult to understand and always seems partial.  Is the "REST API" still in use? ESRI seems to say that we should use ArcGIS Runtime SDK for .NET. Is it the same thing as a "REST API".

How can I create a simple C# script to extract my Survey Data?

EDIT: It seems the documentation wasn't working because I have a proxy on my network. I solved my connection problem by adding this in the App.config file of my C# project:

<system.net>  <defaultProxy useDefaultCredentials="true" /></system.net>

Then for my code I followed this pattern:

-Get a token using the arcGIS Runtime .NET:

var cred = await Esri.ArcGISRuntime.Security.AuthenticationManager.Current.GenerateCredentialAsync(
                                                                new Uri("https://www.arcgis.com/sharing/rest"),
                                                                userName,
                                                                Password) as ArcGISTokenCredential;‍‍‍‍

- Send a POST request using a standard .NET WebClient:

               string responseStr;
               //You service Url will be similar to this
               string ServiceUrl = @"https://services1.arcgis.com/zj13e1cPkTMwv2/ArcGIS/rest/services/service_864a34588bccb03c946c77354/F..."; //Layer 0
                using (var wb = new WebClient())
                {
                   //Request parameters. This one select all data and fields
                    var Request = new NameValueCollection();
                    Request ["where"] = "1=1";
                    Request ["f"] = "json";
                    Request ["outFields"] = "*";
                    Request ["token"] = cred.Token; //Important! Must always be present
                    //Send the request using the REST api
                    var response = wb.UploadValues(ServiceUrl + "query", "POST", Request );
                    responseStr = Encoding.UTF8.GetString(response);
                }
                File.WriteAllText(dossier + fichier, reponseStr); //Export results in a txt file
                Rootobject dataResults = JsonConvert.DeserializeObject<Rootobject>(responseStr); //Build the C# object containing the data. This line will crash for you here. Google about how to convert a JSON string to an object.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I hope this help someone.

[ConnorDennehy1]

I just keep getting this:

ERROR: Can't find a feature service with id: 799b972213c14da0af861a6fc81c6685
TIP:   Navigate to the item details page of your feature service, and get the id from the URL

I have tried it with entering the red and green ID's... am I missing something?

[JamesTedrick]

Hi Connor,

Are you the owner of the feature service?  If so, you should be able to download from it's Item Details page.  If not, the owner may need to enable export.

[SoratoSouza_e_Silva]

The python script that export to CSV, works to form too? No feature service...

The error below appear:

How to solve?

[GeoffLewis]

What version of ArcMap are you using? I don't think script works at version before 10.3 maybe 10.4?

[CristhieDiaz]

Hi, We have data that is being captured in survey123 and I must deliver it to the client in .sav format but in the toobox I can't get how to convert it, can you help me?

[IsmaelChivite]

Hi. Survey123 does not support exporting data directly into .sav files (SPSS). You can, however, export the data into CSV or Excel, and then use a third party tool to convert it into .sav. You can also use IBM PASW (SPSS) .sav | Safe Software to export data from ArcGIS (including Survey123 data) into SPSS .sav files.

[CristhieDiaz]

Hola Ismael primero gracias por tu rápida respuesta, nos podrías sugerir alguna herramienta para convertir que conozcas? por otro lado si tengo la data en ArcPro hay alguna forma de exportarla? o en general en Arcgis no hay ninguna manera directa de hacerlo?.

Muchas Gracias!

[Anonymous User]

Ismael Chivite‌ can you please reshare the python script?

[ShawnCapararo]

Can anyone advise how to select more than one record using the map interface in the Data tab? Holding the shift key down and drawing a box only zooms in and out? Thanks in advance.

[WDFW_HabitatProgram]

@IsmaelChivite  or @JamesTedrick Seems like permissions were added to the download script item, is this identical to SyncSurvey at https://github.com/tedrick/SyncSurvey? 

I had previously found somebody else's attempt at converting SyncSurvey to Python 3, modified that, but am having GlobalID issues (they're doing what GlobalID's do and are becoming unique after being copied from the downloaded replica to a separate feature class in a file geodatabase.

* Update: figured out to add  arcpy.env.preserveGlobalIds = True before the append and added a check for globalids in the field mappings function that changes the target type from globalid to guid... but still curious why the link for the python script above is rejecting my AGOL credentials.