Downloading data captured with Survey123 for ArcGIS

27014
39
11-09-2015 10:30 PM
IsmaelChivite
Esri Frequent Contributor
7 39 27K

In this blog post I will describe how you can download data captured with Survey123 for ArcGIS. There are three basic scenarios where downloading your survey data is useful:

  • Analysis in third party toolsStata, SPSS, SAS, Tableau or  Microsoft Excel are just some examples of the many tools out there that people use to analyze data. You can download data captured with Survey123 for ArcGIS in formats that these tools understand such as CSV. You can also download your data in shapefile and FileGeodatabase format in case you want to use GIS tools compatible with these formats.
  • Back-up: Download the data you capture to have an extra copy. Just in case!
  • Enterprise integration: Download your data so you can load it into your own database (Microsoft SQL Server, Oracle etc) or massage it before you bring it over into your own enterprise system.

I will describe a few different techniques to download your data:

Manually downloading your data from survey123.arcgis.com

The easiest way to download your data is using the http://survey123.arcgis.com web site:

  1. Log into http://survey123.arcgis.com with your ArcGIS  credentials
  2. In the Survey gallery, click on your Survey and go to the Data tab
  3. Scroll down, select the output format you want (CSV, Excel, Shapefile or File Geodatabase) and download your data.

If you are working with Portal for ArcGIS, the technique above will be of no use, because the http://survey123.arcgis.com web site only works against ArcGIS  (for now), but do not worry, the next tip will work with both ArcGIS  and Portal for ArcGIS.

Manually downloading your data from ArcGIS Enterprise or ArcGIS

  1. Log into your ArcGIS organization (in ArcGIS  or Portal for ArcGIS) and go to MyContent.
  2. For every survey you have published, you will find a folder and within it, a Feature Layer. Open the details page of the Feature Layer and select Export Data.

Downloading data manually as described above is fine except when you want to do it very often. Scripting is the way to go if you want to regularly download your survey data. A classic example is when you want to pull data from ArcGIS  daily, or hourly, into your own database for further analysis or QA/QC workflows.

Downloading your data with Python

Since data captured with Survey123 for ArcGIS is stored in Feature Services we can use the standard REST API in ArcGIS to trigger the download. Javascript, Python, PHP, Scala, Java or .Net are some of the environments from which you can invoke the ArcGIS REST API.

My favorite Python script to programmatically download Survey123 data into a geodatabase was written by James Tedrick and is shared in this blog post.  James describes at length how to use the script in his post and there are many useful comments from people that used it.

Alternatively, you can also use this Python script to download data into a local shapefile, file geodatabase or csv file. To run this script:

  • Download the code into a machine with ArcGIS for Desktop 10.3 (or newer)
  • Edit with IDLE of a Python editor to input your own feature service id, credentials and output folder
    • Feature Service ID: In arcgis.com or your own Portal for ArcGIS, navigate to the details page of the feature service you want to download. Get the ID from the URL. It will look something like this: f4a06f02b1dd4b1f8904ed0d4e8713bd
    • Output Format: Choose between 'Shapefile', 'File Geodatabase' or 'CSV'.
    • Credentials: Typically the owner of the Survey. Simply replace the 'XXXXX' and 'YYYYYY'
    • You can define a Portal URL and the output folder where you want the data to be saved.

The advantage of using a Python script is that you can easily run it anytime and get the latest data from your Survey.  You can also schedule your script to run every so often.

39 Comments
RonKnepper
Occasional Contributor II

Do you have any easy tips for dealing with exporting attachments? The only real option I see is to download your data to a FGDB then use a python script to pull this data out.  I think everyone would love to see some kind of tool for exporting attachments (named appropriately)

DougBrowning
MVP Frequent Contributor

Try using ArcRest.  It gives you a full replica copy locally in a GDB including attachments.  It has been working great for me until I hit 33,000+ records.  I have a ticket open on it.

I assume it will do tables also but not sure.

Download ArcREST and install it.  Then here is the simple code.

Python code

from arcrest.security import AGOLTokenSecurityHandler

from arcrest.agol import FeatureService

from arcrest.common.filters import LayerDefinitionFilter

import arcpy

import datetime

# Set these based on your FS and credentials

username = ""

password = ""

url = "http://services2.arcgis.com/as3H1x8rqlUMyL6s/ArcGIS/rest/services/ETC..."

outDir = r"C:\temp"

#Get a local replica of the HFS  note set layers properly

proxy_port = None

proxy_url = None

#_defaultTokenExpiration = 60 #Minutes

agolSH = AGOLTokenSecurityHandler(username=username,password=password)

fs = FeatureService(url=url,securityHandler=agolSH,proxy_port=proxy_port,proxy_url=proxy_url,initialize=True)

result = fs.createReplica(replicaName='Demo', layers='0,1,2,3,4', keep_replica=False, layerQueries=None, geometryFilter=None, returnAttachments=True,returnAttachmentDatabyURL=False,returnAsFeatureClass=True, out_path=outDir)

IsmaelChivite
Esri Frequent Contributor

Thanks Ronald for pointing out the need for an out of the box geoprocessing tool in ArcGIS to export attachments from a featureclass. Please feel free to vote and comment here too: ArcGIS Idea - Batch attachment export

Our friends from Esri Technical support also pointed me to this KB article:  41763 - Batch export attachments from a feature class

GeoffLewis
New Contributor II

Does anyone know why I get the following error from the Download Survey123 Data Python script

Traceback (most recent call last):

  File "C:/Users/lewisg/Desktop/DownloadSurvey123Data.py", line 89, in <module>

    exportItemId = export_data['exportItemId']

KeyError: 'exportItemId'

Line 89 is  # retrieve the itemId for the export:

    exportItemId = export_data['exportItemId']

We want to set up a scheduled task and avoid manual downloads

CarmelConnolly
Regular Contributor

Hi Geoff,

What ItemID are you using? I had this error too but I was using the wrong ItemID, if you browse to the item details of the feature service you want to download, the ItemID will be at the end of the URL.

Carmel

TL2
by
Occasional Contributor III

Ismael Chivite

Another alternative I am tinkering with is the GeoEvent processor to bring my Survey123 data into my SDE.  I have successfully implemented this on some test features and surveys but not to my exact needs.  My issue arises with attachments (photos).  The hosted data store does not provide a rest end point for the attachments table.  When I make a repeat, I do get to work with an table which has a rest endpoint but not with photo attachments (which I would think is just another table). 

Is there anyway for the GeoEvent processor to access the attachments from the Hosted Data Store?  I am thinking the answer is no.

This may only work with surveys without photo attachments?

Thanks.

IsmaelChivite
Esri Frequent Contributor

Adam Mollenkopf​ Looping Adam in case he can help  with this.

GeoffLewis
New Contributor II

Thanks to Jamie Till at Eagle in New Zealand who identified what I had done wrong:

  1. I should have left the portal URL code as is i.e. portal_url = r"https://www.arcgis.com/" and not changed it to our home URL in ArcGISonline
  2. I should have ticked the Export Data and Sync boxes in Feature Layer (Hosted ) Settings

Thanks for the suggestions and advice for what was an ID1ot error 😉

TL2
by
Occasional Contributor III

The python script provided always runs once for me on the first go.  Then everytime after I get " Exception: !!! Export job failed. Status "partial" "

Anyone else experiencing this?

DougBrowning
MVP Frequent Contributor

Is it maybe trying to overwrite the same file?

TL2
by
Occasional Contributor III

The issue had to do with the time.sleep module.  5 seconds was not long enough for the asynchronization.  Increasing the time solved the issue.  

DougBrowning
MVP Frequent Contributor

You can also use ArcRest.  Here is my simple script for that.

from arcrest.security import AGOLTokenSecurityHandler
from arcrest.agol import FeatureService
from arcrest.common.filters import LayerDefinitionFilter

username = "put here"
password = "put here"
# Put url to HFS here
url = "put here"
# The dir to store the GeoDB zip file (it is now a zip with a .gdb in it)
outDir = r"put here"
# A list of layer numbers you want to download. Get these numbers of the rest details page using the url above
layerList = [0,1,2,3,4]


#Get a local replica of the HFS
proxy_port = None
proxy_url = None
agolSH = AGOLTokenSecurityHandler(username=username,password=password)
fs = FeatureService(url=url, securityHandler=agolSH, proxy_port=proxy_port, proxy_url=proxy_url, initialize=True)
result = fs.createReplica(replicaName='Backup', layers=layerList, dataFormat='filegdb', out_path=outDir, wait=False)

TL2
by
Occasional Contributor III

Thanks Doug, I will try that out.  

My main goal is to replicate my HFS in my local SDE.  I have been tinkering with GeoEvent to pass the features over but that will not pass attachments (images) through.  Do you have any suggestions in automating the process of copying HFS survey123 data into SDE on through a script.  Other than the two provided with some appending.

Thanks.

DougBrowning
MVP Frequent Contributor

Yea we have been struggling with that a bit too.  Wipe sde, download data to a gdb, then load into sde is the only way we could think of also.  Unless you get fancy with comparing dates from editor tracking.  Had some luck with giving append tool the HFS directly but not always.  For now I think we will wait for Portal replication that is slated to come out with 10.5 at the end of the year.  It was talked about at the conference.

JoeFlannery
Regular Contributor II

Geoff:

I have followed your instructions (thank you) and I am still getting the "exportItemId" error message.  Might you have any other words of wisdom for this issue?

RobbWright
New Contributor III

Has anybody had success in downloading the Survey123 data *with* the associated photos?  I have been unsuccessful (and am frustrated).  If not with this tool, is there another way to pull down the Survey123 data with photos?

JoeFlannery
Regular Contributor II

Robb:

 

Yes, I have had successfully downloading Survey123 data from AGOL to my on-prem server and am able to see the attached photos.  The attached photos are stored as “Blob” data type in a related table in the FGDB.  The attached photos are “related” to the survey points.  In ArcMap, when I use the Identify tool, I can drill down and see the photo that is attached to the point.  See below.

Attached Photo

JoeFlannery
Regular Contributor II

Robb:

By the way, I am using James Tedrick‌'s python script to bring Survey123 down from AGOL.  James' script will download to enterprise geodatabase or FGDB.

https://community.esri.com/groups/survey123/blog/2016/12/14/migrating-data-from-the-survey-feature-s... 

JamesTedrick
Esri Esteemed Contributor

Hi Robb,

If you need to export the images out of the file geodatabase, I'd suggest you look at the following Knowledge Base article: How To: Batch export attachments from a feature class  

AndrewWatkins
New Contributor

Hello, Great script and very useful. I am new to Python and am having trouble appending a date stamp to the file name in the script. Any help would be great.

Thanks,

Andrew

JamesTedrick
Esri Esteemed Contributor

Hi Andrew,

I'd suggest looking at Python strftime reference - that's pretty comprehensive for creating text strings.

A quick example:

import datetime

datetime.datetime.now().strftime("%Y%m%d-%H%M%S") '20171023-144605'

BretLian
New Contributor II

I'd like to use the python script that downloads to a local gdb in an automated fashion to import data in to an SDE Database, but I'm curious if there is a modification I could make to the script to make the name of the GDB within the zipped folder a consistent one, like "Staging". I can rename the zip folder, but the contents(the GDB itself) has a different id everytime the script is ran.

If I could get a consistent gdb name, I could then automate the appending of data into the corresponding SDE tables.

I'm somewhat inexperienced with python and any direction would be helpful. James's script is fantastic buy doesn't seem to work with photo attachments the way we need it to(attachments table is empty)

Thanks,
Bret

DougBrowning
MVP Frequent Contributor

Bret if you look above I had posted a way to download using ArcRest instead.  It downloads a full gdb with all relationships classes and photos. 

Note you must install ArcRest python package first.

What I left out was the code to rename the file after download.  I also added code here to unzip the file downloaded and then you can rename the gdb (or just ask the zip file\os what it contains and use that - os.listdir - use this as your gdb name).  this unzip code is a little diff due to .gdb file type.

Hope it helps

set the un and pass plus out dir

---download gdb

import arcrest
from arcresthelper import securityhandlerhelper
from arcresthelper import common

import os, time

HFSScope=[]


def trace():
    """
        trace finds the line, the filename
        and error message and returns it
        to the user
    """

    import traceback, inspect, sys
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0]
    filename = inspect.getfile(inspect.currentframe())
    # script name + line number
    line = tbinfo.split(", ")[1]
    # Get Python syntax error
    #
    synerror = traceback.format_exc().splitlines()[-1]
    return line, filename, synerror



if __name__ == "__main__":
    proxy_port = None
    proxy_url = None

    securityinfo = {}
    securityinfo['security_type'] = 'Portal'#LDAP, NTLM, OAuth, Portal, PKI
    securityinfo['username'] = "un"#<UserName>
    securityinfo['password'] = "pass"#<Password>
    securityinfo['org_url'] = "http://www.arcgis.com"
    securityinfo['proxy_url'] = proxy_url
    securityinfo['proxy_port'] = proxy_port
    securityinfo['referer_url'] = None
    securityinfo['token_url'] = None
    securityinfo['certificatefile'] = None
    securityinfo['keyfile'] = None
    securityinfo['client_id'] = None
    securityinfo['secret_id'] = None

    # Assumes a config script that has FriendlyName, ItemID
    path, script    = os.path.split(os.path.realpath(__file__))
    Config          = path.replace('\Scripts','\Config')+'\\'+'BackupAllAGOHFS.txt'.replace("/", "\\")

    # where to save the zip files
    savePath = r"c:\save"


    # read in the config file for list of HFS for Survey123
    BackupAllAGOHFSConfigFile = open(Config, 'r')
    HFSList = BackupAllAGOHFSConfigFile.read().splitlines()
    BackupAllAGOHFSConfigFile.close()

    for HFS in HFSList:
        HFSname = HFS.split(",")[0]
        itemId = HFS.split(",")[1]          # this now comes from the main AGO page with all the settings.
                                            # example itemId = "5a9ecdf30f53466bad14f5dbbedb1014"  http://co.maps.arcgis.com/home/item.html?id=d92ed61e72cb49fd9142fc1817487379
        if len(HFSScope)==0 or HFSname in HFSScope:
            try:
                shh = securityhandlerhelper.securityhandlerhelper(securityinfo=securityinfo)
                if shh.valid == False:
                    print shh.message
                else:
                    print HFSname
                    admin = arcrest.manageorg.Administration(securityHandler=shh.securityhandler)

                    item = admin.content.getItem(itemId)
                    user = admin.content.users.user(username=item.owner)

                    # use to set jsut some layers.  NOT using this
        ##            exportParameters = {"layers":[
        ##                                   {
        ##                                       "id": 0,
        ##                                       "where": "OBJECTID = 1"
        ##                                       }]
        ##                               }

                    # this version waits for the download so no pause needed
                    res  = user.exportItem(title=HFSname,
                                        itemId=itemId,
                                        exportFormat="File Geodatabase",
                                        exportParameters=None,
                                        wait=True)

                    exportItemId = res.id
                    exportItem = admin.content.getItem(exportItemId)
                    itemDataPath = exportItem.itemData(f=None, savePath=savePath)

                    # Delete from AGO list to remove clutter
                    exportItem.userItem.deleteItem()

                    print "Backed up " + HFSname + " to " + itemDataPath

                    # rename file so it makes sense
                    renameFile = os.path.dirname(itemDataPath) + "\\" + HFSname + " Backup " + time.asctime().replace(':', '-') + ".zip"
                    os.rename(itemDataPath, renameFile)

            except (common.ArcRestHelperError),e:
                print "error in function: %s" % e[0]['function']
                print "error on line: %s" % e[0]['line']
                print "error in file name: %s" % e[0]['filename']
                print "with error message: %s" % e[0]['synerror']
                if 'arcpyError' in e[0]:
                    print "with arcpy message: %s" % e[0]['arcpyError']

            except:
                line, filename, synerror = trace()
                print "error on line: %s" % line
                print "error in file name: %s" % filename
                print "with error message: %s" % synerror
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

----download

-----unzip

# use this to unzip a zip file that had a .gdb in it


def unzip(path, zipFile):
    # If the output location does not yet exist, create it
    #
    if not isdir(path):
        os.makedirs(path)

    zip = zipfile.ZipFile(zipFile, 'r')
    for each in zip.namelist():
        print "Attempting unzip"
        arcpy.AddMessage("Extracting " + os.path.basename(each) + " ...")

        # Check to see if the item was written to the zip file with an
        # archive name that includes a parent directory. If it does, create
        # the parent folder in the output workspace and then write the file,
        # otherwise, just write the file to the workspace.
        #
        if not each.endswith('/'):
            print "Splitting files"
            root, name = split(each)
            directory = normpath(join(path, root))
            if not isdir(directory):
                os.makedirs(directory)
            print "Writing files"
            file(join(directory, name), 'wb').write(zip.read(each))
            print "Successful Unzip"


outdir = r"C:\temp"
zipfile = r"C:\temp\file.zip"

unzip(outDir, zipfile)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
YenniferCedeño
New Contributor II
JosephKerski
Esri Frequent Contributor

Very helpful - thank you.  Exporting attachments (photos) is what many of our educational users want to do ... is there an easy way to do this part that does not require writing and running a script?

DougBrowning
MVP Frequent Contributor

Yes just go to the item page for the feature service and click Export Data (this is what the script simulates).

If you do not see this button then go to Setting for the item and check Allow others to Export different formats.

JosephKerski
Esri Frequent Contributor

Thanks Doug ... where do the photos end up once you do this export?

--Joseph K 

DougBrowning
MVP Frequent Contributor

They are in the geodatabase still. 

I am guessing you want the images exported to the file system as a .jpg file old school style?  If so there is no way to do that except for a script.  See this post  https://community.esri.com/groups/survey123/blog/2017/03/22/working-with-your-survey123-photos

and this one  https://support.esri.com/en/technical-article/000011912

NicolasRoger
Occasional Contributor

Can you provide a similar script for .NET (I use C#).  The python script crash at this line: "import ago" so it seems I dont have what it takes.

The documentation on how to extract data with c# is difficult to understand and always seems partial.  Is the "REST API" still in use? ESRI seems to say that we should use ArcGIS Runtime SDK for .NET. Is it the same thing as a "REST API".

How can I create a simple C# script to extract my Survey Data?

EDIT: It seems the documentation wasn't working because I have a proxy on my network. I solved my connection problem by adding this in the App.config file of my C# project:

<system.net>  <defaultProxy useDefaultCredentials="true" /></system.net>

Then for my code I followed this pattern:

-Get a token using the arcGIS Runtime .NET:

var cred = await Esri.ArcGISRuntime.Security.AuthenticationManager.Current.GenerateCredentialAsync(
                                                                new Uri("https://www.arcgis.com/sharing/rest"),
                                                                userName,
                                                                Password) as ArcGISTokenCredential;

- Send a POST request using a standard .NET WebClient:

               

   

               string responseStr;
               //You service Url will be similar to this
               string ServiceUrl = @"https://services1.arcgis.com/zj13e1cPkTMwv2/ArcGIS/rest/services/service_864a34588bccb03c946c77354/F..."; //Layer 0
                using (var wb = new WebClient())
                {
                   //Request parameters. This one select all data and fields
                    var Request = new NameValueCollection();
                    Request ["where"] = "1=1";
                    Request ["f"] = "json";
                    Request ["outFields"] = "*";
                    Request ["token"] = cred.Token; //Important! Must always be present
                    //Send the request using the REST api
                    var response = wb.UploadValues(ServiceUrl + "query", "POST", Request );
                    responseStr = Encoding.UTF8.GetString(response);
                }
                File.WriteAllText(dossier + fichier, reponseStr); //Export results in a txt file
                Rootobject dataResults = JsonConvert.DeserializeObject<Rootobject>(responseStr); //Build the C# object containing the data. This line will crash for you here. Google about how to convert a JSON string to an object.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I hope this help someone.

ConnorDennehy1
New Contributor

I just keep getting this:

ERROR: Can't find a feature service with id: 799b972213c14da0af861a6fc81c6685
TIP:   Navigate to the item details page of your feature service, and get the id from the URL

I have tried it with entering the red and green ID's... am I missing something?

JamesTedrick
Esri Esteemed Contributor

Hi Connor,

Are you the owner of the feature service?  If so, you should be able to download from it's Item Details page.  If not, the owner may need to enable export.

SoratoSouza_e_Silva
Occasional Contributor

The python script that export to CSV, works to form too? No feature service...

The error below appear:

How to solve?

GeoffLewis
New Contributor II

What version of ArcMap are you using? I don't think script works at version before 10.3 maybe 10.4?

CristhieDiaz
New Contributor II

Hi, We have data that is being captured in survey123 and I must deliver it to the client in .sav format but in the toobox I can't get how to convert it, can you help me?

IsmaelChivite
Esri Frequent Contributor

Hi. Survey123 does not support exporting data directly into .sav files (SPSS). You can, however, export the data into CSV or Excel, and then use a third party tool to convert it into .sav. You can also use IBM PASW (SPSS) .sav | Safe Software to export data from ArcGIS (including Survey123 data) into SPSS .sav files.

CristhieDiaz
New Contributor II

Hola Ismael primero gracias por tu rápida respuesta, nos podrías sugerir alguna herramienta para convertir que conozcas? por otro lado si tengo la data en ArcPro hay alguna forma de exportarla? o en general en Arcgis no hay ninguna manera directa de hacerlo?.

Muchas Gracias!

Chris_Counsell
Esri Contributor

Ismael Chivite‌ can you please reshare the python script?

ShawnCapararo
New Contributor II

Can anyone advise how to select more than one record using the map interface in the Data tab? Holding the shift key down and drawing a box only zooms in and out? Thanks in advance.

WDFW_HabitatProgram
New Contributor

@IsmaelChivite  or @JamesTedrick Seems like permissions were added to the download script item, is this identical to SyncSurvey at https://github.com/tedrick/SyncSurvey? 

 

I had previously found somebody else's attempt at converting SyncSurvey to Python 3, modified that, but am having GlobalID issues (they're doing what GlobalID's do and are becoming unique after being copied from the downloaded replica to a separate feature class in a file geodatabase.

* Update: figured out to add  arcpy.env.preserveGlobalIds = True before the append and added a check for globalids in the field mappings function that changes the target type from globalid to guid... but still curious why the link for the python script above is rejecting my AGOL credentials.