Access non-public results from Survey123 outside of agol environment

ChristopheSchnaufer · ‎07-05-2023

First time posting here. I am trying to access Survey123 results that aren't Everyone(public) using the `arcgis` Python package from a machine that does not have ArcGIS installed. Unfortunately, I am only able to access public results in my account (and elsewhere). The end goal is to access data that had been shared with me from another group (and won't be public). The following sample code shows what I've done so far. Any help is appreciated

from arcgis.gis import GIS

# ESRI Survey123 API endpoint
survey123_api_url = 'https://www.arcgis.com'
survey123_username = '<my_username>'
survey123_password = '<my_password>'

# Get a list of non-public Survey123 data
survey_item_id = '88d7e11f82fa44c0a52db4ba435b86ff'
gis = GIS(survey123_api_url, survey123_username, survey123_password)

# Use SurveyManager to see everything available
survey_manager = arcgis.apps.survey123.SurveyManager(gis)
print(survey_manager.surveys) # only contains public items

# Try to get a non-public item
sbi = survey_manager.get(survey_item_id)
print(sbi) # only contains item when it's public

sr = gis.content.search('owner:<my account name>')
print(sr)   # also only contains public items

ChristopheSchnaufer · ‎07-18-2023

I have a solution in hand! Thanks to @ChristopherCounsell and @alex_mapintel for all their help

import arcgis
from arcgis.gis import GIS

# ESRI endpoint
esri_url = 'https://www.arcgis.com'
esri_username = '<username>'
esri_password = '<password>'

# Connect to ArcGIS
gis = GIS(esri_url, esri_username, esri_password)

# Search for accessible content
search_results = gis.content.search('owner:<username>')
for one_search in search_results:
    # Find the feature service that you are looking for
    if one_search.type == 'Feature Service':
        # Choose the layer of interest
        feature_layer = one_search.layers[0]
        # Perform a query to get all of the data
        results = feature_layer.query(where='OBJECTId>=0')
        # Process the returned data (printing in this case)
        for data in results.features:
            print(data.as_dict)

I still am having other issues, but I can not be sure I'm accessing available data

View solution in original post

alex_mapintel · ‎07-05-2023

Hi @ChristopheSchnaufer,

Looks like you will need to import the survey123 module at the beginning of the script. Try adding:

from arcgis.apps import survey123

from there you can just use:

survey_manager = survey123.SurveyManager(gis)

in your script.

All the best,

Alex

ChristopheSchnaufer · ‎07-06-2023

Hello @alex_mapintel ,

Thank you for your response

The code changes you suggested are the equivalent to what I have in line 13 in the original post. Nevertheless, I tried the changes as you suggested and there wasn't a change in the results.

My code from line 13 in the original post:

survey_manager = arcgis.apps.survey123.SurveyManager(gis)

is the equivalent of the following import and creation statements:

# Declare import of survey123 from the arcgis.apps module
from arcgis.apps import survey123

# Stuff happens ...

# Create an instance of SurveyManager
# Equivelant to: survey_manager=arcgis.apps.survey123.SurveyManager(gis)
survey_manager = survey123.SuveryManager(gis)

alex_mapintel · ‎07-06-2023

Hi @ChristopheSchnaufer ,

No Worries. This is what I'd be looking into for the next steps:

1) Can you confirm that if you login to AGOL using a browser, using credentials from the python script, do you see that the item '88d7e11f82fa44c0a52db4ba435b86ff' is shared with you?

2) What license type is associated with the account you are using? Viewer? Does this license type allow you to interrogate survey123 data?

All the best,

ChristopheSchnaufer · ‎07-07-2023

Hello @alex_mapintel,

1) Yes I can log into the browser and see the data, it's in my account - I'm trying that out before I tackle accessing results from someone else. When I made the result associated with the ID Public I can see it with Python and in the browser; when it's not public I can not see it with Python, but I can with the browser. Due to the sensitive nature of the clients results that I will need to access, they will be restricted and not public

2) I have the following when I check my license info:

User type: GIS Professional Advanced
Role: Publisher

ChristopherCounsell · ‎07-11-2023

What are you trying to do? You don't need to work with the survey manager to access 'results'. You can use the feature service (or a view) and work with that. I would use the survey manager if I wanted to say see all the features services targeted by surveys owned by a user. But I'd need to be the owner or an administrator to check this. You won't be either in the given scenario. Sounds like you can remove the Survey Manager from the workflow and ask for feature service access. Unless there is some specific thing you need to be doing, in which case you might need a higher level of permissions (or the client runs the script you develop).

But this isn't necessarily your issue...

sr = gis.content.search('owner:<my account name>')
print(sr)   # also only contains public items

If you are searching using gis.content.search and only returning publicly shared items owned by your account, it sounds like this isn't an issue with Survey Manager. It sounds like your ArcGIS Account does not have the necessary permissions. Given your user is a Publisher, if you have signed in correctly, you should be be able to access non-public content that your account owns.

Add a few lines to check your authentication

from arcgis.gis import GIS

# Connect to ArcGIS Online
url = 'https://www.arcgis.com'
username = '<my_username>'
password = '<my_password>'

gis = GIS(url, username, password)

# Show connection
print(gis)
# Show username of signed in user
user = gis.properties.user.username
print(user)
# Get all content owned by this user
ownerquery = "owner:" + user
content = gis.content.search(query=ownerquery,max_items=-1)
print(content)

This should print

GIS @ https://magisian.maps.arcgis.com version:2023.2
themagisian
[<all your content>]

I don't know why this would be failing for you as your script seems fine (and seems to be returning some content). The most obvious answers are that you are using the wrong username and password or don't own the content.

If this working OK, we'll then need to understand more about what you're actually trying to do, and who will own the content. Like what is shared public? There's several parts to a survey (form, views, service, map and the folder - they should all be together). Get the authentication confirmed first and then get back to us.

ChristopheSchnaufer · ‎07-12-2023

Hello @ChristopherCounsell,

The content I'm currently testing against is my own content which is public, or not public, depending upon the test

I have run the code above as suggested and I have the following output. I am using SSO with the University of Arizona using the below (redacted) username, and don't see my public survey results

GIS @ https://www.arcgis.com version:2023.2
<login username>
[]

When I use my ESRI username in the above query after login, I get my public results but not my restricted ones

What I'm trying to do is pull down non-public survey results data to a local machine using Python (another programming language would be OK if I can get to the data). Currently I am testing against my own account

ChristopherCounsell · ‎07-14-2023

Interesting... maybe it's tied to the SSO but it's showing your login so not sure. What type of SSO? Where are you running this script?

'survey results' aren't an item. You don't need to interact with 'Survey123' at all here. You need access to the Hosted Feature Layers. Couple of scripts:

https://support.esri.com/en-us/knowledge-base/how-to-download-feature-service-items-from-arcgis-onli...

https://developers.arcgis.com/python/guide/download-data/

Want to give one of those a go?

ChristopherCounsell · ‎07-14-2023

Just to add some discussion on the SSO - you may need to use a clientid or interactive login to authenticate (thought it's unusual to see that your logged in name is shown. maybe it's tied to token generation). This is less my area than the content/survey handling but it does appear to be access or authentication related.

https://community.esri.com/t5/arcgis-api-for-python-questions/is-it-possible-to-use-the-arcgis-onlin...

ChristopheSchnaufer · ‎07-17-2023

@ChristopherCounsell I have tried the scripts and haven't made progress. I'm going to recap what I'm trying right now with my own account and ignore any other data.

I have a new account through my employer which didn't have anything in it. I created a quick Survey123 form and made 3 survey entries in it (3 submissions). I am now trying to use Python to download the data I entered into the new form to my local machine.

Here's what I see on this account when I look at "My Content"; everything there was created by Survey123.

When I query with my username (eg. gis.content.search('owner:<username>')) I get two results now (regardless of the Public setting)

[<Item title:"Testing Survey_results" type:Feature Layer Collection owner:<username>, <Item title:"Testing Survey" type:Form owner:<username> >]

The links you sent me appear to want a Feature Service - which I don't appear to have

My data is stubbornly staying on AGOL unless I manually download it