Repeats in S123 Data not Downloading in Pandas Dataframe

[leightmaes]

Hello everyone,

I use Python to download S123 data using the SurveyManager into a pandas dataframe. I can easily do so with one of my surveys that does not have repeats - all data from the survey appears in the pandas df.

However, my surveys that have repeat sections do not appear when I download the survey as a pandas DF. Note that I am not downloading the feature service generated from the survey results, but I am using the Survey ID itself to download the data from the survey - as described here. Is there any way to also extract these repeats into separate DFs using survey_manager.get()?

See my script below:

# use survey manager to pull survey data
survey_manager = SurveyManager(gis)
survey_by_id = survey_manager.get("surveyidhere")

# download survey as pandas DataFrame
survey_df = survey_by_id.download('DF')
survey_df.head()

[jcarlson]

In a survey, repeats are handled by related tables. Outputting to a dataframe is only going to give you a single table, however. You'll need to either export to a format that can handle multiple tables, such as a file geodatabase, or query each layer/table from the service directly, rather than using the survey manager class.

[leightmaes]

Hi Josh,

Thanks for your reply. When you say "query each layer/table from the service directly", what do you mean? I do not want FGDB format, and would prefer to keep my data in a tabular non-spatial format coming out of S123. Is there a way to do this with a feature service instead of the survey manager class?

[jcarlson · Accepted solution]

You can create a Spatially Enabled Dataframe from any layer in your portal, either through the GeoAccessor.from_layer function, or, my preferred method, using the FeatureLayer.query function. By including the parameter as_df=True, whatever is returned from our query comes out as a dataframe. You can optionally limit the fields and whether any geometry is returned.

To do this for each layer, you might try something like this:

fs = gis.content.get('itemid of service')

main_layer = fs.layers[0]
rel_table = fs.tables[0]

main_df = main_layer.query(as_df=True)
rel_df = rel_table.query(as_df=True)

And so on, for however many related tables/layers you're working with.

[leightmaes]

Thanks Josh! This works like a charm.