Web scrape Survey123 records to be used as a quick access links in Excel.

02-25-2021 03:36 PM
New Contributor

I have exported a feature class of all the widgets we have in our city to a spreadsheet in Excel. Every week, we need to fill out a condition assessment survey for each widget. Each widget has an Asset ID that are created by a script and is unique across all our features. The survey results need to be shared among non-GIS users who do not have any Desktop Apps from Esri. They do have AGO accounts and can view the results on survey123.arcgis.com. I would like to add a field to the Excel table to store a link that would take me to a record in survey123.arcgis.com. You can see the link I'm talking about if you go to the Data tab in one of your surveys and click on one record. Look up at the URL and see that you see the item ID for your survey, the objectID of the record, and the extent of the feature. Click on another record, and the URL changes. If you copy and paste that URL into your browser address bar, either the record comes up or you'll need to log in to AGO again.

The problem is that we have hundreds of widgets so clicking on each record then copy/pasting the URLs into the Excel spreadsheet isn't a sustainable solution. I don't know the capabilities of web scraping scripts, so please forgive my ignorance in this next question. Is it possible to recursively grab the URL from each record and using the ObjectID to join the survey result to the features? Remember that over time, each feature will have many survey results (52 per year). I'd love to make this a scheduled task and just monitor the output. I have some Python and SQL experience but I definitely need help getting started. I know that there are added complications of navigating through the login process not to mention making this work for multiple surveys. I wouldn't be against just keeping a list of all the itemIDs in a python list.

0 Kudos
2 Replies
Esri Regular Contributor

Hello @AlanVenneman1

I don't know much about web scraping, since the URL is going to be mostly the same for all features (https://survey123.arcgis.com/surveys/xx....xx/data?objectIds=<objectID>) you can probably use openpyxl a python module that ships with the ArcGIS API for Python to loop through each row and concatenate the URL with the object ID of the current row the loop is working with. 

Thank you,
New Contributor

It's funny, but my boss was just telling me that the link to all the survey results for an asset (not the individual records) was just fine. That being said, I am still curious to see if it can be done. I'll check out that library and see what I can do. Thanks for the reply!

0 Kudos