I'd like to have this script loop through a list of URLs and use a web scraper to search certain things from each of these websites. I have my cursor set up as well as Beautiful soup, but I'm wondering how I identify each item in the list? Can I attach an index number to each one somehow? Here's what I have. If I run this it will print the URLs. I've used Beautiful Soup to find things in HTML before, but I'm not sure how to find things from URLs in a list? from bs4 import BeautifulSoup import urllib import urllib . request import os , arcpy import time #the hosted layer with website urls in 'Website' field fc = r 'https://services.arcgis.com/fGsbyIOAuxHnF97m/arcgis/rest/services/Grab_Go_School_Meals_Location_(View)/FeatureServer/0?token=fbpCfA34sTJ4rzWO2TQn_c38B4TfGWOZ6jTMeFL1m7CNKd9_odI1t_t_hL-YvvePbE3M428FRT-zW-bISRYrGdJ2CnloKrHoHAfMnbGXpJ-5-zZBU6ONK1u0hMv5D-Vy-fnRpqpQP3aiQEke8L9d9jxDVBKWPamqCa0z0ko4IZX3xpIpHPSEKpmwpcJEaK7Z_rai3IBsT5-tqfMKIxnGCwe4SZZED8bDZM9j1T55-LggpjCgpwqWODs4vpj58iMy' #query the webpage and return the html to the variable'soup' html = urllib . request . urlopen ( url ) #parse the downloaded homepage and grab all text soup = BeautifulSoup ( html , 'html.parser' ) #use current time to detect change t = time . ctime ( ) #Search Cursor #fc field where URLs are stored field = [ "Website" ] with arcpy . da . SearchCursor ( fc , field ) as cursor : for row in cursor : print ( row ) ##BeautifulSoup ##count the number of ' ' tags in HTML n = len ( soup . find_all ( 'h2' ) ) print ( n ) #the text of the 26th ' ' tag atts = soup . find_all ( 'h2' ) [ 20 ] . text ( 'http://www.manhattan114.org/index.php/download_file/view/2776/1/' , ) ( 'http://www2.nlsd122.org/files/district/parentsandstudents/message_from_superintendent/2019-2020/mfts_031720.pdf' , ) ( 'https://www.peotoneschools.org/UserFiles/Servers/Server_266769/File/COVID-19%20Email%203.15.20.pdf' , ) ( 'https://manteno5.org/news/what_s_new/c_o_v_i_d-19_updates' , ) ( 'https://www.joliet86.org/student-grab-and-go-meals-available/' , ) ( 'https://www.joliet86.org/student-grab-and-go-meals-available/' , ) ( 'https://www.joliet86.org/student-grab-and-go-meals-available/' , ) ( 'https://www.joliet86.org/student-grab-and-go-meals-available/' , )

bs4 won't parse PDF files, I believe, so you will have to figure out some intermediate step to download the PDF and extract the text. Even after the text is extracted, there are no HTML structure tags with it.

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

SearchCursor and BeautifulSoup

JaredPilbeam2

I'd like to have this script loop through a list of URLs and use a web scraper to search certain things from each of these websites. I have my cursor set up as well as Beautiful soup, but I'm wondering how I identify each item in the list? Can I attach an index number to each one somehow? Here's what I have. If I run this it will print the URLs. I've used Beautiful Soup to find things in HTML before, but I'm not sure how to find things from URLs in a list?

<SPAN class="keyword token">from</SPAN> bs4 <SPAN class="keyword token">import</SPAN> BeautifulSoup
<SPAN class="keyword token">import</SPAN> urllib
<SPAN class="keyword token">import</SPAN> urllib<SPAN class="punctuation token">.</SPAN>request
<SPAN class="keyword token">import</SPAN> os<SPAN class="punctuation token">,</SPAN> arcpy
<SPAN class="keyword token">import</SPAN> time


<SPAN class="comment token">#the hosted layer with website urls in 'Website' field</SPAN>
fc <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'https://services.arcgis.com/fGsbyIOAuxHnF97m/arcgis/rest/services/Grab_Go_School_Meals_Location_(View)/FeatureServer/0?token=fbpCfA34sTJ4rzWO2TQn_c38B4TfGWOZ6jTMeFL1m7CNKd9_odI1t_t_hL-YvvePbE3M428FRT-zW-bISRYrGdJ2CnloKrHoHAfMnbGXpJ-5-zZBU6ONK1u0hMv5D-Vy-fnRpqpQP3aiQEke8L9d9jxDVBKWPamqCa0z0ko4IZX3xpIpHPSEKpmwpcJEaK7Z_rai3IBsT5-tqfMKIxnGCwe4SZZED8bDZM9j1T55-LggpjCgpwqWODs4vpj58iMy'</SPAN>
<SPAN class="comment token">#query the webpage and return the html to the variable'soup'</SPAN>
html <SPAN class="operator token">=</SPAN> urllib<SPAN class="punctuation token">.</SPAN>request<SPAN class="punctuation token">.</SPAN>urlopen<SPAN class="punctuation token">(</SPAN>url<SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">#parse the downloaded homepage and grab all text</SPAN>
soup <SPAN class="operator token">=</SPAN> BeautifulSoup<SPAN class="punctuation token">(</SPAN>html<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'html.parser'</SPAN><SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token">#use current time to detect change</SPAN>
t <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>ctime<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token">#Search Cursor</SPAN>
<SPAN class="comment token">#fc field where URLs are stored</SPAN>
field <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">"Website"</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>fc<SPAN class="punctuation token">,</SPAN> field<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
     <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
        <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>row<SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token">##BeautifulSoup</SPAN>
<SPAN class="comment token">##count the number of '<h1>' tags in HTML</SPAN>
n <SPAN class="operator token">=</SPAN> len<SPAN class="punctuation token">(</SPAN>soup<SPAN class="punctuation token">.</SPAN>find_all<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'h2'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>n<SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">#the text of the 26th '<h2>' tag</SPAN>
atts <SPAN class="operator token">=</SPAN> soup<SPAN class="punctuation token">.</SPAN>find_all<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'h2'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">20</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>text‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'http://www.manhattan114.org/index.php/download_file/view/2776/1/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'http://www2.nlsd122.org/files/district/parentsandstudents/message_from_superintendent/2019-2020/mfts_031720.pdf'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.peotoneschools.org/UserFiles/Servers/Server_266769/File/COVID-19%20Email%203.15.20.pdf'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://manteno5.org/news/what_s_new/c_o_v_i_d-19_updates'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Encuentra más publicaciones etiquetadas con

Accepted answers

All comments

JoshuaBixby

I marked this thread as Assumed Answered because your issues aren't really with cursors but BeautifulSoup, and you are not using cursors anymore and have started a new question about BeautifulSoup.

JaredPilbeam2

I dropped the idea of using a cursor. Now I'm just using Beautifulsoup, but I'm having some additional trouble that seems to be caused by the if else statement. I posted another question: https://community.esri.com/thread/250821-beautifulsoup-if-else-statement

DanPatterson_Retired

sorry... only do soup with lunch

no examples on the web or the beautifulsoup doc site?

Beautiful Soup Documentation — Beautiful Soup 4.4.0 documentation

at crummy.com... got to love it

JaredPilbeam2

Dan,

I've only tested the top half. So, down to line 11. It prints the URLs just fine.

<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'http://www.manhattan114.org/index.php/download_file/view/2776/1/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'http://www2.nlsd122.org/files/district/parentsandstudents/message_from_superintendent/2019-2020/mfts_031720.pdf'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.peotoneschools.org/UserFiles/Servers/Server_266769/File/COVID-19%20Email%203.15.20.pdf'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://manteno5.org/news/what_s_new/c_o_v_i_d-19_updates'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'https://www.joliet86.org/student-grab-and-go-meals-available/'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

DanPatterson_Retired

so, is there some output? or an error? or it does nothing?

JaredPilbeam2

Joshua,

Thanks for the reply. My overall objective is two-part. I want to use the SearchCursor to create a list of URLs from the Website field of a hosted feature layer. That I've done. Secondly, I want to use bs4 to find certain things on each one of the webpages from the list. I'm stuck wondering how I connect the two blocks of code?

Sorry, that end part wasn't even set up for this script yet. It's from something else. This might be a better reference:

<SPAN class="comment token">#the hosted layer with website urls in 'Website' field</SPAN>
fc <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'https://services.arcgis.com/fGsbyIOAuxHnF97m/arcgis/rest/services/Grab_Go_School_Meals_Location_(View)/FeatureServer/0?token=fbpCfA34sTJ4rzWO2TQn_c38B4TfGWOZ6jTMeFL1m7CNKd9_odI1t_t_hL-YvvePbE3M428FRT-zW-bISRYrGdJ2CnloKrHoHAfMnbGXpJ-5-zZBU6ONK1u0hMv5D-Vy-fnRpqpQP3aiQEke8L9d9jxDVBKWPamqCa0z0ko4IZX3xpIpHPSEKpmwpcJEaK7Z_rai3IBsT5-tqfMKIxnGCwe4SZZED8bDZM9j1T55-LggpjCgpwqWODs4vpj58iMy'</SPAN>
<SPAN class="comment token">#use current time to detect change</SPAN>
t <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>ctime<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token">#Search Cursor</SPAN>
<SPAN class="comment token">#fc field where URLs are stored</SPAN>
field <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">"Website"</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>fc<SPAN class="punctuation token">,</SPAN> field<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
     <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
        <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>row<SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token">###Below still under construction###</SPAN>

<SPAN class="comment token">#BeautifulSoup</SPAN>
<SPAN class="comment token">#query the webpage and return the html to the variable'soup'</SPAN>
html <SPAN class="operator token">=</SPAN> urllib<SPAN class="punctuation token">.</SPAN>request<SPAN class="punctuation token">.</SPAN>urlopen<SPAN class="punctuation token">(</SPAN>fc<SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">#parse the downloaded homepage and grab all text</SPAN>
soup <SPAN class="operator token">=</SPAN> BeautifulSoup<SPAN class="punctuation token">(</SPAN>html<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'html.parser'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">#print(soup.prettify())</SPAN>

<SPAN class="comment token">#Count the number of '<h1>' tags in HTML</SPAN>
n <SPAN class="operator token">=</SPAN> len<SPAN class="punctuation token">(</SPAN>soup<SPAN class="punctuation token">.</SPAN>find_all<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'h2'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">#The text of the 26th '<h2>' tag</SPAN>
<SPAN class="comment token">##atts = soup.find_all('h2')[1].text‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

surveyReviewDashboard.png

JoshuaBixby

Can you take a step back and explain your overall objective? From looking at your code, I am not understanding the end part involving using bs4.

Únete a la Conversación

Los miembros registrados pueden publicar, seguir actualizaciones y más. ¿Nuevo aquí? Regístrate gratis.

Enlaces Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.