I think this gets me the length of the text count for "COVID-19" because it prints 8. import requests from bs4 import BeautifulSoup import re url = r 'https://www.bolingbrook.com/coronavirus' #request webpage soup = BeautifulSoup ( requests . get ( url . content , "lxml" ) #find occurences of string print ( len ( soup . find_all ( string = re . compile ( "COVID-19" ) ) ) ) #prints >> > 8 When I do a CTRL+F for "COVID-19" on the webpage I get a count of 5 occurrences. When I do a CTRL+F for "COVID-19" in the Developer tools I get 15. I'm trying to get the count for the total occurrences of the string "COVID-19". How can I set up the code to do that?

Seems like the easiest way to find out what it's doing is to just print out the results of soup . find_all ( string = re . compile ( "COVID-19" ) ) right? Seems that the function returns a list of all matches, so you could just do something like the following: matches = soup . find_all ( string = re . compile ( "COVID-19" ) ) for match in matches : print ( match ) Also I had a bit of a peek at that webpage, and I think in some places it references "Covid-19" and in others "COVID-19", which with your regular expression would only return the latter.

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Beautifulsoup count occurences of string

jpilbeam

I think this gets me the length of the text count for "COVID-19" because it prints 8.

<SPAN class="keyword token">import</SPAN> requests
<SPAN class="keyword token">from</SPAN> bs4 <SPAN class="keyword token">import</SPAN> BeautifulSoup
<SPAN class="keyword token">import</SPAN> re

url <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'https://www.bolingbrook.com/coronavirus'</SPAN>

<SPAN class="comment token">#request webpage</SPAN>
soup <SPAN class="operator token">=</SPAN> BeautifulSoup<SPAN class="punctuation token">(</SPAN>requests<SPAN class="punctuation token">.</SPAN>get<SPAN class="punctuation token">(</SPAN>url<SPAN class="punctuation token">.</SPAN>content<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"lxml"</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">#find occurences of string</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>len<SPAN class="punctuation token">(</SPAN>soup<SPAN class="punctuation token">.</SPAN>find_all<SPAN class="punctuation token">(</SPAN>string<SPAN class="operator token">=</SPAN>re<SPAN class="punctuation token">.</SPAN>compile<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"COVID-19"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

<SPAN class="comment token">#prints</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="number token">8</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

When I do a CTRL+F for "COVID-19" on the webpage I get a count of 5 occurrences. When I do a CTRL+F for "COVID-19" in the Developer tools I get 15.

I'm trying to get the count for the total occurrences of the string "COVID-19". How can I set up the code to do that?

Finde mehr Beiträge mit dem Tag

Accepted answers

JoshuaSharp-Heward

Seems like the easiest way to find out what it's doing is to just print out the results of

soup<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">.</SPAN>find_all<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">(</SPAN>string<SPAN class="" style="color: #a67f59; background: rgba(255, 255, 255, 0.5); border: 0px; font-weight: inherit;">=</SPAN>re<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">.</SPAN>compile<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">(</SPAN><SPAN class="" style="color: #669900; border: 0px; font-weight: inherit;">"COVID-19"</SPAN><SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">)</SPAN><SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">)</SPAN>

right? Seems that the function returns a list of all matches, so you could just do something like the following:

matches <SPAN class="operator token">=</SPAN> soup<SPAN class="punctuation token">.</SPAN>find_all<SPAN class="punctuation token">(</SPAN>string<SPAN class="operator token">=</SPAN>re<SPAN class="punctuation token">.</SPAN>compile<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"COVID-19"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">for</SPAN> match <SPAN class="keyword token">in</SPAN> matches<SPAN class="punctuation token">:</SPAN>
   <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>match<SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Also I had a bit of a peek at that webpage, and I think in some places it references "Covid-19" and in others "COVID-19", which with your regular expression would only return the latter.

All comments

JoshuaBixby

I can't say I fully understand what your COVID-19 searches are meant to reflect, but I am quite certain the results from the Developer tool are not what you are after. HTML is stylized content, and there are plenty of reasons COVID-19 could should up in parts of HTML that would not be what you are trying to capture with your search.

In order to extract meaning from scraping HTML pages, you need a decent understanding ahead of time about the HTML structure of the site/pages. Most web pages today are dynamically generated from content management systems with fairly organized structure, i.e., there are set banners, sidebars, footers, ..., and main content. Which part of the page you want to analyze and how you analyze it varies from question to question.

jpilbeam

Apologies if I wasn't very clear. I was more or less wondering if the len() function was doing what I intended it to, which was to count the occurrences of the COVID-19 element not the length of the string.

The script is meant to find the number of times "COVID-19" occurs in that URL. If that number changes the next time the script runs (through Task Scheduler) it triggers an email to one of the GIS staff members here (I left those parts out for clarification).

I see what you mean, though. You have to really study the HTML in able to get the script to accurately find things. It's a little difficult to find time to know a website that good when you're looking at every municipal and school district website in the county, however. They change constantly.

JoshuaSharp-Heward

Seems like the easiest way to find out what it's doing is to just print out the results of

soup<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">.</SPAN>find_all<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">(</SPAN>string<SPAN class="" style="color: #a67f59; background: rgba(255, 255, 255, 0.5); border: 0px; font-weight: inherit;">=</SPAN>re<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">.</SPAN>compile<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">(</SPAN><SPAN class="" style="color: #669900; border: 0px; font-weight: inherit;">"COVID-19"</SPAN><SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">)</SPAN><SPAN class="" style="color: #999999; border: 0px; font-weight: inherit;">)</SPAN>

right? Seems that the function returns a list of all matches, so you could just do something like the following:

matches <SPAN class="operator token">=</SPAN> soup<SPAN class="punctuation token">.</SPAN>find_all<SPAN class="punctuation token">(</SPAN>string<SPAN class="operator token">=</SPAN>re<SPAN class="punctuation token">.</SPAN>compile<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"COVID-19"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">for</SPAN> match <SPAN class="keyword token">in</SPAN> matches<SPAN class="punctuation token">:</SPAN>
   <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>match<SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Also I had a bit of a peek at that webpage, and I think in some places it references "Covid-19" and in others "COVID-19", which with your regular expression would only return the latter.

jpilbeam

You're right. That's exactly what I did. I put the print statement in, and simply counted the results. There are 8 of them. I was thinking it was counting the length of "COVID-19" because there are 8 characters there too.

#printed this
>>>
COVID-19 Message
Village of Bolingbrook COVID-19 Update 05.06.20
COVID-19 Message
Village of Bolingbrook COVID-19 Update 05.06.20
The Village of Bolingbrook has been in constant communication with both Amita Bolingbrook Hospital and Edward Hospital, along with the Will County and Illinois Department of Public Health to monitor the possible spread of the Coronavirus (COVID-19) in the Bolingbrook area. We are following the most current recommendations for treatment and preventing the potential spread of infection. Protective equipment and procedures are in place to provide emergency medical treatment and safe transport of individuals to the hospital.&nbsp;
COVID-19 HELP
Will County COVID-19 Cases
Here in Northern Illinois, we have had 41 blood drives canceled due to coronavirus concerns, resulting in approximately 1,500 uncollected blood donations.&nbsp; As the number of COVID-19 cases grow, we do expect that number to increase unfortunately.&nbsp; That’s why we are asking organizations to please keep their blood drives and for donors to continue to give.&nbsp; Together, we must ensure a readily available blood supply for patients who are counting on us.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Also, good point. CTRL+F in the developer tools seems to not be case-sensitive. I'll go ahead and mark your answer correct since it was the closest thing to my not-so-clear question.

Nimm an der Diskussion teil

Angemeldete Mitglieder können Beiträge verfassen, Updates folgen und mehr. Neu hier? Registriere ein kostenloses Konto.

Schnellzugriffe

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.