Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Cancel
- Home
- :
- About JaredPilbeam2
JaredPilbeam2
MVP Alum
since
10-17-2017
yesterday
717
Posts created
479
Kudos given
30
Solutions
123
Kudos received
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
My Ideas
(2 Idea Submissions)Latest Contributions by JaredPilbeam2
|
POST
|
" Ditch the list by writing only the string of the date in the list and then comparing it to the string of the date you just retrieved." Joshua, I'm thinking I did just that. But, it sent the email even though they appear to be both strings now? #Find string in webpage
last_update_fra = soup.findAll(string=re.compile("Last Updated"))
#use ''.join to change list object to string
last_update_fra_string = ''.join(last_update_fra)
print(last_update_fra_string)
#write to text file
txt = open(r"C:\Users\jpilbeam\LastUpdate.txt", "w")
txt.write(last_update_fra_string)
txt.close()
txt = open(r"C:\Users\jpilbeam\LastUpdate.txt", "r")
print(txt.read())
if last_update_fra_string == txt:
print("no change")
else:
print("send email") >>>
Last Updated 4/6/2020, 4:45 p.m.
Last Updated 4/6/2020, 4:45 p.m.
send email
>>>
... View more
04-07-2020
11:36 AM
|
0
|
1
|
6284
|
|
POST
|
Randy, Thanks. I'm still working on this. I had another method going based on an answer which isn't working either after testing. It seems like I'm comparing the same thing, so the email never sent when the website eventually changed: python - BeautifulSoup Detect Change Trigger - Stack Overflow So, back here now. I was running the snippets in the interpreter as you've done. When I read the txt file it came back as something different (date has changed as the website has since updated): >>> last_update_fra = ['Last Updated 4/3/2020, 8:28 p.m.']
>>> txt = open(r"C:\Users\jpilbeam\LastUpdate.txt", "r")
>>> type(txt)
<class '_io.TextIOWrapper'>
>>> t = txt.read()
>>> type(t)
<class 'str'>
>>> t
"['Last Updated 4/6/2020, 4:45 p.m.']"
>>> type(last_update_fra)
<class 'list'>
>>> tt = str(last_update_fra)
>>> tt
"['Last Updated 4/3/2020, 8:28 p.m.']"
>>> if t == tt:
... print("match")
...
... View more
04-07-2020
08:41 AM
|
0
|
3
|
6284
|
|
POST
|
Joshua, I have a good grasp on what you're saying, thanks. I'm at the point where I'm checking the two dates against each other. Both the "last_update_fra" and "txt" variables print out the same, but when I do if == there is no result. #Find string
last_update_fra = soup.findAll(string=re.compile("Last Updated"))
print(last_update_fra)
#write "Last Update" to file
txt = open(r"C:\Users\jpilbeam\LastUpdate.txt", "w")
#to write to file parameter has to be a string
txt.write(str(last_update_fra))
txt.close()
#open and read the file
txt = open(r"C:\Users\jpilbeam\LastUpdate.txt", "r")
print(txt.read())
if txt == last_update_fra:
print("good")
>>>
['Last Updated 4/3/2020, 8:28 p.m.']
['Last Updated 4/3/2020, 8:28 p.m.']
>>> Edit: They're not the same. Now I have to figure out why. if txt != last_update_fra:
print("good")
#prints good
... View more
04-05-2020
09:36 AM
|
0
|
0
|
6284
|
|
POST
|
I have a script that uses bs4 to scrape a webpage and grab a string named, "Last Updated 4/3/2020, 8:28 p.m.". I then assign this string to a variable and send it in an email. The script is scheduled to run once a day. However, the date and time on the website change every other day. So, instead of emailing every time I run the script I'd like to set up a trigger so that it sends only when the date is different. How do I configure the script to detect that change? '''Checks municipal websites for changes in meal assistance during C19 pandemic'''
# Import requests (to download the page)
import requests
# Import BeautifulSoup (to parse what we download)
from bs4 import BeautifulSoup
# Import win32com (Python module that controls Outlook)
import win32com.client
from win32com.client import Dispatch, constants
import re
import urllib
import urllib.request
#list of urls
urls = ['http://www.vofil.com/covid19_updates']
#set the headers like we are a browser
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
#download the homepage
response = requests.get(urls[0], headers=headers)
#parse the downloaded homepage and grab all text
soup = BeautifulSoup(response.text, "lxml")
#Find string
last_update_fra = soup.findAll(string=re.compile("Last Updated"))
print(last_update_fra)
#Put something here (if else..?) to trigger an email.
#I left off email block... Here's where the string is in the HTML: <h1>COVID-19 News Updates</h1>
<div><br></div><div>Last Updated 4/3/2020, 12:08 p.m.</div>
... View more
04-04-2020
04:24 PM
|
0
|
8
|
6733
|
|
POST
|
Hi Don, Thanks. I've since figured it out. Here are the full queries: send_email IS NULL Or send_email = '""' And field_2 = 'parcel_or_pin_question?'
send_email IS NULL Or send_email = '""' And field_2 = 'street_or_address_question?'
send_email IS NULL Or send_email = '""' And field_2 = 'tax_related_question?'
send_email IS NULL Or send_email = '""' And field_2 = 'zoning_or_flood_zone_question?'
send_email = '""' Or send_email IS NULL And field_2 = 'other' My feature layer is populated through Survey123. The fields in the layer are populated based off of the entries in the survey. So, for example a survey was just submitted about a 'parcel or pin question'. That field in the layer is in turn populated and once the send emails script runs it will run the query and an email will be sent to the corresponding department (or whoever I have as a recipient). I actually removed Or send_email = '""'. If you have a question about it maybe I can help. By the way, it's a know bug that you can't send to multiple recipients with this tool: ENH-000116673 Provide the ability to enter multiple email addresses in the Recipient Email Address section when using t…
... View more
04-02-2020
12:53 PM
|
0
|
0
|
2285
|
|
POST
|
I did notice that the bs4 search was case sensitive. I've actually looked into ways to deal with that, but forgot about it. I used Dan's approach and it worked. But I had to take it out of the while loop because it was running endlessly. if len(results) <= 3:
do stuff
... View more
04-02-2020
06:36 AM
|
0
|
0
|
4907
|
|
POST
|
I dropped the idea of using a cursor. Now I'm just using Beautifulsoup, but I'm having some additional trouble that seems to be caused by the if else statement. I posted another question: https://community.esri.com/thread/250821-beautifulsoup-if-else-statement
... View more
04-01-2020
07:58 PM
|
0
|
1
|
3719
|
|
POST
|
The if else statement is sending an email no matter what I use in the if part. I'm not sure what the problem is. I tried a bunch of different operators, namely <3, <=3, <2, == -3, == 3 . (1.) I wanted to check the number of times a given word occurred in the HTML. (2.) If it occurred equal to or less than the number given, stop. Anything else, proceed (the script goes on to send an email out). '''Checks school websites for changes in meal assistance during C19 pandemic'''
# Import requests (to download the page)
import requests
# Import BeautifulSoup (to parse what we download)
from bs4 import BeautifulSoup
# Import win32com (to allow email)
import win32com.client
from win32com.client import Dispatch, constants
import re
while True:
#set the url
url = 'https://manteno5.org/news/what_s_new/c_o_v_i_d-19_updates'
#set the headers like we are a browser
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 ( KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
# download the homepage
response = requests.get(url, headers=headers)
# parse the downloaded homepage and grab all text
soup = BeautifulSoup(response.text, "lxml")
#the word I'm looking for
searched_word = 'Grab and Go'
#find how many times the word occurs
results = soup.body.find_all(text=re.compile('{0}'.format(searched_word)), recursive=True)
print('Found the word "{0}" {1} time(s)\n'.format(searched_word, len(results)))
#if the number of times "Grab and Go" occurs on the page is less
#than a given number print "No change"
if str(soup).find(searched_word) == -3:
print("No change")
continue
#but if the word "Grab and Go" occurs any other number of times
else:
#script goes on to send an email...
break It sent the email even though "Grab and Go" only occurred three times? Print out: Found the word "Grab and Go" 3 time(s)
... View more
04-01-2020
07:55 PM
|
0
|
4
|
5107
|
|
POST
|
Dan, I've only tested the top half. So, down to line 11. It prints the URLs just fine. ('http://www.manhattan114.org/index.php/download_file/view/2776/1/',)
('http://www2.nlsd122.org/files/district/parentsandstudents/message_from_superintendent/2019-2020/mfts_031720.pdf',)
('https://www.peotoneschools.org/UserFiles/Servers/Server_266769/File/COVID-19%20Email%203.15.20.pdf',)
('https://manteno5.org/news/what_s_new/c_o_v_i_d-19_updates',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
... View more
03-27-2020
10:34 AM
|
0
|
1
|
3719
|
|
POST
|
Joshua, Thanks for the reply. My overall objective is two-part. I want to use the SearchCursor to create a list of URLs from the Website field of a hosted feature layer. That I've done. Secondly, I want to use bs4 to find certain things on each one of the webpages from the list. I'm stuck wondering how I connect the two blocks of code? Sorry, that end part wasn't even set up for this script yet. It's from something else. This might be a better reference: #the hosted layer with website urls in 'Website' field
fc = r'https://services.arcgis.com/fGsbyIOAuxHnF97m/arcgis/rest/services/Grab_Go_School_Meals_Location_(View)/FeatureServer/0?token=fbpCfA34sTJ4rzWO2TQn_c38B4TfGWOZ6jTMeFL1m7CNKd9_odI1t_t_hL-YvvePbE3M428FRT-zW-bISRYrGdJ2CnloKrHoHAfMnbGXpJ-5-zZBU6ONK1u0hMv5D-Vy-fnRpqpQP3aiQEke8L9d9jxDVBKWPamqCa0z0ko4IZX3xpIpHPSEKpmwpcJEaK7Z_rai3IBsT5-tqfMKIxnGCwe4SZZED8bDZM9j1T55-LggpjCgpwqWODs4vpj58iMy'
#use current time to detect change
t = time.ctime()
#Search Cursor
#fc field where URLs are stored
field = ["Website"]
with arcpy.da.SearchCursor(fc, field) as cursor:
for row in cursor:
print(row)
###Below still under construction###
#BeautifulSoup
#query the webpage and return the html to the variable'soup'
html = urllib.request.urlopen(fc)
#parse the downloaded homepage and grab all text
soup = BeautifulSoup(html, 'html.parser')
#print(soup.prettify())
#Count the number of '<h1>' tags in HTML
n = len(soup.find_all('h2'))
#The text of the 26th '<h2>' tag
##atts = soup.find_all('h2')[1].text
... View more
03-27-2020
09:25 AM
|
0
|
3
|
3719
|
|
POST
|
I'd like to have this script loop through a list of URLs and use a web scraper to search certain things from each of these websites. I have my cursor set up as well as Beautiful soup, but I'm wondering how I identify each item in the list? Can I attach an index number to each one somehow? Here's what I have. If I run this it will print the URLs. I've used Beautiful Soup to find things in HTML before, but I'm not sure how to find things from URLs in a list? from bs4 import BeautifulSoup
import urllib
import urllib.request
import os, arcpy
import time
#the hosted layer with website urls in 'Website' field
fc = r'https://services.arcgis.com/fGsbyIOAuxHnF97m/arcgis/rest/services/Grab_Go_School_Meals_Location_(View)/FeatureServer/0?token=fbpCfA34sTJ4rzWO2TQn_c38B4TfGWOZ6jTMeFL1m7CNKd9_odI1t_t_hL-YvvePbE3M428FRT-zW-bISRYrGdJ2CnloKrHoHAfMnbGXpJ-5-zZBU6ONK1u0hMv5D-Vy-fnRpqpQP3aiQEke8L9d9jxDVBKWPamqCa0z0ko4IZX3xpIpHPSEKpmwpcJEaK7Z_rai3IBsT5-tqfMKIxnGCwe4SZZED8bDZM9j1T55-LggpjCgpwqWODs4vpj58iMy'
#query the webpage and return the html to the variable'soup'
html = urllib.request.urlopen(url)
#parse the downloaded homepage and grab all text
soup = BeautifulSoup(html, 'html.parser')
#use current time to detect change
t = time.ctime()
#Search Cursor
#fc field where URLs are stored
field = ["Website"]
with arcpy.da.SearchCursor(fc, field) as cursor:
for row in cursor:
print(row)
##BeautifulSoup
##count the number of '<h1>' tags in HTML
n = len(soup.find_all('h2'))
print(n)
#the text of the 26th '<h2>' tag
atts = soup.find_all('h2')[20].text ('http://www.manhattan114.org/index.php/download_file/view/2776/1/',)
('http://www2.nlsd122.org/files/district/parentsandstudents/message_from_superintendent/2019-2020/mfts_031720.pdf',)
('https://www.peotoneschools.org/UserFiles/Servers/Server_266769/File/COVID-19%20Email%203.15.20.pdf',)
('https://manteno5.org/news/what_s_new/c_o_v_i_d-19_updates',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
('https://www.joliet86.org/student-grab-and-go-meals-available/',)
... View more
03-27-2020
08:31 AM
|
0
|
8
|
3865
|
|
POST
|
Brilliant. My last notebook was pretty messy, so I started a whole new one. It worked great! Now I have to find a way to run this script from Jupyter via Task Scheduler.
... View more
03-18-2020
02:22 PM
|
0
|
2
|
4643
|
|
POST
|
Here's how I have the url. fc = "https://services.arcgis.com/fGsbyIOAuxHnF97m/arcgis/rest/services/service_0a8c947fd081446eb3a31ef7ca0b966a/FeatureServer/0" I didn't append /0 because I understood that to be the token if the feature wasn't shared.
... View more
03-18-2020
02:12 PM
|
0
|
0
|
4643
|
|
POST
|
Hi Jake, The service is shared with everyone. I put your cursor snippet in a cell and ran it. It printed all the features. So, then i put my code in and got the following error. After reading it I went to the Settings page of the Feature and made sure Add, Update, and Delete Features button was checked. Then I ran the same cell, but it threw the same error.
... View more
03-18-2020
01:55 PM
|
0
|
5
|
4643
|
|
POST
|
This script successfully updates fields in a table based on other fields. I've been testing on a feature class in a FGDB, but my goal is to use this script on a hosted feature layer. For that I'd imagine working in a Jupyter notebook would be the way to go? import arcpy, os
fc = r'\Test\Test.gdb\survey'
fields = ['Email_Category', 'whats_your_question_about',
'tax_related_question', 'zoning_or_flood_zone_question']
with arcpy.da.UpdateCursor(fc, fields) as cursor:
for row in cursor:
if row[1] == 'tax_assessment':
row[0] = 'tax assessment'
if row[2] == 'tax_bill':
row[0] = 'tax bill'
if row[3] == 'flood_zone':
row[0] = 'flood zone'
cursor.updateRow(row)
print("Finished") I have Jupyter all set up and have created an item. But, if I put my script in verbatim I get this error.
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-24-6a7e13cce2ac> in <module>
----> 1 with arcpy.da.UpdateCursor(survey, fields) as cursor:
2 for row in cursor:
3 if row[1] == 'tax_assessment':
4 row[0] = 'tax assessment'
5 if row[2] == 'tax_bill':
RuntimeError: 'in_table' is not a table or a featureclass
... View more
03-18-2020
12:26 PM
|
0
|
7
|
4723
|
My kudoed posts
| Title | Kudos | Posted |
|---|---|---|
| 1 | 3 weeks ago | |
| 1 | 05-04-2026 08:45 AM | |
| 1 | 04-20-2026 01:20 PM | |
| 1 | 07-24-2025 01:27 PM | |
| 1 | 11-13-2025 08:22 AM |
Activity Feed
- Posted Use Select Widget with List widget and Selections Disappear on ArcGIS Experience Builder Questions. Friday
- Kudoed List Widget loses selection when Map is clicked for vijaybadugu. Thursday
- Kudoed Dynamically showing a PDF file in Embed widget for ChristopherKreuter. a week ago
- Kudoed Re: Dynamically showing a PDF file in Embed widget for apgis16. a week ago
- Kudoed Custom print layout for widget that uses map series to print multi-page report and honors element selection? for AllisonMartin. 3 weeks ago
- Kudoed Print a map series within Experience Builder? for HayleyL. 3 weeks ago
- Got a Kudo for Re: Update table so column one stores the value of matching value from column two. 3 weeks ago
- Posted Re: Update table so column one stores the value of matching value from column two on Python Questions. 3 weeks ago
- Posted Re: Update table so column one stores the value of matching value from column two on Python Questions. 3 weeks ago
- Posted Update table so column one stores the value of matching value from column two on Python Questions. 3 weeks ago
- Kudoed Embedded PDFs not showing for Ed_. 3 weeks ago
- Kudoed Open a PDF when Clicking on Polyline in Web Map for MHO_Harbors. 3 weeks ago
- Kudoed Zone Lookup for Brian_McLeer. 3 weeks ago
- Kudoed CSV Geocoder Widget for Brian_McLeer. a month ago
- Kudoed Re: Copying a common widget to be customized for abureaux. 05-18-2026 02:19 PM
- Kudoed Exporting from the widget table directly to PDF for doronwen. 05-14-2026 09:24 AM
- Posted Add CSV Widget on Experience Builder Custom Widgets. 05-13-2026 01:51 PM
- Kudoed Re: Geocode File Location Only Shape X, Shape Y for JonathanLebowitz. 05-08-2026 07:01 AM
- Kudoed Re: Geocode File Location Only Shape X, Shape Y for BobBooth1. 05-07-2026 02:03 PM
- Kudoed Re: Geocode File Location Only Shape X, Shape Y for Laura. 05-07-2026 12:47 PM
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
yesterday
|