topic Using Python to scrape points from a google map embedded on a website in Python Questions

Using Python to scrape points from a google map embedded on a website

AliWertheim1 — Sat, 11 Dec 2021 16:37:49 GMT

Hi everyone,

I am trying to use the requests library + beautiful soup to pull information on the antenna points from the map shown on this website.

BIPT: Sitebeheer

My original plan was to iterate through site numbers, pull out the lat/long information that appears on the left hand panel when a point is clicked, and display that data in Arc. So far I have accessed the <div> element where that information is located when the point is clicked (<div> id = selectedsite ... </div>). But the element turns up empty in python seemingly because nothing is clicked?

This is my first time web scraping and I have limited HTML knowlege, if there is another approach that would be better or any pointers you could offer that would be greatly appreciated

import requests
from bs4 import BeautifulSoup

#api_key = 'AIzaSyAv5Ru8AQCc9ZvfcrdYEg9SGHyDZqv55T0'

#Prettify HTML
source = requests.get("http://www.sites.bipt.be/").text
soup = BeautifulSoup(source, 'lxml')
print(soup.prettify())


# Class where information on the selected site is located - turns up empty
div = soup.find(id='selectedsite')
print(div.prettify())
‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Using Python to scrape points from a google map embedded on a website

DanPatterson_Retired — Wed, 15 Jan 2020 14:40:28 GMT

Are any esri products involved in the process? or is this a pure python question?

If the latter...

Stack Overflow - Where Developers Learn, Share, & Build Careers

might be a better venue

Re: Using Python to scrape points from a google map embedded on a website

AliWertheim1 — Wed, 15 Jan 2020 14:58:07 GMT

Thanks for this advice - I have cross posted on stack overflow as well.

Re: Using Python to scrape points from a google map embedded on a website

BruceHarold — Wed, 15 Jan 2020 15:34:22 GMT

I can see a link to download a spreadsheet which exposes coordinates:

http://www.sites.bipt.be/lijst-liste.xlsx

I simplified it (attached, using Data Interoperability extension).

Then Convert Coordinate Notation tool can expose lat/long better: