When ArcGIS Online updated to HTTPS only, I became aware that one of my attribute tables (a series of business listings) had a ton of HTTP URLs. Not only that, but many of the URLs were old and the business owner had redirected to a new site.
I went searching, and found this article that outlines how to scan through a list of URLs to not only see if it is HTTPS, but to also let me know if it redirects to a new URL.
Check it out below. This can easily be leveraged in ArcGIS Pro and Notebooks.
import requests
import urllib.request
import sys
URLS = ["https://www.websiteA.com/Place/Lafayette_IN.htm",
"http://www.websiteB.com/",
"https://www.websiteC.com/home"]
for URL in URLS:
HTTP_URL = URL.replace("https:","http:") # convert all URLs to HTTP for the first attempt
r = requests.get(HTTP_URL) # first we try http
actual = r.url # return the response URL from the website
print(actual)
