Solving SSL Errors in Python Requests
The Day the Earth Stood Still Web Expired
On September 30th, 2021, Let's Encrypt (a non-profit certificate authority) allowed one of their certificates to expire, causing any certificate that had it as part of it's chain to show as invalid. This caused quite a bit of mayhem but they did eventually issue a replacement and through reboots and other updates, most of the issue went away.
Except for Python virtual environments installed with ArcGIS Pro. I started having some of my automated processes fail around this time and was not able to resolve them even though we had rebooted machines and servers and manually installed the new replacement certificates locally, but nothing worked. Here's a quick example of something that was failing:
import arcpy, os, requests, json
logMessages = ''
r = requests.session()
r.headers.update({'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'})
body = ''' [out:json]
[timeout:120]
;
(
way
["highway"="service"]
(43.6192138,-116.4338378,43.6339976,-116.4136293);
);
out;
>;
out skel qt;'''
OSM_URL3 = 'https://overpass.kumi.systems/api/interpreter'
# reach out to the Overpass API to grab the data
try:
respSuccess = False
numTries = 0
while respSuccess == False and numTries <= 10:
numTries += 1
response = r.post(OSM_URL3, data=body, verify=True)
if response.status_code == 200:
respSuccess = True
logMessages += '''Requesting features from OSM:
Response Status: {}\n'''.format(response.status_code)
except requests.exceptions.RequestException as e:
logMessages += '{}\n'.format(e.args[0].reason)
print(logMessages)
this would always produce a message like: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)
Because these processes were not priority, it took me a couple months to circle around to digging in a bit more. I just figured out the issue and I want to share because I feel like I only understood the watered-down version of how SSL Certificates work and it appears that my frustration stemmed from that partial understanding. Also, it affected my automation that relied on Python web requests to get data.
How I thought certificate checking works
I originally thought that a certificate just needed validated against some authority and then it would allow traffic to flow.
Not How Certificates Are Verified
How Certificates Actually Work (ish)
My previous understanding caused me to feel stuck because I was experiencing SSL errors trying to download data using python requests and getting failures even though I was able to download the same data using a web browser or other client software. So why was Python the only environment indicating an invalid certificate? Because this is roughly how the verification process actually works.
The crucial detail I was missing was that my own computer had a previously downloaded copy of the certificate and when Python was checking the one received from the remote server against the local, my local copy still showed it was expired.
Almost Solved
Once I understood this, I requested some help from my awesome System Administrator and was shown how to remove an expired certificate from my Windows Certificate Store and install the new one. I know, you're thinking I'm going to show you how to do that but I'm not because it didn't work and there's probably plenty of other sites that will teach you how to do that. I'm posting this because I literally couldn't find it anywhere.
I concluded that Python must not be referencing the certificates in my operating systems certificate store and must use it's own. So I did an uninstall and a fresh install of ArcGIS Pro. This still didn't fix it! So I installed pro on another computer and tried to run the same bit of code on the same network connection. THIS WORKED! But why wouldn't my machine work?
The Real Fix
I did some research and fiddling and found indication that the Requests Library uses it's own certificate bundle (a new term for me) and it was stored at the location provided by running the following code:
import requests
print(requests.certs.where())
So, knowing that ArcGIS Pro had already been reinstalled and it didn't help, the installer must not have replaced this file, it must have just checked it and moved on or appended data to it? Not sure there. Either way, I renamed the old one and put a copy of the cert found on the other machine after a new install and BAM! My machine could now perform a web request through Python without throwing an SSL error.
The Easier Fix
If you're having this same issue, you can most likely do an uninstall of ArcGIS Pro, then make sure to delete the certificate in the location indicated by the requests.certs.where() function before re-installing pro.
I'm sure that it's not recommended to tweak too much with most of the items in a Conda virtual environment, but I have no need to have a custom virtual environment because I'm using only libraries that are already included when you install ArcGIS Pro. However, in this one instance, this did the trick.
I'm hopeful that there's a better way to have a certificate in a virtual environment be updated by running a command and that someone may be able to shine some light on this topic.
