Solving SSL Errors in Python Requests

12-08-2021 12:14 PM
Occasional Contributor II

The Day the Earth Stood Still Web Expired

On September 30th, 2021, Let's Encrypt (a non-profit certificate authority) allowed one of their certificates to expire, causing any certificate that had it as part of it's chain to show as invalid. This caused quite a bit of mayhem but they did eventually issue a replacement and through reboots and other updates, most of the issue went away.

Except for Python virtual environments installed with ArcGIS Pro. I started having some of my automated processes fail around this time and was not able to resolve them even though we had rebooted machines and servers and manually installed the new replacement certificates locally, but nothing worked. Here's a quick example of the something that was failing:





import arcpy, os, requests, json

logMessages = ''

r = requests.session()

r.headers.update({'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'})

body = '''  [out:json]
out skel qt;'''

OSM_URL3 = ''

# reach out to the Overpass API to grab the data
	respSuccess = False
	numTries = 0
	while respSuccess == False and numTries <= 10:
		numTries += 1
		response =, data=body, verify=True)
		if response.status_code == 200:
			respSuccess = True
	logMessages += '''Requesting features from OSM:
	Response Status: {}\n'''.format(response.status_code)

except requests.exceptions.RequestException as e:
	logMessages += '{}\n'.format(e.args[0].reason)





this would always produce a message like: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)

Because these processes were not priority, it took me a couple months to circle around to digging in a bit more. I just figured out the issue and I want to share because I feel like I only understood the watered-down version of how SSL Certificates work and it appears that my frustration stemmed from that partial understanding. Also, it affected my automation that relied on Python web requests to get data.

How I thought certificate checking works

I originally thought that a certificate just needed validated against some authority and then it would allow traffic to flow. 

Not How Certificates Are VerifiedNot How Certificates Are Verified


How Certificates Actually Work (ish)

My previous understanding caused me to feel stuck because I was experiencing SSL errors trying to download data using python requests and getting failures even though I was able to download the same data using a web browser or other client software. So why was Python the only environment indicating an invalid certificate? Because this is roughly how the verification process actually works.


 The crucial detail I was missing was that my own computer had a previously downloaded copy of the certificate and when Python was checking the one received from the remote server against the local, my local copy still showed it was expired.

Almost Solved

Once I understood this, I requested some help from my awesome System Administrator and was shown how to remove an expired certificate from my Windows Certificate Store and install the new one. I know, you're thinking I'm going to show you how to do that but I'm not because it didn't work and there's probably plenty of other sites that will teach you how to do that. I'm posting this because I literally couldn't find it anywhere.

I concluded that Python must not be referencing the certificates in my operating systems certificate store and must use it's own. So I did an uninstall and a fresh install of ArcGIS Pro. This still didn't fix it! So I installed pro on another computer and tried to run the same bit of code on the same network connection. THIS WORKED! But why wouldn't my machine work?

The Real Fix

I did some research and fiddling and found indication that the Requests Library uses it's own certificate bundle (a new term for me) and it was stored at the location provided by running the following code:





import requests





So, knowing that ArcGIS Pro had already been reinstalled and it didn't help, the installer must not have replaced this file, it must have just checked it and moved on or appended data to it? Not sure there. Either way, I renamed the old one and put a copy of the cert found on the other machine after a new install and BAM! My machine could now perform a web request through Python without throwing an SSL error.

The Easier Fix

If you're having this same issue, you can most likely do an uninstall of ArcGIS Pro, then make sure to delete the certificate in the location indicated by the requests.certs.where() function before re-installing pro.

I'm sure that it's not recommended to tweak too much with most of the items in a Conda virtual environment, but I have no need to have a custom virtual environment because I'm using only libraries that are already included when you install ArcGIS Pro. However, in this one instance, this did the trick.

I'm hopeful that there's a better way to have a certificate in a virtual environment be updated by running a command and that someone may be able to shine some light on this topic.

2 Replies
New Contributor

I had the same issue when trying to decode some addresses using Nominatim. I really want to thank you for you suggestion, I used cacert.pem from my friend laptop, who the didn't got the error using arcpy.

Occasional Contributor II

Glad to hear. I feel like there must have been something wrong with my install for it not to have updated that file when I reinstalled. Happy coding!

0 Kudos