topic Re: Python script stopped working. Need help in Python Snippets Questions

Python script stopped working. Need help

JohnBrand — Sun, 12 Dec 2021 09:18:05 GMT

I have a small Python script that essentially downloads a zip file, unzips it, and saves the contents (shapefiles) to a folder on my network. It stopped working not too long ago and I'm not sure why. I was testing each line in the Python Consol, and when it got to the last line shown below (zipresp = urlopen(zipurl)), it gave me a bunch of errors that I don't understand. Can anyone tell me more on the errors?

from urllib.request import urlopen
from zipfile import ZipFile

zipurl = "http://apps.ohiodnr.gov/geodata/Statewide/OGWells_statewide.zip"

# Download the file from the URL
zipresp = urlopen(zipurl)

Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\http\client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\http\client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\http\client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\http\client.py", line 1036, in _send_output
self.send(msg)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\http\client.py", line 974, in send
self.connect()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\http\client.py", line 1415, in connect
server_hostname=server_hostname)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\ssl.py", line 817, in __init__
self.do_handshake()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-0dab62d9e32b>", line 7, in <module>
zipresp = urlopen(zipurl)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 564, in error
result = self._call_chain(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 756, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 526, in open
response = self._open(req, data)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 544, in _open
'_open', req)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\urllib\request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>

Re: Python script stopped working. Need help

DanPatterson — Wed, 08 Jul 2020 14:46:59 GMT

John Brand‌ were you always using python 3 when it worked? (changes occurred from python 2.7 into 3 and onward)

Re: Python script stopped working. Need help

JohnBrand — Wed, 08 Jul 2020 15:41:40 GMT

Yes, I was using python 3 when I originally put the script together.

Re: Python script stopped working. Need help

GeoJosh — Wed, 08 Jul 2020 17:51:44 GMT

John,

It looks like the server's certificate can't be validated. Have a look at the top comment in the StackOverflow discussion below. Creating a new SSLContext and passing it with the request should resolve the issue.

python - urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error - Stack Overflow

-Josh

Re: Python script stopped working. Need help

MichaelVolz — Wed, 08 Jul 2020 18:50:34 GMT

Are http requests still allowed? I thought ESRI and the IT industry in general was moving for the more secure https protocol.

Re: Python script stopped working. Need help

JohnBrand — Wed, 08 Jul 2020 19:47:11 GMT

Sorry, I really don't know anything about this stuff.... Who's server certificate are you referring to? Is it good practice to have a certificate verified?

Re: Python script stopped working. Need help

GeoJosh — Wed, 08 Jul 2020 19:58:49 GMT

You're right, but there's still a large number of organizations who haven't made the switch yet. This line of code should work if the OP can bypass the cert verification step OR make the request over HTTPS.

Re: Python script stopped working. Need help

GeoJosh — Wed, 08 Jul 2020 20:02:04 GMT

No worries! Michael Volz above mentioned something that I didn't notice. You're making the request over HTTP instead of HTTPS. Do you get the same behavior if you change the URL to https://apps.ohiodnr.gov/geodata/Statewide/OGWells_statewide.zip?

In my first post, I'm referring to the certificate presented by apps.ohiodnr.gov and yes, ideally this certificate would pass validation in a script like this.

Re: Python script stopped working. Need help

JohnBrand — Wed, 08 Jul 2020 20:45:06 GMT

I changed the script to https and I received the error messages again.

Re: Python script stopped working. Need help

GeoJosh — Wed, 08 Jul 2020 20:50:37 GMT

I'm not sure why that cert is failing validation. As mentioned above, the workaround is to skip the validation process all together. The code below worked for me:

from urllib.request import urlopen
from zipfile import ZipFile
import ssl

zipurl = "https://apps.ohiodnr.gov/geodata/Statewide/OGWells_statewide.zip"

# Download the file from the URL
gcontext = ssl.SSLContext()
zipresp = urlopen(zipurl, context=gcontext)