topic Re: URLLib.Request Errors in Python Questions

URLLib.Request Errors

jaykapalczynski — Fri, 05 Aug 2022 16:36:36 GMT

Hello... I am trying to grab some JSON text that is on a webpage via a WordPress Api output. There is nothing on this page except the JSON string.

If I go to CMD prompt and navigate to my Python folder I can confirm I have python and PIP

C:\Python310>py --version Python 3.10.6 C:\Python310>py -m pip --version pip 22.2.1 from C:\Python310\lib\site-packages\pip (python 3.10)

import arcpy import requests import json import uuid from urllib.request import urlopen import urllib import requests url = "https://somewebsite/" response = urlopen(url) data_json = json.loads(response.read) print (data_json)

I am trying to above code but keep getting this error:

Traceback (most recent call last):
File "C:\Users\t\Desktop\JSONSamples\readJson.py", line 7, in <module>
from urllib.request import urlopen
ImportError: No module named request

Any thoughts? Do I need to install something additional to Python? Any help would be much appreciated.

Re: URLLib.Request Errors

Anonymous User — Fri, 05 Aug 2022 17:03:58 GMT

pip is the mechanism used to get packages, you should check if you have urllib3 installed by opening the python terminal at your environment (like you did) and doing pip list. urllib is part of the standard python library install...

if its not listed, then you can pip install urllib3

Re: URLLib.Request Errors

jaykapalczynski — Fri, 05 Aug 2022 17:19:58 GMT

Maybe a firewall issue on my side? Can I download it and install locally?

I try at this location

Tryin to install urllib3

I can get their via web browser

Re: URLLib.Request Errors

Anonymous User — Fri, 05 Aug 2022 17:55:52 GMT

Note that this is not the python environment that is installed for Pro. Is your IDE pointing to this python install, or Pro's default environment?

If you look at the start menu folder under arcgis, you should see a Python Command Prompt. Starting that should open a cmd terminal with the active Pro, conda environment. pip list should work there to list all installed packages.

it will look something like: (assessors-env) C:\Program Files\ArcGIS\Pro\bin\Python\envs\assessors-env>

I also get different results using the full python instead of py.

Re: URLLib.Request Errors

jaykapalczynski — Fri, 05 Aug 2022 20:00:53 GMT

OK I was able to go into the terminal under GIS and do PIP LIST

Then went to install urllib3 and says that its already satisfied

Let me step back a bit I am running this little snip of code from a file with .py extension through the IDLE interface. I am not running this via ArcGIS Pro. I want to be able to put this .py file on the network and write a .bat file to run in task scheduler. I could be going about this the wrong way ...

It says below that the requirement is already satisfied but when I run through IDLE it give me that error. This is a learning curve and I thank you for your patience and input. Cheers

So maybe as you stated I need to make sure the IDLE interface is pointed to ArcGIS Pro or the correct python install as there is still older versions on this computer - not sure how to check or do that.

Re: URLLib.Request Errors

Anonymous User — Fri, 05 Aug 2022 20:41:24 GMT

sweet! The system python might be different than the Pro's python.

I want to be able to put this .py file on the network and write a .bat file to run in task scheduler. I could be going about this the wrong way ...

You can designate the specific python.exe in Task scheduler so you can skip the bat file step. In the actions settings: paste the path to the Pro env and the path to your script like this:

C\:Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe 'path to your script'

and it will parse the path to your script as an argument, which is ok. It will start your script using Pro's python environment and python interpreter.

... you can have that python.exe path and path to your script in the bat file in the same format and it should tell it to start that interpreter and run this script. Either way, I think you need to tell it which python.exe to use.

---

As for setting the interpreter/ env in IDLE, when you open it, it should list the env/ interpreter at the top:

Python 2.7.18 (v2.7.18:8d21aa21f2, Apr 20 2020, 13:19:08) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license()" for more information.

You can also use: python at a normal cmd prompt and it should list what python interpreter the system (and most likely the python that is being used to execute in bat file) is using.

Re: URLLib.Request Errors

StuartMoore — Mon, 08 Aug 2022 06:28:12 GMT

this might help:

fsURL = "http://yourwebsite.com" payload3 = {'updates': updateAttr, 'f': 'json','token' : aToken} r3 = requests.get(fsURL, params=payload3, proxies=proxies) print(r3.text)

just ignore the payload if you dont need to pass any parameters, same goes for the proxy

forgot to say the above just uses requests

Stu

Re: URLLib.Request Errors

jaykapalczynski — Mon, 08 Aug 2022 13:01:58 GMT

OK so here is where I am currently at. I am pretty confused why this is not working....

Below are my 4 options when I go to run the .py file

Code

This is the URL I am using

import arcpy import requests import json import uuid from urllib.request import urlopen import urllib import requests url = "https://dwr.virginia.gov/-/api/v1/vbwt-sites/" response = urlopen(url) data_json = json.loads(response.read) print (data_json)

1st Option - Edit with IDLE

This open it in IDLE 2.7 which was a prior install of python that is on this computer

Error -

Traceback (most recent call last):
File "C:\Users\tjv36463\Desktop\JSONSamples\readJson.py", line 5, in <module>
from urllib.request import urlopen
ImportError: No module named request

2nd Option - Edit with IDLE (ArcGIS Pro)

If I choose this option

Error -

Traceback (most recent call last):
File "C:\Users\tjv36463\Desktop\JSONSamples\readJson.py", line 13, in <module>
response = urlopen(url)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

3rd Option - Edit with IDLE (Edit with IDLE 3.10 (64-bit))

The 3rd options opens in this

Error -

Traceback (most recent call last):
File "C:\Users\tjv36463\Desktop\JSONSamples\readJson.py", line 1, in <module>
import arcpy
ModuleNotFoundError: No module named 'arcpy'

If I run it from PYTHON command prompt I get this

Re: URLLib.Request Errors

StuartMoore — Mon, 08 Aug 2022 13:16:11 GMT

Hi @jaykapalczynski ,

i just tried this and it seemed to work and pulled back 648 records

import requests import json fsURL = "https://dwr.virginia.gov/-/api/v1/vbwt-sites" r3 = requests.get(fsURL) res = json.loads (r3.text) print(len(res))

that was done with idle for ArcGIS Pro

hope that helps

Stu

Re: URLLib.Request Errors

Anonymous User — Mon, 08 Aug 2022 13:20:18 GMT

Use option 2. Since you are getting an API response (the JSON), you need to use a different urllib method.

PS- its good practise to look at the sites robots.txt file to see what they allow as far as automation/scraping.

https://dwr.virginia.gov/robots.txt

import arcpy import requests import json import uuid import urllib # import requests <- duplicate import url = "https://dwr.virginia.gov/-/api/v1/vbwt-sites/" # add the headers because the server doesnt allow resquests from python response = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'}) infile=urllib.request.urlopen(response).read() data_json = json.loads(infile) print (data_json)

Re: URLLib.Request Errors

Anonymous User — Mon, 08 Aug 2022 13:33:07 GMT

Just for FYI, option one (2.7) IDLE is using the Python 2.7 environment, which you noted is from an old install, (but may still be used for other programs) and since you have been working in 3.10, this 2.7 will not have any of the packages that you installed. And its syntax is different so it would most likely fail trying to interpret the 3.x syntax.

Option 2 is pointing to the IDLE packaged with Pro, so good there.

Option 3 is pointing to the base Python3.10 install for your system and since Pro has it's own, arcpy is not installed into the system environment.

Option 4 is good, but it was getting tripped up trying to decode the page.

Take a look at getting another python editor- like pycharm, vs code, spyder... Set these to your Pro python environment and things may be a little easier for your development.