URLLib.Request Errors

jaykapalczynski · ‎08-05-2022

Hello... I am trying to grab some JSON text that is on a webpage via a WordPress Api output. There is nothing on this page except the JSON string.

If I go to CMD prompt and navigate to my Python folder I can confirm I have python and PIP

C:\Python310>py --version
Python 3.10.6

C:\Python310>py -m pip --version
pip 22.2.1 from C:\Python310\lib\site-packages\pip (python 3.10)

import arcpy
import requests
import json
import uuid
from urllib.request import urlopen
import urllib
import requests

url = "https://somewebsite/"

response = urlopen(url)
data_json = json.loads(response.read)
print (data_json)

I am trying to above code but keep getting this error:

Traceback (most recent call last):
File "C:\Users\t\Desktop\JSONSamples\readJson.py", line 7, in <module>
from urllib.request import urlopen
ImportError: No module named request

Any thoughts? Do I need to install something additional to Python? Any help would be much appreciated.

Anonymous User · ‎08-05-2022

pip is the mechanism used to get packages, you should check if you have urllib3 installed by opening the python terminal at your environment (like you did) and doing pip list. urllib is part of the standard python library install...

if its not listed, then you can pip install urllib3

jaykapalczynski · ‎08-05-2022

Maybe a firewall issue on my side? Can I download it and install locally?

I try at this location

Tryin to install urllib3

I can get their via web browser

Anonymous User · ‎08-05-2022

Note that this is not the python environment that is installed for Pro. Is your IDE pointing to this python install, or Pro's default environment?

If you look at the start menu folder under arcgis, you should see a Python Command Prompt. Starting that should open a cmd terminal with the active Pro, conda environment. pip list should work there to list all installed packages.

it will look something like: (assessors-env) C:\Program Files\ArcGIS\Pro\bin\Python\envs\assessors-env>

I also get different results using the full python instead of py.

jaykapalczynski · ‎08-05-2022

OK I was able to go into the terminal under GIS and do PIP LIST

Then went to install urllib3 and says that its already satisfied

Let me step back a bit I am running this little snip of code from a file with .py extension through the IDLE interface. I am not running this via ArcGIS Pro. I want to be able to put this .py file on the network and write a .bat file to run in task scheduler. I could be going about this the wrong way ...

It says below that the requirement is already satisfied but when I run through IDLE it give me that error. This is a learning curve and I thank you for your patience and input. Cheers

So maybe as you stated I need to make sure the IDLE interface is pointed to ArcGIS Pro or the correct python install as there is still older versions on this computer - not sure how to check or do that.

Anonymous User · ‎08-05-2022

sweet! The system python might be different than the Pro's python.

I want to be able to put this .py file on the network and write a .bat file to run in task scheduler. I could be going about this the wrong way ...

You can designate the specific python.exe in Task scheduler so you can skip the bat file step. In the actions settings: paste the path to the Pro env and the path to your script like this:

C\:Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe 'path to your script'

and it will parse the path to your script as an argument, which is ok. It will start your script using Pro's python environment and python interpreter.

... you can have that python.exe path and path to your script in the bat file in the same format and it should tell it to start that interpreter and run this script. Either way, I think you need to tell it which python.exe to use.

---

As for setting the interpreter/ env in IDLE, when you open it, it should list the env/ interpreter at the top:

Python 2.7.18 (v2.7.18:8d21aa21f2, Apr 20 2020, 13:19:08) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license()" for more information.

You can also use: python at a normal cmd prompt and it should list what python interpreter the system (and most likely the python that is being used to execute in bat file) is using.

jaykapalczynski · ‎08-08-2022

OK so here is where I am currently at. I am pretty confused why this is not working....

Below are my 4 options when I go to run the .py file

Code

This is the URL I am using

import arcpy
import requests
import json
import uuid
from urllib.request import urlopen
import urllib
import requests

url = "https://dwr.virginia.gov/-/api/v1/vbwt-sites/"
response = urlopen(url)
data_json = json.loads(response.read)
print (data_json)

1st Option - Edit with IDLE

This open it in IDLE 2.7 which was a prior install of python that is on this computer

Error -

Traceback (most recent call last):
File "C:\Users\tjv36463\Desktop\JSONSamples\readJson.py", line 5, in <module>
from urllib.request import urlopen
ImportError: No module named request

2nd Option - Edit with IDLE (ArcGIS Pro)

If I choose this option

Error -

Traceback (most recent call last):
File "C:\Users\tjv36463\Desktop\JSONSamples\readJson.py", line 13, in <module>
response = urlopen(url)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

3rd Option - Edit with IDLE (Edit with IDLE 3.10 (64-bit))

The 3rd options opens in this

Error -

Traceback (most recent call last):
File "C:\Users\tjv36463\Desktop\JSONSamples\readJson.py", line 1, in <module>
import arcpy
ModuleNotFoundError: No module named 'arcpy'

If I run it from PYTHON command prompt I get this

StuartMoore · ‎08-08-2022

Hi @jaykapalczynski ,

i just tried this and it seemed to work and pulled back 648 records

import requests
import json

fsURL = "https://dwr.virginia.gov/-/api/v1/vbwt-sites"
r3 = requests.get(fsURL)

res = json.loads (r3.text)

print(len(res))

that was done with idle for ArcGIS Pro

hope that helps

Stu

Anonymous User · ‎08-08-2022

Use option 2. Since you are getting an API response (the JSON), you need to use a different urllib method.

PS- its good practise to look at the sites robots.txt file to see what they allow as far as automation/scraping.

https://dwr.virginia.gov/robots.txt

import arcpy
import requests
import json
import uuid
import urllib
# import requests <- duplicate import

url = "https://dwr.virginia.gov/-/api/v1/vbwt-sites/"

# add the headers because the server doesnt allow resquests from python
response = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
infile=urllib.request.urlopen(response).read()
data_json = json.loads(infile)
print (data_json)

Anonymous User · ‎08-08-2022

Just for FYI, option one (2.7) IDLE is using the Python 2.7 environment, which you noted is from an old install, (but may still be used for other programs) and since you have been working in 3.10, this 2.7 will not have any of the packages that you installed. And its syntax is different so it would most likely fail trying to interpret the 3.x syntax.

Option 2 is pointing to the IDLE packaged with Pro, so good there.

Option 3 is pointing to the base Python3.10 install for your system and since Pro has it's own, arcpy is not installed into the system environment.

Option 4 is good, but it was getting tripped up trying to decode the page.

Take a look at getting another python editor- like pycharm, vs code, spyder... Set these to your Pro python environment and things may be a little easier for your development.