Select to view content in your preferred language

Use Python-API behind Proxy

17246
27
06-13-2017 05:47 AM
MU
by
New Contributor III

Hi,
I'm trying to use the new Python-API with our on-premise system. Our company runs it's own public key infrastructure, so I've configured Portal for ArcGIS with a corresponding certificate. Furthermore, there is a company-wide proxy that we are using.

As a first throw, I'd like to connect to ArcGIS Online by doing this:

from arcgis.gis import GIS
my_gis = GIS()

This crashes with:

c:\Program Files\ArcGIS\Pro\bin\Python>python.exe "C:\temp\Test.py"
Traceback (most recent call last):
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\urllib\request.py", line 1254
, in do_open
 h.request(req.get_method(), req.selector, req.data, headers)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 1106, i
n request
 self._send_request(method, url, body, headers)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 1151, i
n _send_request
 self.endheaders(body)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 1102, i
n endheaders
 self._send_output(message_body)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 934, in
 _send_output
 self.send(msg)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 877, in
 send
 self.connect()
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 1252, i
n connect
 super().connect()
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 853, in
 connect
 self._tunnel()
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\http\client.py", line 832, in
 _tunnel
 message.strip()))
OSError: Tunnel connection failed: 407 authenticationrequired

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "C:\temp\Test.py", line 7, in <module>
 my_gis = GIS()
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\site-packages\arcgis\gis.py",
 line 95, in __init__
 verify_cert=self._verify_cert, client_id=self._client_id)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\site-packages\arcgis\_impl\po
rtalpy.py", line 160, in __init__
 client_id=client_id)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\site-packages\arcgis\_impl\co
nnection.py", line 281, in __init__
 self.login(username, password, expiration, client_id)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\site-packages\arcgis\_impl\co
nnection.py", line 504, in login
 resp = self.post('', { 'f': 'json' }, add_token=False) # probe portal to fin
d auth scheme
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\site-packages\arcgis\_impl\co
nnection.py", line 1031, in post
 resp = opener.open(url, data=encoded_postdata.encode())
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\urllib\request.py", line 466,
 in open
 response = self._open(req, data)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\urllib\request.py", line 484,
 in _open
 '_open', req)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\urllib\request.py", line 444,
 in _call_chain
 result = func(*args)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\urllib\request.py", line 1297
, in https_open
 context=self._context, check_hostname=self._check_hostname)
 File "c:\Program Files\ArcGIS\Pro\bin\Python\lib\urllib\request.py", line 1256
, in do_open
 raise URLError(err)
urllib.error.URLError: <urlopen error Tunnel connection failed: 407 authenticati
onrequired>

Is it related to some proxy issues and if yes, where do I have to configure the proxy?

Tags (2)
27 Replies
Brian_Wilson
Regular Contributor II

If I change the proxy_host to use a name instead of an IP address, the error changes to 

SSLError: Please set verify_cert=False due to encountered SSL error: HTTPSConnectionPool(host='delta.co.clatsop.or.us', port=443): Max retries exceeded with url: /portal/sharing/rest/info?f=json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

and it goes without saying I tried "verify_cert=False"

 

Higher up in the stack trace I see this

MaxRetryError: HTTPSConnectionPool(host='delta.co.clatsop.or.us', port=443): Max retries exceeded with url: /portal/sharing/rest/info?f=json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

0 Kudos
LeonAus
New Contributor III

I am also unable to set the right proxy settings to be able to connect to portal through the python api.

I need to be able to send the https request to a http proxy address and it seems that the python api doesn't support this. What's frustrating is that if I do the same request outside of the python api, using Requests or urllib, I am able to connect. The following request is successful:

proxy = {'http': 'http://<ipaddress>:8080', 'https': 'http://<ipaddress>:8080'}
import requests
x = requests.get('<portaladdress>/portal/sharing/rest/info?f=json', proxies=proxy)

 

But if I try and use the same proxy setting for the python api I get a getaddrinfo error:

>>> from arcgis.gis import GIS
>>> GIS('https://<portaladdress>/portal', username="Leon", password="xxx", verify_cert=False, proxy=proxy)
Traceback (most recent call last):
File "...\python-env\lib\site-packages\urllib3\connection.py", line 170, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "...\python-env\lib\site-packages\urllib3\util\connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "...\python-env\lib\socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "...\python-env\lib\site-packages\urllib3\connectionpool.py", line 706, in urlopen
chunked=chunked,
File "...\python-env\lib\site-packages\urllib3\connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "...\python-env\lib\site-packages\urllib3\connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "...\python-env\lib\site-packages\urllib3\connection.py", line 353, in connect
conn = self._new_conn()
File "...\python-env\lib\site-packages\urllib3\connection.py", line 182, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x00000280CA721108>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "...\python-env\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "...\python-env\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "...\python-env\lib\site-packages\urllib3\util\retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<portal_address>', port=443): Max retries exceeded with url: /portal/sharing/rest/info?f=json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000280CA721108>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 402, in get
verify=self._verify_cert)
File "...\python-env\lib\site-packages\requests\sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "...\python-env\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "...\python-env\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "...\python-env\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='<portal_address>', port=443): Max retries exceeded with url: /portal/sharing/rest/info?f=json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000280CA721108>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "...\python-env\lib\site-packages\arcgis\gis\__init__.py", line 359, in __init__
raise e
File "...\python-env\lib\site-packages\arcgis\gis\__init__.py", line 343, in __init__
trust_env=kwargs.get("trust_env", None))
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 171, in __init__
trust_env=trust_env)
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 199, in __init__
self._product = self._check_product()
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1421, in _check_product
res = self.get(baseurl + '/info', params={'f' : 'json'}, add_token=False)
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 412, in get
"A connection error has occurred: %s" % errCE)
requests.exceptions.ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='<portal_address>', port=443): Max retries exceeded with url: /portal/sharing/rest/info?f=json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000280CA721108>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))


This is the same error if I don't use the proxy keyword at all so I don't think the proxy settings are being taken into account.

I also don't think using the proxy_host and proxy_port options let you send the https traffic to a http proxy so that isn't an option, but when I try to use them I get this error:

>>> GIS('https://e4.cop.ema.np.cp1.homeaffairs.gov.au/portal', username="Leon", password="xxx", verify_cert=False, proxy_host='<ipaddress>', proxy_port=8080)
Traceback (most recent call last):
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 402, in get
verify=self._verify_cert)
File "...\python-env\lib\site-packages\requests\sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "...\python-env\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "...\python-env\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "...\python-env\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "...\python-env\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "...\python-env\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "...\python-env\lib\site-packages\urllib3\connection.py", line 359, in connect
conn = self._connect_tls_proxy(hostname, conn)
File "...\python-env\lib\site-packages\urllib3\connection.py", line 506, in _connect_tls_proxy
ssl_context=ssl_context,
File "...\python-env\lib\site-packages\urllib3\util\ssl_.py", line 432, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
File "...\python-env\lib\site-packages\urllib3\util\ssl_.py", line 474, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock)
File "...\python-env\lib\ssl.py", line 423, in wrap_socket
session=session
File "...\python-env\lib\ssl.py", line 827, in _create
raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "...\python-env\lib\site-packages\arcgis\gis\__init__.py", line 359, in __init__
raise e
File "...\python-env\lib\site-packages\arcgis\gis\__init__.py", line 343, in __init__
trust_env=kwargs.get("trust_env", None))
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 171, in __init__
trust_env=trust_env)
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 199, in __init__
self._product = self._check_product()
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1421, in _check_product
res = self.get(baseurl + '/info', params={'f' : 'json'}, add_token=False)
File "...\python-env\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 426, in get
raise Exception('A general error occurred: %s' % e)
Exception: A general error occurred: check_hostname requires server_hostname

 

 

 

0 Kudos
Alex_at_Exprodat
New Contributor II

I've been having the same trouble. Trying to identify a proxy seems to fail. Here is a snippet of the code from the arcgis module (C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\_impl\_con\_connection.py) using version 1.9.1 where it assigns proxies to the requests session:

def _create_session(self):
    if self._proxy and isinstance(self._proxy, dict):
        proxies = self._proxy
    if self._proxy_port and self._proxy_url:
        url = "%s:%s" % (self._proxy_url, self._proxy_port)
        if self._proxy_password and self._proxy_username:
            proxies = {
                "http": "http://%s:%s@%s"
                % (self._proxy_username, self._proxy_password, url),
                "https": "https://%s:%s@%s"
                % (self._proxy_username, self._proxy_password, url),
            }
        else:
            proxies = {"http": "http://%s" % url, "https": "https://%s" % url}
    else:
        proxies = None

 

In line 3, the proxies variable (which gets passed to the requests session) is populated with the proxy parameter from your GIS('https://<portaladdress>/portal', username="Leon", password="xxx", verify_cert=False, proxy=proxy) call. This is what it should do - your call to requests.get('<portaladdress>/portal/sharing/rest/info?f=json', proxies=proxyproves the connection works using your proxy settings. Note that versions prior to 1.9.1 do not have lines 2-3 above as they do not support the proxy parameter in the GIS() function.

However, line 4 checks for the optional proxy_host and proxy_port parameters in the GIS() call. If they are specified, your proxy parameter gets overwritten. When you use the proxy_host and proxy_port parameters (as others have done prior to 1.9.1), it creates a dictionary and forces http proxy to an http:// address and the https proxy to an https:// address. Some proxy servers are not configured for https, so you end up with proxy errors in the request because of SSL certificate issues, but using verify_cert=False doesn't seem to bypass or ignore those errors.

For example, if you use

gis = GIS('https://<portaladdress>/portal', username='Leon', password='xxx', proxy_host='<ipaddress>', proxy_port=8080)

the code will convert that to this dictionary:

proxies = {
    'http':  'http://<ipaddress>:8080',
    'https': 'https://<ipaddress>:8080'
    }

 

If you do not specify the proxy_host and proxy_port parameters, then line 15-16 will set proxies to None, even if you have specified the proxy parameter. This seems to be a bug in the code because it means the proxy parameter will always be overwritten, making it completely useless.

For testing, I changed line 4 from "if" to "elif" so that it will only run if you have not used the proxy parameter. That seemed to work for me, but I highly recommend cloning your arcgis python environment before making any changes to any modules.

Another option I have seen suggested is to set environment variables HTTP_PROXY and HTTPS_PROXY. With the proxies = None on line 16, I believe the environment variables get ignored by the arcgis module. If those variables are set, I might change line 16 to 

proxies = requests.utils.getproxies()

This should pull proxies that have been set up in the system by default, so even a normal GIS() call will use the system proxy settings without having to manually inject them.

In the end, this seems to work best for me:

def _create_session(self):
    if self._proxy and isinstance(self._proxy, dict):
        proxies = self._proxy
    # if self._proxy_port and self._proxy_url:
    elif self._proxy_port and self._proxy_url:
        url = "%s:%s" % (self._proxy_url, self._proxy_port)
        if self._proxy_password and self._proxy_username:
            proxies = {
                "http": "http://%s:%s@%s"
                % (self._proxy_username, self._proxy_password, url),
                "https": "https://%s:%s@%s"
                % (self._proxy_username, self._proxy_password, url),
            }
        else:
            proxies = {"http": "http://%s" % url, "https": "https://%s" % url}
    else:
        # proxies = None
        proxies = requests.utils.getproxies()

 

LeonAus
New Contributor III

Wow, what an unbelievably comprehensive answer. requests.utils.getproxies() did the trick.

Thanks for the workaround, I'm putting on a hat just so I can take it off...

0 Kudos
LorenzMeyer1
Occasional Contributor

Great answer and explanation. Thanks💪

0 Kudos
EdKnowles
New Contributor III

This may or may not be of interest:

BUG-000140216

When attempting to connect to ArcGIS Online through a forward proxy using ArcGIS API for Python 1.8.4 1.8.5, and 1.9.1, an error message is returned.

0 Kudos
SamDoss
New Contributor

I was having similar issues and I am working with just a http proxy server. Tried your work around Alex_at_Exprodatu, but not could not get it to work. What you described did helped me understand the eccentricities that were going on in in the background.

I stumbled upon something that did work. I used the proxy arg, but declared both http and https variable in the proxy dictionary. I entered the http address into the https dictionary variable. For example

proxy = {'http': 'http://<ipaddress>:8080',
         'https': 'http://<ipaddress>:8080'}

gis = GIS('https://<portaladdress>/portal', username='Leon', password='xxx', proxy=proxy)

 

Alex_at_Exprodat
New Contributor II

A lot will depend on what version of the arcgis module you're working with and your internal proxy configuration. We had to do something similar - adding the http url to the https proxy entry. It sounds like a bad idea, but often the proxy itself doesn't need to be https because it's just passing along messages from the https endpoint. But the Esri code seems to assume that the https entry must be an https proxy url, which is not always the case. It seems better to always declare you own full proxy dictionary with both http and https keys to avoid ambiguity. Thanks for adding that info.

0 Kudos