topic Re: Accessing ArcGIS Online REST Services through a proxy with Python in Python Questions

Accessing ArcGIS Online REST Services through a proxy with Python

TrilliumLevine1 — Thu, 04 Sep 2014 14:14:41 GMT

Hello All,

I'm trying to run a script to access an ArcGIS Online REST web service and am behind an orginizational firewall, so I need to go over an authenticated proxy. My understanding is that urllib2 is supposed to pull the proxy settings from my system automatically so I shouldn't need to add any code dealing with that, but I keep getting a socket timeout error (60010) right after the script attempts to establish a connection -- I think this has to do with the proxy, but maybe I'm wrong. Has anyone else run into this problem and found a sollution? I tried importing the'socket' module and setting the timout value to 'NONE', as well as setting the expiration to a larger value, but the error keeps appearing after 1 minute. I've attached the part of my code dealing with the connection to the service -- does anyone out there have any ideas? Any feedback would be highly appreciated!

import json, urllib, urllib2 import sys, os import arcpy import shutil  class GetToken(object):     # find feature service attachments, send to directory     def urlopen(self, url, data=None):         # open url, send response         referer = "http://www.arcgis.com/arcgis/rest"         req = urllib2.Request(url)         req.add_header('Referer', referer)         if data:             response = urllib2.urlopen(req, data)         else:             response = urllib2.urlopen(req)         return response     def gentoken(self, username, password,         referer = 'www.arcgis.com', expiration=60):         # gets token from referrer         query_dict = {'username': username,             'password': password,             'expiration': str(expiration),             'client': 'referer',             'referer': referer,             'f': 'json'}         query_string = urllib.urlencode(query_dict)         token_url = "https://www.arcgis.com/sharing/rest/generateToken"         token_response = urllib.urlopen(token_url, query_string)         token = json.loads(token_response.read())         if "token" not in token:             print token['messages']             exit()         else:             return token['token']

Re: Accessing ArcGIS Online REST Services through a proxy with Python

JasonScheirer — Thu, 04 Sep 2014 16:39:59 GMT

No, you need to configure your proxies in your code. See this thread on Stack Overflow.

Re: Accessing ArcGIS Online REST Services through a proxy with Python

TrilliumLevine1 — Sun, 12 Dec 2021 00:47:51 GMT

Thanks for your response, Jason. I've tried all the variations mentioned on the link you sent, as well as others. Right now the top of my script looks like below, and I also set my environment variables appropriately for http_proxy and https_proxy. Right now I'm getting an error on the splituser def inside of urllib telling me that a string or buffer is expected.


import sys, os
import arcpy
import shutil
import re


# set proxies
proxy = urllib2.ProxyHandler({
    'http': '192.168.104.103',
    'https': '192.168.104.103'
})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
class GetToken(object):
    # find feature service attachments, send to directory


    def urlopen(self, url, data=None):
        # open url, send response
        referer = "http://www.arcgis.com/arcgis/rest"
        req = urllib2.Request(url)
        req.add_header('Referer', referer)
        if data:
            response = urllib2.urlopen(req, data)
        else:
            response = urllib2.urlopen(req)
        return response


    def gentoken(self, username, password,
        referer = 'www.arcgis.com', expiration=60):
        # gets token from referrer
        query_dict = {'username': username,
            'password': password,
            'expiration': str(expiration),
            'client': 'referer',
            'referer': referer,
            'f': 'json'}
        query_string = urllib.urlencode(query_dict)
        token_url = "https://www.arcgis.com/sharing/rest/generateToken"
        token_response = urllib.urlopen(token_url, query_string)
        token = json.loads(token_response.read())
        if "token" not in token:
            print token['messages']
            exit()
        else:
            return token['token']

Re: Accessing ArcGIS Online REST Services through a proxy with Python

JasonScheirer — Fri, 05 Sep 2014 20:14:25 GMT

Can you post the traceback?

Re: Accessing ArcGIS Online REST Services through a proxy with Python

TrilliumLevine1 — Mon, 08 Sep 2014 08:19:57 GMT

yep, here's the traceback:

Traceback

<module> C:\Users\tle\Desktop\Scripts\dl_extract2.py 161

main C:\Users\tle\Desktop\Scripts\dl_extract2.py 157

__init__ C:\Users\tle\Desktop\Scripts\dl_extract2.py 53

gentoken C:\Users\tle\Desktop\Scripts\dl_extract2.py 40

urlopen C:\Python26\ArcGIS10.0\lib\urllib.py 88

open C:\Python26\ArcGIS10.0\lib\urllib.py 207

open_https C:\Python26\ArcGIS10.0\lib\urllib.py 439

endheaders C:\Python26\ArcGIS10.0\lib\httplib.py 904

_send_output C:\Python26\ArcGIS10.0\lib\httplib.py 776

send C:\Python26\ArcGIS10.0\lib\httplib.py 735

connect C:\Python26\ArcGIS10.0\lib\httplib.py 1112

wrap_socket C:\Python26\ArcGIS10.0\lib\ssl.py 350

__init__ C:\Python26\ArcGIS10.0\lib\ssl.py 118

do_handshake C:\Python26\ArcGIS10.0\lib\ssl.py 293

IOError: [Errno socket error] [Errno 1] _ssl.c:480: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Re: Accessing ArcGIS Online REST Services through a proxy with Python

TrilliumLevine1 — Wed, 10 Sep 2014 10:03:54 GMT

This actually turned out to be a pretty easy fix. All that were needed in my case were the system variables:

http_proxy: http//:user:password@server:port

https_proxy: http//:user:password@server:port

This link explains why the http protocol is used for both.

The following code was not needed -- in fact, the script threw a 404 error when it was included:

proxy = urllib2.ProxyHandler({

'http': '192.168.104.103',

'https': '192.168.104.103'

})

opener = urllib2.build_opener(proxy)

urllib2.install_opener(opener)

Re: Accessing ArcGIS Online REST Services through a proxy with Python

Gisbert61 — Tue, 24 May 2016 06:16:04 GMT

One more discovery:

Turns out that (in my situation, ymmv) the authentication is key. So setting the username in the proxy"definition" is mandatory. Either in the environment setting or in proxies = urllib2.ProxyHandler I needed to add http://username:password@abc.com:1234

hth, Bert