Using ArcGIS Python API to consume standalone ArcGIS REST API

[darrenng123123]

I have access to a ArcGIS REST API service running on a standalone, unfederated Arcgis server. (Not AGOL or ArcGIS Enteprise) ie (https://xxxx.xxxxx.com/arcgis/rest/services/xxxxxx/xxxxMap/MapServer)

Is there any way to consume the REST API services like MapServers, ImageServers using ArcGIS's Python API?
Also, I am working in a secured environment with no internet access. Using MapImageLayer from ArcGIS's Python API forces a GIS object as a parameter by default. And that GIS object calls out to www.arcgis.com for anonymous authentication.

[JoshuaBixby]

You say the environment is secured, but is the service secured as well?  That is, do you have to authenticate with GIS Server to access the service or can you access it anonymously when inside the environment?

[darrenng123123]

Hi Joshua,

No. The service is not secured, i do not need to authenticate with a GIS Server to access the REST API. Yes I can access the REST API anonymously within the environment.

I was meaning to say is that the REST Service is hosted internally within my organization's intranet. And i do not have access to the internet from that environment.

If it helps, I am able to use the community made esri.leaflet package for R to display the mapserver on Rstudio. https://github.com/bhaskarvk/leaflet.esri

[JoshuaBixby]

If the service is unsecured, i.e., no authentication necessary, the following should work:

>>> import arcgis
>>> arcgis.__version__
'1.8.5'
>>> 
>>> from arcgis.features import FeatureLayer
>>> 
>>> svc_url = r"https://localhost.localdomain/arcgis/rest/services/SomeMapService/MapServer"
>>> fl = FeatureLayer(svc_url)
>>> fl.properties

[darrenng123123]

Hi Joshua,

Thanks for the prompt reply. I really appreciate it. I have been knocking my head over this issue for weeks.

I tried your code snippet and encountered the same error as when i was trying to use the MapImageLayer from arcgis.mapping. It seems like the code is trying to anonymously authenticate to ArcGIS online (www.arcgis.com) even when it doesn't need to. It is unable to resolve the hostname as we have a internal DNS that we use for our intranet.

Did your device have access to the internet when trying out this code snippet?

Here is the error: 

---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
    156             conn = connection.create_connection(
--> 157                 (self._dns_host, self.port), self.timeout, **extra_kw
    158             )
/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     60 
---> 61     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
     62         af, socktype, proto, canonname, sa = res
/opt/conda/lib/python3.7/socket.py in getaddrinfo(host, port, family, type, proto, flags)
    747     addrlist = []
--> 748     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    749         af, socktype, proto, canonname, sa = res
gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
NewConnectionError                        Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    671                 headers=headers,
--> 672                 chunked=chunked,
    673             )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    375         try:
--> 376             self._validate_conn(conn)
    377         except (SocketTimeout, BaseSSLError) as e:
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    993         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 994             conn.connect()
    995 
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in connect(self)
    333         # Add certificate verification
--> 334         conn = self._new_conn()
    335         hostname = self.host
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
    168             raise NewConnectionError(
--> 169                 self, "Failed to establish a new connection: %s" % e
    170             )
NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
MaxRetryError                             Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    448                     retries=self.max_retries,
--> 449                     timeout=timeout
    450                 )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    719             retries = retries.increment(
--> 720                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    721             )
/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    435         if new_retry.is_exhausted():
--> 436             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    437 
MaxRetryError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError                           Traceback (most recent call last)
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
    686                                           cert=cert,
--> 687                                           files=files)
    688         except requests.exceptions.SSLError as err:
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs)
    580 
--> 581         return self.request('POST', url, data=data, json=json, **kwargs)
    582 
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    532         send_kwargs.update(settings)
--> 533         resp = self.send(prep, **send_kwargs)
    534 
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs)
    645         # Send the request
--> 646         r = adapter.send(request, **kwargs)
    647 
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    515 
--> 516             raise ConnectionError(e, request=request)
    517 
ConnectionError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError                           Traceback (most recent call last)
<ipython-input-142-edcb80967e08> in <module>
      3 svc_url = r"https://xxxxxxx/arcgis/rest/services/MapServices/xxxxx/MapServer"
      4 
----> 5 fl = FeatureLayer(svc_url)
      6 fl.properties
~/.local/lib/python3.7/site-packages/arcgis/features/layer.py in __init__(self, url, gis, container, dynamic_layer)
     50         if str(url).lower().endswith("/"):
     51             url = url[:-1]
---> 52         super(FeatureLayer, self).__init__(url, gis)
     53         self._storage = container
     54         self._dynamic_layer = dynamic_layer
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, gis)
  11561 
  11562     def __init__(self, url, gis=None):
> 11563         super(Layer, self).__init__(url, gis)
  11564         self.filter = None
  11565         self._time_filter = None
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, gis)
  11412 
  11413         if gis is None:
> 11414             gis = GIS(set_active=False)
  11415             self._gis = gis
  11416             self._con = gis._con
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
    357                                    "argument when connecting to the GIS.")
    358             else:
--> 359                 raise e
    360         try:
    361             if url.lower().find("arcgis.com") > -1 and \
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
    341                                            custom_auth=custom_auth, #token=self._utoken,
    342                                            client_secret=client_secret,
--> 343                                            trust_env=kwargs.get("trust_env", None))
    344             if self._is_hosted_nb_home:
    345                 # For GIS("home") objects, force no referer passed in
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in __init__(self, url, username, password, key_file, cert_file, expiration, referer, proxy_host, proxy_port, connection, workdir, tokenurl, verify_cert, client_id, custom_auth, token, **kwargs)
    171                                       trust_env=trust_env)
    172         #self.get_version(True)
--> 173         self.get_properties(True)
    174 
    175 
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
   1136                     resp = self.con.get(path, ssl=True) # issue seen with key, cert auth
   1137                 if not resp:
-> 1138                     raise e
   1139 
   1140             if resp:
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
   1124             resp = None
   1125             try:
-> 1126                 resp = self.con.post(path, self._postdata(), ssl=True)
   1127             except Exception as e:
   1128                 if not self.con._verify_cert and \
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
    694         except requests.exceptions.ConnectionError as errCE:
    695             raise requests.exceptions.ConnectionError(
--> 696                 "A connection error has occurred: %s" % errCE)
    697         except requests.exceptions.InvalidHeader as errIH:
    698             raise requests.exceptions.InvalidHeader(
ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known'))

[JoshuaBixby · Accepted solution]

Use a ServicesDirectory object as your GIS:

>>> import re
>>> import arcgis
>>> arcgis.__version__
'1.8.5'
>>>
>>> from arcgis.features import FeatureLayer
>>> from arcgis.gis.server import ServicesDirectory
>>>
>>> svc_url = r"https://localhost.localdomain/arcgis/rest/services/someMapService/MapServer"
>>> sd = ServicesDirectory(re.search(r"(http.*/rest/services)", svc_url).group(1))
>>> fl = FeatureLayer(svc_url, gis=sd)
>>> 

[darrenng123123]

You are a life saver. Thanks!

[darrenng123123]

Hi Joshua,

Thanks for all the help thus far.

I successfully created a MapImageLayer object without any error. My intent is to display this image as a map or widget.

However, I realized that  I must create a WebMap object first, before adding this layer. Creating a default WebMap() object also triggers a call to arcgis online, which results in an error. 

Do you know if there is any way to use this MapImageLayer to create a WebMap() object, or to create a WebMap() object from a MapServer REST URL.

Again, thanks for all your help

Here is my code: 

from arcgis.mapping import MapImageLayer
import re
from arcgis.gis.server import ServicesDirectory
from arcgis.mapping import WebMap
svc_url = r"https://xxx.xxx.xxx/arcgis/rest/services/MapServices/xxxxxx/MapServer"
sd = ServicesDirectory(re.search(r"(http.*/rest/services)",svc_url).group(1))
mil = MapImageLayer(svc_url,gis=sd)
wm = WebMap()
wm.add_layer(mil)

Here is the same error code:

---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
    156             conn = connection.create_connection(
--> 157                 (self._dns_host, self.port), self.timeout, **extra_kw
    158             )
/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     60 
---> 61     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
     62         af, socktype, proto, canonname, sa = res
/opt/conda/lib/python3.7/socket.py in getaddrinfo(host, port, family, type, proto, flags)
    747     addrlist = []
--> 748     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    749         af, socktype, proto, canonname, sa = res
gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
NewConnectionError                        Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    671                 headers=headers,
--> 672                 chunked=chunked,
    673             )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    375         try:
--> 376             self._validate_conn(conn)
    377         except (SocketTimeout, BaseSSLError) as e:
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    993         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 994             conn.connect()
    995 
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in connect(self)
    333         # Add certificate verification
--> 334         conn = self._new_conn()
    335         hostname = self.host
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
    168             raise NewConnectionError(
--> 169                 self, "Failed to establish a new connection: %s" % e
    170             )
NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
MaxRetryError                             Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    448                     retries=self.max_retries,
--> 449                     timeout=timeout
    450                 )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    719             retries = retries.increment(
--> 720                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    721             )
/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    435         if new_retry.is_exhausted():
--> 436             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    437 
MaxRetryError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError                           Traceback (most recent call last)
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
    686                                           cert=cert,
--> 687                                           files=files)
    688         except requests.exceptions.SSLError as err:
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs)
    580 
--> 581         return self.request('POST', url, data=data, json=json, **kwargs)
    582 
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    532         send_kwargs.update(settings)
--> 533         resp = self.send(prep, **send_kwargs)
    534 
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs)
    645         # Send the request
--> 646         r = adapter.send(request, **kwargs)
    647 
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    515 
--> 516             raise ConnectionError(e, request=request)
    517 
ConnectionError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError                           Traceback (most recent call last)
<ipython-input-16-6bfc5e47757d> in <module>
      9 mil = MapImageLayer(svc_url,gis=sd)
     10 
---> 11 wm = WebMap()
     12 
     13 wm.add_layer(mil)
~/.local/lib/python3.7/site-packages/arcgis/mapping/_types.py in __init__(self, webmapitem)
    233         # Set up map widget to use in jupyter env: have changes made to
    234         # self._webmapdict get passed to the widge to render
--> 235         self._mapview = MapView(gis=self._gis, item=self)
    236         self._mapview.hide_mode_switch = True
    237         self._mapview._webmap = {}
~/.local/lib/python3.7/site-packages/arcgis/widgets/_mapview/_mapview.py in __init__(self, gis, item, mode, **kwargs)
    692 
    693         # Set up gis object
--> 694         self._setup_gis_properties(gis)
    695 
    696         # Set up miscellanous properties needed on startup
~/.local/lib/python3.7/site-packages/arcgis/widgets/_mapview/_mapview.py in _setup_gis_properties(self, gis)
    958         if gis is None:
    959             from arcgis.gis import GIS
--> 960             gis = GIS(set_active=False)
    961         self.gis = gis
    962 
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
    357                                    "argument when connecting to the GIS.")
    358             else:
--> 359                 raise e
    360         try:
    361             if url.lower().find("arcgis.com") > -1 and \
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
    341                                            custom_auth=custom_auth, #token=self._utoken,
    342                                            client_secret=client_secret,
--> 343                                            trust_env=kwargs.get("trust_env", None))
    344             if self._is_hosted_nb_home:
    345                 # For GIS("home") objects, force no referer passed in
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in __init__(self, url, username, password, key_file, cert_file, expiration, referer, proxy_host, proxy_port, connection, workdir, tokenurl, verify_cert, client_id, custom_auth, token, **kwargs)
    171                                       trust_env=trust_env)
    172         #self.get_version(True)
--> 173         self.get_properties(True)
    174 
    175 
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
   1136                     resp = self.con.get(path, ssl=True) # issue seen with key, cert auth
   1137                 if not resp:
-> 1138                     raise e
   1139 
   1140             if resp:
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
   1124             resp = None
   1125             try:
-> 1126                 resp = self.con.post(path, self._postdata(), ssl=True)
   1127             except Exception as e:
   1128                 if not self.con._verify_cert and \
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
    694         except requests.exceptions.ConnectionError as errCE:
    695             raise requests.exceptions.ConnectionError(
--> 696                 "A connection error has occurred: %s" % errCE)
    697         except requests.exceptions.InvalidHeader as errIH:
    698             raise requests.exceptions.InvalidHeader(
ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

[MarkKinnaman]

This is how I connect to my unfederated Staging environment for testing. Granted this is an open endpoint that doesn't have any secured services on it.

For reference, gisA is AGOL, gisE is Enterprise, and gisT is the staging server.

from arcgis.gis import GIS
from arcgis.gis.server import Server
from dotenv import find_dotenv, load_dotenv

load_dotenv(find_dotenv())
pURL = os.getenv('PORTAL_SITE')
aURL = os.getenv('AGOL_SITE')
sURL = os.getenv('STAGE_SITE')
sUser = os.getenv('STAGE_USERNAME')
sPass = os.getenv('STAGE_PASSWORD')
aUser = os.getenv('AGOL_USERNAME')
aPass = os.getenv('AGOL_PASSWORD')
pUser = os.getenv('ESRI_USERNAME')
pPass = os.getenv('ESRI_PASSWORD')

#Connect to Enterprise GIS
gisE = GIS(url=pURL, username=pUser, password=pPass)

#Connect to AGOL
gisA = GIS(url=aURL, username=aUser, password=aPass, set_active=False)

#Connect to Staging. Does not work for right now. Found a solution: https://community.esri.com/t5/developers-questions/error-connecting-to-arcgis-server-with-arcgis-python-api/m-p/869042?commentID=896496#comment-896496

# gisT = GIS(url=sURL, username=sUser, password=sPass, set_active=False, verify_cert=False)
# from arcgis.gis.server import Server
# sURL = 'https://ESRISTAGE:6443'
# gisT = Server(url=f'{sURL}/ed8fb7ce-e99d-4eb2-af78-b7ef11e3c2c4/admin',token_url=f'{sURL}/ed8fb7ce-e99d-4eb2-af78-b7ef11e3c2c4/tokens/generateToken', username=sUser, password=sPass, verify_cert=False, set_active=False)

#This method for connecting to staging server does work. Must use Server Module for now
gisT = Server(url=f'{sURL}/arcgis/admin', username=sUser, password=sPass, verify_cert=False)

# gisT = GIS(url=f'{sURL}/arcgis/admin', username=sUser, password=sPass, verify_cert=False)


print(gisA)
print(gisE)
print(gisT)

[darrenng123123]

Hi Mark,

Thanks for the info.

Unfortunately, the server is not owned by me. And hence, it is unlikely that i would be able to get an admin account to access the Server. 

I only have access to the rest services that it exposes, ie. MapServer, ImageServer etc.

Using the Server object in your example, are you able to use arcgis.mapping.MapImageLayer by putting it as a parameter instead of the GIS object? Or is the Server object just used for administrative tasks on the ArcGIS Server

Is there any way to consume the REST Services without first authenticating as an admin on the Server?