I have access to a ArcGIS REST API service running on a standalone, unfederated Arcgis server. (Not AGOL or ArcGIS Enteprise) ie (https://xxxx.xxxxx.com/arcgis/rest/services/xxxxxx/xxxxMap/MapServer)
Is there any way to consume the REST API services like MapServers, ImageServers using ArcGIS's Python API?
Also, I am working in a secured environment with no internet access. Using MapImageLayer from ArcGIS's Python API forces a GIS object as a parameter by default. And that GIS object calls out to www.arcgis.com for anonymous authentication.
Solved! Go to Solution.
Use a ServicesDirectory object as your GIS:
>>> import re
>>> import arcgis
>>> arcgis.__version__
'1.8.5'
>>>
>>> from arcgis.features import FeatureLayer
>>> from arcgis.gis.server import ServicesDirectory
>>>
>>> svc_url = r"https://localhost.localdomain/arcgis/rest/services/someMapService/MapServer"
>>> sd = ServicesDirectory(re.search(r"(http.*/rest/services)", svc_url).group(1))
>>> fl = FeatureLayer(svc_url, gis=sd)
>>>
You say the environment is secured, but is the service secured as well? That is, do you have to authenticate with GIS Server to access the service or can you access it anonymously when inside the environment?
Hi Joshua,
No. The service is not secured, i do not need to authenticate with a GIS Server to access the REST API. Yes I can access the REST API anonymously within the environment.
I was meaning to say is that the REST Service is hosted internally within my organization's intranet. And i do not have access to the internet from that environment.
If it helps, I am able to use the community made esri.leaflet package for R to display the mapserver on Rstudio. https://github.com/bhaskarvk/leaflet.esri
If the service is unsecured, i.e., no authentication necessary, the following should work:
>>> import arcgis
>>> arcgis.__version__
'1.8.5'
>>>
>>> from arcgis.features import FeatureLayer
>>>
>>> svc_url = r"https://localhost.localdomain/arcgis/rest/services/SomeMapService/MapServer"
>>> fl = FeatureLayer(svc_url)
>>> fl.properties
Hi Joshua,
Thanks for the prompt reply. I really appreciate it. I have been knocking my head over this issue for weeks.
I tried your code snippet and encountered the same error as when i was trying to use the MapImageLayer from arcgis.mapping. It seems like the code is trying to anonymously authenticate to ArcGIS online (www.arcgis.com) even when it doesn't need to. It is unable to resolve the hostname as we have a internal DNS that we use for our intranet.
Did your device have access to the internet when trying out this code snippet?
Here is the error:
---------------------------------------------------------------------------
gaierror Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
156 conn = connection.create_connection(
--> 157 (self._dns_host, self.port), self.timeout, **extra_kw
158 )
/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
60
---> 61 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
62 af, socktype, proto, canonname, sa = res
/opt/conda/lib/python3.7/socket.py in getaddrinfo(host, port, family, type, proto, flags)
747 addrlist = []
--> 748 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
749 af, socktype, proto, canonname, sa = res
gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
NewConnectionError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
671 headers=headers,
--> 672 chunked=chunked,
673 )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
375 try:
--> 376 self._validate_conn(conn)
377 except (SocketTimeout, BaseSSLError) as e:
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
993 if not getattr(conn, "sock", None): # AppEngine might not have `.sock`
--> 994 conn.connect()
995
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in connect(self)
333 # Add certificate verification
--> 334 conn = self._new_conn()
335 hostname = self.host
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
168 raise NewConnectionError(
--> 169 self, "Failed to establish a new connection: %s" % e
170 )
NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
719 retries = retries.increment(
--> 720 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
721 )
/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
435 if new_retry.is_exhausted():
--> 436 raise MaxRetryError(_pool, url, error or ResponseError(cause))
437
MaxRetryError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
686 cert=cert,
--> 687 files=files)
688 except requests.exceptions.SSLError as err:
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs)
580
--> 581 return self.request('POST', url, data=data, json=json, **kwargs)
582
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs)
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
515
--> 516 raise ConnectionError(e, request=request)
517
ConnectionError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
<ipython-input-142-edcb80967e08> in <module>
3 svc_url = r"https://xxxxxxx/arcgis/rest/services/MapServices/xxxxx/MapServer"
4
----> 5 fl = FeatureLayer(svc_url)
6 fl.properties
~/.local/lib/python3.7/site-packages/arcgis/features/layer.py in __init__(self, url, gis, container, dynamic_layer)
50 if str(url).lower().endswith("/"):
51 url = url[:-1]
---> 52 super(FeatureLayer, self).__init__(url, gis)
53 self._storage = container
54 self._dynamic_layer = dynamic_layer
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, gis)
11561
11562 def __init__(self, url, gis=None):
> 11563 super(Layer, self).__init__(url, gis)
11564 self.filter = None
11565 self._time_filter = None
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, gis)
11412
11413 if gis is None:
> 11414 gis = GIS(set_active=False)
11415 self._gis = gis
11416 self._con = gis._con
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
357 "argument when connecting to the GIS.")
358 else:
--> 359 raise e
360 try:
361 if url.lower().find("arcgis.com") > -1 and \
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
341 custom_auth=custom_auth, #token=self._utoken,
342 client_secret=client_secret,
--> 343 trust_env=kwargs.get("trust_env", None))
344 if self._is_hosted_nb_home:
345 # For GIS("home") objects, force no referer passed in
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in __init__(self, url, username, password, key_file, cert_file, expiration, referer, proxy_host, proxy_port, connection, workdir, tokenurl, verify_cert, client_id, custom_auth, token, **kwargs)
171 trust_env=trust_env)
172 #self.get_version(True)
--> 173 self.get_properties(True)
174
175
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
1136 resp = self.con.get(path, ssl=True) # issue seen with key, cert auth
1137 if not resp:
-> 1138 raise e
1139
1140 if resp:
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
1124 resp = None
1125 try:
-> 1126 resp = self.con.post(path, self._postdata(), ssl=True)
1127 except Exception as e:
1128 if not self.con._verify_cert and \
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
694 except requests.exceptions.ConnectionError as errCE:
695 raise requests.exceptions.ConnectionError(
--> 696 "A connection error has occurred: %s" % errCE)
697 except requests.exceptions.InvalidHeader as errIH:
698 raise requests.exceptions.InvalidHeader(
ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Use a ServicesDirectory object as your GIS:
>>> import re
>>> import arcgis
>>> arcgis.__version__
'1.8.5'
>>>
>>> from arcgis.features import FeatureLayer
>>> from arcgis.gis.server import ServicesDirectory
>>>
>>> svc_url = r"https://localhost.localdomain/arcgis/rest/services/someMapService/MapServer"
>>> sd = ServicesDirectory(re.search(r"(http.*/rest/services)", svc_url).group(1))
>>> fl = FeatureLayer(svc_url, gis=sd)
>>>
You are a life saver. Thanks!
Hi Joshua,
Thanks for all the help thus far.
I successfully created a MapImageLayer object without any error. My intent is to display this image as a map or widget.
However, I realized that I must create a WebMap object first, before adding this layer. Creating a default WebMap() object also triggers a call to arcgis online, which results in an error.
Do you know if there is any way to use this MapImageLayer to create a WebMap() object, or to create a WebMap() object from a MapServer REST URL.
Again, thanks for all your help
Here is my code:
from arcgis.mapping import MapImageLayer
import re
from arcgis.gis.server import ServicesDirectory
from arcgis.mapping import WebMap
svc_url = r"https://xxx.xxx.xxx/arcgis/rest/services/MapServices/xxxxxx/MapServer"
sd = ServicesDirectory(re.search(r"(http.*/rest/services)",svc_url).group(1))
mil = MapImageLayer(svc_url,gis=sd)
wm = WebMap()
wm.add_layer(mil)
Here is the same error code:
---------------------------------------------------------------------------
gaierror Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
156 conn = connection.create_connection(
--> 157 (self._dns_host, self.port), self.timeout, **extra_kw
158 )
/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
60
---> 61 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
62 af, socktype, proto, canonname, sa = res
/opt/conda/lib/python3.7/socket.py in getaddrinfo(host, port, family, type, proto, flags)
747 addrlist = []
--> 748 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
749 af, socktype, proto, canonname, sa = res
gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
NewConnectionError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
671 headers=headers,
--> 672 chunked=chunked,
673 )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
375 try:
--> 376 self._validate_conn(conn)
377 except (SocketTimeout, BaseSSLError) as e:
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
993 if not getattr(conn, "sock", None): # AppEngine might not have `.sock`
--> 994 conn.connect()
995
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in connect(self)
333 # Add certificate verification
--> 334 conn = self._new_conn()
335 hostname = self.host
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
168 raise NewConnectionError(
--> 169 self, "Failed to establish a new connection: %s" % e
170 )
NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )
/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
719 retries = retries.increment(
--> 720 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
721 )
/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
435 if new_retry.is_exhausted():
--> 436 raise MaxRetryError(_pool, url, error or ResponseError(cause))
437
MaxRetryError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
686 cert=cert,
--> 687 files=files)
688 except requests.exceptions.SSLError as err:
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs)
580
--> 581 return self.request('POST', url, data=data, json=json, **kwargs)
582
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534
/opt/conda/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs)
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647
/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
515
--> 516 raise ConnectionError(e, request=request)
517
ConnectionError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
<ipython-input-16-6bfc5e47757d> in <module>
9 mil = MapImageLayer(svc_url,gis=sd)
10
---> 11 wm = WebMap()
12
13 wm.add_layer(mil)
~/.local/lib/python3.7/site-packages/arcgis/mapping/_types.py in __init__(self, webmapitem)
233 # Set up map widget to use in jupyter env: have changes made to
234 # self._webmapdict get passed to the widge to render
--> 235 self._mapview = MapView(gis=self._gis, item=self)
236 self._mapview.hide_mode_switch = True
237 self._mapview._webmap = {}
~/.local/lib/python3.7/site-packages/arcgis/widgets/_mapview/_mapview.py in __init__(self, gis, item, mode, **kwargs)
692
693 # Set up gis object
--> 694 self._setup_gis_properties(gis)
695
696 # Set up miscellanous properties needed on startup
~/.local/lib/python3.7/site-packages/arcgis/widgets/_mapview/_mapview.py in _setup_gis_properties(self, gis)
958 if gis is None:
959 from arcgis.gis import GIS
--> 960 gis = GIS(set_active=False)
961 self.gis = gis
962
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
357 "argument when connecting to the GIS.")
358 else:
--> 359 raise e
360 try:
361 if url.lower().find("arcgis.com") > -1 and \
~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs)
341 custom_auth=custom_auth, #token=self._utoken,
342 client_secret=client_secret,
--> 343 trust_env=kwargs.get("trust_env", None))
344 if self._is_hosted_nb_home:
345 # For GIS("home") objects, force no referer passed in
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in __init__(self, url, username, password, key_file, cert_file, expiration, referer, proxy_host, proxy_port, connection, workdir, tokenurl, verify_cert, client_id, custom_auth, token, **kwargs)
171 trust_env=trust_env)
172 #self.get_version(True)
--> 173 self.get_properties(True)
174
175
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
1136 resp = self.con.get(path, ssl=True) # issue seen with key, cert auth
1137 if not resp:
-> 1138 raise e
1139
1140 if resp:
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force)
1124 resp = None
1125 try:
-> 1126 resp = self.con.post(path, self._postdata(), ssl=True)
1127 except Exception as e:
1128 if not self.con._verify_cert and \
~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs)
694 except requests.exceptions.ConnectionError as errCE:
695 raise requests.exceptions.ConnectionError(
--> 696 "A connection error has occurred: %s" % errCE)
697 except requests.exceptions.InvalidHeader as errIH:
698 raise requests.exceptions.InvalidHeader(
ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
This is how I connect to my unfederated Staging environment for testing. Granted this is an open endpoint that doesn't have any secured services on it.
For reference, gisA is AGOL, gisE is Enterprise, and gisT is the staging server.
from arcgis.gis import GIS
from arcgis.gis.server import Server
from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())
pURL = os.getenv('PORTAL_SITE')
aURL = os.getenv('AGOL_SITE')
sURL = os.getenv('STAGE_SITE')
sUser = os.getenv('STAGE_USERNAME')
sPass = os.getenv('STAGE_PASSWORD')
aUser = os.getenv('AGOL_USERNAME')
aPass = os.getenv('AGOL_PASSWORD')
pUser = os.getenv('ESRI_USERNAME')
pPass = os.getenv('ESRI_PASSWORD')
#Connect to Enterprise GIS
gisE = GIS(url=pURL, username=pUser, password=pPass)
#Connect to AGOL
gisA = GIS(url=aURL, username=aUser, password=aPass, set_active=False)
#Connect to Staging. Does not work for right now. Found a solution: https://community.esri.com/t5/developers-questions/error-connecting-to-arcgis-server-with-arcgis-python-api/m-p/869042?commentID=896496#comment-896496
# gisT = GIS(url=sURL, username=sUser, password=sPass, set_active=False, verify_cert=False)
# from arcgis.gis.server import Server
# sURL = 'https://ESRISTAGE:6443'
# gisT = Server(url=f'{sURL}/ed8fb7ce-e99d-4eb2-af78-b7ef11e3c2c4/admin',token_url=f'{sURL}/ed8fb7ce-e99d-4eb2-af78-b7ef11e3c2c4/tokens/generateToken', username=sUser, password=sPass, verify_cert=False, set_active=False)
#This method for connecting to staging server does work. Must use Server Module for now
gisT = Server(url=f'{sURL}/arcgis/admin', username=sUser, password=sPass, verify_cert=False)
# gisT = GIS(url=f'{sURL}/arcgis/admin', username=sUser, password=sPass, verify_cert=False)
print(gisA)
print(gisE)
print(gisT)
Hi Mark,
Thanks for the info.
Unfortunately, the server is not owned by me. And hence, it is unlikely that i would be able to get an admin account to access the Server.
I only have access to the rest services that it exposes, ie. MapServer, ImageServer etc.
Using the Server object in your example, are you able to use arcgis.mapping.MapImageLayer by putting it as a parameter instead of the GIS object? Or is the Server object just used for administrative tasks on the ArcGIS Server
Is there any way to consume the REST Services without first authenticating as an admin on the Server?