topic Re: Using ArcGIS Python API to consume standalone ArcGIS REST API in ArcGIS API for Python Questions

Using ArcGIS Python API to consume standalone ArcGIS REST API

darrenng123123 — Tue, 18 May 2021 15:19:48 GMT

I have access to a ArcGIS REST API service running on a standalone, unfederated Arcgis server. (Not AGOL or ArcGIS Enteprise) ie (https://xxxx.xxxxx.com/arcgis/rest/services/xxxxxx/xxxxMap/MapServer)

Is there any way to consume the REST API services like MapServers, ImageServers using ArcGIS's Python API?
Also, I am working in a secured environment with no internet access. Using MapImageLayer from ArcGIS's Python API forces a GIS object as a parameter by default. And that GIS object calls out to www.arcgis.com for anonymous authentication.

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

JoshuaBixby — Tue, 18 May 2021 15:23:23 GMT

You say the environment is secured, but is the service secured as well? That is, do you have to authenticate with GIS Server to access the service or can you access it anonymously when inside the environment?

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

MarkKinnaman — Tue, 18 May 2021 16:03:29 GMT

This is how I connect to my unfederated Staging environment for testing. Granted this is an open endpoint that doesn't have any secured services on it.

For reference, gisA is AGOL, gisE is Enterprise, and gisT is the staging server.

from arcgis.gis import GIS from arcgis.gis.server import Server from dotenv import find_dotenv, load_dotenv load_dotenv(find_dotenv()) pURL = os.getenv('PORTAL_SITE') aURL = os.getenv('AGOL_SITE') sURL = os.getenv('STAGE_SITE') sUser = os.getenv('STAGE_USERNAME') sPass = os.getenv('STAGE_PASSWORD') aUser = os.getenv('AGOL_USERNAME') aPass = os.getenv('AGOL_PASSWORD') pUser = os.getenv('ESRI_USERNAME') pPass = os.getenv('ESRI_PASSWORD') #Connect to Enterprise GIS gisE = GIS(url=pURL, username=pUser, password=pPass) #Connect to AGOL gisA = GIS(url=aURL, username=aUser, password=aPass, set_active=False) #Connect to Staging. Does not work for right now. Found a solution: https://community.esri.com/t5/developers-questions/error-connecting-to-arcgis-server-with-arcgis-python-api/m-p/869042?commentID=896496#comment-896496 # gisT = GIS(url=sURL, username=sUser, password=sPass, set_active=False, verify_cert=False) # from arcgis.gis.server import Server # sURL = 'https://ESRISTAGE:6443' # gisT = Server(url=f'{sURL}/ed8fb7ce-e99d-4eb2-af78-b7ef11e3c2c4/admin',token_url=f'{sURL}/ed8fb7ce-e99d-4eb2-af78-b7ef11e3c2c4/tokens/generateToken', username=sUser, password=sPass, verify_cert=False, set_active=False) #This method for connecting to staging server does work. Must use Server Module for now gisT = Server(url=f'{sURL}/arcgis/admin', username=sUser, password=sPass, verify_cert=False) # gisT = GIS(url=f'{sURL}/arcgis/admin', username=sUser, password=sPass, verify_cert=False) print(gisA) print(gisE) print(gisT)

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

darrenng123123 — Tue, 18 May 2021 16:23:19 GMT

Hi Joshua,

No. The service is not secured, i do not need to authenticate with a GIS Server to access the REST API. Yes I can access the REST API anonymously within the environment.

I was meaning to say is that the REST Service is hosted internally within my organization's intranet. And i do not have access to the internet from that environment.

If it helps, I am able to use the community made esri.leaflet package for R to display the mapserver on Rstudio. https://github.com/bhaskarvk/leaflet.esri

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

darrenng123123 — Tue, 18 May 2021 16:19:05 GMT

Hi Mark,

Thanks for the info.

Unfortunately, the server is not owned by me. And hence, it is unlikely that i would be able to get an admin account to access the Server.

I only have access to the rest services that it exposes, ie. MapServer, ImageServer etc.

Using the Server object in your example, are you able to use arcgis.mapping.MapImageLayer by putting it as a parameter instead of the GIS object? Or is the Server object just used for administrative tasks on the ArcGIS Server

Is there any way to consume the REST Services without first authenticating as an admin on the Server?

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

JoshuaBixby — Tue, 18 May 2021 16:25:16 GMT

If the service is unsecured, i.e., no authentication necessary, the following should work:

>>> import arcgis >>> arcgis.__version__ '1.8.5' >>> >>> from arcgis.features import FeatureLayer >>> >>> svc_url = r"https://localhost.localdomain/arcgis/rest/services/SomeMapService/MapServer" >>> fl = FeatureLayer(svc_url) >>> fl.properties

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

darrenng123123 — Tue, 18 May 2021 16:37:52 GMT

Hi Joshua,

Thanks for the prompt reply. I really appreciate it. I have been knocking my head over this issue for weeks.

I tried your code snippet and encountered the same error as when i was trying to use the MapImageLayer from arcgis.mapping. It seems like the code is trying to anonymously authenticate to ArcGIS online (www.arcgis.com) even when it doesn't need to. It is unable to resolve the hostname as we have a internal DNS that we use for our intranet.

Did your device have access to the internet when trying out this code snippet?

Here is the error:

--------------------------------------------------------------------------- gaierror Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self) 156 conn = connection.create_connection( --> 157 (self._dns_host, self.port), self.timeout, **extra_kw 158 ) /opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options) 60 ---> 61 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): 62 af, socktype, proto, canonname, sa = res /opt/conda/lib/python3.7/socket.py in getaddrinfo(host, port, family, type, proto, flags) 747 addrlist = [] --> 748 for res in _socket.getaddrinfo(host, port, family, type, proto, flags): 749 af, socktype, proto, canonname, sa = res gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: NewConnectionError Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 671 headers=headers, --> 672 chunked=chunked, 673 ) /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw) 375 try: --> 376 self._validate_conn(conn) 377 except (SocketTimeout, BaseSSLError) as e: /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn) 993 if not getattr(conn, "sock", None): # AppEngine might not have `.sock` --> 994 conn.connect() 995 /opt/conda/lib/python3.7/site-packages/urllib3/connection.py in connect(self) 333 # Add certificate verification --> 334 conn = self._new_conn() 335 hostname = self.host /opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self) 168 raise NewConnectionError( --> 169 self, "Failed to establish a new connection: %s" % e 170 ) NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: MaxRetryError Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 448 retries=self.max_retries, --> 449 timeout=timeout 450 ) /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 719 retries = retries.increment( --> 720 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 721 ) /opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace) 435 if new_retry.is_exhausted(): --> 436 raise MaxRetryError(_pool, url, error or ResponseError(cause)) 437 MaxRetryError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: ConnectionError Traceback (most recent call last) ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs) 686 cert=cert, --> 687 files=files) 688 except requests.exceptions.SSLError as err: /opt/conda/lib/python3.7/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs) 580 --> 581 return self.request('POST', url, data=data, json=json, **kwargs) 582 /opt/conda/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 532 send_kwargs.update(settings) --> 533 resp = self.send(prep, **send_kwargs) 534 /opt/conda/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs) 645 # Send the request --> 646 r = adapter.send(request, **kwargs) 647 /opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 515 --> 516 raise ConnectionError(e, request=request) 517 ConnectionError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: ConnectionError Traceback (most recent call last) <ipython-input-142-edcb80967e08> in <module> 3 svc_url = r"https://xxxxxxx/arcgis/rest/services/MapServices/xxxxx/MapServer" 4 ----> 5 fl = FeatureLayer(svc_url) 6 fl.properties ~/.local/lib/python3.7/site-packages/arcgis/features/layer.py in __init__(self, url, gis, container, dynamic_layer) 50 if str(url).lower().endswith("/"): 51 url = url[:-1] ---> 52 super(FeatureLayer, self).__init__(url, gis) 53 self._storage = container 54 self._dynamic_layer = dynamic_layer ~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, gis) 11561 11562 def __init__(self, url, gis=None): > 11563 super(Layer, self).__init__(url, gis) 11564 self.filter = None 11565 self._time_filter = None ~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, gis) 11412 11413 if gis is None: > 11414 gis = GIS(set_active=False) 11415 self._gis = gis 11416 self._con = gis._con ~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs) 357 "argument when connecting to the GIS.") 358 else: --> 359 raise e 360 try: 361 if url.lower().find("arcgis.com") > -1 and \ ~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs) 341 custom_auth=custom_auth, #token=self._utoken, 342 client_secret=client_secret, --> 343 trust_env=kwargs.get("trust_env", None)) 344 if self._is_hosted_nb_home: 345 # For GIS("home") objects, force no referer passed in ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in __init__(self, url, username, password, key_file, cert_file, expiration, referer, proxy_host, proxy_port, connection, workdir, tokenurl, verify_cert, client_id, custom_auth, token, **kwargs) 171 trust_env=trust_env) 172 #self.get_version(True) --> 173 self.get_properties(True) 174 175 ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force) 1136 resp = self.con.get(path, ssl=True) # issue seen with key, cert auth 1137 if not resp: -> 1138 raise e 1139 1140 if resp: ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force) 1124 resp = None 1125 try: -> 1126 resp = self.con.post(path, self._postdata(), ssl=True) 1127 except Exception as e: 1128 if not self.con._verify_cert and \ ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs) 694 except requests.exceptions.ConnectionError as errCE: 695 raise requests.exceptions.ConnectionError( --> 696 "A connection error has occurred: %s" % errCE) 697 except requests.exceptions.InvalidHeader as errIH: 698 raise requests.exceptions.InvalidHeader( ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff17d6ce438>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

JoshuaBixby — Tue, 18 May 2021 17:26:52 GMT

Use a ServicesDirectory object as your GIS:

>>> import re >>> import arcgis >>> arcgis.__version__ '1.8.5' >>> >>> from arcgis.features import FeatureLayer >>> from arcgis.gis.server import ServicesDirectory >>> >>> svc_url = r"https://localhost.localdomain/arcgis/rest/services/someMapService/MapServer" >>> sd = ServicesDirectory(re.search(r"(http.*/rest/services)", svc_url).group(1)) >>> fl = FeatureLayer(svc_url, gis=sd) >>>

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

darrenng123123 — Wed, 19 May 2021 03:14:35 GMT

You are a life saver. Thanks!

Re: Using ArcGIS Python API to consume standalone ArcGIS REST API

darrenng123123 — Wed, 19 May 2021 08:01:57 GMT

Hi Joshua,

Thanks for all the help thus far.

I successfully created a MapImageLayer object without any error. My intent is to display this image as a map or widget.

However, I realized that I must create a WebMap object first, before adding this layer. Creating a default WebMap() object also triggers a call to arcgis online, which results in an error.

Do you know if there is any way to use this MapImageLayer to create a WebMap() object, or to create a WebMap() object from a MapServer REST URL.

Again, thanks for all your help

Here is my code:

from arcgis.mapping import MapImageLayer import re from arcgis.gis.server import ServicesDirectory from arcgis.mapping import WebMap svc_url = r"https://xxx.xxx.xxx/arcgis/rest/services/MapServices/xxxxxx/MapServer" sd = ServicesDirectory(re.search(r"(http.*/rest/services)",svc_url).group(1)) mil = MapImageLayer(svc_url,gis=sd) wm = WebMap() wm.add_layer(mil)

Here is the same error code:

--------------------------------------------------------------------------- gaierror Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self) 156 conn = connection.create_connection( --> 157 (self._dns_host, self.port), self.timeout, **extra_kw 158 ) /opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options) 60 ---> 61 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): 62 af, socktype, proto, canonname, sa = res /opt/conda/lib/python3.7/socket.py in getaddrinfo(host, port, family, type, proto, flags) 747 addrlist = [] --> 748 for res in _socket.getaddrinfo(host, port, family, type, proto, flags): 749 af, socktype, proto, canonname, sa = res gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: NewConnectionError Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 671 headers=headers, --> 672 chunked=chunked, 673 ) /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw) 375 try: --> 376 self._validate_conn(conn) 377 except (SocketTimeout, BaseSSLError) as e: /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn) 993 if not getattr(conn, "sock", None): # AppEngine might not have `.sock` --> 994 conn.connect() 995 /opt/conda/lib/python3.7/site-packages/urllib3/connection.py in connect(self) 333 # Add certificate verification --> 334 conn = self._new_conn() 335 hostname = self.host /opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self) 168 raise NewConnectionError( --> 169 self, "Failed to establish a new connection: %s" % e 170 ) NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: MaxRetryError Traceback (most recent call last) /opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 448 retries=self.max_retries, --> 449 timeout=timeout 450 ) /opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 719 retries = retries.increment( --> 720 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 721 ) /opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace) 435 if new_retry.is_exhausted(): --> 436 raise MaxRetryError(_pool, url, error or ResponseError(cause)) 437 MaxRetryError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: ConnectionError Traceback (most recent call last) ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs) 686 cert=cert, --> 687 files=files) 688 except requests.exceptions.SSLError as err: /opt/conda/lib/python3.7/site-packages/requests/sessions.py in post(self, url, data, json, **kwargs) 580 --> 581 return self.request('POST', url, data=data, json=json, **kwargs) 582 /opt/conda/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 532 send_kwargs.update(settings) --> 533 resp = self.send(prep, **send_kwargs) 534 /opt/conda/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs) 645 # Send the request --> 646 r = adapter.send(request, **kwargs) 647 /opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 515 --> 516 raise ConnectionError(e, request=request) 517 ConnectionError: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: ConnectionError Traceback (most recent call last) <ipython-input-16-6bfc5e47757d> in <module> 9 mil = MapImageLayer(svc_url,gis=sd) 10 ---> 11 wm = WebMap() 12 13 wm.add_layer(mil) ~/.local/lib/python3.7/site-packages/arcgis/mapping/_types.py in __init__(self, webmapitem) 233 # Set up map widget to use in jupyter env: have changes made to 234 # self._webmapdict get passed to the widge to render --> 235 self._mapview = MapView(gis=self._gis, item=self) 236 self._mapview.hide_mode_switch = True 237 self._mapview._webmap = {} ~/.local/lib/python3.7/site-packages/arcgis/widgets/_mapview/_mapview.py in __init__(self, gis, item, mode, **kwargs) 692 693 # Set up gis object --> 694 self._setup_gis_properties(gis) 695 696 # Set up miscellanous properties needed on startup ~/.local/lib/python3.7/site-packages/arcgis/widgets/_mapview/_mapview.py in _setup_gis_properties(self, gis) 958 if gis is None: 959 from arcgis.gis import GIS --> 960 gis = GIS(set_active=False) 961 self.gis = gis 962 ~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs) 357 "argument when connecting to the GIS.") 358 else: --> 359 raise e 360 try: 361 if url.lower().find("arcgis.com") > -1 and \ ~/.local/lib/python3.7/site-packages/arcgis/gis/__init__.py in __init__(self, url, username, password, key_file, cert_file, verify_cert, set_active, client_id, profile, **kwargs) 341 custom_auth=custom_auth, #token=self._utoken, 342 client_secret=client_secret, --> 343 trust_env=kwargs.get("trust_env", None)) 344 if self._is_hosted_nb_home: 345 # For GIS("home") objects, force no referer passed in ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in __init__(self, url, username, password, key_file, cert_file, expiration, referer, proxy_host, proxy_port, connection, workdir, tokenurl, verify_cert, client_id, custom_auth, token, **kwargs) 171 trust_env=trust_env) 172 #self.get_version(True) --> 173 self.get_properties(True) 174 175 ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force) 1136 resp = self.con.get(path, ssl=True) # issue seen with key, cert auth 1137 if not resp: -> 1138 raise e 1139 1140 if resp: ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_portalpy.py in get_properties(self, force) 1124 resp = None 1125 try: -> 1126 resp = self.con.post(path, self._postdata(), ssl=True) 1127 except Exception as e: 1128 if not self.con._verify_cert and \ ~/.local/lib/python3.7/site-packages/arcgis/gis/_impl/_con/_connection.py in post(self, path, params, files, **kwargs) 694 except requests.exceptions.ConnectionError as errCE: 695 raise requests.exceptions.ConnectionError( --> 696 "A connection error has occurred: %s" % errCE) 697 except requests.exceptions.InvalidHeader as errIH: 698 raise requests.exceptions.InvalidHeader( ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='www.arcgis.com', port=443): Max retries exceeded with url: /sharing/rest/portals/self (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7baba8dcc0>: Failed to establish a new connection: [Errno -2] Name or service not known'))