topic Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services in ArcGIS GeoEvent Server Questions

GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

DennisGeasan — Thu, 16 Jan 2014 18:01:26 GMT

This is related to an earlier thread where I cannot register an ArcGIS server data store.

The ArcGIS Server data store I'm trying to register is configured as follows:
Machine: Windows Server 2008 R2
ArcGIS Server: 10.2.1
AGS Security: Windows for Users and Roles with the 'Web' (Web Adapter) used for the Authentication Tier and Mode;
Map Services Permissions: Root level - set to 'Private'.

With the above configuration on the AGS machine I cannot register it in 10.2.1 GEP. If I change the root level permissions on the AGS machine to 'Public' then I can register it in GEP. If I set the root level to 'Private' and then change the permissions on a folder to public, then I CAN register the AGS machine in GEP

It is required at this site that all map service permissions be private. How do I accomplish that with the new version of GEP? I tried defining tokens but that doesn't work.

Dennis Geasan

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

JavierDelgadillo — Thu, 16 Jan 2014 20:28:04 GMT

Dennis,

The only mode of authentication supported by GEP at 10.2.1 for Servers and Portals is token based authentication. We do not support any of the other various forms of authentication protocols supported by Web Browsers (like NTLM, Basic, Digest, or PKI). If your WebAdaptor installation requires that (as it seems to), then you cannot configure GEP access to go through the WebAdaptor.

If both GEP and your ArcGIS Server sit behind the firewall, then you should configure GEP to talk directly to your ArcGIS Server using a token obtained using the browser. The ArcGIS datastore registration dialog will show you the link to visit and get a token. You'll need to ensure you copy the referrer exactly as it appears in the registration dialog (beware of copying/pasting whitespace), otherwise GEP will not be able to talk with your ArcGIS Server.

-Javier

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

DennisGeasan — Thu, 16 Jan 2014 22:50:58 GMT

Hello Javidel,

I did have success defining the AGS data store by generating a token using the ArcGIS Server administrators account user ID/password. For using windows Authentication on the AGS machine I guess this is my only option.

However, I am able to use the web adapter when defining a GEP data source. In GEP 10.2 I ran a service with a web adapter connection for several weeks. It still works in 10.2.1 except now I have to set the ArcGIS Server feature access service permissions to 'public access' and I have to include port number 6080 in the GEP data source URL. In 10.2 I did not have to do that. In both versions of ArcGIS Server I am using Windows authentication (users and roles) with the Web Adapter installed. In IIS I have the Web Adapter site configured with 'Anonymous Authentication' disabled and 'Windows Authentication' enabled.

I've generated tokens as you described but the data source fails to connect when I use a windows user account. Attached is more description.

For the customer site I'm working with it is a requirement that all map services be private. And the web adapter is also required for Windows Authentication with the corporate domain. For GEP 10.2.1 it looks like I now have to register the AGS data store using the AGS admin account.

Not sure if that is a good thing or not. Comments?

Also, as a suggestion, a little more description in the help on this subject would be useful. If it is required that GEP connect to AGS using a token then explain that in the help and also don't provide the option to not use a token in the GEP data source dialog.
DG

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

JavierDelgadillo — Tue, 21 Jan 2014 20:38:25 GMT

Will try to answer some of the questions from your PDF here.

We have the option to use a token because some ArcGIS Servers require tokens while others don't. We don't know that when registering the connection. Furthermore, some servers may have some folders/services that are publicly viewable while some are restricted. In these cases, users need the ability to use or not use a token to access the services on their ArcGIS Server.

When choosing the ID/password to enter on the token generation page, enter the credentials for an account that can access the service you want to consume or update. Additionally, set the Expiration to the length of time to the duration you want your DataStore to be valid.

If you mouse-over the red icon in the ArcGIS data store window, what message do you see? This should give you a hint as to why GEP is having trouble accessing the services.

-Javier

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

DennisGeasan — Fri, 10 Dec 2021 22:41:11 GMT

Hello Javidel,
The tokens I am creating are lasting only one day. Yesterday I registered the server using a token that was created using the AGS admin account. After the registration process there was a green check mark in the status column. This morning there is a red circle with a horizontal white bar. When I mouse over the icon the message is "Invalid Token". I am selecting the option in the token generator to allow the token to last one year.

In the token generator, you have the option of setting the response to HTML or JSON. I have been using the JSON response which includes a UNIX time format string for the expiration date. Running that value thru a Unix time format converter it is 24 hours after creating the token, even if I have chosen the one year option.

So yesterday I created two AGS registrations, one based on the JSON response and the second on the HTML response. My thought was that maybe the HTML response provided the correct token. No joy. Today, the status of both entries is now the red icon/"Invalid Token". For both of those entries I used the AGS admin account for the user ID/password.

I tried again to use a windows domain account in the token generator. I get the following response:

{"error":{"code":200,"message":"You are not authorized to access this information","details":"Invalid credentials"}}

The AGS machine I am registering is using Windows for managing roles and users and the Web Adapter is installed and working properly. I tried both styles for the user id (domain\userID and userID alone). Same response to both.

I checked the security section of AGS and I see the User and Role store is actually set to 'ASP.NET'. That is a configuration (one recommended by ESRI) I had to do in 10.2.0 to accommodate issues with AD roles. After the upgrade to 10.2.1 from 10.2.0 this read as "Windows" for the User/Role store but apparently it has reverted to the previous configuration. I will pursue getting this set back to strictly Windows but:

Should I be able to use a Windows user ID/Password in the token generator?

I would rather not have to use tokens. There is the obvious problem of remembering to regenerate the token and if we can't get a token to last more than 24 hours, well....... Using a Windows AD account should be sufficient. Access to folders and services in AGS is defined by the Win AD roles assigned to those folders/services so the token approach is not really needed. But if the GEP code base can only connect to AGS via tokens then I should be able to utilize a Windows AD domain account to generate the token.

DG

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

AndreasEspersen — Fri, 20 Mar 2015 12:00:06 GMT

Hi Javier,

There is one thing i do not understand when adding a data store with a token.

We have a setup with a server holding GEP and ArcGIS Server installed and wanting to register a data store on another server with ArcGIS Server containing feature services.. In order to be able to cluster the server with feature services. We are registering the data store the server with feature services via another server with the web adaptor installed as shown below.

The web adaptor gives access to the arcgis server with feature services to outside our internal network. Therefore we have disabled administrative access on the Web adaptor for security reasons.

The data stores is successfully added but from the log I can see that a call is being made to

https://<Web adaptor URL>/arcgistest/admin/data/findItems?f=json&token=XXXXXXX&types=egdb: GET Request failed(HTTP/1.1 403 Forbidden).

Why is the GEP trying to access an admin URL?

Regards

- Andreas

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

RJSunderman — Tue, 28 Apr 2015 00:57:21 GMT

Hello Andreas -

My apologies that no one has gotten back with you in response to your question.

GeoEvent was designed to rely on GIS-tier communications with an ArcGIS Server site using Server generated tokens. Support for web-tier authentication and authorization using SAML and IWA is being developed, but is limited, and is being introduced beginning with the 10.3.1 product release.

I need to clarify your question. Are we working with two different GIS Server machines, both with ArcGIS Server installed, which are participating in different ArcGIS Server sites? I ask because you mentioned the term "cluster" and I want to make sure that the GIS Servers we are considering are not part of a single ArcGIS Server site cluster.

If the two GIS Servers were participating in the same ArcGIS Server site cluster with the GeoEvent Extension installed on one GIS Server but not on the other we would be working with an unsupported configuration. All of the GIS Servers in a site's cluster must be provisioned exactly the same - this is important when using the GeoEvent Extension.

Next question - is it an option for your first ArcGIS Server, the one which has the GeoEvent Extension installed, to reach the second ArcGIS Server without going through the Web Adapter? Design assumptions built-in to the GeoEvent 10.3 release (and previous releases) assume that GeoEvent will be able to discover services using the ArcGIS REST Services Directory and that the directory will be accessed through either the open REST endpoint http://server‑name.domain:6080/arcgis or the secure endpoint https://server‑name.domain:6443/arcgis. The preference would be that GeoEvent not attempt to go through the web adapter to discover a server's services.

Some support for IWA was implemented in the 10.3.1 product release for registering ArcGIS Server connections as GeoEvent Data Stores. For example, you can specify that GeoEvent should use a token to access the data store and specify the secure endpoint (https://server‑name.domain:6443/arcgis) as the URI to be used. When the token expires, you will have to refresh your Data Store configuration with a new token in order to continue using the data store. You can also, beginning with the 10.3.1 release, select to use web-tier authentication and supply a recognized IWA username and password when registering a data store.

My understanding is that, even when using web-tier authentication, that access must still be made through fully-qualified server-name / port URI ... either the http://server‑name.domain:6080/arcgis endpoint or the secure endpoint https://server‑name.domain:6443/arcgis. I will try to get someone to confirm this.

Hope this information helps -

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

RJSunderman — Tue, 28 Apr 2015 18:42:06 GMT

Andreas -

Some feedback specifically on your question: "Why is the GEP trying to access an admin URL?"

https://<Web adaptor URL>/arcgistest/admin/data/findItems?f=json&token=XXXXXXX&types=egdb: GET Request failed(HTTP/1.1 403 Forbidden).

The GeoEvent Extension is using the admin API to determine if a managed geodatabase has been registered with ArcGIS Server. If there is a managed geodatabase, when configuring a new 'Send Features to a Stream Service' output and selecting to publish a Stream Service, you will have the option to check the 'Store Latest' checkbox and publish a "buddy" feature service which will cache the most recent observation for each received TRACK_ID. If you do not have a managed geodatabase registered with ArcGIS Server, the 'Store Latest' option is not available.

- RJ

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

AndreasEspersen — Wed, 29 Apr 2015 12:13:41 GMT

Hi RJ,

Thanks for getting back - Much appreciated.

A little background information about our setup i might provide some insight why we asked the question.

We are using one ArcGIS for Server with GeoEvent Extension - Lets called this server for Server 1. We have another ArcGIS for Server holding feature services (FS) and map services (MS) (Server 2). The GeoEvent Extension outputs to a FS on server 2. From what I heard from UC and what one of my colleagues was told in Palm Spring this is best practice.

Server 2 make MS and FS available both internally and externally. Some of the applications require to have security enabled. One of them is a service that the GeoEvent Extension updates.

Sometime in the future we would like to add more ArcGIS For Server to Server 2 and create a cluster still holding all FS and MS. We have prepared for this are using the web adaptor in front of the ArcGIS Server holding FS and MS as a load balancer but in the future we might use a physical load balancer. We have followed the architecture described in ArcGIS Help (10.2, 10.2.1, and 10.2.2).

Server 1 is not accessible from outside our domain and only run GeoEvent Extension.

We raised the initially question when we had problems updating FS with Security enabled on server 2 (GIS-Tier with users LDAP). The update would run about 4-6 hours and then stop.

The Geoevent Extension would say everything was still alright. The data store could still be vailidated but the Feature class being updated did not receive any updates. The only thing that caught my attention in the log was was:

'https://<Web adaptor URL>/arcgistest/admin/data/findItems?f=json&token=XXXXXXX&types=egdb: GET Request failed(HTTP/1.1 403 Forbidden).'

If we connect to server 2 with the directy URL then we the error below: properly due a certificate error.

Invalid URL: Please provide a valid url for example: http://<hostname>:<port>/arcgis

So why is it that the geoevent Extension stops after 4-6 hours?

- Andreas

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

DennisGeasan — Wed, 29 Apr 2015 16:08:58 GMT

Andreas,

I found Version 10.2.2 a little more stable. I experienced a similar problem with the extension just stopping after random amounts of time when using a Web Socket "listening" connection as the input. The inputs seem to be more reliable if they poll the data source rather than waiting (listening) for the data source to send the input data.

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

RJSunderman — Fri, 01 May 2015 17:30:47 GMT

So why is it that the geoevent Extension stops after 4-6 hours?

There was an issue we observed, primarily with updating feature services hosted via ArcGIS Online in the 10.3 release. A few users also observed the issue using local feature services they had published to their on-premises Server. The symptom was that GeoEvent would appear to be receiving and processing data (its event counters would increment), but updates to features through a feature service would stop after a variable length of time. We usually found messages in the karaf.log with the text Error: null response or Error Posting to URL ... java.io.IOException: null respons when this specific issue would manifest. The issue was addressed with the 10.3 Patch 1 for GeoEvent, available here from Esri Support.

I am not sure if the issue was actually present in earlier releases (10.2.x). In any case, I'd encourage you to upgrade to the 10.3.1 release which should be publicly available in just a couple of weeks (13-May 2015). The focus for 10.3.1 was on improving the product's stability, however, we did complete a product enhancement with regard to registering an ArcGIS Server connection as a GeoEvent Data Store. You can now specify whether you want to use a token for authentication or if you want to use web-tier authentication and provide a username / password when registering a GeoEvent Data Store. This is new at the 10.3.1 release.

If we connect to server 2 with the directy URL then we the error below: properly due a certificate error.
Invalid URL: Please provide a valid url for example: http://<hostname>:<port>/arcgis

There were enhancements in the area of security certificates introduced in both the 10.3 and 10.3.1 releases. GeoEvent should trust the ArcGIS Server certificate presented by the local server on which it was installed at the 10.3 release (rather than relying on a GeoEvent self-signed certificate). With the 10.3.1 release GeoEvent will also trust any SSL certificates you have imported using the ArcGIS Server Administrator Directory.

After you've upgraded to the latest release, if you find that, after working for a period of time, feature updates stop working, you see log messages you believe are security or SSL certificate related, or need advice on configuring your enterprise's reverse proxy or web-tier authentication ... please go ahead and open new GeoNet discussion threads for the separate issues. Questions on enterprise system architecture and security configuration I will probably have to refer to Esri Tech Support. I'm better positioned to offer advice on questions concerning GeoEvent product functionality.

Best Regards -

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

AndreasEspersen — Tue, 05 May 2015 06:27:49 GMT

Hi RJ,

Thanks for your feedback.

We found a workaround that has been running for almost a week now.

Making a change in the host file on the GEP server we are now accessing the local server with the FS directly and not via the web adaptor.

In this way we avoid the bad URL and can run with security. It is not an optimal solution for us but we can wait for 10.3.1.

Thanks

Andreas

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

ThomasColson — Sun, 28 Jun 2015 22:43:07 GMT

Given that a large chunk of users of ArcGIS Server and GEP are being required to move to Windows AD and PKI Smart Card Auth (nothing else is allowed, period, non-negotiable, if the software doesn't take a smart card, it's getting uninstalled).......I'm wondering when ESRI plans to address the authentication issues that its customers are forced to work with.

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

KDeVogelaere — Wed, 09 Sep 2015 21:21:03 GMT

I have run into a similar issue in GeoEvent Processor 10.2.2,

Machine: Windows Server 2008 R2

ArcGIS Server: 10.2.2

AGS Security: Windows for Users and Roles with the 'Web' (Web Adapter) used for the Authentication Tier and Mode

Map Services Permissions: Folder level - set to 'Private'.

Our DataStore connections to secured ArcGIS Servers have a 365 token expiration and often several days after registering the connection it will show 'Invalid Token' and the services are in 'Error' state. I am unsure what is triggering the issue, but we've resolved this by editing the Datastore connection and re-entering the same token ID, saving the edit, and clicking 'Validate All'.

Invalid Token java.lang.Exception: Invalid Token at com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection.defaultValidate(DefaultArcGISServerConnection.java:571)[226:com.esri.ges.framework.datastore.agsconnection-datastore:10.2.2] at com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection.validate(DefaultArcGISServerConnection.java:589)[226:com.esri.ges.framework.datastore.agsconnection-datastore:10.2.2] at com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection.isUseable(DefaultArcGISServerConnection.java:2376)[226:com.esri.ges.framework.datastore.agsconnection-datastore:10.2.2] at com.esri.ges.transport.featureService.FeatureServiceInboundTransport.isArcGISConnectionConfigurationValid(FeatureServiceInboundTransport.java:401)[251:com.esri.ges.framework.transport.featureservice-transport:10.2.2] at com.esri.ges.transport.featureService.FeatureServiceInboundTransport.onArcGISServerConnectionStatusChange(FeatureServiceInboundTransport.java:422)[251:com.esri.ges.framework.transport.featureservice-transport:10.2.2] at com.esri.ges.manager.datastore.agsconnection.internal.ArcGISServerConnectionManagerImpl$1.run(ArcGISServerConnectionManagerImpl.java:236)[318:com.esri.ges.manager.internal-agsconnectionmanager:10.2.2] at java.lang.Thread.run(Unknown Source)[:1.7.0_51]

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

JavierDelgadillo — Thu, 10 Sep 2015 15:23:47 GMT

K DeVogelaere,

If you're regularly having to update tokens because they're expiring, you may consider introducing the GeoEvent Datastore Proxy into your environment. It's a Java WebApplication designed to act as the endpoint GeoEvent communicates with, but is responsible for updating tokens when connecting to a system that requires username/password for authentication to either the WebTier or GIS Tier.

You can view the project here Esri/geoevent-datastore-proxy · GitHub and download the first release here Release Initial Release of GeoEvent DataStore Proxy · Esri/geoevent-datastore-proxy · GitHub .

If the proxy looks like something you'd be interested in, you could submit a Tech Support incident if you need some help configuring the proxy for your environment.

-Javier

Re: GEP 10.2.1 Datastore - Cannot register AGS w/ Windows AD & non-public services

KDeVogelaere — Thu, 10 Sep 2015 19:32:42 GMT

Thank you Javier for sharing this info, when you say GeoEvent Datastore Proxy is responsible for updating tokens do you mean re-generating new tokens for external ArcGIS Servers? Or bouncing the keep-alive tokens stored within GeoEvent Processor?

I am not convinced the tokens are expiring, using the same token as entered previously to register the ArcGIS Server is still allowing a connection and showing a valid token has been entered. Our Operations team regularly runs security patching on this server about once a month taking a full reboot after the patching is complete. Sorry I do not have enough details to determine the cause of the 'Invalid Token' message at this time.