GeoEvent Server 10.5.1 “No service was found”

JasonSheldon · ‎12-15-2017

ArcGIS Server on this machine is ok, but whenever we try and access the geoevent server manager page we’re getting a “No service was found” message instead of the login page. The geoevent server can’t start up correctly. I tried several fixes from the forums:

verified the license file
removed cache and lock files from C:\Program Files\ArcGIS\Server\GeoEvent\data
tried completely uninstalling, deleting files from c:\ProgramData\ and then reinstalling

In the karaf.log file (attached), I’m seeing this error…

I also see this entry...

The arcgis server is running with a valid third party cert, and I’m not sure why the .pfx file would even be renamed .cer? The other certs in that directory, including the one used for ArcGIS Server are all ok. In arcgis server admin all looks good.

You can access arcgis server manager at:

https://www.napermaps.com:6443/arcgis/manager

You can see from that it’s using a valid cert from Entrust.

Any thoughts or suggestions would be great! Unsure where to go from here.

JasonSheldon · ‎12-15-2017

Just wanted to add that GeoEvent was functioning ok for over a month with the current configuration. This happened about a week ago.

I'm also unable to shutdown the GeoEvent Server service. It times out. I have to kill the process manually, and the associated java.exe just to get it to shut down.

MarkBramer · ‎12-18-2017

Try this:

There is a bug where the certs from C:\Program Files\ArcGIS\Server\framework\etc\certificates do not correctly copy to C:\arcgisserver\config-store\machines\HOSTNAME.ESRI.COM.

It is the keystore file that is important from this location. You can do a diff operation to check if the files are different.

Here’s the fix for the issue.

- Stop GES.

- Then would recommend you to move the files from C:\arcgisserver\config-store\machines\HOSTNAME.ESRI.COM to another location as a backup.

- Now copy the files from C:\Program Files\ArcGIS\Server\framework\etc\certificates and paste in C:\arcgisserver\config-store\machines\HOSTNAME.ESRI.COM.

- Start GES

- Delete browser cache before accessing GES manager

JasonSheldon · ‎12-18-2017

Mark,

Thanks, I was able to get GeoEvent to start up again without having to do the steps you outlined, but I will do those as well as it should clear up that error.

Here is what worked for me:

This fix online pointed in the right direction, but was outdated. Many of the files are in new locations for 10.5.1.

https://esriaustraliatechblog.wordpress.com/2014/08/12/geoevent-processor-wont-start-up-help/

Shutdown ArcGIS Geoevent Server service from Services panel. This would just hang and eventually time out. Looked at the following file to get the PID of the GeoEvent service:

C:\Program Files\ArcGIS\Server\GeoEvent\instance\instance.properties

From taskmanager, clicked Details tab, find the java.exe with that PID and deleted it.

From services panel, shut down ArcGIS Server service

Deleted the entire data folder at:

C:\Program Files\ArcGIS\Server\GeoEvent\data

Deleted the Zookeeper folder at:

C:\arcgisserver\local\zookeeper

Startup ArcGIS Server service in Services - and waited for about 5 minutes or so for it to complete load.

Startup ArcGIS Geoevent server service

Waited a few minutes and it login page loaded ok.

The fixes were all there in the forums, spread out a bit, and outdated with the locations of zookeeper.

RJSunderman · ‎01-04-2018

Hello Jason -

The fixes were all there in the forums, spread out a bit, and outdated with the locations of zookeeper.

I feel like I need to apologize. It must seem like we're changing our integration with zookeeper every release ...

And true to form, at the 10.6 release, the location of the zookeeper files will be changing again.

At the 10.5 / 10.5.1 release

GeoEvent Server uses the Zookeeper platform service provided by ArcGIS Server for its distributed config store
The Zookeeper files are found in the ArcGIS Server's 'local' directory:
- e.g. C:\arcgisserver\local\zookeeper

Beginning with the 10.6 release

GeoEvent Server is introducing a new architectural component: The GeoEvent Gateway
Look for a "What's New" help topic in the to-be-released 10.6 on-line documentation after 9-Jan-2018
- http://enterprise.arcgis.com/en/geoevent/latest/get-started/whats-new-in-geoevent-server.htm
- That should be the final public link, but it's not yet been pushed for public release, so I can't be sure.

What's Changing ...

GeoEvent Gateway will incorporate a message broker (Apache Kafka) and distributed config store (Apache ZK)
ArcGIS Server will still provide its platform service, but GeoEvent Server isn't using that anymore.
- You'll still find ZK files beneath C:\arcgisserver\local\zookeeper, but those are not the droids you looking for
- The GeoEvent Server's zookeeper files will be found beneath C:\ProgramData\Esri\GeoEvent-Gateway
  - e.g. C:\ProgramData\Esri\GeoEvent-Gateway\zookeeper-data

I'll try to prepare a blog that pulls together the steps for administratively resetting GeoEvent Server, by deleting its data\ files, deleting its zookeeper\ files, deleting the new GeoEvent-Gateway\ files (at 10.6) ... and provide clear steps with correct paths for both 10.5 / 10.5.1 and the 10.6 product releases. I'll add a comment to this thread with a link to that blog when it's available.

- RJ

RJSunderman · ‎01-08-2018

Blog has been posted:

https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2018/01/09/howto-administrativ...‌

JoëlHempenius3 · ‎07-29-2018

My GeoEvent is still not working with the signed fields invalid error in the log.

The problems started when I updated Geo Event server from 10.5.1 to 10.6.1, after completing the update of both Arcgis Server and Geo Event server I got the "No service was found." message in the browser. In the Geo Event log, I see the signed fields invalid error message.

First, I tought something was wrong with the certificate, so I removed the old certifcate and reinstalled a the certificate in Arcgis Server Manager.

I uninstalled GeoEvent Server and then reinstalled.

I did the administrative reset as described in the blog

I also copied the certificates as described by Mark Bramer.

Still i"m stuck with the signed fields invalid error and the No service was found.

Any advice?

-Joël Hempenius.

Languages: JavaScript, Python and Dunglish

MarkYerington · ‎08-31-2018

ESRI could you provide some guidance as we to will be migrating soon from 10.5.1 to 10.6.1. When we upgraded the last time to 10.5.1 we were able to just restart the Geoevent service a couple times because of the above error and everything fired back up luckily. Is there a good outline of updating a Geoevent Machine as there seems to be some communications issues between the Geoevent server and ArcGIS server that is required with a dedicated server for the install. We have a lot of people relying on this service and I would like to minimize downtime when upgrading.

Javier_Pintos · ‎10-17-2018

I have the same error:

2018-10-17T10:19:49,051 | ERROR | Curator-Framework-0 | CuratorFrameworkImpl             | 205 - curator-framework - 3.1.0 | Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
     at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) [388:org.apache.zookeeper.zookeeper-geoevent:3.5.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:802) [205:curator-framework:3.1.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:931) [205:curator-framework:3.1.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:883) [205:curator-framework:3.1.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:69) [205:curator-framework:3.1.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:310) [205:curator-framework:3.1.0]
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:?]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:?]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
     at java.lang.Thread.run(Thread.java:748) [?:?]
2018-10-17T10:19:49,052 | ERROR | Curator-Framework-0 | CuratorFrameworkImpl             | 205 - curator-framework - 3.1.0 | Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
     at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:912) [205:curator-framework:3.1.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:883) [205:curator-framework:3.1.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:69) [205:curator-framework:3.1.0]
     at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:310) [205:curator-framework:3.1.0]
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:?]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:?]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
     at java.lang.Thread.run(Thread.java:748) [?:?]

2018-10-17T10:24:05,494 | ERROR | Blueprint Extender: 1 | BlueprintContainerImpl           | 244 - org.apache.aries.blueprint.core - 1.8.0 | Unable to start blueprint container for bundle com.esri.arcgis.discovery.arcgis-stream-container/10.6.0 due to unresolved dependencies [(objectClass=com.esri.ges.manager.datastore.agsconnection.ArcGISServerConnectionManager), (objectClass=com.esri.ges.manager.globalproperty.GlobalPropertyManager), (objectClass=com.esri.ges.messaging.StreamMessaging)]
java.util.concurrent.TimeoutException
     at org.apache.aries.blueprint.container.BlueprintContainerImpl$1.run(BlueprintContainerImpl.java:372) [244:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) [244:org.apache.aries.blueprint.core:1.8.0]
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:?]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:?]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
     at java.lang.Thread.run(Thread.java:748) [?:?]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

*The last exception is the first of many exception of timeout on dependencies.

I tried reinstall many time.

I tried administrative reset.

I tried with certificate too.

But it doesn't works.

Do you have any advice?

Thank you.

Javier.

MarkYerington · ‎10-17-2018

Javier

I would probably try the administrative reset as described in the post. We have not tackled this upgrade, but will be in a few months.

Mark Yerington │GIS Manager│ GISP

Muscatine Area Geographic Information Consortium

c/o Muscatine Power & Water | 3205 Cedar Street | Muscatine, IA 52761

Office: 563.262.3316│Cell: 563.260.4525 Email: myerington@mpw.org│Web: magic-gis.com<https://magic-gis.com/>