rsunderman-esristaff

HowTo:  Administratively Reset GeoEvent Server (10.5.x / 10.6.x)

Blog Post created by rsunderman-esristaff Employee on Jul 25, 2018

GeoEvent Server Automatic Configuration Backup Files

It is possible, and in fact preferred, to create XML snapshots of your ArcGIS GeoEvent Server configuration using GeoEvent Manager (Site > GeoEvent > Configuration Store > Export Configuration).

But what if something has gone sideways and you cannot access GeoEvent Manager? Before you delete GeoEvent Server’s ZooKeeper distributed configuration store, you will want to locate a recent XML configuration and see if recent changes to inputs, outputs, GeoEvent Definitions, and GeoEvent Services are in the configuration file.

Beginning with GeoEvent Server 10.5, a copy of the configuration is exported automatically for you, daily, at 00:00:00 hours (local time).

  • Automatic backup files, by default, are written to the following folder:
    C:\ProgramData\Esri\GeoEvent
  • You can change the folder used by editing the folder registered for 'Automatic Backups':
    Site > GeoEvent > Data Stores > Register Folder
  • You can change when and how often snapshots of your configuration are taken:
    Site > Settings > Configure Global Settings > Automatic Backup Settings

 

GeoEvent Server ZooKeeper Files

At the 10.5 / 10.5.1 release – GeoEvent Server uses the “synchronization service” platform service in ArcGIS Server, which is running an Apache ZooKeeper behind the scenes. Since this is an ArcGIS Server service, the application files are found in the ArcGIS Server 'local' folder (e.g. C:\arcgisserver\local).

If a system administrator wanted to administratively clear a configuration of GeoEvent Server they could stop the ArcGIS Server platform service -- using the Administrative API -- or stop the ArcGIS Server Windows service and delete the files and folders found beneath C:\arcgisserver\local\zookeeper\.

  • You should leave the parent folder, C:\arcgisserver\local\zookeeper intact.
  • You should also confirm with Esri Technical Support that patches, service packs, or hot-fixes you may have installed have not changed how the “synchronization service” platform service is used by other ArcGIS Enterprise components before administratively deleting files from beneath the ArcGIS Server directories. (ArcGIS GeoAnalytics Server, for example, uses the platform service to elect a machine participating in a multiple-machine analytic as the "leader" for an operation.)

Beginning with the 10.6 release – GeoEvent Server is running its own Apache ZooKeeper instance within the ArcGIS GeoEvent Gateway Windows service. If a system administrator wanted to administratively clear a 10.6 configuration of GeoEvent Server they could stop the ArcGIS GeoEvent Gateway Windows service – which will also stops the dependent ArcGIS GeoEvent Server Windows service – and then delete the files and folders found beneath: C:\ProgramData\Esri\GeoEvent-Gateway\zookeeper-data.


GeoEvent Server Kafka File

NOTE: The following only applies to 10.6 and later releases of GeoEvent Server.

Beginning with the 10.6 release – GeoEvent Server is running an Apache Kafka instance as an event message broker within the ArcGIS GeoEvent Gateway Windows service. The message broker uses on-disk topic queues to manage event records. The event records which have been sent from the message broker to a GeoEvent Server instance for processing are recorded within the broker's associated configuration store (e.g. Apache ZooKeeper).

The Kafka message broker provides a transactional message guarantee that the RabbitMQ message broker (used in 10.5.1 and earlier releases) does not provide. If the GeoEvent Gateway on a machine were stopped and restarted, the configuration store will have recorded where event message processing was suspended and will use indexes into the topic queues to resume processing previously received event records.

The topic queue files are closed, new files created, and old files deleted according to configurable data retention strategy. However, if the GeoEvent Gateway were stopped and its ZooKeeper configuration were deleted, the Kafka topic queues will likely be orphaned and potentially large message log files may not be deleted from disk according to the data retention strategy. In this case, a system administrator might need to locate and delete the topic queue files from beneath C:\ProgramData\Esri\GeoEvent-Gateway\kafka.

 

GeoEvent Server Runtime Files

When GeoEvent Server is initially launched, following a new product installation, a number of files are created as the system framework is built. These files, referred to as “cached bundles” are written into a \data folder in the GeoEvent Server installation directory (e.g  C:\Program Files\ArcGIS\Server\GeoEvent\data). Again, if something has gone sideways, a system administrator might want to try deleting these files, forcing the system framework to be rebuilt, before deciding to uninstall and then reinstall GeoEvent Server.

This might be necessary if, for example, you continue to see the message "No Services Found" displayed in a browser window (after several minutes and a browser refresh) when attempting to launch GeoEvent Manager. In this case, deleting the runtime files from the \data folder to force the system framework to be rebuilt may remedy an issue which prevented GeoEvent Server from launching correctly the first time.

Another reason a system administrator may need to force the system framework to be rebuilt might be observing a message that the ArcGIS GeoEvent Server Windows service could not be stopped “in a timely fashion” (when selecting to stop the service using the Windows Task Manager). In this case, an administrator should ensure the process identified in the C:\Program Files\ArcGIS\Server\GeoEvent\instances\instance.properties file has been stopped. Administratively terminating this processes to stop GeoEvent Server can leave the system framework in a bad state, requiring the \data files be deleted so the framework can be rebuilt.

 

Administratively Reset GeoEvent Server

Deleting the Apache ZooKeeper files (to administratively clear the GeoEvent Server configuration), the product’s runtime files (to force the system framework to be rebuilt), and removing previously received event messages (by deleting Kafka topic queues from disk) is how system administrators reset a GeoEvent Server instance to look like the product has just been installed. Below are the steps and system folders you need to access to administratively reset GeoEvent Server at the 10.5.x and 10.6.x releases.

 

If you have custom components in the C:\Program Files\ArcGIS\Server\GeoEvent\deploy folder, move these from the \deploy folder to a local temporary folder, while GeoEvent Server is running, to prevent the component from being restored (from the distributed configuration store) when GeoEvent Server is restarted. Also, make sure you have a copy of the most recent XML export of your GeoEvent Server configuration if you want to save the elements you have created.

10.5.x

  You should confirm with Esri Technical Support that system folders and files you plan to delete before executing the steps below. Files you delete following the steps below are irrecoverable.

  1. Stop the ArcGIS Server Windows service.
    (This will also stop the GeoEvent Server Windows service)
  2. Locate and delete the files and folders beneath C:\Program Files\ArcGIS\Server\GeoEvent\data
    (Leave the \data folder intact)
  3. Locate and delete the files and folders beneath C:\arcgisserver\local\zookeeper
    (Leave the \zookeeper folder intact)
  4. Locate and delete the files and folders beneath C:\ProgramData\Esri\GeoEvent
    (Leave the \GeoEvent folder intact)
  5. Start the ArcGIS Server Windows service.
    (Confirm you can log in to the ArcGIS Server Manager web application)
  6. Start the ArcGIS GeoEvent Server Windows service.

10.6.x

  Note that the lifecycle of the ArcGIS GeoEvent Gateway service is intended to mirror that of the operating system.
  You can administratively reset GeoEvent Server (e.g. deleting its runtime files from its \data folder) without stopping the ArcGIS GeoEvent Gateway service -- unless you also want to administratively delete the ZooKeeper files from the configuration store (which in the 10.6.x are maintained as part of the ArcGIS GeoEvent Gateway service).

  1. Stop the ArcGIS GeoEvent Server Windows service.
  2. Locate and delete the files and folders beneath the following directories (leaving the parent folders intact):
    C:\Program Files\ArcGIS\Server\GeoEvent\data\
    C:\ProgramData\Esri\GeoEvent\
  3. Stop the ArcGIS GeoEvent Gateway Windows service.
    This will also stop the ArcGIS GeoEvent Server Windows service if it is running.
  4. Locate and delete the files and folders beneath the following directories.
    Leave the parent folders (highlighted) intact:
    C:\ProgramData\Esri\GeoEvent-Gateway\zookeeper-data
    C:\Program Files\ArcGIS\Server\GeoEvent\gateway\log
  5. If you delete the zookeeper-data files, you should remove any orphaned topic queues
    by deleting the on-disk Kafka logs (delete the 'logs' sub-folder, leave the 'kafka' folder intact):
    C:\ProgramData\Esri\GeoEvent-Gateway\kafka\logs
  6. Locate and delete the GeoEvent Gateway configuration file (a new file will be rebuilt).
    C:\Program Files\ArcGIS\Server\GeoEvent\etc\com.esri.ges.gateway.cfg
  7. Start the ArcGIS GeoEvent Server Windows service.
    This will start the ArcGIS GeoEvent Gateway service if it has been stopped.
    Confirm you can log in to GeoEvent Manager.

At this point you can also review the contents of the rebuilt com.esri.ges.gateway.cfg file. The GeoEvent Gateway will record its message broker and configuration store port configurations in this file if it was able to launch successfully:

gateway.zookeeper.connect=MY-MACHINE.MY-DOMAIN:4181

gateway.kafka.brokers=MY-MACHINE.MY-DOMAIN:9192

gateway.kafka.topic.partitions=3

gateway.kafka.topic.replication.factor=3

Outcomes