GeoEvent Processor shut down and restart troubles

[AllenScully]

We have occasional server shutdowns on the machine hosting both map/feature services and GeoEvent Extentsion.  Sometimes planned, sometimes not so much.  These shutdowns absolutely destroy our GeoEvent services, to the point where I have to publish new map/feature services and re-configure the existing GeoEvent services to use these new services.  I have tried 'graceful' shutdowns - shutting down the GeoEvent Service, then input+output connectors, then GeoEvent Service as a whole, then ArcServer, and bringing them back up manually in the reverse order) - no luck, things are still completely busted.  

Is there a way to handle this kind of thing?  It's pretty disruptive and frustrating.  

Thanks - 

Allen

[ChrisBeyett]

Hi Allen, 

Sorry to hear about the GeoEvent Extension within your environment, and this is certainly not expected behavior. We can try to narrow down the potential cause, as from what you described could be the ArcGIS Server or GeoEvent Extension. It will be easier to pose a few questions regarding your environment first. 

1. When the machine becomes responsive again after the shutdown occurs, does GeoEvent fail to read the ArcGIS Server map/feature services? Another way of looking at this is when you open the GeoEvent Manager, are the input/outputs showing as green (valid) or red (invalid)? 

2. Do the map/feature services themselves become unresponsive at the REST endpoint? 

3. Can you replicate the behavior without a complete system shutdown? Meaning, are your "graceful" shutdowns occurring without shutting down the machine?

4. Finally, what version of ArcGIS Server/GeoEvent do you have within this system?

Thanks

-Chris

[AllenScully]

Thanks Chris - 

The issue can be re-produced without a full server re-start.  Essentially stopping/restarting the ArcGISGeoEvent service on the server (or ArcServer service) has the same effect.

The input/output connectors appear valid (green not red).  The map services are responsive - you can view data via ArcGIS Javascript from the rest endpoint. 

What does sometimes (not always but more than half the time) work is re-validating the servers in the data store (even though the input/output connectors show as valid, as do the items in the data store), then going into each input/output connector and essentially re-setting the properties, specifically the servers and layers where the map/feature services are coming from/writing to.  

The output feature services and GeoEvent extension are running on 10.4, but some of our input map services are coming from a 10.3.1 ArcServer machine.  I thought this may be the problem, but we encountered similar behavior when running everything on 10.3.1 machines.

[AllenScully]

I should add that this is all in a test environment - we are unfortunately locked into 10.3.1 for now in production until we can get our SDE databases upgraded from SQL Server 2008 to 2014.

[ChrisBeyett]

Hi Allen, 

Without visualizing your environment first hand, to me this behavior could be with the data store connections themselves, or another program on the machine that is blocking the GeoEvent.exe and java.exe process from starting correctly. I'll pose a few more questions to see if we can get it:

1. Within the data store connections in GeoEvent Manager, are you using any authentication such as "Use Token" or "Web Tier Authentication"? 

2. When the ArcGIS Server and GeoEvent Windows Services are started again, how much RAM do the "ArcGISGeoEvent.exe" and "java.exe" processes consume? 

3. Do you have any antivirus on-access scanning or other 3rd party applications installed that could be blocking the processes? 

4. At this point, you can start leveraging the GeoEvent logs to further determine the cause of the behavior. 

     a. Start with the ROOT logger set at "INFO"

     b. Look for any Token or data store failures

     c. The physical files are located in the C:\Program Files\ArcGIS\Server\GeoEvent\data\logs folder, and you can zip that      folder and include it in this thread

Thanks

- Chris