lost geoevent services on server reboot

[BradleyNeish1]

Whenever my server gets restarted, I lose my geoevent services(open up designer, there are no services). I have them saved, so it's pretty easy to get them back. I was just wondering if this is on purpose, or a known issue, or if there's something wrong with my setup?

[RJSunderman]

Hey Bradley -

When you indicate "whenever my server gets restarted", are you saying whenever the server machine itself is rebooted? Do you see this same behavior (opening the Service Designer and finding no services) when you stop and restart the ArcGISGeoEventProcessor windows service? Do you typically exit/close the Service Designer application and the GeoEvent Processor Manager web page before the restart occurs?

I believe you are describing a different manifestation of an issue we"?ve seen before. Let"?s say that for some reason a published GeoEvent Service is not valid when the GeoEvent Processor starts up. Maybe an input component was deleted or failed to load; maybe the bundle supporting a processor component of the service is missing; maybe the GEP configuration was not persisted to disk correctly due to an abrupt system shutdown "? there could be several possible reasons...

If GeoEvent Processor encounters an invalid service on startup, at 10.2.0 the service will not be loaded into the framework. There have been improvements made with 10.2.1 which handle this situation more defensibly. Also with 10.2.1, the application you know as the Service Designer is being retired and its functionality incorporated into the GeoEvent Processor Manager "? so you won"?t have two separate application which you need to "?refresh"? or use to configure and manage your inputs, outputs, and services.

If you"?d like, send us your GeoEventConfig.xml and we can review your configuration. Regardless, I"?m comfortable saying that when the 10.2.1 release is available "? sometime in the next several weeks "? you won"?t continue to experience this issue.

Best Regards -
RJ

[BradleyNeish1]

RJ,

Thanks for replying. I am referring to restarting the machine itself, and it happens whether I have closed the service designer window or not. I have not tested whether this happens if I just start and restart the service.
It's not a big deal since I have the services saved, and don't often restart the machine anyway. Looking forward to the new release and seeing the Service Designer retired and merged into the manager.

Thanks,
b

[DarrylWilson]

Hi There,

I am too finding that the services disappear from the GeoEvent Manager Services page and the Services Designer on server reboot.

The input and outputs are still there, however every time I reboot the services need to be re-created.

I have had to re-boot a number of times as the memory (6GB) was all being used. Also wondering if there is a memory leak anywhere?

Many thanks

Darryl

[RJSunderman]

Hello Darryl -

GeoEvent Processor has a default 2GB limit for RAM allocation. You can configure this for your instance. Locate the product installation folder "�\ArcGIS\Server\GeoEventProcessor\etc on your system and open the ArcGISGeoEventProcessor.cfg file in a text editor. You should be able to locate the configuration setting toward the top of the file:

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

I would recommend monitoring other processes running on the system to see which is consuming the most RAM. Unless you have configured GEP to use more than its default, you shouldn't find that ArcGISGeoEventProcessor.exe is using that much RAM.

The issue with GeoEvent Services disappearing is an issue with 10.2.0 which we have addressed with 10.2.1 (which is effectively a service pack / stabilization release). Can you reevaluate the issue with the 10.2.1 release? This should be publicly available the second week of January. (Regional offices, Business Partners, and Distributors should be receiving their distribution the week of December 19th.)

Thanks -
RJ

[DennisGeasan]

I have also been seeing the same issue.  But today, magically, all the services reappeared after starting the GEP service whereas in the past (for a few weeks) I had to always reload a GE Services from the saved '.ges' file.  In my case I'm running GEP on a VMware Worksation 10 instance.  I don't start the GEP service automatically.  I manually start the service after manually starting ArcGIS Server service.  I do this because it's a development system and I don't want to always run AGS or GEP on system start.  I think what happened today is that I turned off all inputs and outputs that were no longer being utilized in a GE Service.  I've been in the habit of leaving outputs running even if they are not included in a GE Service.  Maybe the best practice is to only enable the inputs/outputs defined in a GE Service. 

In a thread I recently posted I requested help consuming a JSON collection.  The solution provided by Ryan Elliott is one I had unsuccessfully tried several days ago but today it worked so I'm suspicious that whatever was killing the startup of GE Services had also affected my earlier attempt of using a JSON collection in an input.  Of course, it is more likely that I had just buggered something in the input configuration but todays coincidence is curious.

Anyhow, looking forward to the next version.

DG

[DennisGeasan]

GeoEvent Processor has a default 2GB limit for RAM allocation. You can configure this for your instance. Locate the product installation folder �?�\ArcGIS\Server\GeoEventProcessor\etc on your system and open the ArcGISGeoEventProcessor.cfg file in a text editor. You should be able to locate the configuration setting toward the top of the file:

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

On my systems the process "D:\Program Files\ArcGIS\Server\GeoEventProcessor\jre\bin\java.exe" grows to nearly the max heap size. I'm reducing that max heap to 512MB as the server only has 4GB. Any issues with using a smaller max heap?
DG

[RJSunderman]

Hello Dennis -

I'd like to provide you some under-the-hood details which might play into the question we're now discussing. Internally, the GeoEvent Processor is using ActiveMQ to manage the event queues being sent to each node in a GeoEvent Service. Esri Germany reported that they observed a large number of files being written to a tmp_storage folder beneath the ...\data\activemq folder in the product installation directory. This was a disk space consumption issue for them.

We recommended to the Esri Germany team that they increase the wrapper.java.maxmemory setting. Our understanding is that they are feeding a very high volume of events into GeoEvent Processor, which is caching events to disk when the service components are unable to keep up. Our recommendations for system resources are a minimum 6GB of RAM for GEP (vs. the 4GB minimum recommendation for ArcGIS Server). The default sizes for the queues used by the Event Processing framework can also be adjusted; you would only need to consider that if you were working with high volumes of event data. To adjust the queue sizes edit the com.esri.ges.messaging.jms.cfg and increase the defaults to reflect the following:

• com.esri.ges.messaging.jms.destinationPolicy.queue.memoryLimit  (10 megs)
• com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit  (10 megs)
• com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit  (1 gig)

You will need to stop GeoEvent Processor and edit these files as an administrator (assuming they are beneath C:\Program Files\...) then restart GeoEvent Processor in order for the settings to take effect.

If you've observed the java.exe  process which is running with the -Dkaraf.base="C:\<install folder>\GeoEventProcessor" -Dkaraf.data="C:\<install folder>\GeoEventProcessor\data" arguments in its command line actually consuming more than 1024 megs of memory, lowering the wrapper.java.maxmemory value to 512 megs as you suggest will only result in Java throwing out-of-memory exceptions. How many events are you sending into GeoEvent Processor? What inputs are you using? How many different GeoEvent Services do you have running? I think we need to examine the load you are placing on your Server, by GeoEvent Processing as well as any other running services/applications, before we recommend throttling the memory allowed to the Java process.

- RJ

[DennisGeasan]

Hello RJ and thanks for the reply.

I am running 5 Generic JSON inputs. The URL of each input retrieves a JSON collection. Depending on the URL the number of collection items ranges from 35 to about 350. Is each collection item treated as an event? If so then that might explain some other problems I'm encountering based on what you explained.

Each input runs through a field mapper processer that is then directed to an "Update Feature" output.

So 5 inputs, 5 Geoevent Services, and 5 outputs. Each "Update Feature" output goes to a separate AGS feature service.

For all outputs I am capturing the input attribute "$RECEIVED_TIME" so I can determine if an update has actually occurred.

For the input that retrieves 35 items, all records in the target feature class get updated on nearly every run of the input.

The other 4 inputs all retrieve more than 100 items. Usually, not all of the records in the target feature class get updated after an input has run. In fact after about 12 hours, no further updates occur to the target feature classes with inputs items > 100.

The JSON collection from each input represent the locations and some attributes of vehicles at a business site. The intention is to display these on a web map.

Initially I wrote inputs for each vehicle but considering some sites have close to 350 vehicles it is more efficient to make one REST call and get all vehicles in one input.

So the question is, am I attempting to do something GEP is not intended for? If GEP is able to handle this type of input, how do I modify stuff so that the target feature classes are consistently updated.

BTW - the 'java.exe' process has remained at about 600MB since I made the change to max heap size but now one of the ArcSOC.exe services is consuming 1.4GB memory.

Dennis Geasan
GIS Technologies

[DennisGeasan]

com.esri.ges.messaging.jms.destinationPolicy.queue.memoryLimit (10 megs)

com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit (10 megs)

com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit (1 gig)

The second item is listed twice with different values. Which should it be? Is there a third item to adjust?  Thanks.

Dennis Geasan