topic Re: lost geoevent services on server reboot in ArcGIS GeoEvent Server Questions

lost geoevent services on server reboot

BradleyNeish1 — Thu, 17 Oct 2013 14:07:24 GMT

Whenever my server gets restarted, I lose my geoevent services(open up designer, there are no services). I have them saved, so it's pretty easy to get them back. I was just wondering if this is on purpose, or a known issue, or if there's something wrong with my setup?

Re: lost geoevent services on server reboot

RJSunderman — Thu, 17 Oct 2013 18:16:48 GMT

Hey Bradley -

When you indicate "whenever my server gets restarted", are you saying whenever the server machine itself is rebooted? Do you see this same behavior (opening the Service Designer and finding no services) when you stop and restart the ArcGISGeoEventProcessor windows service? Do you typically exit/close the Service Designer application and the GeoEvent Processor Manager web page before the restart occurs?

I believe you are describing a different manifestation of an issue we"?ve seen before. Let"?s say that for some reason a published GeoEvent Service is not valid when the GeoEvent Processor starts up. Maybe an input component was deleted or failed to load; maybe the bundle supporting a processor component of the service is missing; maybe the GEP configuration was not persisted to disk correctly due to an abrupt system shutdown "? there could be several possible reasons...

If GeoEvent Processor encounters an invalid service on startup, at 10.2.0 the service will not be loaded into the framework. There have been improvements made with 10.2.1 which handle this situation more defensibly. Also with 10.2.1, the application you know as the Service Designer is being retired and its functionality incorporated into the GeoEvent Processor Manager "? so you won"?t have two separate application which you need to "?refresh"? or use to configure and manage your inputs, outputs, and services.

If you"?d like, send us your GeoEventConfig.xml and we can review your configuration. Regardless, I"?m comfortable saying that when the 10.2.1 release is available "? sometime in the next several weeks "? you won"?t continue to experience this issue.

Best Regards -
RJ

Re: lost geoevent services on server reboot

BradleyNeish1 — Thu, 17 Oct 2013 19:07:52 GMT

RJ,

Thanks for replying. I am referring to restarting the machine itself, and it happens whether I have closed the service designer window or not. I have not tested whether this happens if I just start and restart the service.
It's not a big deal since I have the services saved, and don't often restart the machine anyway. Looking forward to the new release and seeing the Service Designer retired and merged into the manager.

Thanks,
b

Re: lost geoevent services on server reboot

DarrylWilson — Tue, 10 Dec 2013 01:23:15 GMT

Hi There,

I am too finding that the services disappear from the GeoEvent Manager Services page and the Services Designer on server reboot.

The input and outputs are still there, however every time I reboot the services need to be re-created.

I have had to re-boot a number of times as the memory (6GB) was all being used. Also wondering if there is a memory leak anywhere?

Many thanks

Darryl

Re: lost geoevent services on server reboot

RJSunderman — Tue, 10 Dec 2013 18:22:18 GMT

Hello Darryl -

GeoEvent Processor has a default 2GB limit for RAM allocation. You can configure this for your instance. Locate the product installation folder "�\ArcGIS\Server\GeoEventProcessor\etc on your system and open the ArcGISGeoEventProcessor.cfg file in a text editor. You should be able to locate the configuration setting toward the top of the file:

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

I would recommend monitoring other processes running on the system to see which is consuming the most RAM. Unless you have configured GEP to use more than its default, you shouldn't find that ArcGISGeoEventProcessor.exe is using that much RAM.

The issue with GeoEvent Services disappearing is an issue with 10.2.0 which we have addressed with 10.2.1 (which is effectively a service pack / stabilization release). Can you reevaluate the issue with the 10.2.1 release? This should be publicly available the second week of January. (Regional offices, Business Partners, and Distributors should be receiving their distribution the week of December 19th.)

Thanks -
RJ

Re: lost geoevent services on server reboot

DennisGeasan — Sat, 14 Dec 2013 05:50:58 GMT

I have also been seeing the same issue. But today, magically, all the services reappeared after starting the GEP service whereas in the past (for a few weeks) I had to always reload a GE Services from the saved '.ges' file. In my case I'm running GEP on a VMware Worksation 10 instance. I don't start the GEP service automatically. I manually start the service after manually starting ArcGIS Server service. I do this because it's a development system and I don't want to always run AGS or GEP on system start. I think what happened today is that I turned off all inputs and outputs that were no longer being utilized in a GE Service. I've been in the habit of leaving outputs running even if they are not included in a GE Service. Maybe the best practice is to only enable the inputs/outputs defined in a GE Service.

In a thread I recently posted I requested help consuming a JSON collection. The solution provided by Ryan Elliott is one I had unsuccessfully tried several days ago but today it worked so I'm suspicious that whatever was killing the startup of GE Services had also affected my earlier attempt of using a JSON collection in an input. Of course, it is more likely that I had just buggered something in the input configuration but todays coincidence is curious.

Anyhow, looking forward to the next version.

DG

Re: lost geoevent services on server reboot

DennisGeasan — Tue, 17 Dec 2013 18:22:43 GMT

GeoEvent Processor has a default 2GB limit for RAM allocation. You can configure this for your instance. Locate the product installation folder �?�\ArcGIS\Server\GeoEventProcessor\etc on your system and open the ArcGISGeoEventProcessor.cfg file in a text editor. You should be able to locate the configuration setting toward the top of the file:

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

On my systems the process "D:\Program Files\ArcGIS\Server\GeoEventProcessor\jre\bin\java.exe" grows to nearly the max heap size. I'm reducing that max heap to 512MB as the server only has 4GB. Any issues with using a smaller max heap?
DG

Re: lost geoevent services on server reboot

RJSunderman — Wed, 18 Dec 2013 16:22:42 GMT

Hello Dennis -

I'd like to provide you some under-the-hood details which might play into the question we're now discussing. Internally, the GeoEvent Processor is using ActiveMQ to manage the event queues being sent to each node in a GeoEvent Service. Esri Germany reported that they observed a large number of files being written to a tmp_storage folder beneath the ...\data\activemq folder in the product installation directory. This was a disk space consumption issue for them.

We recommended to the Esri Germany team that they increase the wrapper.java.maxmemory setting. Our understanding is that they are feeding a very high volume of events into GeoEvent Processor, which is caching events to disk when the service components are unable to keep up. Our recommendations for system resources are a minimum 6GB of RAM for GEP (vs. the 4GB minimum recommendation for ArcGIS Server). The default sizes for the queues used by the Event Processing framework can also be adjusted; you would only need to consider that if you were working with high volumes of event data. To adjust the queue sizes edit the com.esri.ges.messaging.jms.cfg and increase the defaults to reflect the following:

com.esri.ges.messaging.jms.destinationPolicy.queue.memoryLimit (10 megs)
com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit (10 megs)
com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit (1 gig)

You will need to stop GeoEvent Processor and edit these files as an administrator (assuming they are beneath C:\Program Files\...) then restart GeoEvent Processor in order for the settings to take effect.

If you've observed the java.exe process which is running with the -Dkaraf.base="C:\<install folder>\GeoEventProcessor" -Dkaraf.data="C:\<install folder>\GeoEventProcessor\data" arguments in its command line actually consuming more than 1024 megs of memory, lowering the wrapper.java.maxmemory value to 512 megs as you suggest will only result in Java throwing out-of-memory exceptions. How many events are you sending into GeoEvent Processor? What inputs are you using? How many different GeoEvent Services do you have running? I think we need to examine the load you are placing on your Server, by GeoEvent Processing as well as any other running services/applications, before we recommend throttling the memory allowed to the Java process.

- RJ

Re: lost geoevent services on server reboot

DennisGeasan — Wed, 18 Dec 2013 17:42:36 GMT

Hello RJ and thanks for the reply.

I am running 5 Generic JSON inputs. The URL of each input retrieves a JSON collection. Depending on the URL the number of collection items ranges from 35 to about 350. Is each collection item treated as an event? If so then that might explain some other problems I'm encountering based on what you explained.

Each input runs through a field mapper processer that is then directed to an "Update Feature" output.

So 5 inputs, 5 Geoevent Services, and 5 outputs. Each "Update Feature" output goes to a separate AGS feature service.

For all outputs I am capturing the input attribute "$RECEIVED_TIME" so I can determine if an update has actually occurred.

For the input that retrieves 35 items, all records in the target feature class get updated on nearly every run of the input.

The other 4 inputs all retrieve more than 100 items. Usually, not all of the records in the target feature class get updated after an input has run. In fact after about 12 hours, no further updates occur to the target feature classes with inputs items > 100.

The JSON collection from each input represent the locations and some attributes of vehicles at a business site. The intention is to display these on a web map.

Initially I wrote inputs for each vehicle but considering some sites have close to 350 vehicles it is more efficient to make one REST call and get all vehicles in one input.

So the question is, am I attempting to do something GEP is not intended for? If GEP is able to handle this type of input, how do I modify stuff so that the target feature classes are consistently updated.

BTW - the 'java.exe' process has remained at about 600MB since I made the change to max heap size but now one of the ArcSOC.exe services is consuming 1.4GB memory.

Dennis Geasan
GIS Technologies

Re: lost geoevent services on server reboot

DennisGeasan — Wed, 18 Dec 2013 23:30:30 GMT

com.esri.ges.messaging.jms.destinationPolicy.queue.memoryLimit (10 megs)

com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit (10 megs)

com.esri.ges.messaging.jms.destinationPolicy.topic.memoryLimit (1 gig)

The second item is listed twice with different values. Which should it be? Is there a third item to adjust? Thanks.

Dennis Geasan

Re: lost geoevent services on server reboot

RJSunderman — Wed, 18 Dec 2013 23:46:20 GMT

Is each collection item treated as an event?

Yes. When a JSON structure is received and the node being used as the root is a list, GeoEvent Processor parses the list and sends each item in the list to the adapter as a separate GeoEvent.

5 inputs, 5 Geoevent Services, and 5 outputs. Each "Update Feature" output goes to a separate AGS feature service.

As a service layout, this is reasonable. What we have to manage is the total number of events being received and processed every second so as to not overwhelm the server. Updating a feature service is currently a bottleneck in the event processing workflow. GeoEvent Processor can generally handle on the order of 800 events per second without any processing or filtering being performed. When updating a feature service, the event traffic needs to be throttled back to 200 - 300 events per second (total, across all running GeoEvent Services). We are developing high capacity stream services with high availability and cluster processing for our next major product release - but that is not going to be publicly available until sometime mid- to late-2014. I've attached two files to this thread which you might find helpful for capacity planning and general product performance.

The other 4 inputs all retrieve more than 100 items. Usually, not all of the records in the target feature class get updated after an input has run. In fact after about 12 hours, no further updates occur to the target feature classes with inputs items > 100.

I don't believe that what the feature service does with the event data it receives will affect resource consumption. That is, it is just as expensive to query a feature service and update 50 features as it is to query the feature service and update zero features. As long as we're not making PUT and GET calls too frequently, on too large a feature dataset, GeoEvent Processor should take the JSON in, convert it to a GeoEvent, query the feature service and post the event data. It shouldn't matter that after a half-day all necessary features have been processed and the dataset is considered up-to-date. How many total features are in that dataset and how many events are being received a second is what matters.

BTW - the 'java.exe' process has remained at about 600MB since I made the change to max heap size but now one of the ArcSOC.exe services is consuming 1.4GB memory.

The ArcSOC.exe processes are part of the ArcGIS for Server product. If indeed the issue is with feature service update, it might make sense that those processes are consuming the lion share of the server's memory. I might recommend scaling back your solution to run one of the five GeoEvent Services you have for a period of time to monitor the system's resources and then slowly scale up to either add a second service or stop a "smaller" service to begin running one that handles more data. Running all five services when 80% of those are each expecting a few hundred events at a shot might just be overwhelming your server.

- RJ

Re: lost geoevent services on server reboot

RJSunderman — Wed, 18 Dec 2013 23:50:52 GMT

Apologies - forgot to attach the promised performance planning and capacity documents.

Re: lost geoevent services on server reboot

DennisGeasan — Thu, 19 Dec 2013 00:22:17 GMT

As long as we're not making PUT and GET calls too frequently, on too large a feature dataset, GeoEvent Processor should take the JSON in, convert it to a GeoEvent, query the feature service and post the event data. It shouldn't matter that after a half-day all necessary features have been processed and the dataset is considered up-to-date. How many total features are in that dataset and how many events are being received a second is what matters.

Attached is a screen capture of the GEP monitor with the 5 inputs and outputs. First page is the GE Services. Second page is Inputs(top) and Outputs(bottom). They are all less than 3/s. Is this the measure you are looking for? If not how do I get the number of events per second?

Each feature class has a different number of records but they range from 34 to about 333 records.

DG

Re: lost geoevent services on server reboot

AndreasEspersen — Thu, 13 Nov 2014 16:35:21 GMT

HI RJ,

Last night and today i have also experienced losing my geoevent services on 10.2.2 after restarting the windows service for GEP and had to set it up all from scratch again.

Is there a way to export geoevent services like you could with the old tool Service Designer and import them again afterwards. That would also be beneficial moving from test to production.

We have also experienced some weird behavior with our outputs that the feature service we are outputting to suddenly have changed after restart of GEP Windows Service

Regards

Andy

Re: lost geoevent services on server reboot

ThomasPaschke — Thu, 13 Nov 2014 22:06:50 GMT

Hi Andreas,

You can save the configuration of your GEP (including the service) by exporting a configuration xml file. This will save your inputs, outputs, services, GEDs, Tags, GeoFences & Data Stores.

To do so go to Site -> Configuration Store -> Export Configuration. To restore or migrate a configuration use the Import Configuration functionality on the same site. If you only want to import/export selected features the xml file can be edited.

Hope that helps.

Best,

Thomas

Re: lost geoevent services on server reboot

AndreasEspersen — Fri, 14 Nov 2014 06:58:51 GMT

Hi Thomas,

Much Appreciated

Do you know why GEP drops the geoevent services?

Regards

Andy

Re: lost geoevent services on server reboot

ThomasPaschke — Fri, 14 Nov 2014 17:21:05 GMT

Although we made a good effort to fix all the use cases, in rare cases the validation of the service can fail which is why the service is not showing up in the user interface. It should not be deleted and still be on disk.

Re: lost geoevent services on server reboot

BradleyHardesty — Fri, 19 Dec 2014 18:00:41 GMT

I have also experienced this loss of GEP services and have to re-import my site configuration every time my server computer is rebooted or the GEP extension is restarted in windows services. Looking for a resolution.

Re: lost geoevent services on server reboot

RJSunderman — Fri, 27 Feb 2015 19:31:54 GMT

The ArcGIS 10.3 GeoEvent Extension Patch 1 for Server, now available on the Esri Support site at the link below, includes a fix which I expect will address the issue you are describing.

http://support.esri.com/en/downloads/patches-servicepacks/view/productid/66/metaid/2181

Specifically, we discovered that GeoEvent Services, published in the 10.3 release, which included filters with spatial conditions (INSIDE, WITHIN, CROSSES, etc.) caused the GeoEvent Services to disappear when either the GeoEvent product's service was restarted, or the GIS Server was rebooted. The upcoming 10.3.1 release will also address the issues addressed in Patch 1 above.

- RJ