temp_storage issue in GeoEvent 10.2.2

arunepuri1 · ‎08-06-2015

Hi All,

I am using GeoEvent Processor 10.2.2 and the temp_storage (\Program Files\ArcGIS\Server\GeoEventProcessor\data\activemq\localhost\localhost\tmp_storage) is increasing daily by10 GB.

As of now only two GEP Services with interval 1 Second.

1. What is the data it's storing in this folder.

2. What is the way to automatically clean then,

3. What is the way to clean the logs by it's date or size(GEP logs setting the option is not there).

4. How make GEP server High available(Data Center and Data Recovery) system without manual intervention.

Thanks,

Arun Epuri

RJSunderman · ‎08-17-2015

Hello Arun -

The issue you’re seeing, in which files are written to the following system folder, is a known 10.2.2 behavior.

…\data\activemq\localhost\localhost\tmp_storage

When a GeoEvent Service's inbound connector receives more event data than the GeoEvent Service is able to process the unprocessed events are held in memory, until the product's cache is full. Unprocessed events then get written out to disk. ActiveMQ was not doing a very good job of cleaning up event data it had persisted to disk … which is why you are seeing files begin to pile-up in the tmp_storage folder.

There is not much you can do at the 10.2.2 release to address this. You can edit the com.esri.ges.messaging.jms.cfg file, found in the GeoEventProcessor\etc folder) to increase the memory limits so GEP will hold more messages in memory, so that not so many files are written to disk. I've attached a screenshot showing the properties ... but these edits are really just masking the issue. If an event provider is consistently sending GeoEvent more event data than its GeoEvent Services can reasonably handle the event backlog will grow and eventually cause the system to fail.

If unprocessed events are being dumped to disk, you will likely see something in the karaf.log along the lines of: "org.apache.activemq.broker.region.Queue" … "Usage Manager Memory Limit reached.

Producer (ID:XXX) stopped to prevent flooding queue".

By the time I've seen messages like the above logged I've already pushed GeoEvent into a bad state. I've had to stop GeoEvent Windows Service, delete everything from beneath …\GeoEventProcessor\data (to forcefully reset the product) and then restart.

The better solution is to upgrade to the 10.3.1 product release. The product does a much better job of managing its event backlog at the 10.3 and 10.3.1 releases, working to reduce the backlog during periods in which event data is not being received.

Testing high volume / high velocity data for 10.3 we found that adding 4 - 8 GB of premium RAM to a system significantly increased event throughput. One of our holistic testers, in an Esri lab, managed to saturate the network card before depleting the system's RAM. (They ended up crashing the system, so remember that there are system resources in play other than RAM, DISK I/O, and CPU.)

Hope this information is helpful -

RJ

View solution in original post

RJSunderman · ‎08-17-2015

Hello Arun -

The issue you’re seeing, in which files are written to the following system folder, is a known 10.2.2 behavior.

…\data\activemq\localhost\localhost\tmp_storage

When a GeoEvent Service's inbound connector receives more event data than the GeoEvent Service is able to process the unprocessed events are held in memory, until the product's cache is full. Unprocessed events then get written out to disk. ActiveMQ was not doing a very good job of cleaning up event data it had persisted to disk … which is why you are seeing files begin to pile-up in the tmp_storage folder.

There is not much you can do at the 10.2.2 release to address this. You can edit the com.esri.ges.messaging.jms.cfg file, found in the GeoEventProcessor\etc folder) to increase the memory limits so GEP will hold more messages in memory, so that not so many files are written to disk. I've attached a screenshot showing the properties ... but these edits are really just masking the issue. If an event provider is consistently sending GeoEvent more event data than its GeoEvent Services can reasonably handle the event backlog will grow and eventually cause the system to fail.

If unprocessed events are being dumped to disk, you will likely see something in the karaf.log along the lines of: "org.apache.activemq.broker.region.Queue" … "Usage Manager Memory Limit reached.

Producer (ID:XXX) stopped to prevent flooding queue".

By the time I've seen messages like the above logged I've already pushed GeoEvent into a bad state. I've had to stop GeoEvent Windows Service, delete everything from beneath …\GeoEventProcessor\data (to forcefully reset the product) and then restart.

The better solution is to upgrade to the 10.3.1 product release. The product does a much better job of managing its event backlog at the 10.3 and 10.3.1 releases, working to reduce the backlog during periods in which event data is not being received.

Testing high volume / high velocity data for 10.3 we found that adding 4 - 8 GB of premium RAM to a system significantly increased event throughput. One of our holistic testers, in an Esri lab, managed to saturate the network card before depleting the system's RAM. (They ended up crashing the system, so remember that there are system resources in play other than RAM, DISK I/O, and CPU.)

Hope this information is helpful -

RJ

arunepuri1 · ‎08-19-2015

Thanks RJ for the detailed information.