Error executing reactive work due to unable to create native thread: possibly out of memory or process/resource limits reached.

Question

Hello, everyone:

I keep reading data from a websocket interface, and after about 2,000,000 events recieved, We will have  an error in the log which was reported by "org.apache.camel.impl.engine.DefaultReactiveExecutor" :

Error executing reactive work due to unable to create native thread: possibly out of memory or process/resource limits reached. This exception is ignored. java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.lang.Thread.start0(Native Method)

I increased the memory limit in the GeoEvent/etc/ArcGISGeoEvent-wrapper.cfg

# Minimum and Maximum Java Heap Sizes
wrapper.java.additional.15=-Xms1g
wrapper.java.additional.16=-Xmx16g

the above error still exist.

is there any other settings I can modify to elimiate the above outofmemory error?

Thanks a lot.

RJSunderman · Answer

@JeffYang2023  -- I don't know if this will help. In an unrelated discussion looking at one of GeoEvent Server's processors which can potentially create a large number of threads, I was told that a user can only create so many threads. The commands below were run on a Mac, so I assume there are similar commands you can run within your Linux (?) environment.

ulimit -u
5568

sysctl kern.num_threads
kern.num_threads: 40960

sysctl kern.num_taskthreads
kern.num_taskthreads: 8192

Does this mean that a process owner on a given machine can only instantiate around 5500 threads, while the kernel can handle perhaps 41000 threads?  I'm not sure, but the limit for a number of "task" threads is much less, and all three are orders of magnitude less that the 2,000,000 event records you observe being ingest before seeing the DefaultReactiveExecutor engine log the exception.

I was pointed to this article on stackoverflow which suggests that you may have reached a limit on the number of open files, if the process thread(s) are consuming a large number of file descriptors / handles. the article suggests punching the ulimit up from 5k to 65k ... but I really don't know what the ramifications of that might be for overall system stability.

You also might take a look at this article from mastertheboss.com which walks through some suggestions for addressing the Java OutOfMemoryError.

* What input have you configured to receive the sensor data?  I'm assuming it is either the Receive JSON on a WebSocket or the Subscribe to an External WebSocket for JSON input?

* Is the data being ingest by the input in one extremely large batch?  Or is the data coming in as several batches with some period of time between batches (and it is not until you eventually reach 2 Million records you see the error)?

I'm wondering if the issue is that hundreds-of-thousands of data records received all at once as a single massive batch of data is causing the issue, versus a potential resource leak in the inbound connector you're using where you can receive 250 data records each second and it takes over 2 hours to receive a sufficient number of data records to trigger the exception.

===

I would advise that if you need to use the -Xms and -Xmx switches to increase the JVM RAM allocation that you set their values the same. This disables dynamic memory allocation. The way Java works, if you set a minimum of 1g and a maximum of 16g, when Java determines the JVM needs to be resized it will instantiate a new (larger) JVM and copy the running state into the new instance. This isn't as much of a problem if the system creates a 4g instance to copy over a currently running 2g instance (temporarily consuming 6 gigs). But if it were trying to dynamically scale 12g to 16g?  That's a lot of temporary memory being consumed to copy data from one JVM to another.

It is reportedly more stable, if you really need that much memory, to allocate a minimum (-Xms) to 16g and set the maximum (-Xmx) to 16g as well -- to prevent the dynamic resizing.

JeffYang2023 · Answer

full exception trace is here, thanks a lot:

Error executing reactive work due to unable to create native thread: possibly out of memory or process/resource limits reached. This exception is ignored. java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.lang.Thread.start0(Native Method) ~[?:?] at java.lang.Thread.start(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:?] at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?] at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:318) ~[?:?] at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:471) ~[?:?] at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:193) ~[?:?] at org.apache.camel.impl.engine.DefaultReactiveExecutor.schedule(DefaultReactiveExecutor.java:59) ~[?:?] at org.apache.camel.processor.MulticastProcessor.lambda$schedule$1(MulticastProcessor.java:336) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?]