Is it safe to delete Opensearch dump files resulting from JVM errors in Data Store and Portal?

[StephenKaranja]

Hi everyone,

We're running ArcGIS Enterprise 11.5 in a single-machine deployment. The free space on the C: drive where ArcGIS Enterprise is installed is running low. Upon investigating, we've found out that it is Opensearch minidump files (.mdmp) that are taking up most of the space. These are in the following locations for Data Store and Portal:

C:\Program Files\ArcGIS\DataStore\framework\runtime\opensearch

C:\Program Files\ArcGIS\Portal\framework\runtime\ds\framework\runtime\opensearch

The dump files seem to result from a fatal error involving the JRE. An example of a log file that goes with the dump file is attached.

The top portion of the log file reads:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x00007ffafbb977fb, pid=10080, tid=12392
#
# JRE version: OpenJDK Runtime Environment Temurin-11.0.18+10 (11.0.18+10) (build 11.0.18+10)
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.18+10 (11.0.18+10, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V [jvm.dll+0x6777fb]
#
# Core dump will be written. Default location: C:\Program Files\ArcGIS\DataStore\framework\runtime\opensearch\hs_err_pid10080.mdmp
#
# If you would like to submit a bug report, please visit:
# https://github.com/adoptium/adoptium-support/issues
#

What we'd like to know is: Are these object dump files safe to delete to free up space?

Regards,

Stephen

[HenryLindemann]

Hi @StephenKaranja ,
Yes, you can delete the dump files, please make sure to only delete the .mdmp files as there are other configs in the file locations. If you can make a backup before making any system changes.

If you get a lot of these dumps then investigate your system something is wrong, like running out memory, or maybe even datastores automated backup running and causing an out of disk scenario.

The hs_err_pidXXXX.mdmp files are created when the Java Virtual Machine (JVM) running OpenSearch encounters a fatal error and crashes (e.g., out-of-memory, segmentation fault, EXCEPTION_IN_PAGE_ERROR, etc.).

Kind Regards
Henry

[StephenKaranja]

Hi @HenryLindemann,

Thanks a lot for your reply.

We opted to move some of the core dump files to another drive to create some more free space on C:. We had created a full backup of the system before this.

We investigated the system and think it is likely running out of memory that could be causing this. The memory usage is at about 100% and we think this causes Portal for ArcGIS to go down as well reporting a "no buffer space available" error. We had posted about this here.

We have requested the IT department to increase the RAM on the server machine to enable us monitor the issue and see whether it stops occurring. We'll keep you posted on the outcome of this.

Regards,

Stephen