The java application runs slower and finally runs out of memory or you see an error in the logs or in the console output
OutOfMemoryError: Java heap space or
OutOfMemoryError: gc overhead limit exceeded.
Such problems can have many causes.
One possible cause is that the java application, in our case, CRX / CQ was started from the command line with the default heap memory settings of Java. This means that the jvm parameter
-Xmx was not specified. CRX or CQ need at least 256 MB of heap allocated to run. If this is the problem, then starting from the command line,ensure that the heap memory settings are set. Example:
java -Xmx512m -jar *.jar
If this is not the case, then your application could be retaining too many objects without releasing them for Garbage Collection. This is called a memory leak, see here for more information. See the section below on how to analyze memory problems in java applications.
To automatically create a heap dump when running out of memory, you can add the jvm parameter
-XX:+HeapDumpOnOutOfMemoryError to generate a heap dump automatically when the application throws an OutOfMemoryError. For example,
java -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -jar *.jar
This generates a heap dump file (
java_...hprof) in the process's working directory whenever the java process runs out of memory. The process can continue to run after the heap dump was generated. Usually, one heap dump file is enough to analyze the problem.
Note: if you are using
crx-quickstart/server/start script to start your CRX instance then you can add
-XX:+HeapDumpOnOutOfMemoryError to the
CQ_JVM_OPTS variable (make sure that the variable is uncommented as well). For example:
After adding this parameter and restarting the CRX instance, verify that the new jvm option is set. Run
ps -ef | grep java from the command-line. Then check whether you see
-XX:+HeapDumpOnOutOfMemoryError as a parameter of the CRX java process.
If you need to specify a different directory to generate the heap dump in due to disk space restrictions, then you can add the
-XX:HeapDumpPath=/path/to/generate/heapdump parameter to tell the jvm where to put the file.
See here for a reference of debugging related jvm parameters.
To manually generate a heap dump, run this command (jmap and jps can be found in the
bin folder of your jdk home directory):
ps -ef | grep javaor
Ctrl+Shift+Escthen go to View => Select Columns => PID (Process Identifier) or
/path/to/generate/heapdumpfile.hprofwith the location you would like to generate the heap dump file, and replace
1234with the pid that you looked up in the previous step.
jmap -dump:format=b,file=/path/to/generate/heapdumpfile.hprof 1234
You'll first have to change the default JVM settings regarding the Dump agents, to generate the correct dumps on user signal. There are several kinds of dumps, but you generally need the full system dump to perform a thorough memory analysis. Add the following arguments:
This "user" event happening when the JVM receives the SIGQUIT (Linux, AIX®, z/OS®, and i5/OS™) or SIGBREAK (Windows) signal from the operating system.
For more information, see the vendor's documentation here.
Warning: Heap dump files are large and can be up to the same size on disk as your max heap -Xmx jvm parameter configuration. Make sure that you have enough disk space allocated to the directory where the dump file is generated.
A good tool to analyze heap dumps is EclipseMAT (Eclipse Memory Analyzer): http://www.eclipse.org/mat/
This tool can't analyze IBM JVM generated dumps. For those, there are several possibilities. IBM HeapAnalyzer will do well for the heap dumps in PHD or classic format.
For a complete system dump analysis, use the IBM Support Assistant Workbench , with IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer Version 1.2 installed on top.
The heap histogram is a simple measurement of the number of live objects and memory used per Java class. Unfortunately, depending on the Java installation, the required tools may not be available or may not always work. To create a heap histogram, you first need the process id of the Java process. To get it, run
ps or (if available), run:
This Java tool gets the process ids of all running Java processes. Example:
327 3332 sun.tools.jps.Jps 3313 crx-quickstart-....jar
Now run the following command:
jmap -histo 3313
The list is sorted by total memory required (shallow: excluding referenced objects). The first 20 lines of the output are the most interesting. Example output:
JVM version is 1.5.0_20-141 Iterating over heap. This may take a while... Warning: skipping invalid TLAB for thread t@62211 Warning: skipping invalid TLAB for thread t@62467 ... Size Count Class description ------------------------------------------------------- 10592904 12916 byte 10285840 75255 * ConstMethodKlass 6283176 58388 char 6042304 14928 int 4995752 116201 * SymbolKlass 4220896 75255 * MethodKlass 4196512 6969 * ConstantPoolKlass 2928560 6969 * InstanceKlassKlass 2631008 6066 * ConstantPoolCacheKlass 2395872 149742 org.apache.jackrabbit.core.query.lucene.DocId$PlainDocId 1476008 7003 java.util.HashMap$Entry 1396128 58172 java.lang.String 1070232 44593 java.util.HashMap$Entry 753984 10036 short 735464 54 org.apache.jackrabbit.core.query.lucene.DocId 720192 7502 java.lang.Class 640704 13348 com.day.crx.persistence.tar.index.IndexEntry ...
To help analyzing the problem, we also need to know the following information: