Symptoms

The java application runs slower and finally runs out of memory or you see an error in the logs or in the console output OutOfMemoryError: Java heap space or OutOfMemoryError: gc overhead limit exceeded.

Cause

Such problems can have many causes.

One possible cause is that the java application, in our case, CRX / CQ was started from the command line with the default heap memory settings of Java. This means that the jvm parameter -Xmx was not specified. CRX or CQ need at least 256 MB of heap allocated to run. If this is the problem, then starting from the command line,ensure that the heap memory settings are set. Example:

java -Xmx512m -jar *.jar

If this is not the case, then your application could be retaining too many objects without releasing them for Garbage Collection. This is called a memory leak, see here for more information. See the section below on how to analyze memory problems in java applications.

Analysis, solution

Create heap dump

Generate a heap dump automatically

To automatically create a heap dump when running out of memory, you can add the jvm parameter -XX:+HeapDumpOnOutOfMemoryError to generate a heap dump automatically when the application throws an OutOfMemoryError. For example,

java -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -jar *.jar

This generates a heap dump file (java_...hprof) in the process's working directory whenever the java process runs out of memory. The process can continue to run after the heap dump was generated. Usually, one heap dump file is enough to analyze the problem.

Note: if you are using crx-quickstart/server/start script to start your CRX instance then you can add -XX:+HeapDumpOnOutOfMemoryError to the CQ_JVM_OPTS variable (make sure that the variable is uncommented as well). For example:

CQ_JVM_OPTS='-XX:+HeapDumpOnOutOfMemoryError'

After adding this parameter and restarting the CRX instance, verify that the new jvm option is set. Run ps -ef | grep java from the command-line. Then check whether you see -XX:+HeapDumpOnOutOfMemoryError as a parameter of the CRX java process.

If you need to specify a different directory to generate the heap dump in due to disk space restrictions, then you can add the -XX:HeapDumpPath=/path/to/generate/heapdump parameter to tell the jvm where to put the file.

See here for a reference of debugging related jvm parameters.

Generate a heap dump manually

Sun/Oracle JVM

To manually generate a heap dump, run this command (jmap and jps can be found in the bin folder of your jdk home directory):

  1. look up the pid of the java process you are generating a heap dump for.
    • In Unix or Linux this can be done with ps -ef | grep java or jps -l
    • In Windows this can be done by opening the task manager, press Ctrl+Shift+Esc then go to View => Select Columns => PID (Process Identifier) or jps -l
  2. Run the jmap command below, replace /path/to/generate/heapdumpfile.hprof with the location you would like to generate the heap dump file, and replace 1234 with the pid that you looked up in the previous step.
    jmap -dump:format=b,file=/path/to/generate/heapdumpfile.hprof 1234
    

IBM JVM

You'll first have to change the default JVM settings regarding the Dump agents, to generate the correct dumps on user signal. There are several kinds of dumps, but you generally need the full system dump to perform a thorough memory analysis. Add the following arguments:

Xdump:heap:opts=PHD+CLASSIC:events=user -Xdump:system:events=user

This "user" event happening when the JVM receives the SIGQUIT (Linux, AIX®, z/OS®, and i5/OS™) or SIGBREAK (Windows) signal from the operating system.

For more information, see the vendor's documentation here.

Warning: Heap dump files are large and can be up to the same size on disk as your max heap -Xmx jvm parameter configuration. Make sure that you have enough disk space allocated to the directory where the dump file is generated.

Analyze heap dump

A good tool to analyze heap dumps is EclipseMAT (Eclipse Memory Analyzer): http://www.eclipse.org/mat/

This tool can't analyze IBM JVM generated dumps. For those, there are several possibilities. IBM HeapAnalyzer will do well for the heap dumps in PHD or classic format.

For a complete system dump analysis, use the IBM Support Assistant Workbench , with IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer Version 1.2 installed on top.

Heap histogram

The heap histogram is a simple measurement of the number of live objects and memory used per Java class. Unfortunately, depending on the Java installation, the required tools may not be available or may not always work. To create a heap histogram, you first need the process id of the Java process. To get it, run ps or (if available), run:

jps -l

This Java tool gets the process ids of all running Java processes. Example:

327 
3332 sun.tools.jps.Jps
3313 crx-quickstart-....jar

Now run the following command:

jmap -histo 3313

The list is sorted by total memory required (shallow: excluding referenced objects). The first 20 lines of the output are the most interesting. Example output:

JVM version is 1.5.0_20-141
Iterating over heap. This may take a while...
Warning: skipping invalid TLAB for thread t@62211
Warning: skipping invalid TLAB for thread t@62467
...
Size	Count	Class description
-------------------------------------------------------
10592904	12916	byte[]
10285840	75255	* ConstMethodKlass
6283176	58388	char[]
6042304	14928	int[]
4995752	116201	* SymbolKlass
4220896	75255	* MethodKlass
4196512	6969	* ConstantPoolKlass
2928560	6969	* InstanceKlassKlass
2631008	6066	* ConstantPoolCacheKlass
2395872	149742	org.apache.jackrabbit.core.query.lucene.DocId$PlainDocId
1476008	7003	java.util.HashMap$Entry[]
1396128	58172	java.lang.String
1070232	44593	java.util.HashMap$Entry
753984	10036	short[]
735464	54	org.apache.jackrabbit.core.query.lucene.DocId[]
720192	7502	java.lang.Class
640704	13348	com.day.crx.persistence.tar.index.IndexEntry
...

Additional information

To help analyzing the problem, we also need to know the following information:

  • CRX or CQ version, including a listing of all installed hot fixes version number.
  • Operating system, JVM vendor, and version.

References

[1] http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html
[2] http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp#DebuggingOptions

Za ta izdelek je bila izdana neprenosljiva licenca za priznanje avtorstva, nekomercialno uporabo in skupno rabo pod enakimi pogoji 3.0  Objav v storitvah Twitter™ in Facebook ne urejajo pogoji pogodbe Creative Commons.

Pravna obvestila   |   Pravilnik o zasebnosti v spletu