Analyze AEM Java thread dumps using IBM Thread Analyzer tool.
- Review the threads with stack depth of 10 lines or longer. Those are usually the threads of most interest. Take notes on threads that are of interest.
- Sort by thread "State"
- Scroll down to the "Runnable" threads. Runnable threads are the ones that were actively taking up CPU time when the thread dump was taken.
Note: When reviewing the "Runnable" threads, you can ignore threads listed in the Threads that can be ignored section on the bottom of this page.
- Find runnable threads that are part of the application, for example, background job threads or request threads (request threads have names like this 127.0.0.1  GET /content/sites/global/en/sitemap.static-delivery.httpd.html HTTP/1.1). Once you find them then click on them one by one.
- For each request thread, you can find out when the user's browser made the request to the server by looking at the timestamp in the thread name. For example, in the thread name above, the timestamp (in millisecond unix epoch format) is 1347028187737. We can convert that epoch number to a date / time using www.epochconverter.com. Each thread dump shows the date and time when it was taken. You can take the difference in time between the request time and the thread dump time to see how long a request has been active for.
- After reviewing the request threads, scroll through the other "Runnable" threads. Once you have found a "Runnable" thread of interest, then look at the middle panel, "Waiting threads". Threads listed there are waiting for the selected thread to release a monitor. If you do not see any waiting threads, then your selected thread could still be the owner of a Lock (see implementing classes of Lock for details). For example, with a ReentrantReadWriteLock you cannot tell which thread is the lock holder as locks implement multiple monitors internally. So you might have to look at the source code to match it up with a thread that could be the lock holder.
- If the thread had a lock or monitor that many other threads were waiting on then go through the rest of the thread dumps to see if you can find other threads that have the same issue. See if the same thread still exists in the other dumps (in IBM TDA you can select multiple thread dumps and click the "Compare Threads" button* to view a thread's state across multiple thread dumps.
- In this view you can see the thread across multiple thread dumps to see if it is a long running thread. Basically if the thread is in Runnable state across multiple dumps and has a long stack then that usually means it is a long running thread.
- If you didn't find much looking at the runnable threads then go back to the thread listing, select a thread dump then click on "Monitor Detail" button* on the top panel. IBM TDA will open a window showing a tree view of monitor owning threads and their waiting threads. Note: It might display some thread pool threads like the servlet engine thread pool monitor, idle threads could be ignored. You can usually tell a thread is an idle thread pool thread because most of the time they only have 10 stack lines or less.
- If you captured "top -H -b -n1 -p <javapid>" output in addition to thread dumps then you can cross reference the thread level CPU utilization. Open the top output and get the process id of the threads that are utilizing the CPU. Convert the process id to hexadecimal then search for that hexadecimal value in the corresponding thread dump file. The id should match the "nid" of one of the threads.
- If the matching thread utilizing the most CPU is the "VM Thread" or any "GC" threads then you might have a memory issue. Repeat the same exercise for more thread dumps and top output and if there is a pattern of these threads taking CPU time, you have a memory issue.
- If you have confirmed the memory issue, then capture a heap dump next time the problem occurs. See this article for more details on capturing and analyzing heap dumps.
- VM Thread: This is a VM system thread.
- Threads starting with GC task thread: These are garbage collection threads.
- Threads with names similar to -  in code at java.net.PlainSocketImpl.socketAccept(Native Method): These are threads from the servlet engine's thread pool waiting on new connections.