The performance of AEM on Java 7 suddenly decreases, and thread dumps show no evidence of a culprit thread; many threads are in RUNNABLE state and executing portion of codes that are known to not have any flaws.
The root cause of this issue is that Java 7 suffers from the CodeCache problem described in this Stack Overflow question and answer.
In summary, "In Java 7, once the CodeCache gets full, the JIT is stopped, then almost a third of the CodeCache area is flushed but then JIT is not re-enabled."
This scenario results in much slower execution of the Java code that was either evicted from the code cache or was never executed by JVM. Therefore some requests that hit only JIT compiled code will still respond with the usual response times whereas other requests will take much longer.
There are three possible solutions:
These arguments specify a code cache size of 90 MB (instead of the default 48 MB) and disable code cache flushing, so that if the code cache overflows, existing hot code is not running in interpreter mode (only new code is not compiled any more). Monitor the code cache size for some time (for example using jconsole) to ensure that the configured maximum value is not too low. Note that after you upgrade to Java 8, you must remove these settings.