Many custom applications call web services from AEM. These applications use Apache Commons HTTP Client or other libraries. When the back-end systems experience performance issues then AEM would experience slow response times. In addition, if too many threads hang, this can lead to slow JVM garbage collections, out of memory errors, OS thread exhaustion, etc.
When capturing thread dumps or heap dumps from AEM, you observe many threads waiting on the web service calls.
Below , is an example stack trace from a JVM thread dump. It was captured from an AEM instance with an application experiencing a badly performing back-end service.
The thread dump showed a few issues:
- The request thread below was stuck waiting on a web service that wasn't responding. No socket read timeout was set so the thread was waiting forever. See here for the solution.
- Hundreds of threads were waiting on the single request thread. This was because the thread pool was configured to be single threaded, queued first in, first out. Having so many pending threads caused high memory utilization. See here for the solution.
- The web service that was being called was taking too long to respond. This caused the aformentioned thread pile up.
Note the highlighted stack lines:
- In yellow, you can see that the custom application is using Spring Framework's RestTemplate library to do a web service call.
- In orange, you can see that Spring Framework uses Apache Commons HTTP Client for its web service calls.
- In red, you can see that the thread was stuck in SocketInputStream.socketRead which means it is waiting on the web service for a response.