The site is overloaded by abnormal traffic.

Environment

CQ5.x, AEM 6.x, AEM Dispatcher

Cause

Potential causes could be any of the following:

  • Denial of Service attack
  • Search bot or scraping bot hitting expensive URLs
  • Extra traffic spike due to popular article, press release, etc.

Resolution

To debug such an issue, it is best to have proper logging enabled at the dispatcher level:

1. Enable logging of the X-Forwarded-For header:

In the Apache HTTP Server's access_log, add this to the httpd.conf file in the dispatcher servers:

LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

In Microsoft IIS add X-Forwarded-For in the "Advanced Logging" configuration.

That would log the correct end-user IP.

Use the output showing the IP and User-Agent to analyze whether it is a malicious attack and block the offending IPs if it makes sense.

2. Set dispatcher.log's log level to debug:

Apache:
In the httpd configuration files there should be a section for the dispatcher.  Set DispatcherLogLevel to 3:

DispatcherLogLevel 3

IIS:

Modify the dis_iis.ini and set the loglevel to 3:

loglevel=3

3. Review this documentation and webinar on dispatcher caching. Take steps to improve dispatcher caching, that helps avoid outages caused by traffic spikes:

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License  Twitter™ and Facebook posts are not covered under the terms of Creative Commons.

Legal Notices   |   Online Privacy Policy