August 8 2014 update on email notification issue

As a result of the email issue on August 8 2014, the Adobe Social operations team started a thorough investigation of the entire mail infrastructure. This infrastructure includes several redundant machines from which mail gets routed in case any fail. When the influx of emails began on August 8, we found that the primary server had failed and restarted the mail routing process. As part of the investigation, we found that some of the redundant mail servers were in the same state and the operations team restarted them. This is why there were several more emails that were sent.

The goal of this work is to A) understand the state of this infrastructure, B) make sure that we know how it is functioning, and most importantly, C) set up alerts in case it fails again. As our operations team continued to go through the process to find any other issues, it is possible that more emails were sent.

We apologize for any inconvenience this has caused.