Gmail outage: An Investigation

Delicious Bookmark this on Delicious

By Ragu. Posted on 10 June 2011.

Google’s Gmail system had a loss of availability for 0.08% of users from all over the world on Feb 27. You can always go to Apps Status Dashboard to ensure there are no issues currently with Google Apps you are using.

The affected users had login errors and empty mail boxes. Google Engineers analyzed and identified the root cause was a bug injected from a recent software update for the storage before they started the restoration process.

The affected users’ accounts were blocked while they were working to avoid data integrity. So the affected users when they tried to log in had a message “Sorry, your account has been disabled”. The unaffected accounts by the software bug had been restored while the affected accounts had their other apps restored except Gmail. Google stores these data in multiple datacenters as well as in tapes. They retrieved the messages not available online from tape backup. Since they were restoring from the tapes, it took a longer time depending on the size of the messages. While this process, the messages sent were not delivered to the receivers.

The Google Apps and Gmail were restored 100% on Mar 3 and had a confirmation on the Google Help Forum. Google handled the situation properly by posting the updates on Google Blog which they created particularly to serve this issue.

As a result of this outage, they want to develop and expand testing tools to identify this kind of bug, develop tools to alert to stop the propagation of the issues, automate the restoration process, develop tools for users to use while there is a breakdown, implement tools to update the users while disruption.

For more information, please go to Google Apps Incident Report.

HTML Comment Box is loading comments...