By replacing those old components, they came back incorrectly with an error about being about consumption at zero. This outage would have happened earlier if the company had not kept it for a grace period. Unfortunately, that fix ended, and its automated systems began to behave as if the problem were real. Google had security guards in place to avoid such issues, but they were not created to handle the specific case that happened Monday morning.
“We apologize for the inconvenience this incident has had on our customers and their businesses,” Google said. “We take any event that seriously affects the availability and reliability of our customers, especially incidents made in multiple regions.”
While the company’s engineers were able to fix the problem relatively quickly, Google says it plans to implement new measures to prevent a similar situation in the future. In particular, one of his goals is to do a better job of communicating when an outage takes his services. It also plans to improve its monitoring systems so that it can catch incorrect configurations sooner.
