Samantha Schaevitz was in the hometown of a community at Huridocs, a nonprofit of human rights, when she received the call. Schaevitz works on site-trusted engineering at Google; they are the ones who keep the ship stable when things get choppy. And by February of this year, when large parts of Asia shut down in an attempt to slow the spread of the novel coronavirus, Google Meet found itself on water. They needed Schaevitz back at work.
Google launched Meet in 2017 as a business-oriented alternative to its Hangouts chat service. (Google has gradually phased out Hangouts and sent users to Meet and Chat as part of the ever-muddled messaging platform strategy.) As the coronavirus spread and more countries issued orders to stay home, people flocked to video chat services for work and to check on family and friends. Google saw Meet undergo 30 times growth in the early months of the pandemic; soon enough, the service had a maximum of 100 million attendees at meetings each day. That’s a lot.
Amid all the profound changes that people have made in response to Covid-19, the infrastructure that the internet has undergirds has experienced a shift in usage patterns as people trade office hours for home insulation. The companies that handle these systems have mostly managed the new needs of users. “You essentially took the highlight and extended it over a much longer period of the day,” says Ben Treynor Sloss, Google Vice President of Engineering. “The use went up, but it was mostly that the use looked more like peak most of the day, instead of the peaks going up dramatically.” However, some services saw the use far above normal.
Google prepares them on a regular basis for emergencies through their disaster and invasion testing, or DIRT. In these exercises, about 10,000 workers will simultaneously simulate dealing with a type of crisis, ranging from a localized natural disaster to a Godzilla attack. However, the Candid-19 pandemic revealed that it surpassed even the most dramatic scenarios of the company.
“We typically had a simulated event at the regional level,” says Treynor Sloss. “We never did DIRT for a world-class event, in part, to be honest, because it does not seem likely.” There was also a practical concern: convincingly ridiculing an incident with global impact would downgrade the experiences of actual Google users, a cardinal sin in the world of DIRT.
That all meant that Schaevitz, who was leading the response to the incident for Google Meet, and the teams involved had to figure things out. Especially since it became clear that they were accepting far more new users than their most ambitious early projections.
“In the beginning, we started planning for a doubling of our footprint, which is already huge. That is not the normal growth curve. We soon realized that this would not be enough, ”says Schaevitz. “We were constantly trying to make progress in building more runways … so that we had time to find a solution if things were to come up over a longer time horizon, instead of just waking up every day and being “What’s up with the news?”
Complicating the challenge was that the Google engineers involved in the answer even worked from home, spread across four offices in three countries. “All the people who worked on it – and this is a large number of teams – even the people who worked on it in the same place, have never really been in the room since it started,” says Schaevitz, based in Zurich, Switzerland. At a technical level that proved provable enough; as you might imagine, Google prioritizes web-based tools that are accessible from anywhere. But coordinating the 24-hour-per-day operation remotely requires setting up redundancies for more than just bandwidth. In a blog post detailing the response, Schaevitz described how everyone in a role of responding to an incident got a “standby”, basically an understudy that could go into if the main character got sick or had time to leave. (A particularly precautionary measure during a global health crisis.)
.
