Last Monday, Sacramento reached a scalding high of 113 degrees Fahrenheit, smashing the previous daily record of 108. The same day, a Twitter data centre located in the California state capital failed, according to a report from CNN that cites a leaked internal company email.
“On September 5th, Twitter experienced the loss of its Sacramento (SMF) datacenter region due to extreme weather. The unprecedented event resulted in the total shutdown of physical equipment in SMF,” Carrie Fernandez, the social media company’s vice president of engineering, wrote in an internal message to Twitter engineers on Friday, according to CNN.
The server centre outage hasn’t impacted site users, instead it just made Twitter more vulnerable, by taking its back-up data storage offline. But the failure highlights an increasingly worrying vulnerability of tech companies’ in the climate change era. Servers need to stay cool to run properly, but maintaining their temperatures is a costly challenge, especially when they’re located in places like California’s Central Valley.
Other tech giants have also recently experienced similar outages. During a heatwave in the United Kingdom earlier this year, both Oracle and Google servers went down. And large scale crypto mining operations in Texas had to reduce operations in July to account for heatwaves and a stressed power grid.
Last week’s heatwave was one of the most extreme ever for the West. It brought triple-digit temperatures to almost all of California, and shattered nearly 1,000 heat records. The intense weather taxed the electric grid, and the state only narrowly avoided blackouts (mostly thanks to text blasts that got people to reduce their energy usage).
But while the electricity stayed on, Twitter’s servers didn’t. The outage put Twitter in a “non-redundant state,” Fernandez wrote in the staff memo, as reported by CNN.
The company has data centres in multiple locations around the country including Atlanta, Georgia and Portland, Oregon, according to the news outlet. Those centres store duplicate data by design, to avoid data loss and total site outages. But as of Friday, Fernandez warned, “if we lose one of those remaining data centres, we many not be able to serve traffic to all of Twitter’s users.”
In his whistleblower complaint, Peiter “Mudge” Zatko specifically referenced the vulnerability and “insufficient data centre redundancy,” of Twitter’s system. From CNN:
“Even a temporary but overlapping outage of a small number of data centres would likely result in the service [Twitter] going offline for weeks, months, or permanently,” according to Zatko’s whistleblower disclosure. (Twitter has criticised Zatko and broadly defended itself against the allegations, saying the disclosure paints a “false narrative” of the company.)
Since the Sacramento failure occurred on Monday and the staff memo was sent on Friday, there’ve been at least five days of server disruption. It’s unclear if the data centre has been brought back online in the interim, or if the the outage is ongoing.
Twitter declined to respond to any of Gizmodo’s questions regarding their data centres. Instead, in an email to Gizmodo, a company spokesperson wrote, “There have been no disruptions impacting the ability for people to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to ship updates and will continue working to provide a seamless Twitter experience.”