IOI
As Antigone Davis was appearing on CNBC to defend Facebook against a whistleblower’s claims and the administration of research data suggesting Instagram may harm teens, Facebook went completely offline.
Just before noon ET, the outage began. It took nearly six hours to resolve. It’s the worst outage Facebook has experienced since it went offline for more than 24 hours in 2019, and it’s hurting smaller businesses and creators who depend on services such as Facebook for their income.
A configuration issue caused the outage on Monday evening, according to Facebook. Users’ data is not believed to have been impacted.
At 5:30PM ET, DNSchecker.org demonstrated that most ISP DNS servers were successfully connecting to Facebook.com after failing all tests for most of the day. Our experience with Facebook and Instagram was restored a few minutes later; however, the DNS fixes may have taken some time to spread throughout the internet.
A team of Facebook engineers was sent to the company’s US data centers to resolve the issue
“We are aware that some people have difficulty accessing our apps and products,” tweets Facebook communications exec Andy Stone. We’re working to restore services as quickly as possible.” Mike Schroepfer, about to step down next year as CTO, tweeted, “We’re experiencing networking issues and our teams are working as fast as possible to debug and restore.”
Nearly all of Facebook’s internal communication and work systems had been crippled by the outage. As reported, several employees communicated through their work-provided Outlook emails, even though they couldn’t receive email from external addresses. Any employee who needed to log in with their work email was blocked, but those who had been using Google Docs and Zoom before the outage could still use them.
Several engineers had been dispatched to Facebook’s data centers in the US to try and fix the problem, two sources familiar with the matter said. Therefore, the already longest outage in Facebook’s history could’ve extended further.
You can see the problems on Down Detector (or in your Twitter feed). Despite unclear reasons as to why so many people were unable to access the platforms, their DNS records suggested the problem being DNS, similar to last week’s Slack outage (DNS is always the culprit).
VP of Cloudflare Dane Knecht said that Facebook’s border gateway protocol routes are mysteriously no longer available. Some have speculated about hackers or an internal protest over the whistleblower report, but there is no evidence to suggest anything malicious behind it.
While Instagram.com flashed a 5xx Server Error message, the Facebook site just said that something went wrong. Oculus as well was affected. The browser worked, but social features and adding new games were not available. Users could load existing games and use the browser.
The six-hour outage on Monday has finally been explained by Facebook
Instagram, Messenger, Whatsapp, and Oculus VR were all brought offline by a configuration change to Facebook’s routers – not a hack or attempt to access user data. Facebook’s explanation doesn’t go into much detail, but it sounds like the machines in its data centers couldn’t talk to one another – the company says “this disruption in network traffic caused our data centers to stop communicating, resulting in a service disruption.”
Facebook’s CEO Mark Zuckerberg apologized Monday evening and said the platforms would be returning to live service shortly. “Sorry for the disruption today — I know how much you rely on our services to stay connected with the people you care about.”
On Monday, the company went down from approximately 11:40AM ET to 3:05PM ET. The site was down for over 24 hours, which was the worst outage since 2019. According to reports, employees weren’t able to communicate on the company message board, and some used Outlook email accounts provided by the company to do so.
Facebook’s sites could not be found because of a DNS routing issue which started from a routine BGP update gone wrong.
Facebook’s outage coincided with a planned congressional hearing for whistleblower Frances Haugen. Haugen, a former product manager on Facebook’s Civic Integrity group, provided a trove of internal documents to Wall Street Journal reporters. She told 60 Minutes on Sunday that Facebook “pays for its profits with our safety.”
