2021 Roblox outage

The image shown during site maintenance.

The 2021 Roblox outage was an incident that occurred on October 28, 2021, at 5:00 p.m. PST, or midnight UTC, when Roblox began to have major site-wide issues due to an external system glitch. Players were having issues with navigating the site and using Roblox. Most players were denied access to usually public experiences, and most pages would redirect to an Error 400 Bad Request.

18 hours after the malfunctioning started, Roblox closed the site for maintenance for the first time since 2019. The site was progressively reopened 63 hours after closing, with full access to the site restored after 72.75 hours, making this the longest outage in Roblox history.

False rumors claimed that the Chipotle Boorito Maze promotion or other traffic congestions caused the outage, but this was quickly denied by the official Roblox Twitter account and later by the postmortem blog post. The shutdown led to major internet coverage from Roblox users and non-users alike on YouTube, Twitter, Discord and other social media and news platforms.

Timeline

The Roblox Status page documenting the incident, seen on October 31, 2021, at 6:00 PDT

An error message seen on Windows 10 when trying to open Roblox Studio during the maintenance

Note: Hour 1 is 5 PM PST.

Before October 28, 2021

While the issues directly connected to the outage did not officially start until the 28th, there were some notable issues in the days before that some people consider the start of the outage:

October 25, 2021

• At 11 AM PST, many people start having problems with uploading clothing, audio assets, gamepasses, and models as well as issues with filing support tickets.

October 27, 2021

• At 7 PM PST, the Community Creations tab of the marketplace is reported by one user on the Developer Forum as broken.

October 28, 2021

• During the hours leading up to the event – Users report minor issues with their avatar, and some experiences' lagging more than usual.
• 1 hour before the outage – At around 4 PM PST, the bug begins to reach most servers. These servers begin to crash, and some users are unable to join games. Avatars in-game only appear as the default blocky avatar. Most people who are online on Roblox start to notice lag. By the end of the hour, servers begin to crash rapidly, starting the outage.
• Hour 1
  • It is reported that Roblox had lost 99% of its active users.
  • status.roblox.com reports an "Active Incident" and a "Service Disruption" across the website, mobile app, Xbox app, experiences, Roblox Studio, data stores, experience–joining, and avatar. They report that "many player experiences are impacted".
• Hour 2 (6 PM PST)
  • The website remains accessible, but the pages are completely dysfunctional. Assets on pages, such as avatars, thumbnails, and rating percentages do not load.
  • The glitch reaches users on the mobile app, who report being logged out of Roblox, which sparks concerns of their accounts possibly being compromised.
  • The chat filter is affected, causing chat messages to not load properly.
  • By the end of the hour, the last users go offline.
• Hour 3
  • For the first time since 2016, active-user-count websites report 0 Roblox users online.
  • Some people report that they are able to sporadically load into the website.^{⁠[citation needed]}

• Hour 6 (10 PM PST)
  • Roblox Studio becomes inaccessible to users.
  • Users report that the visit button is not visible on experiences' pages, and is replaced with “Unable to verify that you have access to this experience, please try again later”, and the website has become "impossible" to navigate.
  • It is reported that trying to view pages for experiences will almost always show a HTTP 400 "Bad Request" error.
  • status.roblox.com reports that they "identified the root cause as (an) internal system issue."
  • Mobile users are completely locked out of the app, as they aren't able to register or login. Some users thought this was their accounts getting banned.

October 29, 2021

• Hour 14 (7 AM PST)
  • Users report very limited access to Roblox and an ability to visit some experiences, despite the platform's fragile state.
  • status.roblox.com reports that they "are actively working on the incident", and that "some users started to have limited access."
• Hour 15 – @Roblox tweets the following: "We know you're having issues using Roblox right now. We're sorry and working hard to get things back to normal."
• Hour 17 (10 AM PST) - The Roblox website is officially closed for "maintenance". This is the first time since 2019 that the website has been fully locked.
• Hour 20 (1 PM PST) – @Roblox tweets: "Still making progress on today’s outage. We'll continue to keep you updated. Once again, we apologize for the delay. We know that this outage was not related to any specific experiences or partnerships on the platform", clearing up an initial rumor that an event sponsored by Chipotle contributed to the early stages of the outage.
• Hour 21: – status.roblox.com updates to say that "the team continues active work on full site recovery".
• Hour 22 – Roblox is #38 in Twitter's worldwide trending hashtags due to the outage.

October 30, 2021

• Hour 46 (3 PM PST)
  • @Roblox tweets: "We [believe we] have identified an underlying internal cause of the outage. We're in the process of performing the necessary engineering and maintenance work to get Roblox back up and running ASAP. Thanks for your patience." The tweet was deleted shortly after.
  • status.roblox.com reports "We believe we have identified an underlying internal cause of the outage. We're in the process of performing the necessary engineering and maintenance work to get Roblox back up and running. Thanks for your patience."

October 31, 2021

• Hour 60 (5 AM PST) – Roblox breaks its previous outage record of just under 60 hours (August 2016), marking its longest-ever outage in its history.
• Hour 62 (7 AM PST) –
  • status.roblox.com updates to say "We believe we have identified an underlying internal cause of the outage. We're in the process of performing the necessary engineering and maintenance work to get Roblox back up and running. Thanks for your patience".
  • The first users report the ability to open the website.
• Hour 64 – @Roblox tweets "We have identified root cause and solution. We are working on getting things back online and we will keep you updated throughout the day". status.roblox.com reports the same thing a minute later.
• Hour 65 (10 AM PST) – The ability to open the site started to be rolled out worldwide. For users able to access the website, a banner is placed stating the following: "We are incrementally opening to groups of users and will continue rolling out."
• Hour 68 (1 PM PST)
  • status.roblox.com announces that "Traffic is being allowed incrementally. Some, but not all players will have access".
  • 8 minutes later, @Roblox tweets: "Quick update as we work to get things back to normal. We are incrementally bringing regions back online".
• Hour 72 –
  • @Roblox tweets: "Roblox is back online everywhere! Thank you for your continued patience as we get back to normal."
  • 15 minutes before the start of the 73rd hour, status.roblox.com updates to report that "Normal operations are restored", as the website's banner updates to say: "Roblox is back online! Thank you for your patience as we get back to normal.", thus bringing an end to the outage for all users.

Aftermath

Refund sent to a developer for a scheduled advertisement after the initial outage.

Message sent to a developer reporting on compensation issued by Roblox for lost earnings during the outage.

A few hours after the platform was put back up online and was fully functional, Roblox CEO David Baszucki issued an apology on the Roblox Blog about the events that occurred during the past 73 hours. At the end of this blog, Baszucki stated that Roblox would take action on preventing another major site–wide failure like this in the future, as well as stating they will publish a post-mortem with more details about the cause of the outage once their analysis was complete. It was also stated on the Developer Forum that no user persistence data was lost and would issue refunds to developers for advertisements that were not able to run due to the outage. The outage also caused Roblox to lose more than $1.5 billion in market cap on November 1, 2021, a period in which would take in at least $15 million in revenue.Even 20 hours after the end of the outage, there were still some avatar shop API errors and close to zero discoverability for newly released accessories. This was only reported by a very small portion of users. It did not affect any experiences.

On November 3, 2021, Roblox began to refund ads that could not run because of the outage.

On November 8, 2021, Roblox announced that they would provide appropriate compensation to developers for lost revenue due to the outage by estimating "Robux earned from marketplace and in-experience transactions, and also Premium Payouts". A few hours later, Roblox began this process.

On January 20, 2022, the post-mortem blog post was published by Roblox staff, explaining that the outage was caused by a misconfiguration in a core resource management service running on the experience discovery and platform servers. Furthermore, the time required to diagnose the issue was prolonged due to Roblox's heavy reliance on the impacted systems. In the same blog post, Roblox explained that they have taken the necessary steps to prevent outages at this scale from happening again and intend to expand to multiple datacenters to further mitigate outages.

Criticism and rumors

Many users have speculated the outage was fueled by a recently–released Chipotle event where visitors could earn vouchers for free burritos. Multiple rumors spread through platforms such as Reddit and Twitter that Roblox was shutting down due to several reasons, including inappropriate content on their website and bankruptcy. During the early stages of the outage, people rumored that experience loading failures were due to experiences not properly saving, and that users should avoid joining any experience.

As the outage extended past the 24–hour mark, members of the community expressed frustration about possibly losing their login streaks that experiences often use to reward visitors. Developers and designers noted that they lost Robux to advertisements placed in the 24–hour period preceding the outage, due to the decreased site traffic, but were later refunded.

Many of the people were also involved with holding up protest signs about wanting to bring Roblox back online.

People have criticized the length of the outage and the incompetence of engineers to identify the problem for nearly 48 hours. Several influencers made mock videos about Roblox's official Twitter posts.