Roblox Wiki
Advertisement
Roblox Wiki

The image shown during site maintenance.

On October 28, 2021, at 4:45 PM PST/12:45 AM UTC, Roblox servers began to experience major, platform-wide system disruptions across the website and APIs, leading to website services like the Avatar Editor being unable to load, huge chat issues, and in-experience performance's becoming very slow. Later, many users were kicked from experiences, and, eventually, all users were. Roblox Studio also later began to function improperly due to these issues.

18 hours after the outage started, Roblox placed the site under maintenance for the first time since 2019. As Roblox incrementally reopened access to the website, the first users could join experiences 63 hours after the start of the outage, while others took longer to gain access. The website was fully reopened worldwide at 5:45 PM PST on October 31, 2021.

The outage lasted for 72 hours and 45 minutes, or just over 3 days, making it the longest outage in Roblox history.

It was widely believed to be caused by the Chipotle Boorito Maze event or other traffic congestions, but this was quickly denied by the Roblox Twitter account and later by the postmortem blog post.[1] The shutdown led to major internet coverage from Roblox users and non-users alike on YouTube, Twitter, Discord and other social media and news platforms.

Timeline

The Roblox Status page documenting the incident, seen on October 31, 2021, at 6:00 PDT

An error message seen on Windows 10 when trying to open Roblox Studio during the maintenance

Note: Hour 1 is 5 PM PST.

October 27, 2021

  • At 7 PM PST, the Community Creations tab of the Avatar Shop is reported by one user on the Developer Forum as broken, and even though the outage wouldn't begin until nearly 21 hours after this post was made, some have considered this to possibly be the start of the outage.[2]
Last_moments_of_roblox

Last moments of roblox

Footage of Roblox before it was put under maintenance

October 28, 2021

  • During the hours leading up to the event – Users report minor issues with their avatar, and some experiences' lagging more than usual.
  • 1 hour before the outage – At around 4 PM PST, the bug begins to reach most servers. These servers begin to crash, and some users are unable to join games. Most people who are online on Roblox start to notice lag. By the end of the hour, servers begin to crash rapidly.
  • Hour 1
    • It is reported that Roblox had lost 99% of its active users.
    • status.roblox.com reports an "Active Incident" and a "Service Disruption" across the website, mobile app, Xbox app, experiences, Roblox Studio, data stores, experience–joining, and avatar. They report that "many player experiences are impacted".
  • Hour 2 (6 PM PST)
    • The website remains accessible, but the pages are completely dysfunctional. Assets on pages, such as avatars, thumbnails, and rating percentages do not load.
    • The glitch reaches users on the mobile app, who report being logged out of Roblox, which sparks concerns of their accounts possibly being compromised.
    • The chat filter is affected, causing chat messages to not load properly.
    • By the end of the hour, the last users go offline.
  • Hour 3
    • For the first time since 2016, active-user-count websites report 0 Roblox users online.
    • Some people report that they are able to sporadically load into the website.[citation needed]
  • Hour 5 (10 PM PST)
    • Roblox Studio becomes inaccessible to users.
    • Users report that the visit button is not visible on experiences' pages, and is replaced with “Unable to verify that you have access to this experience, please try again later”, and the website has become "impossible" to navigate.
    • It is reported that trying to view pages for experiences will almost always show a HTTP 400 "Bad Request" error.
    • status.roblox.com reports that they "identified the root cause as (an) internal system issue."
    • Mobile users are completely locked out of the app, as they aren't able to register or login.

October 29, 2021

  • Hour 14 (7 AM PST)
    • Users report very limited access to Roblox and an ability to visit some experiences, despite the platform's fragile state.
    • status.roblox.com reports that they "are actively working on the incident", and that "some users started to have limited access."
  • Hour 15@Roblox tweets the following: "We know you're having issues using Roblox right now. We're sorry and working hard to get things back to normal."
  • Hour 17 (10 AM PST) - The Roblox website is officially closed for "maintenance". This is the first time since 2019 that the website has been fully locked.
  • Hour 20 (1 PM PST)@Roblox tweets: "Still making progress on today’s outage. We'll continue to keep you updated. Once again, we apologize for the delay. We know that this outage was not related to any specific experiences or partnerships on the platform", clearing up an initial rumor that an event sponsored by Chipotle contributed to the early stages of the outage.
  • Hour 21: – status.roblox.com updates to say that "the team continues active work on full site recovery".
  • Hour 22 – Roblox is #38 in Twitter's worldwide trending hashtags due to the outage.[3]

October 30, 2021

  • Hour 46 (3 PM PST)
    • @Roblox tweets: "We [believe we] have identified an underlying internal cause of the outage. We're in the process of performing the necessary engineering and maintenance work to get Roblox back up and running ASAP. Thanks for your patience." The tweet was deleted shortly after.
    • status.roblox.com reports "We believe we have identified an underlying internal cause of the outage. We're in the process of performing the necessary engineering and maintenance work to get Roblox back up and running. Thanks for your patience."

October 31, 2021

  • Hour 60 (5 AM PST) – Roblox breaks its previous outage record of just under 60 hours (August 2016), marking its longest-ever outage in its history.
  • Hour 62 (7 AM PST)
    • status.roblox.com updates to say "We believe we have identified an underlying internal cause of the outage. We're in the process of performing the necessary engineering and maintenance work to get Roblox back up and running. Thanks for your patience".
    • The first users report the ability to open the website.
  • Hour 64@Roblox tweets "We have identified root cause and solution. We are working on getting things back online and we will keep you updated throughout the day". status.roblox.com reports the same thing a minute later.
  • Hour 65 (10 AM PST) – The ability to open the site started to be rolled out worldwide. For users able to access the website, a banner is placed stating the following: "We are incrementally opening to groups of users and will continue rolling out."[4]
  • Hour 68 (1 PM PST)
    • status.roblox.com announces that "Traffic is being allowed incrementally. Some, but not all players will have access".
    • 8 minutes later, @Roblox tweets: "Quick update as we work to get things back to normal. We are incrementally bringing regions back online".
  • Hour 72
    • @Roblox tweets: "Roblox is back online everywhere! Thank you for your continued patience as we get back to normal."
    • 15 minutes before the 73rd hour, status.roblox.com updates to report that "Normal operations are restored".
    • The website's banner updates to say: "Roblox is back online! Thank you for your patience as we get back to normal."

Aftermath

A few hours after the platform was put back up online and was fully functional, Roblox CEO David Baszucki issued an apology[5] on the Roblox Blog about the events that occurred during the past 73 hours. At the end of this blog, Baszucki stated that Roblox would take action on preventing another major site–wide failure like this in the future, as well as stating they will publish a post-mortem with more details about the cause of the outage once their analysis was complete. It was also stated on the Developer Forum that no user persistence data was lost and would issue refunds to developers for advertisements that were not able to run due to the outage.[6] The outage also caused Roblox to lose more than $1.5 billion in market cap on November 1, 2021, a period in which would take in at least $15 million in revenue.[7]

Refund sent to the creator for their advertisement after the initial outage.

Even 20 hours after the end of the outage, there were still some Catalog API errors and they were currently close to zero discoverability for newly released accessories in the avatar shop. This only happened to a very small portion of users. They did not affect development/mechanics of experiences.[8]

On November 3, 2021, Roblox began to roll out refunds to developers after their advertisements could not run on the platform during the outage.[9]

On November 8, 2021, Roblox announced that they would provide appropriate compensation to developers for lost revenue due to the outage by estimating "Robux earned from marketplace and in-experience transactions, and also Premium Payouts". A few hours later, Roblox began this process.[10]

On January 20, 2022, a blog post was published by Roblox staff, explaining that the outage was caused by a misconfiguration in a core service responsible for server discovery and platform configuration. Furthermore, the time required to diagnose the issue was prolonged due to Roblox's heavy reliance on the impacted systems. In the same blog post, Roblox explained that they have taken the necessary steps to prevent outages at this scale from happening again and intend to expand to multiple datacenters to further mitigate outages.[11]

Criticism and rumors

Many users have speculated the outage was fueled by a recently–released Chipotle event where visitors could earn vouchers for free burritos.[12] Multiple rumors spread through platforms such as Reddit and Twitter that Roblox was shutting down due to several reasons, including inappropriate content on their website and bankruptcy.[13] During the early stages of the outage, people rumored that experience loading failures were due to experiences not properly saving, and that users should avoid joining any experience.[14]

As the outage extended past the 24–hour mark, members of the community expressed frustration about possibly losing their login streaks that experiences often use to reward visitors. Some people were very sad and angry that they lost months or even years of streak. Developers and designers noted that they lost Robux to advertisements placed in the 24–hour period preceding the outage, due to the decreased site traffic.

People have criticized the length of the outage and the inability of engineers to identify the problem for nearly 48 hours. Several influencers made mock videos about Roblox's official Twitter posts.[15]

References

Advertisement