Storing trillions of chat messages
Design Discord's Message Storage
Built from what Discord's engineers published about storing billions, then trillions, of messages: choose a partition key that keeps every read small, survive a channel full of deletions and a channel everyone opens at once, then move the whole thing to a new database while it is running.
Advanced, about 45 minutes, 9 stages
The situation
A chat platform organises conversations into servers, and servers into channels. Opening a channel loads its latest 50 messages; scrolling up loads older pages; clicking a reply or a search result jumps straight to one message. Messages can be edited and deleted.
The platform stores about 120 million messages a day and the number is climbing fast. Reads and writes are roughly equal, and reads are scattered: most servers are small groups of friends, a few are public communities with hundreds of thousands of members. The current database is a single replica set whose data and indexes no longer fit in memory, and read latency has become unpredictable.
Discord described this exact situation in 2017, and what happened over the five years after it in 2023. This investigation follows the decisions they made and the failures they wrote about.
What it has to do
Functional
- Send a message to a channel.
- Load the latest page of messages in a channel, and page backwards through history.
- Jump to a specific message by its ID.
- Edit and delete messages.
Non-functional
- Loading a channel is fast whatever the channel's size or age (p99 well under 100 ms).
- An acknowledged message is never lost, even if a database node dies.
- Capacity grows by adding nodes, without manual resharding.
- One busy channel never slows down unrelated channels.
Constraints and assumptions
- About 120 million new messages a day, growing several-fold a year.
- Roughly equal reads and writes; reads are spread randomly across millions of channels.
- A small infrastructure team: operating the store must not need constant manual work.
- A message is about 1 KB including metadata.
- Message IDs are 64-bit Snowflakes: a millisecond timestamp, a worker number and a sequence, so they sort by time.
- New messages reach online members through a separate WebSocket gateway; this investigation is about storing and reading them.
Interview questions it prepares you for
- “Design Discord.”
- “Design the message storage for a chat app like Slack or WhatsApp.”
- “How would you store and paginate billions of chat messages?”
- “One key in your database is getting far more traffic than the rest. What do you do?”
Read and practise next
How Discord built it · Storing chat history forever, in their engineers' own words
Concepts to know first: Partitioning, Generating unique identifiers.
Similar systems: Design a Collaborative Editor (Google Docs), Design a Notification System.