A home timeline at 300,000 reads a second
Design a News Feed (Twitter Timeline)
Built from Twitter's own account of its home timeline: decide when the work of 'one post, many followers' happens, survive accounts with thirty million followers, and stop precomputing timelines for people who never come back.
Intermediate, about 45 minutes, 9 stages
The situation
On a social network, people post short messages and follow other accounts. The home timeline shows the latest posts from everyone you follow, newest first, and it is the first thing every app open loads.
At the scale Twitter described in 2012–13: about 150 million active users, around 400 million posts a day (roughly 5,000 a second, 12,000 or more during big live events), and about 300,000 home-timeline reads a second. Most accounts have a few hundred followers. A handful have tens of millions.
Raffi Krikorian, then running Twitter's timeline infrastructure, described how they built this in his talk "Timelines at Scale". This investigation walks through the same decisions.
What it has to do
Functional
- Post a message.
- Follow and unfollow accounts.
- Read the home timeline, newest first, with pagination.
- Delete a post, after which nobody sees it.
Non-functional
- The home timeline loads in tens of milliseconds.
- A new post appears in followers' timelines within a few seconds.
- One account's post never delays delivery for everyone else.
- Losing a cache node loses no posts.
Constraints and assumptions
- About 300,000 timeline reads a second at peak.
- About 5,000 posts a second on average, 12,000+ during big events.
- Some accounts have more than 30 million followers.
- Posts are stored durably in a sharded database; the social graph is a separate service.
- People mostly read the first page or two of their timeline.
- Keeping the latest 800 entries per user is enough; older pages can be slower.
- Post IDs are time-sortable 64-bit IDs, so sorting by ID is sorting by time.
Interview questions it prepares you for
- “Design Twitter.”
- “Design the Facebook or Instagram news feed.”
- “Fan-out on write or fan-out on read? Defend your choice.”
- “How do you handle a celebrity with 50 million followers in your feed design?”
Read and practise next
How Twitter (X) built it · Fan-out for home timelines, in their engineers' own words
Concepts to know first: Caching, Asynchronous processing.
Similar systems: Design a Notification System, Design Discord's Message Storage.