System Design Problems
Design a News Feed System
A news feed system displays a personalized stream of posts from users a person follows. The core challenge is fan-out: a single post by a popular user must be delivered to millions of followers in near real-time.
- Fan-out on Write β Pre-compute feeds when a post is published (push model)
- Fan-out on Read β Compute feeds dynamically when a user loads their feed (pull model)
- Hybrid Approach β Combine both strategies based on follower count
The fundamental tension: pre-computing feeds is fast to read but expensive to write; computing on-the-fly is cheap to write but slow to read.
Requirements
Functional Requirements
- Users can create posts (text, images, videos)
- Users see a personalized feed of posts from people they follow
- Feed is sorted by time (or relevance/ranking)
- Users can like, comment, and share posts
- New posts appear in followers' feeds within seconds
- Support for trending topics and discovery
Non-Functional Requirements
- Latency: Feed loads in < 200ms
- Throughput: 500K feed reads/sec, 50K posts/sec
- Freshness: New posts visible within 5 seconds
- Consistency: Eventual consistency is acceptable
- Scalability: 500M users, 100M daily active users
The read-to-write ratio is extreme: 10:1 or higher. This drives the architecture toward read optimization and pre-computation.
Back-of-the-Envelope Estimation
Feed Traffic Estimation
- 100M DAU Γ 10 feed views/day = 1B feed reads/day
- QPS_read = 1B / 86400 β 11,600 QPS
- Peak read QPS (3x): ~35,000 QPS
- 50K posts/sec Γ avg 200 followers = 10M fan-outs/sec
- Storage per post: 500 bytes metadata + 1 KB media ref = 1.5 KB
- Daily new posts: 50K Γ 86400 = 4.3B posts/day (metadata: ~6.5 TB/day)
Feed Generation Strategies
DfFan-out on Write (Push)
When a user publishes a post, the system immediately writes the post reference to each follower's pre-computed feed list. Reads are fast because the feed is already assembled.
DfFan-out on Read (Pull)
When a user loads their feed, the system queries all the users they follow, fetches their recent posts, merges and ranks them, and returns the result. Writes are cheap but reads are expensive.
Hybrid Strategy
The hybrid approach handles the celebrity problem:
Celebrity Threshold
Here,
- =Follower count cutoff (typically 10K-50K)
- =Pre-compute feed for regular users
- =Compute on-the-fly for celebrities
Set the celebrity threshold dynamically. If a user is approaching the threshold, transition them from push to pull gradually. Monitor fan-out latency to auto-adjust.
Detailed Design
Data Models
// Post
{
post_id: "p_123",
user_id: "u_456",
content: "Hello world!",
media_urls: ["https://..."],
created_at: "2026-06-20T10:00:00Z",
like_count: 42,
comment_count: 5
}
// Feed (per user, stored in Redis sorted set)
// Key: feed:{user_id}
// Score: post timestamp
// Value: post_id
Feed Storage
Use Redis sorted sets for feed storage:
DfFeed Cache Structure
Each user's feed is a Redis sorted set where scores are post timestamps. This enables efficient time-ordered retrieval and automatic expiration of old posts.
ZADD feed:u_789 1687267200 p_123
ZADD feed:u_789 1687267100 p_122
ZREVRANGE feed:u_789 0 19 // Get 20 most recent posts
Ranking Algorithm
Feed items can be ranked by time or by relevance:
Feed Ranking Score
Here,
- =Time decay factor (newer = higher)
- =Likes, comments, shares normalized
- =User's interaction history with author
- alpha, eta, gamma=Weighting constants (tunable)
Ranking Score Calculation
For a post 2 hours old with 100 likes, 20 comments, and high affinity:
recency = e^(-2/24) β 0.92 engagement = log(100 + 20 Γ 5 + 1) / log(1000) β 0.85 affinity = 0.9 (frequent interaction)
score = 0.5 Γ 0.92 + 0.3 Γ 0.85 + 0.2 Γ 0.9 = 0.891
Real-time Updates
For real-time feed updates, use WebSockets or Server-Sent Events (SSE):
- User connects via WebSocket
- When a followed user posts, push the new post via WebSocket
- Client inserts the post at the top of the feed
- For batch updates, poll every 30 seconds
WebSocket connections are expensive to maintain. Use them selectively for highly active users and fall back to polling for others.
Scaling Considerations
Database Sharding
Partition the posts table by user_id hash:
shard = hash(user_id) % NUM_SHARDS
This co-locates all posts by the same user on the same shard, enabling efficient queries for "get all posts by user X."
Feed Cache Sizing
Feed Cache Memory
Here,
- =Active users with cached feeds
- =Posts per feed (typically 500)
- =Bytes per feed entry (typically 64 bytes)
Feed Cache Calculation
For 100M active users: memory = 100M Γ 500 Γ 64 bytes = 3.2 TB
Distribute across a Redis cluster of 50 nodes: ~64 GB per node.
Practice Exercises
-
Design: How would you implement a "For You" personalized feed that uses machine learning to rank posts based on user preferences? What features would you use?
-
Scale: If a celebrity with 50M followers publishes a post, estimate the time and resources needed to fan out the post to all followers using the push model.
-
Consistency: How would you handle the case where a user unfollows someone but still sees their posts in the feed? Design a feed invalidation strategy.
-
Trade-offs: Compare push, pull, and hybrid feed generation for a system with 100M users where 0.1% are celebrities (10M+ followers).
Key Takeaways:
- Fan-out on write provides fast reads but expensive writes; fan-out on read is the opposite
- The hybrid model (push for regular users, pull for celebrities) is the industry standard
- Redis sorted sets are ideal for feed storage with time-ordered retrieval
- Ranking algorithms combine recency, engagement, and affinity signals
- WebSocket connections enable real-time feed updates
What to Learn Next
-> Design Chat System Real-time messaging with WebSocket and presence tracking.
-> Event-Driven Architecture Event sourcing, CQRS, and asynchronous communication.
-> Caching Strategies Cache-aside, write-through, and distributed caching patterns.
-> Message Queues Kafka, RabbitMQ, and event-driven fan-out.
-> Design Recommendation System ML-based content ranking and personalization.
-> Data Replication Replication strategies for high-availability data access.