Designing a Feed System

1. What Is the Requirement?

As long as the user keeps swiping downward, new pieces of information keep appearing.

2. Why Build This Requirement?

3. Requirement Analysis

1. Roles in the system and what each role can do
   • Publisher: publish posts
   • Follower: view posts from followed publishers
2. What the flow looks like when each role performs an action
   • Flowchart
   • Sequence diagram

4. Design

4.1. Storage Design

Choose storage according to functional requirements, request volume, data volume, and storage model.

• Function
  • Redis: fast
  • Kafka: burst absorption, asynchronous processing, decoupling
• Request volume
  • Consumer-side traffic
• Data volume
  • Daily data volume = daily request count × bytes written per request
  • Retention period
• Data-model design
  • Publisher post list
  • Publisher follower list
  • Publishers followed by a follower
  • Posts from publishers followed by a follower

4.2. API Design

4.2.1. Publisher Publishes a Post

• Req
  • content
• Rsp
  • code
  • msg

4.2.2. Follower Reads Posts from Followed Publishers

• Req
  • page
  • page_size
• Rsp
  • code
  • msg
  • total_page
  • blog list
    • uid
    • content
    • createtime

4.3. Architecture Design

Consider security, high concurrency, high availability, and maintainability together.

1. Split into microservices
2. Draw the application architecture diagram
3. Draw the sequence diagram

4.3.1. Pull vs Push

1. Pull
   • When a follower reads, iterate over followed publishers, read their posts, and aggregate them.
   • This is also called read fan-out: one feed read expands into N reads (N is the number of followed publishers) plus one aggregation operation.
   • Advantage: no extra storage is wasted.
   • Disadvantage: reads are heavy; a large follow count is problematic.
   • Suitable when the follow count is small.
2. Push
   • When a publisher posts, iterate over followers and write the post into each follower’s inbox.
   • This is also called write fan-out: each post expands into M writes (M is the number of followers).
   • Advantage: reads are fast.
   • Disadvantage: data duplication; a huge follower count is problematic.
   • Suitable when the follower count is small.
3. Pull + Push
   • Use write fan-out when the follower count is small.
   • Use read fan-out for publishers with many followers; active followers can still use write fan-out.
   • Summary: one question is which publishers count as large accounts, for which follower count can be a signal. Another is which followers are active, for which recent login time can be a signal.

4.4. Code Design

• Class diagram
• Component diagram

5. Effort Estimation

• Estimate one API at roughly 0.5–2 days.

6. Development

7. Testing

Functional testing, unit testing, API testing, load testing, etc.

8. Operations

1. Deployment
   • Deployment diagram
   • Machine count: estimate overall request volume, estimate actual QPS, then divide by measured per-machine QPS
2. Organize logs and monitoring for critical paths

9. Summary

1. Compare solutions
2. Problems encountered and how they were solved
3. Design highlights
   • Why a component was introduced
4. Pain points and improvements
   • What happens when request volume and data volume grow by N times

10. References

• What Is a Feed?
• Read Fan-Out and Write Fan-Out