Documentation Index

Fetch the complete documentation index at: https://mintlify.com/twitter/the-algorithm/llms.txt

Use this file to discover all available pages before exploring further.

​

System architecture

X’s Recommendation Algorithm is built on a shared set of data sources, machine learning models, and software frameworks that power multiple product surfaces. This architecture enables code reuse, consistent quality, and rapid iteration across different recommendation experiences.

​

Architecture overview

Product surfaces at X are built on three core layers:

1. Data Layer - Real-time user actions, post metadata, and user signals
2. Model Layer - Graph embeddings, ranking models, and content understanding
3. Service Layer - Candidate generation, ranking, filtering, and serving

​

For You Timeline architecture

The diagram below illustrates how major services and jobs interconnect to construct a For You Timeline: The For You Timeline represents the most complex product surface, utilizing nearly all components in the recommendation system.

​

Data components

The data layer provides foundational signals and storage for the recommendation system.

​

Tweetypie

Tweetypie is the core tweet service that manages all tweet data operations. It provides:

• Tweet creation, reading, and mutation APIs
• Hydration of tweet metadata and features
• Denormalization of tweet data for efficient serving
• Caching and storage optimization

Reference: tweetypie/server/README.md

​

Unified User Actions

Provides a real-time stream of all user actions across X, including:

• Favorites, retweets, replies, quotes
• Follows, unfollows, mutes, blocks
• Clicks, video views, profile visits
• Notification opens and tab clicks

This stream feeds into multiple downstream systems for real-time personalization. Reference: unified_user_actions/README.md

​

User Signal Service

Centralizes retrieval of user signals used across recommendation systems:

• Explicit signals - Direct user actions (likes, follows, bookmarks)
• Implicit signals - Behavioral data (clicks, dwell time, video views)
• Aggregated and filtered for privacy and quality

Reference: user-signal-service/README.md

​

Model components

The model layer includes graph-based algorithms, embeddings, and neural networks for understanding users and content.

​

SimClusters

SimClusters is a general-purpose representation layer based on overlapping communities. It provides:

• KnownFor - Which communities a producer (account) is known for
• InterestedIn - Which communities a consumer (user) is interested in
• Tweet embeddings - Community representation of tweets based on favs
• Topic embeddings - Community representation of topics

SimClusters covers the top 20M producers and ~145K communities, enabling:

• Consumer-based tweet recommendations
• Producer-based tweet recommendations
• Tweet similarity calculations
• Topic-based content discovery

Reference: src/scala/com/twitter/simclusters_v2/README.md

​

TwHIN

Twitter Heterogeneous Information Network (TwHIN) provides dense graph embeddings learned from the full user-tweet interaction graph. Unlike SimClusters’ sparse community-based embeddings, TwHIN creates dense vector representations that capture fine-grained relationships.

​

Real Graph

Predicts the probability that one user will interact with another user, used for:

• Follow recommendations
• Out-of-network content discovery
• Social graph understanding

Reference: src/scala/com/twitter/interaction_graph/README.md

​

Software frameworks

The service layer provides frameworks for building, serving, and monitoring recommendation systems.

​

Product Mixer

Product Mixer is the core framework for building recommendation products. It provides:

• Pipelines - Structured execution flow (Product → Mixer → Candidate → Scoring)
• Components - Reusable building blocks for candidate sources, filters, scorers
• Composition - Mix heterogeneous content (tweets, ads, users)
• Monitoring - Built-in observability and debugging

All modern recommendation surfaces (For You, Following, Notifications) are built on Product Mixer. Reference: product-mixer/README.md

​

Navi

High-performance model serving infrastructure written in Rust:

• Serves TensorFlow, PyTorch, and ONNX models
• Optimized for low latency and high throughput
• Powers real-time ranking in the recommendation pipeline

Reference: navi/README.md

​

For You Timeline components

The For You Timeline uses specialized components for each stage of the recommendation pipeline.

​

Candidate sources

​

Search Index (Earlybird)

Earlybird is X’s real-time search engine, providing:

• Inverted index of recent tweets
• In-network tweet retrieval
• Light Ranker scoring for initial ranking
• Powers ~50% of For You Timeline content

Reference: src/java/com/twitter/search/README.md

​

User Tweet Entity Graph (UTEG)

Built on the GraphJet framework, UTEG maintains an in-memory graph of user-tweet interactions:

• Real-time updates from user actions
• Graph traversal for candidate generation
• Supports multiple edge types (favorite, retweet, reply)
• Enables collaborative filtering at scale

Reference: src/scala/com/twitter/recos/user_tweet_entity_graph/README.md

​

Ranking components

​

Heavy Ranker

The Heavy Ranker is a deep neural network that:

• Uses approximately 6,000 features per tweet
• Predicts multiple engagement types (like, retweet, reply, etc.)
• Multi-task learning to optimize for various objectives
• Primary determinant of final tweet ranking

​

Mixing and filtering

​

Home Mixer

Home Mixer orchestrates the entire For You Timeline construction:

1. Fetches candidates from multiple sources in parallel
2. Hydrates features for ranking
3. Applies Heavy Ranker scoring
4. Filters and applies heuristics (diversity, balance, feedback)
5. Mixes tweets with ads, who-to-follow modules, prompts
6. Adds product features (conversation modules, social context)

Reference: home-mixer/README.md

​

Visibility Filters

Ensures content safety and quality through:

• Hard filtering (blocked, muted authors)
• Legal compliance (DMCA, country-specific restrictions)
• NSFW content filtering based on user settings
• Abusive content detection
• Coarse-grained downranking for quality

Reference: visibilitylib/README.md

​

Recommended Notifications

Recommended Notifications use a similar but specialized architecture: Reference: pushservice/README.md

​

Data flow

The typical data flow through the system:

​

Scalability

The architecture handles massive scale:

• ~1 billion tweets evaluated down to thousands of candidates
• ~145K communities in SimClusters covering 20M producers
• Real-time updates to graphs and embeddings
• Billions of requests daily across product surfaces

​

Next steps