Skip to main content

​
Overview

This guide will help you set up your development environment to work with X’s Recommendation Algorithm. The codebase is a complex, production-scale system written primarily in Scala, with components in Java, Python, and Rust.

​
Prerequisites

Before you begin, ensure you have the following installed:
1

Install Java Development Kit (JDK)

Install JDK 11 or later. The project uses Java 11 as the runtime platform.
# On Ubuntu/Debian
sudo apt-get update
sudo apt-get install openjdk-11-jdk

# On macOS with Homebrew
brew install openjdk@11
2

Install Bazel

The project uses Bazel as its build system. Install Bazel following the official installation guide.
# On Ubuntu/Debian
sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel

# On macOS with Homebrew
brew install bazel
Verify your installation:
bazel --version
3

Install Scala (Optional)

While Bazel manages Scala dependencies, you may want to install Scala for IDE support.
# On Ubuntu/Debian
sudo apt-get install scala

# On macOS with Homebrew
brew install scala
4

Install Python

Some components require Python 3.7 or later for machine learning models and scripts.
# On Ubuntu/Debian
sudo apt-get install python3 python3-pip

# On macOS with Homebrew
brew install [email protected]
5

Install Rust (for Navi)

The Navi model serving component is written in Rust.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

​
Clone the Repository

Clone the X Recommendation Algorithm repository: 
git clone https://github.com/twitter/the-algorithm.git
cd the-algorithm

​
Repository Structure

Understanding the repository structure will help you navigate the codebase:

​
Core Data Services

  • tweetypie/ - Core service for reading and writing post data
  • unified_user_actions/ - Real-time stream of user actions
  • user-signal-service/ - Centralized platform for user signals (likes, replies, clicks)

​
Models and Algorithms

  • src/scala/com/twitter/simclusters_v2/ - SimClusters community detection
  • trust_and_safety_models/ - NSFW and abusive content detection
  • src/scala/com/twitter/interaction_graph/ - Real-Graph user interaction predictions
  • src/scala/com/twitter/graph/batch/job/tweepcred/ - TweepCred reputation algorithm

​
Software Frameworks

  • navi/ - High-performance ML model serving (Rust)
  • product-mixer/ - Framework for building content feeds
  • timelines/data_processing/ml_util/aggregation_framework/ - Feature aggregation
  • representation-manager/ - Embedding retrieval service
  • twml/ - Legacy ML framework (TensorFlow v1)

​
Product Surfaces

  • home-mixer/ - Main service for Home Timeline (For You, Following, Lists)
  • pushservice/ - Recommendation service for notifications
  • tweet-mixer/ - Coordination layer for out-of-network tweet candidates
  • follow-recommendations-service/ - Account and post recommendations

​
Candidate Sources and Ranking

  • src/java/com/twitter/search/ - Search index for in-network posts
  • src/scala/com/twitter/recos/ - GraphJet-based recommendation services
  • timelineranker/ - Legacy ranking service
  • visibilitylib/ - Content filtering and compliance

​
Source Code Organization

  • src/java/ - Java implementations
  • src/scala/ - Scala implementations
  • src/python/ - Python ML models and scripts
  • src/thrift/ - Thrift service definitions

​
IDE Setup

1

Install IntelliJ IDEA

Download and install IntelliJ IDEA (Community or Ultimate edition).
2

Install Plugins

Install the following plugins:
  • Scala
  • Bazel (by Google)
3

Import Project

  1. Open IntelliJ IDEA
  2. Select “Open” and choose the repository root directory
  3. When prompted, import as a Bazel project
  4. Configure the project SDK to use Java 11

​
Visual Studio Code

1

Install VS Code

Download and install Visual Studio Code.
2

Install Extensions

Install the following extensions:
  • Scala (Metals)
  • Bazel
  • Python
  • rust-analyzer (for Navi)
3

Open Project

Open the repository root directory in VS Code. Metals will automatically detect the Scala code and provide IDE features.

​
Verify Your Setup

Test your development environment by checking if you can query the build targets: 
# List all build targets
bazel query //...

# Check a specific component
bazel query //home-mixer/...
If these commands run without errors, your development environment is ready!

​
Next Steps

Build docs developers (and LLMs) love

Get started for freeTalk to us