Sources

Sources are the first stage of the Home Mixer pipeline, responsible for retrieving candidate posts from various backend services. The Home Mixer implements two primary sources: Thunder (in-network) and Phoenix (out-of-network).

Overview

Sources implement the Source trait which provides an asynchronous get_candidates method that returns a list of PostCandidate objects. Each source can optionally implement an enable method to conditionally activate based on query parameters.

ThunderSource

The Thunder source retrieves in-network posts from users that the viewer follows. It queries the Thunder service’s AMP cluster to fetch posts from the user’s following list.

Implementation

pub struct ThunderSource {
    pub thunder_client: Arc<ThunderClient>,
}

Configuration

Arc<ThunderClient>

required

Client for communicating with the Thunder service clusters

ThunderCluster

default:"Amp"

The Thunder cluster to query (defaults to Amp cluster)

usize

Maximum number of posts to retrieve from Thunder service (configured via THUNDER_MAX_RESULTS)

Request Parameters

When querying Thunder, the source constructs a request with:

user_id: The viewer’s user ID
following_user_ids: List of user IDs the viewer follows
max_results: Maximum posts to retrieve
algorithm: Retrieval algorithm (defaults to “default”)
exclude_tweet_ids: Posts to exclude from results

Output Fields

i64

The unique identifier of the post

u64

The user ID of the post author

Option<u64>

If the post is a reply, the ID of the parent post

Vec<u64>

List of ancestor post IDs in the conversation thread. Includes the immediate parent and conversation root if different.

ServedType

Set to ForYouInNetwork to indicate this candidate came from the user’s network

Code Example

home-mixer/sources/thunder_source.rs

let request = GetInNetworkPostsRequest {
    user_id: query.user_id as u64,
    following_user_ids: following_list.iter().map(|&id| id as u64).collect(),
    max_results: p::THUNDER_MAX_RESULTS,
    exclude_tweet_ids: vec![],
    algorithm: "default".to_string(),
    debug: false,
    is_video_request: false,
};

let response = client
    .get_in_network_posts(request)
    .await
    .map_err(|e| format!("ThunderSource: {}", e))?;

let candidates: Vec<PostCandidate> = response
    .into_inner()
    .posts
    .into_iter()
    .map(|post| PostCandidate {
        tweet_id: post.post_id,
        author_id: post.author_id as u64,
        in_reply_to_tweet_id: post.in_reply_to_post_id,
        ancestors,
        served_type: Some(pb::ServedType::ForYouInNetwork),
        ..Default::default()
    })
    .collect();

PhoenixSource

The Phoenix source retrieves out-of-network posts using the Phoenix retrieval service. It leverages the user’s action sequence to find relevant posts from outside their following graph.

Implementation

pub struct PhoenixSource {
    pub phoenix_retrieval_client: Arc<dyn PhoenixRetrievalClient + Send + Sync>,
}

Configuration

Arc<dyn PhoenixRetrievalClient>

required

Client for communicating with the Phoenix retrieval service

usize

Maximum number of posts to retrieve (configured via PHOENIX_MAX_RESULTS)

Conditional Activation

Phoenix source includes an enable method that checks query parameters:

home-mixer/sources/phoenix_source.rs

fn enable(&self, query: &ScoredPostsQuery) -> bool {
    !query.in_network_only
}

The source is disabled when in_network_only is true, ensuring only Thunder results are used.

Requirements

Phoenix source requires the user_action_sequence to be present in the query. This sequence captures the user’s recent interactions and is used by Phoenix to understand user preferences.

Output Fields

i64

The unique identifier of the post

u64

The user ID of the post author

Option<u64>

If the post is a reply, the ID of the parent post

ServedType

Set to ForYouPhoenixRetrieval to indicate this candidate came from Phoenix

Code Example

home-mixer/sources/phoenix_source.rs

let sequence = query
    .user_action_sequence
    .as_ref()
    .ok_or_else(|| "PhoenixSource: missing user_action_sequence".to_string())?;

let response = self
    .phoenix_retrieval_client
    .retrieve(user_id, sequence.clone(), p::PHOENIX_MAX_RESULTS)
    .await
    .map_err(|e| format!("PhoenixSource: {}", e))?;

let candidates: Vec<PostCandidate> = response
    .top_k_candidates
    .into_iter()
    .flat_map(|scored_candidates| scored_candidates.candidates)
    .filter_map(|scored_candidate| scored_candidate.candidate)
    .map(|tweet_info| PostCandidate {
        tweet_id: tweet_info.tweet_id as i64,
        author_id: tweet_info.author_id,
        in_reply_to_tweet_id: Some(tweet_info.in_reply_to_tweet_id),
        served_type: Some(pb::ServedType::ForYouPhoenixRetrieval),
        ..Default::default()
    })
    .collect();

Source Pipeline Integration

Both sources are typically used together in the Phoenix candidate pipeline. Thunder provides in-network candidates while Phoenix provides out-of-network candidates, creating a diverse feed that balances familiar content with discovery.

Typical Flow

Thunder Source queries the user’s following graph
Phoenix Source (if enabled) retrieves out-of-network recommendations
Candidates from both sources are merged
Subsequent pipeline stages apply filters and scoring

Error Handling

Both sources return Result<Vec<PostCandidate>, String>, allowing the pipeline to handle failures gracefully. Common error scenarios include:

No available service channels
Network timeouts
Missing required query fields
Service-level errors

Filters - Filter candidates after source retrieval
Scorers - Score candidates for ranking
Phoenix Scorer - Scores Phoenix candidates using ML predictions

Candidate Pipeline Traits

Phoenix Models

Home Mixer Components

Overview

ThunderSource

Implementation

Configuration

Request Parameters

Output Fields

Code Example

PhoenixSource

Implementation

Configuration

Conditional Activation

Requirements

Output Fields

Code Example

Source Pipeline Integration

Typical Flow

Error Handling

​Overview

​ThunderSource

​Implementation

​Configuration

​Request Parameters

​Output Fields

​Code Example

​PhoenixSource

​Implementation

​Configuration

​Conditional Activation

​Requirements

​Output Fields

​Code Example

​Source Pipeline Integration

​Typical Flow

​Error Handling

​Related Components

Overview

ThunderSource

Implementation

Configuration

Request Parameters

Output Fields

Code Example

PhoenixSource

Implementation

Configuration

Conditional Activation

Requirements

Output Fields

Code Example

Source Pipeline Integration

Typical Flow

Error Handling

Related Components