Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/facebookresearch/LoRe/llms.txt

Use this file to discover all available pages before exploring further.

LoRe’s approach to personalization is to give every user a compact fingerprint — a weight vector w_i over K shared reward directions — rather than a full separate model. Because w_i lives on the probability simplex and has only K parameters, it is cheap to learn, easy to interpret, and transferable to users the model has never seen before.

​
User representation

Each user i is represented by a vector w_i ∈ ℝ^K. After a softmax transformation, w_i lies on the K-dimensional probability simplex: all entries are positive and sum to 1. The reward for user i on a response with feature vector x is: 
r_i(x) = x · V · w_i
V (shape [features × K]) is shared across all users. Only w_i is user-specific, making the per-user parameter count equal to K regardless of the embedding dimension.

​
Joint learning of W and V

W (the matrix of all users’ weight vectors, shape [N × K]) and V are learned simultaneously from the complete set of users’ preference data during training. Both are nn.Parameter objects inside LoRe_regularized: 
self.W = nn.Parameter(torch.rand(num_classes, num_basis_vectors, device=device))
self.V = nn.Parameter(torch.randn(num_features, num_basis_vectors, device=device))
The alternating minimization loop updates W and V in separate steps each iteration, so each parameter set is optimized while the other is held fixed. See Low-rank reward modeling explained for the full training loop.

​
User splits: seen vs. unseen

LoRe experiments divide users into two groups:
SplitDescription
Seen users (train_workers)Appear in both train and test datasets. W is learned jointly for these users.
Unseen users (test_workers)Appear only at test time. Their w_i must be inferred from a small held-out sample.
This split tests two distinct capabilities:
1

Seen user, unseen prompts

The learned w_i for a seen user is applied to prompts that were not part of training. This evaluates how well the reward generalizes to new content for a known user.
2

Unseen user, unseen prompts

A brand new user provides a few preference examples. PersonalizeBatch adapts a fresh w_i using the frozen V from training. This evaluates few-shot transfer to new users.
In run_regularized, this distinction maps directly to two evaluation blocks: 
# Seen users: W_joint was learned jointly, evaluated on held-out prompts
accuracies_seen = eval_multiple(
    W_joint, [V_joint.detach() for _ in range(N)], test_features_sparse
)

# Unseen users: W_few_shot is adapted from V_joint using few-shot data
W_few_shot = learn_multiple_few_shot(
    train_features_unseen, V_joint.detach(),
    num_iterations=500, learning_rate=0.5
)
accuracies_unseen = eval_multiple(
    W_few_shot, [V_joint.detach() for _ in range(N_unseen)],
    test_features_sparse_unseen
)

​
The PersonalizeBatch class

PersonalizeBatch handles adaptation for unseen users. It takes the pretrained V as a fixed input and only optimizes per-user weight vectors w: 
class PersonalizeBatch(nn.Module):
    def __init__(self, num_classes, num_features, num_basis_vectors,
                 num_iterations, learning_rate):
        super(PersonalizeBatch, self).__init__()
        self.w = nn.ParameterList([
            nn.Parameter(torch.randn(num_basis_vectors))
            for _ in range(num_classes)
        ])
        self.optimizer = optim.Adam(self.parameters(), lr=learning_rate)

    def forward(self, X, V):
        nll = 0
        for i, x in enumerate(X):
            V_w = V @ F.softmax(self.w[i])      # project to user direction
            logits = x @ V_w / 100.0
            log_likelihood = torch.log(torch.sigmoid(logits))
            nll += (-log_likelihood.sum()) / len(x)
        return nll

    def train(self, X, V):
        for j in range(self.num_iterations):
            self.optimizer.zero_grad()
            loss = self.forward(X, V)
            loss.backward()
            self.optimizer.step()
        return [F.softmax(self.w[i]).detach() for i in range(len(X))]
V is never updated inside PersonalizeBatch — gradients only flow through self.w. This is what makes few-shot adaptation cheap: optimizing K parameters per user rather than the full [features × K] basis.

​
Generating synthetic user populations

For the PersonalLLM dataset, LoRe can generate synthetic user populations using a Dirichlet distribution. The Dirichlet naturally produces vectors on the simplex, matching the structure of softmax-normalized w_i: 
def generate_popupulation(alpha, N):
    return np.random.dirichlet(alpha, N)
  • alpha is the Dirichlet concentration parameter (a K-dimensional vector).
  • N is the number of synthetic users to generate.
  • Small alpha values (e.g. [0.1, 0.1, ...]) produce users who strongly prefer one basis direction.
  • Large alpha values (e.g. [10, 10, ...]) produce users with nearly uniform mixtures.
This population generation is used in PersonalLLM experiments to simulate diverse preference distributions before collecting real user data.
Conceptual layout of LoRe’s personalization structure:
Feature space (dim = 4096)
        │
        V  [4096 × K]          ← shared by all users, learned jointly
        │
        ├── w_1 [K]  → r_1(x) = x · V · w_1   (user 1)
        ├── w_2 [K]  → r_2(x) = x · V · w_2   (user 2)
        ├── ...
        └── w_N [K]  → r_N(x) = x · V · w_N   (user N)

Seen users:    w_i learned jointly with V during training
Unseen users:  w_i adapted via PersonalizeBatch, V fixed
The rank K determines how many independent reward axes exist in the population. Increasing K allows finer-grained personalization but requires more data per user to learn w_i reliably.

Build docs developers (and LLMs) love

Get started for freeTalk to us