Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/StarTrail-org/PixelRAG/llms.txt

Use this file to discover all available pages before exploring further.

Retrieving over images requires an embedding model that understands visual content — not just text. Standard text embedding models cannot represent a rendered table, a chart, or a diagram in a way that lets a search query like “benchmark results for GPT-4” find the right screenshot tile. PixelRAG uses Qwen3-VL-Embedding-2B, a vision-language model, to embed both document screenshots and text queries into a shared vector space where visual content is retrievable by meaning.

​
The embedding model

Base model

Qwen/Qwen3-VL-Embedding-2B — a 2-billion parameter vision-language embedding model. The model processes image inputs through a ViT vision encoder followed by a language model, and produces fixed-length embedding vectors via last-token pooling.

LoRA fine-tune

The base model is LoRA fine-tuned on screenshot data from Wikipedia using contrastive learning with hard negatives. The trained adapters are published at Chrisyichuan/wiki-screenshot-embedding-lora. The best checkpoint is lora_vit/ckpt200 (ViT LoRA at step 200, ~102 MB).
Key technical details:
  • Pooling: Last-token pooling over the language model’s last_hidden_state (post-RMSNorm). The index of the last non-padding token is computed from the attention mask.
  • Normalization: L2 normalization applied to the pooled vector before storage. All embeddings in the index and all query embeddings are L2 normalized, so inner product equals cosine similarity.
  • Precision: Embeddings are stored as float16 in .npz shards to reduce disk and memory footprint.
  • Patch alignment: Image dimensions are rounded to multiples of 28 pixels (one Qwen3-VL patch) before processing. Chunks shorter than 28px are merged into the previous chunk rather than embedded as-is.
from peft import PeftModel
from transformers import AutoModel

base = AutoModel.from_pretrained("Qwen/Qwen3-VL-Embedding-2B")

# Best checkpoint: ViT-LoRA, step 200
model = PeftModel.from_pretrained(
    base,
    "Chrisyichuan/wiki-screenshot-embedding-lora",
    subfolder="lora_vit/ckpt200",
)

​
Query encoding

The serve API accepts three forms of query input, all producing L2-normalized float32 vectors of the same dimension as the index.
A text query is encoded using the model’s chat template, with a system instruction prepended:
{
  "queries": [{"text": "What is the speed of light?"}],
  "n_docs": 5
}
The default instruction used by pixelrag serve is:
“Retrieve images or text relevant to the user’s query.”
You can override it per-request:
{
  "queries": [{"text": "benchmark comparison table"}],
  "instruction": "Find the screenshot that best answers this question.",
  "n_docs": 5
}
An image query passes a base64-encoded image alongside (or instead of) a text string. The image is decoded and passed to the processor together with the chat-template text prompt.
{
  "queries": [{
    "image": "<base64-encoded PNG or JPEG>",
    "text": "Find pages similar to this one"
  }],
  "n_docs": 5
}
Both data:image/png;base64,... data URLs and raw base64 strings are accepted.
If you have already computed query embeddings externally, you can pass them directly. All queries in the request must use the embedding field (mixing with text/image fields in one request is not allowed).The vectors must be L2 normalized — the server re-normalizes them before search to ensure alignment with the index.
{
  "queries": [{"embedding": [0.012, -0.034, ...]}],
  "n_docs": 5
}

​
Indexing internals

FAISS IndexIVFFlat

The default index is a FAISS IndexIVFFlat with inner product metric (METRIC_INNER_PRODUCT). Because all vectors are L2 normalized, inner product equals cosine similarity — no separate normalization step is needed at search time.
  • nlist — number of IVF clusters (default: 4096; auto-adjusted for small indexes)
  • nprobe — number of clusters to scan per query (default: 128; overridable per request via nprobe field)
  • K-means training uses up to 500K sampled vectors

Qdrant

The Qdrant backend stores vectors with COSINE distance (equivalent to inner product on normalized vectors) and supports:
  • Quantization (e.g. int8 scalar) to reduce memory usage
  • Disk-backed vectors for indexes that exceed VRAM
  • Payload filtering on tile_height and article_id
  • Append and recreate modes for incremental updates
Memory-mapped loading for large indexes: 
# Enable mmap for multi-100GB FAISS indexes
PIXELRAG_INDEX_MMAP=1 pixelrag serve --index-dir ./index --port 30001
Setting PIXELRAG_INDEX_MMAP=1 instructs the FAISS backend to memory-map the index file rather than loading it entirely into RAM. This is essential when serving the pre-built Wikipedia index (~217 GB for the base FAISS index).

​
LoRA fine-tuning results

The LoRA fine-tune applies LoRA adapters to both the ViT vision encoder and the language model (--lora-vit), trained with contrastive loss over hard negatives. Evaluation setup — miniv8 test set:
PropertyValue
Questions400 SimpleQA questions
Candidate tiles7,426 rendered Wikipedia screenshot tiles
MetricQA score (fraction of questions answered correctly by a VLM reader given the retrieved tile)
Gradergpt-4.1-2025-04-14 comparing reader answers to gold answers
Results:
ModelQA Score
Untrained base (Qwen/Qwen3-VL-Embedding-2B)~0.715–0.730
LoRA fine-tuned (lora_vit/ckpt200)≈ 0.785
The fine-tuned model was trained on Chrisyichuan/screenshot-training-natural-filtered-v2 — a dataset of 104K query–image pairs with 2 hard negatives each, generated via LLM-augmented query synthesis and hard-negative mining from Wikipedia screenshots.
The QA score peaks around step 150–250 of the training run and may decay slightly beyond that point due to overfitting. The published lora_vit/ckpt200 adapter is the best overall checkpoint from the original run.
Training configuration highlights: 
CUDA_VISIBLE_DEVICES=<GPU> uv run python train_contrastors.py \
    --data-split-dir "$DATA_ROOT/screenshot-training-natural-filtered-v2" \
    --text-warmup-steps 50 \
    --text-data-dir "$DATA_ROOT/text-qa-pair" \
    --max-steps 350 \
    --batch-size 64 \
    --num-hard-negatives 2 \
    --lr 7e-6 \
    --lora-vit \
    --output-dir ./output
The --lora-vit flag — applying LoRA to the ViT vision encoder in addition to the language model — is the single largest contributor to the QA score improvement over the base model. For the complete training recipe, environment setup, dataset download instructions, and a step-by-step guide to reproducing the run, see the Training guide.

Build docs developers (and LLMs) love

Get started for freeTalk to us