Documentation Index
Fetch the complete documentation index at: https://mintlify.com/avnlp/dspy-opt/llms.txt
Use this file to discover all available pages before exploring further.
The TriviaQA pipeline targets the TriviaQA benchmark, a reading comprehension dataset composed of question-answer-evidence triples written by trivia enthusiasts. Each question is paired with independently gathered evidence documents from Wikipedia and the web, covering a wide variety of content types — from music lyrics and airline reviews to historical date records and news articles. The pipeline captures this content diversity through a typed metadata schema that includes an enum-constrained content_type field and a numeric year field for temporal filtering.
Dataset
| Property | Value |
|---|
| HuggingFace ID | mandarjoshi/trivia_qa |
| Train split | train |
| Test split | validation |
| Weaviate collection | TriviaQA |
| Complexity type | Trivia, factoid QA |
Pipeline Class
The TriviaQARAG class is defined in triviaqa_rag_module.py and subclasses dspy.Module. It wires together the five shared utility stages and exposes a single forward() entry point that handles the full retrieve-then-read flow.
from dspy_opt.triviaqa.triviaqa_rag_module import TriviaQARAG
class TriviaQARAG(dspy.Module):
def __init__(
self,
query_rewriter: QueryRewriter,
sub_query_generator: SubQueryGenerator,
metadata_extractor: MetadataExtractor,
metadata_schema: Dict[str, Any],
weaviate_retriever: WeaviateRetriever,
embedding_model: SentenceTransformer,
top_k: int = 3,
): ...
def forward(self, question: str) -> dspy.Prediction:
"""Execute the complete RAG pipeline."""
forward() Return Fields
forward() returns a dspy.Prediction containing the following fields:
| Field | Description |
|---|
question | The original input question (passed through unchanged) |
rewritten_query | Search-optimized version of the question produced by QueryRewriter |
sub_queries | List of decomposed sub-queries from SubQueryGenerator |
retrieved_context | Deduplicated list of passages returned by WeaviateRetriever |
answer | Concise answer generated by dspy.ChainOfThought |
reasoning | Explanation of how the answer was derived |
content_type and a numeric year field, enabling precise filtering based on the content category and temporal scope of evidence documents.
| Field | Type | Description |
|---|
content_type | string (enum: review, lyrics, trivia, date_info, news) | Category of the content |
primary_entity | string | Main subject (airline, song, TV show, etc.) |
year | number | Publication or relevant year |
metadata_schema:
properties:
content_type:
type: "string"
enum: ["review", "lyrics", "trivia", "date_info", "news"]
description: "Category of the content"
primary_entity:
type: "string"
description: "Main subject (airline, song, TV show, etc.)"
year:
type: "number"
description: "Publication or relevant year"
The content_type enum restricts the metadata extractor to a fixed vocabulary of content categories. This makes the Weaviate filter highly predictable and prevents the LLM from hallucinating free-form type values that would fail to match indexed documents.
Models
| Role | Model |
|---|
| Answer LLM | groq/qwen3-32b |
| Extractor LLM | groq/llama-3.3-70b-versatile |
| Embedding | Qwen/Qwen3-Embedding-0.6B |
| Evaluator LLM | groq/qwen3-32b |
Scripts
| Script | Description |
|---|
triviaqa_indexing.py | Load dataset from HuggingFace, extract metadata, embed, and store in Weaviate |
triviaqa_rag_module.py | Pipeline class definition — imported by optimizer and evaluation scripts |
triviaqa_rag_mipro.py | Run MIPROv2 optimization |
triviaqa_rag_copro.py | Run COPRO optimization |
triviaqa_rag_bootstrap_few_shot.py | Run BootstrapFewShot optimization |
triviaqa_rag_simba.py | Run SIMBA optimization |
triviaqa_rag_gepa.py | Run GEPA optimization |
triviaqa_rag_evaluation.py | Evaluate the optimized pipeline with DeepEval metrics |
Configuration Files
| File | Description |
|---|
triviaqa_indexing_config.yml | Indexing parameters: embedding model, metadata schema, collection name |
triviaqa_rag_mipro_config.yml | MIPROv2 parameters: max_bootstrapped_demos, max_labeled_demos, auto |
triviaqa_rag_copro_config.yml | COPRO parameters: breadth, depth, init_temperature |
triviaqa_rag_bootstrap_few_shot_config.yml | BootstrapFewShot parameters: max_bootstrapped_demos, max_rounds |
triviaqa_rag_simba_config.yml | SIMBA parameters: bsize, num_candidates, max_steps, max_demos |
triviaqa_rag_gepa_config.yml | GEPA parameters: max_full_evals, reflection_minibatch_size, candidate_selection_strategy |
triviaqa_rag_evaluation_config.yml | Evaluation settings and DeepEval metric thresholds |
MIPROv2 Configuration
answer_llm:
model: "groq/qwen3-32b"
api_key_env: "GROQ_API_KEY"
extractor_llm:
model: "groq/llama-3.3-70b-versatile"
api_key_env: "GROQ_API_KEY"
embedding:
embedding_model: "Qwen/Qwen3-Embedding-0.6B"
tokenizer_kwargs:
padding_side: "left"
weaviate:
url_env: "WEAVIATE_URL"
api_key_env: "WEAVIATE_API_KEY"
collection_name: "TriviaQA"
top_k: 5
train_dataset:
name: "mandarjoshi/trivia_qa"
split: "train"
test_dataset:
name: "mandarjoshi/trivia_qa"
split: "validation"
optimizer:
max_bootstrapped_demos: 3
max_labeled_demos: 16
auto: "medium"
Running the Pipeline
All scripts must be run from the triviaqa/ directory so that relative config file paths resolve correctly.
# Index documents into Weaviate
cd src/dspy_opt/triviaqa
python triviaqa_indexing.py
# Run MIPROv2 optimization
cd src/dspy_opt/triviaqa
python triviaqa_rag_mipro.py
# Run SIMBA optimization
cd src/dspy_opt/triviaqa
python triviaqa_rag_simba.py
# Run GEPA optimization
cd src/dspy_opt/triviaqa
python triviaqa_rag_gepa.py
# Evaluate the optimized pipeline
cd src/dspy_opt/triviaqa
python triviaqa_rag_evaluation.py
Programmatic Usage
import dspy
from sentence_transformers import SentenceTransformer
from dspy_opt.triviaqa.triviaqa_rag_module import TriviaQARAG
from dspy_opt.utils.metadata_extractor import MetadataExtractor
from dspy_opt.utils.query_rewriter import QueryRewriter
from dspy_opt.utils.sub_query_generator import SubQueryGenerator
from dspy_opt.utils.weaviate_retriever import WeaviateRetriever
# Configure LLMs
answer_lm = dspy.LM("groq/qwen3-32b", api_key="your-groq-api-key")
extractor_lm = dspy.LM("groq/llama-3.3-70b-versatile", api_key="your-groq-api-key")
dspy.configure(lm=answer_lm)
# Initialize components
query_rewriter = QueryRewriter()
sub_query_generator = SubQueryGenerator()
metadata_extractor = MetadataExtractor(extractor_llm=extractor_lm)
embedding_model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
retriever = WeaviateRetriever(
weaviate_url="your-weaviate-url",
weaviate_api_key="your-weaviate-api-key",
collection_name="TriviaQA",
top_k=5,
)
metadata_schema = {
"properties": {
"content_type": {
"type": "string",
"enum": ["review", "lyrics", "trivia", "date_info", "news"],
"description": "Category of the content",
},
"primary_entity": {
"type": "string",
"description": "Main subject (airline, song, TV show, etc.)",
},
"year": {"type": "number", "description": "Publication or relevant year"},
}
}
# Build and run the pipeline
pipeline = TriviaQARAG(
query_rewriter=query_rewriter,
sub_query_generator=sub_query_generator,
metadata_extractor=metadata_extractor,
metadata_schema=metadata_schema,
weaviate_retriever=retriever,
embedding_model=embedding_model,
top_k=5,
)
result = pipeline("Which band released the album 'Dark Side of the Moon' in 1973?")
print(result.answer)
print(result.reasoning)