Documentation Index
Fetch the complete documentation index at: https://mintlify.com/avnlp/dspy-opt/llms.txt
Use this file to discover all available pages before exploring further.
The FreshQA pipeline targets the SealQA benchmark, a dynamic QA dataset that mixes standard single-hop factoid questions with questions that contain hidden false premises — requiring the model to both retrieve relevant evidence and actively debunk incorrect assumptions. The pipeline stores documents in a FreshQA Weaviate collection and uses lightweight title / category metadata to sharpen retrieval filters before hybrid search.
Dataset
| Property | Value |
|---|
| HuggingFace ID | vtllms/sealqa |
| Subset | longseal |
| Split | test |
| Weaviate collection | FreshQA |
| Complexity type | Single-hop, false-premise debunking |
Pipeline Class
The FreshQARAG class is defined in freshqa_rag_module.py and subclasses dspy.Module. It wires together all five shared utility components and exposes a single forward() entry point.
from dspy_opt.freshqa.freshqa_rag_module import FreshQARAG
class FreshQARAG(dspy.Module):
def __init__(
self,
query_rewriter: QueryRewriter,
sub_query_generator: SubQueryGenerator,
metadata_extractor: MetadataExtractor,
metadata_schema: Dict[str, Any],
weaviate_retriever: WeaviateRetriever,
embedding_model: SentenceTransformer,
top_k: int = 3,
): ...
def forward(self, question: str) -> dspy.Prediction:
"""Execute the complete RAG pipeline."""
forward() Return Fields
forward() returns a dspy.Prediction containing the following fields:
| Field | Description |
|---|
question | The original input question (passed through unchanged) |
rewritten_query | Search-optimized version of the question produced by QueryRewriter |
sub_queries | List of decomposed sub-queries from SubQueryGenerator |
retrieved_context | Deduplicated list of passages returned by WeaviateRetriever |
answer | Concise answer generated by dspy.ChainOfThought |
reasoning | Explanation of how the answer was derived |
MetadataExtractor parses each query against the following JSON schema to produce Weaviate filter values:
| Field | Type | Description |
|---|
title | string | The main title or name of the subject |
category | string | Primary category or type of content |
metadata_schema:
properties:
title:
type: "string"
description: "The main title or name of the subject"
category:
type: "string"
description: "Primary category or type of content"
Models
| Role | Model |
|---|
| Answer LLM | groq/qwen3-32b |
| Extractor LLM | groq/llama-3.3-70b-versatile |
| Embedding | Qwen/Qwen3-Embedding-0.6B |
| Evaluator LLM | groq/qwen3-32b |
Scripts
| Script | Description |
|---|
freshqa_indexing.py | Load dataset from HuggingFace, extract metadata, embed, and store in Weaviate |
freshqa_rag_module.py | Pipeline class definition — imported by optimizer and evaluation scripts |
freshqa_rag_mipro.py | Run MIPROv2 optimization |
freshqa_rag_copro.py | Run COPRO optimization |
freshqa_rag_bootstrap_few_shot.py | Run BootstrapFewShot optimization |
freshqa_rag_simba.py | Run SIMBA optimization |
freshqa_rag_gepa.py | Run GEPA optimization |
freshqa_rag_evaluation.py | Evaluate the optimized pipeline with DeepEval metrics |
Configuration Files
| File | Description |
|---|
freshqa_indexing_config.yml | Indexing parameters: embedding model, metadata schema, collection name |
freshqa_rag_mipro_config.yml | MIPROv2 parameters: max_bootstrapped_demos, max_labeled_demos, auto |
freshqa_rag_copro_config.yml | COPRO parameters: breadth, depth, init_temperature |
freshqa_rag_bootstrap_few_shot_config.yml | BootstrapFewShot parameters: max_bootstrapped_demos, max_rounds |
freshqa_rag_simba_config.yml | SIMBA parameters: bsize, num_candidates, max_steps, max_demos |
freshqa_rag_gepa_config.yml | GEPA parameters: max_full_evals, reflection_minibatch_size, candidate_selection_strategy |
freshqa_rag_evaluation_config.yml | Evaluation settings and DeepEval metric thresholds |
MIPROv2 Configuration
The default MIPROv2 configuration (freshqa_rag_mipro_config.yml) is shown below. All other optimizer configs follow the same top-level structure but differ in the optimizer section.
answer_llm:
model: "groq/qwen3-32b"
api_key_env: "GROQ_API_KEY"
extractor_llm:
model: "groq/llama-3.3-70b-versatile"
api_key_env: "GROQ_API_KEY"
embedding:
model: "Qwen/Qwen3-Embedding-0.6B"
tokenizer_kwargs:
padding_side: "left"
weaviate:
url_env: "WEAVIATE_URL"
api_key_env: "WEAVIATE_API_KEY"
collection_name: "FreshQA"
top_k: 5
dataset:
name: "vtllms/sealqa"
subset: "longseal"
split: "test"
optimizer:
max_bootstrapped_demos: 3
max_labeled_demos: 16
auto: "medium"
Running the Pipeline
All scripts must be run from the freshqa/ directory so that relative config file paths resolve correctly.
# Index documents into Weaviate
cd src/dspy_opt/freshqa
python freshqa_indexing.py
# Run MIPROv2 optimization
cd src/dspy_opt/freshqa
python freshqa_rag_mipro.py
# Run SIMBA optimization
cd src/dspy_opt/freshqa
python freshqa_rag_simba.py
# Run GEPA optimization
cd src/dspy_opt/freshqa
python freshqa_rag_gepa.py
# Evaluate the optimized pipeline
cd src/dspy_opt/freshqa
python freshqa_rag_evaluation.py
Programmatic Usage
import dspy
from sentence_transformers import SentenceTransformer
from dspy_opt.freshqa.freshqa_rag_module import FreshQARAG
from dspy_opt.utils.metadata_extractor import MetadataExtractor
from dspy_opt.utils.query_rewriter import QueryRewriter
from dspy_opt.utils.sub_query_generator import SubQueryGenerator
from dspy_opt.utils.weaviate_retriever import WeaviateRetriever
# Configure LLMs
answer_lm = dspy.LM("groq/qwen3-32b", api_key="your-groq-api-key")
extractor_lm = dspy.LM("groq/llama-3.3-70b-versatile", api_key="your-groq-api-key")
dspy.configure(lm=answer_lm)
# Initialize components
query_rewriter = QueryRewriter()
sub_query_generator = SubQueryGenerator()
metadata_extractor = MetadataExtractor(extractor_llm=extractor_lm)
embedding_model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
retriever = WeaviateRetriever(
weaviate_url="your-weaviate-url",
weaviate_api_key="your-weaviate-api-key",
collection_name="FreshQA",
top_k=5,
)
metadata_schema = {
"properties": {
"title": {"type": "string", "description": "The main title or name of the subject"},
"category": {"type": "string", "description": "Primary category or type of content"},
}
}
# Build and run the pipeline
pipeline = FreshQARAG(
query_rewriter=query_rewriter,
sub_query_generator=sub_query_generator,
metadata_extractor=metadata_extractor,
metadata_schema=metadata_schema,
weaviate_retriever=retriever,
embedding_model=embedding_model,
top_k=5,
)
result = pipeline("Is the Great Wall of China visible from space?")
print(result.answer)
print(result.reasoning)