evaluate_model_predictions.py — Model Prediction Evaluator

evaluate_model_predictions.py takes a JSONL predictions file from inference_base.py or inference_qlora.py and evaluates each prediction against the reference answer and source context from the test split. It produces both a per-sample scored JSONL and an aggregate summary JSON, which can then be exported to CSV for cross-model comparison using convert_eval_to_csv.py.

Usage

python scripts/evaluate_model_predictions.py \
  --input outputs/gemma4-base/test_predictions.jsonl \
  --output outputs/gemma4-base/test_eval.json

Use --limit to evaluate a subset during development:

python scripts/evaluate_model_predictions.py \
  --input outputs/gemma4-base/test_predictions.jsonl \
  --output outputs/gemma4-base/test_eval.json \
  --limit 25

Use --estimate-only to preview approximate token volume before incurring API costs:

python scripts/evaluate_model_predictions.py \
  --input outputs/gemma4-base/test_predictions.jsonl \
  --estimate-only

CLI Flags

Flag	Type	Default	Description
`--input`	`Path`	(required)	JSONL predictions file generated by `inference_base.py` or `inference_qlora.py`.
`--output`	`Path`	`<input_stem>_eval.json`	Summary report JSON. Defaults to the same directory as `--input` with `_eval.json` suffix.
`--output-jsonl`	`Path`	`<output>.jsonl`	Per-sample scored JSONL (incremental detail file). Defaults to the same path as `--output` with `.jsonl` extension.
`--limit`	`int`	`None`	Evaluate only the first N examples from the input file.
`--timeout-seconds`	`int`	`180`	Per-sample timeout in seconds for Ragas metric calls.
`--llm-model`	`str`	`gpt-5.4-mini`	OpenAI model used by LLM-based Ragas metrics (faithfulness, answer relevancy, answer correctness).
`--embedding-model`	`str`	`text-embedding-3-small`	OpenAI embedding model used by semantic metrics (answer correctness, semantic similarity).
`--max-completion-tokens`	`int`	`2048`	Output token limit for the judge LLM. Increasing this reduces `IncompleteOutputException` errors on long clinical responses.
`--metrics`	`str` (one or more)	All four metrics	Metrics to compute. Choices: `faithfulness`, `answer_relevancy`, `answer_correctness`, `semantic_similarity`.
`--resume`	`bool`	`True`	Resume from `--output-jsonl` if it exists, skipping already-evaluated `qa_id`s. Use `--no-resume` to recalculate from scratch.
`--estimate-only`	flag	`False`	Print an approximate token volume estimate without calling OpenAI or Ragas.

Metrics Computed

The evaluator uses Ragas metrics backed by an OpenAI LLM judge and embedding model:

Metric	Type	What it measures
`faithfulness`	LLM-based	Whether the generated answer is grounded in the provided source context. Scores high when every claim in the answer can be traced to the context.
`answer_relevancy`	LLM + embedding	Whether the generated answer directly addresses the question. Penalizes answers that are technically correct but off-topic.
`answer_correctness`	LLM + embedding	Combined factual and semantic correctness of the generated answer relative to the reference answer.
`semantic_similarity`	Embedding-based	Cosine similarity between the embedding of the generated answer and the reference answer. Purely semantic — does not involve a judge LLM.

Each metric is scored asynchronously per sample with a configurable timeout. If a metric call fails or times out, the field is set to null and an error field records the exception.

Output Files

test_eval.jsonl (per-sample detail, written incrementally): Each line is a scored record with the following fields: qa_id, model_role, model_name, adapter_dir, dataset_variant, question, generated_answer, reference_answer, source_context, metadata, plus one numeric field per metric (faithfulness, answer_relevancy, answer_correctness, semantic_similarity). LLM-based metrics also include a <metric>_reason field when Ragas returns a rationale. Failed metrics are null and the error field is populated. test_eval.json (aggregate summary):

{
  "script_version": "model_prediction_eval_v2_resumable",
  "generated_at": "2025-01-01T00:00:00+00:00",
  "input_file": "outputs/gemma4-base/test_predictions.jsonl",
  "detail_file": "outputs/gemma4-base/test_eval.jsonl",
  "sampled_predictions": 328,
  "metrics": ["faithfulness", "answer_relevancy", "answer_correctness", "semantic_similarity"],
  "ragas_models": {
    "llm_model": "gpt-5.4-mini",
    "embedding_model": "text-embedding-3-small",
    "max_completion_tokens": 2048
  },
  "summary": {
    "predictions_evaluated": 328,
    "predictions_with_error": 0,
    "avg_faithfulness": 0.8412,
    "avg_answer_relevancy": 0.9103,
    "avg_answer_correctness": 0.7654,
    "avg_semantic_similarity": 0.8891
  }
}

CSV Export

Use convert_eval_to_csv.py to convert evaluation outputs from one or more models into three CSV files for comparison and publication:

python scripts/convert_eval_to_csv.py \
  --outputs-dir outputs/ \
  --out-dir csv_outputs/

Flag	Type	Default	Description
`--outputs-dir`	`str`	`outputs`	Directory containing model subdirectories, each with a `test_eval.jsonl` file.
`--out-dir`	`str`	`csv_outputs`	Directory to write the resulting CSV files.

The script discovers all subdirectories under --outputs-dir, loads their test_eval.jsonl files, and produces three outputs:

File	Format	Content
`csv_outputs/master_eval.csv`	Long format	One row per `(model, qa_id)` with all metric scores and metadata fields (question type, difficulty, source PDF, section, pages, topics, etc.)
`csv_outputs/model_summary.csv`	One row per model	Aggregate scores: `avg_faithfulness`, `avg_answer_relevancy`, `avg_answer_correctness`, `avg_semantic_similarity`, prediction counts, error counts
`csv_outputs/wide_comparison.csv`	Wide format	One row per `qa_id` with side-by-side generated answers and metric scores for every model as separate columns

Comparing Across Models

Run inference with the base model

Generate baseline predictions using the unmodified model on the test split:

python scripts/inference_base.py \
  --model-name google/gemma-4-E2B-it \
  --output-prefix outputs/gemma4-base/test

Run inference with the QLoRA adapter

Generate predictions using the fine-tuned adapter from train_qlora_trl.py:

python scripts/inference_qlora.py \
  --adapter-dir outputs/gemma4-grounded \
  --output-prefix outputs/gemma4-grounded/test

Evaluate both prediction files

Score each predictions file against the reference answers:

python scripts/evaluate_model_predictions.py \
  --input outputs/gemma4-base/test_predictions.jsonl \
  --output outputs/gemma4-base/test_eval.json

python scripts/evaluate_model_predictions.py \
  --input outputs/gemma4-grounded/test_predictions.jsonl \
  --output outputs/gemma4-grounded/test_eval.json

Export and compare with convert_eval_to_csv.py

Merge all eval outputs into structured CSVs for analysis:

python scripts/convert_eval_to_csv.py \
  --outputs-dir outputs/ \
  --out-dir csv_outputs/

The resulting csv_outputs/wide_comparison.csv places all model scores side-by-side per question for direct comparison.

Pre-computed evaluation results are already present in the repository under outputs/. The following model runs are available without re-running inference or evaluation:

outputs/gemma4-base/ — baseline Gemma 4 predictions and eval scores
outputs/gemma4-grounded/ — QLoRA-fine-tuned Gemma 4 (grounded variant) scores
outputs/medgemma-base/ — baseline MedGemma 1.5 4B predictions and eval scores
outputs/medgemma-grounded/ — QLoRA-fine-tuned MedGemma 1.5 4B (grounded variant) scores

Corpus Scripts

QA Scripts

Model Scripts

evaluate_model_predictions.py — Model Prediction Evaluator

Usage

CLI Flags

Metrics Computed

Output Files

CSV Export

Comparing Across Models

Build docs developers (and LLMs) love

Corpus Scripts

QA Scripts

Model Scripts

Documentation Index

​Usage

​CLI Flags

​Metrics Computed

​Output Files

​CSV Export

​Comparing Across Models

Build docs developers (and LLMs) love

Usage

CLI Flags

Metrics Computed

Output Files

CSV Export

Comparing Across Models