Documentation Index
Fetch the complete documentation index at: https://mintlify.com/NicolasHoyosDevss/MaternaQA-es/llms.txt
Use this file to discover all available pages before exploring further.
inference_base.py runs the MaternaQA-es test set through an unmodified base model — no adapter is loaded. The generated predictions form the baseline used to measure the improvement gained from QLoRA fine-tuning. The output file is a JSONL of per-sample predictions that can be fed directly into evaluate_model_predictions.py for scoring.
Usage
<output-prefix>_predictions.jsonl. For example, --output-prefix outputs/gemma4-base/test produces outputs/gemma4-base/test_predictions.jsonl.
CLI Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--model-name | str | (required) | Hugging Face model ID or local path of the base model. |
--model-class | str | auto | Model loading class. auto uses AutoModelForImageTextToText for Gemma 4/MedGemma and AutoModelForCausalLM for others. Choices: auto, causal-lm, image-text-to-text. |
--dataset-root | Path | datasets/obstetrics/qa/publication | Root folder containing the publication variant subdirectories. |
--dataset-variant | str | sft_grounded | SFT variant to read the test split from. Choices: sft_closed_book, sft_grounded. |
--output-prefix | Path | (required) | Path prefix for the output file. The script appends _predictions.jsonl to produce the final filename. |
--max-new-tokens | int | 512 | Maximum number of new tokens the model may generate per prediction. |
--temperature | float | 0.7 | Sampling temperature. Only applied when --do-sample is set. |
--top-p | float | 0.9 | Nucleus sampling probability threshold. Only applied when --do-sample is set. |
--do-sample | flag | False | Use stochastic sampling. By default the script uses greedy decoding for reproducible evaluation. |
--limit | int | None | Maximum number of test examples to process. Useful for smoke tests. |
--trust-remote-code | flag | False | Pass trust_remote_code=True to model and tokenizer loaders. |
--attn-implementation | str | None | Optional attention backend override: eager, sdpa, flash_attention_2, etc. |
--load-in-4bit | bool | True | Load the base model in 4-bit NF4 quantization to keep inference memory-comparable to training and reduce VRAM usage. |
--resume | bool | True | Resume from an existing <output-prefix>_predictions.jsonl, skipping already-processed qa_ids. Use --no-resume to overwrite the file from scratch. |
What It Does
- Loads the base model in 4-bit NF4 quantization (matching the training setup) using
BitsAndBytesConfigwithbnb_4bit_quant_type="nf4"and double quantization enabled. The model is loaded withdevice_map="auto"and placed ineval()mode. - Reads the test split from
<dataset-root>/<dataset-variant>/test.jsonl. The--dataset-variantflag controls whether the model receives source context (sft_grounded) or only the question (sft_closed_book). - Builds a prompt for each example by extracting all messages up to (but not including) the last assistant turn, then applying the tokenizer’s chat template with
add_generation_prompt=True. - Generates a prediction using greedy decoding by default (set
--do-samplefor stochastic generation). The new tokens are decoded and stripped of special tokens. - Writes results incrementally to the output JSONL file, flushing after every prediction so the run can be safely interrupted and resumed.
Output File
Each line in<output-prefix>_predictions.jsonl is a JSON object with the following fields:
| Field | Description |
|---|---|
id | qa_id from dataset metadata, or sequential index as fallback |
model_role | Always "base" for predictions from this script |
model_name | The value of --model-name |
adapter_dir | Always null (no adapter loaded) |
dataset_variant | The value of --dataset-variant |
question | Extracted question text |
generated_answer | The model’s generated prediction |
reference_answer | The ground-truth answer from the test split |
source_context | Source context from metadata.contexto_fuente |
prompt_text | The full rendered prompt string sent to the model |
metadata | Full metadata dict from the original JSONL record |
The repository includes pre-computed baseline predictions ready for evaluation. You can use them directly without re-running inference:
outputs/gemma4-base/test_predictions.jsonloutputs/medgemma-base/test_predictions.jsonl
Next Step
Pipe the predictions file intoevaluate_model_predictions.py to compute faithfulness, answer relevancy, answer correctness, and semantic similarity scores: