The MaternaQA-es pipeline is an 8-step reproducible process that turns raw clinical PDFs into a published, auditable Spanish-language Q&A dataset. Every intermediate artifact — raw pages, clean pages, chunks, Q&A pairs, and evaluation reports — is written to disk so the entire construction chain is traceable back to its source documents. Running the pipeline from scratch produces byte-identical outputs given the same inputs and seed, making the dataset creation process fully reproducible for peer review and extension.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/NicolasHoyosDevss/MaternaQA-es/llms.txt
Use this file to discover all available pages before exploring further.
The 8 Pipeline Steps
Source Curation
Clinical PDFs are selected and documented in a manifest (
document_manifest.py). Each PDF is registered with its filename, path, size, page count, document type, inclusion status, language, and any pages flagged for potential OCR. Excluded documents record a reason for exclusion, keeping the selection process auditable.Text Extraction
extract_pdfs.py reads each PDF page-by-page. The primary extractor is PyMuPDF, which provides fast and stable block-level text parsing. When a page yields too little content, pdfplumber acts as a layout-aware fallback. Pages that remain below the minimum character threshold after both methods are flagged needs_ocr and preserved in the inventory for manual review.Cleaning and Filtering
clean_text.py removes documentary noise without aggressively normalizing clinical content. Pages with repeated headers and footers, page numbers, table-of-contents content, reference-heavy sections, fragmented extraction artifacts, or OCR-required content are marked and dropped. Every page retains an audit record — is_kept, drop_reason, section type, extraction method, and document type — for downstream inspection.Document Segmentation (Chunking)
build_lm_dataset.py groups clean pages into clinically-useful chunks with configurable token limits and overlap. Chunk boundaries respect document structure so that each unit contains enough context for Q&A generation without exceeding model input limits. Every chunk receives a stable chunk_id, source PDF, page range, clean text, section type, content role, clinical topics, token estimate, and clinical score.Topic Enrichment
content_enrichment.py assigns clinical scores and topic tags to each chunk from the 18-topic taxonomy (prenatal_care, postpartum, preterm_labor, labor_induction, vaginal_birth, cesarean, hemorrhage, preeclampsia, diabetes_gestational, infection, fetal_monitoring, newborn_care, ultrasound, genetics, contraception, infertility, menopause, gynecologic_oncology). The clinical_score gates which chunks are eligible for Q&A generation.Synthetic Q&A Generation
generate_synthetic_qa.py uses OpenAI Structured Outputs to generate clinically-grounded Spanish Q&A pairs from enriched corpus chunks. The async pipeline processes all eligible chunks concurrently and writes resumable checkpoints so interrupted runs can continue without duplicating work. Six question types — factual, definicion, comparacion, razonamiento, aplicacion, and hipotetico — are generated across three difficulty levels.Quality Control
Quality is assessed in two layers. A custom operational judge estimates faithfulness, answer relevancy, roundtrip consistency, and issues a quality verdict for each pair during generation. Ragas (
evaluate_qa_with_ragas.py) then provides formal, comparable metrics over stratified samples from each split. The two layers serve distinct roles: the judge filters and diagnoses during construction; Ragas provides the reportable evaluation.Publication Preparation
prepare_qa_publication_variants.py exports three clean variants from the audited Q&A files: sft_closed_book (question → answer, for parametric fine-tuning), sft_grounded (context + question → answer, for evidence-grounded fine-tuning), and qa_flat_jsonl (flat records with full metadata, for auditing and RAG evaluation). Internal evaluation fields are removed from the publication variants.Run the Full Pipeline
To rebuild the entire corpus from PDF extraction through audit reports in one command:--generate-qa:
Run Incrementally
Pipeline Outputs
Corpus LM
datasets/obstetrics/lm/Train, validation, and test JSONL files containing annotated chunks ready for language model training and Q&A generation.QA Final
datasets/obstetrics/qa/final/Per-split Q&A artifacts: raw.jsonl (auditable pairs with full metadata), sft.jsonl (messages format), progress.json, and generation_report.json.Publication Variants
datasets/obstetrics/qa/publication/Three clean publication-ready variants: sft_closed_book/, sft_grounded/, and qa_flat_jsonl/ — each split into train, validation, and test files.Audit Reports
artifacts/obstetrics/reports/Cleaning report, build report, table extraction report, and the final audit report covering duplicate detection, split statistics, and integrity checks.Full Pipeline Flags
The following arguments are accepted byrun_full_pipeline.py to control corpus construction behavior:
| Flag | Default | Description |
|---|---|---|
--input-dir | pdfs/obstetrics/ | Source directory containing clinical PDFs |
--validation-ratio | 0.05 | Fraction of documents assigned to the validation split |
--test-ratio | 0.05 | Fraction of documents assigned to the test split |
--seed | 42 | Random seed for reproducible document-level splits |
--min-clinical-score | 5 | Minimum clinical score required to include a chunk |
--min-tokens | 500 | Minimum token count for a chunk to be published |
--max-tokens | 1200 | Maximum token count per chunk before splitting |
--overlap-tokens | 80 | Token overlap between consecutive chunks |
--generate-qa | false | When set, runs the synthetic QA generation step (requires OPENAI_API_KEY) |
Detailed Documentation
For a deeper look at each phase, see the dedicated pages:- Corpus Construction — PDF extraction, page cleaning, chunking, and topic enrichment
- QA Generation — Synthetic Q&A generation with OpenAI Structured Outputs
- Evaluation — Ragas formal evaluation and model benchmarking