Mindloom Training Corpus: Data Sources and Model Pipeline

Mindloom’s ML capabilities rest on a bilingual (English + Spanish) technical content corpus assembled from public sources, cleaned and labelled across 8 thematic categories. From that corpus, a two-notebook pipeline produces a single model.joblib artifact that the inference service loads at startup. This page covers how both are built — from raw data through training to the moment the artifact lands in OCI Object Storage.

The Corpus

The corpus is stored as JSONL files — one record per line — in the data/processed/ directory. Three files are produced and uploaded to OCI Object Storage:

File	Purpose
`processed/train_corpus.jsonl`	Training split
`processed/test_corpus.jsonl`	English evaluation split
`processed/test_corpus_es.jsonl`	Spanish evaluation split (cross-lingual transfer)

Each line follows a fixed schema with all fields mandatory:

{
  "id": "...",
  "title": "...",
  "body": "...",
  "category": "...",
  "source": "...",
  "url": "...",
  "language": "es"
}

An additional quality field (derived from source engagement signals — votes on Stack Overflow, reactions on Dev.to) is used internally by the ETL notebook for weighted sampling. It is not persisted in the contents table and does not appear in the training features.

Data Sources

The corpus draws from three public sources, chosen to cover all 8 categories across both languages:

Kaggle StackSample

stackoverflow/stacksample — English questions labelled via weighted tag voting

Dev.to API

Technical articles in both English and Spanish — the closest match to the content Mindloom indexes

Stack Exchange Dumps

es.stackoverflow.com for Spanish; security, dba, and softwareengineering for underrepresented categories

Category Labels

Every document is assigned exactly one of the 8 categories. The labelling rules live in CATEGORY_TAGS inside the EDA/ETL notebook.

Category
Backend
Frontend
Móvil
Datos e IA
DevOps y Cloud
Bases de datos
Seguridad
Fundamentos

EDA and ETL Notebook

File: data/TechMind_01_EDA_ETL.ipynb — runs in Google Colab. This notebook handles the full data preparation pipeline:

Extraction

Fetches data from Kaggle (via Colab Secrets credentials), the Dev.to API, and Stack Exchange archive dumps.

Exploration and cleaning

Performs EDA, removes low-quality documents, normalizes text, and applies deduplication.

Category labelling

Maps source tags and metadata to one of the 8 Mindloom categories using CATEGORY_TAGS.

Stratified split and export

Splits into train/test with random_state=42 and exports to the three JSONL files. Uploads them to the OCI Object Storage bucket.

Use random_state=42 on every sampling operation and stratified split in this notebook. Without it, two runs of the same code on the same source data will produce different class distributions and make metrics non-reproducible.

Model Training Notebook

File: notebook/TechMind_02_Model_Training.ipynb — runs in Google Colab or Jupyter. This notebook reads the processed JSONL files from OCI Object Storage and trains the full pipeline. It requires processed/*.jsonl to already be uploaded by the EDA/ETL notebook.

Load corpus

Downloads train_corpus.jsonl (and optionally the test files) from OCI Object Storage.

Embed the full corpus

Encodes every document body with "passage: " prefix using intfloat/multilingual-e5-small, producing L2-normalized float32 vectors. This covers the complete corpus, not just the training split.

Train TF-IDF and SVD baseline

Fits a TfidfVectorizer (baseline_vectorizer) on the training text, then fits TruncatedSVD on the resulting matrix. Also fits a separate TfidfVectorizer (keyword_vectorizer) for keyword extraction.

Train the main classifier

Builds hybrid features (E5 embedding concatenated with SVD-reduced TF-IDF, both L2-normalized) and trains a LogisticRegression classifier.

Train K-means and UMAP

Fits KMeans and UMAP on the corpus embeddings. Both are stored as fitted model objects so the inference service can assign cluster IDs and map coordinates to documents it never saw during training.

Evaluate

Computes macro-F1 on the English and Spanish test sets. Metrics are stored in the meta dict and returned by GET /model/info.

Serialize and publish

Serializes the artifact dict to model.joblib using joblib, uploads the artifact and corpus_index.npz, then updates models/latest.txt.

The Serialized Artifact

The training notebook serializes a Python dict to model.joblib using joblib.dump(). Every key in this dict is a contract: the inference service reads them by name, and renaming or removing a key is a breaking change that will crash the service at startup.

Key	Type	Description
`classifier`	`LogisticRegression`	Main classifier (E5 + SVD-reduced TF-IDF features → category)
`label_encoder`	`LabelEncoder`	Maps integer indices ↔ category name strings
`keyword_vectorizer`	`TfidfVectorizer`	Bilingual keyword extraction (top 5 terms per document)
`baseline_vectorizer`	`TfidfVectorizer`	Input to SVD and to the explainability baseline
`baseline_classifier`	`LogisticRegression`	Explainability baseline; coefficients used in `_get_explanation()`
`svd`	`TruncatedSVD`	Reduces TF-IDF matrix to `svd_components` dimensions
`kmeans`	`KMeans`	Assigns cluster IDs to new content at request time
`umap_reducer`	`UMAP`	Projects new content to 2D map coordinates at request time
`meta`	`dict`	Version, embedding model name, `dim`, `feature_dim`, `svd_components`, E5 prefixes, categories, metrics, `train_size`

meta["embedding_model"] stores the name of the transformer (e.g., "intfloat/multilingual-e5-small"). The inference service uses this name to load the model from the local HuggingFace cache baked into the Docker image. Transformer weights are never stored in the joblib — doing so would inflate the artifact from ~0.24 MB to ~470 MB.

The `meta` Dict Structure

The meta dict is returned verbatim by GET /model/info. Its fields map directly to the ModelInfoResponse Pydantic schema:

{
    "version": "v1",
    "embedding_model": "intfloat/multilingual-e5-small",
    "dim": 384,
    "feature_dim": 884,       # dim + svd_components
    "svd_components": 500,
    "classifier_c": 4.0,
    "doc_prefix": "passage: ",
    "query_prefix": "query: ",
    "categories": [...],       # list of 8 category name strings
    "n_clusters": 8,
    "terms_by_category": {...},
    "metrics": {
        "embedding_macro_f1_en": ...,
        "embedding_macro_f1_es": ...,
        "tfidf_macro_f1_en": ...,
        "tfidf_macro_f1_es": ...,
        "embedding_macro_f1_es_reliable": ...,
        "es_reliable_categories": 8,
        "es_min_support": 30
    },
    "train_size": ...
}

Publishing a New Model

To promote a retrained model to production:

Upload the model artifact

oci os object put --bucket-name techmind-data \
  --file model.joblib --name models/v2/model.joblib

Upload the corpus index

oci os object put --bucket-name techmind-data \
  --file corpus_index.npz --name models/v2/corpus_index.npz

Update the pointer — last

echo "models/v2/" > latest.txt
oci os object put --bucket-name techmind-data \
  --file latest.txt --name models/latest.txt

The inference service reads models/latest.txt on every startup. The next container restart will automatically download and load the new version — no Docker image rebuild or API redeployment required.

latest.txt must be updated after both model.joblib and corpus_index.npz have finished uploading. If the pointer is updated first and the inference container restarts during the upload window, it will attempt to download a file that does not yet exist and fail to start — taking the entire API stack down with it.

Retraining Workflow

To retrain with new data, run both notebooks end-to-end in order (TechMind_01_EDA_ETL.ipynb then TechMind_02_Model_Training.ipynb), then publish the new joblib as described above. No code changes are needed in the inference service, the API, or the web frontend.

For how the model artifact is loaded and used at runtime, see Inference Service.

Get Started

Core Features

Deployment

Data & ML

Mindloom Training Corpus: Data Sources and Model Pipeline

The Corpus

Data Sources

Kaggle StackSample

Dev.to API

Stack Exchange Dumps

Category Labels

EDA and ETL Notebook

Model Training Notebook

The Serialized Artifact

The `meta` Dict Structure

Publishing a New Model

Retraining Workflow

Build docs developers (and LLMs) love

Get Started

Core Features

Deployment

Data & ML

Documentation Index

​The Corpus

​Data Sources

Kaggle StackSample

Dev.to API

Stack Exchange Dumps

​Category Labels

​EDA and ETL Notebook

​Model Training Notebook

​The Serialized Artifact

​The meta Dict Structure

​Publishing a New Model

​Retraining Workflow

Build docs developers (and LLMs) love

The Corpus

Data Sources

Category Labels

EDA and ETL Notebook

Model Training Notebook

The Serialized Artifact

The `meta` Dict Structure

Publishing a New Model

Retraining Workflow