Experiment Loop

Input: Dataset + Target + Task Type + (optional) Constraints
                    │
                    ▼
            ┌────────────────┐
            │ DATA PROFILING │  Analyze schema, distributions, missing values
            └───────┬────────┘
                    │
                    ▼
            ┌────────────────┐
            │ BASELINE MODEL │  Simple model to establish performance floor
            └───────┬────────┘
                    │
                    ▼
    ┌────────────────────────────────────┐
    │          ITERATION LOOP            │
    │                                    │
    │  1. Experiment Design (Gemini)     │  ← hypothesis, model, params
    │  2. Code Generation (Jinja2)       │  ← validated Python script
    │  3. Execution (subprocess)         │  ← train, evaluate, capture metrics
    │  4. Results Analysis (Gemini)      │  ← trends, comparisons, insights
    │  5. Hypothesis Generation (Gemini) │  ← ranked next steps
    │  6. Termination Check              │  ← continue or stop?
    │                                    │
    │  Repeat until termination...       │
    └───────────────┬────────────────────┘
                    │
                    ▼
          ┌───────────────────┐
          │ REPORT GENERATION │  Gemini writes narrative Markdown report
          └─────────┬─────────┘
                    │
                    ▼
Output: Best Model + Report + Visualizations + MLflow Experiment + Code

DataProfile(
    n_rows=20640,
    n_columns=9,
    numeric_columns=["MedInc", "HouseAge", "AveRooms", ...],
    categorical_columns=[],
    target_column="MedHouseVal",
    target_type="continuous",
    missing_values={"total_bedrooms": 207},
    target_stats={
        "mean": 2.07,
        "std": 1.15,
        "skew": 0.98,  # Right-skewed → consider log transform
    }
)

Iteration 0: baseline
  Model: LinearRegression
  RMSE: 0.7343
  Execution Time: 0.8s

{
  "experiment_name": "xgboost_tuned_learning_rate",
  "hypothesis": "Lower learning rate with more estimators may reduce overfitting",
  "model_type": "XGBRegressor",
  "model_params": {
    "n_estimators": 200,
    "learning_rate": 0.05,
    "max_depth": 6
  },
  "preprocessing": {
    "missing_values": "median",
    "scaling": "standard",
    "target_transform": "log"
  },
  "reasoning": "Previous iteration showed XGBoost overfit with learning_rate=0.1. Reducing to 0.05 and increasing n_estimators should improve generalization."
}

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor
import json

# Load data
df = pd.read_csv("data/sample/california_housing.csv")

# Handle missing values (median imputation)
# ... preprocessing code ...

# Apply target transform (log)
y_train_transformed = np.log1p(y_train)
y_test_transformed = np.log1p(y_test)

# Train model
model = XGBRegressor(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=6,
    random_state=42,
    n_jobs=-1
)
model.fit(X_train_scaled, y_train_transformed)

# Evaluate
y_pred_transformed = model.predict(X_test_scaled)
y_pred = np.expm1(y_pred_transformed)  # Inverse log transform

rmse = np.sqrt(mean_squared_error(y_test, y_pred))
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

# Output JSON
result = {
    "metrics": {"rmse": rmse, "mae": mae, "r2": r2},
    "model_path": "outputs/models/model_iter_3.pkl",
    "success": True
}
print(json.dumps(result))

process = subprocess.run(
    ["python", script_path],
    capture_output=True,
    text=True,
    timeout=600,  # 10 minutes
)

{"metrics": {"rmse": 0.1332, ...}, "success": true}

ExperimentResult(
    experiment_name="xgboost_tuned_learning_rate",
    iteration=3,
    model_type="XGBRegressor",
    model_params={"n_estimators": 200, "learning_rate": 0.05, "max_depth": 6},
    metrics={"rmse": 0.1332, "mae": 0.0987, "r2": 0.8654},
    execution_time=12.4,
    success=True,
)

{
  "key_observations": [
    "XGBoost with lower learning rate (0.05) outperformed iteration 2's 0.1 by 10.3%",
    "Log transformation of target variable remains critical — iteration 1 without it had RMSE 2x worse",
    "Tree-based models (iterations 2-3) consistently outperform linear models (iteration 0)",
    "Diminishing returns observed — improvement from iteration 2 to 3 is only 2.1%"
  ],
  "reasoning": "The lower learning rate prevented overfitting while increased n_estimators maintained model capacity. However, we may be approaching the limit of this dataset's predictability."
}

{
  "analysis_summary": "XGBoost shows strong performance but diminishing returns suggest we're near optimal for this approach",
  "exploration_vs_exploitation": "balanced",
  "hypotheses": [
    {
      "hypothesis_id": "h1",
      "statement": "Fine-tune XGBoost regularization (alpha/lambda) to squeeze out remaining 1-2% improvement",
      "rationale": "Current model shows slight overfitting based on train/test gap",
      "suggested_model": "XGBRegressor",
      "suggested_params": {"reg_alpha": 0.5, "reg_lambda": 1.0},
      "confidence_score": 0.72,
      "priority": 1
    },
    {
      "hypothesis_id": "h2",
      "statement": "Try LightGBM as alternative gradient booster with better handling of categorical features",
      "rationale": "LightGBM often matches XGBoost performance with faster training",
      "suggested_model": "LGBMRegressor",
      "confidence_score": 0.65,
      "priority": 2
    }
  ],
  "reasoning": "Focus on exploitation (h1) since we have a strong baseline, but keep one exploratory option (h2) in case LightGBM surprises us."
}

should_stop, reason = self.state.should_terminate()
if should_stop:
    self.state.termination_reason = reason
    self.state.phase = ExperimentPhase.COMPLETED
    break

# ML Experiment Report: california_housing

## Executive Summary

This report summarizes an automated ML experiment session on the California Housing dataset for regression. A total of 20 experiments were conducted (18 successful), optimizing for RMSE. The best performing model was xgboost_tuned_regularization with RMSE = 0.1289. This represents an 82.4% improvement over the baseline LinearRegression model.

## Best Model

**Model**: XGBRegressor
**Experiment**: xgboost_tuned_regularization
**Iteration**: 17
**Primary Metric (rmse)**: 0.128943

**All Metrics**:
| Metric | Value |
|--------|-------|
| mae | 0.0876 |
| r2 | 0.8854 |
| rmse | 0.1289 |

**Hyperparameters**:
| Parameter | Value |
|-----------|-------|
| learning_rate | 0.05 |
| max_depth | 6 |
| n_estimators | 200 |
| reg_alpha | 0.5 |

def save_state(self):
    state_path = self.output_dir / f"state_{self.state.session_id}.json"
    self.state.save(state_path)

python -m src.main run \
  --resume outputs/state_a3f9c2d1.json \
  --verbose

[Phase 1: Data Profiling]
✓ Profiled 20,640 rows × 9 columns
✓ Target: MedHouseVal (continuous, right-skewed)
✓ Primary metric: rmse

[Phase 2: Baseline]
Iteration 0: baseline
  Model: LinearRegression
  RMSE: 0.7343 ← baseline

[Phase 3: Iteration Loop]
Iteration 1: random_forest_initial
  Model: RandomForestRegressor
  RMSE: 0.2456 (66.5% better than baseline) ← new best

Iteration 2: log_transform_target
  Model: RandomForestRegressor
  RMSE: 0.1421 (42.2% better than iteration 1) ← new best

Iteration 3: xgboost_initial
  Model: XGBRegressor
  RMSE: 0.1332 (6.3% better than iteration 2) ← new best

... iterations 4-16 ...

Iteration 17: xgboost_tuned_regularization
  Model: XGBRegressor
  RMSE: 0.1289 (3.2% better than previous best) ← new best

Iteration 18: lightgbm_alternative
  Model: LGBMRegressor
  RMSE: 0.1301 (no improvement)

Iteration 19: xgboost_final_tuning
  Model: XGBRegressor
  RMSE: 0.1291 (no improvement)

Iteration 20: ensemble_stacking
  Model: StackingRegressor
  RMSE: 0.1288 (0.08% better) ← marginal, diminishing returns

✓ Termination: Maximum iterations reached

[Phase 4: Finalization]
✓ Best model: xgboost_tuned_regularization (RMSE: 0.1289)
✓ Generated 3 visualizations
✓ Report saved: outputs/reports/report_california_housing_20260302_143022.md
✓ MLflow tracking: mlflow ui --backend-store-uri file:./outputs/mlruns