Training Models

Model training in Neurenix is performed using the run command. This page provides additional context and examples specific to training workflows.

Overview

The neurenix run command is used to train models. It executes your training script with configurable parameters and manages the training environment. For complete command reference, see the run command documentation.

Quick Start

# Initialize a project
neurenix init --name my-model

# Navigate to project
cd my-model

# Train with default settings
neurenix run train.py

Training Workflows

Basic Training

# Train with config file
neurenix run train.py --config config.json

Custom Hyperparameters

# Override training parameters
neurenix run train.py \
  --epochs 100 \
  --batch-size 64 \
  --learning-rate 0.001

GPU Training

# Auto-detect and use GPU
neurenix run train.py --device auto

# Force specific GPU
neurenix run train.py --device cuda

# Force CPU training
neurenix run train.py --device cpu

Distributed Training

For distributed training across multiple GPUs or nodes, configure your training script to use Neurenix’s distributed training utilities:

import neurenix
from neurenix.distributed import DistributedTrainer

config = neurenix.load_config("config.json")

# Initialize distributed training
trainer = DistributedTrainer(
    model=neurenix.create_model(config["model"]),
    backend="nccl"  # or "gloo" for CPU
)

trainer.train(
    train_data,
    val_data,
    epochs=config["training"]["epochs"]
)

Then run with:

neurenix run train_distributed.py --device auto

Monitoring Training

While training is running, you can monitor progress in real-time:

# In another terminal
neurenix monitor --log-dir logs

This displays:

Current epoch and batch
Training and validation metrics
Loss curves
Hardware utilization

Configuration Examples

Image Classification

{
  "model": {
    "type": "resnet",
    "depth": 50,
    "num_classes": 10
  },
  "training": {
    "batch_size": 128,
    "epochs": 90,
    "learning_rate": 0.1,
    "optimizer": "sgd",
    "momentum": 0.9,
    "weight_decay": 0.0001,
    "scheduler": {
      "type": "step",
      "step_size": 30,
      "gamma": 0.1
    }
  },
  "hardware": {
    "device": "auto",
    "precision": "mixed"
  }
}

Natural Language Processing

{
  "model": {
    "type": "transformer",
    "num_layers": 12,
    "hidden_size": 768,
    "num_heads": 12
  },
  "training": {
    "batch_size": 32,
    "epochs": 3,
    "learning_rate": 0.00005,
    "optimizer": "adamw",
    "warmup_steps": 1000,
    "max_grad_norm": 1.0
  },
  "hardware": {
    "device": "auto",
    "precision": "fp16"
  }
}

Reinforcement Learning

{
  "model": {
    "type": "ppo",
    "policy_network": {
      "layers": [256, 256],
      "activation": "relu"
    },
    "value_network": {
      "layers": [256, 256],
      "activation": "relu"
    }
  },
  "training": {
    "num_steps": 2048,
    "num_epochs": 10,
    "batch_size": 64,
    "learning_rate": 0.0003,
    "gamma": 0.99,
    "gae_lambda": 0.95
  },
  "hardware": {
    "device": "auto"
  }
}

Training Script Template

Here’s a comprehensive training script template:

import neurenix
from neurenix.nn import Module
from neurenix.optim import Adam
from neurenix.data import DataLoader
import os

# Load configuration
config_path = os.environ.get('NEURENIX_CONFIG', 'config.json')
config = neurenix.load_config(config_path)

# Set device
neurenix.set_device(
    config["hardware"]["device"],
    precision=config["hardware"].get("precision", "float32")
)

# Load data
print("Loading datasets...")
train_data = neurenix.load_dataset("data/train")
val_data = neurenix.load_dataset("data/val")

train_loader = DataLoader(
    train_data,
    batch_size=config["training"]["batch_size"],
    shuffle=True
)
val_loader = DataLoader(
    val_data,
    batch_size=config["training"]["batch_size"]
)

# Create model
print("Creating model...")
model = neurenix.create_model(config["model"])

# Create optimizer
optimizer = Adam(
    model.parameters(),
    lr=config["training"]["learning_rate"]
)

# Create scheduler if specified
scheduler = None
if "scheduler" in config["training"]:
    scheduler = neurenix.create_scheduler(
        optimizer,
        config["training"]["scheduler"]
    )

# Train
print("Starting training...")
neurenix.train(
    model,
    train_loader,
    val_loader,
    optimizer=optimizer,
    scheduler=scheduler,
    epochs=config["training"]["epochs"],
    log_dir="logs"
)

# Save model
print("Saving model...")
neurenix.save_model(model, "models/model.nrx")
print("Training completed!")

Best Practices

1. Use Version Control

Track your training configurations:

git add config.json train.py
git commit -m "Training configuration for experiment 1"

2. Save Checkpoints

Modify your training script to save checkpoints:

neurenix.train(
    model,
    train_loader,
    val_loader,
    checkpoint_dir="checkpoints",
    save_frequency=5  # Save every 5 epochs
)

3. Log Everything

Enable comprehensive logging:

neurenix.train(
    model,
    train_loader,
    val_loader,
    log_dir="logs",
    log_metrics=["loss", "accuracy", "f1_score"],
    log_frequency=10  # Log every 10 batches
)

4. Validate Regularly

Run validation during training to catch overfitting:

neurenix run train.py

# Monitor validation metrics
neurenix monitor --log-dir logs --metrics val_loss,val_accuracy

5. Resume from Checkpoints

# Load checkpoint if it exists
if os.path.exists("checkpoints/latest.nrx"):
    model = neurenix.load_model("checkpoints/latest.nrx")
    print("Resumed from checkpoint")

Troubleshooting

Out of Memory

# Reduce batch size
neurenix run train.py --batch-size 16

# Use mixed precision
# Add to config.json: "precision": "mixed"

Slow Training

# Verify GPU usage
neurenix hardware --info

# Use auto device selection
neurenix run train.py --device auto

Poor Convergence

# Try different learning rate
neurenix run train.py --learning-rate 0.0001

# Increase epochs
neurenix run train.py --epochs 100

Commands

Overview

Quick Start

Training Workflows

Basic Training

Custom Hyperparameters

GPU Training

Distributed Training

Monitoring Training

Configuration Examples

Image Classification

Natural Language Processing

Reinforcement Learning

Training Script Template

Best Practices

1. Use Version Control

2. Save Checkpoints

3. Log Everything

4. Validate Regularly

5. Resume from Checkpoints

Troubleshooting

Out of Memory

Slow Training

Poor Convergence

See Also

​Overview

​Quick Start

​Training Workflows

​Basic Training

​Custom Hyperparameters

​GPU Training

​Distributed Training

​Monitoring Training

​Configuration Examples

​Image Classification

​Natural Language Processing

​Reinforcement Learning

​Training Script Template

​Best Practices

​1. Use Version Control

​2. Save Checkpoints

​3. Log Everything

​4. Validate Regularly

​5. Resume from Checkpoints

​Troubleshooting

​Out of Memory

​Slow Training

​Poor Convergence

​See Also

Overview

Quick Start

Training Workflows

Basic Training

Custom Hyperparameters

GPU Training

Distributed Training

Monitoring Training

Configuration Examples

Image Classification

Natural Language Processing

Reinforcement Learning

Training Script Template

Best Practices

1. Use Version Control

2. Save Checkpoints

3. Log Everything

4. Validate Regularly

5. Resume from Checkpoints

Troubleshooting

Out of Memory

Slow Training

Poor Convergence

See Also