Schema Migrations

​

Overview

​

Migration System

​

How Migrations Work

• Version-numbered - Each migration has a unique version number (e.g., 00001, 00002)
• Scope-specific - Different database implementations (SQLite, PostgreSQL) have separate migrations
• Hash-verified - Migration file contents are hashed to detect tampering
• Sequential - Migrations must be applied in order
• Idempotent - Safe to run multiple times

​

Migration Files

<version>-<description>.<scope>.sql

• 00001-collections.sqlite.sql
• 00002-segments.sqlite.sql
• 00003-collection-dimension.sqlite.sql
• 00004-tenants-databases.sqlite.sql

​

Migration Subsystems

1. SysDB (chromadb/migrations/sysdb/) - System database for collections, segments, tenants
2. Embeddings Queue (chromadb/migrations/embeddings_queue/) - Write-ahead log for embeddings
3. MetaDB (chromadb/migrations/metadb/) - Metadata and configuration storage

​

Configuration

​

Migration Modes

from chromadb.config import Settings
import chromadb

# Apply migrations automatically (default)
client = chromadb.Client(Settings(
    migrations="apply"
))

# Validate migrations only (no changes)
client = chromadb.Client(Settings(
    migrations="validate"
))

# Skip migrations entirely
client = chromadb.Client(Settings(
    migrations="none"
))

export MIGRATIONS=apply  # or validate, none

​

Migration Mode Descriptions

• apply (default) - Automatically apply all pending migrations
• validate - Check that all migrations are applied, raise error if not
• none - Skip migration checks entirely (not recommended)

​

Hash Algorithm

client = chromadb.Client(Settings(
    migrations_hash_algorithm="sha256"  # or "md5" (default)
))

export MIGRATIONS_HASH_ALGORITHM=sha256

​

Upgrading Chroma

​

Standard Upgrade Process

pip install --upgrade chromadb

import chromadb

# Migrations apply automatically
client = chromadb.Client(Settings(
    is_persistent=True,
    persist_directory="./chroma_data"
))

​

Upgrading from Legacy Versions

# Install migration tool
pip install chroma-migrate

# Run migration
chroma-migrate

​

Version Compatibility

• Major versions (e.g., 1.0.0 → 2.0.0) - May require manual migration
• Minor versions (e.g., 0.4.0 → 0.5.0) - Automatic migrations
• Patch versions (e.g., 0.4.1 → 0.4.2) - No migrations needed

​

Migration Validation

​

Pre-Upgrade Validation

from chromadb.config import Settings
import chromadb

# This will raise an exception if migrations are needed
client = chromadb.Client(Settings(
    migrations="validate",
    is_persistent=True,
    persist_directory="./chroma_data"
))

​

Migration Status

import sqlite3

conn = sqlite3.connect("./chroma_data/chroma.sqlite3")
cursor = conn.cursor()

# View all applied migrations
cursor.execute("""
    SELECT dir, version, filename, hash
    FROM migrations
    ORDER BY dir, version
""")

for row in cursor.fetchall():
    print(f"{row[0]}: v{row[1]} - {row[2]} ({row[3][:8]}...)")

conn.close()

sysdb: v1 - 00001-collections.sqlite.sql (a1b2c3d4...)
sysdb: v2 - 00002-segments.sqlite.sql (e5f6g7h8...)
sysdb: v3 - 00003-collection-dimension.sqlite.sql (i9j0k1l2...)
...

​

Manual Migration Procedures

​

Backing Up Data

# Backup persistent directory
cp -r ./chroma_data ./chroma_data.backup

# Or backup just the database
cp ./chroma_data/chroma.sqlite3 ./chroma_data/chroma.sqlite3.backup

​

Applying Migrations Manually

from chromadb.db.impl.sqlite import SqliteDB
from chromadb.config import System, Settings

# Create system with migrations disabled
system = System(Settings(
    migrations="none",
    is_persistent=True,
    persist_directory="./chroma_data"
))

db = system.instance(SqliteDB)
db.start()

# Apply migrations
db.apply_migrations()

db.stop()

​

Rolling Back Migrations

1. Stop Chroma
2. Restore from backup
3. Downgrade Chroma version

# Restore backup
rm -rf ./chroma_data
mv ./chroma_data.backup ./chroma_data

# Install previous version
pip install chromadb==0.4.24

​

Migration Errors

​

UninitializedMigrationsError

client = chromadb.Client(Settings(
    migrations="apply"  # Will create migration table
))

​

UnappliedMigrationsError

# Apply pending migrations
client = chromadb.Client(Settings(
    migrations="apply"
))

​

InconsistentVersionError

1. Restore original migration files
2. Or restore from backup and re-migrate

​

InconsistentHashError

1. Restore original migration file from the correct Chroma version
2. Or restore from backup

​

Custom Migrations

​

Creating Custom Migrations

-- 99999-custom-indexes.sqlite.sql
CREATE INDEX IF NOT EXISTS idx_custom_field
ON collections(custom_field);

CREATE INDEX IF NOT EXISTS idx_created_at
ON segments(created_at);

chromadb/migrations/sysdb/99999-custom-indexes.sqlite.sql

​

Production Migration Strategy

​

Pre-Production Testing

1. Test in staging environment
   # Copy production data to staging
   cp -r /prod/chroma_data /staging/chroma_data
   
   # Test migration in staging
   cd /staging
   pip install chromadb==<new-version>
   python test_migration.py
   
2. Verify migration success
   import chromadb
   
   client = chromadb.Client(Settings(
       is_persistent=True,
       persist_directory="/staging/chroma_data"
   ))
   
   # Verify collections
   collections = client.list_collections()
   print(f"Found {len(collections)} collections")
   
   # Verify queries work
   for collection in collections:
       results = collection.get(limit=10)
       print(f"{collection.name}: {len(results['ids'])} items")
   
3. Performance testing
   import time
   
   # Benchmark queries
   start = time.time()
   results = collection.query(
       query_embeddings=test_embeddings,
       n_results=10
   )
   print(f"Query time: {time.time() - start:.3f}s")
   

​

Production Migration

1. Schedule maintenance window
2. Backup production data
   cp -r /prod/chroma_data /backups/chroma_data_$(date +%Y%m%d_%H%M%S)
   
3. Stop Chroma server
   systemctl stop chroma
   
4. Upgrade package
   pip install --upgrade chromadb
   
5. Apply migrations
   python apply_migrations.py
   
6. Verify migration
   python verify_migration.py
   
7. Start Chroma server
   systemctl start chroma
   
8. Monitor for issues
   tail -f /var/log/chroma/chroma.log
   

​

Rollback Plan

#!/bin/bash
# rollback.sh

echo "Rolling back Chroma..."

# Stop server
systemctl stop chroma

# Restore backup
rm -rf /prod/chroma_data
cp -r /backups/chroma_data_backup /prod/chroma_data

# Downgrade package
pip install chromadb==<previous-version>

# Start server
systemctl start chroma

echo "Rollback complete"

​

Best Practices

1. Always backup before upgrading - No exceptions
2. Test in staging first - Never upgrade production directly
3. Use migrations="validate" in CI/CD - Catch migration issues early
4. Monitor migration logs - Watch for warnings or errors
5. Keep migration history - Maintain records of when migrations were applied
6. Use consistent hash algorithm - Choose sha256 for production, stick with it
7. Schedule maintenance windows - Don’t migrate during peak traffic
8. Verify after migration - Test queries, check data integrity
9. Have rollback plan ready - Document and test rollback procedures
10. Follow release notes - Read migration notes for each version

​

Migration History

​

Version 0.4.0

• Introduced new migration system
• Added tenant and database support
• Migrated from legacy DuckDB format

​

Version 0.5.0

• Added collection dimension tracking
• Improved segment management
• Enhanced metadata indexing

​

Version 0.5.6

• Automatic purging improvements
• Storage optimization

​

Troubleshooting

​

Migration hangs or takes too long

1. Check database size:
   du -sh ./chroma_data
   
2. Monitor migration progress:
   tail -f chroma.log | grep -i migration
   
3. For very large databases, consider:
   • Increasing timeouts
   • Running during off-peak hours
   • Temporarily increasing resources

​

Database locked errors

1. Ensure Chroma is stopped
2. Check for stale connections:
   lsof ./chroma_data/chroma.sqlite3
   
3. Kill any processes holding locks

​

Corruption after failed migration

1. Restore from backup immediately
2. Investigate failure cause
3. Fix underlying issue
4. Re-attempt migration

​

Next Steps

• Configure Observability to monitor migrations
• Review Configuration options
• Optimize Performance after migrations