Compression - th0th

Overview

th0th’s compression system reduces context size by 70-98% while preserving code structure and readability. Unlike naive truncation, it intelligently extracts signatures, types, and architecture while removing implementation details.

Why compression matters: AI models have token limits (4K-200K). Sending compressed context lets you include 10x more files in a single query.

Compression Strategy

Structure-Preserving Compression

The core principle: Keep what AI needs to understand, remove what it doesn’t.

// Original code (450 tokens)
async function authenticateUser(credentials: Credentials): Promise<User> {
  const { email, password } = credentials;
  
  // Find user in database
  const user = await db.users.findUnique({
    where: { email },
    include: { roles: true, permissions: true }
  });
  
  if (!user) {
    throw new AuthenticationError('User not found');
  }
  
  // Verify password hash
  const isValid = await bcrypt.compare(password, user.passwordHash);
  
  if (!isValid) {
    throw new AuthenticationError('Invalid password');
  }
  
  // Update last login timestamp
  await db.users.update({
    where: { id: user.id },
    data: { lastLoginAt: new Date() }
  });
  
  return user;
}

// Compressed code (45 tokens, 90% reduction)
async function authenticateUser(credentials: Credentials): Promise<User>

What Gets Preserved?

Function/method signatures
Type definitions and interfaces
Class declarations
Import/export statements
JSDoc comments
Public API surface

What Gets Removed?

Function bodies and implementation logic
Local variables and temporary state
Error handling details
Database queries
Logging and debugging code
Internal helper functions

Compression Strategies

th0th supports multiple compression strategies via the CompressionStrategy enum:

CODE_STRUCTURE
SEMANTIC_DEDUP
IDENTITY

Default and most effectiveExtracts structural elements only:

// Input
export class UserService {
  private db: Database;
  
  constructor(db: Database) {
    this.db = db;
  }
  
  async createUser(data: UserData): Promise<User> {
    // validation logic
    const user = await this.db.users.create({ data });
    // post-creation hooks
    return user;
  }
}

// Output
export class UserService
async createUser(data: UserData): Promise<User>

Reduction: 85-95%

Remove duplicate semantic contentEliminates repeated patterns:

// Input (repetitive code)
function handleUserCreated(user) { /* ... */ }
function handleUserUpdated(user) { /* ... */ }
function handleUserDeleted(user) { /* ... */ }

// Output (deduplicated)
function handleUserCreated(user) { /* ... */ }
// ... 2 similar handlers

Reduction: 30-50%

Implementation Details

CodeCompressor Architecture

class CodeCompressor implements ICompressor {
  async compress(
    content: string,
    strategy?: CompressionStrategy
  ): Promise<CompressedContent> {
    // 1. Extract structure (imports, classes, functions)
    const structure = this.extractStructure(content);
    
    // 2. Convert to compressed text
    const compressed = this.structureToText(structure);
    
    // 3. Calculate metrics
    const originalTokens = estimateTokens(content);
    const compressedTokens = estimateTokens(compressed);
    
    // 4. Return with metadata
    return CompressedContentModel.create(
      content,
      compressed,
      strategy,
      language,
      preservedElements
    );
  }
}

Structure Extraction Process

Parse Lines

Split content into lines and track state (in comment, in class, etc.)

Detect Blocks

Identify imports, interfaces, classes, functions using regex patterns

Track Brace Depth

Monitor { and } to understand scope boundaries

Extract Signatures

Keep declarations, remove bodies (everything after opening {)

Preserve Comments

Include JSDoc and documentation comments

Pattern Matching

The compressor recognizes patterns across multiple languages:

const patterns = {
  // TypeScript/JavaScript
  class: /^\s*(export\s+)?(class|interface|type|enum)\s+\w+/,
  function: /^\s*(export\s+)?(async\s+)?function\s+\w+/,
  arrow: /^\s*(export\s+)?const\s+\w+\s*=\s*(async\s*)?\(/,
  
  // Python
  pythonDef: /^\s*(async\s+)?def\s+\w+/,
  pythonClass: /^\s*class\s+\w+/,
  
  // Go
  goFunc: /^\s*func\s+(\([^)]+\)\s+)?\w+/,
  
  // Rust
  rustFn: /^\s*(pub\s+)?(async\s+)?fn\s+\w+/,
  rustStruct: /^\s*(pub\s+)?struct\s+\w+/,
};

Compression Output

CompressedContent Model

interface CompressedContent {
  original: string;           // Original uncompressed content
  compressed: string;         // Compressed version
  compressionRatio: number;   // e.g., 0.92 = 92% reduction
  tokensSaved: number;        // Tokens saved
  strategy: CompressionStrategy;
  language: string;           // Detected language
  metadata: {
    originalTokens: number;
    compressedTokens: number;
    originalSize: number;     // bytes
    compressedSize: number;   // bytes
    preservedElements: string[];  // List of what was kept
  };
}

Example Output

TypeScript Class Compression

Input (28 lines, ~520 tokens):

import { Database } from './database';
import { User, UserData } from './types';

export class UserService {
  private db: Database;
  private logger: Logger;
  
  constructor(db: Database, logger: Logger) {
    this.db = db;
    this.logger = logger;
  }
  
  async createUser(data: UserData): Promise<User> {
    this.logger.info('Creating user', { email: data.email });
    
    const existing = await this.db.users.findUnique({
      where: { email: data.email }
    });
    
    if (existing) {
      throw new Error('User already exists');
    }
    
    const user = await this.db.users.create({ data });
    this.logger.info('User created', { id: user.id });
    
    return user;
  }
}

Output (7 lines, ~85 tokens, 84% reduction):

import { Database } from './database';
import { User, UserData } from './types';

export class UserService
async createUser(data: UserData): Promise<User>

Python Module Compression

Input (35 lines, ~610 tokens):

import bcrypt
from typing import Optional
from .database import Database
from .models import User, UserData

class UserService:
    def __init__(self, db: Database):
        self.db = db
    
    async def authenticate(
        self, 
        email: str, 
        password: str
    ) -> Optional[User]:
        """Authenticate user with email and password."""
        user = await self.db.users.find_one({"email": email})
        
        if not user:
            return None
        
        if not bcrypt.checkpw(password.encode(), user.password_hash):
            return None
        
        return user
    
    async def create_user(self, data: UserData) -> User:
        """Create a new user."""
        hashed = bcrypt.hashpw(data.password.encode(), bcrypt.gensalt())
        user_dict = {**data.dict(), "password_hash": hashed}
        user = await self.db.users.insert_one(user_dict)
        return User(**user)

Output (9 lines, ~120 tokens, 80% reduction):

import bcrypt
from typing import Optional
from .database import Database
from .models import User, UserData

class UserService
async def authenticate(self, email: str, password: str) -> Optional[User]
async def create_user(self, data: UserData) -> User

Performance Characteristics

Compression Ratios by Language

TypeScript

85-95% reductionRich type info preserved

Python

75-85% reductionType hints preserved

Go

80-90% reductionInterface defs preserved

Rust

85-95% reductionTrait bounds preserved

Speed

Small File (< 500 lines)
Medium File (500-2000 lines)
Large File (> 2000 lines)

< 10ms per fileRegex-based parsing is very fast

Compression is on-demand by default. Files are only compressed when requested, not during indexing.

Use Cases

1. Context Window Optimization

Problem: AI model has 8K token limit, project has 50 relevant files totaling 100K tokens. Solution: Compress all 50 files to ~10K tokens, fit everything in context.

const files = await search('authentication logic', projectId);

// Compress each file
const compressed = await Promise.all(
  files.map(f => compressor.compress(f.content))
);

// Total tokens: 100K → 10K (90% reduction)

2. Codebase Overview

Problem: Need to understand project structure without reading thousands of lines. Solution: Compress entire codebase to show API surface only.

const overview = await th0th.compress({
  projectPath: '/path/to/project',
  strategy: 'code_structure',
  include: ['src/**/*.ts']
});

// AI sees: all classes, functions, types (no implementation)

3. Incremental Context Building

Problem: As conversation progresses, context fills up with old code. Solution: Compress older context, keep recent files uncompressed.

// Recent file (full detail)
const currentFile = await readFile('auth.ts');

// Related files (compressed for context)
const relatedCompressed = await Promise.all(
  relatedFiles.map(f => compressor.compress(f))
);

// Send to AI: current (full) + related (compressed)

Advanced Features

Language Detection

Automatic language detection for language-specific parsing:

function detectLanguage(content: string): string {
  // Dart-specific patterns
  if (/\b(library|part of|dart:|package:)/.test(content)) {
    return 'dart';
  }
  
  // TypeScript vs JavaScript
  if (content.includes('interface ') || content.includes(': ')) {
    return 'typescript';
  }
  if (content.includes('import ') && content.includes('from ')) {
    return 'javascript';
  }
  
  // Python
  if (content.includes('def ') && content.includes(':')) {
    return 'python';
  }
  
  // Go, Rust, etc...
  return 'unknown';
}

Decompression

Compression is reversible (original is stored):

const compressed = await compressor.compress(code);

// Later: get original back
const original = await compressor.decompress(compressed);
assert(original === code);  // true

Compression Estimation

Preview compression ratio without full compression:

const ratio = await compressor.estimateCompression(content);
console.log(`Expected ${(ratio * 100).toFixed(0)}% reduction`);
// => "Expected 87% reduction"

Best Practices

When to Compress

Large files (> 500 lines): Always compressBackground context: Compress related filesCodebase overviews: Compress all filesHistorical context: Compress older messages

When NOT to Compress

Active editing: Keep full detail for file being modifiedSmall files (< 100 lines): Compression overhead not worth itConfig files: Often need full content (env vars, etc.)Documentation: Prose doesn’t compress well

Optimization Tips

Batch compression: Process multiple files in parallelCache results: Compressed output rarely changesProgressive detail: Start compressed, expand on demandMonitor tokens: Use estimateTokens() to track usage

Metrics & Monitoring

th0th tracks compression performance via MetricsCollector:

interface CompressionMetrics {
  totalCompressions: number;
  totalTokensSaved: number;
  averageRatio: number;
  compressionsByLanguage: Record<string, {
    count: number;
    avgRatio: number;
  }>;
}

// Example output
const metrics = MetricsCollector.getCompressionMetrics();
console.log(metrics);
// {
//   totalCompressions: 1247,
//   totalTokensSaved: 892341,
//   averageRatio: 0.89,  // 89% reduction
//   compressionsByLanguage: {
//     typescript: { count: 856, avgRatio: 0.91 },
//     python: { count: 234, avgRatio: 0.82 },
//     go: { count: 157, avgRatio: 0.88 }
//   }
// }

Limitations

Not suitable for:

Algorithmic code: Implementation details matter
Configuration: Need exact values
Data structures: Field names and types insufficient
Tests: Assertions require full logic

Compression works best for understanding architecture and API surfaces. For detailed debugging, use uncompressed code.

Future Enhancements

Planned improvements:

AST-based parsing: More accurate than regex for complex code
Semantic compression: Use embeddings to identify truly redundant code
Adaptive compression: Adjust ratio based on available context budget
Comment preservation: Keep important TODO/FIXME/NOTE comments
Call graph awareness: Preserve function relationships

Semantic Search

Find relevant code before compressing it

Architecture

How compression fits into the overall system

Smart Chunking

Chunking strategy affects compression efficiency

API Reference

Complete compression API docs

Get Started

Integration

Core Concepts

Configuration

Operations

Documentation Index

​Overview

​Compression Strategy

​Structure-Preserving Compression

​Compression Strategies

​Implementation Details

​CodeCompressor Architecture

​Structure Extraction Process

​Pattern Matching

​Compression Output

​CompressedContent Model

​Example Output

​Performance Characteristics

​Compression Ratios by Language

TypeScript

Python

Go

Rust

​Speed

​Use Cases

​1. Context Window Optimization

​2. Codebase Overview

​3. Incremental Context Building

​Advanced Features

​Language Detection

​Decompression

​Compression Estimation

​Best Practices

When to Compress

When NOT to Compress

Optimization Tips

​Metrics & Monitoring

​Limitations

​Future Enhancements

​Related Topics

Semantic Search

Architecture

Smart Chunking

API Reference

Build docs developers (and LLMs) love

Overview

Compression Strategy

Structure-Preserving Compression

Compression Strategies

Implementation Details

CodeCompressor Architecture

Structure Extraction Process

Pattern Matching

Compression Output

CompressedContent Model

Example Output

Performance Characteristics

Compression Ratios by Language

Speed

Use Cases

1. Context Window Optimization

2. Codebase Overview

3. Incremental Context Building

Advanced Features

Language Detection

Decompression

Compression Estimation

Best Practices

Metrics & Monitoring

Limitations

Future Enhancements

Related Topics