CLS Compilation Pipeline: From Source to Interpreter

Every .clsx file follows the same path from source text to running program: the Lexer converts characters into a token stream, the Parser assembles that stream into a typed AST, optional middleware passes (TypeChecker, NameResolver, Optimizer) validate and refine the AST at compile time, and finally the tree-walking interpreter visits the AST nodes recursively to evaluate expressions and execute statements. This page traces each stage in detail, notes the two available backends, describes the packaging flow for production builds, and closes with the current milestone status.

Pipeline Diagram

.clsx  →  Lexer  →  Parser  →  AST  →  Tree-walker (Interpreter)
                                    → JSON backend (AST dump)
                                    → WASM codegen (future → .clbin)

The middleware passes sit between the Parser and the Interpreter and are all optional — the interpreter can execute a raw AST directly, which is how clx run works in its fast path.

Stage 1: Lexer

Source: cls-core/src/frontend/lexer.rs The Lexer struct holds the source as a Vec<char> and walks it character by character, tracking line and col for every token it emits. Its public interface is:

let mut lexer = Lexer::new(source_str);
let tokens: Vec<SpannedToken> = lexer.tokenize()?;

Each SpannedToken pairs a Token variant with a Span (start_line, start_col, end_line, end_col). The lexer handles:

Strings — single ', double ", and backtick ` delimiters, with escape sequences (\n, \t, \r, \\, \', \")
Numbers — integer and float literals
Identifiers and keywords — Keyword variants for the full CLS keyword set; falls back to Token::Identifier for user names
CMX markup — when < is followed by an identifier or uppercase letter, peek_is_cmx_start() returns true and lex_cmx() tokenizes the entire tag, storing CmxTokens in an internal cmx_buffer that is drained on subsequent calls to next_token
Operators and symbols — two- and three-character lookahead for ==, !=, ->, ::, ++, \|, etc.
Comments — # causes the lexer to skip to end-of-line; all annotation content (# @description …) is consumed here and is later recovered by maptype by re-scanning the raw source text

Stage 2: Parser

Source: cls-core/src/frontend/parser.rs The Parser is a recursive-descent parser that consumes a Vec<SpannedToken> and produces a Module AST node:

let mut parser = Parser::new(tokens);
let module: Module = parser.parse()?;

Module holds a flat Vec<Statement>. The parser dispatches on the current token to choose the right parse_* method:

Token	Parser method
`var` / `const` / `let`	`parse_var_decl` / `parse_const_decl`
`function` / `async`	`parse_function_decl`
`class`	`parse_class_decl`
`interface`	`parse_interface_decl`
`structure`	`parse_structure_decl`
`if`	`parse_if_statement` (with `elif` chains)
`while` / `loop` / `for`	`parse_while_statement` / `parse_loop_statement` / `parse_for_statement`
`switch`	`parse_switch_statement`
`try`	`parse_try_statement`
`import` / `from`	`parse_import` / `parse_from_import`
`return`	`parse_return_statement`
CMX expression	`parse_cmx_element`

On a parse error the parser calls recover() to advance past the problematic token and attempts to continue, collecting multiple diagnostics in a single pass.

Stage 3: AST

Source: cls-core/src/frontend/ast.rs The AST is a set of Rust enums. The top-level variants are:

Module — root node, owns a Vec<Statement> and a Span
Statement — covers all statement forms: VarDecl, ConstDecl, FunctionDecl, ClassDecl, InterfaceDecl, StructureDecl, TypeAlias, EnumDecl, ModuleDecl, NamespaceDecl, If, While, Loop, For, ForEach, Switch, Try, With, Return, Import, FromImport, Include, Break, Continue, Expression, Cmx, Config, Meta
Expression — Binary, Unary, Call, MemberAccess, Index, Array, Tuple, Record, ArrowFunction, Conditional, Assignment, Identifier, NamespaceAccess, Parenthesized, StringInterpolation, Cmx, Await, Literal

Every compound node embeds a Span so that diagnostics and go-to-definition can report precise source locations.

Stage 4: Middleware (Optional)

Middleware passes are compile-time only. They do not mutate the AST (except the Optimizer) and all run before the interpreter.

TypeChecker

Source: cls-core/src/middleware/typeck.rs

let mut checker = TypeChecker::new(types_config);
checker.check(&module)?;
let diagnostics = checker.diagnostics();

TypeChecker maintains a stack of scopes (Vec<HashMap<String, Type>>) and walks every statement and expression. It:

Validates assignment compatibility between declared and inferred types
Resolves generic type parameters at call sites
Checks function call arities
Reports type mismatches and unknown names as Diagnostic { severity: Error | Warning }

All core intrinsics (print, input, int, float, str, bool, len, type, now, exit, sleep, throw) are pre-registered in the root scope at construction time. Type checking is gated by TypesConfig::check — when check: false the pass returns immediately with no diagnostics.

NameResolver

Source: cls-core/src/middleware/resolver.rs

let mut resolver = NameResolver::new();
resolver.resolve(&module)?;
let diagnostics = resolver.diagnostics();

The NameResolver verifies that every identifier reference has a reachable declaration in the enclosing scope chain. Errors are collected in resolver.diagnostics() and published as LSP diagnostics by the server.

Optimizer

Source: cls-core/src/middleware/optimizer.rs

let optimizer = Optimizer::new();
optimizer.optimize(&mut module);

The Optimizer performs in-place AST rewrites. It recursively visits every Statement and Expression, currently traversing the full tree (constant folding infrastructure is present but the folding pass is marked as a planned TODO). All control-flow forms — If, While, Loop, For, ForEach, Switch, Try, With — are traversed so that any future rewrite pass sees every expression in the program.

Stage 5: Tree-walker Interpreter

Source: cls-runtime/src/interpreter.rs The Interpreter is a tree-walking executor. It holds an Environment chain (a linked list of scope frames) and visits each Statement and Expression node recursively:

// Simplified call from a node
let mut interpreter = Interpreter::new(resolver, intrinsics);
interpreter.run(&module)?;

Key execution behaviors:

Environment chain — variable lookup walks from the innermost scope outward; return unwinds the call stack via a Rust Err(ControlFlow::Return(value))
Function calls — creates a new scope frame, binds arguments to parameter names, executes the function body, and unwraps the return value
Module loading — import "mod" calls Interpreter::load_module_source, which invokes the node-configured ModuleResolver to retrieve source text, runs it through the full Lexer → Parser → Interpreter pipeline in a fresh scope, and caches the exported Value::Record for subsequent imports
CMX evaluation — CMX expressions produce a Value::Record with tag, props, and children keys for lowercase tags; uppercase tags are resolved as function calls
Async/await — async functions and await expressions are tracked in the interpreter state; the Tokio runtime (in the node) drives async execution

Backends

JSON Backend

clx ast src/main.clsx --json

Dumps the fully-parsed AST as pretty-printed JSON to stdout. Useful for debugging the parser and for tooling that consumes the AST without running the interpreter.

WASM Backend (Planned)

The WASM codegen backend lives in cls-core/src/backend/ and will produce .clbin bytecode files. A .clbin will be embedded inside a .clsapp or .clslib zip and executed by a WASM-capable clxr runtime, enabling CLS programs to run in browsers and serverless environments without shipping the tree-walker.

Packaging Flow

clx build serialises a project into a self-contained .clsapp archive:

Resolve all imports

Starting from the entry point, clx build recursively resolves every import statement using the same ModuleResolver that the interpreter uses at runtime.

Serialize ASTs

Each resolved module’s AST is serialised (currently as JSON via the JSON backend) and written into the zip archive under its module path.

Bundle resources

Static assets referenced via res:// paths are copied into the archive.

Write the manifest

The cls.json manifest is embedded in the archive root so clxr can find the entry point.

At runtime, clxr app.clsapp opens the zip via the VfsResolver and resolves the entry point through the res:// protocol — the same mechanism used internally by Interpreter::load_module_source during import resolution.

Milestones

Phase	Description	Status
F1	Workspace + crates + base nodes	✅ Complete
F2	Pipeline: lexer → parser → tree-walker	✅ Complete
F3	Type checker + name resolver + optimizer	✅ Complete
F4	Stdlib: math, json, fs, http, intrinsics	✅ Complete
F5a	ModuleResolver + imports	✅ Complete
F5b	Exports + user modules	✅ Complete
F6	Migration `ccls` → `clx`, `.ccls` → `.clsx`	✅ Complete
F7	VFS + ClsLib indexing	✅ Complete
—	LSP server + VS Code extension	✅ Complete
—	Type maps + autocompletion	✅ Complete
—	Async/await syntax	✅ Complete
—	Structure + Interface	✅ Complete
—	CMX with reference lookup	✅ Complete
—	Error system with traceback	✅ Complete
Future	WASM backend (`.clbin`)	🚧 Planned
Future	WASM runtime in `clxr`	🚧 Planned
Future	Registry + package publishing	🚧 Planned

VS Code Extension

Architecture

CLS Compilation Pipeline: From Source to Interpreter

Pipeline Diagram

Stage 1: Lexer

Stage 2: Parser

Stage 3: AST

Stage 4: Middleware (Optional)

TypeChecker

NameResolver

Optimizer

Stage 5: Tree-walker Interpreter

Backends

JSON Backend

WASM Backend (Planned)

Packaging Flow

Milestones

Build docs developers (and LLMs) love

VS Code Extension

Architecture

Documentation Index

​Pipeline Diagram

​Stage 1: Lexer

​Stage 2: Parser

​Stage 3: AST

​Stage 4: Middleware (Optional)

​TypeChecker

​NameResolver

​Optimizer

​Stage 5: Tree-walker Interpreter

​Backends

​JSON Backend

​WASM Backend (Planned)

​Packaging Flow

​Milestones

Build docs developers (and LLMs) love

Pipeline Diagram

Stage 1: Lexer

Stage 2: Parser

Stage 3: AST

Stage 4: Middleware (Optional)

TypeChecker

NameResolver

Optimizer

Stage 5: Tree-walker Interpreter

Backends

JSON Backend

WASM Backend (Planned)

Packaging Flow

Milestones