Parallel Execution

HTTPSpec can run multiple test files in parallel using a configurable thread pool, dramatically improving test suite performance while maintaining sequential execution within each file.

How Parallel Execution Works

Thread Pool Architecture

When you run HTTPSpec with multiple test files, it:

Discovers all .http and .httpspec files
Creates a thread pool with N worker threads
Distributes files across the thread pool
Executes each file independently in parallel
Collects results and reports summary

Implementation in main.zig:57-71:

// Set up thread pool and reporter.
var pool: std.Thread.Pool = undefined;
try pool.init(.{
    .allocator = allocator,
    .n_jobs = threads,
});
defer pool.deinit();

var wg: std.Thread.WaitGroup = .{};
var reporter = TestReporter.BasicReporter.init();

// Run all tests in parallel.
for (files.items) |path| {
    pool.spawnWg(&wg, runTest, .{ allocator, &reporter, path, stderr });
}
wg.wait();

What Runs in Parallel

Different Files

Each .http or .httpspec file runs in its own thread from the pool.

Example:

tests/
├── user_tests.http      # Runs in parallel
├── order_tests.http     # Runs in parallel
└── auth_tests.http      # Runs in parallel

All three files execute simultaneously on different threads.

What Runs Sequentially

Requests Within a File

Requests within the same file always execute sequentially, top to bottom.

Example in user_tests.http:

### Request 1 - Executes first
POST https://api.example.com/users

### Request 2 - Executes after Request 1
GET https://api.example.com/users/123

### Request 3 - Executes after Request 2
DELETE https://api.example.com/users/123

Configuring Thread Pool Size

Control parallelism with the HTTP_THREAD_COUNT environment variable:

# Default: 1 thread (sequential execution)
httpspec ./tests/

# 4 parallel threads
HTTP_THREAD_COUNT=4 httpspec ./tests/

# 8 parallel threads
HTTP_THREAD_COUNT=8 httpspec ./tests/

# 16 parallel threads for maximum performance
HTTP_THREAD_COUNT=16 httpspec ./tests/

Finding the Optimal Thread Count

The optimal thread count depends on:

Number of CPU cores: Start with your CPU core count
Number of test files: No benefit to more threads than files
API rate limits: Too many threads might hit rate limits
Network I/O: HTTP requests are I/O-bound, so you can use more threads than CPU cores

Recommended starting points:

Scenario	Recommended Thread Count
Local development	4-8 threads
CI/CD pipeline	8-16 threads
Rate-limited APIs	2-4 threads
< 10 test files	Number of files

Since HTTP tests are I/O-bound (waiting for network responses), you can safely use more threads than CPU cores. Start with 2x your core count and adjust based on performance.

Performance Benefits

Example: 12 Test Files

Without parallelization (1 thread):

File 1: 2s
File 2: 1.5s
File 3: 3s
...
Total: 24s

With parallelization (4 threads):

Thread 1: File 1 (2s) + File 5 (2s) + File 9 (1s) = 5s
Thread 2: File 2 (1.5s) + File 6 (3s) + File 10 (1s) = 5.5s
Thread 3: File 3 (3s) + File 7 (2s) + File 11 (1s) = 6s
Thread 4: File 4 (2.5s) + File 8 (1.5s) + File 12 (1s) = 5s
Total: ~6s (4x speedup!)

Real-World Example

# Slow: Run tests sequentially
time httpspec ./test_files/
# Result: 45.2 seconds

# Fast: Run with 8 threads
time HTTP_THREAD_COUNT=8 httpspec ./test_files/
# Result: 7.1 seconds (6.4x speedup!)

Test Isolation

Each test file runs in complete isolation:

Memory Isolation

Every test file gets its own arena allocator in main.zig:113-116:

// Create arena allocator for this test to provide memory isolation
var arena = std.heap.ArenaAllocator.init(base_allocator);
defer arena.deinit(); // Automatically frees all test allocations
const allocator = arena.allocator();

This ensures:

No memory leaks between tests
Automatic cleanup when test completes
Tests cannot interfere with each other’s memory

Request Independence

Tests running in parallel cannot share state. Each file must be self-contained.

### user_tests.http
POST https://api.example.com/users
{"name": "Alice"}

//# status == 201

GET https://api.example.com/users/123
//# status == 200

Best Practices

1. Design for Parallelism

Make each test file independent:

tests/
├── create_user_alice.http    # Creates user "alice"
├── create_user_bob.http      # Creates user "bob"
└── create_user_charlie.http  # Creates user "charlie"

Each file creates a different user, so they can run in parallel without conflicts.

2. Use Unique Identifiers

Avoid conflicts by using unique data:

### Create user with unique email
POST https://api.example.com/users
Content-Type: application/json

{
  "name": "Test User",
  "email": "test-{{ $timestamp }}@example.com"
}

//# status == 201

Note: HTTPSpec doesn’t currently support variable substitution or timestamp generation. Use unique static values or implement this in your API.

3. Avoid Shared Resources

Don’t have multiple tests modify the same resource:

### test_1.http
PUT https://api.example.com/settings/feature_a
{"enabled": true}

### test_2.http
PUT https://api.example.com/settings/feature_b
{"enabled": true}

Keep dependent requests in the same file:

user_workflow.http

### Create user
POST https://api.example.com/users
{"name": "Alice"}

//# status == 201

### Update user (depends on creation)
PUT https://api.example.com/users/123
{"name": "Alice Updated"}

//# status == 200

### Delete user (depends on existence)
DELETE https://api.example.com/users/123

//# status == 204

5. Monitor Performance

Measure speedup as you add threads:

# Benchmark different thread counts
for threads in 1 2 4 8 16; do
  echo "Testing with $threads threads:"
  time HTTP_THREAD_COUNT=$threads httpspec ./tests/
done

Look for diminishing returns:

thread:  45s (baseline)
threads: 24s (1.9x speedup)
threads: 13s (3.5x speedup)
threads:  8s (5.6x speedup)
threads: 7s (6.4x speedup) ← diminishing returns
threads: 7s (6.4x speedup) ← no improvement

6. Consider API Rate Limits

Some APIs limit concurrent requests:

# GitHub API allows ~60 req/min unauthenticated
# Use fewer threads to stay under limit
HTTP_THREAD_COUNT=2 httpspec ./github_tests/

# Your internal API might handle more
HTTP_THREAD_COUNT=16 httpspec ./internal_tests/

Handling Failures

Independent Failure Handling

When one test file fails, others continue:

tests/
├── test_1.http  # Passes ✓
├── test_2.http  # Fails ✗
├── test_3.http  # Passes ✓
└── test_4.http  # Passes ✓

Output:

[Fail] in tests/test_2.http:5 Expected status 200, got 404

Pass: 3
Fail: 1

All four files ran in parallel, and three succeeded despite one failure.

Early Exit Within File

Failures only stop the current file (see Sequential Testing):

test_2.http

### Request 1
GET https://api.example.com/endpoint1
//# status == 200  # Fails!

### Request 2
GET https://api.example.com/endpoint2  # Never executes

But test_1.http, test_3.http, and test_4.http complete successfully in parallel.

Implementation Details

Thread Count Parsing

From main.zig:26:

// Determine thread count from environment.
const threads = std.process.parseEnvVarInt("HTTP_THREAD_COUNT", usize, 10) catch 1;

Defaults to 1 thread if:

Environment variable not set
Invalid value provided
Parse error occurs

File Discovery

HTTPSpec recursively finds all test files in main.zig:161-185:

fn listHttpFiles(allocator: std.mem.Allocator, dir: []const u8) ![][]const u8 {
    var files: std.ArrayList([]const u8) = .empty;
    
    var dir_entry = try std.fs.cwd().openDir(dir, .{ .iterate = true });
    var it = dir_entry.iterate();
    while (try it.next()) |entry| {
        if (entry.kind == .directory) {
            // Recurse into subdirectories
            const sub_files = try listHttpFiles(allocator, subdir);
            for (sub_files) |file| try files.append(allocator, file);
        } else if (std.mem.eql(u8, std.fs.path.extension(entry.name), ".http") or
            std.mem.eql(u8, std.fs.path.extension(entry.name), ".httpspec"))
        {
            // Add .http and .httpspec files
            try files.append(allocator, file_path);
        }
    }
    return files.toOwnedSlice(allocator);
}

Work Distribution

The thread pool distributes work automatically:

// Run all tests in parallel.
for (files.items) |path| {
    pool.spawnWg(&wg, runTest, .{ allocator, &reporter, path, stderr });
}
wg.wait();

Each pool.spawnWg call:

Finds an available thread in the pool
Assigns the file to that thread
Executes runTest function with the file path

The WaitGroup (wg) ensures the main thread waits for all tests to complete.

Comparison: Sequential vs Parallel

Aspect	Sequential (1 thread)	Parallel (N threads)
Speed	Slowest	N× faster (approx)
Resource usage	Minimal	Higher CPU/memory
Debugging	Easier	More complex
Setup	None required	Set `HTTP_THREAD_COUNT`
Use case	Development, debugging	CI/CD, large test suites
Dependencies	Can share state	Must be independent

Common Issues

Race Conditions

Two tests modifying the same resource can cause race conditions:

test_a.http

PUT https://api.example.com/counter
{"value": 100}

test_b.http

PUT https://api.example.com/counter
{"value": 200}

Since both run in parallel, the final counter value is unpredictable. Solution: Use different resources or make tests in the same file.

Resource Exhaustion

Too many threads can overwhelm your system:

# Bad: More threads than CPU cores + files
HTTP_THREAD_COUNT=128 httpspec ./tests/  # Only 5 files!

Solution: Match thread count to file count and system capacity.

API Rate Limiting

Parallel requests might trigger rate limits:

Error: 429 Too Many Requests

Solution: Reduce thread count or add delays between requests (not currently supported in HTTPSpec).

Tips for Large Test Suites

1. Organize by Domain

tests/
├── auth/
│   ├── login.http
│   └── logout.http
├── users/
│   ├── create.http
│   └── update.http
└── orders/
    ├── place_order.http
    └── cancel_order.http

2. Balance File Sizes

Distribute requests evenly across files for better parallelization:

file_1.http: 3 requests
file_2.http: 3 requests
file_3.http: 3 requests
file_4.http: 3 requests

With 4 threads, the unbalanced case still takes as long as file_1.http alone.

3. Use CI/CD Optimization

In continuous integration:

.github/workflows/test.yml

- name: Run HTTPSpec tests
  run: HTTP_THREAD_COUNT=16 httpspec ./tests/
  env:
    HTTP_THREAD_COUNT: 16

4. Profile Your Tests

Identify slow tests:

# Run with timing
for file in tests/*.http; do
  echo "Testing $file"
  time httpspec "$file"
done

Optimize or split slow tests into multiple files.

Next Steps

Sequential Testing

Learn how requests execute within files

Writing Tests

Best practices for organizing test files

Get Started

File Format

Command Reference

Guides

Examples

Reference

​How Parallel Execution Works

​Thread Pool Architecture

​What Runs in Parallel

Different Files

​What Runs Sequentially

Requests Within a File

​Configuring Thread Pool Size

​Finding the Optimal Thread Count

​Performance Benefits

​Example: 12 Test Files

​Real-World Example

​Test Isolation

​Memory Isolation

​Request Independence

​Best Practices

​1. Design for Parallelism

​2. Use Unique Identifiers

​3. Avoid Shared Resources

​4. Group Related Sequential Tests

​5. Monitor Performance

​6. Consider API Rate Limits

​Handling Failures

​Independent Failure Handling

​Early Exit Within File

​Implementation Details

​Thread Count Parsing

​File Discovery

​Work Distribution

​Comparison: Sequential vs Parallel

​Common Issues

​Race Conditions

​Resource Exhaustion

​API Rate Limiting

​Tips for Large Test Suites

​1. Organize by Domain

​2. Balance File Sizes

​3. Use CI/CD Optimization

​4. Profile Your Tests

​Next Steps

Sequential Testing

Writing Tests

Build docs developers (and LLMs) love

How Parallel Execution Works

Thread Pool Architecture

What Runs in Parallel

What Runs Sequentially

Configuring Thread Pool Size

Finding the Optimal Thread Count

Performance Benefits

Example: 12 Test Files

Real-World Example

Test Isolation

Memory Isolation

Request Independence

Best Practices

1. Design for Parallelism

2. Use Unique Identifiers

3. Avoid Shared Resources

4. Group Related Sequential Tests

5. Monitor Performance

6. Consider API Rate Limits

Handling Failures

Independent Failure Handling

Early Exit Within File

Implementation Details

Thread Count Parsing

File Discovery

Work Distribution

Comparison: Sequential vs Parallel

Common Issues

Race Conditions

Resource Exhaustion

API Rate Limiting

Tips for Large Test Suites

1. Organize by Domain

2. Balance File Sizes

3. Use CI/CD Optimization

4. Profile Your Tests

Next Steps