Troubleshooting

This guide helps you diagnose and resolve common issues when running Clementine.

Installation Issues

RISC Zero Installation Fails

Symptom: cargo-risczero installation fails

Problem: RISC Zero toolchain fails to install or wrong version is installed.Solution:

# Install the correct version (2.1.0)
curl -L https://risczero.com/install | bash
rzup install cargo-risczero 2.1.0
rzup install r0vm 2.1.0
rzup install rust 1.88.0

Verify the installation:

cargo risczero --version

Symptom: xcrun metal errors on macOS

Problem: Metal compiler errors when building on macOS.Solution:

# Install Xcode Command Line Tools
xcode-select --install

Also install Xcode from the App Store if the error persists.

Symptom: Compilation fails on Ubuntu with SSL errors

Problem: Missing system dependencies on Ubuntu/Debian.Solution:

sudo apt update
sudo apt install build-essential libssl-dev pkg-config

Runtime Issues

Server Won’t Start

Symptom: Stack overflow errors

Problem: Server crashes with stack overflow errors.Solution: Set the minimum stack size:

export RUST_MIN_STACK=33554432

Add this to your shell profile (~/.bashrc or ~/.zshrc) to make it permanent.

Symptom: Database connection errors

Problem: Cannot connect to PostgreSQL database.Solution:

Verify PostgreSQL is running:
```
docker ps | grep clementine-test-db
```

Check connection settings in your config file:

[db]
host = "localhost"
port = 5432
user = "clementine"
password = "clementine"
dbname = "clementine"

If using Docker, ensure the container is running:

docker run --name clementine-test-db \
  -e POSTGRES_USER=clementine \
  -e POSTGRES_PASSWORD=clementine \
  -e POSTGRES_DB=clementine \
  -p 5432:5432 \
  --restart always \
  -d postgres:15 \
  bash -c "exec docker-entrypoint.sh postgres -c 'max_connections=1000'"

Symptom: TLS certificate errors

Problem: Server fails to start with certificate-related errors.Solution:

Generate certificates for testing:
```
./scripts/generate_certs.sh
```

Verify certificate paths in config:

[rpc]
server_cert_path = "./certs/server/server.pem"
server_key_path = "./certs/server/server.key"
ca_cert_path = "./certs/ca/ca.pem"

Ensure certificate files exist and are readable:
```
ls -la certs/server/
```

For production, use properly signed certificates from a trusted CA, not self-signed certificates.

Symptom: Bitcoin node connection fails

Problem: Cannot connect to Bitcoin node.Solution:

Verify Bitcoin Core is running (v29.0 or later required)

Check RPC credentials in config:

[bitcoin]
rpc_url = "http://localhost:8332"
rpc_user = "bitcoin"
rpc_password = "bitcoin"
network = "regtest"  # or "testnet4" or "mainnet"

Ensure Bitcoin Core RPC is enabled in bitcoin.conf:

server=1
rpcuser=bitcoin
rpcpassword=bitcoin
rpcallowip=127.0.0.1

RPC Authentication Issues

Symptom: Client certificate verification fails

Problem: RPC calls are rejected with authentication errors.Solution: Clementine uses mutual TLS (mTLS) for authentication:

Verifier/Operator methods can only be called by the aggregator:
- Ensure aggregator is using the correct client certificate
- Path specified in aggregator_cert_path in config
Internal methods can only be called by the entity’s own certificate:
- Use the entity’s client certificate (client_cert_path)

Verify certificate configuration:

[rpc]
client_cert_path = "./certs/client/client.pem"
client_key_path = "./certs/client/client.key"
aggregator_cert_path = "./certs/aggregator/aggregator.pem"

The aggregator does not enforce client certificates but does use TLS for encryption.

Testing Issues

Test Failures

Symptom: Tests fail with RISC0 errors

Problem: Circuit-related tests fail.Solution:

Enable dev mode for testing:
```
export RISC0_DEV_MODE=1
```

Optionally download BitVM cache:

wget https://static.testnet.citrea.xyz/common/bitvm_cache_dev.bin -O bitvm_cache_dev.bin
export BITVM_CACHE_PATH=/path/to/bitvm_cache_dev.bin

Symptom: Unit tests pass but integration tests fail

Problem: Integration tests require more setup than unit tests.Solution:

Ensure PostgreSQL is running
Ensure Bitcoin node is running (for applicable tests)

Check that all environment variables are set:

export RISC0_DEV_MODE=1
export RUST_MIN_STACK=33554432

Run integration tests separately:
```
cargo test_integration
```

You can run unit and integration tests separately using the cargo aliases defined in .cargo/config.toml:

cargo test_unit_debug - Unit tests only
cargo test_integration - Integration tests only

Performance Issues

Slow Performance

Symptom: Circuit generation is very slow

Problem: BitVM cache generation takes a long time.Solution: Download pre-generated cache:

# For dev mode
wget https://static.testnet.citrea.xyz/common/bitvm_cache_dev.bin -O bitvm_cache_dev.bin
export BITVM_CACHE_PATH=/path/to/bitvm_cache_dev.bin

# For production
wget https://static.testnet.citrea.xyz/common/bitvm_cache_v3.bin -O bitvm_cache.bin
export BITVM_CACHE_PATH=/path/to/bitvm_cache.bin

If not downloaded, the cache will be generated automatically on first run, which can take significant time.

Symptom: Database queries are slow

Problem: PostgreSQL performance degradation.Solution:

Increase max connections if needed:

docker run --name clementine-test-db \
  -e POSTGRES_USER=clementine \
  -e POSTGRES_PASSWORD=clementine \
  -e POSTGRES_DB=clementine \
  -p 5432:5432 \
  --restart always \
  -d postgres:15 \
  bash -c "exec docker-entrypoint.sh postgres -c 'max_connections=1000'"

Monitor database performance
Consider running migrations if schema is outdated

Debugging

Getting More Information

Enable verbose logging

Increase log verbosity to debug issues:

./target/release/clementine-core verifier \
  --config /path/to/config.toml \
  --verbose 5  # Logs everything

Log levels:

1: Errors only
2: Warnings
3: Info
4: Debug
5: Trace (everything)

Enable full backtraces

Get detailed error backtraces:

RUST_LIB_BACKTRACE=full ./target/release/clementine-core verifier --config /path/to/config.toml

Debug Tokio tasks

For debugging async task issues:

Uncomment console-subscriber dependency in Cargo.toml
Uncomment console_subscriber::init(); in src/utils.rs
Rebuild with console support:
```
cargo build_console
```
Run your service and access the console:
```
tokio-console
```

This provides real-time visibility into Tokio task execution, helping identify deadlocks and performance issues.

Security

Certificate Management

Proper certificate management is critical for production deployments.

Best practices for TLS certificates

Never commit private keys to version control
Use proper CAs in production, not self-signed certificates
Rotate certificates regularly to minimize exposure
Use distinct certificates for different services/entities

Store certificates securely with appropriate file permissions:

chmod 600 certs/server/server.key
chmod 600 certs/client/client.key
chmod 644 certs/server/server.pem
chmod 644 certs/client/client.pem

Monitor certificate expiration and renew before expiry

Common Error Messages

Error	Possible Cause	Solution
`stack overflow`	`RUST_MIN_STACK` not set	`export RUST_MIN_STACK=33554432`
`Connection refused (database)`	PostgreSQL not running	Start PostgreSQL container
`Connection refused (bitcoin)`	Bitcoin node not running	Start Bitcoin Core
`Certificate verify failed`	Wrong certificate or missing CA	Check certificate paths and regenerate
`RISC0 proof generation failed`	Wrong RISC0 version or mode	Verify version 2.1.0 and set `RISC0_DEV_MODE=1` for tests
`BitVM cache not found`	Cache not downloaded	Download cache or let it generate
`Authentication failed`	mTLS certificate mismatch	Verify client certificate matches expected cert in config

Getting Help

If you’re still experiencing issues:

Search existing issues: Check GitHub Issues for similar problems
Review logs: Run with --verbose 5 to get detailed logs
Check configuration: Verify all paths and settings in your config file
Create an issue: If you’ve found a bug, create a new issue with:
- Clementine version
- Operating system
- Full error message and backtrace
- Steps to reproduce
- Configuration (sanitize sensitive data)
Security issues: For security vulnerabilities, see Security Reporting

Join the Citrea Discord for community support and real-time help.

Additional Resources

Troubleshooting

Troubleshooting

Installation Issues

RISC Zero Installation Fails

Runtime Issues

Server Won’t Start

RPC Authentication Issues

Testing Issues

Test Failures

Performance Issues

Slow Performance

Debugging

Getting More Information

Security

Certificate Management

Common Error Messages

Getting Help

Build docs developers (and LLMs) love

Additional Resources

​Troubleshooting

​Installation Issues

​RISC Zero Installation Fails

​Runtime Issues

​Server Won’t Start

​RPC Authentication Issues

​Testing Issues

​Test Failures

​Performance Issues

​Slow Performance

​Debugging

​Getting More Information

​Security

​Certificate Management

​Common Error Messages

​Getting Help

Build docs developers (and LLMs) love

Troubleshooting

Installation Issues

RISC Zero Installation Fails

Runtime Issues

Server Won’t Start

RPC Authentication Issues

Testing Issues

Test Failures

Performance Issues

Slow Performance

Debugging

Getting More Information

Security

Certificate Management

Common Error Messages

Getting Help