Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/CelestoAI/SmolVM/llms.txt

Use this file to discover all available pages before exploring further.

This guide covers common issues you might encounter when using SmolVM and how to resolve them.

​
Diagnostics

SmolVM includes a built-in diagnostic tool to check your system configuration: 
# Auto-detect backend and check prerequisites
smolvm doctor

# Check specific backend
smolvm doctor --backend firecracker
smolvm doctor --backend qemu

# CI-friendly JSON output
smolvm doctor --json --strict
The smolvm doctor command validates:
  • KVM availability (Linux/Firecracker)
  • Firecracker binary installation
  • QEMU installation and HVF support (macOS)
  • Network configuration (nftables, iproute2)
  • System permissions

​
Common Issues

Problem: The Firecracker backend requires KVM hardware virtualization.Solution:
  1. Verify KVM is available:
ls -l /dev/kvm
  1. If missing, enable virtualization in your BIOS/UEFI settings
  2. Add your user to the kvm group: 
sudo usermod -aG kvm $USER
newgrp kvm  # Or log out and back in
  1. Verify permissions:
# Should show rw-rw---- with kvm group
ls -l /dev/kvm
Alternative: Use the QEMU backend if KVM is unavailable:
from smolvm import SmolVM
vm = SmolVM(backend="qemu")
Problem: The firecracker executable is not in PATH.Solution:
  1. Run the system setup script:
sudo ./scripts/system-setup.sh --configure-runtime
  1. Or install manually using the HostManager:
from smolvm.host import HostManager

host = HostManager()
host.install_firecracker()
  1. Verify installation:
which firecracker
# Should show /usr/local/bin/firecracker or ~/.smolvm/bin/firecracker
Problem: User lacks permissions to create TAP networking devices.Solution:
  1. Run the network setup script:
sudo ./scripts/system-setup.sh --configure-runtime
  1. Or configure manually:
# Allow user to create TAP devices
sudo setcap cap_net_admin+ep $(which ip)

# Enable IP forwarding
sudo sysctl -w net.ipv4.ip_forward=1
echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf
  1. Verify nftables is installed:
sudo nft list ruleset
Problem: VM starts successfully but SSH connection fails.Solution:
  1. Check VM status:
from smolvm import SmolVM

manager = SmolVM()
vm_info = manager.get("vm-xxxxx")
print(f"Status: {vm_info.status}")
print(f"IP: {vm_info.network.guest_ip}")
print(f"SSH Port: {vm_info.network.ssh_host_port}")
  1. Test network connectivity:
# Ping guest IP (Firecracker backend)
ping 172.16.0.2

# Test SSH port forwarding
nc -zv localhost 2200
  1. Check firewall rules:
sudo nft list ruleset | grep 2200
  1. Examine VM logs:
cat ~/.local/state/smolvm/vm-xxxxx.log
  1. Increase SSH timeout:
from smolvm import SmolVM, VMConfig

config = VMConfig(ssh_timeout=60.0)  # Increase from default 30s
vm = SmolVM(config)
Problem: Multiple SmolVM processes trying to access the state database simultaneously.Solution:
  1. The state database uses exclusive locks for writes. Ensure only one process modifies VMs at a time.
  2. Check for stale processes: 
ps aux | grep firecracker
ps aux | grep qemu-system
  1. Reconcile VM state:
from smolvm import SmolVM

manager = SmolVM()
stale_vms = manager.reconcile()
print(f"Cleaned up {len(stale_vms)} stale VMs")
  1. If persistent, manually remove the lock:
# WARNING: Only do this if no SmolVM processes are running
rm ~/.local/state/smolvm/smolvm.db-wal
rm ~/.local/state/smolvm/smolvm.db-shm
Problem: All IPs in the 172.16.0.2-254 range are allocated.Solution:
  1. List all VMs:
from smolvm import SmolVM

manager = SmolVM()
vms = manager.list_vms()
print(f"Total VMs: {len(vms)}")
  1. Clean up stopped VMs:
from smolvm.types import VMState

stopped_vms = manager.list_vms(status=VMState.STOPPED)
for vm in stopped_vms:
    manager.delete(vm.vm_id)
  1. Use the CLI cleanup command:
smolvm cleanup --all
  1. The IP pool supports 253 concurrent VMs. If you need more, consider implementing a custom IP allocator.
Problem: QEMU process terminates immediately after start.Solution:
  1. Verify QEMU installation:
qemu-system-aarch64 --version
qemu-system-x86_64 --version
  1. Check HVF acceleration support:
qemu-system-aarch64 -accel help
# Should list 'hvf' for Hypervisor.framework
  1. Reinstall QEMU via Homebrew:
brew uninstall qemu
brew install qemu
  1. Check VM logs for kernel panic:
cat ~/.local/state/smolvm/vm-xxxxx.log
  1. Ensure kernel and rootfs are compatible:
from smolvm import VMConfig
from smolvm.build import download_assets

# Download matching kernel and rootfs
assets = download_assets()
config = VMConfig(
    kernel_path=assets["kernel"],
    rootfs_path=assets["rootfs"]
)
Problem: VM is marked as ERROR and cannot be restarted.Solution:
  1. Check VM details:
from smolvm import SmolVM

manager = SmolVM()
vm_info = manager.get("vm-xxxxx")
print(f"Status: {vm_info.status}")
print(f"PID: {vm_info.pid}")
  1. Examine logs:
cat ~/.local/state/smolvm/vm-xxxxx.log
  1. Delete and recreate:
manager.delete("vm-xxxxx")

# Create fresh VM
new_vm = manager.create(config)
manager.start(new_vm.vm_id)
  1. Run reconciliation to clean up stale state:
stale_vms = manager.reconcile()
Problem: SmolVM cannot create or write to any data directory.Solution:
  1. Check directory permissions:
ls -ld ~/.local/state/smolvm
ls -ld /var/lib/smolvm
  1. Create directory manually:
mkdir -p ~/.local/state/smolvm
chmod 755 ~/.local/state/smolvm
  1. Set explicit data directory:
from pathlib import Path
from smolvm import SmolVM

data_dir = Path("/tmp/smolvm-data")
manager = SmolVM(data_dir=data_dir)
  1. Use environment variable:
export SMOLVM_DATA_DIR=/tmp/smolvm-data

​
Debugging Tips

​
Enable Debug Logging

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

from smolvm import SmolVM

manager = SmolVM()
# Now you'll see detailed debug output

​
Inspect Firecracker Socket

Manually query the Firecracker API: 
from smolvm.api import FirecrackerClient
from pathlib import Path

socket_path = Path("/tmp/fc-vm-xxxxx.sock")
client = FirecrackerClient(socket_path)
info = client.get_instance_info()
print(info)
client.close()

​
Check Network Rules

# List all nftables rules
sudo nft list ruleset

# Check NAT rules for specific VM
sudo nft list table nat | grep smolvm

# List TAP devices
ip link show | grep tap

# Show routes to guest VMs
ip route show | grep 172.16.0

​
Monitor VM Processes

# List all Firecracker processes
ps aux | grep firecracker

# List all QEMU processes  
ps aux | grep qemu-system

# Check resource usage
top -p $(pgrep -d',' firecracker)

​
Getting Help

If you’re still experiencing issues:
  1. Check the GitHub Issues for similar problems
  2. Run smolvm doctor --json and include the output in your bug report
  3. Include relevant logs from ~/.local/state/smolvm/*.log
  4. Join the Celesto AI Slack community

​
Known Limitations

  • IP Pool: Maximum 253 concurrent VMs (172.16.0.2-254)
  • SSH Port Pool: Maximum 800 concurrent VMs (ports 2200-2999)
  • Firecracker: Linux only (requires KVM)
  • QEMU: Slower boot times compared to Firecracker
  • SSH Trust Model: Host keys not strictly verified by default (see SECURITY.md)

Build docs developers (and LLMs) love

Get started for freeTalk to us