Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/mikronita/mikrom/llms.txt

Use this file to discover all available pages before exploring further.

mikrom-agent is the worker daemon that runs on each compute node in the Mikrom cluster. It receives placement commands from mikrom-scheduler over NATS, then drives the full lifecycle of microVMs on that host—creating, starting, pausing, resuming, and cleaning up Firecracker or Cloud Hypervisor VMs. In addition to VM lifecycle management, the agent reports host and VM metrics back to the scheduler, streams workload logs to the control plane, loads the mikrom-agent-ebpf data-plane program, and starts the tundra-nat64 translator that enables IPv4 egress from the IPv6-only internal network. Port: 5003

​
Responsibilities

VM Lifecycle

Creates, starts, stops, pauses, resumes, and destroys Firecracker and Cloud Hypervisor microVMs in response to scheduler commands.

Worker Registration

Registers the worker node with the scheduler on startup and sends periodic heartbeats carrying CPU, RAM, and disk capacity metrics.

Log Streaming

Captures stdout/stderr from running workloads and ships them to the control plane for display in the dashboard and CLI.

eBPF Data Plane

Embeds and loads the mikrom-agent-ebpf program for host-side network handling and metrics collection.

NAT64 Translation

Starts a singleton tundra-nat64 translator on the host bridge, enabling IPv4 egress from workloads via DNS64 answers from mikrom-dns.

Host Metrics

Collects and reports CPU, RAM, disk, and per-VM metrics to the scheduler for capacity-aware placement decisions.

​
Hypervisor Support

Firecracker is the default hypervisor for application workloads. It provides sub-second boot times and strong isolation with a minimal device model.
  • Used for standard app deployments.
  • Boots using the Zig-built mikrom-init binary embedded in the microVM image.
  • Socket-based API for lifecycle management.

​
eBPF Data Plane

The agent embeds the compiled mikrom-agent-ebpf artifact at build time. The build script (build.rs) reads the program from target/bpfel-unknown-none/release/mikrom-agent-ebpf and embeds it into the agent binary. At runtime, the agent loads it into the kernel for host-side networking support and metrics collection. 
Build pipeline:
  mikrom-agent-ebpf  ──(cargo build)──►  target/bpfel-unknown-none/release/mikrom-agent-ebpf
                                                    │
  mikrom-agent build.rs  ◄──embeds────────────────┘
                                                    │
  mikrom-agent runtime  ──loads into kernel────────┘
The eBPF build uses Rust nightly with -Z build-std=core. Keep the eBPF build path in sync with the Dagger CI runner when modifying the build pipeline.

​
NATS Command Handling

The agent subscribes to NATS subjects controlled by the scheduler. The communication model is request-reply: the scheduler sends a placement command and waits for the agent’s acknowledgement within the configured timeout. 
mikrom-scheduler  ──NATS request──►  mikrom-agent  ──creates VM──►  microVM
                  ◄──NATS reply────  (ack / error)

​
Timeout Configuration

All timeout values are runtime-configurable via environment variables. Defaults are conservative to accommodate a range of hardware.

​
NATS Timeouts

VariableDefaultDescription
AGENT_NATS_CONNECT_TIMEOUT_SECS5Timeout for the initial NATS connection
AGENT_NATS_MAX_BACKOFF_SECS60Maximum backoff between NATS reconnect attempts
AGENT_NATS_CIRCUIT_BREAKER_COOLDOWN_SECS300Cooldown before retrying after a circuit-breaker trip

​
Cloud Hypervisor Timeouts

VariableDefaultDescription
AGENT_CLOUD_HYPERVISOR_SOCKET_WAIT_TIMEOUT_SECS10Time to wait for the Cloud Hypervisor socket to appear
AGENT_CLOUD_HYPERVISOR_API_CONNECT_TIMEOUT_SECS5Timeout for HTTP API connection to Cloud Hypervisor
AGENT_CLOUD_HYPERVISOR_API_STATUS_TIMEOUT_SECS30Timeout waiting for a status response
AGENT_CLOUD_HYPERVISOR_API_HEADER_TIMEOUT_SECS10Timeout reading HTTP response headers
AGENT_CLOUD_HYPERVISOR_API_BODY_TIMEOUT_SECS60Timeout reading HTTP response body
AGENT_CLOUD_HYPERVISOR_CONFIGURE_CLIENT_TIMEOUT_SECS5Timeout for the configure-phase HTTP client
AGENT_CLOUD_HYPERVISOR_CONFIGURE_REQUEST_TIMEOUT_SECS6Timeout for individual configure requests
AGENT_CLOUD_HYPERVISOR_CONFIGURE_BACKOFF_MAX_SECS5Maximum backoff during configure retries

​
Firecracker Timeouts

VariableDefaultDescription
FC_SOCKET_WAIT_TIMEOUT_SECS120Time to wait for the Firecracker socket (non-jailer boot path)
FC_SOCKET_WAIT_CHROOT_SECS10Time to wait for the socket inside a chroot
FC_API_CONNECT_TIMEOUT_SECS2Timeout for HTTP API connection to Firecracker
FC_API_STATUS_TIMEOUT_SECS30Timeout waiting for a status response
FC_API_HEADER_TIMEOUT_SECS10Timeout reading HTTP response headers
FC_API_BODY_TIMEOUT_SECS60Timeout reading HTTP response body
FC_PROCESS_TERMINATE_TIMEOUT_SECS10Grace period for process termination (SIGTERM)
FC_PROCESS_KILL_TIMEOUT_SECS2Grace period before sending SIGKILL
FC_VFS_TERMINATE_TIMEOUT_SECS5Grace period for VFS process termination
FC_VFS_KILL_TIMEOUT_SECS2Grace period before sending VFS SIGKILL
FC_SOCKET_WAIT_TIMEOUT_SECS defaults to 120 seconds to accommodate slower hosts on the non-jailer boot path. Lower this value only if your hardware consistently produces faster Firecracker startup times.

​
Runtime Dependencies

  • NATS — command and control channel from the scheduler
  • Firecracker or Cloud Hypervisor — hypervisor binary on the host
  • mikrom-init — Zig-built init binary embedded in microVM images; must be present in the image root
  • mikrom-network — WireGuard mesh connectivity; must be running on the same host
  • mikrom-dns — provides DNS64 answers that feed the NAT64 translator
  • mikrom-agent-ebpf — compiled eBPF payload, embedded at build time

​
Key Source Files

FilePurpose
src/main.rsProcess bootstrap and runtime wiring
src/agent.rsVM orchestration and scheduler integration
src/metrics.rsHost metric collection
src/ebpf/Agent-side integration for the compiled eBPF payload
build.rsEmbeds the eBPF artifact from target/bpfel-unknown-none/release

​
Debian Package

A Debian package is available for production deployments. 
make deb-agent   # Build the .deb for mikrom-agent
On systemd deployments, load DT_API_TOKEN from /etc/mikrom/dynatrace.env instead of embedding it in agent.env. A template is provided at debian/etc/mikrom/dynatrace.env.example.

​
Local Development

make run-agent     # Start the agent against local infrastructure
make ci-smoke      # fmt + clippy
make ci-fast       # Full workspace test suite
make ci-full       # Adds release build + eBPF validation

​
Testing

  • Prefer workspace-level CI profiles for the full agent + eBPF path.
  • The NATS integration test is opt-in: make ci-external-tests sets MIKROM_RUN_NATS_TESTS=1.
  • Ceph storage tests require a live Ceph cluster: set MIKROM_RUN_CEPH_TESTS=1 or use the dedicated ceph-tests GitHub Actions job on a self-hosted runner labeled ceph.
  • For end-to-end NAT64/DNS64 smoke validation, follow the steps in docs/nat64-dns64-smoke-checklist.md.
When making changes to VM lifecycle behavior, always validate against the scheduler and networking services together rather than running the agent in isolation. The three services form a tightly coupled placement-and-boot pipeline.

Build docs developers (and LLMs) love

Get started for freeTalk to us