Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via a Prometheus HTTP endpoint.Configuration
Configure the Prometheus metrics endpoint in yourconfig.toml:
config.toml
Accessing Metrics
Once configured, you can query the metrics endpoint using curl or any Prometheus-compatible scraper:Metrics are currently only provided for developer and diagnostic use. The endpoint or data provided may break or change in incompatible ways at any time.
Metric Types
Firedancer reports three metric types following the Prometheus data model:metric type
A cumulative metric representing a monotonically increasing counter.
metric type
A single numerical value that can go arbitrarily up or down.
metric type
Samples observations like packet sizes and counts them in buckets.
Available Metrics
Link Metrics
Metrics for all inter-tile communication links:counter
The number of times the link reader has consumed a fragment.
counter
The total number of bytes read by the link consumer.
counter
The number of fragments that were filtered and not consumed.
counter
The total number of bytes read by the link consumer that were filtered.
counter
The number of times the link has been overrun while polling.
counter
The number of input overruns detected while reading metadata by the consumer.
counter
The number of times the consumer was detected as rate limiting by the producer.
Tile Metrics
Metrics available for all tiles:gauge
The process ID of the tile.
gauge
The thread ID of the tile. Always the same as the PID in production, but might be different in development.
gauge
Index of the CPU last executed on.
counter
The number of involuntary context switches.
counter
The number of voluntary context switches.
counter
The number of major page faults.
counter
The number of minor page faults.
gauge
The current status of the tile: 0 is booting, 1 is running, 2 is shutdown.
gauge
The last UNIX timestamp in nanoseconds that the tile heartbeated.
gauge
Whether the tile is currently backpressured or not, either 1 or 0.
counter
Number of times the tile has had to wait for one or more consumers to catch up to resume publishing.
Tile-Specific Metrics
IPEcho Tile
gauge
The current shred version used by the validator.
gauge
The number of active connections to the ipecho service.
counter
The number of connections that have been made and closed normally.
counter
The number of connections that have been made and closed abnormally.
Snapshot Control Tile
gauge
State of the snapshot control tile.
gauge
Number of bytes read so far from the full snapshot. Might decrease if snapshot load is aborted and restarted.
gauge
Total size of the full snapshot file.
gauge
Number of bytes read so far from the incremental snapshot.
gauge
The predicted slot from which replay starts after snapshot loading finishes.
Integration with Monitoring Tools
Prometheus
Add Firedancer as a scrape target in yourprometheus.yml:
prometheus.yml
Grafana
Once metrics are being scraped by Prometheus, you can create Grafana dashboards to visualize:- Tile health and backpressure
- Link throughput and overruns
- Context switches and page faults
- Snapshot loading progress
For a complete list of all available metrics including tile-specific counters for net, quic, verify, dedup, pack, bank, poh, shred, store, sign, and other tiles, refer to the full metrics output from your running validator.