Skip to main content
This page covers common troubleshooting steps when encountering errors while building and running Firedancer. If these don’t resolve your issue, seek help in the #firedancer-operators channel on the Solana Tech Discord or file an issue on GitHub.

​
Building Issues

​
General Recommendations

1

Clean build from scratch

Do a fresh clone of the repository following the Getting Started guide. Verify you’re using a supported compiler.
2

Install dependencies

Always run the dependency script after cloning:
3

Update existing repositories correctly

If updating an existing clone, update the solana submodule after pulling:

​
Missing cargo Binary

Error:
Solution: This typically happens due to a race condition during Rust toolchain installation. Reinstall the toolchain (replace 1.75.0 with your version):

​
Out of Memory During Build

Symptom: Build fails with various errors when compiling Agave components. Solution: You need around 32GiB of available memory to build Firedancer. If you run out of memory, make can return various errors. Consider:
  • Closing other applications
  • Reducing parallel jobs: make -j4 fdctl solana (instead of -j)
  • Adding swap space temporarily
  • Building on a machine with more RAM

​
Compiler Version Issues

Solution: Firedancer requires GCC 8.5 or higher. Only GCC versions 11, 12, and 13 are officially supported and tested. Check your GCC version:
If your version is unsupported, install a supported version through your package manager.

​
Configuration Issues

​
General Recommendations

If fdctl configure init all fails, try removing all existing configuration first:
You can also re-run a specific stage:
Ensure the same config.toml is used for all commands:
Ensure your configuration file is valid TOML. Common issues:
  • Missing quotes around strings
  • Incorrect array syntax
  • Duplicate keys
  • Invalid section headers
You can validate TOML syntax using online validators or command-line tools.
fdctl often prints helpful suggestions in error messages. Look for:
  • Specific capability requirements
  • Permission issues
  • Missing system resources
  • Configuration conflicts

​
Hugetlbfs Allocation Failures

Error: Cannot allocate huge pages Solutions:
Run hugetlbfs configuration immediately after system boot. Memory fragmentation later prevents allocating large contiguous blocks.
  • Reboot and run fdctl configure init hugetlbfs before other applications start
  • Check available memory: cat /proc/meminfo | grep Huge
  • Verify no other applications are holding huge pages

​
Network Device Configuration Errors

Error: ethtool commands fail Solutions:
  • Verify the network interface name in your config matches the actual interface:
  • Check if the interface supports required features:
  • Ensure you’re running with root privileges (not just capabilities)

​
Sysctl Parameter Conflicts

Error: Cannot set sysctl values Solutions:
  • Check current values:
  • Verify you have CAP_SYS_ADMIN or root privileges
  • Check for conflicting sysctl configurations in /etc/sysctl.conf or /etc/sysctl.d/

​
Runtime Issues

​
General Recommendations

Ensure the same config.toml is used for both configure and run commands.

​
Validator Fails to Start

Checklist:
1

Verify configuration

All stages should pass. If not, run init:
2

Check permissions

  • Verify you’re running as root: whoami
  • Verify the user in config.toml exists: id <username>
  • Ensure the user has access to keypair files
3

Verify keypairs exist

4

Check ledger directory

Ensure the ledger directory is accessible and has sufficient space:

​
XDP/AF_XDP Errors

Error: Cannot attach XDP program Solutions:
  • Verify kernel version (minimum 4.18 required):
  • Check if another XDP program is attached:
    Look for “xdp” in the output.
  • Verify network driver supports XDP:
    Well-tested drivers: ixgbe, i40e, ice

​
Network Connectivity Issues

Symptom: Validator cannot connect to cluster Solutions:
  • Verify gossip entrypoints are correct for your cluster
  • Check firewall rules allow required ports
  • Ensure network interface is up:
  • Test basic connectivity:

​
Tile Crashes or Hangs

Symptom: One or more tiles stop responding Solutions:
  • Check if hyperthreaded pairs are conflicting (see initialization guide)
  • Verify CPU affinity settings don’t overlap
  • Monitor system resources:
  • Check for out-of-memory conditions:

​
RPC Not Responding

Symptom: Cannot connect to RPC endpoint Solutions:
  • Verify RPC is enabled in config:
  • Check if port is listening:
  • Verify firewall allows connections to RPC port
  • Test RPC endpoint:

​
Performance Issues

​
High CPU Usage on Non-Tile Processes

Symptom: ksoftirqd or other kernel processes consuming CPU Solutions:
  • Verify IRQ affinity configuration
  • Check if network interrupts are properly configured
  • Ensure tiles are pinned to correct CPU cores

​
Poor Network Performance

Solutions:
  • Verify ethtool configuration is correct:
  • Check network device statistics:
  • Ensure sufficient network queues for net tiles
  • Consider using dedicated mode for ethtool-channels

​
Slow Block Processing

Solutions:
  • Increase tile counts in configuration if CPU cores are available
  • Monitor tile backpressure with fdctl monitor
  • Verify disk I/O is not bottlenecked:

​
Getting Help

If these troubleshooting steps don’t resolve your issue:
  1. Gather diagnostic information:
    • Firedancer version: git describe --tags
    • Kernel version: uname -r
    • GCC version: gcc --version
    • Network driver: ethtool -i <interface>
    • Error messages and logs
  2. Check existing issues:
  3. Ask for help:
    • Solana Tech Discord: #firedancer-operators channel
    • File a GitHub issue with diagnostic information
When reporting issues, include your configuration file (with sensitive data like keypairs removed) and relevant error messages.