Debug the Linux Kernel with ftrace, perf, and kprobes

Kernel debugging requires a different toolkit than userspace debugging because the kernel runs without the safety net of an operating system beneath it. This page covers the spectrum of kernel debugging techniques, from simple log messages to post-mortem crash analysis, using real interfaces documented in the kernel source.

printk and pr_* macros

printk is the kernel’s primary logging function. Messages written with printk go into the kernel ring buffer and appear on the console if their log level is at or below the current console log level.

/* Basic printk with an explicit log level */
printk(KERN_INFO "my_driver: device found at IRQ %d\n", irq);

/* Preferred short-hand macros */
pr_emerg("System is about to crash\n");
pr_alert("Action required immediately\n");
pr_crit("Critical error in %s\n", __func__);
pr_err("Failed to allocate memory: %d\n", ret);
pr_warn("Deprecated API called from %pS\n", (void *)_RET_IP_);
pr_info("Device initialized successfully\n");
pr_debug("Buffer address: %px size: %zu\n", buf, size);

Read the ring buffer from userspace with dmesg:

# Show all messages with timestamps
dmesg -T

# Follow new messages as they arrive
dmesg -w

# Show only error-level messages and above
dmesg --level=err,crit,alert,emerg

# Clear the ring buffer (requires root)
dmesg -c

pr_debug and dev_dbg compile to no-ops unless either DEBUG is defined in the source file or dynamic debug is enabled at runtime. This means they add zero overhead in production.

Dynamic debug

Dynamic debug allows you to enable and disable individual pr_debug and dev_dbg call sites at runtime without recompiling the kernel. Enable it by setting CONFIG_DYNAMIC_DEBUG=y. If /proc/dynamic_debug/control exists, your kernel supports dynamic debug. View the catalog of all available debug sites:

head -n7 /proc/dynamic_debug/control
# filename:lineno [module]function flags format
# init/main.c:1424 [main]run_init_process =_ "  with arguments:\012"

The third column shows the current flags. =p means the site is enabled and will print. Enable specific debug messages by writing a query to the control file:

# Enable all debug messages from a module
echo "module usb +p" > /proc/dynamic_debug/control

# Enable messages in a specific source file
echo "file drivers/net/ethernet/intel/e1000e/netdev.c +p" > /proc/dynamic_debug/control

# Enable messages in a specific function
echo "func tcp_rcv_established +p" > /proc/dynamic_debug/control

# Disable all enabled sites
echo "-p" > /proc/dynamic_debug/control

To see dynamic debug output on the console you may need to raise the console log level: echo 8 > /proc/sys/kernel/printk or boot with loglevel=8.

ftrace — function tracer

ftrace is a kernel tracing framework built around the tracefs filesystem, typically mounted at /sys/kernel/tracing. It provides function-level tracing, latency measurement, and a rich event system.

Mount tracefs

mount -t tracefs nodev /sys/kernel/tracing
# Or add to /etc/fstab:
# tracefs  /sys/kernel/tracing  tracefs  defaults  0  0

Choose a tracer

# List available tracers
cat /sys/kernel/tracing/available_tracers
# blk function_graph wakeup_dl wakeup_rt wakeup function nop

# Enable the function tracer
echo function > /sys/kernel/tracing/current_tracer

Filter which functions to trace

# Trace only functions matching a pattern
echo 'tcp_*' > /sys/kernel/tracing/set_ftrace_filter

# Trace a specific function
echo schedule > /sys/kernel/tracing/set_ftrace_filter

# Clear the filter (trace all functions)
echo > /sys/kernel/tracing/set_ftrace_filter

Start and read the trace

# Enable tracing
echo 1 > /sys/kernel/tracing/tracing_on

# ... perform the operation you want to trace ...

# Disable tracing
echo 0 > /sys/kernel/tracing/tracing_on

# Read the trace output
cat /sys/kernel/tracing/trace

The function_graph tracer adds call depth and duration, making it easier to follow execution flow:

echo function_graph > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
# ... trigger activity ...
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace

trace-cmd

trace-cmd is a userspace tool that wraps the tracefs interface into a simpler command-line API:

# Record a function trace for 3 seconds
trace-cmd record -p function -l 'ext4_*' sleep 3

# Report the recorded trace
trace-cmd report

perf — performance analysis

perf is the primary performance analysis tool for Linux. It exposes hardware performance counters, software events, and kernel tracepoints through a unified interface.

# Record CPU cycles, instructions, cache misses for a command
perf stat ls -la /usr/bin

# Record with specific events
perf stat -e cycles,instructions,cache-misses,cache-references ls

For meaningful function names in perf report, build the kernel with CONFIG_DEBUG_INFO=y and install the kernel debug symbols package for your distribution.

kprobes and tracepoints

Kprobes lets you dynamically insert breakpoints at nearly any instruction in the kernel and run a handler when the breakpoint is hit. A kretprobe fires when a specified function returns, giving you access to both the return value and the execution time.

#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
};

static int handler_pre(struct kprobe *p, struct pt_regs *regs)
{
    pr_info("do_sys_open called from %pS\n", (void *)regs->ip);
    return 0;
}

static int __init kprobe_example_init(void)
{
    kp.pre_handler = handler_pre;
    return register_kprobe(&kp);
}

Tracepoints are static instrumentation points compiled into the kernel. Unlike kprobes they have no overhead when disabled and a minimal, well-defined overhead when enabled.

# List all available tracepoints in the scheduler subsystem
ls /sys/kernel/tracing/events/sched/

# Enable the sched_switch tracepoint
echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable

# Read the trace
cat /sys/kernel/tracing/trace

KASAN — Kernel Address Sanitizer

KASAN is a dynamic memory safety error detector that finds out-of-bounds and use-after-free bugs. Enable it with CONFIG_KASAN=y. It has three modes:

Mode	Config	Use case
Generic KASAN	`CONFIG_KASAN_GENERIC`	Development debugging on x86_64, arm64, and others
Software Tag-Based	`CONFIG_KASAN_SW_TAGS`	Testing on arm64 with moderate overhead
Hardware Tag-Based	`CONFIG_KASAN_HW_TAGS`	Production use on arm64 CPUs with MTE support

When KASAN detects a bug it prints a report to the kernel log:

BUG: KASAN: slab-out-of-bounds in copy_from_kernel_nofault+0x...
Read of size 8 at addr ffff88800f2a5000 by task syz-executor/1234

Generic KASAN can increase memory usage by 2–5× and slow execution by 30–100%. Do not use CONFIG_KASAN_GENERIC in production environments.

KFENCE — Kernel Electric-Fence

KFENCE is a low-overhead sampling-based memory error detector designed to run in production. It detects heap out-of-bounds access, use-after-free, and invalid-free errors. Enable it with CONFIG_KFENCE=y. Unlike KASAN, KFENCE does not instrument every allocation. It guards a sample of allocations and relies on the statistical guarantee that, given enough uptime across a fleet, bugs will be caught.

# Set the sample interval (milliseconds between guarded allocations)
# At boot:
kfence.sample_interval=100

# At runtime:
echo 100 > /sys/module/kfence/parameters/sample_interval

# View KFENCE error reports
dmesg | grep KFENCE

A KFENCE report looks like this:

BUG: KFENCE: out-of-bounds read in test_out_of_bounds_read+0x...
Out-of-bounds read at 0x... (1B right of kfence-#0):
 test_out_of_bounds_read+0x... [kfence_test]

Kernel oops and panic analysis

When the kernel encounters an unrecoverable error it prints an oops message (if the error is recoverable) or a panic message (if it is not). The message contains the faulting instruction, register state, and a call stack.

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
Oops: 0002 [#1] SMP PTI
CPU: 0 PID: 4424 Comm: insmod Tainted: P        W  O      4.20.0
RIP: 0010:my_oops_init+0x13/0x1000 [kpanic]

The Tainted: field tells you whether the kernel’s integrity is in question. A P flag means a proprietary module was loaded; W means a warning occurred; O means an out-of-tree module was loaded. Decode the taint flags:

# Check taint status of the running kernel
cat /proc/sys/kernel/tainted

# Decode the taint number using the kernel tool
sh tools/debugging/kernel-chktaint

Decode the RIP address back to a symbol and source line:

# Translate an address to a symbol
addr2line -e vmlinux -i 0xffffffff81234567

# Or use the scripts provided with the kernel
scripts/decode_stacktrace.sh vmlinux < oops.txt

kdump — post-mortem crash analysis

kdump captures a complete memory dump of the failed kernel by booting a small capture kernel into a reserved region of memory. The dump is written to disk and analysed later with the crash tool.

Reserve memory for the capture kernel

Add crashkernel=256M to your kernel command line (adjust the size based on available RAM).

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet crashkernel=256M"

Regenerate the GRUB config and reboot.

Install and configure kdump

Debian / Ubuntu
Fedora / RHEL

sudo apt-get install kdump-tools crash linux-crashdump
sudo systemctl enable kdump-tools
sudo systemctl start kdump-tools

sudo dnf install kexec-tools crash
sudo systemctl enable kdump
sudo systemctl start kdump

Verify kdump is ready

# Should report that the crash kernel is loaded
cat /proc/sys/kernel/kexec_crash_loaded
# 1

# Check the reserved memory region
cat /proc/iomem | grep "Crash kernel"

Analyse the dump with crash

After a kernel panic, the dump is saved to the path configured in /etc/kdump.conf (commonly /var/crash/).

crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux \
      /var/crash/$(ls /var/crash | tail -1)/vmcore

# Inside the crash shell:
crash> bt          # backtrace of the crashing thread
crash> log         # print the kernel message log
crash> ps          # list processes at the time of crash
crash> vm          # display virtual memory information
crash> quit

ftrace documentation

Full reference for the ftrace framework and all available tracers.

perf wiki

Comprehensive perf usage guide including flame graphs and PMU events.

KASAN documentation

Detailed KASAN configuration and report interpretation guide.

kdump documentation

Setting up kdump and analysing crash dumps with the crash tool.

Get Started

Development Guide

Kernel Internals

Driver Development

Administration

Contributing

Debug the Linux Kernel with ftrace, perf, and kprobes

printk and pr_* macros

Dynamic debug

ftrace — function tracer

trace-cmd

perf — performance analysis

kprobes and tracepoints

KASAN — Kernel Address Sanitizer

KFENCE — Kernel Electric-Fence

Kernel oops and panic analysis

kdump — post-mortem crash analysis

ftrace documentation

perf wiki

KASAN documentation

kdump documentation

Build docs developers (and LLMs) love

Get Started

Development Guide

Kernel Internals

Driver Development

Administration

Contributing

Documentation Index

​printk and pr_* macros

​Dynamic debug

​ftrace — function tracer

​trace-cmd

​perf — performance analysis

​kprobes and tracepoints

​KASAN — Kernel Address Sanitizer

​KFENCE — Kernel Electric-Fence

​Kernel oops and panic analysis

​kdump — post-mortem crash analysis

ftrace documentation

perf wiki

KASAN documentation

kdump documentation

Build docs developers (and LLMs) love

printk and pr_* macros

Dynamic debug

ftrace — function tracer

trace-cmd

perf — performance analysis

kprobes and tracepoints

KASAN — Kernel Address Sanitizer

KFENCE — Kernel Electric-Fence

Kernel oops and panic analysis

kdump — post-mortem crash analysis