Skip to content

bpftrace How-to Guides

What this page covers

Task recipes for bpftrace 0.27.x: install or build it, check that a kernel is ready, discover probes, run one-liners, write reusable scripts, upgrade old scripts, feed output into other tools, and debug failures. Commands come from the upstream README, docs/language.md, docs/stdlib.md, docs/migration_guide.md and the bpftrace(8) man page (checked 2026-09-25). Flags and defaults are listed in Reference. Background is in Explanation.

Syntax differs between versions

Examples use current syntax: lowercase begin/end, . for field access (args.filename), let declarations. Provider names are case-insensitive since 0.24, so BEGIN still works. Older distro builds may reject newer constructs such as duration literals, for ranges or macros. Check bpftrace --version first.

Install bpftrace

Distribution packages (from the upstream README install matrix):

sudo apt install bpftrace          # Debian, Ubuntu
sudo dnf install bpftrace          # Fedora, CentOS Stream (CRB)
sudo apk add bpftrace              # Alpine
sudo pacman -S bpftrace            # Arch Linux
sudo zypper install bpftrace       # openSUSE Tumbleweed
sudo emerge -av bpftrace           # Gentoo
nix-shell -p bpftrace              # nixpkgs

Then confirm what you got, because docs must match the binary:

bpftrace --version

For the current upstream release on an older distro, download the static AppImage asset for your architecture (x86_64 or arm64, suffixed by architecture since 0.27.0) from the GitHub Releases page. The README also documents a nightly AppImage built from master:

declare -A suffixes=([x86_64]="X64" [amd64]="AMD64");
declare prefix="bpftrace/bpftrace/workflows/binary/master/bpftrace";
declare url="https://nightly.link/${prefix}-${suffixes[$(uname -m)]}.zip";
curl -L -o bpftrace.zip "${url}" && unzip bpftrace.zip

Build From Source

The repo uses git submodules (libbpf is vendored), so clone recursively:

git clone --recurse-submodules https://github.com/bpftrace/bpftrace
cd bpftrace

Nix is the recommended build path and is what CI uses:

nix develop                          # enter the dev shell
cmake -B build -DCMAKE_BUILD_TYPE=Release
make -C build -j$(nproc)
./build/src/bpftrace --version

Or build a package directly. Flakes skip submodules unless asked:

nix build .?submodules=1#
sudo ./result/bin/bpftrace -e 'begin { print("hello world!") }'

Distro-native builds need recent LLVM (18 to 23 for 0.27), libclang, bcc and libelf development packages; upstream's Dockerfiles under docker/ list them per distro. To link a system libbpf instead of the vendored one, add -DUSE_SYSTEM_LIBBPF=On. Pick an LLVM major with -DLLVM_REQUESTED_VERSION=<major>.

Verify Kernel Readiness

Minimum kernel for 0.27 is 6.1. Check the version, BTF, and the feature report:

uname -r
ls -l /sys/kernel/btf/vmlinux        # kernel BTF present?
sudo bpftrace --info                 # kernel features + build features

Check the required config options (path depends on distro; some expose /proc/config.gz):

grep -E "CONFIG_BPF_SYSCALL=|CONFIG_BPF_JIT=|CONFIG_DEBUG_INFO_BTF=|CONFIG_KPROBE_EVENTS=|CONFIG_UPROBE_EVENTS=|CONFIG_FTRACE_SYSCALLS=" /boot/config-$(uname -r)
zgrep -E "CONFIG_BPF_SYSCALL=|CONFIG_KPROBE_EVENTS=" /proc/config.gz

From a source checkout, the upstream script checks the full list documented in docs/dependency_support.md:

./scripts/check_kernel_features.sh

The full option list is in Reference. Missing options explain most "probe not found" failures on trimmed vendor kernels.

Discover What You Can Trace

Listing is always step one on an unknown host:

sudo bpftrace -l 'tracepoint:syscalls:*openat*'
sudo bpftrace -l 'kprobe:tcp_*'
sudo bpftrace -l '*sleep*'                       # any provider
sudo bpftrace -l 'fentry:bpf:*'                  # running BPF programs you can attach to
sudo bpftrace -l my_script.bt                    # probes a script would attach

Add -v to see argument names and types before writing a script:

sudo bpftrace -lv 'tracepoint:syscalls:sys_enter_openat'
sudo bpftrace -lv 'fentry:tcp_reset'
sudo bpftrace -lv 'struct css_task_iter'         # struct layout from BTF

One-Liner Cookbook

Trace file opens with process name and path:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-6d %-16s %s\n", pid, comm, str(args.filename)); }'

Count syscalls by process name (map printed on Ctrl-C):

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

Syscall rate per second, drained asynchronously:

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @syscalls = count(); }
interval:1s { print(@syscalls); clear(@syscalls); }'

Log2 histogram of vfs_read return values (bytes read):

sudo bpftrace -e 'kretprobe:vfs_read { @bytes = hist(retval); }'

Block I/O request size histogram:

sudo bpftrace -e 'tracepoint:block:block_rq_issue { @ = hist(args.bytes); }'

Profile kernel stacks at 99 Hz (feed into a flame graph):

sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'

Typed kernel arguments with fentry (needs BTF):

sudo bpftrace -e 'fexit:fget { printf("fd %d name %s\n", args.fd, str(retval.f_path.dentry.d_name.name)); }'

Page faults sampled one in one hundred, by process:

sudo bpftrace -e 'software:faults:100 { @[comm] = count(); }'

Trace only one process, or a command you launch:

sudo bpftrace -p 1234 -e 'uprobe:/bin/bash:readline { printf("arg0: %d\n", arg0); }'
sudo bpftrace -c 'sleep 5' -e 'kprobe:do_nanosleep /pid == cpid/ { printf("%d sleeping\n", pid); }'

Linear histogram and time series (tseries is behind the unstable_tseries flag, which warns by default):

interval:ms:1 { @ = lhist(rand % 10, 0, 10, 1); }
@ = tseries(@v, 1s, 5, "max")

Run the Bundled Tools

The repo tools/ directory ships 38 maintained scripts (opensnoop.bt, biolatency.bt, runqlat.bt, tcpretrans.bt, and more; full list in Reference). Distro packages install them, but the directory varies; find them with dpkg -L bpftrace or rpm -ql bpftrace. Tools on master may use unreleased features, so use the copy from the release/0.Y.x branch that matches your binary.

sudo ./tools/opensnoop.bt
sudo ./tools/opensnoop.bt -- --errname          # named options via getopt()

Write a Reusable Script

A script file puts sections in a fixed order: C definitions, then config and import, then map declarations, macros and probes. This example uses features stable since 0.25:

#!/usr/bin/env bpftrace

config = {
    missing_probes = "warn";
    max_strlen = 256
}

let @opens = lruhash(10000);

macro fname(p) { str(p) }

begin {
    printf("Tracing openat... Hit Ctrl-C to end.\n");
}

tracepoint:syscalls:sys_enter_openat {
    @opens[comm, fname(args.filename)] = count();
}

end {
    print(@opens, 20);
    clear(@opens);
}

Make it executable and pass named options after --, read with getopt():

chmod 755 opens.bt
sudo ./opens.bt
sudo bpftrace -e 'begin { print((getopt("aa", 1), getopt("bb"))); }' -- --aa=20 --bb

Share code between scripts with imports (paths resolve relative to the importing script, and world-writable directories are refused):

import "helpers.bt";
import "my_c_lib.bpf.c";

Format and dry-run before sharing:

bpftrace --fmt opens.bt
sudo bpftrace --dry-run opens.bt

Test and Benchmark Scripts

Since 0.25, test: and bench: probes are ignored in normal runs and executed in dedicated modes:

sudo bpftrace --test my_tests.bt
sudo bpftrace --bench my_bench.bt

Run only matching probes of a large script:

sudo bpftrace --probe-filter 'tcp' big_script.bt

Upgrade Old Scripts

Common fixes when a script written for 0.21 or earlier fails on 0.25+ (from the upstream migration guide):

Error or symptom Fix
Undefined or undeclared variable: $x Declare with let $x; in the outer block (0.22 block scoping)
delete() takes up to 2 arguments delete(@m, key) per key; tuples for composite keys
Integer size mismatch ... 'uint32' Cast (uint64)pid (pid/tid are uint32 since 0.22)
Maps no longer print on kill -USR1 Add self:signal:SIGUSR1 { print(@m); }
Script exits with an attach error on some hosts Add config = { missing_probes = "warn" } (fatal by default since 0.24)
Tracepoint args fails on a kernel without BTF Export BPFTRACE_BTF=/path/to/vmlinux.btf (needed since 0.25)
sarg0 unknown Read stack arguments via reg("sp") (sarg removed in 0.25)
Deprecation warning for while Rewrite as for ($i : 0..N) { ... }

Feed Output Into Other Tools

JSON output is NDJSON (one JSON document per line), which suits jq and log shippers:

sudo bpftrace -f json -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:5s { print(@); clear(@); }' | jq -c .

Write to a file with explicit buffering, and silence status messages (errors still go to stderr):

sudo bpftrace -q -B line -o /var/tmp/opens.log opens.bt

Run as a background unit (requires a build with -DENABLE_SYSTEMD=1); systemd-run returns once probes are attached:

sudo systemd-run --unit=bpftrace --service-type=notify bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'
sudo systemctl stop bpftrace

Debug Verifier and Attach Failures

sudo bpftrace -d verifier -e '...'           # BPF verifier log
sudo bpftrace -d libbpf -e '...'             # libbpf log
sudo bpftrace -d codegen-opt -e '...'        # optimized LLVM IR
sudo bpftrace -k script.bt                   # warn when probe_read helpers fail
sudo bpftrace -v script.bt                   # per-probe load/attach details

Troubleshooting

Permission denied or missing capabilities

bpftrace needs CAP_BPF, CAP_PERFMON, CAP_DAC_READ_SEARCH and CAP_DAC_OVERRIDE (checked since 0.25). In containers this means privileged mode or explicit capability grants plus access to tracefs and BTF. Plain root inside a restricted namespace can still fail at attach.

  • Kernel lockdown blocks loading. Common with Secure Boot. Upstream options: disable Secure Boot, sudo mokutil --disable-validation and reboot, or lift lockdown temporarily with SysRq+x.
  • "tracepoint not found" / attach error. The event does not exist on this kernel. Re-run -l discovery rather than assuming portability, or set missing_probes = "warn".
  • BPF stack limit of 512 bytes exceeded. Use fewer or smaller strings (pid instead of comm), fewer map keys, split work across probes, or lower on_stack_limit so large objects move to pre-allocated memory.
  • "Kernel headers not found". Only needed for #include-based scripts. Point BPFTRACE_KERNEL_SOURCE at the headers, or rely on BTF instead.
  • Verifier rejects the program. Read -d verifier output. Unbounded loops, oversized stack objects and misaligned accesses are typical causes.
  • Dropped events / missing lines. The ring buffer overflowed. Aggregate in maps instead of printf() per event, or raise perf_rb_pages.
  • Empty output but clean start. Confirm events actually occur (generate traffic). -p also filters kprobes and tracepoints by PID.
  • Slow sync map checks. Casting or comparing a count()/sum() map iterates all CPUs. Move threshold logic into an interval consumer.
  • macOS or Windows. Unsupported. Point investigations at a Linux VM.

Field Notes

  • Pair sessions: leave the openat one-liner running while you reproduce an app bug, then paste the output into the incident channel.
  • For repeat use, keep .bt scripts in the ops repo and treat changes as code reviews.
  • On busy hosts prefer fentry and tracepoints over kprobes, and sampling probes (profile:hz:99) over tracing every event.

Sources