Skip to content

How-to Guides

Task recipes for working through the bpf-developer-tutorial: preparing a host, building and running eunomia-bpf and libbpf lessons, checking kernel prerequisites for the newer lessons, debugging, and cleaning up. Commands come from the upstream lesson READMEs and CI workflow (read 2026-09-25). Package lists and version facts are in Reference: Toolchain Requirements.

Prerequisites

  • A Linux host with kernel 4.8 or later. Lesson 1 suggests 5.15+ or 6.2+ and a recent Ubuntu. Lesson 0 recommends 5.15+ or 6.9+ for full feature coverage (BPF arena, tokens). Individual lessons need up to 7.0 (see Reference: Kernel Baseline Spread).
  • Root or sudo access for loading programs and reading trace output.
  • Basic C knowledge and familiarity with processes and system calls.

Match the CI baseline

Upstream CI builds every libbpf lesson on ubuntu-24.04 with distro clang/llvm. Using the same release (or a newer one with an HWE kernel) avoids most toolchain surprises.

Isolate a Practice Environment

  1. Use a disposable VM. Ubuntu 24.04 matches the CI runner. Snapshot it before the security lessons (24-28, 34, 51) so you can roll back.
  2. Use a mainline test VM for bleeding-edge lessons. Lessons 52 (7.0), 54 (6.19) and 53 (6.16) need very recent kernels. Do not run them on production or daily-driver systems.
  3. Keep networking lessons off your uplink. XDP, tc, TCX and qdisc lessons attach to a named interface. Use veth pairs or loopback where the lesson allows it (lesson 50 defaults to loopback).
  4. Use an emulator for Android. Lesson 22 used an Android Studio emulator image, which keeps experiments off a personal handset.
  5. Check the unprivileged-BPF setting. The lessons assume privileged loading anyway, so leaving unprivileged BPF disabled is fine:
sysctl kernel.unprivileged_bpf_disabled

Install the Toolchain

For the libbpf-based lessons (11 onward), install clang, libelf and zlib as lesson 11 documents:

# Debian/Ubuntu
sudo apt install clang libelf1 libelf-dev zlib1g-dev

# Fedora/CentOS
sudo dnf install clang elfutils-libelf elfutils-libelf-devel zlib-devel

To mirror the full CI package set on Ubuntu 24.04:

sudo apt-get install -y --no-install-recommends \
  libelf1 libelf-dev zlib1g-dev make git clang llvm pkg-config \
  build-essential dwarves linux-headers-generic-hwe-24.04

For the eunomia-bpf lessons (1-10), lesson 1 only needs clang and llvm plus the two eunomia-bpf binaries. ecc compiles kernel-side code into a package. ecli runs packages:

$ sudo apt install clang llvm
$ wget https://aka.pw/bpf-ecli -O ecli && chmod +x ./ecli
$ wget https://github.com/eunomia-bpf/eunomia-bpf/releases/latest/download/ecc && chmod +x ./ecc
$ ./ecc -h
eunomia-bpf compiler
Usage: ecc [OPTIONS] <SOURCE_PATH> [EXPORT_EVENT_HEADER]

The eunomia-bpf README also documents downloading ecli directly from https://github.com/eunomia-bpf/eunomia-bpf/releases/latest/download/ecli. ecc also needs libclang. On aarch64 use ecc-aarch64 and ecli-aarch64.

ecli -h output changed

Lesson 1 still shows the old one-line usage (ecli [--help] [--version] [--json] [--no-cache] url-and-args). Current releases (v1.0.38, 2026-03-08) print subcommands run, push and pull. The remote HTTP mode (ecli client, ecli-server) was removed in March 2026.

No local toolchain at all? The Docker image compiles any directory containing *.bpf.c and headers:

docker run -it -v `pwd`/:/src/ ghcr.io/eunomia-bpf/ecc-`uname -m`:latest

Get the Tutorial Source

Clone with submodules. The libbpf lessons build against the vendored bpftool (which carries libbpf) and blazesym submodules:

git clone https://github.com/eunomia-bpf/bpf-developer-tutorial.git
cd bpf-developer-tutorial
git submodule update --init --recursive

Each lesson directory is independent: build one lesson, not the whole repo.

Compile and Run an eunomia-bpf Lesson

Lesson 1 walkthrough (src/1-helloworld):

  1. Compile the kernel-side program into a package:

    $ ./ecc minimal.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...

  2. Run it as root:

    $ sudo ./ecli run package.json
Running eBPF program...

  3. In another terminal, read the tracepoint output:

    $ sudo cat /sys/kernel/debug/tracing/trace_pipe | grep "BPF triggered sys_enter_write"

  4. If the output is quiet, generate some writes:

    echo "test" > /tmp/test.txt

Stop ecli with Ctrl+C. Events stop immediately.

Compile and Run a libbpf Lesson

Lesson 11 (src/11-bootstrap) is the reference pattern for every Makefile-based lesson:

$ cd src/11-bootstrap
$ make
  BPF      .output/bootstrap.bpf.o
  GEN-SKEL .output/bootstrap.skel.h
  CC       .output/bootstrap.o
  BINARY   bootstrap
$ sudo ./bootstrap

Build from the repository root the way CI does, with a timed smoke run:

make -C src/13-tcpconnlat
sudo timeout -s 2 3 src/13-tcpconnlat/tcpconnlat

The Rust profiler in lesson 12 uses the same make entry point but needs Rust and Cargo installed first:

cd src/12-profile
make
sudo ./profile

Run Prebuilt Tools Without Compiling

ecli can pull and run a precompiled program from an OCI registry:

$ sudo ./ecli run ghcr.io/eunomia-bpf/execve:latest
[79130] node -> /bin/sh -c which ps
[79131] sh -> which ps

Prefer local builds for anything sensitive (see Explanation: Artifact Supply Chain).

Check Kernel Prerequisites

Confirm BTF is available for CO-RE (most lessons from 2 onward need it):

ls -l /sys/kernel/btf/vmlinux
zcat /proc/config.gz 2>/dev/null | grep CONFIG_DEBUG_INFO_BTF || grep CONFIG_DEBUG_INFO_BTF /boot/config-$(uname -r)

Check a lesson-specific option before building. Replace the option name with the one from Reference: Kernel Config Options:

grep -E 'CONFIG_SCHED_CLASS_EXT|CONFIG_NET_XGRESS|CONFIG_NET_SCH_BPF|CONFIG_BPF_LSM' /boot/config-$(uname -r)

For BPF LSM lessons (19, 54), bpf must also be in the active LSM list:

cat /sys/kernel/security/lsm

Run the sched_ext Lessons

Lesson 44 builds scx_simple from the kernel source tree on a 6.12+ kernel with CONFIG_SCHED_CLASS_EXT=y:

uname -r                          # needs 6.12 or later
cd linux/tools/sched_ext && make  # inside a matching kernel source tree
sudo ./scx_simple -f              # -f selects FIFO mode, -v verbose

While it runs, the kernel reports sched_ext state in sysfs:

cat /sys/kernel/sched_ext/state

Terminate the scx_simple process to return all tasks to the default scheduler. SysRq-S also switches back, and SysRq-D triggers a debug dump.

Build the 2026 Lessons (50-54)

These lessons build from the repository root with make, following the lesson READMEs:

make -C src/50-tcx && sudo src/50-tcx/tcx_demo        # monitors loopback until Ctrl+C
make -C src/54-exec-image-inspector clean
make -C src/54-exec-image-inspector -j2

Lesson 51 defaults to a dry run. Only --apply destroys connections:

sudo ./tcp_quarantine 127.0.0.1:42063
sudo ./tcp_quarantine --apply 127.0.0.1:55490 127.0.0.1:42063

If lesson 53 is killed before cleanup and leaves a bpf_pacer root qdisc behind, find the interface that shows it and remove it:

tc qdisc show | grep bpf_pacer
sudo tc qdisc del dev ACTUAL_IFACE root

Debugging Recipes

  • List loaded BPF programs and confirm attachment:

    sudo bpftool prog list

  • Enumerate syscall tracepoints when choosing a SEC() hook name:

    sudo ls /sys/kernel/debug/tracing/events/syscalls/

  • Watch bpf_printk output (shared by all BPF programs, so filter it):

    sudo cat /sys/kernel/debug/tracing/trace_pipe

Clean Up After Experiments

Lesson 28 teaches that pinned programs outlive their launcher. Check for residue instead of assuming teardown:

sudo bpftool prog show
sudo bpftool map show
sudo bpftool link show
ls /sys/fs/bpf/
sudo rm -f /sys/fs/bpf/<pinned-path>   # only pins you created

Common Issues

Tracing silently off

No trace_pipe output despite "Running eBPF program..." usually means tracing is disabled. Enable it: 

$ sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/tracing_on'

bpf_printk limits (lesson 1)

At most three format arguments, a globally shared pipe and measurable overhead at high event rates. Real tools move to perf event arrays and ring buffers (lessons 7-8).

  • Missing /sys/kernel/btf/vmlinux: the kernel was built without CONFIG_DEBUG_INFO_BTF. CO-RE lessons will fail to load. Use a distro kernel with BTF enabled.
  • Syscall tracepoint attach fails with "No such file or directory": the kernel lacks CONFIG_FTRACE_SYSCALLS. Lesson 22 hit this on the Android emulator kernel.
  • Permission denied attaching: most hooks require root. XDP/tc lessons also need a real or veth interface named on the command line.
  • fentry errors on arm64: fentry needs 5.5 on x86_64 but 6.0 on arm64 (lesson 3).
  • Build errors about bpf_session_cookie or file dynptr kfuncs: lessons 52 and 54 carry workarounds for the vendored vmlinux.h. Rebuild from a clean tree (make clean) with updated submodules before patching anything.

Start Your Own Project

When you move past single-lesson examples, start from an org template: libbpf-starter-template (C), cilium-ebpf-starter-template (Go), libbpf-rs-starter-template (Rust) or eunomia-template. Each ships a one-command Makefile, a Dockerfile that publishes to GitHub Packages, and GitHub Actions for build, test and release. The README's example of running a template-built image:

sudo docker run --rm -it --privileged ghcr.io/eunomia-bpf/libbpf-rs-template:latest

Sources