Skip to content

Explanation

How the bpf-developer-tutorial is put together and why: the curriculum layout, the CO-RE toolchain it teaches (both the eunomia-bpf ecc/ecli path and the libbpf skeleton path), how event data leaves the kernel, the newer kernel features the 2026 lessons cover, the generated compatibility matrix, and the security model for running the exercises. Look-up tables (lesson index, SEC() names, kernel config options, capabilities) live in Reference.

Curriculum Layout

The repository holds one directory per tutorial under src/. The generated compatibility matrix tracks 65 of them as of 2026-09-25: numbered lessons 0-54, six features/* lessons, three xpu/* lessons and cgroup. Each is self-contained: kernel-side C (*.bpf.c), a user-space loader in C, Go or Rust depending on the lesson, an English README.md and a Chinese README.zh.md, a Makefile, and a .config metadata file that drives the compatibility matrix. Shared dependencies (libbpf and bpftool through the libbpf/bpftool submodule, libbpf/blazesym, pre-generated vmlinux.h headers) sit in src/third_party/.

Tier Lessons Framework focus
Getting started 0-10 eunomia-bpf ecc/ecli. kprobe, fentry, uprobe, tracepoints. Hash maps, perf event arrays, ring buffers, histograms
Advanced projects 11-21 libbpf user-space programs (lesson 11 is derived from libbpf-bootstrap). Rust/libbpf-rs profiler (12), USDT, memleak, LSM, tc and XDP basics
In-depth: networking 23, 29, 41-42, 46, 50, 53 L7 socket filters, sockops, XDP tcpdump/load balancer/packet generator, TCX links, BPF qdisc egress pacer
In-depth: tracing 30-33, 37, 39-40, 48, 52 sslsniff via uprobes, Go goroutine states, wall-clock (on-CPU + off-CPU) profiler, funclatency, Rust/nginx/MySQL tracing, energy monitoring, fsession
In-depth: security 24-28, 34, 51, 54 (plus 19) Process/file hiding, bpf_send_signal, privilege-escalation demos, pinned-program lifecycle, syscall argument rewrite, TCP quarantine, exec image inspection
Kernel features 35-36, 38, 43 + features/* User ring buffer, userspace runtimes (bpftime), BTF-uprobe CO-RE, custom kfuncs, arena, iterators, token, workqueues, dynptr, struct_ops
Schedulers (sched_ext) 44-45 scx_simple and scx_nest on Linux 6.12+
GPU/XPU 47 + xpu/* CUDA API tracing via uprobes, CUPTI flamegraph profiler, DRM GPU and Intel NPU driver tracepoints
Platform 22, 49, cgroup Android emulator experiments, HID-BPF device fixes, cgroup policy control

Two non-lesson assets round out the set: a bpftrace tutorial port (src/bpftrace-tutorial) for one-liner-style learning, and lesson 18's curated research-paper index. Lesson 32 (wall-clock profiler) is present, built in CI and listed in the matrix, but missing from the README table of contents.

Deliberately thin on theory

The README states that the tutorial "does not cover complex concepts and scenario introductions". Lesson 0 instead prescribes a learning plan: read ebpf.io and docs.ebpf.io (5-7 hours), then try the bpftrace tutorial (about 1 hour), the BCC Python tutorial (3-4 hours), a libbpf-bootstrap example (2 hours) and lessons 1-10 of this repo (3-4 hours).

The CO-RE Toolchain Taught by the Tutorial

All lessons follow Compile Once, Run Everywhere (CO-RE): C code is compiled with clang against a BTF-derived vmlinux.h, producing BPF bytecode with relocation records that libbpf patches at load time using the running kernel's BTF (/sys/kernel/btf/vmlinux). The same object then works across kernel versions without recompiling. The course uses two build paths on top of that: eunomia-bpf for lessons 1-10 and plain libbpf with bpftool-generated skeletons from lesson 11 on.

The diagram shows both build paths converging on libbpf and the kernel's verifier and JIT.

flowchart TB
    subgraph Author["Authoring (src/NN-lesson/)"]
        SRC["minimal.bpf.c<br/>SEC('tp/syscalls/sys_enter_write')"]
        VMLINUX["vmlinux.h<br/>BTF-derived kernel types"]
        USER["bootstrap.c / main.rs<br/>user-space loader"]
    end

    subgraph Eunomia["eunomia-bpf path (lessons 1-10)"]
        ECC["ecc<br/>clang wrapper"]
        PKG["package.json or Wasm module<br/>BPF object + exported event schema"]
        OCI["OCI registry<br/>ghcr.io/eunomia-bpf/..."]
        ECLI["ecli run<br/>bpf-loader-rs"]
    end

    subgraph Libbpf["libbpf path (lessons 11+)"]
        CLANG["clang -target bpf<br/>NN.bpf.o"]
        SKEL["bpftool gen skeleton<br/>NN.skel.h"]
        BIN["user binary<br/>linked with libbpf"]
    end

    subgraph Kernel["Linux kernel"]
        BTF["/sys/kernel/btf/vmlinux<br/>CO-RE relocation target"]
        VERIFIER["BPF verifier"]
        JIT["JIT compiler"]
        HOOKS["Hooks: tracepoints, kprobe/fentry,<br/>uprobe, XDP/tcx, LSM, struct_ops"]
        MAPS["Maps: hash, ringbuf,<br/>perf event array, histograms"]
    end

    SRC --> ECC
    VMLINUX --> ECC
    ECC --> PKG
    PKG --> OCI
    PKG --> ECLI
    OCI --> ECLI
    SRC --> CLANG
    VMLINUX --> CLANG
    CLANG --> SKEL
    SKEL --> BIN
    USER --> BIN
    ECLI -->|"libbpf load"| VERIFIER
    BIN -->|"bpf() syscall"| VERIFIER
    BTF -.->|"relocations applied by libbpf"| VERIFIER
    VERIFIER --> JIT
    JIT --> HOOKS
    HOOKS --> MAPS
    MAPS -->|"ring buffer / perf events / map reads"| BIN
    MAPS --> ECLI

Key design points visible across lessons:

  • Kernel/user separation: every program splits into kernel logic plus a user-space loader that handles attach, event polling and display. eunomia-bpf removes the loader for early lessons by generating it from the exported event struct, which is why lesson 1 is only about 25 lines of C.
  • Skeletons: from lesson 11 on, bpftool gen skeleton turns the BPF object into a *.skel.h header with typed open, load, attach and destroy functions, the libbpf-bootstrap convention (make prints BPF, GEN-SKEL, CC, BINARY steps).
  • Minimal-example discipline: lesson 1's tracepoint program demonstrates bpf_get_current_pid_tgid() filtering and bpf_printk output to /sys/kernel/debug/tracing/trace_pipe. The BPF_NO_GLOBAL_DATA macro keeps it loadable on kernels older than 5.2.

Later tiers swap the harness and keep the same kernel-side shape: libbpf C loaders from lesson 11, libbpf-rs in lessons 12 and 37, cilium/ebpf in the Go starter template, and bpftime or wasm-bpf as alternative load-and-execute runtimes discussed in lesson 36.

Runtime Event Flow

The lifecycle taught by the early course arc, from compiled artifact to observed data:

sequenceDiagram
    participant U as ecli (user terminal)
    participant L as bpf-loader-rs / libbpf
    participant K as Kernel verifier and JIT
    participant T as Tracepoint sys_enter_write
    participant M as trace_pipe or ring buffer
    U->>L: sudo ./ecli run package.json
    L->>K: bpf(BPF_PROG_LOAD) with CO-RE relocations applied
    K->>K: verify safety, JIT to native code
    L->>T: attach via perf_event_open or BPF link
    Note over T,M: every matching kernel event runs the BPF function
    T->>M: bpf_printk (lesson 1), later bpf_perf_event_output or bpf_ringbuf_output
    M->>U: events streamed until Ctrl+C closes the link

Hook Taxonomy Across the Curriculum

The course touches almost every major program type: tracepoints and raw tracepoints, kprobe/kretprobe, fentry/fexit and the new fsession, uprobes and USDT, BPF LSM, socket filters, sockops/sk_msg, classic tc, TCX links, XDP, cgroup hooks, iterators and struct_ops. The full table with real SEC() names per lesson is in Reference: Program Types and Attach Points.

The rebuild of the same unlink monitor with kprobe (lesson 2) and then fentry (lesson 3) is the curriculum's clearest teaching device for moving off legacy probe types. Lesson 52 continues the story: fsession replaces the fentry/fexit pair plus hash map with one program and a per-invocation session cookie.

Data Plumbing Evolution

The event-export mechanisms are introduced in dependency order, each lifting a limit of the previous one:

  1. bpf_printk (lesson 1): zero-setup debugging. Shared global pipe, at most three format arguments, measurable overhead at high event rates.
  2. Perf event array (lesson 7, execsnoop): structured events through per-CPU buffers. Events from different CPUs can arrive out of order, and each CPU's buffer is sized separately.
  3. Ring buffer (lesson 8, exitsnoop): one shared memory-mapped buffer (kernel 5.8+), preserving event order across CPUs with better memory efficiency.
  4. Histogram maps (lesson 9, runqlat): in-kernel log2 bucketing of latencies. Only aggregate buckets cross the kernel/user boundary, not raw events.
  5. User ring buffer (lesson 35, kernel 6.1+): reverses direction, letting user space send asynchronous messages to BPF programs.
  6. BPF arena (features/bpf_arena, 6.9+): shared memory pages mapped in both kernel and user space for zero-copy data structures.

Kernel Features Covered by the 2026 Lessons

The lessons added in 2026 track features that landed in recent mainline kernels. Each lesson states its own requirements table.

Lesson Kernel feature Min kernel What changes for the developer
50-tcx TCX links (SEC("tcx/ingress")) 6.6 tc programs become BPF links owned by an fd, with explicit BPF_F_BEFORE/BPF_F_AFTER ordering instead of numeric priorities. The lesson notes Cilium already migrated its dataplane to TCX
51-tcp-quarantine bpf_sock_destroy kfunc with socket iterators 6.5 Destroy selected established TCP connections from BPF. Dry-run by default
53-egress-pacer BPF qdisc (Qdisc_ops via struct_ops) 6.16 A whole queuing discipline (enqueue, dequeue, init, reset, destroy) written in BPF
54-exec-image-inspector BPF task work + file dynptr 6.19 Read the executed file's ELF header safely after exec from an LSM hook
52-fsession-latency fsession program type 7.0 (released April 2026) One program runs at function entry and return, with a session cookie replacing the entry-timestamp hash map

Vendored headers lag the kernel

Lessons 52 and 54 carry workarounds because the repository's vmlinux.h snapshot predates Linux 6.19/7.0: lesson 52 renames the old bpf_session_cookie/bpf_session_is_return declarations and redeclares them with the new ctx argument, and lesson 54 declares the new kfuncs locally in bpf_experimental.h. Regenerating vmlinux.h from a 6.19+/7.0+ kernel removes the need.

Sched_ext Track

Lessons 44-45 teach struct_ops-based CPU scheduling on Linux 6.12+, where sched_ext was merged. A BPF program defines the scheduling policy through callbacks and dispatch queues (DSQs). The scheduler can be switched on and off at runtime without a reboot, and if the BPF scheduler errors out the kernel reverts tasks to the default fair scheduler. Debugging hooks include the sched_ext_dump tracepoint and SysRq-D.

  • Lesson 44 walks through scx_simple, built from the kernel's tools/sched_ext/ directory. It runs in global weighted virtual-time mode or FIFO mode (-f).
  • Lesson 45 reproduces scx_nest from the sched-ext/scx repository, which Meta developed from the Inria paper "OS Scheduling with Nest: Keeping Tasks Close Together on Warm Cores". It packs tasks onto a small set of recently used ("warm") cores so those cores keep high frequencies. It suits low-utilization workloads on single-socket or single-CCX hosts and can hurt workloads that prefer spreading across cores.

Lesson 0 cites production adoption: Meta running sched_ext schedulers (scx_layered) on over one million machines. The tutorial gives no source for that number. The upstream sched-ext/scx README says only that Meta "is in the process of mass production deployment" and publishes no machine count (checked 2026-09-28), so treat the figure as the tutorial's own.

Framework Comparison Frame

Lessons 0 and 1 set out the framework choices the curriculum teaches:

Framework Model Trade-off emphasized
BCC Python front-end, runtime compilation on each target Rich helpers, but heavy dependencies (clang/LLVM on every host) and repeated compiles
libbpf / CO-RE Ahead-of-time compiled object, skeleton headers One compile runs everywhere. More C boilerplate
cilium/ebpf (Go) Go-native loading and management of BPF objects Idiomatic Go apps, still needs C for the kernel side
libbpf-rs Rust wrapper over libbpf Memory safety in user space. Same CO-RE kernel objects
eunomia-bpf Kernel-code-only authorship, packaged JSON/Wasm artifacts Fastest path from .bpf.c to a distributable tool, at the cost of an org-specific toolchain

Compatibility Verification System

The compatibility matrix (src/compatibility.md, published at eunomia.dev/tutorials/compatibility) was added in July 2026. It is generated, not hand-written: scripts/generate_compatibility.py reads each lesson's .config, validates the fields and writes the table. Upstream says the file must not be edited directly.

The pipeline from lesson metadata to published evidence:

flowchart LR
    CFG[".config per lesson<br/>kernel_min, kernel_min_basis,<br/>architectures, btf, kernel_config,<br/>hardware, root, test_status"]
    GEN["scripts/generate_compatibility.py<br/>validates allowed values"]
    MATRIX["src/compatibility.md<br/>65 rows"]
    CI["test-libbpf.yml on ubuntu-24.04<br/>make -C src/NN, timed sudo run"]
    SYNC["trigger-sync.yml<br/>downstream site sync"]
    SITE["eunomia.dev/tutorials"]
    CFG --> GEN
    GEN --> MATRIX
    CI -->|"evidence for ci-runtime / ci-build"| CFG
    MATRIX --> SYNC
    SYNC --> SITE

For each lesson the matrix records the minimum kernel with its basis (tutorial docs, first version of a required feature, or repository baseline), architectures, BTF requirement, core CONFIG_* options, hardware, root requirement and test status. Status distribution on 2026-09-25: 23 CI runtime, 31 CI build, 8 not in CI, 3 docs only (details in Reference: Test Status Distribution).

This is why a kernel spread from 4.8 to 7.0 coexists with confident instructions: each claim traces back to declared metadata plus automated evidence where feasible. Upstream is explicit about the limits: "Not in CI" does not imply a manual test, and a distro kernel can disable a listed option even when its version number is high enough.

Architectural nuance captured by the matrix

Lesson 3 has split baselines: fentry works on x86_64 from 5.5 but arm64 needs 6.0. Lessons 22, 28, 31, 47, 51-54 and all xpu/* lessons are declared x86_64 only.

Platform Portability Notes

Lesson 22 is an exploration report from 2023, not a supported production path. It runs eunomia-bpf inside a Debian chroot (the eadb approach) on an Android 13 x86_64 emulator image (kernel 5.15.41, Pixel 6 AVD). BTF was enabled by default, so CO-RE examples such as exec monitoring worked, but syscall tracepoints failed because that kernel lacked CONFIG_FTRACE_SYSCALLS. At the time, the lesson notes, Android did not support dynamic loading of eBPF programs well.

Lesson 49 covers HID-BPF: input-device fixups loaded as BPF (SEC("struct_ops/hid_device_event")) without patching drivers. It is the clearest example in the course of eBPF as a general driver-extension mechanism rather than pure observability. The cgroup lesson shows policy enforcement with cgroup/connect4, cgroup/dev and cgroup/sysctl hooks.

GPU and XPU Track

The GPU material comes from the same group that builds bpftime's GPU support:

  • Lesson 47 attaches uprobes to CUDA runtime API calls (cudaMalloc, cudaMemcpy, cudaLaunchKernel) to trace GPU work from the host side. It needs an NVIDIA CUDA GPU.
  • xpu/flamegraph combines eBPF CPU stack capture at cudaLaunchKernel() with CUPTI activity records to produce a CPU-to-GPU flamegraph (the lesson profiles Qwen3 LLM inference as its example).
  • xpu/gpu-kernel-driver uses stable DRM scheduler tracepoints, working across Intel, AMD and Nouveau drivers, with both eBPF and bpftrace.
  • xpu/npu-kernel-driver traces the Intel NPU (ivpu) driver on 6.2+.

None of the xpu/* lessons run in CI because the runners have no GPU or NPU.

Research Grounding

Lesson 18 ties the practical track to systems research the organization considers foundational: XRP in-kernel storage functions (OSDI '22 Best Paper), Jitterbug verified BPF JITs (OSDI '20), Electrode eBPF-accelerated Paxos (up to 128.4% throughput, NSDI '23), BMC in-kernel Memcached cache (up to 18x throughput, NSDI '21), hXDP FPGA offload (OSDI '20) and lambda-IO computational storage (FAST '23). The maintainers' own bpftime work appeared at OSDI '25 ("Extending Applications Safely and Efficiently").

Benchmarks

The tutorial publishes no performance benchmarks of its own, which is appropriate for instructional material. Performance claims inside lessons are attributed: bpftime's "up to 10x" uprobe speedup over kernel uprobes comes from the bpftime project's own benchmarks, and the paper speedups above carry venue citations. Treat both as upstream claims, not independently reproduced here.

Security Model

Security considerations for using this tutorial: the privilege the exercises need, the sensitivity of the data they capture, the dual-use lessons, and supply-chain trust for prebuilt artifacts.

Privilege Model

Every runnable lesson is marked "Root: Required" in the matrix. On kernels 5.8+ the work splits across CAP_BPF, CAP_PERFMON and CAP_NET_ADMIN (table in Reference: Privilege Model), and BPF token (6.9+, features/bpf_token) can delegate a subset of those rights into a user namespace.

Lesson 0's framing

The tutorial presents the verifier as the safety boundary: programs are statically checked before execution, preventing kernel crashes and unsafe memory access, and the JIT then produces native code. These guarantees protect the kernel, not the data an authorized tracer collects.

Data Sensitivity While Learning

Several lessons decode sensitive material by design:

  • Lesson 30 (sslsniff) attaches uprobes to TLS libraries (OpenSSL, GnuTLS, NSS) and prints plaintext, so running it captures the HTTPS traffic of local applications.
  • Lessons 5, 15 and 37 capture user-space function arguments (shell input, JVM GC telemetry, Rust functions).
  • Lessons 39-40 record nginx requests and MySQL queries.

Treat these like credential-dumping tools: run them only on machines you own or are explicitly authorized to instrument. Shared hosts, employer laptops and shared CI runners are poor practice targets even when root is available. Setup steps for a disposable lab are in How-to Guides: Isolate a Practice Environment.

Dual-Use Lessons

A deliberate course arc (lessons 24-28, 34, 51) teaches offense-shaped primitives such as hiding processes, killing processes from BPF, tampering with file reads, rewriting syscall arguments and destroying TCP connections. The point is defender familiarity: knowing what an attacker with CAP_BPF can do is what makes BPF LSM detection (lessons 19, 54) and BPF-token delegation worth learning. None of the lessons go beyond well-known kernel-programmability techniques. The per-lesson caution table is in Reference: Dual-Use Lesson Warnings.

Artifact Supply Chain

ecli runs either locally compiled packages or remote ones such as ghcr.io/eunomia-bpf/execve:latest. Remote mode executes third-party-compiled bytecode with your privileges, and the eunomia-bpf and tutorial READMEs read for this note describe no signing or provenance scheme for these images. For anything security-sensitive, compile locally from source you have read (ecc minimal.bpf.c) and compare any package.json against the .bpf.c it claims to match.

Lifecycle Residue

Lesson 28 shows that programs pinned in bpffs (/sys/fs/bpf) or held by a pinned link keep running after their loader exits. Teardown is therefore not automatic: an orphaned tracing program silently keeps collecting, which is a privacy leak in exactly the spirit of the tracing lessons. Clean-up commands are in How-to Guides: Clean Up After Experiments.

Sources