Skip to content

Networking

Domain summary

Container Network Interface (CNI) plugins for Kubernetes: pod networking, network policy enforcement, encryption, and observability across cluster nodes. The domain covers the three most common open-source CNIs. Cilium (eBPF, CNCF Graduated) is the feature-rich option, Calico (Tigera, five data planes, BGP) is the policy- and routing-centric option, and Flannel (SUSE-maintained, VXLAN) is the minimal overlay.

← Knowledge Base

How the Topics Relate

Each CNI plugs into the kubelet through the CNI spec, but they differ in data plane and in how much they do beyond pod connectivity. Canal (Calico policy on Flannel networking) and Cilium chaining are the two common ways the plugins are combined.

flowchart TB
    KUBELET["kubelet + container runtime<br/>(CNI ADD / DEL)"]
    KUBELET --> FL["Flannel<br/>flanneld: VXLAN / host-gw / WireGuard"]
    KUBELET --> CA["Calico<br/>Felix: iptables / nftables / eBPF / HNS / VPP<br/>BIRD: BGP"]
    KUBELET --> CI["Cilium<br/>cilium-agent: eBPF (TC/tcx, XDP, socket)"]
    FL -->|"Canal: Calico policy on Flannel networking"| CA
    FL -->|"CNI chaining: Cilium policy on Flannel"| CI
    FL -.->|"policy add-ons"| NP["kube-network-policies (Helm option)<br/>K3s embedded kube-router"]
    CA --> CAX["Whisker / Goldmane flow logs<br/>Calico Ingress Gateway (Envoy Gateway)"]
    CI --> CIX["Hubble flow observability<br/>Gateway API, Cluster Mesh, Tetragon"]

Topics

Plugin Description Latest Version License
Calico Tigera's CNI and policy engine with BGP, VXLAN or IP-in-IP routing and five data planes (iptables, nftables, eBPF, Windows HNS, VPP). Ordered policy tiers, and the same policy model for VMs and hosts v3.32.2 (2026-08-29) Apache-2.0 (Open Source). Enterprise and Cloud editions are proprietary
Cilium eBPF-based CNI with identity-based L3-L7 policy, kube-proxy replacement, Gateway API, sidecar-free mesh, Cluster Mesh, Hubble observability and the Tetragon sub-project. CNCF Graduated 1.20.2 (2026-09-15) Apache-2.0 (eBPF code GPL-2.0 / BSD-2-Clause)
Flannel Minimal layer 3 fabric: one flanneld per node, VXLAN by default (host-gw, WireGuard). Default in K3s and the networking half of Canal. No built-in NetworkPolicy v0.28.9 (2026-08-07) Apache-2.0

Comparisons

Comparison Scope
CNI Comparison Calico vs Cilium vs Flannel: features, performance estimates, resource use, operations, multi-cluster, migrations, edge support, and a decision flowchart

All comparisons: comparisons index.

When to Use Which

This summarizes the CNI Comparison decision guide.

Situation Pick Why
Production cluster that needs L7 policy, flow observability, kube-proxy replacement or multi-cluster in open source Cilium Hubble, L7 HTTP/gRPC/DNS policy, Cluster Mesh are all Apache-2.0
On-prem fabric that peers with top-of-rack routers over BGP, or ordered policy tiers Calico Native BGP (BIRD), tiers with Deny/Pass, global and host policy
Windows worker nodes Calico (or Flannel for connectivity only) Calico has a Windows HNS data plane. Cilium is Linux only
Protecting VMs or bare-metal hosts outside Kubernetes Calico Host endpoints and non-Kubernetes workloads. Cilium removed external workloads in 1.18
Kernel older than 5.10 (other than RHEL 8.x backports) Calico iptables or Flannel Cilium 1.18+ and Calico eBPF need 5.10 (RHEL 8 backports excepted)
Dev, CI, labs, K3s or small edge clusters Flannel One DaemonSet, ~50Mi memory request, default in K3s
Flannel cluster that now needs NetworkPolicy Add a policy controller, then consider Calico or Cilium kube-network-policies (Helm), K3s kube-router, Canal or Cilium chaining

Landscape

eBPF is reshaping Kubernetes networking. It lets CNI plugins implement packet forwarding, load balancing, network policy and observability in the Linux kernel, without long iptables chains or userspace proxies. Cilium led this shift. It is the data plane behind GKE Dataplane V2 and AKS's "Azure CNI Powered by Cilium", and EKS supports it through Helm. Calico responded by adding its own eBPF data plane alongside its iptables and nftables (GA in 3.31) backends.

The sidecar-less trend from the service mesh world is spreading into CNI territory. Cilium ships a sidecar-free mesh (eBPF plus a per-node Envoy). Calico 3.32 bundles Istio ambient mode (tech preview) and offers application layer policy through Dikastes alongside Istio's Envoy. Network policy enforcement has grown from basic L3/L4 Kubernetes NetworkPolicy to identity-aware policies (CiliumNetworkPolicy, Calico GlobalNetworkPolicy with tiers) and, in Cilium, L7 filtering by HTTP path, gRPC method or DNS FQDN. Upstream ClusterNetworkPolicy is now supported by both Cilium 1.20 and Calico 3.32.

Both Cilium and Calico now implement the Kubernetes Gateway API: Cilium natively (Gateway API v1.6.1 in 1.20) and Calico through Calico Ingress Gateway, a hardened Envoy Gateway build (GA in 3.31).

IPAM Diversity

IPAM strategies vary significantly across CNI plugins: host-scope IPAM for overlay networks (Flannel), cluster-scope IPAM with block borrowing for routed networks (Calico BGP), and cloud-provider IPAM that allocates ENIs or VPC IPs directly to pods (AWS VPC CNI, Azure CNI, Cilium ENI mode). The choice of IPAM strategy has downstream implications for VPC address space consumption, pod-to-external-service routing, and multi-cluster networking.

Encryption in transit has also moved into the CNI layer. Cilium supports transparent WireGuard and IPsec encryption between nodes (ztunnel-based mTLS is beta). Calico offers WireGuard encryption for its eBPF and iptables data planes. Flannel has a WireGuard backend. None of these replaces mTLS workload identity from a service mesh.

Key Concepts

CNI Plugin Interface

The Container Network Interface specification defines a JSON-based contract between the container runtime and network plugins. When a pod is scheduled, the kubelet invokes the CNI binary with ADD to attach a network interface and assign an IP, and DEL to clean up when the pod terminates.

CNI plugins can be chained: a primary plugin handles IP assignment and routing, while secondary plugins add bandwidth limiting, port mapping, or IPAM delegation. The spec is deliberately minimal, which is why CNI plugins vary enormously in capability, from Flannel's simple VXLAN overlay to Cilium's full eBPF networking stack with service mesh integration.

eBPF vs iptables

Performance Implications

iptables processes packets through a linear chain of rules. This design causes O(n) degradation as the number of services and network policies grows, which becomes noticeable in clusters with thousands of services. eBPF attaches programs directly to kernel hooks (XDP, TC, socket) and uses hash-map lookups whose cost does not grow with the rule count. Cilium's eBPF kube-proxy replacement removes the iptables service chains. Calico's eBPF data plane also bypasses iptables and can replace kube-proxy, while Calico keeps iptables and nftables as alternative data planes.

The kernel version is a practical constraint. Cilium 1.18 to 1.20 require Linux 5.10 or later (or RHEL 8.10's 4.18 kernel), and some features need more (netkit 6.8). Calico's baseline is also Linux 5.10 (its eBPF data plane also runs on RHEL 8.4 kernel 4.18.0-305+), and its nftables data plane needs 5.13+. nftables offers better performance than legacy iptables through atomic rule replacement and native set/map data structures, but it is still a rule-evaluation engine rather than a programmable data path. Flannel's own nftables mode (for masquerading rules) is experimental.

Network Policy

Kubernetes-native NetworkPolicy resources define L3/L4 ingress and egress rules scoped to pods via label selectors. The API is intentionally limited: it lacks cluster-wide scope, explicit deny rules and ordering, and L7 filtering. The upstream ClusterNetworkPolicy API addresses cluster-wide admin policy.

Extended policy capabilities by CNI:

  • Calico (Open Source): GlobalNetworkPolicy (cluster-wide, ordered tiers, Deny/Pass actions), staged policies, host endpoint policy, NetworkSet (external CIDR grouping), and HTTP method/path rules through Application Layer Policy (Dikastes running with Istio's Envoy). DNS policy is a Calico Enterprise and Calico Cloud feature.
  • Cilium: CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy with L7 HTTP and gRPC filtering, DNS-based FQDN rules, and identity-aware enforcement that survives pod IP changes. Kafka L7 policy was removed in 1.20.
  • Flannel: flanneld enforces no NetworkPolicy. Options: the Helm chart's netpol.enabled=true (deploys SIG Network's kube-network-policies, since v0.25.5), K3s's embedded kube-router controller, Canal (Calico policy on Flannel networking, RKE2's default), or Cilium chaining.

IPAM (IP Address Management)

The strategy for allocating IP addresses to pods across a cluster. Host-local IPAM assigns a subnet per node (a /24 by default in Flannel) from a cluster CIDR: simple, but it can waste addresses. Calico's IPAM hands out /26 blocks and lets nodes borrow from each other for efficient utilization.

Cloud-provider IPAM (AWS VPC CNI, Cilium ENI mode) assigns real VPC IPs to pods. This enables direct pod-to-VPC-resource communication without NAT but consumes VPC address space. Dual-stack (IPv4+IPv6) networking has been GA since Kubernetes 1.23, and Calico, Cilium and Flannel all support it.

Overlay vs Routed

Overlay networks (VXLAN, Geneve) encapsulate pod traffic in outer UDP headers. This enables pod networking across any underlying L3 infrastructure without node network cooperation. Routed networks (BGP peering, direct routing) advertise pod CIDRs to the physical network. This avoids encapsulation overhead but requires L3 fabric cooperation.

Calico supports both modes: VXLAN (or IP-in-IP) for cloud environments without BGP support, and native BGP peering for on-premises networks where route reflectors distribute pod routes. Flannel defaults to a VXLAN overlay and offers host-gw (direct routes on a shared L2 network), prioritizing simplicity. Cilium supports VXLAN, Geneve and native routing modes, and has a BGP control plane for advertising pod and service routes.

  • Kubernetes: the cluster networking model and where the CNI fits
  • Service Mesh: Istio ambient/ztunnel, Linkerd and Envoy Gateway, which overlap with Cilium's mesh and Calico Ingress Gateway
  • eBPF Developer Tutorial: eBPF fundamentals behind Cilium, Tetragon and Calico's eBPF data plane
  • Observability: where Hubble and Calico flow logs feed into metrics and tracing stacks

Sources

Open Questions

  • As eBPF becomes the dominant data plane, will there be a practical reason to deploy Flannel (pure overlay, no eBPF) outside of resource-constrained edge environments like K3s on small ARM clusters?
  • With Cilium and Calico both implementing the Gateway API (and Calico bundling Istio ambient), will CNIs keep building their own L7 stacks, or converge on upstream Envoy Gateway and Istio components?
  • With Cilium now providing service mesh, Hubble observability and Tetragon runtime security alongside CNI, does the "do one thing well" Unix philosophy apply, or is an integrated networking platform the right architectural choice?
  • No independent, reproducible benchmark compares Cilium 1.20 eBPF with Calico 3.32 eBPF on current kernels. See the single-source third-party figures in the CNI Comparison.