eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack

· By José Manuel Román Fernández Checa
Contents

TL;DR

eBPF is a sandboxed virtual machine inside the Linux kernel that runs verified code at well-defined hooks: kprobes, tracepoints, socket events, network drivers. Before eBPF, changing kernel behaviour meant recompiling it or loading an arbitrary module; with eBPF, you load a small program that passes a formal verifier and runs at native speed with memory safety. In networking, this translates into the fact that the packet does not have to travel through the traditional TCP/IP stack: an eBPF program in the NIC driver (XDP) can drop, forward or rewrite the packet before the kernel has done its first alloc; a program on cgroup hooks (sock_ops) can redirect connections to another socket without the packet ever leaving the machine. Cilium is the CNI that has taken this to its logical conclusion: it replaces kube-proxy with pure eBPF (O(1) instead of the O(N) of iptables), routes pod-to-pod without VXLAN where it can, evaluates Network Policies with BPF maps, and since 1.16 it has remade its BGP control plane with a new set of CRDs, CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, CiliumBGPNodeConfigOverride, which replace the monolithic CiliumBGPPeeringPolicy that is already deprecated. This post goes down the three layers (basic eBPF → eBPF networking → Cilium) and ends with the operational CRDs.

The analogy: signed plugins for the kernel

Think of the browser. Twenty years ago, extending a browser meant compiling a native binary and loading it: any extension could crash it, corrupt memory, read your bank cookies. Today, extensions are JavaScript in a sandbox with a manifest that declares permissions, a runtime that enforces the isolation and a store that signs the code. The extension does not touch the browser binary; it lives in a controlled world and can only talk to the browser through defined APIs. Result: massive extensibility with a bounded attack surface.

eBPF is exactly that for the Linux kernel. Loading a classic .ko module means loading native code with full access to kernel memory: one bug and the system is gone. eBPF is a bytecode VM with a static verifier, a controlled allocator, JIT to native hardware after passing the verifier, and a set of kernel “helpers” it can call. The eBPF program can read the kernel memory the verifier allows it to read, and only that. It cannot enter infinite loops (the verifier demands that it terminate). It cannot jump to arbitrary addresses. It cannot dereference pointers without having validated them first. And, most importantly: it is user code, loaded at runtime, executing inside the kernel at native speed.

The consequences are visible from miles away. Before, observing traffic in production meant patching the kernel or loading a risky module. Today, bpftrace -e 'tracepoint:net:net_dev_xmit { @[args->dev->name] = count(); }' gives you a histogram of packets per interface in three lines and zero downtime. Before, replacing iptables with something faster meant rewriting the netfilter subsystem. Today, Cilium loads 60 KB of eBPF bytecode into XDP and unseats iptables with a hash map.

Basic eBPF: what it is and what it is not

The origin and the scope

The name comes from Berkeley Packet Filter, a 1992 idea (McCanne and Jacobson) for filtering packets with a mini-bytecode that tcpdump used internally. In 2014, Alexei Starovoitov renamed it eBPF and extended it enormously: 11 64-bit registers instead of 2 32-bit ones, a 512-byte stack, maps as structures shared with userspace, JIT to native hardware, and a far more sophisticated formal verifier. From being a packet filter, it became a generic kernel extensibility mechanism.

Today eBPF is used for four things:

  1. Networking: XDP, TC, cgroup hooks, socket ops, lightweight tunnels.
  2. Observability: kprobes, uprobes, tracepoints, USDT. The basis of projects like bpftrace, bcc, Pixie, Parca.
  3. Security: BPF LSM (Linux Security Module in eBPF), syscall blocking with seccomp-bpf. Falco, Tetragon, Tracee.
  4. Scheduling: sched_ext (kernel 6.12+), process schedulers written entirely in eBPF. Still at a very early stage.

The VM

An eBPF program is compiled from C (or Rust, or Go with cilium/ebpf) to eBPF bytecode, not to x86/arm64 directly. The kernel loader (via the bpf() syscall) passes that bytecode through the verifier:

  • It reconstructs the control flow graph.
  • It performs static analysis of every possible path: every instruction has to be reachable, every memory access has to be within known bounds, every pointer has to have been validated.
  • It rejects loops without a known upper bound. Recent kernels admit bounded loops (the bpf_loop helper), but the counter is always finite.
  • It rejects calls to helpers or kfuncs that the hook’s program type does not allow.

If the verifier accepts the program, the JIT translates it to the host’s native code (x86, arm64, etc.) and it stays attached to its hook. From then on it runs every time the hook’s event occurs, with no context switch to userspace, with no syscall cost. Latencies on the order of hundreds of nanoseconds per invocation.

Maps: the bridge to userspace

An isolated eBPF program is not much use. What makes it useful are maps: data structures shared between the kernel program and userspace. There are several types:

  • BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH: hash tables with or without LRU eviction.
  • BPF_MAP_TYPE_ARRAY, BPF_MAP_TYPE_PERCPU_ARRAY: arrays, optionally per-CPU to avoid contention.
  • BPF_MAP_TYPE_RINGBUF, BPF_MAP_TYPE_PERF_EVENT_ARRAY: channels for streaming events to userspace.
  • BPF_MAP_TYPE_PROG_ARRAY: arrays of eBPF programs for tail calls (chaining programs without returning to the base kernel).

Userspace reads and writes these maps via bpf() syscalls; the kernel program reads and writes them directly. It is the basis of any eBPF system: the kernel program collects data into a map, the userspace daemon reads it. Cilium does exactly this: the userland agent (Go) manages the policy and translates it into map entries; the eBPF programs living in XDP/TC read the maps and apply the decisions.

CO-RE: compile once, run on any kernel

A classic nightmare of kernel modules: they are tied to the exact kernel version they were compiled against. Distributing a precompiled module for a fleet of machines with different distros was impossible.

eBPF solves this with CO-RE (Compile Once, Run Everywhere): the bytecode includes relocations that the loader resolves on each specific kernel by consulting BTF (BPF Type Format), a representation of the kernel’s struct layouts that the kernel itself publishes. Result: a single eBPF binary works on kernels 5.10, 5.15, 6.1 and 6.8 without recompiling, because the loader adjusts the struct access offsets at runtime.

This is what has allowed productive eBPF distributions to exist. Without CO-RE, every kernel would be a porting project.

eBPF in networking: the hooks that matter

Inside the Linux network subsystem, eBPF has several hooks. The ones relevant to CNIs:

XDP — eXpress Data Path

XDP is the earliest hook: it runs in the NIC driver, before the packet enters the kernel proper. There is no sk_buff (the struct the rest of the kernel uses to represent packets); there is only a pointer to a RAM buffer with the received bytes.

The actions an XDP program can return:

  • XDP_DROP: discard the packet immediately. The driver drops it and frees the buffer. Cost: nanoseconds. Use case: DDoS mitigation. Cloudflare processed more than 8 million packets/second per CPU with XDP for dropping SYN floods.
  • XDP_PASS: let the packet continue to the normal kernel. It becomes an sk_buff and enters the traditional stack.
  • XDP_TX: send it back out on the same interface after possible modifications. Useful for L4 load balancers that rewrite the destination and return it.
  • XDP_REDIRECT: send the packet to another interface or to a map (to forward to userspace via AF_XDP, or to another NIC, or to a pod’s veth).
  • XDP_ABORTED: error (increments a counter, drops).

Real use cases:

  • Cloudflare L3 DDoS protection: XDP rules that drop millions of packets/s.
  • Facebook Katran: an L4 load balancer that rewrites the destination IP and returns it on the same interface. Handles 10× more connections per server than classic IPVS.
  • Cilium XDP acceleration: Service load balancing at the lowest layer possible.

TC (Traffic Control) — clsact with BPF

XDP is very fast but limited: the packet does not have an sk_buff yet and many decisions (conntrack, NAT, encapsulation with metadata) are easier when it does. The TC clsact with BPF hook runs after building the sk_buff but before the routing and netfilter decisions. Actions:

  • TC_ACT_OK: the packet continues through the stack.
  • TC_ACT_SHOT: drop.
  • TC_ACT_REDIRECT: redirect to another interface.
  • TC_ACT_PIPE, TC_ACT_STOLEN: pipeline control for combining with other qdiscs.

Use cases:

  • Stateful network policy: Cilium evaluates L3-L7 policies in TC with the full sk_buff and conntrack available.
  • Marking and QoS: traffic marking so the scheduler applies priorities.
  • Overlay encapsulation: adding VXLAN/Geneve headers when the mode is tunnel.

XDP and TC combine: XDP for the cheap and early stuff (DDoS, simple LB), TC for what needs an skb and state.

Cgroup hooks: sock_ops and CGROUP_SOCK_ADDR

The most radical conceptual step: hooks that are not in the network layer but in the socket layer. Relevant types:

  • BPF_PROG_TYPE_CGROUP_SOCK_ADDR: invoked when a process in a cgroup calls connect(), bind(), sendto(). The eBPF program can rewrite the destination address before the connection goes out. This is what lets Cilium do Service load balancing without the packet entering the network stack: if the client tries to connect to 10.96.0.1:443 (a ClusterIP), an eBPF program on this hook rewrites the destination to the real IP of the backend pod before the syscall continues.
  • BPF_PROG_TYPE_SOCK_OPS: invoked on TCP events (creation, established, retransmission). It allows tuning socket parameters at runtime and, most importantly, pairing local sockets via bpf_sk_assign to shortcut without the packet travelling over the network.

This is the “third layer” of the bypass: it is not just faster, it is conceptually different. The packet is not built, not serialised, does not traverse the IP layer or the TCP layer. It is the difference between speeding up a road and discovering that for some journeys you do not need to take the car at all.

The long road: what the traditional TCP/IP stack looks like

To appreciate what eBPF saves, it is worth tracing a packet’s full journey through the Linux stack. Take the case “packet arrives on a NIC, goes to a local process”:

NIC (DMA into the driver's ring buffer)
  ↓
driver: napi_schedule, poll, allocates sk_buff
  ↓
[XDP hook]                    ← if there is an XDP program, it is decided here
  ↓
netif_receive_skb
  ↓
__netif_receive_skb_core
  ↓
[TC ingress clsact + BPF]     ← if there is a TC ingress program
  ↓
packet_type handlers (IP, ARP...)
  ↓
ip_rcv → ip_rcv_core
  ↓
[netfilter NF_INET_PRE_ROUTING]   ← iptables PREROUTING
  ↓
routing decision (FIB lookup)
  ↓
[netfilter NF_INET_LOCAL_IN] or [NF_INET_FORWARD]
  ↓
tcp_v4_rcv → tcp_v4_do_rcv
  ↓
tcp_rcv_established
  ↓
sk_data_ready
  ↓
process reads with recv()/read()

Every arrow is a function call with a measurable cost. Every netfilter hook walks all the registered iptables/nftables rules. With kube-proxy in iptables mode and 5,000 Services × 10 endpoints each, there are on the order of 150,000 rules evaluated sequentially at NF_INET_PRE_ROUTING. Published benchmarks show latencies of tens of microseconds per packet in large Kubernetes clusters in the netfilter step alone, before the application receives anything.

And that is the normal path. On the way out the same thing happens in reverse: tcp_sendmsg → ip_output → NF_INET_LOCAL_OUT → routing → NF_INET_POSTROUTING → dev_queue_xmit → driver → NIC.

How Cilium skips this stack

Cilium does not eliminate the TCP/IP stack; it is still there for the cases that need it. What it does is shortcuts at the points where it hurts.

Shortcut 1 — XDP for the Service datapath

For a cluster with 5,000 Services, kube-proxy iptables has an O(N) cost in evaluating rules (even with iptables-restore --noflush and tricks, it is still linear in the number of chains the packet traverses).

Cilium replaces it like this:

  • Every Service and its endpoints live in an eBPF hash map.
  • When a packet comes in destined for a ClusterIP, Cilium’s XDP program does an O(1) lookup in that map and obtains the backend endpoint.
  • It rewrites the destination and does XDP_TX (returns it on the same interface towards the backend) or XDP_REDIRECT (sends it to the corresponding local pod’s veth).

This means the cost does not grow with the number of Services. 100 Services or 100,000, constant lookup in the map. Published benchmarks show latency reductions of 30-50% in clusters with many Services compared with kube-proxy iptables, and of an order of magnitude compared with IPVS in some cases.

Shortcut 2 — socket-LB: the packet is never built

Cilium 1.6+ introduced socket-level load balancing, based on cgroup hooks. It works like this:

  • When a pod calls connect(10.96.0.1:443) (a Service’s ClusterIP), the syscall enters the kernel.
  • Before the kernel builds anything network-related, an eBPF program on CGROUP_SOCK_ADDR/connect4 intercepts it and rewrites the destination address to the real IP of the backend pod.
  • The kernel carries on with the connect as if the client had written 10.0.0.42:8080 directly.

Why does it matter? Because when the backend pod is on the same node, this shortcut turns a call that would have involved:

syscall connect → kernel stack → veth → bridge → veth → kernel stack → syscall accept

into:

syscall connect (with the destination rewritten) → direct loopback

The TCP/IP stack is literally avoided. There is no encapsulated packet, no journey through veth pairs, no netfilter. L7 pod-to-pod latencies on the same node drop to local communication levels (~5-15 µs instead of ~30-50 µs for services with kube-proxy iptables and traditional veth).

Shortcut 3 — pod-to-pod direct routing

The traditional overlay mode (Flannel, Calico VXLAN) encapsulates every pod-to-pod packet in VXLAN/Geneve. Every packet carries an extra 50-byte header, requires encap/decap, and consumes MTU.

Cilium supports direct routing: the pod CIDRs are advertised to the underlying fabric (with BGP, which is where the control plane we will look at comes in) and the physical routers route the pod-to-pod packets without encapsulating. The packet leaves a pod with its original IP as source and the destination pod’s IP as dest, the node’s NIC hands it to the network, the network routes it, it arrives at the destination node and is delivered to the pod. Zero encap, full MTU, minimal latency.

Cilium does this via eBPF programs in TC that rewrite the necessary headers and decide whether the packet goes via encap or direct according to the policy configured per node.

Shortcut 4 — Network Policy in TC with maps

Network Policies in classic CNIs are usually translated into iptables rules, another factor that explodes linearly. Cilium evaluates them in eBPF programs that read identity maps: every workload has a numeric identifier computed from its labels, and the policy is a map (src_identity, dst_identity, port, proto) → allow|deny. One hash map lookup per packet.

This also enables Cilium’s L7 policies (HTTP, gRPC, Kafka filtering): the eBPF program recognises the L7 handshake, selectively redirects to the embedded Envoy proxy (which lives as a sidecar of the datapath, not as a pod sidecar) and only on that subset does it pay the cost of the L7 proxy. All the L3/L4 traffic stays on the eBPF fast path.

Cilium: the architecture

Cilium combines two planes:

  • Agent (Go): lives as a DaemonSet on every node. It is the “slow” part: it translates the intent expressed in CRDs (CiliumNetworkPolicy, CiliumBGPClusterConfig, etc.) into entries in eBPF maps. It talks to the Kubernetes API server to discover endpoints, services, pods. It embeds a GoBGP for the BGP control plane. It embeds an Envoy for L7 policies.
  • Datapath (eBPF): the programs loaded into XDP, TC, cgroup hooks. They are the “fast” part: they see every packet, read the maps the agent maintains, and decide in nanoseconds.

This separation is what makes Cilium operationally comfortable: the intent is expressed in YAML, the agent materialises it into maps, the maps are read by the datapath. If the agent goes down temporarily, the datapath keeps working with the last loaded configuration. As in any well-built control/data plane system.

BGP Control Plane v2: the CRDs you have to know

Cilium has had BGP support for several years. The first version used a single monolithic CRD, CiliumBGPPeeringPolicy, which mixed node configuration, peers, timers and advertisements into a single object. Since Cilium 1.16 there is BGP Control Plane v2, which breaks that configuration into separate CRDs with clear responsibilities. CiliumBGPPeeringPolicy (the cilium.io/v2alpha1 API) is deprecated and migration warnings appear in the operator logs if you still use it.

The new CRDs (the cilium.io/v2 API):

1. CiliumBGPClusterConfig

Defines BGP instances and the peers they connect to, from the cluster’s perspective. Which nodes apply this configuration is selected with a nodeSelector.

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp-cluster
spec:
  nodeSelector:
    matchLabels:
      bgp-policy: rack-1                   # only nodes with this label
  bgpInstances:
  - name: instance-65000
    localASN: 65000
    peers:
    - name: top-of-rack-1a
      peerASN: 64512
      peerAddress: 10.0.1.1
      peerConfigRef:
        name: tor-shared-config            # → reference to CiliumBGPPeerConfig
    - name: top-of-rack-1b
      peerASN: 64512
      peerAddress: 10.0.1.2
      peerConfigRef:
        name: tor-shared-config

A BGP instance is the abstraction “this node takes part in BGP with this local ASN and these peers”. Several can coexist on the same node (multi-instance for multi-VRF).

2. CiliumBGPPeerConfig

Defines the shared parameters of the peering: timers, address families, transport, MD5 password, graceful restart, etc. It is referenced from CiliumBGPClusterConfig via peerConfigRef. This avoids repeating the same configuration for every peer when there are dozens of them.

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tor-shared-config
spec:
  timers:
    holdTimeSeconds: 30
    keepAliveTimeSeconds: 10
    connectRetryTimeSeconds: 5
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: bgp                       # → binds to CiliumBGPAdvertisement
  - afi: ipv6
    safi: unicast
    advertisements:
      matchLabels:
        advertise: bgp
  authentication:
    password:
      name: bgp-md5-secret                   # Secret with the MD5 password
      key: password

A single CiliumBGPPeerConfig can be referenced by many different peers. You change timers or families in one place.

3. CiliumBGPAdvertisement

Declares which prefixes are advertised: the node’s pod CIDRs, the ClusterIPs and ExternalIPs of Services, the IPs assigned by CiliumLoadBalancerIPPool for type=LoadBalancer Services. They are bound to CiliumBGPPeerConfig via labels.

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: services-and-pods
  labels:
    advertise: bgp                          # ← the label the PeerConfig matches
spec:
  advertisements:
  - advertisementType: PodCIDR              # advertises the node's pod CIDR
    attributes:
      communities:
        standard:
        - "65000:100"
  - advertisementType: Service              # advertises ClusterIPs / LoadBalancer IPs
    service:
      addresses:
      - LoadBalancerIP
      - ClusterIP
      - ExternalIP
    selector:
      matchLabels:
        bgp-advertise: "true"               # only Services with this label
    attributes:
      communities:
        standard:
        - "65000:200"
      localPreference: 200

The granularity is very fine: you can advertise different types of prefixes with different BGP communities, different local-preference, different path attributes, and filter Services with label selectors. This was literally impossible with CiliumBGPPeeringPolicy v1.

4. CiliumBGPNodeConfig (auto-generated)

This CRD is not configured by hand. The Cilium operator generates one per node from the CiliumBGPClusterConfig that applies to that node. It is the materialised per-node state that each node’s agent reads to bring up its peerings. If you want to see what BGP configuration is actually running on a node, kubectl get ciliumbgpnodeconfig <nodename> -o yaml shows you.

5. CiliumBGPNodeConfigOverride

Optional. It allows overriding the generated configuration for a specific node when you need something non-standard. Use cases:

  • Pinning the BGP router-id to a specific IP (useful when the node has several interfaces).
  • Specifying the peer’s local address when there are several outgoing interfaces.
  • Changing timers only for one problematic node.
apiVersion: cilium.io/v2
kind: CiliumBGPNodeConfigOverride
metadata:
  name: node-rack1-master01                  # the name must match the node's
spec:
  bgpInstances:
  - name: instance-65000
    routerID: 10.0.1.10                      # router-id override
    peers:
    - name: top-of-rack-1a
      localAddress: 10.0.1.10                # specific local interface

Relationship diagram

Cilium BGP Control Plane v2 CRDs and their relationshipsCiliumBGPClusterConfignodeSelector + bgpInstancesCiliumBGPPeerConfigtimers, families, authCiliumBGPAdvertisementpod CIDR, Service IPsCiliumBGPNodeConfigauto-generated by the operatorCiliumBGPNodeConfigOverrideoptional, by node namepeerConfigRefbinds via labelsadvertisementsoperatoroverridesolid arrows: direct YAML references. Dashed: links by label selector or lateral coordination.

CiliumLoadBalancerIPPool: it complements, it is not BGP

Although it is not strictly a BGP CRD, it is worth mentioning: CiliumLoadBalancerIPPool is the CRD that supplies IPs to type=LoadBalancer Services. It defines a range (10.20.0.0/24, for example) that Cilium assigns automatically to LoadBalancer Services. Combined with a CiliumBGPAdvertisement that advertises LoadBalancerIP, it gives the complete cycle: new Service → IP assigned from the pool → BGP advertisement to the routers → IP routable from the corporate network, with no external balancer.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: lb-pool-rack1
spec:
  blocks:
  - start: "10.20.0.10"
    stop: "10.20.0.250"
  serviceSelector:
    matchLabels:
      lb-pool: rack1

Full manifest: pod CIDRs + LoadBalancer Services advertised to a redundant ToR pair

A realistic example of a cluster with two top-of-rack switches as BGP peers, both in the same AS (64512), Cilium in AS 65000:

# 1. CiliumBGPPeerConfig — shared config for both ToRs
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tor-peers
spec:
  timers:
    holdTimeSeconds: 30
    keepAliveTimeSeconds: 10
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: bgp

# 2. CiliumBGPAdvertisement — what gets advertised
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: pods-and-lb
  labels:
    advertise: bgp
spec:
  advertisements:
  - advertisementType: PodCIDR
  - advertisementType: Service
    service:
      addresses: [LoadBalancerIP]
    selector:
      matchExpressions:
      - { key: io.kubernetes.service.namespace, operator: NotIn, values: [kube-system] }

# 3. CiliumBGPClusterConfig — which nodes talk to which peers
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cluster-bgp
spec:
  nodeSelector:
    matchLabels:
      bgp: enabled
  bgpInstances:
  - name: instance-65000
    localASN: 65000
    peers:
    - name: tor-a
      peerASN: 64512
      peerAddress: 10.0.1.1
      peerConfigRef: { name: tor-peers }
    - name: tor-b
      peerASN: 64512
      peerAddress: 10.0.1.2
      peerConfigRef: { name: tor-peers }

# 4. CiliumLoadBalancerIPPool — range of LB IPs
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: lb-corporate
spec:
  blocks:
  - cidr: "10.20.0.0/24"

Four objects. Before, in v1, it was a single CiliumBGPPeeringPolicy that mixed everything together and turned out to be hard to maintain in clusters of more than 5 nodes with heterogeneous configuration. The new separation is longer but clearly factorable: one PeerConfig per peer type, one Advertisement per advertisement policy, one ClusterConfig connecting nodes with peers.

Common operational traps

routingMode: tunnel mode with BGP

BGP only makes sense with direct routing (routingMode: native). If you have tunnel mode (VXLAN/Geneve) and configure BGP, you will advertise pod CIDRs but the packets will still go out encapsulated, producing confusing behaviour (sometimes via tunnel, sometimes direct depending on routes). Configure routingMode: native and disable the tunnel.

eBPF host routing vs kube-proxy replacement

They are two different things. kubeProxyReplacement: true enables the replacement of kube-proxy (the Services). bpf.hostRouting: true enables the host’s iptables bypass (the node’s routing decisions are made with eBPF instead of the traditional FIB). The second needs kernel 5.10+ with all the bpf features enabled; if you do not have that kernel, it falls back to legacy mode and the performance is only “almost as good”.

Aggressive BGP timers over flapping NICs

With holdTimeSeconds: 9 / keepAliveSeconds: 3, a NIC that blinks for 5 seconds breaks the BGP session and all the advertised routes disappear from the fabric. That node’s pods become unreachable until the session is re-established. For clusters on hardware with suspect NICs, use the conservative values (holdTime: 30, keepAlive: 10) and consider graceful restart explicitly (it is already in the example above).

Advertising ClusterIP to the corporate network

Advertising ClusterIP to external routers is rarely what you want: these are internal Service IPs, not designed to be reached from outside the cluster. For external exposure, use LoadBalancerIP from a CiliumLoadBalancerIPPool. Advertising ClusterIP only makes sense in very specific topologies (multi-cluster mesh with shared service discovery).

Mixing v2alpha1 (CiliumBGPPeeringPolicy) and v2 (CiliumBGPClusterConfig)

It does not work well. The operator emits warnings in the logs about the use of the deprecated API, and conflicts between what the peering policy defines and what the cluster config defines can produce strange states. Migrate from one to the other in a single pass; do not run both.

MD5 password and MTU

If you configure an MD5 password in CiliumBGPPeerConfig.authentication, the TCP header is larger. On links with a tight MTU (1500 - 50 for the upstream fabric’s VXLAN, for example), the BGP handshake can fragment and die silently. Either use MTU 9000 between nodes and ToR, or make sure the MSS values are negotiated correctly.

What we have not covered

  • Cilium Cluster Mesh: federation of several Cilium clusters so their Services can see each other. It fits with BGP when you want native routing between clusters; it has its own CRDs.
  • L7 Policies and the embedded Envoy: HTTP/gRPC/Kafka policy. Another layer of eBPF + proxy that deserves its own post.
  • Hubble: eBPF-based traffic observability that Cilium exposes. Flow log dashboards with zero impact on latency.
  • Transparent WireGuard: pod-to-pod encryption without sidecars, controlled by Cilium via eBPF redirect to a kernel WireGuard dataplane.
  • Gateway API in Cilium: the successor to Ingress, with first-class support since Cilium 1.16+.
  • eBPF for LLM serving: the natural connection with the previous inference series. There is recent work using eBPF for multi-tenant fairness on GPUs and for token tracking; paper territory, not production yet.

In other words, there is material left for another three posts in today’s series. Let us take them in order.

References

Conceptual and project:

XDP, TC and firewalling:

Cilium:

Cross-references: