Network Namespace Routing Debugging: When veth Pairs Don't Forward

A missing ip_forward in a nested network namespace cost us an afternoon — traffic entered the veth pair, ARP resolved, and packets died silently at the namespace boundary. This checklist is what we now run before touching CNI plugins, rootless containers, or test harnesses that spin up netns by hand.

The failure (2025-02-03)

Setup:

• Host: Ubuntu 22.04, kernel 5.15
• Test harness creates ip netns add test-ns, veth pair veth0 (host) / veth1 (ns)
• Host side: 192.168.100.1/24, namespace side: 192.168.100.2/24
• Goal: ping from host → namespace and namespace → host

Symptom: host → ns worked; ns → host failed. tcpdump on veth0 showed ICMP echo request arriving on host; no reply left veth1.

Not DNS. Not iptables on INPUT (policy ACCEPT). Not wrong subnet.

Root cause

Inside test-ns, we had not enabled forwarding on veth1 and host net.ipv4.ip_forward was 1 globally but the namespace had its own sysctl tree defaulting to 0.

For L3 forwarding through a veth (namespace acting as router to another interface), you need ip_forward=1 in every namespace that forwards, not just the host.

ip netns exec test-ns sysctl -w net.ipv4.ip_forward=1
# persistent in harness:
echo 1 > /proc/sys/net/ipv4/ip_forward  # from within netns

Our case was simpler — we expected direct connected subnet behavior. The actual bug was rp_filter strict mode on the host dropping replies because the route lookup did not expect sources from the veth subnet on that interface.

Fix:

sysctl -w net.ipv4.conf.veth0.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=2  # loose mode, safer global default

Reverse path filtering bites nested namespaces constantly.

Checklist (run in order)

1. Link and address sanity

ip netns exec test-ns ip -br addr
ip -br addr show veth0
ip netns exec test-ns ip route

Expect connected route for peer /24 or explicit default via peer.

2. Reachability at L2

ip netns exec test-ns arping -I veth1 192.168.100.1

If ARP fails, check veth peer index pairing: ip link show veth1 → peer veth0.

3. Forwarding flags per namespace

ip netns exec test-ns cat /proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/ip_forward  # host

Both must be 1 if routing between interfaces inside ns.

4. rp_filter

sysctl net.ipv4.conf.{all,default,veth0,veth1}.rp_filter

Log values. Strict 1 on wrong iface = silent drop.

5. iptables/nftables — all tables

iptables-save
ip netns exec test-ns iptables-save
nft list ruleset

Check FORWARD chain policy, Docker/CNI injected rules, -i veth+ -j DROP leftovers.

Kubernetes: iptables -L -n -v | grep -i drop on node.

6. conntrack and NAT

If SNAT is involved, verify reverse path and nf_conntrack_max not exhausted:

cat /proc/sys/net/netfilter/nf_conntrack_count

7. tcpdump both ends simultaneously

tcpdump -i veth0 -n icmp &
ip netns exec test-ns tcpdump -i veth1 -n icmp

Pattern: request on one side only → routing; request both, reply one → filter or rp_filter.

Nested namespace gotcha

ip netns add inner inside outer creates a nested netns. Sysctls in outer do not propagate. Our CI script now templates:

for ns in host outer inner; do
  ip netns exec $ns sysctl -w net.ipv4.ip_forward=1
done

Document in runbook — do not assume.

MTU and fragmentation in veth paths

A separate afternoon lost to MTU mismatch: host veth MTU 1500, inner namespace default route via tunnel with MTU 1400, TCP MSS not clamped. Symptom: small pings work, large HTTPS fails. Fix with iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu or consistent MTU on both sides.

For IPv6, duplicate address detection and router advertisements behave differently per namespace — verify accept_ra sysctl if default routes vanish after link bounce.

CNI-specific notes

• Flannel VXLAN: check flannel.1 and route to pod CIDR via gateway in main table
• Calico: verify BPF vs iptables dataplane — calico-node logs on IP pool mismatch
• kind/minikube: double NAT; host firewall (ufw) blocking bridge traffic — disable ufw or allow kind bridge

For production observability of drops, see eBPF observability — but fix forwarding first; probes won't fix wrong sysctl.

What I'd do next

Add a pre-flight netns self-test to our integration test runner: create ns, veth, ping both directions, fail in 2 seconds with sysctl dump on failure. We would have caught rp_filter in CI, not on a laptop.

Automate collection of ip -d link, ip rule, and full sysctl net.* into incident tickets — one script, attached to every "network weird" PagerDuty alert.

Memory and cgroup limits on nodes running heavy CNI churn interact with conntrack table size — cross-reference cgroups v2 memory limits when debugging OOM during network stress tests.