Wireguard in Production: Latency Profile, Key Rotation, and the Failure Mode Nobody Documents

WireGuard is fast, minimal, and cryptographically sensible — until you need key rotation without dropping active sessions across 140 edge nodes. The man page explains Cryptokey Routing; it does not explain the 90-second outage you get when you mishandle pre-shared key rollover on asymmetric routes.

Lab note from production WireGuard 1.0.20210914 (kernel module on 6.1) + userspace wg-quick on Ubuntu 22.04 jump hosts.

Latency profile (why we chose it)

Compared to OpenVPN 2.5 (same hardware, same path):

Metric	WireGuard	OpenVPN (UDP)
p50 RTT overhead	+0.3 ms	+1.8 ms
p99 RTT overhead	+0.9 ms	+6.2 ms
Handshake CPU (per conn)	Lower	~3x
Throughput 1 Gbps link	940 Mbps	720 Mbps

Measured ping -i 0.01 tunnel vs direct, 10k samples, same-AZ. WireGuard's ChaCha20-Poly1305 on modern x86 with AES-NI available still wins on small packet latency — kernel implementation path is short.

We use WireGuard for site-to-site (HQ ↔ AWS VPC) and admin access, not full client VPN for 2000 employees — different scale than Tailscale's model.

Key rotation: what the docs skip

WireGuard identity is the public key. Rotating a peer's key means updating PublicKey on the remote AllowedIPs peer entry. There is no "grace period" in protocol — old key stops working when remote removes it.

Failure mode we hit

1. Generated new keypair on edge router wg genkey | tee ...
2. Updated HQ server config with new public key, wg syncconf applied
3. Removed old public key from HQ immediately
4. Edge still had old private key active until wg-quick down && up

90 seconds of asymmetric routing: edge sends with old key, HQ rejects. BGP didn't flap; TCP sessions reset. Monitoring blip.

Rotation procedure that works

Dual-key overlap window:

1. Generate new keypair; add second peer block on HQ for same AllowedIPs with new public key (temporarily duplicate routes — WireGuard allows two peers if AllowedIPs differ; same IPs need careful fwmark routing)
2. On edge: add new interface or second peer entry pointing to HQ with new keys
3. Verify bi-directional handshake (wg show latest handshake < 2 min)
4. Remove old peer from both sides
5. Total overlap: 15 minutes scheduled; zero dropped sessions in last 8 rotations

For mesh (multiple peers), use staged rotation: one peer pair at a time. Automate with Ansible; manual wg set at scale is error-prone.

Pre-shared key (PSK) rotation

PSK adds quantum resistance argument; rotation is independent of keypair rotation. Changing PSK requires both sides simultaneous update — no dual-PSK mode. Schedule maintenance window or use keypair rotation overlap only without PSK change mid-window.

Other failure modes nobody warned us about

MTU / PMTUD black holes

Default MTU=1420 in wg-quick. Path with PPPoE or bad ICMP filtering → silent TCP hangs on large transfers. Fix:

MTU = 1280
PostUp = iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

AllowedIPs routing loops

Hub-and-spoke: edge sets AllowedIPs=0.0.0.0/0, HQ sets AllowedIPs=10.50.0.0/16. Edge advertises default via WireGuard; HQ responds; asymmetric return if HQ also has default route via same tunnel. Document topology; use separate routing tables / fwmark.

See network namespace routing debugging — same class of bug in veth setups.

PersistentKeepalive on NAT

Edge behind CGNAT needs PersistentKeepalive = 25. Without it, inbound init fails after NAT binding expires. Corporate WiFi: 25s; datacenter: omit (saves packets).

Monitoring WireGuard health

Metrics we export via node_exporter textfile collector from cron every 60s:

wg show all latest-handshakes dump

Alert if latest handshake > 180s on any peer — NAT binding or routing issue before user tickets. Handshake metric more actionable than ping through tunnel (ping can succeed while throughput is asymmetric).

We log RX/TX bytes per peer — sudden RX drop with stable TX often means AllowedIPs mismatch post-rotation, not cable fault.

Comparison to Tailscale

We run Tailscale on laptops; WireGuard site-to-site on infrastructure. Tailscale's coordination server solves key distribution we manual with Ansible. Cost trade: Tailscale per-seat vs ops time on raw WireGuard. Hybrid is fine — don't force one VPN religion.

Security adjacent

WireGuard gives encryption, not authorization. We pair with:

• SSH only via tunnel for admin (no WireGuard → full network)
• iptables/nftables on HQ limiting source IPs per public key
• systemd hardening on wg-quick@ units

What I'd do next

1. Evaluate WireGuard dynamic routing (BIRD/BGP over tunnel) vs static AllowedIPs — scaling pain at 140 nodes
2. Test kernel 6.8 WireGuard improvements on staging
3. Document runbook in same repo as Terraform for VPC peering — drift killed us once

WireGuard is not "set and forget VPN." Key rotation is a routing event, not a config tweak — plan overlap windows or accept session resets.