CI for Firmware: Hardware-in-the-Loop Testing on a $200 Budget

Firmware CI that only runs gcc -Wall -Werror and host-side mocks is incomplete. It catches syntax, not timing, not flash wear, not "this GPIO toggles the wrong pin on real silicon." We built a hybrid pipeline—QEMU for breadth, one real board per target family for depth—for roughly $200 in hardware plus a GitHub Actions runner we already had.

Architecture

push → lint/build (all targets)
     → unit tests (Unity on host)
     → QEMU smoke (Zephyr native_sim / arm virt)
     → HIL farm (self-hosted runner + USB hub)
     → artifact: signed .bin + test report

Hardware bill (~$198):

• 2× STM32F411 Black Pill ($8 each) — regression for Cortex-M4 app
• 1× nRF52840 DK ($35) — BLE stack smoke
• 1× Raspberry Pi 4 2GB ($45, used) — self-hosted runner + OpenOCD server
• 1× powered USB 2.0 hub ($18) — flaky without powered hub
• 3× ST-Link V2 clones ($9 each) — treat as disposable
• Jumper wire, SD card, enclosure: remainder

No $10k rack. No proprietary fixture plates. Boards sit in anti-static bags with pogo-pin SWD where we need repeatability on one product.

QEMU layer

Zephyr's native_sim and qemu_cortex_m3 run in GitHub-hosted runners for every PR. Tests cover:

• State machine transitions
• CBOR encode/decode round trips
• Bootloader metadata parsing (no flash write)

Limitation: no ADC nonlinearity, no I2C clock stretching bugs, no brown-out behavior. QEMU is a filter, not proof.

Reproducible workspaces matter—our west.yml pin strategy is documented in Zephyr west workspace reproducible 2024.

HIL layer on the Pi runner

Self-hosted runner (actions-runner 2.311.0) on Pi 4, Ubuntu 22.04 arm64.

Each board connects via ST-Link; OpenOCD 0.12.0 serves GDB. Test harness (pytest + pyOCD or OpenOCD telnet)流程:

1. Flash candidate .hex from CI artifact
2. Reset and wait for UART banner (TEST_READY\n)
3. Send test vectors over UART or SWO
4. Assert pass/fail GPIO pattern on known pins
5. Capture serial log as CI artifact

# excerpt — .github/workflows/firmware-hil.yml
hil-smoke:
  runs-on: [self-hosted, linux, arm64, hil-farm]
  needs: build
  steps:
    - uses: actions/download-artifact@v4
    - run: |
        openocd -f interface/stlink.cfg -f target/stm32f4x.cfg \
          -c "program build/zephyr/zephyr.hex verify reset exit"
        python3 tools/hil/uart_test_runner.py --port /dev/ttyACM0 --suite smoke

Flaky test root causes we hit: cheap ST-Links dropping SWD under USB hub contention (fixed with powered hub and USBAutosuspend=-1), boards bricked by tests that forgot to disable watchdog (added pre-flash bootloader recovery script).

OTA and bootloader coverage

One HIL slot runs OTA update bootloader design scenarios: swap A/B slots, power-cut mid-write (relay on GPIO), signature reject path. That test cannot run in QEMU with fidelity—we use a Pololu relay board ($13) on nRST and flash power.

Cost vs coverage matrix

Layer	Cost	Catches
Host Unity	$0	Logic, parsing, crypto unit paths
QEMU	$0 (hosted)	RTOS integration, coarse HAL
HIL 1× board	~$50/board	Silicon-specific, timing, GPIO
HIL + fault injection	+$15 relay	Power loss, brown-out

We run full HIL on main and nightly; PRs get QEMU + one rotating board to keep queue time under 12 minutes.

What I'd do next

1. Add MCUboot test vectors aligned with production keys in a segregated HSM slot—never production keys on the farm.
2. Track flash cycle count per HIL board; retire after 10k program cycles to avoid worn-flash false failures.
3. Open-source the harness once we scrub customer-specific pin maps—placeholder repo linked in metadata is the intent.

Self-hosted runner hardening

The Pi runner sits on an isolated VLAN—no inbound SSH from internet, outbound only to GitHub and internal artifact registry. USB autosuspend disabled via udev; ST-Links on 30 cm cables max. We rotate runner registration token quarterly and treat the Pi SD card as disposable (USB SSD boot since month 4—SD corruption killed a weekend).

Parallel HIL jobs: one job per USB root port; sharing one port for two ST-Links reproduced "target not halted" at 15% rate.

Artifact retention and reproducibility

CI retains last 50 .hex artifacts; HIL reruns fetch by commit SHA. Reproducing field bug from six-month-old firmware required digging S3 archive—now tag releases trigger indefinite artifact pin for shipped versions.

Developer local HIL

Engineers can run hil-smoke locally against USB board—same script as CI, --skip-flash for rapid iteration. Reduced "works on CI only" when ST-Link serial differed; script accepts HIL_PORT env override.

QEMU version pin

QEMU 8.1 vs 8.0 changed arm virt RTC behavior—one test flaky until pinned in CI container digest. HIL catches silicon; QEMU catches integration—both need reproducible pins.

Hardware-in-the-loop is not optional for firmware you ship at volume. It does not require enterprise budget—just one real chip in the loop and discipline about what QEMU is allowed to sign off on.