OTA Update Bootloader Design: Lessons From Three Shipped Products

A/B partition, signature check, rollback counter — every decision I got wrong the first time, and what I'd do differently today across three shipped products (wearable hub, industrial gateway, consumer sensor puck).

Answer first

Treat OTA as a state machine with evidence, not a flash write script. Minimum viable secure OTA: dual slots, cryptographic verify before swap, anti-rollback monotonic counter in protected storage, and a known-good fallback that doesn't depend on the primary app booting.

Product 1: wearable (nRF52840, 1 MB internal flash)

What we shipped v1

• Single backup slot + MCUboot 1.10
• Ed25519 signature, key baked in OTP
• BLE GATT transfer, 128-byte chunks
• Swap on reboot only

What went wrong

Failed upgrade brick rate: 0.8% first 10k devices.

Root causes:

1. Power loss during swap — battery at 5% allowed OTA start; swap truncated mid-page.
2. No download completeness check — CRC on wire, not on full image hash before mark-pending.
3. Rollback counter in app NVS — corrupt NVS after bad flash → counter reset → downgraded to known CVE firmware once.

v2 fixes

• Minimum 30% battery + on charger for swap commit
• SHA-256 over staged image in external SPI before boot_set_pending
• Monotonic counter in UICR/user UICR region, incremented only after verify — MCUboot security counter hook
• MCUboot slot alignment fixed — image trailer at 32-byte boundary per tool requirement

Brick rate → 0.02% over 18 months (remaining failures: flash wear on abused test units).

Product 2: gateway (i.MX RT1064, NuttX, external QSPI)

Dual 8 MB slots, Ethernet OTA, RSA-3072 (customer compliance mandate — Ed25519 would be fine technically).

Wrong decision: streaming write directly to inactive slot

Network stall mid-image left partial slot that passed header magic check but failed signature — bootloader looped reset until serial recovery.

Right decision: download to staging partition (third region)

Flow: staging → verify → copy to inactive slot → mark pending → reboot. Extra copy cost 40 s on 4 MB image; acceptable at monthly update cadence.

Tied into NuttX mount/unmount rules — no mounted FS on inactive slot during copy.

Product 3: consumer puck (ESP32-S3, 8 MB flash, Wi-Fi)

Used ESP-IDF OTA partitions — learned different lessons:

• Anti-rollback eFuse enabled late — field units could install old signed image with bug
• Rollback via esp_ota_mark_app_invalid_rollback_and_reboot — works if you test it; we didn't in CI first month → one bad release needed manual UART flash for 200 units

Cross-product lesson: test rollback path in manufacturing fixture, not only happy path OTA.

Design patterns that survived review

Slot layout

[ Bootloader 48 KB fixed ]
[ Slot A primary app ]
[ Slot B upgrade app   ]
[ Staging / scratch    ]  ← optional but worth flash cost
[ NVS / config         ]  ← never erase in OTA path

Keep NVS partition stable across updates — UUID migration in app, not partition table churn per release.

State machine (explicit)

States: IDLE → DOWNLOADING → STAGED → VERIFIED → PENDING_SWAP → TEST_BOOT → CONFIRMED | REVERT

Persist state enum + image hash in NVS before swap. On boot, TEST_BOOT runs hardware self-test (IMU WHO_AM_I, flash CRC spot check). CONFIRM only after 24 h uptime or user ack — your product policy chooses; pick one and document.

Signature verify location

Verify in bootloader only — app verifies again redundantly OK; never verify only in app.

Keys: prod key in HSM at signing CI; dev key separate OID in image header — MCUboot imgtool supports multiple keys; don't reuse dev key in prod builds (obvious; caught in audit twice).

Rollback counter semantics

Monotonic security version in image header, not app semver. Customer sees 2.4.1; bootloader sees security_counter=17.

Mismatch with USB-PD powered devices: OTA during PD contract renegotiation caused brownout on one board — gate OTA on stable 5 V/9 V rail (see power sequencing).

Security incidents we avoided

• Downgrade to pre-encryption firmware — blocked by counter
• Bitflip in transit — caught by signature (SHA alone in app would miss targeted header patch)
• Malicious mirror server — TLS + signature; TLS alone insufficient

Metrics we track

Metric	Target
OTA success rate (confirmed)	>99.5%
Bricked requiring UART	<0.05%
Mean download time	product SLA
Rollback events / month	trending down postmortem

What I'd do differently on product 1 from day zero

1. Staging partition — even on 1 MB flash (shrink app by 128 KB once, forever safer)
2. Factory test: intentional power-cut during swap on golden unit
3. Postmortem template linked — postmortem writing — every OTA failure gets doc within 72 h

What I'd do next

Unified OTA telemetry schema across BLE/Wi-Fi/Ethernet transports — same state enum uplink to fleet dashboard. Correlates failures with battery %, RSSI, flash ID.

Open-source our staging→verify→swap library — internal only today; too many teams re-learn power-loss brick math.

OTA isn't firmware feature #47. It's the trust boundary between your signing key and every device in the field. Design it like one.

Telemetry we wish we'd had from day one

Per-device OTA event stream:

• download_bytes, stage, verify_result, swap_result, boot_test_result
• battery_mv, on_charger, rssi (BLE path)
• security_counter_before/after

Correlates 0.8% brick rate with battery <10% — obvious in hindsight, invisible without fields.

Legal and customer comms

Consumer puck OTA failure → support script must distinguish bricked (UART recovery) vs rolled back (silent, user unaware). We conflated terms in one app release note — support volume spiked.

A/B vs single-slot economics

Single backup slot saves flash but removes rollback target during failed test boot if swap already happened. A/B costs flash; pays for itself first time you avoid UART truck roll.

What I'd do next