Zephyr vs FreeRTOS Scheduling: Notes From a Migration

We migrated a mid-complexity sensor hub from FreeRTOS 10.4 to Zephyr 3.4. The scheduler semantics differ more than the docs admit — especially around priority inheritance, tickless idle, and who owns timer callbacks.

Context

Product: wearable sensor hub, nRF52840 + external SPI flash, 6 threads at peak, BLE stack, 200 Hz IMU fusion, OTA slot behind MCUboot patterns we kept unchanged.

Why migrate: upstream Nordic support shifting to Zephyr; hiring pool; native DeviceTree for board variants.

Timeline: 11 weeks firmware + 4 weeks soak before fleet OTA.

What we expected vs. what happened

Area	FreeRTOS 10.4	Zephyr 3.4	Surprise
Preemption	Priority-based, config MAX_PRIORITIES=5	Cooperative + preemptive threads, many priorities	Zephyr system workqueue runs at priority -1 — starved our "high" sensor thread once
Mutex PI	Optional, we enabled	k_mutex PI on by default	Different lock ordering exposed latent ABBA in logging path
Timers	Timer service task	Timer workqueue + expiry in ISR context rules	Callback doing k_msgq_put blocked in ISR path — assert in debug
Tickless	Nordic port custom	SysTick + HFCLK dependency	40 µs jitter on fusion deadline vs. 15 µs FreeRTOS
Stack watermark	uxTaskGetStackHighWaterMark	CONFIG_THREAD_STACK_INFO	Comparable; Zephyr threads default smaller stacks — two overflows week 2

None fatal. All subtle.

Scheduler semantics that bit us

1. System workqueue priority

Zephyr's system workqueue handles deferred ISR work and some stack internals. Default priority is negative (above app threads in Zephyr's numbering where lower number = higher priority).

Our IMU thread at priority 5 lost CPU to deferred workqueue jobs from the BLE stack when connection interval tightened. Symptom: fusion timestamp gaps exactly one connection event apart.

Fix: raised IMU thread priority; moved heavy BLE deferred work to dedicated ble_wq with capped queue depth; documented in devicetree chosen nothing — this is Kconfig:

CONFIG_SYSTEM_WORKQUEUE_PRIORITY=7

Read Zephyr's priority docs in Zephyr order, not FreeRTOS inverted intuition.

2. k_timer vs. k_work_delayable

FreeRTOS xTimerCreate callbacks run in timer service task — can block on mutex with PI.

Zephyr k_timer expiry runs in interrupt context; user handler must be ISR-safe. We ported a 10 ms housekeeping timer that flushed logs with mutex — kernel panic in CONFIG_ASSERT builds.

Fix: k_work_schedule from timer ISR; flush in thread context. Pattern should have been default from day one.

3. Priority ceiling on sensor bus mutex

Shared I2C bus: IMU + barometer + fuel gauge. FreeRTOS mutex with PI masked priority inversion when BLE thread grabbed bus for NVM metadata read.

Zephyr k_mutex + same thread priorities — one 80 ms stall when barometer long read collided with IMU DRDY thread.

Fix: split buses on rev C hardware; software-only fix was dedicated sensor acquisition thread owning bus — serializes reads, adds latency, shippable.

Cross-link: Rust no_std DMA notes discuss ISR boundaries — same discipline applies to timer callbacks.

4. Tickless and SoftDevice replacement

migrated off SoftDevice to Zephyr BLE host — different radio scheduling. FreeRTOS tickless idle suppressed ticks during long BLE radio events via Nordic hook; Zephyr needed CONFIG_BT_CTLR_SDC_TICKLESS tuning.

Missed fusion deadlines dropped from 2% to 0.1% after aligning LL tick with sensor timer using gptmr hardware trigger instead of pure OS timer.

Migration methodology

1. Thread mapping spreadsheet — FreeRTOS task name → Zephyr thread name, priority, stack, blocking calls.
2. Run both on duplicate CI targets for 2 weeks — same tests, compare latency histograms.
3. Soak test 14 days — battery powered, log sched stats (CONFIG_SCHED_THREAD_USAGE) hourly.

Don't big-bang migrate radio + sensors + OTA in one branch. We did OTA last.

When FreeRTOS was still better for us

• Deterministic 200 Hz control loop on a secondary MCU (STM32G0) — stayed FreeRTOS; no Zephyr port time justified.
• Contract manufacturer had FreeRTOS factory test — duplicated for Zephyr cost more than NRE saved.

When Zephyr won

• DeviceTree board variants (6 SKUs → one board overlay set)
• west update reproducible builds vs. submodule drift
• Native logging subsystem with backend switching
• POSIX subset for NuttX-curious engineers prototyping with native_sim

Numbers after 6 months fleet

• Kernel assert rate: 0 (was 3/week first month during bring-up)
• Average fusion jitter: 22 µs (was 18 µs FreeRTOS — acceptable trade)
• Engineering velocity on new sensor drivers: +30% (driver model clearer)
• Image size: +180 KB flash (bootloader headroom OK)

What I'd do next

Run Zephyr latmon benchmark on every board rev before signing scheduler config. Add automated test: IMU thread must run within 500 µs of DRDY GPIO 99.9% over 1-hour harness.

If starting migration today: Zephyr 3.6 LTS, not 3.4 — LL tick fixes we cherry-picked are mainline now.

Don't assume "RTOS is RTOS." Scheduler policy is product behavior.

FreeRTOS config we migrated from

For comparison — our FreeRTOSConfig.h highlights:

configMAX_PRIORITIES 5
configUSE_PREEMPTION 1
configUSE_MUTEXES 1
configUSE_TIMERS 1
configTIMER_TASK_PRIORITY 2
configTIMER_QUEUE_LENGTH 10

Zephyr doesn't map 1:1 — timer task priority 2 in FreeRTOS was below sensor thread 3 in our numbering (inverted sense). Spreadsheet column "FreeRTOS numeric" vs "Zephyr numeric" vs "effective preempts whom" prevented one class of mistakes; not all.

k_thread_priority_set during bring-up

Temporary boost for bring-up tests only — we left a +2 priority boost on IMU thread "for debug" that shipped in one RC build. Fusion improved; BLE starved. Static analysis grep for k_thread_priority_set in CI now.

Message queue depth sizing

Zephyr K_MSGQ_DEFINE depths copied from FreeRTOS queue lengths without accounting for larger struct payloads in new fusion message format. EAGAIN on queue put in ISR — silent drop of IMU samples. Fixed depths after CONFIG_ASSERT caught it; production had 2 weeks of rare dropouts before assert build hit fleet beta.

What I'd do next