Heap Fragmentation in Long-Running Embedded Systems: Diagnosis and Mitigation
A product that runs for months will fragment its heap in ways that 24-hour QA testing never reveals. Static allocation, pool allocators, and when to give up on dynamic memory.
A firmware image that passes 24-hour QA can still fail at month three with malloc(128) returning NULL despite "plenty" of free bytes—classic heap fragmentation on long-running embedded systems. We diagnosed this on a connected appliance (FreeRTOS 10.4, newlib nano malloc, 64 KB heap) that leaked no bytes according to xPortGetFreeHeapSize() yet could not allocate a 256-byte JSON buffer after 11 weeks in soak.
How fragmentation manifests
mallocfails; largest free block (heap_walkcustom tool) is 96 bytes while total free is 18 KB- Slow performance creep as allocator searches bins
- Nightly reboot "fixes" it—red flag for production
24-hour test loops miss this because allocation pattern never reaches steady-state churn.
Diagnosis tools
1. Heap walker wrapping malloc/free (linker wrap or -Wl,--wrap=malloc):
void heap_stats(size_t *total_free, size_t *largest_block, size_t *frag_pct) {
/* walk free list headers — newlib dlmalloc structure specific */
}
Log frag_pct = 1 - (largest_block / total_free) hourly to flash.
2. FreeRTOS heap tracing
configUSE_TRACE_FACILITY + vHeapWalk if using heap_4/heap_5—shows allocated block sizes distribution.
3. Host replay
Record allocation sequence from soak log; replay on Linux with same allocator build—reproduce offline.
Our culprit: alternating malloc(512) for MQTT payload and malloc(64) for topic strings, freed out of order, over 400k cycles.
Mitigation strategies (ordered)
Static allocation (best when feasible)
All buffers fixed at compile time. Zero fragmentation. Cost: RAM always reserved for peak.
We moved MQTT RX buffer to static 1024 B ring—problem gone for that subsystem.
Memory pools
typedef struct {
uint8_t blocks[POOL_COUNT][BLOCK_SIZE];
bool used[POOL_COUNT];
} mem_pool_t;
void *pool_alloc(mem_pool_t *p);
void pool_free(mem_pool_t *p, void *blk);
Fixed block sizes; O(1) alloc/free; internal fragmentation only. Used for network packet buffers.
Block allocator / slab for uniform objects
All sensor_reading_t same size—slab allocator with 128 slots.
Replace newlib malloc with tlsf or memheap
TLSF (Two Level Segregated Fit) bounds fragmentation for real-time claims—verify license and code size (~4 KB). Zephyr sys_heap alternative on migration path.
Related: linker script anatomy mistakes if .heap size undersized from start.
Stop dynamic allocation in loops
Obvious but violated constantly—cJSON_Print in 1 Hz loop without reuse.
When to give up on dynamic heap
- Safety-certified codebase requiring deterministic behavior
- Uptime measured in years without maintenance window
- MCU with <32 KB RAM—fragmentation overhead unbearable
Keep heap for init-only allocations (parse config once at boot), never steady-state.
Flash logging of heap stats interacts with flash write endurance—do not log every second.
Results after refactor
| Metric | Before | After (pools + static MQTT) |
|---|---|---|
| Soak duration to alloc fail | ~11 weeks | >52 weeks (stopped test) |
| Largest free block at week 4 | 96 B | 14 KB |
| Peak heap used | 41 KB | 38 KB |
What I'd do next
- CI soak test 72 h minimum with allocation logging assert:
largest_free > 4 KBevery hour. - Ban implicit malloc in lwIP path—custom
mem_mallocpool sized from traffic model. - Document allocation budget per module in architecture review—KB signed off like CPU %.
newlib nano vs newlib full
--specs=nano.specs shrinks malloc metadata but changes fragmentation behavior—we reproduced field failure only on nano build. CI now runs soak on both spec variants weekly; one hour machine time, caught regression when switching toolchains.
FreeRTOS heap_4 vs libc malloc
Mixing pvPortMalloc for RTOS objects and libc malloc for app code split heaps—xPortGetFreeHeapSize() lied about app allocations. Standardized on heap_4 for everything with wrapper functions; libc malloc banned in application code via -Wl,--wrap=malloc trap in CI.
Fragmentation metric automation
Nightly soak job asserts largest_free >= 0.25 * total_heap—failure opens ticket automatically. Caught regression when third-party JSON library added realloc churn in minor version bump; pinned library until vendor fixed pool-friendly allocator option.
Static analysis complement
Cppcheck malloc leak check missed custom pool aliasing—only runtime heap walk caught double-free in error path. Use static analysis plus soak; neither alone sufficient on embedded.
Production telemetry
Fleet reports heap_largest_free percentile weekly—p5 below 2 KB triggers firmware review before field failure. Privacy-safe aggregate only; no PII in heap stats. Caught one SKU with third-party PNG decoder leak after 45 days deployed.
Long-lived object pools sizing
Sized MQTT pool at peak concurrent messages from load test plus 20%—undersized pool forced fallback to malloc and recreated fragmentation in staging. Size pools from measured peak, not guess.
valgrind on host sim
Algorithm modules linked against host libc run under valgrind nightly—found 48-byte leak in JSON parse path invisible to Unity asserts. Not exhaustive for target, cheap net for portable code. Pair with longest-block metric on device; together they catch different failure classes.
xPortGetFreeHeapSize() lying comfortably is worse than no metric. Measure largest block or stop pretending the heap is healthy.
Manish Bookreader
Electronics enthusiast, Embedded Systems Expert, Linux/Networking programmer, and Software Engineer passionate about AI, electronics, books, and cooking.
