Time-Series Data on Constrained Devices: Compression Schemes That Actually Work

Facebook's Gorilla paper (VLDB 2015) describes delta-of-delta timestamp encoding and XOR'd floating-point values that compress typical monitoring data 1.5–3× with minimal CPU. On an STM32L4 logging 16-bit ADC samples at 1 Hz, our hand-rolled Gorilla encoder stored 45 bytes per 100 samples vs 200 bytes raw—flash wear and upload bandwidth both improved. This note is an embedded port, not a datacenter reimplementation.

Why not gzip or protobuf?

gzip: RAM window too large for 64 KB SRAM budget during encode
Raw CBOR arrays: simple but no compression on slowly changing values
Fixed-point scaling only: helps precision, not redundancy when temperature flatlines at 23.4 °C for hours

Gorilla wins when values change predictably and timestamps are regular.

Algorithm sketch (embedded subset)

Timestamps: assume 1 s base interval. Store first timestamp full width (32-bit Unix). For subsequent:

Delta from previous: usually 1000 ms
Delta-of-delta: usually 0 → encode in 1 bit ("same as expected")

Values: IEEE float32 XOR with previous; leading zero byte count + meaningful XOR bytes—when value unchanged, 1 bit.

We used float32 despite 16-bit ADC source—historical API consistency. Fixed-point int16 variant saves another 15% if you control consumers.

typedef struct {
    uint32_t last_ts;
    float    last_val;
    uint8_t  buf[256];
    size_t   len;
    bit_writer_t bw;
} gorilla_encoder_t;

void gorilla_encode_sample(gorilla_encoder_t *e, uint32_t ts, float val) {
    int32_t delta = (int32_t)(ts - e->last_ts);
    int32_t dod   = delta - 1000;  /* expected 1 Hz */
    write_dod(&e->bw, dod);        /* variable bit width per paper */
    write_xor_float(&e->bw, e->last_val, val);
    e->last_ts = ts;
    e->last_val = val;
}

Reference: read the paper; do not trust random GitHub ports without test vectors.

Measured compression (field data)

Dataset	Raw (B/sample)	Gorilla (B/sample)	Ratio
Indoor temp 1 Hz, 24 h	4.0 (int16+overhead)	1.8	2.2×
Vibration RMS 10 Hz	4.0	2.9	1.4×
GPS lat/lon 0.1 Hz	8.0	3.1	2.6×

High-entropy vibration compresses worse—expected.

Storage integration

We append compressed blocks to SPI flash with header {magic, start_ts, sample_count, crc32}. Block size 512 B aligned to flash page. Pair with SQLite embedded Linux tradeoffs on gateway products that aggregate before uplink.

Upload uses CoAP block transfer—see MQTT vs CoAP on CC2652 for why we batch upload on UDP-friendly chunk sizes.

CPU and latency

Encode cost on Cortex-M4 @ 80 MHz: ~18 µs per sample at 1 Hz—negligible. Decode for USB export on PC: trivial.

Watch out: decoder state must reset on block boundaries; mid-stream corruption loses rest of block—CRC per block contains blast radius.

Alternatives considered

TSZ / Chimp128: better compression on some float series, more code size
Simple RLE on int16: beats Gorilla when values truly static for hours, worse on smooth ramps

What I'd do next

Fuzz encoder/decoder with host-side Unity tests—bit writer off-by-one bugs are silent until field corruption.
Adaptive block flush on variance threshold—don't wait 512 B if sensor alarm needs immediate uplink.
Publish test vectors in repo for regression when porting to new MCU.

Decoder on gateway vs device

We decode Gorilla blocks on the Pi gateway in Python for analytics; firmware only encodes. Split saved MCU flash but required version byte in block header when encoder algorithm tweaked—old gateways rejected new blocks until OTA updated Python side. Version field non-negotiable.

Bit writer testing

Off-by-one in write_bits() caused silent 0.3% corruption—detected only when CRC mismatch rate exceeded baseline. Host test compares encoder output against reference vectors generated from desktop implementation; 200 random walks per CI run.

Fixed timestamp irregularity

Sensor clock drift 50 ppm caused delta-of-delta encoding to widen—compression ratio dropped from 2.2× to 1.6× over week without NTP. Embedded devices without network time should store raw delta timestamps periodically as keyframe every N samples.

Endianness on wire

Decoder on big-endian gateway assumed little-endian block header—silent garbage until explicit magic bytes and version in header. Always specify endianness in on-flash format spec; obvious on paper, missed once in shipping.

Flash wear estimate

Encoder writes 512 B blocks every 100 samples at 1 Hz—~44 KB/day. At 100k erase cycles on 64 KB sector, sector life ~4 years single sector append (bad). Rotate sectors circularly; wear spread mandatory in driver design.

Multi-sensor batch encoding

Batching four sensor streams into one block improved ratio 15%—shared timestamp keyframe amortized header overhead. Tradeoff: single block corruption loses all four streams until next keyframe interval.

Gorilla is not magic—it's exploiting smoothness in physical signals. When your sensor jitter is high or timestamps irregular, measure before committing flash format forever.