Linux io_uring at the Edge: Latency Profiles on a Cortex-A53

Lab note — 2025-04-12, bench ID EDGE-IOUR-03

io_uring on a quad-core Cortex-A53 at 1.4 GHz is not a datacenter toy. On our Radxa Rock Pi S (RK3328, 1 GB DDR3, kernel 6.1.52-rockchip, liburing 2.4), moving a synchronous read/write microbenchmark to io_uring cut p99 latency by 31% and reduced CPU time per 4 KB op by ~22%. The gains are real on constrained silicon — with caveats about memory ordering, fixed buffers, and when the setup cost eats the win.

Setup

Hardware:

• SoC: Rockchip RK3328, 4× Cortex-A53 @ 1.408 GHz (cpufreq performance governor)
• RAM: 1 GB LPDDR3 — memory bandwidth is the hidden bottleneck
• Storage: SanDisk Ultra microSD (UHS-I, real-world ~38 MB/s sequential)
• Kernel: 6.1.52-rockchip, CONFIG_IO_URING=y

Software:

• liburing 2.4 built with -O2 -march=armv8-a+crc
• Benchmark: 1 million random 4 KB reads/writes within a 512 MB file, O_DIRECT, single-threaded submitter

Comparison baselines:

1. pread / pwrite in a loop
2. io_uring SQ poll mode (IORING_SETUP_SQPOLL) — disabled on this board after thermal throttling
3. io_uring default async submission

Findings

Syscall amortization dominates on A53

Each pread costs roughly 1.2–1.8 µs of userspace + kernel entry/exit on this kernel build (measured with clock_gettime(CLOCK_MONOTONIC_RAW) around the syscall only). Batching 32 ops per io_uring_submit dropped entries to one per batch.

Mode	p50 (µs)	p99 (µs)	CPU % (1 core)
pread loop	48	210	78
io_uring (batch 32)	41	144	61
io_uring + registered buffers	38	128	57

Registered buffers (io_uring_register_buffers) helped more on reads than writes — consistent with less memcpy on the completion path.

SQPOLL is not free on edge

With IORING_SETUP_SQPOLL, a kernel thread spins at 100% of one A53 core even at idle. On a board without a heatsink, that triggered thermal throttling within 4 minutes and increased tail latency. SQPOLL might make sense on a fan-cooled i.MX8 or RK3588 big core; on passively cooled A53, skip it.

Memory bandwidth floor

When we increased batch size to 128, gains flattened — profiling with perf stat showed stall_frontend and memory controller saturation. The A53 is not syscall-bound at that point; it is waiting on DDR3.

This matches what we see in container memory pressure scenarios — see cgroups v2 memory limits for how the same silicon behaves under Kubernetes.

O_DIRECT and alignment

Without O_DIRECT, the page cache masked syscall overhead and io_uring looked less impressive (~8% p99 win). For edge log ingestion and sensor spool files where we already bypass cache, io_uring is the right default.

Buffer alignment: 4096 bytes. Unaligned buffers failed registration with -EINVAL on our kernel.

Minimal pattern that shipped

We used this in a firmware log shipper (Rust 1.77, io-uring crate 0.6):

// Pseudocode — production uses liburing directly
io_uring_queue_init(32, &ring, 0);
io_uring_register_buffers(&ring, iovecs, 32);

for (each chunk) {
    prep_read_fixed(fd, buf_index, len, offset);
    if (pending == 32) {
        io_uring_submit(&ring);
        wait_cqe(&ring);
    }
}

Key tradeoff: fixed buffer pool size caps in-flight I/O. For our 256 KB spool segments, 32 × 4 KB buffers was enough; streaming multi-MB objects needed a different queue depth.

When I would not use io_uring on edge

• Single-digit ops per second — setup and buffer registration dominate.
• eMMC with high write amplification — fix the storage layer first; io_uring will not fix wear or fsync latency on cheap eMMC.
• Kernels < 5.10 — backport quality varies; vendor BSP kernels often lag.

Kernel tunables we touched

On RK3328 we also tested io_uring with IORING_SETUP_IOPOLL for polled completion on NVMe-over-USB3 (ASMedia ASM1153 enclosure). IOPOLL reduced p99 by another 8% but pinned one core during sustained reads — unacceptable on a gateway that also runs Modbus polling. Document the governor setting in your BSP README; schedutil vs performance changed our p99 spread by 12% independent of io_uring.

For multi-threaded submitters, io_uring_register_ring_fd and per-thread rings avoided lock contention on ring->sq. Two threads sharing one ring performed worse than two synchronous pread threads at our concurrency level (2). Your mileage varies above 4 submitters.

Failure modes during bring-up

• -ENOMEM on io_uring_queue_init with depth 4096 on 1 GB RAM — cap queue depth at 64 for edge.
• EBADF after fork without IORING_SETUP_NO_MMAP awareness — child processes must not share rings unless designed for it.
• Stale CQEs after signal interruption — we wrap submit/wait in retry loops for EINTR on embedded daemons without SA_RESTART on all signals.

What I'd do next

Re-run with io_uring opcodes for network (IORING_OP_SENDMSG) on the same board for our MQTT telemetry path — syscall batching might matter more than storage on that workload.

Also compare against mmap + ring buffer for the specific case of append-only logs where we control the writer — sometimes the simplest map wins on 1 GB RAM devices because you avoid double buffering entirely.

Database maintenance workloads on larger edge gateways (Postgres on ARM) intersect with Postgres vacuum tuning — different problem, same silicon budget mindset.