Edge Inference Benchmarking: Raspberry Pi 5 vs Hailo-8 vs ONNX on x86

Bench log — 2025-03-22, same model, three targets

YOLOv8n (Ultralytics 8.1.0, ONNX export opset 17) on a fixed 640×640 input, 500-frame test clip (warehouse person detection, 30 fps source). Three deployment targets: Raspberry Pi 5 (8 GB), Hailo-8 M.2 on Pi 5 carrier, and Intel N100 mini PC (x86, ONNX Runtime 1.17, OpenVINO EP disabled for apples-to-apples). Power measured at wall with Uni-T UT210E inline meter. Numbers are one afternoon — repeatable within ~5%, not lab certification.

Hardware

Target	CPU / accelerator	RAM	Notes
Pi 5	BCM2712 4× Cortex-A76 @ 2.4 GHz	8 GB	Active cooler, --performance governor
Pi 5 + Hailo-8	Hailo-8 (26 TOPS marketed), PCIe Gen2 x1	8 GB	HailoRT 4.17.0, hailo-yolo postprocess on CPU
N100 box	4× E-core @ 3.4 GHz burst	16 GB	Beelink S12 Pro, Ubuntu 22.04

Model: YOLOv8n FP16 ONNX on x86; INT8 HEF compiled for Hailo; Pi 5 native uses ONNX Runtime 1.17 CPU EP (no ARM GPU delegate in our stack).

Pre/post processing overhead

Numbers above are model inference only. JPEG decode + letterbox resize on Pi 5 CPU added 28 ms p50 before ORT — total pipeline 170 ms, 5.9 FPS end-to-end. Hailo path still decoded on CPU; moving decode to ISP or V4L2 M2M is on the roadmap.

N100 with OpenVINO EP (re-tested after initial ORT run): YOLOv8n INT8 via POT — 31 ms p50, 28 FPS, mAP 0.699. Worth it if you standardize on Intel edge; we stayed ORT for portable ONNX in this bench.

Environmental conditions

Bench at 22°C ambient. Pi 5 in enclosure at 35°C cabinet: Hailo throttled at 41 FPS → 38 FPS; Pi CPU path throttled harder (6.8 → 5.1 FPS). Spec thermal in deployment location, not on open bench.

BOM rough-order (USD, 2025)

Line	Cost
Pi 5 8 GB	~$80
Hailo-8 M.2	~$70
Active cooler + case	~$25
N100 mini PC	~$140
USB3 powered hub	~$30

Engineering time for Hailo compile pipeline: ~8 hours initial, ~1 hour per model iteration — factor into TCO vs N100 where ONNX drops in without HEF.

Reproducibility notes for this bench

All raw CSVs and ONNX export commands live in bench/edge-yolo-v8n-2025-03/ internal repo. Camera exposure fixed at 1/60 s, ISO 400, warehouse clip lit with 5000 K panels — detection scores vary under different lighting; do not compare mAP across labs without matching capture conditions.

Results

Platform	p50 latency (ms)	p99 (ms)	FPS sustained	Wall power (W)	Power per frame (mJ)
Pi 5 CPU ORT	142	310	6.8	8.4	1235
Pi 5 + Hailo-8	18	28	52	9.1	175
N100 CPU ORT	48	89	19	11.2	589

Hailo wins on throughput per watt for this model size. Pi 5 alone is viable for ≤7 FPS single stream — one camera, non-safety-critical.

Accuracy check (mAP@0.5 on holdout 200 frames)

Platform	mAP@0.5	Notes
PyTorch FP32 reference	0.712	baseline
Pi 5 ORT FP16	0.708	acceptable
Hailo INT8	0.691	-2.1 pts — verify on your classes
N100 ORT FP16	0.708	matches Pi ORT

INT8 on Hailo lost small-object boxes at distance — same class of issue as LLM quant cliffs in quantization war stories.

Hidden costs

Hailo compile cycle: 45 min per model revision on dev laptop (Hailo Dataflow Compiler 3.25). Budget engineer time, not just BOM.

PCIe x1 on Pi 5: sufficient for YOLOv8n; YOLOv8m saturated at ~22 FPS — bottleneck shifts to link.

Thermal on Pi 5 sustained: without active cooler, CPU path throttled at 4.2 minutes — p99 unusable. Add $12 fan.

N100 power: idle 6 W, burst 15 W — better duty-cycled than Hailo for sparse inference (1 frame / 10 s).

Software commands (repro)

Pi 5 ORT:

pip install onnxruntime==1.17.0 ultralytics==8.1.0
yolo export model=yolov8n.pt format=onnx opset=17
python bench_ort.py --model yolov8n.onnx --frames 500

Hailo:

hailo compiler yolov8n.onnx --hw-arch hailo8
hailortcli run-hef yolov8n.hef --input test_640.bin

Decision matrix (what I'd ship)

• Battery solar edge, 1 cam, 5 FPS OK: Pi 5 CPU + aggressive frame skip
• Multi-cam retail, 30 FPS aggregate: Hailo-8 or move to Jetson Orin Nano (not benchmarked here — next bench day)
• Factory floor with x86 maintenance spare: N100 — easier ops, ONNX Runtime docs, no HEF compile

Always run your classes through Hailo INT8 before BOM lock — marketing TOPS ≠ your mAP.

Drift and monitoring

Edge models still need input drift checks — camera vibration, lighting seasonality. Log thumbnail embeddings upstream; tie to ML drift monitoring patterns even at the edge.

Next bench

• RK3588 NPU path with RKNN-Toolkit2 2.0
• Jetson Orin Nano Super vs Hailo-8 on YOLOv8s
• Power including PoE switch overhead for 4-cam install

Compare total cost: Hailo module $70 + Pi 5 $80 vs N100 box $140 — engineering time dominates if you retrain weekly.