Clients often arrive with a model file and a board in a drawer — and a question: "will this work?" Our job is to make that combination reliable across power cycling, outdoor temperatures, sustained inference loads, camera stream interruptions, and a fleet that needs over-the-air updates without an engineer on-site. This article covers the methodology and results of head-to-head inference benchmarks on Jetson Nano, Raspberry Pi 4, and Raspberry Pi 5 — and the decisions that matter more than benchmark numbers in real deployments.

How we benchmarked — and why it matters

We ran the same quantised object detection model (YOLO variant, TensorRT on Jetson, ONNX Runtime on Pi) across all three boards under identical conditions: same camera stream, same resolution, same ambient temperature, sustained for two hours. We measured inference latency at p50/p95/p99, CPU and memory utilisation under load, thermal behaviour and throttle onset, and power draw at the board rail.

The two-hour sustained run is the part most lab comparisons skip. A board that hits 30 FPS in a 60-second burst may throttle to 18 FPS after 20 minutes in a closed enclosure. If your use case runs continuously — and most field deployments do — the burst number is the marketing number, not the production number. We use the sustained-run metric for every hardware decision.

We also profiled boot time to first inference and recovery time after a simulated power loss. Field devices lose power unexpectedly. A system that takes four minutes to resume inference after a power event is a support burden. That recovery path is part of the product.

Board-by-board: what we found

Raspberry Pi 5 is the strongest choice for cost-sensitive CPU inference workloads with GPIO requirements. The performance-per-dollar gap over Pi 4 is significant for inference tasks: ONNX Runtime throughput improved by roughly 35-40% on our test model. If CUDA is not required and the budget matters, Pi 5 is where most new projects start.

Jetson Nano still wins when CUDA acceleration and a stable vision stack are required. The TensorRT pipeline on Nano outperforms ONNX Runtime on Pi 5 for GPU-bound vision workloads — and the CUDA ecosystem means fewer surprises when integrating standard perception libraries. For anything running quantised models with serious throughput requirements, Nano is the conservative choice with the most mature tooling.

Banana Pi enters the picture when I/O layout, supplier diversity, or specific peripheral requirements matter more than ecosystem familiarity. We use it on projects where the client has procurement constraints or where specific interface combinations (dual Ethernet, SATA, PCIe) are needed on a Pi form factor. The software support ecosystem is thinner, so expect more driver bringup time.

None of these boards are universally "best." They are trade-offs that should be made explicitly against the specific constraint set: latency budget, power budget, enclosure thermal design, supplier requirements, and ecosystem familiarity of the team maintaining the device. We write that constraint set down before selecting hardware — not after a board arrives and turns out to be the wrong choice.

 

Profile power under camera load before finalising mechanical design

  • A board that runs at 55C in open air may throttle at 72C in a closed enclosure in summer.
  • Thermal throttle onset under sustained load is the number that matters, not peak specs.
  • Run two-hour sustained profiles before signing off on enclosure design — not after.

Cameras, cabling, and enclosure design

CSI versus USB camera is not a philosophical debate — it is a latency and cabling debate. On small boards, the camera path often contributes more to end-to-end inference latency than the model itself. CSI cameras on Pi eliminate USB overhead and scheduling jitter; USB cameras offer more flexibility and longer cable runs. Profile both against your latency budget with the specific sensor you intend to use, not a proxy sensor from a different product family.

Lens selection affects the perception system, not just the optics. Focal length and distortion characteristics need to be accounted for in the preprocessing pipeline. A lens swap mid-project that was not profiled as a system change will degrade model performance in ways that look like inference regression until the root cause is found.

3D enclosures are part of the perception system, not packaging added at the end. Vent placement affects thermal profile. Lens port position affects camera FOV and distortion at the edges. Mount flex from cable tension affects camera angle under vibration. We design STEP and STL files alongside firmware bringup so pilot units look and behave like production units — not lab stacks held together with zip ties.

Life after prototype: OTA, provisioning, and fleet visibility

Shipping ten devices is a project; shipping ten thousand is a products and operations problem. The gap between the two is the tooling: OTA update delivery, device provisioning pipelines, certificate management, and a dashboard that tells you which devices are alive, on which firmware version, and why this specific one has not checked in.

We build OTA paths with A/B partitions and rollback hooks from the first device onwards, not as a retrofit when the fleet is large enough to make manual updates painful. Retrofitting OTA into an established fleet requires touching every device — exactly the problem OTA is supposed to solve. Starting with it means the first firmware update to a hundred devices is as easy as the update to ten.

Provisioning should be boring and repeatable. Factory burn-in scripts, device certificates, and tenant assignment run from a CLI with no manual SSH steps. If provisioning a new device requires tribal knowledge or a specific engineer, it is a support liability at scale.

Device dashboards reuse the same observability patterns as the server stack: health heartbeat, inference throughput, queue depth, thermal warnings, last successful update timestamp. If the dashboard cannot answer "is device 4,872 healthy right now and what version is it running," the fleet is not operable — it is just managed by intuition.