A board support package that boots in the lab is a prototype. A BSP that keeps a device updatable, debuggable, and serviceable after three years in a warehouse, a field enclosure, or a clinical cart is a product. The difference between those two states is not luck or board quality — it is the engineering discipline applied to the BSP itself. This article covers how we structure embedded Linux BSPs for devices that need to survive contact with the real world long after the project team has moved on.
What a BSP really includes
A BSP is commonly described as the kernel, drivers, and bootloader for a specific board. In practice, a production BSP includes considerably more: pinmux configuration with explicit documentation of every pin assignment and its rationale, power domain sequencing that handles the actual power-up and power-down order of your peripheral combination, factory flashing procedures that a contract manufacturer can run without tribal knowledge, and a reproducible build system that generates the same image from source on any build host.
We use Yocto for most production BSPs. The learning curve is steep and the tooling is opinionated, but the reproducibility guarantees matter for long-lived products. A Yocto build from a locked manifest produces a bit-identical image regardless of when or where it runs — which means the firmware running on unit 4,000 is provably the same as the firmware that passed certification on unit 1, not a drift-accumulated approximation of it.
BSP milestones are aligned with the mechanical and thermal design from the start. The enclosure thermal model, the antenna clearance requirements, and the USB hub power budget all affect BSP configuration choices — power limits, thermal throttle policy, USB current allocation. Discovering those constraints in week ten of a twelve-week project produces a redesign. Discovering them in week two produces a configuration change.
Documentation is a first-class deliverable, not a post-project task. Every BSP we ship includes: a hardware revision compatibility matrix, a pin assignment table with function and signal direction, a driver version table with upstream patch status, and a factory calibration procedure. The test for good documentation is whether an engineer who was not on the project can bring up a new board from source without a call to the original team.
Kernel configuration and driver strategy
We start from the minimal kernel configuration for the target SoC and add only what the hardware requires. A Raspbian-style "everything enabled" kernel works for development; a production device with 200 unnecessary kernel modules loaded has a larger attack surface, slower boot times, and more scheduling interference than a device with a purposefully minimal configuration.
Out-of-tree drivers are a long-term maintenance liability. When a driver for a specific sensor or peripheral cannot be upstreamed, we version-lock the kernel to the last revision the driver supports and document the constraint explicitly. Silent kernel upgrades that break peripheral drivers are a common source of field failures in devices that receive OS-level security updates without full regression testing.
Camera driver validation deserves particular attention. Camera sensor firmware and kernel driver versions interact in ways that are not always documented by the sensor vendor. We validate the exact combination of sensor firmware, kernel driver version, and ISP configuration on the physical camera modules that will ship — not a representative unit from a different lot — before locking the BSP for production.
Real-time kernel patches are applied selectively rather than universally. PREEMPT_RT improves worst-case latency but adds scheduling overhead that can reduce throughput for non-latency-critical workloads. We profile the actual control loop requirements against the stock and RT kernels and apply RT only when the worst-case latency data justifies the throughput cost.
OTA that actually recovers from failure
An OTA update that fails silently and leaves the device in an unbootable state is worse than no OTA at all. The minimum viable OTA architecture for production devices has three properties: atomic updates (the update either fully applies or does not apply at all), verified images (cryptographic signature check before any partition is modified), and automatic rollback on failed boot (the device returns to the previous known-good state if the new image fails to reach a defined health check).
A/B partition schemes implement atomicity without a complex transaction system. The inactive partition receives the update while the active partition continues running; the bootloader switches to the new partition only after verification succeeds; the boot counter decrements on each boot attempt and triggers rollback if it reaches zero without a successful health check. This scheme has been deployed in millions of Android and ChromeOS devices and is well understood.
OTA is also a telemetry system. A device that silently accepts or rejects updates without reporting status creates a fleet management blind spot. Every update attempt should report: device ID, firmware version before and after, update duration, verification result, and boot result. This telemetry feeds the fleet dashboard and allows the operations team to identify systematic update failures before they affect a significant portion of the fleet.
Delta updates reduce bandwidth requirements for fleets on cellular or metered connections. A full image update for a Pi-class device may be 500MB; a delta from the previous version may be 20MB. For fleets of thousands of devices on metered connections, the difference is material in both cost and update success rate. We implement delta generation and verification as part of the OTA server pipeline.
Debug tooling and field serviceability
Remote debug access for field devices requires planning before deployment, not improvisation during an incident. We provision each device with a read-only debug interface — typically a restricted SSH key with limited command permissions — that allows log retrieval, health check execution, and diagnostic data collection without full shell access. Full shell access requires a separate authentication step with audit logging.
Kernel panic capture and post-mortem analysis are built into the BSP from the first build. Kdump or pstore-based crash recording means that a device that panics in the field can report the crash dump on next boot rather than losing the failure information. Without this, field crashes are often impossible to root-cause from telemetry alone.
Hardware-level debug through JTAG or SWD is preserved in production units rather than depopulated. Repopulating debug headers after a production issue requires board rework and delays diagnosis. Leaving headers populated with a documented pinout allows low-level debug when a device arrives from the field with a failure that cannot be diagnosed through software tools.
Field serviceability documentation covers replacement procedures for every component likely to need replacement over the device lifetime: SD card swap procedures that preserve configuration, camera module replacement with recalibration steps, and battery replacement for devices with backup power. Service documentation written after the fact requires the original engineer; service documentation written during design is used by field technicians without that dependency.
