Introduction
Welcome to Learning copper-rs, a hands-on guide to building robots with the Copper deterministic robotics operating system.
Most people learn Rust with the amazing Rust Book, so we built it for copper-rs.
This book has been written with the help of AI, for the sake of speed, proper grammar and formatting, but each and every line has been reviewed by a human.
The source for this book is available on GitHub. Feedback and contributions are welcome.
This book will take you from zero to a running Copper project, one concept at a time. By the end, you’ll understand how to define task graphs, write sensor drivers and processing pipelines, and run everything with sub-microsecond deterministic scheduling.
Reading version markers
A marker means “available since this version” for a feature or the API shown. Under a heading, it applies to that section; beside a paragraph or list item, it applies only there. Features that predate 1.0 have no marker. Paired patch markers give the minimum patch on each release line when a feature was backported.
This edition’s examples use 1.3.0-dev. Sections marked v1.3 currently require that development checkout; see Setting Up. Older markers identify when a capability became available, but the examples use this edition’s APIs and tools. Keep your application and its logreaders on the same version. Experimental features are identified separately in their chapters.
Who is this for?
This book is aimed at:
- Robotics developers coming from ROS / ROS 2 who want to explore a Rust-native alternative
- Rust developers interested in robotics and real-time systems
- Beginners who want a gentle, progressive introduction to Copper’s architecture
No prior Copper experience is required. Familiarity with Rust syntax is helpful but any experience with other programming language should allow you to get through.
Let’s get started!
What is copper-rs?
Copper is a deterministic robotics runtime written in Rust. Think of it as a “game engine for robots”: describe your system declaratively and Copper will create a custom scheduler and run it deterministically from cloud simulation down to embedded controllers.
The big idea
You describe your robot as a task graph – a directed graph of components that produce, process, and consume data. Copper reads this graph at compile time and generates a custom deterministic scheduler. At runtime, your tasks execute in a precise, pre-computed order with zero overhead.
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Sensor │────▶│ Algorithm│────▶│ Actuator │
│ (Source)│ │ (Task) │ │ (Sink) │
└──────────┘ └──────────┘ └──────────┘
Key features
- Rust-first – Ergonomic and safe.
- Sub-microsecond latency – Zero-alloc, data-oriented runtime.
- Deterministic replay – Every run, bit-for-bit identical.
- Interoperable with ROS 2 – Bridges via Zenoh, opening the path for a progressive migration.
- Runs anywhere – From Linux servers, workstations, and SBCs to bare-metal MPUs.
- Built to ship – One stack from simulation to production.
How it works (in 30 seconds)
- You define your tasks (Rust structs that implement a trait).
- You wire them together in a configuration file (
copperconfig.ron). - A compile-time macro reads the config and generates the scheduler.
- At runtime, Copper calls your tasks’
process()methods in the optimal order, passing pre-allocated messages between them.
That’s it. No topic discovery, no callback registration, no middleware configuration. Define, wire, run.
copper-rs vs ROS
If you’re coming from ROS or ROS 2, this chapter maps the concepts you already know to their Copper equivalents. The mental model is similar. Both are component-based frameworks with message passing, but the execution model is fundamentally different.
Concept mapping
| ROS 2 | copper-rs | Notes |
|---|---|---|
| Node | Task | A unit of computation |
| Publisher | CuSrcTask | Produces data (sensor driver) |
| Subscriber + Publisher | CuTask | Processes data (algorithm) |
| Subscriber | CuSinkTask | Consumes data (actuator) |
.msg file | Rust struct with derives | Message definition |
| Topic | Connection in copperconfig.ron | Data channel between tasks |
| Launch file + YAML params | copperconfig.ron | Graph topology + task parameters |
colcon build | cargo build | Build system |
package.xml / CMakeLists.txt | Cargo.toml | Dependency management |
| Executor | #[copper_runtime] macro | Generated deterministic scheduler |
rosbag | Unified Logger + cu29-export | Record and replay |
| Parameter server | ComponentConfig in RON | Per-task key-value configuration |
Key differences
Scheduling
In ROS 2, nodes are separate processes (or threads in a composed executor). Callbacks fire asynchronously when messages arrive via DDS. The execution order depends on network timing and OS scheduling – it’s non-deterministic.
In Copper, tasks run in the same process and are called synchronously in a compile-time-determined order. Every cycle, every task runs in the exact same sequence. There are no callbacks, no races, no surprises.
Message passing
In ROS 2, messages are serialized, sent over DDS (a network middleware), deserialized, and delivered to callbacks. This adds latency and allocations.
In Copper, messages are pre-allocated buffers in shared memory. A task writes its output directly into a buffer that the next task reads from. No serialization, no copies, no allocations on the hot path.
Replay
In ROS 2, rosbag records and replays topic messages. Replay is approximate – timing
jitter, OS scheduling, and node startup order can cause differences.
In Copper, replay is deterministic. The unified logger records every message and periodic state snapshots (“keyframes”). Given the same log, replay produces identical results every time, down to the bit.
Configuration
In ROS 2, you typically write a launch file (Python or XML), separate YAML parameter files, and topic remappings. These are resolved at runtime.
In Copper, everything is in one RON file (copperconfig.ron) that is read at
compile time. If your config references a task type that doesn’t exist, you get a
compile error, not a runtime crash.
Bridges
The good news is that you do not have to chose between Copper and ROS: Copper’s Zenoh bridge lets you run both side by side, so you can migrate incrementally.
Setting Up Your Environment
Use the development checkout named in the edition notice. Before building a project, we need Rust 1.95 or newer and the Copper project bootstrap tool.
Install Rust
Follow the official installation guide at https://rust-lang.org/tools/install/, or run:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
After installation, make sure cargo is available and recent enough:
cargo --version
rustc --version
Install cargo-cunew
v1.0
cargo-cunew scaffolds a new Copper project from the official Copper templates:
cargo install cargo-cunew
Generate our first simple Copper project
Generate a new project directly:
cargo cunew --source git --git-branch master my_project
This selects the development code used by these chapters and generates a complete project. All Copper dependencies use the same source. For a released application, select a release supported by the feature you want to use.
For more details, see the official documentation: https://copper-project.github.io/copper-rs/Project-Templates/
What you get
The generated project contains everything you need:
my_project/
├── build.rs # Build script (required by Copper logging)
├── Cargo.toml # Dependencies
├── copperconfig.ron # Task graph definition
├── justfile # Helper commands (logreader, replay, DAG rendering)
└── src/
├── main.rs # Runtime entry point
├── logreader.rs # Offline log export utility
├── resim.rs # Replay / remote-debug entry point
├── view.rs # Graph and schedule viewer helper
├── pgs.rs # Profile-guided scheduling helper
├── pgs_candidate.rs # Candidate runtime for measured scheduling
└── tasks.rs # Your task implementations
In the next chapter, we’ll explore what each of these files does.
Try it
You can try to compile and run your project:
cd my_project
cargo run
It will compile and run, and you’ll start to receive some messages:
00:00:01.1781 [Debug] Received message: 42
00:00:01.1781 [Debug] Sink Received message: 43
Kill the process and let’s move to the next chapter.
Project Structure Overview
Let’s look at what’s inside a Copper project and understand the role of each file. Here’s the layout generated by the template:
my_project/
├── build.rs # Build script (sets up logging index)
├── Cargo.toml # Dependencies and project metadata
├── copperconfig.ron # Task graph definition
├── justfile # Helper commands for logs, replay, and DAG rendering
└── src/
├── main.rs # Runtime entry point
├── tasks.rs # Your task implementations
├── logreader.rs # Log export utility
├── resim.rs # Replay / remote-debug entry point
├── view.rs # Graph and schedule viewer helper
├── pgs.rs # Profile-guided scheduling helper
└── pgs_candidate.rs # Candidate runtime for measured scheduling
Which files do I actually work on?
Great news: in the next chapters, we will mainly focus on two files:
| File | What you do there | ROS 2 equivalent |
|---|---|---|
tasks.rs | Implement tasks + define message types | Node source files + .msg files |
copperconfig.ron | Define the task graph, wire connections, set parameters | Launch file + YAML params |
The rest is scaffolding that you set up once and rarely change:
| File | Role | How often you touch it |
|---|---|---|
main.rs | Boilerplate: create logger, build runtime, call run() | Rarely |
build.rs | Sets an env var for Copper’s logging macros | Never |
logreader.rs | CLI tool to decode and export Copper’s binary logs | Rarely |
resim.rs | Replay target for deterministic re-simulation and remote debug | Rarely |
view.rs | Launches the matching graph and schedule viewers | Rarely |
pgs.rs, pgs_candidate.rs | Compare scheduling candidates from recorded timings | When tuning schedules |
justfile | Shortcuts for common commands like just log and just graph | Occasionally |
Cargo.toml | Dependencies | When adding new hardware driver crates |
The mental model
Think of it this way:
copperconfig.ronis the architecture of your robot – what components exist and how they connect. It’s declarative.tasks.rsis the behavior of your robot – what each component actually does. It’s imperative Rust code.main.rsis the engine startup – it boots the runtime that executes your architecture and behavior. You don’t need to understand it deeply to get started.
In the next chapters, we’ll dive into each of these files, starting with the task graph configuration.
The Task Graph: copperconfig.ron
The file copperconfig.ron is the heart of your robot’s architecture. It defines what
tasks exist, how they connect, and what parameters they receive. Copper reads
this file at compile time to generate a deterministic scheduler.
The format is RON (Rusty Object Notation) – a human-readable data format designed for Rust.
The complete example
Here is the copperconfig.ron from our template project:
(
tasks: [
(
id: "src",
type: "tasks::MySource",
),
(
id: "t-0",
type: "tasks::MyTask",
),
(
id: "sink",
type: "tasks::MySink",
),
],
cnx: [
(
src: "src",
dst: "t-0",
msg: "crate::tasks::MyPayload",
),
(
src: "t-0",
dst: "sink",
msg: "crate::tasks::MyPayload",
),
],
)
This defines a three-task pipeline:
MySource ──▶ MyTask ──▶ MySink
"src" "t-0" "sink"
We will discuss these 3 traits (Source, Task, Sink) later.
For the moment, let’s focus on the content of the file.
The tasks section
Each entry in the tasks array declares one task:
(
id: "src", // Unique string identifier
type: "tasks::MySource", // Rust type that implements the task
kind: source, // Optional but recommended: source | task | sink
),
-
id– A unique name for this task instance. Used to reference it in connections. -
type– The fully qualified path to the Rust struct (relative to your crate root). -
kind– v1.0 Declares whether the Rust type implementsCuSrcTask,CuTask, orCuSinkTask. Usesource,task, orsink. Copper can still infer the ordinary roles for legacy configs when the graph shape is unambiguous, but explicitkindis the preferred form.v1.3 For a transform implementing
CuStatelessTask, usekind: stateless_task; see the working transform example.
Optional task fields
Beyond id and type, each task entry supports several optional fields:
-
config– A key-value map of parameters passed to the task’snew()constructor as anOption<&ComponentConfig>. We’ll see how to read them in the Task Anatomy chapter.( id: "gpio", type: "cu_rp_gpio::RPGpio", config: { "pin": 4, }, ), -
missions– A list of mission IDs in which this task is active. Copper supports defining multiple “missions” (configurations of the same robot for different scenarios). A task only gets instantiated if the current mission is in its list. If omitted, the task is active in all missions.( id: "lidar", type: "tasks::LidarDriver", missions: ["outdoor", "mapping"], ), -
background– When set totrue, a compatible task runs on a background thread instead of the critical path. Useful for stages that do heavy or blocking work (network I/O, disk writes) that shouldn’t affect the deterministic scheduling of other tasks.( id: "telemetry", type: "tasks::TelemetryUploader", background: true, ),v1.0 Sources can also run in the background. To run a source or task on a specific thread pool instead of the default
backgroundone, use the object formbackground: (pool: "name"). See Thread Pools, Affinity, and Real-Time Scheduling for declaring pools with custom sizing, CPU affinity, and real-time priority. -
background_process_empty– v1.1.4 v1.2.2 On a regular background task, set this totrueto dispatch jobs even when inputs are empty. The default isfalse; completed results are still collected on empty ticks. See background pool behavior for an example. -
logging– Controls whether Copper’s unified logger records the output messages of this task. Setenabled: falseto reduce log size for high-frequency or uninteresting tasks.( id: "fast-sensor", type: "tasks::HighRateSensor", logging: (enabled: false), ),
The cnx (connections) section
Each entry in cnx wires one task’s output to another’s input:
(
src: "src", // Producing task's id
dst: "t-0", // Consuming task's id
msg: "crate::tasks::MyPayload", // Rust type of the message payload
),
-
src– Theidof the task producing the message. -
dst– Theidof the task consuming the message. -
msg– The fully qualified Rust type of the payload (see next chapter for a focus on this). -
missions(optional) – A list of mission IDs in which this connection is active, just like themissionsfield on tasks. If omitted, the connection is active in all missions.( src: "lidar", dst: "mapper", msg: "crate::tasks::PointCloud", missions: ["outdoor", "mapping"], ),
If a source or regular task intentionally exposes an output that is unused in a given
graph, point that connection at dst: "__nc__". This keeps the output message type
explicit without adding a dummy sink.
Compile-time constants
v1.1
Robot geometry and other values that define the compiled application can live in the
top-level constants array. Numeric constants can select a Rust storage type or an SI
quantity and unit:
constants: [
(id: "MAX_DETECTIONS", storage: usize, value: 64),
(
id: "CAMERA_TRANSLATION",
module: "frames::camera",
quantity: length,
value: [0.18, 0.0, 0.31],
),
]
This generates constants::MAX_DETECTIONS and
frames::camera::CAMERA_TRANSLATION. Values with a physical quantity are normalized to
the coherent SI unit during compilation.
For a struct with a const constructor, provide an explicit Rust type and expression
instead of value:
(
id: "WORLD_TO_ROBOT",
module: "transforms",
type: "cu_transform::TypedTransform3D<f32, cu_transform::WorldFrame, cu_transform::RobotFrame>",
expression: "cu_transform::TypedTransform3D::<f32, cu_transform::WorldFrame, cu_transform::RobotFrame>::from_translation_euler_xyz(crate::frames::robot::TRANSLATION, crate::frames::robot::ROTATION)",
)
Expressions may reference other generated constants and compose values through const
methods. Rust checks the declared type and rejects non-const operations. Use fully qualified
dependency paths and crate::... paths because the expression is compiled in its generated
module. See examples/cu_config_constants for typed transforms and include merging.
How this compares to ROS
In ROS 2, you’d create publishers and subscribers on named topics, and they’d find each
other at runtime via DDS discovery. In Copper, connections are explicit and resolved at
compile time. If you reference a task id that doesn’t exist, you get a compile error –
not a silent runtime misconfiguration.
There’s more
The configuration file we’ve seen here is minimal on purpose. Real-world Copper projects can use additional top-level sections for monitoring, logging tuning, rate limiting, missions, and modular composition. We’ll cover these throughout the book as we go.
Defining Messages
Messages (also called payloads) are the data that flows between tasks. In ROS, you’d
define these in .msg files and run a code generator. In Copper, they’re just Rust structs
with the right derives.
A custom payload
Here’s the message type from our template project (in src/tasks.rs):
#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
value: i32,
}
}
Use Copper’s codec
Import the encoding derives from cu29::bincode and add
#[bincode(crate = "cu29::bincode")] to each type deriving Encode or Decode.
The attribute tells generated code where to find Copper’s codec. This keeps your
payloads, logreader, and runtime on the same encoding implementation, including
when you enable self-describing logs.
You only need the existing cu29 dependency for the codec; keep serde for its
own derives. If a copied example fails with an unresolved bincode import,
check both the import and the derive attribute against the complete example
above.
What each derive does
Every derive on a payload struct has a specific purpose:
| Derive | Purpose |
|---|---|
Default | Required. Copper pre-allocates message buffers at startup. Default provides the initial “empty” value. |
Encode, Decode | Binary serialization for Copper’s zero-alloc message buffers and logging. These come from cu-bincode. |
Serialize, Deserialize | Used for configuration parsing, log export (MCAP, JSON), and tooling. These come from serde. |
Reflect | Runtime type introspection for monitoring tools and simulation integration. |
Debug | Human-readable printing for development. |
Clone | Allows copying messages when needed (e.g., forking data to multiple consumers). |
You do not really have to worry about all of these derives for now. Just add them each time you define a message, and we’ll see how they come into action later.
Using primitive types
For simple cases, you don’t need a custom struct at all. Primitive types like i32, f64,
and bool already implement all the required traits:
#![allow(unused)]
fn main() {
// In copperconfig.ron:
// msg: "i32"
// In your task:
type Output<'m> = output_msg!(i32);
}
This is great for prototyping. As your robot grows, you’ll likely define richer message types with multiple fields.
Latched state updates
v1.0 Not every payload is a fresh sample. Some values are low-rate state: camera calibration, static transforms, lookup tables, map metadata, and similar data that usually stays the same for many cycles.
Re-sending the full value every cycle is wasteful, but making it implicit would hurt determinism and replay. Copper therefore models this explicitly:
CuLatchedStateUpdate<T>is the message that flows across the connectionCuLatchedState<T>is the consumer-side cache stored in your task state
The update variants are:
NoChange– keep the previously latched valueSet(value)– replace the previously latched valueClear– remove the previously latched value
The important mental model is that Copper does not automatically “remember the last
message” for you. The state transition is the message. Each downstream task that needs the
value keeps its own CuLatchedState<T> and applies updates as they arrive.
Producer side
A source or task that discovers calibration data might publish it once with Set(...),
then emit NoChange on later cycles until something changes:
#![allow(unused)]
fn main() {
type Output<'m> = output_msg!(CuLatchedStateUpdate<CameraCalibration>);
fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
if let Some(calibration) = self.pending_calibration.take() {
output.set_payload(CuLatchedStateUpdate::Set(calibration));
} else {
output.set_payload(CuLatchedStateUpdate::NoChange);
}
Ok(())
}
}
If the cached value becomes invalid, emit CuLatchedStateUpdate::Clear.
Consumer side
The consumer stores the latest latched value in its own task state:
#![allow(unused)]
fn main() {
pub struct DepthProjector {
calibration: CuLatchedState<CameraCalibration>,
}
fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>,
output: &mut Self::Output<'_>) -> CuResult<()> {
if let Some(update) = input.payload() {
self.calibration.update(update);
}
let Some(calibration) = self.calibration.get() else {
return Ok(());
};
output.set_payload(project_depth_with(calibration));
Ok(())
}
}
This pattern keeps the common case cheap, because NoChange is just a tiny explicit
update, while still making the state evolution visible in logs and deterministic replay.
Using units directly in payloads
Copper exposes the cu29-units wrappers (through cu29::units) so your payload fields can
carry units directly instead of raw f32 values.
#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use cu29::units::si::f32::{Length, Time, Velocity};
use cu29::units::si::length::{inch, meter};
use cu29::units::si::time::second;
use cu29::units::si::velocity::{kilometer_per_hour, meter_per_second};
use serde::{Deserialize, Serialize};
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct WheelSample {
pub distance: Length,
pub dt: Time,
pub speed: Velocity,
}
impl WheelSample {
pub fn from_raw(distance_m: f32, dt_s: f32) -> Self {
let distance = Length::new::<meter>(distance_m);
let dt = Time::new::<second>(dt_s);
// m / s -> m/s
let speed: Velocity = (distance.into_uom() / dt.into_uom()).into(); // this is type safe
Self {
distance,
dt,
speed,
}
}
pub fn distance_in_inches(&self) -> f32 {
self.distance.get::<inch>()
}
pub fn speed_mps(&self) -> f32 {
self.speed.get::<meter_per_second>()
}
pub fn speed_kph(&self) -> f32 {
self.speed.get::<kilometer_per_hour>()
}
}
}
This gives you unit-safe fields in messages, unit-safe math when building messages, and explicit
conversions when consuming them. Wrapper types support same-dimension arithmetic (+, -) and
scalar scale (* f32, / f32) directly; for cross-dimension operations (like Length / Time),
compute with the underlying uom quantity and convert back with .into() (or from_uom).
Designing good payloads
A few tips for payload design:
- Keep payloads small. They’re pre-allocated and copied between cycles. Large payloads waste memory and cache space.
- Use fixed-size types. Avoid
StringorVecon the critical path. Prefer arrays, fixed-size buffers, or enums. - One struct per “topic”. Each connection in
copperconfig.roncarries exactly one message type. If you need to send different kinds of data, define different structs and use separate connections.
Example: an IMU payload
Here’s what a more realistic payload might look like for an IMU sensor (from here):
#![allow(unused)]
fn main() {
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct ImuPayload {
pub accel_x: Acceleration,
pub accel_y: Acceleration,
pub accel_z: Acceleration,
pub gyro_x: AngularVelocity,
pub gyro_y: AngularVelocity,
pub gyro_z: AngularVelocity,
pub temperature: ThermodynamicTemperature,
}
}
In the next chapter, we’ll see how tasks produce and consume these messages.
For more advanced unit algebra, dimensions, and available units, see the underlying
uom crate docs.
Writing Tasks: tasks.rs
In the previous chapters we looked at copperconfig.ron (the architecture) and message
payloads (the data). Now let’s look at tasks.rs – where the actual behavior lives.
Just like the task graph chapter, we’ll start with the complete file and then break it down piece by piece.
The complete file
Here is the full tasks.rs from our template project:
#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};
// Define a message type
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
value: i32,
}
// Defines a source (ie. driver)
#[derive(Default, Reflect)]
pub struct MySource {}
impl Freezable for MySource {}
impl CuSrcTask for MySource {
type Resources<'r> = ();
type Output<'m> = output_msg!(MyPayload);
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {})
}
fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
output.set_payload(MyPayload { value: 42 });
Ok(())
}
}
// Defines a processing task
#[derive(Reflect)]
pub struct MyTask {}
impl Freezable for MyTask {}
impl CuTask for MyTask {
type Resources<'r> = ();
type Input<'m> = input_msg!(MyPayload);
type Output<'m> = output_msg!(MyPayload);
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {})
}
fn process(
&mut self,
_ctx: &CuContext,
input: &Self::Input<'_>,
output: &mut Self::Output<'_>,
) -> CuResult<()> {
debug!("Received message: {}", input.payload().unwrap().value);
output.set_payload(MyPayload { value: 43 });
Ok(())
}
}
// Defines a sink (ie. actuation)
#[derive(Default, Reflect)]
pub struct MySink {}
impl Freezable for MySink {}
impl CuSinkTask for MySink {
type Resources<'r> = ();
type Input<'m> = input_msg!(MyPayload);
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {})
}
fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>) -> CuResult<()> {
debug!("Sink Received message: {}", input.payload().unwrap().value);
Ok(())
}
}
}
That’s the entire file. Let’s walk through it section by section.
The imports
#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};
}
cu29::prelude::*– Brings in everything you need from Copper: task traits,CuContext,ComponentConfig,CuResult,Freezable,Reflect, theinput_msg!/output_msg!macros, and thedebug!logging macro.cu29::bincodeandserde– For the serialization derives onMyPayload(covered in the Defining Messages chapter).
The message type
#![allow(unused)]
fn main() {
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
value: i32,
}
}
We already covered this in the previous chapter. This is the data that flows between tasks
through the connections defined in copperconfig.ron.
The three task traits
The file defines three structs, each implementing a different trait. Copper provides three task traits for the three roles a task can play in the pipeline:
| Trait | Role | Has Input? | Has Output? | ROS Analogy |
|---|---|---|---|---|
CuSrcTask | Produces data | No | Yes | Publisher / driver node |
CuTask | Transforms data | Yes | Yes | Subscriber + Publisher |
CuSinkTask | Consumes data | Yes | No | Subscriber / actuator node |
Let’s look at each one.
Source Task: CuSrcTask – MySource
#![allow(unused)]
fn main() {
#[derive(Default, Reflect)]
pub struct MySource {}
impl Freezable for MySource {}
impl CuSrcTask for MySource {
type Resources<'r> = ();
type Output<'m> = output_msg!(MyPayload);
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {})
}
fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
output.set_payload(MyPayload { value: 42 });
Ok(())
}
}
}
A source is the entry point of data into the pipeline. It has no input – it generates data, typically by reading from hardware (a camera, an IMU, a GPIO pin).
What happens each cycle: The runtime calls process(), and MySource writes a
MyPayload { value: 42 } into the pre-allocated output buffer. Downstream tasks will
read this value.
Notice the process() signature: it only has output, no input.
Processing Task: CuTask – MyTask
#![allow(unused)]
fn main() {
#[derive(Reflect)]
pub struct MyTask {}
impl Freezable for MyTask {}
impl CuTask for MyTask {
type Resources<'r> = ();
type Input<'m> = input_msg!(MyPayload);
type Output<'m> = output_msg!(MyPayload);
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {})
}
fn process(
&mut self,
_ctx: &CuContext,
input: &Self::Input<'_>,
output: &mut Self::Output<'_>,
) -> CuResult<()> {
debug!("Received message: {}", input.payload().unwrap().value);
output.set_payload(MyPayload { value: 43 });
Ok(())
}
}
}
A processing task sits in the middle of the pipeline. It reads from upstream and writes downstream.
What happens each cycle: The runtime first runs the upstream source, then calls this
task’s process() with the source’s output as input. The task reads the value, logs it,
and writes a new value to its own output for the sink downstream.
Notice the process() signature: it has both input and output.
Sink Task: CuSinkTask – MySink
#![allow(unused)]
fn main() {
#[derive(Default, Reflect)]
pub struct MySink {}
impl Freezable for MySink {}
impl CuSinkTask for MySink {
type Resources<'r> = ();
type Input<'m> = input_msg!(MyPayload);
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {})
}
fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>) -> CuResult<()> {
debug!("Sink Received message: {}", input.payload().unwrap().value);
Ok(())
}
}
}
A sink is the end of the pipeline. It receives data but produces no output. Typically this drives an actuator, writes to a display, or sends data to an external system.
What happens each cycle: The runtime calls process() with the upstream task’s output.
The sink reads the value and does something with it (here, it logs it).
Notice the process() signature: it only has input, no output.
Deterministic tasks in live replay
A live Copper twin can reconstruct a synchronous task from captured inputs,
recorded time, and frozen state. Select it with streaming: (replay: reconstruct)
in the shared graph. Keep logging enabled on that task and all its input
producers; missing replay inputs are rejected at compile time. See
task reconstruction
for supported codecs, a camera pipeline example, and development verification.
Tying it back to the task graph
Remember our copperconfig.ron:
tasks: [
( id: "src", type: "tasks::MySource" ),
( id: "t-0", type: "tasks::MyTask" ),
( id: "sink", type: "tasks::MySink" ),
],
Each type field points to one of the structs we just defined. The connections in cnx
determine which task’s output feeds into which task’s input. The Rust compiler verifies
that the message types match at build time.
In the next chapter, we’ll dissect our task to look more closely at each associated type and method in order to understand exactly what they do.
Task Anatomy: Types and Methods
Now that you’ve seen the three task traits, let’s look closely at the associated types and methods that make them work.
Associated Types
type Resources<'r> and impl Freezable
We’ll cover these in detail in the
Advanced Task Features chapter. For now, notice that both are
defined as empty – type Resources<'r> = () and impl Freezable for MyTask {} – which
is all you need for a simple project.
type Input<'m> (CuTask and CuSinkTask only)
#![allow(unused)]
fn main() {
type Input<'m> = input_msg!(MyPayload);
}
Declares what messages this task receives from upstream. The input_msg!() macro wraps
your payload type into Copper’s message container (CuMsg<T>), which carries:
- The payload itself (your struct)
- Metadata (timestamps, status flags)
- Time of Validity (
tov)
Multiple inputs: If a task receives data from multiple upstream tasks, list the types separated by commas:
#![allow(unused)]
fn main() {
type Input<'m> = input_msg!(SensorA, SensorB);
}
In process(), the input parameter becomes a tuple that you can destructure:
#![allow(unused)]
fn main() {
fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>, ...) -> CuResult<()> {
let (sensor_a_msg, sensor_b_msg) = *input;
// Use sensor_a_msg.payload() and sensor_b_msg.payload()
Ok(())
}
}
type Output<'m> (CuSrcTask and CuTask only)
#![allow(unused)]
fn main() {
type Output<'m> = output_msg!(MyPayload);
}
Declares what messages this task produces for downstream. The output buffer is
pre-allocated by the runtime at startup. You don’t create messages – you fill them
using set_payload().
Methods
fn new(config, resources) -> CuResult<Self>
#![allow(unused)]
fn main() {
fn new(
_config: Option<&ComponentConfig>,
_resources: Self::Resources<'_>,
) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {})
}
}
The constructor. Called once when the runtime builds the task graph.
config: Option<&ComponentConfig> is a key-value map from the task’s config block
in copperconfig.ron. For example, if your RON file has:
(
id: "motor",
type: "tasks::MotorDriver",
config: { "pin": 4, "max_speed": 1.0 },
),
You can read the values in new():
#![allow(unused)]
fn main() {
fn new(config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self> {
let cfg = config.ok_or("MotorDriver requires a config block")?;
let pin: u8 = cfg
.get::<u8>("pin")?
.ok_or_else(|| CuError::from("MotorDriver missing `pin`"))?;
let max_speed: f64 = cfg
.get::<f64>("max_speed")?
.ok_or_else(|| CuError::from("MotorDriver missing `max_speed`"))?;
Ok(Self { pin, max_speed })
}
}
We’ll discuss resources in future chapters.
fn process() – the main loop
This is where your task does its work. The runtime calls it every cycle. The signature depends on the trait:
| Trait | Signature |
|---|---|
CuSrcTask | process(&mut self, ctx, output) |
CuTask | process(&mut self, ctx, input, output) |
CuSinkTask | process(&mut self, ctx, input) |
In our simple example, the source ignores most parameters and just writes a value:
#![allow(unused)]
fn main() {
fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
output.set_payload(MyPayload { value: 42 });
Ok(())
}
}
Let’s look at what each parameter gives you.
Reading input
input.payload() returns Option<&T> – an Option because the message could be empty
(e.g., the upstream task had nothing to send this cycle). In production you should handle
None; in our example we just unwrap:
#![allow(unused)]
fn main() {
let value = input.payload().unwrap();
}
Writing output
output.set_payload(value) writes your data into a buffer that was pre-allocated at
startup.
v1.0 For values that should persist on the consumer side until explicitly replaced, Copper also has a generic latched-state pattern:
CuLatchedStateUpdate<T>::NoChange– keep the cached valueCuLatchedStateUpdate<T>::Set(value)– replace the cached valueCuLatchedStateUpdate<T>::Clear– remove the cached value
Downstream tasks can store that state in CuLatchedState<T> and apply updates as they
arrive. This is a good fit for calibration bundles, static transforms, lookup tables, and
other low-rate metadata that should not be rebuilt every cycle. The full producer and
consumer pattern is covered in Defining Messages.
The context: &CuContext
Every lifecycle callback receives a ctx parameter of type &CuContext. It dereferences
to Copper’s monotonic RobotClock, so ctx.now() gives you the current Copper time, and
it also carries runtime metadata such as ctx.cl_id(), ctx.instance_id(), and the
currently executing task ID.
ctx.now() returns a CuTime measured in nanoseconds on the application’s
timeline. By default, that timeline starts when the clock is constructed. Tasks
should use it rather than std::time::SystemTime::now() or
std::time::Instant::now() so recorded runs can be replayed.
v1.3
With shared reference time, the timeline can instead use a
shared reference epoch, such as TAI. Task code still reads ctx.now().
In our simple project we prefix the parameter with _ because we don’t use it. But on a
real robot you’d use it like this:
Timestamp your output (typical for source tasks):
#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
output.set_payload(MyPayload { value: read_sensor() });
output.tov = Tov::Time(ctx.now());
Ok(())
}
}
Compute a time delta (e.g., for a PID controller):
#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, input: &Self::Input<'_>, output: &mut Self::Output<'_>) -> CuResult<()> {
let now = ctx.now();
let dt = now - self.last_time; // CuDuration in nanoseconds
self.last_time = now;
let error = self.target - input.payload().unwrap().value;
let correction = self.kp * error + self.ki * self.integral * dt.as_secs_f64();
// ...
Ok(())
}
}
Detect a timeout:
#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, input: &Self::Input<'_>) -> CuResult<()> {
if input.payload().is_none() {
let elapsed = ctx.now() - self.last_seen;
if elapsed > CuDuration::from_millis(100) {
debug!("Sensor timeout! No data for {}ms", elapsed.as_millis());
}
}
Ok(())
}
}
Why not use the system clock? Because Copper supports deterministic replay. When
you replay a recorded run, the runtime feeds your tasks the exact same clock values from
the original recording. If you used std::time, the replay would have different
timestamps and your tasks would behave differently. With CuContext (and its underlying
RobotClock), same clock + same inputs = same outputs, every time.
The golden rule of process()
Because process() runs on the real-time critical path (potentially thousands of
times per second), you should avoid heap allocation inside it. Operations like
Vec::push(), String::from(), or Box::new() ask the system allocator for memory,
which can take an unpredictable amount of time and cause your cycle to miss its deadline.
Copper’s architecture is designed so you never need to allocate in process(): messages
are pre-allocated, and the structured logger writes to pre-mapped memory. Keep your
process() fast and predictable.
So far we’ve focused on new() and process() – the two methods you’ll always
implement. But Copper tasks have a richer lifecycle with optional hooks for setup,
teardown, and work that doesn’t need to run on the critical path. Let’s look at that next.
Task Lifecycle
Beyond new() and process(), Copper tasks have a full lifecycle with optional hooks for
setup, teardown, and non-critical work. Understanding this lifecycle helps you put the
right code in the right place.
The full lifecycle
new() → start() → [ preprocess() → process() → postprocess() ] → stop()
└──────────── repeats every cycle ────────────┘
Lifecycle methods
| Method | When | Thread | What to do here |
|---|---|---|---|
new() | Once, at construction | Main | Read config, initialize state |
start() | Once, before the first cycle | Main | Open file handles, initialize hardware, allocate buffers |
preprocess() | Every cycle, before process() | Best-effort | Heavy prep work: decompression, FFT, parsing |
process() | Every cycle | Critical path | Core logic. Keep it fast. No allocations. |
postprocess() | Every cycle, after process() | Best-effort | Telemetry, non-critical logging, statistics |
stop() | Once, after the last cycle | Main | Cleanup: close files, stop hardware, free resources |
The two threads
Copper splits each cycle across two execution contexts:
Critical path thread
This is where process() runs. Tasks execute back-to-back in the order determined by
the task graph topology. The runtime minimizes latency and jitter on this thread. You must
avoid allocations, system calls, and anything that could block.
Best-effort thread
This is where preprocess() and postprocess() run. The runtime schedules these to
minimize interference with the critical path. You can safely do heavier work here:
I/O, allocations, logging, network calls.
What if preprocess() is late?
The critical path never waits for the best-effort thread. If preprocess() takes
longer than expected and hasn’t finished when the critical path is ready, process() runs
anyway – with whatever data is available from the previous cycle (or nothing, if it’s the
first one).
This is intentional: in a real-time system, a late result is a wrong result. It’s better
to run your control loop on slightly stale data than to miss a deadline. Your process()
should handle this gracefully:
#![allow(unused)]
fn main() {
fn preprocess(&mut self, _ctx: &CuContext) -> CuResult<()> {
// Heavy work on the best-effort thread -- might be slow
self.decoded_image = Some(decode_jpeg(&self.raw_buffer));
Ok(())
}
fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>,
output: &mut Self::Output<'_>) -> CuResult<()> {
// Use whatever is ready. If preprocess was late, decoded_image
// still holds the previous cycle's result (or None on first cycle).
if let Some(ref image) = self.decoded_image {
output.set_payload(run_inference(image));
}
Ok(())
}
}
The same applies to postprocess(): if it falls behind, the next cycle’s process()
still runs on time.
Example: when to use each method
Imagine an IMU driver task:
new() → Read the SPI bus config from ComponentConfig
start() → Open the SPI device, configure the sensor registers
preprocess() → (not needed for this task)
process() → Read raw bytes from SPI, convert to ImuReading, set_payload()
postprocess() → Log statistics (sample rate, error count)
stop() → Close the SPI device
Or a computer vision task:
new() → Load the model weights
start() → Initialize the inference engine
preprocess() → Decode the JPEG image from the camera (heavy, OK on best-effort thread)
process() → Run inference on the decoded image, output detections
postprocess() → Update FPS counter, send telemetry
stop() → Release GPU resources
All lifecycle methods are optional
new() and process() are required – everything else has a default no-op implementation.
You only implement the lifecycle methods you need:
#![allow(unused)]
fn main() {
impl CuTask for MyTask {
type Resources<'r> = ();
type Input<'m> = input_msg!(MyPayload);
type Output<'m> = output_msg!(MyPayload);
// Required: constructor
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
where Self: Sized {
Ok(Self {})
}
// Required: core logic
fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>,
output: &mut Self::Output<'_>) -> CuResult<()> {
// your core logic
Ok(())
}
// Optionally implement any of these:
// fn start(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
// fn stop(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
// fn preprocess(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
// fn postprocess(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
}
}
Freeze and thaw (state snapshots)
Copper automatically logs every message flowing between tasks. But messages alone aren’t enough to reproduce a task’s behavior – you also need its internal state.
Consider a PID controller that accumulates error over time. If you want to replay from minute 7 of a 10-minute run to debug a crash, you need to know what the accumulated error was at minute 7. Without state snapshots, you’d have to replay from the very start and wait 7 minutes to get there.
That’s what freeze and thaw solve. The Freezable trait gives each task two hooks:
freeze()– Save the task’s internal state. Called periodically by the runtime to create “keyframes.”thaw()– Restore the task’s state from a saved snapshot.
These are not part of the per-cycle loop. They run at a much lower rate and are independent of the critical path:
┌─── cycle ───┐ ┌─── cycle ───┐ ┌─── cycle ───┐
... ─── process() ─── process() ─── ... ─── process() ─── ...
│ │
freeze() freeze()
(keyframe) (keyframe)
Think of it like a video codec: process() runs every frame, while freeze() saves a
keyframe at a low rate. During replay, the runtime jumps to the nearest keyframe before
minute 7, restores every task’s state via thaw(), and replays from there – no need to
start from the beginning.
v1.3
Tasks implementing CuStatelessTask do not need to implement Freezable.
Their per-cycle callbacks receive &self, and the runtime records an empty
keyframe component for them. Restoring a keyframe leaves these task instances
unchanged. Keep any state that affects later outputs in a task implementing
CuTask, with freeze and thaw preserving that state.
For CuSrcTask, CuTask, and CuSinkTask implementations with no state to
snapshot (like our simple MySource, MyTask, MySink), the empty
impl Freezable is sufficient. We’ll cover how to implement freeze and thaw
for stateful tasks in the Advanced Task Features chapter.
The Remaining Files and Running
We’ve covered copperconfig.ron and tasks.rs – the two files you’ll edit most. Now
let’s look at main.rs, build.rs, and the relevant Cargo.toml bits. These are mostly
boilerplate that you write once and rarely touch. We’ll come back to logreader.rs in
Logging and Replaying Data and replay via resim.rs later.
main.rs – the entry point
v1.2 This entry point uses typed application handles: running consumes the initialized handle and returns a stopped handle.
pub mod tasks;
use cu29::prelude::*;
use std::path::Path;
use std::thread::sleep;
use std::time::Duration;
const PREALLOCATED_STORAGE_SIZE: Option<usize> = Some(1024 * 1024 * 100);
#[copper_runtime(config = "copperconfig.ron")]
struct MyProjectApplication {}
fn main() {
let logger_path = "logs/my-project.copper";
if let Some(parent) = Path::new(logger_path).parent() {
if !parent.exists() {
std::fs::create_dir_all(parent).expect("Failed to create logs directory");
}
}
debug!("Logger created at {}.", logger_path);
debug!("Creating application... ");
let application = MyProjectApplication::builder()
.with_log_path(logger_path, PREALLOCATED_STORAGE_SIZE)
.expect("Failed to setup logger.")
.build()
.expect("Failed to create application.");
debug!("Running... starting clock: {}.", application.clock().now());
let stopped = application.run_until_shutdown().expect("Failed to run application.");
debug!("End of program: {}.", stopped.clock().now());
sleep(Duration::from_secs(1));
}
Here’s what each part does:
pub mod tasks; – Brings in your tasks.rs. The task types you defined there
(e.g., tasks::MySource) are what copperconfig.ron references.
#[copper_runtime(config = "copperconfig.ron")] – The key macro. At compile time,
it reads your config file, parses the task graph, computes a topological execution order,
and generates a custom runtime struct with a deterministic scheduler. It also creates a
builder struct named MyProjectApplicationBuilder. The struct itself is empty – all the
generated code is injected by the macro.
PREALLOCATED_STORAGE_SIZE – How much memory (in bytes) to pre-allocate for the
structured log. 100 MB is a reasonable default.
MyProjectApplication::builder() – Creates the generated application builder.
You provide the pieces you want to override, such as the clock, config override, or
resource factory.
.with_log_path(...).build() – Wires
everything together: creates each task by calling their new() constructors,
pre-allocates all message buffers, and sets up the scheduler.
application.run_until_shutdown() – Starts the deterministic execution loop. Calls start() on
all tasks, then enters the cycle loop (preprocess -> process -> postprocess for
each task, in topological order), and continues until you stop the application (Ctrl+C).
It consumes the initialized handle and returns a stopped handle, so the code uses
stopped.clock() after the run.
application.clock() – Returns the runtime clock handle. The builder creates a
default clock unless you override it with .with_clock(...), which is mainly useful in
tests or simulation.
build.rs – build-time setup
v1.1 Keep the template’s complete build script:
fn main() {
cu29_build::setup();
}
The generated Cargo.toml already includes cu29-build under
[build-dependencies]. This helper prepares the interned string index used by
Copper’s structured logging and the other build-time metadata. Keep its Copper
source and version aligned with cu29.
Controlling the Loop Frequency
If you’ve run our project, you noticed that the output scrolls extremely fast. That’s because by default, Copper runs the task graph as fast as possible – there’s no rate limiter. On a modern machine, that can mean hundreds of thousands of cycles per second.
In this chapter, we’ll see how fast the loop actually runs and how to set a target frequency.
How fast is it running?
Let’s modify MySource in tasks.rs to print the time at each cycle. Replace the
process() method:
#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
debug!("Source at {}µs", ctx.now().as_micros());
output.set_payload(MyPayload { value: 42 });
Ok(())
}
}
Run the project and look at the timestamps:
00:00:00.5296 [Debug] Source at 529632
00:00:00.5297 [Debug] Received message: 42
00:00:00.5298 [Debug] Sink Received message: 43
00:00:00.5300 [Debug] Source at 530005
00:00:00.5301 [Debug] Received message: 42
00:00:00.5302 [Debug] Sink Received message: 43
...
The values are in microseconds. The gap between two Source at ... lines is a few hundred
microseconds. Without a rate limiter, the loop runs as fast as it can – potentially
thousands of cycles per second, pegging a CPU core at 100%. Way too fast for most applications – and it will peg a CPU
core at 100%.
Setting a target frequency
Copper provides a simple way to rate-limit the execution loop. Add a runtime section to
your copperconfig.ron:
(
tasks: [
(
id: "src",
type: "tasks::MySource",
),
(
id: "t-0",
type: "tasks::MyTask",
),
(
id: "sink",
type: "tasks::MySink",
),
],
cnx: [
(
src: "src",
dst: "t-0",
msg: "crate::tasks::MyPayload",
),
(
src: "t-0",
dst: "sink",
msg: "crate::tasks::MyPayload",
),
],
runtime: (
rate_target_hz: 1,
),
)
The only change is the runtime section at the bottom. rate_target_hz: 1 tells Copper
to target 1 cycle per second (1 Hz).
Run again:
00:00:00.0005 [Debug] Source at 510
00:00:00.0005 [Debug] Received message: 42
00:00:00.0005 [Debug] Sink Received message: 43
00:00:01.0000 [Debug] Source at 1000019
00:00:01.0001 [Debug] Received message: 42
00:00:01.0001 [Debug] Sink Received message: 43
Now each full cycle (source -> task -> sink) runs once per second.
How it works
rate_target_hz acts as a rate limiter, not a scheduler. After each complete cycle,
the runtime checks how much time has elapsed. If the cycle finished faster than the target
period (e.g., under 10ms for 100 Hz), the runtime waits for the remaining time. If the
cycle took longer than the target period, the next cycle starts immediately – no time is
wasted.
This means:
- Your actual frequency is at most
rate_target_hz. - If your tasks are too slow for the target, the loop runs at whatever rate it can sustain.
- Without the
runtimesection, the loop runs flat-out with no pause between cycles.
For host-side applications that need tighter cadence at higher rates, Copper also ships an
optional high-precision-limiter crate feature. That mode keeps the same
runtime.rate_target_hz config, but the std runtime switches from pure sleep-based waiting
to a hybrid sleep-then-spin limiter with absolute deadlines. It burns more CPU near each
tick, so it is meant for metronomic loops, not for the default “save cycles” use case.
Difference with ROS
In ROS, each node controls its own frequency. A camera node publishes at 30 Hz, an IMU
node publishes at 200 Hz, and a planner node runs at 10 Hz – all independently. Nodes
are loosely coupled via topics, and each one has its own rospy.Rate() or rclcpp::Rate
timer that governs how often it publishes.
Copper works differently. There is one global loop that executes the entire task graph
in sequence: source -> processing -> sink, back-to-back, in topological order. Every task
runs every cycle, as fast as possible. The rate_target_hz setting doesn’t make
individual tasks run at different speeds – it tells the runtime how long to wait
between cycles so the whole pipeline doesn’t run flat-out and peg the CPU.
ROS: Node A (30 Hz) ──┐
Node B (200 Hz) ──┼── independent timers, publish on topics
Node C (10 Hz) ──┘
Copper: [ Source → Task → Sink ] → wait → [ Source → Task → Sink ] → wait → ...
└──── one cycle ────────┘ └──── one cycle ────────┘
global rate_target_hz
The key insight: in Copper, all tasks share the same cadence. If you need different parts of your system to run at different rates (e.g., a fast inner control loop and a slow planner), you’d use separate task graphs or implement logic inside a task to skip cycles. But for most applications, a single frequency for the whole pipeline is simpler and avoids the synchronization headaches that come with multiple independent timers.
Shared reference time
v1.3
Two robots exchanging timestamped measurements need timestamps with the same
meaning. A shared reference clock gives ctx.now() a named epoch and lets the
runtime stop consuming measurements when its synchronization uncertainty exceeds
your chosen limit. Task code still uses ordinary Copper timestamps.
See synchronization in the monitor
Use this experimental feature from the copper-rs checkout named in the
edition notice. The
Linux demo needs Rust, just, ptp4l, phc2sys, pmc, ethtool, and ip, plus
an active Ethernet interface exposing /dev/ptp0 and read access to that device.
Use a test network: the demo advertises a local PTP master on that interface.
From the repository root, in an interactive terminal, run:
cd examples/cu_clock_sync
just
The recipe asks for sudo to start the PTP services it needs, then opens
ConsoleMon’s CLK panel. It cleans up the services it started when you exit;
existing services are reused. If services are already configured with suitable
permissions, just run starts only the Copper application.
Look for Clock / reference, a shared epoch, Offset from reference, and Estimated uncertainty. The demo’s 20 ms uncertainty limit decides when the clock becomes unusable; it is not an accuracy target. A local reference sourced from the host’s own clock demonstrates synchronization without establishing absolute accuracy. Local calendar time appears only when the provider supplies a valid TAI minus UTC offset.
For another PHC device or software timestamping, follow the example’s Linux setup guide.
Configure your application’s reference
The clock-sync feature makes ctx.now() use a shared
reference epoch. The provider chooses that epoch: Linux PTP adapters expose TAI,
while board and GNSS providers can supply bare GPS or application-defined
nanoseconds without conversion. Enable
clock-sync on cu29, enable linux-phc on cu-ptp, then select an owned resource:
resources: [(
id: "ptp",
provider: "cu_ptp::LinuxPtpBundle",
config: {
"device": "/dev/ptp0",
"management": "/var/run/ptp4l",
"reference_error_ns": 10000,
},
)],
runtime: (clock: (parent: "ptp.reference", max_error_ns: 100000, sample_interval_ns: 100000000)),
The resource reads a PHC already disciplined by ptp4l. Match its device, port and
domain to the service. Choose a measured upstream uncertainty bound through
reference_error_ns; Copper adds capture, drift and remaining phase error.
Acquisition finishes before tasks start. Task code continues to stamp messages
with ordinary ctx.now() and Tov::Time.
The generated Serial runtime samples references outside task process hooks and
checks clock health before each iteration. Clock clones share the same monotonic
curve. ctx.clock.sync_status() reports domain/session, sample age, phase,
drift and estimated error. Reference loss enters bounded holdover. Exceeding the
error budget or age limit fails the iteration before consumers execute;
restarting acquires the reference again. This experimental API supports the Serial plan. Pipeline scheduling is rejected
because correction changes must be coordinated across in-flight CopperLists.
Optional runtime fields are acquisition_timeout_ns (10000000000),
sample_interval_ns (1000000000), max_age_ns (5000000000),
drift_bound_ppb (100000) and max_slew_ppb (500000). The error budget can expire
before the maximum age. Raw time drives cadence and deadlines independently of
corrections.
Each RobotClock owns its hardware-counter calibration, including when
clock-sync is disabled. clock.raw_now() returns a CuInstant on the
undisciplined local timeline, while clock.now() and clock.recent() return
execution timestamps as CuTime. Both use nanoseconds; intervals use
CuDuration. These distinct types prevent raw readings from being used as
message timestamps or execution timestamps from being used as raw deadlines. Clones share its origin and calibration; independently
constructed clocks have separate raw origins. Use ctx.now() for message
timestamps and raw time for reference capture and elapsed-time measurements.
CuInstant readings use an explicit clock handle. Use clock.raw_now() for counter reads, clock.busy_wait_for(duration)
for a raw elapsed-time wait, and clock.busy_wait_until(raw_deadline) for a
deadline from that clock or one of its clones. Raw reads and waits are available
with or without synchronization. Mock clocks require their control handle to
advance time while waiting. For typed readings from the control handle, use
control.raw_now(). Its existing now() accessor retains the legacy view of
the raw counter as CuTime; execution time comes from the associated clock.
For a captured raw reading, CuInstant::from_nanos(nanos) reconstructs an
instant using the capturing clock’s origin. ClockSnapshot::at(instant) maps
it to execution CuTime through the recorded epoch and rate.
For a local experiment without a PHC, LinuxSystemPtpBundle reads the system UTC
clock associated with software-timestamped ptp4l. It requires the known TAI−UTC
offset. A board bundle can export BoardPtp<HZ> with BSP hooks for its existing
PTP service and a known counter frequency. Initialize and extend the board timer
in the BSP; keep Cortex-M counter reads in the foreground.
Unified logs record each published correction in the RuntimeLifecycle stream.
Enable clock-sync in recorded replay applications and matching logreaders.
Replay restores curves, quality, source and epoch without starting reference I/O. Unconfigured
clocks keep their local epoch and constructor APIs.
ctx.clock.origin() identifies both the advertised upstream source and the epoch
of ctx.now(). For example, a GNSS receiver distributed through PTP can report
ClockSource::Gnss with ClockEpoch::Tai; a provider supplying bare GPS time
reports ClockSource::Gnss with ClockEpoch::Gps. Source describes provenance,
while epoch identifies what timestamp zero means. Unknown provenance stays unknown.
The accessor performs no counter read, allocation, reference I/O or conversion.
Board providers return source and epoch through BoardPtpHooks::origin, matching
the nanoseconds returned by read. Reference observations carry this metadata with
the captured time. For initialization without live synchronization,
RobotClock::from_ref_time_with_origin(ns, origin) selects a named epoch; use
from_ref_time_with_rtc_and_origin when supplying RTC calibration functions.
An already synchronized clock keeps its epoch during resynchronization; selecting
a different epoch requires a new clock.
ConsoleMon’s CLK panel shows numeric robot time, source and epoch. For TAI with
a valid provider-supplied tai_minus_utc_seconds, it also shows the local timezone
as an ISO 8601 date to whole seconds, including the timezone offset. The monitor
performs this conversion only while drawing CLK. If the TAI minus UTC offset is
unknown, local time is unavailable. GPS and application-defined values stay numeric;
calendar conversion belongs to application display/debug code.
Follow the runnable Linux example for software-grandmaster and hardware-grandmaster setup. The component’s reference guide describes Linux resource parameters and the board contract.
Adding the Console Monitor
In the previous chapter we controlled the loop frequency. But how do we know how fast each
task actually runs? How much time does process() take? What’s the jitter?
Copper comes with a built-in TUI (terminal user interface) monitor called CuConsoleMon that displays live statistics for every task in your pipeline – mean execution time, standard deviation, min, max, and more.
Step 1: Add the dependency
Add cu-consolemon to your Cargo.toml next to your existing Copper dependencies.
The only new dependency is cu-consolemon.
If your project already uses Copper through git or local path dependencies, keep that
same source style and add cu-consolemon from the same Copper release. If you use
crates.io, use the same published Copper release as your existing cu29 dependency.
Step 2: Enable it in copperconfig.ron
Add a monitor section to your copperconfig.ron:
(
tasks: [
(
id: "src",
type: "tasks::MySource",
),
(
id: "t-0",
type: "tasks::MyTask",
),
(
id: "sink",
type: "tasks::MySink",
),
],
cnx: [
(
src: "src",
dst: "t-0",
msg: "crate::tasks::MyPayload",
),
(
src: "t-0",
dst: "sink",
msg: "crate::tasks::MyPayload",
),
],
runtime: (
rate_target_hz: 1,
),
monitor: (
type: "cu_consolemon::CuConsoleMon",
),
)
Step 3: Run
cargo run
Instead of the scrolling debug output you’re used to, you’ll see a live terminal dashboard that takes over your terminal. Navigate to the latency tab (you can click on it)
The latency tab shows one row per task, plus an End2End row at the bottom for the entire pipeline. Here’s what each column means:
| Column | Meaning |
|---|---|
| Task | The task id from your copperconfig.ron |
| Min | Fastest process() execution seen across all cycles |
| Max | Slowest process() execution seen (watch for spikes here) |
| Mean | Average process() execution time |
| Stddev | Standard deviation – how much the execution time varies. Lower is more predictable |
| Jitter (mean) | Average difference between consecutive cycle times. Indicates how stable the timing is |
| Jitter (max) | Worst-case jitter seen. A large value means occasional timing spikes |
The End2End row measures the full pipeline latency: from the start of src to the end
of sink. In our example, the entire pipeline completes in about 187 µs on average, with
a worst case of 375 µs.
This is extremely useful for:
- Verifying your frequency: Are your tasks fast enough for the target
rate_target_hz? - Finding bottlenecks: Which task takes the most time?
- Detecting jitter: A high Std Dev means unpredictable execution time, which could be a problem for real-time control.
Difference with ROS
In ROS 2, getting this kind of per-node timing data typically requires external tools like
ros2 topic hz, ros2 topic delay, or hooking up a tracing framework. Each tool gives
you one metric at a time, for one topic.
In Copper, the console monitor is built into the runtime. One line of config gives you a live dashboard of every task in the system, with no external tooling needed.
Press Ctrl+C to stop the application as usual.
The other tabs
The monitor has six tabs, accessible by pressing 1 through 6 or clicking on them.
1 - SYS: System Info
Displays your system information: OS, kernel, CPU, memory, and the Copper version. Useful for documenting what hardware a recording was made on.
2 - DAG: Task Graph
A visual representation of your task graph. Each box is a task, showing its id, Rust
type, and the message types on its connections. This is your copperconfig.ron rendered
live – a quick way to verify the pipeline is wired correctly.
4 - BW: Bandwidth
Shows memory and disk bandwidth consumed by the runtime. The left panel (Memory BW) reports the observed loop rate, CopperList size, and raw memory throughput. The right panel (Disk / Encoding) shows the serialized size of each CopperList, encoding efficiency, structured log bandwidth, and total disk write rate. Useful for sizing your log storage and checking that the logger can keep up.
v1.2 With LogStream configured, BW also shows a Telemetry / TX panel per destination: local submissions, budget, drops, and recovery activity. Two-way streams add receiver reports and the effective FEC repair interval. See link health and feedback for setup and interpretation.
5 - MEM: Memory Pools
Displays the status of Copper’s pre-allocated memory pools: pool ID, used vs total buffers, buffer size, handles in use, and allocation rate. In our simple project the pools are empty, but on a real robot with large messages (images, point clouds), this helps you verify that buffers are correctly sized and not exhausted.
6 - LOG: Debug Output
A scrollable view of all debug!() log output from your tasks – the same messages you’d
see without the monitor, but captured inside the TUI. You can scroll through the history
with hjkl or arrow keys. The bottom bar also shows keyboard shortcuts: r to reset
latency statistics, q to quit.
Logging and Replaying Data
Every time you’ve run our project so far, Copper has been quietly recording everything.
Look at main.rs – the generated app builder’s .with_log_path(...) call initializes a unified logger
that writes to logs/my-project.copper. Every cycle, the runtime serializes every message
exchanged between tasks (the CopperList) and writes it to that file.
In this chapter, we’ll explore what’s in that log file, how to read it back, and how to replay recorded data through the pipeline.
What gets logged?
Copper’s unified logger captures two kinds of data in a single .copper file:
-
Structured text logs – Every
debug!(),info!(),warn!(), anderror!()call from your tasks. These are stored in an efficient binary format (not as text strings), so they’re extremely fast to write and compact on disk. -
CopperList data – The complete set of message payloads exchanged between tasks in each cycle. In our project, that means every
MyPayload { value: 42 }fromMySourceand everyMyPayload { value: 43 }fromMyTask, along with precise timestamps.
This is different from most robotics frameworks where logging is opt-in and you have to explicitly record topics. In Copper, every message is logged by default. The runtime does this automatically as part of its execution loop – no extra code needed.
Typed logging codecs
v1.0
A task’s logging.codec or logging.codecs setting selects a typed codec for its
recorded payloads, such as lossless PNG or FFV1 image compression. This lets the
encoding follow the payload’s structure while keeping codec selection in RON.
Keep the logreader built for the application version that produced a recording. The codec’s representation is part of the encoded content, whose layout can change between application versions. The unified-log file and section header version describes the outer storage layout.
Step 1: Generate a log file
Make sure your project is in the state from the previous chapters, with the 1 Hz rate
limiter. Here’s the copperconfig.ron for reference:
(
tasks: [
(
id: "src",
type: "tasks::MySource",
),
(
id: "t-0",
type: "tasks::MyTask",
),
(
id: "sink",
type: "tasks::MySink",
),
],
cnx: [
(
src: "src",
dst: "t-0",
msg: "crate::tasks::MyPayload",
),
(
src: "t-0",
dst: "sink",
msg: "crate::tasks::MyPayload",
),
],
runtime: (
rate_target_hz: 1,
),
)
Note: If you still have the
monitorsection from the previous chapter, remove it for now. The console monitor takes over the terminal and makes it harder to see the debug output.
Run the project and let it execute for 5-10 seconds, then press Ctrl+C:
cargo run
00:00:03.9986 [Debug] Received message: 42
00:00:03.9987 [Debug] Sink Received message: 43
00:00:04.9979 [Debug] Source at 4997916
00:00:04.9980 [Debug] Received message: 42
00:00:04.9981 [Debug] Sink Received message: 43
...
After stopping, check the logs/ directory:
ls -lh logs/
-rw-r--r-- 1 user user 4.0K logs/my-project.copper
That .copper file contains everything: every message, every timestamp, every debug line.
Note: The log path is hardcoded to
"logs/my-project.copper"inmain.rs. Each run overwrites the previous log file – there is no automatic rotation or timestamping. If you want to keep a log from a previous session, rename or move the file before running the project again.
Step 2: The log reader
If you look at your project, you’ll notice there’s already a file you haven’t used yet:
src/logreader.rs. The project template ships with a built-in log reader. Let’s look at
it:
pub mod tasks;
use cu29::prelude::*;
use cu29_export::run_cli;
// This will create the CuStampedDataSet that is specific to your copper project.
// It is used to instruct the log reader how to decode the logs.
gen_cumsgs!("copperconfig.ron");
#[cfg(feature = "logreader")]
fn main() {
run_cli::<CuStampedDataSet>().expect("Failed to run the export CLI");
}
This is a small but powerful program. Let’s break it down:
gen_cumsgs!("copperconfig.ron") – This macro reads your task graph and generates a
CuStampedDataSet type that knows the exact message types used in your pipeline. The log
reader needs this to decode the binary data in the .copper file – without it, the bytes
would be meaningless.
run_cli::<CuStampedDataSet>() – This is Copper’s built-in export CLI. It provides
several subcommands for extracting data from .copper files. By passing the generated
CuStampedDataSet type, you tell it exactly how to decode your project’s messages.
#[cfg(feature = "logreader")] – The log reader is gated behind a Cargo feature flag
so its dependencies (like cu29-export) are only compiled when you actually need them.
Step 3: Extract text logs
The first thing you can do with the log reader is extract the structured text logs – the
debug!() messages from your tasks. Remember, these aren’t stored as text in the
.copper file; they’re stored as compact binary indices. The log reader reconstructs the
human-readable text using the string index that was built at compile time.
Run:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper extract-text-log target/debug/cu29_log_index
The arguments are:
logs/my-project.copper– The log file to readextract-text-log– The subcommand to extract text logstarget/debug/cu29_log_index– The path to the string index directory (generated during compilation bybuild.rs)
You’ll see output like:
25.501 µs [Debug]: Logger created at logs/my-project.copper.
45.903 µs [Debug]: Creating application...
64.282 µs [Debug]: CuConfig: Reading configuration from file: copperconfig.ron
669.067 µs [Debug]: Running... starting clock: 666.866 µs.
823.766 µs [Debug]: Source at 822
870.122 µs [Debug]: Received message: 42
887.054 µs [Debug]: Sink Received message: 43
1.000 s [Debug]: Source at 1000206
1.000 s [Debug]: Received message: 42
1.000 s [Debug]: Sink Received message: 43
2.000 s [Debug]: Source at 1999631
...
This is the same output you saw scrolling by when the application was running – but reconstructed from the binary log after the fact.
Step 4: Extract CopperLists
The more interesting subcommand extracts the CopperList data – the actual message payloads from every cycle:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper extract-copperlists
Serialized CopperLists contain the cycle id and msgs. Lifecycle state is
runtime bookkeeping and is omitted from binary logs, JSON, Python exports, and
remote-debug snapshots. Decode messages with the producing application’s typed
logreader or its embedded catalog; the file encapsulation version describes the
file and section layout.
The output is JSON by default. Here’s what the first CopperList looks like:
{
"id": 0,
"msgs": [
{
"payload": {
"value": 42
},
"tov": "None",
"metadata": {
"process_time": {
"start": 822050,
"end": 867803
},
"status_txt": ""
}
},
{
"payload": {
"value": 43
},
"tov": "None",
"metadata": {
"process_time": {
"start": 869282,
"end": 885259
},
"status_txt": ""
}
}
]
}
Every message from every cycle is there, exactly as it was produced. Each entry in msgs
corresponds to a connection in your task graph (in order: src→t-0, then t-0→sink).
Along with the payload, you get:
metadata.process_time– The start and end timestamps (in nanoseconds) of the task’sprocess()call that produced this message. This is the same timing data the console monitor uses for its latency statistics.tov– “Time of validity”, an optional timestamp that the task can set to indicate when the data was actually captured (useful for hardware drivers with their own clocks).status_txt– An optional status string the task can set for diagnostics.
This is the raw data you’d use for offline analysis, regression testing, or replay.
Why this matters: replay
Recording data is useful for post-mortem analysis, but the real power of Copper’s logging is deterministic replay. Because every message and its timestamp is recorded, you can feed logged data back into the pipeline and reproduce the exact same execution – without any hardware.
This means you can:
- Debug without hardware: Record a session on the real robot, then replay it on your laptop to test processing logic.
- Regression test: Record a known-good session, then replay it after code changes to verify the pipeline still produces the same results.
- Analyze edge cases: When your robot encounters an unusual situation, the log captures it. You can replay that exact moment over and over while you debug.
The key insight is that all downstream tasks don’t know the difference – they receive
the same MyPayload messages with the same timestamps, whether they come from live
hardware or a log file.
Remote debug and Time Traveler
When exported CopperLists are not enough, a replay binary can expose the log through Copper’s remote debug API. A debugger such as Time Traveler can then seek or step through the timeline, replay batches, inspect task state, query message schemas, and read structured logs from the same session.
Replay CLI and structured log queries
v1.0 Replay targets use three standard arguments:
--debug-base <zenoh-prefix>
--log-base <recorded-log>
--replay-log-base <replay-output>
Replay and remote-debug servers should be optimized. A normal Cargo dev build is unoptimized and can make seeking, stepping, payload decoding, and state inspection much slower. Copper’s generated projects therefore define this profile:
[profile.debug-optimized]
inherits = "dev"
opt-level = 3
debug = 2
debug-assertions = true
The generated just resim and just resim-debug recipes select it with
--profile debug-optimized. This keeps Copper debug!() structured log entries, debug
assertions, and debugger information while optimizing the replay server. Use the same
profile for hand-written replay recipes rather than running the server with Cargo’s
unoptimized dev profile.
Application replay binaries normally use cu29::replay::ReplayCli, or flatten
ReplayArgs into a larger CLI. The most useful remote methods are:
nav.seek,nav.step, andnav.replayfor navigationtimeline.get_clandtimeline.listfor recorded messages and timingstate.inspect,state.read, andstate.searchfor task stateschema.get_typeandschema.get_outputsfor debugger-driven renderinglogs.stringsandlogs.listfor structured logs
Session leases and deferred payloads
v1.1
Each replay server grants one active debug-session lease. While the debugger is otherwise
idle, it sends health.ping with the active session_id every second. Any session-scoped
request also renews the lease. After 3 seconds without session activity, the server
automatically closes the session so the debugger can reconnect after a crash or lost
connection. A second session.open returns SessionBusy while the lease is active.
To inspect two logs or maintain two independent timeline cursors concurrently, launch two
replay-server processes with different --debug-base and --replay-log-base values.
Handle-backed payloads stay lazy. A normal response contains only a compact CuHandle
descriptor. When a CBOR request explicitly sets include_handle_contents: true, selected
handle buffers are returned out of line in the response’s attachments array. This lets
an inspector load a camera frame or point cloud on demand without copying every large
buffer while it scrubs the timeline.
The debug value format preserves values that JSON alone cannot represent faithfully,
including 128-bit integers, non-finite floats, and maps with non-string keys. Schema
queries describe enums, maps, and fixed-capacity CuArrayVec values, and
schema.get_outputs supports paging output slots and their field catalogs separately.
Linux shared-memory setup
v1.1
On Linux, the default local remote-debug transport uses a 1 GiB Zenoh shared-memory pool
and requires replies of 256 KiB or larger to use SHM. Both the Copper process and the
debugger need sufficient RLIMIT_MEMLOCK, and /dev/shm must have room for the pool.
Configure the launching shell before starting either process:
sudo prlimit --pid $$ --memlock=2147483648:2147483648
ulimit -l 2097152
df -h /dev/shm
The ulimit -l value is in KiB. For systemd use LimitMEMLOCK=2G; for Docker configure
both --ulimit memlock=2147483648:2147483648 and --shm-size=2g.
Copper checks these resources at startup and refuses to silently copy a large reply over
the Unix socket when SHM allocation fails. Tests can use a smaller explicit
RemoteDebugShmConfig on both server and client. Non-Linux platforms retain the previous
4 MiB pool and 3 KiB threshold without Linux-specific preflight.
How the unified logger works under the hood
Copper’s logger is designed for zero-impact logging on the critical path. Here’s how:
-
Pre-allocated memory slabs – At startup,
.with_log_path(...)allocates a large contiguous block of memory (controlled byPREALLOCATED_STORAGE_SIZE– 100 MB in our project). CopperLists are written into this pre-allocated buffer without any dynamic allocation. -
Binary serialization – Messages are serialized using
bincode, not formatted as text. This is why your payloads need theEncode/Decodederives. Binary serialization is orders of magnitude faster thanformat!()orserde_json. -
Memory-mapped I/O – The pre-allocated slabs are memory-mapped to the
.copperfile. The OS handles flushing to disk asynchronously, so the robot’s critical path never blocks on disk I/O. -
Structured text logging – Even
debug!()calls don’t format strings at runtime. Instead, Copper stores a compact index and the raw values. The actual string formatting happens only when you read the log – not when you write it. This is why thebuild.rssets upLOG_INDEX_DIR– it’s building a string table at compile time.
This means logging in Copper is almost free on the hot path. You can log everything without worrying about performance – which is exactly why it logs everything by default.
Logging and logstream extensibility
To use a custom payload, codec, storage backend, or packet transport, choose the extension point below. This map follows task messages and state snapshots through the APIs in this edition’s 1.3.0-dev checkout.
Purple means a Rust type selected in RON: you can name your own implementation. Teal means an implementation supplied in Rust. Blue marks public APIs or records; gray marks core machinery. A setting that only tunes an algorithm does not make that algorithm replaceable.
| Replace or extend | Where to connect your implementation |
|---|---|
| Task | tasks[].type in copperconfig.ron: name your CuTask or CuSrcTask implementation. |
| Message payload | cnx[].msg: name your payload type. The CuMsg envelope stays the same. |
| State snapshot | Implement Freezable::freeze and thaw on your task. Generated capture calls these methods. |
| Payload codec | Define a CuLogCodec<Payload> type in logging.codecs[].type, give it an id, then select that id with tasks[].logging.codec or the per-message logging.codecs map on the task. |
| Logger and section storage | Supply your UnifiedLogWrite<S> and SectionStorage implementations through the generated app’s builder.with_logger::<S, L>(logger). |
| Byte storage backend | Implement ByteStorage and ByteRegion, construct ByteLogger with that backend, and inject it through with_logger. |
| Packet transport | Implement CuStreamTx; select its type and resource through log_streaming.destinations[].transport.{type, resource}. The resource comes from a bundle declared under resources. Rust construction can instead inject the transport with builder.with_logstream(transport, config). |
| Semantic record sink | Implement WriteStream<Record>. Attaching an arbitrary sink to generated app output requires changing the output wiring; there is no RON sink-type slot. |
The payload codec changes payload encoding inside the record; bincode remains
the serialization framework. Likewise, logstream’s link, fec, and
recovery_interval settings tune the built-in sender rather than select a custom
sender algorithm. Replacing those mechanisms requires a core change.
The runtime owns message slots, snapshot order, output handoff, and buffer reuse. Local storage and logstream branch from completed records, so a streaming transport does not need to read the robot’s slab files. For transport wiring and sender settings, continue with Streaming Logs and Live Telemetry and its reference.
Controlling what gets logged
Sometimes you don’t want to log everything. A high-frequency sensor producing megabytes per second can fill up your log storage quickly. Copper provides two ways to control this:
Per-task logging control
In copperconfig.ron, you can disable logging for specific tasks:
(
id: "fast-sensor",
type: "tasks::HighRateSensor",
logging: (enabled: false),
),
This stops the runtime from recording that task’s output messages in the CopperList log. The task still runs normally – it just doesn’t contribute to the log file.
Global logging settings
The logging section in copperconfig.ron lets you tune the logger globally:
logging: (
slab_size_mib: 1024,
section_size_mib: 100,
),
Task-state keyframes
v1.2 Task-state keyframes can be disabled independently of CopperList and structured logging:
logging: (
enable_keyframe_logging: false,
),
This is a compile-time application property. The generated runtime omits keyframe capture code when it is disabled, which is useful for applications whose background workers may still be active at a CopperList boundary. Replay then has message history but no task-state snapshots to restore.
Streaming to a ground station
v1.2 To collect a native archive on another machine while the robot runs, continue to Streaming Logs and Live Telemetry. The guide covers UDP setup, structured robot logs, live task reconstruction, optional receiver feedback, and recovery after packet loss. Per-task logging controls also determine which inputs are available for reconstruction.
Difference with ROS
In ROS, data recording is a separate tool: rosbag2. You start a ros2 bag record
process alongside your running nodes, tell it which topics to subscribe to, and it saves
messages into a SQLite database or MCAP file. Replay is done with ros2 bag play, which
republishes the messages on the same topics.
ROS:
Run: ros2 launch my_robot.launch.py
Record: ros2 bag record /camera /imu /cmd_vel ← separate process
Replay: ros2 bag play my_bag/ ← republishes on topics
Copper:
Run: cargo run ← logging is automatic
Read: cargo run --features logreader --bin my-project-logreader ...
Replay: feed CopperLists back into the pipeline ← deterministic
Key differences:
| ROS 2 (rosbag2) | Copper (unified logger) | |
|---|---|---|
| Opt-in vs automatic | You must explicitly record topics | Everything is logged by default |
| Separate process | ros2 bag record runs alongside | Built into the runtime – zero config |
| Format | SQLite / MCAP | Custom binary .copper format |
| Performance impact | Adds subscriber overhead per topic | Near-zero – pre-allocated, memory-mapped |
| Replay mechanism | Republishes on topics | Feed CopperLists directly into tasks |
| Deterministic | Timing depends on DDS, not guaranteed | Timestamps are recorded, replay is deterministic |
| Text logging | Separate (rosout, spdlog) | Unified – text and data in one file |
The biggest philosophical difference: in ROS, recording is something you do. In Copper,
recording is something that happens. You don’t configure it, you don’t start it, you
don’t choose which topics to record. The runtime records everything, always. You only need
to decide what to exclude (via logging: (enabled: false)) if storage is a concern.
This “record everything by default” approach is what makes Copper’s deterministic replay possible. Since every message and every timestamp is captured automatically, you can always go back and reproduce any moment of your robot’s execution.
Keep recordings across restarts and bound disk use
The normal .with_log_path(...) setup creates a fresh recording.
v1.2 Use append when you want another run of the same application in that recording.
v1.3 Use rollover for a robot that must keep recording within a fixed storage budget, retaining recent history as older sections are reclaimed.
This exercise uses the single-crate project from
The Remaining Files and Running. Keep its
src/tasks.rs and task graph, and remove any rate_target_hz setting for the
longer rollover exercise below.
1. Create a short recording
In copperconfig.ron, add or replace the top-level logging section:
logging: (
slab_size_mib: 4,
section_size_mib: 1,
enable_keyframe_logging: false,
),
This exercise uses tasks with no changing internal state and disables keyframe capture to fit the small budget. For a stateful replay application, keep keyframes and budget for their stream as well.
Replace src/main.rs with this complete entry point:
pub mod tasks;
use cu29::prelude::*;
use std::path::Path;
use std::sync::{Arc, Mutex};
#[copper_runtime(config = "copperconfig.ron")]
struct MyProjectApplication {}
fn main() -> Result<(), Box<dyn std::error::Error>> {
std::fs::create_dir_all("logs")?;
let logger = UnifiedLoggerBuilder::new()
.write(true)
.create(true)
.file_base_name(Path::new("logs/retained.copper"))
.preallocated_size(4 * 1024 * 1024)
.append(false)
.build()?;
let UnifiedLogger::Write(logger) = logger else {
return Err("Expected a write-capable logger".into());
};
let app = MyProjectApplication::builder()
.with_unified_logger(Arc::new(Mutex::new(logger)))
.build()?;
let mut running = app.start().expect("start application");
for _ in 0..10 {
running.run_one_iteration()?;
}
running.stop().expect("stop application");
Ok(())
}
Run from the project root:
cargo run
It records ten cycles and exits normally. Normal teardown closes the streams
and logger so the next process can append. Keep the entire retained.copper
slab family.
2. Append another run
v1.2
Change .append(false) to .append(true) in that entry point, then run it again:
cargo run
v1.3 Use the logreader’s run-selection commands to inspect the appended recording:
cargo run --features logreader --bin my-project-logreader -- logs/retained.copper list-runs
cargo run --features logreader --bin my-project-logreader -- logs/retained.copper --run 1 extract-copperlists
These commands use the my-project-logreader binary from the book’s first
project; use your generated binary name if you named it differently. The list
now contains two runs. Selecting run 1 exports the second set of ten cycles,
whose CopperList IDs start again for the new construction.
Append requires a cleanly closed log, the same slab allocation size, and matching application metadata and configuration. With self-describing logs it also checks the complete shared catalog. A crash or metadata mismatch rejects append. Use a fresh path when you change the application, mission set, or configuration.
3. Bound the recent history
v1.3
Add .rollover(8 * 1024 * 1024) before .build() on the logger builder and
change the loop to for _ in 0..300_000. Keep .append(true) to continue the
recording already created. Run a release build to avoid printing every debug
line while generating enough cycles to fill the budget:
cargo run --release
cargo run --features logreader --bin my-project-logreader -- logs/retained.copper list-runs
cargo run --features logreader --bin my-project-logreader -- logs/retained.copper fsck
fsck shows a retained CopperList ID range ending at 299999, with the earlier
IDs missing after rollover. Those reported holes are the intentionally reclaimed
history. The start time can also be unknown once its lifecycle event is gone.
The capacity is eight MiB across the logical log; slabs are its backing files. Rollover reclaims the oldest closed data sections as more space is needed. Application metadata and any payload catalog remain available. Those static sections also consume capacity, so leave room for them and the active streams; one MiB sections fit the budget used here.
Run indices follow the first retained section of each run. After rollover,
list runs again before choosing --run: old runs can disappear, and start times
can become unknown when lifecycle events have rotated out. The reader still
knows the mission of each surviving data section. Read order follows the log’s
section links, rather than the numerical order of slab filenames.
Rollover can discard earlier samples and keyframes, so archive a recording you need for full-history analysis or replay before they are reclaimed. If an open section prevents reclamation, the logger returns a space error; it does not silently overwrite a write in progress.
Continue with Self-Describing Logs to share these recordings with readers who do not have your application’s payload types.
Self-Describing Logs
v1.3
Suppose you receive a robot’s .copper recording and want to plot wheel speed.
With an application-specific logreader, you need the Rust payload types used by
that application. A self-describing log carries a catalog that tells an
offline reader how to decode those payloads, including field names and storage
units. You can inspect the recording with Copper’s standalone tools and export
the samples without building the robot’s logreader.
Use this feature when you share recordings with another team, keep logs for later analysis, or collect recordings from several applications. This feature is experimental; use the checkout named in the edition notice for the examples below.
1. Try a complete recording
From the root of a copper-rs checkout, enter the wheel-sensor example:
cd examples/cu_self_describing_logs
just
This records ten cycles in logs/wheel.copper, displays the embedded catalog,
and checks the recording with fsck --deep. The graph is a wheel source, a
filter, and a sink. Each cycle captures the source and filter payloads; the sink
contributes its execution metadata.
Look at the catalog’s WheelSample fields. Alongside encoder ticks it describes
distance, speed, a timestamp, acceleration, temperatures, and sensor state. The
reader obtains those names and types from the recording itself.
Export the captured cycles from the same directory:
just extract --export-format jsonl > samples.jsonl
Each JSONL line is one CopperList, with its id and ordered msgs. A message
contains its payload, timestamps, and capture information. Check that
samples.jsonl has ten lines, with source and filtered wheel samples in each
cycle. Exporting Data continues with catalog dumps, CSV,
and payload-size diagnostics using this same recording.
2. Enable it in your application
The project templates expose an application feature. For an existing app, add
this forwarding entry to its Cargo.toml:
[features]
self-describing-logs = ["cu29/self-describing-logs"]
Keep the usual build script calling cu29_build::setup() and your ordinary
builder:
let app = App::builder()
.with_log_path("logs/app.copper", Some(32 * 1024 * 1024))?
.build()?;
Run your application’s recording binary with the feature enabled:
cargo run --features self-describing-logs
Copper builds and saves the catalog during application construction, before initializing resources. It describes the output types declared in your RON graph, including types from dependency crates. One catalog covers every compiled mission and records each mission’s output-slot order.
Payload capture still follows your logging configuration. Keep
enable_task_logging: true and enable logging on the outputs you want to
inspect. A catalog describes a type; a task with logging: (enabled: false)
does not contribute its payload bytes to the recording.
3. Describe your payloads
Ordinary encoding derives supply the descriptions when the feature is enabled. For example, a wheel payload can use Copper’s units directly:
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use cu29::units::si::f32::{Length, Velocity};
#[derive(Clone, Debug, Default, Serialize, Deserialize, Encode, Decode, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct WheelSample {
pub ticks: u32,
pub distance: Length,
pub speed: Velocity,
pub timestamp: CuTime,
}
The catalog follows nested fields and enum variants automatically. A reusable
payload crate can add #[bincode(describe)] to provide descriptions even when
the recording feature is disabled.
Units tell you how to interpret the stored value. A length created from
centimetres is stored in metres, velocity uses m·s⁻¹, and CuTime stores
nanoseconds. Use the catalog’s storage unit when plotting or converting exported
numbers. The unit used to construct a Rust quantity does not change its storage
unit.
If you implement Encode by hand, also implement ValueDecode with a recipe
that matches the bytes your encoder writes. For an Orientation wrapper whose
encoder writes exactly an array of four f32 values:
impl cu29::bincode::ValueDecode for Orientation {
const DECODE: &'static cu29::bincode::ValueDecodeSpec =
<[f32; 4] as cu29::bincode::ValueDecode>::DECODE;
}
Check that the recipe decodes the complete encoded value, including consecutive values. A custom logging codec also needs a recipe for its recorded representation. If startup rejects a task/type/codec combination, check that encoder’s description.
4. Read your own recording
From the copper-rs repository root, pass the recording’s base path to the
standalone reader. Replace logs/app.copper below with your recording’s path:
just logextract logs/app.copper catalog
just logextract logs/app.copper extract-copperlists --export-format jsonl > samples.jsonl
just logextract logs/app.copper fsck --deep
just logextract builds the standalone reader with catalog support. Keep the
complete slab family with the recording. Structured text reconstruction still
uses the producing build’s cu29_log_index.
For recordings containing several runs and for detailed validation reports, continue to Exporting Data. For dictionary-based analysis, see Python Support.
Recording on an embedded target
The producer feature supports no_std. Schema serialization and compression
finish at startup using bounded working memory, without allocating a complete
catalog buffer. The recording loop keeps its ordinary native encoding pass.
Your logger backend still determines its own storage allocation requirements.
The startup traversal supports up to 256 distinct reachable native types,
including nested field types. A graph exceeding that limit returns a startup
error. Host-side catalog readers enable cu29/decode-catalog and use std to
allocate decoded value trees; cu29-export/self-describing-logs selects that
reader feature automatically.
Append reuses a matching catalog, and rollover retains it so surviving samples remain decodable. See recording retention for the storage budget and the history that rollover can reclaim.
The wheel example provides the complete application. The format reference describes the encoding for readers that implement their own tooling.
Exporting Data to Other Formats
In the logging chapter, we learned how Copper records everything
into .copper files and how the log reader can extract text logs and CopperList data as
JSON. But JSON is just the beginning. Real-world workflows often need data in other
formats – CSV for spreadsheets and data science tools, MCAP for visualization in
Foxglove, or statistics JSON for offline performance analysis.
The log reader CLI you already have supports all of these. In this chapter, we’ll explore
every export format and diagnostic tool available in cu29-export.
Choose a reader
For the wheel recording from the previous chapter, use the standalone
just logextract reader or the example’s just recipes. For your first
project’s log, use the matching src/logreader.rs binary from
Logging and Replaying Data.
The following sections state the starting directory for each command.
Inspecting a self-describing recording
v1.3 Continue from the ten-cycle wheel recording in Self-Describing Logs. That chapter covers recording, payload descriptions, storage units, and your first JSONL export. Here we use the existing log to export the catalog and inspect storage costs.
Export the catalog
From the copper-rs repository root, enter the example directory you used earlier:
cd examples/cu_self_describing_logs
just catalog --export-format ron > catalog.ron
just catalog --export-format json > catalog.json
These files include the complete field descriptions, mission output-slot maps, and recorded configuration. Use them when another tool needs the descriptions.
For your own log, run just logextract <base-path> catalog --export-format json
from the repository root, substituting its base path. For multiple runs, see
Selecting a recorded run.
Interpret missing payloads
When inspecting JSONL exports, use capture status to distinguish an absent payload from one excluded by logging. Compact recordings preserve that information; flat recordings can lose the original presence information when capture is suppressed. Each exported cycle retains its slot identity, payload, time of validity, and metadata.
Use an existing application logreader
App logreaders use their native typed decoder by default. If the logreader’s
cu29-export dependency enables self-describing-logs, run this from the
application directory to select the catalog:
logreader logs/robot.copper extract-copperlists --decoder catalog
Structured text still requires the producing build’s string index. For direct Python access to the catalog and decoded dictionaries, follow Catalog-Based Offline Analysis.
Selecting a recorded run
v1.3 A unified log can contain several runs, for example when you append a restart or construct another mission. Each run has its own CopperList sequence and clock. Use the application’s built logreader to find the run you want:
logreader logs/robot.copper list-runs
logreader logs/robot.copper --run 1 fsck --dump-runtime-lifecycle
logreader logs/robot.copper --run 1 extract-copperlists
Run these commands from the application directory. Here logreader means its
built logreader executable. With the standalone reader, use
just logextract <base-path> list-runs from the copper-rs repository root, then
put --run N before the subcommand. Indices are zero-based and follow the
first retained section of each run. They differ
from instance_id, which can repeat across process restarts. list-runs shows
the mission, application, start time when available, and whether shutdown
completed. After rollover removes startup events, the retained sections still
identify their run and mission; the missing timestamps remain unknown.
A single-run log is selected automatically. Use --run for a multi-run log,
including when extracting text, computing statistics, or exporting MCAP:
logreader logs/robot.copper --run 1 extract-text-log target/debug/cu29_log_index
logreader logs/robot.copper --run 1 log-stats --output run-1.json
logreader logs/robot.copper --run 1 export-mcap --output run-1.mcap
The selected run supplies the recorded configuration and mission. Statistics
use those by default; --config and --mission override them. Use an
application logreader and string index from the producing build.
CSV export
We already saw how extract-copperlists outputs JSON by default. But if you’re working
with spreadsheets, pandas, or any data analysis tool, CSV is often more convenient.
In the wheel example directory, export the recording you already made:
just extract --export-format csv > samples.csv
For your first project’s typed logreader, return to that project directory.
The --export-format flag (or -e for short) switches the output format:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper extract-copperlists --export-format csv
The output looks like this:
id, src_time, src_tov, src,, t_0_time, t_0_tov, t_0,, sink_time, sink_tov, sink,
0, [672.083 µs – 734.327 µs], None, {"value":42}, [735.631 µs – 750.445 µs], None, {"value":43}
1, [1.000 s – 1.000 s], None, {"value":42}, [1.000 s – 1.000 s], None, {"value":43}
2, [2.000 s – 2.000 s], None, {"value":42}, [2.000 s – 2.000 s], None, {"value":43}
...
Each row is one CopperList (one execution cycle). The columns are:
id– The CopperList sequence number.<task>_time– The process time range (start-end in nanoseconds) for that task’sprocess()call.<task>_tov– The time of validity, if the task set one.<task>– The payload as inline JSON.
The column headers are generated from your task IDs, so they’ll match whatever you named
your tasks in copperconfig.ron.
Tip: You can redirect the output to a file and open it in any spreadsheet application or load it into a pandas DataFrame:
cargo run --features logreader --bin my-project-logreader -- \ logs/my-project.copper extract-copperlists -e csv > data.csvimport pandas as pd df = pd.read_csv("data.csv")
Checking log integrity with fsck
Before doing any analysis on a log file, it’s good practice to verify its integrity. The
fsck subcommand reads the entire log and checks that all sections are well-formed:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper fsck
The log checked out OK.
=== Statistics ===
Total time -> 1.999 s
Total used size -> 328 bytes
Logging rate -> 0.00 MiB/s (effective)
# of CL -> 2
CL rate -> 1.00 Hz
CL total size -> 129 bytes
# of Keyframes -> 1
KF rate -> 0.50 Hz
KF total size -> 7 bytes
# of SL entries -> 13
SL total size -> 192 bytes
Even without the verbose flag, fsck gives you a useful summary:
- Total time – How long the recording lasted.
- CL rate – How many CopperLists per second were logged (should match your
rate_target_hz). - Keyframes – Periodic snapshots of frozen task state, used for seeking during replay.
- SL entries – The number of structured log lines (
debug!(),info!(), etc.).
Verbose mode
Add -v for section-by-section details, or -vv for maximum verbosity:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper fsck -v
This prints every section header as it’s read, along with the CopperList ID ranges and time ranges within each section. It’s invaluable for debugging corrupted logs – you can see exactly where the corruption starts.
Deep validation and payload storage costs
v1.3
For the wheel recording, return to examples/cu_self_describing_logs in the
copper-rs checkout and run:
just fsck
The example’s fsck recipe enables deep validation. Expect ten decoded
CopperLists and twenty captured payloads: one source and one filtered sample
per cycle. Invalid or truncated data returns an error with its location.
Keyframe envelopes are checked too; replay still needs the application’s
decoder for frozen task state.
The report includes captured-payload counts and bytes. Its table groups payloads by task and message type, sorted by total bytes, with each group’s share, mean size, and size range. Use it to find the outputs consuming most of your storage before reducing capture. These sizes count encoded payload bytes, excluding CopperList metadata.
The catalog’s compressed size includes its catalog headers, continuation framing, and integrity footer; the decompressed size is the encoded catalog body. This shows startup description overhead separately from the samples accumulated during the run.
Log statistics
The log-stats subcommand computes per-edge statistics from the log and writes them as a
JSON file. This is useful for offline analysis of your task graph’s performance:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper log-stats --output stats.json --config copperconfig.ron
This produces a stats.json file with detailed statistics for every edge in your task
graph:
{
"schema_version": 1,
"config_signature": "fnv1a64:166f6eae65f80d15",
"mission": null,
"edges": [
{
"src": "src",
"dst": "sink",
"msg": "crate::tasks::MyPayload",
"samples": 3,
"none_samples": 0,
"valid_time_samples": 3,
"total_raw_bytes": 12,
"avg_raw_bytes": 4.0,
"rate_hz": 1.0004343150476716,
"throughput_bytes_per_sec": 6.002605890286029
},
{
"src": "src",
"dst": "t-0",
"msg": "crate::tasks::MyPayload",
"samples": 3,
"none_samples": 0,
"valid_time_samples": 3,
"total_raw_bytes": 12,
"avg_raw_bytes": 4.0,
"rate_hz": 1.0004343150476716,
"throughput_bytes_per_sec": 6.002605890286029
},
{
"src": "t-0",
"dst": "sink",
"msg": "crate::tasks::MyPayload",
"samples": 3,
"none_samples": 0,
"valid_time_samples": 3,
"total_raw_bytes": 12,
"avg_raw_bytes": 4.0,
"rate_hz": 1.0004283749265024,
"throughput_bytes_per_sec": 6.002570249559014
}
],
"perf": {
"samples": 3,
"valid_time_samples": 3,
"end_to_end": {
"min_ns": 74413,
"max_ns": 178994,
"mean_ns": 114416.0,
"stddev_ns": 46095.0
},
"jitter": {
"min_ns": 15429,
"max_ns": 104581,
"mean_ns": 60005.0,
"stddev_ns": 44576.0
}
}
}
For each edge in the graph, you get:
samples– Total number of CopperLists processed.none_samples– How many times the payload wasNone(the task produced no output for that cycle).rate_hz– The measured message rate on this edge.avg_raw_bytes– Average payload size in bytes.throughput_bytes_per_sec– Sustained data throughput on this edge.
The perf section gives you pipeline-level timing:
end_to_end– Latency from the first task’sprocess()start to the last task’sprocess()end, across the full pipeline.jitter– Variation in that end-to-end latency between consecutive cycles.
Using with missions
If your project uses missions, pass the --mission flag to select which
mission’s graph to use for the edge mapping:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper log-stats --config copperconfig.ron --mission normal
MCAP export for Foxglove
MCAP is an open-source container format designed for multimodal robotics data. It’s the native format of Foxglove, a powerful web-based visualization tool for robotics. Exporting your Copper logs to MCAP lets you visualize your pipeline’s data in Foxglove’s timeline, plot panels, and 3D views.
Enabling the MCAP feature
MCAP export requires an additional feature flag. Add mcap to your logreader’s feature
dependencies in Cargo.toml:
[features]
logreader = ["dep:cu29-export"]
logreader-mcap = ["logreader", "cu29-export/mcap"]
And make sure your existing cu29-export dependency enables the "mcap" feature.
Note: If your project already depends on
cu29-exportviagit, a localpath, or crates.io, keep that same source and Copper release and just add the"mcap"feature.
Automatic payload schemas
v1.2
gen_cumsgs! records the concrete payload type for every output slot. The MCAP exporter
uses that generated metadata to build each JSON Schema automatically, including when one
task has multiple output types. Your logreader does not need a task-ID-to-schema table:
pub mod tasks;
use cu29::prelude::*;
use cu29_export::run_cli;
gen_cumsgs!("copperconfig.ron");
fn main() {
run_cli::<CuStampedDataSet>().expect("Failed to run the export CLI");
}
For ordinary derived serialization, reflection is enough. If a reusable payload has a
custom Serialize implementation whose wire shape differs from its reflected fields,
the payload crate implements SerializedPayloadSchema beside that serializer and adds
SerializedPayloadSchema to its #[reflect(...)] attribute. That one payload-owned
definition is then discovered by every logreader automatically. Application authors do
not repeat the schema or maintain per-task matches.
Exporting to MCAP
Once the feature is enabled, a new export-mcap subcommand becomes available:
cargo run --features logreader-mcap --bin my-project-logreader -- \
logs/my-project.copper export-mcap --output data.mcap
Exporting copperlists to MCAP format: data.mcap
MCAP Export: 5 CopperLists → 10 messages, 2 channels, 2 schemas
The export creates one MCAP channel per task output (named /<task-id>), and one
schema per message type. Schemas are generated automatically at compile time using
serde-reflection – the exporter introspects your Rust types and produces JSON Schema
definitions that Foxglove can use to understand the data structure.
Each MCAP message contains:
payload– Your task’s output, serialized as JSON.tov– The time of validity.process_time– Start and end timestamps of the task’sprocess()call.status_txt– Any status text the task set.
Inspecting MCAP files
The mcap-info subcommand lets you inspect an MCAP file without opening Foxglove:
cargo run --features logreader-mcap --bin my-project-logreader -- \
logs/my-project.copper mcap-info data.mcap
=== MCAP File Info ===
File: data.mcap
Size: 2617 bytes (0.00 MB)
=== Statistics ===
Total messages: 9
Channels: 3
Schemas: 3
=== Channels ===
/sink [json] (schema: copper.sink): 3 messages
/src [json] (schema: copper.src): 3 messages
/t_0 [json] (schema: copper.t_0): 3 messages
=== Schemas ===
Schema: copper.sink (encoding: jsonschema)
Schema: copper.src (encoding: jsonschema)
Schema: copper.t_0 (encoding: jsonschema)
Add --schemas to print the full JSON Schema for each channel, or -n 3 to show the
first 3 sample messages per channel:
cargo run --features logreader-mcap --bin my-project-logreader -- \
logs/my-project.copper mcap-info data.mcap --schemas -n 1
Opening in Foxglove
Once you have an .mcap file, open it in Foxglove:
- Go to app.foxglove.dev (or use the desktop app).
- Click “Open local file” and select your
.mcapfile. - Your channels (
/src,/t-0, etc.) appear in the sidebar. - Add a Raw Messages panel to see the JSON payloads, or a Plot panel to graph numeric fields over time.
Because the MCAP file includes JSON Schema definitions, Foxglove knows the structure of your messages and can offer autocomplete for field names in plots.
Foxglove has plenty of tutorials on how to set up the Panels. Here is a basic UI showing our data with plots and raw messages:
Summary of subcommands
Here’s a quick reference for all the log reader subcommands:
| Subcommand | What it does | Feature needed |
|---|---|---|
extract-text-log <index> | Reconstructs human-readable text logs | – |
extract-copperlists | Dumps CopperList data (JSON or CSV) | – |
extract-copperlists -e csv | Dumps CopperList data as CSV | – |
fsck | Checks log integrity and prints statistics | – |
log-stats | Computes per-edge statistics as JSON | – |
export-mcap --output f.mcap | Exports to MCAP format for Foxglove | mcap |
mcap-info <file> | Inspects an MCAP file | mcap |
Streaming Logs and Live Telemetry
v1.2
LogStream sends your robot’s execution log to a ground station while the robot
runs. The ground station records a native .copper archive, displays typed task
outputs and structured logs, and can run selected tasks locally to reconstruct
outputs you chose not to transmit.
This chapter takes you from the runnable UDP demo to your own sender and ground
application. Use matching application code, task graph, Copper revision, and
encoding features on both machines. Keep the producing build’s cu29_log_index
with its recordings so you can render structured logs later.
Robot tasks ──→ onboard .copper log
│
└──→ LogStream worker ──→ UDP or framed serial ──→ received .copper archive
│
live twin
│
your UI or analysis
Forward error correction (FEC) adds repair packets that can recover lost data within configured bounds. Longer outages leave explicit gaps. A keyframe saves task state; a recovery point binds that keyframe to the stream manifest and a CopperList boundary so the receiver can resume safely. Starting late does not retrieve expired history.
For custom transports, codecs, and storage backends, see the logging and logstream extensibility map.
1. Try the UDP demo
Use the source checkout named in the edition notice. The demo requires the checkout’s Rust toolchain, Just, and Python 3. From that checkout:
cd examples/cu_logstream_demo
just
This starts separate sender and receiver processes, compares the received archive
against the onboard recording, checks its integrity, and runs offline replay.
Each automated run prints a fresh output directory under the example’s logs/.
For the interactive view, run these in separate terminals, both in the example directory. Start telemetry first:
# Terminal 1: ground station
just telemetry
# Terminal 2: robot, with Copper's console monitor
just sender
The graph is encoders → kinematics. Joint angles cross the link; the ground
station runs the same kinematics task to calculate the arm pose. The robot arm
is labeled reconstructed locally; payload not transmitted. The robot log pane
shows simulated encoder temperature and supply-voltage diagnostics sent as
structured logs.
| Control | Action |
|---|---|
1, 2, Tab | Select Live, Health, or cycle between them. |
Space | Pause both display readers; recording continues. |
Arrow keys or hjkl | Scroll Health details. |
q, Escape, Ctrl-C | Close telemetry and finalize its archive. |
The sender runs for about a minute. Telemetry stays open after it finishes. Pause the view and resume to see display misses without interrupting recording.
The manual demo commands replace recordings at their chosen base paths. Supply new paths to retain previous runs:
just telemetry 0.0.0.0:7447 logs/ground-session-2.copper
# On the robot; replace this example address with the ground station's address:
just sender 192.168.1.20:7447 logs/robot-session-2.copper
Use just receiver 0.0.0.0:7447 logs/received-session-2.copper for headless
recording. This demo receiver exits after one second without a datagram, or fails
if no traffic arrives within 15 seconds; start the sender promptly. These timeout
and overwrite behaviors belong to the demo, not the generated twin API.
2. Configure your robot’s sender
Enable the logstream feature on your existing cu29 dependency and add
cu29-logstream-udp from the same Copper revision. Ground code that directly uses
LogStream types also needs cu29-logstream with its std feature. Follow the
demo Cargo manifest
for a workspace example; keep your application’s existing dependency source.
Add the following top-level blocks to your copperconfig.ron, alongside your
existing tasks and cnx. These are the demo’s working UDP settings; change
remote_addr for your ground station:
logging: (
enable_task_logging: true,
enable_keyframe_logging: true,
keyframe_interval: 16,
slab_size_mib: 16,
section_size_mib: 1,
copperlist_count: 8,
),
resources: [
(
id: "network",
provider: "cu29_logstream_udp::CuUdpLogStreamResources",
config: {
"bind_addr": "127.0.0.1:0",
"remote_addr": "127.0.0.1:7447",
"send_buffer_bytes": 262144,
"ttl": 1,
},
),
],
log_streaming: (
destinations: [
(
id: "ground",
transport: (
type: "cu29_logstream_udp::CuUdpLogStreamTx",
resource: "network.tx",
),
link: (
mtu_bytes: 1200,
bitrate_bps: 2000000,
memory_budget_kib: 576,
max_latency_ms: 250,
burst_packets: 8,
),
fec: (
continuous: (
field: Gf256,
window_symbols: 64,
repair_every_source_symbols: 4,
repair_density: Full,
),
objects: (
max_object_bytes: 65536,
repair_symbols_per_block: 2,
),
),
recovery_interval: 16,
max_record_bytes: 4096,
),
],
),
Keep your usual application builder, local log path, clock, and execution loop. The generated runtime wires the configured transport and sender workers. The demo’s mock clock, impairment wrapper, and fixed iteration loop are test harness code and are not needed in your application.
network.tx is a resource binding, not a task-graph edge. The
graph renderer shows it in
the resource table as system: logstream (ground).
| Setting | How to choose it |
|---|---|
mtu_bytes | Maximum Copper packet size accepted by your carrier; 1200 is the demo profile. |
bitrate_bps | Budget for all Copper packet bytes, including repairs, recovery, and structured logs. Leave room for UDP/IP or serial overhead. |
memory_budget_kib | Sender buffer budget. The demo needs 576 KiB; larger records need more storage. This is not a total process-memory limit. |
max_latency_ms | Expiry deadline for ordinary queued data, measured from encoded-record admission. |
burst_packets | Allowed transmission burst; reduce it for a slow carrier. |
window_symbols | Continuous FEC recovery window; the generated sender supports at most 64 symbols, and receiver limits also apply. |
repair_every_source_symbols | One continuous repair after this many source symbols. Lower means more repair traffic. |
max_object_bytes | Finite-object bound, including recovery data; it must fit the sender’s supported RaptorQ geometry. |
max_record_bytes | Maximum complete CopperList record, including its record header. |
recovery_interval | Recovery boundaries in CopperLists, not milliseconds. Must be a nonzero multiple of logging.keyframe_interval. |
For example, keyframes every 20 lists and recovery points every 100 lists select recovery boundaries at IDs 0, 100, 200, and so on. Streaming needs a nonzero keyframe interval even if you disable local keyframe recording. Keeping local keyframes, as above, also makes the onboard recording useful for stateful replay.
Start with the demo profile and captured task outputs. Measure actual transmission and drop counters before reducing bandwidth or adding reconstruction.
3. Start a ground-side twin
Build a simulation application from the same graph and task types. The generated
twin builder owns receiving, archive writing, recovery, and replay; you supply
a packet receiver and consume the resulting frames. This integration sketch uses
Ground as your generated application type:
use cu29::prelude::*;
use cu29_logstream_udp::CuUdpLogStreamConfig;
use std::time::Duration;
#[copper_runtime(config = "copperconfig.ron", sim_mode = true)]
struct Ground {}
// Inside your fallible ground application entrypoint:
let (_, rx) = CuUdpLogStreamConfig::new("0.0.0.0:7447".parse()?).open()?;
let (mut twin, mut frames) = Ground::twin(rx)
.with_log_path("logs/received-session-1.copper")
.spawn()?;
// Repeat this on your UI or analysis thread while the application is running:
frames.wait_timeout(Duration::from_millis(50));
let status = frames.status();
while let Some(update) = frames.try_read() {
render(&update.frame.copperlist); // Your presentation function.
account_for_display_misses(update.missed);
}
// At application shutdown:
let final_status = twin.stop()?;
Use the generated output accessors for your graph. In the demo these include
get_encoders_output() and get_kinematics_output(). Their payloads remain typed
Rust values; your UI does not need to decode the stream again.
Choose a fresh archive path for each sender session. The builder rejects an
existing base file or first slab. Keep the twin handle alive while receiving;
stop() finalizes recording and reports worker errors and final counters.
Dropping the handle also stops and joins its workers.
For recording alone, insert .archive_only() before .spawn(). This disables
task reconstruction and frame production while preserving archival and status.
The default twin accepts the demo’s 1200-byte packet, 64-symbol window, 4 KiB record, and 64 KiB recovery-object profile. Increasing sender limits does not increase receiver limits automatically. Set the builder bounds below for larger records and recovery objects. For other FEC geometry or custom routing, use the custom receiver API with explicit bounds.
Size storage and receiver buffers
v1.3
Size the ground station’s receiver for the largest serialized capture and for
pauses in replay or UI consumption. This experimental builder API requires
cu29/logstream. In your ground station’s existing
Ground::twin(rx) setup, use named constants to make the storage and buffering
choices explicit before starting the workers:
use cu29::logstream::{FiniteObjectLimits, SessionRouterLimits};
use std::num::NonZeroUsize;
// Budget for one complete serialized CopperList, including captured messages
// and metadata, and for one manifest, keyframe, or structured log record.
const MAX_RECORD_BYTES: usize = 1024 * 1024;
const MAX_OBJECT_BYTES: u64 = 64 * 1024;
// Fit the larger bound plus a 512-byte section header and 32-byte continuity
// envelope; round up to a convenient multiple of 512 bytes.
const SECTION_BYTES: usize = 2 * 1024 * 1024;
// Hold many sections per backing file: this slab is 16 times the section size.
const SLAB_BYTES: usize = 32 * 1024 * 1024;
// Buffer short replay slowdowns, keeping the retained payload count bounded.
const REPLAY_CAPACITY: NonZeroUsize = NonZeroUsize::new(32).unwrap();
// Independent reader buffers: completed CopperLists and structured log entries.
const FRAME_CAPACITY: NonZeroUsize = NonZeroUsize::new(64).unwrap();
const LOG_CAPACITY: NonZeroUsize = NonZeroUsize::new(64).unwrap();
let limits = SessionRouterLimits {
max_record_bytes: MAX_RECORD_BYTES,
finite_objects: FiniteObjectLimits {
max_object_bytes: MAX_OBJECT_BYTES,
..SessionRouterLimits::default().finite_objects
},
..Default::default()
};
let (mut twin, mut frames) = Ground::twin(rx)
.with_log_path("logs/received.copper")
.with_receiver_limits(limits)
.with_section_size(SECTION_BYTES)
.with_slab_size(SLAB_BYTES)
.with_replay_capacity(REPLAY_CAPACITY)
.with_frame_capacity(FRAME_CAPACITY)
.with_log_capacity(LOG_CAPACITY)
.spawn()?;
Choose record bounds for the largest complete serialized CopperList, including all captured outputs and metadata. The sender’s advertised record and object bounds must fit the receiver’s bounds. Finite objects carry manifests, keyframes (task-state snapshots), and structured log entries. A section groups archive records and must fit the larger record/object bound plus a 512-byte header and 32 bytes of continuity envelope allowance. Slab and section sizes must be multiples of 512 bytes; each section must fit in a slab. Prefer slabs roughly 10–100 times larger than sections to reduce file turnover. Larger slabs reserve more disk space at a time; the archive grows by adding slabs as needed.
Defaults are 16 MiB slabs, 128 KiB sections, 4 KiB records, 64 KiB finite objects,
32 replay events and pending captures per buffer, and 64 unread reconstructed
CopperLists and structured log entries per reader buffer. Increase record/object
limits together with section size. spawn() validates local settings before
creating directories or starting workers. Sender requirements are checked when
its manifest arrives. FEC geometry supports the 1200-byte-MTU, 64-symbol profile.
Receiver routing and object concurrency limits are in SessionRouterLimits;
one twin accepts exactly one sender session.
Replay buffering absorbs temporary reconstruction slowdowns. Overflow requires
recovery from a matching keyframe while recording continues. Frame buffering
holds completed CopperLists for UI or analysis; a frame here is one iteration’s
captured inputs and reconstructed outputs. Structured log buffering holds the
robot’s logging calls, such as info! and debug!, for twin.take_log_reader().
Both reader buffers replace the oldest unread item when full; the archive keeps
the received data. Larger buffers retain more payloads and entries in memory.
Choose counts together with byte limits for the ground station’s memory budget,
and size the sender’s streaming memory and link bandwidth for its payloads.
Keep display loss separate from recording loss
The frame reader retains 64 display frames by default, overwriting the oldest
unread frame when full. Set .with_frame_capacity(128.try_into()?) on the builder
to change display retention. update.missed counts overwritten display frames;
frames.status() reads an independent status slot without consuming frames.
A slow, paused, or dropped reader never makes recording wait. A borrowed frame remains valid while publication continues, but payload allocations and your own chart history consume additional memory. The receiver and UI still share a process failure boundary.
For async presentation, use frames.ready().await instead of polling. Consume
unread frames and status changes before waiting again. Custom integrations can
use telemetry_channel and a scheduling waker; see the
receiver reference.
4. Reduce traffic with task reconstruction
A live twin can execute ordinary synchronous tasks using captured inputs, recorded clock values, and restored task state. Mark a suitable task in the shared graph:
(
id: "kinematics",
type: "cu_logstream_demo::tasks::Kinematics",
streaming: (replay: reconstruct),
),
The robot still runs the task and records its full output locally. Its streamed capture omits the selected output payload, including in repairs and repeated recovery boundaries. The twin reconstructs that output before downstream tasks consume it, restoring sender metadata. Reconstructed values appear in live frames and offline replay output; the received archive keeps the captured bytes.
Omitting streaming keeps the default replay: capture. Sources and bridge
receives stay captured. Reconstruction requires a logged, ordinary synchronous
task; background and anytime tasks are unsupported. A graph using reconstruction
must use the lossless native compressed codec and full handle capture: custom
codecs, flat CopperList encoding, and selective handle policies are rejected.
Choose tasks whose outputs depend on recorded inputs, Copper’s clock, and state
covered by Freezable. Avoid external side effects and unrecorded inputs such as
host wall time or hardware reads. Matching code is necessary, but does not by
itself make a nondeterministic algorithm reproducible.
Place the capture boundary where the required inputs exist
Every incoming connection of every reconstructed task must come from a node with logging enabled. A producer can itself reconstruct, so chains of reconstructed tasks are valid. Copper checks all incoming branches and each mission graph at compile time, including when transports are injected in code.
For a camera pipeline, the following task settings let you omit images while sending detections and reconstructing tracking and planning. This is a fragment to apply to your own task types and connections:
tasks: [
(id: "camera", type: "tasks::Camera", logging: (enabled: false)),
(id: "detect", type: "tasks::Detect", streaming: (replay: capture)),
(id: "track", type: "tasks::Track", streaming: (replay: reconstruct)),
(id: "plan", type: "tasks::Plan", streaming: (replay: reconstruct)),
],
With camera → detect → track → plan, detect supplies captured detections;
track and plan have the inputs they need. Keep logging enabled on all three.
Changing detect to reconstruct while camera logging is disabled fails at
compile time: the diagnostic names detect, camera, and the image message
type. Either enable camera logging or keep detector output captured. Similarly,
an additional unlogged camera input directly into track makes that task invalid.
Check reconstruction during development
Enable cu29/logstream-verify on both ends to compare reconstructed outputs
against sender digests. In the demo, use separate terminals:
just telemetry-verify
# Other terminal:
just sender-verify
Matching frames are labeled Verified. Normal operation labels them Reconstructed and does not send the optional verification digest. A mismatch suppresses reconstructed frames until the next matching recovery point; native recording continues. Verification is opt-in even in a Rust debug build, and its live digests do not change the native archive format.
5. Read structured robot logs
Configured streaming destinations automatically forward structured entries
alongside local logging. Message templates and parameter names travel as interned
IDs; values retain their types. String values are transmitted if your application
explicitly logs strings. Ground rendering needs the producing application’s
cu29_log_index, not an unrelated ground build’s string table.
The twin exposes an independent log reader once:
let mut logs = twin.take_log_reader().expect("fresh twin log reader");
// In your presentation loop, with the producing build's string table loaded:
while let Some(update) = logs.try_read() {
let line = rebuild_logline(&strings, &update.frame.entry)?;
show_robot_log(line, update.missed);
}
This ring retains 64 entries and publishes only after archival succeeds.
CuTwinStatus::structured_logs counts archived entries. Pausing or dropping this
reader does not stop task frames or recording.
The demo looks for cu29_log_index beside its executable. To select the producing
build’s index explicitly, run the already-built binary from the example directory:
../../target/debug/cu-logstream-demo telemetry \
--listen 0.0.0.0:7447 --log-base logs/telemetry-session-3.copper \
--log-index /path/to/producing-build/cu29_log_index
Structured transmission uses bounded buffers and the same destination bitrate
budget. Oversized entries or exhausted buffers increment inbox_drops; local
records remain intact. Link loss can also leave missing remote log entries.
Keep the onboard recording when you need the complete local history.
6. Add feedback and inspect link health
One-way streaming works without a return path. To send receiver health reports
back and optionally adapt continuous FEC, configure both endpoints explicitly.
In the demo, run just telemetry-two-way and just sender-two-way in separate
terminals. Telemetry listens on port 7447 and reports to the robot on port 7448.
Use just telemetry-two-way lossy to exercise sustained source loss.
In your own sender’s destination, add the following block. Also give network
a fixed bind_addr, such as 0.0.0.0:7448, so the ground can address it:
feedback: (
transport: (
type: "cu29_logstream_udp::CuUdpLogStreamRx",
resource: "network.rx",
),
report_interval_ms: 100,
timeout_ms: 1000,
adaptation: (
min_repair_every_source_symbols: 1,
max_repair_every_source_symbols: 16,
),
),
On the ground, create a TX endpoint targeting that robot address and pass it to the twin builder. Replace the example robot address below:
let mut socket = CuUdpLogStreamConfig::new("0.0.0.0:7447".parse()?);
socket.remote_addr = Some("192.168.1.10:7448".parse()?);
let (tx, rx) = socket.open()?;
let (mut twin, mut frames) = Ground::twin(rx)
.with_feedback(tx.expect("remote_addr creates a TX endpoint"))
.with_log_path("logs/received-two-way-session-1.copper")
.spawn()?;
The sender manifest advertises feedback support and cadence. Sharing a UDP socket does not enable feedback automatically. Give each logical receive endpoint one owner; cloned UDP receivers compete for packets rather than broadcasting them.
Omit adaptation for reports only. With adaptation enabled, the baseline
repair_every_source_symbols must lie within the configured bounds. Loss raises
protection; sustained healthier reports allow less redundancy. After timeout,
feedback becomes stale and the repair interval gradually returns to baseline.
Bitrate, burst, memory, latency, FEC window/field/density, and object FEC stay fixed.
Feedback failure never stops capture or autonomous recovery.
In the robot’s console monitor, open BW to see one Telemetry / TX panel
per destination. Arrow keys or hjkl scroll large panels.
| Reading | Interpretation |
|---|---|
| TX bytes/packets, budget, drops, recovery activity | Local submission and sender health; a successful UDP send does not confirm delivery. |
| One-way / Disabled | No feedback was configured. |
| Two-way / Waiting | Feedback is configured but no valid report has arrived. |
| Active | Receiver measurements and effective repair interval are available. |
Stale or failed; receiver values n/a | Receiver measurements are unavailable or out of date. Check the return path. |
| Source loss / FEC recovery | Finalized symbol outcomes; excludes the active coding window and unseen history or tails. |
Reports are advisory, not per-record acknowledgements. Custom monitors can read
worker snapshots through runtime.log_streams(); the presentation thread samples
them without adding monitoring callbacks to task execution.
7. Inspect recordings and exercise recovery
From the demo directory, point its tools at a received archive base:
just cl logs/received-session-1.copper
just fsck logs/received-session-1.copper
just resim logs/received-session-1.copper logs/replay-session-1.copper
just resim-debug logs/replay-session-1.copper logs/debug-session-1.copper
For your own app, use its matching logreader and replay binaries as described in
Logging and Replaying Data. Keep the whole slab family
(name.copper, name_0.copper, and subsequent slabs where present), and pass the
base path to the tools. Always use a different path for replay output.
Extract received structured logs with the producing build’s index:
../../target/debug/cu-logstream-demo-logreader \
logs/received-session-1.copper extract-text-log \
../../target/debug/cu29_log_index
The received archive retains sender payload bytes, timestamps, keyframes, and
stream continuity records. fsck reports missing ranges and verified recovery
boundaries. Replay cannot cross a gap without a matching keyframe; resuming later
does not fill the gap. A clean receiver shutdown does not establish that an
unobserved sender tail was complete.
Share and retain a received archive
v1.3
To read the archive with the standalone tools from the
self-describing log exercise,
enable cu29/self-describing-logs in the sender and generated twin. Build them
from the same application definitions and encoding features. The twin writes
the matching catalog at archive startup, including streamed payload recipes
when onboard task logging is disabled. Archive configuration capture is currently
unavailable; supply the application’s configuration explicitly for analysis
that needs it.
For bounded ground-station storage, add this option before .spawn() in the
builder from Step 3:
.with_log_rollover(256 * 1024 * 1024)
This bounds storage to 256 MiB. Capacity must exceed the configured section size, be a multiple of 512 bytes, and fit static metadata plus an archive section. The archive uses the same rollover behavior as onboard recording, keeping the catalog while reclaiming old data.
The sender’s repeated manifest identifies its run, runtime instance, and mission. Received sections retain that identity through rollover and late joins. Use the twin generated for the sender’s active mission; a new sender construction needs a fresh session and archive. Runtime lifecycle events remain in the onboard log.
Exercise stream recovery
Exercise the failure modes before deploying:
| Demo command | What it exercises |
|---|---|
just run loss | Recover a dropped CopperList with FEC. |
just run lossy | Sustained source-packet loss. |
just run outage | Resume after an interruption that leaves missing history. |
just run late | Join an already running sender. |
just run restart | Start a fresh receiver during a run. |
just run idle | Recover the initial boundary after captures have stopped. |
just check | Run all scenarios, including clean reception. |
just run-two-way all | Run the scenarios with feedback enabled. |
The sender repeats its manifest and latest complete recovery bundle on a 250 ms
local deadline, even without new captures. Repetition shares the bitrate budget,
so this is a scheduling deadline rather than a delivery guarantee. Pending
bundles finish before newer ones replace them. Pacing uses RobotClock and does
not require clock synchronization with the ground station.
Troubleshooting
| Symptom | Check or change |
|---|---|
| No packets at the ground | Verify numeric bind/remote addresses, ports, routing, and firewall access. Loopback addresses only reach the same machine. |
| Packets arrive but no frames | Inspect recording/replay status: the twin may be waiting for a valid manifest/recovery point, recovering after a gap, or configured as archive-only. |
| Compile error about an unlogged replay input | Enable the named producer’s logging or capture the named consumer’s output. Check every fan-in branch. |
| Startup rejects the stream/profile | Use matching application/schema/encoding features; fit the receiver’s record, object, symbol, and window bounds. |
| Archive already exists | Choose a fresh base path and keep each sender session separate. |
| Display misses rise while archive progress continues | Reduce UI work or increase display capacity. These misses do not mean packets were lost. |
| Sender drops or source gaps rise | Check encoded record sizes, latency expiry, link capacity, FEC overhead, and memory limits. More repair traffic also consumes the link budget. |
| Robot log text is wrong or cannot be rendered | Load the exact producing build’s string index. |
| Feedback stays Waiting or becomes Stale | Check the robot’s fixed bind port, ground TX destination, .with_feedback(...), and single receive ownership. |
| Reconstruction stalls while recording continues | Inspect replay queue pressure, source gaps, and verification failures; reconstruction resumes at a matching recovery point. |
For serial transport, receiver internals, and current framing details, continue to LogStream Transport and Receiver Reference.
LogStream Transport and Receiver Reference
v1.2 Start with Streaming Logs and Live Telemetry for the UDP walkthrough. This reference covers custom carriers, receiver ownership, and format details needed when sizing or extending an integration.
Serial and transparent radios
A raw serial byte stream needs packet framing before LogStream can decode it.
cu29-logstream-serial supplies SerialLogStreamTx<S, N> and
SerialLogStreamRx<S, N> over cu_serial::SerialIo.
SerialLogStreamTxResources<S, N> consumes a serial resource and exports tx;
the RX provider consumes serial and exports rx. Each provider owns its serial
resource; choose one direction per carrier with these adapters.
A UART can feed an HC-12 radio resource, whose serial output feeds the LogStream
framing resource. Define concrete application aliases for those generic
providers, then bind the final tx slot to your streaming destination. The
serial adapter
and HC-12 examples
provide the transport and radio details. Do not mix raw application bytes and
LogStream packets on the same stream without an explicit multiplexing protocol.
The serial adapter wraps each packet in 0x7e delimiters, escapes 0x7e and
0x7d, and appends an escaped four-byte big-endian CRC32C of the unescaped packet.
RX verifies and removes that checksum before handing the packet to LogStream.
Invalid or oversized frames are discarded; the next delimiter resynchronizes RX.
Both serial peers must use this framing.
The compile-time frame capacity N bounds packets to (N - 2) / 2 - 4 bytes,
allowing worst-case escaping. The default N = 514 therefore permits a 252-byte
packet, not the UDP demo’s 1200-byte MTU. Configure MTU within this bound, use a
small burst (one for slow links), and budget for checksum, escaping, and UART
start/stop bits as well as Copper traffic.
The scheduled sender drains pending serial writes while idle and during bounded
shutdown. If you drive an endpoint directly, including in no_std, keep calling
CuStreamTx::poll_pending() until it returns false. Each poll performs at most
one nonblocking UART write. WouldBlock means retry the packet after the current
frame drains; successful submission only means the adapter accepted it.
Custom receivers
Use the generated twin when its profile fits. For explicit receiver bounds or a
custom archive pipeline, construct SessionRouter and supply packets through
CuStreamRx. Its manifest describes decoder geometry and the application schema;
the receiver checks those requirements against its own allocation limits before
constructing decoders.
| API | Use |
|---|---|
NativeArchive<P> | Archive full captures using the application’s generated dataset type. |
CaptureArchive<P> | Preserve selective captures for live or offline reconstruction; used by the generated twin. |
cu29_export::stream_continuity_reader | Read archived manifests, gap ranges, and recovery boundaries programmatically. |
telemetry::telemetry_channel | Publish typed, already decoded values to an independent presentation reader. |
SessionRouter lends SessionEventRef values to callbacks. Encoded bytes remain
receiver-owned; if a callback fails, drain_events can retry delivery. Copy with
event.to_owned() only when you need a separately owned event. Make your callback
safe for retries; the generated twin reports archive failures instead of hiding
them behind an application retry loop.
ReceivedManifest includes the decoded manifest and its original verified
record. Pass it to the archive constructor. Recovery events also carry their
original record, allowing archival without recreating the bytes. Publish frames
only after archive.accept(...) succeeds. To publish structured logs yourself,
take the owned entry with archive.take_structured_log() immediately afterward.
The telemetry channel has one publisher and one reader, bounded retention, and
an independent coalesced status slot. ready().await, wait_timeout, and
register_waker support different event loops. A registered waker should only
schedule work or unpark a thread. It must not process data or block the receiving
worker. Consume frames, changed status, and publisher closure before waiting again.
Assembly buffers are recycled and grow within configured bounds as needed. Borrowed delivery avoids mandatory per-event copies, but session discovery, RaptorQ decoding, task-state restoration, and payload decoding can allocate. The presentation ring is also a host-side queue, not a real-time wait-free primitive. Consult the receiver source when changing capacities or callback ownership.
Storage, pacing, and shutdown
Each generated destination has a sender worker with bounded encoded-record pools, packet queues, continuous FEC storage, and retained recovery data. CopperLists and keyframe capture objects are released before transmission waits on the carrier. Structured logging adds a bounded copy of the successful local encoded entry into each destination’s preallocated storage.
The destination reserves four structured-record buffers, each capped at 4096
bytes including the record header, or max_object_bytes if smaller. These buffers
count toward memory_budget_kib. Thread stacks, allocator/channel bookkeeping,
and bounded-input RaptorQ scratch allocations are additional; this setting is a
buffer budget, not a whole-process memory ceiling.
Replay traffic, recovery traffic, and structured logs share byte-deficit
scheduling with weights 3:1:1. Unused capacity is available to other ready
lanes. The bitrate/burst budget covers every lane, while separate bounded packet
queues keep structured traffic from filling the replay queue. Ordinary data
expires after max_latency_ms; the latest complete recovery bundle remains
available for periodic retransmission until replaced or stopped.
Pool exhaustion, queue overflow, expiry, carrier backpressure, and shutdown
shedding have counters. SenderMonitor::snapshot() publishes worker health at
10 Hz, and shutdown statistics/failures are written through Copper structured
logging. Shutdown stops repetition and drains only to the configured latency
deadline. The no_std SenderCore needs a caller to provide CuTime and drive
poll; immediate codec sinks do not enforce pacing themselves.
Compatibility and wire sizing
Build sender, receiver, and logreader for the same producing application and Copper revision. Packets, semantic records, RLC fragments, and session manifests carry no version fields. Schema validation is not a compatibility negotiation mechanism. Unified-log encapsulation version 1 describes the file and section layout, not the encoded application content.
Serialized CopperLists contain id and msgs; lifecycle state stays in memory.
Timestamp-delta and backreference helpers preserve the compressed metadata bytes.
Ordinary integers use the codec’s standard integer encoding.
Current packed header sizes are:
| Layer | Bytes |
|---|---|
| Continuous RLC source packet | 32 |
| Continuous RLC repair packet | 36 |
| RaptorQ object packet | 51 |
| Semantic record | 45 |
| Protected RLC fragment | 20 |
Multi-byte packet-header fields are big endian. The generated sender uses min(mtu_bytes - 51, 1128) bytes per symbol,
reserving the largest current packet header and respecting its fixed storage cap.
At MTU 1200, symbols are therefore at most 1128 bytes. Maximum RLC source, RLC repair, and RaptorQ packets are therefore 1160,
1164, and 1179 bytes respectively. Use actual packet lengths for bandwidth
accounting. The wire encoder
is the authority for a custom implementation.
Packet extent comes from the carrier; there is no redundant payload length or packet sequence counter. FEC identifiers and record identities provide recovery and deduplication. Protected fragments retain record length and fragment index; fragment count and payload extent are derived. The 11-byte RaptorQ OTI omits the library’s reserved zero byte, which decoding restores.
A semantic record’s payload length comes from its reassembled extent. Its BLAKE3 digest covers record kind, object ID, derived length, and payload; appended or truncated payloads fail validation. Recovery points reference the exact manifest and keyframe digests.
Manifests carry identity, decoder requirements, optional feedback cadence and destination key, and application schema. Sender bitrate, latency, adaptation, repair policy, and memory limits remain local. The receiver also enforces its own finite-object and buffering limits; a sender cannot raise them through a manifest.
Both data and feedback packets require carrier-provided packet integrity; neither has an inner CRC. UDP relies on normal network/link checksum protection, while the serial adapter supplies CRC32C. Delivery, ordering, and uniqueness are not guaranteed. Feedback reports retain receiver sequence numbers so stale or duplicate reports do not refresh health. Record digests independently check reassembled content and recovery references.
Missions
So far, our project has a single pipeline: MySource → MyTask → MySink. Every time we
run it, the same three tasks execute in the same order. But real-world robots often need
to operate in different modes – a drone might have a “takeoff” mode and a “cruise”
mode, a warehouse robot might switch between “navigate” and “charge”, or you might want
a “simulation” mode that uses mock drivers instead of real hardware.
In many frameworks, you’d handle this with if statements scattered across your code, or
by maintaining separate launch files. Copper takes a different approach: missions.
What are missions?
A mission is a named variant of your task graph. You declare all possible tasks and
connections in a single copperconfig.ron, then tag each one with the missions it belongs
to. At build time, Copper generates a separate application builder for each mission. At
runtime, you choose which builder to use.
The key properties:
- Tasks without a
missionsfield are shared – they exist in every mission. - Tasks with a
missionsfield are selective – they only exist in the listed missions. - Connections follow the same rule – tag them with
missionsto make them mission-specific. - No recompilation needed to switch – all missions are compiled at once. You pick which one to run.
Step 1: Define missions in copperconfig.ron
Let’s modify our project to support two missions:
- “normal” – The full pipeline we’ve been using: source → processing → sink.
- “direct” – A shortcut that skips the processing task: source → sink.
This is a simple but realistic scenario. Imagine MyTask does some expensive computation
(image processing, path planning). During testing or in a degraded mode, you might want to
bypass it and send raw data straight to the sink.
Replace your copperconfig.ron with:
(
missions: [(id: "normal"), (id: "direct")],
tasks: [
(
id: "src",
type: "tasks::MySource",
),
(
id: "t-0",
type: "tasks::MyTask",
missions: ["normal"],
),
(
id: "sink",
type: "tasks::MySink",
),
],
cnx: [
(
src: "src",
dst: "t-0",
msg: "crate::tasks::MyPayload",
missions: ["normal"],
),
(
src: "t-0",
dst: "sink",
msg: "crate::tasks::MyPayload",
missions: ["normal"],
),
(
src: "src",
dst: "sink",
msg: "crate::tasks::MyPayload",
missions: ["direct"],
),
],
runtime: (
rate_target_hz: 1,
),
)
Let’s break down what changed:
The missions declaration
missions: [(id: "normal"), (id: "direct")],
This top-level array declares all available missions. Each mission is just an ID – a string that you’ll reference elsewhere.
Shared tasks
src and sink have no missions field. This means they participate in every
mission. They are the common backbone of the pipeline.
Mission-specific tasks
(
id: "t-0",
type: "tasks::MyTask",
missions: ["normal"],
),
MyTask is tagged with missions: ["normal"]. It only exists in the “normal” mission.
When running the “direct” mission, this task is simply not instantiated.
Mission-specific connections
The connections are where the graph really diverges:
Mission "normal": src ──▶ t-0 ──▶ sink
Mission "direct": src ──────────▶ sink
In “normal”, data flows through the processing task. In “direct”, the source connects
directly to the sink, bypassing MyTask entirely.
Notice that the connection src → sink in the “direct” mission uses the same message type
(crate::tasks::MyPayload) as the other connections. This works because MySink already
accepts MyPayload as input – the message types must be compatible regardless of which
path the data takes.
Step 2: Update main.rs
When you declare missions, the #[copper_runtime] macro no longer generates a single
builder. Instead, it creates a module for each mission, named after the mission ID.
Each module contains its own builder type.
For our project, the macro generates:
normal::MyProjectApplicationBuilder– builds the “normal” pipelinedirect::MyProjectApplicationBuilder– builds the “direct” pipeline
Update main.rs to select a mission:
pub mod tasks;
use cu29::prelude::*;
use std::path::Path;
use std::thread::sleep;
use std::time::Duration;
const PREALLOCATED_STORAGE_SIZE: Option<usize> = Some(1024 * 1024 * 100);
#[copper_runtime(config = "copperconfig.ron")]
struct MyProjectApplication {}
// Import the per-mission application types
use normal::MyProjectApplication as NormalApp;
use direct::MyProjectApplication as DirectApp;
fn main() {
// Pick the mission from the first command-line argument (default: "normal")
let mission = std::env::args()
.nth(1)
.unwrap_or_else(|| "normal".to_string());
let logger_path = "logs/my-project.copper";
if let Some(parent) = Path::new(logger_path).parent() {
if !parent.exists() {
std::fs::create_dir_all(parent).expect("Failed to create logs directory");
}
}
let clock = RobotClock::default();
debug!("Logger created at {}.", logger_path);
match mission.as_str() {
"normal" => {
debug!("Starting mission: normal");
let app = NormalApp::builder()
.with_clock(clock.clone())
.with_log_path(logger_path, PREALLOCATED_STORAGE_SIZE)
.expect("Failed to setup logger.")
.build()
.expect("Failed to create application.");
app.run_until_shutdown().expect("Failed to run application.");
}
"direct" => {
debug!("Starting mission: direct");
let app = DirectApp::builder()
.with_clock(clock.clone())
.with_log_path(logger_path, PREALLOCATED_STORAGE_SIZE)
.expect("Failed to setup logger.")
.build()
.expect("Failed to create application.");
app.run_until_shutdown().expect("Failed to run application.");
}
other => {
eprintln!("Unknown mission: '{}'. Available: normal, direct", other);
std::process::exit(1);
}
}
debug!("End of program.");
sleep(Duration::from_secs(1));
}
The important changes:
- Two
usestatements import the builders from the generated mission modules. - A
matchon the mission name selects which builder to use. Here we read it from the command line, but you could also use an environment variable, a config file, or any other mechanism. - Each branch is independent – the “normal” branch builds a three-task pipeline, the “direct” branch builds a two-task pipeline.
No changes to tasks.rs are needed. The task implementations are the same – missions
only control which tasks are instantiated and wired, not how they behave.
Step 3: Run it
Run the “normal” mission (the default):
cargo run
00:00:00.0006 [Debug] Source at 630
00:00:00.0006 [Debug] Received message: 42
00:00:00.0006 [Debug] Sink Received message: 43
00:00:01.0002 [Debug] Source at 1000259
00:00:01.0004 [Debug] Received message: 42
00:00:01.0004 [Debug] Sink Received message: 43
00:00:01.9999 [Debug] Source at 1999931
00:00:02.0000 [Debug] Received message: 42
00:00:02.0001 [Debug] Sink Received message: 43
This is the full pipeline. The source produces 42, MyTask transforms it to 43, and the
sink receives 43.
Now run the “direct” mission:
cargo run -- direct
00:00:00.0005 [Debug] Source at 549
00:00:00.0005 [Debug] Sink Received message: 42
00:00:00.9999 [Debug] Source at 999945
00:00:01.0000 [Debug] Sink Received message: 42
00:00:01.9992 [Debug] Source at 1999286
00:00:01.9994 [Debug] Sink Received message: 42
00:00:02.9987 [Debug] Source at 2998704
00:00:02.9988 [Debug] Sink Received message: 42
Notice the difference: there is no “Received message: 42” line from MyTask, and the sink
receives 42 (the raw value from the source) instead of 43. MyTask was never
instantiated – the data went straight from source to sink.
Same binary, same tasks, different wiring. No recompilation.
How it works under the hood
When the #[copper_runtime] macro processes a configuration with missions, it:
- Parses all missions from the top-level
missionsarray. - For each mission, filters the task list and connection list to include only:
- Tasks/connections with no
missionsfield (shared across all missions) - Tasks/connections whose
missionsarray contains this mission’s ID
- Tasks/connections with no
- Generates a Rust module for each mission (named after the mission ID), containing a builder type, the filtered task graph, and the scheduling code.
All of this happens at compile time. The final binary contains the code for every mission, but only the tasks belonging to the selected mission are instantiated at runtime.
A task can belong to multiple missions
A task isn’t limited to a single mission. If you have a task that’s needed in several (but not all) missions, list them:
(
id: "safety-monitor",
type: "tasks::SafetyMonitor",
missions: ["navigate", "charge", "manual"],
),
This task is active in three missions but excluded from others (say, “simulation” or “diagnostics”).
When to use missions
Missions are most useful when you have:
- Hardware vs simulation: Use real drivers in one mission, mock drivers in another.
- Operating modes: Different task graphs for different phases of operation (startup, cruise, landing).
- Platform variants: The same codebase running on different hardware – one mission for the prototype with basic sensors, another for the production model with full sensor suite.
- Debug vs production: A mission with extra logging/monitoring tasks for development, and a lean mission for deployment.
Difference with ROS
In ROS 2, switching between configurations typically means maintaining separate launch files or using launch arguments with conditionals:
# ROS 2: launch file with conditionals
use_sim = LaunchConfiguration('use_sim')
Node(
package='my_robot',
executable='lidar_driver',
condition=UnlessCondition(use_sim),
),
Node(
package='my_robot',
executable='fake_lidar',
condition=IfCondition(use_sim),
),
ros2 launch my_robot robot.launch.py use_sim:=true
This works, but the logic lives in Python launch files that are completely separate from your node code. Errors (wrong topic names, missing remappings) only show up at runtime.
In Copper, all missions live in copperconfig.ron:
Copper:
copperconfig.ron ← missions, tasks, connections
cargo run ← default mission
cargo run -- direct ← alternative mission
Key differences:
| ROS 2 | Copper | |
|---|---|---|
| Where | Python launch files + YAML params | copperconfig.ron |
| Validation | Runtime (nodes may fail to connect) | Compile time (macro checks the graph) |
| Granularity | Per-node conditionals | Per-task and per-connection tagging |
| Switching | Launch arguments | Builder selection in main.rs |
| All variants visible | Spread across files and conditionals | One file, all missions visible |
The main advantage: one file shows every mission and which tasks run in each.
From Project to Workspace
In Setting Up Your Environment, we generated a single-crate project with cu_project. That flat structure
is perfect for getting started, but as your robot grows – more sensors, more algorithms,
shared components across multiple robots – a single crate with everything in tasks.rs
becomes hard to manage.
Copper provides a second template, cu_full, that scaffolds a Cargo workspace with
a clear separation between applications and reusable components. In this chapter, we’ll
generate one and understand its layout.
Generating a workspace
v1.0
Use cargo-cunew with the workspace template:
cargo cunew --template workspace --source git --git-branch master my_workspace
This creates a my_workspace/ directory with a full workspace layout.
What you get
my_workspace/
├── Cargo.toml # Workspace root
├── justfile # Automation helpers
├── apps/
│ └── cu_example_app/ # Your first application
│ ├── Cargo.toml
│ ├── build.rs
│ ├── copperconfig.ron
│ └── src/
│ ├── main.rs
│ ├── logreader.rs
│ ├── resim.rs
│ ├── pgs.rs
│ ├── pgs_candidate.rs
│ ├── messages.rs
│ └── tasks/
│ ├── mod.rs
│ ├── local_example_src.rs
│ ├── local_example_task.rs
│ └── local_example_sink.rs
├── tools/
│ └── cu29_view_helper/ # Graph and schedule launcher
├── components/
│ ├── bridges/
│ ├── monitors/
│ ├── payloads/
│ ├── sinks/
│ ├── sources/
│ └── tasks/
└── doc/
That’s a lot more structure than our simple my_project/. Let’s walk through it.
The workspace root: Cargo.toml
The top-level Cargo.toml defines the workspace and shared dependencies:
[workspace]
members = [
"apps/cu_example_app",
"components/bridges/cu_example_shared_bridge",
"tools/cu29_view_helper",
]
resolver = "2"
[workspace.dependencies]
cu29 = { git = "https://github.com/copper-project/copper-rs", branch = "master" }
cu29-build = { git = "https://github.com/copper-project/copper-rs", branch = "master" }
cu29-export = { git = "https://github.com/copper-project/copper-rs", branch = "master" }
serde = { version = "1.0", features = ["derive"] }
This example selects the checkout used in Setting Up. Keep the same Copper source
for all three crates; a generated workspace records the source you selected when
scaffolding it. The codec is available through cu29::bincode, so application
crates do not need a separate bincode dependency.
Every crate in the workspace references these shared dependencies with
workspace = true in its own Cargo.toml. This means you define dependency versions
once at the workspace level, and all crates stay in sync.
When you add a new application or component, you add it to the members list.
The apps/ directory
This is where your application crates live. Each app is a standalone binary that owns its own runtime configuration, log storage, logreader, and replay target.
The example app (cu_example_app) looks very similar to the my_project we built in
earlier chapters, but with two key differences.
Messages are in their own file
Instead of defining MyPayload inside tasks.rs, the workspace template puts message
types in a dedicated messages.rs:
#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
pub value: i32,
}
}
This makes it easier to share message types – other files within the app import them
with use crate::messages::MyPayload.
Tasks are in their own directory
Instead of one big tasks.rs, each task gets its own file under src/tasks/:
src/tasks/
├── mod.rs # Re-exports all tasks
├── local_example_src.rs # MySource
├── local_example_task.rs # MyTask
└── local_example_sink.rs # MySink
The mod.rs ties them together:
#![allow(unused)]
fn main() {
mod local_example_sink;
mod local_example_src;
mod local_example_task;
pub use local_example_sink::MySink;
pub use local_example_src::MySource;
pub use local_example_task::MyTask;
}
From the rest of the codebase, you still write tasks::MySource – the internal file
structure is hidden behind the module.
This is standard Rust module organization, but it matters as your robot grows. When you
have 15 tasks, having them in separate files with clear names is much easier to navigate
than scrolling through a 1000-line tasks.rs.
The copperconfig.ron is identical
The task graph configuration works exactly the same way. The only difference is that
message paths reference crate::messages::MyPayload instead of
crate::tasks::MyPayload, because the message type moved to its own module:
(
tasks: [
(
id: "src",
type: "tasks::MySource",
),
(
id: "t-0",
type: "tasks::MyTask",
),
(
id: "sink",
type: "tasks::MySink",
),
],
cnx: [
(
src: "src",
dst: "t-0",
msg: "crate::messages::MyPayload",
),
(
src: "t-0",
dst: "sink",
msg: "crate::messages::MyPayload",
),
],
)
The components/ directory
This is where reusable components live – code that can be shared across multiple applications or even published for other people to use.
The directory is organized by category:
| Directory | Purpose | Example |
|---|---|---|
sources/ | Sensor drivers that produce data | Camera driver, IMU reader |
sinks/ | Actuator drivers that consume data | Motor controller, GPIO writer |
tasks/ | Processing algorithms | PID controller, path planner |
bridges/ | Interfaces to external systems | Zenoh bridge, ROS bridge |
monitors/ | Monitoring and visualization | Console TUI, web dashboard |
payloads/ | Shared message types | Sensor payloads, spatial types |
The template generates these directories with placeholder .keep files. They’re empty,
waiting for you to add your own components as your project grows. We’ll cover how to
create shared components and how to reuse existing ones from the Copper ecosystem in the
Reusing Community Components chapter.
Where do message types go?
You might wonder: should messages go in messages.rs inside the app, or in a component
crate under components/payloads/? The answer depends on who needs them.
App-local messages stay in messages.rs inside the app. If MyPayload is only used
by the tasks within cu_example_app, it belongs right there. This is the most common case
when you’re starting out – and it’s exactly where the template puts it.
Shared messages go into a component crate when multiple apps or components need the
same type. For example, if you have two robots that both use the same sensor data format,
you’d create a crate under components/payloads/ and have both apps depend on it.
Ecosystem messages are already defined in Copper’s built-in payload crates (like
cu-sensor-payloads for common sensor types). You don’t write these – you just depend on
them. We’ll explore them in the
Reusing Community Components chapter.
Here’s the rule of thumb:
| Question | Put messages in… |
|---|---|
| Only used by tasks within one app? | apps/my_app/src/messages.rs |
| Shared between multiple apps in your workspace? | A crate under components/payloads/ |
| Already defined by an existing Copper component? | Just depend on that crate |
When in doubt, start local. You can always move a message type into a shared crate later when a second consumer appears.
Running the workspace
From the workspace root, run the example app with:
cargo run -p cu_example_app
The -p flag tells Cargo which workspace member to build and run. This is different from
the simple project where cargo run was enough – in a workspace with multiple binaries,
you need to be explicit.
Simple project vs workspace: when to switch
You don’t need to start with a workspace. Here’s a simple rule of thumb:
| Situation | Use |
|---|---|
| Learning, prototyping, single-robot projects | cu_project (simple) |
| Multiple robots sharing components | cu_full (workspace) |
| Components you want to publish or reuse | cu_full (workspace) |
| Team projects with clear module boundaries | cu_full (workspace) |
The good news: migrating from a simple project to a workspace is just moving files around
and updating Cargo.toml paths. The task code, message types, and copperconfig.ron
format are identical in both cases.
Modular Configuration
In the previous chapter we saw how a workspace separates code into multiple crates.
But what about the configuration? As your robot grows, a single copperconfig.ron with
dozens of tasks and connections becomes hard to read and maintain. What if you have two
motors that use the same driver with different parameters? You’d have to duplicate the
task entry and hope you keep them in sync.
Copper solves this with modular configuration: the ability to split your RON file into smaller pieces and compose them with parameter substitution.
The includes section
The key feature is the includes array at the top level of copperconfig.ron. Each entry
specifies a path to another RON file and an optional set of parameters:
(
tasks: [],
cnx: [],
includes: [
(
path: "sensor.ron",
params: { "id": "front", "pin": 4 },
),
],
)
When Copper processes this configuration, it reads sensor.ron, substitutes the
parameters, and merges the resulting tasks and connections into the main configuration.
Selecting graph fragments with Cargo features
v1.1
Use when on an include when a Cargo feature also enables an optional component crate:
includes: [
(path: "graph/base.ron"),
(
path: "graph/jetson_mipi.ron",
when: Feature("jetson-mipi"),
),
]
The complete included fragment is removed when the feature is off. Copper skips the file before reading, validation, and code generation, so task types and message types from an unavailable target crate never become Rust references.
The available predicates compose without putting conditions on individual graph elements:
when: Not(Feature("camera"))
when: All([Feature("camera"), Feature("recording")])
when: Any([Feature("mipi"), Feature("usb-camera")])
Forward the consuming crate’s active features from its build script:
[features]
default = []
jetson-mipi = ["dep:cu-gstreamer"]
[build-dependencies]
cu29-build = "1.1"
// build.rs
fn main() {
cu29_build::setup();
}
The optional dependency may also live in a target-specific dependency table. See
examples/cu_feature_gated_camera for the complete Linux/aarch64 camera example.
Parameter substitution with {{param}}
Inside an included file, parameters are referenced using double curly braces:
// sensor.ron
(
tasks: [
(
id: "sensor_{{id}}",
type: "tasks::SensorDriver",
config: { "pin": {{pin}} },
),
],
cnx: [],
)
When included with params: { "id": "front", "pin": 4 }, this expands to:
(
tasks: [
(
id: "sensor_front",
type: "tasks::SensorDriver",
config: { "pin": 4 },
),
],
cnx: [],
)
The {{id}} becomes front and {{pin}} becomes 4. Simple text substitution –
it works in task IDs, config values, message types, or anywhere else in the RON file.
Reusing the same file with different parameters
The real power shows up when you include the same file multiple times with different parameters. This is the robotics equivalent of “instantiate the same component twice with different settings.”
Consider a robot with two motors – left and right. They use the same driver code, but different GPIO pins:
// motors.ron
(
tasks: [
(
id: "motor_{{id}}",
type: "tasks::MotorDriver",
config: { "pin": {{pin}}, "direction": "{{direction}}" },
),
],
cnx: [],
)
In your main configuration, include it twice:
(
tasks: [],
cnx: [],
includes: [
(
path: "motors.ron",
params: { "id": "left", "pin": 4, "direction": "forward" },
),
(
path: "motors.ron",
params: { "id": "right", "pin": 5, "direction": "reverse" },
),
],
)
This produces two tasks: motor_left on pin 4 and motor_right on pin 5. One file
defines the motor pattern; the main config just says “give me two of them with these
settings.”
A complete example
Here’s a more realistic configuration that combines local tasks, included subsystems, monitoring, and logging:
(
tasks: [
(
id: "planner",
type: "tasks::PathPlanner",
),
],
cnx: [
(
src: "sensor_front",
dst: "planner",
msg: "crate::messages::SensorData",
),
(
src: "sensor_rear",
dst: "planner",
msg: "crate::messages::SensorData",
),
(
src: "planner",
dst: "motor_left",
msg: "crate::messages::MotorCommand",
),
(
src: "planner",
dst: "motor_right",
msg: "crate::messages::MotorCommand",
),
],
monitor: ( type: "cu_consolemon::CuConsoleMon" ),
logging: (
slab_size_mib: 1024,
section_size_mib: 100,
),
includes: [
(
path: "sensor.ron",
params: { "id": "front", "pin": 2 },
),
(
path: "sensor.ron",
params: { "id": "rear", "pin": 3 },
),
(
path: "motors.ron",
params: { "id": "left", "pin": 4, "direction": "forward" },
),
(
path: "motors.ron",
params: { "id": "right", "pin": 5, "direction": "reverse" },
),
],
)
The main config file focuses on architecture – how subsystems connect to each other.
The included files focus on component definition – what a sensor or motor looks like.
The connections in the main file reference tasks by their expanded IDs (sensor_front,
motor_left), which are predictable from the template + parameters.
Recursive includes
Included files can themselves include other files. This lets you build hierarchical configurations:
copperconfig.ron
├── includes left_arm.ron
│ ├── includes shoulder_motor.ron
│ └── includes elbow_motor.ron
└── includes right_arm.ron
├── includes shoulder_motor.ron
└── includes elbow_motor.ron
Each level can pass different parameters down, so shoulder_motor.ron is written once but
instantiated four times (left shoulder, left elbow, right shoulder, right elbow) – each
with its own pin assignments and IDs.
Where to put included files
There’s no strict rule, but a common convention in workspace projects is to keep included
RON files next to the main copperconfig.ron:
apps/cu_example_app/
├── copperconfig.ron # Main config (includes the others)
├── sensor.ron # Sensor subsystem template
├── motors.ron # Motor subsystem template
└── src/
└── ...
For very large projects, you might create a config/ subdirectory and use relative paths
in the includes:
includes: [
( path: "config/sensor.ron", params: { ... } ),
]
Instance-specific overrides for swarms
When several robots share the same subsystem code but need different calibration data, keep the base subsystem graph static and use a multi-Copper umbrella file with a conventional instance override tree:
(
subsystems: [
(id: "robot", config: "robot_base.ron"),
],
interconnects: [],
instance_overrides_root: "instances",
)
Copper will then probe:
instances/<instance_id>/<subsystem_id>.ron
So instance 17 for subsystem robot resolves:
instances/17/robot.ron
That file applies only config overlays, not topology changes:
(
set: [
(
path: "tasks/imu/config",
value: {
"gyro_bias": [0.012, -0.004, 0.008],
},
),
],
)
Supported paths are intentionally small:
tasks/<id>/configresources/<id>/configbridges/<id>/config
This keeps the graph static while letting each robot carry its own calibration or hardware
parameters in git. See examples/cu_instance_overrides_demo/ for a minimal example.
Difference with ROS
In ROS 2, configuration reuse is handled through launch file composition and YAML parameter files. You can include other launch files and remap parameters:
# ROS 2: composing launch files
IncludeLaunchDescription(
PythonLaunchDescriptionSource('motor_launch.py'),
launch_arguments={'motor_id': 'left', 'pin': '4'}.items(),
)
Copper’s approach is similar in spirit but different in execution:
| ROS 2 | Copper | |
|---|---|---|
| Format | Python launch files + YAML | RON files with {{param}} substitution |
| Reuse | IncludeLaunchDescription | includes array in RON |
| Parameters | Launch arguments + YAML files | params map with text substitution |
| Validation | Runtime | Compile time |
| Nesting | Launch files can include other launch files | RON files can include other RON files |
The main advantage of Copper’s approach is simplicity: parameterization remains text
substitution, while the only conditional composition is a small typed predicate on whole
includes. There is no Python logic or per-task if/else chain. The result is a flat list
of tasks and connections that Copper validates at compile time.
Further reading
- Modular Configuration in the official Copper documentation.
- Configuration reference for the full RON schema.
examples/modular_config_example/in the copper-rs repository for a working example.
Reusing Community Components
One of the biggest advantages of building your robot with Copper is that you don’t have to write everything from scratch. The copper-rs repository ships with a growing collection of ready-made components – sensor drivers, algorithms, bridges, and shared message types – that you can drop into your project.
In this chapter, we’ll explore what’s available and how to use it.
The components directory
The copper-rs repository organizes its components by role:
components/
├── sources/ Sensor drivers that produce data
├── sinks/ Actuator drivers that consume data
├── tasks/ Processing algorithms (input → output)
├── bridges/ Interfaces to external systems (Zenoh, ROS 2)
├── monitors/ Monitoring and visualization tools
├── payloads/ Shared message types
├── libs/ Supporting libraries
├── res/ Platform-specific resource bundles (HAL)
└── testing/ Test helpers (network injection, mocks)
Each component is a standalone Rust crate that implements one or more Copper tasks. You
add it as a dependency, reference it in copperconfig.ron, and you’re done – no glue
code needed.
The ecosystem is actively growing. Browse the components directory in the copper-rs repository to see what’s currently available – you’ll find LiDAR drivers, IMU drivers, camera capture, PID controllers, AprilTag detection, Zenoh and ROS 2 bridges, and more.
How to use a component
Let’s say you want to add a PID controller to your project. The process is:
1. Add the dependency
In your application’s Cargo.toml:
[dependencies]
cu-pid = { path = "../../components/tasks/cu_pid" }
cu-sensor-payloads = { path = "../../components/payloads/cu_sensor_payloads" }
Or if the component is published on crates.io, use the same Copper release for both crates that the rest of your project already uses.
In a workspace, you’d add it to the workspace-level Cargo.toml first:
[workspace.dependencies]
cu-pid = { path = "../../components/tasks/cu_pid" }
cu-sensor-payloads = { path = "../../components/payloads/cu_sensor_payloads" }
Then reference it in the app’s Cargo.toml:
[dependencies]
cu-pid = { workspace = true }
cu-sensor-payloads = { workspace = true }
2. Reference it in copperconfig.ron
Some reusable Copper tasks are generic. cu-pid is one of them, so the usual pattern is
to define a small local alias in your app code first:
#![allow(unused)]
fn main() {
use cu_pid::GenericPIDTask;
pub type PidTask = GenericPIDTask<cu_sensor_payloads::ImuPayload>;
}
Then add the task to your configuration using that alias:
(
id: "pid",
type: "tasks::PidTask",
config: {
"kp": 1.0,
"ki": 0.1,
"kd": 0.05,
},
),
The config section passes parameters that the task reads in its new() constructor. If
a component exposes a concrete, non-generic task type, you can reference the crate path
directly; for generic tasks like cu-pid, a local alias keeps the RON type path simple.
3. Wire it up
Connect it to your existing tasks:
cnx: [
(
src: "imu",
dst: "pid",
msg: "cu_sensor_payloads::ImuPayload",
),
(
src: "pid",
dst: "motor",
msg: "cu_pid::PIDControlOutputPayload",
),
],
That’s it. Beyond the small alias for generic components, there is no adapter layer. The component is a Copper task like any other – it just happens to live in a separate crate.
Using shared payload types
When two components need to exchange data, they must agree on a message type. This is where the payloads crates come in.
For example, cu-sensor-payloads defines common sensor types that multiple source drivers
produce. If you use the BMI088 IMU driver (cu-bmi088), it outputs a type from
cu-sensor-payloads. Any downstream task that accepts that same type can consume the data
without any conversion.
This is the component ecosystem’s contract: drivers produce standard payload types, and
algorithms consume them. Swap a Bosch IMU for an InvenSense IMU, and the downstream
pipeline doesn’t change – both produce the same ImuPayload type.
Writing your own reusable components
If you’ve built a task that could be useful to others – or even just to your future self across projects – you can extract it into a component crate:
- Create a library crate under
components/(pick the right category). - Move your task struct and its
implintolib.rs. - Move shared message types into the crate or use existing ones from
cu-sensor-payloads. - Add it to the workspace
memberslist in the rootCargo.toml. - Reference it from your application’s
Cargo.tomlandcopperconfig.ron.
The workspace template already has placeholder directories with .keep files for each
component category. Just replace the placeholder with your crate.
Difference with ROS
In ROS 2, reusing components means installing packages (via apt or building from source)
and then referencing their nodes in your launch files. The discovery is runtime-based –
nodes find each other through DDS topics.
In Copper, reusing components means adding a Rust dependency and referencing the task
type in copperconfig.ron. The wiring is resolved at compile time.
| ROS 2 | Copper | |
|---|---|---|
| Discovery | Packages via apt / source build | Crates via Cargo.toml |
| Integration | Launch files + topic remapping | copperconfig.ron + type paths |
| Message compatibility | .msg files + code generation | Shared Rust types (payloads crates) |
| Validation | Runtime (topics may not match) | Compile time (types must match) |
| Sharing | ROS package index | crates.io / git dependencies |
The biggest difference is the compile-time guarantee. In ROS, you can wire two nodes
together with mismatched message types and only find out when you run the system. In
Copper, if your PID controller expects ImuPayload and your driver produces
CameraFrame, the compiler tells you immediately.
Automating Tasks with Just
Running a logreader or rendering a graph involves binary names, features, and
paths. A justfile gives those commands short names so you can spend your time
changing the robot instead of reconstructing command lines.
This chapter uses the templates named in the
edition notice and assumes just is available.
Start in the project directory from Setting Up, or the workspace
root from From Project to Workspace. Run just --list to see
the recipes your generated project provides.
Read a recording in two commands
After running the application once, use:
just log
just cl
log reconstructs the structured text with the producing build’s string index.
cl exports each recorded CopperList, including task payloads and timestamps.
For the first project, look for the source value 42 and processing value 43.
Both commands default to the application’s normal log path.
v1.3 The template recipes accept a different application or recording. For a different recording in a single-crate project:
just cl logs/another-session.copper
In the workspace template, the first argument selects the application and the second selects its recording:
just cl cu_example_app apps/cu_example_app/logs/cu_example_app.copper
Recipe arguments are positional. When you target another app, provide its log
path too; the default log path still names cu_example_app.
See the graph and the execution schedule
v1.1 The graph answers “what connects to what?” The schedule answers “where and in what order will that work execute?” Both views help you check a RON edit before running hardware.
v1.3 The current templates use these viewer recipes. From either template’s root:
just graph
just sched
The recipes open SVG files in your default viewer. In a single-crate project,
the outputs are graph.svg and schedule.svg. In the workspace template they
are apps/cu_example_app/graph.svg and
apps/cu_example_app/schedule.svg.
For our three-task app, the graph shows src -> t-0 -> sink. Its default serial
schedule runs those operations on one worker. After selecting the
Pipeline planner, inspect the
schedule to see the process stages placed on separate workers and CopperLists
overlapping across stages.
Resource tables list the consumers of each resource, including configured
LogStream transports. A destination using network.tx, for example, appears as
system: logstream (ground) in the Used by column. Use this to check shared
hardware ownership alongside the task graph.
Add observed timing
v1.1 A schedule diagram explains placement and ordering; it does not establish how fast your app runs. Record a representative run, then add its measured timings.
v1.3 With the current templates, run:
just graph-log
just sched-log
These recipes use the application’s logreader to obtain statistics from its normal log. Keep the log and configuration from the same application version. Look for an expensive stage or a large gap between stages, then use the performance chapters to investigate it.
Target another configuration
v1.3 In a single-crate project, render an alternate configuration without replacing your application config:
just graph copperconfig.ron graph-check.svg
just sched copperconfig.ron schedule-check.svg
In a workspace, supply the app, config, and output paths:
just graph cu_example_app apps/cu_example_app/copperconfig.ron graph-check.svg
just sched cu_example_app apps/cu_example_app/copperconfig.ron schedule-check.svg
The first argument is the Cargo application package name. Both templates also
provide a Rust viewer helper for additional options, including mission
selection. For example, in the workspace from the
missions exercise, select normal with:
cargo run -p cu29-view-helper -- schedule --app cu_example_app --mission normal --output schedule-normal.svg --open
Use a mission ID that your selected app actually declares.
Replay a failure
The generated recipes run replay through the debug-optimized Cargo profile:
just resim
just resim-debug
resim replays the recording. resim-debug serves a replay session to a remote
debug client. Compiler optimization keeps seeking practical while preserving
Copper’s debug logs, assertions, and debugger information. See
Logging and Replaying Data for the replay workflow.
Automate your own repeated commands
A recipe is a name followed by indented shell commands. In the workspace
justfile, add this complete recipe:
run:
cargo run -p cu_example_app
Then run:
just run
It starts the same application you would launch with the Cargo command. Use
just for operations you repeat, keeping explicit arguments for the paths and
applications that change.
Advanced Task Features
In the earlier chapters we treated #[derive(Reflect)], impl Freezable, and
type Resources<'r> = () as required boilerplate. This chapter explains what each of
these does and when you’d want to customize them.
#[derive(Reflect)]
The Reflect derive macro enables runtime introspection on your task struct.
Monitoring tools and simulation environments use it to inspect a task’s fields at runtime
without knowing the concrete type at compile time.
#![allow(unused)]
fn main() {
#[derive(Reflect)]
pub struct MyController {
kp: f64,
ki: f64,
accumulated_error: f64,
}
}
With Reflect derived, a monitoring UI could display kp, ki, and
accumulated_error as live values while the robot is running.
In practice: always add #[derive(Reflect)] to your task structs. It enables
monitoring and simulation tooling when you need it.
Freezable – State Serialization
The Freezable trait enables Copper’s deterministic replay system. The runtime
periodically takes “snapshots” (keyframes) of every task’s internal state, much like
keyframes in a video codec. During replay, it can jump to any snapshot instead of
replaying from the very beginning.
Tasks with no state to freeze
For tasks that hold no internal state (or state that can be trivially reconstructed), the empty implementation is all you need:
#![allow(unused)]
fn main() {
impl Freezable for MySource {}
}
This tells Copper “there’s nothing to snapshot for this task.” This is what we’ve been using throughout the book, and it’s correct for all our tasks so far – they don’t carry any state between cycles.
Stateful tasks
If your task maintains state that affects its behavior across cycles – for example, a PID
controller accumulating error, a Kalman filter maintaining a covariance matrix, or a
counter tracking message sequence numbers – you should implement the freeze and thaw
methods.
The Freezable trait uses bincode’s Encoder and Decoder for serialization:
#![allow(unused)]
fn main() {
use cu29::prelude::*;
use cu29::bincode::enc::Encoder;
use cu29::bincode::de::Decoder;
use cu29::bincode::error::{EncodeError, DecodeError};
#[derive(Reflect)]
pub struct PidController {
kp: f64,
ki: f64,
kd: f64,
accumulated_error: f64,
previous_error: f64,
}
impl Freezable for PidController {
fn freeze<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> {
// Serialize the fields that change at runtime
cu29::bincode::Encode::encode(&self.accumulated_error, encoder)?;
cu29::bincode::Encode::encode(&self.previous_error, encoder)?;
Ok(())
}
fn thaw<D: Decoder>(&mut self, decoder: &mut D) -> Result<(), DecodeError> {
self.accumulated_error = cu29::bincode::Decode::decode(decoder)?;
self.previous_error = cu29::bincode::Decode::decode(decoder)?;
Ok(())
}
}
}
freeze serializes the task’s mutable state using bincode’s Encode trait.
thaw deserializes it back using Decode. The runtime calls these automatically at
keyframe boundaries.
Rule of thumb: Only serialize fields that change during
process(). Configuration fields likekp,ki,kdare set once innew()and don’t need to be frozen – they’ll be reconstructed fromcopperconfig.ronduring replay.
When does this matter?
If you never use Copper’s replay features, the empty impl Freezable is fine for every
task. But if you plan to record and replay robot runs (one of Copper’s strongest features),
implementing Freezable correctly means the replay system can seek to any point in the
log efficiently rather than replaying from the start.
Think of it like a video file: without keyframes, you’d have to decode from the beginning
every time you want to seek. With keyframes (state snapshots), you can jump to any point
and resume from there. Freezable is how Copper creates those keyframes for your task’s
internal state.
A transform that can process independent cycles
v1.3
An image conversion or a fixed arithmetic transform often depends only on its
current input and immutable settings. Such work can use CuStatelessTask: its
per-cycle callbacks borrow &self, allowing an execution plan to overlap calls
for different CopperLists. Use it when each cycle can be processed independently.
A controller whose next output depends on an accumulated error belongs in
CuTask, with that error included in its frozen state.
Try this in the single-crate project from
Writing Tasks. The source still
produces MyPayload { value: 42 } and the sink still receives MyPayload.
1. Replace the processing task
In src/tasks.rs, replace the MyTask struct and its trait implementations
with this complete block. Keep the file’s existing imports, payload, source,
and sink:
#![allow(unused)]
fn main() {
#[derive(Reflect)]
pub struct MyTask {}
impl Freezable for MyTask {}
impl CuStatelessTask for MyTask {
type Resources<'r> = ();
type Input<'m> = input_msg!(MyPayload);
type Output<'m> = output_msg!(MyPayload);
fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self> {
Ok(Self {})
}
fn process(
&self,
ctx: &CuContext,
input: &Self::Input<'_>,
output: &mut Self::Output<'_>,
) -> CuResult<()> {
if let Some(payload) = input.payload() {
output.set_payload(MyPayload { value: payload.value * 2 });
output.tov = input.tov;
debug!(ctx, "Doubled input: {}", payload.value * 2);
} else {
output.clear_payload();
}
Ok(())
}
}
}
The arithmetic uses only the current input. output remains mutable because it
belongs to this CopperList; the task instance is shared through &self.
2. Declare the task kind
Replace the t-0 entry in copperconfig.ron with:
(
id: "t-0",
type: "tasks::MyTask",
kind: stateless_task,
),
Keep the existing src -> t-0 -> sink connections. The explicit kind tells
Copper to call CuStatelessTask rather than CuTask; it cannot infer this
choice from the connections.
3. Run and observe the transform
From the project root:
cargo run
The task reports Doubled input: 84, and the sink reports
Sink Received message: 84. Stop the app with Ctrl-C. You have changed the
transform’s execution contract while keeping its message type and graph wiring.
The serial plan is sufficient for this exercise; overlapping execution requires
a plan that schedules independent occurrences concurrently.
Choosing the contract
CuStatelessTask requires Send + Sync. preprocess, process, and
postprocess all use &self; construction, start, stop, and restoring a
keyframe remain exclusive operations. Immutable calibration or configuration
can live in the struct. An empty Freezable implementation is appropriate when
there is no changing state to restore.
Any interior mutability or shared resource must preserve the same output when
calls overlap. A lock makes access safe but does not make order-dependent
algorithm state deterministic. Use CuTask for that state. Stateless nodes
cannot use background or anytime-task configuration.
Resources – Hardware Injection
The type Resources<'r> associated type declares what hardware or system resources a
task needs. Resources represent physical endpoints – serial ports, GPIO controllers, SPI
buses, cameras – or shared services bound through a resource bundle.
Worker thread pools are configured separately, in the
runtimesection of the config – see Thread Pools, Affinity, and Real-Time Scheduling.
No resources
Most tasks in a simple project don’t need external resources:
#![allow(unused)]
fn main() {
type Resources<'r> = ();
}
This means “give me nothing.” The _resources parameter in new() is just (). This is
what we’ve used throughout the book.
The problem resources solve
On a real robot, a sensor driver needs access to a hardware peripheral – say, a serial port or an SPI bus. Where does that handle come from? You have a few options:
-
Create it inside
new()– Works, but what if two tasks need the same bus? Or if the hardware needs platform-specific initialization that your task shouldn’t know about? -
Pass it as a global – Not great for testing, portability, or safety.
-
Have the runtime inject it – This is what Resources does. You declare what you need, and the runtime provides it.
How it works
Resources involve three pieces:
1. A resource provider (called a “bundle”) is declared in copperconfig.ron:
resources: [
(
id: "board",
provider: "crate::resources::BoardBundle",
config: { "uart_device": "/dev/ttyUSB0" },
),
],
The bundle is a Rust type that knows how to create the actual hardware handles (open the serial port, initialize GPIO, etc.).
2. Tasks declare which resources they need in copperconfig.ron:
(
id: "imu",
type: "tasks::ImuDriver",
resources: { "serial": "board.uart0" },
),
This says: “the imu task needs a resource it calls serial, and it should be bound to
the uart0 resource from the board bundle.”
3. The task declares a Resources type in its Rust implementation that knows how to
pull the bound resources from the runtime’s resource manager:
#![allow(unused)]
fn main() {
impl CuSrcTask for ImuDriver {
type Resources<'r> = ImuResources<'r>;
type Output<'m> = output_msg!(ImuPayload);
fn new(
config: Option<&ComponentConfig>,
resources: Self::Resources<'_>,
) -> CuResult<Self> {
// resources.serial is the hardware handle, injected by the runtime
Ok(Self { port: resources.serial })
}
// ...
}
}
Why this design?
Resources keep your tasks portable. An IMU driver doesn’t need to know which serial
port to open – that’s a deployment detail. On one robot it might be /dev/ttyUSB0, on
another it might be a different bus entirely. By moving the hardware binding to the
configuration, the same task code works on different platforms without changes.
Resources can also be shared. A shared bus (like I2C) can be bound to multiple tasks.
The resource manager handles the ownership semantics: exclusive resources are moved to a
single task, shared resources are behind an Arc and each consumer receives a cloned handle.
v1.1
The same pattern applies to GPU services. With the cu29/cuda feature, a bundle can own a
shared CUDA context and CuCudaPool, while tasks pass
CuHandle<CudaSliceWrapper<T>> through the graph. Keep CUDA streams task-local unless a
component explicitly defines shared-stream ordering. CUDA handles are process-local, so
their producing nodes should use logging: (handle_content: none); they cannot currently
be reconstructed during replay. CuHandle::storage_id can confirm that two stages saw the
same pooled allocation rather than a replacement copy.
When do you need resources?
For the projects in this book, type Resources<'r> = () is all you need. Resources become
important when you:
- Write drivers that talk to real hardware (serial, SPI, GPIO, USB)
- Need multiple tasks to share a hardware bus
- Want the same task to work across different boards without code changes
- Want to swap real hardware for mocks in testing
The examples/cu_resources_test/ in the copper-rs repository is a complete working example
that demonstrates bundles, shared resources, owned resources, and mission-specific resource
bindings.
Composing resource bundles
v1.2 A hardware handle sometimes needs several layers before a task can use it: a UART becomes a configured radio, and the radio becomes a framed telemetry transport. A resource bundle can consume another bundle’s output, so each layer owns its startup and configuration while tasks receive the finished handle.
Inspect a complete radio application
The HC-12 component contains a complete example of this composition. From the root of the copper-rs checkout used for this book, run its software checks:
cd components/res/cu_hc12
just
The tests exercise radio startup with simulated serial, pin, and delay resources;
no radio hardware is needed. For the application wiring, read
examples/echo.ron
and
examples/echo.rs
in this component. The RON configuration declares three bundles:
boardexports a nonblocking UART and its adapter’s RTS pin.timingexports a startup delay.radioconsumes those three resources and exports the configured HC-12 serial handle. The byte bridge takes that handle and echoes received bytes.
In that existing radio declaration, the input bindings are:
resources: {
"serial": "board.serial0",
"set": "board.serial0_rts",
"delay": "timing.delay",
},
The left side names an input required by the consuming provider; the right side names an output from another bundle. Copper constructs providers in dependency order. Cycles, missing providers, inactive mission providers, and unknown binding names fail compilation. Concrete resource types and exclusive ownership are checked at startup. Changing these input bindings requires a rebuild.
Run with two radios
On Unix, connect two HC-12 radios through USB-to-TTL adapters whose compatible
3.3 V active-low RTS# outputs connect to SET, with common grounds. Both UARTs
must match their modules’ saved baud rate (factory default: 9600, 8N1). The
example enables the Linux resource provider’s serial-rts feature. Its
serial0_rts: true configuration exports the owned SET-pin handle and disables
hardware flow control.
From the same component directory, replace these device paths with the two adapters on your machine, then run:
just pair /dev/ttyUSB0 /dev/ttyUSB1
The program uses the baud rate and channel from examples/echo.ron, configures
both radios, and verifies 1-, 31-, 128-, and 256-byte transfers in each direction.
A mismatch or a five-second timeout fails the check. Run it while both radios
are idle. To echo on just one end of a pair instead, edit the device and channel
in examples/echo.ron, then run just echo.
Adapt the layers
A composing provider declares its inputs with resources!, exposes their names
through ResourceBundle::INPUT_NAMES, and retrieves them with
bundle.inputs::<Inputs>(manager)? in its build method. Consume owned inputs
or clone shared handles, then register the wrapped output. See
the HC-12 provider
for a complete implementation.
For telemetry, a SerialLogStreamTxResources<Radio> bundle can consume
radio.serial and export telemetry.tx for a LogStream destination. This
replaces the byte bridge as the radio’s owner. Follow the component’s
LogStream setup
for concrete type aliases, feature setup, and a conservative link budget.
Resource composition establishes startup dependencies; it does not add
per-cycle task-message edges.
Summary
| Feature | Purpose | When to customize |
|---|---|---|
#[derive(Reflect)] | Runtime introspection for monitoring/simulation | Always use the derive; the feature flag controls cost |
impl Freezable | State snapshots for deterministic replay | Implement freeze/thaw for stateful tasks |
type Resources<'r> | Hardware/service dependency injection | Declare resource types when your task needs hardware handles |
For most of the projects you’ll build while learning Copper, the defaults are fine:
#[derive(Reflect)] on the struct, empty impl Freezable, and type Resources<'r> = ().
These become important as you move from prototyping to deploying on real hardware with
real replay requirements.
Writing a Bridge
So far, our Copper apps have been made from sources, tasks, and sinks. Those are enough when every endpoint is independent. A bridge is for the next case: one external connection or protocol has several logical channels that should share the same state.
This chapter builds a tiny bridge with one incoming channel and one outgoing channel. It does not talk to real hardware. That is intentional: the goal is to learn the bridge shape without also learning serial ports, sockets, ROS 2, CAN, or motor protocols.
The bridge will keep two pieces of state:
#![allow(unused)]
fn main() {
connected: bool,
messages_seen: u64,
}
connected changes in the bridge lifecycle. messages_seen increments when the bridge
receives a command. When the bridge sends a status message, it can see the same
messages_seen value. That small bit of shared state is the reason this belongs in one
bridge instead of two unrelated tasks.
The graph will look like this:
counter/command_in ──▶ CountCommands ──▶ counter/status_out
Rx channel task Tx channel
Step 1: Add message types
Create a new file called src/messages.rs:
#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};
#[derive(Default, Debug, Clone, Serialize, Deserialize, Encode, Decode, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct CommandPayload {
pub requested_count: u64,
}
#[derive(Default, Debug, Clone, Serialize, Deserialize, Encode, Decode, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct StatusPayload {
pub cycles: u64,
}
}
These are normal Copper payloads. The bridge receives CommandPayload from the outside
world and sends StatusPayload back out.
Checkpoint: after this step, src/messages.rs should contain only payload types. It
should not mention bridges or tasks yet.
Step 2: Add a task between the bridge channels
Edit src/tasks.rs and add this task:
#![allow(unused)]
fn main() {
use crate::messages::{CommandPayload, StatusPayload};
use cu29::prelude::*;
#[derive(Default, Reflect)]
pub struct CountCommands {
cycles: u64,
}
impl Freezable for CountCommands {}
impl CuTask for CountCommands {
type Resources<'r> = ();
type Input<'m> = input_msg!(CommandPayload);
type Output<'m> = output_msg!(StatusPayload);
fn new(
_config: Option<&ComponentConfig>,
_resources: Self::Resources<'_>,
) -> CuResult<Self> {
Ok(Self { cycles: 0 })
}
fn process<'i, 'o>(
&mut self,
_ctx: &CuContext,
input: &Self::Input<'i>,
output: &mut Self::Output<'o>,
) -> CuResult<()> {
if input.payload().is_some() {
self.cycles += 1;
output.set_payload(StatusPayload {
cycles: self.cycles,
});
} else {
output.clear_payload();
}
Ok(())
}
}
}
This task is deliberately small. It counts how many command messages reached it and emits that count as status.
If your tasks.rs already has use cu29::prelude::*;, do not add it twice. Add only the
use crate::messages::{CommandPayload, StatusPayload}; line and the CountCommands
struct/implementation.
Checkpoint: the task should compile as an ordinary CuTask: one input payload, one
output payload, no resources.
Step 3: Declare bridge channels
Create a new file called src/bridges.rs:
#![allow(unused)]
fn main() {
use crate::messages::{CommandPayload, StatusPayload};
use cu29::prelude::*;
rx_channels! {
pub struct CounterRxChannels : CounterRxId {
command_in => CommandPayload = "counter/command_in",
}
}
tx_channels! {
pub struct CounterTxChannels : CounterTxId {
status_out => StatusPayload = "counter/status_out",
}
}
}
The rx_channels! macro declares channels that enter Copper from the outside world. The
tx_channels! macro declares channels that leave Copper for the outside world.
Each channel has three pieces:
| Piece | Example | Meaning |
|---|---|---|
| Channel id | command_in | Name used by the mission config |
| Payload type | CommandPayload | Message type on that channel |
| Default route | "counter/command_in" | External route/topic/path if the config does not override it |
Checkpoint: after this step, the bridge has channel declarations but no bridge struct yet. That is normal.
Step 4: Implement the bridge state
In the same src/bridges.rs file, add the bridge struct:
#![allow(unused)]
fn main() {
#[derive(Default, Reflect)]
pub struct CounterBridge {
connected: bool,
messages_seen: u64,
}
impl Freezable for CounterBridge {}
}
This is the shared state owned by the bridge instance.
For this tutorial, messages_seen is used only for debug output, so the empty
Freezable implementation is fine. If bridge state affects real outputs during replay,
implement freeze and thaw as shown in the previous chapter.
Checkpoint: the state should belong to the bridge, not to a global variable. Bridge
methods will receive &mut self, so they can update this state safely.
Step 5: Implement CuBridge::new
Continue in src/bridges.rs:
#![allow(unused)]
fn main() {
impl CuBridge for CounterBridge {
type Resources<'r> = ();
type Tx = CounterTxChannels;
type Rx = CounterRxChannels;
fn new(
_config: Option<&ComponentConfig>,
_tx_channels: &[BridgeChannelConfig<<Self::Tx as BridgeChannelSet>::Id>],
_rx_channels: &[BridgeChannelConfig<<Self::Rx as BridgeChannelSet>::Id>],
_resources: Self::Resources<'_>,
) -> CuResult<Self>
where
Self: Sized,
{
Ok(Self {
connected: false,
messages_seen: 0,
})
}
}
Unlike a task constructor, a bridge constructor receives the configured Tx and Rx channel lists. Real bridges use those lists to read routes, topic names, baud rates, QoS settings, or other per-channel configuration. This first bridge ignores them.
Checkpoint: this code opens the impl CuBridge for CounterBridge block but does not
close it yet. The next steps add lifecycle, receive, and send methods inside the same
block.
Step 6: Add lifecycle methods
Still inside the impl CuBridge for CounterBridge block, add:
#![allow(unused)]
fn main() {
fn start(&mut self, _ctx: &CuContext) -> CuResult<()> {
self.connected = true;
debug!("CounterBridge connected");
Ok(())
}
fn stop(&mut self, _ctx: &CuContext) -> CuResult<()> {
self.connected = false;
debug!("CounterBridge disconnected after {} messages", self.messages_seen);
Ok(())
}
}
start runs before bridge I/O begins. stop runs when the application shuts down. A real
bridge might open and close a serial port here. Our tutorial bridge only flips a boolean.
Checkpoint: after this step, connected is changed only by lifecycle methods. That
makes it easy to reason about.
Step 7: Receive commands
Add the receive method inside the same impl block:
#![allow(unused)]
fn main() {
fn receive<'a, Payload>(
&mut self,
_ctx: &CuContext,
channel: &'static BridgeChannel<CounterRxId, Payload>,
msg: &mut CuMsg<Payload>,
) -> CuResult<()>
where
Payload: CuMsgPayload + 'a,
{
match channel.id() {
CounterRxId::CommandIn => {
self.messages_seen += 1;
msg.set_payload(Payload::default());
debug!("CounterBridge received command {}", self.messages_seen);
}
}
Ok(())
}
}
The method is generic over Payload because a real bridge can have many Rx channels with
different payload types. This example has only one Rx channel, so the match has one arm.
The important part is the state change:
#![allow(unused)]
fn main() {
self.messages_seen += 1;
}
That state will still be available when the bridge sends status later in the same graph.
Checkpoint: receive should write into the provided msg buffer with
msg.set_payload(...). Do not allocate a new message.
Step 8: Send status
Finish the impl CuBridge block with send and the final closing brace:
#![allow(unused)]
fn main() {
fn send<'a, Payload>(
&mut self,
_ctx: &CuContext,
channel: &'static BridgeChannel<CounterTxId, Payload>,
msg: &CuMsg<Payload>,
) -> CuResult<()>
where
Payload: CuMsgPayload + 'a,
{
match channel.id() {
CounterTxId::StatusOut => {
if self.connected {
debug!(
"CounterBridge sent status; bridge has seen {} commands",
self.messages_seen
);
} else {
warn!("CounterBridge tried to send while disconnected");
}
debug!("Status payload present: {}", msg.payload().is_some());
}
}
Ok(())
}
}
}
This is where the example shows shared bridge state. receive increments
messages_seen; send reads it. Both methods are part of the same bridge instance.
Checkpoint: the send method should consume the msg passed by the runtime. It
should not call the CountCommands task directly. The graph decides what data reaches the
Tx channel.
Step 9: Register the new modules
Edit src/main.rs. Near the existing module declarations, add:
#![allow(unused)]
fn main() {
pub mod bridges;
pub mod messages;
}
If your file already has:
#![allow(unused)]
fn main() {
pub mod tasks;
}
then the top of the file should now look like:
#![allow(unused)]
fn main() {
pub mod bridges;
pub mod messages;
pub mod tasks;
}
Checkpoint: copperconfig.ron will reference bridges::CounterBridge,
tasks::CountCommands, and messages::..., so all three modules must be visible from
the crate root.
Step 10: Wire the bridge in copperconfig.ron
Replace the simple three-task pipeline with this bridge graph:
(
tasks: [
(
id: "count_commands",
type: "tasks::CountCommands",
),
],
bridges: [
(
id: "counter",
type: "bridges::CounterBridge",
channels: [
Rx(
id: "command_in",
route: "counter/command_in",
),
Tx(
id: "status_out",
route: "counter/status_out",
),
],
),
],
cnx: [
(
src: "counter/command_in",
dst: "count_commands",
msg: "messages::CommandPayload",
),
(
src: "count_commands",
dst: "counter/status_out",
msg: "messages::StatusPayload",
),
],
)
The important syntax is the endpoint path:
bridge_id/channel_id
counter/command_in is an Rx channel, so it can only be a source endpoint.counter/status_out is a Tx channel, so it can only be a destination endpoint.
Checkpoint: the direction should read naturally:
external command -> bridge Rx -> task -> bridge Tx -> external status
If you accidentally wire a Tx channel as a source, or an Rx channel as a destination, Copper should reject the configuration.
Step 11: Build and run
Run a compile check first:
cargo check
If that passes, run the app:
cargo run
You should see debug logs showing that the bridge connected, received commands, and sent status messages. The exact output depends on your logger and monitor setup, but the important pattern is:
CounterBridge connected
CounterBridge received command 1
CounterBridge sent status; bridge has seen 1 commands
Checkpoint: if you see both receive and send logs, the bridge is wired correctly. If
you see receive logs but no send logs, check the count_commands -> counter/status_out
connection. If you see no bridge logs, check that pub mod bridges; exists in main.rs
and that copperconfig.ron has a bridges section.
What this example leaves out
This bridge is intentionally fake. It does not open a socket, parse bytes, or talk to a device. Those details are transport-specific, and adding them now would hide the important shape:
- Declare Rx and Tx channels.
- Store shared state in the bridge struct.
- Use
startandstopfor connection lifecycle. - Use
receiveto fill Copper messages from outside data. - Use
sendto consume Copper messages and write them outside. - Wire bridge channels with
bridge_id/channel_idincopperconfig.ron.
Once that shape is clear, real bridges are mostly a matter of replacing the fake receive and send bodies with protocol code.
Examples to read next
After you finish this chapter, the cleanest examples to read are:
| Example | Why read it |
|---|---|
examples/cu_bridge_test | Shows bridge scheduling and graph wiring with small in-repo bridge implementations. Start here if you want to see Rx-to-task, task-to-Tx, loopback, and fanout shapes. |
examples/cu_resources_test/src/bridges.rs | Shows a compact custom bridge that receives injected resources. Read this after the resources chapter. |
examples/cu_zenoh_bridge_demo | Shows a real middleware bridge used by two Copper apps. This is a good next step once the simple bridge API is familiar. |
There are also protocol-specific bridges such as MSP, CRSF, DSHOT, ROS 2, and Iceoryx2. Those are useful references, but they are not the first examples to read: each one adds domain-specific protocol details on top of the bridge mechanics.
Keep sensor data fresh with a Zenoh bridge
v1.3 A network publisher can send data faster than your robot processes it. If the receiver queues every sample, the control loop can keep receiving messages while quietly falling seconds behind. For a sensor such as odometry, you usually want the newest available reading. For a command stream, you may need every event.
cu-zenoh-bridge makes that choice per receive channel.
It consumes at most one queued sample per receive call. The default is a
one-entry ring: a new arrival replaces the queued older sample.
Try two apps on one machine
Start from the root of the copper-rs checkout used for this book. Build both demo binaries before starting either process:
cargo build -p cu-zenoh-bridge-demo --bin zenoh-ping --bin zenoh-pong
In terminal 1, from that root:
cargo run -p cu-zenoh-bridge-demo --bin zenoh-pong -- --iterations 100 --instance-id 2
While it is running, use terminal 2 from the same root:
cargo run -p cu-zenoh-bridge-demo --bin zenoh-ping -- --iterations 20 --instance-id 1
The apps discover each other as local Zenoh peers. Look for replies tagged
pong-bincode, pong-json, and pong-cbor in the ping app. Each channel uses
its configured representation over one bridge session.
Choose what waits for the next iteration
In examples/cu_zenoh_bridge_demo/pong_config.ron, replace the ping_bin Rx
entry inside the bridge’s channels list with this explicit latest-sample policy:
Rx(
id: "ping_bin",
route: "demo/ping/bin",
config: {
"wire_format": "bincode",
"queue_mode": "ring",
"ring_size": 1,
},
),
Rebuild and rerun the two commands. The wiring and payload types stay the same;
this setting determines which arrival the next receive call reads. In your own
sensor channel, use the same config entries. Setting ring_size alone also
selects a ring of that depth.
To retain every queued event instead, replace that entry with:
Rx(
id: "ping_bin",
route: "demo/ping/bin",
config: {
"wire_format": "bincode",
"queue_mode": "fifo",
},
),
FIFO delivers queued samples in arrival order. Use it for commands or events only when the consumer can keep up: its queue can grow without bound if arrivals outpace receive calls. A deeper ring keeps a bounded backlog and discards the oldest queued sample when full; depth one gives the freshest pending sample.
To see the difference, in src/bin/pong.rs within the demo replace both
Duration::from_millis(200) sleeps with Duration::from_millis(1000).
Repeat the runs first with the ring and then with FIFO. Ping still sends every
200 ms. With the ring, the replies’ sequence numbers skip intermediate arrivals.
With FIFO, replies progress through the older queued numbers. Exact numbers
depend on peer discovery and timing; watch the increasing lag from the current
ping sequence. Restore the sleeps after the exercise.
Apply it to a robot
Choose a queue policy for each Rx channel according to what the message means:
| Data | Starting choice | Reason |
|---|---|---|
| Current pose, temperature, or sensor sample | ring, depth 1 | Old samples should not delay the current reading |
| A short burst whose recent history matters | ring, bounded depth | Retain a finite backlog |
| Commands and events that must each be consumed | fifo | Preserve queued events while ensuring the consumer keeps up |
ring_size must be positive. Set queue_mode on Rx channels; a Tx setting is
rejected during construction. During replay, Copper injects recorded bridge
outputs, so the live subscriber’s queue policy does not change the recorded
values.
When to make a bridge
Use a bridge when the channels belong to the same external connection or protocol:
- One serial port with commands in and telemetry out
- One CAN bus with several message IDs
- One middleware session with several topics
- One ESC bus with command and telemetry channels
Use plain sources and sinks when the endpoints are independent and do not need shared transport state.
Summary
A bridge is not just a source and a sink placed next to each other. It is one component
that owns shared protocol state and exposes several typed channels to Copper. In this
chapter, CounterBridge owned connected and messages_seen; its Rx and Tx methods both
used that state through one bridge instance.
The small counter example is not useful as a robot driver, but it is useful as a template:
start with channel declarations, add the bridge state, implement CuBridge, and then wire
the channels into the graph.
Interfacing a Simulator
The usual simulator integration is an event callback loop:
simulator tick
-> fill Copper's simulated inputs for this tick
-> let Copper run the tasks that should run
-> read Copper's outputs
-> apply those outputs to the simulator
-> advance the simulator
That is the mental model to start with. Copper does not currently ship a ready-made integration for every simulator. The Bevy examples are the concrete examples in this repository, and users have also integrated Copper successfully with simulators such as Isaac Sim, CARLA, and MuJoCo. In all of those cases, the job is to write a thin adapter at the simulator boundary.
If the simulator runs in another process, that adapter can use whatever RPC or message transport fits your setup: sockets, shared memory, middleware topics, or the simulator’s own API. The important part is still the same: each simulator step produces observations for Copper, Copper runs the graph, and commands are sent back to the simulator at a known point in the simulation step.
The callback shape
In simulation mode, Copper generates a SimStep enum from your copperconfig.ron. Each
variant corresponds to one task step in the graph.
Your simulator callback receives those steps while Copper is running an iteration. For each step it returns one of two common answers:
| Return value | Meaning |
|---|---|
SimOverride::ExecuteByRuntime | Copper should run that task normally. Use this for estimation, control, planning, and other algorithm tasks. |
SimOverride::ExecutedBySim | The simulator handled that step. Use this for simulated sensors and actuators. |
The common pattern is:
- Build the Copper runtime in
sim_mode = true. - Register a simulation callback.
- On each simulator tick, update Copper’s clock to the simulator time.
- When Copper reaches a simulated hardware task, fill its output from simulator state.
- When Copper reaches a simulated actuator task, read the command and store it for the simulator.
- Let all ordinary algorithm tasks run normally.
For example, a balancing robot simulator might do this:
Bevy physics tick
-> set Copper time to the current physics time
-> encoder task output is filled from wheel position in the sim
-> angle sensor task output is filled from body orientation in the sim
-> controller tasks run normally
-> motor task output is captured
-> motor force is applied to the simulated body
The controller does not know this is a simulation. It sees the same kind of messages it would see on the real robot.
What to implement first
For any simulator that can call into Copper directly, start with a small adapter layer owned by the simulator:
#![allow(unused)]
fn main() {
struct MySimCopperAdapter {
copper: MyRobotSim,
clock: RobotClock,
clock_mock: RobotClockMock,
last_motor_command: MotorCommand,
}
}
Then wire it into your simulator’s tick/update function:
#![allow(unused)]
fn main() {
fn simulator_tick(adapter: &mut MySimCopperAdapter, sim: &mut MySimulator) -> CuResult<()> {
adapter.clock_mock.set_value(sim.time_nanos());
let mut callback = |step| {
match step {
my_robot::SimStep::Imu(CuTaskCallbackState::Process(_, output)) => {
output.set_payload(sim.read_imu());
SimOverride::ExecutedBySim
}
my_robot::SimStep::Motor(CuTaskCallbackState::Process(input, _output)) => {
if let Some(command) = input.payload() {
adapter.last_motor_command = *command;
}
SimOverride::ExecutedBySim
}
_ => SimOverride::ExecuteByRuntime,
}
};
adapter.copper.run_one_iteration(&mut callback)?;
sim.apply_motor_command(adapter.last_motor_command);
sim.step();
Ok(())
}
}
Treat this as pseudocode: the generated runtime type and step names come from your
copperconfig.ron, and your payload types will be your own. The important point is where
the simulator code lives. It lives in the simulation adapter, not inside the controller
tasks.
In practice, simulated sensors write to output, simulated actuators read from input,
and the estimation/control/planning tasks return ExecuteByRuntime.
For a multi-output source, the Process callback receives the same complete output tuple
declared by the real CuSrcTask. Fill each simulated port in place, then return
ExecutedBySim; downstream tasks consume those messages through the normal Copper graph.
Types to look for
When adapting a simulator, these are the Copper pieces to grep for in existing examples:
| Item | Why it matters |
|---|---|
#[copper_runtime(..., sim_mode = true)] | Generates the simulation-aware runtime and SimStep enum for your graph. |
SimStep | The generated enum you match in the simulation callback. The variants come from task ids in copperconfig.ron. |
CuTaskCallbackState::Process(input, output) | Gives the callback access to a task’s input and output messages for that step. |
SimOverride::ExecutedBySim | Return this when the simulator filled the output or consumed the actuator command. |
SimOverride::ExecuteByRuntime | Return this when Copper should run the task normally. |
RobotClock::mock() and RobotClockMock::set_value(...) | Let the simulator drive Copper time from simulation time. |
run_one_iteration(&mut callback) | Runs one Copper iteration while giving your callback a chance to handle simulated steps. |
The most useful files to read are:
# From a checkout of the main Copper repository:
rg "sim_mode|SimStep|SimOverride|CuTaskCallbackState|run_one_iteration|RobotClock::mock" extra-examples/examples/cu_rp_balancebot/src
rg "sim_mode|SimStep|SimOverride|CuTaskCallbackState|run_one_iteration|RobotClock::mock" extra-examples/examples/cu_flight_controller/src
The first line in the tick function is not incidental:
#![allow(unused)]
fn main() {
adapter.clock_mock.set_value(sim.time_nanos());
}
For simulation, Copper should usually see simulation time, not wall-clock time. If the simulator pauses, Copper time pauses. If the simulator runs faster than real time, the timestamps still describe the simulated run. Decide here whether one simulator tick equals one Copper tick, whether Copper runs on every physics substep, and whether commands apply immediately or on the next simulator step.
Pick an ownership pattern
Choose the shape that matches who owns the main loop.
The simulator owns the loop
This is the most common shape for an interactive simulator.
Your simulator has the main loop. Each simulator tick calls Copper, Copper runs the graph, and the simulator applies the resulting commands.
Use this when:
- the simulator has a physics or rendering loop
- the simulator already controls pause, reset, and stepping
- you want interactive visualization
- you want one Copper tick per simulator tick, or one Copper tick every N simulator ticks
The Bevy examples in the main Copper repository’s extra-examples/ folder use this style:
| Example | What it demonstrates |
|---|---|
examples/cu_rp_balancebot | Bevy owns the physics tick. The simulation callback fills encoder/ADC-style task outputs from the world and applies motor output back into the world. |
examples/cu_flight_controller | Bevy owns the flight simulation. Copper runs the flight-control and compute graphs while callbacks preempt hardware endpoints, including the multi-output ZED source, and downstream tasks consume their normal Copper messages. |
These examples are useful even if you are not using Bevy. Look for the simulator callback layer: that is the part to copy conceptually.
In cu_rp_balancebot, the callback does exactly this:
Balposis a simulated angle sensor, so the callback writes an ADC payload from the simulated body orientation.Railposis a simulated encoder, so the callback writes encoder ticks from the simulated cart position.Motoris a simulated actuator, so the callback reads the motor command and applies a force in the physics world.- all other steps return
ExecuteByRuntime.
Copper owns the loop
Use this when the simulator is a small model or library that Copper can step directly.
In this shape, the Copper app drives the timing. Simulated sensor tasks read from the model, normal tasks run, and simulated actuator tasks write back to the model. This is a good fit for headless tests and deterministic algorithm checks.
Use this when:
- the simulator has no strict main loop of its own
- rendering is not important
- you want batch tests or faster-than-real-time runs
- the simulator can be stepped from Rust without blocking
Useful Resources
Throughout this book, we’ve been learning Copper by building projects step by step. But the copper-rs repository itself is a goldmine of resources that go far beyond what we’ve covered. This chapter is a guided tour of where to find documentation, examples, and tools in the copper-rs repository.
Official documentation
The Copper project maintains a documentation site with detailed guides on every aspect of the framework:
copper-project.github.io/copper-rs
Here are the most useful pages:
| Page | What you’ll find |
|---|---|
| Copper Application Overview | Full walkthrough of a task graph and runtime |
| Copper RON Configuration Reference | Complete schema for copperconfig.ron |
| Task Lifecycle | Detailed explanation of new, start, process, stop |
| Modular Configuration | Includes, parameter substitution, composition |
| Build and Deploy | Building for different targets and deploying |
| Supported Platforms | What hardware and OS combinations are supported |
| Baremetal Development | Running Copper on microcontrollers without an OS |
| Copper Component Catalog | Live catalog of drivers, algorithms, and bridges |
| FAQ | Common questions and answers |
Bookmark the configuration reference in particular – it’s the definitive source for every
field you can put in copperconfig.ron.
The examples/ directory
The repository ships with a large collection of working examples. Each one is a complete,
buildable project with its own copperconfig.ron, task implementations, and main.rs.
When you want to learn a new feature, reading the corresponding example is often the
fastest way to understand how it works in practice.
Browse them at github.com/copper-project/copper-rs/tree/master/examples, or run any example from the repository root:
cargo run -p cu_missions
Project templates
We used cu_project (see Setting Up Your Environment) and cu_full (see From Project to Workspace) to scaffold our projects, but
the bundled template tree has more to offer:
support/cargo_cunew/templates/
├── cu_project/ # Simple single-crate project
├── cu_full/ # Multi-crate workspace with components
└── README.md # Detailed usage guide
The support/cargo_cunew/templates/README.md guide documents both cargo cunew and the
direct cargo generate developer path, including the just gen-workspace / just gen-project
shortcuts for template work inside the repo.
Docker images
The support/docker/ directory contains Dockerfiles for building Copper projects in
containerized environments:
Dockerfile.ubuntu– Standard Ubuntu-based build environmentDockerfile.ubuntu-cuda– Ubuntu with CUDA support for GPU-accelerated tasks
These are useful for CI pipelines or for building on machines where you don’t want to install the full Rust toolchain.
Cross-compilation support
The support/ directory also includes helpers for cross-compiling and deploying to
embedded targets:
just cross-armv7-deploy # Build and deploy to ARMv7 (e.g., Raspberry Pi)
just cross-riscv64-deploy # Build and deploy to RISC-V 64
These commands build release binaries for the target architecture and scp them to the
robot along with the copperconfig.ron.
The Discord
Finally, the Copper project has an active Discord server where you can ask questions, share your projects, and get help from the community and the framework authors.
Python Support
Copper has first-class Python support, but there are two very different ways to use it.
If you only remember one thing from this chapter, remember this:
Python for offline log analysis is reasonable. Python on the live task path is a prototyping hack and should not be treated as a production architecture.
Catalog-Based Offline Analysis
v1.3 If you want to analyse a self-describing recording without rebuilding its application or registering Rust payload types, use Copper’s catalog reader. It returns ordinary Python dictionaries for the recorded values.
Reuse the wheel recording from Self-Describing Logs. On Linux, with Rust and Python 3 available, run this complete sequence from the copper-rs repository root, using the checkout named in the edition notice:
cargo build -p cu29-export --lib --features python-extension-module,self-describing-logs
mkdir -p target/python
cp target/debug/libcu29_export.so target/python/libcu29_export.so
PYTHONPATH=target/python python3 - <<'PYTHON'
import libcu29_export as cu
path = "examples/cu_self_describing_logs/logs/wheel.copper"
catalog = cu.value_decode_catalog_unified(path)
count = 0
for cl in cu.copperlist_value_iterator_unified(path):
print(cl["id"], cl["msgs"][0]["payload"])
count += 1
assert count == 10
PYTHON
python-extension-module builds an importable extension and enables the Python
API; self-describing-logs enables the catalog reader. The temporary
PYTHONPATH makes this built module available to that Python command.
You see the source wheel sample for each of the ten cycles. Inspect the catalog when you need field descriptions or storage units for plotting.
For a multi-run recording, pass run=1 to
copperlist_value_iterator_unified to select the zero-based run index you found
with list-runs. A single-run log needs no run argument.
value_decode_catalog_unified takes only the path: its shared catalog describes
all compiled missions. Each CopperList uses the selected run’s mission and output
order.
Each cycle dictionary has id and msgs, whose entries retain payload, time
of validity, metadata, and capture status. Integers keep full precision. A
corrupt record raises IOError and stops iteration. These APIs are experimental
and run offline.
Application-Specific Offline Analysis
If you already expose Rust payload types through an application-specific PyO3
module, you can read typed CopperLists alongside structured logs and runtime
lifecycle records. The cu_flight_controller example shows this approach in
src/python_module.rs. Use it when your analysis needs those Rust types; the
catalog reader above supplies ordinary dictionaries without that registration.
Runtime Python Tasks
Copper also provides cu-python-task, which lets one task delegate its
process(...) body to a Python script.
This is intentionally a prototyping feature, not a performance feature.
The reason is simple: as soon as Python enters the live execution path, you lose the main properties Copper is designed to preserve:
- low latency
- low jitter
- predictable allocation behavior
- tight control over the realtime path
Python allocates constantly, and the Rust/Python boundary adds even more overhead. That is enough to ruin the realtime characteristics of the stack. Compared to a native Rust Copper task, the performance is abysmal.
The Two Execution Modes
process
In process mode, Copper spawns a separate Python interpreter and exchanges length-prefixed CBOR frames over stdin/stdout.
Advantages:
- the GIL is not inside the Copper process
- interpreter failures are more isolated
Costs:
- CBOR serialization and deserialization every cycle
- copies and allocations every cycle
- interprocess overhead
- more latency and jitter
embedded
In embedded mode, Copper calls Python directly inside the process through PyO3.
Advantages:
- no child process
- no extra CBOR IPC layer
- usually somewhat less overhead than
process
Costs:
- the GIL is now inside the Copper process
- Python exceptions happen inside the runtime process
- values still have to be converted and allocated every cycle
- still not suitable for realtime use
In this workspace, embedded mode is also not supported on macOS.
What It Is Actually For
The intended workflow is narrow:
- prototype one task quickly in Python
- confirm the algorithm is doing the right thing
- rewrite that task in Rust immediately afterward
If you want, you can use an LLM to draft the Rust rewrite once the Python behavior is stable. But the final destination should still be Rust, not a permanent Python task.
Use this feature as a disposable bridge, not as the architecture.
Performance Basics
This chapter should serve two different readers:
- If you already know how Copper executes a cycle and you just want the knobs, jump to Tuning Knob Reference.
- If you are still building the mental model, read straight through, then continue to Troubleshooting Performance.
Start with one simple idea
By default, Copper runs one CopperList at a time on one thread.
That is the baseline model. Most performance questions are just variations of:
- is one task too slow?
- is the end of the CopperList too slow?
- is the OS getting behind on disk writeback?
- does this graph have enough CPU work to profit from overlap?
Those are different problems, so they do not use the same tuning knob.
What one Copper cycle looks like
At a high level, one cycle looks like this:
- Create or reuse a CopperList.
- Run the generated DAG in topological order.
- Record timings and monitoring data.
- Serialize and log the completed CopperList.
- Recycle the CopperList and start the next cycle.
That last part is important. Even when the task path is fast, the cycle is not really over until Copper has dealt with the finished CopperList.
Two places where time is spent
When a user says “my loop is too slow”, the time is usually being lost in one of two places:
Inside the DAG
This means one or more tasks are expensive.
Typical signs:
- one task dominates the LAT tab
- end-to-end latency is high because task execution itself is high
Typical fixes:
- optimize one task
- run one task in the background
- parallelize inside one task
- use
parallel-rtif the graph is a long CPU pipeline
After the DAG finishes
This means the task work is acceptable, but the completed CopperList is expensive to finish.
Typical signs:
- task latencies look acceptable
- the BW tab shows significant serialization or disk cost
Typical fixes:
async-cl-io- reducing logged data
mmap-fsyncif the problem appears later under writeback pressure
What Copper does with the finished CopperList
Copper’s unified logger writes the finished CopperList to a memory-mapped log.
By default:
- the task outputs are serialized at the end of the cycle
- the logger writes to memory-mapped slabs
- when a section is closed, Copper asks the OS to flush that range asynchronously
This is usually fast enough, but it creates an important beginner trap:
- the system can look fine at first
- dirty pages accumulate in the background
- later, the machine starts paying the bill
So “it collapses after a while” is often a logging and writeback story, not a task story.
What to do next
Once this mental model is clear, do not keep guessing. Look at the numbers:
The normal flow should be:
- understand the baseline model
- inspect LAT, BW, MEM, and
log-stats - follow the troubleshooting tree
- only then choose a tuning knob
Tuning Knob Reference
This is the fast path for readers who already know what kind of bottleneck they have.
| Knob | What it changes | Typical symptom |
|---|---|---|
async-cl-io | Moves CopperList serialization and logging to a dedicated thread | Task latencies are fine, but end-of-CL overhead is too high |
parallel-rt + runtime.planner | Enables an explicitly selected concurrent execution plan | Several CPU-bound stages are back to back and the global rate is too low |
mmap-fsync | Forces file sync_all() on section flush | The system runs fine for a while, then collapses under dirty-page / writeback pressure |
background: true | Runs one compatible source or task on the background thread pool and returns None while it is still busy | One isolated stage is too slow, but it does not need to block every cycle |
Task-local thread pool / parallel for | Parallelizes the internals of one task | One hot task has obvious internal data parallelism |
logging: (enabled: false) or enable_task_logging: false | Reduces logged bytes | Bandwidth and serialized size are too high |
logging.copperlist_count | Increases the number of preallocated in-flight CopperLists | Async or parallel modes stall because they run out of CopperList slots |
Feature switches
The runtime features are Cargo features on cu29. A simple way to expose them in your app is
to forward them from your crate:
[features]
async-cl-io = ["cu29/async-cl-io"]
parallel-rt = ["cu29/parallel-rt"]
mmap-fsync = ["cu29/mmap-fsync"]
Then you can run:
cargo run --features async-cl-io
cargo run --features parallel-rt
cargo run --features mmap-fsync
Select a pipeline plan
v1.3
parallel-rt enables the concurrent executor. Your RON
configuration chooses the plan. The default planner is Serial, so enabling
only the Cargo feature keeps serial execution.
Use Pipeline when several CPU-heavy stages can work on different CopperLists
at the same time. While a later stage handles cycle A, an earlier stage can
start cycle B. Each cycle still follows the graph’s dependencies.
Try it in the single-crate project from Writing Tasks. The small arithmetic graph makes the setup visible; its work is too cheap to promise a throughput improvement.
1. Forward the executor feature
Add this entry to the application’s existing [features] table:
parallel-rt = ["cu29/parallel-rt"]
2. Select the plan and its capacity
Keep the tasks and connections in copperconfig.ron. Add the following
runtime and logging sections inside its top-level parentheses, replacing
those sections if they already exist:
runtime: (
planner: (kind: Pipeline, config: {"max_in_flight": 4}),
),
logging: (
copperlist_count: 4,
),
max_in_flight bounds concurrent cycles. copperlist_count reserves their
message storage at startup and must be at least that large. Start with a small
limit: more slots increase memory use, and extra in-flight work can increase
latency without improving throughput.
The example removes the 1 Hz limiter so it does not hold back the pipeline.
Keep or restore rate_target_hz when your application needs a fixed maximum
loop rate.
3. Run and inspect
From the project root:
cargo run --features parallel-rt
The source and sink still report 42 and 43. Stop with Ctrl-C. With the
template’s viewer recipes from Setting Up, run:
just sched
The schedule shows separate process-stage workers and overlap between cycles.
The rt pool’s affinity and policy control these workers; its threads value
does not determine how many stages Pipeline creates. See
Thread Pools before assigning cores or priorities.
To return to serial execution, change the planner to (kind: Serial) and run
cargo run. To choose worker placement and ordering from recorded timings,
continue to Profile-Guided Scheduling, whose
candidates use explicit schedules.
Two distinctions that matter
parallel-rt is not the same as background: true
parallel-rt is a runtime-level pipeline. Multiple CopperLists can be in flight at the
same time, and each generated process stage keeps FIFO order for determinism.
background: true is a stage-level escape hatch. One source or task moves to the
background thread pool and may return None for some cycles while its previous run is
still finishing.
parallel-rt is not the same as task-local parallelism
parallel-rt parallelizes the DAG across CopperLists.
Task-local parallelism parallelizes the inside of one task.
If one task is much heavier than the rest, fixing that task directly is often better than adding runtime-level parallelism around it.
Background tasks and thread pools
If a source or task is marked with background: true, it runs on a worker thread from the
background thread pool. If you do not declare that pool, the runtime creates a default one
for you.
This makes background stages easy to turn on, but it does not mean they are always the right choice. They work best when skipping or sampling intermediate cycles is acceptable.
v1.0
You can size the background pool, pin it to specific cores, or run a stage on a separate
pool entirely. See Thread Pools, Affinity, and Real-Time Scheduling for
the runtime.thread_pools configuration.
Reading Performance Metrics
Performance tuning starts with the right numbers. In Copper, the useful views are already built in:
- live, in the console monitor
- offline, from
log-stats
If you have not added the monitor yet, start with Adding the Console Monitor. If you want offline stats from a log, see Exporting Data to Other Formats.
Live signals
LAT tab
Use the latency tab when you want to know whether the problem is inside the DAG itself.
Look for:
- one task with a much larger mean or max than the others
- high jitter on one task
- a high End2End latency even though no single task stands out
What it usually means:
- one hot task: optimize that task, background it, or parallelize inside it
- many moderate CPU-bound tasks in a row: consider
parallel-rt
BW tab
Use the bandwidth tab when the DAG seems fine but the loop still feels too expensive.
This tab is the first place to look for:
- observed loop rate
- CopperList size
- raw memory throughput
- serialized size
- total disk write rate
What it usually means:
- latency is fine, but BW is high at the end of the cycle: consider
async-cl-io - disk / encoding numbers are simply too large: reduce what you log
- numbers are fine for a while, then the machine slows down: suspect writeback pressure and
test
mmap-fsync
MEM tab
Use the memory pools tab when the system looks unstable or starts failing only under load.
Look for:
- used buffers close to total buffers
- handles in use staying high
- obvious pool exhaustion
What it usually means:
- increase pool sizes
- reduce how long buffers stay alive
- if many CopperLists are in flight, check whether your pool sizing still makes sense
Offline signals with log-stats
The log-stats command adds two kinds of evidence:
- pipeline timing in
perf - per-edge size and throughput in
edges
Typical command:
cargo run --features logreader --bin my-project-logreader -- \
logs/my-project.copper log-stats --output stats.json --config copperconfig.ron
The most useful fields for tuning are:
| Field | Why it matters |
|---|---|
perf.end_to_end.* | Confirms real pipeline latency over a recorded session |
perf.jitter.* | Shows whether the problem is mostly spikes rather than mean cost |
edges[].avg_raw_bytes | Shows which edge is large |
edges[].throughput_bytes_per_sec | Shows which edge is pushing the most sustained traffic |
edges[].rate_hz | Shows whether the effective rate on that edge matches your expectation |
edges[].none_samples | Useful when a task sometimes emits nothing, especially with background behavior |
Symptom to first metric
| Symptom | Check first |
|---|---|
| “The loop is too slow” | LAT tab, then End2End in log-stats |
| “Tasks look fine but the cycle still ends late” | BW tab |
| “It works for a while, then degrades” | BW tab over time, then test mmap-fsync |
| “One task is clearly the culprit” | LAT tab row for that task |
| “A specific edge is too heavy” | avg_raw_bytes and throughput_bytes_per_sec in log-stats |
| “The system fails only under load” | MEM tab |
| “Async or parallel mode did not help much” | Check whether logging.copperlist_count is too low |
A simple workflow
- Start with the LAT tab and BW tab.
- Confirm the hypothesis with
log-stats. - Change one knob at a time.
- Re-run and compare the same metrics.
Then use the decision tree:
Troubleshooting Performance
This chapter is meant to be navigated. Start at the first question and follow the links.
Before you start:
- use the LAT, BW, and MEM tabs from Adding the Console Monitor
- if you need offline evidence, generate
stats.jsonwithlog-stats
Start Here
Q1: Are you missing your target rate?
Q2: Are you actually rate-limited by config?
Check runtime.rate_target_hz.
Q3: Is one task clearly dominating the latency tab?
Q4: Can that task finish later without blocking every cycle?
Q5: Do you have a long CPU-bound pipeline?
Think: several pure compute stages back to back, little blocking I/O, little waiting on hardware.
Q6: Is the problem mostly end-of-CopperList overhead or late collapse?
- Large end-of-cycle overhead right away, while task latencies are acceptable:
Enable
async-cl-io - The system runs well at first, then degrades over time:
Enable
mmap-fsync - Neither: Continue to Q7
Q7: Are you pushing too much data?
Check the BW tab and log-stats:
-
large serialized CopperLists
-
high disk write rate
-
one edge with large
avg_raw_bytesorthroughput_bytes_per_sec
Q8: Are you running out of buffers or in-flight CopperLists?
- Pool pressure in MEM tab: Tune memory pools
- Too few in-flight CopperLists for async or parallel execution: Increase CopperList slots
- Neither: Reduce thread oversubscription
Outcomes
Rate Target Is the Limiter
If runtime.rate_target_hz is lower than the rate you want, Copper is doing exactly what you
asked.
runtime: (
rate_target_hz: 100,
),
If you need a higher rate:
- raise
rate_target_hz - or remove it and run at best effort
If the system starts missing deadlines after that change, go back to Q3 or Q5.
Enable async-cl-io
Use this when the task path is fine, but the end of the CopperList is too expensive.
What it does:
- keeps the DAG execution on the main loop
- queues the completed CopperList to a dedicated serializer thread
- recycles the slot later when serialization finishes
Minimal feature forwarding:
[features]
async-cl-io = ["cu29/async-cl-io"]
cargo run --features async-cl-io
v1.1.4 v1.2.2 Regular background tasks skip empty inputs by default:
(
id: "heavy",
type: "tasks::HeavyTask",
background: true,
),
Empty inputs then skip dispatch. Copper still collects a completed result once,
even on an empty-input cycle, and emits an empty output until another result is
ready. Inputs arriving while a job is running are skipped. Set
background_process_empty: true to dispatch empty inputs. This setting defaults
to false and works with named pools and background anytime tasks wrapped as
regular tasks. See background pool behavior for
a configuration example. Keyframe replay preserves a pending job’s saved input
and whether its completed result has already been collected.
Good fit:
src -> task_a -> task_b -> sink
The LAT tab looks acceptable, but BW shows large serialized CL cost.
Bad fit:
src -> huge_cpu_task -> sink
The hot spot is inside huge_cpu_task itself. Moving CL serialization off-thread does not fix that.
Two checks after enabling it:
- if the benefit is small, your real bottleneck is elsewhere
- if the serializer now waits for free CopperLists, raise
logging.copperlist_count
Enable parallel-rt
v1.3 Use this when you have several CPU-bound stages back to back and want higher global throughput.
What it does:
- keeps deterministic FIFO order per generated process stage
- lets multiple CopperLists be in flight at the same time
- turns the runtime into a stage pipeline instead of a strictly one-CL-at-a-time loop
Minimal feature forwarding:
[features]
parallel-rt = ["cu29/parallel-rt"]
The feature enables the executor; select runtime.planner: (kind: Pipeline, ...)
and reserve enough CopperLists as shown in the
pipeline setup, then run:
cargo run --features parallel-rt
Good fit:
src -> cpu_a -> cpu_b -> cpu_c -> cpu_d -> sink
Why:
- many stages can work on different CopperLists at the same time
- no single stage is overwhelmingly larger than the others
Bad fit:
src -> giant_cpu_task -> sink
Why:
- the giant task is still the throughput limiter
- you usually get more by optimizing or parallelizing that task directly
Also a poor fit:
camera -> wait_for_io -> bridge -> sink
Why:
- the graph is dominated by waiting and I/O, not by a CPU pipeline
Two checks after enabling it:
- if workers look idle, you may need a larger
logging.copperlist_count - if throughput does not move, the graph probably lacks enough useful overlap
Enable mmap-fsync
Use this when the system runs well for a while, then starts degrading, and you suspect the OS is accumulating too many dirty pages from the memory-mapped logger.
What it does:
- keeps the memory-mapped logger
- adds synchronous file
sync_all()on section flush - trades peak throughput for more explicit writeback
Minimal feature forwarding:
[features]
mmap-fsync = ["cu29/mmap-fsync"]
cargo run --features mmap-fsync
Good fit:
The robot behaves well for a while, BW looks reasonable, then the machine starts stalling.
Bad fit:
The loop is already compute-bound from the first second.
After enabling it, measure the cost. If the throughput penalty is too large:
- reduce what you log
- experiment with
section_size_mibandslab_size_mib - or accept the default async writeback and provision the storage path better
Background One Task
Use this when one isolated source or task is too slow, but it does not need to block the whole loop.
(
id: "heavy",
type: "tasks::HeavyTask",
background: true,
),
What it means semantically:
- Copper runs that source or task on the
backgroundthread pool - while it is still busy,
process()returnsNone - downstream tasks therefore see missing output for some cycles
Good fit:
src -> heavy_optional_stage -> sink
Why:
- it is acceptable for the stage to sample or skip intermediate cycles
Bad fit:
src -> estimator -> controller -> actuator
Why:
- the controller path usually needs one coherent output per cycle
If you do not declare a background pool, Copper creates a default one. To size it, pin it
to specific cores, or run this stage on a separate pool, see
Thread Pools, Affinity, and Real-Time Scheduling.
Parallelize Inside One Task
Use this when the real bottleneck is inside one task, not in the DAG structure.
Typical cases:
- per-pixel or per-point computation
- large reduction or map kernels
- CPU-heavy loops that can be split cleanly
Preferred approach:
- keep the DAG simple
- use a controlled parallel loop inside the task, or a declared thread pool
- measure the task again in the LAT tab
Good fit:
src -> expensive_image_task -> sink
Bad fit:
The real issue is serialized logging or OS writeback.
Reduce Logging Volume
Use this when BW and log-stats show that the logger is simply carrying too much data.
The first lever is per-task logging:
(
id: "fast-sensor",
type: "tasks::FastSensor",
logging: (enabled: false),
),
The second lever is global task logging:
logging: (
enable_task_logging: false,
),
v1.2 If CopperLists are still needed but task-state snapshots are not, disable keyframes only:
logging: (
enable_keyframe_logging: false,
),
This compiles keyframe capture out of the generated runtime while retaining CopperList and structured logs. It also avoids attempting to freeze background work at a CopperList boundary; the tradeoff is that replay cannot restore task state from keyframes.
Good fit:
One or two high-rate edges dominate avg_raw_bytes and throughput_bytes_per_sec.
Bad fit:
The hot spot is a CPU-heavy task, but the logged payloads are small.
The practical rule is simple:
- keep logging on for the edges you actually need for replay or diagnosis
- turn it off for bulky intermediate traffic first
Increase CopperList Slots
Use this when async or parallel execution is underfilled because there are not enough preallocated CopperLists available.
logging: (
copperlist_count: 8,
),
This value is consumed by the generated runtime, so it needs to be compiled into the binary.
Good fit:
async-cl-io or parallel-rt is enabled, but the runtime still behaves as if only a tiny number
of CopperLists can be active.
Bad fit:
The graph has no useful overlap to exploit.
More slots help only when there is real work to overlap.
Tune Memory Pools
Use this when the MEM tab shows pressure on pooled buffers or handles.
Good fit:
handle-backed payloads stay alive across several stages and the pool remains near full.
Actions:
- increase pool size
- reduce payload lifetime
- reduce the number of simultaneously retained CopperLists if that is what keeps the buffers alive
Bad fit:
The pool is healthy and the real issue is CPU or disk.
Reduce Thread Oversubscription
Use this when you stacked too many concurrency mechanisms at once:
parallel-rt- several
background: truetasks - task-local Rayon pools or custom worker pools
Symptoms:
- throughput does not improve
- jitter gets worse
- the machine is busy, but the useful rate barely moves
The fix is usually to simplify first:
- use
parallel-rtfor DAG-level overlap - use
background: truefor one isolated non-blocking stage - use task-local parallelism for one hot algorithm
Do not enable all three everywhere by default.
Thread Pools, Affinity, and Real-Time Scheduling
v1.0
By default, Copper runs your task graph on one thread: the global loop executes
source -> processing -> sink back-to-back every cycle. Background tasks and a concurrent plan such as Pipeline step outside that
single thread, and both need
worker threads to run on. This chapter is about controlling those workers: how many there
are, which CPU cores they run on, and what scheduling priority the OS gives them.
On a laptop the defaults are fine. On a real robot, where a latency-critical control loop shares a machine with heavy background work (logging, vision, networking), you often want to isolate the critical work onto dedicated cores and give it real-time priority so the OS never lets background work preempt it. That is what thread pools, CPU affinity, and real-time scheduling are for.
Note: This is a
std-only, host-side feature. Onno_std/ embedded targets there are no threads to schedule and this whole chapter does not apply.
Declaring thread pools
Thread pools are declared in the runtime section of copperconfig.ron, next to
rate_target_hz:
runtime: (
thread_pools: [
( id: "rt", threads: 4, affinity: [2, 3, 4, 5], policy: Fifo(priority: 80) ),
( id: "background", threads: 2, affinity: [0, 1] ),
( id: "vision", threads: 2, policy: Nice(10), on_error: Strict ),
],
),
Each pool has:
id– a unique name. Tasks and the runtime refer to pools by this name. Two names are reserved (see below):backgroundandrt.threads– how many worker threads the pool runs.affinity(optional) – the CPU cores the workers may run on.policy(optional) – the OS scheduling policy and priority. Defaults toFair.on_error(optional) – what to do if the affinity or policy cannot be applied. Defaults toWarn.
Scheduling policies
The policy controls how the OS scheduler treats the pool’s worker threads. On Linux these
map directly onto the POSIX scheduling policies.
| Policy | What it is | When to use it |
|---|---|---|
Fair | Normal fair time-sharing (SCHED_OTHER / CFS). The OS shares the CPU across threads and nothing starves. | The default. Everything that is not latency-critical. |
Nice(n) | Fair scheduler with a niceness bias, n in -20..=19 (lower is more favorable). A soft hint, not a guarantee. | Bias a pool below or above normal work without leaving the fair scheduler – e.g. Nice(10) for heavy work that should yield to the control loop. |
Fifo(priority: p) | SCHED_FIFO hard real-time, p in 1..=99. Runs ahead of every fair thread and is not time-sliced – it runs until it blocks or a higher-priority thread preempts it. | The latency-critical pipeline. Pin it with affinity so a busy worker cannot starve other work on the same core. |
RoundRobin(priority: p) | SCHED_RR, same as Fifo except threads at the same priority are time-sliced in turn. | Several real-time workers that share a priority and should interleave fairly. |
Rule of thumb: reach for
Nicefirst. Real-time policies (Fifo/RoundRobin) are powerful but easy to misuse – a busyFifoworker with noaffinitycan starve the rest of the system. Use them for genuinely time-critical work, and pin them.
Real-time policies are Linux-only and usually require the CAP_SYS_NICE capability (run
as root, grant the capability, or raise the limit in /etc/security/limits.conf). Negative
niceness (Nice(-1) and below) needs the same capability – without it the request falls
back per the pool’s on_error. On non-Linux hosts only Fair is applied directly; Nice,
Fifo, and RoundRobin all fail according to the pool’s on_error. CPU affinity is
cross-platform regardless.
CPU affinity
affinity is a list of logical CPU cores the pool’s workers are allowed to run on. When
set, workers are pinned Spread: worker i is pinned to affinity[i % affinity.len()].
So a pool with threads: 4 and affinity: [2, 3, 4, 5] pins one worker to each of
cores 2-5:
worker 0 -> core 2
worker 1 -> core 3
worker 2 -> core 4
worker 3 -> core 5
The point of affinity is isolation. If your real-time pool owns cores 2-5 and your
background pool is confined to cores 0-1, the OS will not schedule background work onto the
cores running your control loop. Combined with a Fifo policy, this is how you keep jitter
out of the critical path.
If you omit affinity, the OS is free to run the workers on any core.
Choosing cores
Copper does not auto-detect kernel work or CPU topology – the affinity list is taken at
face value. Two things on Linux are worth checking before you pick cores for a
latency-critical pool:
- Where the kernel handles IRQs. Hardware interrupts (NIC, NVMe, USB, timers) run on
whichever CPUs the kernel binds them to, and a busy IRQ adds jitter to that core.
cat /proc/interruptsshows per-CPU interrupt counts;cat /proc/irq/<n>/smp_affinity_listshows the affinity of a specific IRQ. By default most distros either pin IRQs to CPU 0 or letirqbalancespread them; either way, the conservative move is to keep your real-timeaffinityoff the cores that show large counts in/proc/interrupts. For full isolation, the standard tool is the kernel command line (isolcpus=,nohz_full=,rcu_nocbs=), which removes the listed cores from the general scheduler entirely. - Hyperthread siblings. Two logical CPUs that share one physical core compete for the
same execution units, so pinning a real-time pool across a sibling pair gives almost no
isolation from background work landing on the other sibling.
lscpu -eshows theCORE/CPUmapping;/sys/devices/system/cpu/cpuN/topology/thread_siblings_listgives the pair directly. The conservative move is to pick one logical CPU per physical core for thertpool and leave the siblings tobackgroundor to the OS.
The explicit list is intentionally the lowest-level knob – a list of logical CPU ids the
OS understands – and there are no higher-level selectors (BigCores, NumaNode, “avoid
IRQs”, “avoid siblings”) yet. Future versions can layer those on top without changing the
list form, so the recommendation today is to inspect your target machine with the tools
above and write the list explicitly.
Reserved pools: background and rt
Two pool names have special meaning.
background
When a source or task is marked background: true, it runs on the pool named background.
If you do not declare one, Copper creates a default background pool for you: 2 worker
threads, no affinity, Fair scheduling. Declare it explicitly when you want to size it or
pin it:
runtime: (
thread_pools: [
( id: "background", threads: 2, affinity: [0, 1] ),
],
),
v1.1.4 v1.2.2
Regular background tasks skip dispatch when inputs are empty, while still
collecting completed results. If a task must run on empty ticks, set
background_process_empty: true on its node:
(
id: "telemetry",
type: "tasks::TelemetryUploader",
background: true,
background_process_empty: true,
),
This option applies to regular background tasks, including background anytime tasks wrapped as regular tasks. Background sources have no inputs to skip.
A task can also opt into a different pool with the object form of background:
(
id: "detector",
type: "vision::Detector",
background: (pool: "vision"),
),
This runs detector on the vision pool instead of background – useful when you want a
class of heavy tasks scheduled separately from the rest of your background work. The pool
name is resolved at build time: referencing a pool that isn’t declared in
runtime.thread_pools, or pointing a background task at the reserved rt pool, is a
compile-time error.
See The Task Graph for the background field, and
Performance Basics for when backgrounding a stage is the right
call.
rt
v1.3
The rt in parallel-rt stands for runtime – the generated execution engine that
runs the task graph – not “real-time”. The pool named rt configures workers created by the
Pipeline planner: it spawns one worker per
generated process stage, and the rt pool lets you pin and prioritize those workers
exactly like any other pool:
runtime: (
thread_pools: [
( id: "rt", threads: 4, affinity: [2, 3, 4, 5], policy: Fifo(priority: 80) ),
],
),
The rt pool is consumed directly by the engine: only its affinity and policy are
applied to the stage workers (Spread across the stage index). parallel-rt spawns one
worker per generated process stage, so the threads: field has no effect on the rt pool
– size affinity to match the stage count if you want exact one-core-per-stage pinning.
Unlike other pools, you do not bind tasks to it and you cannot use it as a background
target.
When it cannot be applied: on_error
Setting CPU affinity or a real-time priority can fail. The common cases are platform-
specific: on Linux, a Fifo / RoundRobin priority (or Nice below 0) without
CAP_SYS_NICE; on Windows, any non-Fair policy at all, since the POSIX real-time
policies are not implemented there; and on either OS, an affinity entry that names a
core the machine does not have. on_error chooses what happens:
Warn(default) – log a warning and fall back to default scheduling. The pool still runs; it just does not get the requested affinity/priority. This keeps unprivileged dev/laptop runs working out of the box.Strict– fail at startup. Use this on a deployed real-time robot, where running the control loop at default priority is a fault you want to catch loudly rather than discover as jitter in the field.
( id: "control", threads: 2, policy: Fifo(priority: 90), on_error: Strict ),
The rt-scheduling feature
Affinity and scheduling are gated behind the rt-scheduling Cargo feature on cu29,
forwarded from your crate the same way as the other runtime features:
[features]
rt-scheduling = ["cu29/rt-scheduling"]
cargo run --features rt-scheduling
With the feature off, runtime.thread_pools still works – pools are built with the
requested thread counts – but affinity and policy are ignored and a warning is emitted.
This lets you keep one config that runs unprivileged in development and pins to real-time on
the target by flipping a feature flag. CPU affinity is cross-platform; the real-time
policies are Linux-only.
Determinism and replay
Pool configuration is performance-only. Affinity and policy change when and where work
runs, never what each task computes or in what order results commit to a copperlist. By
design, swapping in or out an rt pool produces a bit-identical normalized CopperList
stream in both sync and parallel-rt modes, and the runtime matrix integration tests guard
this as a regression.
The practical consequence is that an offline replay or resim run on a different machine
keeps producing the same output as the live capture, even when the pool config cannot be
honored: with the default on_error: Warn the runtime logs and falls back to default
scheduling, and the stream is unchanged. If you need to force neutralization (for
example, to suppress warnings on a CI host that cannot pin, or to pick cores at runtime),
mutate config.runtime.thread_pools before building the app.
Good fit / Bad fit
Good fit:
core 0-1: background pool (logging, telemetry) Nice / Fair
core 2-5: rt pool (control pipeline) Fifo, pinned
Why:
- the critical pipeline owns dedicated cores at real-time priority
- everything else is confined elsewhere and yields to it
Bad fit:
( id: "everything", threads: 8, policy: Fifo(priority: 99) )
Why:
- a single high-priority
Fifopool with no affinity, sized to every core, can starve the OS itself - real-time priority is for isolating specific work, not for “make it all fast”
Summary
| Field | Purpose | Default |
|---|---|---|
id | Pool name; background and rt are reserved | – (required) |
threads | Number of worker threads | – (required) |
affinity | CPU cores the workers are pinned to (Spread) | unpinned |
policy | OS scheduling policy/priority | Fair |
on_error | Behavior when affinity/policy cannot be applied | Warn |
For complete working examples, see
examples/cu_background_task
(both background forms) and
examples/cu_runtime_matrix
(rt and background pools).
Profile-Guided Scheduling
v1.3 Profile-Guided Scheduling (PGS) uses timings from a representative Copper run to propose execution schedules for your application. It is an offline workflow: the robot records its normal unified log, a host-side logreader generates candidates, and you rebuild and measure those candidates before selecting one.
Workflow
Start with a reference run and its measured performance. Generate candidate plans, select one, then compile and run it under the same workload. Compare the measured results with the reference, test another candidate or replan, and keep the best plan.
The important rule is: a prediction is a candidate to test, not a result to deploy. Task costs change when work is moved between CPUs, caches, and scheduling policies. The measured ranking can select the original schedule, and that is a valid outcome.
PGS changes the execution plan: which worker runs each operation, the order on each worker, CPU placement, and how many CopperLists may be in flight. It does not change the task graph or task implementations. Generated candidates are normal explicit Copper plans, validated against the static graph at build time.
Before you start
Use the same application revision, target-class machine, launch options, and representative workload throughout the comparison. A desktop profile is useful for testing the workflow, but it does not predict a robot computer with different CPUs, devices, or load.
You need:
- An application-specific logreader using
cu29_export::run_cli, as described in Logging and Replaying Data. - A unified log from one mission with runtime lifecycle records and CopperList process timestamps.
- Payload presence logged at the endpoints of every latency chain.
- The
parallel-rtfeature forwarded by the application, as described in Performance Basics.
For handle-backed payloads, handle_content: none is sufficient for PGS: it preserves
message metadata and handle presence without recording the referenced buffer. A terminal
sink has no output payload, so PGS infers that it fired from its directly connected
producer. Keep payload presence for that producer as well.
Record long enough to cover steady state, periodic firing patterns, bursts, and the slow paths that matter to the robot. PGS requires only two firings for a contracted source, but two samples are not a useful latency distribution.
1. Write the scheduling contract
The contract states what the application needs and which machine resources PGS may use. Put
it in a separate file such as schedule.ron:
(
chains: [
(
id: "sense_to_command",
source: "camera",
sink: "motor_command",
deadline_ms: 20,
),
],
sources: [
(
task: "camera",
period_ms: 10,
),
],
cpus: [2, 3, 4, 5],
max_in_flight: 4,
headroom: 0.2,
worker_policy: Fair,
)
| Field | Meaning | How to choose it |
|---|---|---|
chains | End-to-end paths whose latency matters | Name a source, a reachable sink, and the real application deadline |
sources | Input delivery requirements | Give the expected period of sources whose throughput must be preserved; omit sources you do not want scored |
cpus | Exact logical CPUs available to generated workers | Reserve CPUs intentionally; inspect affinity, IRQ load, and sibling cores as described in Thread Pools |
max_in_flight | Largest pipeline depth PGS may explore | Start conservatively; emitted configs raise logging.copperlist_count when needed, which increases preallocated memory |
headroom | Fraction of deadlines and worker windows kept free | 0.2 means candidates enter the headroom penalty above 80% utilization |
worker_policy | Policy applied to generated workers | Start with Fair; Fifo(priority: N) and RoundRobin(priority: N) require deployment support and leave two higher priorities for PGS |
A chain sample is the sink’s process end minus the source’s time of validity. If the source did not set a time of validity, its process start is used. Every chain id must be unique, every deadline and source period must be positive, and every sink must be reachable from its source in the selected mission.
2. Record the baseline
Run the normal application on the deployment system under the workload you care about. Use a distinct log path so the comparison cannot overwrite it:
cargo run --release --bin my-project -- --log logs/pgs-baseline.copper
The exact application command and --log argument are project-specific. What matters is
that the resulting unified log contains the effective configuration, the mission, process
timestamps, and the message presence needed by the contract.
If one log contains several recorded runs, inspect it first:
cargo run --release --features logreader --bin my-project-logreader -- \
logs/pgs-baseline.copper list-runs
Pass the global --run <index> option when optimizing a particular run. PGS deliberately
uses the configuration and mission recorded in that run; there are no separate config or
mission overrides that could silently describe a different application.
3. Generate candidates
Run optimize-schedule through the application’s typed logreader:
cargo run --release --features logreader --bin my-project-logreader -- \
logs/pgs-baseline.copper optimize-schedule \
--contract schedule.ron \
--output target/pgs \
--candidates 3
The search is deterministic: the same log, contract, and application version produce the same candidates. Requesting more candidates gives you more distinct schedules to measure; it does not make the first candidate more thoroughly optimized.
The command prints a readable ranking and writes the complete evidence under target/pgs/:
| Artifact | What you can learn or do with it |
|---|---|
contract.ron | Confirm the normalized requirements used by this run |
profile.ron | Inspect per-operation fired/skipped costs, percentiles, firing patterns, measured chains, source delivery, mission, and graph signature |
plan-N.plan.ron | Inspect or archive the standalone validated CuPlan |
plan-N.config.ron | Build the application with the original recorded config plus that exact explicit plan |
predictions.ron | Compare predicted chain latency, source delivery, worker load, response time, CPU, and score for every candidate |
report.txt | Read the same ranked summary printed in the terminal |
Reading the score
Scores are compared from left to right. A later tier cannot compensate for a worse earlier tier. The predicted score contains:
- Source delivery deficit.
- Number of chains predicted to miss their deadlines.
- Number of chains predicted to consume the configured headroom.
- Sum of chain latency divided by deadline.
- Maximum predicted worker load.
This means PGS does not optimize only deadlines. It first protects contracted source
delivery, then deadline feasibility and margin, then total normalized latency, and finally
load balance. A source affects the first tier only when it appears in sources.
Worker response is the model’s bounded response time for one cycle. unbounded means the
response-time calculation did not converge within its safety cap; do not treat that worker
as viable merely because another reported number looks small.
4. Build and run candidates
Build each candidate from its emitted plan-N.config.ron. The safest setup is a dedicated
candidate binary copied from the application’s normal entry point, with only the compile-time
config path and output log path changed:
#[copper_runtime(config = "target/pgs/plan-1.config.ron")]
struct PgsCandidate {}
Keep the same resources, mission, component parameters, workload, release profile, and
launch options as the baseline. Enable parallel-rt, because a generated explicit schedule
may contain several workers:
cargo run --release --features parallel-rt --bin my-project-pgs-candidate -- \
--log logs/pgs-plan-1.copper
Run on the same target under comparable system load. Warm-up, thermal throttling, IRQ placement, logging pressure, and competing processes can all move the result. If those conditions are part of normal robot operation, keep them in the measurement rather than creating an unrealistically quiet benchmark.
5. Rank measurements and select
Feed candidate logs back into the same command. The positional log remains the baseline:
cargo run --release --features logreader --bin my-project-logreader -- \
logs/pgs-baseline.copper optimize-schedule \
--contract schedule.ron \
--output target/pgs \
--measure plan-1=logs/pgs-plan-1.copper \
--measure plan-2=logs/pgs-plan-2.copper
This regenerates the predictions and adds:
| Artifact | Meaning |
|---|---|
measurements.ron | Every measured candidate, best first, with p50/p99/max latency, misses, delivered source rate, and the corresponding prediction |
report.txt | Predicted ranking followed by the measured ranking |
selected.plan.ron | The standalone plan belonging to the best measured row, including the original plan when baseline wins |
Measured ranking uses delivered source rate and measured p99 chain latency. Its score omits the predicted worker-load tier because the log comparison ranks observed end-to-end results. Deadline misses are still reported for every chain, even though p99 is the ranking statistic.
Use the winning plan-N.config.ron when a candidate wins; keep the original configuration
when baseline wins. selected.plan.ron is the compact plan artifact for inspection,
versioning, or tooling. Before deployment, repeat the winning candidate across the robot’s
expected operating range. Multiple measurement arguments may use names such as
plan-1/r2=logs/pgs-plan-1-r2.copper; each run remains a separate row so variability stays
visible.
Diagnosing rejected inputs
PGS fails instead of filling missing evidence with guessed costs. Common input failures are:
| Error or symptom | Cause and action |
|---|---|
| No recorded configuration or runtime lifecycle record | Record a fresh log with the current application runtime and use the matching logreader build |
| No CopperList with process timestamps | Verify that the selected run contains executed CopperLists, not only lifecycle or text-log records |
| No process timestamps for an operation | The operation did not execute in the captured workload, or the log does not contain its timing metadata; exercise that path and record again |
| A chain has no samples | Its source and sink never both fired in a CopperList; preserve endpoint payload presence and exercise the complete path |
| A contracted source fired fewer than twice | Record a longer representative window or correct the source id/period |
Contract and comparison failures usually indicate a mismatch:
| Error or symptom | Cause and action |
|---|---|
| More than one mission in the selected run | Record or select one mission at a time |
| Unknown task or unreachable sink | Correct the task ids or define a chain that follows the selected mission’s graph |
| Profiles were recorded on different graphs or missions | Rebuild and record baseline and candidates from the same topology, component config, resource bindings, and mission |
| No valid plan on the requested CPUs | Revisit the CPU set, deadlines, headroom, and in-flight capacity; do not weaken requirements merely to produce a file |
| Candidate fails to apply its scheduling policy | Use Fair while validating the workflow, or configure the required affinity and Linux real-time privileges described in Thread Pools |