Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

Welcome to Learning copper-rs, a hands-on guide to building robots with the Copper deterministic robotics operating system.

Most people learn Rust with the amazing Rust Book, so we built it for copper-rs.

This book has been written with the help of AI, for the sake of speed, proper grammar and formatting, but each and every line has been reviewed by a human.

The source for this book is available on GitHub. Feedback and contributions are welcome.

This book will take you from zero to a running Copper project, one concept at a time. By the end, you’ll understand how to define task graphs, write sensor drivers and processing pipelines, and run everything with sub-microsecond deterministic scheduling.

Reading version markers

v1.0 v1.1 v1.2 v1.3

A marker means “available since this version” for a feature or the API shown. Under a heading, it applies to that section; beside a paragraph or list item, it applies only there. Features that predate 1.0 have no marker. Paired patch markers give the minimum patch on each release line when a feature was backported.

This edition’s examples use 1.3.0-dev. Sections marked v1.3 currently require that development checkout; see Setting Up. Older markers identify when a capability became available, but the examples use this edition’s APIs and tools. Keep your application and its logreaders on the same version. Experimental features are identified separately in their chapters.

Who is this for?

This book is aimed at:

  • Robotics developers coming from ROS / ROS 2 who want to explore a Rust-native alternative
  • Rust developers interested in robotics and real-time systems
  • Beginners who want a gentle, progressive introduction to Copper’s architecture

No prior Copper experience is required. Familiarity with Rust syntax is helpful but any experience with other programming language should allow you to get through.

Let’s get started!

What is copper-rs?

Copper is a deterministic robotics runtime written in Rust. Think of it as a “game engine for robots”: describe your system declaratively and Copper will create a custom scheduler and run it deterministically from cloud simulation down to embedded controllers.

The big idea

You describe your robot as a task graph – a directed graph of components that produce, process, and consume data. Copper reads this graph at compile time and generates a custom deterministic scheduler. At runtime, your tasks execute in a precise, pre-computed order with zero overhead.

  ┌──────────┐     ┌──────────┐     ┌──────────┐
  │  Sensor  │────▶│ Algorithm│────▶│ Actuator │
  │  (Source)│     │  (Task)  │     │  (Sink)  │
  └──────────┘     └──────────┘     └──────────┘

Key features

  • Rust-first – Ergonomic and safe.
  • Sub-microsecond latency – Zero-alloc, data-oriented runtime.
  • Deterministic replay – Every run, bit-for-bit identical.
  • Interoperable with ROS 2 – Bridges via Zenoh, opening the path for a progressive migration.
  • Runs anywhere – From Linux servers, workstations, and SBCs to bare-metal MPUs.
  • Built to ship – One stack from simulation to production.

How it works (in 30 seconds)

  1. You define your tasks (Rust structs that implement a trait).
  2. You wire them together in a configuration file (copperconfig.ron).
  3. A compile-time macro reads the config and generates the scheduler.
  4. At runtime, Copper calls your tasks’ process() methods in the optimal order, passing pre-allocated messages between them.

That’s it. No topic discovery, no callback registration, no middleware configuration. Define, wire, run.

copper-rs vs ROS

If you’re coming from ROS or ROS 2, this chapter maps the concepts you already know to their Copper equivalents. The mental model is similar. Both are component-based frameworks with message passing, but the execution model is fundamentally different.

Concept mapping

ROS 2copper-rsNotes
NodeTaskA unit of computation
PublisherCuSrcTaskProduces data (sensor driver)
Subscriber + PublisherCuTaskProcesses data (algorithm)
SubscriberCuSinkTaskConsumes data (actuator)
.msg fileRust struct with derivesMessage definition
TopicConnection in copperconfig.ronData channel between tasks
Launch file + YAML paramscopperconfig.ronGraph topology + task parameters
colcon buildcargo buildBuild system
package.xml / CMakeLists.txtCargo.tomlDependency management
Executor#[copper_runtime] macroGenerated deterministic scheduler
rosbagUnified Logger + cu29-exportRecord and replay
Parameter serverComponentConfig in RONPer-task key-value configuration

Key differences

Scheduling

In ROS 2, nodes are separate processes (or threads in a composed executor). Callbacks fire asynchronously when messages arrive via DDS. The execution order depends on network timing and OS scheduling – it’s non-deterministic.

In Copper, tasks run in the same process and are called synchronously in a compile-time-determined order. Every cycle, every task runs in the exact same sequence. There are no callbacks, no races, no surprises.

Message passing

In ROS 2, messages are serialized, sent over DDS (a network middleware), deserialized, and delivered to callbacks. This adds latency and allocations.

In Copper, messages are pre-allocated buffers in shared memory. A task writes its output directly into a buffer that the next task reads from. No serialization, no copies, no allocations on the hot path.

Replay

In ROS 2, rosbag records and replays topic messages. Replay is approximate – timing jitter, OS scheduling, and node startup order can cause differences.

In Copper, replay is deterministic. The unified logger records every message and periodic state snapshots (“keyframes”). Given the same log, replay produces identical results every time, down to the bit.

Configuration

In ROS 2, you typically write a launch file (Python or XML), separate YAML parameter files, and topic remappings. These are resolved at runtime.

In Copper, everything is in one RON file (copperconfig.ron) that is read at compile time. If your config references a task type that doesn’t exist, you get a compile error, not a runtime crash.

Bridges

The good news is that you do not have to chose between Copper and ROS: Copper’s Zenoh bridge lets you run both side by side, so you can migrate incrementally.

Setting Up Your Environment

Use the development checkout named in the edition notice. Before building a project, we need Rust 1.95 or newer and the Copper project bootstrap tool.

Install Rust

Follow the official installation guide at https://rust-lang.org/tools/install/, or run:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

After installation, make sure cargo is available and recent enough:

cargo --version
rustc --version

Install cargo-cunew

v1.0 cargo-cunew scaffolds a new Copper project from the official Copper templates:

cargo install cargo-cunew

Generate our first simple Copper project

Generate a new project directly:

cargo cunew --source git --git-branch master my_project

This selects the development code used by these chapters and generates a complete project. All Copper dependencies use the same source. For a released application, select a release supported by the feature you want to use.

For more details, see the official documentation: https://copper-project.github.io/copper-rs/Project-Templates/

What you get

The generated project contains everything you need:

my_project/
├── build.rs              # Build script (required by Copper logging)
├── Cargo.toml            # Dependencies
├── copperconfig.ron      # Task graph definition
├── justfile              # Helper commands (logreader, replay, DAG rendering)
└── src/
    ├── main.rs           # Runtime entry point
    ├── logreader.rs      # Offline log export utility
    ├── resim.rs          # Replay / remote-debug entry point
    ├── view.rs           # Graph and schedule viewer helper
    ├── pgs.rs            # Profile-guided scheduling helper
    ├── pgs_candidate.rs  # Candidate runtime for measured scheduling
    └── tasks.rs          # Your task implementations

In the next chapter, we’ll explore what each of these files does.

Try it

You can try to compile and run your project:

cd my_project
cargo run

It will compile and run, and you’ll start to receive some messages:

00:00:01.1781 [Debug] Received message: 42
00:00:01.1781 [Debug] Sink Received message: 43

Kill the process and let’s move to the next chapter.

Project Structure Overview

Let’s look at what’s inside a Copper project and understand the role of each file. Here’s the layout generated by the template:

my_project/
├── build.rs              # Build script (sets up logging index)
├── Cargo.toml            # Dependencies and project metadata
├── copperconfig.ron      # Task graph definition
├── justfile              # Helper commands for logs, replay, and DAG rendering
└── src/
    ├── main.rs           # Runtime entry point
    ├── tasks.rs          # Your task implementations
    ├── logreader.rs      # Log export utility
    ├── resim.rs          # Replay / remote-debug entry point
    ├── view.rs           # Graph and schedule viewer helper
    ├── pgs.rs            # Profile-guided scheduling helper
    └── pgs_candidate.rs  # Candidate runtime for measured scheduling

Which files do I actually work on?

Great news: in the next chapters, we will mainly focus on two files:

FileWhat you do thereROS 2 equivalent
tasks.rsImplement tasks + define message typesNode source files + .msg files
copperconfig.ronDefine the task graph, wire connections, set parametersLaunch file + YAML params

The rest is scaffolding that you set up once and rarely change:

FileRoleHow often you touch it
main.rsBoilerplate: create logger, build runtime, call run()Rarely
build.rsSets an env var for Copper’s logging macrosNever
logreader.rsCLI tool to decode and export Copper’s binary logsRarely
resim.rsReplay target for deterministic re-simulation and remote debugRarely
view.rsLaunches the matching graph and schedule viewersRarely
pgs.rs, pgs_candidate.rsCompare scheduling candidates from recorded timingsWhen tuning schedules
justfileShortcuts for common commands like just log and just graphOccasionally
Cargo.tomlDependenciesWhen adding new hardware driver crates

The mental model

Think of it this way:

  • copperconfig.ron is the architecture of your robot – what components exist and how they connect. It’s declarative.
  • tasks.rs is the behavior of your robot – what each component actually does. It’s imperative Rust code.
  • main.rs is the engine startup – it boots the runtime that executes your architecture and behavior. You don’t need to understand it deeply to get started.

In the next chapters, we’ll dive into each of these files, starting with the task graph configuration.

The Task Graph: copperconfig.ron

The file copperconfig.ron is the heart of your robot’s architecture. It defines what tasks exist, how they connect, and what parameters they receive. Copper reads this file at compile time to generate a deterministic scheduler.

The format is RON (Rusty Object Notation) – a human-readable data format designed for Rust.

The complete example

Here is the copperconfig.ron from our template project:

(
    tasks: [
        (
            id: "src",
            type: "tasks::MySource",
        ),
        (
            id: "t-0",
            type: "tasks::MyTask",
        ),
        (
            id: "sink",
            type: "tasks::MySink",
        ),
    ],
    cnx: [
        (
            src: "src",
            dst: "t-0",
            msg: "crate::tasks::MyPayload",
        ),
        (
            src: "t-0",
            dst: "sink",
            msg: "crate::tasks::MyPayload",
        ),
    ],
)

This defines a three-task pipeline:

MySource  ──▶  MyTask  ──▶  MySink
  "src"        "t-0"        "sink"

We will discuss these 3 traits (Source, Task, Sink) later.
For the moment, let’s focus on the content of the file.

The tasks section

Each entry in the tasks array declares one task:

(
    id: "src",                    // Unique string identifier
    type: "tasks::MySource",      // Rust type that implements the task
    kind: source,                 // Optional but recommended: source | task | sink
),
  • id – A unique name for this task instance. Used to reference it in connections.

  • type – The fully qualified path to the Rust struct (relative to your crate root).

  • kind – v1.0 Declares whether the Rust type implements CuSrcTask, CuTask, or CuSinkTask. Use source, task, or sink. Copper can still infer the ordinary roles for legacy configs when the graph shape is unambiguous, but explicit kind is the preferred form.

    v1.3 For a transform implementing CuStatelessTask, use kind: stateless_task; see the working transform example.

Optional task fields

Beyond id and type, each task entry supports several optional fields:

  • config – A key-value map of parameters passed to the task’s new() constructor as an Option<&ComponentConfig>. We’ll see how to read them in the Task Anatomy chapter.

    (
        id: "gpio",
        type: "cu_rp_gpio::RPGpio",
        config: {
            "pin": 4,
        },
    ),
    
  • missions – A list of mission IDs in which this task is active. Copper supports defining multiple “missions” (configurations of the same robot for different scenarios). A task only gets instantiated if the current mission is in its list. If omitted, the task is active in all missions.

    (
        id: "lidar",
        type: "tasks::LidarDriver",
        missions: ["outdoor", "mapping"],
    ),
    
  • background – When set to true, a compatible task runs on a background thread instead of the critical path. Useful for stages that do heavy or blocking work (network I/O, disk writes) that shouldn’t affect the deterministic scheduling of other tasks.

    (
        id: "telemetry",
        type: "tasks::TelemetryUploader",
        background: true,
    ),
    

    v1.0 Sources can also run in the background. To run a source or task on a specific thread pool instead of the default background one, use the object form background: (pool: "name"). See Thread Pools, Affinity, and Real-Time Scheduling for declaring pools with custom sizing, CPU affinity, and real-time priority.

  • background_process_empty – v1.1.4 v1.2.2 On a regular background task, set this to true to dispatch jobs even when inputs are empty. The default is false; completed results are still collected on empty ticks. See background pool behavior for an example.

  • logging – Controls whether Copper’s unified logger records the output messages of this task. Set enabled: false to reduce log size for high-frequency or uninteresting tasks.

    (
        id: "fast-sensor",
        type: "tasks::HighRateSensor",
        logging: (enabled: false),
    ),
    

The cnx (connections) section

Each entry in cnx wires one task’s output to another’s input:

(
    src: "src",                          // Producing task's id
    dst: "t-0",                          // Consuming task's id
    msg: "crate::tasks::MyPayload",      // Rust type of the message payload
),
  • src – The id of the task producing the message.

  • dst – The id of the task consuming the message.

  • msg – The fully qualified Rust type of the payload (see next chapter for a focus on this).

  • missions (optional) – A list of mission IDs in which this connection is active, just like the missions field on tasks. If omitted, the connection is active in all missions.

    (
        src: "lidar",
        dst: "mapper",
        msg: "crate::tasks::PointCloud",
        missions: ["outdoor", "mapping"],
    ),
    

If a source or regular task intentionally exposes an output that is unused in a given graph, point that connection at dst: "__nc__". This keeps the output message type explicit without adding a dummy sink.

Compile-time constants

v1.1 Robot geometry and other values that define the compiled application can live in the top-level constants array. Numeric constants can select a Rust storage type or an SI quantity and unit:

constants: [
    (id: "MAX_DETECTIONS", storage: usize, value: 64),
    (
        id: "CAMERA_TRANSLATION",
        module: "frames::camera",
        quantity: length,
        value: [0.18, 0.0, 0.31],
    ),
]

This generates constants::MAX_DETECTIONS and frames::camera::CAMERA_TRANSLATION. Values with a physical quantity are normalized to the coherent SI unit during compilation.

For a struct with a const constructor, provide an explicit Rust type and expression instead of value:

(
    id: "WORLD_TO_ROBOT",
    module: "transforms",
    type: "cu_transform::TypedTransform3D<f32, cu_transform::WorldFrame, cu_transform::RobotFrame>",
    expression: "cu_transform::TypedTransform3D::<f32, cu_transform::WorldFrame, cu_transform::RobotFrame>::from_translation_euler_xyz(crate::frames::robot::TRANSLATION, crate::frames::robot::ROTATION)",
)

Expressions may reference other generated constants and compose values through const methods. Rust checks the declared type and rejects non-const operations. Use fully qualified dependency paths and crate::... paths because the expression is compiled in its generated module. See examples/cu_config_constants for typed transforms and include merging.

How this compares to ROS

In ROS 2, you’d create publishers and subscribers on named topics, and they’d find each other at runtime via DDS discovery. In Copper, connections are explicit and resolved at compile time. If you reference a task id that doesn’t exist, you get a compile error – not a silent runtime misconfiguration.

There’s more

The configuration file we’ve seen here is minimal on purpose. Real-world Copper projects can use additional top-level sections for monitoring, logging tuning, rate limiting, missions, and modular composition. We’ll cover these throughout the book as we go.

Defining Messages

Messages (also called payloads) are the data that flows between tasks. In ROS, you’d define these in .msg files and run a code generator. In Copper, they’re just Rust structs with the right derives.

A custom payload

Here’s the message type from our template project (in src/tasks.rs):

#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};

#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
    value: i32,
}
}

Use Copper’s codec

Import the encoding derives from cu29::bincode and add #[bincode(crate = "cu29::bincode")] to each type deriving Encode or Decode. The attribute tells generated code where to find Copper’s codec. This keeps your payloads, logreader, and runtime on the same encoding implementation, including when you enable self-describing logs.

You only need the existing cu29 dependency for the codec; keep serde for its own derives. If a copied example fails with an unresolved bincode import, check both the import and the derive attribute against the complete example above.

What each derive does

Every derive on a payload struct has a specific purpose:

DerivePurpose
DefaultRequired. Copper pre-allocates message buffers at startup. Default provides the initial “empty” value.
Encode, DecodeBinary serialization for Copper’s zero-alloc message buffers and logging. These come from cu-bincode.
Serialize, DeserializeUsed for configuration parsing, log export (MCAP, JSON), and tooling. These come from serde.
ReflectRuntime type introspection for monitoring tools and simulation integration.
DebugHuman-readable printing for development.
CloneAllows copying messages when needed (e.g., forking data to multiple consumers).

You do not really have to worry about all of these derives for now. Just add them each time you define a message, and we’ll see how they come into action later.

Using primitive types

For simple cases, you don’t need a custom struct at all. Primitive types like i32, f64, and bool already implement all the required traits:

#![allow(unused)]
fn main() {
// In copperconfig.ron:
//   msg: "i32"

// In your task:
type Output<'m> = output_msg!(i32);
}

This is great for prototyping. As your robot grows, you’ll likely define richer message types with multiple fields.

Latched state updates

v1.0 Not every payload is a fresh sample. Some values are low-rate state: camera calibration, static transforms, lookup tables, map metadata, and similar data that usually stays the same for many cycles.

Re-sending the full value every cycle is wasteful, but making it implicit would hurt determinism and replay. Copper therefore models this explicitly:

  • CuLatchedStateUpdate<T> is the message that flows across the connection
  • CuLatchedState<T> is the consumer-side cache stored in your task state

The update variants are:

  • NoChange – keep the previously latched value
  • Set(value) – replace the previously latched value
  • Clear – remove the previously latched value

The important mental model is that Copper does not automatically “remember the last message” for you. The state transition is the message. Each downstream task that needs the value keeps its own CuLatchedState<T> and applies updates as they arrive.

Producer side

A source or task that discovers calibration data might publish it once with Set(...), then emit NoChange on later cycles until something changes:

#![allow(unused)]
fn main() {
type Output<'m> = output_msg!(CuLatchedStateUpdate<CameraCalibration>);

fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
    if let Some(calibration) = self.pending_calibration.take() {
        output.set_payload(CuLatchedStateUpdate::Set(calibration));
    } else {
        output.set_payload(CuLatchedStateUpdate::NoChange);
    }
    Ok(())
}
}

If the cached value becomes invalid, emit CuLatchedStateUpdate::Clear.

Consumer side

The consumer stores the latest latched value in its own task state:

#![allow(unused)]
fn main() {
pub struct DepthProjector {
    calibration: CuLatchedState<CameraCalibration>,
}

fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>,
           output: &mut Self::Output<'_>) -> CuResult<()> {
    if let Some(update) = input.payload() {
        self.calibration.update(update);
    }

    let Some(calibration) = self.calibration.get() else {
        return Ok(());
    };

    output.set_payload(project_depth_with(calibration));
    Ok(())
}
}

This pattern keeps the common case cheap, because NoChange is just a tiny explicit update, while still making the state evolution visible in logs and deterministic replay.

Using units directly in payloads

Copper exposes the cu29-units wrappers (through cu29::units) so your payload fields can carry units directly instead of raw f32 values.

#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use cu29::units::si::f32::{Length, Time, Velocity};
use cu29::units::si::length::{inch, meter};
use cu29::units::si::time::second;
use cu29::units::si::velocity::{kilometer_per_hour, meter_per_second};
use serde::{Deserialize, Serialize};

#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct WheelSample {
    pub distance: Length,
    pub dt: Time,
    pub speed: Velocity,
}

impl WheelSample {
    pub fn from_raw(distance_m: f32, dt_s: f32) -> Self {
        let distance = Length::new::<meter>(distance_m);
        let dt = Time::new::<second>(dt_s);

        // m / s -> m/s
        let speed: Velocity = (distance.into_uom() / dt.into_uom()).into(); // this is type safe

        Self {
            distance,
            dt,
            speed,
        }
    }

    pub fn distance_in_inches(&self) -> f32 {
        self.distance.get::<inch>()
    }

    pub fn speed_mps(&self) -> f32 {
        self.speed.get::<meter_per_second>()
    }

    pub fn speed_kph(&self) -> f32 {
        self.speed.get::<kilometer_per_hour>()
    }
}
}

This gives you unit-safe fields in messages, unit-safe math when building messages, and explicit conversions when consuming them. Wrapper types support same-dimension arithmetic (+, -) and scalar scale (* f32, / f32) directly; for cross-dimension operations (like Length / Time), compute with the underlying uom quantity and convert back with .into() (or from_uom).

Designing good payloads

A few tips for payload design:

  • Keep payloads small. They’re pre-allocated and copied between cycles. Large payloads waste memory and cache space.
  • Use fixed-size types. Avoid String or Vec on the critical path. Prefer arrays, fixed-size buffers, or enums.
  • One struct per “topic”. Each connection in copperconfig.ron carries exactly one message type. If you need to send different kinds of data, define different structs and use separate connections.

Example: an IMU payload

Here’s what a more realistic payload might look like for an IMU sensor (from here):

#![allow(unused)]
fn main() {
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct ImuPayload {
    pub accel_x: Acceleration,
    pub accel_y: Acceleration,
    pub accel_z: Acceleration,
    pub gyro_x: AngularVelocity,
    pub gyro_y: AngularVelocity,
    pub gyro_z: AngularVelocity,
    pub temperature: ThermodynamicTemperature,
}
}

In the next chapter, we’ll see how tasks produce and consume these messages.

For more advanced unit algebra, dimensions, and available units, see the underlying uom crate docs.

Writing Tasks: tasks.rs

In the previous chapters we looked at copperconfig.ron (the architecture) and message payloads (the data). Now let’s look at tasks.rs – where the actual behavior lives.

Just like the task graph chapter, we’ll start with the complete file and then break it down piece by piece.

The complete file

Here is the full tasks.rs from our template project:

#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};

// Define a message type
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
    value: i32,
}

// Defines a source (ie. driver)
#[derive(Default, Reflect)]
pub struct MySource {}

impl Freezable for MySource {}

impl CuSrcTask for MySource {
    type Resources<'r> = ();
    type Output<'m> = output_msg!(MyPayload);

    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
    where
        Self: Sized,
    {
        Ok(Self {})
    }

    fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
        output.set_payload(MyPayload { value: 42 });
        Ok(())
    }
}

// Defines a processing task
#[derive(Reflect)]
pub struct MyTask {}

impl Freezable for MyTask {}

impl CuTask for MyTask {
    type Resources<'r> = ();
    type Input<'m> = input_msg!(MyPayload);
    type Output<'m> = output_msg!(MyPayload);

    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
    where
        Self: Sized,
    {
        Ok(Self {})
    }

    fn process(
        &mut self,
        _ctx: &CuContext,
        input: &Self::Input<'_>,
        output: &mut Self::Output<'_>,
    ) -> CuResult<()> {
        debug!("Received message: {}", input.payload().unwrap().value);
        output.set_payload(MyPayload { value: 43 });
        Ok(())
    }
}

// Defines a sink (ie. actuation)
#[derive(Default, Reflect)]
pub struct MySink {}

impl Freezable for MySink {}

impl CuSinkTask for MySink {
    type Resources<'r> = ();
    type Input<'m> = input_msg!(MyPayload);

    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
    where
        Self: Sized,
    {
        Ok(Self {})
    }

    fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>) -> CuResult<()> {
        debug!("Sink Received message: {}", input.payload().unwrap().value);
        Ok(())
    }
}
}

That’s the entire file. Let’s walk through it section by section.

The imports

#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};
}
  • cu29::prelude::* – Brings in everything you need from Copper: task traits, CuContext, ComponentConfig, CuResult, Freezable, Reflect, the input_msg! / output_msg! macros, and the debug! logging macro.
  • cu29::bincode and serde – For the serialization derives on MyPayload (covered in the Defining Messages chapter).

The message type

#![allow(unused)]
fn main() {
#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
    value: i32,
}
}

We already covered this in the previous chapter. This is the data that flows between tasks through the connections defined in copperconfig.ron.

The three task traits

The file defines three structs, each implementing a different trait. Copper provides three task traits for the three roles a task can play in the pipeline:

TraitRoleHas Input?Has Output?ROS Analogy
CuSrcTaskProduces dataNoYesPublisher / driver node
CuTaskTransforms dataYesYesSubscriber + Publisher
CuSinkTaskConsumes dataYesNoSubscriber / actuator node

Let’s look at each one.

Source Task: CuSrcTask – MySource

#![allow(unused)]
fn main() {
#[derive(Default, Reflect)]
pub struct MySource {}

impl Freezable for MySource {}

impl CuSrcTask for MySource {
    type Resources<'r> = ();
    type Output<'m> = output_msg!(MyPayload);

    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
    where
        Self: Sized,
    {
        Ok(Self {})
    }

    fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
        output.set_payload(MyPayload { value: 42 });
        Ok(())
    }
}
}

A source is the entry point of data into the pipeline. It has no input – it generates data, typically by reading from hardware (a camera, an IMU, a GPIO pin).

What happens each cycle: The runtime calls process(), and MySource writes a MyPayload { value: 42 } into the pre-allocated output buffer. Downstream tasks will read this value.

Notice the process() signature: it only has output, no input.

Processing Task: CuTask – MyTask

#![allow(unused)]
fn main() {
#[derive(Reflect)]
pub struct MyTask {}

impl Freezable for MyTask {}

impl CuTask for MyTask {
    type Resources<'r> = ();
    type Input<'m> = input_msg!(MyPayload);
    type Output<'m> = output_msg!(MyPayload);

    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
    where
        Self: Sized,
    {
        Ok(Self {})
    }

    fn process(
        &mut self,
        _ctx: &CuContext,
        input: &Self::Input<'_>,
        output: &mut Self::Output<'_>,
    ) -> CuResult<()> {
        debug!("Received message: {}", input.payload().unwrap().value);
        output.set_payload(MyPayload { value: 43 });
        Ok(())
    }
}
}

A processing task sits in the middle of the pipeline. It reads from upstream and writes downstream.

What happens each cycle: The runtime first runs the upstream source, then calls this task’s process() with the source’s output as input. The task reads the value, logs it, and writes a new value to its own output for the sink downstream.

Notice the process() signature: it has both input and output.

Sink Task: CuSinkTask – MySink

#![allow(unused)]
fn main() {
#[derive(Default, Reflect)]
pub struct MySink {}

impl Freezable for MySink {}

impl CuSinkTask for MySink {
    type Resources<'r> = ();
    type Input<'m> = input_msg!(MyPayload);

    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
    where
        Self: Sized,
    {
        Ok(Self {})
    }

    fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>) -> CuResult<()> {
        debug!("Sink Received message: {}", input.payload().unwrap().value);
        Ok(())
    }
}
}

A sink is the end of the pipeline. It receives data but produces no output. Typically this drives an actuator, writes to a display, or sends data to an external system.

What happens each cycle: The runtime calls process() with the upstream task’s output. The sink reads the value and does something with it (here, it logs it).

Notice the process() signature: it only has input, no output.

Deterministic tasks in live replay

A live Copper twin can reconstruct a synchronous task from captured inputs, recorded time, and frozen state. Select it with streaming: (replay: reconstruct) in the shared graph. Keep logging enabled on that task and all its input producers; missing replay inputs are rejected at compile time. See task reconstruction for supported codecs, a camera pipeline example, and development verification.

Tying it back to the task graph

Remember our copperconfig.ron:

tasks: [
    ( id: "src",  type: "tasks::MySource" ),
    ( id: "t-0",  type: "tasks::MyTask"   ),
    ( id: "sink", type: "tasks::MySink"   ),
],

Each type field points to one of the structs we just defined. The connections in cnx determine which task’s output feeds into which task’s input. The Rust compiler verifies that the message types match at build time.

In the next chapter, we’ll dissect our task to look more closely at each associated type and method in order to understand exactly what they do.

Task Anatomy: Types and Methods

Now that you’ve seen the three task traits, let’s look closely at the associated types and methods that make them work.

Associated Types

type Resources<'r> and impl Freezable

We’ll cover these in detail in the Advanced Task Features chapter. For now, notice that both are defined as empty – type Resources<'r> = () and impl Freezable for MyTask {} – which is all you need for a simple project.

type Input<'m> (CuTask and CuSinkTask only)

#![allow(unused)]
fn main() {
type Input<'m> = input_msg!(MyPayload);
}

Declares what messages this task receives from upstream. The input_msg!() macro wraps your payload type into Copper’s message container (CuMsg<T>), which carries:

  • The payload itself (your struct)
  • Metadata (timestamps, status flags)
  • Time of Validity (tov)

Multiple inputs: If a task receives data from multiple upstream tasks, list the types separated by commas:

#![allow(unused)]
fn main() {
type Input<'m> = input_msg!(SensorA, SensorB);
}

In process(), the input parameter becomes a tuple that you can destructure:

#![allow(unused)]
fn main() {
fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>, ...) -> CuResult<()> {
    let (sensor_a_msg, sensor_b_msg) = *input;
    // Use sensor_a_msg.payload() and sensor_b_msg.payload()
    Ok(())
}
}

type Output<'m> (CuSrcTask and CuTask only)

#![allow(unused)]
fn main() {
type Output<'m> = output_msg!(MyPayload);
}

Declares what messages this task produces for downstream. The output buffer is pre-allocated by the runtime at startup. You don’t create messages – you fill them using set_payload().

Methods

fn new(config, resources) -> CuResult<Self>

#![allow(unused)]
fn main() {
fn new(
    _config: Option<&ComponentConfig>,
    _resources: Self::Resources<'_>,
) -> CuResult<Self>
where
    Self: Sized,
{
    Ok(Self {})
}
}

The constructor. Called once when the runtime builds the task graph.

config: Option<&ComponentConfig> is a key-value map from the task’s config block in copperconfig.ron. For example, if your RON file has:

(
    id: "motor",
    type: "tasks::MotorDriver",
    config: { "pin": 4, "max_speed": 1.0 },
),

You can read the values in new():

#![allow(unused)]
fn main() {
fn new(config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self> {
    let cfg = config.ok_or("MotorDriver requires a config block")?;
    let pin: u8 = cfg
        .get::<u8>("pin")?
        .ok_or_else(|| CuError::from("MotorDriver missing `pin`"))?;
    let max_speed: f64 = cfg
        .get::<f64>("max_speed")?
        .ok_or_else(|| CuError::from("MotorDriver missing `max_speed`"))?;
    Ok(Self { pin, max_speed })
}
}

We’ll discuss resources in future chapters.

fn process() – the main loop

This is where your task does its work. The runtime calls it every cycle. The signature depends on the trait:

TraitSignature
CuSrcTaskprocess(&mut self, ctx, output)
CuTaskprocess(&mut self, ctx, input, output)
CuSinkTaskprocess(&mut self, ctx, input)

In our simple example, the source ignores most parameters and just writes a value:

#![allow(unused)]
fn main() {
fn process(&mut self, _ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
    output.set_payload(MyPayload { value: 42 });
    Ok(())
}
}

Let’s look at what each parameter gives you.

Reading input

input.payload() returns Option<&T> – an Option because the message could be empty (e.g., the upstream task had nothing to send this cycle). In production you should handle None; in our example we just unwrap:

#![allow(unused)]
fn main() {
let value = input.payload().unwrap();
}

Writing output

output.set_payload(value) writes your data into a buffer that was pre-allocated at startup.

v1.0 For values that should persist on the consumer side until explicitly replaced, Copper also has a generic latched-state pattern:

  • CuLatchedStateUpdate<T>::NoChange – keep the cached value
  • CuLatchedStateUpdate<T>::Set(value) – replace the cached value
  • CuLatchedStateUpdate<T>::Clear – remove the cached value

Downstream tasks can store that state in CuLatchedState<T> and apply updates as they arrive. This is a good fit for calibration bundles, static transforms, lookup tables, and other low-rate metadata that should not be rebuilt every cycle. The full producer and consumer pattern is covered in Defining Messages.

The context: &CuContext

Every lifecycle callback receives a ctx parameter of type &CuContext. It dereferences to Copper’s monotonic RobotClock, so ctx.now() gives you the current Copper time, and it also carries runtime metadata such as ctx.cl_id(), ctx.instance_id(), and the currently executing task ID.

ctx.now() returns a CuTime measured in nanoseconds on the application’s timeline. By default, that timeline starts when the clock is constructed. Tasks should use it rather than std::time::SystemTime::now() or std::time::Instant::now() so recorded runs can be replayed.

v1.3 With shared reference time, the timeline can instead use a shared reference epoch, such as TAI. Task code still reads ctx.now().

In our simple project we prefix the parameter with _ because we don’t use it. But on a real robot you’d use it like this:

Timestamp your output (typical for source tasks):

#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
    output.set_payload(MyPayload { value: read_sensor() });
    output.tov = Tov::Time(ctx.now());
    Ok(())
}
}

Compute a time delta (e.g., for a PID controller):

#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, input: &Self::Input<'_>, output: &mut Self::Output<'_>) -> CuResult<()> {
    let now = ctx.now();
    let dt = now - self.last_time;  // CuDuration in nanoseconds
    self.last_time = now;

    let error = self.target - input.payload().unwrap().value;
    let correction = self.kp * error + self.ki * self.integral * dt.as_secs_f64();
    // ...
    Ok(())
}
}

Detect a timeout:

#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, input: &Self::Input<'_>) -> CuResult<()> {
    if input.payload().is_none() {
        let elapsed = ctx.now() - self.last_seen;
        if elapsed > CuDuration::from_millis(100) {
            debug!("Sensor timeout! No data for {}ms", elapsed.as_millis());
        }
    }
    Ok(())
}
}

Why not use the system clock? Because Copper supports deterministic replay. When you replay a recorded run, the runtime feeds your tasks the exact same clock values from the original recording. If you used std::time, the replay would have different timestamps and your tasks would behave differently. With CuContext (and its underlying RobotClock), same clock + same inputs = same outputs, every time.

The golden rule of process()

Because process() runs on the real-time critical path (potentially thousands of times per second), you should avoid heap allocation inside it. Operations like Vec::push(), String::from(), or Box::new() ask the system allocator for memory, which can take an unpredictable amount of time and cause your cycle to miss its deadline.

Copper’s architecture is designed so you never need to allocate in process(): messages are pre-allocated, and the structured logger writes to pre-mapped memory. Keep your process() fast and predictable.


So far we’ve focused on new() and process() – the two methods you’ll always implement. But Copper tasks have a richer lifecycle with optional hooks for setup, teardown, and work that doesn’t need to run on the critical path. Let’s look at that next.

Task Lifecycle

Beyond new() and process(), Copper tasks have a full lifecycle with optional hooks for setup, teardown, and non-critical work. Understanding this lifecycle helps you put the right code in the right place.

The full lifecycle

new()  →  start()  →  [ preprocess() → process() → postprocess() ]  →  stop()
                       └──────────── repeats every cycle ────────────┘

Lifecycle methods

MethodWhenThreadWhat to do here
new()Once, at constructionMainRead config, initialize state
start()Once, before the first cycleMainOpen file handles, initialize hardware, allocate buffers
preprocess()Every cycle, before process()Best-effortHeavy prep work: decompression, FFT, parsing
process()Every cycleCritical pathCore logic. Keep it fast. No allocations.
postprocess()Every cycle, after process()Best-effortTelemetry, non-critical logging, statistics
stop()Once, after the last cycleMainCleanup: close files, stop hardware, free resources

The two threads

Copper splits each cycle across two execution contexts:

Critical path thread

This is where process() runs. Tasks execute back-to-back in the order determined by the task graph topology. The runtime minimizes latency and jitter on this thread. You must avoid allocations, system calls, and anything that could block.

Best-effort thread

This is where preprocess() and postprocess() run. The runtime schedules these to minimize interference with the critical path. You can safely do heavier work here: I/O, allocations, logging, network calls.

What if preprocess() is late?

The critical path never waits for the best-effort thread. If preprocess() takes longer than expected and hasn’t finished when the critical path is ready, process() runs anyway – with whatever data is available from the previous cycle (or nothing, if it’s the first one).

This is intentional: in a real-time system, a late result is a wrong result. It’s better to run your control loop on slightly stale data than to miss a deadline. Your process() should handle this gracefully:

#![allow(unused)]
fn main() {
fn preprocess(&mut self, _ctx: &CuContext) -> CuResult<()> {
    // Heavy work on the best-effort thread -- might be slow
    self.decoded_image = Some(decode_jpeg(&self.raw_buffer));
    Ok(())
}

fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>,
           output: &mut Self::Output<'_>) -> CuResult<()> {
    // Use whatever is ready. If preprocess was late, decoded_image
    // still holds the previous cycle's result (or None on first cycle).
    if let Some(ref image) = self.decoded_image {
        output.set_payload(run_inference(image));
    }
    Ok(())
}
}

The same applies to postprocess(): if it falls behind, the next cycle’s process() still runs on time.

Example: when to use each method

Imagine an IMU driver task:

new()           → Read the SPI bus config from ComponentConfig
start()         → Open the SPI device, configure the sensor registers
preprocess()    → (not needed for this task)
process()       → Read raw bytes from SPI, convert to ImuReading, set_payload()
postprocess()   → Log statistics (sample rate, error count)
stop()          → Close the SPI device

Or a computer vision task:

new()           → Load the model weights
start()         → Initialize the inference engine
preprocess()    → Decode the JPEG image from the camera (heavy, OK on best-effort thread)
process()       → Run inference on the decoded image, output detections
postprocess()   → Update FPS counter, send telemetry
stop()          → Release GPU resources

All lifecycle methods are optional

new() and process() are required – everything else has a default no-op implementation. You only implement the lifecycle methods you need:

#![allow(unused)]
fn main() {
impl CuTask for MyTask {
    type Resources<'r> = ();
    type Input<'m> = input_msg!(MyPayload);
    type Output<'m> = output_msg!(MyPayload);

    // Required: constructor
    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self>
    where Self: Sized {
        Ok(Self {})
    }

    // Required: core logic
    fn process(&mut self, _ctx: &CuContext, input: &Self::Input<'_>,
               output: &mut Self::Output<'_>) -> CuResult<()> {
        // your core logic
        Ok(())
    }

    // Optionally implement any of these:
    // fn start(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
    // fn stop(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
    // fn preprocess(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
    // fn postprocess(&mut self, ctx: &CuContext) -> CuResult<()> { ... }
}
}

Freeze and thaw (state snapshots)

Copper automatically logs every message flowing between tasks. But messages alone aren’t enough to reproduce a task’s behavior – you also need its internal state.

Consider a PID controller that accumulates error over time. If you want to replay from minute 7 of a 10-minute run to debug a crash, you need to know what the accumulated error was at minute 7. Without state snapshots, you’d have to replay from the very start and wait 7 minutes to get there.

That’s what freeze and thaw solve. The Freezable trait gives each task two hooks:

  • freeze() – Save the task’s internal state. Called periodically by the runtime to create “keyframes.”
  • thaw() – Restore the task’s state from a saved snapshot.

These are not part of the per-cycle loop. They run at a much lower rate and are independent of the critical path:

         ┌─── cycle ───┐  ┌─── cycle ───┐        ┌─── cycle ───┐
... ─── process() ─── process() ─── ... ─── process() ─── ...
                              │                         │
                          freeze()                  freeze()
                        (keyframe)                (keyframe)

Think of it like a video codec: process() runs every frame, while freeze() saves a keyframe at a low rate. During replay, the runtime jumps to the nearest keyframe before minute 7, restores every task’s state via thaw(), and replays from there – no need to start from the beginning.

v1.3 Tasks implementing CuStatelessTask do not need to implement Freezable. Their per-cycle callbacks receive &self, and the runtime records an empty keyframe component for them. Restoring a keyframe leaves these task instances unchanged. Keep any state that affects later outputs in a task implementing CuTask, with freeze and thaw preserving that state.

For CuSrcTask, CuTask, and CuSinkTask implementations with no state to snapshot (like our simple MySource, MyTask, MySink), the empty impl Freezable is sufficient. We’ll cover how to implement freeze and thaw for stateful tasks in the Advanced Task Features chapter.

The Remaining Files and Running

We’ve covered copperconfig.ron and tasks.rs – the two files you’ll edit most. Now let’s look at main.rs, build.rs, and the relevant Cargo.toml bits. These are mostly boilerplate that you write once and rarely touch. We’ll come back to logreader.rs in Logging and Replaying Data and replay via resim.rs later.

main.rs – the entry point

v1.2 This entry point uses typed application handles: running consumes the initialized handle and returns a stopped handle.

pub mod tasks;

use cu29::prelude::*;
use std::path::Path;
use std::thread::sleep;
use std::time::Duration;

const PREALLOCATED_STORAGE_SIZE: Option<usize> = Some(1024 * 1024 * 100);

#[copper_runtime(config = "copperconfig.ron")]
struct MyProjectApplication {}

fn main() {
    let logger_path = "logs/my-project.copper";
    if let Some(parent) = Path::new(logger_path).parent() {
        if !parent.exists() {
            std::fs::create_dir_all(parent).expect("Failed to create logs directory");
        }
    }
    debug!("Logger created at {}.", logger_path);
    debug!("Creating application... ");
    let application = MyProjectApplication::builder()
        .with_log_path(logger_path, PREALLOCATED_STORAGE_SIZE)
        .expect("Failed to setup logger.")
        .build()
        .expect("Failed to create application.");
    debug!("Running... starting clock: {}.", application.clock().now());

    let stopped = application.run_until_shutdown().expect("Failed to run application.");
    debug!("End of program: {}.", stopped.clock().now());
    sleep(Duration::from_secs(1));
}

Here’s what each part does:

pub mod tasks; – Brings in your tasks.rs. The task types you defined there (e.g., tasks::MySource) are what copperconfig.ron references.

#[copper_runtime(config = "copperconfig.ron")] – The key macro. At compile time, it reads your config file, parses the task graph, computes a topological execution order, and generates a custom runtime struct with a deterministic scheduler. It also creates a builder struct named MyProjectApplicationBuilder. The struct itself is empty – all the generated code is injected by the macro.

PREALLOCATED_STORAGE_SIZE – How much memory (in bytes) to pre-allocate for the structured log. 100 MB is a reasonable default.

MyProjectApplication::builder() – Creates the generated application builder. You provide the pieces you want to override, such as the clock, config override, or resource factory.

.with_log_path(...).build() – Wires everything together: creates each task by calling their new() constructors, pre-allocates all message buffers, and sets up the scheduler.

application.run_until_shutdown() – Starts the deterministic execution loop. Calls start() on all tasks, then enters the cycle loop (preprocess -> process -> postprocess for each task, in topological order), and continues until you stop the application (Ctrl+C). It consumes the initialized handle and returns a stopped handle, so the code uses stopped.clock() after the run.

application.clock() – Returns the runtime clock handle. The builder creates a default clock unless you override it with .with_clock(...), which is mainly useful in tests or simulation.

build.rs – build-time setup

v1.1 Keep the template’s complete build script:

fn main() {
    cu29_build::setup();
}

The generated Cargo.toml already includes cu29-build under [build-dependencies]. This helper prepares the interned string index used by Copper’s structured logging and the other build-time metadata. Keep its Copper source and version aligned with cu29.

Controlling the Loop Frequency

If you’ve run our project, you noticed that the output scrolls extremely fast. That’s because by default, Copper runs the task graph as fast as possible – there’s no rate limiter. On a modern machine, that can mean hundreds of thousands of cycles per second.

In this chapter, we’ll see how fast the loop actually runs and how to set a target frequency.

How fast is it running?

Let’s modify MySource in tasks.rs to print the time at each cycle. Replace the process() method:

#![allow(unused)]
fn main() {
fn process(&mut self, ctx: &CuContext, output: &mut Self::Output<'_>) -> CuResult<()> {
    debug!("Source at {}µs", ctx.now().as_micros());
    output.set_payload(MyPayload { value: 42 });
    Ok(())
}
}

Run the project and look at the timestamps:

00:00:00.5296 [Debug] Source at 529632
00:00:00.5297 [Debug] Received message: 42
00:00:00.5298 [Debug] Sink Received message: 43
00:00:00.5300 [Debug] Source at 530005
00:00:00.5301 [Debug] Received message: 42
00:00:00.5302 [Debug] Sink Received message: 43
...

The values are in microseconds. The gap between two Source at ... lines is a few hundred microseconds. Without a rate limiter, the loop runs as fast as it can – potentially thousands of cycles per second, pegging a CPU core at 100%. Way too fast for most applications – and it will peg a CPU core at 100%.

Setting a target frequency

Copper provides a simple way to rate-limit the execution loop. Add a runtime section to your copperconfig.ron:

(
    tasks: [
        (
            id: "src",
            type: "tasks::MySource",
        ),
        (
            id: "t-0",
            type: "tasks::MyTask",
        ),
        (
            id: "sink",
            type: "tasks::MySink",
        ),
    ],
    cnx: [
        (
            src: "src",
            dst: "t-0",
            msg: "crate::tasks::MyPayload",
        ),
        (
            src: "t-0",
            dst: "sink",
            msg: "crate::tasks::MyPayload",
        ),
    ],
    runtime: (
        rate_target_hz: 1,
    ),
)

The only change is the runtime section at the bottom. rate_target_hz: 1 tells Copper to target 1 cycle per second (1 Hz).

Run again:

00:00:00.0005 [Debug] Source at 510
00:00:00.0005 [Debug] Received message: 42
00:00:00.0005 [Debug] Sink Received message: 43
00:00:01.0000 [Debug] Source at 1000019
00:00:01.0001 [Debug] Received message: 42
00:00:01.0001 [Debug] Sink Received message: 43

Now each full cycle (source -> task -> sink) runs once per second.

How it works

rate_target_hz acts as a rate limiter, not a scheduler. After each complete cycle, the runtime checks how much time has elapsed. If the cycle finished faster than the target period (e.g., under 10ms for 100 Hz), the runtime waits for the remaining time. If the cycle took longer than the target period, the next cycle starts immediately – no time is wasted.

This means:

  • Your actual frequency is at most rate_target_hz.
  • If your tasks are too slow for the target, the loop runs at whatever rate it can sustain.
  • Without the runtime section, the loop runs flat-out with no pause between cycles.

For host-side applications that need tighter cadence at higher rates, Copper also ships an optional high-precision-limiter crate feature. That mode keeps the same runtime.rate_target_hz config, but the std runtime switches from pure sleep-based waiting to a hybrid sleep-then-spin limiter with absolute deadlines. It burns more CPU near each tick, so it is meant for metronomic loops, not for the default “save cycles” use case.

Difference with ROS

In ROS, each node controls its own frequency. A camera node publishes at 30 Hz, an IMU node publishes at 200 Hz, and a planner node runs at 10 Hz – all independently. Nodes are loosely coupled via topics, and each one has its own rospy.Rate() or rclcpp::Rate timer that governs how often it publishes.

Copper works differently. There is one global loop that executes the entire task graph in sequence: source -> processing -> sink, back-to-back, in topological order. Every task runs every cycle, as fast as possible. The rate_target_hz setting doesn’t make individual tasks run at different speeds – it tells the runtime how long to wait between cycles so the whole pipeline doesn’t run flat-out and peg the CPU.

ROS:    Node A (30 Hz)  ──┐
        Node B (200 Hz) ──┼── independent timers, publish on topics
        Node C (10 Hz)  ──┘

Copper: [ Source → Task → Sink ] → wait → [ Source → Task → Sink ] → wait → ...
        └──── one cycle ────────┘         └──── one cycle ────────┘
                         global rate_target_hz

The key insight: in Copper, all tasks share the same cadence. If you need different parts of your system to run at different rates (e.g., a fast inner control loop and a slow planner), you’d use separate task graphs or implement logic inside a task to skip cycles. But for most applications, a single frequency for the whole pipeline is simpler and avoids the synchronization headaches that come with multiple independent timers.

Shared reference time

v1.3 Two robots exchanging timestamped measurements need timestamps with the same meaning. A shared reference clock gives ctx.now() a named epoch and lets the runtime stop consuming measurements when its synchronization uncertainty exceeds your chosen limit. Task code still uses ordinary Copper timestamps.

See synchronization in the monitor

Use this experimental feature from the copper-rs checkout named in the edition notice. The Linux demo needs Rust, just, ptp4l, phc2sys, pmc, ethtool, and ip, plus an active Ethernet interface exposing /dev/ptp0 and read access to that device. Use a test network: the demo advertises a local PTP master on that interface. From the repository root, in an interactive terminal, run:

cd examples/cu_clock_sync
just

The recipe asks for sudo to start the PTP services it needs, then opens ConsoleMon’s CLK panel. It cleans up the services it started when you exit; existing services are reused. If services are already configured with suitable permissions, just run starts only the Copper application.

Look for Clock / reference, a shared epoch, Offset from reference, and Estimated uncertainty. The demo’s 20 ms uncertainty limit decides when the clock becomes unusable; it is not an accuracy target. A local reference sourced from the host’s own clock demonstrates synchronization without establishing absolute accuracy. Local calendar time appears only when the provider supplies a valid TAI minus UTC offset.

For another PHC device or software timestamping, follow the example’s Linux setup guide.

Configure your application’s reference

The clock-sync feature makes ctx.now() use a shared reference epoch. The provider chooses that epoch: Linux PTP adapters expose TAI, while board and GNSS providers can supply bare GPS or application-defined nanoseconds without conversion. Enable clock-sync on cu29, enable linux-phc on cu-ptp, then select an owned resource:

resources: [(
    id: "ptp",
    provider: "cu_ptp::LinuxPtpBundle",
    config: {
        "device": "/dev/ptp0",
        "management": "/var/run/ptp4l",
        "reference_error_ns": 10000,
    },
)],
runtime: (clock: (parent: "ptp.reference", max_error_ns: 100000, sample_interval_ns: 100000000)),

The resource reads a PHC already disciplined by ptp4l. Match its device, port and domain to the service. Choose a measured upstream uncertainty bound through reference_error_ns; Copper adds capture, drift and remaining phase error. Acquisition finishes before tasks start. Task code continues to stamp messages with ordinary ctx.now() and Tov::Time.

The generated Serial runtime samples references outside task process hooks and checks clock health before each iteration. Clock clones share the same monotonic curve. ctx.clock.sync_status() reports domain/session, sample age, phase, drift and estimated error. Reference loss enters bounded holdover. Exceeding the error budget or age limit fails the iteration before consumers execute; restarting acquires the reference again. This experimental API supports the Serial plan. Pipeline scheduling is rejected because correction changes must be coordinated across in-flight CopperLists.

Optional runtime fields are acquisition_timeout_ns (10000000000), sample_interval_ns (1000000000), max_age_ns (5000000000), drift_bound_ppb (100000) and max_slew_ppb (500000). The error budget can expire before the maximum age. Raw time drives cadence and deadlines independently of corrections.

Each RobotClock owns its hardware-counter calibration, including when clock-sync is disabled. clock.raw_now() returns a CuInstant on the undisciplined local timeline, while clock.now() and clock.recent() return execution timestamps as CuTime. Both use nanoseconds; intervals use CuDuration. These distinct types prevent raw readings from being used as message timestamps or execution timestamps from being used as raw deadlines. Clones share its origin and calibration; independently constructed clocks have separate raw origins. Use ctx.now() for message timestamps and raw time for reference capture and elapsed-time measurements.

CuInstant readings use an explicit clock handle. Use clock.raw_now() for counter reads, clock.busy_wait_for(duration) for a raw elapsed-time wait, and clock.busy_wait_until(raw_deadline) for a deadline from that clock or one of its clones. Raw reads and waits are available with or without synchronization. Mock clocks require their control handle to advance time while waiting. For typed readings from the control handle, use control.raw_now(). Its existing now() accessor retains the legacy view of the raw counter as CuTime; execution time comes from the associated clock.

For a captured raw reading, CuInstant::from_nanos(nanos) reconstructs an instant using the capturing clock’s origin. ClockSnapshot::at(instant) maps it to execution CuTime through the recorded epoch and rate.

For a local experiment without a PHC, LinuxSystemPtpBundle reads the system UTC clock associated with software-timestamped ptp4l. It requires the known TAI−UTC offset. A board bundle can export BoardPtp<HZ> with BSP hooks for its existing PTP service and a known counter frequency. Initialize and extend the board timer in the BSP; keep Cortex-M counter reads in the foreground.

Unified logs record each published correction in the RuntimeLifecycle stream. Enable clock-sync in recorded replay applications and matching logreaders. Replay restores curves, quality, source and epoch without starting reference I/O. Unconfigured clocks keep their local epoch and constructor APIs.

ctx.clock.origin() identifies both the advertised upstream source and the epoch of ctx.now(). For example, a GNSS receiver distributed through PTP can report ClockSource::Gnss with ClockEpoch::Tai; a provider supplying bare GPS time reports ClockSource::Gnss with ClockEpoch::Gps. Source describes provenance, while epoch identifies what timestamp zero means. Unknown provenance stays unknown. The accessor performs no counter read, allocation, reference I/O or conversion.

Board providers return source and epoch through BoardPtpHooks::origin, matching the nanoseconds returned by read. Reference observations carry this metadata with the captured time. For initialization without live synchronization, RobotClock::from_ref_time_with_origin(ns, origin) selects a named epoch; use from_ref_time_with_rtc_and_origin when supplying RTC calibration functions. An already synchronized clock keeps its epoch during resynchronization; selecting a different epoch requires a new clock.

ConsoleMon’s CLK panel shows numeric robot time, source and epoch. For TAI with a valid provider-supplied tai_minus_utc_seconds, it also shows the local timezone as an ISO 8601 date to whole seconds, including the timezone offset. The monitor performs this conversion only while drawing CLK. If the TAI minus UTC offset is unknown, local time is unavailable. GPS and application-defined values stay numeric; calendar conversion belongs to application display/debug code.

Follow the runnable Linux example for software-grandmaster and hardware-grandmaster setup. The component’s reference guide describes Linux resource parameters and the board contract.

Adding the Console Monitor

In the previous chapter we controlled the loop frequency. But how do we know how fast each task actually runs? How much time does process() take? What’s the jitter?

Copper comes with a built-in TUI (terminal user interface) monitor called CuConsoleMon that displays live statistics for every task in your pipeline – mean execution time, standard deviation, min, max, and more.

Step 1: Add the dependency

Add cu-consolemon to your Cargo.toml next to your existing Copper dependencies. The only new dependency is cu-consolemon.

If your project already uses Copper through git or local path dependencies, keep that same source style and add cu-consolemon from the same Copper release. If you use crates.io, use the same published Copper release as your existing cu29 dependency.

Step 2: Enable it in copperconfig.ron

Add a monitor section to your copperconfig.ron:

(
    tasks: [
        (
            id: "src",
            type: "tasks::MySource",
        ),
        (
            id: "t-0",
            type: "tasks::MyTask",
        ),
        (
            id: "sink",
            type: "tasks::MySink",
        ),
    ],
    cnx: [
        (
            src: "src",
            dst: "t-0",
            msg: "crate::tasks::MyPayload",
        ),
        (
            src: "t-0",
            dst: "sink",
            msg: "crate::tasks::MyPayload",
        ),
    ],
    runtime: (
        rate_target_hz: 1,
    ),
    monitor: (
        type: "cu_consolemon::CuConsoleMon",
    ),
)

Step 3: Run

cargo run

Instead of the scrolling debug output you’re used to, you’ll see a live terminal dashboard that takes over your terminal. Navigate to the latency tab (you can click on it)

The latency tab shows one row per task, plus an End2End row at the bottom for the entire pipeline. Here’s what each column means:

ColumnMeaning
TaskThe task id from your copperconfig.ron
MinFastest process() execution seen across all cycles
MaxSlowest process() execution seen (watch for spikes here)
MeanAverage process() execution time
StddevStandard deviation – how much the execution time varies. Lower is more predictable
Jitter (mean)Average difference between consecutive cycle times. Indicates how stable the timing is
Jitter (max)Worst-case jitter seen. A large value means occasional timing spikes

The End2End row measures the full pipeline latency: from the start of src to the end of sink. In our example, the entire pipeline completes in about 187 µs on average, with a worst case of 375 µs.

This is extremely useful for:

  • Verifying your frequency: Are your tasks fast enough for the target rate_target_hz?
  • Finding bottlenecks: Which task takes the most time?
  • Detecting jitter: A high Std Dev means unpredictable execution time, which could be a problem for real-time control.

Difference with ROS

In ROS 2, getting this kind of per-node timing data typically requires external tools like ros2 topic hz, ros2 topic delay, or hooking up a tracing framework. Each tool gives you one metric at a time, for one topic.

In Copper, the console monitor is built into the runtime. One line of config gives you a live dashboard of every task in the system, with no external tooling needed.

Press Ctrl+C to stop the application as usual.

The other tabs

The monitor has six tabs, accessible by pressing 1 through 6 or clicking on them.

1 - SYS: System Info

Displays your system information: OS, kernel, CPU, memory, and the Copper version. Useful for documenting what hardware a recording was made on.

2 - DAG: Task Graph

A visual representation of your task graph. Each box is a task, showing its id, Rust type, and the message types on its connections. This is your copperconfig.ron rendered live – a quick way to verify the pipeline is wired correctly.

4 - BW: Bandwidth

Shows memory and disk bandwidth consumed by the runtime. The left panel (Memory BW) reports the observed loop rate, CopperList size, and raw memory throughput. The right panel (Disk / Encoding) shows the serialized size of each CopperList, encoding efficiency, structured log bandwidth, and total disk write rate. Useful for sizing your log storage and checking that the logger can keep up.

v1.2 With LogStream configured, BW also shows a Telemetry / TX panel per destination: local submissions, budget, drops, and recovery activity. Two-way streams add receiver reports and the effective FEC repair interval. See link health and feedback for setup and interpretation.

5 - MEM: Memory Pools

Displays the status of Copper’s pre-allocated memory pools: pool ID, used vs total buffers, buffer size, handles in use, and allocation rate. In our simple project the pools are empty, but on a real robot with large messages (images, point clouds), this helps you verify that buffers are correctly sized and not exhausted.

6 - LOG: Debug Output

A scrollable view of all debug!() log output from your tasks – the same messages you’d see without the monitor, but captured inside the TUI. You can scroll through the history with hjkl or arrow keys. The bottom bar also shows keyboard shortcuts: r to reset latency statistics, q to quit.

Logging and Replaying Data

Every time you’ve run our project so far, Copper has been quietly recording everything. Look at main.rs – the generated app builder’s .with_log_path(...) call initializes a unified logger that writes to logs/my-project.copper. Every cycle, the runtime serializes every message exchanged between tasks (the CopperList) and writes it to that file.

In this chapter, we’ll explore what’s in that log file, how to read it back, and how to replay recorded data through the pipeline.

What gets logged?

Copper’s unified logger captures two kinds of data in a single .copper file:

  1. Structured text logs – Every debug!(), info!(), warn!(), and error!() call from your tasks. These are stored in an efficient binary format (not as text strings), so they’re extremely fast to write and compact on disk.

  2. CopperList data – The complete set of message payloads exchanged between tasks in each cycle. In our project, that means every MyPayload { value: 42 } from MySource and every MyPayload { value: 43 } from MyTask, along with precise timestamps.

This is different from most robotics frameworks where logging is opt-in and you have to explicitly record topics. In Copper, every message is logged by default. The runtime does this automatically as part of its execution loop – no extra code needed.

Typed logging codecs

v1.0 A task’s logging.codec or logging.codecs setting selects a typed codec for its recorded payloads, such as lossless PNG or FFV1 image compression. This lets the encoding follow the payload’s structure while keeping codec selection in RON.

Keep the logreader built for the application version that produced a recording. The codec’s representation is part of the encoded content, whose layout can change between application versions. The unified-log file and section header version describes the outer storage layout.

Step 1: Generate a log file

Make sure your project is in the state from the previous chapters, with the 1 Hz rate limiter. Here’s the copperconfig.ron for reference:

(
    tasks: [
        (
            id: "src",
            type: "tasks::MySource",
        ),
        (
            id: "t-0",
            type: "tasks::MyTask",
        ),
        (
            id: "sink",
            type: "tasks::MySink",
        ),
    ],
    cnx: [
        (
            src: "src",
            dst: "t-0",
            msg: "crate::tasks::MyPayload",
        ),
        (
            src: "t-0",
            dst: "sink",
            msg: "crate::tasks::MyPayload",
        ),
    ],
    runtime: (
        rate_target_hz: 1,
    ),
)

Note: If you still have the monitor section from the previous chapter, remove it for now. The console monitor takes over the terminal and makes it harder to see the debug output.

Run the project and let it execute for 5-10 seconds, then press Ctrl+C:

cargo run
00:00:03.9986 [Debug] Received message: 42
00:00:03.9987 [Debug] Sink Received message: 43
00:00:04.9979 [Debug] Source at 4997916
00:00:04.9980 [Debug] Received message: 42
00:00:04.9981 [Debug] Sink Received message: 43
...

After stopping, check the logs/ directory:

ls -lh logs/
-rw-r--r-- 1 user user 4.0K  logs/my-project.copper

That .copper file contains everything: every message, every timestamp, every debug line.

Note: The log path is hardcoded to "logs/my-project.copper" in main.rs. Each run overwrites the previous log file – there is no automatic rotation or timestamping. If you want to keep a log from a previous session, rename or move the file before running the project again.

Step 2: The log reader

If you look at your project, you’ll notice there’s already a file you haven’t used yet: src/logreader.rs. The project template ships with a built-in log reader. Let’s look at it:

pub mod tasks;

use cu29::prelude::*;
use cu29_export::run_cli;

// This will create the CuStampedDataSet that is specific to your copper project.
// It is used to instruct the log reader how to decode the logs.
gen_cumsgs!("copperconfig.ron");

#[cfg(feature = "logreader")]
fn main() {
    run_cli::<CuStampedDataSet>().expect("Failed to run the export CLI");
}

This is a small but powerful program. Let’s break it down:

gen_cumsgs!("copperconfig.ron") – This macro reads your task graph and generates a CuStampedDataSet type that knows the exact message types used in your pipeline. The log reader needs this to decode the binary data in the .copper file – without it, the bytes would be meaningless.

run_cli::<CuStampedDataSet>() – This is Copper’s built-in export CLI. It provides several subcommands for extracting data from .copper files. By passing the generated CuStampedDataSet type, you tell it exactly how to decode your project’s messages.

#[cfg(feature = "logreader")] – The log reader is gated behind a Cargo feature flag so its dependencies (like cu29-export) are only compiled when you actually need them.

Step 3: Extract text logs

The first thing you can do with the log reader is extract the structured text logs – the debug!() messages from your tasks. Remember, these aren’t stored as text in the .copper file; they’re stored as compact binary indices. The log reader reconstructs the human-readable text using the string index that was built at compile time.

Run:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper extract-text-log target/debug/cu29_log_index

The arguments are:

  • logs/my-project.copper – The log file to read
  • extract-text-log – The subcommand to extract text logs
  • target/debug/cu29_log_index – The path to the string index directory (generated during compilation by build.rs)

You’ll see output like:

25.501 µs [Debug]: Logger created at logs/my-project.copper.
45.903 µs [Debug]: Creating application... 
64.282 µs [Debug]: CuConfig: Reading configuration from file: copperconfig.ron
669.067 µs [Debug]: Running... starting clock: 666.866 µs.
823.766 µs [Debug]: Source at 822
870.122 µs [Debug]: Received message: 42
887.054 µs [Debug]: Sink Received message: 43
1.000 s [Debug]: Source at 1000206
1.000 s [Debug]: Received message: 42
1.000 s [Debug]: Sink Received message: 43
2.000 s [Debug]: Source at 1999631
...

This is the same output you saw scrolling by when the application was running – but reconstructed from the binary log after the fact.

Step 4: Extract CopperLists

The more interesting subcommand extracts the CopperList data – the actual message payloads from every cycle:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper extract-copperlists

Serialized CopperLists contain the cycle id and msgs. Lifecycle state is runtime bookkeeping and is omitted from binary logs, JSON, Python exports, and remote-debug snapshots. Decode messages with the producing application’s typed logreader or its embedded catalog; the file encapsulation version describes the file and section layout.

The output is JSON by default. Here’s what the first CopperList looks like:

{
  "id": 0,
  "msgs": [
    {
      "payload": {
        "value": 42
      },
      "tov": "None",
      "metadata": {
        "process_time": {
          "start": 822050,
          "end": 867803
        },
        "status_txt": ""
      }
    },
    {
      "payload": {
        "value": 43
      },
      "tov": "None",
      "metadata": {
        "process_time": {
          "start": 869282,
          "end": 885259
        },
        "status_txt": ""
      }
    }
  ]
}

Every message from every cycle is there, exactly as it was produced. Each entry in msgs corresponds to a connection in your task graph (in order: src→t-0, then t-0→sink). Along with the payload, you get:

  • metadata.process_time – The start and end timestamps (in nanoseconds) of the task’s process() call that produced this message. This is the same timing data the console monitor uses for its latency statistics.
  • tov – “Time of validity”, an optional timestamp that the task can set to indicate when the data was actually captured (useful for hardware drivers with their own clocks).
  • status_txt – An optional status string the task can set for diagnostics.

This is the raw data you’d use for offline analysis, regression testing, or replay.

Why this matters: replay

Recording data is useful for post-mortem analysis, but the real power of Copper’s logging is deterministic replay. Because every message and its timestamp is recorded, you can feed logged data back into the pipeline and reproduce the exact same execution – without any hardware.

This means you can:

  • Debug without hardware: Record a session on the real robot, then replay it on your laptop to test processing logic.
  • Regression test: Record a known-good session, then replay it after code changes to verify the pipeline still produces the same results.
  • Analyze edge cases: When your robot encounters an unusual situation, the log captures it. You can replay that exact moment over and over while you debug.

The key insight is that all downstream tasks don’t know the difference – they receive the same MyPayload messages with the same timestamps, whether they come from live hardware or a log file.

Remote debug and Time Traveler

When exported CopperLists are not enough, a replay binary can expose the log through Copper’s remote debug API. A debugger such as Time Traveler can then seek or step through the timeline, replay batches, inspect task state, query message schemas, and read structured logs from the same session.

Replay CLI and structured log queries

v1.0 Replay targets use three standard arguments:

--debug-base <zenoh-prefix>
--log-base <recorded-log>
--replay-log-base <replay-output>

Replay and remote-debug servers should be optimized. A normal Cargo dev build is unoptimized and can make seeking, stepping, payload decoding, and state inspection much slower. Copper’s generated projects therefore define this profile:

[profile.debug-optimized]
inherits = "dev"
opt-level = 3
debug = 2
debug-assertions = true

The generated just resim and just resim-debug recipes select it with --profile debug-optimized. This keeps Copper debug!() structured log entries, debug assertions, and debugger information while optimizing the replay server. Use the same profile for hand-written replay recipes rather than running the server with Cargo’s unoptimized dev profile.

Application replay binaries normally use cu29::replay::ReplayCli, or flatten ReplayArgs into a larger CLI. The most useful remote methods are:

  • nav.seek, nav.step, and nav.replay for navigation
  • timeline.get_cl and timeline.list for recorded messages and timing
  • state.inspect, state.read, and state.search for task state
  • schema.get_type and schema.get_outputs for debugger-driven rendering
  • logs.strings and logs.list for structured logs

Session leases and deferred payloads

v1.1 Each replay server grants one active debug-session lease. While the debugger is otherwise idle, it sends health.ping with the active session_id every second. Any session-scoped request also renews the lease. After 3 seconds without session activity, the server automatically closes the session so the debugger can reconnect after a crash or lost connection. A second session.open returns SessionBusy while the lease is active.

To inspect two logs or maintain two independent timeline cursors concurrently, launch two replay-server processes with different --debug-base and --replay-log-base values.

Handle-backed payloads stay lazy. A normal response contains only a compact CuHandle descriptor. When a CBOR request explicitly sets include_handle_contents: true, selected handle buffers are returned out of line in the response’s attachments array. This lets an inspector load a camera frame or point cloud on demand without copying every large buffer while it scrubs the timeline.

The debug value format preserves values that JSON alone cannot represent faithfully, including 128-bit integers, non-finite floats, and maps with non-string keys. Schema queries describe enums, maps, and fixed-capacity CuArrayVec values, and schema.get_outputs supports paging output slots and their field catalogs separately.

Linux shared-memory setup

v1.1 On Linux, the default local remote-debug transport uses a 1 GiB Zenoh shared-memory pool and requires replies of 256 KiB or larger to use SHM. Both the Copper process and the debugger need sufficient RLIMIT_MEMLOCK, and /dev/shm must have room for the pool. Configure the launching shell before starting either process:

sudo prlimit --pid $$ --memlock=2147483648:2147483648
ulimit -l 2097152
df -h /dev/shm

The ulimit -l value is in KiB. For systemd use LimitMEMLOCK=2G; for Docker configure both --ulimit memlock=2147483648:2147483648 and --shm-size=2g.

Copper checks these resources at startup and refuses to silently copy a large reply over the Unix socket when SHM allocation fails. Tests can use a smaller explicit RemoteDebugShmConfig on both server and client. Non-Linux platforms retain the previous 4 MiB pool and 3 KiB threshold without Linux-specific preflight.

How the unified logger works under the hood

Copper’s logger is designed for zero-impact logging on the critical path. Here’s how:

  1. Pre-allocated memory slabs – At startup, .with_log_path(...) allocates a large contiguous block of memory (controlled by PREALLOCATED_STORAGE_SIZE – 100 MB in our project). CopperLists are written into this pre-allocated buffer without any dynamic allocation.

  2. Binary serialization – Messages are serialized using bincode, not formatted as text. This is why your payloads need the Encode / Decode derives. Binary serialization is orders of magnitude faster than format!() or serde_json.

  3. Memory-mapped I/O – The pre-allocated slabs are memory-mapped to the .copper file. The OS handles flushing to disk asynchronously, so the robot’s critical path never blocks on disk I/O.

  4. Structured text logging – Even debug!() calls don’t format strings at runtime. Instead, Copper stores a compact index and the raw values. The actual string formatting happens only when you read the log – not when you write it. This is why the build.rs sets up LOG_INDEX_DIR – it’s building a string table at compile time.

This means logging in Copper is almost free on the hot path. You can log everything without worrying about performance – which is exactly why it logs everything by default.

Logging and logstream extensibility

To use a custom payload, codec, storage backend, or packet transport, choose the extension point below. This map follows task messages and state snapshots through the APIs in this edition’s 1.3.0-dev checkout.

Task messages and state flowing through logging and logstream extension points

Purple means a Rust type selected in RON: you can name your own implementation. Teal means an implementation supplied in Rust. Blue marks public APIs or records; gray marks core machinery. A setting that only tunes an algorithm does not make that algorithm replaceable.

Replace or extendWhere to connect your implementation
Tasktasks[].type in copperconfig.ron: name your CuTask or CuSrcTask implementation.
Message payloadcnx[].msg: name your payload type. The CuMsg envelope stays the same.
State snapshotImplement Freezable::freeze and thaw on your task. Generated capture calls these methods.
Payload codecDefine a CuLogCodec<Payload> type in logging.codecs[].type, give it an id, then select that id with tasks[].logging.codec or the per-message logging.codecs map on the task.
Logger and section storageSupply your UnifiedLogWrite<S> and SectionStorage implementations through the generated app’s builder.with_logger::<S, L>(logger).
Byte storage backendImplement ByteStorage and ByteRegion, construct ByteLogger with that backend, and inject it through with_logger.
Packet transportImplement CuStreamTx; select its type and resource through log_streaming.destinations[].transport.{type, resource}. The resource comes from a bundle declared under resources. Rust construction can instead inject the transport with builder.with_logstream(transport, config).
Semantic record sinkImplement WriteStream<Record>. Attaching an arbitrary sink to generated app output requires changing the output wiring; there is no RON sink-type slot.

The payload codec changes payload encoding inside the record; bincode remains the serialization framework. Likewise, logstream’s link, fec, and recovery_interval settings tune the built-in sender rather than select a custom sender algorithm. Replacing those mechanisms requires a core change.

The runtime owns message slots, snapshot order, output handoff, and buffer reuse. Local storage and logstream branch from completed records, so a streaming transport does not need to read the robot’s slab files. For transport wiring and sender settings, continue with Streaming Logs and Live Telemetry and its reference.

Controlling what gets logged

Sometimes you don’t want to log everything. A high-frequency sensor producing megabytes per second can fill up your log storage quickly. Copper provides two ways to control this:

Per-task logging control

In copperconfig.ron, you can disable logging for specific tasks:

(
    id: "fast-sensor",
    type: "tasks::HighRateSensor",
    logging: (enabled: false),
),

This stops the runtime from recording that task’s output messages in the CopperList log. The task still runs normally – it just doesn’t contribute to the log file.

Global logging settings

The logging section in copperconfig.ron lets you tune the logger globally:

logging: (
    slab_size_mib: 1024,
    section_size_mib: 100,
),

Task-state keyframes

v1.2 Task-state keyframes can be disabled independently of CopperList and structured logging:

logging: (
    enable_keyframe_logging: false,
),

This is a compile-time application property. The generated runtime omits keyframe capture code when it is disabled, which is useful for applications whose background workers may still be active at a CopperList boundary. Replay then has message history but no task-state snapshots to restore.

Streaming to a ground station

v1.2 To collect a native archive on another machine while the robot runs, continue to Streaming Logs and Live Telemetry. The guide covers UDP setup, structured robot logs, live task reconstruction, optional receiver feedback, and recovery after packet loss. Per-task logging controls also determine which inputs are available for reconstruction.

Difference with ROS

In ROS, data recording is a separate tool: rosbag2. You start a ros2 bag record process alongside your running nodes, tell it which topics to subscribe to, and it saves messages into a SQLite database or MCAP file. Replay is done with ros2 bag play, which republishes the messages on the same topics.

ROS:
  Run:    ros2 launch my_robot.launch.py
  Record: ros2 bag record /camera /imu /cmd_vel       ← separate process
  Replay: ros2 bag play my_bag/                        ← republishes on topics

Copper:
  Run:    cargo run                                     ← logging is automatic
  Read:   cargo run --features logreader --bin my-project-logreader ...
  Replay: feed CopperLists back into the pipeline       ← deterministic

Key differences:

ROS 2 (rosbag2)Copper (unified logger)
Opt-in vs automaticYou must explicitly record topicsEverything is logged by default
Separate processros2 bag record runs alongsideBuilt into the runtime – zero config
FormatSQLite / MCAPCustom binary .copper format
Performance impactAdds subscriber overhead per topicNear-zero – pre-allocated, memory-mapped
Replay mechanismRepublishes on topicsFeed CopperLists directly into tasks
DeterministicTiming depends on DDS, not guaranteedTimestamps are recorded, replay is deterministic
Text loggingSeparate (rosout, spdlog)Unified – text and data in one file

The biggest philosophical difference: in ROS, recording is something you do. In Copper, recording is something that happens. You don’t configure it, you don’t start it, you don’t choose which topics to record. The runtime records everything, always. You only need to decide what to exclude (via logging: (enabled: false)) if storage is a concern.

This “record everything by default” approach is what makes Copper’s deterministic replay possible. Since every message and every timestamp is captured automatically, you can always go back and reproduce any moment of your robot’s execution.

Keep recordings across restarts and bound disk use

The normal .with_log_path(...) setup creates a fresh recording.

v1.2 Use append when you want another run of the same application in that recording.

v1.3 Use rollover for a robot that must keep recording within a fixed storage budget, retaining recent history as older sections are reclaimed.

This exercise uses the single-crate project from The Remaining Files and Running. Keep its src/tasks.rs and task graph, and remove any rate_target_hz setting for the longer rollover exercise below.

1. Create a short recording

In copperconfig.ron, add or replace the top-level logging section:

logging: (
    slab_size_mib: 4,
    section_size_mib: 1,
    enable_keyframe_logging: false,
),

This exercise uses tasks with no changing internal state and disables keyframe capture to fit the small budget. For a stateful replay application, keep keyframes and budget for their stream as well.

Replace src/main.rs with this complete entry point:

pub mod tasks;

use cu29::prelude::*;
use std::path::Path;
use std::sync::{Arc, Mutex};

#[copper_runtime(config = "copperconfig.ron")]
struct MyProjectApplication {}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    std::fs::create_dir_all("logs")?;
    let logger = UnifiedLoggerBuilder::new()
        .write(true)
        .create(true)
        .file_base_name(Path::new("logs/retained.copper"))
        .preallocated_size(4 * 1024 * 1024)
        .append(false)
        .build()?;
    let UnifiedLogger::Write(logger) = logger else {
        return Err("Expected a write-capable logger".into());
    };
    let app = MyProjectApplication::builder()
        .with_unified_logger(Arc::new(Mutex::new(logger)))
        .build()?;
    let mut running = app.start().expect("start application");
    for _ in 0..10 {
        running.run_one_iteration()?;
    }
    running.stop().expect("stop application");
    Ok(())
}

Run from the project root:

cargo run

It records ten cycles and exits normally. Normal teardown closes the streams and logger so the next process can append. Keep the entire retained.copper slab family.

2. Append another run

v1.2 Change .append(false) to .append(true) in that entry point, then run it again:

cargo run

v1.3 Use the logreader’s run-selection commands to inspect the appended recording:

cargo run --features logreader --bin my-project-logreader -- logs/retained.copper list-runs
cargo run --features logreader --bin my-project-logreader -- logs/retained.copper --run 1 extract-copperlists

These commands use the my-project-logreader binary from the book’s first project; use your generated binary name if you named it differently. The list now contains two runs. Selecting run 1 exports the second set of ten cycles, whose CopperList IDs start again for the new construction.

Append requires a cleanly closed log, the same slab allocation size, and matching application metadata and configuration. With self-describing logs it also checks the complete shared catalog. A crash or metadata mismatch rejects append. Use a fresh path when you change the application, mission set, or configuration.

3. Bound the recent history

v1.3 Add .rollover(8 * 1024 * 1024) before .build() on the logger builder and change the loop to for _ in 0..300_000. Keep .append(true) to continue the recording already created. Run a release build to avoid printing every debug line while generating enough cycles to fill the budget:

cargo run --release
cargo run --features logreader --bin my-project-logreader -- logs/retained.copper list-runs
cargo run --features logreader --bin my-project-logreader -- logs/retained.copper fsck

fsck shows a retained CopperList ID range ending at 299999, with the earlier IDs missing after rollover. Those reported holes are the intentionally reclaimed history. The start time can also be unknown once its lifecycle event is gone.

The capacity is eight MiB across the logical log; slabs are its backing files. Rollover reclaims the oldest closed data sections as more space is needed. Application metadata and any payload catalog remain available. Those static sections also consume capacity, so leave room for them and the active streams; one MiB sections fit the budget used here.

Run indices follow the first retained section of each run. After rollover, list runs again before choosing --run: old runs can disappear, and start times can become unknown when lifecycle events have rotated out. The reader still knows the mission of each surviving data section. Read order follows the log’s section links, rather than the numerical order of slab filenames.

Rollover can discard earlier samples and keyframes, so archive a recording you need for full-history analysis or replay before they are reclaimed. If an open section prevents reclamation, the logger returns a space error; it does not silently overwrite a write in progress.

Continue with Self-Describing Logs to share these recordings with readers who do not have your application’s payload types.

Self-Describing Logs

v1.3 Suppose you receive a robot’s .copper recording and want to plot wheel speed. With an application-specific logreader, you need the Rust payload types used by that application. A self-describing log carries a catalog that tells an offline reader how to decode those payloads, including field names and storage units. You can inspect the recording with Copper’s standalone tools and export the samples without building the robot’s logreader.

Use this feature when you share recordings with another team, keep logs for later analysis, or collect recordings from several applications. This feature is experimental; use the checkout named in the edition notice for the examples below.

1. Try a complete recording

From the root of a copper-rs checkout, enter the wheel-sensor example:

cd examples/cu_self_describing_logs
just

This records ten cycles in logs/wheel.copper, displays the embedded catalog, and checks the recording with fsck --deep. The graph is a wheel source, a filter, and a sink. Each cycle captures the source and filter payloads; the sink contributes its execution metadata.

Look at the catalog’s WheelSample fields. Alongside encoder ticks it describes distance, speed, a timestamp, acceleration, temperatures, and sensor state. The reader obtains those names and types from the recording itself.

Export the captured cycles from the same directory:

just extract --export-format jsonl > samples.jsonl

Each JSONL line is one CopperList, with its id and ordered msgs. A message contains its payload, timestamps, and capture information. Check that samples.jsonl has ten lines, with source and filtered wheel samples in each cycle. Exporting Data continues with catalog dumps, CSV, and payload-size diagnostics using this same recording.

2. Enable it in your application

The project templates expose an application feature. For an existing app, add this forwarding entry to its Cargo.toml:

[features]
self-describing-logs = ["cu29/self-describing-logs"]

Keep the usual build script calling cu29_build::setup() and your ordinary builder:

let app = App::builder()
    .with_log_path("logs/app.copper", Some(32 * 1024 * 1024))?
    .build()?;

Run your application’s recording binary with the feature enabled:

cargo run --features self-describing-logs

Copper builds and saves the catalog during application construction, before initializing resources. It describes the output types declared in your RON graph, including types from dependency crates. One catalog covers every compiled mission and records each mission’s output-slot order.

Payload capture still follows your logging configuration. Keep enable_task_logging: true and enable logging on the outputs you want to inspect. A catalog describes a type; a task with logging: (enabled: false) does not contribute its payload bytes to the recording.

3. Describe your payloads

Ordinary encoding derives supply the descriptions when the feature is enabled. For example, a wheel payload can use Copper’s units directly:

use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use cu29::units::si::f32::{Length, Velocity};

#[derive(Clone, Debug, Default, Serialize, Deserialize, Encode, Decode, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct WheelSample {
    pub ticks: u32,
    pub distance: Length,
    pub speed: Velocity,
    pub timestamp: CuTime,
}

The catalog follows nested fields and enum variants automatically. A reusable payload crate can add #[bincode(describe)] to provide descriptions even when the recording feature is disabled.

Units tell you how to interpret the stored value. A length created from centimetres is stored in metres, velocity uses m·s⁻¹, and CuTime stores nanoseconds. Use the catalog’s storage unit when plotting or converting exported numbers. The unit used to construct a Rust quantity does not change its storage unit.

If you implement Encode by hand, also implement ValueDecode with a recipe that matches the bytes your encoder writes. For an Orientation wrapper whose encoder writes exactly an array of four f32 values:

impl cu29::bincode::ValueDecode for Orientation {
    const DECODE: &'static cu29::bincode::ValueDecodeSpec =
        <[f32; 4] as cu29::bincode::ValueDecode>::DECODE;
}

Check that the recipe decodes the complete encoded value, including consecutive values. A custom logging codec also needs a recipe for its recorded representation. If startup rejects a task/type/codec combination, check that encoder’s description.

4. Read your own recording

From the copper-rs repository root, pass the recording’s base path to the standalone reader. Replace logs/app.copper below with your recording’s path:

just logextract logs/app.copper catalog
just logextract logs/app.copper extract-copperlists --export-format jsonl > samples.jsonl
just logextract logs/app.copper fsck --deep

just logextract builds the standalone reader with catalog support. Keep the complete slab family with the recording. Structured text reconstruction still uses the producing build’s cu29_log_index.

For recordings containing several runs and for detailed validation reports, continue to Exporting Data. For dictionary-based analysis, see Python Support.

Recording on an embedded target

The producer feature supports no_std. Schema serialization and compression finish at startup using bounded working memory, without allocating a complete catalog buffer. The recording loop keeps its ordinary native encoding pass. Your logger backend still determines its own storage allocation requirements.

The startup traversal supports up to 256 distinct reachable native types, including nested field types. A graph exceeding that limit returns a startup error. Host-side catalog readers enable cu29/decode-catalog and use std to allocate decoded value trees; cu29-export/self-describing-logs selects that reader feature automatically.

Append reuses a matching catalog, and rollover retains it so surviving samples remain decodable. See recording retention for the storage budget and the history that rollover can reclaim.

The wheel example provides the complete application. The format reference describes the encoding for readers that implement their own tooling.

Exporting Data to Other Formats

In the logging chapter, we learned how Copper records everything into .copper files and how the log reader can extract text logs and CopperList data as JSON. But JSON is just the beginning. Real-world workflows often need data in other formats – CSV for spreadsheets and data science tools, MCAP for visualization in Foxglove, or statistics JSON for offline performance analysis.

The log reader CLI you already have supports all of these. In this chapter, we’ll explore every export format and diagnostic tool available in cu29-export.

Choose a reader

For the wheel recording from the previous chapter, use the standalone just logextract reader or the example’s just recipes. For your first project’s log, use the matching src/logreader.rs binary from Logging and Replaying Data. The following sections state the starting directory for each command.

Inspecting a self-describing recording

v1.3 Continue from the ten-cycle wheel recording in Self-Describing Logs. That chapter covers recording, payload descriptions, storage units, and your first JSONL export. Here we use the existing log to export the catalog and inspect storage costs.

Export the catalog

From the copper-rs repository root, enter the example directory you used earlier:

cd examples/cu_self_describing_logs
just catalog --export-format ron > catalog.ron
just catalog --export-format json > catalog.json

These files include the complete field descriptions, mission output-slot maps, and recorded configuration. Use them when another tool needs the descriptions.

For your own log, run just logextract <base-path> catalog --export-format json from the repository root, substituting its base path. For multiple runs, see Selecting a recorded run.

Interpret missing payloads

When inspecting JSONL exports, use capture status to distinguish an absent payload from one excluded by logging. Compact recordings preserve that information; flat recordings can lose the original presence information when capture is suppressed. Each exported cycle retains its slot identity, payload, time of validity, and metadata.

Use an existing application logreader

App logreaders use their native typed decoder by default. If the logreader’s cu29-export dependency enables self-describing-logs, run this from the application directory to select the catalog:

logreader logs/robot.copper extract-copperlists --decoder catalog

Structured text still requires the producing build’s string index. For direct Python access to the catalog and decoded dictionaries, follow Catalog-Based Offline Analysis.

Selecting a recorded run

v1.3 A unified log can contain several runs, for example when you append a restart or construct another mission. Each run has its own CopperList sequence and clock. Use the application’s built logreader to find the run you want:

logreader logs/robot.copper list-runs
logreader logs/robot.copper --run 1 fsck --dump-runtime-lifecycle
logreader logs/robot.copper --run 1 extract-copperlists

Run these commands from the application directory. Here logreader means its built logreader executable. With the standalone reader, use just logextract <base-path> list-runs from the copper-rs repository root, then put --run N before the subcommand. Indices are zero-based and follow the first retained section of each run. They differ from instance_id, which can repeat across process restarts. list-runs shows the mission, application, start time when available, and whether shutdown completed. After rollover removes startup events, the retained sections still identify their run and mission; the missing timestamps remain unknown.

A single-run log is selected automatically. Use --run for a multi-run log, including when extracting text, computing statistics, or exporting MCAP:

logreader logs/robot.copper --run 1 extract-text-log target/debug/cu29_log_index
logreader logs/robot.copper --run 1 log-stats --output run-1.json
logreader logs/robot.copper --run 1 export-mcap --output run-1.mcap

The selected run supplies the recorded configuration and mission. Statistics use those by default; --config and --mission override them. Use an application logreader and string index from the producing build.

CSV export

We already saw how extract-copperlists outputs JSON by default. But if you’re working with spreadsheets, pandas, or any data analysis tool, CSV is often more convenient.

In the wheel example directory, export the recording you already made:

just extract --export-format csv > samples.csv

For your first project’s typed logreader, return to that project directory. The --export-format flag (or -e for short) switches the output format:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper extract-copperlists --export-format csv

The output looks like this:

id, src_time, src_tov, src,, t_0_time, t_0_tov, t_0,, sink_time, sink_tov, sink,
0, [672.083 µs – 734.327 µs], None, {"value":42}, [735.631 µs – 750.445 µs], None, {"value":43}
1, [1.000 s – 1.000 s], None, {"value":42}, [1.000 s – 1.000 s], None, {"value":43}
2, [2.000 s – 2.000 s], None, {"value":42}, [2.000 s – 2.000 s], None, {"value":43}
...

Each row is one CopperList (one execution cycle). The columns are:

  • id – The CopperList sequence number.
  • <task>_time – The process time range (start-end in nanoseconds) for that task’s process() call.
  • <task>_tov – The time of validity, if the task set one.
  • <task> – The payload as inline JSON.

The column headers are generated from your task IDs, so they’ll match whatever you named your tasks in copperconfig.ron.

Tip: You can redirect the output to a file and open it in any spreadsheet application or load it into a pandas DataFrame:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper extract-copperlists -e csv > data.csv
import pandas as pd
df = pd.read_csv("data.csv")

Checking log integrity with fsck

Before doing any analysis on a log file, it’s good practice to verify its integrity. The fsck subcommand reads the entire log and checks that all sections are well-formed:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper fsck
The log checked out OK.
        === Statistics ===
  Total time       -> 1.999 s
  Total used size  -> 328 bytes
  Logging rate     -> 0.00 MiB/s (effective)

  # of CL          -> 2
  CL rate          -> 1.00 Hz
  CL total size    -> 129 bytes

  # of Keyframes   -> 1
  KF rate          -> 0.50 Hz
  KF total size    -> 7 bytes

  # of SL entries  -> 13
  SL total size    -> 192 bytes

Even without the verbose flag, fsck gives you a useful summary:

  • Total time – How long the recording lasted.
  • CL rate – How many CopperLists per second were logged (should match your rate_target_hz).
  • Keyframes – Periodic snapshots of frozen task state, used for seeking during replay.
  • SL entries – The number of structured log lines (debug!(), info!(), etc.).

Verbose mode

Add -v for section-by-section details, or -vv for maximum verbosity:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper fsck -v

This prints every section header as it’s read, along with the CopperList ID ranges and time ranges within each section. It’s invaluable for debugging corrupted logs – you can see exactly where the corruption starts.

Deep validation and payload storage costs

v1.3 For the wheel recording, return to examples/cu_self_describing_logs in the copper-rs checkout and run:

just fsck

The example’s fsck recipe enables deep validation. Expect ten decoded CopperLists and twenty captured payloads: one source and one filtered sample per cycle. Invalid or truncated data returns an error with its location. Keyframe envelopes are checked too; replay still needs the application’s decoder for frozen task state.

The report includes captured-payload counts and bytes. Its table groups payloads by task and message type, sorted by total bytes, with each group’s share, mean size, and size range. Use it to find the outputs consuming most of your storage before reducing capture. These sizes count encoded payload bytes, excluding CopperList metadata.

The catalog’s compressed size includes its catalog headers, continuation framing, and integrity footer; the decompressed size is the encoded catalog body. This shows startup description overhead separately from the samples accumulated during the run.

Log statistics

The log-stats subcommand computes per-edge statistics from the log and writes them as a JSON file. This is useful for offline analysis of your task graph’s performance:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper log-stats --output stats.json --config copperconfig.ron

This produces a stats.json file with detailed statistics for every edge in your task graph:

{
  "schema_version": 1,
  "config_signature": "fnv1a64:166f6eae65f80d15",
  "mission": null,
  "edges": [
    {
      "src": "src",
      "dst": "sink",
      "msg": "crate::tasks::MyPayload",
      "samples": 3,
      "none_samples": 0,
      "valid_time_samples": 3,
      "total_raw_bytes": 12,
      "avg_raw_bytes": 4.0,
      "rate_hz": 1.0004343150476716,
      "throughput_bytes_per_sec": 6.002605890286029
    },
    {
      "src": "src",
      "dst": "t-0",
      "msg": "crate::tasks::MyPayload",
      "samples": 3,
      "none_samples": 0,
      "valid_time_samples": 3,
      "total_raw_bytes": 12,
      "avg_raw_bytes": 4.0,
      "rate_hz": 1.0004343150476716,
      "throughput_bytes_per_sec": 6.002605890286029
    },
    {
      "src": "t-0",
      "dst": "sink",
      "msg": "crate::tasks::MyPayload",
      "samples": 3,
      "none_samples": 0,
      "valid_time_samples": 3,
      "total_raw_bytes": 12,
      "avg_raw_bytes": 4.0,
      "rate_hz": 1.0004283749265024,
      "throughput_bytes_per_sec": 6.002570249559014
    }
  ],
  "perf": {
    "samples": 3,
    "valid_time_samples": 3,
    "end_to_end": {
      "min_ns": 74413,
      "max_ns": 178994,
      "mean_ns": 114416.0,
      "stddev_ns": 46095.0
    },
    "jitter": {
      "min_ns": 15429,
      "max_ns": 104581,
      "mean_ns": 60005.0,
      "stddev_ns": 44576.0
    }
  }
}

For each edge in the graph, you get:

  • samples – Total number of CopperLists processed.
  • none_samples – How many times the payload was None (the task produced no output for that cycle).
  • rate_hz – The measured message rate on this edge.
  • avg_raw_bytes – Average payload size in bytes.
  • throughput_bytes_per_sec – Sustained data throughput on this edge.

The perf section gives you pipeline-level timing:

  • end_to_end – Latency from the first task’s process() start to the last task’s process() end, across the full pipeline.
  • jitter – Variation in that end-to-end latency between consecutive cycles.

Using with missions

If your project uses missions, pass the --mission flag to select which mission’s graph to use for the edge mapping:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper log-stats --config copperconfig.ron --mission normal

MCAP export for Foxglove

MCAP is an open-source container format designed for multimodal robotics data. It’s the native format of Foxglove, a powerful web-based visualization tool for robotics. Exporting your Copper logs to MCAP lets you visualize your pipeline’s data in Foxglove’s timeline, plot panels, and 3D views.

Enabling the MCAP feature

MCAP export requires an additional feature flag. Add mcap to your logreader’s feature dependencies in Cargo.toml:

[features]
logreader = ["dep:cu29-export"]
logreader-mcap = ["logreader", "cu29-export/mcap"]

And make sure your existing cu29-export dependency enables the "mcap" feature.

Note: If your project already depends on cu29-export via git, a local path, or crates.io, keep that same source and Copper release and just add the "mcap" feature.

Automatic payload schemas

v1.2 gen_cumsgs! records the concrete payload type for every output slot. The MCAP exporter uses that generated metadata to build each JSON Schema automatically, including when one task has multiple output types. Your logreader does not need a task-ID-to-schema table:

pub mod tasks;

use cu29::prelude::*;
use cu29_export::run_cli;

gen_cumsgs!("copperconfig.ron");

fn main() {
    run_cli::<CuStampedDataSet>().expect("Failed to run the export CLI");
}

For ordinary derived serialization, reflection is enough. If a reusable payload has a custom Serialize implementation whose wire shape differs from its reflected fields, the payload crate implements SerializedPayloadSchema beside that serializer and adds SerializedPayloadSchema to its #[reflect(...)] attribute. That one payload-owned definition is then discovered by every logreader automatically. Application authors do not repeat the schema or maintain per-task matches.

Exporting to MCAP

Once the feature is enabled, a new export-mcap subcommand becomes available:

cargo run --features logreader-mcap --bin my-project-logreader -- \
    logs/my-project.copper export-mcap --output data.mcap
Exporting copperlists to MCAP format: data.mcap
MCAP Export: 5 CopperLists → 10 messages, 2 channels, 2 schemas

The export creates one MCAP channel per task output (named /<task-id>), and one schema per message type. Schemas are generated automatically at compile time using serde-reflection – the exporter introspects your Rust types and produces JSON Schema definitions that Foxglove can use to understand the data structure.

Each MCAP message contains:

  • payload – Your task’s output, serialized as JSON.
  • tov – The time of validity.
  • process_time – Start and end timestamps of the task’s process() call.
  • status_txt – Any status text the task set.

Inspecting MCAP files

The mcap-info subcommand lets you inspect an MCAP file without opening Foxglove:

cargo run --features logreader-mcap --bin my-project-logreader -- \
    logs/my-project.copper mcap-info data.mcap
=== MCAP File Info ===
File: data.mcap
Size: 2617 bytes (0.00 MB)

=== Statistics ===
Total messages: 9
Channels: 3
Schemas: 3

=== Channels ===
  /sink [json] (schema: copper.sink): 3 messages
  /src [json] (schema: copper.src): 3 messages
  /t_0 [json] (schema: copper.t_0): 3 messages

=== Schemas ===
Schema: copper.sink (encoding: jsonschema)

Schema: copper.src (encoding: jsonschema)

Schema: copper.t_0 (encoding: jsonschema)

Add --schemas to print the full JSON Schema for each channel, or -n 3 to show the first 3 sample messages per channel:

cargo run --features logreader-mcap --bin my-project-logreader -- \
    logs/my-project.copper mcap-info data.mcap --schemas -n 1

Opening in Foxglove

Once you have an .mcap file, open it in Foxglove:

  1. Go to app.foxglove.dev (or use the desktop app).
  2. Click “Open local file” and select your .mcap file.
  3. Your channels (/src, /t-0, etc.) appear in the sidebar.
  4. Add a Raw Messages panel to see the JSON payloads, or a Plot panel to graph numeric fields over time.

Because the MCAP file includes JSON Schema definitions, Foxglove knows the structure of your messages and can offer autocomplete for field names in plots.

Foxglove has plenty of tutorials on how to set up the Panels. Here is a basic UI showing our data with plots and raw messages:

Summary of subcommands

Here’s a quick reference for all the log reader subcommands:

SubcommandWhat it doesFeature needed
extract-text-log <index>Reconstructs human-readable text logs–
extract-copperlistsDumps CopperList data (JSON or CSV)–
extract-copperlists -e csvDumps CopperList data as CSV–
fsckChecks log integrity and prints statistics–
log-statsComputes per-edge statistics as JSON–
export-mcap --output f.mcapExports to MCAP format for Foxglovemcap
mcap-info <file>Inspects an MCAP filemcap

Streaming Logs and Live Telemetry

v1.2 LogStream sends your robot’s execution log to a ground station while the robot runs. The ground station records a native .copper archive, displays typed task outputs and structured logs, and can run selected tasks locally to reconstruct outputs you chose not to transmit.

This chapter takes you from the runnable UDP demo to your own sender and ground application. Use matching application code, task graph, Copper revision, and encoding features on both machines. Keep the producing build’s cu29_log_index with its recordings so you can render structured logs later.

Robot tasks ──→ onboard .copper log
     │
     └──→ LogStream worker ──→ UDP or framed serial ──→ received .copper archive
                                                           │
                                                      live twin
                                                           │
                                                  your UI or analysis

Forward error correction (FEC) adds repair packets that can recover lost data within configured bounds. Longer outages leave explicit gaps. A keyframe saves task state; a recovery point binds that keyframe to the stream manifest and a CopperList boundary so the receiver can resume safely. Starting late does not retrieve expired history.

For custom transports, codecs, and storage backends, see the logging and logstream extensibility map.

1. Try the UDP demo

Use the source checkout named in the edition notice. The demo requires the checkout’s Rust toolchain, Just, and Python 3. From that checkout:

cd examples/cu_logstream_demo
just

This starts separate sender and receiver processes, compares the received archive against the onboard recording, checks its integrity, and runs offline replay. Each automated run prints a fresh output directory under the example’s logs/.

For the interactive view, run these in separate terminals, both in the example directory. Start telemetry first:

# Terminal 1: ground station
just telemetry
# Terminal 2: robot, with Copper's console monitor
just sender

The graph is encoders → kinematics. Joint angles cross the link; the ground station runs the same kinematics task to calculate the arm pose. The robot arm is labeled reconstructed locally; payload not transmitted. The robot log pane shows simulated encoder temperature and supply-voltage diagnostics sent as structured logs.

ControlAction
1, 2, TabSelect Live, Health, or cycle between them.
SpacePause both display readers; recording continues.
Arrow keys or hjklScroll Health details.
q, Escape, Ctrl-CClose telemetry and finalize its archive.

The sender runs for about a minute. Telemetry stays open after it finishes. Pause the view and resume to see display misses without interrupting recording.

The manual demo commands replace recordings at their chosen base paths. Supply new paths to retain previous runs:

just telemetry 0.0.0.0:7447 logs/ground-session-2.copper
# On the robot; replace this example address with the ground station's address:
just sender 192.168.1.20:7447 logs/robot-session-2.copper

Use just receiver 0.0.0.0:7447 logs/received-session-2.copper for headless recording. This demo receiver exits after one second without a datagram, or fails if no traffic arrives within 15 seconds; start the sender promptly. These timeout and overwrite behaviors belong to the demo, not the generated twin API.

2. Configure your robot’s sender

Enable the logstream feature on your existing cu29 dependency and add cu29-logstream-udp from the same Copper revision. Ground code that directly uses LogStream types also needs cu29-logstream with its std feature. Follow the demo Cargo manifest for a workspace example; keep your application’s existing dependency source.

Add the following top-level blocks to your copperconfig.ron, alongside your existing tasks and cnx. These are the demo’s working UDP settings; change remote_addr for your ground station:

    logging: (
        enable_task_logging: true,
        enable_keyframe_logging: true,
        keyframe_interval: 16,
        slab_size_mib: 16,
        section_size_mib: 1,
        copperlist_count: 8,
    ),
    resources: [
        (
            id: "network",
            provider: "cu29_logstream_udp::CuUdpLogStreamResources",
            config: {
                "bind_addr": "127.0.0.1:0",
                "remote_addr": "127.0.0.1:7447",
                "send_buffer_bytes": 262144,
                "ttl": 1,
            },
        ),
    ],
    log_streaming: (
        destinations: [
            (
                id: "ground",
                transport: (
                    type: "cu29_logstream_udp::CuUdpLogStreamTx",
                    resource: "network.tx",
                ),
                link: (
                    mtu_bytes: 1200,
                    bitrate_bps: 2000000,
                    memory_budget_kib: 576,
                    max_latency_ms: 250,
                    burst_packets: 8,
                ),
                fec: (
                    continuous: (
                        field: Gf256,
                        window_symbols: 64,
                        repair_every_source_symbols: 4,
                        repair_density: Full,
                    ),
                    objects: (
                        max_object_bytes: 65536,
                        repair_symbols_per_block: 2,
                    ),
                ),
                recovery_interval: 16,
                max_record_bytes: 4096,
            ),
        ],
    ),

Keep your usual application builder, local log path, clock, and execution loop. The generated runtime wires the configured transport and sender workers. The demo’s mock clock, impairment wrapper, and fixed iteration loop are test harness code and are not needed in your application.

network.tx is a resource binding, not a task-graph edge. The graph renderer shows it in the resource table as system: logstream (ground).

SettingHow to choose it
mtu_bytesMaximum Copper packet size accepted by your carrier; 1200 is the demo profile.
bitrate_bpsBudget for all Copper packet bytes, including repairs, recovery, and structured logs. Leave room for UDP/IP or serial overhead.
memory_budget_kibSender buffer budget. The demo needs 576 KiB; larger records need more storage. This is not a total process-memory limit.
max_latency_msExpiry deadline for ordinary queued data, measured from encoded-record admission.
burst_packetsAllowed transmission burst; reduce it for a slow carrier.
window_symbolsContinuous FEC recovery window; the generated sender supports at most 64 symbols, and receiver limits also apply.
repair_every_source_symbolsOne continuous repair after this many source symbols. Lower means more repair traffic.
max_object_bytesFinite-object bound, including recovery data; it must fit the sender’s supported RaptorQ geometry.
max_record_bytesMaximum complete CopperList record, including its record header.
recovery_intervalRecovery boundaries in CopperLists, not milliseconds. Must be a nonzero multiple of logging.keyframe_interval.

For example, keyframes every 20 lists and recovery points every 100 lists select recovery boundaries at IDs 0, 100, 200, and so on. Streaming needs a nonzero keyframe interval even if you disable local keyframe recording. Keeping local keyframes, as above, also makes the onboard recording useful for stateful replay.

Start with the demo profile and captured task outputs. Measure actual transmission and drop counters before reducing bandwidth or adding reconstruction.

3. Start a ground-side twin

Build a simulation application from the same graph and task types. The generated twin builder owns receiving, archive writing, recovery, and replay; you supply a packet receiver and consume the resulting frames. This integration sketch uses Ground as your generated application type:

use cu29::prelude::*;
use cu29_logstream_udp::CuUdpLogStreamConfig;
use std::time::Duration;

#[copper_runtime(config = "copperconfig.ron", sim_mode = true)]
struct Ground {}

// Inside your fallible ground application entrypoint:
let (_, rx) = CuUdpLogStreamConfig::new("0.0.0.0:7447".parse()?).open()?;
let (mut twin, mut frames) = Ground::twin(rx)
    .with_log_path("logs/received-session-1.copper")
    .spawn()?;

// Repeat this on your UI or analysis thread while the application is running:
frames.wait_timeout(Duration::from_millis(50));
let status = frames.status();
while let Some(update) = frames.try_read() {
    render(&update.frame.copperlist); // Your presentation function.
    account_for_display_misses(update.missed);
}

// At application shutdown:
let final_status = twin.stop()?;

Use the generated output accessors for your graph. In the demo these include get_encoders_output() and get_kinematics_output(). Their payloads remain typed Rust values; your UI does not need to decode the stream again.

Choose a fresh archive path for each sender session. The builder rejects an existing base file or first slab. Keep the twin handle alive while receiving; stop() finalizes recording and reports worker errors and final counters. Dropping the handle also stops and joins its workers.

For recording alone, insert .archive_only() before .spawn(). This disables task reconstruction and frame production while preserving archival and status.

The default twin accepts the demo’s 1200-byte packet, 64-symbol window, 4 KiB record, and 64 KiB recovery-object profile. Increasing sender limits does not increase receiver limits automatically. Set the builder bounds below for larger records and recovery objects. For other FEC geometry or custom routing, use the custom receiver API with explicit bounds.

Size storage and receiver buffers

v1.3 Size the ground station’s receiver for the largest serialized capture and for pauses in replay or UI consumption. This experimental builder API requires cu29/logstream. In your ground station’s existing Ground::twin(rx) setup, use named constants to make the storage and buffering choices explicit before starting the workers:

use cu29::logstream::{FiniteObjectLimits, SessionRouterLimits};
use std::num::NonZeroUsize;

// Budget for one complete serialized CopperList, including captured messages
// and metadata, and for one manifest, keyframe, or structured log record.
const MAX_RECORD_BYTES: usize = 1024 * 1024;
const MAX_OBJECT_BYTES: u64 = 64 * 1024;
// Fit the larger bound plus a 512-byte section header and 32-byte continuity
// envelope; round up to a convenient multiple of 512 bytes.
const SECTION_BYTES: usize = 2 * 1024 * 1024;
// Hold many sections per backing file: this slab is 16 times the section size.
const SLAB_BYTES: usize = 32 * 1024 * 1024;
// Buffer short replay slowdowns, keeping the retained payload count bounded.
const REPLAY_CAPACITY: NonZeroUsize = NonZeroUsize::new(32).unwrap();
// Independent reader buffers: completed CopperLists and structured log entries.
const FRAME_CAPACITY: NonZeroUsize = NonZeroUsize::new(64).unwrap();
const LOG_CAPACITY: NonZeroUsize = NonZeroUsize::new(64).unwrap();

let limits = SessionRouterLimits {
    max_record_bytes: MAX_RECORD_BYTES,
    finite_objects: FiniteObjectLimits {
        max_object_bytes: MAX_OBJECT_BYTES,
        ..SessionRouterLimits::default().finite_objects
    },
    ..Default::default()
};
let (mut twin, mut frames) = Ground::twin(rx)
    .with_log_path("logs/received.copper")
    .with_receiver_limits(limits)
    .with_section_size(SECTION_BYTES)
    .with_slab_size(SLAB_BYTES)
    .with_replay_capacity(REPLAY_CAPACITY)
    .with_frame_capacity(FRAME_CAPACITY)
    .with_log_capacity(LOG_CAPACITY)
    .spawn()?;

Choose record bounds for the largest complete serialized CopperList, including all captured outputs and metadata. The sender’s advertised record and object bounds must fit the receiver’s bounds. Finite objects carry manifests, keyframes (task-state snapshots), and structured log entries. A section groups archive records and must fit the larger record/object bound plus a 512-byte header and 32 bytes of continuity envelope allowance. Slab and section sizes must be multiples of 512 bytes; each section must fit in a slab. Prefer slabs roughly 10–100 times larger than sections to reduce file turnover. Larger slabs reserve more disk space at a time; the archive grows by adding slabs as needed.

Defaults are 16 MiB slabs, 128 KiB sections, 4 KiB records, 64 KiB finite objects, 32 replay events and pending captures per buffer, and 64 unread reconstructed CopperLists and structured log entries per reader buffer. Increase record/object limits together with section size. spawn() validates local settings before creating directories or starting workers. Sender requirements are checked when its manifest arrives. FEC geometry supports the 1200-byte-MTU, 64-symbol profile. Receiver routing and object concurrency limits are in SessionRouterLimits; one twin accepts exactly one sender session.

Replay buffering absorbs temporary reconstruction slowdowns. Overflow requires recovery from a matching keyframe while recording continues. Frame buffering holds completed CopperLists for UI or analysis; a frame here is one iteration’s captured inputs and reconstructed outputs. Structured log buffering holds the robot’s logging calls, such as info! and debug!, for twin.take_log_reader(). Both reader buffers replace the oldest unread item when full; the archive keeps the received data. Larger buffers retain more payloads and entries in memory. Choose counts together with byte limits for the ground station’s memory budget, and size the sender’s streaming memory and link bandwidth for its payloads.

Keep display loss separate from recording loss

The frame reader retains 64 display frames by default, overwriting the oldest unread frame when full. Set .with_frame_capacity(128.try_into()?) on the builder to change display retention. update.missed counts overwritten display frames; frames.status() reads an independent status slot without consuming frames.

A slow, paused, or dropped reader never makes recording wait. A borrowed frame remains valid while publication continues, but payload allocations and your own chart history consume additional memory. The receiver and UI still share a process failure boundary.

For async presentation, use frames.ready().await instead of polling. Consume unread frames and status changes before waiting again. Custom integrations can use telemetry_channel and a scheduling waker; see the receiver reference.

4. Reduce traffic with task reconstruction

A live twin can execute ordinary synchronous tasks using captured inputs, recorded clock values, and restored task state. Mark a suitable task in the shared graph:

(
    id: "kinematics",
    type: "cu_logstream_demo::tasks::Kinematics",
    streaming: (replay: reconstruct),
),

The robot still runs the task and records its full output locally. Its streamed capture omits the selected output payload, including in repairs and repeated recovery boundaries. The twin reconstructs that output before downstream tasks consume it, restoring sender metadata. Reconstructed values appear in live frames and offline replay output; the received archive keeps the captured bytes.

Omitting streaming keeps the default replay: capture. Sources and bridge receives stay captured. Reconstruction requires a logged, ordinary synchronous task; background and anytime tasks are unsupported. A graph using reconstruction must use the lossless native compressed codec and full handle capture: custom codecs, flat CopperList encoding, and selective handle policies are rejected.

Choose tasks whose outputs depend on recorded inputs, Copper’s clock, and state covered by Freezable. Avoid external side effects and unrecorded inputs such as host wall time or hardware reads. Matching code is necessary, but does not by itself make a nondeterministic algorithm reproducible.

Place the capture boundary where the required inputs exist

Every incoming connection of every reconstructed task must come from a node with logging enabled. A producer can itself reconstruct, so chains of reconstructed tasks are valid. Copper checks all incoming branches and each mission graph at compile time, including when transports are injected in code.

For a camera pipeline, the following task settings let you omit images while sending detections and reconstructing tracking and planning. This is a fragment to apply to your own task types and connections:

tasks: [
    (id: "camera", type: "tasks::Camera", logging: (enabled: false)),
    (id: "detect", type: "tasks::Detect", streaming: (replay: capture)),
    (id: "track", type: "tasks::Track", streaming: (replay: reconstruct)),
    (id: "plan", type: "tasks::Plan", streaming: (replay: reconstruct)),
],

With camera → detect → track → plan, detect supplies captured detections; track and plan have the inputs they need. Keep logging enabled on all three. Changing detect to reconstruct while camera logging is disabled fails at compile time: the diagnostic names detect, camera, and the image message type. Either enable camera logging or keep detector output captured. Similarly, an additional unlogged camera input directly into track makes that task invalid.

Check reconstruction during development

Enable cu29/logstream-verify on both ends to compare reconstructed outputs against sender digests. In the demo, use separate terminals:

just telemetry-verify
# Other terminal:
just sender-verify

Matching frames are labeled Verified. Normal operation labels them Reconstructed and does not send the optional verification digest. A mismatch suppresses reconstructed frames until the next matching recovery point; native recording continues. Verification is opt-in even in a Rust debug build, and its live digests do not change the native archive format.

5. Read structured robot logs

Configured streaming destinations automatically forward structured entries alongside local logging. Message templates and parameter names travel as interned IDs; values retain their types. String values are transmitted if your application explicitly logs strings. Ground rendering needs the producing application’s cu29_log_index, not an unrelated ground build’s string table.

The twin exposes an independent log reader once:

let mut logs = twin.take_log_reader().expect("fresh twin log reader");
// In your presentation loop, with the producing build's string table loaded:
while let Some(update) = logs.try_read() {
    let line = rebuild_logline(&strings, &update.frame.entry)?;
    show_robot_log(line, update.missed);
}

This ring retains 64 entries and publishes only after archival succeeds. CuTwinStatus::structured_logs counts archived entries. Pausing or dropping this reader does not stop task frames or recording.

The demo looks for cu29_log_index beside its executable. To select the producing build’s index explicitly, run the already-built binary from the example directory:

../../target/debug/cu-logstream-demo telemetry \
    --listen 0.0.0.0:7447 --log-base logs/telemetry-session-3.copper \
    --log-index /path/to/producing-build/cu29_log_index

Structured transmission uses bounded buffers and the same destination bitrate budget. Oversized entries or exhausted buffers increment inbox_drops; local records remain intact. Link loss can also leave missing remote log entries. Keep the onboard recording when you need the complete local history.

One-way streaming works without a return path. To send receiver health reports back and optionally adapt continuous FEC, configure both endpoints explicitly. In the demo, run just telemetry-two-way and just sender-two-way in separate terminals. Telemetry listens on port 7447 and reports to the robot on port 7448. Use just telemetry-two-way lossy to exercise sustained source loss.

In your own sender’s destination, add the following block. Also give network a fixed bind_addr, such as 0.0.0.0:7448, so the ground can address it:

feedback: (
    transport: (
        type: "cu29_logstream_udp::CuUdpLogStreamRx",
        resource: "network.rx",
    ),
    report_interval_ms: 100,
    timeout_ms: 1000,
    adaptation: (
        min_repair_every_source_symbols: 1,
        max_repair_every_source_symbols: 16,
    ),
),

On the ground, create a TX endpoint targeting that robot address and pass it to the twin builder. Replace the example robot address below:

let mut socket = CuUdpLogStreamConfig::new("0.0.0.0:7447".parse()?);
socket.remote_addr = Some("192.168.1.10:7448".parse()?);
let (tx, rx) = socket.open()?;
let (mut twin, mut frames) = Ground::twin(rx)
    .with_feedback(tx.expect("remote_addr creates a TX endpoint"))
    .with_log_path("logs/received-two-way-session-1.copper")
    .spawn()?;

The sender manifest advertises feedback support and cadence. Sharing a UDP socket does not enable feedback automatically. Give each logical receive endpoint one owner; cloned UDP receivers compete for packets rather than broadcasting them.

Omit adaptation for reports only. With adaptation enabled, the baseline repair_every_source_symbols must lie within the configured bounds. Loss raises protection; sustained healthier reports allow less redundancy. After timeout, feedback becomes stale and the repair interval gradually returns to baseline. Bitrate, burst, memory, latency, FEC window/field/density, and object FEC stay fixed. Feedback failure never stops capture or autonomous recovery.

In the robot’s console monitor, open BW to see one Telemetry / TX panel per destination. Arrow keys or hjkl scroll large panels.

ReadingInterpretation
TX bytes/packets, budget, drops, recovery activityLocal submission and sender health; a successful UDP send does not confirm delivery.
One-way / DisabledNo feedback was configured.
Two-way / WaitingFeedback is configured but no valid report has arrived.
ActiveReceiver measurements and effective repair interval are available.
Stale or failed; receiver values n/aReceiver measurements are unavailable or out of date. Check the return path.
Source loss / FEC recoveryFinalized symbol outcomes; excludes the active coding window and unseen history or tails.

Reports are advisory, not per-record acknowledgements. Custom monitors can read worker snapshots through runtime.log_streams(); the presentation thread samples them without adding monitoring callbacks to task execution.

7. Inspect recordings and exercise recovery

From the demo directory, point its tools at a received archive base:

just cl logs/received-session-1.copper
just fsck logs/received-session-1.copper
just resim logs/received-session-1.copper logs/replay-session-1.copper
just resim-debug logs/replay-session-1.copper logs/debug-session-1.copper

For your own app, use its matching logreader and replay binaries as described in Logging and Replaying Data. Keep the whole slab family (name.copper, name_0.copper, and subsequent slabs where present), and pass the base path to the tools. Always use a different path for replay output.

Extract received structured logs with the producing build’s index:

../../target/debug/cu-logstream-demo-logreader \
    logs/received-session-1.copper extract-text-log \
    ../../target/debug/cu29_log_index

The received archive retains sender payload bytes, timestamps, keyframes, and stream continuity records. fsck reports missing ranges and verified recovery boundaries. Replay cannot cross a gap without a matching keyframe; resuming later does not fill the gap. A clean receiver shutdown does not establish that an unobserved sender tail was complete.

Share and retain a received archive

v1.3 To read the archive with the standalone tools from the self-describing log exercise, enable cu29/self-describing-logs in the sender and generated twin. Build them from the same application definitions and encoding features. The twin writes the matching catalog at archive startup, including streamed payload recipes when onboard task logging is disabled. Archive configuration capture is currently unavailable; supply the application’s configuration explicitly for analysis that needs it.

For bounded ground-station storage, add this option before .spawn() in the builder from Step 3:

.with_log_rollover(256 * 1024 * 1024)

This bounds storage to 256 MiB. Capacity must exceed the configured section size, be a multiple of 512 bytes, and fit static metadata plus an archive section. The archive uses the same rollover behavior as onboard recording, keeping the catalog while reclaiming old data.

The sender’s repeated manifest identifies its run, runtime instance, and mission. Received sections retain that identity through rollover and late joins. Use the twin generated for the sender’s active mission; a new sender construction needs a fresh session and archive. Runtime lifecycle events remain in the onboard log.

Exercise stream recovery

Exercise the failure modes before deploying:

Demo commandWhat it exercises
just run lossRecover a dropped CopperList with FEC.
just run lossySustained source-packet loss.
just run outageResume after an interruption that leaves missing history.
just run lateJoin an already running sender.
just run restartStart a fresh receiver during a run.
just run idleRecover the initial boundary after captures have stopped.
just checkRun all scenarios, including clean reception.
just run-two-way allRun the scenarios with feedback enabled.

The sender repeats its manifest and latest complete recovery bundle on a 250 ms local deadline, even without new captures. Repetition shares the bitrate budget, so this is a scheduling deadline rather than a delivery guarantee. Pending bundles finish before newer ones replace them. Pacing uses RobotClock and does not require clock synchronization with the ground station.

Troubleshooting

SymptomCheck or change
No packets at the groundVerify numeric bind/remote addresses, ports, routing, and firewall access. Loopback addresses only reach the same machine.
Packets arrive but no framesInspect recording/replay status: the twin may be waiting for a valid manifest/recovery point, recovering after a gap, or configured as archive-only.
Compile error about an unlogged replay inputEnable the named producer’s logging or capture the named consumer’s output. Check every fan-in branch.
Startup rejects the stream/profileUse matching application/schema/encoding features; fit the receiver’s record, object, symbol, and window bounds.
Archive already existsChoose a fresh base path and keep each sender session separate.
Display misses rise while archive progress continuesReduce UI work or increase display capacity. These misses do not mean packets were lost.
Sender drops or source gaps riseCheck encoded record sizes, latency expiry, link capacity, FEC overhead, and memory limits. More repair traffic also consumes the link budget.
Robot log text is wrong or cannot be renderedLoad the exact producing build’s string index.
Feedback stays Waiting or becomes StaleCheck the robot’s fixed bind port, ground TX destination, .with_feedback(...), and single receive ownership.
Reconstruction stalls while recording continuesInspect replay queue pressure, source gaps, and verification failures; reconstruction resumes at a matching recovery point.

For serial transport, receiver internals, and current framing details, continue to LogStream Transport and Receiver Reference.

LogStream Transport and Receiver Reference

v1.2 Start with Streaming Logs and Live Telemetry for the UDP walkthrough. This reference covers custom carriers, receiver ownership, and format details needed when sizing or extending an integration.

Serial and transparent radios

A raw serial byte stream needs packet framing before LogStream can decode it. cu29-logstream-serial supplies SerialLogStreamTx<S, N> and SerialLogStreamRx<S, N> over cu_serial::SerialIo. SerialLogStreamTxResources<S, N> consumes a serial resource and exports tx; the RX provider consumes serial and exports rx. Each provider owns its serial resource; choose one direction per carrier with these adapters.

A UART can feed an HC-12 radio resource, whose serial output feeds the LogStream framing resource. Define concrete application aliases for those generic providers, then bind the final tx slot to your streaming destination. The serial adapter and HC-12 examples provide the transport and radio details. Do not mix raw application bytes and LogStream packets on the same stream without an explicit multiplexing protocol.

The serial adapter wraps each packet in 0x7e delimiters, escapes 0x7e and 0x7d, and appends an escaped four-byte big-endian CRC32C of the unescaped packet. RX verifies and removes that checksum before handing the packet to LogStream. Invalid or oversized frames are discarded; the next delimiter resynchronizes RX. Both serial peers must use this framing.

The compile-time frame capacity N bounds packets to (N - 2) / 2 - 4 bytes, allowing worst-case escaping. The default N = 514 therefore permits a 252-byte packet, not the UDP demo’s 1200-byte MTU. Configure MTU within this bound, use a small burst (one for slow links), and budget for checksum, escaping, and UART start/stop bits as well as Copper traffic.

The scheduled sender drains pending serial writes while idle and during bounded shutdown. If you drive an endpoint directly, including in no_std, keep calling CuStreamTx::poll_pending() until it returns false. Each poll performs at most one nonblocking UART write. WouldBlock means retry the packet after the current frame drains; successful submission only means the adapter accepted it.

Custom receivers

Use the generated twin when its profile fits. For explicit receiver bounds or a custom archive pipeline, construct SessionRouter and supply packets through CuStreamRx. Its manifest describes decoder geometry and the application schema; the receiver checks those requirements against its own allocation limits before constructing decoders.

APIUse
NativeArchive<P>Archive full captures using the application’s generated dataset type.
CaptureArchive<P>Preserve selective captures for live or offline reconstruction; used by the generated twin.
cu29_export::stream_continuity_readerRead archived manifests, gap ranges, and recovery boundaries programmatically.
telemetry::telemetry_channelPublish typed, already decoded values to an independent presentation reader.

SessionRouter lends SessionEventRef values to callbacks. Encoded bytes remain receiver-owned; if a callback fails, drain_events can retry delivery. Copy with event.to_owned() only when you need a separately owned event. Make your callback safe for retries; the generated twin reports archive failures instead of hiding them behind an application retry loop.

ReceivedManifest includes the decoded manifest and its original verified record. Pass it to the archive constructor. Recovery events also carry their original record, allowing archival without recreating the bytes. Publish frames only after archive.accept(...) succeeds. To publish structured logs yourself, take the owned entry with archive.take_structured_log() immediately afterward.

The telemetry channel has one publisher and one reader, bounded retention, and an independent coalesced status slot. ready().await, wait_timeout, and register_waker support different event loops. A registered waker should only schedule work or unpark a thread. It must not process data or block the receiving worker. Consume frames, changed status, and publisher closure before waiting again.

Assembly buffers are recycled and grow within configured bounds as needed. Borrowed delivery avoids mandatory per-event copies, but session discovery, RaptorQ decoding, task-state restoration, and payload decoding can allocate. The presentation ring is also a host-side queue, not a real-time wait-free primitive. Consult the receiver source when changing capacities or callback ownership.

Storage, pacing, and shutdown

Each generated destination has a sender worker with bounded encoded-record pools, packet queues, continuous FEC storage, and retained recovery data. CopperLists and keyframe capture objects are released before transmission waits on the carrier. Structured logging adds a bounded copy of the successful local encoded entry into each destination’s preallocated storage.

The destination reserves four structured-record buffers, each capped at 4096 bytes including the record header, or max_object_bytes if smaller. These buffers count toward memory_budget_kib. Thread stacks, allocator/channel bookkeeping, and bounded-input RaptorQ scratch allocations are additional; this setting is a buffer budget, not a whole-process memory ceiling.

Replay traffic, recovery traffic, and structured logs share byte-deficit scheduling with weights 3:1:1. Unused capacity is available to other ready lanes. The bitrate/burst budget covers every lane, while separate bounded packet queues keep structured traffic from filling the replay queue. Ordinary data expires after max_latency_ms; the latest complete recovery bundle remains available for periodic retransmission until replaced or stopped.

Pool exhaustion, queue overflow, expiry, carrier backpressure, and shutdown shedding have counters. SenderMonitor::snapshot() publishes worker health at 10 Hz, and shutdown statistics/failures are written through Copper structured logging. Shutdown stops repetition and drains only to the configured latency deadline. The no_std SenderCore needs a caller to provide CuTime and drive poll; immediate codec sinks do not enforce pacing themselves.

Compatibility and wire sizing

Build sender, receiver, and logreader for the same producing application and Copper revision. Packets, semantic records, RLC fragments, and session manifests carry no version fields. Schema validation is not a compatibility negotiation mechanism. Unified-log encapsulation version 1 describes the file and section layout, not the encoded application content.

Serialized CopperLists contain id and msgs; lifecycle state stays in memory. Timestamp-delta and backreference helpers preserve the compressed metadata bytes. Ordinary integers use the codec’s standard integer encoding.

Current packed header sizes are:

LayerBytes
Continuous RLC source packet32
Continuous RLC repair packet36
RaptorQ object packet51
Semantic record45
Protected RLC fragment20

Multi-byte packet-header fields are big endian. The generated sender uses min(mtu_bytes - 51, 1128) bytes per symbol, reserving the largest current packet header and respecting its fixed storage cap. At MTU 1200, symbols are therefore at most 1128 bytes. Maximum RLC source, RLC repair, and RaptorQ packets are therefore 1160, 1164, and 1179 bytes respectively. Use actual packet lengths for bandwidth accounting. The wire encoder is the authority for a custom implementation.

Packet extent comes from the carrier; there is no redundant payload length or packet sequence counter. FEC identifiers and record identities provide recovery and deduplication. Protected fragments retain record length and fragment index; fragment count and payload extent are derived. The 11-byte RaptorQ OTI omits the library’s reserved zero byte, which decoding restores.

A semantic record’s payload length comes from its reassembled extent. Its BLAKE3 digest covers record kind, object ID, derived length, and payload; appended or truncated payloads fail validation. Recovery points reference the exact manifest and keyframe digests.

Manifests carry identity, decoder requirements, optional feedback cadence and destination key, and application schema. Sender bitrate, latency, adaptation, repair policy, and memory limits remain local. The receiver also enforces its own finite-object and buffering limits; a sender cannot raise them through a manifest.

Both data and feedback packets require carrier-provided packet integrity; neither has an inner CRC. UDP relies on normal network/link checksum protection, while the serial adapter supplies CRC32C. Delivery, ordering, and uniqueness are not guaranteed. Feedback reports retain receiver sequence numbers so stale or duplicate reports do not refresh health. Record digests independently check reassembled content and recovery references.

Missions

So far, our project has a single pipeline: MySource → MyTask → MySink. Every time we run it, the same three tasks execute in the same order. But real-world robots often need to operate in different modes – a drone might have a “takeoff” mode and a “cruise” mode, a warehouse robot might switch between “navigate” and “charge”, or you might want a “simulation” mode that uses mock drivers instead of real hardware.

In many frameworks, you’d handle this with if statements scattered across your code, or by maintaining separate launch files. Copper takes a different approach: missions.

What are missions?

A mission is a named variant of your task graph. You declare all possible tasks and connections in a single copperconfig.ron, then tag each one with the missions it belongs to. At build time, Copper generates a separate application builder for each mission. At runtime, you choose which builder to use.

The key properties:

  • Tasks without a missions field are shared – they exist in every mission.
  • Tasks with a missions field are selective – they only exist in the listed missions.
  • Connections follow the same rule – tag them with missions to make them mission-specific.
  • No recompilation needed to switch – all missions are compiled at once. You pick which one to run.

Step 1: Define missions in copperconfig.ron

Let’s modify our project to support two missions:

  • “normal” – The full pipeline we’ve been using: source → processing → sink.
  • “direct” – A shortcut that skips the processing task: source → sink.

This is a simple but realistic scenario. Imagine MyTask does some expensive computation (image processing, path planning). During testing or in a degraded mode, you might want to bypass it and send raw data straight to the sink.

Replace your copperconfig.ron with:

(
    missions: [(id: "normal"), (id: "direct")],
    tasks: [
        (
            id: "src",
            type: "tasks::MySource",
        ),
        (
            id: "t-0",
            type: "tasks::MyTask",
            missions: ["normal"],
        ),
        (
            id: "sink",
            type: "tasks::MySink",
        ),
    ],
    cnx: [
        (
            src: "src",
            dst: "t-0",
            msg: "crate::tasks::MyPayload",
            missions: ["normal"],
        ),
        (
            src: "t-0",
            dst: "sink",
            msg: "crate::tasks::MyPayload",
            missions: ["normal"],
        ),
        (
            src: "src",
            dst: "sink",
            msg: "crate::tasks::MyPayload",
            missions: ["direct"],
        ),
    ],
    runtime: (
        rate_target_hz: 1,
    ),
)

Let’s break down what changed:

The missions declaration

missions: [(id: "normal"), (id: "direct")],

This top-level array declares all available missions. Each mission is just an ID – a string that you’ll reference elsewhere.

Shared tasks

src and sink have no missions field. This means they participate in every mission. They are the common backbone of the pipeline.

Mission-specific tasks

(
    id: "t-0",
    type: "tasks::MyTask",
    missions: ["normal"],
),

MyTask is tagged with missions: ["normal"]. It only exists in the “normal” mission. When running the “direct” mission, this task is simply not instantiated.

Mission-specific connections

The connections are where the graph really diverges:

Mission "normal":   src ──▶ t-0 ──▶ sink
Mission "direct":   src ──────────▶ sink

In “normal”, data flows through the processing task. In “direct”, the source connects directly to the sink, bypassing MyTask entirely.

Notice that the connection src → sink in the “direct” mission uses the same message type (crate::tasks::MyPayload) as the other connections. This works because MySink already accepts MyPayload as input – the message types must be compatible regardless of which path the data takes.

Step 2: Update main.rs

When you declare missions, the #[copper_runtime] macro no longer generates a single builder. Instead, it creates a module for each mission, named after the mission ID. Each module contains its own builder type.

For our project, the macro generates:

  • normal::MyProjectApplicationBuilder – builds the “normal” pipeline
  • direct::MyProjectApplicationBuilder – builds the “direct” pipeline

Update main.rs to select a mission:

pub mod tasks;

use cu29::prelude::*;
use std::path::Path;
use std::thread::sleep;
use std::time::Duration;

const PREALLOCATED_STORAGE_SIZE: Option<usize> = Some(1024 * 1024 * 100);

#[copper_runtime(config = "copperconfig.ron")]
struct MyProjectApplication {}

// Import the per-mission application types
use normal::MyProjectApplication as NormalApp;
use direct::MyProjectApplication as DirectApp;

fn main() {
    // Pick the mission from the first command-line argument (default: "normal")
    let mission = std::env::args()
        .nth(1)
        .unwrap_or_else(|| "normal".to_string());

    let logger_path = "logs/my-project.copper";
    if let Some(parent) = Path::new(logger_path).parent() {
        if !parent.exists() {
            std::fs::create_dir_all(parent).expect("Failed to create logs directory");
        }
    }
    let clock = RobotClock::default();
    debug!("Logger created at {}.", logger_path);

    match mission.as_str() {
        "normal" => {
            debug!("Starting mission: normal");
            let app = NormalApp::builder()
                .with_clock(clock.clone())
                .with_log_path(logger_path, PREALLOCATED_STORAGE_SIZE)
                .expect("Failed to setup logger.")
                .build()
                .expect("Failed to create application.");
            app.run_until_shutdown().expect("Failed to run application.");
        }
        "direct" => {
            debug!("Starting mission: direct");
            let app = DirectApp::builder()
                .with_clock(clock.clone())
                .with_log_path(logger_path, PREALLOCATED_STORAGE_SIZE)
                .expect("Failed to setup logger.")
                .build()
                .expect("Failed to create application.");
            app.run_until_shutdown().expect("Failed to run application.");
        }
        other => {
            eprintln!("Unknown mission: '{}'. Available: normal, direct", other);
            std::process::exit(1);
        }
    }

    debug!("End of program.");
    sleep(Duration::from_secs(1));
}

The important changes:

  1. Two use statements import the builders from the generated mission modules.
  2. A match on the mission name selects which builder to use. Here we read it from the command line, but you could also use an environment variable, a config file, or any other mechanism.
  3. Each branch is independent – the “normal” branch builds a three-task pipeline, the “direct” branch builds a two-task pipeline.

No changes to tasks.rs are needed. The task implementations are the same – missions only control which tasks are instantiated and wired, not how they behave.

Step 3: Run it

Run the “normal” mission (the default):

cargo run
00:00:00.0006 [Debug] Source at 630
00:00:00.0006 [Debug] Received message: 42
00:00:00.0006 [Debug] Sink Received message: 43
00:00:01.0002 [Debug] Source at 1000259
00:00:01.0004 [Debug] Received message: 42
00:00:01.0004 [Debug] Sink Received message: 43
00:00:01.9999 [Debug] Source at 1999931
00:00:02.0000 [Debug] Received message: 42
00:00:02.0001 [Debug] Sink Received message: 43

This is the full pipeline. The source produces 42, MyTask transforms it to 43, and the sink receives 43.

Now run the “direct” mission:

cargo run -- direct
00:00:00.0005 [Debug] Source at 549
00:00:00.0005 [Debug] Sink Received message: 42
00:00:00.9999 [Debug] Source at 999945
00:00:01.0000 [Debug] Sink Received message: 42
00:00:01.9992 [Debug] Source at 1999286
00:00:01.9994 [Debug] Sink Received message: 42
00:00:02.9987 [Debug] Source at 2998704
00:00:02.9988 [Debug] Sink Received message: 42

Notice the difference: there is no “Received message: 42” line from MyTask, and the sink receives 42 (the raw value from the source) instead of 43. MyTask was never instantiated – the data went straight from source to sink.

Same binary, same tasks, different wiring. No recompilation.

How it works under the hood

When the #[copper_runtime] macro processes a configuration with missions, it:

  1. Parses all missions from the top-level missions array.
  2. For each mission, filters the task list and connection list to include only:
    • Tasks/connections with no missions field (shared across all missions)
    • Tasks/connections whose missions array contains this mission’s ID
  3. Generates a Rust module for each mission (named after the mission ID), containing a builder type, the filtered task graph, and the scheduling code.

All of this happens at compile time. The final binary contains the code for every mission, but only the tasks belonging to the selected mission are instantiated at runtime.

A task can belong to multiple missions

A task isn’t limited to a single mission. If you have a task that’s needed in several (but not all) missions, list them:

(
    id: "safety-monitor",
    type: "tasks::SafetyMonitor",
    missions: ["navigate", "charge", "manual"],
),

This task is active in three missions but excluded from others (say, “simulation” or “diagnostics”).

When to use missions

Missions are most useful when you have:

  • Hardware vs simulation: Use real drivers in one mission, mock drivers in another.
  • Operating modes: Different task graphs for different phases of operation (startup, cruise, landing).
  • Platform variants: The same codebase running on different hardware – one mission for the prototype with basic sensors, another for the production model with full sensor suite.
  • Debug vs production: A mission with extra logging/monitoring tasks for development, and a lean mission for deployment.

Difference with ROS

In ROS 2, switching between configurations typically means maintaining separate launch files or using launch arguments with conditionals:

# ROS 2: launch file with conditionals
use_sim = LaunchConfiguration('use_sim')

Node(
    package='my_robot',
    executable='lidar_driver',
    condition=UnlessCondition(use_sim),
),
Node(
    package='my_robot',
    executable='fake_lidar',
    condition=IfCondition(use_sim),
),
ros2 launch my_robot robot.launch.py use_sim:=true

This works, but the logic lives in Python launch files that are completely separate from your node code. Errors (wrong topic names, missing remappings) only show up at runtime.

In Copper, all missions live in copperconfig.ron:

Copper:
  copperconfig.ron    ← missions, tasks, connections
  cargo run           ← default mission
  cargo run -- direct ← alternative mission

Key differences:

ROS 2Copper
WherePython launch files + YAML paramscopperconfig.ron
ValidationRuntime (nodes may fail to connect)Compile time (macro checks the graph)
GranularityPer-node conditionalsPer-task and per-connection tagging
SwitchingLaunch argumentsBuilder selection in main.rs
All variants visibleSpread across files and conditionalsOne file, all missions visible

The main advantage: one file shows every mission and which tasks run in each.

From Project to Workspace

In Setting Up Your Environment, we generated a single-crate project with cu_project. That flat structure is perfect for getting started, but as your robot grows – more sensors, more algorithms, shared components across multiple robots – a single crate with everything in tasks.rs becomes hard to manage.

Copper provides a second template, cu_full, that scaffolds a Cargo workspace with a clear separation between applications and reusable components. In this chapter, we’ll generate one and understand its layout.

Generating a workspace

v1.0 Use cargo-cunew with the workspace template:

cargo cunew --template workspace --source git --git-branch master my_workspace

This creates a my_workspace/ directory with a full workspace layout.

What you get

my_workspace/
├── Cargo.toml                          # Workspace root
├── justfile                            # Automation helpers
├── apps/
│   └── cu_example_app/                 # Your first application
│       ├── Cargo.toml
│       ├── build.rs
│       ├── copperconfig.ron
│       └── src/
│           ├── main.rs
│           ├── logreader.rs
│           ├── resim.rs
│           ├── pgs.rs
│           ├── pgs_candidate.rs
│           ├── messages.rs
│           └── tasks/
│               ├── mod.rs
│               ├── local_example_src.rs
│               ├── local_example_task.rs
│               └── local_example_sink.rs
├── tools/
│   └── cu29_view_helper/               # Graph and schedule launcher
├── components/
│   ├── bridges/
│   ├── monitors/
│   ├── payloads/
│   ├── sinks/
│   ├── sources/
│   └── tasks/
└── doc/

That’s a lot more structure than our simple my_project/. Let’s walk through it.

The workspace root: Cargo.toml

The top-level Cargo.toml defines the workspace and shared dependencies:

[workspace]
members = [
    "apps/cu_example_app",
    "components/bridges/cu_example_shared_bridge",
    "tools/cu29_view_helper",
]
resolver = "2"

[workspace.dependencies]
cu29 = { git = "https://github.com/copper-project/copper-rs", branch = "master" }
cu29-build = { git = "https://github.com/copper-project/copper-rs", branch = "master" }
cu29-export = { git = "https://github.com/copper-project/copper-rs", branch = "master" }
serde = { version = "1.0", features = ["derive"] }

This example selects the checkout used in Setting Up. Keep the same Copper source for all three crates; a generated workspace records the source you selected when scaffolding it. The codec is available through cu29::bincode, so application crates do not need a separate bincode dependency.

Every crate in the workspace references these shared dependencies with workspace = true in its own Cargo.toml. This means you define dependency versions once at the workspace level, and all crates stay in sync.

When you add a new application or component, you add it to the members list.

The apps/ directory

This is where your application crates live. Each app is a standalone binary that owns its own runtime configuration, log storage, logreader, and replay target.

The example app (cu_example_app) looks very similar to the my_project we built in earlier chapters, but with two key differences.

Messages are in their own file

Instead of defining MyPayload inside tasks.rs, the workspace template puts message types in a dedicated messages.rs:

#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};

#[derive(Default, Debug, Clone, Encode, Decode, Serialize, Deserialize, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct MyPayload {
    pub value: i32,
}
}

This makes it easier to share message types – other files within the app import them with use crate::messages::MyPayload.

Tasks are in their own directory

Instead of one big tasks.rs, each task gets its own file under src/tasks/:

src/tasks/
├── mod.rs                    # Re-exports all tasks
├── local_example_src.rs      # MySource
├── local_example_task.rs     # MyTask
└── local_example_sink.rs     # MySink

The mod.rs ties them together:

#![allow(unused)]
fn main() {
mod local_example_sink;
mod local_example_src;
mod local_example_task;

pub use local_example_sink::MySink;
pub use local_example_src::MySource;
pub use local_example_task::MyTask;
}

From the rest of the codebase, you still write tasks::MySource – the internal file structure is hidden behind the module.

This is standard Rust module organization, but it matters as your robot grows. When you have 15 tasks, having them in separate files with clear names is much easier to navigate than scrolling through a 1000-line tasks.rs.

The copperconfig.ron is identical

The task graph configuration works exactly the same way. The only difference is that message paths reference crate::messages::MyPayload instead of crate::tasks::MyPayload, because the message type moved to its own module:

(
    tasks: [
        (
            id: "src",
            type: "tasks::MySource",
        ),
        (
            id: "t-0",
            type: "tasks::MyTask",
        ),
        (
            id: "sink",
            type: "tasks::MySink",
        ),
    ],
    cnx: [
        (
            src: "src",
            dst: "t-0",
            msg: "crate::messages::MyPayload",
        ),
        (
            src: "t-0",
            dst: "sink",
            msg: "crate::messages::MyPayload",
        ),
    ],
)

The components/ directory

This is where reusable components live – code that can be shared across multiple applications or even published for other people to use.

The directory is organized by category:

DirectoryPurposeExample
sources/Sensor drivers that produce dataCamera driver, IMU reader
sinks/Actuator drivers that consume dataMotor controller, GPIO writer
tasks/Processing algorithmsPID controller, path planner
bridges/Interfaces to external systemsZenoh bridge, ROS bridge
monitors/Monitoring and visualizationConsole TUI, web dashboard
payloads/Shared message typesSensor payloads, spatial types

The template generates these directories with placeholder .keep files. They’re empty, waiting for you to add your own components as your project grows. We’ll cover how to create shared components and how to reuse existing ones from the Copper ecosystem in the Reusing Community Components chapter.

Where do message types go?

You might wonder: should messages go in messages.rs inside the app, or in a component crate under components/payloads/? The answer depends on who needs them.

App-local messages stay in messages.rs inside the app. If MyPayload is only used by the tasks within cu_example_app, it belongs right there. This is the most common case when you’re starting out – and it’s exactly where the template puts it.

Shared messages go into a component crate when multiple apps or components need the same type. For example, if you have two robots that both use the same sensor data format, you’d create a crate under components/payloads/ and have both apps depend on it.

Ecosystem messages are already defined in Copper’s built-in payload crates (like cu-sensor-payloads for common sensor types). You don’t write these – you just depend on them. We’ll explore them in the Reusing Community Components chapter.

Here’s the rule of thumb:

QuestionPut messages in…
Only used by tasks within one app?apps/my_app/src/messages.rs
Shared between multiple apps in your workspace?A crate under components/payloads/
Already defined by an existing Copper component?Just depend on that crate

When in doubt, start local. You can always move a message type into a shared crate later when a second consumer appears.

Running the workspace

From the workspace root, run the example app with:

cargo run -p cu_example_app

The -p flag tells Cargo which workspace member to build and run. This is different from the simple project where cargo run was enough – in a workspace with multiple binaries, you need to be explicit.

Simple project vs workspace: when to switch

You don’t need to start with a workspace. Here’s a simple rule of thumb:

SituationUse
Learning, prototyping, single-robot projectscu_project (simple)
Multiple robots sharing componentscu_full (workspace)
Components you want to publish or reusecu_full (workspace)
Team projects with clear module boundariescu_full (workspace)

The good news: migrating from a simple project to a workspace is just moving files around and updating Cargo.toml paths. The task code, message types, and copperconfig.ron format are identical in both cases.

Modular Configuration

In the previous chapter we saw how a workspace separates code into multiple crates. But what about the configuration? As your robot grows, a single copperconfig.ron with dozens of tasks and connections becomes hard to read and maintain. What if you have two motors that use the same driver with different parameters? You’d have to duplicate the task entry and hope you keep them in sync.

Copper solves this with modular configuration: the ability to split your RON file into smaller pieces and compose them with parameter substitution.

The includes section

The key feature is the includes array at the top level of copperconfig.ron. Each entry specifies a path to another RON file and an optional set of parameters:

(
    tasks: [],
    cnx: [],
    includes: [
        (
            path: "sensor.ron",
            params: { "id": "front", "pin": 4 },
        ),
    ],
)

When Copper processes this configuration, it reads sensor.ron, substitutes the parameters, and merges the resulting tasks and connections into the main configuration.

Selecting graph fragments with Cargo features

v1.1 Use when on an include when a Cargo feature also enables an optional component crate:

includes: [
    (path: "graph/base.ron"),
    (
        path: "graph/jetson_mipi.ron",
        when: Feature("jetson-mipi"),
    ),
]

The complete included fragment is removed when the feature is off. Copper skips the file before reading, validation, and code generation, so task types and message types from an unavailable target crate never become Rust references.

The available predicates compose without putting conditions on individual graph elements:

when: Not(Feature("camera"))
when: All([Feature("camera"), Feature("recording")])
when: Any([Feature("mipi"), Feature("usb-camera")])

Forward the consuming crate’s active features from its build script:

[features]
default = []
jetson-mipi = ["dep:cu-gstreamer"]

[build-dependencies]
cu29-build = "1.1"
// build.rs
fn main() {
    cu29_build::setup();
}

The optional dependency may also live in a target-specific dependency table. See examples/cu_feature_gated_camera for the complete Linux/aarch64 camera example.

Parameter substitution with {{param}}

Inside an included file, parameters are referenced using double curly braces:

// sensor.ron
(
    tasks: [
        (
            id: "sensor_{{id}}",
            type: "tasks::SensorDriver",
            config: { "pin": {{pin}} },
        ),
    ],
    cnx: [],
)

When included with params: { "id": "front", "pin": 4 }, this expands to:

(
    tasks: [
        (
            id: "sensor_front",
            type: "tasks::SensorDriver",
            config: { "pin": 4 },
        ),
    ],
    cnx: [],
)

The {{id}} becomes front and {{pin}} becomes 4. Simple text substitution – it works in task IDs, config values, message types, or anywhere else in the RON file.

Reusing the same file with different parameters

The real power shows up when you include the same file multiple times with different parameters. This is the robotics equivalent of “instantiate the same component twice with different settings.”

Consider a robot with two motors – left and right. They use the same driver code, but different GPIO pins:

// motors.ron
(
    tasks: [
        (
            id: "motor_{{id}}",
            type: "tasks::MotorDriver",
            config: { "pin": {{pin}}, "direction": "{{direction}}" },
        ),
    ],
    cnx: [],
)

In your main configuration, include it twice:

(
    tasks: [],
    cnx: [],
    includes: [
        (
            path: "motors.ron",
            params: { "id": "left",  "pin": 4, "direction": "forward" },
        ),
        (
            path: "motors.ron",
            params: { "id": "right", "pin": 5, "direction": "reverse" },
        ),
    ],
)

This produces two tasks: motor_left on pin 4 and motor_right on pin 5. One file defines the motor pattern; the main config just says “give me two of them with these settings.”

A complete example

Here’s a more realistic configuration that combines local tasks, included subsystems, monitoring, and logging:

(
    tasks: [
        (
            id: "planner",
            type: "tasks::PathPlanner",
        ),
    ],
    cnx: [
        (
            src: "sensor_front",
            dst: "planner",
            msg: "crate::messages::SensorData",
        ),
        (
            src: "sensor_rear",
            dst: "planner",
            msg: "crate::messages::SensorData",
        ),
        (
            src: "planner",
            dst: "motor_left",
            msg: "crate::messages::MotorCommand",
        ),
        (
            src: "planner",
            dst: "motor_right",
            msg: "crate::messages::MotorCommand",
        ),
    ],
    monitor: ( type: "cu_consolemon::CuConsoleMon" ),
    logging: (
        slab_size_mib: 1024,
        section_size_mib: 100,
    ),
    includes: [
        (
            path: "sensor.ron",
            params: { "id": "front", "pin": 2 },
        ),
        (
            path: "sensor.ron",
            params: { "id": "rear", "pin": 3 },
        ),
        (
            path: "motors.ron",
            params: { "id": "left",  "pin": 4, "direction": "forward" },
        ),
        (
            path: "motors.ron",
            params: { "id": "right", "pin": 5, "direction": "reverse" },
        ),
    ],
)

The main config file focuses on architecture – how subsystems connect to each other. The included files focus on component definition – what a sensor or motor looks like. The connections in the main file reference tasks by their expanded IDs (sensor_front, motor_left), which are predictable from the template + parameters.

Recursive includes

Included files can themselves include other files. This lets you build hierarchical configurations:

copperconfig.ron
├── includes left_arm.ron
│   ├── includes shoulder_motor.ron
│   └── includes elbow_motor.ron
└── includes right_arm.ron
    ├── includes shoulder_motor.ron
    └── includes elbow_motor.ron

Each level can pass different parameters down, so shoulder_motor.ron is written once but instantiated four times (left shoulder, left elbow, right shoulder, right elbow) – each with its own pin assignments and IDs.

Where to put included files

There’s no strict rule, but a common convention in workspace projects is to keep included RON files next to the main copperconfig.ron:

apps/cu_example_app/
├── copperconfig.ron        # Main config (includes the others)
├── sensor.ron              # Sensor subsystem template
├── motors.ron              # Motor subsystem template
└── src/
    └── ...

For very large projects, you might create a config/ subdirectory and use relative paths in the includes:

includes: [
    ( path: "config/sensor.ron", params: { ... } ),
]

Instance-specific overrides for swarms

When several robots share the same subsystem code but need different calibration data, keep the base subsystem graph static and use a multi-Copper umbrella file with a conventional instance override tree:

(
    subsystems: [
        (id: "robot", config: "robot_base.ron"),
    ],
    interconnects: [],
    instance_overrides_root: "instances",
)

Copper will then probe:

instances/<instance_id>/<subsystem_id>.ron

So instance 17 for subsystem robot resolves:

instances/17/robot.ron

That file applies only config overlays, not topology changes:

(
    set: [
        (
            path: "tasks/imu/config",
            value: {
                "gyro_bias": [0.012, -0.004, 0.008],
            },
        ),
    ],
)

Supported paths are intentionally small:

  • tasks/<id>/config
  • resources/<id>/config
  • bridges/<id>/config

This keeps the graph static while letting each robot carry its own calibration or hardware parameters in git. See examples/cu_instance_overrides_demo/ for a minimal example.

Difference with ROS

In ROS 2, configuration reuse is handled through launch file composition and YAML parameter files. You can include other launch files and remap parameters:

# ROS 2: composing launch files
IncludeLaunchDescription(
    PythonLaunchDescriptionSource('motor_launch.py'),
    launch_arguments={'motor_id': 'left', 'pin': '4'}.items(),
)

Copper’s approach is similar in spirit but different in execution:

ROS 2Copper
FormatPython launch files + YAMLRON files with {{param}} substitution
ReuseIncludeLaunchDescriptionincludes array in RON
ParametersLaunch arguments + YAML filesparams map with text substitution
ValidationRuntimeCompile time
NestingLaunch files can include other launch filesRON files can include other RON files

The main advantage of Copper’s approach is simplicity: parameterization remains text substitution, while the only conditional composition is a small typed predicate on whole includes. There is no Python logic or per-task if/else chain. The result is a flat list of tasks and connections that Copper validates at compile time.

Further reading

Reusing Community Components

One of the biggest advantages of building your robot with Copper is that you don’t have to write everything from scratch. The copper-rs repository ships with a growing collection of ready-made components – sensor drivers, algorithms, bridges, and shared message types – that you can drop into your project.

In this chapter, we’ll explore what’s available and how to use it.

The components directory

The copper-rs repository organizes its components by role:

components/
├── sources/      Sensor drivers that produce data
├── sinks/        Actuator drivers that consume data
├── tasks/        Processing algorithms (input → output)
├── bridges/      Interfaces to external systems (Zenoh, ROS 2)
├── monitors/     Monitoring and visualization tools
├── payloads/     Shared message types
├── libs/         Supporting libraries
├── res/          Platform-specific resource bundles (HAL)
└── testing/      Test helpers (network injection, mocks)

Each component is a standalone Rust crate that implements one or more Copper tasks. You add it as a dependency, reference it in copperconfig.ron, and you’re done – no glue code needed.

The ecosystem is actively growing. Browse the components directory in the copper-rs repository to see what’s currently available – you’ll find LiDAR drivers, IMU drivers, camera capture, PID controllers, AprilTag detection, Zenoh and ROS 2 bridges, and more.

How to use a component

Let’s say you want to add a PID controller to your project. The process is:

1. Add the dependency

In your application’s Cargo.toml:

[dependencies]
cu-pid = { path = "../../components/tasks/cu_pid" }
cu-sensor-payloads = { path = "../../components/payloads/cu_sensor_payloads" }

Or if the component is published on crates.io, use the same Copper release for both crates that the rest of your project already uses.

In a workspace, you’d add it to the workspace-level Cargo.toml first:

[workspace.dependencies]
cu-pid = { path = "../../components/tasks/cu_pid" }
cu-sensor-payloads = { path = "../../components/payloads/cu_sensor_payloads" }

Then reference it in the app’s Cargo.toml:

[dependencies]
cu-pid = { workspace = true }
cu-sensor-payloads = { workspace = true }

2. Reference it in copperconfig.ron

Some reusable Copper tasks are generic. cu-pid is one of them, so the usual pattern is to define a small local alias in your app code first:

#![allow(unused)]
fn main() {
use cu_pid::GenericPIDTask;

pub type PidTask = GenericPIDTask<cu_sensor_payloads::ImuPayload>;
}

Then add the task to your configuration using that alias:

(
    id: "pid",
    type: "tasks::PidTask",
    config: {
        "kp": 1.0,
        "ki": 0.1,
        "kd": 0.05,
    },
),

The config section passes parameters that the task reads in its new() constructor. If a component exposes a concrete, non-generic task type, you can reference the crate path directly; for generic tasks like cu-pid, a local alias keeps the RON type path simple.

3. Wire it up

Connect it to your existing tasks:

cnx: [
    (
        src: "imu",
        dst: "pid",
        msg: "cu_sensor_payloads::ImuPayload",
    ),
    (
        src: "pid",
        dst: "motor",
        msg: "cu_pid::PIDControlOutputPayload",
    ),
],

That’s it. Beyond the small alias for generic components, there is no adapter layer. The component is a Copper task like any other – it just happens to live in a separate crate.

Using shared payload types

When two components need to exchange data, they must agree on a message type. This is where the payloads crates come in.

For example, cu-sensor-payloads defines common sensor types that multiple source drivers produce. If you use the BMI088 IMU driver (cu-bmi088), it outputs a type from cu-sensor-payloads. Any downstream task that accepts that same type can consume the data without any conversion.

This is the component ecosystem’s contract: drivers produce standard payload types, and algorithms consume them. Swap a Bosch IMU for an InvenSense IMU, and the downstream pipeline doesn’t change – both produce the same ImuPayload type.

Writing your own reusable components

If you’ve built a task that could be useful to others – or even just to your future self across projects – you can extract it into a component crate:

  1. Create a library crate under components/ (pick the right category).
  2. Move your task struct and its impl into lib.rs.
  3. Move shared message types into the crate or use existing ones from cu-sensor-payloads.
  4. Add it to the workspace members list in the root Cargo.toml.
  5. Reference it from your application’s Cargo.toml and copperconfig.ron.

The workspace template already has placeholder directories with .keep files for each component category. Just replace the placeholder with your crate.

Difference with ROS

In ROS 2, reusing components means installing packages (via apt or building from source) and then referencing their nodes in your launch files. The discovery is runtime-based – nodes find each other through DDS topics.

In Copper, reusing components means adding a Rust dependency and referencing the task type in copperconfig.ron. The wiring is resolved at compile time.

ROS 2Copper
DiscoveryPackages via apt / source buildCrates via Cargo.toml
IntegrationLaunch files + topic remappingcopperconfig.ron + type paths
Message compatibility.msg files + code generationShared Rust types (payloads crates)
ValidationRuntime (topics may not match)Compile time (types must match)
SharingROS package indexcrates.io / git dependencies

The biggest difference is the compile-time guarantee. In ROS, you can wire two nodes together with mismatched message types and only find out when you run the system. In Copper, if your PID controller expects ImuPayload and your driver produces CameraFrame, the compiler tells you immediately.

Automating Tasks with Just

Running a logreader or rendering a graph involves binary names, features, and paths. A justfile gives those commands short names so you can spend your time changing the robot instead of reconstructing command lines.

This chapter uses the templates named in the edition notice and assumes just is available. Start in the project directory from Setting Up, or the workspace root from From Project to Workspace. Run just --list to see the recipes your generated project provides.

Read a recording in two commands

After running the application once, use:

just log
just cl

log reconstructs the structured text with the producing build’s string index. cl exports each recorded CopperList, including task payloads and timestamps. For the first project, look for the source value 42 and processing value 43. Both commands default to the application’s normal log path.

v1.3 The template recipes accept a different application or recording. For a different recording in a single-crate project:

just cl logs/another-session.copper

In the workspace template, the first argument selects the application and the second selects its recording:

just cl cu_example_app apps/cu_example_app/logs/cu_example_app.copper

Recipe arguments are positional. When you target another app, provide its log path too; the default log path still names cu_example_app.

See the graph and the execution schedule

v1.1 The graph answers “what connects to what?” The schedule answers “where and in what order will that work execute?” Both views help you check a RON edit before running hardware.

v1.3 The current templates use these viewer recipes. From either template’s root:

just graph
just sched

The recipes open SVG files in your default viewer. In a single-crate project, the outputs are graph.svg and schedule.svg. In the workspace template they are apps/cu_example_app/graph.svg and apps/cu_example_app/schedule.svg.

For our three-task app, the graph shows src -> t-0 -> sink. Its default serial schedule runs those operations on one worker. After selecting the Pipeline planner, inspect the schedule to see the process stages placed on separate workers and CopperLists overlapping across stages.

Resource tables list the consumers of each resource, including configured LogStream transports. A destination using network.tx, for example, appears as system: logstream (ground) in the Used by column. Use this to check shared hardware ownership alongside the task graph.

Add observed timing

v1.1 A schedule diagram explains placement and ordering; it does not establish how fast your app runs. Record a representative run, then add its measured timings.

v1.3 With the current templates, run:

just graph-log
just sched-log

These recipes use the application’s logreader to obtain statistics from its normal log. Keep the log and configuration from the same application version. Look for an expensive stage or a large gap between stages, then use the performance chapters to investigate it.

Target another configuration

v1.3 In a single-crate project, render an alternate configuration without replacing your application config:

just graph copperconfig.ron graph-check.svg
just sched copperconfig.ron schedule-check.svg

In a workspace, supply the app, config, and output paths:

just graph cu_example_app apps/cu_example_app/copperconfig.ron graph-check.svg
just sched cu_example_app apps/cu_example_app/copperconfig.ron schedule-check.svg

The first argument is the Cargo application package name. Both templates also provide a Rust viewer helper for additional options, including mission selection. For example, in the workspace from the missions exercise, select normal with:

cargo run -p cu29-view-helper -- schedule --app cu_example_app --mission normal --output schedule-normal.svg --open

Use a mission ID that your selected app actually declares.

Replay a failure

The generated recipes run replay through the debug-optimized Cargo profile:

just resim
just resim-debug

resim replays the recording. resim-debug serves a replay session to a remote debug client. Compiler optimization keeps seeking practical while preserving Copper’s debug logs, assertions, and debugger information. See Logging and Replaying Data for the replay workflow.

Automate your own repeated commands

A recipe is a name followed by indented shell commands. In the workspace justfile, add this complete recipe:

run:
  cargo run -p cu_example_app

Then run:

just run

It starts the same application you would launch with the Cargo command. Use just for operations you repeat, keeping explicit arguments for the paths and applications that change.

Advanced Task Features

In the earlier chapters we treated #[derive(Reflect)], impl Freezable, and type Resources<'r> = () as required boilerplate. This chapter explains what each of these does and when you’d want to customize them.

#[derive(Reflect)]

The Reflect derive macro enables runtime introspection on your task struct. Monitoring tools and simulation environments use it to inspect a task’s fields at runtime without knowing the concrete type at compile time.

#![allow(unused)]
fn main() {
#[derive(Reflect)]
pub struct MyController {
    kp: f64,
    ki: f64,
    accumulated_error: f64,
}
}

With Reflect derived, a monitoring UI could display kp, ki, and accumulated_error as live values while the robot is running.

In practice: always add #[derive(Reflect)] to your task structs. It enables monitoring and simulation tooling when you need it.

Freezable – State Serialization

The Freezable trait enables Copper’s deterministic replay system. The runtime periodically takes “snapshots” (keyframes) of every task’s internal state, much like keyframes in a video codec. During replay, it can jump to any snapshot instead of replaying from the very beginning.

Tasks with no state to freeze

For tasks that hold no internal state (or state that can be trivially reconstructed), the empty implementation is all you need:

#![allow(unused)]
fn main() {
impl Freezable for MySource {}
}

This tells Copper “there’s nothing to snapshot for this task.” This is what we’ve been using throughout the book, and it’s correct for all our tasks so far – they don’t carry any state between cycles.

Stateful tasks

If your task maintains state that affects its behavior across cycles – for example, a PID controller accumulating error, a Kalman filter maintaining a covariance matrix, or a counter tracking message sequence numbers – you should implement the freeze and thaw methods.

The Freezable trait uses bincode’s Encoder and Decoder for serialization:

#![allow(unused)]
fn main() {
use cu29::prelude::*;
use cu29::bincode::enc::Encoder;
use cu29::bincode::de::Decoder;
use cu29::bincode::error::{EncodeError, DecodeError};

#[derive(Reflect)]
pub struct PidController {
    kp: f64,
    ki: f64,
    kd: f64,
    accumulated_error: f64,
    previous_error: f64,
}

impl Freezable for PidController {
    fn freeze<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> {
        // Serialize the fields that change at runtime
        cu29::bincode::Encode::encode(&self.accumulated_error, encoder)?;
        cu29::bincode::Encode::encode(&self.previous_error, encoder)?;
        Ok(())
    }

    fn thaw<D: Decoder>(&mut self, decoder: &mut D) -> Result<(), DecodeError> {
        self.accumulated_error = cu29::bincode::Decode::decode(decoder)?;
        self.previous_error = cu29::bincode::Decode::decode(decoder)?;
        Ok(())
    }
}
}

freeze serializes the task’s mutable state using bincode’s Encode trait. thaw deserializes it back using Decode. The runtime calls these automatically at keyframe boundaries.

Rule of thumb: Only serialize fields that change during process(). Configuration fields like kp, ki, kd are set once in new() and don’t need to be frozen – they’ll be reconstructed from copperconfig.ron during replay.

When does this matter?

If you never use Copper’s replay features, the empty impl Freezable is fine for every task. But if you plan to record and replay robot runs (one of Copper’s strongest features), implementing Freezable correctly means the replay system can seek to any point in the log efficiently rather than replaying from the start.

Think of it like a video file: without keyframes, you’d have to decode from the beginning every time you want to seek. With keyframes (state snapshots), you can jump to any point and resume from there. Freezable is how Copper creates those keyframes for your task’s internal state.

A transform that can process independent cycles

v1.3 An image conversion or a fixed arithmetic transform often depends only on its current input and immutable settings. Such work can use CuStatelessTask: its per-cycle callbacks borrow &self, allowing an execution plan to overlap calls for different CopperLists. Use it when each cycle can be processed independently. A controller whose next output depends on an accumulated error belongs in CuTask, with that error included in its frozen state.

Try this in the single-crate project from Writing Tasks. The source still produces MyPayload { value: 42 } and the sink still receives MyPayload.

1. Replace the processing task

In src/tasks.rs, replace the MyTask struct and its trait implementations with this complete block. Keep the file’s existing imports, payload, source, and sink:

#![allow(unused)]
fn main() {
#[derive(Reflect)]
pub struct MyTask {}

impl Freezable for MyTask {}

impl CuStatelessTask for MyTask {
    type Resources<'r> = ();
    type Input<'m> = input_msg!(MyPayload);
    type Output<'m> = output_msg!(MyPayload);

    fn new(_config: Option<&ComponentConfig>, _resources: Self::Resources<'_>) -> CuResult<Self> {
        Ok(Self {})
    }

    fn process(
        &self,
        ctx: &CuContext,
        input: &Self::Input<'_>,
        output: &mut Self::Output<'_>,
    ) -> CuResult<()> {
        if let Some(payload) = input.payload() {
            output.set_payload(MyPayload { value: payload.value * 2 });
            output.tov = input.tov;
            debug!(ctx, "Doubled input: {}", payload.value * 2);
        } else {
            output.clear_payload();
        }
        Ok(())
    }
}
}

The arithmetic uses only the current input. output remains mutable because it belongs to this CopperList; the task instance is shared through &self.

2. Declare the task kind

Replace the t-0 entry in copperconfig.ron with:

(
    id: "t-0",
    type: "tasks::MyTask",
    kind: stateless_task,
),

Keep the existing src -> t-0 -> sink connections. The explicit kind tells Copper to call CuStatelessTask rather than CuTask; it cannot infer this choice from the connections.

3. Run and observe the transform

From the project root:

cargo run

The task reports Doubled input: 84, and the sink reports Sink Received message: 84. Stop the app with Ctrl-C. You have changed the transform’s execution contract while keeping its message type and graph wiring. The serial plan is sufficient for this exercise; overlapping execution requires a plan that schedules independent occurrences concurrently.

Choosing the contract

CuStatelessTask requires Send + Sync. preprocess, process, and postprocess all use &self; construction, start, stop, and restoring a keyframe remain exclusive operations. Immutable calibration or configuration can live in the struct. An empty Freezable implementation is appropriate when there is no changing state to restore.

Any interior mutability or shared resource must preserve the same output when calls overlap. A lock makes access safe but does not make order-dependent algorithm state deterministic. Use CuTask for that state. Stateless nodes cannot use background or anytime-task configuration.

Resources – Hardware Injection

The type Resources<'r> associated type declares what hardware or system resources a task needs. Resources represent physical endpoints – serial ports, GPIO controllers, SPI buses, cameras – or shared services bound through a resource bundle.

Worker thread pools are configured separately, in the runtime section of the config – see Thread Pools, Affinity, and Real-Time Scheduling.

No resources

Most tasks in a simple project don’t need external resources:

#![allow(unused)]
fn main() {
type Resources<'r> = ();
}

This means “give me nothing.” The _resources parameter in new() is just (). This is what we’ve used throughout the book.

The problem resources solve

On a real robot, a sensor driver needs access to a hardware peripheral – say, a serial port or an SPI bus. Where does that handle come from? You have a few options:

  1. Create it inside new() – Works, but what if two tasks need the same bus? Or if the hardware needs platform-specific initialization that your task shouldn’t know about?

  2. Pass it as a global – Not great for testing, portability, or safety.

  3. Have the runtime inject it – This is what Resources does. You declare what you need, and the runtime provides it.

How it works

Resources involve three pieces:

1. A resource provider (called a “bundle”) is declared in copperconfig.ron:

resources: [
    (
        id: "board",
        provider: "crate::resources::BoardBundle",
        config: { "uart_device": "/dev/ttyUSB0" },
    ),
],

The bundle is a Rust type that knows how to create the actual hardware handles (open the serial port, initialize GPIO, etc.).

2. Tasks declare which resources they need in copperconfig.ron:

(
    id: "imu",
    type: "tasks::ImuDriver",
    resources: { "serial": "board.uart0" },
),

This says: “the imu task needs a resource it calls serial, and it should be bound to the uart0 resource from the board bundle.”

3. The task declares a Resources type in its Rust implementation that knows how to pull the bound resources from the runtime’s resource manager:

#![allow(unused)]
fn main() {
impl CuSrcTask for ImuDriver {
    type Resources<'r> = ImuResources<'r>;
    type Output<'m> = output_msg!(ImuPayload);

    fn new(
        config: Option<&ComponentConfig>,
        resources: Self::Resources<'_>,
    ) -> CuResult<Self> {
        // resources.serial is the hardware handle, injected by the runtime
        Ok(Self { port: resources.serial })
    }
    // ...
}
}

Why this design?

Resources keep your tasks portable. An IMU driver doesn’t need to know which serial port to open – that’s a deployment detail. On one robot it might be /dev/ttyUSB0, on another it might be a different bus entirely. By moving the hardware binding to the configuration, the same task code works on different platforms without changes.

Resources can also be shared. A shared bus (like I2C) can be bound to multiple tasks. The resource manager handles the ownership semantics: exclusive resources are moved to a single task, shared resources are behind an Arc and each consumer receives a cloned handle.

v1.1 The same pattern applies to GPU services. With the cu29/cuda feature, a bundle can own a shared CUDA context and CuCudaPool, while tasks pass CuHandle<CudaSliceWrapper<T>> through the graph. Keep CUDA streams task-local unless a component explicitly defines shared-stream ordering. CUDA handles are process-local, so their producing nodes should use logging: (handle_content: none); they cannot currently be reconstructed during replay. CuHandle::storage_id can confirm that two stages saw the same pooled allocation rather than a replacement copy.

When do you need resources?

For the projects in this book, type Resources<'r> = () is all you need. Resources become important when you:

  • Write drivers that talk to real hardware (serial, SPI, GPIO, USB)
  • Need multiple tasks to share a hardware bus
  • Want the same task to work across different boards without code changes
  • Want to swap real hardware for mocks in testing

The examples/cu_resources_test/ in the copper-rs repository is a complete working example that demonstrates bundles, shared resources, owned resources, and mission-specific resource bindings.

Composing resource bundles

v1.2 A hardware handle sometimes needs several layers before a task can use it: a UART becomes a configured radio, and the radio becomes a framed telemetry transport. A resource bundle can consume another bundle’s output, so each layer owns its startup and configuration while tasks receive the finished handle.

Inspect a complete radio application

The HC-12 component contains a complete example of this composition. From the root of the copper-rs checkout used for this book, run its software checks:

cd components/res/cu_hc12
just

The tests exercise radio startup with simulated serial, pin, and delay resources; no radio hardware is needed. For the application wiring, read examples/echo.ron and examples/echo.rs in this component. The RON configuration declares three bundles:

  • board exports a nonblocking UART and its adapter’s RTS pin.
  • timing exports a startup delay.
  • radio consumes those three resources and exports the configured HC-12 serial handle. The byte bridge takes that handle and echoes received bytes.

In that existing radio declaration, the input bindings are:

resources: {
    "serial": "board.serial0",
    "set": "board.serial0_rts",
    "delay": "timing.delay",
},

The left side names an input required by the consuming provider; the right side names an output from another bundle. Copper constructs providers in dependency order. Cycles, missing providers, inactive mission providers, and unknown binding names fail compilation. Concrete resource types and exclusive ownership are checked at startup. Changing these input bindings requires a rebuild.

Run with two radios

On Unix, connect two HC-12 radios through USB-to-TTL adapters whose compatible 3.3 V active-low RTS# outputs connect to SET, with common grounds. Both UARTs must match their modules’ saved baud rate (factory default: 9600, 8N1). The example enables the Linux resource provider’s serial-rts feature. Its serial0_rts: true configuration exports the owned SET-pin handle and disables hardware flow control.

From the same component directory, replace these device paths with the two adapters on your machine, then run:

just pair /dev/ttyUSB0 /dev/ttyUSB1

The program uses the baud rate and channel from examples/echo.ron, configures both radios, and verifies 1-, 31-, 128-, and 256-byte transfers in each direction. A mismatch or a five-second timeout fails the check. Run it while both radios are idle. To echo on just one end of a pair instead, edit the device and channel in examples/echo.ron, then run just echo.

Adapt the layers

A composing provider declares its inputs with resources!, exposes their names through ResourceBundle::INPUT_NAMES, and retrieves them with bundle.inputs::<Inputs>(manager)? in its build method. Consume owned inputs or clone shared handles, then register the wrapped output. See the HC-12 provider for a complete implementation.

For telemetry, a SerialLogStreamTxResources<Radio> bundle can consume radio.serial and export telemetry.tx for a LogStream destination. This replaces the byte bridge as the radio’s owner. Follow the component’s LogStream setup for concrete type aliases, feature setup, and a conservative link budget. Resource composition establishes startup dependencies; it does not add per-cycle task-message edges.

Summary

FeaturePurposeWhen to customize
#[derive(Reflect)]Runtime introspection for monitoring/simulationAlways use the derive; the feature flag controls cost
impl FreezableState snapshots for deterministic replayImplement freeze/thaw for stateful tasks
type Resources<'r>Hardware/service dependency injectionDeclare resource types when your task needs hardware handles

For most of the projects you’ll build while learning Copper, the defaults are fine: #[derive(Reflect)] on the struct, empty impl Freezable, and type Resources<'r> = (). These become important as you move from prototyping to deploying on real hardware with real replay requirements.

Writing a Bridge

So far, our Copper apps have been made from sources, tasks, and sinks. Those are enough when every endpoint is independent. A bridge is for the next case: one external connection or protocol has several logical channels that should share the same state.

This chapter builds a tiny bridge with one incoming channel and one outgoing channel. It does not talk to real hardware. That is intentional: the goal is to learn the bridge shape without also learning serial ports, sockets, ROS 2, CAN, or motor protocols.

The bridge will keep two pieces of state:

#![allow(unused)]
fn main() {
connected: bool,
messages_seen: u64,
}

connected changes in the bridge lifecycle. messages_seen increments when the bridge receives a command. When the bridge sends a status message, it can see the same messages_seen value. That small bit of shared state is the reason this belongs in one bridge instead of two unrelated tasks.

The graph will look like this:

counter/command_in  ──▶  CountCommands  ──▶  counter/status_out
      Rx channel              task                 Tx channel

Step 1: Add message types

Create a new file called src/messages.rs:

#![allow(unused)]
fn main() {
use cu29::bincode::{Decode, Encode};
use cu29::prelude::*;
use serde::{Deserialize, Serialize};

#[derive(Default, Debug, Clone, Serialize, Deserialize, Encode, Decode, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct CommandPayload {
    pub requested_count: u64,
}

#[derive(Default, Debug, Clone, Serialize, Deserialize, Encode, Decode, Reflect)]
#[bincode(crate = "cu29::bincode")]
pub struct StatusPayload {
    pub cycles: u64,
}
}

These are normal Copper payloads. The bridge receives CommandPayload from the outside world and sends StatusPayload back out.

Checkpoint: after this step, src/messages.rs should contain only payload types. It should not mention bridges or tasks yet.

Step 2: Add a task between the bridge channels

Edit src/tasks.rs and add this task:

#![allow(unused)]
fn main() {
use crate::messages::{CommandPayload, StatusPayload};
use cu29::prelude::*;

#[derive(Default, Reflect)]
pub struct CountCommands {
    cycles: u64,
}

impl Freezable for CountCommands {}

impl CuTask for CountCommands {
    type Resources<'r> = ();
    type Input<'m> = input_msg!(CommandPayload);
    type Output<'m> = output_msg!(StatusPayload);

    fn new(
        _config: Option<&ComponentConfig>,
        _resources: Self::Resources<'_>,
    ) -> CuResult<Self> {
        Ok(Self { cycles: 0 })
    }

    fn process<'i, 'o>(
        &mut self,
        _ctx: &CuContext,
        input: &Self::Input<'i>,
        output: &mut Self::Output<'o>,
    ) -> CuResult<()> {
        if input.payload().is_some() {
            self.cycles += 1;
            output.set_payload(StatusPayload {
                cycles: self.cycles,
            });
        } else {
            output.clear_payload();
        }
        Ok(())
    }
}
}

This task is deliberately small. It counts how many command messages reached it and emits that count as status.

If your tasks.rs already has use cu29::prelude::*;, do not add it twice. Add only the use crate::messages::{CommandPayload, StatusPayload}; line and the CountCommands struct/implementation.

Checkpoint: the task should compile as an ordinary CuTask: one input payload, one output payload, no resources.

Step 3: Declare bridge channels

Create a new file called src/bridges.rs:

#![allow(unused)]
fn main() {
use crate::messages::{CommandPayload, StatusPayload};
use cu29::prelude::*;

rx_channels! {
    pub struct CounterRxChannels : CounterRxId {
        command_in => CommandPayload = "counter/command_in",
    }
}

tx_channels! {
    pub struct CounterTxChannels : CounterTxId {
        status_out => StatusPayload = "counter/status_out",
    }
}
}

The rx_channels! macro declares channels that enter Copper from the outside world. The tx_channels! macro declares channels that leave Copper for the outside world.

Each channel has three pieces:

PieceExampleMeaning
Channel idcommand_inName used by the mission config
Payload typeCommandPayloadMessage type on that channel
Default route"counter/command_in"External route/topic/path if the config does not override it

Checkpoint: after this step, the bridge has channel declarations but no bridge struct yet. That is normal.

Step 4: Implement the bridge state

In the same src/bridges.rs file, add the bridge struct:

#![allow(unused)]
fn main() {
#[derive(Default, Reflect)]
pub struct CounterBridge {
    connected: bool,
    messages_seen: u64,
}

impl Freezable for CounterBridge {}
}

This is the shared state owned by the bridge instance.

For this tutorial, messages_seen is used only for debug output, so the empty Freezable implementation is fine. If bridge state affects real outputs during replay, implement freeze and thaw as shown in the previous chapter.

Checkpoint: the state should belong to the bridge, not to a global variable. Bridge methods will receive &mut self, so they can update this state safely.

Step 5: Implement CuBridge::new

Continue in src/bridges.rs:

#![allow(unused)]
fn main() {
impl CuBridge for CounterBridge {
    type Resources<'r> = ();
    type Tx = CounterTxChannels;
    type Rx = CounterRxChannels;

    fn new(
        _config: Option<&ComponentConfig>,
        _tx_channels: &[BridgeChannelConfig<<Self::Tx as BridgeChannelSet>::Id>],
        _rx_channels: &[BridgeChannelConfig<<Self::Rx as BridgeChannelSet>::Id>],
        _resources: Self::Resources<'_>,
    ) -> CuResult<Self>
    where
        Self: Sized,
    {
        Ok(Self {
            connected: false,
            messages_seen: 0,
        })
    }
}

Unlike a task constructor, a bridge constructor receives the configured Tx and Rx channel lists. Real bridges use those lists to read routes, topic names, baud rates, QoS settings, or other per-channel configuration. This first bridge ignores them.

Checkpoint: this code opens the impl CuBridge for CounterBridge block but does not close it yet. The next steps add lifecycle, receive, and send methods inside the same block.

Step 6: Add lifecycle methods

Still inside the impl CuBridge for CounterBridge block, add:

#![allow(unused)]
fn main() {
    fn start(&mut self, _ctx: &CuContext) -> CuResult<()> {
        self.connected = true;
        debug!("CounterBridge connected");
        Ok(())
    }

    fn stop(&mut self, _ctx: &CuContext) -> CuResult<()> {
        self.connected = false;
        debug!("CounterBridge disconnected after {} messages", self.messages_seen);
        Ok(())
    }
}

start runs before bridge I/O begins. stop runs when the application shuts down. A real bridge might open and close a serial port here. Our tutorial bridge only flips a boolean.

Checkpoint: after this step, connected is changed only by lifecycle methods. That makes it easy to reason about.

Step 7: Receive commands

Add the receive method inside the same impl block:

#![allow(unused)]
fn main() {
    fn receive<'a, Payload>(
        &mut self,
        _ctx: &CuContext,
        channel: &'static BridgeChannel<CounterRxId, Payload>,
        msg: &mut CuMsg<Payload>,
    ) -> CuResult<()>
    where
        Payload: CuMsgPayload + 'a,
    {
        match channel.id() {
            CounterRxId::CommandIn => {
                self.messages_seen += 1;
                msg.set_payload(Payload::default());
                debug!("CounterBridge received command {}", self.messages_seen);
            }
        }
        Ok(())
    }
}

The method is generic over Payload because a real bridge can have many Rx channels with different payload types. This example has only one Rx channel, so the match has one arm.

The important part is the state change:

#![allow(unused)]
fn main() {
self.messages_seen += 1;
}

That state will still be available when the bridge sends status later in the same graph.

Checkpoint: receive should write into the provided msg buffer with msg.set_payload(...). Do not allocate a new message.

Step 8: Send status

Finish the impl CuBridge block with send and the final closing brace:

#![allow(unused)]
fn main() {
    fn send<'a, Payload>(
        &mut self,
        _ctx: &CuContext,
        channel: &'static BridgeChannel<CounterTxId, Payload>,
        msg: &CuMsg<Payload>,
    ) -> CuResult<()>
    where
        Payload: CuMsgPayload + 'a,
    {
        match channel.id() {
            CounterTxId::StatusOut => {
                if self.connected {
                    debug!(
                        "CounterBridge sent status; bridge has seen {} commands",
                        self.messages_seen
                    );
                } else {
                    warn!("CounterBridge tried to send while disconnected");
                }
                debug!("Status payload present: {}", msg.payload().is_some());
            }
        }
        Ok(())
    }
}
}

This is where the example shows shared bridge state. receive increments messages_seen; send reads it. Both methods are part of the same bridge instance.

Checkpoint: the send method should consume the msg passed by the runtime. It should not call the CountCommands task directly. The graph decides what data reaches the Tx channel.

Step 9: Register the new modules

Edit src/main.rs. Near the existing module declarations, add:

#![allow(unused)]
fn main() {
pub mod bridges;
pub mod messages;
}

If your file already has:

#![allow(unused)]
fn main() {
pub mod tasks;
}

then the top of the file should now look like:

#![allow(unused)]
fn main() {
pub mod bridges;
pub mod messages;
pub mod tasks;
}

Checkpoint: copperconfig.ron will reference bridges::CounterBridge, tasks::CountCommands, and messages::..., so all three modules must be visible from the crate root.

Step 10: Wire the bridge in copperconfig.ron

Replace the simple three-task pipeline with this bridge graph:

(
    tasks: [
        (
            id: "count_commands",
            type: "tasks::CountCommands",
        ),
    ],
    bridges: [
        (
            id: "counter",
            type: "bridges::CounterBridge",
            channels: [
                Rx(
                    id: "command_in",
                    route: "counter/command_in",
                ),
                Tx(
                    id: "status_out",
                    route: "counter/status_out",
                ),
            ],
        ),
    ],
    cnx: [
        (
            src: "counter/command_in",
            dst: "count_commands",
            msg: "messages::CommandPayload",
        ),
        (
            src: "count_commands",
            dst: "counter/status_out",
            msg: "messages::StatusPayload",
        ),
    ],
)

The important syntax is the endpoint path:

bridge_id/channel_id

counter/command_in is an Rx channel, so it can only be a source endpoint.
counter/status_out is a Tx channel, so it can only be a destination endpoint.

Checkpoint: the direction should read naturally:

external command -> bridge Rx -> task -> bridge Tx -> external status

If you accidentally wire a Tx channel as a source, or an Rx channel as a destination, Copper should reject the configuration.

Step 11: Build and run

Run a compile check first:

cargo check

If that passes, run the app:

cargo run

You should see debug logs showing that the bridge connected, received commands, and sent status messages. The exact output depends on your logger and monitor setup, but the important pattern is:

CounterBridge connected
CounterBridge received command 1
CounterBridge sent status; bridge has seen 1 commands

Checkpoint: if you see both receive and send logs, the bridge is wired correctly. If you see receive logs but no send logs, check the count_commands -> counter/status_out connection. If you see no bridge logs, check that pub mod bridges; exists in main.rs and that copperconfig.ron has a bridges section.

What this example leaves out

This bridge is intentionally fake. It does not open a socket, parse bytes, or talk to a device. Those details are transport-specific, and adding them now would hide the important shape:

  • Declare Rx and Tx channels.
  • Store shared state in the bridge struct.
  • Use start and stop for connection lifecycle.
  • Use receive to fill Copper messages from outside data.
  • Use send to consume Copper messages and write them outside.
  • Wire bridge channels with bridge_id/channel_id in copperconfig.ron.

Once that shape is clear, real bridges are mostly a matter of replacing the fake receive and send bodies with protocol code.

After you finish this chapter, the cleanest examples to read are:

ExampleWhy read it
examples/cu_bridge_testShows bridge scheduling and graph wiring with small in-repo bridge implementations. Start here if you want to see Rx-to-task, task-to-Tx, loopback, and fanout shapes.
examples/cu_resources_test/src/bridges.rsShows a compact custom bridge that receives injected resources. Read this after the resources chapter.
examples/cu_zenoh_bridge_demoShows a real middleware bridge used by two Copper apps. This is a good next step once the simple bridge API is familiar.

There are also protocol-specific bridges such as MSP, CRSF, DSHOT, ROS 2, and Iceoryx2. Those are useful references, but they are not the first examples to read: each one adds domain-specific protocol details on top of the bridge mechanics.

Keep sensor data fresh with a Zenoh bridge

v1.3 A network publisher can send data faster than your robot processes it. If the receiver queues every sample, the control loop can keep receiving messages while quietly falling seconds behind. For a sensor such as odometry, you usually want the newest available reading. For a command stream, you may need every event.

cu-zenoh-bridge makes that choice per receive channel. It consumes at most one queued sample per receive call. The default is a one-entry ring: a new arrival replaces the queued older sample.

Try two apps on one machine

Start from the root of the copper-rs checkout used for this book. Build both demo binaries before starting either process:

cargo build -p cu-zenoh-bridge-demo --bin zenoh-ping --bin zenoh-pong

In terminal 1, from that root:

cargo run -p cu-zenoh-bridge-demo --bin zenoh-pong -- --iterations 100 --instance-id 2

While it is running, use terminal 2 from the same root:

cargo run -p cu-zenoh-bridge-demo --bin zenoh-ping -- --iterations 20 --instance-id 1

The apps discover each other as local Zenoh peers. Look for replies tagged pong-bincode, pong-json, and pong-cbor in the ping app. Each channel uses its configured representation over one bridge session.

Choose what waits for the next iteration

In examples/cu_zenoh_bridge_demo/pong_config.ron, replace the ping_bin Rx entry inside the bridge’s channels list with this explicit latest-sample policy:

Rx(
    id: "ping_bin",
    route: "demo/ping/bin",
    config: {
        "wire_format": "bincode",
        "queue_mode": "ring",
        "ring_size": 1,
    },
),

Rebuild and rerun the two commands. The wiring and payload types stay the same; this setting determines which arrival the next receive call reads. In your own sensor channel, use the same config entries. Setting ring_size alone also selects a ring of that depth.

To retain every queued event instead, replace that entry with:

Rx(
    id: "ping_bin",
    route: "demo/ping/bin",
    config: {
        "wire_format": "bincode",
        "queue_mode": "fifo",
    },
),

FIFO delivers queued samples in arrival order. Use it for commands or events only when the consumer can keep up: its queue can grow without bound if arrivals outpace receive calls. A deeper ring keeps a bounded backlog and discards the oldest queued sample when full; depth one gives the freshest pending sample.

To see the difference, in src/bin/pong.rs within the demo replace both Duration::from_millis(200) sleeps with Duration::from_millis(1000). Repeat the runs first with the ring and then with FIFO. Ping still sends every 200 ms. With the ring, the replies’ sequence numbers skip intermediate arrivals. With FIFO, replies progress through the older queued numbers. Exact numbers depend on peer discovery and timing; watch the increasing lag from the current ping sequence. Restore the sleeps after the exercise.

Apply it to a robot

Choose a queue policy for each Rx channel according to what the message means:

DataStarting choiceReason
Current pose, temperature, or sensor samplering, depth 1Old samples should not delay the current reading
A short burst whose recent history mattersring, bounded depthRetain a finite backlog
Commands and events that must each be consumedfifoPreserve queued events while ensuring the consumer keeps up

ring_size must be positive. Set queue_mode on Rx channels; a Tx setting is rejected during construction. During replay, Copper injects recorded bridge outputs, so the live subscriber’s queue policy does not change the recorded values.

When to make a bridge

Use a bridge when the channels belong to the same external connection or protocol:

  • One serial port with commands in and telemetry out
  • One CAN bus with several message IDs
  • One middleware session with several topics
  • One ESC bus with command and telemetry channels

Use plain sources and sinks when the endpoints are independent and do not need shared transport state.

Summary

A bridge is not just a source and a sink placed next to each other. It is one component that owns shared protocol state and exposes several typed channels to Copper. In this chapter, CounterBridge owned connected and messages_seen; its Rx and Tx methods both used that state through one bridge instance.

The small counter example is not useful as a robot driver, but it is useful as a template: start with channel declarations, add the bridge state, implement CuBridge, and then wire the channels into the graph.

Interfacing a Simulator

The usual simulator integration is an event callback loop:

simulator tick
    -> fill Copper's simulated inputs for this tick
    -> let Copper run the tasks that should run
    -> read Copper's outputs
    -> apply those outputs to the simulator
    -> advance the simulator

That is the mental model to start with. Copper does not currently ship a ready-made integration for every simulator. The Bevy examples are the concrete examples in this repository, and users have also integrated Copper successfully with simulators such as Isaac Sim, CARLA, and MuJoCo. In all of those cases, the job is to write a thin adapter at the simulator boundary.

If the simulator runs in another process, that adapter can use whatever RPC or message transport fits your setup: sockets, shared memory, middleware topics, or the simulator’s own API. The important part is still the same: each simulator step produces observations for Copper, Copper runs the graph, and commands are sent back to the simulator at a known point in the simulation step.

The callback shape

In simulation mode, Copper generates a SimStep enum from your copperconfig.ron. Each variant corresponds to one task step in the graph.

Your simulator callback receives those steps while Copper is running an iteration. For each step it returns one of two common answers:

Return valueMeaning
SimOverride::ExecuteByRuntimeCopper should run that task normally. Use this for estimation, control, planning, and other algorithm tasks.
SimOverride::ExecutedBySimThe simulator handled that step. Use this for simulated sensors and actuators.

The common pattern is:

  1. Build the Copper runtime in sim_mode = true.
  2. Register a simulation callback.
  3. On each simulator tick, update Copper’s clock to the simulator time.
  4. When Copper reaches a simulated hardware task, fill its output from simulator state.
  5. When Copper reaches a simulated actuator task, read the command and store it for the simulator.
  6. Let all ordinary algorithm tasks run normally.

For example, a balancing robot simulator might do this:

Bevy physics tick
    -> set Copper time to the current physics time
    -> encoder task output is filled from wheel position in the sim
    -> angle sensor task output is filled from body orientation in the sim
    -> controller tasks run normally
    -> motor task output is captured
    -> motor force is applied to the simulated body

The controller does not know this is a simulation. It sees the same kind of messages it would see on the real robot.

What to implement first

For any simulator that can call into Copper directly, start with a small adapter layer owned by the simulator:

#![allow(unused)]
fn main() {
struct MySimCopperAdapter {
    copper: MyRobotSim,
    clock: RobotClock,
    clock_mock: RobotClockMock,
    last_motor_command: MotorCommand,
}
}

Then wire it into your simulator’s tick/update function:

#![allow(unused)]
fn main() {
fn simulator_tick(adapter: &mut MySimCopperAdapter, sim: &mut MySimulator) -> CuResult<()> {
    adapter.clock_mock.set_value(sim.time_nanos());

    let mut callback = |step| {
        match step {
            my_robot::SimStep::Imu(CuTaskCallbackState::Process(_, output)) => {
                output.set_payload(sim.read_imu());
                SimOverride::ExecutedBySim
            }

            my_robot::SimStep::Motor(CuTaskCallbackState::Process(input, _output)) => {
                if let Some(command) = input.payload() {
                    adapter.last_motor_command = *command;
                }
                SimOverride::ExecutedBySim
            }

            _ => SimOverride::ExecuteByRuntime,
        }
    };

    adapter.copper.run_one_iteration(&mut callback)?;
    sim.apply_motor_command(adapter.last_motor_command);
    sim.step();
    Ok(())
}
}

Treat this as pseudocode: the generated runtime type and step names come from your copperconfig.ron, and your payload types will be your own. The important point is where the simulator code lives. It lives in the simulation adapter, not inside the controller tasks.

In practice, simulated sensors write to output, simulated actuators read from input, and the estimation/control/planning tasks return ExecuteByRuntime.

For a multi-output source, the Process callback receives the same complete output tuple declared by the real CuSrcTask. Fill each simulated port in place, then return ExecutedBySim; downstream tasks consume those messages through the normal Copper graph.

Types to look for

When adapting a simulator, these are the Copper pieces to grep for in existing examples:

ItemWhy it matters
#[copper_runtime(..., sim_mode = true)]Generates the simulation-aware runtime and SimStep enum for your graph.
SimStepThe generated enum you match in the simulation callback. The variants come from task ids in copperconfig.ron.
CuTaskCallbackState::Process(input, output)Gives the callback access to a task’s input and output messages for that step.
SimOverride::ExecutedBySimReturn this when the simulator filled the output or consumed the actuator command.
SimOverride::ExecuteByRuntimeReturn this when Copper should run the task normally.
RobotClock::mock() and RobotClockMock::set_value(...)Let the simulator drive Copper time from simulation time.
run_one_iteration(&mut callback)Runs one Copper iteration while giving your callback a chance to handle simulated steps.

The most useful files to read are:

# From a checkout of the main Copper repository:
rg "sim_mode|SimStep|SimOverride|CuTaskCallbackState|run_one_iteration|RobotClock::mock" extra-examples/examples/cu_rp_balancebot/src
rg "sim_mode|SimStep|SimOverride|CuTaskCallbackState|run_one_iteration|RobotClock::mock" extra-examples/examples/cu_flight_controller/src

The first line in the tick function is not incidental:

#![allow(unused)]
fn main() {
adapter.clock_mock.set_value(sim.time_nanos());
}

For simulation, Copper should usually see simulation time, not wall-clock time. If the simulator pauses, Copper time pauses. If the simulator runs faster than real time, the timestamps still describe the simulated run. Decide here whether one simulator tick equals one Copper tick, whether Copper runs on every physics substep, and whether commands apply immediately or on the next simulator step.

Pick an ownership pattern

Choose the shape that matches who owns the main loop.

The simulator owns the loop

This is the most common shape for an interactive simulator.

Your simulator has the main loop. Each simulator tick calls Copper, Copper runs the graph, and the simulator applies the resulting commands.

Use this when:

  • the simulator has a physics or rendering loop
  • the simulator already controls pause, reset, and stepping
  • you want interactive visualization
  • you want one Copper tick per simulator tick, or one Copper tick every N simulator ticks

The Bevy examples in the main Copper repository’s extra-examples/ folder use this style:

ExampleWhat it demonstrates
examples/cu_rp_balancebotBevy owns the physics tick. The simulation callback fills encoder/ADC-style task outputs from the world and applies motor output back into the world.
examples/cu_flight_controllerBevy owns the flight simulation. Copper runs the flight-control and compute graphs while callbacks preempt hardware endpoints, including the multi-output ZED source, and downstream tasks consume their normal Copper messages.

These examples are useful even if you are not using Bevy. Look for the simulator callback layer: that is the part to copy conceptually.

In cu_rp_balancebot, the callback does exactly this:

  • Balpos is a simulated angle sensor, so the callback writes an ADC payload from the simulated body orientation.
  • Railpos is a simulated encoder, so the callback writes encoder ticks from the simulated cart position.
  • Motor is a simulated actuator, so the callback reads the motor command and applies a force in the physics world.
  • all other steps return ExecuteByRuntime.

Copper owns the loop

Use this when the simulator is a small model or library that Copper can step directly.

In this shape, the Copper app drives the timing. Simulated sensor tasks read from the model, normal tasks run, and simulated actuator tasks write back to the model. This is a good fit for headless tests and deterministic algorithm checks.

Use this when:

  • the simulator has no strict main loop of its own
  • rendering is not important
  • you want batch tests or faster-than-real-time runs
  • the simulator can be stepped from Rust without blocking

Useful Resources

Throughout this book, we’ve been learning Copper by building projects step by step. But the copper-rs repository itself is a goldmine of resources that go far beyond what we’ve covered. This chapter is a guided tour of where to find documentation, examples, and tools in the copper-rs repository.

Official documentation

The Copper project maintains a documentation site with detailed guides on every aspect of the framework:

copper-project.github.io/copper-rs

Here are the most useful pages:

PageWhat you’ll find
Copper Application OverviewFull walkthrough of a task graph and runtime
Copper RON Configuration ReferenceComplete schema for copperconfig.ron
Task LifecycleDetailed explanation of new, start, process, stop
Modular ConfigurationIncludes, parameter substitution, composition
Build and DeployBuilding for different targets and deploying
Supported PlatformsWhat hardware and OS combinations are supported
Baremetal DevelopmentRunning Copper on microcontrollers without an OS
Copper Component CatalogLive catalog of drivers, algorithms, and bridges
FAQCommon questions and answers

Bookmark the configuration reference in particular – it’s the definitive source for every field you can put in copperconfig.ron.

The examples/ directory

The repository ships with a large collection of working examples. Each one is a complete, buildable project with its own copperconfig.ron, task implementations, and main.rs. When you want to learn a new feature, reading the corresponding example is often the fastest way to understand how it works in practice.

Browse them at github.com/copper-project/copper-rs/tree/master/examples, or run any example from the repository root:

cargo run -p cu_missions

Project templates

We used cu_project (see Setting Up Your Environment) and cu_full (see From Project to Workspace) to scaffold our projects, but the bundled template tree has more to offer:

support/cargo_cunew/templates/
├── cu_project/     # Simple single-crate project
├── cu_full/        # Multi-crate workspace with components
└── README.md       # Detailed usage guide

The support/cargo_cunew/templates/README.md guide documents both cargo cunew and the direct cargo generate developer path, including the just gen-workspace / just gen-project shortcuts for template work inside the repo.

Docker images

The support/docker/ directory contains Dockerfiles for building Copper projects in containerized environments:

  • Dockerfile.ubuntu – Standard Ubuntu-based build environment
  • Dockerfile.ubuntu-cuda – Ubuntu with CUDA support for GPU-accelerated tasks

These are useful for CI pipelines or for building on machines where you don’t want to install the full Rust toolchain.

Cross-compilation support

The support/ directory also includes helpers for cross-compiling and deploying to embedded targets:

just cross-armv7-deploy     # Build and deploy to ARMv7 (e.g., Raspberry Pi)
just cross-riscv64-deploy   # Build and deploy to RISC-V 64

These commands build release binaries for the target architecture and scp them to the robot along with the copperconfig.ron.

The Discord

Finally, the Copper project has an active Discord server where you can ask questions, share your projects, and get help from the community and the framework authors.

Python Support

Copper has first-class Python support, but there are two very different ways to use it.

If you only remember one thing from this chapter, remember this:

Python for offline log analysis is reasonable. Python on the live task path is a prototyping hack and should not be treated as a production architecture.

Catalog-Based Offline Analysis

v1.3 If you want to analyse a self-describing recording without rebuilding its application or registering Rust payload types, use Copper’s catalog reader. It returns ordinary Python dictionaries for the recorded values.

Reuse the wheel recording from Self-Describing Logs. On Linux, with Rust and Python 3 available, run this complete sequence from the copper-rs repository root, using the checkout named in the edition notice:

cargo build -p cu29-export --lib --features python-extension-module,self-describing-logs
mkdir -p target/python
cp target/debug/libcu29_export.so target/python/libcu29_export.so
PYTHONPATH=target/python python3 - <<'PYTHON'
import libcu29_export as cu

path = "examples/cu_self_describing_logs/logs/wheel.copper"
catalog = cu.value_decode_catalog_unified(path)
count = 0
for cl in cu.copperlist_value_iterator_unified(path):
    print(cl["id"], cl["msgs"][0]["payload"])
    count += 1
assert count == 10
PYTHON

python-extension-module builds an importable extension and enables the Python API; self-describing-logs enables the catalog reader. The temporary PYTHONPATH makes this built module available to that Python command.

You see the source wheel sample for each of the ten cycles. Inspect the catalog when you need field descriptions or storage units for plotting.

For a multi-run recording, pass run=1 to copperlist_value_iterator_unified to select the zero-based run index you found with list-runs. A single-run log needs no run argument. value_decode_catalog_unified takes only the path: its shared catalog describes all compiled missions. Each CopperList uses the selected run’s mission and output order.

Each cycle dictionary has id and msgs, whose entries retain payload, time of validity, metadata, and capture status. Integers keep full precision. A corrupt record raises IOError and stops iteration. These APIs are experimental and run offline.

Application-Specific Offline Analysis

If you already expose Rust payload types through an application-specific PyO3 module, you can read typed CopperLists alongside structured logs and runtime lifecycle records. The cu_flight_controller example shows this approach in src/python_module.rs. Use it when your analysis needs those Rust types; the catalog reader above supplies ordinary dictionaries without that registration.

Runtime Python Tasks

Copper also provides cu-python-task, which lets one task delegate its process(...) body to a Python script.

This is intentionally a prototyping feature, not a performance feature.

The reason is simple: as soon as Python enters the live execution path, you lose the main properties Copper is designed to preserve:

  • low latency
  • low jitter
  • predictable allocation behavior
  • tight control over the realtime path

Python allocates constantly, and the Rust/Python boundary adds even more overhead. That is enough to ruin the realtime characteristics of the stack. Compared to a native Rust Copper task, the performance is abysmal.

The Two Execution Modes

process

In process mode, Copper spawns a separate Python interpreter and exchanges length-prefixed CBOR frames over stdin/stdout.

Advantages:

  • the GIL is not inside the Copper process
  • interpreter failures are more isolated

Costs:

  • CBOR serialization and deserialization every cycle
  • copies and allocations every cycle
  • interprocess overhead
  • more latency and jitter

embedded

In embedded mode, Copper calls Python directly inside the process through PyO3.

Advantages:

  • no child process
  • no extra CBOR IPC layer
  • usually somewhat less overhead than process

Costs:

  • the GIL is now inside the Copper process
  • Python exceptions happen inside the runtime process
  • values still have to be converted and allocated every cycle
  • still not suitable for realtime use

In this workspace, embedded mode is also not supported on macOS.

What It Is Actually For

The intended workflow is narrow:

  1. prototype one task quickly in Python
  2. confirm the algorithm is doing the right thing
  3. rewrite that task in Rust immediately afterward

If you want, you can use an LLM to draft the Rust rewrite once the Python behavior is stable. But the final destination should still be Rust, not a permanent Python task.

Use this feature as a disposable bridge, not as the architecture.

Performance Basics

This chapter should serve two different readers:

Start with one simple idea

By default, Copper runs one CopperList at a time on one thread.

That is the baseline model. Most performance questions are just variations of:

  • is one task too slow?
  • is the end of the CopperList too slow?
  • is the OS getting behind on disk writeback?
  • does this graph have enough CPU work to profit from overlap?

Those are different problems, so they do not use the same tuning knob.

What one Copper cycle looks like

At a high level, one cycle looks like this:

  1. Create or reuse a CopperList.
  2. Run the generated DAG in topological order.
  3. Record timings and monitoring data.
  4. Serialize and log the completed CopperList.
  5. Recycle the CopperList and start the next cycle.

That last part is important. Even when the task path is fast, the cycle is not really over until Copper has dealt with the finished CopperList.

Two places where time is spent

When a user says “my loop is too slow”, the time is usually being lost in one of two places:

Inside the DAG

This means one or more tasks are expensive.

Typical signs:

  • one task dominates the LAT tab
  • end-to-end latency is high because task execution itself is high

Typical fixes:

  • optimize one task
  • run one task in the background
  • parallelize inside one task
  • use parallel-rt if the graph is a long CPU pipeline

After the DAG finishes

This means the task work is acceptable, but the completed CopperList is expensive to finish.

Typical signs:

  • task latencies look acceptable
  • the BW tab shows significant serialization or disk cost

Typical fixes:

  • async-cl-io
  • reducing logged data
  • mmap-fsync if the problem appears later under writeback pressure

What Copper does with the finished CopperList

Copper’s unified logger writes the finished CopperList to a memory-mapped log.

By default:

  • the task outputs are serialized at the end of the cycle
  • the logger writes to memory-mapped slabs
  • when a section is closed, Copper asks the OS to flush that range asynchronously

This is usually fast enough, but it creates an important beginner trap:

  • the system can look fine at first
  • dirty pages accumulate in the background
  • later, the machine starts paying the bill

So “it collapses after a while” is often a logging and writeback story, not a task story.

What to do next

Once this mental model is clear, do not keep guessing. Look at the numbers:

The normal flow should be:

  1. understand the baseline model
  2. inspect LAT, BW, MEM, and log-stats
  3. follow the troubleshooting tree
  4. only then choose a tuning knob

Tuning Knob Reference

This is the fast path for readers who already know what kind of bottleneck they have.

KnobWhat it changesTypical symptom
async-cl-ioMoves CopperList serialization and logging to a dedicated threadTask latencies are fine, but end-of-CL overhead is too high
parallel-rt + runtime.plannerEnables an explicitly selected concurrent execution planSeveral CPU-bound stages are back to back and the global rate is too low
mmap-fsyncForces file sync_all() on section flushThe system runs fine for a while, then collapses under dirty-page / writeback pressure
background: trueRuns one compatible source or task on the background thread pool and returns None while it is still busyOne isolated stage is too slow, but it does not need to block every cycle
Task-local thread pool / parallel forParallelizes the internals of one taskOne hot task has obvious internal data parallelism
logging: (enabled: false) or enable_task_logging: falseReduces logged bytesBandwidth and serialized size are too high
logging.copperlist_countIncreases the number of preallocated in-flight CopperListsAsync or parallel modes stall because they run out of CopperList slots

Feature switches

The runtime features are Cargo features on cu29. A simple way to expose them in your app is to forward them from your crate:

[features]
async-cl-io = ["cu29/async-cl-io"]
parallel-rt = ["cu29/parallel-rt"]
mmap-fsync = ["cu29/mmap-fsync"]

Then you can run:

cargo run --features async-cl-io
cargo run --features parallel-rt
cargo run --features mmap-fsync

Select a pipeline plan

v1.3 parallel-rt enables the concurrent executor. Your RON configuration chooses the plan. The default planner is Serial, so enabling only the Cargo feature keeps serial execution.

Use Pipeline when several CPU-heavy stages can work on different CopperLists at the same time. While a later stage handles cycle A, an earlier stage can start cycle B. Each cycle still follows the graph’s dependencies.

Try it in the single-crate project from Writing Tasks. The small arithmetic graph makes the setup visible; its work is too cheap to promise a throughput improvement.

1. Forward the executor feature

Add this entry to the application’s existing [features] table:

parallel-rt = ["cu29/parallel-rt"]

2. Select the plan and its capacity

Keep the tasks and connections in copperconfig.ron. Add the following runtime and logging sections inside its top-level parentheses, replacing those sections if they already exist:

runtime: (
    planner: (kind: Pipeline, config: {"max_in_flight": 4}),
),
logging: (
    copperlist_count: 4,
),

max_in_flight bounds concurrent cycles. copperlist_count reserves their message storage at startup and must be at least that large. Start with a small limit: more slots increase memory use, and extra in-flight work can increase latency without improving throughput.

The example removes the 1 Hz limiter so it does not hold back the pipeline. Keep or restore rate_target_hz when your application needs a fixed maximum loop rate.

3. Run and inspect

From the project root:

cargo run --features parallel-rt

The source and sink still report 42 and 43. Stop with Ctrl-C. With the template’s viewer recipes from Setting Up, run:

just sched

The schedule shows separate process-stage workers and overlap between cycles. The rt pool’s affinity and policy control these workers; its threads value does not determine how many stages Pipeline creates. See Thread Pools before assigning cores or priorities.

To return to serial execution, change the planner to (kind: Serial) and run cargo run. To choose worker placement and ordering from recorded timings, continue to Profile-Guided Scheduling, whose candidates use explicit schedules.

Two distinctions that matter

parallel-rt is not the same as background: true

parallel-rt is a runtime-level pipeline. Multiple CopperLists can be in flight at the same time, and each generated process stage keeps FIFO order for determinism.

background: true is a stage-level escape hatch. One source or task moves to the background thread pool and may return None for some cycles while its previous run is still finishing.

parallel-rt is not the same as task-local parallelism

parallel-rt parallelizes the DAG across CopperLists.

Task-local parallelism parallelizes the inside of one task.

If one task is much heavier than the rest, fixing that task directly is often better than adding runtime-level parallelism around it.

Background tasks and thread pools

If a source or task is marked with background: true, it runs on a worker thread from the background thread pool. If you do not declare that pool, the runtime creates a default one for you.

This makes background stages easy to turn on, but it does not mean they are always the right choice. They work best when skipping or sampling intermediate cycles is acceptable.

v1.0 You can size the background pool, pin it to specific cores, or run a stage on a separate pool entirely. See Thread Pools, Affinity, and Real-Time Scheduling for the runtime.thread_pools configuration.

Reading Performance Metrics

Performance tuning starts with the right numbers. In Copper, the useful views are already built in:

  • live, in the console monitor
  • offline, from log-stats

If you have not added the monitor yet, start with Adding the Console Monitor. If you want offline stats from a log, see Exporting Data to Other Formats.

Live signals

LAT tab

Use the latency tab when you want to know whether the problem is inside the DAG itself.

Look for:

  • one task with a much larger mean or max than the others
  • high jitter on one task
  • a high End2End latency even though no single task stands out

What it usually means:

  • one hot task: optimize that task, background it, or parallelize inside it
  • many moderate CPU-bound tasks in a row: consider parallel-rt

BW tab

Use the bandwidth tab when the DAG seems fine but the loop still feels too expensive.

This tab is the first place to look for:

  • observed loop rate
  • CopperList size
  • raw memory throughput
  • serialized size
  • total disk write rate

What it usually means:

  • latency is fine, but BW is high at the end of the cycle: consider async-cl-io
  • disk / encoding numbers are simply too large: reduce what you log
  • numbers are fine for a while, then the machine slows down: suspect writeback pressure and test mmap-fsync

MEM tab

Use the memory pools tab when the system looks unstable or starts failing only under load.

Look for:

  • used buffers close to total buffers
  • handles in use staying high
  • obvious pool exhaustion

What it usually means:

  • increase pool sizes
  • reduce how long buffers stay alive
  • if many CopperLists are in flight, check whether your pool sizing still makes sense

Offline signals with log-stats

The log-stats command adds two kinds of evidence:

  • pipeline timing in perf
  • per-edge size and throughput in edges

Typical command:

cargo run --features logreader --bin my-project-logreader -- \
    logs/my-project.copper log-stats --output stats.json --config copperconfig.ron

The most useful fields for tuning are:

FieldWhy it matters
perf.end_to_end.*Confirms real pipeline latency over a recorded session
perf.jitter.*Shows whether the problem is mostly spikes rather than mean cost
edges[].avg_raw_bytesShows which edge is large
edges[].throughput_bytes_per_secShows which edge is pushing the most sustained traffic
edges[].rate_hzShows whether the effective rate on that edge matches your expectation
edges[].none_samplesUseful when a task sometimes emits nothing, especially with background behavior

Symptom to first metric

SymptomCheck first
“The loop is too slow”LAT tab, then End2End in log-stats
“Tasks look fine but the cycle still ends late”BW tab
“It works for a while, then degrades”BW tab over time, then test mmap-fsync
“One task is clearly the culprit”LAT tab row for that task
“A specific edge is too heavy”avg_raw_bytes and throughput_bytes_per_sec in log-stats
“The system fails only under load”MEM tab
“Async or parallel mode did not help much”Check whether logging.copperlist_count is too low

A simple workflow

  1. Start with the LAT tab and BW tab.
  2. Confirm the hypothesis with log-stats.
  3. Change one knob at a time.
  4. Re-run and compare the same metrics.

Then use the decision tree:

Troubleshooting Performance

This chapter is meant to be navigated. Start at the first question and follow the links.

Before you start:

Start Here

Q1: Are you missing your target rate?

Q2: Are you actually rate-limited by config?

Check runtime.rate_target_hz.

Q3: Is one task clearly dominating the latency tab?

Q4: Can that task finish later without blocking every cycle?

Q5: Do you have a long CPU-bound pipeline?

Think: several pure compute stages back to back, little blocking I/O, little waiting on hardware.

Q6: Is the problem mostly end-of-CopperList overhead or late collapse?

Q7: Are you pushing too much data?

Check the BW tab and log-stats:

  • large serialized CopperLists

  • high disk write rate

  • one edge with large avg_raw_bytes or throughput_bytes_per_sec

  • Yes

  • No

Q8: Are you running out of buffers or in-flight CopperLists?

Outcomes

Rate Target Is the Limiter

If runtime.rate_target_hz is lower than the rate you want, Copper is doing exactly what you asked.

runtime: (
    rate_target_hz: 100,
),

If you need a higher rate:

  • raise rate_target_hz
  • or remove it and run at best effort

If the system starts missing deadlines after that change, go back to Q3 or Q5.

Enable async-cl-io

Use this when the task path is fine, but the end of the CopperList is too expensive.

What it does:

  • keeps the DAG execution on the main loop
  • queues the completed CopperList to a dedicated serializer thread
  • recycles the slot later when serialization finishes

Minimal feature forwarding:

[features]
async-cl-io = ["cu29/async-cl-io"]
cargo run --features async-cl-io

v1.1.4 v1.2.2 Regular background tasks skip empty inputs by default:

(
    id: "heavy",
    type: "tasks::HeavyTask",
    background: true,
),

Empty inputs then skip dispatch. Copper still collects a completed result once, even on an empty-input cycle, and emits an empty output until another result is ready. Inputs arriving while a job is running are skipped. Set background_process_empty: true to dispatch empty inputs. This setting defaults to false and works with named pools and background anytime tasks wrapped as regular tasks. See background pool behavior for a configuration example. Keyframe replay preserves a pending job’s saved input and whether its completed result has already been collected.

Good fit:

src -> task_a -> task_b -> sink

The LAT tab looks acceptable, but BW shows large serialized CL cost.

Bad fit:

src -> huge_cpu_task -> sink

The hot spot is inside huge_cpu_task itself. Moving CL serialization off-thread does not fix that.

Two checks after enabling it:

  • if the benefit is small, your real bottleneck is elsewhere
  • if the serializer now waits for free CopperLists, raise logging.copperlist_count

Enable parallel-rt

v1.3 Use this when you have several CPU-bound stages back to back and want higher global throughput.

What it does:

  • keeps deterministic FIFO order per generated process stage
  • lets multiple CopperLists be in flight at the same time
  • turns the runtime into a stage pipeline instead of a strictly one-CL-at-a-time loop

Minimal feature forwarding:

[features]
parallel-rt = ["cu29/parallel-rt"]

The feature enables the executor; select runtime.planner: (kind: Pipeline, ...) and reserve enough CopperLists as shown in the pipeline setup, then run:

cargo run --features parallel-rt

Good fit:

src -> cpu_a -> cpu_b -> cpu_c -> cpu_d -> sink

Why:

  • many stages can work on different CopperLists at the same time
  • no single stage is overwhelmingly larger than the others

Bad fit:

src -> giant_cpu_task -> sink

Why:

  • the giant task is still the throughput limiter
  • you usually get more by optimizing or parallelizing that task directly

Also a poor fit:

camera -> wait_for_io -> bridge -> sink

Why:

  • the graph is dominated by waiting and I/O, not by a CPU pipeline

Two checks after enabling it:

  • if workers look idle, you may need a larger logging.copperlist_count
  • if throughput does not move, the graph probably lacks enough useful overlap

Enable mmap-fsync

Use this when the system runs well for a while, then starts degrading, and you suspect the OS is accumulating too many dirty pages from the memory-mapped logger.

What it does:

  • keeps the memory-mapped logger
  • adds synchronous file sync_all() on section flush
  • trades peak throughput for more explicit writeback

Minimal feature forwarding:

[features]
mmap-fsync = ["cu29/mmap-fsync"]
cargo run --features mmap-fsync

Good fit:

The robot behaves well for a while, BW looks reasonable, then the machine starts stalling.

Bad fit:

The loop is already compute-bound from the first second.

After enabling it, measure the cost. If the throughput penalty is too large:

  • reduce what you log
  • experiment with section_size_mib and slab_size_mib
  • or accept the default async writeback and provision the storage path better

Background One Task

Use this when one isolated source or task is too slow, but it does not need to block the whole loop.

(
    id: "heavy",
    type: "tasks::HeavyTask",
    background: true,
),

What it means semantically:

  • Copper runs that source or task on the background thread pool
  • while it is still busy, process() returns None
  • downstream tasks therefore see missing output for some cycles

Good fit:

src -> heavy_optional_stage -> sink

Why:

  • it is acceptable for the stage to sample or skip intermediate cycles

Bad fit:

src -> estimator -> controller -> actuator

Why:

  • the controller path usually needs one coherent output per cycle

If you do not declare a background pool, Copper creates a default one. To size it, pin it to specific cores, or run this stage on a separate pool, see Thread Pools, Affinity, and Real-Time Scheduling.

Parallelize Inside One Task

Use this when the real bottleneck is inside one task, not in the DAG structure.

Typical cases:

  • per-pixel or per-point computation
  • large reduction or map kernels
  • CPU-heavy loops that can be split cleanly

Preferred approach:

  • keep the DAG simple
  • use a controlled parallel loop inside the task, or a declared thread pool
  • measure the task again in the LAT tab

Good fit:

src -> expensive_image_task -> sink

Bad fit:

The real issue is serialized logging or OS writeback.

Reduce Logging Volume

Use this when BW and log-stats show that the logger is simply carrying too much data.

The first lever is per-task logging:

(
    id: "fast-sensor",
    type: "tasks::FastSensor",
    logging: (enabled: false),
),

The second lever is global task logging:

logging: (
    enable_task_logging: false,
),

v1.2 If CopperLists are still needed but task-state snapshots are not, disable keyframes only:

logging: (
    enable_keyframe_logging: false,
),

This compiles keyframe capture out of the generated runtime while retaining CopperList and structured logs. It also avoids attempting to freeze background work at a CopperList boundary; the tradeoff is that replay cannot restore task state from keyframes.

Good fit:

One or two high-rate edges dominate avg_raw_bytes and throughput_bytes_per_sec.

Bad fit:

The hot spot is a CPU-heavy task, but the logged payloads are small.

The practical rule is simple:

  • keep logging on for the edges you actually need for replay or diagnosis
  • turn it off for bulky intermediate traffic first

Increase CopperList Slots

Use this when async or parallel execution is underfilled because there are not enough preallocated CopperLists available.

logging: (
    copperlist_count: 8,
),

This value is consumed by the generated runtime, so it needs to be compiled into the binary.

Good fit:

async-cl-io or parallel-rt is enabled, but the runtime still behaves as if only a tiny number
of CopperLists can be active.

Bad fit:

The graph has no useful overlap to exploit.

More slots help only when there is real work to overlap.

Tune Memory Pools

Use this when the MEM tab shows pressure on pooled buffers or handles.

Good fit:

handle-backed payloads stay alive across several stages and the pool remains near full.

Actions:

  • increase pool size
  • reduce payload lifetime
  • reduce the number of simultaneously retained CopperLists if that is what keeps the buffers alive

Bad fit:

The pool is healthy and the real issue is CPU or disk.

Reduce Thread Oversubscription

Use this when you stacked too many concurrency mechanisms at once:

  • parallel-rt
  • several background: true tasks
  • task-local Rayon pools or custom worker pools

Symptoms:

  • throughput does not improve
  • jitter gets worse
  • the machine is busy, but the useful rate barely moves

The fix is usually to simplify first:

  • use parallel-rt for DAG-level overlap
  • use background: true for one isolated non-blocking stage
  • use task-local parallelism for one hot algorithm

Do not enable all three everywhere by default.

Thread Pools, Affinity, and Real-Time Scheduling

v1.0 By default, Copper runs your task graph on one thread: the global loop executes source -> processing -> sink back-to-back every cycle. Background tasks and a concurrent plan such as Pipeline step outside that single thread, and both need worker threads to run on. This chapter is about controlling those workers: how many there are, which CPU cores they run on, and what scheduling priority the OS gives them.

On a laptop the defaults are fine. On a real robot, where a latency-critical control loop shares a machine with heavy background work (logging, vision, networking), you often want to isolate the critical work onto dedicated cores and give it real-time priority so the OS never lets background work preempt it. That is what thread pools, CPU affinity, and real-time scheduling are for.

Note: This is a std-only, host-side feature. On no_std / embedded targets there are no threads to schedule and this whole chapter does not apply.

Declaring thread pools

Thread pools are declared in the runtime section of copperconfig.ron, next to rate_target_hz:

runtime: (
    thread_pools: [
        ( id: "rt",         threads: 4, affinity: [2, 3, 4, 5], policy: Fifo(priority: 80) ),
        ( id: "background", threads: 2, affinity: [0, 1] ),
        ( id: "vision",     threads: 2, policy: Nice(10), on_error: Strict ),
    ],
),

Each pool has:

  • id – a unique name. Tasks and the runtime refer to pools by this name. Two names are reserved (see below): background and rt.
  • threads – how many worker threads the pool runs.
  • affinity (optional) – the CPU cores the workers may run on.
  • policy (optional) – the OS scheduling policy and priority. Defaults to Fair.
  • on_error (optional) – what to do if the affinity or policy cannot be applied. Defaults to Warn.

Scheduling policies

The policy controls how the OS scheduler treats the pool’s worker threads. On Linux these map directly onto the POSIX scheduling policies.

PolicyWhat it isWhen to use it
FairNormal fair time-sharing (SCHED_OTHER / CFS). The OS shares the CPU across threads and nothing starves.The default. Everything that is not latency-critical.
Nice(n)Fair scheduler with a niceness bias, n in -20..=19 (lower is more favorable). A soft hint, not a guarantee.Bias a pool below or above normal work without leaving the fair scheduler – e.g. Nice(10) for heavy work that should yield to the control loop.
Fifo(priority: p)SCHED_FIFO hard real-time, p in 1..=99. Runs ahead of every fair thread and is not time-sliced – it runs until it blocks or a higher-priority thread preempts it.The latency-critical pipeline. Pin it with affinity so a busy worker cannot starve other work on the same core.
RoundRobin(priority: p)SCHED_RR, same as Fifo except threads at the same priority are time-sliced in turn.Several real-time workers that share a priority and should interleave fairly.

Rule of thumb: reach for Nice first. Real-time policies (Fifo / RoundRobin) are powerful but easy to misuse – a busy Fifo worker with no affinity can starve the rest of the system. Use them for genuinely time-critical work, and pin them.

Real-time policies are Linux-only and usually require the CAP_SYS_NICE capability (run as root, grant the capability, or raise the limit in /etc/security/limits.conf). Negative niceness (Nice(-1) and below) needs the same capability – without it the request falls back per the pool’s on_error. On non-Linux hosts only Fair is applied directly; Nice, Fifo, and RoundRobin all fail according to the pool’s on_error. CPU affinity is cross-platform regardless.

CPU affinity

affinity is a list of logical CPU cores the pool’s workers are allowed to run on. When set, workers are pinned Spread: worker i is pinned to affinity[i % affinity.len()]. So a pool with threads: 4 and affinity: [2, 3, 4, 5] pins one worker to each of cores 2-5:

worker 0 -> core 2
worker 1 -> core 3
worker 2 -> core 4
worker 3 -> core 5

The point of affinity is isolation. If your real-time pool owns cores 2-5 and your background pool is confined to cores 0-1, the OS will not schedule background work onto the cores running your control loop. Combined with a Fifo policy, this is how you keep jitter out of the critical path.

If you omit affinity, the OS is free to run the workers on any core.

Choosing cores

Copper does not auto-detect kernel work or CPU topology – the affinity list is taken at face value. Two things on Linux are worth checking before you pick cores for a latency-critical pool:

  • Where the kernel handles IRQs. Hardware interrupts (NIC, NVMe, USB, timers) run on whichever CPUs the kernel binds them to, and a busy IRQ adds jitter to that core. cat /proc/interrupts shows per-CPU interrupt counts; cat /proc/irq/<n>/smp_affinity_list shows the affinity of a specific IRQ. By default most distros either pin IRQs to CPU 0 or let irqbalance spread them; either way, the conservative move is to keep your real-time affinity off the cores that show large counts in /proc/interrupts. For full isolation, the standard tool is the kernel command line (isolcpus=, nohz_full=, rcu_nocbs=), which removes the listed cores from the general scheduler entirely.
  • Hyperthread siblings. Two logical CPUs that share one physical core compete for the same execution units, so pinning a real-time pool across a sibling pair gives almost no isolation from background work landing on the other sibling. lscpu -e shows the CORE/CPU mapping; /sys/devices/system/cpu/cpuN/topology/thread_siblings_list gives the pair directly. The conservative move is to pick one logical CPU per physical core for the rt pool and leave the siblings to background or to the OS.

The explicit list is intentionally the lowest-level knob – a list of logical CPU ids the OS understands – and there are no higher-level selectors (BigCores, NumaNode, “avoid IRQs”, “avoid siblings”) yet. Future versions can layer those on top without changing the list form, so the recommendation today is to inspect your target machine with the tools above and write the list explicitly.

Reserved pools: background and rt

Two pool names have special meaning.

background

When a source or task is marked background: true, it runs on the pool named background. If you do not declare one, Copper creates a default background pool for you: 2 worker threads, no affinity, Fair scheduling. Declare it explicitly when you want to size it or pin it:

runtime: (
    thread_pools: [
        ( id: "background", threads: 2, affinity: [0, 1] ),
    ],
),

v1.1.4 v1.2.2 Regular background tasks skip dispatch when inputs are empty, while still collecting completed results. If a task must run on empty ticks, set background_process_empty: true on its node:

(
    id: "telemetry",
    type: "tasks::TelemetryUploader",
    background: true,
    background_process_empty: true,
),

This option applies to regular background tasks, including background anytime tasks wrapped as regular tasks. Background sources have no inputs to skip.

A task can also opt into a different pool with the object form of background:

(
    id: "detector",
    type: "vision::Detector",
    background: (pool: "vision"),
),

This runs detector on the vision pool instead of background – useful when you want a class of heavy tasks scheduled separately from the rest of your background work. The pool name is resolved at build time: referencing a pool that isn’t declared in runtime.thread_pools, or pointing a background task at the reserved rt pool, is a compile-time error.

See The Task Graph for the background field, and Performance Basics for when backgrounding a stage is the right call.

rt

v1.3 The rt in parallel-rt stands for runtime – the generated execution engine that runs the task graph – not “real-time”. The pool named rt configures workers created by the Pipeline planner: it spawns one worker per generated process stage, and the rt pool lets you pin and prioritize those workers exactly like any other pool:

runtime: (
    thread_pools: [
        ( id: "rt", threads: 4, affinity: [2, 3, 4, 5], policy: Fifo(priority: 80) ),
    ],
),

The rt pool is consumed directly by the engine: only its affinity and policy are applied to the stage workers (Spread across the stage index). parallel-rt spawns one worker per generated process stage, so the threads: field has no effect on the rt pool – size affinity to match the stage count if you want exact one-core-per-stage pinning. Unlike other pools, you do not bind tasks to it and you cannot use it as a background target.

When it cannot be applied: on_error

Setting CPU affinity or a real-time priority can fail. The common cases are platform- specific: on Linux, a Fifo / RoundRobin priority (or Nice below 0) without CAP_SYS_NICE; on Windows, any non-Fair policy at all, since the POSIX real-time policies are not implemented there; and on either OS, an affinity entry that names a core the machine does not have. on_error chooses what happens:

  • Warn (default) – log a warning and fall back to default scheduling. The pool still runs; it just does not get the requested affinity/priority. This keeps unprivileged dev/laptop runs working out of the box.
  • Strict – fail at startup. Use this on a deployed real-time robot, where running the control loop at default priority is a fault you want to catch loudly rather than discover as jitter in the field.
( id: "control", threads: 2, policy: Fifo(priority: 90), on_error: Strict ),

The rt-scheduling feature

Affinity and scheduling are gated behind the rt-scheduling Cargo feature on cu29, forwarded from your crate the same way as the other runtime features:

[features]
rt-scheduling = ["cu29/rt-scheduling"]
cargo run --features rt-scheduling

With the feature off, runtime.thread_pools still works – pools are built with the requested thread counts – but affinity and policy are ignored and a warning is emitted. This lets you keep one config that runs unprivileged in development and pins to real-time on the target by flipping a feature flag. CPU affinity is cross-platform; the real-time policies are Linux-only.

Determinism and replay

Pool configuration is performance-only. Affinity and policy change when and where work runs, never what each task computes or in what order results commit to a copperlist. By design, swapping in or out an rt pool produces a bit-identical normalized CopperList stream in both sync and parallel-rt modes, and the runtime matrix integration tests guard this as a regression.

The practical consequence is that an offline replay or resim run on a different machine keeps producing the same output as the live capture, even when the pool config cannot be honored: with the default on_error: Warn the runtime logs and falls back to default scheduling, and the stream is unchanged. If you need to force neutralization (for example, to suppress warnings on a CI host that cannot pin, or to pick cores at runtime), mutate config.runtime.thread_pools before building the app.

Good fit / Bad fit

Good fit:

core 0-1: background pool (logging, telemetry)   Nice / Fair
core 2-5: rt pool (control pipeline)             Fifo, pinned

Why:

  • the critical pipeline owns dedicated cores at real-time priority
  • everything else is confined elsewhere and yields to it

Bad fit:

( id: "everything", threads: 8, policy: Fifo(priority: 99) )

Why:

  • a single high-priority Fifo pool with no affinity, sized to every core, can starve the OS itself
  • real-time priority is for isolating specific work, not for “make it all fast”

Summary

FieldPurposeDefault
idPool name; background and rt are reserved– (required)
threadsNumber of worker threads– (required)
affinityCPU cores the workers are pinned to (Spread)unpinned
policyOS scheduling policy/priorityFair
on_errorBehavior when affinity/policy cannot be appliedWarn

For complete working examples, see examples/cu_background_task (both background forms) and examples/cu_runtime_matrix (rt and background pools).

Profile-Guided Scheduling

v1.3 Profile-Guided Scheduling (PGS) uses timings from a representative Copper run to propose execution schedules for your application. It is an offline workflow: the robot records its normal unified log, a host-side logreader generates candidates, and you rebuild and measure those candidates before selecting one.

Workflow

Start with a reference run and its measured performance. Generate candidate plans, select one, then compile and run it under the same workload. Compare the measured results with the reference, test another candidate or replan, and keep the best plan.

PGS workflow: compile and run a reference, extract a profile, propose and select plans, compile and run candidates, and compare measured performance.

The important rule is: a prediction is a candidate to test, not a result to deploy. Task costs change when work is moved between CPUs, caches, and scheduling policies. The measured ranking can select the original schedule, and that is a valid outcome.

PGS changes the execution plan: which worker runs each operation, the order on each worker, CPU placement, and how many CopperLists may be in flight. It does not change the task graph or task implementations. Generated candidates are normal explicit Copper plans, validated against the static graph at build time.

Before you start

Use the same application revision, target-class machine, launch options, and representative workload throughout the comparison. A desktop profile is useful for testing the workflow, but it does not predict a robot computer with different CPUs, devices, or load.

You need:

  1. An application-specific logreader using cu29_export::run_cli, as described in Logging and Replaying Data.
  2. A unified log from one mission with runtime lifecycle records and CopperList process timestamps.
  3. Payload presence logged at the endpoints of every latency chain.
  4. The parallel-rt feature forwarded by the application, as described in Performance Basics.

For handle-backed payloads, handle_content: none is sufficient for PGS: it preserves message metadata and handle presence without recording the referenced buffer. A terminal sink has no output payload, so PGS infers that it fired from its directly connected producer. Keep payload presence for that producer as well.

Record long enough to cover steady state, periodic firing patterns, bursts, and the slow paths that matter to the robot. PGS requires only two firings for a contracted source, but two samples are not a useful latency distribution.

1. Write the scheduling contract

The contract states what the application needs and which machine resources PGS may use. Put it in a separate file such as schedule.ron:

(
    chains: [
        (
            id: "sense_to_command",
            source: "camera",
            sink: "motor_command",
            deadline_ms: 20,
        ),
    ],
    sources: [
        (
            task: "camera",
            period_ms: 10,
        ),
    ],
    cpus: [2, 3, 4, 5],
    max_in_flight: 4,
    headroom: 0.2,
    worker_policy: Fair,
)
FieldMeaningHow to choose it
chainsEnd-to-end paths whose latency mattersName a source, a reachable sink, and the real application deadline
sourcesInput delivery requirementsGive the expected period of sources whose throughput must be preserved; omit sources you do not want scored
cpusExact logical CPUs available to generated workersReserve CPUs intentionally; inspect affinity, IRQ load, and sibling cores as described in Thread Pools
max_in_flightLargest pipeline depth PGS may exploreStart conservatively; emitted configs raise logging.copperlist_count when needed, which increases preallocated memory
headroomFraction of deadlines and worker windows kept free0.2 means candidates enter the headroom penalty above 80% utilization
worker_policyPolicy applied to generated workersStart with Fair; Fifo(priority: N) and RoundRobin(priority: N) require deployment support and leave two higher priorities for PGS

A chain sample is the sink’s process end minus the source’s time of validity. If the source did not set a time of validity, its process start is used. Every chain id must be unique, every deadline and source period must be positive, and every sink must be reachable from its source in the selected mission.

2. Record the baseline

Run the normal application on the deployment system under the workload you care about. Use a distinct log path so the comparison cannot overwrite it:

cargo run --release --bin my-project -- --log logs/pgs-baseline.copper

The exact application command and --log argument are project-specific. What matters is that the resulting unified log contains the effective configuration, the mission, process timestamps, and the message presence needed by the contract.

If one log contains several recorded runs, inspect it first:

cargo run --release --features logreader --bin my-project-logreader -- \
    logs/pgs-baseline.copper list-runs

Pass the global --run <index> option when optimizing a particular run. PGS deliberately uses the configuration and mission recorded in that run; there are no separate config or mission overrides that could silently describe a different application.

3. Generate candidates

Run optimize-schedule through the application’s typed logreader:

cargo run --release --features logreader --bin my-project-logreader -- \
    logs/pgs-baseline.copper optimize-schedule \
    --contract schedule.ron \
    --output target/pgs \
    --candidates 3

The search is deterministic: the same log, contract, and application version produce the same candidates. Requesting more candidates gives you more distinct schedules to measure; it does not make the first candidate more thoroughly optimized.

The command prints a readable ranking and writes the complete evidence under target/pgs/:

ArtifactWhat you can learn or do with it
contract.ronConfirm the normalized requirements used by this run
profile.ronInspect per-operation fired/skipped costs, percentiles, firing patterns, measured chains, source delivery, mission, and graph signature
plan-N.plan.ronInspect or archive the standalone validated CuPlan
plan-N.config.ronBuild the application with the original recorded config plus that exact explicit plan
predictions.ronCompare predicted chain latency, source delivery, worker load, response time, CPU, and score for every candidate
report.txtRead the same ranked summary printed in the terminal

Reading the score

Scores are compared from left to right. A later tier cannot compensate for a worse earlier tier. The predicted score contains:

  1. Source delivery deficit.
  2. Number of chains predicted to miss their deadlines.
  3. Number of chains predicted to consume the configured headroom.
  4. Sum of chain latency divided by deadline.
  5. Maximum predicted worker load.

This means PGS does not optimize only deadlines. It first protects contracted source delivery, then deadline feasibility and margin, then total normalized latency, and finally load balance. A source affects the first tier only when it appears in sources.

Worker response is the model’s bounded response time for one cycle. unbounded means the response-time calculation did not converge within its safety cap; do not treat that worker as viable merely because another reported number looks small.

4. Build and run candidates

Build each candidate from its emitted plan-N.config.ron. The safest setup is a dedicated candidate binary copied from the application’s normal entry point, with only the compile-time config path and output log path changed:

#[copper_runtime(config = "target/pgs/plan-1.config.ron")]
struct PgsCandidate {}

Keep the same resources, mission, component parameters, workload, release profile, and launch options as the baseline. Enable parallel-rt, because a generated explicit schedule may contain several workers:

cargo run --release --features parallel-rt --bin my-project-pgs-candidate -- \
    --log logs/pgs-plan-1.copper

Run on the same target under comparable system load. Warm-up, thermal throttling, IRQ placement, logging pressure, and competing processes can all move the result. If those conditions are part of normal robot operation, keep them in the measurement rather than creating an unrealistically quiet benchmark.

5. Rank measurements and select

Feed candidate logs back into the same command. The positional log remains the baseline:

cargo run --release --features logreader --bin my-project-logreader -- \
    logs/pgs-baseline.copper optimize-schedule \
    --contract schedule.ron \
    --output target/pgs \
    --measure plan-1=logs/pgs-plan-1.copper \
    --measure plan-2=logs/pgs-plan-2.copper

This regenerates the predictions and adds:

ArtifactMeaning
measurements.ronEvery measured candidate, best first, with p50/p99/max latency, misses, delivered source rate, and the corresponding prediction
report.txtPredicted ranking followed by the measured ranking
selected.plan.ronThe standalone plan belonging to the best measured row, including the original plan when baseline wins

Measured ranking uses delivered source rate and measured p99 chain latency. Its score omits the predicted worker-load tier because the log comparison ranks observed end-to-end results. Deadline misses are still reported for every chain, even though p99 is the ranking statistic.

Use the winning plan-N.config.ron when a candidate wins; keep the original configuration when baseline wins. selected.plan.ron is the compact plan artifact for inspection, versioning, or tooling. Before deployment, repeat the winning candidate across the robot’s expected operating range. Multiple measurement arguments may use names such as plan-1/r2=logs/pgs-plan-1-r2.copper; each run remains a separate row so variability stays visible.

Diagnosing rejected inputs

PGS fails instead of filling missing evidence with guessed costs. Common input failures are:

Error or symptomCause and action
No recorded configuration or runtime lifecycle recordRecord a fresh log with the current application runtime and use the matching logreader build
No CopperList with process timestampsVerify that the selected run contains executed CopperLists, not only lifecycle or text-log records
No process timestamps for an operationThe operation did not execute in the captured workload, or the log does not contain its timing metadata; exercise that path and record again
A chain has no samplesIts source and sink never both fired in a CopperList; preserve endpoint payload presence and exercise the complete path
A contracted source fired fewer than twiceRecord a longer representative window or correct the source id/period

Contract and comparison failures usually indicate a mismatch:

Error or symptomCause and action
More than one mission in the selected runRecord or select one mission at a time
Unknown task or unreachable sinkCorrect the task ids or define a chain that follows the selected mission’s graph
Profiles were recorded on different graphs or missionsRebuild and record baseline and candidates from the same topology, component config, resource bindings, and mission
No valid plan on the requested CPUsRevisit the CPU set, deadlines, headroom, and in-flight capacity; do not weaken requirements merely to produce a file
Candidate fails to apply its scheduling policyUse Fair while validating the workflow, or configure the required affinity and Linux real-time privileges described in Thread Pools