Skip to content

README.md

Latest commit

 

History

History
272 lines (191 loc) · 7.12 KB

README.md

File metadata and controls

272 lines (191 loc) · 7.12 KB

Robocodec

License: MulanPSL-2.0 Rust codecov

Robocodec is a robotics data format library for reading, writing, and converting MCAP and ROS1 bag files. It provides a unified API with automatic format detection, parallel processing, and support for multiple message encodings (CDR, Protobuf, JSON) and schema types (ROS .msg, ROS2 IDL, OMG IDL).

Why Robocodec?

  • Clean API - Only RoboReader, RoboWriter, RoboRewriter exposed at top level
  • Auto-Detection - Format detected from file extension or URL scheme
  • Fast - Parallel processing with rayon, zero-copy memory-mapped files
  • S3-Native - First-class support for s3:// URLs (AWS S3, MinIO, Alibaba OSS, etc.)
  • Transformations - Topic/type renaming during file rewriting

Quick Start

Rust

# Cargo.toml
[dependencies]
robocodec = "0.1"
use robocodec::RoboReader;

// Format auto-detected from extension
let reader = RoboReader::open("data.mcap")?;
println!("Found {} channels", reader.channels().len());

Python (from source)

Python bindings are available but must be built from source:

git clone https://github.com/archebase/robocodec.git
cd robocodec
make build-python-dev
from robocodec import RoboReader

reader = RoboReader("data.mcap")
print(f"Found {len(reader.channels)} channels")

Note: PyPI release is coming soon. For now, build from source using the instructions above.

Common Tasks

Read messages from a file

use robocodec::RoboReader;

let reader = RoboReader::open("file.mcap")?;

// List all channels
for channel in reader.channels() {
    println!("{}: {} messages", channel.topic, channel.message_count);
}

// Get message count
println!("Total messages: {}", reader.message_count());

Write messages to a file

use robocodec::RoboWriter;

let mut writer = RoboWriter::create("output.mcap")?;
let channel_id = writer.add_channel("/topic", "MessageType", "cdr", None)?;
// ... write messages ...
writer.finish()?;

Read decoded messages

use robocodec::RoboReader;

let reader = RoboReader::open("file.mcap")?;

for result in reader.decoded()? {
    let msg = result?;
    println!("Topic: {}", msg.topic());
    println!("Data: {:?}", msg.message);
    println!("Log time: {:?}", msg.log_time);
}

Read from S3

Robocodec supports reading directly from S3-compatible storage using s3:// URLs:

use robocodec::RoboReader;

// MCAP format
let reader = RoboReader::open("s3://my-bucket/path/to/data.mcap")?;
println!("Found {} channels", reader.channels().len());

// ROS1 BAG format (streaming read)
let reader = RoboReader::open("s3://my-bucket/path/to/data.bag")?;
println!("Found {} channels", reader.channels().len());

// RRD format (streaming read)
let reader = RoboReader::open("s3://my-bucket/path/to/data.rrd")?;
println!("Found {} channels", reader.channels().len());

S3-compatible services (AWS S3, Alibaba Cloud OSS, MinIO, etc.) require credentials via environment variables:

# AWS S3
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION="us-east-1"  # optional, defaults to us-east-1

# For Alibaba Cloud OSS, MinIO, or other S3-compatible services
export AWS_ACCESS_KEY_ID="your-oss-access-key"
export AWS_SECRET_ACCESS_KEY="your-oss-secret-key"

Note: While we use AWS-standard environment variable names for compatibility, robocodec works with any S3-compatible storage service.

Custom S3 endpoints

For S3-compatible services with custom endpoints:

Option 1: Environment variable (global)

export S3_ENDPOINT="http://localhost:9000"  # MinIO
export S3_ENDPOINT="https://oss-cn-hangzhou.aliyuncs.com"  # Alibaba OSS

Option 2: URL query parameter (per-request)

use robocodec::RoboReader;

// MinIO running locally
let reader = RoboReader::open("s3://bucket/data.mcap?endpoint=http://localhost:9000")?;

// Alibaba Cloud OSS (Hangzhou region)
let reader = RoboReader::open(
    "s3://bucket/data.mcap?endpoint=https://oss-cn-hangzhou.aliyuncs.com"
)?;

Rewrite files with transformations

The rewriter processes files in the same format, optionally applying topic and type transformations:

use robocodec::{RoboRewriter, TransformBuilder, RewriteOptions};

let transform = TransformBuilder::new()
    .with_topic_rename("/old/topic", "/new/topic")
    .build();

let options = RewriteOptions::default().with_transforms(transform);
let rewriter = RoboRewriter::with_options("input.mcap", options)?;
rewriter.rewrite("output.mcap")?;

Note: The rewriter preserves the same format. Cross-format conversion is not currently supported.

Installation

Rust Users

Add to Cargo.toml:

[dependencies]
robocodec = "0.1"

Optional features:

robocodec = { version = "0.1", features = ["jemalloc"] }
Feature Description Default
s3 S3-compatible storage support (AWS S3, MinIO, etc.) ✅ Yes
python Python bindings ❌ No
jemalloc Use jemalloc allocator (Linux only) ❌ No
python Python bindings ❌ No
jemalloc Use jemalloc allocator (Linux only) ❌ No

Python Users

Build from source (PyPI release coming soon):

git clone https://github.com/archebase/robocodec.git
cd robocodec
make build-python-dev

Supported Formats

Format Read Write
MCAP ✅ ✅
ROS1 Bag ✅ ✅
RRF2 (Rerun) ✅ ✅

Note: RRF2 support is compatible with Rerun 0.27+. Earlier versions use a different format and are not supported.

Message Encodings

Encoding Description
CDR Common Data Representation (ROS1/ROS2)
Protobuf Protocol Buffers
JSON JSON encoding

Schema Support

  • ROS .msg files (ROS1)
  • ROS2 IDL (Interface Definition Language)
  • OMG IDL (Object Management Group)

License

MulanPSL v2 - see LICENSE

Development

Testing

make test              # Run all tests
make test-rust         # Run Rust tests only
make test-python       # Run Python tests only

Fuzzing

Robocodec includes comprehensive fuzzing infrastructure for parser security and robustness testing:

./scripts/fuzz_init.sh  # Initialize fuzzing infrastructure (one-time setup)
make fuzz               # Quick fuzzing check (30s per target)
make fuzz-all           # Extended fuzzing (1min per target)
make fuzz-mcap          # Fuzz MCAP parser only

For detailed fuzzing documentation, see docs/FUZZING.md.

Benchmarks

make bench              # Run performance benchmarks
make bench-compare      # Compare against baseline

Links
⚡