Add support for rosidl::Buffer in rosidl C path for rclpy by nvcyc · Pull Request #943 · ros2/rosidl

nvcyc · 2026-03-16T09:29:29Z

Description

This pull request makes the rosidl C layer Buffer-aware and adds Python bindings for rosidl::Buffer, enabling rclpy message types to work with vendor-backed buffer memory.

This pull request consists of the following key changes:

rosidl_runtime_c: Adds is_rosidl_buffer field (bool) in each primitive sequence struct (e.g., rosidl_runtime_c__uint8__Sequence). When true, data points to an rosidl::Buffer<T>*.
rosidl_generator_c: For messages with uint8[] fields, the generated fini logic (msg__functions.c.em) checks msg->field.is_rosidl_buffer and clears the sequence fields without calling normal sequence fini, avoiding double-free of a borrowed rosidl::Buffer* used by the Python C extension.
rosidl_typesupport_introspection_c: The goal for the changes in C introspection is to keep it working with CPU-based path. The introspection accessors throw exceptions for non-CPU backends.
rosidl_buffer_py (new): Python bindings (pybind11) for rosidl::Buffer.

Is this user-facing behavior change?

This pull request does not change existing rclpy behavior. The is_rosidl_buffer flag defaults to false, so all existing C sequence code paths are unchanged. The rosidl_buffer_py.Buffer class is new and only used when vendor-backed (non-CPU) buffers are received by rclpy subscribers. CPU-based data continues to be delivered as array.array.

Did you use Generative AI?

Yes. Claude (claude-4.6-opus) via Cursor was used to assist with creating an initial prototype version of the changes contained in this PR.

Additional Information

This PR is part of the broader ROS 2 native buffer feature introduced in this post.

It depends on PRs #941, #942 and ros2/rosidl_typesupport_fastrtps#144 (core buffer types, C++ rosidl changes, and C++ serialization).

hidmic · 2026-03-17T02:50:58Z

+      return py::isinstance<PyBuffer>(obj);
+    }, "Check if the given object is an rosidl_buffer.Buffer");
+
+  m.def("_get_buffer_ptr", [](PyBuffer & buf) -> uintptr_t {


@nvcyc meta: how would one reach to the underlying memory safely if we type erase like this?

_get_buffer_ptr is strictly internal plumbing. It's mainly used by the generated Python-to-C conversion code (_msg_support.c.em) during publish to hand the rosidl::Buffer<uint8_t>* back to the C sequence struct (where it is plain C code that can't easily handle typed rosidl::Buffer<uint8_t>*). We don't expect end users to call it.

Yeah, I see now. We are basically using the C message as a vehicle for a type erased, heap allocated rosidl::Buffer. On publish you need to borrow. On callback (ie. on take), you need to transfer ownership from C to Python.

I can't say I love it but I understand it.

hidmic

Looks correct.

hidmic · 2026-03-20T20:31:47Z

+  {
+    if (buffer_->get_backend_type() == "cpu") {
+      return py::bytes(
+        reinterpret_cast<const char *>(buffer_->data()),


@nvcyc meta: I missed this the first time. Copying the entire buffer isn't great. Returning a memoryview or implemeting the array protocol for CPU buffers may be more appropriate.

to_bytes() exists mainly for supporting rosidl_runtime_py (message_to_ordereddict, message_to_yaml) to serialize buffer data to dicts, YAML, and CSV.

For users to interact with a buffer, it should either be through

backend vendor provided APIs

converting to the CPU equivalent type in rclpy, which is array.array via to_arrary() if desired

hidmic · 2026-03-20T20:43:14Z

+      return py::isinstance<PyBuffer>(obj);
+    }, "Check if the given object is an rosidl_buffer.Buffer");
+
+  m.def("_get_buffer_ptr", [](PyBuffer & buf) -> uintptr_t {


Yeah, I see now. We are basically using the C message as a vehicle for a type erased, heap allocated rosidl::Buffer. On publish you need to borrow. On callback (ie. on take), you need to transfer ownership from C to Python.

I can't say I love it but I understand it.

InvincibleRMC · 2026-03-21T23:30:17Z

I would love to see some type stubs for the python Buffer class!

mjcarroll · 2026-04-01T20:35:24Z

Does this CI also cover 942?

nvcyc · 2026-04-01T20:39:00Z

Yes. This PR (#943) is (re)based on #942, so checking out this #943 branch will include changes in #942. So I think the CI I triggered above should be valid for both #942 and #943.

nvcyc · 2026-04-01T21:18:31Z

Pulls: #943, ros2/rosidl_typesupport_fastrtps#144, ros2/rosidl_runtime_py#39, ros2/rosidl_python#250
Gist: https://gist.githubusercontent.com/nvcyc/63f913be77e0ac65d906ccf58f94aca7/raw/597b9f91085df45556fea2b8a350e6b04438e57f/ros2.repos
BUILD args: --continue-on-error --packages-above-and-dependencies rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py
TEST args: --packages-above rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18798

Linux
Linux-aarch64
Linux-rhel
Windows

nvcyc · 2026-04-02T03:04:14Z

Pulls: #943, ros2/rosidl_typesupport_fastrtps#144, ros2/rosidl_runtime_py#39, ros2/rosidl_python#250, ros2/rosidl_core#14, ros2/rmw_cyclonedds#575
Gist: https://gist.githubusercontent.com/nvcyc/dfd06fa52cb41f00f46ec87a19a6a8a6/raw/fa6cc1a80d16030d21aea20bed2065b88f8e3900/ros2.repos
BUILD args: --continue-on-error --packages-above-and-dependencies rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime
TEST args: --packages-above rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18801

Linux
Linux-aarch64
Linux-rhel
Windows

wjwwood

With the other comments addressed, this lgtm

nvcyc · 2026-04-02T20:42:31Z

Pulls: #943, ros2/rosidl_typesupport_fastrtps#144, ros2/rosidl_runtime_py#39, ros2/rosidl_python#250, ros2/rosidl_core#14, ros2/rmw_cyclonedds#575
Gist: https://gist.githubusercontent.com/nvcyc/d13227291610b614acad252daf9858cb/raw/fa6cc1a80d16030d21aea20bed2065b88f8e3900/ros2.repos
BUILD args: --continue-on-error --packages-above-and-dependencies rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime
TEST args: --packages-above rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18807

Linux
Linux-aarch64
Linux-rhel
Windows

nvcyc · 2026-04-03T17:25:43Z

Pulls: #943, ros2/rosidl_typesupport_fastrtps#144, ros2/rosidl_runtime_py#39, ros2/rosidl_python#250, ros2/rosidl_core#14, ros2/rmw_cyclonedds#575
Gist: https://gist.githubusercontent.com/nvcyc/2ecd2ae717fce6a9f8c805c18a54ff26/raw/fa6cc1a80d16030d21aea20bed2065b88f8e3900/ros2.repos
BUILD args: --continue-on-error --packages-above-and-dependencies rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime
TEST args: --packages-above rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18817

Linux
Linux-aarch64
Linux-rhel
Windows

nvcyc · 2026-04-03T22:13:25Z

Triggered with custom ros2.repos that contain all PRs: https://raw.githubusercontent.com/nvcyc/ros2/refs/heads/rolling-native-buffer-prs/ros2.repos

Started by user CY Chen
Running as SYSTEM
Building on the built-in node in workspace /var/lib/jenkins/workspace/ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18827

Linux
Linux-aarch64
Linux-rhel
Windows

Signed-off-by: CY Chen <cyc@nvidia.com>

nvcyc · 2026-04-04T17:22:11Z

The above force-push is to rebase #942 and #943 to rolling.

nvcyc · 2026-04-04T23:50:34Z

Pulls: #943, ros2/rosidl_typesupport_fastrtps#144, ros2/rosidl_runtime_py#39, ros2/rosidl_python#250, ros2/rosidl_core#14, ros2/rmw_fastrtps#867
Gist: https://gist.githubusercontent.com/nvcyc/2b81ce4a59e71ec57d7d52f520fab590/raw/a28b02c5a17f2e12f08af57ffeb8479e1e399f98/ros2.repos
BUILD args: --continue-on-error --packages-above-and-dependencies rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime rmw_fastrtps_cpp rmw_fastrtps_shared_cpp
TEST args: --packages-above rosidl_buffer rosidl_buffer_py rosidl_runtime_c rosidl_typesupport_introspection_c rosidl_typesupport_introspection_tests rosidl_generator_cpp rosidl_runtime_cpp rosidl_typesupport_introspection_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_runtime_py rosidl_generator_py rosidl_core_runtime rmw_fastrtps_cpp rmw_fastrtps_shared_cpp
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18835

Linux
Linux-aarch64
Linux-rhel
Windows

sloretz · 2026-04-06T17:24:17Z

Trying Windows CI with --packages-ignore rmw_zenoh_cpp test_rmw_zenoh_cpp zenoh_cpp_vendor zenoh_security_tools:

sloretz · 2026-04-06T22:59:54Z

Windows CI minus zenoh is green, and all 6 PRs are aproved, so I'm merging them

* Update rosidl cpp path to emit rosidl::Buffer for uint8[] type Signed-off-by: CY Chen <cyc@nvidia.com> * Remove unused has_buffer_fields_ field Signed-off-by: CY Chen <cyc@nvidia.com> * Add support for rosidl::Buffer in rosidl C path for rclpy (#943) * Add support for rosidl::Buffer in rosidl C path for rclpy Signed-off-by: CY Chen <cyc@nvidia.com> * Add rosidl_buffer_py to rosidl_runtime_packages group Signed-off-by: CY Chen <cyc@nvidia.com> * Remove unused has_buffer_fields_ field from introspection C Signed-off-by: CY Chen <cyc@nvidia.com> * Simplify rosidl_buffer Python bindings Signed-off-by: CY Chen <cyc@nvidia.com> * Add owns_rosidl_buffer in sequence struct to handle buffer lifecycle Signed-off-by: CY Chen <cyc@nvidia.com> * Add Python type stub for Buffer class Signed-off-by: CY Chen <cyc@nvidia.com> * Remove unused test assertion for introspection Signed-off-by: CY Chen <cyc@nvidia.com> * Add comments for the limitation of _take_buffer_from_ptr Signed-off-by: CY Chen <cyc@nvidia.com> * Revert adding a line in msg__functions.c.em Signed-off-by: CY Chen <cyc@nvidia.com> * Update rosidl_buffer/c_helpers.h to use the right comment style Signed-off-by: CY Chen <cyc@nvidia.com> --------- Signed-off-by: CY Chen <cyc@nvidia.com> --------- Signed-off-by: CY Chen <cyc@nvidia.com>

nvcyc requested review from MiguelCompany, ahcorde, hidmic, karsten-nvidia, mjcarroll, skyegalaxy and tfoote March 16, 2026 09:30

hidmic reviewed Mar 17, 2026

View reviewed changes

hidmic reviewed Mar 18, 2026

View reviewed changes

Comment thread rosidl_runtime_c/src/primitives_sequence_functions.c

hidmic reviewed Mar 18, 2026

View reviewed changes

Comment thread rosidl_buffer_py/src/rosidl_buffer_py.cpp

nvcyc mentioned this pull request Mar 19, 2026

Add rosidl_buffer_py as build_export_depend with explicit group resol… ros2/rosidl_core#14

Merged

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from 0f9fd11 to 986d123 Compare March 19, 2026 08:52

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch 2 times, most recently from e0d2f52 to 9d7675f Compare March 20, 2026 05:21

nvcyc mentioned this pull request Mar 20, 2026

Update rosidl cpp path to emit rosidl::Buffer for uint8[] type #942

Merged

hidmic approved these changes Mar 20, 2026

View reviewed changes

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from fce6039 to 6375777 Compare March 21, 2026 23:44

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from 3085e51 to f06ac85 Compare March 22, 2026 08:41

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from b4e6596 to 4dcb08e Compare March 22, 2026 08:41

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from f06ac85 to 20ba671 Compare March 22, 2026 17:56

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from 4dcb08e to 897f0bc Compare March 22, 2026 17:56

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from 20ba671 to 4623bdf Compare March 29, 2026 04:26

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from 0a33311 to 9f1a637 Compare March 29, 2026 04:29

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from 4623bdf to d04fd63 Compare March 29, 2026 18:57

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from 9f1a637 to 4a1fce2 Compare March 29, 2026 18:57

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from d04fd63 to 863e827 Compare March 29, 2026 19:27

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from 4a1fce2 to 3a47ecd Compare March 29, 2026 19:28

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from 863e827 to c3f83ab Compare March 29, 2026 20:23

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from 3a47ecd to 2ac2f2a Compare March 29, 2026 20:23

nvcyc mentioned this pull request Apr 1, 2026

Add support for rosidl::Buffer in rosidl Python path for rclpy ros2/rosidl_python#250

Merged

nvcyc mentioned this pull request Apr 2, 2026

Key support squash ros2/rmw_cyclonedds#575

Merged

wjwwood approved these changes Apr 2, 2026

View reviewed changes

Comment thread rosidl_buffer/include/rosidl_buffer/c_helpers.h Outdated

Comment thread rosidl_buffer_py/src/rosidl_buffer_py.cpp

Comment thread rosidl_runtime_c/src/primitives_sequence_functions.c

nvcyc mentioned this pull request Apr 3, 2026

Add support for rosidl::Buffer-aware per-endpoint pub/sub ros2/rmw_fastrtps#867

Merged

nvcyc force-pushed the native_buffer/2-rosidl_cpp branch from 2bd85fd to 3a8edd9 Compare April 4, 2026 17:11

nvcyc added 10 commits April 4, 2026 17:11

Add support for rosidl::Buffer in rosidl C path for rclpy

dcca52f

Signed-off-by: CY Chen <cyc@nvidia.com>

Add rosidl_buffer_py to rosidl_runtime_packages group

2764556

Signed-off-by: CY Chen <cyc@nvidia.com>

Remove unused has_buffer_fields_ field from introspection C

6ca8e42

Signed-off-by: CY Chen <cyc@nvidia.com>

Simplify rosidl_buffer Python bindings

ffc50fb

Signed-off-by: CY Chen <cyc@nvidia.com>

Add owns_rosidl_buffer in sequence struct to handle buffer lifecycle

96ca0e5

Signed-off-by: CY Chen <cyc@nvidia.com>

Add Python type stub for Buffer class

358d16f

Signed-off-by: CY Chen <cyc@nvidia.com>

Remove unused test assertion for introspection

5ac628c

Signed-off-by: CY Chen <cyc@nvidia.com>

Add comments for the limitation of _take_buffer_from_ptr

5cfbba9

Signed-off-by: CY Chen <cyc@nvidia.com>

Revert adding a line in msg__functions.c.em

3632263

Signed-off-by: CY Chen <cyc@nvidia.com>

Update rosidl_buffer/c_helpers.h to use the right comment style

22ffbb9

Signed-off-by: CY Chen <cyc@nvidia.com>

nvcyc force-pushed the native_buffer/4-rosidl_c_py branch from 6d21fcd to 22ffbb9 Compare April 4, 2026 17:12

sloretz merged commit d3cb77d into native_buffer/2-rosidl_cpp Apr 6, 2026
3 checks passed

sloretz deleted the native_buffer/4-rosidl_c_py branch April 6, 2026 23:02

Conversation

nvcyc commented Mar 16, 2026

Description

Is this user-facing behavior change?

Did you use Generative AI?

Additional Information

Uh oh!

hidmic Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

nvcyc Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hidmic Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hidmic left a comment

Choose a reason for hiding this comment

Uh oh!

hidmic Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

nvcyc Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

hidmic Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

InvincibleRMC commented Mar 21, 2026

Uh oh!

mjcarroll commented Apr 1, 2026

Uh oh!

nvcyc commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvcyc commented Apr 1, 2026

Uh oh!

nvcyc commented Apr 2, 2026

Uh oh!

wjwwood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvcyc commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvcyc commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvcyc commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvcyc commented Apr 4, 2026

Uh oh!

nvcyc commented Apr 4, 2026

Uh oh!

sloretz commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sloretz commented Apr 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nvcyc Mar 20, 2026 •

edited

Loading

nvcyc commented Apr 1, 2026 •

edited

Loading

nvcyc commented Apr 2, 2026 •

edited

Loading

nvcyc commented Apr 3, 2026 •

edited

Loading

nvcyc commented Apr 3, 2026 •

edited

Loading

sloretz commented Apr 6, 2026 •

edited

Loading