Add rosidl_buffer and rosidl_buffer_backend for native Buffer type support by nvcyc · Pull Request #941 · ros2/rosidl

nvcyc · 2026-03-13T18:00:46Z

Description

This pull request adds rosidl_buffer and rosidl_buffer_backend packages - core C++ buffer types for the ROS 2 native buffer feature. This package introduces rosidl::Buffer<T>, a polymorphic container that's designed to replace std::vector<T> for uint8[] message fields (to be adopted in later pull requests; we focus on only the core buffer types in this pull request.) The native buffer type and its backend class enable vendor-specific memory buffer implementation (CUDA, ROCm, etc.) while maintaining backward compatibility for existing CPU-based std::vector<T> user code.

This pull request consists of the following key components:

Buffer<T>: PIMPL-based container providing std::vector<T>-compatible API. All vector-compatible operations (element access, iterators, modifiers) are CPU-only and throw std::runtime_error for non-CPU backends. Backend management APIs (get_backend_type(), get_impl(), to_vector()) work for all backends.
BufferImplBase<T>: Minimal abstract base class of buffer implementation. All backend-specific APIs are to be provided by the vendor-specific backend implementations.
CpuBufferImpl<T>: CPU-based buffer implementation wrapping std::vector<T>.
BufferBackend: Abstract interface for vendor-specific buffer backend implementations.

Is this user-facing behavior change?

This pull request does not change existing behavior.
When adopted in later pull requests, message types with uint8[] fields (e.g., sensor_msgs/msg/Image.data) will use rosidl::Buffer<uint8_t> instead of std::vector<uint8_t>. For CPU backends (the default), Buffer<T> is a transparent drop-in replacement.

Did you use Generative AI?

Yes. Claude (claude-4.6-opus) via Cursor was used to assist with the std::vector<T> compatibility feature in the rosidl::Buffer<T> class and generate portions of the unit tests.

Additional Information

This package/PL is part of the broader ROS2 native buffer feature introduced in this post.

hidmic

Amazing work @nvcyc!

hidmic

Looks correct.

…pport Signed-off-by: CY Chen <cyc@nvidia.com>

Signed-off-by: CY Chen <cyc@nvidia.com>

MiguelCompany

LGTM

ahcorde · 2026-03-27T08:19:33Z

Pulls: #941
Gist: https://gist.githubusercontent.com/ahcorde/d14f1064681a43ab6af3d7e1c2bbd285/raw/17346b358df7d0e4cab5d800a790e46b6c26dea0/ros2.repos
BUILD args: --packages-above-and-dependencies rosidl_buffer rosidl_buffer_backend
TEST args: --packages-above rosidl_buffer rosidl_buffer_backend
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18681

Linux
Linux-aarch64
Linux-rhel
Windows

Signed-off-by: CY Chen <cyc@nvidia.com>

…nter Signed-off-by: CY Chen <cyc@nvidia.com>

Signed-off-by: CY Chen <cyc@nvidia.com>

nvcyc · 2026-03-30T00:51:21Z

Pulls: #941
Gist: https://gist.githubusercontent.com/nvcyc/874d4eb64eb8543bcd349cd9e355a788/raw/17346b358df7d0e4cab5d800a790e46b6c26dea0/ros2.repos
BUILD args: --continue-on-error --packages-above-and-dependencies rosidl_buffer rosidl_buffer_backend
TEST args: --packages-above rosidl_buffer rosidl_buffer_backend
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/18701

Linux
Linux-aarch64
Linux-rhel
Windows

nvcyc · 2026-03-30T15:00:43Z

Re-triggered the CI just for the Windows platform:

Windows

mjcarroll

LGTM!

A new package was added to `rcl_logging` called `rcl_logging_implementation`. It depends on `rcpputils`, which I wasn't able to get building. Therefore, we are skipping building that package using `COLCON_IGNORE`. https://github.com/ros2/rclc needs to be pinned to an older version until ros2/rcl#1269 is added in https://github.com/micro-ROS/rcl. https://github.com/ros2/rosidl needs to be pinned to an older version due to breaking changes made with ros2/rosidl#941 This PR also fixes micro-ROS#251.

) ROS 2 Rolling merged ros2/rosidl#941 and ros2/rosidl#942 as part of the native buffer feature. The ROSIDL_RUNTIME_C__PRIMITIVE_SEQUENCE macro now emits two extra boolean fields (is_rosidl_buffer, owns_rosidl_buffer) in every primitive sequence struct, growing them from 24 to 32 bytes. Non-primitive sequences (e.g. Parameter__Sequence) are unchanged at 24 bytes. The rclnodejs code generator produces ref-struct-di definitions for these sequence types. Because the generated layout lacked the new fields, every message containing a primitive sequence had misaligned field offsets when serialized, causing Fast DDS to segfault in get_serialized_size_* during rcl_publish. Changes: 1. rosidl_gen/templates/message-template.js — Detect Rolling+ at generation time via DistroUtils.getDistroId() and compute needsRosidlBufferFields once per message. Conditionally emit is_rosidl_buffer and owns_rosidl_buffer in the generated sequence struct only for primitive-package types (Bool, Byte, Int8, …, Float64, String) on Rolling+. Generated files get clean, static struct definitions with no runtime require() or ternary. 2. .github/workflows/linux-x64-build-and-test.yml — Add rosidl_buffer_py and pybind11 to rosdep --skip-keys for the Rolling nightly install, since these new packages from the native buffer feature have unresolvable rosdep keys in the tarball context. Verified: Rolling: test-parameters (33), test-parameter-service (9), test-action-client (19) — all pass. Jazzy: test-parameters (33), test-parameter-service (9), test-action-client (18+1 pending) — all pass. Fix: #1481

A new package was added to `rcl_logging` called `rcl_logging_implementation`. It depends on `rcpputils`, which I wasn't able to get building. Therefore, we are skipping building that package using `COLCON_IGNORE`. https://github.com/ros2/rclc needs to be pinned to an older version until ros2/rcl#1269 is added in https://github.com/micro-ROS/rcl. https://github.com/ros2/rosidl needs to be pinned to an older version due to breaking changes made with ros2/rosidl#941 This PR also fixes micro-ROS#251.

A new package was added to `rcl_logging` called `rcl_logging_implementation`. It depends on `rcpputils`, which I wasn't able to get building. Therefore, we are skipping building that package using `COLCON_IGNORE`. https://github.com/ros2/rclc needs to be pinned to an older version until ros2/rcl#1269 is added in https://github.com/micro-ROS/rcl. https://github.com/ros2/rosidl needs to be pinned to an older version due to breaking changes made with ros2/rosidl#941

Compiling pcl_ros package in rolling results in errors like this: In file included from /build/perception_pcl-release-release-rolling-pcl_ros-2.8.0-1/tools/combined_pointcloud_to_pcd.cpp:47: /ros-rolling-pcl-conversions-2.8.0-r1/include/pcl_conversions/pcl_conversions/pcl_conversions.h: In function 'void pcl_conversions::moveFromPCL(pcl::PCLImage&, sensor_msgs::msg::Image&)': /ros-rolling-pcl-conversions-2.8.0-r1/include/pcl_conversions/pcl_conversions/pcl_conversions.h:172:16: error: 'using sensor_msgs::msg::Image_<std::allocator<void> >::_data_type = class rosidl::Buffer<unsigned char, std::allocator<unsigned char> >' {aka 'class rosidl::Buffer<unsigned char, std::allocator<unsigned char> >'} has no member named 'swap' 172 | image.data.swap(pcl_image.data); | ^~~~ These are caused by ros2/rosidl#941 and followup PRs, which change the type of `uint8[]` message fields from `std::vector<uint8_t>` to `rosidl::Buffer<uint8_t>`. To maintain the previous functionality, explicit typecasting is needed at a few places. This causes invocation of the conversion operator [1], which returns a reference to the underlying std::vector. [1]: https://github.com/ros2/rosidl/blob/7c4e0f90f2979c16c906d65058fc7966360f52e1/rosidl_buffer/include/rosidl_buffer/buffer.hpp#L420 Signed-off-by: Michal Sojka <michal.sojka@cvut.cz>

Compiling this in rolling results in the following error: /build/bag2_to_image-release-release-rolling-bag2_to_image-0.1.1-1/src/bag2_to_image.cpp: In constructor 'bag2_to_image::Bag2ToImageNode::Bag2ToImageNode(const rclcpp::NodeOptions&)': /build/bag2_to_image-release-release-rolling-bag2_to_image-0.1.1-1/src/bag2_to_image.cpp:165:52: error: no matching function for call to 'cv::Mat::Mat(sensor_msgs::msg::CompressedImage_<std::allocator<void> >::_data_type&)' 165 | auto mat = cv::imdecode(cv::Mat(image->data), imdecode_flag_); | ^ These are caused by ros2/rosidl#941 and followup PRs, which change the type of `uint8[]` message fields from `std::vector<uint8_t>` to `rosidl::Buffer<uint8_t>`. To maintain the previous functionality, explicit typecasting is needed. This causes invocation of the conversion operator [1], which returns a reference to the underlying std::vector. [1]: https://github.com/ros2/rosidl/blob/7c4e0f90f2979c16c906d65058fc7966360f52e1/rosidl_buffer/include/rosidl_buffer/buffer.hpp#L420 Signed-off-by: Michal Sojka <michal.sojka@cvut.cz>

nvcyc force-pushed the native_buffer/1-rosidl_buffer branch 3 times, most recently from cee2760 to ea59c56 Compare March 14, 2026 20:16

nvcyc mentioned this pull request Mar 14, 2026

Add rosidl_buffer for native Buffer type support #938

Closed

nvcyc requested review from MiguelCompany, ahcorde, hidmic, karsten-nvidia, mjcarroll, skyegalaxy and tfoote March 14, 2026 20:26

hidmic reviewed Mar 15, 2026

View reviewed changes

Comment thread rosidl_buffer/include/rosidl_buffer/buffer.hpp Outdated

Comment thread rosidl_buffer/include/rosidl_buffer/buffer.hpp Outdated

Comment thread rosidl_buffer/include/rosidl_buffer/cpu_buffer_impl.hpp Outdated

Comment thread rosidl_buffer/include/rosidl_buffer/buffer_impl_base.hpp

This was referenced Mar 15, 2026

Update rosidl cpp path to emit rosidl::Buffer for uint8[] type #942

Merged

Add support for rosidl::Buffer in rosidl C path for rclpy #943

Merged

nvcyc requested a review from hidmic March 19, 2026 09:12

hidmic approved these changes Mar 20, 2026

View reviewed changes

MiguelCompany suggested changes Mar 20, 2026

View reviewed changes

Comment thread rosidl_buffer/include/rosidl_buffer/buffer.hpp Outdated

Comment thread rosidl_buffer/include/rosidl_buffer/buffer.hpp Outdated

Comment thread rosidl_buffer/include/rosidl_buffer/buffer.hpp

nvcyc requested a review from MiguelCompany March 22, 2026 17:47

nvcyc added 4 commits March 22, 2026 17:55

Add rosidl_buffer and rosidl_buffer_backend for native Buffer type su…

f35cabe

…pport Signed-off-by: CY Chen <cyc@nvidia.com>

Reinforce rosidl::Buffer APIs

027b70e

Signed-off-by: CY Chen <cyc@nvidia.com>

Add passing Allocator to CpuBufferImpl

e4032e9

Signed-off-by: CY Chen <cyc@nvidia.com>

Rename backend aux info to backend metadata in BufferBackend

c53b788

Signed-off-by: CY Chen <cyc@nvidia.com>

nvcyc force-pushed the native_buffer/1-rosidl_buffer branch from 326b4f8 to c53b788 Compare March 22, 2026 17:55

MiguelCompany approved these changes Mar 23, 2026

View reviewed changes

ahcorde approved these changes Mar 26, 2026

View reviewed changes

nvcyc added 2 commits March 29, 2026 04:18

Add buffer descriptor type support operators

8cd8b0b

Signed-off-by: CY Chen <cyc@nvidia.com>

Update descriptor ops to use non-owning pointers

6ec582e

Signed-off-by: CY Chen <cyc@nvidia.com>

nvcyc force-pushed the native_buffer/1-rosidl_buffer branch from a19b54f to 6ec582e Compare March 29, 2026 19:25

Update from_descriptor_with_endpoint to return type-erased unique poi…

18bf5f1

…nter Signed-off-by: CY Chen <cyc@nvidia.com>

Add kMaxBufferDescriptorSize for descriptor upper bound size

80da71f

Signed-off-by: CY Chen <cyc@nvidia.com>

hidmic approved these changes Mar 30, 2026

View reviewed changes

mjcarroll approved these changes Mar 31, 2026

View reviewed changes

ahcorde merged commit 4e122a5 into rolling Mar 31, 2026
4 checks passed

ahcorde deleted the native_buffer/1-rosidl_buffer branch March 31, 2026 07:27

ahcorde mentioned this pull request Apr 1, 2026

Native support for tensor datatypes ros2/ros2#1736

Open

minggangw mentioned this pull request Apr 8, 2026

Fix SIGSEGV on ROS 2 Rolling caused by rosidl sequence ABI change RobotWebTools/rclnodejs#1480

Merged

Amronos mentioned this pull request Apr 11, 2026

Fix rolling builds micro-ROS/micro_ros_espidf_component#323

Open

minggangw mentioned this pull request Apr 13, 2026

Roadmap: Prepare rclnodejs for ROS 2 Lyrical Luth RobotWebTools/rclnodejs#1458

Open

28 tasks

wentasah mentioned this pull request Apr 16, 2026

pcl_conversions: Fix compile errors caused by rosidl_buffer introduction ros-perception/perception_pcl#529

Open

wentasah mentioned this pull request Apr 16, 2026

Fix compile error caused by rosidl_buffer introduction wep21/bag2_to_image#8

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add rosidl_buffer and rosidl_buffer_backend for native Buffer type support#941

Add rosidl_buffer and rosidl_buffer_backend for native Buffer type support#941
ahcorde merged 8 commits intorollingfrom
native_buffer/1-rosidl_buffer

nvcyc commented Mar 13, 2026

Uh oh!

hidmic left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hidmic left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MiguelCompany left a comment

Uh oh!

ahcorde commented Mar 27, 2026

Uh oh!

nvcyc commented Mar 30, 2026

Uh oh!

nvcyc commented Mar 30, 2026

Uh oh!

mjcarroll left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

nvcyc commented Mar 13, 2026

Description

Is this user-facing behavior change?

Did you use Generative AI?

Additional Information

Uh oh!

hidmic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hidmic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MiguelCompany left a comment

Choose a reason for hiding this comment

Uh oh!

ahcorde commented Mar 27, 2026

Uh oh!

nvcyc commented Mar 30, 2026

Uh oh!

nvcyc commented Mar 30, 2026

Uh oh!

mjcarroll left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants