Improve quorum queue Ra system configurability by kjnilsson · Pull Request #15962 · rabbitmq/rabbitmq-server

kjnilsson · 2026-04-08T10:34:32Z

Summary

Make all quorum queue Ra configuration parameters explicit and configurable via rabbitmq.conf.

This PR expands on the initial work to make segment sizing configurable by exposing all quorum queue Ra system parameters through the quorum_queue.* configuration namespace. This allows operators to fine-tune Raft log and WAL behavior specifically for quorum queues, with backward compatibility for the legacy raft.* configuration options.

Motivation

Previously, quorum queue Ra system parameters could only be configured via the global raft.* settings or the pre-launch default configuration. This PR provides dedicated quorum_queue.* configuration options that:

Make the configuration flow explicit and discoverable
Allow quorum queue-specific tuning without affecting other Ra-based systems (Khepri, streams)
Maintain backward compatibility with existing raft.* configurations
Centralize defaults in one place for easier maintenance

Changes

New `quorum_queue.*` Configuration Options

quorum_queue.segment_max_size_bytes (default: 64 MB)
quorum_queue.segment_max_entries (default: 4096)
quorum_queue.wal_max_size_bytes (default: 512 MB)
quorum_queue.wal_max_entries (default: 500,000)
quorum_queue.wal_max_batch_size (default: 4096)
quorum_queue.wal_compute_checksums (default: true)
quorum_queue.segment_compute_checksums (default: true)
quorum_queue.max_append_entries_rpc_batch_size (default: 16)
quorum_queue.compress_mem_tables (default: true)
quorum_queue.snapshot_chunk_size (default: 1 MB)

Implementation Details

Schema Mappings: Added Cuttlefish schema mappings for each quorum_queue.* setting that map to rabbit application environment variables
Backward Compatibility: Each quorum_queue.* setting falls back to the corresponding raft.* setting if not explicitly configured
Explicit Configuration: All parameters are explicitly passed to the quorum_queues Ra system configuration during initialization, making the data flow transparent
Defaults Alignment: All default values match the RabbitMQ prelaunch defaults, ensuring consistent behavior
Code Simplification: Simplified WalMaxEntries handling to use application:get_env with macro defaults instead of complex case statement logic

Files Changed

deps/rabbit/src/rabbit_ra_systems.erl: Updated to read quorum_queue configuration and apply defaults via macros
deps/rabbit/priv/schema/rabbit.schema: Added mappings and translations for all quorum_queue.* settings
deps/rabbit/docs/rabbitmq.conf.example: Documented all new configuration options
deps/rabbit/test/config_schema_SUITE_data/rabbit.snippets: Added test coverage for all new settings

Test Plan

Configuration schema tests verify all mappings parse correctly
Default values align with Ra's built-in defaults
Backward compatibility verified via fallback to raft.* settings
Build compiles successfully without linter errors

Notes

All defaults match the values previously set in rabbit_prelaunch_conf.erl
Configuration is centralized via macros for easier future maintenance
No breaking changes; existing configurations continue to work unchanged

Copilot

Pull request overview

Adds configurability for Raft log segment max size for quorum queues by introducing a raft.segment_max_size_bytes setting in rabbitmq.conf and wiring it through RabbitMQ’s config schema into the quorum-queues Ra system configuration.

Changes:

Add a new Cuttlefish schema mapping/translation for raft.segment_max_size_bytes.
Pass segment_max_size_bytes into the quorum_queues Ra system config (with a default macro).
Extend schema tests and the example rabbitmq.conf with the new setting.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
deps/rabbit/test/config_schema_SUITE_data/rabbit.snippets	Adds a schema snippet asserting `raft.segment_max_size_bytes` maps into the RabbitMQ app env.
deps/rabbit/src/rabbit_ra_systems.erl	Reads `rabbit.quorum_segment_max_size_bytes` and injects `segment_max_size_bytes` into the quorum queues Ra system config.
deps/rabbit/priv/schema/rabbit.schema	Introduces mapping/translation for `raft.segment_max_size_bytes`.
deps/rabbit/docs/rabbitmq.conf.example	Documents the new `raft.segment_max_size_bytes` option in the Raft settings section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add support for configuring Ra's segment_max_size_bytes parameter via rabbitmq.conf as raft.segment_max_size_bytes. This allows operators to control the maximum size of Raft log segment files for quorum queues. The configuration: - Maps quorum_queue.segment_max_size_bytes in rabbitmq.conf to rabbit.quorum_segment_max_size_bytes in the application environment - Defaults to Ra's built-in default of 64 MB when not configured - Is explicitly applied to the quorum_queues Ra system configuration for transparency and maintainability Also add a macro QUORUM_DEFAULT_SEGMENT_MAX_SIZE_B to align with Ra's default value.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…urable Add support for configuring all quorum queue Ra system parameters via rabbitmq.conf using the quorum_queue.* namespace. This allows operators to fine-tune Raft log and WAL behavior specifically for quorum queues. New quorum_queue.* configuration options: - quorum_queue.wal_compute_checksums (default: true) - quorum_queue.segment_compute_checksums (default: true) - quorum_queue.max_append_entries_rpc_batch_size (default: 16) - quorum_queue.compress_mem_tables (default: true) - quorum_queue.segment_max_size_bytes (default: 64 MB) - quorum_queue.segment_max_entries (default: 4096) - quorum_queue.wal_max_size_bytes (default: 512 MB) - quorum_queue.wal_max_entries (default: 500,000) - quorum_queue.wal_max_batch_size (default: 4096) - quorum_queue.snapshot_chunk_size (default: 1 MB) Each quorum_queue.* setting has a fallback to the corresponding raft.* setting for backward compatibility with existing configurations. All configuration is mapped to rabbit application environment variables and explicitly applied to the quorum_queues Ra system during initialization, making the data flow transparent and maintainable. Add macros for all default values that match the RabbitMQ prelaunch defaults, centralizing configuration defaults in one place for easier maintenance. Simplify WalMaxEntries handling to use application:get_env with macro default instead of complex case statement checking DefaultConfig.

Improve quorum queue Ra system configurability (backport #15962)

Expose all coordination Ra system parameters through new coordination.* configuration options, following the pattern established in PR #15962 for quorum queues. This allows operators to fine-tune Raft log and WAL behavior specifically for the coordination system without affecting other Ra-based systems (quorum queues, Khepri). New coordination.* configuration options: - coordination.wal_compute_checksums (default: true) - coordination.segment_compute_checksums (default: true) - coordination.wal_max_size_bytes (default: 64 MB) - coordination.wal_max_entries (default: from Ra's default_config) - coordination.wal_max_batch_size (default: from Ra's default_config) - coordination.segment_max_size_bytes (default: from Ra's default_config) - coordination.segment_max_entries (default: from Ra's default_config) - coordination.max_append_entries_rpc_batch_size (default: from Ra's default_config) - coordination.compress_mem_tables (default: true) - coordination.snapshot_chunk_size (default: from Ra's default_config) Each coordination.* setting has a fallback to its default value, ensuring backward compatibility. The wal_max_size_bytes parameter maintains the existing hardcoded default of 64 MB. Implementation details: 1. Added macros for coordination defaults in rabbit_ra_systems.erl 2. Refactored get_config(coordination) to read from application environment variables with sensible defaults, matching the quorum_queues pattern 3. Added Cuttlefish schema mappings for all new configuration options 4. Added comprehensive documentation in rabbitmq.conf.example 5. Added schema validation tests in rabbit.snippets 6. Added dedicated test suite (rabbit_ra_systems_SUITE) to validate defaults All configuration is explicitly passed to the coordination Ra system during initialization, making the data flow transparent and maintainable. Backward compatibility: Existing deployments continue to work unchanged, with coordination.wal_max_size_bytes defaulting to 64 MB.

kjnilsson requested a review from Copilot April 8, 2026 10:34

kjnilsson added the backport-v4.3.x label Apr 8, 2026

kjnilsson added this to the 4.4.0 milestone Apr 8, 2026

kjnilsson changed the title ~~Qq segment max size bytes config~~ QQ: segment max size bytes config Apr 8, 2026

Copilot started reviewing on behalf of kjnilsson April 8, 2026 10:35 View session

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch from 3184402 to 3b4492f Compare April 8, 2026 10:36

kjnilsson changed the title ~~QQ: segment max size bytes config~~ QQ: make segment_max_size_bytes configurable Apr 8, 2026

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread deps/rabbit/src/rabbit_ra_systems.erl

Comment thread deps/rabbit/priv/schema/rabbit.schema Outdated

Comment thread deps/rabbit/docs/rabbitmq.conf.example

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch from 3b4492f to d4b664c Compare April 8, 2026 11:42

kjnilsson requested a review from Copilot April 8, 2026 11:43

Copilot started reviewing on behalf of kjnilsson April 8, 2026 11:46 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread deps/rabbit/priv/schema/rabbit.schema Outdated

Comment thread deps/rabbit/priv/schema/rabbit.schema Outdated

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch from d4b664c to 48825ac Compare April 8, 2026 15:40

kjnilsson requested a review from Copilot April 8, 2026 15:43

Copilot started reviewing on behalf of kjnilsson April 8, 2026 15:44 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread deps/rabbit/priv/schema/rabbit.schema

Comment thread deps/rabbit/docs/rabbitmq.conf.example Outdated

Comment thread deps/rabbit/test/config_schema_SUITE_data/rabbit.snippets Outdated

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch from 48825ac to eacdd0c Compare April 8, 2026 16:02

kjnilsson requested a review from Copilot April 8, 2026 16:02

Copilot started reviewing on behalf of kjnilsson April 8, 2026 16:03 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Comment thread deps/rabbit/priv/schema/rabbit.schema Outdated

kjnilsson requested a review from Copilot April 9, 2026 08:15

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread deps/rabbit/src/rabbit_ra_systems.erl Outdated

Comment thread deps/rabbit/src/rabbit_ra_systems.erl

Comment thread deps/rabbit/priv/schema/rabbit.schema Outdated

Copilot started reviewing on behalf of kjnilsson April 9, 2026 09:41 View session

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch from ab1b76a to 607448d Compare April 9, 2026 09:43

kjnilsson requested a review from Copilot April 9, 2026 09:49

kjnilsson changed the title ~~QQ: make segment_max_size_bytes configurable~~ Improve quorum queue Ra system configurability Apr 9, 2026

Copilot AI reviewed Apr 9, 2026

View reviewed changes

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch 2 times, most recently from 1265826 to 8b6bb2f Compare April 9, 2026 10:11

kjnilsson marked this pull request as ready for review April 9, 2026 10:12

ansd approved these changes Apr 9, 2026

View reviewed changes

Copilot started reviewing on behalf of kjnilsson April 9, 2026 10:20 View session

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch from 8b6bb2f to fd35cd7 Compare April 9, 2026 10:29

kjnilsson marked this pull request as draft April 9, 2026 10:42

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch 2 times, most recently from 73280ee to c970f7d Compare April 9, 2026 11:57

kjnilsson force-pushed the qq-segment-max-size-bytes-config branch from c970f7d to 42927f0 Compare April 9, 2026 14:02

kjnilsson marked this pull request as ready for review April 9, 2026 16:05

michaelklishin merged commit 8bb0954 into main Apr 10, 2026
362 of 363 checks passed

michaelklishin deleted the qq-segment-max-size-bytes-config branch April 10, 2026 03:04

mergify Bot mentioned this pull request Apr 10, 2026

Improve quorum queue Ra system configurability (backport #15962) #15997

Merged

michaelklishin added a commit that referenced this pull request Apr 10, 2026

Merge pull request #15997 from rabbitmq/mergify/bp/v4.3.x/pr-15962

dd4bc26

Improve quorum queue Ra system configurability (backport #15962)

kjnilsson mentioned this pull request Apr 10, 2026

Make coordination Ra system parameters configurable via rabbitmq.conf #16013

Draft

Conversation

kjnilsson commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

New quorum_queue.* Configuration Options

Implementation Details

Files Changed

Test Plan

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kjnilsson commented Apr 8, 2026 •

edited

Loading

New `quorum_queue.*` Configuration Options