Commit 897581f
committed
Fix local shovels crashing with plugin queue types
Local shovels crashed with a badarg error when consuming from plugin
queue types such as JMS or delayed queues:
```
crasher:
initial call: rabbit_shovel_worker:init/1
pid: <0.1237.0>
registered_name: []
exception exit: {badarg,
[{persistent_term,get,
[{rabbit_global_counters,'local-shovel',
rabbit_jms_queue}],
[{error_info,#{module => erl_erts_errors}}]},
{rabbit_global_counters,fetch,2,
[{file,"rabbit_global_counters.erl"},{line,274}]},
{rabbit_global_counters,messages_delivered,3,
[{file,"rabbit_global_counters.erl"},{line,223}]},
{rabbit_local_shovel,'-handle_deliver/3-fun-0-',3,
[{file,"rabbit_local_shovel.erl"},{line,667}]},
{lists,foldl,3,[{file,"lists.erl"},{line,2466}]},
{rabbit_local_shovel,handle_deliver,3,
[{file,"rabbit_local_shovel.erl"},{line,665}]},
{lists,foldl,3,[{file,"lists.erl"},{line,2466}]},
{rabbit_shovel_worker,handle_msg,2,
[{file,"rabbit_shovel_worker.erl"},{line,116}]}]}
in function gen_server2:terminate/3 (gen_server2.erl:1174)
```
The crash occurred because rabbit_local_shovel:boot_step/0 only
initialized global counters for the three core queue types
(classic, quorum, stream). When a local shovel delivered a message
from a plugin queue type, rabbit_global_counters:messages_delivered/3
attempted to look up a counter that was never created.
The root cause was a missing ordering guarantee between queue type
registrations and the local shovel counter initialization during
node boot. Previously, the boot step dependency chain was:
```
pre_boot -> rabbit_global_counters -.
\
-> external_infrastructure
/
pre_boot -> rabbit_registry --------'
```
Queue type registrations (classic, quorum, stream, JMS, delayed) all
required rabbit_registry but had no ordering relationship with the
local shovel counter boot step. Both competed to run before
external_infrastructure with no mutual dependency.
Introduce a new boot step phase "queue_type_registrations" that serves
as a synchronization point:
- All queue type registration boot steps enable it
- The local shovel counter boot step requires it (along with
rabbit_global_counters)
The resulting dependency chain is:
```
pre_boot -> rabbit_global_counters -.
\
pre_boot -> rabbit_registry ----. -> local shovel counters
\ /
queue type regs --> queue_type_registrations
```
This guarantees that rabbit_queue_type:known_queue_type_modules()
returns all registered queue types when boot_step/0 runs, so
counters are initialized for every queue type without hardcoding.
Here are the relevant boot step logs:
```
2026-03-23 12:51:11.142127+01:00 [info] <0.217.0> Running boot step rabbit_classic_queue_type defined by app rabbit
2026-03-23 12:51:11.142138+01:00 [debug] <0.217.0> Applying MFA: M = rabbit_registry, F = register, A = [queue,<<"classic">>,
2026-03-23 12:51:11.142138+01:00 [debug] <0.217.0> rabbit_classic_queue]
2026-03-23 12:51:11.142191+01:00 [debug] <0.217.0> Finished MFA: M = rabbit_registry, F = register, A = [queue,<<"classic">>,
2026-03-23 12:51:11.142191+01:00 [debug] <0.217.0> rabbit_classic_queue]
2026-03-23 12:51:11.142210+01:00 [info] <0.217.0> Running boot step rabbit_quorum_queue_type defined by app rabbit
2026-03-23 12:51:11.142229+01:00 [debug] <0.217.0> Applying MFA: M = rabbit_registry, F = register, A = [queue,<<"quorum">>,
2026-03-23 12:51:11.142229+01:00 [debug] <0.217.0> rabbit_quorum_queue]
2026-03-23 12:51:11.142268+01:00 [debug] <0.217.0> Finished MFA: M = rabbit_registry, F = register, A = [queue,<<"quorum">>,
2026-03-23 12:51:11.142268+01:00 [debug] <0.217.0> rabbit_quorum_queue]
2026-03-23 12:51:11.142286+01:00 [info] <0.217.0> Running boot step rabbit_stream_queue defined by app rabbit
2026-03-23 12:51:11.142300+01:00 [debug] <0.217.0> Applying MFA: M = rabbit_registry, F = register, A = [queue,<<"stream">>,
2026-03-23 12:51:11.142300+01:00 [debug] <0.217.0> rabbit_stream_queue]
2026-03-23 12:51:11.142335+01:00 [debug] <0.217.0> Finished MFA: M = rabbit_registry, F = register, A = [queue,<<"stream">>,
2026-03-23 12:51:11.142335+01:00 [debug] <0.217.0> rabbit_stream_queue]
...
2026-03-23 12:51:11.142702+01:00 [info] <0.217.0> Running boot step delayed_queue_type defined by app rabbitmq_delayed_queue
2026-03-23 12:51:11.142709+01:00 [debug] <0.217.0> Applying MFA: M = rabbit_registry, F = register, A = [queue,<<"delayed">>,
2026-03-23 12:51:11.142709+01:00 [debug] <0.217.0> rabbitmq_delayed_queue]
2026-03-23 12:51:11.142733+01:00 [debug] <0.217.0> Finished MFA: M = rabbit_registry, F = register, A = [queue,<<"delayed">>,
2026-03-23 12:51:11.142733+01:00 [debug] <0.217.0> rabbitmq_delayed_queue]
2026-03-23 12:51:11.142745+01:00 [info] <0.217.0> Running boot step jms_queue_type defined by app rabbitmq_jms
2026-03-23 12:51:11.142795+01:00 [debug] <0.217.0> Applying MFA: M = rabbit_registry, F = register, A = [queue,<<"jms">>,
2026-03-23 12:51:11.142795+01:00 [debug] <0.217.0> rabbit_jms_queue]
2026-03-23 12:51:11.142825+01:00 [debug] <0.217.0> Finished MFA: M = rabbit_registry, F = register, A = [queue,<<"jms">>,
2026-03-23 12:51:11.142825+01:00 [debug] <0.217.0> rabbit_jms_queue]
2026-03-23 12:51:11.142837+01:00 [info] <0.217.0> Running boot step queue_type_registrations defined by app rabbit
2026-03-23 12:51:11.142847+01:00 [info] <0.217.0> Running boot step rabbit_global_local_shovel_counters defined by app rabbitmq_shovel
2026-03-23 12:51:11.142876+01:00 [debug] <0.217.0> Applying MFA: M = rabbit_local_shovel, F = boot_step, A = []
2026-03-23 12:51:11.143000+01:00 [debug] <0.217.0> Finished MFA: M = rabbit_local_shovel, F = boot_step, A = []
```
This commit makes the following test suite green:
```
make -C deps/rabbitmq_jms ct-jms_queue_shovel
```1 parent 33409c3 commit 897581f
5 files changed
Lines changed: 12 additions & 7 deletions
File tree
- deps
- rabbitmq_shovel/src
- rabbit/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
82 | 86 | | |
83 | 87 | | |
84 | 88 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
95 | | - | |
| 95 | + | |
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
| 123 | + | |
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | | - | |
| 133 | + | |
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
99 | | - | |
100 | | - | |
101 | | - | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
| |||
0 commit comments