Node failing with a schema_integrity_check_failed after successful Khepri migration (4.1 to 4.2)

[skry-dev]

Describe the bug

I am upgrading my RabbitMQ cluster from 4.1 to 4.2 (running on Erlang 27). I am experiencing a persistent boot failure on nodes after enabling Khepri. When I restart the one of the cluster node, it was get error.
I deploy Rabbitmq as docker container to each node. I upgraded cluster one by one. My cluster was healthy before upgrade.

error.txt

Reproduction steps

1. Upgraded all nodes to RabbitMQ 4.2.
2. Verified all nodes were running and clustered.
3. Enabled Khepri using: rabbitmqctl enable_feature_flag khepri_db --opt-in.
4. Monitored logs and saw: Migration to Khepri completed / finished.
5. Verified status with rabbit_metastore:is_khepri_enabled() which returned true.
6. Verified khepri_db feature flag is enabled on all nodes.
7. Restart one of the cluster node.
8. The Problem: When I restart a node (rolling restart), it fails to boot with the following error:
   {error, {schema_integrity_check_failed, [{table_missing, rabbit_user}, {table_missing, rabbit_vhost}, ... ]}}
   Even after deleting the Mnesia directory and performing a reset followed by join_cluster, the node still encounters the same integrity check failure upon the next manual restart.

rabbit-config.txt
error.txt
docker-compose.yml

Expected behavior

I want to restart nodes without any error.

Additional context

I can not upgrade my environment because of this error. I try to delete all data and restart docker containers with 4.2.0, enabled khepri. But it didn't work.

[michaelklishin]

@skry-dev according to the error, something in your cluster still requires Mnesia but it is, well, not started any more since you have migrated to Khepri.

This is almost certainly a 3rd party plugin and not something in RabbitMQ.

Neither your rabbitmq.conf nor the Compose deployment file show any plugins but it is almost certainly a plugin such as rabbitmq_rtopic_exchange or rabbitmq_delayed_exchange that explicitly depend on Mnesia.

The latter was updated to start Mnesia separately but then archived.

That's all we can suggest given the amount of information you have shared. Also, the most recent RabbitMQ 4.2.x release is 4.2.5, and 4.2.6 is shipping in less than two weeks.

[michaelklishin]

Specifically, the plugin in question calls either rabbit_table:check_schema_integrity/1 or rabbit_table:check_schema_integrity/0 (likely the latter), and RabbitMQ with Khepri enabled
returns an empty list of Mnesia tables to verify:

definitions() ->
    case rabbit_khepri:get_feature_state() of
        enabled -> [];
        _       -> mandatory_definitions()
    end.

Plus we are not aware of any such failures except for 3rd party plugins that are Mnesia-specific, and 4.2.x has been out for about 6 months now.

[skry-dev]

Hi,
When I was listed my plugins, I saw that you are right. I'm using that plugins:

[E*] rabbitmq_auth_backend_ldap
[E*] rabbitmq_delayed_message_exchange
[E*] rabbitmq_management
[e*] rabbitmq_management_agent
[E*] rabbitmq_prometheus
[E*] rabbitmq_top
[e*] rabbitmq_web_dispatch

One of the plugins (rabbitmq_delayed_message_exchange) you said is causing the problem is on my list.

My question is, should we completely disable this plugin when upgrading to 4.2.x?

[michaelklishin]

@skry-dev that plugin was updated to start Mnesia, so you probably also run an old version.

That said, that plugin is no longer maintained, so you should move away from it in general.