NVMe Multipath & docs

[gabrielgbs97]

For the NVMe kernel module, the nvme_core module handles the multipathing functionality in modern kernels. It's important to note that the nvme_core.multipath=Y parameter is related to Device Mapper (DM) multipath. In RHEL 10, this parameter has been removed in favor of the nvme_core kernel module, which enables multipathing by default.

Are we missing anything? We are integrating OpenEBS on top of EL10 nodes and encountered the deprecation of vme_core.multipath=Y.

Source: OpenEBS Documentation

[tiagolobocastro]

the nvme_core.multipath=Y parameter is related to Device Mapper (DM) multipath

Not soure what you mean here, nvme_core.multipath is enabling native nvme multipath and not the device mapper multipath

I think we should modify the docs @balaharish7

The kernel parameter nvme_core.multipath=Y is mandatory to enable HA functionality. Without this configuration, volume target failover is not possible.

This is not accurate, the parameter is not necessary as long as /sys/module/nvme_core/parameters/multipath = Y.
Same thing in the pre-requisites.

[tiagolobocastro]

Interesting, I wonder why they don't want to support it at all. In some cases DM multipath may be interesting as it adds features like pause/resume.

[tiagolobocastro]

BTW, on AlmaLinux 10 node:

cat: /sys/module/nvme_core/parameters/multipath: No such file or directory ls: cannot access '/sys/module/nvme_core/parameters/multipath': No such file or directory

RedHat states that this is enabled by default, but not sure where this can be verified...

Is this on a system with nvme_core loaded?
If we can't read the multipath then IIRC we're going to consider this node as ANA incapable.
We may need another way of verifying for multipath if this is the case

[gabrielgbs97]

nvme_core

Yes, (re)loaded via modprobe to verify, loaded across reboots with

echo "nvme_tcp" | sudo tee /etc/modules-load.d/nvme_tcp.conf

If we can't read the multipath then IIRC we're going to consider this node as ANA incapable.
We may need another way of verifying for multipath if this is the case

My nodes have:
openebs.io/csi-node.nvme-ana=false

[gabrielgbs97]

So, as of now, I think OpenEBS Mayastor NVME ANA would not work on RHEL 10 and derivates. I cannot find traces of multipath across /sys/module/nvme_core tree. Another fragile strategy would be detecting node as ANA capable as default, if:

• nvme_core is loaded
• /sys/module/nvme_core/parameters/multipath does not exist

Otherwise consider it false. If we look at Linux kernel docs (6.16.0-rc5):
https://docs.kernel.org/admin-guide/nvme-multipath.html#

It only talks about /sys/module/nvme_core/parameters/iopolicy as a related parameter with multipath. Not sure if it's present on RHEL9 when /sys/module/nvme_core/parameters/multipath = N ...

[tiagolobocastro]

At least for me on 6.15.2 vm it's still there:

[root@nixos:/localpv]# uname -a
Linux nixos 6.15.2 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jun 10 11:17:11 UTC 2025 x86_64 GNU/Linux

[root@nixos:/localpv]# cat /sys/module/nvm^C

[root@nixos:/localpv]# modprobe nvme_core

[root@nixos:/localpv]# cat /sys/module/nvme_core/parameters/multipath 
Y

[gabrielgbs97]

A Linux Kernel developer may put some light on thsi, but:
https://issues.redhat.com/browse/RHEL-67045?jql=text%20~%20%22nvme%20multipath%22

Here RH states:

In RHEL-10 we are no longer supporting DMMP multi-pathing with NVMe. Therfore, we want to remove all of the code associated with CONFIG_NVME_MULTIPATH from upstream kernel, and modify all user level components that my be dependent upon the kernel configuration and sysfs variables.

&dev_attr_ana_grpid.attr,
&dev_attr_ana_state.attr,
&subsys_attr_iopolicy.attr,

The following kernel code paths will be affected.
C symbol: CONFIG_NVME_MULTIPATH

My take: all CONFIG_NVME_MULTIPATH related code is removed and native multipath is enabled hardcoded. It's just that we don't know a simple interface to know it.

[tiagolobocastro]

Also:

Jira:RHEL-68504[[1]](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/10.0_release_notes/overview#ftn.PrivateTicketFootnote)

The nvme_core.multipath parameter has been removed

In RHEL 10, the use of DM multipath with NVMe devices over RDMA and FC is no longer supported. As a consequence, the nvme_core.multipath parameter has been removed, the native NVMe multipath is enabled by default, and it can no longer be disabled.

Bug fixes and support for using DM multipath with NVMe devices over RDMA and FC are provided only through the end of the RHEL 9 lifecycle. Note that DM multipath was never supported with NVMe over TCP in any version of RHEL.

[gabrielgbs97]

So, how can we mark nodes as ANA capable?

[tiagolobocastro]

I've raised a PR modifying the behavior when the multipath is not found. Looks like we had already considered a case where the module is present but not the parameter, though we were considering it as non-ana.

[tiagolobocastro]

Unfortunately on 5.15.72-talos the multipath parameter is not present, and the system is not multipath enabled :(

[gabrielgbs97]

On my RHEL 10 nodes, I experience nvme_tcp request disconnections/timeouts. I do not know if this is related to ANA disabled capabilities in mayastor/spdk internals. See console screen output:

AlmaLinux 10.0 (Purple Lion)
Kernel 6.12.0-55.29.1.el10_0.x86_64 on x86_64

OpenEBS 4.3.3

nvme nvme1: failed to send request -13 (repeats)
nvme nume1: failed to connect socket: -111 or -110 (repeats)
nvme nume1: Connect Invalid Data Parameter...
nvme nume1: Connect Invalid
Data Parameter, subsysnqn "nqn.2019-05. io.openebs:..."

1 month ago, cluster was perfectly usable. Online failover for non-ANA capable nodes would be not possible, right?

[tiagolobocastro]

I've checked with Alma 10 and iopolicy is there, so I think we can use it

[gabrielgbs97]

Sorry I was OOF, yes I can confirm this. We talked about this in https://github.com/orgs/openebs/discussions/3987#discussioncomment-13709105 , I just didn't know if it could play as a /sys/module/nvme_core/parameters/multipath substitute.

[tiagolobocastro]

Oh that's right we did. On my approach I don't check when multipath=N, only when multipath is not present: openebs/mayastor-control-plane#1005

[tiagolobocastro]

Fix is merged to develop branch, will be part of the next minor release @gabrielgbs97

[joshuacox]

FWIW - I have been adding nvme_tcp to machines that host openebs, but sometimes I use this shotgun blast to fix nodes with issues:

sudo modprobe nvme_tcp nvme_fabrics nvme_auth nvme_keyring nvme_tcp nvme_core