Memory Leak with ZFS Record Size=512B, Using Latest Code

[saju-p]

System information

Type	Version/Name
Distribution Name	RHEL / RockyLinux
Distribution Version	RHEL 10.1 / RockyLinux 9.4
Kernel Version	6.12.0-55.9.1.el10_0.x86_64 / 5.14.0-427.37.1.el9_4.x86_64
Architecture	x86_64
OpenZFS Version	zfs-2.4.1 (tag zfs-2.4.1, commit 1c702dd). Also reproduced with zfs-2.4.0 (tag zfs-2.4.0, commit 7433349).

Describe the problem you're observing

We run FIO tests for 60-90 minutes. With ZFS record size=512B, it fails. However, it passes with ZFS record size=128K.
Previously, it used to pass for both record sizes when we were testing 2.3.4. We will do more testing on the intermittent tags to find out where it broke.

ZFS 2.4.1, ZFS 2.4.0, ZFS 2.4.0 RC4
ashift=12,(4K ) fio block size = 4K, zfs record size=128K, fio jobs=64, size=2G
WRITE-> PASS
READ -> PASS

ashift=12,(4K ) fio block size = 4K, zfs record size=512B, fio jobs=64, size=2G
WRITE-> PASS
READ -> FAIL. Memory leak and crashed/hanged

[amotin]

"zfs record size=512B" with "ashift=12,(4K)" is not sane. It will likely consume 10 times more storage than needed. There will also be enormous memory overhead. All thus is obviously not a reason for a leak, but are you sure you have one?

[saju-p]

@amotin , You are correct — recordsize=512B with ashift=12 is technically valid but not practical: at 64 jobs × 2G, it generates ~134M ZFS records and the metadata alone exhausts available RAM. We retested version 2.3.4 and found it also has the issue that we missed earlier.

We have updated our test matrix to start at recordsize=4K (aligned with ashift) and the issue does not reproduce.

We could update the help section to recommend that the 'recordsize' should be greater than or equal to the 'ashift' size.