RFE: speed up SEEK_DATA syscalls

[devZer0]

file access on ZFS can be pathologically slow with SEEK_DATA, as txg_sync is getting triggered, when file contents changing during lseek

this can lead to pathological slow access pattern, like for example being reported here

https://bugzilla.proxmox.com/show_bug.cgi?id=7012

i.e. for example when writing zero data to qcow2 on top of btrfs from inside vm ( dd if=/dev/zero of=/dev/sdo bs=1024k oflag=sync) , i easily get >500MB/s on btrfs , whereas with zfs it drops down to 5MB/s. (on rotating disks with compression set to zstd, detect-zeroes=on for qcow2 disk)

that's a 100 times worse, and especially unfortunate, because effectively no real data from inside vm is being written to physical disk at all , bacause zero write inside vm should be either unmapped or compressed to nearly nothing. so, basically writes which don't hit the pyhsical disk at all can be dead slow, though because of "metadata & sync overhead".

so maybe there is room for improvement here.

i think i have seen other slowness reports related to SEEK_DATA/SEEK_HOLE, but don't remember what/where.

[devZer0]

i have this table generated from chatgpt while having a session trying to understand what's going on.

i can't tell if it's 100% correct, but it may show why other fs besides JFS don't suffer from this type of problem - which is in sync with what i observe via performance testing (details in the proxmox ticket above)

[robn]

SEEK_DATA/SEEK_HOLE perform hole detection. As currently implemented, it only considers the on-disk state, not any in-flight changes. By default, it will wait until in-flight changes are on disk before returning a result. If you would prefer performance over accuracy, you can set the tunable zfs_dmu_offset_next_sync to 0, which will cause it to always assume in-flight changes will yield data, not holes. Depending on the workload however, that may end up reducing performance, as it may then result in the calling program doing additional work that it wouldn't otherwise have to do. You'll need to test.

Incidentally, your dd command has oflag=sync, which is forcing all these writes to be synchronous, which definitely hurts performance. That command isn't doing seeks though, so I'm not sure what you're expecting in this context.

Comparisons with other filesystems are largely irrelevant, as they have different construction and make different correctness guarantees.

I suspect you'll likely do a lot better by not laying qcow2 on top ZFS, or at least, not doubling up on features. For example, ZFS is already doing zero detection and hole punching in its compression system; having qcow2 also do it is just double-handling.

[devZer0]

That command isn't doing seeks though, so I'm not sure what you're expecting in this context

no it doesn't, but when issued inside VM, the qemu/kvm layer will turn that into seeks, as in proxmox by default qemu layer tries to detect zero writes , and turn that into unmap operation with qcow2 virtual disk files. and that's why it lseeks at that level, and this is where the performance issue starts...

yes, i know i can workaround with turning of zero detection in proxmox, but there is (yet) no setting in the gui to disable that.

what's mentioned here may a rare/special use case and you are right, that one should consider other options - but you never know if there are other cases where this happens , which won't get reported because not being recognized by the end user....

[robn]

Yeah, so to the extent that ZFS is involved here, it's really more of a "tuning guide" kind of thing - "for best results with this combination of software, set X, Y and Z".

Beyond that, not much more we can do in the short term. I think any serious improvement is going to require considering dirty buffers not yet committed to disk, and that's incredibly difficult to do for a bunch of reasons, and I'm not sure that its worth the effort and complexity (to me and my corporate masters, at least; others may have different ideas).