Consider using zstandard dictionary to compress chunks

[ddorian]

Hi,

Something like: https://github.com/facebook/rocksdb/wiki/Dictionary-Compression
Should provide much better compression especially when using groupname to group together similar documents.

Makes sense ?

[jobergum]

Vespa uses zstd as default for summary log store already

vespa/searchcore/src/vespa/searchcore/config/proton.def

Line 223 in fb855d5

summary.log.chunk.compression.type enum {NONE, LZ4, ZSTD} default=ZSTD

[ddorian]

Yes, but do you compress using a shared dictionary ?
I'm assuming you use something like a sstable in rocksdb and chunks are stored together in an immutable file after compaction..

[jobergum]

No we don't use any shared dictionaries, default chunk size is 64KB so multiple documents might be compressed in the same chunk.

[ddorian]

I know you don't.
The issue is to enable it.
lower chunks, faster decompression-speed

[vekterli]

Document store files are append-only (though not immutable like SSTables AFAIK) so using a shared dictionary across chunks certainly has great potential to improve compression ratios. We've considered this but haven't gotten around to implementing it yet. But it's on our radar! IIRC doc store files (unlike SSTable files) use a header rather than a footer, so we'd probably need to do some a-priori random sampling of documents to generate a reasonably good candidate dictionary to write before starting to compress and write the chunks themselves.