zstd

Zstandard compression for Go, transpiled, CGo-free.

Can be built with CGO_ENABLED=0 and do not need system libzstd, pkg-config, a C compiler, or an external zstd binary at runtime.

The module exposes two packages:

• go.dw1.io/zstd: the Go API for application code.
• go.dw1.io/zstd/abi: the low-level generated libzstd ABI for generated or systems packages that need raw zstd symbols.

Generated ABI output is checked in. Users do not need to run the generator to import the module.

Status

This repository currently generates ABI files for:

• linux/amd64
• linux/arm64

Other targets fail clearly from go.dw1.io/zstd/abi instead of silently using a different implementation.

The checked-in source baseline is upstream zstd v1.5.7 at commit f8745da6ff1ad1e7bab384bd1f9d742439278e99.

Install

go get go.dw1.io/zstd

One-Shot Compression

package main

import (
	"fmt"

	"go.dw1.io/zstd"
)

func main() {
	src := []byte("hello zstd")

	compressed, err := zstd.Compress(nil, src, zstd.CompressOptions{
		Level:    zstd.DefaultLevel(),
		Checksum: true,
	})
	if err != nil {
		panic(err)
	}

	decompressed, err := zstd.Decompress(nil, compressed, zstd.DecompressOptions{})
	if err != nil {
		panic(err)
	}

	fmt.Println(string(decompressed))
}

For frames without a declared content size, one-shot decompression requires an explicit limit:

decompressed, err := zstd.Decompress(nil, compressed, zstd.DecompressOptions{
	MaxDecodedSize: 8 << 20,
})

Use NewReader for large or unknown-size streams where allocating the full decoded payload up front is not appropriate.

Streaming

var compressed bytes.Buffer

w, err := zstd.NewWriter(&compressed, zstd.WriterOptions{
	CompressOptions: zstd.CompressOptions{Checksum: true},
})
if err != nil {
	panic(err)
}
if _, err := w.Write([]byte("streamed zstd")); err != nil {
	panic(err)
}
if err := w.Close(); err != nil {
	panic(err)
}

r, err := zstd.NewReader(bytes.NewReader(compressed.Bytes()), zstd.ReaderOptions{})
if err != nil {
	panic(err)
}
out, err := io.ReadAll(r)
if err != nil {
	panic(err)
}
if err := r.Close(); err != nil {
	panic(err)
}

Writer supports Write, Flush, Close, and Reset. Reader supports incremental reads, multiple frames, skippable frames, dictionaries, window-size limits, and Reset.

Dictionaries

Raw dictionaries can be passed directly:

dict := []byte("customer status invoice total account")
src := []byte("customer account status: invoice total paid")

compressed, err := zstd.Compress(nil, src, zstd.CompressOptions{Dict: dict})
if err != nil {
	panic(err)
}
decompressed, err := zstd.Decompress(nil, compressed, zstd.DecompressOptions{Dict: dict})
if err != nil {
	panic(err)
}

For repeated workloads, compile dictionaries once with NewCDict and NewDDict. The package also exposes TrainDictionary, FinalizeDictionary, DictID, and DictHeaderSize.

Warning

CDict and DDict values are immutable after construction and may be shared for read-only use. Mutable compressors, decompressors, readers, and writers are NOT safe for concurrent mutation.

Frame Metadata

Use the metadata APIs to inspect frames before deciding how to decode them:

header, err := zstd.FrameHeaderOf(compressed)
if err != nil {
	panic(err)
}

fmt.Println(header.ContentSize, header.WindowSize, header.DictID, header.Checksum)

Available helpers include ContentSize, FrameCompressedSize, FrameHeaderSize, FrameDictID, IsFrame, and IsSkippableFrame.

Unknown content size and malformed content-size errors are distinct. A forged or very large content-size field does not cause implicit allocation without a user provided limit.

Errors

zstd result-code failures are wrapped in typed Go errors:

• CompressionError
• DecompressionError
• DictionaryError
• FrameError
• StreamError
• LimitError

Use errors.As to inspect categories and the underlying *zstd.Error details:

_, err := zstd.Decompress(nil, []byte("not a zstd frame"), zstd.DecompressOptions{
	MaxDecodedSize: 1024,
})

var zerr *zstd.DecompressionError
if errors.As(err, &zerr) {
	fmt.Println(zerr.Err.Operation, zerr.Err.Code, zerr.Err.Name)
}

Stream errors wrap the underlying io.Reader or io.Writer error.

Raw ABI Package

go.dw1.io/zstd/abi exposes the generated libzstd-like ABI for low-level consumers. It is intentionally version-coupled to the pinned upstream source and uses ccgo/modernc runtime conventions.

Application code should prefer the root package. Generated packages that need raw zstd symbols can depend on go.dw1.io/zstd/abi instead of carrying a private libzstd copy.

Generation

The generator lives in internal/cmd/genzstd and is wired through go generate:

go generate ./...

The generated package includes:

• abi/zstd_linux_amd64.go
• abi/zstd_linux_arm64.go
• abi/SOURCE-MANIFEST.txt
• abi/SYMBOLS.txt
• ABI symbol compile tests
• unsupported-target build failure stubs

SOURCE-MANIFEST.txt records the zstd source commit, header version, selected source files, macros, include directories, local patch hashes, target list, and ccgo version. SYMBOLS.txt records the public header symbols expected in the ABI package.

Check that committed generated output is current with:

go run ./internal/cmd/genzstd -check

Verification

Useful local checks:

CGO_ENABLED=0 go test ./...
go test ./...
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go test ./...
go run ./internal/cmd/genzstd -check

The test suite covers one-shot APIs, streaming, reusable contexts, dictionaries, dictionary training/finalization, frame metadata, typed errors, upstream golden fixtures, same-source official libzstd oracle comparisons where a C compiler is available, generated-consumer ABI usage, unsupported targets, and fuzz targets.

Short fuzz smoke:

go test -run '^$' -fuzz=FuzzRoundTrip -fuzztime=1s .

License

This module preserves the upstream zstd BSD-style license path. See LICENSE.