Zeta — an F# implementation of DBSP for .NET 10

Zeta is an F# implementation of Database Stream Processing (DBSP) — the algebra of streaming incremental view maintenance — targeting .NET 10 with near-zero per-operation allocations and a surface that feels native to both F# and C#.

"DBSP" is the algorithm (Budiu et al., VLDB'23); "Zeta" is this library. Academic references in docs, tests, and proofs stay as DBSP; product and API references are Zeta. See docs/NAMING.md for the full split.

The thesis: reproducible stability

Maintainer directive, 2026-04-22:

is obvious to all personas who come across our project the whole point is reproducable stability

See AGENTS.md §"The purpose: reproducible stability".

What DBSP is

DBSP defines a tiny, complete calculus for incremental computation over changing relations. Three primitives — delay (z^-1), differentiation (D), and integration (I) — together with lifting (↑) let you transform any query Q into its incremental form Q^Δ = D ∘ Q^↑ ∘ I. Plain-English definitions for every term here live in docs/GLOSSARY.md. Key identities:

• I ∘ D = D ∘ I = id (bijection on streams)
• (Q1 ∘ Q2)^Δ = Q1^Δ ∘ Q2^Δ (chain rule)
• Linear Q: Q^Δ = Q (trivial incrementalization)
• Bilinear Q (e.g. join): (a ⋈ b)^Δ = Δa ⋈ Δb + z^-1(I(a)) ⋈ Δb + Δa ⋈ z^-1(I(b))
• distinct^Δ: the paper's H function, work bounded by |Δ|

See src/Core/Incremental.fs for the implementation of these theorems and tests/Tests.FSharp/Circuit/Incremental.Tests.fs for the equivalence proofs as executable tests.

What Zeta adds on top

Zeta is not just the paper's three-primitive kernel — it ships an implementation + a catalogue of operators, sketches, CRDTs, and runtime primitives built on top:

• Kernel primitives. Delay, Integrate, Differentiate, Constant — the paper's three plus a Constant operator. See src/Core/Primitive.fs.
• Operators. Map, filter, join (inner + left-outer), groupBy-sum, consolidate, index, distinct with H-incrementalization. See src/Core/Operators.fs and src/Core/Advanced.fs.
• Aggregates and windowing. Sum, average, scalar-fold, sliding window, lag-1, watermark + speculative-watermark. See src/Core/Aggregate.fs, src/Core/Window.fs, src/Core/Watermark.fs.
• Sketches. Bloom + CountingBloom, Count-Min, HyperLogLog, HyperMinHash, Kll, Haar, Tropical. See src/Core/BloomFilter.fs, src/Core/CountMin.fs, src/Core/Sketch.fs.
• CRDT family. G-counter, PN-counter, OR-set, LWW, DeltaCrdt. See src/Core/Crdt.fs and src/Core/DeltaCrdt.fs.
• Recursion & hierarchy. Fixed-point Recursive, ClosureTable, NestedCircuit, HigherOrder, Residuated. See src/Core/Recursive.fs, src/Core/Hierarchy.fs.
• Storage & durability. Spine family (Balanced / Disk / SpineAsync / SpineSelector), Merkle, FastCdc content-defined chunking, checkpoint / durability modes, Witness-Durable Commit skeleton. See src/Core/Spine.fs and src/Core/Durability.fs.
• Runtime. Mailbox + work-stealing runtimes, consistent-hash sharding, information-theoretic sharder, chaos environment, deterministic simulation harness, metrics + tracing. See src/Core/Runtime.fs, src/Core/ConsistentHash.fs, src/Core/ChaosEnv.fs.
• Wire + SIMD. Arrow IPC serializer, FsPickler serializer, hardware-CRC, SIMD merge / dispatch. See src/Core/ArrowSerializer.fs, src/Core/Simd.fs.
• Plugins. src/Bayesian/BayesianAggregate.fs — online Bayesian posteriors (Beta / Normal-Inverse-Gamma / Dirichlet) as first-class operators. Writing a plugin? Start with docs/PLUGIN-AUTHOR.md for the public IOperator<'T> surface, the capability-tag system, and the PluginHarness test loop. Full design rationale at docs/research/plugin-api-design.md.

The paper's algebra is the invariant; Zeta is the F#/.NET realisation + a research surface for operators that compose correctly under the paper's rules.

Quick tour

open Zeta.Core

let circuit = Circuit.create ()
let orders = circuit.ZSetInput<string * int64> ()

let totals =
    circuit.GroupBySum(
        orders.Stream,
        System.Func<_,_>(fst),
        System.Func<_,_>(snd))

let view = circuit.Output(circuit.IntegrateZSet totals)

async {
    orders.Send(ZSet.ofKeys [ "alice", 100L ; "bob", 50L ])
    do! circuit.StepAsync () |> Async.AwaitTask

    for e in view.Current do
        printfn "%A -> weight %d" e.Key e.Weight
}
|> Async.RunSynchronously

And the same thing, C#-side:

using Zeta.Core;

var circuit = new Circuit();
var orders = circuit.ZSetInput<(string, long)>();

var totals = circuit.GroupBySum(orders.Stream,
    t => t.Item1,
    t => t.Item2);

var view = circuit.Output(circuit.IntegrateZSet(totals));

orders.Send(ZSetModule.ofKeys(new[] { ("alice", 100L), ("bob", 50L) }));
await circuit.StepAsync();

foreach (var e in view.Current)
    Console.WriteLine($"{e.Key} -> weight {e.Weight}");

Performance design

Pattern	Applied	Why
ReadOnlySpan<T> on hot loops	✓	JIT elides bounds checks, vectorizes
ArrayPool<T>.Shared.Rent / Return	✓	Scratch buffers never hit GC
GC.AllocateUninitializedArray	✓	Skip zero-init for blittable T
ImmutableCollectionsMarshal.AsImmutableArray	✓	One alloc per output; no double-copy
Struct IComparer<T> + MemoryExtensions.Sort	✓	Monomorphized sort, no delegate alloc
CollectionsMarshal.GetValueRefOrAddDefault	✓	One hash lookup instead of two
IsReferenceOrContainsReferences<T>	✓	JIT-constant-folded clearArray
[<InlineIfLambda>] + let inline	✓	Monomorphize lambda body at callsite
[<Struct; IsReadOnly>] everywhere	✓	Zero-copy cursors and handles
Channel / ConcurrentQueue for I/O	✓	Lock-free many-producer inputs
backgroundTask CE	✓	F#'s library-safe async (no SC capture)
ValueTask.CompletedTask sync path	✓	No state-machine alloc on sync ticks

Layout

src/Core/            F# core library
├── Algebra.fs            Weight (= int64) and helpers
├── Pool.fs               ArrayPool + exact-size allocation helpers
├── ZSet.fs               Z-set algebra — add, neg, map, filter, join, distinct
├── IndexedZSet.fs        Key-indexed Z-sets for efficient joins
├── Circuit.fs            Op / Stream / Circuit scheduler
├── Primitive.fs          z^-1, integrate, differentiate
├── Operators.fs          Linear/bilinear operators, extension methods
├── Handles.fs            Input/output handles (lock-free)
└── Incremental.fs        Q^Δ = D ∘ Q ∘ I transformation helpers
tests/Tests.FSharp/  F# tests — xUnit v3 + FsUnit + FsCheck
tests/Tests.CSharp/  C# tests — xUnit v3
bench/Benchmarks/    BenchmarkDotNet + MemoryDiagnoser
samples/Demo/        Working example

Building and testing

dotnet build -c Release
dotnet test  -c Release
dotnet run --project samples/Demo -c Release
dotnet run --project bench/Benchmarks -c Release -- --filter "*"

Running the F# static analyzers

Two analyzer packs are wired in: G-Research.FSharp.Analyzers and Ionide.Analyzers. Run them via the FSharp.Analyzers.Build MSBuild target:

dotnet tool install --global fsharp-analyzers
dotnet msbuild src/Core/Core.fsproj \
  -t:AnalyzeFSharpProject \
  -p:Configuration=Release \
  -p:FSharpAnalyzersExeHost=

Contributing

Three entry points, pick whichever fits:

• First time contributing, or directing an AI to do the work for you? Read docs/FIRST-PR.md. UI-first, plain-language, no git or F# required.
• Comfortable with GitHub, git, and F#? Read CONTRIBUTING.md. Shorter; assumes the tools.
• An AI agent coordinating work without a human in the loop? Read docs/AGENT-CLAIM-PROTOCOL.md for the git-native claim spec, and docs/AGENT-ISSUE-WORKFLOW.md for the dual-track principle and platform adapters.

Acknowledgements

Zeta follows the algebra of Budiu, McSherry, Ryzhyk, and Tannen (arXiv:2203.16684). The Rust reference implementation is Feldera; Zeta keeps the same operator surface while leaning hard into .NET-native patterns — ArrayPool, Span<T>, struct comparers, CollectionsMarshal — instead of mirroring Rust's trace/spine machinery verbatim.

Zeta follows a glass-halo transparency commitment: human-maintainer and agent decisions are equally visible in the same public version-controlled substrate — neither party can hide a move from the other. The bilateral / symmetric form is original to Zeta; see docs/ALIGNMENT.md §"Symmetric transparency: the glass halo" for the full definition and mutual commitment.