Skip to content

feat(vortex): add streaming commitment prototype (behind flag)#2727

Draft
srinathln7 wants to merge 1 commit intomainfrom
srinath/prover-vortex-opt
Draft

feat(vortex): add streaming commitment prototype (behind flag)#2727
srinathln7 wants to merge 1 commit intomainfrom
srinath/prover-vortex-opt

Conversation

@srinathln7
Copy link
Copy Markdown
Contributor

@srinathln7 srinathln7 commented Apr 2, 2026

Note: This PR is currently a technical spike that emerged unexpectedly while I was working on a different task. It is currently experimental in nature with Claude. I am still in the process of verifying the results and assessing the full implications of these changes. As such, this is a Draft PR and is not yet ready for formal review. Have not yet tested with a real trace file. As of now, it is created purely for my reference - to be revisited later.

Implements batched RS-extension + incremental Ring-SIS accumulation to reduce peak memory during the commitment phase. The IncrementalHasher accumulates SIS column hashes in NTT domain across row batches, correctly handling limb placement when batches don't align with SIS key polynomial boundaries. Enabled via WithStreamingCommitment(batchSize) option.

Benchmark results (Apple M4 Pro, 8192 cols, rate=2):

  • batch=N (single batch): 15-37% memory reduction, ~0% wall-clock overhead
  • batch=N/2: ~3-10% wall-clock overhead
  • Smaller batches show increasing overhead due to generic (non-AVX512) path

Correctness verified: streaming produces identical Merkle roots, SIS hashes, and encoded matrices as the current path across 13 test configurations.


Note

Medium Risk
Introduces a new incremental Ring-SIS hashing/commitment implementation and switches prover behavior behind a flag; correctness depends on subtle limb-to-polynomial placement and NTT-domain accumulation across batches.

Overview
Adds a streaming Ring-SIS column hashing implementation (IncrementalHasher) that accumulates per-column SIS hashes in the NTT domain across row batches and finalizes with a single inverse FFT to match TransversalHash output.

Introduces CommitMerkleWithSISStreaming, which RS-encodes rows in batches and feeds them into the incremental hasher before building the same Merkle commitment, and factors SIS-hash-to-leaf compression into sisHashesToMerkleLeaves.

Wires the new path into the Vortex compiler/prover behind WithStreamingCommitment(batchSize) / Ctx.StreamingCommitment, and adds tests + benchmarks to verify streaming matches the existing commitment/hashes/encoding across multiple batch sizes.

Written by Cursor Bugbot for commit 0ca0f60. This will update automatically on new commits. Configure here.

Implements batched RS-extension + incremental Ring-SIS accumulation
to reduce peak memory during the commitment phase. The IncrementalHasher
accumulates SIS column hashes in NTT domain across row batches, correctly
handling limb placement when batches don't align with SIS key polynomial
boundaries. Enabled via WithStreamingCommitment(batchSize) option.

Benchmark results (Apple M4 Pro, 8192 cols, rate=2):
- batch=N (single batch): 15-37% memory reduction, ~0% wall-clock overhead
- batch=N/2: ~3-10% wall-clock overhead
- Smaller batches show increasing overhead due to generic (non-AVX512) path

Correctness verified: streaming produces identical Merkle roots, SIS hashes,
and encoded matrices as the current path across 13 test configurations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@srinathln7 srinathln7 marked this pull request as draft April 2, 2026 15:11
@srinathln7 srinathln7 requested a deployment to docker-build-and-e2e April 2, 2026 15:11 — with GitHub Actions Waiting
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.11%. Comparing base (f838178) to head (0ca0f60).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #2727   +/-   ##
=========================================
  Coverage     59.11%   59.11%           
  Complexity     1632     1632           
=========================================
  Files           459      459           
  Lines         18621    18621           
  Branches       2023     2023           
=========================================
  Hits          11008    11008           
  Misses         6941     6941           
  Partials        672      672           
Flag Coverage Δ *Carryforward flag
hardhat 95.29% <ø> (ø)
kotlin 55.07% <ø> (ø) Carriedforward from f838178

*This pull request uses carry forward flags. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants