feat(vortex): add streaming commitment prototype (behind flag)#2727
Draft
srinathln7 wants to merge 1 commit intomainfrom
Draft
feat(vortex): add streaming commitment prototype (behind flag)#2727srinathln7 wants to merge 1 commit intomainfrom
srinathln7 wants to merge 1 commit intomainfrom
Conversation
Implements batched RS-extension + incremental Ring-SIS accumulation to reduce peak memory during the commitment phase. The IncrementalHasher accumulates SIS column hashes in NTT domain across row batches, correctly handling limb placement when batches don't align with SIS key polynomial boundaries. Enabled via WithStreamingCommitment(batchSize) option. Benchmark results (Apple M4 Pro, 8192 cols, rate=2): - batch=N (single batch): 15-37% memory reduction, ~0% wall-clock overhead - batch=N/2: ~3-10% wall-clock overhead - Smaller batches show increasing overhead due to generic (non-AVX512) path Correctness verified: streaming produces identical Merkle roots, SIS hashes, and encoded matrices as the current path across 13 test configurations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2727 +/- ##
=========================================
Coverage 59.11% 59.11%
Complexity 1632 1632
=========================================
Files 459 459
Lines 18621 18621
Branches 2023 2023
=========================================
Hits 11008 11008
Misses 6941 6941
Partials 672 672
*This pull request uses carry forward flags. Click here to find out more. 🚀 New features to boost your workflow:
|
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: This PR is currently a technical spike that emerged unexpectedly while I was working on a different task. It is currently experimental in nature with Claude. I am still in the process of verifying the results and assessing the full implications of these changes. As such, this is a
DraftPR and is not yet ready for formal review. Have not yet tested with a real trace file. As of now, it is created purely for my reference - to be revisited later.Implements batched RS-extension + incremental Ring-SIS accumulation to reduce peak memory during the commitment phase. The IncrementalHasher accumulates SIS column hashes in NTT domain across row batches, correctly handling limb placement when batches don't align with SIS key polynomial boundaries. Enabled via WithStreamingCommitment(batchSize) option.
Benchmark results (Apple M4 Pro, 8192 cols, rate=2):
Correctness verified: streaming produces identical Merkle roots, SIS hashes, and encoded matrices as the current path across 13 test configurations.
Note
Medium Risk
Introduces a new incremental Ring-SIS hashing/commitment implementation and switches prover behavior behind a flag; correctness depends on subtle limb-to-polynomial placement and NTT-domain accumulation across batches.
Overview
Adds a streaming Ring-SIS column hashing implementation (
IncrementalHasher) that accumulates per-column SIS hashes in the NTT domain across row batches and finalizes with a single inverse FFT to matchTransversalHashoutput.Introduces
CommitMerkleWithSISStreaming, which RS-encodes rows in batches and feeds them into the incremental hasher before building the same Merkle commitment, and factors SIS-hash-to-leaf compression intosisHashesToMerkleLeaves.Wires the new path into the Vortex compiler/prover behind
WithStreamingCommitment(batchSize)/Ctx.StreamingCommitment, and adds tests + benchmarks to verify streaming matches the existing commitment/hashes/encoding across multiple batch sizes.Written by Cursor Bugbot for commit 0ca0f60. This will update automatically on new commits. Configure here.