feat(ant-node): persist record index to eliminate startup CPU spike#3484
Open
tomdif wants to merge 1 commit intomaidsafe:mainfrom
Open
feat(ant-node): persist record index to eliminate startup CPU spike#3484tomdif wants to merge 1 commit intomaidsafe:mainfrom
tomdif wants to merge 1 commit intomaidsafe:mainfrom
Conversation
…tup CPU spike Cache the record metadata index (key, ValidationType, DataTypes) to disk so that node restarts read one small msgpack file instead of doing a parallel AES-256-GCM-SIV decrypt of every record on disk (up to 16K files). This eliminates the 40+ second 100% CPU spike on startup after binary upgrades. - Add PersistedRecordIndex struct serialized via rmp_serde - On startup, try cached index first; fall back to full decrypt scan - Validate index with encryption_seed match + spot-check files on disk - Flush index on payment, on initial construction, and on Drop - Cap rayon threads in full-scan fallback to (available_parallelism - 1) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
available_parallelism - 1, reserving one core for the OS/tokio runtimeHow it works
try_load_cached_index()reads a smallrecord_indexmsgpack file fromhistoric_quote_dirupdate_records_from_an_existing_store()full-decrypt scan entirelyDropSecurity
Test plan
cargo check -p ant-node— passescargo clippy -p ant-node --all-targets -- -Dwarnings— passes cleancargo test --release --package ant-node --lib— 76/76 tests pass (includingcan_store_after_restart)