Bonsai to archive migration

[jframe]

PR description

fixes #9608

This improves the sync of Bonsai archive to sync as Bonsai and then migration to Bonsai archive. This significantly improves the time taken to sync a Bonsai archive node. It does this by waiting until the node is fully in sync and then processing trie logs sequentially.

There is an impact to the engine API performance during the migration. But this is part of the syncing so some degradation is acceptable and the nodes are able to stay in sync during the migration.

• Uses the Synchronizer.subscribeInSync instead being an additional sync step as the initial sync complete event is never fired for full sync and I want to use to the same mechanism for all networks. Placing a step in the Full sync pipeline wouldn't work for post-merge testnets that use backward sync
• Low priority transactions are used for the migrations so that RocksDB throttles migration writes under compaction pressure rather than impacting the engine API. This made the most difference.
• There is a possible edge case with finishing the migration where a block could be missed switching to archive mode. I have deliberately a fix out of this PR as this will be irrelevant once hybrid mode is implemented in the next PR which will be able to handle missing archive data.

Migration Test Results

Network	Test	Run 1	Run 2	Run 3
Hoodi	Migration time	3h 12m	3h 40m	—
Mainnet	Migration time	7d 13h 51m	7d 4h 27m	7d 4h 27m

Migration Approaches to mitigate engine API times

engine_newPayloadV4 — quantiles during migration (seconds)

	p50 mean	p50 peak	p95 mean	p95 peak	p99 mean	p99 peak
control (no migration)	0.28 s	0.81 s	0.81 s	2.88 s	1.06 s	2.95 s
migrate-10 — no protection	1.75 s	4.40 s	4.52 s	48.07 s	6.08 s	54.98 s
migrate-11 — pause on engine API	1.46 s	17.44 s	3.15 s	39.64 s	3.59 s	47.67 s
migrate-12 — low-priority writes	1.22 s	1.80 s	2.47 s	4.94 s	2.82 s	5.71 s

engine_forkchoiceUpdatedV3 — quantiles during migration (seconds)

	p50 mean	p50 peak	p95 mean	p95 peak	p99 mean	p99 peak
control (no migration)	0.16 s	0.23 s	0.33 s	0.57 s	0.38 s	0.76 s
migrate-10 — no protection	1.46 s	4.09 s	5.65 s	11.96 s	7.30 s	13.32 s
migrate-11 — pause on engine API	0.80 s	2.93 s	3.64 s	14.54 s	4.54 s	17.82 s
migrate-12 — low-priority writes	0.26 s	0.52 s	0.57 s	3.68 s	0.73 s	4.60 s

Fixed Issue(s)

fixes #9608

Thanks for sending a pull request! Have you done the following?

• Checked out our contribution guidelines?
• Considered documentation and added the doc-change-required label to this PR if updates are required.
• Considered the changelog and included an update if required.
• For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

• spotless: ./gradlew spotlessApply
• unit tests: ./gradlew build
• acceptance tests: ./gradlew acceptanceTest
• integration tests: ./gradlew integrationTest
• reference tests: ./gradlew ethereum:referenceTests:referenceTests
• hive tests: Engine or other RPCs modified?

[Copilot]

Pull request overview

Adds support for migrating an existing Bonsai node from flat DB mode to archive mode without a full resync, and wires it into controller startup so the archiver is enabled only after migration completes.

Changes:

• Introduces BonsaiFlatDbToArchiveMigrator to backfill archive-keyed flat DB entries from trie logs and persist migration progress.
• Extends BonsaiArchiveFlatDbStrategy with overloads that accept an explicit BonsaiContext for archive writes/removals.
• Updates storage/controller flow: notify subscribers on strategy upgrade, mark strategy pointer volatile, and start migration after the node is in sync.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/pathbased/bonsaiarchive/BonsaiFlatDbToArchiveMigrator.java	New migrator that replays trie logs into archive-keyed flat DB entries and records progress.
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/pathbased/bonsai/storage/flat/BonsaiArchiveFlatDbStrategy.java	Adds context-based overloads used by the migrator to write/delete archive entries at a given block context.
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/pathbased/bonsai/storage/BonsaiWorldStateKeyValueStorage.java	Notifies subscribers to clear caches after switching flat DB strategy to avoid stale snapshot reads.
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/pathbased/common/storage/flat/FlatDbStrategyProvider.java	Makes the strategy reference volatile for cross-thread visibility when switching strategies.
app/src/main/java/org/hyperledger/besu/controller/BesuControllerBuilder.java	Starts migration post-sync (when in FULL mode) and enables archiver only after successful migration.
ethereum/core/src/test/java/org/hyperledger/besu/ethereum/trie/pathbased/bonsaiarchive/BonsaiFlatDbToArchiveMigratorTest.java	New unit tests covering migrator behavior (progress, concurrency, failures, block catch-up).
ethereum/core/src/test/java/org/hyperledger/besu/ethereum/trie/pathbased/bonsai/storage/BonsaiWorldStateKeyValueStorageTest.java	Adds test ensuring subscribers are notified to clear caches on strategy upgrade.

[usmansaleem]

Code review

Found 1 issue:

1. Resource leak: transaction not rolled back on exception in migrateBlocks()

In migrateBlocks(), a SegmentedKeyValueStorageTransaction is opened via startLowPriorityTransaction() on each loop iteration, but there is no try/finally or try-with-resources to guarantee rollback or cleanup if processBlock() throws or the IllegalStateException for a missing trie log is raised. The transaction is neither committed nor rolled back on the exception path, leaking the underlying native RocksDB transaction object. Since the migration runs over millions of blocks on mainnet (7+ days), any transient failure in processBlock() will leak native memory. Wrapping the transaction usage in try { ... tx.commit(); } catch (...) { tx.rollback(); throw; } (or making the transaction AutoCloseable and using try-with-resources) would fix this.

besu/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/pathbased/bonsaiarchive/BonsaiFlatDbToArchiveMigrator.java

Lines 149 to 158 in f819dcf

	final SegmentedKeyValueStorageTransaction tx =
	worldStateStorage.getComposedWorldStateStorage().startLowPriorityTransaction();
	if (maybeTrieLog.isPresent()) {
	processBlock(maybeTrieLog.get(), blockNumber, tx);
	migratedBlockNumber.incrementAndGet();
	} else if (blockNumber > 0) {
	throw new IllegalStateException("No trie log found for block " + blockNumber);
	}
	saveProgress(blockNumber, tx);
	tx.commit();

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[jframe]

Code review

Found 1 issue:

1. Resource leak: transaction not rolled back on exception in migrateBlocks()

In migrateBlocks(), a SegmentedKeyValueStorageTransaction is opened via startLowPriorityTransaction() on each loop iteration, but there is no try/finally or try-with-resources to guarantee rollback or cleanup if processBlock() throws or the IllegalStateException for a missing trie log is raised. The transaction is neither committed nor rolled back on the exception path, leaking the underlying native RocksDB transaction object. Since the migration runs over millions of blocks on mainnet (7+ days), any transient failure in processBlock() will leak native memory. Wrapping the transaction usage in try { ... tx.commit(); } catch (...) { tx.rollback(); throw; } (or making the transaction AutoCloseable and using try-with-resources) would fix this.

besu/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/pathbased/bonsaiarchive/BonsaiFlatDbToArchiveMigrator.java

Lines 149 to 158 in f819dcf

	final SegmentedKeyValueStorageTransaction tx =
	worldStateStorage.getComposedWorldStateStorage().startLowPriorityTransaction();
	if (maybeTrieLog.isPresent()) {
	processBlock(maybeTrieLog.get(), blockNumber, tx);
	migratedBlockNumber.incrementAndGet();
	} else if (blockNumber > 0) {
	throw new IllegalStateException("No trie log found for block " + blockNumber);
	}
	saveProgress(blockNumber, tx);
	tx.commit();

🤖 Generated with Claude Code

• If this code review was useful, please react with 👍. Otherwise, react with 👎.

The general throws case is handled already by RocksDbTransaction. While I don't think it's much of an issue we could create an empty transaction since we create that before trying to get the trielog. So I'll change it to only create the transaction if we aren't going to exit early with the IllegalStateException

[usmansaleem]

LGTM