Skip to content

Headers backward#9510

Merged
pinges merged 261 commits intobesu-eth:mainfrom
pinges:HeadersBackward
Feb 6, 2026
Merged

Headers backward#9510
pinges merged 261 commits intobesu-eth:mainfrom
pinges:HeadersBackward

Conversation

@pinges
Copy link
Copy Markdown
Contributor

@pinges pinges commented Nov 28, 2025

This pull request introduces several improvements and fixes related to snap sync.

Snap sync chain download is using two pipelines now.

  • The first pipeline downloads the headers. The download is starting from a trusted pivot block. For mainnet that trusted pivot block is a safe block that was sent by the CL (FCU). Headers are downloaded backwards from that trusted block and each block can be validated using the parent block hash contained in the previous header. The last header downloaded is block 1, which is validated by the parent block hash of block 2, and in addition validated against the known genesis block by checking whether the parent block hash matches the block hash of the genesis block (block 0).
  • All headers are written to the database and are fully trusted at the end of the header download.
  • The second pipeline now uses the trusted headers to download the bodies and receipts. This download starts from the genesis block or a known checkpoint. Mainnet blocks and receipts by default are downloaded starting at the first block of the Paris hard fork. The first step of the second pipeline retrieves the headers from the database. Blocks and receipts are requested from peers and are validated against the block hash or the receipts root contained in the header.
  • The storing of the bodies and receipts was optimized by doing less validation as these are validated against fully trusted headers now.

Test and Configuration Enhancements:

  • Fixed BFT sync acceptance tests. BFT SNAP and CHECKPOINT sync tests now use snap.

Build and Miscellaneous Fixes:

  • Fixed test resource generation in the Gradle build script to only generate the test blockchain if the binary does not already exist.

Syncing Times compared to latest
*Chain download is 2-3 hours faster than latest, but as state download with state heal is slower this does not make a big difference in total sync time.

image

@github-project-automation github-project-automation bot moved this to Backlog in RC 25.12.0 Dec 4, 2025
Gabriel-Trintinalia and others added 28 commits December 15, 2025 10:25
* Bump spec tests

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>

* changelog

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>

---------

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Usman Saleem <usman@usmans.info>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* Add test summary reporting to block-test command

The block-test command now tracks and displays a summary of test
execution results, including total tests, passed/failed counts, and
a list of failed tests with their failure reasons. This improves
usability when running large test suites by providing a clear
overview of test outcomes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

* Refactor block-test summary output for better maintainability

Address code review feedback by:
- Extract separator string into SEPARATOR constant to reduce duplication
- Reuse failureReason variable in output statements instead of duplicating
  formatting logic

This improves code maintainability by consolidating the failure message
formatting in a single location.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

* Address review feedback: simplify TestResults implementation

- Remove unused testName parameter from recordPass() method
- Remove totalTests field and calculate it dynamically in printSummary()
  from passed + failed counts
- Simplify SEPARATOR constant by removing embedded newline and use
  explicit out.println() for consistent formatting
- Add hasTests() helper method to check if any tests were executed

This reduces redundant state tracking and improves code clarity.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>

---------

Signed-off-by: Bhargava Shastry <bhargava.shastry@ethereum.org>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Usman Saleem <usman@usmans.info>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* Implement era1 file export

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Clean up OutputStreamFactory to pass inspections

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix javadoc

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix javadoc

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix more compile checks

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix more compile checks

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* fix more compile checks

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Output era1 files to supplied directory

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Implement accumulator and block index

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix compile errors

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix javadoc

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Couple of fixes, add logging

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Implement our own merkleizer, try many things

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix accumulator calculations

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix block number to file number calculation

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Correctly use accumulator hash in era1 filename

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Update manual test to use nimbus.team source for expected era1 file

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add javadoc to new Era1* files

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BlocksSubCommandTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add MerkleizerTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor to produce Era1FileWriter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add unit test for Era1FileWriter, fix discovered bug in Era1FileWriter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor to produce Era1Accumulator and add unit test

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor to produce Era1BlockIndexConverter and add unit test

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Unit test Era1BlockExporter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix javadoc issues

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Rename difficultysForFile to difficultiesForFile

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Move network selection outside of loop

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Make blocks export option non-hidden, add changelog entry

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Use network option in era1 block exporter

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Change Era1BlockExporter.export arguments to start/end block, and handle file number conversion internally

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add additional Merkleizer tests and guard against empty leaves

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BlocksSubCommandTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Move new rlp encoder classes into encoding package, add encodeWrapped for BlockBodyEncoder, add TransactionReceiptEncodingConfiguration parameter to TransactionReceiptEncoder.encode

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

---------

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>
Signed-off-by: Matilda-Clerke <matilda.clerke@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…u-eth#9320)

This reverts commit 42306f9.

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…esu-eth#9194)

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* remove parentBeaconBlockRoot from getPayload result to match the specs

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* add burn-in tasks

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…eth#9183)

Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>

Allows for enabling and disabling stack, storage, returnData, etc.. for these 2 specific RPC endpoints: debug_traceStandardBlockToFile and debug_traceStandardBadBlockToFile. It does not change any previous defaults for memory which is shown by default.

Example of usage:

curl --location --globoff 'http://localhost:8545' --data '{
    "jsonrpc": "2.0",
    "method": "debug_standardTraceBlockToFile",
    "params": [
        "0x2476b23a03dd87f272724a12213dc4a1ca8f13a1aa9d128a690e1b7d7d312e5e",
        {
            "txHash": "0xdfd5028da0f2f306ae71dc558c26bb1935b13ec3a0e909223eaa77bf761189a8",
            "disableStack": true,
            "disableMemory": false,
            "disableStorage": false
        }
    ],
    "id": 1
}'

curl --location --globoff 'http://localhost:8545/' --data '{
    "jsonrpc": "2.0",
    "method": "debug_standardTraceBadBlockToFile",
    "params": [
        "0x2476b23a03dd87f272724a12213dc4a1ca8f13a1aa9d128a690e1b7d7d312e5e",
        {
            "txHash": "0xdfd5028da0f2f306ae71dc558c26bb1935b13ec3a0e909223eaa77bf761189a8",
            "disableStack": true,
            "disableMemory": false,
            "disableStorage": false
        }
    ],
    "id": 1
}'

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…lock to be completed (besu-eth#9332)

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Also remove sigleton pattern from ReferenceTestProtocolSchedules.

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* update netty

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…fer value is zero (besu-eth#9315)

Signed-off-by: Miroslav Kovar <miroslavkovar@protonmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…ng with peer task system usage (besu-eth#9178)

* Remove GetBlockFromPeerTask and RetryingGetBlockFromPeersTask, replacing with peer task system usage

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BackwardSyncContextTest

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Use preferred peer first for block retrieval in BlockPropagationManager

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Add new GetHeadersFromPeerTask constructor which doesn't take a block number and use it in SyncStepStep

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Refactor PeerTaskExecutor usage in BlockPropagationManager for better readability

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix a != that should have been ==

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Re-add AbstractBlockPropagationManagerTest.shouldRequestBlockFromOtherPeersIfFirstPeerFails and modify to use mock PeerTaskExecutor

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Re-add BackwardSyncContextTest.whenBlockNotFoundInPeers_shouldRemoveBlockFromQueueAndProgressInNextSession

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Fix BlocksSubCommandTest again, but differently

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

* Remove commented code

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>

---------

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
…su-eth#9337)

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* assert no errors before asserting method calls

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* exclude icebox'd issues from being marked stale

Signed-off-by: jflo <justin+github@florentine.us>

* Update .github/workflows/stale-issues.yml

Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Justin Florentine <justin+github@florentine.us>

---------

Signed-off-by: jflo <justin+github@florentine.us>
Signed-off-by: Justin Florentine <justin+github@florentine.us>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
* add instructions for DCO into DCO.md

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
@pinges pinges marked this pull request as ready for review December 18, 2025 06:50
Copilot AI review requested due to automatic review settings December 18, 2025 06:50
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@pinges pinges requested a review from Copilot December 19, 2025 07:04
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

} else if (taskResult.responseCode() != PeerTaskExecutorResponseCode.SUCCESS
|| taskResult.result().isEmpty()) {
ethPeer.recordUselessResponse("headers");
return getHeader(ethContext, getEthPeer(ethPeers, task), ethPeers, task);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're happy to retry with different peers anyway, could we just call peerTaskExecutor.execute instead and let the PeerTaskExecutor handle retrying?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been talking to Matilda about that and I think when we are adding a bit of functionality to the PeerTaskExecutor we should be able to simplify this code quite a bit. I would prefer to do that in a follow up PR.

@macfarla macfarla marked this pull request as draft January 19, 2026 23:31
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
@pinges pinges marked this pull request as ready for review January 28, 2026 01:51
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
@pinges pinges merged commit c3896bc into besu-eth:main Feb 6, 2026
46 checks passed
pinges added a commit that referenced this pull request Feb 10, 2026
…9510 and #9621)

Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.