Reduce tx p2p broadcast bandwidth and memory used

[fab-10]

Reduce P2P transaction broadcast bandwidth

Overview

A ground-up rework of transaction broadcast and announcement tracking to significantly reduce both memory usage and redundant P2P traffic.

With default configuration an average of 21% less Java heap memory used, total P2P transaction inbound traffic see at least 80% reduction.

The bandwidth optimization comes from a refactoring of the PeerTransactionTracker, where all the related queues about tx broadcast are centralized and consolidated, this allowed to specifically reduce duplicate requests the same tx announcement, that were the main reason of the higher bandwidth usage, then there are other smaller improvements, to avoid not useful work, like taking in account confirmed txs and invalid txs dropped from the pool, plus a fine tuning of marking a tx as seen asap.

The memory optimization comes without losing any memory, since from collected data the amount of distinct hashes tracked across all peers (with the default configuration of 200_000 max tracked per peer) was always below the 300_000, so in this PR the global hash tracking map, with a max size of 300_000, will behave at least the same in remembering seen transactions.

The core change replaces the previous per-peer Set<Hash> tracking (one set per peer) with a single global LRUMap<Hash, PeersSeenState> where each PeersSeenState holds two compact BitSets — one for full transactions seen, one for announcements seen — indexed by a per-peer slot number. This eliminates the dominant memory cost: instead of N_peers × M_hashes hash objects, memory scales with M_hashes only, with O(1) per-peer membership checks via bit operations.

Key changes

PeerTransactionTracker — complete rewrite

• Replaces ConcurrentHashMap<EthPeer, Set<Hash>> with LRUMap<Hash, PeersSeenState>, where PeersSeenState contains two BitSets (transactions / announcements) indexed by peer slot
• Implements EthPeers.ConnectCallback to assign stable slot indices to peers on connect, and clears slot bits on disconnect for index reuse
• Implements BlockAddedObserver: marks confirmed transactions as seen-by-all on HEAD_ADVANCED / CHAIN_REORG events, suppressing rebroadcast of mined transactions
• Separates full-transaction queues (transactionsToSend) from announcement queues (announcementsToSend), both bounded by the new --Xmax-send-queue-size-per-peer option
• New receivedTransactions / receivedAnnouncements entry points for deduplication at ingress; new claimAnnouncementsToRequestFromPeer / consumedAnnouncements for the fetcher loop
• claimTransactionsToSendToPeer / claimTransactionHashesToSendToPeer (bulk) replaced with single-item claimTransactionToSendToPeer / claimAnnouncementToSendToPeer

BufferedGetPooledTransactionsFromPeerFetcher — rewrite

• Pulls announcement state from PeerTransactionTracker instead of managing its own queue
• Retry loop follows the ETH/68 spec: gaps before the last returned transaction are treated as unavailable and dropped; retries continue until an empty or failed response signals end of data
• Request batches are bounded by both hash count (MAX_HASHES) and cumulative byte size

TransactionsMessageSender / NewPooledTransactionHashesMessageSender — rewrite

• Both senders now claim transactions one at a time from the tracker and apply the seen-filter inline, removing the separate deduplication pass
• TransactionsMessage.SizeLimitedBuilder (inline class) replaces the deleted LimitedTransactionsMessages

NewPooledTransactionHashesMessageProcessor / TransactionsMessageProcessor

• Use the new receivedAnnouncements / receivedTransactions tracker entry points, which return only the fresh subset for further processing

ETH68-only announcements

• Pre-ETH68 transaction announcement support removed; TransactionAnnouncement now always carries type and size

RemovalReason

• New stopBroadcasting() method; evicted transactions are removed from all send queues immediately, preventing stale broadcasts

CLI / configuration

• --Xmax-tracked-seen-txs-per-peer renamed to --Xmax-tracked-seen-txs (old name kept as deprecated alias) and default raised from 200 000 to 300 000 — the limit is now global, not per-peer
• New --Xmax-send-queue-size-per-peer (default 100 000) caps per-peer send queues

Metrics

• Added byte-level P2P metrics: p2p_bytes_outbound / p2p_bytes_inbound

Testing

• PeerTransactionTrackerTest, TransactionsMessageSenderTest, NewPooledTransactionHashesMessageSenderTest, BufferedGetPooledTransactionsFromPeerFetcherTest substantially extended and updated to match the new API and registration model
• LimitedTransactionsMessagesTest removed (class deleted); TransactionsMessageTest added for SizeLimitedBuilder
• TransactionAnnouncementEncoderDecoderTest added for the updated announcement encoding

Fixes

fix #8932

Metrics

P2P Transaction Bandwidth

This graph shows where the main bandwidth reduction occurs, there are less GetPooledTransactions requests, that result in much less incoming traffic from the PooledTransactions response.
p2p-a* are the control instances and p2p-b* are the instances running this PR

Total P2P Transaction Inbound traffic see at least 80% reduction

Memory

[Copilot]

Pull request overview

Reworks transaction broadcast/announcement tracking to reduce redundant P2P traffic and memory usage by centralizing “seen” state and reshaping the sending/fetching flow around per-peer queues and deduplication at ingress.

Changes:

• Replaces per-peer seen sets with a global LRU map keyed by tx hash and per-peer slot-indexed BitSets, plus bounded per-peer send/request queues.
• Refactors senders/processors/fetcher to claim work from PeerTransactionTracker (single-item claiming, size-bounded batching, and retry behavior aligned to ETH/68).
• Updates CLI/config defaults, adds new queue-size option, and refreshes/extends tests for the new APIs.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/PeerTransactionTracker.java	Centralizes seen-state & queues; adds connect/disconnect slotting and block-added handling.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/task/BufferedGetPooledTransactionsFromPeerFetcher.java	Fetch loop now pulls announcements from tracker and retries per ETH/68 rules.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionsMessageSender.java	Moves to inline filtering + size-limited builder and single-item claiming.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/NewPooledTransactionHashesMessageSender.java	Moves to single-item claiming and batch send from tracker-managed queue.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/NewPooledTransactionHashesMessageProcessor.java	Delegates dedupe/enqueue to tracker and schedules fetcher task differently.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/messages/TransactionsMessage.java	Adds SizeLimitedBuilder and tightens constructor visibility.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/RemovalReason.java	Adds stopBroadcasting() to allow immediate queue cleanup on eviction/replacement.
app/src/main/java/org/hyperledger/besu/cli/options/TransactionPoolOptions.java	Renames tracked-seen option to global, adds per-peer send-queue cap.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/transactions/TransactionPoolConfiguration.java	Adds global tracked-seen limit + per-peer send-queue size defaults.
ethereum/eth/build.gradle	Adds commons-collections4 dependency used by new tracker structures.
ethereum/eth/src/test/...	Updates and adds tests to match new tracker/sender/fetcher APIs and behaviors.

[ahamlat]

I see that PeerTransactionTracker.java is becoming a centralized hub, and most of the methods are synchronized (on this). This can have some impact on performance related to the shared lock. After discussing this with @fab-10, and by checking the profiling on lock instances, I'm going to approve for now.
The reason is that I don't see any impact on performance, I don't see more locks than before, and @fab-10 is aware of this strict design. If we observe lock contention under higher peer counts or transaction throughput (e.g., spam events), we should revisit with thread safe data structures and fine grained synchronisation.
The profiling below is during 120 seconds.

[fab-10]

@macfarla I am merging this one, but if you still have observations, please let me know and I will address them in a follow-up PR