Downstream clients connecting entail huge memory spike and stalls process

[abailly]

When a downstream client connects to Amaru to sync up and the intersection points it provides are "old" or even Genesis, Amaru memory consumption surges (from <300MB to ~1.5GB on preview) and there's a significant amount of time spent finding relevant headers.

The following trace shows that it takes about 50s to find the headers to send to client.

Sep 04 14:34:50 clermont amaru[1329417]: 2025-09-04T12:34:50.137731Z DEBUG amaru::stages::consensus::forward_chain::client_protocol: finding headers between (90333249, 4fb9cff59ef56a3661d45ab7e445593d743241a95f5f9ecbd6f7751ce67472cb) and [(61789671, 846addc03cf9b816cf956daf0e8ae029aa3a466ac3e4ba52b8d526eac7118aab)]
Sep 04 14:35:36 clermont amaru[1329417]: 2025-09-04T12:35:36.657826Z DEBUG amaru::stages::consensus::forward_chain::client_protocol: intersection found: Tip((61789671, 846addc03cf9b816cf956daf0e8ae029aa3a466ac3e4ba52b8d526eac7118aab), 2559288)

Moreover, while the client is synchronising the upstream amaru node is stalled and does not synchronise with its own upstream peer.

While the migration to pallas-network2 (see #420) should make our network stack more robust, it will take some time so we probably want a stop-gap solution before that.

[abailly]

Once the downstream node is synced, the bump recedes a bit:

[etorreborre]

The problematic function is find_headers_between which potentially loads the whole chain from the beginning for a downstream peer that has never been synchronized.

[abailly]

front-loading only the hashes gives the following memory consumption pattern when 3 nodes connect and sync from one upstream node:

Obviously, while it's much better than before it's still far from being scalable.