light client: rotate unresponsive witness to avoid noisy logs

[martintomazic]

Following https://docs.oasis.io/node/run-your-node/paratime-client-node#stateless-client-node-optional, using oasis grpc provider and latest code on the master (mode: client, local_storage: false)

If I take the latest height, this would be the pattern for every height:

{"caller":"client.go:491","hash":"AC34F6A698353DCDA2D2FF20F619D30825D3E4ACC4BA648C7164AFADE329ABC0","height":29886446,"level":"info","module":"cometbft:base","msg":"Header has already been verified","ts":"2026-05-08T11:14:47.190366232Z"}
{"caller":"client.go:1015","error":"client failed to respond","height":29886447,"level":"info","module":"cometbft:base","msg":"error from light block request from primary, replacing...","primary":{},"ts":"2026-05-08T11:14:47.200692796Z"}
{"caller":"client.go:491","hash":"AC34F6A698353DCDA2D2FF20F619D30825D3E4ACC4BA648C7164AFADE329ABC0","height":29886446,"level":"info","module":"cometbft:base","msg":"Header has already been verified","ts":"2026-05-08T11:14:47.284405731Z"}
{"caller":"client.go:491","hash":"AC34F6A698353DCDA2D2FF20F619D30825D3E4ACC4BA648C7164AFADE329ABC0","height":29886446,"level":"info","module":"cometbft:base","msg":"Header has already been verified","ts":"2026-05-08T11:14:47.284528815Z"}
{"caller":"client.go:491","hash":"AC34F6A698353DCDA2D2FF20F619D30825D3E4ACC4BA648C7164AFADE329ABC0","height":29886446,"level":"info","module":"cometbft:base","msg":"Header has already been verified","ts":"2026-05-08T11:14:47.388494195Z"}
{"caller":"core.go:609","height":29886446,"level":"warn","module":"cometbft/stateless/core","msg":"skipping verification of block results","ts":"2026-05-08T11:14:47.560754931Z"}

If however I start with trust root that is a bit older e.g. oasis network trust:

{"caller":"client.go:568","hash":"E4317969EA82ACED09C6DCE0868269518A4470F61F77A604D6D38029502F497D","height":29764201,"level":"info","module":"cometbft:base","msg":"VerifyHeader","ts":"2026-05-08T11:19:12.00210945Z"}
{"caller":"detector.go:95","err":"client failed to respond","level":"info","module":"cometbft:base","msg":"error in light block request to witness","ts":"2026-05-08T11:19:12.021760619Z"}
{"caller":"client.go:568","hash":"DE190FFE985C5B690EE853A8D6CBF6AF99D2D2BB33D92BA216479E81610728C2","height":29764202,"level":"info","module":"cometbft:base","msg":"VerifyHeader","ts":"2026-05-08T11:19:12.22007535Z"}
{"caller":"detector.go:95","err":"client failed to respond","level":"info","module":"cometbft:base","msg":"error in light block request to witness","ts":"2026-05-08T11:19:12.239326754Z"}
{"caller":"client.go:568","hash":"27C1028A466454994FD41E2134811109B767D20E4A6A08B3FE0BC3216F8542A3","height":29764203,"level":"info","module":"cometbft:base","msg":"VerifyHeader","ts":"2026-05-08T11:19:12.441585294Z"}
{"caller":"detector.go:95","err":"client failed to respond","level":"info","module":"cometbft:base","msg":"error in light block request to witness","ts":"2026-05-08T11:19:12.460740344Z"}
{"caller":"client.go:568","hash":"1AED8CB38F2662ED90791FE8C77F0AFA8BD9787678EF5F4944119A0E16D56970","height":29764204,"level":"info","module":"cometbft:base","msg":"VerifyHeader","ts":"2026-05-08T11:19:12.708120145Z"}
{"caller":"detector.go:95","err":"client failed to respond","level":"info","module":"cometbft:base","msg":"error in light block request to witness","ts":"2026-05-08T11:19:12.727061388Z"}

The node progresses normally but logs error for every version?

Solution

Make it so that if light client witness is not responsive it is replaced.

Currently we maintain witness slots for the cmbft light client, backed by the actual peers. Since detector is not giving feedback to our peermanager, and since detector just logs this error, I am not sure there is a simple fix. This is not critical anyways since if all witnesses fail the verification is rejected, the annoying part is noisy logs.

[martintomazic]

cc @anzoman light client syncing seems to be working, the only thing I spot is that logging is a bit off.