pubsub: connection pool contention prevents control messages to be sent back

[itizir]

Client

PubSub

Tested/observed with v1.43.0 default receive settings, and v2.3.0 custom receive settings.

Expected behavior

Expect control messages (e.g. Ack, ModAck) to be sent by the client unobstructed by whatever work happens on the data fetching and processing side.

Actual behavior

We have been having similar problems as described in the issues linked below: within the context of a service fetching messages from a number of different subscriptions with various levels of backlog, and processing the events at various rates (some of the streams being effectively stalled), we see delays in fetching pending messages and elevated numbers of duplicate sends due to the client attempts at sending back acks timing out.

We first observed this with the v1 client that has a default receiver settings of 10 goroutines. Switching to the v2 client resolves or mitigates the problem, but only if we don't touch the new default of 1 goroutine.

Investigation

The likely explanation is that when the client tries to send back acks, it fails to grab hold of a connection from the pool if iterator workers are hogging them on slow/blocking pull stream requests. Increasing the connection pool size also resolves the issue.

To put bolts and braces on this, ideally the client would be using separate connection pools for data fetching and control messages. Otherwise might be at least worth better documenting this behaviour and/or making the config check more strict to try and avoid such situations.

Additional context

Likely related in various ways to #1247, #1485, #9727, #10437, #1584.

[hongalex]

Hi, thanks for filing an issue. This is still on my radar.

To put bolts and braces on this, ideally the client would be using separate connection pools for data fetching and control messages. Otherwise might be at least worth better documenting this behaviour and/or making the config check more strict to try and avoid such situations.

I think this is an interesting suggestion, though it would require us to instantiate new clients essentially for doing lease management (what you described as control messages). I would need to do more testing to make sure this is an actual problem (trying to reproduce) and then having some benchmarks to ensure that additional client overhead isn't too bad.

[itizir]

Oh yes for sure: deciding what the right approach is here is certainly not trivial, and you will have the necessary context I do not have... That's why I didn't open a PR with suggested fixes! 😄

No urgency anyway: for now it seems that with the right settings (in terms of fetching goroutines and connection pool), it performs fine.
If you have trouble reproducing, let me know and I'll be happy share more details to hopefully allow you to investigate this further.

[hongalex]

within the context of a service fetching messages from a number of different subscriptions with various levels of backlog

How many subscriptions are you fetching from that made you notice this initially? Basically I think if it's 10 (and each subscription opens 10 streams), then this behavior is inline with what we expect from max streams. If it's only a couple then that would change how we would approach this.

[itizir]

Well, I wouldn't say we noticed it based on a number of subscriptions, and I didn't really test whether there was a threshold before it becomes apparent, etc.

I could try and see what the minimum number of subscriptions is, but as a reference point, the test environment in which I managed to reproduce and diagnose the issue probably has around 40 subscriptions or so. But I think that the crucial ingredient is subscriptions with a large backlog of messages. And I could clearly observe that what mattered was the relative numbers for number of workers and number of connections in the pool...

Let me know if you want me to try with <10 and see if I still observe the same behaviour.

[hongalex]

Yeah this is more or less in line with intended behavior of the client library. 40 subscriptions is a lot, and if you have 10 streams opened for each, this would be 400 streams, which overwhelms the default connection pool size.

I think the fix to this is definitely just improving our documentation (as you suggested) for larger number of streams. Interestingly enough, we used to have documentation around this in the v1 client, but I think this got dropped somehow. I'll open a PR to get this added back.

Lastly, I thought about it some more and I'm not a huge fan of creating different clients for streaming vs acks/modacks. Users pulling from only a few subscriptions won't benefit from this change and would double the overhead for both the clients and the server.

[itizir]

Ah ok, fair enough! And thanks for pointing out the docs! I had missed that... Wonder if some mentions in code comments would help highlight it 🤔
I'll also do some other tests but this probably explains the behaviour indeed.

In v2, the default number of workers was dropped from 10 to 1, so this mitigates the potential for the issue by a good bit. But so the way one is supposed to manage scaling of subscriptions would be to increase the size of the connection pool, or split clients up, is that right?

[itizir]

Yup, can confirm: it's the 40 subscriptions/400 streams limit for 4 connections that we're hitting. In particular in the test environment we managed to reproduce that we had just over 40 subscriptions indeed. Interesting coincidence, hahah!

[hongalex]

In v2, the default number of workers was dropped from 10 to 1, so this mitigates the potential for the issue by a good bit. But so the way one is supposed to manage scaling of subscriptions would be to increase the size of the connection pool, or split clients up, is that right?

That's correct. I will add that if the work is computationally expensive, or all the streams have high utilization, it may even be better to split across multiple machines.

I'll leave this issue open until I address the documentation improvements that were suggested.

[itizir]

Thanks for the tips and clarifications!