Cancellation of Socket.ConnectAsync intermitently hangs

[ManickaP]

When Socket.ConnectAsync is cancelled it might lead to a deadlock.

The cancellation code for MultipleConnectAsync tries to acquire a lock which is held by the connectAsync continuation (triggered by DoDnsCallback). The continuation then tries to dispose of the cancellation registration which will wait for all the callbacks if they've already been triggered.

This is easily reproducible with our HTTP stress suite (for HTTP 1.1 runs longer than 30 mins).

The biggest impact of this issue is that it prevents us from getting reasonable results from HTTP 1.1 stress pipeline on master. The pipeline gets cancelled on timeout and no results are available (AzDO empty logs).

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[geoffkizer]

In AttemptConnection, when ConnectAsync completes synchronously, we are calling InternalConnectCallback directly. This is bad since the caller of AttemptConnection should be holding the lock here.

We should instead return to the caller and do the call to InternalConnectCallback outside the lock.

Additionally, we should add asserts wherever we take the lock, but especially in the callbacks like InternalConnectCallback, that the lock is not already held by this thread -- basically, Debug.Assert(!Monitor.IsEntered(_lockObject)).

And if a method is assuming the caller is holding the lock, like AttemptConnection is, we should add an assert that the lock is actually held. This is valuable not just to detect bugs, but also to document the expectation of the function.