Timeouts: hop out of queue processing as soon as we know no more are eligible for timeout (#2217)

Since we're in a head-of-queue style situation here, the first non-timeout we hit means we're done - hop out and release the lock. Small docs fix tucked in here because reasons.

Author: NickCraver

Directory.Build.props

@@ -10,7 +10,7 @@
     <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
     <CodeAnalysisRuleset>$(MSBuildThisFileDirectory)Shared.ruleset</CodeAnalysisRuleset>
     <MSBuildWarningsAsMessages>NETSDK1069</MSBuildWarningsAsMessages>
-    <NoWarn>NU5105</NoWarn>
+    <NoWarn>NU5105;NU1507</NoWarn>
     <PackageReleaseNotes>https://stackexchange.github.io/StackExchange.Redis/ReleaseNotes</PackageReleaseNotes>
     <PackageProjectUrl>https://github.com/StackExchange/StackExchange.Redis/</PackageProjectUrl>
     <PackageLicenseExpression>MIT</PackageLicenseExpression>

docs/ReleaseNotes.md

@@ -6,6 +6,7 @@
 - Adds: `IConnectionMultiplexer` now implements `IAsyncDisposable` ([#2161 by kimsey0](https://github.com/StackExchange/StackExchange.Redis/pull/2161))
 - Adds: `IConnectionMultiplexer.GetServers()` to get all `IServer` instances for a multiplexer ([#2203 by NickCraver](https://github.com/StackExchange/StackExchange.Redis/pull/2203))
 - Fix [#2016](https://github.com/StackExchange/StackExchange.Redis/issues/2016): Align server selection with supported commands (e.g. with writable servers) to reduce `Command cannot be issued to a replica` errors ([#2191 by slorello89](https://github.com/StackExchange/StackExchange.Redis/pull/2191))
+- Performance: Optimization around timeout processing to reduce lock contention in the case of many items that haven't yet timed out during a heartbeat ([#2217 by NickCraver](https://github.com/StackExchange/StackExchange.Redis/pull/2217))
 
 ## 2.6.48
 

docs/Timeouts.md

@@ -87,7 +87,7 @@ By default Redis Timeout exception(s) includes useful information, which can hel
 |qu | Queue-Awaiting-Write : {int}|There are x operations currently waiting in queue to write to the redis server.| 
 |qs | Queue-Awaiting-Response : {int}|There are x operations currently awaiting replies from redis server.| 
 |aw | Active-Writer: {bool}|| 
-|bw | Backlog-Writer: {enum} | Possible values are     Inactive, Started, CheckingForWork, CheckingForTimeout, RecordingTimeout, WritingMessage, Flushing, MarkingInactive, RecordingWriteFailure, RecordingFault,SettingIdle,Faulted|
+|bw | Backlog-Writer: {enum} | Possible values are     Inactive, Started, CheckingForWork, CheckingForTimeout, RecordingTimeout, WritingMessage, Flushing, MarkingInactive, RecordingWriteFailure, RecordingFault, SettingIdle, SpinningDown, Faulted|
 |rs | Read-State: {enum}|Possible values are NotStarted, Init, RanToCompletion, Faulted, ReadSync, ReadAsync, UpdateWriteTime, ProcessBuffer, MarkProcessed, TryParseResult, MatchResult, PubSubMessage, PubSubPMessage, Reconfigure, InvokePubSub, DequeueResult, ComputeResult, CompletePendingMessage, NA| 
 |ws | Write-State: {enum}| Possible values are Initializing, Idle, Writing, Flushing, Flushed, NA| 
 |in | Inbound-Bytes : {long}|there are x bytes waiting to be read from the input stream from redis| 

src/StackExchange.Redis/PhysicalConnection.cs

@@ -665,15 +665,24 @@ internal void OnBridgeHeartbeat()
                     {
                         // We only handle async timeouts here, synchronous timeouts are handled upstream.
                         // Those sync timeouts happen in ConnectionMultiplexer.ExecuteSyncImpl() via Monitor.Wait.
-                        if (msg.ResultBoxIsAsync && msg.HasTimedOut(now, timeout, out var elapsed))
+                        if (msg.HasTimedOut(now, timeout, out var elapsed))
                         {
-                            bool haveDeltas = msg.TryGetPhysicalState(out _, out _, out long sentDelta, out var receivedDelta) && sentDelta >= 0 && receivedDelta >= 0;
-                            var timeoutEx = ExceptionFactory.Timeout(multiplexer, haveDeltas
-                                ? $"Timeout awaiting response (outbound={sentDelta >> 10}KiB, inbound={receivedDelta >> 10}KiB, {elapsed}ms elapsed, timeout is {timeout}ms)"
-                                : $"Timeout awaiting response ({elapsed}ms elapsed, timeout is {timeout}ms)", msg, server);
-                            multiplexer.OnMessageFaulted(msg, timeoutEx);
-                            msg.SetExceptionAndComplete(timeoutEx, bridge); // tell the message that it is doomed
-                            multiplexer.OnAsyncTimeout();
+                            if (msg.ResultBoxIsAsync)
+                            {
+                                bool haveDeltas = msg.TryGetPhysicalState(out _, out _, out long sentDelta, out var receivedDelta) && sentDelta >= 0 && receivedDelta >= 0;
+                                var timeoutEx = ExceptionFactory.Timeout(multiplexer, haveDeltas
+                                    ? $"Timeout awaiting response (outbound={sentDelta >> 10}KiB, inbound={receivedDelta >> 10}KiB, {elapsed}ms elapsed, timeout is {timeout}ms)"
+                                    : $"Timeout awaiting response ({elapsed}ms elapsed, timeout is {timeout}ms)", msg, server);
+                                multiplexer.OnMessageFaulted(msg, timeoutEx);
+                                msg.SetExceptionAndComplete(timeoutEx, bridge); // tell the message that it is doomed
+                                multiplexer.OnAsyncTimeout();
+                            }
+                        }
+                        else
+                        {
+                            // This is a head-of-line queue, which means the first thing we hit that *hasn't* timed out means no more will timeout
+                            // and we can stop looping and release the lock early.
+                            break;
                         }
                         // Note: it is important that we **do not** remove the message unless we're tearing down the socket; that
                         // would disrupt the chain for MatchResult; we just preemptively abort the message from the caller's