-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
We are having an issue with one of our services. It seems the connection multiplexer keeps breaking. It was fine once, we just recylced that pod. But it has kept coming back.
We have 1 cluster (in AWS) with 2 nodes (1 replica).
We keep 1 connection multiplexer alive for the entire pod lifetime, stored in dependency injection.
We have 4 pods (servers) all using the same Redis, but only 1 is getting the timeouts, on every single call.
So I believe the connection multiplexer is breaking, but why could this be? And what is the best path of action to take? Dispose and make a new one? We don't really see this problem on any other services that have their own Redis instances.
StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=1KiB, 5352ms elapsed, timeout is 5000ms), command=INFO, next: INFO, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 1396, cur-in: 0, sync-ops: 5182, async-ops: 7745870, serverEndpoint: redacted.0001.usw2.cache.amazonaws.com:6379, conn-sec: 337520.93, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: redacted-queue-development-849c7d7985-f6649(SE.Redis-v2.8.16.12844), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=16,Max=32767), POOL: (Threads=6,QueuedItems=0,CompletedItems=195372535,Timers=251), v: 2.8.16.12844 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at Redacted.Service.Diagnostics.RedisMonitor.Monitor() in /src/Redacted.Service/Diagnostics/RedisMonitor.cs:line 29
at Redacted.Service.Diagnostics.MonitorBase`1.Monitor(CancellationToken stoppingToken) in /src/Redacted.Service/Diagnostics/MonitorBase.cs:line 58
Redis-Message: INFO
Redis-Message-Next: INFO
Redis-OpsSinceLastHeartbeat: 0
Redis-Queue-Awaiting-Write: 0
Redis-Queue-Awaiting-Response: 1
Redis-Active-Writer: False
Redis-Backlog-Writer: SpinningDown
Redis-Read-State: ReadAsync
Redis-Write-State: Idle
Redis-Inbound-Bytes: 0
Redis-Inbound-Pipe-Bytes: 0
Redis-Outbound-Pipe-Bytes: 0
Redis-Last-Result-Bytes: 1396
Redis-Inbound-Buffer-Bytes: 0
Redis-Sync-Ops: 5182
Redis-Async-Ops: 7745870
Redis-Server-Endpoint: redacted.0001.usw2.cache.amazonaws.com:6379
Redis-Server-Connected-Seconds: 337520.93
Redis-Abort-On-Connect: 1
Redis-Multiplexer-Connects: 1/1/0
Redis-Manager: 10 of 10 available
Redis-Client-Name: redacted-queue-development-849c7d7985-f6649(SE.Redis-v2.8.16.12844)
Redis-ThreadPool-IO-Completion: (Busy=0,Free=1000,Min=1,Max=1000)
Redis-ThreadPool-Workers: (Busy=3,Free=32764,Min=16,Max=32767)
Redis-ThreadPool-Items: (Threads=6,QueuedItems=0,CompletedItems=195372535,Timers=251)
Redis-Busy-Workers: 3
Redis-Version: 2.8.16.12844
redis-command: INFO
request-sent-status: Sent
redis-server: redacted.0001.usw2.cache.amazonaws.com:6379
Commandstatus: Sent
I believe the timeouts aren't valid because they only happen on 1 pod and the Redis is relatively underutilised
