Socket closed unexpectedly

[diegothucao]

Hi mates,
Redis often throw error like above Socket closed unexpectedly, could we turn socket connection off from redis?
Thanks,
Thu

[leibale]

Hi, does your redis server seats behind some kind of a proxy/load balancer? Can you please share some more details on your redis/node configuration?

[diegothucao]

I tried both ways

{
url: setting.ADDRESS_REDIS_URL,
}

And I tried some other way

{
socket: {
port: setting.ADDRESS_REDIS_PORT,
host: setting.ADDRESS_REDIS_HOST,
noDelay: false,
keepAlive: -1,
connectTimeout: 0
},
url: setting.ADDRESS_REDIS_URL,
}

The save/ get value is working but it throw socket issue a lot in logs

Thanks

[guy-microsoft]

Same issue here - it happens in my NodeJS application every 10 minutes, and reconnecting successfully shortly after it.

[leibale]

@diegothucao @guy-microsoft do you use some kind of a proxy/load balancer?

[diegothucao]

no, we do not have it
could we turn socket connect off from redis, for example I just use redis to save and get key, I want to off socket connection.
In previous version, seems it works stablly

[leibale]

What do you mean by "turn socket connect off from redis"?

[diegothucao]

because we are facing issue of socket connection closes, could we do not use socket feature from redis, just normally save and get key... Like we have option like that

[chayim]

@diegothucao Are you saying that you're using a direct connect socket - and that's failing (i.e a unix socket), or that when connecting via a tcp username and host socket things are failing? Can you please bit a bit more explicit?

[phanvanluan94]

I got the same issue.

[diegothucao]

@chayim

{
url: setting.ADDRESS_REDIS_URL,
}

connect like above so it opens say socket close unexpected

[anshul-kai]

I have a multi-server, multi-process application and I'm using redis all over the place. The only instances of redis clients that throw this error are where I'm connecting to the same redis server while using worker threads ie. Connection exists in the main thread and worker thread and the error is constantly thrown every minute. Not an issue when using version 3.

[leibale]

@a-koka can you please share the code that reproduces the error?

[anshul-kai]

@leibale here's a stripped-down version of my code. Hope it's reproducible.

'use strict';

const { Worker, isMainThread } = require('worker_threads');

if (isMainThread) {
  new Worker(__filename);

  require('../common/redis-client').getSubClient().then(redisSubClient => {
    redisSubClient.subscribe('another-channel', data => {
      // do something
    });
  });
} else {
  require('../common/redis-client').getSubClient().then(redisSubClient => {
    redisSubClient.subscribe('channel', data => {
      // do something
    });
  });
}

redis-client.js

'use strict';

const redis = require('redis');

const { REDIS_HOST, REDIS_MAIN_PORT } = process.env;

async function _createClient(port, index) {
  let client;

  client = redis.createClient({ socket: { host: REDIS_HOST, port } });
    await client.connect();

  client.on('error', function(err) {
    console.error('Redis Error:' + err);
  });

  return client;
}

// Each subscribe requires a dedicated redis client.
module.exports.getSubClient = async index => {
  const subClient = await _createClient(REDIS_MAIN_PORT, index);
  return subClient;
};

[brianzable]

This is affecting our application too, very similar client code.

[leibale]

@a-koka I've just run this code, unfortunately I didn't manage to reproduce the crash (I even tried using "CLIENT LIST SKIPME" from redis-cli, it fires "Redis Error:Error: Socket closed unexpectedly" error on the client, but it does not crash the process, and a new socket is created):

const { Worker, isMainThread } = require('worker_threads'),
    redis = require('redis');

(async () => {
    const client = redis.createClient();

    await client.connect();

    client.on('error', err => console.error('Redis Error:' + err));

    if (isMainThread) {
        new Worker(__filename);

        client.subscribe('channel1', message => {
            console.log('channel1', message);
        });
    } else {
        client.subscribe('channel2', message => {
            console.log('channel2', message);
        });
    }
})();

[anshul-kai]

@leibale Would you try publishing data on those two redis channels every 33 ms from another process? If that doesn't help, I am happy to hop on a call, share my screen and capture debugging data.

[leibale]

@a-koka This didn't do it either :(

const { Worker, isMainThread } = require('worker_threads'),
    redis = require('redis');

async function client() {
    const client = redis.createClient();

    client.on('error', err => console.error('Redis Error', err));

    await client.connect();

    return client;
}

async function run(subscribeTo, publishTo) {
    const [subscriber, publisher] = await Promise.all([
        client(),
        client()
    ]);

    await subscriber.subscribe(subscribeTo, message => {
        console.log(subscribeTo, message);
    });

    setInterval(async () => {
        try {
            await publisher.publish(publishTo, 'message');
        } catch (err) {
            console.error('Publish Error', err);
        }
    }, 10);
}

(async () => {
    if (isMainThread) {
        new Worker(__filename);
        await run('channel1', 'channel2');
    } else {
        await run('channel2', 'channel1');
    }
})();

I'll be on this zoom call for the next 30 minutes, if you can, please join and help me debug it

[anshul-kai]

@leibale I briefly connected to let you know that I have an upcoming meeting. I can join using the same link in roughly one hour from now at 1:30 PM Pacific.

[leibale]

@a-koka Sure, take your time :)

Edit: apparently zoom closes idle meetings automatically... please ping me when you are ready

[anshul-kai]

@leibale I'm ready!

[leibale]

@a-koka https://redis.zoom.us/j/94282790711?pwd=OTZIVFVjREdodUd5cnMzN2ZLSHZiQT09

[anshul-kai]

Great chatting with you @leibale and thanks for your help. Below is a summary of the issue and how to fix it.

The issue only happens when subscribing to a channel that receives a lot of data. In my case, this was 100 MB every second.

When this happens, redis closes the underlying connection with the following error:
redis replication scheduled to be closed ASAP for overcoming of output buffer limits.

In node-redis version 3, the connection is closed but no errors are logged. There is an auto-reconnect due to which the issue is never noticed.

In node-redis version 4, the connection is closed and errors logged.

The fix is as simple as increasing the client-output-buffer-limit on redis. I've set it to unlimited as values up to 512 MB didn't work for me.
redis-cli -p 6379 config set client-output-buffer-limit "pubsub 0 0 60"

You might need to increase the maxmemory too, I did not need this.
redis-cli -p 6379 config set maxmemory "1gb"

Persist the configuration to file if needed.
redis-cli -p 6379 config rewrite

[diegothucao]

Thanks @leibale

[diegothucao]

I checked but not see any new version, it was updated officially ?

[leibale]

@diegothucao 4.1.0 will be released in a few days

[bchr02]

I too am experiencing this issue, so I am looking forward to 4.1.0 getting released.

[h2mpeg]

@leibale i have a same problem, i was update node-redis to 4.1.0

`var redis = require("redis");
const fs = require('fs');

const redisClient = redis.createClient({
url: rediss://api3-redis-master.api3-pro.svc.cluster.local:6379,
socket: {
tls: true,
rejectUnauthorized: false,
ca: fs.readFileSync('ks8-ca-cert.crt', encoding='ascii'),
servername: "api3-redis-master.api3-pro.svc.cluster.local"
},
legacyMode: true
});

redisClient.on('error', function(err){
console.log('Redis Client Error', err);
});
redisClient.on('connect', () => {
console.log(Redis connected);
});

(async () => {
// Connect to redis server
await redisClient.connect();
redisClient.get('test_tls', function(err, result){
console.log(err, result);
});
})();`

After 30 minutes when starting nodejs start, redis gives error and reconnects itself

SocketClosedUnexpectedlyError: Socket closed unexpectedly
at TLSSocket. (/data/www/public_html/cloud_api3/node_modules/@redis/client/dist/lib/client/socket.js:156:118)
at Object.onceWrapper (node:events:646:26)
at TLSSocket.emit (node:events:538:35)
at node:net:687:12
at TCP.done (node:_tls_wrap:580:7)

[KunalBurangi]

@leibale , Any update on why the Socket closes unexpectedly ??

[tuanpt-0634]

You should check for client timeout in redis server config => https://redis.io/docs/reference/clients/#client-timeouts

[teamspatialtech]

Great chatting with you @leibale and thanks for your help. Below is a summary of the issue and how to fix it.

The issue only happens when subscribing to a channel that receives a lot of data. In my case, this was 100 MB every second.

When this happens, redis closes the underlying connection with the following error: redis replication scheduled to be closed ASAP for overcoming of output buffer limits.

In node-redis version 3, the connection is closed but no errors are logged. There is an auto-reconnect due to which the issue is never noticed.

In node-redis version 4, the connection is closed and errors logged but there is no auto-reconnect (or re-subscribe).

The fix is as simple as increasing the client-output-buffer-limit on redis. I've set it to unlimited as values up to 512 MB didn't work for me. redis-cli -p 6379 config set client-output-buffer-limit "pubsub 0 0 60"

You might need to increase the maxmemory too, I did not need this. redis-cli -p 6379 config set maxmemory "1gb"

Persist the configuration to file if needed. redis-cli -p 6379 config rewrite

Not work for me i set the Limit to 5GB still get same error

client error SocketClosedUnexpectedlyError: Socket closed unexpectedly
at Socket. (/node_modules/@redis/client/dist/lib/client/socket.js:156:118)

any help @leibale ?
in my case node, js Redis is 4.1.0 and Redis Server is 3.2 on Redis to go

Downgrading the nodejs redis work fine

[thanhdkn]

@leibale
any update for this issue?
Using v4.1.0 I got below error for every 390 seconds

SocketClosedUnexpectedlyError: Socket closed unexpectedly
    at Socket.<anonymous> (/usr/share/nginx/html/node_modules/@redis/client/dist/lib/client/socket.js:156:118)
    at Object.onceWrapper (node:events:642:26)
    at Socket.emit (node:events:527:28)
    at Socket.emit (node:domain:475:12)
    at TCP.<anonymous> (node:net:715:12)

My source code is below:

    const redisClient = createClient({
      url: 'redis://' + REDIS_HOST + ':' + REDIS_PORT
    });
    redisClient.on("error", function (err) {
      console.log(err);
    });
    await redisClient.connect();

[burakkarakus]

I have the same problem, and the only workaround is to downgrade it to 3.1.2...

[mahfuz67]

downgrading redis?

[dwanderton]

+1

[hashhesh]

what about heroku hoppy tier plan since config command is not available then, what's the workarounds? @a-koka

[anshul-kai]

Can you change the underlying config file? I'm not too familiar with heroku but the underlying redis configuration needs to be updated which can be done via CLI or config file.

[hashhesh]

hi @a-koka thanks a million for your reply. unfortunately i can't do that with hoppy tier, but i my issue is solved once i use redis cloud free plan no it works fine.
Thanks again man

[leibale]

@thanhdkn sorry for the huge delay... if the error is happening every X time, it's probably due to a proxy/load balancer in between redis and your server that does not honor TCP keep alive (such as redis on azure, which will close idle connections after 10m, see here). You can use pingInterval to solve it, it'll send PING on interval to make sure the connection will not be dropped.

[thanhdkn]

I am not sure that error still happens now. I changed to use Redis on GCP, do not see the error.

…

On Sun, 22 Jan 2023 at 01:21 Leibale Eidelman ***@***.***> wrote: @thanhdkn <https://github.com/thanhdkn> sorry for the huge delay... if the error is happening every X time, it's probably due to a proxy/load balancer in between redis and your server that does not honor TCP keep alive (such as redis on azure, which will close idle connections after 10m, see here <https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-best-practices-connection#idle-timeout>). You can use pingInterval to solve it, it'll send PING on interval to make sure the connection will not be dropped. — Reply to this email directly, view it on GitHub <#2058 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFAAM2SIJCXZPEZRNJNJY3WTQSL5ANCNFSM5R2I4HJA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[leibale]

Unlike redis on azure, redis on GCP does honor TCP keep alive (as it should), so you actually don't need it :)

[Max-SDSharp]

I solved the problem with pingInterval:

const redisClient = createClient({
url: process.env.REDIS_URL,
pingInterval: 10000,
});

[geoshak]

@msalvatti that solved for me too, thanks!!!