Description
- Version: node-redis - 2.5.3 - redis 2.8.12
- Platform: node 5.6.0
Hello. I would like to share an issue that happened to me in production yesterday. I am using node-redis and creating my client as follows:
var client = redis.createClient(REDIS_URL.port, REDIS_URL.hostname, {
auth_pass: (REDIS_URL.auth || '').split(':')[1],
detect_buffers: true,
});
Seems to be very simple. Yesterday, our application stopped working for about an hour before I rebooted our Heroku dynos, and everything went back to normal. Our Redis provider, RedisLabs, was doing some maintenance, and changed the IP address (and resulting machines) behind their hostname. This caused our current redis connection to die.
We received ECONNRESET, then about 15 seconds of ECONNREFUSED as node-redis attempted a reconnect (during which I am postulating that it was delayed on redislab's DNS record TTL). Then, we received a few NOAUTH errors, before going silent for the duration of the remainder of the hour until reboot.
I have spent some time debugging this today, and I would like to share my findings. While the connection was down, and node-redis was attempting to reconnect, commands were piling up in the offline_queue. Upon reconnection, AUTH was not the first command to run, but instead, the queued offline commands ran, thus spitting some NOAUTH errors at us.
I have been able to reproduce this repeatedly today with some EC2 boxes and twiddling some DNS settings. The fix I have settled on is adding enable_offline_queue: false to the createClient options.
If my understanding is correct, and I had not done anything incorrectly (which is quite possible), I find this to be an easy trap for other node-redis users to fall into. I am posting this issue as to notify any other node-redis users who might have faced this, or who may face this in the future.
Thank you for an otherwise wonderful npm package.