If a secondary goes down, nodes take a long time to refresh

[dblock]

We had a secondary go down, causing a heartbeat check failure in MongoDB and the following in the MongoDB logs:

Sun Sep 28 13:27:21.905 [rsHealthPoll] couldn't connect to c0.gravity.member4.mongolayer.com:27017: couldn't connect to server c0.gravity.member4.mongolayer.com:27017 

This made our response time 5x slower, taking about 400ms to create a new Moped session. I didn't debug where exactly it was sitting, but I am going to imagine it's not too hard to replicate.

I would expect something a bit more efficient, IMO node rediscovery should not be blocking a request.

[wandenberg]

Hi @dblock could you check if the code on #352 fix this?
If not, let me know.

[dblock]

We'll try to upgrade production to moped HEAD this week.

[johnnyshields]

FYI I believe this is the same bug I reported on Mongoid https://github.com/mongoid/mongoid/issues/3892