Skip to content

Redis LOADING error should cause different node to be queried during pipeline reads in redis-cluster #1031

@koenoki

Description

@koenoki

When processing pipeline read commands in redis-cluster if LOADING error is encountered we retry to the same node because failedCmds is not updated in pipelineReadCmds() (see https://github.com/go-redis/redis/blob/master/cluster.go#L1332). This means that if there's an extended period of time in which a node responds with LOADING error, which can happen eg after a failover recovery event in which an entire read replica needs to be replicated from the master, no pipeline reads can succeed. We unfortunately experienced this very scenario in production recently.

Is it by design that upon LOADING error is encountered during pipeline reads go-redis doesn't attempt to look for a different node to query, and if so what is/are the rationale? One factor we considered was consistency, but in redis cluster there's already no data consistency guarantee across shards anyways. Or is this an oversight that should be fixed?

As a reference for non-pipeline reads defaultProcess() already handles LOADING error properly (see https://github.com/go-redis/redis/blob/master/cluster.go#L946).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions