Default Cassandra replication factor considered harmful

[jtlisi]

cortex/pkg/chunk/cassandra/storage_client.go

Line 60 in 9a9648f

f.IntVar(&cfg.ReplicationFactor, "cassandra.replication-factor", 1, "Replication factor to use in Cassandra.")

Cortex will initialize a keyspace with a replication factor of 1 by default. Even if this value is changed later, it will not alter the Keyspace with the new value. This is not ideal and unless a user carefully ensures this value is set when bootstrapping cortex, they could end up with a cassandra backend that is not fault tolerant. Ideally, this value should default to 3, since this allows for a Quorum to be reached without any particular Cassandra node.

[pracucci]

Cassandra backend storage is still marked experimental. We can introduce a config breaking change here, moving from a default of 1 to a default of 3. What do you think?

[jtlisi]

I think that makes sense.