log_source_ips_enabled breaks distributor

[zdykstra]

When log_source_ips_enabled: true is set in the server: configuration block, remote Prometheus instances are unable to push their metrics to Cortex.

To Reproduce
Steps to reproduce the behavior:

1. Use the configuration file at https://cortexmetrics.io/docs/configuration/single-process-config/ , add log_source_ips_enabled: true to the server: configuration section
2. Start a single process with Cortex v1.4.0
3. Point a load balancer to the single process listen IP / port 9009. In this case, HAProxy 2.1 is being used. The configuration file can be provided.
4. Configure a remote_write block in a test Prometheus installation, send metrics to http://loadbalancer:9009/api/v1/push

Expected behavior
Metrics will ingest normally.

Environment:

• Infrastructure: Deployed as a docker container
• Deployment tool: Marathon on top of Mesos

Storage Engine

• Blocks
• Chunks

Additional Context
All metric pushes from Prometheus result in this error being repeatedly logged by the single process Cortex instance.

Cortex log entry

level=warn ts=2020-10-06T16:27:56.886050773Z caller=logging.go:71 traceID=7d0aee8362015afb sourceIPs="10.30.127.101, 1.2.3.4" msg="POST /api/v1/push (500) 1.058849ms Response: \"rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 3818; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.21.0; X-Forwarded-For: 10.30.127.101; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: testing; "

Prometheus log entry

ts=2020-10-06T16:14:27.816Z caller=dedupe.go:112 component=remote level=warn remote_name=2303eb url=http://10.30.56.58:9009/api/v1/push msg="Failed to send batch, retrying" err="server returned HTTP status 500 Internal Server Error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR"

[pracucci]

@MichelHollands Could you investigate it, since you worked on that, please?