Skip to content

Conversation

pracucci
Copy link
Contributor

@pracucci pracucci commented Feb 18, 2020

What this PR does:
Today @pstibrany was working on #2088 and while testing it in a dev cluster we realized the read path wasn't working despite all unit tests and integration tests passed. The issue was caused by the query-frontend --> querier worker which is currently not tested by integration tests. This PR adds such test.

To prove this test would have spotted the issue in #2088:

$ hub pr checkout 2088
$ git merge --squash add-query-frontend-integration-test

# Manually edit modules.go to remove the quick fix

$ git diff
diff --git a/pkg/cortex/modules.go b/pkg/cortex/modules.go
index 81c715036..e1aa7c695 100644
--- a/pkg/cortex/modules.go
+++ b/pkg/cortex/modules.go
@@ -260,7 +260,7 @@ func (t *Cortex) initQuerier(cfg *Config) (err error) {
        api.Register(promRouter)

        subrouter := t.server.HTTP.PathPrefix("/api/prom").Subrouter()
-       subrouter.PathPrefix("/api/v1").Handler(fakeRemoteAddr(t.httpAuthMiddleware.Wrap(promRouter)))
+       subrouter.PathPrefix("/api/v1").Handler(t.httpAuthMiddleware.Wrap(promRouter))
        subrouter.Path("/read").Handler(t.httpAuthMiddleware.Wrap(querier.RemoteReadHandler(queryable)))
        subrouter.Path("/validate_expr").Handler(t.httpAuthMiddleware.Wrap(http.HandlerFunc(t.distributor.ValidateExprHandler)))
        subrouter.Path("/chunks").Handler(t.httpAuthMiddleware.Wrap(querier.ChunksHandler(queryable)))

$ make ./cmd/cortex/.uptodate
$ go test -timeout 30s github.com/cortexproject/cortex/integration -run ^(TestQueryFrontendWithChunksStorage)$ -v -count=1

=== RUN   TestQueryFrontendWithChunksStorage
Starting consul
consul: ==> Starting Consul agent...
consul: ==> Consul agent running!
consul: Version: 'v0.9.4'
consul: Node ID: '554902f8-bd7c-d7f1-81c5-be8d2db844e3'
consul: Node name: 'consul'
consul: Datacenter: 'dc1' (Segment: '<all>')
consul: Server: true (Bootstrap: false)
consul: Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
consul: ==> Log data will now stream in as it occurs:
Ports for container: e2e-cortex-test-consul Mapping: map[8500:32856]
Starting dynamodb
Ports for container: e2e-cortex-test-dynamodb Mapping: map[8000:32857]
dynamodb: Initializing DynamoDB Local with the following configuration:
dynamodb: Port:	8000
dynamodb: InMemory:	true
dynamodb: DbPath:	null
dynamodb: SharedDb:	true
dynamodb: shouldDelayTransientStatuses:	false
dynamodb: CorsParams:	*
Starting table-manager
Ports for container: e2e-cortex-test-table-manager Mapping: map[80:32858]
Starting query-frontend
query-frontend: level=info ts=2020-02-18T16:29:31.690960512Z caller=cortex.go:252 msg=initialising module=runtime-config
query-frontend: level=info ts=2020-02-18T16:29:31.691092144Z caller=manager.go:68 msg="runtime config disabled: file not specified"
query-frontend: level=info ts=2020-02-18T16:29:31.691116819Z caller=cortex.go:252 msg=initialising module=server
query-frontend: level=info ts=2020-02-18T16:29:31.692799198Z caller=server.go:147 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
query-frontend: level=info ts=2020-02-18T16:29:31.693291284Z caller=cortex.go:252 msg=initialising module=overrides
query-frontend: level=info ts=2020-02-18T16:29:31.693384803Z caller=cortex.go:252 msg=initialising module=query-frontend
query-frontend: level=info ts=2020-02-18T16:29:31.69341854Z caller=main.go:100 msg="Starting Cortex" version="(version=, branch=, revision=)"
Ports for container: e2e-cortex-test-query-frontend Mapping: map[80:32860 9095:32859]
Starting distributor
Ports for container: e2e-cortex-test-distributor Mapping: map[80:32861]
Starting querier
consul: ==> Newer Consul version available: 1.7.0 (currently running: 0.9.4)
Ports for container: e2e-cortex-test-querier Mapping: map[80:32862]
Starting ingester
querier: level=info ts=2020-02-18T16:29:33.868435298Z caller=cortex.go:252 msg=initialising module=memberlist-kv
querier: level=info ts=2020-02-18T16:29:33.868492877Z caller=cortex.go:252 msg=initialising module=server
querier: level=info ts=2020-02-18T16:29:33.868776323Z caller=server.go:147 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
querier: level=info ts=2020-02-18T16:29:33.868900406Z caller=cortex.go:252 msg=initialising module=runtime-config
querier: level=info ts=2020-02-18T16:29:33.86892777Z caller=manager.go:68 msg="runtime config disabled: file not specified"
querier: level=info ts=2020-02-18T16:29:33.868940132Z caller=cortex.go:252 msg=initialising module=overrides
querier: level=info ts=2020-02-18T16:29:33.868950255Z caller=cortex.go:252 msg=initialising module=store
querier: level=info ts=2020-02-18T16:29:33.903905675Z caller=cortex.go:252 msg=initialising module=ring
querier: level=info ts=2020-02-18T16:29:33.904204857Z caller=cortex.go:252 msg=initialising module=distributor
querier: level=info ts=2020-02-18T16:29:33.908408591Z caller=cortex.go:252 msg=initialising module=querier-chunk-store
querier: level=info ts=2020-02-18T16:29:33.908513176Z caller=cortex.go:252 msg=initialising module=querier
querier: level=info ts=2020-02-18T16:29:33.910261023Z caller=main.go:100 msg="Starting Cortex" version="(version=, branch=, revision=)"
querier: level=info ts=2020-02-18T16:29:33.939840017Z caller=client.go:212 msg="value is nil" key=collectors/ring index=1
Ports for container: e2e-cortex-test-ingester Mapping: map[80:32863]
Killing ingester
ingester: level=warn ts=2020-02-18T16:29:35.142926281Z caller=transfer.go:422 msg="transfer attempt failed" err="cannot find ingester to transfer chunks to: no pending ingesters" attempt=1 max_retries=10
Killing querier
querier: level=info ts=2020-02-18T16:29:35.699513087Z caller=signals.go:55 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
querier: level=info ts=2020-02-18T16:29:35.700223812Z caller=cortex.go:278 msg=stopping module=querier
querier: level=error ts=2020-02-18T16:29:35.700294643Z caller=worker.go:123 msg="error from DNS watcher" err="watcher has been closed"
querier: level=error ts=2020-02-18T16:29:35.700322256Z caller=worker.go:177 msg="error processing requests" err="rpc error: code = Canceled desc = context canceled"
querier: level=info ts=2020-02-18T16:29:35.700364803Z caller=cortex.go:278 msg=stopping module=querier-chunk-store
querier: level=info ts=2020-02-18T16:29:35.700377297Z caller=cortex.go:278 msg=stopping module=store
querier: level=info ts=2020-02-18T16:29:35.700402641Z caller=cortex.go:278 msg=stopping module=distributor
querier: level=info ts=2020-02-18T16:29:35.700422988Z caller=cortex.go:278 msg=stopping module=overrides
querier: level=info ts=2020-02-18T16:29:35.700505124Z caller=cortex.go:278 msg=stopping module=ring
querier: level=info ts=2020-02-18T16:29:35.700514359Z caller=cortex.go:278 msg=stopping module=memberlist-kv
querier: level=info ts=2020-02-18T16:29:35.700524004Z caller=cortex.go:278 msg=stopping module=runtime-config
querier: level=info ts=2020-02-18T16:29:35.700532689Z caller=cortex.go:278 msg=stopping module=server
Killing distributor
Killing query-frontend
query-frontend: level=info ts=2020-02-18T16:29:36.726489135Z caller=signals.go:55 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
query-frontend: level=info ts=2020-02-18T16:29:36.726605981Z caller=cortex.go:278 msg=stopping module=query-frontend
query-frontend: level=info ts=2020-02-18T16:29:36.726633893Z caller=cortex.go:278 msg=stopping module=overrides
query-frontend: level=info ts=2020-02-18T16:29:36.726649405Z caller=cortex.go:278 msg=stopping module=runtime-config
query-frontend: level=info ts=2020-02-18T16:29:36.726665066Z caller=cortex.go:278 msg=stopping module=server
Killing table-manager
Killing dynamodb
Killing consul
--- FAIL: TestQueryFrontendWithChunksStorage (17.24s)
    /workspace/src/github.com/cortexproject/cortex/integration/query_frontend_test.go:75: 
        	Error Trace:	query_frontend_test.go:75
        	            				query_frontend_test.go:24
        	Error:      	Received unexpected error:
        	            	execution: missing port in address
        	Test:       	TestQueryFrontendWithChunksStorage
FAIL
FAIL	github.com/cortexproject/cortex/integration	17.361s
FAIL
Error: Tests failed.

Which issue(s) this PR fixes:
N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Contributor

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this Marco!

Copy link
Contributor

@jtlisi jtlisi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pracucci pracucci merged commit 5f7340c into cortexproject:master Feb 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants