HTTP handler for ingester shutdown (#1746)

* HTTP handler for ingester shutdown

Signed-off-by: Ganesh Vernekar <[email protected]>

* Add /shutdown endpoint

Signed-off-by: Ganesh Vernekar <[email protected]>

* Avoid panic on multiple shutdown calls to ingester

Signed-off-by: Ganesh Vernekar <[email protected]>

* Add entry in CHANGELOG and docs/apis.md

Signed-off-by: Ganesh Vernekar <[email protected]>

* Fix review comments

Signed-off-by: Ganesh Vernekar <[email protected]>

Author: codesome

CHANGELOG.md

@@ -31,6 +31,7 @@ This release adds support for Redis as an alternative to Memcached, and also inc
 * [CHANGE] In table-manager, default DynamoDB capacity was reduced from 3,000 units to 1,000 units. We recommend you do not run with the defaults: find out what figures are needed for your environment and set that via `-dynamodb.periodic-table.write-throughput` and `-dynamodb.chunk-table.write-throughput`.
 * [FEATURE] Add Redis support for caching #1612
 * [FEATURE] Allow spreading chunk writes across multiple S3 buckets	#1625
+* [FEATURE] Added `/shutdown` endpoint for ingester to shutdown all operations of the ingester. #1746
 * [ENHANCEMENT] Upgraded Prometheus to 2.12.0 and Alertmanager to 0.19.0. #1597
 * [ENHANCEMENT] Cortex is now built with Go 1.13 #1675, #1676, #1679
 * [ENHANCEMENT] Many optimisations, mostly impacting ingester and querier: #1574, #1624, #1638, #1644, #1649, #1654, #1702
@@ -67,4 +68,4 @@ This release has several exciting features, the most notable of them being setti
   * `ha-tracker.cluster` is now `distributor.ha-tracker.cluster`
 * [FEATURE] You can specify "heap ballast" to reduce Go GC Churn #1489
 * [BUGFIX] HA Tracker no longer always makes a request to Consul/Etcd when a request is not from the active replica #1516
-* [BUGFIX] Queries are now correctly cancelled by the query-frontend #1508
+* [BUGFIX] Queries are now correctly cancelled by the query-frontend #1508

docs/apis.md

@@ -127,3 +127,10 @@ Error Response: BadRequest(400)
 These API endpoints will disable/enable the current Rule and Alertmanager configuration for a tenant.
 
 Note that setting a new config will effectively "re-enable" the Rules and Alertmanager configuration for a tenant.
+
+#### Ingester Shutdown
+
+`POST /shutdown` - Shutdown all operations of an ingester. Shutdown operations performed are similar to when an ingester is gracefully shutting down, including flushing of chunks if no other ingester is in `PENDING` state. Ingester does not terminate after calling this endpoint.
+
+- Normal Response Codes: NoContent(204)
+- Error Response Codes: Unauthorized(401)

pkg/cortex/modules.go

@@ -253,6 +253,7 @@ func (t *Cortex) initIngester(cfg *Config) (err error) {
 	grpc_health_v1.RegisterHealthServer(t.server.GRPC, t.ingester)
 	t.server.HTTP.Path("/ready").Handler(http.HandlerFunc(t.ingester.ReadinessHandler))
 	t.server.HTTP.Path("/flush").Handler(http.HandlerFunc(t.ingester.FlushHandler))
+	t.server.HTTP.Path("/shutdown").Handler(http.HandlerFunc(t.ingester.ShutdownHandler))
 	return
 }
 

pkg/ingester/ingester.go

@@ -254,12 +254,26 @@ func (i *Ingester) loop() {
 
 // Shutdown beings the process to stop this ingester.
 func (i *Ingester) Shutdown() {
-	// First wait for our flush loop to stop.
-	close(i.quit)
-	i.done.Wait()
+	select {
+	case <-i.quit:
+		// Ingester was already shutdown.
+		return
+	default:
+		// First wait for our flush loop to stop.
+		close(i.quit)
+		i.done.Wait()
+
+		// Next initiate our graceful exit from the ring.
+		i.lifecycler.Shutdown()
+	}
+}
 
-	// Next initiate our graceful exit from the ring.
-	i.lifecycler.Shutdown()
+// ShutdownHandler triggers the following set of operations in order:
+//     * Change the state of ring to stop accepting writes.
+//     * Flush all the chunks.
+func (i *Ingester) ShutdownHandler(w http.ResponseWriter, r *http.Request) {
+	i.Shutdown()
+	w.WriteHeader(http.StatusNoContent)
 }
 
 // StopIncomingRequests is called during the shutdown process.