Unable to receive messages : Group coordinator not available

[nitinpi]

Description

We were using kafka cluster v 0.10.1 on HDP 2.6.3.2 and were using the confluent.kafka.strongname dotnet connector v0.11.3 and everything was working fine.

Recently upgraded the kafka cluster to v1.1.0 on HDP-2.6.5.3005. We are unable to receive any messages from kafka using the confluent.kafka.strongname connector v0.11.3 or the latest version v1.0-beta2.

Used the standard producer consumer example given on confluent's examples to reproduce the problem.

Also note that we are using this new kafka cluster from our Spark scala code using the spark scala kafka connector and it works without any problems. Problem is only on our .NET side of things consuming messages from kafka.

How to reproduce

My consumer code that repro's the problem is given below. Please note that the cluster named vnq5eu is the one not working and the cluster named van3eu works and is commented out in the code.

public class Program
{
///

/// In this example
/// - offsets are manually committed.
/// - no extra thread is created for the Poll (Consume) loop.
///

public static void Run_Consume(List topics, CancellationToken cancellationToken)
{
var config = new ConsumerConfig
{

		//BootstrapServers = "wn0-van3eu:9092,wn1-van3eu:9092,wn2-van3eu:9092,wn3-van3eu:9092,wn4-van3eu:9092,wn5-van3eu:9092,wn6-van3eu:9092,wn7-van3eu:9092,wn8-van3eu:9092,wn9-van3eu:9092",
		BootstrapServers = "wn0-vnq5eu:9092,wn1-vnq5eu:9092,wn2-vnq5eu:9092,wn3-vnq5eu:9092,wn4-vnq5eu:9092,wn5-vnq5eu:9092,wn6-vnq5eu:9092,wn7-vnq5eu:9092,wn8-vnq5eu:9092,wn9-vnq5eu:9092",
		GroupId = "csharp-consumer",
		EnableAutoCommit = false,
		StatisticsIntervalMs = 5000,
		SessionTimeoutMs = 6000,
		AutoOffsetReset = AutoOffsetResetType.Earliest,
		ApiVersionRequest = true,
		//Debug = "consumer,cgrp,topic,fetch,protocol,broker,generic,feature,all",
		Debug = "all",
		
	};

	const int commitPeriod = 5;

	using (var consumer = new Consumer<Ignore, string>(config))
	{
		// Note: All event handlers are called on the main .Consume thread.

		// Raised when the consumer has been notified of a new assignment set.
		// You can use this event to perform actions such as retrieving offsets
		// from an external source / manually setting start offsets using
		// the Assign method. You can even call Assign with a different set of
		// partitions than those in the assignment. If you do not call Assign
		// in a handler of this event, the consumer will be automatically
		// assigned to the partitions of the assignment set and consumption
		// will start from last committed offsets or in accordance with
		// the auto.offset.reset configuration parameter for partitions where
		// there is no committed offset.
		consumer.OnPartitionsAssigned += (_, partitions)
			=> Console.WriteLine($"Assigned partitions: [{string.Join(", ", partitions)}], member id: {consumer.MemberId}");

		// Raised when the consumer's current assignment set has been revoked.
		consumer.OnPartitionsRevoked += (_, partitions)
			=> Console.WriteLine($"Revoked partitions: [{string.Join(", ", partitions)}]");

		consumer.OnPartitionEOF += (_, tpo)
			=> Console.WriteLine($"Reached end of topic {tpo.Topic} partition {tpo.Partition}, next message will be at offset {tpo.Offset}");

		consumer.OnError += (_, e)
			=> Console.WriteLine($"Error: {e.Reason}");
			
		consumer.OnLog += (_, kafkaLog)
			=> Console.WriteLine($"Kafka log message: {kafkaLog.Message}");

// consumer.OnStatistics += (_, json)
// => Console.WriteLine($"Statistics: {json}");

		consumer.Subscribe(topics);

		while (!cancellationToken.IsCancellationRequested)
		{
			try
			{
				var consumeResult = consumer.Consume(cancellationToken);
				Console.WriteLine($"Topic: {consumeResult.Topic} Partition: {consumeResult.Partition} Offset: {consumeResult.Offset} {consumeResult.Value}");

				if (consumeResult.Offset % commitPeriod == 0)
				{
					// The Commit method sends a "commit offsets" request to the Kafka
					// cluster and synchronously waits for the response. This is very
					// slow compared to the rate at which the consumer is capable of
					// consuming messages. A high performance application will typically
					// commit offsets relatively infrequently and be designed handle
					// duplicate messages in the event of failure.
					var committedOffsets = consumer.Commit(consumeResult);
					Console.WriteLine($"Committed offset: {committedOffsets}");
				}
			}
			catch (ConsumeException e)
			{
				Console.WriteLine($"Consume error: {e.Error}");
			}
		}

		consumer.Close();
	}
}

public static void Main(string[] args)
{
	//string brokerList = "wn0-vnq5eu:9092,wn1-vnq5eu:9092,wn2-vnq5eu:9092,wn3-vnq5eu:9092,wn4-vnq5eu:9092,wn5-vnq5eu:9092,wn6-vnq5eu:9092,wn7-vnq5eu:9092,wn8-vnq5eu:9092,wn9-vnq5eu:9092";
	//var brokerList = "wn0-van3u:9092,wn1-van3u:9092,wn2-van3u:9092,wn3-van3u:9092,wn4-van3u:9092,wn5-van3u:9092,wn6-van3u:9092,wn7-van3u:9092,wn8-van3u:9092,wn9-van3u:9092";

	List<string> topics = new List<string>();
	topics.Add("EOP_SecurityEventLog");

	Console.WriteLine($"Started consumer, Ctrl-C to stop consuming");

	CancellationTokenSource cts = new CancellationTokenSource();
	Console.CancelKeyPress += (_, e) =>
	{
		e.Cancel = true; // prevent the process from terminating.
		cts.Cancel();
	};

	Run_Consume(topics, cts.Token);
}

}

---

Debug logs given below :

Started consumer, Ctrl-C to stop consuming
Kafka log message: [thrd:main]: Group "csharp-consumer" received op SUBSCRIBE (v0) in state wait-broker (join state init, v1 vs 0)
Kafka log message: [thrd:main]: Group "csharp-consumer": subscribe to new subscription of 1 topics (join state init)
Kafka log message: [thrd:main]: Group "csharp-consumer": unsubscribe from current unset subscription of 0 topics (leave group=no, join state init, v1)
Kafka log message: [thrd:main]: Group "csharp-consumer": resetting group leader info: unsubscribe
Kafka log message: [thrd:main]: Group "csharp-consumer" changed join state init -> wait-unassign (v1, state wait-broker)
Kafka log message: [thrd:main]: Group "csharp-consumer": unassign done in state wait-broker (join state wait-unassign): without new assignment: unassign (no previous assignment)
Kafka log message: [thrd:main]: Group "csharp-consumer" changed join state wait-unassign -> init (v1, state wait-broker)
Kafka log message: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 42ms: coordinator query
Kafka log message: [thrd:main]: Group "csharp-consumer": no broker available for coordinator query: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Connecting to ipv4#10.250.26.37:9092 (plaintext) with socket 2172
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Connected to ipv4#10.250.26.37:9092
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Connected (#1)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: Broadcasting state change
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received ApiVersionResponse (v0, 264 bytes, CorrId 1, rtt 0.43ms)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Broker API support:
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey Produce (0) Versions 0..5
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey Fetch (1) Versions 0..7
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey Offset (2) Versions 0..2
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey Metadata (3) Versions 0..5
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey LeaderAndIsr (4) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey StopReplica (5) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey Unknown-6? (6) Versions 0..4
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey Unknown-7? (7) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey OffsetCommit (8) Versions 0..3
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey OffsetFetch (9) Versions 0..3
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey GroupCoordinator (10) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey JoinGroup (11) Versions 0..2
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey Heartbeat (12) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey LeaveGroup (13) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey SyncGroup (14) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DescribeGroups (15) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey ListGroups (16) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey SaslHandshake (17) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey ApiVersion (18) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey CreateTopics (19) Versions 0..2
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DeleteTopics (20) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DeleteRecords (21) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey InitProducerId (22) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey OffsetForLeaderEpoch (23) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey AddPartitionsToTxn (24) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey AddOffsetsToTxn (25) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey EndTxn (26) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey WriteTxnMarkers (27) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey TxnOffsetCommit (28) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DescribeAcls (29) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey CreateAcls (30) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DeleteAcls (31) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DescribeConfigs (32) Versions 0..1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey AlterConfigs (33) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey AlterReplicaLogDirs (34) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DescribeLogDirs (35) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey SaslAuthenticate (36) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey CreatePartitions (37) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey CreateDelegationToken (38) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey RenewDelegationToken (39) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey ExpireDelegationToken (40) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DescribeDelegationToken (41) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: ApiKey DeleteGroups (42) Versions 0..0
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature MsgVer1: Produce (2..2) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature MsgVer1: Fetch (2..2) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature MsgVer1
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature MsgVer2: Produce (3..3) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature MsgVer2: Fetch (4..4) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature MsgVer2
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature ApiVersion: ApiVersion (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature ApiVersion
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerGroupCoordinator: GroupCoordinator (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature BrokerGroupCoordinator
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerBalancedConsumer: GroupCoordinator (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature BrokerBalancedConsumer
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature ThrottleTime: Produce (1..2) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature ThrottleTime: Fetch (1..2) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature ThrottleTime
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature Sasl: JoinGroup (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature Sasl
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature SaslHandshake: SaslHandshake (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature SaslHandshake
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature LZ4: GroupCoordinator (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature LZ4
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature OffsetTime: Offset (1..1) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature OffsetTime
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Feature IdempotentProducer: InitProducerId (0..0) supported by broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Enabling feature IdempotentProducer
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: Broadcasting state change
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Request metadata for brokers only: connected
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 2)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received MetadataResponse (v2, 906 bytes, CorrId 2, rtt 0.71ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: ===== Received metadata: connected =====
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: ClusterId: byVwg5n8TduqpDCoyVyjiw, ControllerId: 1001
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: 10 brokers, 0 topics
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #0/10: wn9-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1001
Kafka log message: [thrd:main]: wn9-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1001: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn9-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1001: Added new broker with NodeId 1001
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #1/10: wn6-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1010
Kafka log message: [thrd:main]: wn6-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1010: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn6-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1010: Added new broker with NodeId 1010
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #2/10: wn5-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1004
Kafka log message: [thrd:wn9-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn9-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1001: Enter main broker thread
Kafka log message: [thrd:main]: wn5-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1004: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn5-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1004: Added new broker with NodeId 1004
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #3/10: wn3-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1007
Kafka log message: [thrd:main]: wn3-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1007: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn3-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1007: Added new broker with NodeId 1007
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #4/10: wn1-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1006
Kafka log message: [thrd:wn5-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn5-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1004: Enter main broker thread
Kafka log message: [thrd:main]: wn1-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1006: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn1-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1006: Added new broker with NodeId 1006
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #5/10: wn4-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1009
Kafka log message: [thrd:main]: wn4-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1009: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn4-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1009: Added new broker with NodeId 1009
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #6/10: wn8-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1003
Kafka log message: [thrd:main]: wn8-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1003: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn8-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1003: Added new broker with NodeId 1003
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #7/10: wn2-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1005
Kafka log message: [thrd:wn1-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn1-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1006: Enter main broker thread
Kafka log message: [thrd:wn8-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn8-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1003: Enter main broker thread
Kafka log message: [thrd:main]: wn2-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1005: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn2-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1005: Added new broker with NodeId 1005
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #8/10: wn7-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1008
Kafka log message: [thrd:main]: wn7-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1008: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn7-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1008: Added new broker with NodeId 1008
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Broker #9/10: wn0-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092 NodeId 1002
Kafka log message: [thrd:main]: wn0-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1002: Enabled low-latency ops queue wake-ups
Kafka log message: [thrd:main]: wn0-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1002: Added new broker with NodeId 1002
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: ClusterId update "" -> "byVwg5n8TduqpDCoyVyjiw"
Kafka log message: [thrd:wn6-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn6-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1010: Enter main broker thread
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: ControllerId update -1 -> 1001
Kafka log message: [thrd:main]: Broadcasting state change
Kafka log message: [thrd:main]: Group "csharp-consumer": no topics in metadata matched subscription
Kafka log message: [thrd:main]: Group "csharp-consumer" management reassigned from broker :0/internal to wn0-vnq5eu:9092/bootstrap
Kafka log message: [thrd:main]: Group "csharp-consumer" management unassigned from broker handle :0/internal
Kafka log message: [thrd:main]: Group "csharp-consumer" management assigned to broker wn0-vnq5eu:9092/bootstrap
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 3)
Kafka log message: [thrd:wn3-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn3-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1007: Enter main broker thread
Kafka log message: [thrd:wn2-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn2-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1005: Enter main broker thread
Kafka log message: [thrd:wn7-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn7-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1008: Enter main broker thread
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 3, rtt 0.70ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:wn4-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn4-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1009: Enter main broker thread
Kafka log message: [thrd:wn0-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:]: wn0-vnq5eu.a1jovikq1ulelazcsaxseaazoh.bx.internal.cloudapp.net:9092/1002: Enter main broker thread
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 4)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 4, rtt 0.63ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 5)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 5, rtt 13.46ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 6)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 6, rtt 1.75ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 7)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 7, rtt 1.89ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 8)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 8, rtt 0.55ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 9)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 9, rtt 1.62ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 10)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 10, rtt 3.40ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 11)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 11, rtt 0.49ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 12)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 12, rtt 0.62ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 13)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 13, rtt 0.58ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 14)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 14, rtt 37.61ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 15)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 15, rtt 0.53ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 16)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 16, rtt 0.93ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 17)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 17, rtt 0.47ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 18)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 18, rtt 0.46ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 19)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 19, rtt 0.74ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 20)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 20, rtt 0.66ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 21)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 21, rtt 0.64ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 22)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 22, rtt 0.98ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 23)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 23, rtt 0.82ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 24)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 24, rtt 0.57ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 25)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 25, rtt 0.77ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 26)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 26, rtt 0.98ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 27)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 27, rtt 0.60ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 28)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 28, rtt 0.51ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 29)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 29, rtt 0.54ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 30)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 30, rtt 15.97ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 31)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 31, rtt 0.76ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 32)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 32, rtt 0.69ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 33)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 33, rtt 9.91ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 34)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 34, rtt 0.62ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 35)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 35, rtt 0.65ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 36)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 36, rtt 0.69ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 37)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 37, rtt 0.55ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 38)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 38, rtt 9.15ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 39)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 39, rtt 9.44ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 40)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 40, rtt 2.10ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 41)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 41, rtt 25.75ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer": querying for coordinator: intervaled in state wait-broker
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Sent GroupCoordinatorRequest (v0, 38 bytes @ 0, CorrId 42)
Kafka log message: [thrd:wn0-vnq5eu:9092/bootstrap]: wn0-vnq5eu:9092/bootstrap: Received GroupCoordinatorResponse (v0, 12 bytes, CorrId 42, rtt 8.16ms)
Kafka log message: [thrd:main]: wn0-vnq5eu:9092/bootstrap: Group "csharp-consumer" GroupCoordinator response error: Broker: Group coordinator not available

Checklist

Please provide the following information:

• Confluent.Kafka nuget version:1.0.0-beta2
• Apache Kafka version: 1.1.0
• Client configuration:
• Operating system:
• Provide logs (with "debug" : "..." as necessary in configuration)
• Provide broker log excerpts
• Critical issue

[nitinpi]

Please advise as we are currently blocked on using this version of kafka cluster with the confluent.kafka.strongname library latest version 1.0.0-beta2.

[nitinpi]

Also wanted to add the following info about running kafka console consumer :

/opt/kafka/bin/kafka-console-consumer.sh --zookeeper zk0.vnq5eu:2181 --topic mytopic --from-beginning - this works
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server [brokerlist] --topic mytopic --from-beginning - this doesn't...it just sits there waiting for messages but does nothing.

[mhowlett]

the client is successfully issuing a find group coordinator request ('GroupCoordinatorRequest'), which is responding with coordinator not available. so this is a broker side problem, but unfortunately I don't have great instincts as to what the problem is likely to be. recommend investigating broker logs for clues. maybe something related to the __consumer_offsets topic. maybe something related to security - do you have any fancy security settings / acls etc?

[nitinpi]

Thanks @mhowlett . I wanted to update this thread to give an update on what fixed our issues. We restarted zookeepers and brokers on all our new clusters and things just started working without any other code changes. Glad to get this resolved. As you said, I think it might be something related to consumer_offsets though when I described that topic using the kafka topics cmdline, the leaders were all the correct ones.