SimpleConsumer throws error when committing offsets

[sibiryakov]

2016-05-26 16:29:04,777 DEBUG    kafka.consumer.simple calling _get_message block=True timeout=0.85804104805
2016-05-26 16:29:04,777 DEBUG    kafka.consumer.simple internal queue empty, fetching more messages
2016-05-26 16:29:04,777 DEBUG    kafka.conn      <BrokerConnection host=localhost/::1 port=9092> Request 50: FetchRequest_v0(replica_id=-1, max_wait_time=858, min_bytes=1, topics=[(topic='fro
ntier-done', partitions=[(partition=0, offset=12224, max_bytes=1048576)])])
2016-05-26 16:29:04,825 DEBUG    kafka.consumer  Committing new offsets for frontier-done, partitions [0]
2016-05-26 16:29:04,825 DEBUG    kafka.consumer  Commit offset 12221 in SimpleConsumer: group=general, topic=frontier-done, partition=0
2016-05-26 16:29:04,826 DEBUG    kafka.conn      <BrokerConnection host=localhost/::1 port=9092> Request 51: OffsetCommitRequest_v0(consumer_group='general', topics=[(topic='frontier-done', p
artitions=[(partition=0, offset=12221, metadata=None)])])
2016-05-26 16:29:04,857 ERROR    messagebus.kafka this should not happen - are you threading?
Traceback (most recent call last):
  File "/Users/sibiryakov/src/sh/crawl-frontier/frontera/contrib/messagebus/kafkabus.py", line 59, in get_messages
    timeout=timeout):
  File "/Users/sibiryakov/src/sh/kafka-python/kafka/consumer/simple.py", line 285, in get_messages
    update_offset=False)
  File "/Users/sibiryakov/src/sh/kafka-python/kafka/consumer/simple.py", line 320, in _get_message
    self._fetch()
  File "/Users/sibiryakov/src/sh/kafka-python/kafka/consumer/simple.py", line 379, in _fetch
    fail_on_error=False
  File "/Users/sibiryakov/src/sh/kafka-python/kafka/client.py", line 662, in send_fetch_request
    KafkaProtocol.decode_fetch_response)
  File "/Users/sibiryakov/src/sh/kafka-python/kafka/client.py", line 283, in _send_broker_aware_request
    conn.recv()
  File "/Users/sibiryakov/src/sh/kafka-python/kafka/conn.py", line 436, in recv
    raise Errors.KafkaError('this should not happen - are you threading?')
KafkaError: this should not happen - are you threading?
2016-05-26 16:29:04,857 ERROR    kafka.conn      <BrokerConnection host=localhost/::1 port=9092>: socket disconnected
2016-05-26 16:29:04,859 DEBUG    messagebus.kafka <BrokerConnection host=localhost/::1 port=9092>:4499316496
2016-05-26 16:29:04,859 ERROR    kafka.consumer  FailedPayloadsError saving offsets:

[dpkp]

looks like the same as / similar to #674 ?

[sibiryakov]

@dpkp Well, the stack trace is different. According to my tests it works somehow if auto-commit is disabled and used with dedicated KafkaClient instance.

[dpkp]

Reopen if this still happens on latest release.