Python 3.10 wheel build for confluent-kafka 1.7.0

[Gyllsdorff]

Description

Python 3.10 have been released but the latest confluent-kafka release 1.7.0 don't have a prebuilt wheel for it.

It is easy to build it from source but it slows down some of the image build procedures quite a bit.

How to reproduce

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion())
• Apache Kafka broker version:
• Client configuration: {...}: None
• Operating system: Linux
• Provide client logs (with 'debug': '..' as necessary): none
• Provide broker log excerpts: none
• Critical issue: No

[grutts]

Is there a plan to build a wheel for 3.10?

[david-walker-gfs]

Interested in this too. Seems 3.9 doesn't yet exist either.

[ahtik]

@david-walker-gfs, 3.9 wheels are available, https://pypi.org/project/confluent-kafka/1.7.0/#files

[johng]

This would be nice

[david-walker-gfs]

@david-walker-gfs, 3.9 wheels are available, https://pypi.org/project/confluent-kafka/1.7.0/#files

Thanks for the link!

[edenhill]

v1.8.2 is now released, with py 3.6-3.9 wheels. We'll look into v3.10 wheels for the next release.

[grutts]

This project releases about 3 times per year. Would you consider releasing a patch (v1.8.3) earlier than the typical release frequency, with 3.10 wheels included?

[edenhill]

The slow release cadence is due to severe CI issues during the latter part of 2021. We're soon moving to another solution which will allow us to release more often.

[grutts]

Good news! Thanks

[wkrt7]

When do you plan to release the wheel for Python 3.10?

[edumorlom]

3.10 would be great

[arroyofnxio]

Hi @edenhill! Is there any way the community can help to prioritize this issue?

[saabeilin]

Upvoting this as well ;-)

[grutts]

Hi @edenhill. Do you have an idea of when the next release, with this change included, will be? The release cadence of the project doesn't seem to have changed. Even a 1.8.3 release which just adds the wheels would be helpful.

[MicaelJarniac]

We're also relying on this. One thing that could help in the meantime is some clear instructions as to how we could build the lib ourselves.

[Lightjohn]

What would be the process to build 3.10 wheels?

[johng]

Any update here? It's been over 5 months since the last release

[AntonOfTheWoods]

While it is definitely annoying they aren't keeping up with the maintainer-recommended versions of Python, simply installing librdkafka-dev (debian derivatives), or the equivalent for your platform seems to do the trick.

[lpsinger]

confluent-kafka-python 1.9.0 is out now, but there are still no wheels for Python 3.10. Python 3.10 has been out for over 8 months. Please build wheels for Python 3.10!

[dnnsthnnr]

I did some digging on my own. Seems like the wheel scripts or for windows depend on cibuildwheel version 1.11.0, which does not support python 3.10 as it is from May 2021.

Is the support for python 3.10 as simple as bumping the cibuildwheel version to at least v2.1.3 where 3.10 was finally supported? Similar to this commit.

I cannot see if there is a sideffect to bumping cibuildwheel version, but I am happy to give a try.

[Lightjohn]

When I tried to build it locally:
In tools/wheels/build-wheels.sh:

• I had to bump cibuildwheel to newer (latest in my case) line 55
• Play a bit with CIBW_BUILD env line 12 (which I added because I only wanted one wheel)
  And it was enough for me.

[AntonOfTheWoods]

Further to my above comment that all that was required was installing librdkafka-dev, 1.9.0 now requires that you get the latest version, which of course is not in any distribution's repos. So you have to either add their repo or manually download and install the package. This is all just way to much hassle when there are equally performant alternatives that Just Work. I'm outta here.

[disrupted]

This is all just way to much hassle when there are equally performant alternatives that Just Work. I'm outta here.

Can you suggest an alternative?

[edenhill]

Python 3.10 wheels are now available on PyPi for Linux, OSX (x64) and Windows

[lpsinger]

Thank you so much!

[andreagrandi]

@edenhill hi Magnus, I'm not sure if I should open a new issue or reopen this one, but I just had the same problem when doing docker-compose up on a project which uses confluent-kafka==2.0.2 from a MBP with M2 CPU and we are using this base image: FROM python:3.10.9-slim-bullseye AS base so Python 3.10.9

I first found this post https://stackoverflow.com/questions/72163907/unable-to-install-confluent-kafka-fatal-error-librdkafka-rdkafka-h-no-such-f and I was able to work around the issue by adding this to my Dockerfile:

RUN apt-get update && \
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && \
  cd /tmp && git clone https://github.com/edenhill/librdkafka && \
  cd librdkafka && git checkout tags/v2.0.2 && \
  ./configure && make && make install && \
  ldconfig &&\
  cd ../ && rm -rf librdkafka

but I do wonder why that library isn't automatically installed. If I remove those steps, when I pip install confluent-kafka inside the Docker image, it tries to rebuild the library and I get this error:

#0 6.567       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.10 -c /tmp/pip-install-7wr9tzff/confluent-kafka_bbefb7599e2f466dafd974ca7ca21b1c/src/confluent_kafka/src/Admin.c -o build/temp.linux-aarch64-cpython-310/tmp/pip-install-7wr9tzff/confluent-kafka_bbefb7599e2f466dafd974ca7ca21b1c/src/confluent_kafka/src/Admin.o
#0 6.567       In file included from /tmp/pip-install-7wr9tzff/confluent-kafka_bbefb7599e2f466dafd974ca7ca21b1c/src/confluent_kafka/src/Admin.c:17:
#0 6.567       /tmp/pip-install-7wr9tzff/confluent-kafka_bbefb7599e2f466dafd974ca7ca21b1c/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
#0 6.567          23 | #include <librdkafka/rdkafka.h>
#0 6.567             |          ^~~~~~~~~~~~~~~~~~~~~~
#0 6.567       compilation terminated.
#0 6.567       error: command '/usr/bin/gcc' failed with exit code 1
#0 6.567       [end of output]

What am I missing? Thanks for your help

[pranavrth]

Hey @andreagrandi,

We don't have linux arm64 wheel as of now. We will provide that in the next release. Please check this issue where I am providing the updates.

[andreagrandi]

We don't have linux arm64 wheel as of now. We will provide that in the next release. Please check #1466 where I am providing the updates.

hi @pranavrth and thanks for your reply!

I do apologise for the maybe dumb question, but I'm a bit confused with architectures and Docker 😅

The existing wheel is valid for M1/M2 CPUs but only if I pip install confluent-kafka directly on my Mac and not from a Dockerfile running on this Mac, is it correct?

If yes, how can our Dockerfile work when building on our CI (we use CircleCI) or AWS? I always assumed those servers where using arm64 CPUs 🤔

Thanks again for your help

[ChillarAnand]

The existing wheel is valid for M1/M2 CPUs but only if I pip install confluent-kafka directly on my Mac and not from a Dockerfile running on this Mac, is it correct?

@andreagrandi Yes. Currently, wheels are available for x86_64 & arm64. But they are not yet available for linux-aarch64.

A docker running on Mac M1(arm64) will be using linux-aarch64. You can run uname -a in the container and see the architecture.

If you want to run it Docker inside Mac M1, you have to build it from source. See this thread on more details. confluentinc/librdkafka#3546 (comment)