PEP517

[Alessandro-Barbieri]

Could you move the buildsystem to PEP517?

[edenhill]

Can you explain what problems it would solve?

[Alessandro-Barbieri]

Because I'm packaging for gentoo overlay ::guru and now non PEP517 build are considered legacy
see https://projects.gentoo.org/python/guide/distutils.html#the-pep-517-and-legacy-modes

[edenhill]

It is unlikely that we'll spend time migrating to PEP517 until the current build system is considered a blocker.
But if you want to submit a PR we would definitely take a look and see if it is a feasible way forward.

Do note that our build and release process is a bit complex due to the multi-platform support and bundled C libraries.

[mhowlett]

cross referencing with: #1403

[ekeric13]

So how do we install this on a linux system if we are using python 3.11? In a dockerfile of "FROM python:3.11-slim" I cannot install "confluent-kafka" as it gives me:

"""
#0 9.999 Note: This error originates from the build backend, and is likely not a problem with poetry but with confluent-kafka (2.0.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "confluent-kafka (==2.0.2)"'.
"""

[RajendraP]

Faced same issue for my build as well -
Note: This error originates from the build backend, and is likely not a problem with poetry but with confluent-kafka (1.9.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "confluent-kafka (==1.9.2)"'.

[sergeykad]

So how do we install this on a linux system if we are using python 3.11? In a dockerfile of "FROM python:3.11-slim" I cannot install "confluent-kafka" as it gives me:

""" #0 9.999 Note: This error originates from the build backend, and is likely not a problem with poetry but with confluent-kafka (2.0.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "confluent-kafka (==2.0.2)"'. """

Is there a workaround for this issue or the only alternative is to switch to another Kafka client? Any suggestions on how to solve it? I am using confluent-kafka 2.3.0

[rc-ctream]

We are facing the same issue, is there any working solution how to solve this issue?

[cuongquach]

I have same issue when try to install with Docker python 3.12 . Any active attentions from teams for this issue ?

  at /usr/local/lib/python3.12/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with confluent-kafka (2.1.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "confluent-kafka (==2.1.1)"'.

[vkroy100]

Hi @mhowlett @edenhill @ekeric13 @rc-ctream @RajendraP @Alessandro-Barbieri @sergeykad @cuongquach , I am getting the below error while using python3.10.14-alpine as base image , if anyone else has successfully found a turnaround or fixes then please help here it is urgent.

Error :
68.42 Note: This error originates from the build backend, and is likely not a problem with poetry but with confluent-kafka (2.3.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "confluent-kafka (==2.3.0)"'.

[MedAzizTousli]

Same issue here.

[jankatins]

Just renewing the error message: poetry now errors with

  copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-cpython-313/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.linux-x86_64-cpython-313/tmp/tmpk5x32vb1/confluent-kafka-2.4.0/src/confluent_kafka/src
  gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection -O3 -fPIC -I/tmp/tmphjw89ndb/.venv/include -I/usr/include/python3.13 -c /tmp/tmpk5x32vb1/confluent-kafka-2.4.0/src/confluent_kafka/src/Admin.c -o build/temp.linux-x86_64-cpython-313/tmp/tmpk5x32vb1/confluent-kafka-2.4.0/src/confluent_kafka/src/Admin.o
  In file included from /tmp/tmpk5x32vb1/confluent-kafka-2.4.0/src/confluent_kafka/src/Admin.c:17:
  /tmp/tmpk5x32vb1/confluent-kafka-2.4.0/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
     23 | #include <librdkafka/rdkafka.h>
        |          ^~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/lib64/ccache/gcc' failed with exit code 1
  

  at ~/.local/share/pipx/venvs/poetry/lib/python3.12/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with confluent-kafka (2.4.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "confluent-kafka (==2.4.0)"'.

:-(

[Leandro-bezerra-silva]

Hi there,

I got a similar problem installing confluent-kafka 2.4.0 in public.ecr.aws/lambda/python:3.11@sha256:885e7899bbfb56dbc34401eb07b5bee738f0909032d5342479acf229f196a4a2

creating build/temp.linux-aarch64-cpython-311/tmp/tmp1tq0kchw/confluent-kafka-2.4.0/src/confluent_kafka/src
  gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/tmpnxfgozm6/.venv/include -I/var/lang/include/python3.11 -c /tmp/tmp1tq0kchw/confluent-kafka-2.4.0/src/confluent_kafka/src/Admin.c -o build/temp.linux-aarch64-cpython-311/tmp/tmp1tq0kchw/confluent-kafka-2.4.0/src/confluent_kafka/src/Admin.o
  error: command 'gcc' failed: No such file or directory
  

  at /var/lang/lib/python3.11/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with confluent-kafka (2.4.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "confluent-kafka (==2.4.0)"'.

Thanks