confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found

[datteswararao]

Can any one help me with this error?

$sudo -H /usr/local/bin/pip install confluent-kafka
Collecting confluent-kafka
Downloading confluent-kafka-0.9.4.tar.gz (40kB)
100% |████████████████████████████████| 40kB 1.3MB/s
Building wheels for collected packages: confluent-kafka
Running setup.py bdist_wheel for confluent-kafka ... error
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;file='/private/tmp/pip-build-qMzR09/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmp_c59HNpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.11-x86_64-2.7
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
copying confluent_kafka/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.macosx-10.11-x86_64-2.7
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
In file included from confluent_kafka/src/confluent_kafka.c:17:
confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found
#include <librdkafka/rdkafka.h>
^
1 error generated.
error: command 'clang' failed with exit status 1

---

Failed building wheel for confluent-kafka
Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
Running setup.py install for confluent-kafka ... error
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;file='/private/tmp/pip-build-qMzR09/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-VSGjNS-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.11-x86_64-2.7
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
copying confluent_kafka/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.macosx-10.11-x86_64-2.7
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
In file included from confluent_kafka/src/confluent_kafka.c:17:
confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found
#include <librdkafka/rdkafka.h>
^
1 error generated.
error: command 'clang' failed with exit status 1

----------------------------------------

Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;file='/private/tmp/pip-build-qMzR09/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-VSGjNS-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-qMzR09/confluent-kafka/

[datteswararao]

i tried the link - #45
but none of the solutions offered is working. I still getting the same error.

[edenhill]

Make sure to install librdkafka first, through homebrew: homebrew install librdkafka
(edit: removed sudo)

[dgrotto]

@edenhill I am facing the same issue as @datteswararao - brew specifically advises (and denies) running installs with elevated privileges.

Simply running brew install librdkafka does install the lib and rdkafka.h is in the install include dir:

$ ls -b /usr/local/Cellar/librdkafka/0.9.5/include/librdkafka/
rdkafka.h	rdkafkacpp.h

Any additional advice is greatly appreciated!

[ewencp]

@dgrotto When you install librdkafka via brew, it should add some symlinks, e.g. from my current installation:

$ ls -lah /usr/local/include/librdkafka
lrwxr-xr-x 1 ewencp admin 45 May 11 22:11 /usr/local/include/librdkafka -> ../Cellar/librdkafka/0.9.5/include/librdkafka

I think this step is done by the brew link command. This is normally done during installation of the formula, but perhaps something went wrong. You can use the brew link command to force the symlinking.

Note that the suggested setup for homebrew is to have its prefix (/usr/local) owned by your normal user such that you will never need to sudo with brew. However, running various commands with sudo (including installing non-brew code to that prefix) can result in the permissions being changed and breaking brew. You'd need to fix up these permissions (brew doctor might help you with it/tell you it is a problem).

[dgrotto]

@ewencp Thanks for the response and suggestion. I took a look at the symlinks created by the brew install and they look the same as what you have:

$ ls -lah /usr/local/include/librdkafka
lrwxr-xr-x  1 david.grotto  admin    45B Apr 24 13:23 /usr/local/include/librdkafka -> ../Cellar/librdkafka/0.9.5/include/librdkafka

I checked permissions on /usr/local and they look healthy, too. How does setup go about looking for these dependencies? Could it be something simple like a PATH problem? Thanks again!

[dgrotto]

I have mostly solved my problem. Apparently brew does not check to ensure that the search path of /usr/local/include exists. This required that I install Xcode command line tools:

xcode-select --install

This added the appropriate #include search paths.

I still have some permissions problems to clean up after this install - SIP in Sierra doesn't let pip get away with some of its no-nos, but this gets past my biggest hurdle! Thanks for the comments.

@datteswararao Thanks for opening this issue. This may help you, too.

Answer mostly found here: http://stackoverflow.com/questions/23905661/on-mac-g-clang-fails-to-search-usr-local-include-and-usr-local-lib-by-def

[rjurney]

xcode-select --install fixed my OS X install issues. After this command, brew install librdkafka works and then so does pip install confluent-kafka.

[naveenwashere]

brew install librdkafka worked for me! Thanks!

[edenhill]

We're looking to ship binary wheels containing all dependencies for the upcoming release, if you'd like to try it out, install the latest confluent_kafka pre-release from test.pypi.org with:

$ pip install --pre --index-url https://test.pypi.org/simple/ confluent_kafka

Verify that it is working:

$ python -c 'import confluent_kafka ; print confluent_kafka.libversion()'

It should print ('0.11.4-RC...', ...).

Please report both success and failure.

Do note that these wheels contain librdkafka.

Thank you!

[dgrotto]

Python 2.7.13
('0.11.4-RC1', 722121)

Python 3.6.2
('0.11.4-RC1', 722121)

[polimreis]

Since El Capitan 10.11, Apple has created a System Integrity Protection(SIP) that does not even allow a super user to change anything in these folders: /System, /bin, /sbin, or /usr (except /usr /local).

Pre requiremets form MacOS El Capitan or newer:

brew install pkg-config
brew upgrade openssl

echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile
export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include

Because of this, I need clone and compile librdkafka by my own, and changed the prefix fom /usr to /usr/local

git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
./configure --clean
./configure --prefix /usr/local
make
sudo make install

[arae]

Just so you know the binaries for linux on https://test.pypi.org/simple/ seem to be only for python2.7. Any plans to publish python3.6 linux binaries? I'm trying this on amazon lambda.

[edenhill]

@area It should provide binaries for py 2.7, 3.3, 3.4, 3.5, 3.6:
https://test.pypi.org/project/confluent-kafka/0.11.4rc1/#files

[manango]

@edenhill when will the change to bundle librdkafka be released?

[rnpridgeon]

@manango, looks like it was released on April 9th.

https://test.pypi.org/project/confluent-kafka/0.11.4/#history

I'll go ahead and mark this as closed for now, please feel free to reopen should you run into additional issues.

[ulkas]

solved by using newest docker image (python 3.9.2)

FROM python:3
...
RUN pip3 install -r requirements.txt

[manikandan-cv]

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include/
export C_INCLUDE_PATH
LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib
export LIBRARY_PATH

Added them to the ~/.zprofile ( you can add to ~/.bashrc )

now worked

[slominskir]

On MacOS Montery 12.2.1 and Homebrew 3.4.4 I found paths were different, but had same problem fixed with environment variables as suggested above.

> brew install librdkafka
...
> setenv C_INCLUDE_PATH /usr/local/Cellar/librdkafka/1.8.2/include/
> setenv LIBRARY_PATH /usr/local/Cellar/librdkafka/1.8.2/lib
> python setup.py build

Ideally the setup.py would just work without
(1) having to manually install librdkafka or
(2) having to set environment variables

At least it works on MacOS. Try building this project on Windows...

[seunggabi]

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include/ export C_INCLUDE_PATH LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib export LIBRARY_PATH

Added them to the ~/.zprofile ( you can add to ~/.bashrc )

now worked

it is works for me!

[srivkrnt]

https://medium.com/@sri.vkrnt/using-confluent-kafka-on-apple-silicon-5d9150d198a3

You can follow this article to resolve the issues with confluent-kafka.

[yagoazedias]

The solution proposed by @srivkrnt worked to me, I just had to change the version from 1.8.2 to the one that was installed in my machine.

[joshcull3n]

https://medium.com/@sri.vkrnt/using-confluent-kafka-on-apple-silicon-5d9150d198a3

You can follow this article to resolve the issues with confluent-kafka.

bro why isn't this in the official kafka documentation?

[srinidhi567]

Solution provided by @srivkrnt worked for me. I had to change the version to 1.9.2. Before setting up the env variable, just check the version,
/opt/homebrew/Cellar/librdkafka/1.9.2/include

[srivkrnt]

@joshcull3n not sure. Though someone should add it.

[ghost]

For those running versions not listed above, homebrew seems to link headers and libs to an unversioned path:

export C_INCLUDE_PATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

EDIT: PS - thanks to everybody above for help figuring this out!

[minhjh]

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include/ export C_INCLUDE_PATH LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib export LIBRARY_PATH

Added them to the ~/.zprofile ( you can add to ~/.bashrc )

now worked

thanks, it's work

[oliolivia]

In my case, I tried to install confluent-kafka=1.8.2, but the version is not compatible, even with librdkafka installed.
I then tried "pip3 install confluent-kafka" and succeed with a version 2.0.2, this still works after I uninstall librdkafka.