confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found

[datteswararao]

Can any one help me with this error?

$sudo -H /usr/local/bin/pip install confluent-kafka
Collecting confluent-kafka
Downloading confluent-kafka-0.9.4.tar.gz (40kB)
100% |████████████████████████████████| 40kB 1.3MB/s
Building wheels for collected packages: confluent-kafka
Running setup.py bdist_wheel for confluent-kafka ... error
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;file='/private/tmp/pip-build-qMzR09/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmp_c59HNpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.11-x86_64-2.7
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
copying confluent_kafka/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.macosx-10.11-x86_64-2.7
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
In file included from confluent_kafka/src/confluent_kafka.c:17:
confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found
#include <librdkafka/rdkafka.h>
^
1 error generated.
error: command 'clang' failed with exit status 1

---

Failed building wheel for confluent-kafka
Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
Running setup.py install for confluent-kafka ... error
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;file='/private/tmp/pip-build-qMzR09/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-VSGjNS-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.11-x86_64-2.7
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
copying confluent_kafka/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/kafkatest
creating build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/init.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.11-x86_64-2.7/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.macosx-10.11-x86_64-2.7
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka
creating build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.macosx-10.11-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
In file included from confluent_kafka/src/confluent_kafka.c:17:
confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found
#include <librdkafka/rdkafka.h>
^
1 error generated.
error: command 'clang' failed with exit status 1

----------------------------------------

Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;file='/private/tmp/pip-build-qMzR09/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-VSGjNS-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-qMzR09/confluent-kafka/

[datteswararao]

i tried the link - #45
but none of the solutions offered is working. I still getting the same error.

[edenhill]

Make sure to install librdkafka first, through homebrew: homebrew install librdkafka
(edit: removed sudo)

[dgrotto]

@edenhill I am facing the same issue as @datteswararao - brew specifically advises (and denies) running installs with elevated privileges.

Simply running brew install librdkafka does install the lib and rdkafka.h is in the install include dir:

$ ls -b /usr/local/Cellar/librdkafka/0.9.5/include/librdkafka/
rdkafka.h	rdkafkacpp.h

Any additional advice is greatly appreciated!

[ewencp]

@dgrotto When you install librdkafka via brew, it should add some symlinks, e.g. from my current installation:

$ ls -lah /usr/local/include/librdkafka
lrwxr-xr-x 1 ewencp admin 45 May 11 22:11 /usr/local/include/librdkafka -> ../Cellar/librdkafka/0.9.5/include/librdkafka

I think this step is done by the brew link command. This is normally done during installation of the formula, but perhaps something went wrong. You can use the brew link command to force the symlinking.

Note that the suggested setup for homebrew is to have its prefix (/usr/local) owned by your normal user such that you will never need to sudo with brew. However, running various commands with sudo (including installing non-brew code to that prefix) can result in the permissions being changed and breaking brew. You'd need to fix up these permissions (brew doctor might help you with it/tell you it is a problem).

[dgrotto]

@ewencp Thanks for the response and suggestion. I took a look at the symlinks created by the brew install and they look the same as what you have:

$ ls -lah /usr/local/include/librdkafka
lrwxr-xr-x  1 david.grotto  admin    45B Apr 24 13:23 /usr/local/include/librdkafka -> ../Cellar/librdkafka/0.9.5/include/librdkafka

I checked permissions on /usr/local and they look healthy, too. How does setup go about looking for these dependencies? Could it be something simple like a PATH problem? Thanks again!

[dgrotto]

I have mostly solved my problem. Apparently brew does not check to ensure that the search path of /usr/local/include exists. This required that I install Xcode command line tools:

xcode-select --install

This added the appropriate #include search paths.

I still have some permissions problems to clean up after this install - SIP in Sierra doesn't let pip get away with some of its no-nos, but this gets past my biggest hurdle! Thanks for the comments.

@datteswararao Thanks for opening this issue. This may help you, too.

Answer mostly found here: http://stackoverflow.com/questions/23905661/on-mac-g-clang-fails-to-search-usr-local-include-and-usr-local-lib-by-def

[rjurney]

xcode-select --install fixed my OS X install issues. After this command, brew install librdkafka works and then so does pip install confluent-kafka.

[naveenwashere]

brew install librdkafka worked for me! Thanks!

[edenhill]

We're looking to ship binary wheels containing all dependencies for the upcoming release, if you'd like to try it out, install the latest confluent_kafka pre-release from test.pypi.org with:

$ pip install --pre --index-url https://test.pypi.org/simple/ confluent_kafka

Verify that it is working:

$ python -c 'import confluent_kafka ; print confluent_kafka.libversion()'

It should print ('0.11.4-RC...', ...).

Please report both success and failure.

Do note that these wheels contain librdkafka.

Thank you!

[dgrotto]

Python 2.7.13
('0.11.4-RC1', 722121)

Python 3.6.2
('0.11.4-RC1', 722121)

[polimreis]

Since El Capitan 10.11, Apple has created a System Integrity Protection(SIP) that does not even allow a super user to change anything in these folders: /System, /bin, /sbin, or /usr (except /usr /local).

Pre requiremets form MacOS El Capitan or newer:

brew install pkg-config
brew upgrade openssl

echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile
export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include

Because of this, I need clone and compile librdkafka by my own, and changed the prefix fom /usr to /usr/local

git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
./configure --clean
./configure --prefix /usr/local
make
sudo make install