Pip install not linking the librdkafka libraries

[yeshwanth43]

Description

pip3 install of confluent-kafka is not linking with librdkafka libraries

[root@host user1]# ldd /usr/local/lib64/python3.6/site-packages/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so
	linux-vdso.so.1 =>  (0x00007ffef31bc000)
	librdkafka-bf5f31b2.so.1 => /usr/local/lib64/python3.6/site-packages/confluent_kafka/../confluent_kafka.libs/librdkafka-bf5f31b2.so.1 (0x00007f2d9d7d5000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2d9d5ab000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2d9d1dd000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f2d9cedb000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f2d9ccd6000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f2d9cace000)
	/lib64/ld-linux-x86-64.so.2 (0x000055a9a7d3c000)

i don't know why, it is pointing to /usr/local/lib64/python3.6/site-packages/confluent_kafka/../confluent_kafka.libs/librdkafka-bf5f31b2.so.1 , instead it should point to /usr/local/lib/librdkafka.so
i have spent hours trying to figure why pip install is behaving in this way
i created a new VM and tried the same, strangely it's a different behavior

[root@host2 user1]# ldd /usr/local/lib64/python3.6/site-packages/confluent_kafka-1.4.2-py3.6-linux-x86_64.egg/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so 
	linux-vdso.so.1 =>  (0x00007fff0f7f1000)
	librdkafka.so.1 => /lib/librdkafka.so.1 (0x00007fef97789000)
	libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007fef97264000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fef97047000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fef96c79000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fef96977000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fef96759000)
	libssl.so.10 => /lib64/libssl.so.10 (0x00007fef964e7000)
	libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fef96084000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fef95e6d000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fef95c69000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fef95a61000)
	/lib64/ld-linux-x86-64.so.2 (0x000055a640178000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fef9585d000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fef95643000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fef9540c000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fef951be000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fef94ed5000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fef94ca2000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fef94a9d000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fef9488d000)
	libfreebl3.so => /lib64/libfreebl3.so (0x00007fef94689000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fef94485000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fef9425d000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fef93ffb000)

i noticed the directory structure is different from vm1 to vm2
on vm1 it is /usr/local/lib64/python3.6/site-packages/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so
on vm2 it is /usr/local/lib64/python3.6/site-packages/confluent_kafka-1.4.2-py3.6-linux-x86_64.egg/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so

can someone please explain me why this is happening.
i am running everything as root.

following is the environment i am running on

[root@host1 user1]# cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)
[root@host1 user1]# yum list installed |grep librdkafka
librdkafka-devel.x86_64              1.1.0_confluent5.3.3-1.el7 @Confluent.dist 
librdkafka1.x86_64                   1.1.0_confluent5.3.3-1.el7 @Confluent.dist 
[root@host1 user1]# yum list installed |grep sasl
cyrus-sasl.x86_64                    2.1.26-23.el7              @base           
cyrus-sasl-devel.x86_64              2.1.26-23.el7              @base           
cyrus-sasl-gssapi.x86_64             2.1.26-23.el7              @base           
cyrus-sasl-lib.x86_64                2.1.26-23.el7              @base           
cyrus-sasl-plain.x86_64              2.1.26-23.el7              @base           
[root@host1 user1]# yum list installed |grep gss
cyrus-sasl-gssapi.x86_64             2.1.26-23.el7              @base           
gssproxy.x86_64                      0.7.0-28.el7               @base           
python-gssapi.x86_64                 1.2.0-3.el7                @base           
[root@host1 user1]# pip3 show confluent_kafka
Name: confluent-kafka
Version: 1.4.2
Summary: Confluent's Python client for Apache Kafka
Home-page: https://github.com/confluentinc/confluent-kafka-python
Author: Confluent Inc
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib64/python3.6/site-packages
Requires: 
Required-by: 
[root@host1 user1]# 

Sample Code:

#!/usr/bin/python3

from confluent_kafka import Producer
import sys

if __name__ == '__main__':


    topic = 'sm'

    # Producer Configuration
    conf = {'bootstrap.servers': 'host1.example.com:9093',
            'session.timeout.ms': 6000,
            'enable.auto.commit': 'TRUE',
            'security.protocol':'SASL_SSL',
            #'sasl.enabled.mechanisms': 'GSSAPI',
            'sasl.mechanisms': 'GSSAPI',
            'sasl.kerberos.principal': '[email protected]',
            'sasl.kerberos.keytab': '/root/kt/user1.kt',
            'ssl.ca.location': '/root/kt/root.pem',
            'ssl.certificate.location':'/root/kt/root.pem',
            'sasl.kerberos.service.name':'kafka',
            'debug':'all',            
            'default.topic.config': {'auto.offset.reset': 'smallest'}
            }

    # Create Producer instance
    p = Producer(**conf)
    p.produce(topic,'testing message from VM ######## !'.encode('utf-8'))
    p.flush()

Error Log:

Traceback (most recent call last):
  File "confKafka.py", line 3, in <module>
    from confluent_kafka import Producer
  File "/usr/lib64/python3.6/site-packages/confluent_kafka/__init__.py", line 19, in <module>
    from .deserializing_consumer import DeserializingConsumer
  File "/usr/lib64/python3.6/site-packages/confluent_kafka/deserializing_consumer.py", line 19, in <module>
    from confluent_kafka.cimpl import Consumer as _ConsumerImpl
ImportError: librdkafka-bf5f31b2.so.1: cannot open shared object file: No such file or directory

i found #438 #241 and #158 similar issues, but didn't helped to resolve the issue.

Any help is greatly appreciated.

How to reproduce

• Running into issue when trying to install confluent-kafka on a new centos VM, running basic producer code

Checklist

Please provide the following information:

• confluent-kafka-python: 1.4.2 and librdkafka version : 1.1.0_confluent5.3.3-1.el7
• Operating system: CentOS Linux release 7.8.2003 (Core)

[edenhill]

I can't reproduce this on centos:7 docker image.

Steps:

$ yum install python3
$ pip3 install confluent_kafka
$ python3 -c 'import confluent_kafka; print(confluent_kafka.libversion())'
('1.4.2', 17040127)