Building Apache Kafka

Building Apache Kafka

General Notes:

• When following the steps below, please use a standard permission user unless otherwise specified.

• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Apache Kafka binaries are available and can be downloaded from here. To use these binaries, different Java flavors can be installed on mentioned distributions.

Note: Apache Kafka(v2.8.0) with scala 2.13 was verified at the time of creation of these instructions

Step 1: Install Dependencies

  export SOURCE_ROOT=/<source_root>/

Extract Apache Kafka binary tar to $SOURCE_ROOT and follow steps given below.

• RHEL (7.8, 7.9)

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9)

      sudo yum install -y wget tar git hostname unzip procps snappy bzip2 bzip2-devel curl gcc-c++ make which zlib-devel diffutils

  • Build GCC 7.3.0:

      cd $SOURCE_ROOT
      mkdir gcc
      cd gcc
      wget https://ftpmirror.gnu.org/gcc/gcc-7.3.0/gcc-7.3.0.tar.xz
      tar -xf gcc-7.3.0.tar.xz
      cd gcc-7.3.0
      ./contrib/download_prerequisites
      mkdir objdir
      cd objdir
      ../configure --prefix=/opt/gcc --enable-languages=c,c++ --with-arch=zEC12 --with-long-double-128 \
           --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu                  \
           --enable-threads=posix --with-system-zlib --disable-multilib
      make -j 8
      sudo make install
      sudo ln -sf /opt/gcc/bin/gcc /usr/bin/gcc
      sudo ln -sf /opt/gcc/bin/g++ /usr/bin/g++
      sudo ln -sf /opt/gcc/bin/g++ /usr/bin/c++
      export PATH=/opt/gcc/bin:"$PATH"
      export LD_LIBRARY_PATH=/opt/gcc/lib64:"$LD_LIBRARY_PATH"
      export C_INCLUDE_PATH=/opt/gcc/lib/gcc/s390x-linux-gnu/7.3.0/include
      export CPLUS_INCLUDE_PATH=/opt/gcc/lib/gcc/s390x-linux-gnu/7.3.0/include
      sudo ln -sf /opt/gcc/lib64/libstdc++.so.6.0.24 /lib64/libstdc++.so.6
      sudo ln -sf /opt/gcc/lib64/libatomic.so.1 /lib64/libatomic.so.1

    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9) from here

• RHEL (8.2, 8.3, 8.4)

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9)

      sudo yum install -y wget tar git hostname unzip procps snappy bzip2 bzip2-devel curl gcc-c++ make which zlib-devel diffutils

    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9) from here

• SLES (12 SP5)

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9)

      sudo zypper install -y wget tar unzip snappy-devel libzip2 bzip2 curl gcc7 gcc7-c++ make which zlib-devel git
      sudo ln -sf /usr/bin/gcc-7 /usr/bin/gcc
      sudo ln -sf /usr/bin/g++-7 /usr/bin/g++
      sudo ln -sf /usr/bin/gcc /usr/bin/cc

    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9) from here

• SLES (15 SP2)

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9)

      sudo zypper install -y unzip snappy-devel libzip5 bzip2 curl gcc-c++ make which zlib-devel tar wget git gzip gawk

    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9) from here

• Ubuntu (18.04, 20.04, 21.04)

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9)

      sudo apt-get -y install wget tar hostname unzip zlib1g-dev libbz2-dev liblz4-dev libzstd-dev git make gcc-7 g++-7 curl
      sudo rm -rf /usr/bin/gcc /usr/bin/g++ /usr/bin/cc
      sudo ln -sf /usr/bin/gcc-7 /usr/bin/gcc
      sudo ln -sf /usr/bin/g++-7 /usr/bin/g++
      sudo ln -sf /usr/bin/gcc /usr/bin/cc

    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9) from here.

Step 2: Build and Create rocksdbjni-5.18.4.jar for s390x

Note: Building rocksdbjni require java 8. It can be installed from here.

• Set JAVA_HOME and PATH for JDK8

    export JAVA_HOME=<path to java8>
    export PATH=$JAVA_HOME/bin:$PATH

• Build and Create Rocksdb Jar

  cd $SOURCE_ROOT
  git clone git://github.com/facebook/rocksdb.git
  cd rocksdb
  git checkout v5.18.4
  sed -i '1656s/ARCH/MACHINE/g' Makefile
  PORTABLE=1 make shared_lib
  make rocksdbjava
  

• Replace Rocksdbjni jar

  cp $SOURCE_ROOT/rocksdb/java/target/rocksdbjni-5.18.4-linux64.jar <path to Apache Kafka download directory>/libs/rocksdbjni-5.18.4.jar
  

Step 3: Set environment variables

• Update JAVA_HOME and PATH variable to switch to JDK11

  export JAVA_HOME=<path to java11>
  export PATH=$JAVA_HOME/bin:$PATH

Step 4: Verification

Follow official quickstart guide given here to verify the installation.

References:

http://kafka.apache.org/
https://kafka.apache.org/quickstart