Building Apache Kafka

Building Apache Kafka

ATTENTION!!! Apache Kafka uses Log4j. Please see details here, for the updates on security vulnerabilities.

The instructions provided below specify the steps to build Apache Kafka 3.0.0 on Linux on IBM Z for following distributions:

• RHEL (7.8, 7.9, 8.2, 8.4)
• SLES (12 SP5, 15 SP2, 15 SP3)
• Ubuntu (18.04, 20.04, 21.04, 21.10)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.

• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Apache Kafka

Step 1: Build using script

If you want to build Apache Kafka using manual steps, go to STEP 2.

Use the following commands to build Apache Kafka using the build script. Please make sure you have wget installed.

• With Eclipse Adoptium Temurin Runtime (previously known as AdoptOpenJDK hotspot)

  wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ApacheKafka/3.0.0/build_kafka_AdoptiumTemurin.sh
  
  # Build Apache Kafka
  bash build_kafka_AdoptiumTemurin.sh

  Alternatively, you can use IBM Semeru Runtime (previously known as AdoptOpenJDK openj9) Java

  wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ApacheKafka/3.0.0/build_kafka_IBMSemeru.sh
  
  # Build Apache Kafka
  bash build_kafka_IBMSemeru.sh

If the build completes successfully, go to STEP 6 for test execution. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

Step 2: Install Dependencies

  export SOURCE_ROOT=/<source_root>/

• RHEL (7.8, 7.9, 8.2, 8.4)

  • With Eclipse Adoptium Temurin Runtime (previously known as AdoptOpenJDK hotspot)

      sudo yum install -y wget tar git curl ca-certificates

    • Download and install Eclipse Adoptium Temurin Runtime (Java 11) from here.

  • With IBM Semeru Runtime (previously known as AdoptOpenJDK openj9)

      sudo yum install -y wget tar git curl ca-certificates

    • Download and Install IBM Semeru Runtime (Java 11) from here.

• SLES (12 SP5, 15 SP2, 15 SP3)

  • With Eclipse Adoptium Temurin Runtime (previously known as AdoptOpenJDK hotspot)

      sudo zypper install -y wget tar git curl gzip

    • Download and install Eclipse Adoptium Temurin Runtime (Java 11) from here.

  • With IBM Semeru Runtime (previously known as AdoptOpenJDK openj9)

      sudo zypper install -y wget tar git curl gzip

    • Download and Install IBM Semeru Runtime (Java 11) from here.

• Ubuntu (18.04, 20.04, 21.04, 21.10)

  • With Eclipse Adoptium Temurin Runtime (previously known as AdoptOpenJDK hotspot)

      sudo apt-get update
      sudo apt-get -y install wget tar git curl

    • Download and install Eclipse Adoptium Temurin Runtime (Java 11) from here.

  • With IBM Semeru Runtime (previously known as AdoptOpenJDK openj9)

      sudo apt-get update
      sudo apt-get -y install wget tar git curl

    • Download and Install IBM Semeru Runtime (Java 11) from here.

Note:

• At the time of creation of these build instructions, Apache Kafka was verified with JDK 11 version (Temurin-11.0.13+8) and (Semeru-11.0.13+8_openj9-0.29.0).

Step 3: Download the source code and build the jar files

 cd $SOURCE_ROOT
 export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ApacheKafka/3.0.0/patch"
 git clone https://github.com/apache/kafka.git
 cd kafka
 git checkout 3.0.0
 curl -sSL $PATCH_URL/IBMSemeru.patch | git apply               (Only for IBM Semeru Runtime Java)
 ./gradlew -PscalaOptimizerMode=method jar                      (Only for IBM Semeru Runtime Java)
 ./gradlew jar                                                  (Only for Adoptium Temurin Runtime Java)

Step 4: Build and Create rocksdbjni-6.19.3.jar for s390x

• Build Rocksdbjni jar using build script

  cd $SOURCE_ROOT
  wget https://github.com/linux-on-ibm-z/scripts/blob/master/RocksDB/v6.19.3/build_rocksdb.sh
  bash build_rocksdb.sh

• Replace Rocksdbjni jar

  cd $SOURCE_ROOT
  cp rocksdb/java/target/rocksdbjni-6.19.3-linux64.jar ./rocksdbjni-6.19.3.jar
  find ./kafka/ ~/.gradle/ -name 'rocksdbjni-6.19.3.jar' -print0 | xargs -0 -n1 cp ./rocksdbjni-6.19.3.jar
  # Optional, verify the installed jars have s390x support
  find ./kafka/ ~/.gradle/ -name 'rocksdbjni-6.19.3.jar' -print0 | xargs -0 -n1 unzip -l | grep s390x

Step 5: Set environment variables

• Update JAVA_HOME and PATH variable to switch to your JDK11

  export JAVA_HOME=<path to java11>
  export PATH=$JAVA_HOME/bin:$PATH

Step 6: Testing (Optional)

 cd $SOURCE_ROOT/kafka
 ./gradlew test -PscalaOptimizerMode=method --continue        (Only for IBM Semeru Runtime Java)
 ./gradlew test --continue                                    (Only for Adoptium Temurin Runtime Java)

Note:

• If any test fails due to timeout, try running it individually.
• If the testing process hangs and stops making progress, it might be helpful to increase the limit of opening files using command ulimit -n <new_value> and restart the tests
• You could also try to use forkEvery = 1 gradle option for testing to reduce the number of test case failures

Step 7: Verification

Follow official quickstart guide given here to verify the installation.

Note:

• You might need to add the arguments --partitions 1 --replication-factor 1 when running Step 3 of the quickstart guide to create new topics. It is a known issue that has been reported to the Kafka community here.

References:

• http://kafka.apache.org/