Building Bazel

Building Bazel

Following instructions specify the steps to build Bazel version 4.1.0 on Linux on IBM Z:

• Ubuntu (18.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1 : Building Bazel

1.1) Build using script

If you'd like to build Bazel using the manual steps, please go to STEP 1.2.

Use the following commands to build Bazel using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/4.1.0/build_bazel.sh

# Build Bazel
bash build_bazel.sh   [Provide -t option for executing build with tests]

In case of any errors, check logs for more details or go to STEP 1.2 to follow the manual build steps.

1.2) Install dependencies

 export SOURCE_ROOT=/<source_root>/
 sudo apt-get update
 sudo apt-get install -y wget curl openjdk-11-jdk unzip patch build-essential zip python3 git

• If applicable, link /usr/bin/python to /usr/bin/python3

  sudo ln -sf /usr/bin/python3 /usr/bin/python

• Set JAVA_HOME:

   export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-s390x
   export PATH=$JAVA_HOME/bin:$PATH

1.3) Build

• Download Bazel 4.1.0 distribution archive:

   cd $SOURCE_ROOT
   mkdir bazel && cd bazel
   wget https://github.com/bazelbuild/bazel/releases/download/4.1.0/bazel-4.1.0-dist.zip
   unzip bazel-4.1.0-dist.zip
   chmod -R +w .

• Apply the patches:

   curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/4.1.0/patch/bazel.patch | git apply

  Note: Netty and netty-tcnative jar are needed to fix test case failures related to remote execution. If remote execution functionality is not required then directly go to Build Bazel section.

  • Build netty-tcnative 2.0.24:

  cd $SOURCE_ROOT
  sudo apt-get install -y ninja-build cmake perl golang libssl-dev autoconf automake libtool make tar maven default-jdk libapr1-dev
  git clone https://github.com/netty/netty-tcnative.git
  cd netty-tcnative
  git checkout netty-tcnative-parent-2.0.24.Final
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/4.1.0/patch/netty-tcnative.patch | git apply
  mvn install

  • Build netty 4.1.48:

  cd $SOURCE_ROOT
  git clone https://github.com/netty/netty.git
  cd netty
  git checkout netty-4.1.48.Final
  ./mvnw clean install -DskipTests

  • Copy the netty and netty-tcnative jar into the respective bazel directory and apply the patch to use them:

  cp $SOURCE_ROOT/netty-tcnative/boringssl-static/target/netty-tcnative-boringssl-static-2.0.24.Final-linux-s390_64.jar $SOURCE_ROOT/bazel/third_party/netty_tcnative/netty-tcnative-boringssl-static-2.0.24.Final.jar
  cp $SOURCE_ROOT/netty/all/target/netty-all-4.1.48.Final.jar $SOURCE_ROOT/bazel/third_party/netty/
  cd $SOURCE_ROOT/bazel
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/4.1.0/patch/bazel-netty.patch | git apply

• Build Bazel:

  bash ./compile.sh
  export PATH=$PATH:$SOURCE_ROOT/bazel/output/

• Bazel version:

  bazel --version
  bazel 4.1.0- (@non-git)

Note: While building Bazel, if build fails with an error java.lang.OutOfMemoryError: Java heap space, apply the patch and rebuild Bazel as mentioned below.

curl -o compile.patch https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/4.1.0/patch/compile.patch
patch --ignore-whitespace $SOURCE_ROOT/bazel/scripts/bootstrap/compile.sh < compile.patch
cd $SOURCE_ROOT/bazel/
bash ./compile.sh
export PATH=$PATH:$SOURCE_ROOT/bazel/output/

Step 2: Execute Test Suite (Optional)

• Run complete test suite:

  cd $SOURCE_ROOT/bazel
  bazel test --flaky_test_attempts=3 --build_tests_only --copt="-Wimplicit-fallthrough=0" --local_test_jobs=12 --show_progress_rate_limit=5 --terminal_columns=143 --show_timestamps --verbose_failures --keep_going --jobs=32 --test_timeout=1200 -- //src/... //third_party/ijar/...

• Run individual test:

  bazel test --flaky_test_attempts=3 --build_tests_only --copt="-Wimplicit-fallthrough=0" --local_test_jobs=12 --show_progress_rate_limit=5 --terminal_columns=143 --show_timestamps --verbose_failures --keep_going --jobs=32 -- //src/<module_name>:<testcase_name>

  For example:

  bazel test --flaky_test_attempts=3 --build_tests_only --copt="-Wimplicit-fallthrough=0" --local_test_jobs=12 --show_progress_rate_limit=5 --terminal_columns=143 --show_timestamps --verbose_failures --keep_going --jobs=32 -- //src/test/cpp/util:md5_test

  Note: 1. Following tests are failing on s390x and x86: //src/java_tools/import_deps_checker/javatests/com/google/devtools/build/importdeps:string_annotation_value_test //src/test/shell/bazel:bazel_bootstrap_distfile_test //src/test/shell/bazel:bazel_determinism_test //src/test/shell/bazel:python_version_test //src/test/shell/bazel:srcs_test //src/test/shell/bazel:workspace_resolved_test

  2. Some tests from below modules failed to build on both s390x and x86: //src/java_tools/import_deps_checker/javatests/com/google/devtools/build/importdeps //src/test/java/com/google/devtools/build/android/desugar

  3. There also might be certain test cases that timeout. Those will pass after rerunning individually //src/test/java/com/google/devtools/build/lib/blackbox/tests/workspace:WorkspaceBlackBoxTest //src/test/shell/bazel:bazel_bootstrap_distfile_tar_test //src/test/shell/bazel:bazel_proto_library_test //src/test/shell/integration:process_wrapper_test

References:

http://bazel.io/ https://bazel.build/basics/getting_started.html https://docs.bazel.build/versions/4.1.0/install-compile-source.html https://github.com/bazelbuild/bazel/issues/8833#issuecomment-629039759