Skip to content

Building Bazel

Jump to bottom
linuxonz edited this page Feb 23, 2024 · 31 revisions

Building Bazel

Following instructions specify the steps to build Bazel version 7.0.2 on Linux on IBM Z:

  • Ubuntu (20.04, 22.04)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1 : Building Bazel

1.1) Build using script

If you'd like to build Bazel using the manual steps, please go to STEP 1.2.

Use the following commands to build Bazel using the build script. Please make sure you have wget installed.

  wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/7.0.2/build_bazel.sh

  # Build Bazel
  bash build_bazel.sh   [Provide -t option for executing build with tests]

In case of any errors, check logs for more details or go to STEP 1.2 to follow the manual build steps.

1.2) Install dependencies  

  • Set environment variable SOURCE_ROOT:
  export SOURCE_ROOT=/<source_root>/
  sudo apt-get update
  • Ubuntu 20.04
  sudo apt-get install -y --no-install-recommends bind9-host build-essential coreutils curl dnsutils ed expect file git gnupg2 iproute2 iputils-ping mkisofs lcov less libssl-dev lsb-release netcat-openbsd openjdk-11-jdk-headless zip zlib1g-dev unzip wget python2 python2-dev python-is-python3 python3 python3-dev python3-pip python3-requests python3-setuptools python3-six python3-wheel python3-yaml
  • Ubuntu 22.04
  sudo apt-get install -y --no-install-recommends bind9-host build-essential coreutils curl dnsutils ed expect file git gnupg2 iproute2 iputils-ping mkisofs lcov less libssl-dev lsb-release netcat-openbsd openjdk-11-jdk-headless zip zlib1g-dev unzip wget python2 python2-dev python3
  • Set JAVA_HOME:
  export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-s390x
  export PATH=$JAVA_HOME/bin:$PATH

1.3) Build

  • Download and patch rules_java 7.1.0:
  cd $SOURCE_ROOT
  git clone -b 7.1.0 https://github.com/bazelbuild/rules_java.git
  cd rules_java
  curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/7.0.2/patch/rules_java_7.1.0.patch | git apply
  • Download Bazel 7.0.2 distribution archive:
  cd $SOURCE_ROOT
  wget https://github.com/bazelbuild/bazel/releases/download/7.0.2/bazel-7.0.2-dist.zip
  mkdir -p dist/bazel && cd dist/bazel
  unzip -q ../../bazel-7.0.2-dist.zip
  chmod -R +w .
  • Build the bootstrap bazel:
  cd ${SOURCE_ROOT}/dist/bazel
  env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
  • Checkout the Bazel 7.0.2 source code and apply the patch:
  cd $SOURCE_ROOT
  git clone --depth 1 -b 7.0.2 https://github.com/bazelbuild/bazel.git
  cd bazel
  curl -sSLO https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/7.0.2/patch/bazel.patch
  sed -i "s#RULES_JAVA_ROOT_PATH#${SOURCE_ROOT}#g" bazel.patch
  patch -p1 < bazel.patch
  rm -f bazel.patch

Note: Netty and netty-tcnative jar are needed to fix test case failures related to remote execution. If remote execution functionality is not required then directly go to Build Bazel section.

  • Install and build netty-tcnative 2.0.61:
  cd $SOURCE_ROOT
  wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/netty-tcnative/2.0.61/build_netty.sh
  bash build_netty.sh -y
  • Build netty 4.1.93:
  cd $SOURCE_ROOT
  git clone https://github.com/netty/netty.git
  cd netty
  git checkout netty-4.1.93.Final
  mvn clean install -DskipTests
  • Copy the netty and netty-tcnative jar into the respective bazel directory:
  cp $SOURCE_ROOT/netty-tcnative/openssl-classes/target/netty-tcnative-classes-2.0.61.Final.jar \
      $SOURCE_ROOT/netty-tcnative/boringssl-static/target/netty-tcnative-boringssl-static-2.0.61.Final-linux-s390_64.jar \
      $SOURCE_ROOT/netty/buffer/target/netty-buffer-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/codec/target/netty-codec-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/codec-http/target/netty-codec-http-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/codec-http2/target/netty-codec-http2-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/common/target/netty-common-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/handler/target/netty-handler-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/handler-proxy/target/netty-handler-proxy-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/resolver/target/netty-resolver-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/resolver-dns/target/netty-resolver-dns-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/transport/target/netty-transport-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/transport-classes-epoll/target/netty-transport-classes-epoll-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/transport-classes-kqueue/target/netty-transport-classes-kqueue-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/transport-native-unix-common/target/netty-transport-native-unix-common-4.1.93.Final-linux-s390_64.jar \
      $SOURCE_ROOT/netty/transport-native-kqueue/target/netty-transport-native-kqueue-4.1.93.Final.jar \
      $SOURCE_ROOT/netty/transport-native-epoll/target/netty-transport-native-epoll-4.1.93.Final-linux-s390_64.jar \
      $SOURCE_ROOT/bazel/third_party
  • Build Bazel:
  cd $SOURCE_ROOT/bazel
  ${SOURCE_ROOT}/dist/bazel/output/bazel build -c opt --stamp --embed_label "7.0.2" //src:bazel //src:bazel_jdk_minimal //src:test_repos //src/main/java/...
  mkdir output
  cp bazel-bin/src/bazel output/bazel
  • Bazel version:
  cd $SOURCE_ROOT/bazel
  ./output/bazel --version
  bazel 7.0.2

Step 2: Execute Test Suite (Optional)

  • Run complete test suite:
  cd $SOURCE_ROOT/bazel
  ./output/bazel --host_jvm_args=-Xmx2g test -c opt --build_tests_only --flaky_test_attempts=3 --test_timeout=3600 --show_progress_rate_limit=5 --terminal_columns=143 --show_timestamps --verbose_failures \
    -- //scripts/... //src/java_tools/... //src/test/... //src/tools/execlog/... //src/tools/singlejar/... //src/tools/workspacelog/... //third_party/ijar/... -//tools/android/... //tools/aquery_differ/... //tools/python/... \
    -//src/java_tools/import_deps_checker/... -//src/test/shell/bazel/android/... -//src/test/java/com/google/devtools/build/android/... -//src/test/shell/bazel:bazel_determinism_test -//src/test/shell/bazel:bazel_android_tools_test
  • Run individual test:
  cd $SOURCE_ROOT/bazel
  ./output/bazel test -c opt --cache_test_results=no --build_tests_only --flaky_test_attempts=3 --test_timeout=3600 --show_progress_rate_limit=5 --show_timestamps --verbose_failures -- //<module_name>:<testcase_name>

For example:

  cd $SOURCE_ROOT/bazel
  ./output/bazel test -c opt --cache_test_results=no --build_tests_only --flaky_test_attempts=3 --test_timeout=3600 --show_progress_rate_limit=5 --show_timestamps --verbose_failures -- //src/test/cpp/util:md5_test

Notes:

1. Some tests have been excluded in the bazel test command above:

  • //src/java_tools/import_deps_checker/... fails on all platforms
  • //src/test/shell/bazel/android/... fails because an android sdk is required
  • //src/test/java/com/google/devtools/build/android/... fails because an android sdk is required
  • //src/test/shell/bazel:bazel_determinism_test fails because some archive files which must be rebuilt during the test on s390x result in new file sizes and md5sum values
  • //src/test/shell/bazel:bazel_android_tools_test fails on both s390x and x86_64

2. The following test would be skipped, since prebuilt ijar and singlejar binaries haven't been released for s390x:

  • //src/test/shell/bazel:bazel_java_test_defaults_prebuilt

3. The following test would fail unless ipv6 is enabled on the system:

  • //src/test/java/com/google/devtools/build/lib/bazel/bzlmod:BzlmodTests

4. The following test may fail with an out of memory error if building on a machine with less than 4GB of memory:

  • //src/test/java/com/google/devtools/build/lib/graph:GraphTests

5. Below test cases fail on both s390x and x86_64 on all distros:

//src/test/java/com/google/devtools/build/lib/bazel/bzlmod:BzlmodTests   
//src/test/java/com/google/devtools/build/lib/sandbox:SpawnRunnerTests   
//src/test/java/com/google/devtools/build/lib/shell:CommandUsingLinuxSandboxTest 
//src/test/shell/bazel:bazel_spawnstats_test                             
//src/test/shell/bazel:embedded_tools_deps_test                          
//src/test/shell/integration:test_test                                   
//src/test/tools:daemonize_test                                          
//src/test/shell/bazel:bazel_example_test                                
//src/test/shell/integration:bazel_hardened_sandboxed_worker_test        
//src/test/shell/integration:sandboxing_test                             
//src/test/shell/bazel:external_integration_test

6. Below test cases fail on both s390x and x86_64 on Ubuntu20.04:

//src/test/java/com/google/devtools/build/lib/buildtool:OutputArtifactConflictTest
//src/test/shell/bazel/remote:remote_execution_test
//src/test/shell/bazel:cc_integration_test

7. The following test passes with Python v3.9:

  • //src/test/java/com/google/devtools/build/lib/query2/cquery:ProtoOutputFormatterCallbackTest

8. The following test fails on s390x and is currently being investigated:

  • //src/test/py/bazel:bazel_module_test

9. More information on running Bazel in a low memory environment:

  • https://bazel.build/advanced/performance/memory

Step 3: Use the new module-focused external dependencies management system in Bazel (Optional)

The new module-focused MODULE.bazel system can be enabled by the flag --enable_bzlmod, for example:

  cd $SOURCE_ROOT/bazel
  ./output/bazel build --enable_bzlmod //src:bazel_nojdk

References:

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally