Building CockroachDB

Building CockroachDB

The instructions provided below specify the steps to build CockroachDB version 20.2.5 on Linux on IBM Z for the following distributions:

• Ubuntu (18.04, 20.04, 20.10)
• SLES (12 SP5, 15 SP2)
• RHEL (7.8, 7.9, 8.1, 8.2, 8.3)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1: Build and Install CockroachDB

1.1) Build using script

If you'd like to build CockroachDB using the manual steps, please go to STEP 1.2.

Use the following commands to build CockroachDB using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/CockroachDB/20.2.5/build_crdb.sh

# Build CockroachDB
bash build_crdb.sh   [Provide -t option for executing build with tests]

In case of any errors, check logs for more details or go to STEP 1.2 to follow the manual build steps.

1.2) Install the Dependencies

export SOURCE_ROOT=/<source_root>/

• Ubuntu (18.04, 20.04, 20.10)

sudo apt-get update && sudo apt-get -y install g++ autoconf automake cmake wget libncurses5-dev bison xz-utils patch git curl

• SLES (12 SP5)

sudo zypper install -y autoconf automake wget ncurses-devel bison patch tar gzip cmake gawk xz python gcc7-c++ zlib-devel bzip2 curl git

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 100
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 100
sudo update-alternatives --install /usr/bin/cpp cpp /usr/bin/cpp-7 100

• SLES (15 SP2)

sudo zypper install -y autoconf automake wget ncurses-devel bison patch tar gzip cmake gawk xz gcc-c++ git python curl

• If applicable, link /usr/bin/s390x-linux-gnu-gcc to /usr/bin/gcc on SLES

sudo ln -s /usr/bin/gcc /usr/bin/s390x-linux-gnu-gcc

• RHEL (7.8, 7.9)

sudo subscription-manager repos --enable=rhel-7-server-for-system-z-rhscl-rpms
sudo yum install -y git autoconf automake wget ncurses-devel bison patch tar gzip xz make bzip2 zlib-devel gcc-c++ curl diffutils

• RHEL (8.1, 8.2, 8.3)

sudo yum install -y gcc-c++ autoconf cmake git wget make ncurses-devel curl xz diffutils bison
sudo ln /usr/bin/gcc /usr/bin/s390x-linux-gnu-gcc

1.3) Build and install CMake (Only for RHEL 7.x and SLES 12 SP5)

cd $SOURCE_ROOT
wget https://github.com/Kitware/CMake/releases/download/v3.13.4/cmake-3.13.4.tar.gz
tar -xvzf cmake-3.13.4.tar.gz
cd cmake-3.13.4
./bootstrap
make
sudo make install
cmake --version

1.4) Enable Software Collections: (Only for RHEL 7.x)

requires GCC 7. It is available on RHEL 7.x via the 'Software Collections' packaging system. You can find out more about Software Collections here: https://www.softwarecollections.org/en/.

Following command will enable GCC 7. The changes are not persistent and these commands will need to be re-run every time a new terminal session is started.

scl enable devtoolset-7 bash

1.5) Install Go 1.13.15

cd $SOURCE_ROOT
wget https://storage.googleapis.com/golang/go1.13.15.linux-s390x.tar.gz
sudo tar -C /usr/local -xzf go1.13.15.linux-s390x.tar.gz
export PATH=/usr/local/go/bin:$PATH

1.6) Install Nodejs and yarn

cd $SOURCE_ROOT
wget https://nodejs.org/dist/v12.18.2/node-v12.18.2-linux-s390x.tar.xz
chmod ugo+r node-v12.18.2-linux-s390x.tar.xz
sudo tar -C /usr/local -xf node-v12.18.2-linux-s390x.tar.xz
export PATH=$PATH:/usr/local/node-v12.18.2-linux-s390x/bin
node -v

# For Ubuntu and RHEL
sudo env PATH=$PATH npm install -g yarn

# For SLES
sudo chmod ugo+w -R /usr/local/node-v12.18.2-linux-s390x
env PATH=$PATH npm install -g yarn

1.7) Change the ownership of .config (Only for Ubuntu)

cd
mkdir -p .config
sudo chown -R $(whoami):$(whoami) .config

1.8) Build and Install CockroachDB

• Download source code

  export GOPATH=$SOURCE_ROOT
  cd $SOURCE_ROOT
  mkdir -p $(go env GOPATH)/src/github.com/cockroachdb
  cd $(go env GOPATH)/src/github.com/cockroachdb
  git clone https://github.com/cockroachdb/cockroach
  cd cockroach
  git checkout v20.2.5
  git submodule update --init --recursive # Fetch the submodules

• Apply patches to code using below commands:

  cd $SOURCE_ROOT/src/github.com/cockroachdb/cockroach
   curl -sSL https://github.com/cockroachdb/cockroach/commit/0de1fd4d19b9cf39ea2c4cce668714c8ec9394b4.patch | git apply -
   curl -sSL https://github.com/ianlancetaylor/cgosymbolizer/commit/d43e30eacb43da92f28d523438f977af0a7b6072.patch | git apply --directory=vendor/github.com/ianlancetaylor/cgosymbolizer -
   curl -sSL https://github.com/apache/arrow/commit/aca707086160afd92da62aa2f9537a284528e48a.patch | git apply --directory=vendor/github.com/apache/arrow --exclude '**/array/bufferbuilder_numeric_test.go' -
   curl -sSL https://github.com/cockroachdb/snappy/commit/060abe547466cfd7cec09e53846e9e205d019045.patch | git apply --directory=c-deps/snappy -

• Build and install

  cd $(go env GOPATH)/src/github.com/cockroachdb/cockroach
  make build
  sudo env GOPATH=$GOPATH PATH=$PATH make install
  sudo mkdir -p /usr/local/lib/cockroach
  sudo cp lib/libgeos.so /usr/local/lib/cockroach/
  sudo cp lib/libgeos_c.so /usr/local/lib/cockroach/

Step 2: Testing (Optional)

cd $(go env GOPATH)/src/github.com/cockroachdb/cockroach
make test

There are currently known test failures on s390x.

Many tests check for exact floating point values however due to optimizations on the s390x platform that cause small differences, tests in these packages will fail.

• github.com/cockroachdb/cockroach/pkg/ccl/logictestccl
• github.com/cockroachdb/cockroach/pkg/geo
• github.com/cockroachdb/cockroach/pkg/geo/geogfn
• github.com/cockroachdb/cockroach/pkg/geo/geoindex
• github.com/cockroachdb/cockroach/pkg/geo/geomfn
• github.com/cockroachdb/cockroach/pkg/sql/logictest
• github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder
• github.com/cockroachdb/cockroach/pkg/sql/opt/memo
• github.com/cockroachdb/cockroach/pkg/sql/sem/tree
• github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval_test
• github.com/cockroachdb/cockroach/pkg/geo/geogfn
• github.com/cockroachdb/cockroach/pkg/geo/geographiclib

Tests in these packages fail because CPU information is reported in a platform dependent format. Ref. https://groups.google.com/g/cockroach-db/c/NNVvOUrMBik

• github.com/cockroachdb/cockroach/pkg/server
• github.com/cockroachdb/cockroach/pkg/server/diagnostics

Tests in these packages fail because of platform dependent filenames in the stack traces.

• github.com/cockroachdb/cockroach/pkg/sql
• github.com/cockroachdb/cockroach/pkg/util/log

Tests in this package fail due to platform dependent hash function values.

• github.com/cockroachdb/cockroach/pkg/sql/colexec

These specific tests in the github.com/cockroachdb/cockroach/pkg/sql/logictest package fail because the binary encoding of the sql plan is platform dependent but the plan is correct and is decoded correctly.

• github.com/cockroachdb/cockroach/pkg/sql/logictest
  • TestLogic/5node/dist_vectorize
  • TestLogic/5node/explain_analyze_plans
  • TestLogic/5node-disk/dist_vectorize
  • TestLogic/5node-spec-planning/explain_analyze_plans

Note:

1. Some test cases might experience time out problem. They should pass after increasing the timeout and run the test again. To run individual test with an increased time out. For example, github.com/cockroachdb/cockroach/pkg/sql/logictest could fail due to timeout, we do the following:

cd $(go env GOPATH)/src/github.com/cockroachdb/cockroach/pkg/sql/logictest
go test . -timeout 1h

Step 3: Verify

cockroach version

Output should look like:

Build Tag:        v20.2.5-dirty
Build Time:       2021/02/22 23:39:50
Distribution:     CCL
Platform:         linux s390x (s390x-linux-gnu)
Go Version:       go1.13.15
C Compiler:       gcc 10.2.0
Build Commit ID:  162c5ac4968cf31c0ed54cd29aa8aeccd66247bb
Build Type:       development

Please visit this link to start a local cluster and CockroachDB SQL.

References:

https://www.cockroachlabs.com/docs https://www.cockroachlabs.com/docs/stable/learn-cockroachdb-sql.html#before-you-begin https://github.com/cockroachdb/cockroach https://wiki.crdb.io/wiki/spaces/CRDB/pages/181338446/Getting+and+building+from+source