Skip to content

Building GlusterFS

Jump to bottom
aborkar-ibm edited this page May 27, 2021 · 38 revisions

Building GlusterFS

Following versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • RHEL (8.1, 8.2, 8.3) has 6.0
  • Ubuntu 18.04 has 3.13.2
  • Ubuntu 20.04 has 7.2
  • SLES 15 SP2 has 3.12.3

The instructions provided below specify the steps to build GlusterFS 9.0 on Linux on IBM Z for following distributions:

  • RHEL (7.8, 7.9, 8.1, 8.2, 8.3)
  • SLES (12 SP5, 15 SP2)
  • Ubuntu (18.04, 20.04)

General Notes:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it
  • For Openssl build on SLES 15.x, latest kernel version is needed.

Build and Install GlusterFS

Step 1: Build using script

If you want to build GlusterFS using manual steps, go to Step 2.

Use the following commands to build GlusterFS using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/9.0/build_glusterfs.sh
# Build GlusterFS
bash build_glusterfs.sh   [Provide -h option to check usage]

If the build completes successfully, go to STEP 5. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

Step 2: Install the dependencies

  • Set environment variables:

    export SOURCE_ROOT=/<source_root>/
export PATCH_URL=https://github.com/linux-on-ibm-z/scripts/tree/master/GlusterFS/9.0/patch

  • RHEL (7.8, 7.9)

    yum install -y autoconf automake bison bzip2 curl flex fuse-devel gcc git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel lvm2 make pkgconfig python python3 readline-devel wget zlib-devel patch which

    • Build urcu

      cd $SOURCE_ROOT
wget https://lttng.org/files/urcu/userspace-rcu-0.10.2.tar.bz2
tar xvjf userspace-rcu-0.10.2.tar.bz2
cd userspace-rcu-0.10.2
./configure --prefix=/usr --libdir=/usr/lib64
make
make install

    • Build openssl

      cd $SOURCE_ROOT
wget https://www.openssl.org/source/old/1.1.1/openssl-1.1.1d.tar.gz
tar xvf openssl-1.1.1d.tar.gz
cd openssl-1.1.1d
./config --prefix=/usr/local --openssldir=/usr/local no-weak-ssl-ciphers no-tls1 no-tls1-method
make
make install
ldconfig /usr/local/lib64
export PATH=/usr/local/bin:$PATH

  • RHEL (8.1, 8.2, 8.3)

    yum install -y autoconf automake bison bzip2 flex fuse-devel gcc-c++ git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel lvm2 make openssl-devel pkgconfig python3 readline-devel wget zlib-devel tar gzip libtirpc-devel patch rpcgen userspace-rcu-devel which diffutils xz

    • Build GCC

      cd $SOURCE_ROOT
wget https://ftpmirror.gnu.org/gcc/gcc-7.5.0/gcc-7.5.0.tar.xz
tar -xf gcc-7.5.0.tar.xz
cd gcc-7.5.0
./contrib/download_prerequisites
mkdir objdir
cd objdir
../configure --prefix=/opt/gcc --enable-languages=c,c++ --with-arch=zEC12 --with-long-double-128 \
       --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu --enable-threads=posix
       --with-system-zlib --disable-multilib
make -j 8
make install
ln -sf /opt/gcc/bin/gcc /usr/bin/gcc
ln -sf /opt/gcc/bin/g++ /usr/bin/g++
ln -sf /opt/gcc/bin/g++ /usr/bin/c++
export PATH=/opt/gcc/bin:"$PATH"

  • SLES (12 SP5)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc git glib2-devel libacl-devel libaio-devel librdmacm1 libtool libuuid-devel libxml2-devel lvm2 make pkg-config python2 rdma-core-devel readline-devel zlib-devel which patch gawk wget

    • Build urcu

      cd $SOURCE_ROOT
wget https://lttng.org/files/urcu/userspace-rcu-0.10.2.tar.bz2
tar xvf userspace-rcu-0.10.2.tar.bz2
cd userspace-rcu-0.10.2
./configure --prefix=/usr --libdir=/usr/lib64
make
make install

    • Build openssl

      cd $SOURCE_ROOT
wget https://www.openssl.org/source/old/1.1.1/openssl-1.1.1d.tar.gz
tar xvf openssl-1.1.1d.tar.gz
cd openssl-1.1.1d
./config --prefix=/usr/local --openssldir=/usr/local no-weak-ssl-ciphers no-tls1 no-tls1-method
make
make install
ldconfig /usr/local/lib64
export PATH=/usr/local/bin:$PATH

  • SLES (15 SP2)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc git-core glib2-devel libacl-devel libaio-devel librdmacm1 libtool liburcu-devel libuuid-devel libxml2-devel lvm2 make pkg-config python3 python3-xattr rdma-core-devel readline-devel zlib-devel which gawk dmraid

    • Build openssl

      cd $SOURCE_ROOT
wget https://www.openssl.org/source/old/1.1.1/openssl-1.1.1d.tar.gz
tar xvf openssl-1.1.1d.tar.gz
cd openssl-1.1.1d
./config --prefix=/usr/local --openssldir=/usr/local no-weak-ssl-ciphers no-tls1 no-tls1-method
make
make install
ldconfig /usr/local/lib64
export PATH=/usr/local/bin:$PATH

  • Ubuntu (18.04)

    apt-get update
apt-get install -y autoconf automake bison curl flex gcc git libacl1-dev libaio-dev libfuse-dev libglib2.0-dev libibverbs-dev librdmacm-dev libreadline-dev libtool liburcu-dev libxml2-dev lvm2 make openssl pkg-config python3 uuid-dev zlib1g-dev patch wget

    • Build openssl

      cd $SOURCE_ROOT
wget https://www.openssl.org/source/old/1.1.1/openssl-1.1.1d.tar.gz
tar xvf openssl-1.1.1d.tar.gz
cd openssl-1.1.1d
./config --prefix=/usr/local --openssldir=/usr/local no-weak-ssl-ciphers no-tls1 no-tls1-method
make
make install
ldconfig /usr/local/lib64
export PATH=/usr/local/bin:$PATH

  • Ubuntu 20.04

    apt-get update
apt-get install -y autoconf automake bison curl flex gcc g++ git libacl1-dev libaio-dev libfuse-dev libglib2.0-dev libibverbs-dev librdmacm-dev libreadline-dev libssl-dev libtool liburcu-dev libxml2-dev lvm2 make openssl pkg-config python3 uuid-dev zlib1g-dev patch wget bash

    • Build GCC

      cd $SOURCE_ROOT
wget https://ftpmirror.gnu.org/gcc/gcc-7.5.0/gcc-7.5.0.tar.xz
tar -xf gcc-7.5.0.tar.xz
cd gcc-7.5.0
./contrib/download_prerequisites
mkdir objdir
cd objdir
../configure --prefix=/opt/gcc --enable-languages=c,c++ --with-arch=zEC12 --with-long-double-128 \
       --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu --enable-threads=posix
       --with-system-zlib --disable-multilib
make -j 8
make install
ln -sf /opt/gcc/bin/gcc /usr/bin/gcc
ln -sf /opt/gcc/bin/g++ /usr/bin/g++
ln -sf /opt/gcc/bin/g++ /usr/bin/c++
export PATH=/opt/gcc/bin:"$PATH"

Step 3: Download, Configure and Build GlusterFS

  • Download and configure source

    Note: : If git clone fails with SSL certificate issue, execute git config --global http.sslVerify false command and then do git clone.

    cd $SOURCE_ROOT
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v9.0
./autogen.sh
./configure --enable-gnfs                                              # For RHEL, SLES 15.x and Ubuntu
./configure --enable-gnfs --disable-events                             # For SLES 12 SP5

  • Patch xlators/performance/io-threads/src/io-threads.h: (For RHEL 8.x, SLES and Ubuntu only)

    This patch fixes the ./tests/features/fuse-lru-limit.t, ./tests/features/weighted-rebalance.t, ./tests/basic/ec/ec-1468261.t test case failures.

    cd $SOURCE_ROOT/glusterfs
wget -O - $PATCH_URL/io-threads.h.diff
git apply io-threads.h.diff

  • Patch xlators/features/bit-rot/src/stub/bit-rot-stub.c:

    This patch fixes the ./tests/bitrot/bug-1207627-bitrot-scrub-status.t test case failure.

    cd $SOURCE_ROOT/glusterfs
wget -O - $PATCH_URL/bit-rot-stub.c.diff
git apply bit-rot-stub.c.diff

  • Patch xlators/nfs/server/src/nfs.c: (For RHEL 8.x only)

    This patch fixes the ./tests/bugs/glusterd/optimized-basic-testcases.t test case failure and crash in libtirpc library.

    cd $SOURCE_ROOT/glusterfs
wget -O - $PATCH_URL/nfs.c.diff
git apply nfs.c.diff

  • Build and install

    make
make install
ldconfig /usr/local/lib

Step 4: Run the test cases (optional)

4.1) Install the testing dependencies

  • RHEL (7.8, 7.9)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gcc-c++ gdb net-tools nfs-utils psmisc pyxattr vim xfsprogs yajl popt-devel python3-pip python3-devel
ln -s /usr/local/lib/libgfchangelog.so.0 /lib64/libgfchangelog.so   # For RHEL 7.9 only
ldconfig /usr/local/lib   # For RHEL 7.9 only
export LANG=en_US.UTF-8   # For RHEL 7.9 only

  • RHEL (8.1, 8.2, 8.3)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gcc-c++ gdb net-tools nfs-utils psmisc vim xfsprogs yajl redhat-rpm-config python3-devel python3-pyxattr python3-prettytable perl-Test-Harness popt-devel

  • SLES 12 SP5

    zypper install -y acl attr bc bind-utils boost-devel gcc-c++ gdb libexpat-devel libxml2-tools net-tools nfs-utils psmisc vim xfsprogs python3-xattr popt-devel python3-pip

  • SLES (15 SP2)

    zypper install -y acl attr bc bind-utils gdb libxml2-tools net-tools-deprecated nfs-utils psmisc thin-provisioning-tools vim xfsprogs python3-xattr python3-PrettyTable libselinux-devel selinux-tools thin-provisioning-tools popt-devel python3-xattr

  • Ubuntu (18.04, 20.04)

    apt-get install -y acl attr bc dbench dnsutils libxml2-utils net-tools nfs-common psmisc python3-pyxattr python3-prettytable thin-provisioning-tools vim xfsprogs yajl-tools rpm2cpio gdb cpio selinux-utils

  • Install pstack command

    • Ubuntu

      cd $SOURCE_ROOT
wget http://rpmfind.net/linux/opensuse/update/leap/15.1/oss/x86_64/gdb-8.3.1-lp151.4.3.1.x86_64.rpm
rpm2cpio gdb-8.3.1-lp151.4.3.1.x86_64.rpm| cpio -idmv
mv ./usr/bin/gstack /usr/bin/
rm -r ./etc ./usr
rm gdb-8.3.1-lp151.4.3.1.x86_64.rpm
ln -sf `which gstack` /usr/bin/pstack

    • SLES

      ln -sf `which gstack` /usr/bin/pstack

  • Install Python modules prettytable and pyxattr

    pip3 install prettytable          # For SLES 12 SP5 only
pip3 install prettytable pyxattr  # For RHEL 7.x only

  • Build DBENCH (For RHEL and SLES only)

    cd $SOURCE_ROOT
git clone https://github.com/sahlberg/dbench
cd dbench
git checkout caa52d347171f96eef5f8c2d6ab04a9152eaf113
./autogen.sh
./configure --datadir=/usr/local/share/doc/loadfiles/
make
make install

  • Build thin-provisioning-tools (For RHEL and SLES 12 SP5 only)

    cd $SOURCE_ROOT
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
git checkout v0.7.6
autoreconf
./configure
make
make install

  • Build yajl (For SLES only)

    cd $SOURCE_ROOT
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
./configure
make install

4.2) Patch and run tests

  • The following patch fixes the test cases listed below that fail due to hardcoded hash values for little-endian systems:

    • ./tests/bugs/posix/bug-1619720.t
    • ./tests/bugs/replicate/bug-1655854-support-dist-to-rep3-arb-conversion.t
    • ./tests/bugs/distribute/bug-1600379.t
    • ./tests/bugs/glusterd/bug-1699339.t
    • ./tests/bugs/distribute/bug-1786679.t
    • ./tests/bugs/glusterfs/bug-902610.t
    • ./tests/basic/changelog/changelog-rename.t
    • ./tests/basic/distribute/dir-heal.t
    • ./tests/basic/distribute/spare_file_rebalance.t
    • ./tests/basic/namespace.t
    cd $SOURCE_ROOT/glusterfs
wget -O - $PATCH_URL/hash-tests.diff
git apply hash-tests.diff

  • The following patch fixes test cases failing due to system configuration:

    • ./tests/bugs/shard/bug-1251824.t
    • ./tests/bugs/shard/bug-1468483.t
    • ./tests/basic/distribute/throttle-rebal.t
    • ./tests/bugs/nfs/bug-847622.t
    • ./tests/bugs/quota/bug-1293601.t
    • ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
    cd $SOURCE_ROOT/glusterfs
wget -O - $PATCH_URL/test-patch.diff
git apply test-patch.diff

  • Patch run-tests.sh for SLES 12 SP5 only, see this issue.)

    cd $SOURCE_ROOT/glusterfs
wget -O - $PATCH_URL/run-tests.sh.diff
git apply run-tests.sh.diff

  • Run the test cases

    cd $SOURCE_ROOT/glusterfs
export LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH   #For SLES 15 SP2 Only
ldconfig                                                   #For SLES 15 SP2 Only
./run-tests.sh

Note:

  • If any of the tests fail with error volume create: patchy: failed: Host <hostname> is not in 'Peer in Cluster' state, make changes to file /etc/hosts by copying the host details entry that is present in the beginning to the section IPv6 capable hosts and replace 0.0.0.0 with the IP address of the host. It should be similar to : 0.0.0.0 <hostname>.xyz.com hostname

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test cases:

    • Password-less SSH login has to be set up. Refer to the Setting Up the Environment for Geo-replication in the documentation.
    • The locale of the system must be UTF-8 for the test cases to pass._

  • The is_nfs_export_available function from nfs.rc used in some tests may fail intermittently. You may try to enable rpcbind as that may fix some of the test cases. The community is aware of this behaviour, refer here for more info.

  • For ./tests/bugs/glusterfs/bug-866459.t test case: This test is under investigation, refer to this issue for details.

  • For tests failing due to /mnt/glusterfs/0/: Transport endpoint is not connected, refer to the issue.

  • For ./tests/bugs/glusterfs-server/bug-861542.t test case: If the hostname of the machine is long, the test case will fail. The hostname of the machine can be shortened temporarily via the hostname command to get the test case to pass.

  • The following test cases fail if the file system is of type ext4. See issue

    • ./tests/bugs/posix/bug-990028.t

  • The test case ./tests/basic/gfapi/gfapi-copy-file-range.t fails for SLES (12 SP5, 15 SP2) on both x86_64 and s390x due to a system call failure. This is beacuse of the unsupported xfs reflink, it can be enabled via echo 'Y' > /sys/module/xfs/parameters/allow_unsupported.

  • The following test cases fail on Ubuntu if the nobody group is missing. The group can be added through the following command groupadd --non-unique --gid $(getent group nogroup | awk -F : -e '{print $3}') nobody:

    • ./tests/bugs/rpc/bug-954057.t
    • ./tests/bugs/glusterfs-server/bug-887145.t

  • The following test cases fail on RHEL 8 due to a kernel bug, updating the kernel via yum update kernel fixes the issue:

    • ./tests/basic/playground/template-xlator-sanity.t
    • ./tests/basic/trace.t

  • Tests under ./tests/bugs/snapshot require the snapd service to be stopped (systemctl stop snapd) if installed:.

  • Certain test cases timeout in run-tests.sh. They pass when run individually through prove -vf <testcase>. Following are few of the known test cases that fail due to timeout.

    • ./tests/basic/afr/entry-self-heal.t
    • ./tests/basic/afr/split-brain-healing-ctime.t
    • ./tests/basic/afr/split-brain-healing.t
    • ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t

  • The test case ./tests/basic/ctime/ctime-utimesat.t passes but very rarely.

  • Certain test cases such as ./tests/basic/afr/split-brain-healing.t will pass upon rerun or rebooting the system.

  • Additional information on the GlusterFS test framework can be found here.

Step 5: Run the GlusterFS Daemon (Optional)

glusterd

Step 6: Testing the GlusterFS volume (Optional)

Refer Documentation for instructions.

Reference

https://www.gluster.org/ https://github.com/gluster/glusterfs http://docs.gluster.org/en/latest/Developer-guide/Building-GlusterFS/

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally