Skip to content

Building GlusterFS

Jump to bottom
aborkar-ibm edited this page Jun 8, 2021 · 38 revisions

Building GlusterFS

Following versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • RHEL (8.1, 8.2, 8.3) has 6.0
  • SLES 15 SP2 has 3.12.3
  • Ubuntu 18.04 has 3.13.2
  • Ubuntu 20.04 has 7.2
  • Ubuntu 21.04 has 9.0

The instructions provided below specify the steps to build or install GlusterFS 9.2 on Linux on IBM Z for following distributions:

  • RHEL (7.8, 7.9, 8.1, 8.2, 8.3)
  • SLES (12 SP5, 15 SP2)

Note: For Ubuntu (18.04, 20.04, 21.04), please refer to this link to install GlusterFS from the community GlusterFS PPA (ppa:gluster/glusterfs-9).

General Notes:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Step 1: Build and Install GlusterFS

Step 1.1: Build using script

If you want to build GlusterFS using manual steps, go to Step 1.2.

Use the following commands to build GlusterFS using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/9.2/build_glusterfs.sh
# Build GlusterFS
bash build_glusterfs.sh   [Provide -h option to check usage]

If the build completes successfully, go to STEP 3. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

Step 1.2: Install the dependencies

  • Set environment variables:

    export SOURCE_ROOT=/<source_root>/
export PREFIX=/usr/local
export PATCH_URL=https://github.com/linux-on-ibm-z/scripts/tree/master/GlusterFS/9.2/patch

PATH=${PREFIX}/bin${PATH:+:${PATH}}
export PATH

PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PREFIX}/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
export PKG_CONFIG_PATH

LD_LIBRARY_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export LD_LIBRARY_PATH

LD_RUN_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
export LD_RUN_PATH

  • RHEL (7.8, 7.9)

    yum install -y autoconf automake bison bzip2 curl flex fuse-devel devtoolset-7 git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel lvm2 make pkgconfig python python3 readline-devel wget zlib-devel patch which
#Switch to GCC 7
source /opt/rh/devtoolset-7/enable

  • RHEL (8.1, 8.2, 8.3)

    yum install -y autoconf automake bison bzip2 flex fuse-devel gcc-c++ git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel liburing-devel lvm2 make openssl-devel pkgconfig python3 readline-devel wget zlib-devel tar gzip libtirpc-devel patch rpcgen userspace-rcu-devel which diffutils xz

  • SLES (12 SP5)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc7 gcc7-c++ git glib2-devel libacl-devel libaio-devel librdmacm1 libtool libuuid-devel libxml2-devel lvm2 make pkg-config python2 rdma-core-devel readline-devel zlib-devel which patch gawk wget popt-devel
update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-7 40
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 40
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 40
update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-7 40

  • SLES (15 SP2)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc-c++ git-core glib2-devel libacl-devel libaio-devel librdmacm1 libtool liburcu-devel libuuid-devel libxml2-devel lvm2 make pkg-config python3 python3-xattr rdma-core-devel readline-devel openssl-devel zlib-devel which gawk dmraid popt-devel

Step 1.3: Build urcu (Only on RHEL 7.x and SLES 12 SP5)

  cd $SOURCE_ROOT
  wget https://lttng.org/files/urcu/userspace-rcu-0.10.2.tar.bz2
  tar xvjf userspace-rcu-0.10.2.tar.bz2
  cd userspace-rcu-0.10.2
  ./configure --prefix=${PREFIX}
  make
  make install

Step 1.4: Build openssl (Only on RHEL 7.x and SLES 12 SP5)

  cd $SOURCE_ROOT
  wget https://www.openssl.org/source/old/1.1.1/openssl-1.1.1d.tar.gz
  tar xvf openssl-1.1.1d.tar.gz
  cd openssl-1.1.1d
  ./config --prefix=${PREFIX} --openssldir=${PREFIX} no-weak-ssl-ciphers no-tls1 no-tls1-method
  make
  make install

Step 1.5: Build GCC 7.5.0 (Only on RHEL 8.x)

  cd $SOURCE_ROOT
  wget https://ftpmirror.gnu.org/gcc/gcc-7.5.0/gcc-7.5.0.tar.xz
  tar -xf gcc-7.5.0.tar.xz
  cd gcc-7.5.0
  ./contrib/download_prerequisites
  mkdir objdir
  cd objdir
  ../configure --prefix=${PREFIX} --enable-languages=c,c++ --with-arch=zEC12 --with-long-double-128 --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu --enable-threads=posix --with-system-zlib --disable-multilib
  make -j 8
  make install
  update-alternatives --install /usr/bin/cc cc ${PREFIX}/bin/gcc 40
  update-alternatives --install /usr/bin/gcc gcc ${PREFIX}/bin/gcc 40
  update-alternatives --install /usr/bin/g++ g++ ${PREFIX}/bin/g++ 40
  update-alternatives --install /usr/bin/c++ c++ ${PREFIX}/bin/g++ 40

Step 1.6: Build liburing (Only on RHEL 7.x and SLES)

  • Download the source

      cd $SOURCE_ROOT
  wget --no-check-certificate https://github.com/axboe/liburing/archive/refs/tags/liburing-2.0.tar.gz
  tar xvf liburing-2.0.tar.gz

  • Patch test/io_uring_register.c: (Only on RHEL 7.x and SLES 12 SP5)

      cd $SOURCE_ROOT/liburing-liburing-2.0
  wget --no-check-certificate $PATCH_URL/liburing.diff
  patch -p1 < liburing.diff

  • Build and install

      ./configure --prefix=${PREFIX}
  make
  make install

Step 1.7: Download, Configure and Build GlusterFS

  • Download and configure source

    Note: : If git clone fails with SSL certificate issue, execute git config --global http.sslVerify false command and then do git clone.

    cd $SOURCE_ROOT
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v9.2
./autogen.sh
./configure --enable-gnfs                                              # For RHEL and SLES 15.x
./configure --enable-gnfs --disable-events                             # For SLES 12 SP5

  • Patch xlators/performance/io-threads/src/io-threads.h: (For RHEL 8.x and SLES only)

    This patch fixes the ./tests/features/weighted-rebalance.t test case failure.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/io-threads.h.diff
git apply io-threads.h.diff

  • Patch xlators/features/bit-rot/src/stub/bit-rot-stub.c:

    This patch fixes the ./tests/bitrot/bug-1207627-bitrot-scrub-status.t test case failure.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/bit-rot-stub.c.diff
git apply bit-rot-stub.c.diff

  • Patch xlators/nfs/server/src/nfs.c: (For RHEL 8.x only)

    This patch fixes the ./tests/bugs/glusterd/optimized-basic-testcases.t test case failure and crash in libtirpc library.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/nfs.c.diff
git apply nfs.c.diff

  • Build and install

    make
make install

Step 2: Run the test cases (optional)

2.1) Install the testing dependencies

  • RHEL (7.8, 7.9)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gdb net-tools nfs-utils psmisc pyxattr vim xfsprogs yajl popt-devel python3-pip

  • RHEL (8.1, 8.2, 8.3)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gdb net-tools nfs-utils psmisc vim xfsprogs yajl redhat-rpm-config python3-devel python3-pyxattr python3-prettytable perl-Test-Harness popt-devel

  • SLES 12 SP5

    zypper install -y acl attr bc bind-utils boost-devel gdb libexpat-devel libxml2-tools net-tools nfs-utils psmisc vim xfsprogs python3-xattr popt-devel python3-pip

  • SLES (15 SP2)

    zypper install -y acl attr bc bind-utils gdb libxml2-tools net-tools-deprecated nfs-utils psmisc thin-provisioning-tools vim xfsprogs python3-xattr python3-PrettyTable libselinux-devel selinux-tools thin-provisioning-tools popt-devel

  • Link pstack command (Only on SLES)

    ln -sf `which gstack` /usr/bin/pstack

  • Install Python modules prettytable (Only on RHEL 7.x and SLES 12 SP5)

    pip3 install prettytable

  • Build DBENCH

    cd $SOURCE_ROOT
git clone https://github.com/sahlberg/dbench
cd dbench
git checkout caa52d347171f96eef5f8c2d6ab04a9152eaf113
./autogen.sh
./configure --datadir=/usr/local/share/doc/loadfiles/
make
make install

  • Build thin-provisioning-tools (For RHEL and SLES 12 SP5 only)

    cd $SOURCE_ROOT
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
git checkout v0.7.6
autoreconf
./configure
make
make install

  • Build yajl (For SLES only)

    cd $SOURCE_ROOT
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
./configure
make install

2.2) Patch and run tests

  • The following patch fixes the test cases listed below that fail due to hardcoded hash values for little-endian systems:

    • ./tests/bugs/posix/bug-1619720.t
    • ./tests/bugs/replicate/bug-1655854-support-dist-to-rep3-arb-conversion.t
    • ./tests/bugs/distribute/bug-1600379.t
    • ./tests/bugs/glusterd/bug-1699339.t
    • ./tests/bugs/distribute/bug-1786679.t
    • ./tests/bugs/glusterfs/bug-902610.t
    • ./tests/basic/changelog/changelog-rename.t
    • ./tests/basic/distribute/dir-heal.t
    • ./tests/basic/distribute/spare_file_rebalance.t
    • ./tests/basic/namespace.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/hash-tests.diff
git apply hash-tests.diff

  • The following patch fixes test cases failing due to system configuration:

    • ./tests/bugs/shard/bug-1251824.t
    • ./tests/bugs/shard/bug-1468483.t
    • ./tests/basic/distribute/throttle-rebal.t
    • ./tests/bugs/nfs/bug-847622.t
    • ./tests/bugs/glusterfs-server/bug-877992.t
    • ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/test-patch.diff
git apply test-patch.diff

  • Patch run-tests.sh (On SLES 12 SP5 only, see this issue):

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/run-tests.sh.diff
git apply run-tests.sh.diff

  • Fix the ./tests/00-geo-rep/00-georep-verify-non-root-setup.t test case failure (Only on RHEL (7.8, 7.9, 8.2)):

    ln -s /usr/local/lib/libgfchangelog.so.0 /lib64/libgfchangelog.so
ldconfig /usr/local/lib
ldconfig /usr/local/lib64

  • Run the test cases:

    cd $SOURCE_ROOT/glusterfs
export LANG=en_US.UTF-8   # For RHEL 7.9 only
./run-tests.sh

Note:

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test cases:

    • Password-less SSH login has to be set up. Refer to the Setting Up the Environment for Geo-replication in the documentation.
    • The locale of the system must be UTF-8 for the test cases to pass.

  • The is_nfs_export_available function from nfs.rc used in some tests may fail intermittently. You may try to enable rpcbind as that may fix some of the test cases. The community is aware of this behaviour, refer here for more info.

  • For ./tests/bugs/glusterfs/bug-866459.t test case: This test is under investigation, refer to this issue for details.

  • For tests failing due to /mnt/glusterfs/0/: Transport endpoint is not connected, refer to the issue.

  • For ./tests/bugs/glusterfs-server/bug-861542.t test case: If the hostname of the machine is long, the test case will fail. The hostname of the machine can be shortened temporarily via the hostname command to get the test case to pass.

  • The following test case fails if the file system is of type ext4. See issue

    • ./tests/bugs/posix/bug-990028.t

  • The test case ./tests/basic/gfapi/gfapi-copy-file-range.t fails for SLES (12 SP5, 15 SP2) on both x86_64 and s390x due to a system call failure. This is beacuse of the unsupported xfs reflink, it can be enabled via echo 'Y' > /sys/module/xfs/parameters/allow_unsupported.

  • The following test cases fail on RHEL 8.1 due to a kernel bug, updating the kernel via yum update kernel fixes the issue:

    • ./tests/basic/playground/template-xlator-sanity.t
    • ./tests/basic/trace.t
    • ./tests/bugs/distribute/bug-1161311.t

  • Tests under ./tests/bugs/snapshot require the snapd service to be stopped (systemctl stop snapd) if installed:.

  • Certain test cases such as ./tests/bugs/replicate/issue-1254-prioritize-enospc.t will timeout in run-tests.sh. They pass when run individually through prove -vf <testcase>.

  • The test case ./tests/basic/ctime/ctime-utimesat.t fails intermittently.

  • Certain test cases such as ./tests/basic/afr/split-brain-healing.t or ./tests/bugs/index/bug-1559004-EMLINK-handling.t will pass upon rerun or rebooting the system.

  • Additional information on the GlusterFS test framework can be found here.

Step 3: Run the GlusterFS Daemon (Optional)

glusterd

Similar logs should be output in /var/log/glusterfs/glusterd.log:

[2021-05-12 16:43:51.548664 +0000] I [MSGID: 100030] [glusterfsd.c:2687:main] 0-glusterd: Started running version [{arg=glusterd}, {version=9.2}, {cmdlinestr=glusterd}]

Step 4: Test the GlusterFS volume (Optional)

Refer to Documentation for instructions.

Reference

https://www.gluster.org/ https://github.com/gluster/glusterfs http://docs.gluster.org/en/latest/Developer-guide/Building-GlusterFS/

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally