Skip to content

Building GlusterFS

Jump to bottom
aborkar-ibm edited this page Mar 23, 2022 · 39 revisions

Building GlusterFS

Following versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • RHEL (8.2, 8.4, 8.5) has 6.0
  • SLES 15 SP3 has 3.12.3
  • Ubuntu 18.04 has 3.13.2
  • Ubuntu 20.04 has 7.2
  • Ubuntu 21.10 has 9.2

The instructions provided below specify the steps to build or install GlusterFS 10.1 on Linux on IBM Z for following distributions:

  • RHEL (8.2, 8.4, 8.5)
  • SLES 15 SP3

Note: For Ubuntu (18.04, 20.04, 21.10), please refer to this link to install GlusterFS from the community GlusterFS PPA (ppa:gluster/glusterfs-10).

General Notes:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Step 1: Build and Install GlusterFS

Step 1.1: Build using script

If you want to build GlusterFS using manual steps, go to Step 1.2.

Use the following commands to build GlusterFS using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/10.1/build_glusterfs.sh
# Build GlusterFS
bash build_glusterfs.sh   [Provide -h option to check usage]

If the build completes successfully, go to STEP 3. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

Step 1.2: Install the dependencies

  • Set environment variables:

    export SOURCE_ROOT=/<source_root>/
export PREFIX=/usr/local
export PATCH_URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/10.1/patch

PATH=${PREFIX}/bin:${PREFIX}/sbin${PATH:+:${PATH}}
export PATH

PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PREFIX}/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
export PKG_CONFIG_PATH

LD_LIBRARY_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export LD_LIBRARY_PATH

LD_RUN_PATH=${PREFIX}/lib:${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
export LD_RUN_PATH

  • RHEL (8.2, 8.4, 8.5)

    yum install -y autoconf automake bison bzip2 flex fuse-devel gcc-c++ git glib2-devel libacl-devel libaio-devel libibverbs-devel librdmacm-devel libtool libxml2-devel libuuid-devel liburing-devel lvm2 make binutils openssl-devel pkgconfig python3 readline-devel wget zlib-devel tar gzip libtirpc-devel patch rpcgen userspace-rcu-devel which diffutils xz gperftools gperf

  • SLES (15 SP3)

    zypper install -y autoconf automake bison cmake flex fuse-devel gcc-c++ git-core glib2-devel libacl-devel libaio-devel librdmacm1 libtool liburcu-devel libuuid-devel libxml2-devel lvm2 make pkg-config python3 python3-xattr rdma-core-devel readline-devel openssl-devel zlib-devel which gawk dmraid popt-devel gperftools-devel gperf gperftools libtirpc-devel rpcgen liburing-devel

Step 1.3: Download, Configure and Build GlusterFS

  • Download and configure source

    Note: : If git clone fails with SSL certificate issue, execute git config --global http.sslVerify false command and then do git clone.

    cd $SOURCE_ROOT
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v10.1
./autogen.sh
./configure --enable-gnfs                                              # For RHEL 
./configure --enable-gnfs --disable-linux-io_uring                     # For SLES 15 SP3

  • Patch xlators/performance/io-threads/src/io-threads.h:

    This patch fixes the ./tests/features/weighted-rebalance.t test case failure.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/io-threads.h.diff
git apply io-threads.h.diff

  • Patch xlators/features/bit-rot/src/stub/bit-rot-stub.c:

    This patch fixes the ./tests/bitrot/bug-1207627-bitrot-scrub-status.t test case failure.

    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/bit-rot-stub.c.diff
git apply bit-rot-stub.c.diff

  • Build and install

    make
make install

Step 2: Run the test cases (optional)

2.1) Install the testing dependencies

  • RHEL (8.2, 8.4, 8.5)

    yum install -y acl attr bc bind-utils boost-devel docbook-style-xsl expat-devel gdb net-tools nfs-utils psmisc vim xfsprogs yajl redhat-rpm-config python3-devel python3-pyxattr python3-prettytable perl-Test-Harness popt-devel

  • SLES (15 SP3)

    zypper install -y acl attr bc bind-utils gdb libxml2-tools net-tools-deprecated nfs-utils psmisc thin-provisioning-tools vim xfsprogs python3-xattr python3-PrettyTable libselinux-devel selinux-tools popt-devel

  • Link pstack command (For SLES only)

    ln -sf `which gstack` /usr/bin/pstack

  • Build DBENCH

    cd $SOURCE_ROOT
git clone https://github.com/sahlberg/dbench
cd dbench
git checkout caa52d347171f96eef5f8c2d6ab04a9152eaf113
./autogen.sh
./configure --datadir=/usr/local/share/doc/loadfiles/
make
make install

  • Build thin-provisioning-tools (For RHEL only)

    cd $SOURCE_ROOT
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
git checkout v0.7.6
autoreconf
./configure
make
make install

  • Build yajl (For SLES only)

    cd $SOURCE_ROOT
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
./configure
make install

2.2) Patch and run tests

  • The following patch fixes the test cases listed below that fail due to hardcoded hash values for little-endian systems:

    • ./tests/bugs/posix/bug-1619720.t
    • ./tests/bugs/replicate/bug-1655854-support-dist-to-rep3-arb-conversion.t
    • ./tests/bugs/distribute/bug-1600379.t
    • ./tests/bugs/glusterd/bug-1699339.t
    • ./tests/bugs/distribute/bug-1786679.t
    • ./tests/bugs/glusterfs/bug-902610.t
    • ./tests/basic/changelog/changelog-rename.t
    • ./tests/basic/distribute/dir-heal.t
    • ./tests/basic/distribute/spare_file_rebalance.t
    • ./tests/basic/namespace.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/hash-tests.diff
git apply hash-tests.diff

  • The following patch fixes test cases failing due to system configuration:

    • ./tests/bugs/shard/bug-1251824.t
    • ./tests/bugs/shard/bug-1468483.t
    • ./tests/basic/distribute/throttle-rebal.t
    • ./tests/bugs/nfs/bug-847622.t
    • ./tests/bugs/glusterfs-server/bug-877992.t
    • ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
    cd $SOURCE_ROOT/glusterfs
wget --no-check-certificate $PATCH_URL/test-patch.diff
git apply test-patch.diff 
wget --no-check-certificate $PATCH_URL/test-patch_sles.diff  (For SLES only)
git apply test-patch_sles.diff     (For SLES only)

  • Fix the ./tests/00-geo-rep/00-georep-verify-non-root-setup.t test case failure (For RHEL only):

    ln -s /usr/local/lib/libgfchangelog.so.0 /lib64/libgfchangelog.so
ldconfig /usr/local/lib
ldconfig /usr/local/lib64

  • Run the test cases:

    cd $SOURCE_ROOT/glusterfs
./run-tests.sh

Note:

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test cases:

    • Password-less SSH login has to be set up. Refer to the Setting Up the Environment for Geo-replication in the documentation.
    • The locale of the system must be UTF-8 for the test cases to pass.

  • The is_nfs_export_available function from nfs.rc used in some tests may fail intermittently. You may try to enable rpcbind as that may fix some of the test cases. The community is aware of this behaviour, refer here for more info.

  • For ./tests/bugs/glusterfs/bug-866459.t test case: This test is under investigation, refer to this issue for details.

  • For tests failing due to /mnt/glusterfs/0/: Transport endpoint is not connected, refer to the issue.

  • For ./tests/bugs/glusterfs-server/bug-861542.t test case: If the hostname of the machine is long, the test case will fail. The hostname of the machine can be shortened temporarily via the hostname command to get the test case to pass.

  • The following test case fails if the file system is of type ext4. See issue

    • ./tests/bugs/posix/bug-990028.t

  • Tests under ./tests/bugs/snapshot require the snapd service to be stopped (systemctl stop snapd) if installed:.

  • Certain test cases such as ./tests/bugs/replicate/issue-1254-prioritize-enospc.t will timeout in run-tests.sh. They pass when run individually through prove -vf <testcase>.

  • Test case ./tests/bugs/glusterd/optimized-basic-testcases.t and ./tests/bugs/glusterfs/bug-879490.t fails on RHEL for x86_64 and s390x , refer to the issue-3300 and issue-3301 respectively.

  • Test case ./tests/bugs/nfs/bug-1157223-symlink-mounting.t fails on SLES 15 Sp3, refer issue for details.

  • The test case ./tests/basic/ctime/ctime-utimesat.t fails intermittently.

  • Additional information on the GlusterFS test framework can be found here.

Step 3: Run the GlusterFS Daemon (Optional)

glusterd

Similar logs should be output in /var/log/glusterfs/glusterd.log:

[2022-03-14 07:24:05.654438 +0000] I [MSGID: 100030] [glusterfsd.c:2769:main] 0-glusterd: Started running version [{arg=glusterd}, {version=10.1}, {cmdlinestr=glusterd}]

Step 4: Test the GlusterFS volume (Optional)

Refer to Documentation for instructions.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally