Skip to content

Building GlusterFS

Jump to bottom
aborkar-ibm edited this page Nov 22, 2018 · 38 revisions

Building GlusterFS

Below versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • Ubuntu 16.04 - 3.7.6
  • Ubuntu 18.04 - 3.13.2

The instructions provided below specify the steps to build GlusterFS 4.1.5 on Linux on IBM Z for following distributions:

  • RHEL (7.3, 7.4, 7.5)
  • SLES (12 SP3, 15)
  • Ubuntu (16.04, 18.04)

General Note:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Building and Installing GlusterFS

Step 1: Build using script

If you want to build GlusterFS using manual steps, go to STEP 2.

Use the following commands to build GlusterFS using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/GlusterFS/build_glusterfs.sh

# Build GlusterFS
bash build_glusterfs.sh   [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 5. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

Step 2: Install the dependencies

  • RHEL (7.3, 7.4, 7.5)

    yum install wget git make gcc-c++ libaio-devel boost-devel expat-devel autoconf autoheader automake libtool flex bison openssl-devel libacl-devel sqlite-devel libxml2-devel python-devel python attr yajl nfs-utils xfsprogs popt-static sysvinit-tools psmisc libibverbs-devel librdmacm-devel readline-devel lvm2-devel glib2-devel fuse-devel bc libuuid-devel

  • SLES (12 SP3, 15)

    zypper install wget which git make gcc-c++ libaio-devel boost-devel autoconf automake cmake libtool flex bison lvm2-devel libacl-devel python-devel python attr xfsprogs sysvinit-tools psmisc bc libopenssl-devel libxml2-devel sqlite3 sqlite3-devel popt-devel nfs-utils libyajl2 python-xml net-tools libuuid-devel

  • Ubuntu (16.04)

    apt-get update
apt-get install make automake autoconf libtool flex bison pkg-config libssl-dev libxml2-dev python-dev libaio-dev libibverbs-dev librdmacm-dev libreadline-dev liblvm2-dev libglib2.0-dev liburcu-dev libcmocka-dev libsqlite3-dev libacl1-dev wget tar dbench git xfsprogs attr nfs-common yajl-tools sqlite3 libxml2-utils thin-provisioning-tools bc uuid-dev

  • Ubuntu (18.04)

    apt-get update
apt-get install make automake autoconf libtool flex bison pkg-config libssl1.0-dev libxml2-dev python-dev libaio-dev libibverbs-dev librdmacm-dev libreadline-dev liblvm2-dev libglib2.0-dev liburcu-dev libcmocka-dev libsqlite3-dev libacl1-dev wget tar dbench git xfsprogs attr nfs-common yajl-tools sqlite3 libxml2-utils thin-provisioning-tools bc uuid-dev

  • Install other dependencies

    • Building URCU (ONLY for RHEL and SLES)

      cd /<source_root>/
git clone git://git.liburcu.org/userspace-rcu.git
cd userspace-rcu
./bootstrap 
./configure
make
make install
ldconfig

    • Building thin-provisioning-tools (ONLY for RHEL)

      cd /<source_root>/
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
autoreconf
./configure
make
make install

Step 3: Download , Configure and Build GlusterFS

  • Download and configure source

    cd /<source_root>/
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v4.1.5
./autogen.sh
./configure --enable-gnfs			# for RHEL and Ubuntu
./configure --enable-gnfs --disable-events	# for SLES

  • Update rculist-extra.h (Only for RHEL)

    rm contrib/userspace-rcu/rculist-extra.h
cp /usr/local/include/urcu/rculist.h contrib/userspace-rcu/rculist-extra.h

  • Make Changes to the file /<source_root>/glusterfs/xlators/performance/io-threads/src/io-threads.h (Only for Ubuntu & SLES)

    diff --git a/xlators/performance/io-threads/src/io-threads.h b/xlators/performance/io-threads/src/io-threads.h
index 7124169b7..f9f87c29b 100644
--- a/xlators/performance/io-threads/src/io-threads.h
+++ b/xlators/performance/io-threads/src/io-threads.h
@@ -36,7 +36,7 @@ struct iot_conf;
 #define IOT_DEFAULT_THREADS     16
 #define IOT_MAX_THREADS         64

-#define IOT_THREAD_STACK_SIZE   ((size_t)(256*1024))
+#define IOT_THREAD_STACK_SIZE   ((size_t)(512*1024))

 typedef struct {
		 struct list_head        clients;

  • Build and install

    make
make install

  • Set environment variables

    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
ldconfig

Step 4: Run the test cases (optional)

  • Building xfsprogs

    • Install Dependencies
      • For Ubuntu 
        apt-get install gettext libblkid-dev
      • For RHEL 
        yum install gettext libblkid-devel
      • For SLES 
        zypper install libblkid-devel gettext-tools
    • Build xfsprogs 
      cd /<source_root>/
wget -c https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/xfsprogs/xfsprogs-4.15.1.tar.gz
tar -xvf xfsprogs-4.15.1.tar.gz
cd xfsprogs-4.15.1 
make
make install

  • Building DBENCH (Only for RHEL and SLES)

    cd /<source_root>/
git clone https://github.com/sahlberg/dbench
cd dbench
./autogen.sh
./configure
make
make install

  • Building yajl (Only for SLES)

    cd /<source_root>/
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
mkdir build
cd build/
cmake ..
make
export PATH=$PATH:$PWD/yajl-2.1.0/bin

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/distribute/throttle-rebal.t

    diff --git a/tests/basic/distribute/throttle-rebal.t b/tests/basic/distribute/throttle-rebal.t
index f4823cf4f..1cc3cac5d 100644
--- a/tests/basic/distribute/throttle-rebal.t
+++ b/tests/basic/distribute/throttle-rebal.t
@@ -17,7 +17,7 @@ function set_throttle {
}
  
#Determine number of cores
-cores=$(cat /proc/cpuinfo | grep processor | wc -l)
+cores=$(cat /proc/cpuinfo | grep -w processor | wc -l)
if [ "$cores" == "" ]; then
  echo "Could not get number of cores available"
fi

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/mount-nfs-auth.t

    diff --git a/tests/basic/mount-nfs-auth.t b/tests/basic/mount-nfs-auth.t
index 0d4e01abb..fd13bbcf8 100755
--- a/tests/basic/mount-nfs-auth.t
+++ b/tests/basic/mount-nfs-auth.t
@@ -15,16 +15,17 @@ TEST glusterd
TEST pidof glusterd
TEST $CLI volume info

-H0IP=$(ip addr show |grep -w inet |grep -v 127.0.0.1|awk '{ print $2 }'| cut -d "/" -f 1)
+#H0IP=$(ip addr show |grep -w inet |grep -v 127.0.0.1|awk '{ print $2 }'| cut -d "/" -f 1)
+H0IP=$(hostname -i | awk '{ print $1 }')
 H0IP6=$(host $HOSTNAME | grep IPv6 | awk '{print $NF}')

# Export variables for allow & deny
-EXPORT_ALLOW="/$V0 $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
-EXPORT_ALLOW_SLASH="/$V0/ $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW="/$V0 $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW_SLASH="/$V0/ $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
 EXPORT_DENY="/$V0 1.2.3.4(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"

# Netgroup variables for allow & deny
-NETGROUP_ALLOW="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 ($H0,,)"
+NETGROUP_ALLOW="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 ($H0IP,,)"
NETGROUP_DENY="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 (1.2.3.4,,)"

V0L1="$V0/L1"
@@ -32,8 +33,8 @@ V0L2="$V0L1/L2"
V0L3="$V0L2/L3"

# Other variations for allow & deny
-EXPORT_ALLOW_RO="/$V0 $H0(sec=sys,ro,anonuid=0) @ngtop(sec=sys,ro,anonuid=0)"
-EXPORT_ALLOW_L1="/$V0L1 $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW_RO="/$V0 $H0IP(sec=sys,ro,anonuid=0) @ngtop(sec=sys,ro,anonuid=0)"
+EXPORT_ALLOW_L1="/$V0L1 $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
EXPORT_WILDCARD="/$V0 *(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"

function build_dirs () {

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/tier/frequency-counters.t

    diff --git a/tests/basic/tier/frequency-counters.t b/tests/basic/tier/frequency-counters.t
index 08e05df12..d12b79c37 100644
--- a/tests/basic/tier/frequency-counters.t
+++ b/tests/basic/tier/frequency-counters.t
@@ -6,8 +6,8 @@
NUM_BRICKS=3
-DEMOTE_FREQ=10
-PROMOTE_FREQ=10
+DEMOTE_FREQ=35
+PROMOTE_FREQ=35

NUM_FILES=5
TEST_DIR=test

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/namespace.t

    diff --git a/tests/basic/namespace.t b/tests/basic/namespace.t
index 25a6b98b4..06dc6dda6 100644
--- a/tests/basic/namespace.t
+++ b/tests/basic/namespace.t
@@ -6,9 +6,9 @@
# These hashes are a result of calling SuperFastHash
# on the corresponding folder names.
-NAMESPACE_HASH=28153613
-NAMESPACE2_HASH=3926991974
-NAMESPACE3_HASH=3493960770
+NAMESPACE_HASH=3253352021
+NAMESPACE2_HASH=458775276
+NAMESPACE3_HASH=1268089390

function check_brick_multiplex() {
		 local ret=$($CLI volume info|grep "cluster.brick-multiplex"|cut -d" " -f2)
@@ -110,12 +110,12 @@ EXPECT_WITHIN 10 "Y" check_samples LOOKUP $NAMESPACE_HASH /namespace patchy0

mkdir -p $N0/namespace{2,3}
EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE2_HASH /namespace2 patchy0
-EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE3_HASH /namespace3 patchy0
+EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE3_HASH /namespace3 patchy6

touch $N0/namespace2/file
touch $N0/namespace3/file
EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE2_HASH /namespace2/file patchy0
-EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE3_HASH /namespace3/file patchy0
+EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE3_HASH /namespace3/file patchy6

truncate -s 0 $N0/namespace/bar
EXPECT_WITHIN 10 "Y" check_samples TRUNCATE $NAMESPACE_HASH /namespace/bar patchy0

  • Make Changes to the file /<source_root>/glusterfs/run-tests.sh(Only for Ubuntu and SLES, Refer to the issue)

    diff --git a/run-tests.sh b/run-tests.sh
index 8c55972dc..ded762676 100755
--- a/run-tests.sh
+++ b/run-tests.sh
@@ -42,10 +42,10 @@ function check_dependencies()
 fi

     # Check for netstat
-    env netstat --version > /dev/null 2>&1
-    if [ $? -ne 0 ]; then
-        MISSING="$MISSING netstat"
-    fi
+ #    env netstat --version > /dev/null 2>&1
+ #   if [ $? -ne 0 ]; then
+ #       MISSING="$MISSING netstat"
+ #   fi

  • Run the test cases

    cd /<source_root>/glusterfs
./run-tests.sh

Note:

  • If any of the tests fail with error volume create: patchy: failed: Host is not in 'Peer in Cluster' state, make changes to file /etc/hosts by copying the host details entry that is present in the beginning to the section IPv6 capable hosts and replace 0.0.0.0 with the IP address of the host. It should be similar to : 0.0.0.0 .xyz.com hostname

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test case : Password-less SSH login has to be set up. Refer the Setting Up the Environment for Geo-replication in the Documentation

  • For trash.t test case : The test case trash is not enabled by default and is considered an experimental feature in Gluster. Hence, the test failure can be ignored. Refer issue for details

  • For ssl-ciphers.t test case(Only for Sles 15) : The test case is failing on s390x as well as on Intel x86. Refer issue for details

  • Additional information on GlusterFS test can be found here.

Step 5: Run the GlusterFS Daemon (Optional)

glusterd

Step 6: Testing the GlusterFS volume (Optional)

Refer Documentation for instructions.

Reference

https://www.gluster.org/
https://github.com/gluster/glusterfs
http://docs.gluster.org/en/latest/Developer-guide/Building-GlusterFS/

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally