Skip to content

Gluster_sp4

Jump to bottom
Cnaik edited this page Apr 17, 2019 · 5 revisions

Building GlusterFS

Below versions of GlusterFS are available in respective distributions at the time of creation of these build instructions:

  • Ubuntu 16.04 - 3.7.6
  • Ubuntu 18.04 - 3.13.2

The instructions provided below specify the steps to build GlusterFS 4.1.5 on Linux on IBM Z for following distributions:

  • RHEL (7.3, 7.4, 7.5)
  • SLES (12 SP4, 15)
  • Ubuntu (16.04, 18.04)

General Note:

  • When following the steps below please use a super user unless otherwise specified
  • A directory /<source root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Building and Installing GlusterFS

Step 1: Install the dependencies

  • RHEL (7.3, 7.4, 7.5)

    yum install wget git make gcc-c++ libaio-devel boost-devel expat-devel autoconf autoheader automake libtool flex bison openssl-devel libacl-devel sqlite-devel libxml2-devel python-devel python attr yajl nfs-utils xfsprogs popt-static sysvinit-tools psmisc libibverbs-devel librdmacm-devel readline-devel lvm2-devel glib2-devel fuse-devel bc libuuid-devel

  • SLES (12 SP4, 15)

    zypper install wget which git make gcc-c++ libaio-devel boost-devel autoconf automake cmake libtool flex bison lvm2-devel libacl-devel python-devel python attr xfsprogs sysvinit-tools psmisc bc libopenssl-devel libxml2-devel sqlite3 sqlite3-devel popt-devel nfs-utils libyajl2 python-xml net-tools libuuid-devel

  • Ubuntu (16.04)

    apt-get update
apt-get install make automake autoconf libtool flex bison pkg-config libssl-dev libxml2-dev python-dev libaio-dev libibverbs-dev librdmacm-dev libreadline-dev liblvm2-dev libglib2.0-dev liburcu-dev libcmocka-dev libsqlite3-dev libacl1-dev wget tar dbench git xfsprogs attr nfs-common yajl-tools sqlite3 libxml2-utils thin-provisioning-tools bc uuid-dev

  • Ubuntu (18.04)

    apt-get update
apt-get install make automake autoconf libtool flex bison pkg-config libssl1.0-dev libxml2-dev python-dev libaio-dev libibverbs-dev librdmacm-dev libreadline-dev liblvm2-dev libglib2.0-dev liburcu-dev libcmocka-dev libsqlite3-dev libacl1-dev wget tar dbench git xfsprogs attr nfs-common yajl-tools sqlite3 libxml2-utils thin-provisioning-tools bc uuid-dev

  • Install other dependencies

    • Building URCU (ONLY for RHEL and SLES)

      cd /<source_root>/
git clone git://git.liburcu.org/userspace-rcu.git
cd userspace-rcu
./bootstrap 
./configure
make
make install
ldconfig

    • Building thin-provisioning-tools (ONLY for RHEL)

      cd /<source_root>/
git clone https://github.com/jthornber/thin-provisioning-tools
cd thin-provisioning-tools
autoreconf
./configure
make
make install

Step 2: Download , Configure and Build GlusterFS

  • Download and configure source

    cd /<source_root>/
git clone https://github.com/gluster/glusterfs
cd glusterfs
git checkout v4.1.5
./autogen.sh
./configure --enable-gnfs			# for RHEL and Ubuntu
./configure --enable-gnfs --disable-events	# for SLES

  • Update rculist-extra.h (Only for RHEL)

    rm contrib/userspace-rcu/rculist-extra.h
cp /usr/local/include/urcu/rculist.h contrib/userspace-rcu/rculist-extra.h

  • Make Changes to the file /<source_root>/glusterfs/xlators/performance/io-threads/src/io-threads.h (Only for Ubuntu & SLES)

    diff --git a/xlators/performance/io-threads/src/io-threads.h b/xlators/performance/io-threads/src/io-threads.h
index 7124169b7..f9f87c29b 100644
--- a/xlators/performance/io-threads/src/io-threads.h
+++ b/xlators/performance/io-threads/src/io-threads.h
@@ -36,7 +36,7 @@ struct iot_conf;
 #define IOT_DEFAULT_THREADS     16
 #define IOT_MAX_THREADS         64

-#define IOT_THREAD_STACK_SIZE   ((size_t)(256*1024))
+#define IOT_THREAD_STACK_SIZE   ((size_t)(512*1024))

 typedef struct {
		 struct list_head        clients;

  • Build and install

    make
make install

  • Set environment variables

    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
ldconfig

Step 3: Run the test cases (optional)

  • Building xfsprogs

    • Install Dependencies
      • For Ubuntu 
        apt-get install gettext libblkid-dev
      • For RHEL 
        yum install gettext libblkid-devel
      • For SLES 
        zypper install libblkid-devel gettext-tools
    • Build xfsprogs 
      cd /<source_root>/
wget -c https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/xfsprogs/xfsprogs-4.15.1.tar.gz
tar -xvf xfsprogs-4.15.1.tar.gz
cd xfsprogs-4.15.1 
make
make install

  • Building DBENCH (Only for RHEL and SLES)

    cd /<source_root>/
git clone https://github.com/sahlberg/dbench
cd dbench
./autogen.sh
./configure
make
make install

  • Building yajl (Only for SLES)

    cd /<source_root>/
git clone https://github.com/lloyd/yajl
cd yajl
git checkout 2.1.0
mkdir build
cd build/
cmake ..
make
export PATH=$PATH:$PWD/yajl-2.1.0/bin

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/distribute/throttle-rebal.t

    diff --git a/tests/basic/distribute/throttle-rebal.t b/tests/basic/distribute/throttle-rebal.t
index f4823cf4f..1cc3cac5d 100644
--- a/tests/basic/distribute/throttle-rebal.t
+++ b/tests/basic/distribute/throttle-rebal.t
@@ -17,7 +17,7 @@ function set_throttle {
}
  
#Determine number of cores
-cores=$(cat /proc/cpuinfo | grep processor | wc -l)
+cores=$(cat /proc/cpuinfo | grep -w processor | wc -l)
if [ "$cores" == "" ]; then
  echo "Could not get number of cores available"
fi

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/mount-nfs-auth.t

    diff --git a/tests/basic/mount-nfs-auth.t b/tests/basic/mount-nfs-auth.t
index 0d4e01abb..fd13bbcf8 100755
--- a/tests/basic/mount-nfs-auth.t
+++ b/tests/basic/mount-nfs-auth.t
@@ -15,16 +15,17 @@ TEST glusterd
TEST pidof glusterd
TEST $CLI volume info

-H0IP=$(ip addr show |grep -w inet |grep -v 127.0.0.1|awk '{ print $2 }'| cut -d "/" -f 1)
+#H0IP=$(ip addr show |grep -w inet |grep -v 127.0.0.1|awk '{ print $2 }'| cut -d "/" -f 1)
+H0IP=$(hostname -i | awk '{ print $1 }')
 H0IP6=$(host $HOSTNAME | grep IPv6 | awk '{print $NF}')

# Export variables for allow & deny
-EXPORT_ALLOW="/$V0 $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
-EXPORT_ALLOW_SLASH="/$V0/ $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW="/$V0 $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW_SLASH="/$V0/ $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
 EXPORT_DENY="/$V0 1.2.3.4(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"

# Netgroup variables for allow & deny
-NETGROUP_ALLOW="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 ($H0,,)"
+NETGROUP_ALLOW="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 ($H0IP,,)"
NETGROUP_DENY="ngtop ng1000\nng1000 ng999\nng999 ng1\nng1 ng2\nng2 (1.2.3.4,,)"

V0L1="$V0/L1"
@@ -32,8 +33,8 @@ V0L2="$V0L1/L2"
V0L3="$V0L2/L3"

# Other variations for allow & deny
-EXPORT_ALLOW_RO="/$V0 $H0(sec=sys,ro,anonuid=0) @ngtop(sec=sys,ro,anonuid=0)"
-EXPORT_ALLOW_L1="/$V0L1 $H0(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
+EXPORT_ALLOW_RO="/$V0 $H0IP(sec=sys,ro,anonuid=0) @ngtop(sec=sys,ro,anonuid=0)"
+EXPORT_ALLOW_L1="/$V0L1 $H0IP(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"
EXPORT_WILDCARD="/$V0 *(sec=sys,rw,anonuid=0) @ngtop(sec=sys,rw,anonuid=0)"

function build_dirs () {

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/tier/frequency-counters.t

    diff --git a/tests/basic/tier/frequency-counters.t b/tests/basic/tier/frequency-counters.t
index 08e05df12..d12b79c37 100644
--- a/tests/basic/tier/frequency-counters.t
+++ b/tests/basic/tier/frequency-counters.t
@@ -6,8 +6,8 @@
NUM_BRICKS=3
-DEMOTE_FREQ=10
-PROMOTE_FREQ=10
+DEMOTE_FREQ=35
+PROMOTE_FREQ=35

NUM_FILES=5
TEST_DIR=test

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/namespace.t

    diff --git a/tests/basic/namespace.t b/tests/basic/namespace.t
index 25a6b98b4..06dc6dda6 100644
--- a/tests/basic/namespace.t
+++ b/tests/basic/namespace.t
@@ -6,9 +6,9 @@
# These hashes are a result of calling SuperFastHash
# on the corresponding folder names.
-NAMESPACE_HASH=28153613
-NAMESPACE2_HASH=3926991974
-NAMESPACE3_HASH=3493960770
+NAMESPACE_HASH=3253352021
+NAMESPACE2_HASH=458775276
+NAMESPACE3_HASH=1268089390

function check_brick_multiplex() {
		 local ret=$($CLI volume info|grep "cluster.brick-multiplex"|cut -d" " -f2)
@@ -110,12 +110,12 @@ EXPECT_WITHIN 10 "Y" check_samples LOOKUP $NAMESPACE_HASH /namespace patchy0

mkdir -p $N0/namespace{2,3}
EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE2_HASH /namespace2 patchy0
-EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE3_HASH /namespace3 patchy0
+EXPECT_WITHIN 10 "Y" check_samples MKDIR $NAMESPACE3_HASH /namespace3 patchy6

touch $N0/namespace2/file
touch $N0/namespace3/file
EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE2_HASH /namespace2/file patchy0
-EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE3_HASH /namespace3/file patchy0
+EXPECT_WITHIN 10 "Y" check_samples CREATE $NAMESPACE3_HASH /namespace3/file patchy6

truncate -s 0 $N0/namespace/bar
EXPECT_WITHIN 10 "Y" check_samples TRUNCATE $NAMESPACE_HASH /namespace/bar patchy0

  • Make Changes to the file /<source_root>/glusterfs/tests/bugs/distribute/bug-1161311.t

    diff --git a/tests/bugs/distribute/bug-1161311.t b/tests/bugs/distribute/bug-1161311.t
index d88642edc..06c2d230f 100755
--- a/tests/bugs/distribute/bug-1161311.t
+++ b/tests/bugs/distribute/bug-1161311.t
@@ -89,6 +89,8 @@ done
 ## into separate bricks when brick count is 3
 TEST mv $M0/dir1/FILE2 $M0/dir1/FILE1

+brick_loc=$(get_backend_paths $M0/dir1/FILE1)
+
 # unmount and remount the volume
 EXPECT_WITHIN $UMOUNT_TIMEOUT "Y" force_umount $M0
 TEST glusterfs -s $H0 --volfile-id $V0 $M0;
@@ -98,7 +100,7 @@ TEST $CLI volume rebalance $V0 start force

 # Wait for FILE to get the sticky bit on, so that file is under
 # active rebalance, before creating the links
-TEST checksticky $B0/${V0}3/dir1/FILE1
+TEST checksticky $brick_loc

 # Create the links
 ## FILE3 FILE5 FILE7 have hashes, c8c91469 566d26ce 22ce7eba
@@ -119,8 +121,8 @@ cd /
 
 # Ideally for this test to have done its job, the file should still be
 # under migration, so check the sticky bit again
-TEST checksticky $B0/${V0}3/dir1/FILE1

+TEST checksticky $brick_loc
 # Wait for rebalance to complete
 EXPECT_WITHIN $REBALANCE_TIMEOUT "completed" rebalance_status_field $V0

  • Make Changes to the file /<source_root>/glusterfs/tests/bugs/distribute/bug-1193636.t

    diff --git a/tests/bugs/distribute/bug-1193636.t b/tests/bugs/distribute/bug-1193636.t
index ccde02edc..e7b8e9e25 100644
--- a/tests/bugs/distribute/bug-1193636.t
+++ b/tests/bugs/distribute/bug-1193636.t
@@ -2,6 +2,7 @@

 . $(dirname $0)/../../include.rc
 . $(dirname $0)/../../volume.rc
+. $(dirname $0)/../../dht.rc


 checksticky () {
@@ -41,12 +42,13 @@ dd if=/dev/zero of=$M0/dir1/FILE2 bs=64k count=10240
 # act on the file
 TEST mv $M0/dir1/FILE2 $M0/dir1/FILE1

+brick_loc=$(get_backend_paths $M0/dir1/FILE1)
+
 build_tester $(dirname $0)/bug-1193636.c

 TEST $CLI volume rebalance $V0 start force

-TEST checksticky $B0/${V0}3/dir1/FILE1
-
+TEST checksticky $brick_loc
 TEST setfattr -n "user.test1" -v "test1" $M0/dir1/FILE1
 TEST setfattr -n "user.test2" -v "test1" $M0/dir1/FILE1
 TEST setfattr -n "user.test3" -v "test1" $M0/dir1/FILE1

  • Make Changes to the file /<source_root>/glusterfs/tests/bugs/nfs/bug-847622.t

    diff --git a/tests/bugs/nfs/bug-847622.t b/tests/bugs/nfs/bug-847622.t
index 3b836745a..f0b3e0f3d 100755
--- a/tests/bugs/nfs/bug-847622.t
+++ b/tests/bugs/nfs/bug-847622.t
@@ -28,7 +28,7 @@ cd $N0

 # simple getfacl setfacl commands
 TEST touch testfile
-TEST setfacl -m u:14:r testfile
+TEST setfacl -m u:14:r $B0/brick0/testfile
 TEST getfacl testfile

 cd

  • Make Changes to the file /<source_root>/glusterfs/tests/basic/ec/ec-1468261.t

     diff --git a/tests/basic/ec/ec-1468261.t b/tests/basic/ec/ec-1468261.t
index 027310295..61f567a9a 100644
--- a/tests/basic/ec/ec-1468261.t
+++ b/tests/basic/ec/ec-1468261.t
@@ -4,7 +4,7 @@
 # changelog enabled on EC volume.
 ###

-SCRIPT_TIMEOUT=300
+SCRIPT_TIMEOUT=600

 . $(dirname $0)/../../include.rc
 . $(dirname $0)/../../volume.rc

  • Make Changes to the file /<source_root>/glusterfs/run-tests.sh(Only for Ubuntu and SLES, Refer to the issue)

    diff --git a/run-tests.sh b/run-tests.sh
index 8c55972dc..ded762676 100755
--- a/run-tests.sh
+++ b/run-tests.sh
@@ -42,10 +42,10 @@ function check_dependencies()
 fi

     # Check for netstat
-    env netstat --version > /dev/null 2>&1
-    if [ $? -ne 0 ]; then
-        MISSING="$MISSING netstat"
-    fi
+ #    env netstat --version > /dev/null 2>&1
+ #   if [ $? -ne 0 ]; then
+ #       MISSING="$MISSING netstat"
+ #   fi

  • Run the test cases

    cd /<source_root>/glusterfs
./run-tests.sh

Note:

  • If any of the tests fail with error volume create: patchy: failed: Host is not in 'Peer in Cluster' state, make changes to file /etc/hosts by copying the host details entry that is present in the beginning to the section IPv6 capable hosts and replace 0.0.0.0 with the IP address of the host. It should be similar to : 0.0.0.0 .xyz.com hostname

  • If you want to use the kernel NFS server or client on a Gluster storage server, make sure to disable the Gluster NFS server:

    gluster volume set <VOLUME> nfs.disable true

  • For geo-replication test case : Password-less SSH login has to be set up. Refer the Setting Up the Environment for Geo-replication in the Documentation

  • The changes for io-threads.h will allow weighted-rebalance.t test case to pass.

  • Increase the run_timeout value in run-tests.sh to fix tests failing due to timeout issue.

  • The test failures below are observed on s390x due to the little endian/big endian specific hash values calculations and can be ignored. Community is looking into it. Refer to this issue for further info.
    /tests/bugs/posix/bug-1619720.t
    /tests/features/trash.t

  • For bug-1207627-bitrot-scrub-status.t test case : This test failure is under investigation, refer to this issue for details.

  • This test is under investigation : /tests/bugs/glusterfs/bug-866459.t.Refer issue for details.

  • For /tests/bugs/glusterfs/bug-902610.t test case : This feature is not supported by Gluster anymore and it will be removed in the later version. Refer to this issue for details.

  • For ssl-ciphers.t test case(Only for Sles 15) : The test case is failing on s390x as well as on Intel x86. Refer issue for details

  • Few tests from bugs module are failing on Intel x86 as well. Refer issue for details

  • Additional information on GlusterFS test can be found here.

Step 4: Run the GlusterFS Daemon (Optional)

glusterd

Step 5: Testing the GlusterFS volume (Optional)

Refer Documentation for instructions.

Reference

https://www.gluster.org/
https://github.com/gluster/glusterfs
http://docs.gluster.org/en/latest/Developer-guide/Building-GlusterFS/

Clone this wiki locally