Building Apache Mesos

The instructions provided below specify the steps to build Apache Mesos version 1.7.0 on Linux on IBM Z for the following distributions:

RHEL (7.3, 7.4, 7.5)
SLES (12 SP3, 15)
Ubuntu (16.04, 18.04)

General Notes:

When following the steps below please use a standard permission user unless otherwise specified
A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Step 1: Building and Installing Apache Mesos

1.1) Build using script

If you want to build mesos using manual steps, go to STEP 1.2.

Use the following commands to build mesos using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Mesos/1.7.0/build_mesos.sh

# Build mesos
bash build_mesos.sh

If the build completes successfully, go to STEP 3. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

1.2) Install dependencies

RHEL (7.3, 7.4, 7.5)

With IBM SDK

sudo yum install make git tar wget java-1.8.0-ibm-devel gcc gcc-c++ patch libzip-devel zlib-devel libcurl-devel apr apr-util apr-devel subversion subversion-devel cyrus-sasl-md5 cyrus-sasl-devel python-devel which autoconf automake libtool bzip2 unzip openssl openssl-devel gperftools-devel

With OpenJDK

sudo yum install make git tar wget java-1.8.0-openjdk-devel gcc gcc-c++ patch libzip-devel zlib-devel libcurl-devel apr apr-util apr-devel subversion subversion-devel cyrus-sasl-md5 cyrus-sasl-devel python-devel which autoconf automake libtool bzip2 unzip openssl openssl-devel gperftools-devel

SLES 12 SP3

With IBM SDK

sudo zypper install wget tar gcc gcc-c++ git patch java-1_8_0-ibm java-1_8_0-ibm-devel libzypp-devel libapr1 libapr1-devel subversion subversion-devel cyrus-sasl-devel cyrus-sasl-crammd5 python-devel libclang autoconf automake libtool bzip2 unzip make

With OpenJDK

sudo zypper install wget tar gcc gcc-c++ git patch java-1_8_0-openjdk-devel libzypp-devel libapr1 libapr1-devel subversion subversion-devel cyrus-sasl-devel cyrus-sasl-crammd5 python-devel libclang autoconf automake libtool bzip2 unzip make

SLES 15

With IBM SDK

sudo zypper install wget tar gcc gcc-c++ git patch java-1_8_0-ibm java-1_8_0-ibm-devel libzypp-devel apr-devel libapr1 subversion subversion-devel cyrus-sasl-devel cyrus-sasl-crammd5 python-devel libclang5 autoconf automake libtool bzip2 unzip python-xml

With OpenJDK

sudo zypper install wget tar gcc gcc-c++ git patch java-1_8_0-openjdk-devel libzypp-devel apr-devel libapr1 subversion subversion-devel cyrus-sasl-devel cyrus-sasl-crammd5 python-devel libclang5 autoconf automake libtool bzip2 unzip python-xml

Ubuntu (16.04, 18.04)

With IBM SDK

sudo apt-get update
sudo apt-get install tar wget git build-essential python-dev python-six python-virtualenv libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev libssl-dev autoconf automake libtool bzip2 unzip libgflags-dev libgtest-dev pkg-config clang libc++-dev

Download IBM Java 8 SDK binary from IBM Java 8 and follow the instructions as per given in the link. Update JAVA_HOME and PATH accordingly.

With OpenJDK

sudo apt-get update
sudo apt-get install tar wget git build-essential python-dev python-six python-virtualenv openjdk-8-jdk libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev libssl-dev autoconf automake libtool bzip2 unzip libgflags-dev libgtest-dev pkg-config clang libc++-dev

Install Maven 3.3.9 (on RHEL and SLES only)

cd /<source_root>/
wget http://www-us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
tar zxvf apache-maven-3.3.9-bin.tar.gz
export M2_HOME=/<source_root>/apache-maven-3.3.9
export PATH=/<source_root>/apache-maven-3.3.9/bin:$PATH

Docker is a prerequisite for running the Apache Mesos test cases. Docker packages are provided for SLES and Ubuntu in there respective repositories. Instructions for installing Docker on RHEL can be found here.

Set environment variables only for OpenJDK:

 export JAVA_HOME=/usr/lib/jvm/java-1.8.0					# Only for RHEL
 export JAVA_HOME=/usr/lib64/jvm/java-1.8.0					# Only for SLES
 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-s390x				# Only for Ubuntu
 export JAVA_TOOL_OPTIONS='-Xmx2048M'
 export PATH=$JAVA_HOME/bin:$PATH

Set environment variables only for IBM SDK:

 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-ibm					# Only for RHEL
 export JAVA_HOME=/usr/lib64/jvm/java-1.8.0-ibm					# Only for SLES
 export JVM_DIR=$JAVA_HOME/jre/lib/s390x/default
 export JAVA_TEST_LDFLAGS="-L$JVM_DIR -R$JVM_DIR -Wl,-ljvm -ldl"
 export JAVA_JVM_LIBRARY=$JAVA_HOME/jre/lib/s390x/default/libjvm.so
 export PATH=$JAVA_HOME/bin:$PATH

1.3) Download source code and apply patch as given below

cd /<source_root>/
git clone https://github.com/apache/mesos
cd mesos/
git checkout 1.7.0

Bundling gRPC 1.11.0

cd /<source_root/mesos/3rdparty/
git clone -b v1.11.0 https://github.com/grpc/grpc.git grpc-1.11.0
cd grpc-1.11.0/
git submodule update --init third_party/cares
cd ../
tar zcvf grpc-1.11.0.tar.gz --exclude .git grpc-1.11.0
rm -rf grpc-1.11.0

Modify /<source_root>/mesos/3rdparty/versions.am:

@@ -26,7 +26,7 @@
 GLOG_VERSION = 0.3.3
 GOOGLETEST_VERSION = 1.8.0
 GPERFTOOLS_VERSION = 2.5
-GRPC_VERSION = 1.10.0
+GRPC_VERSION = 1.11.0
 HTTP_PARSER_VERSION = 2.6.2
 JEMALLOC_VERSION = 5.0.1
 LEVELDB_VERSION = 1.19

Modify /<source_root>/mesos/src/python/native_common/ext_modules.py.in:

@@ -126,7 +126,7 @@


     # We link different grpc library variants based on whether SSL is enabled.
-    grpc = os.path.join('3rdparty', 'grpc-1.10.0')
+    grpc = os.path.join('3rdparty', 'grpc-1.11.0')
     grpc_variant = '_unsecure' if '@ENABLE_SSL_TRUE@' == '#' else ''
     libgrpcpp = os.path.join(abs_top_builddir, grpc, 'libs', 'opt', 'libgrpc++%s.a' % grpc_variant)
     libgrpc = os.path.join(abs_top_builddir, grpc, 'libs', 'opt', 'libgrpc%s.a' % grpc_variant)

Append the following lines to file /<source_root>/mesos/3rdparty/protobuf-3.5.0.patch:

diff --git a/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h b/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h
index 0b0b06c..075c406 100644
--- a/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h
+++ b/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h
@@ -146,6 +146,14 @@ inline Atomic64 NoBarrier_Load(volatile const Atomic64* ptr) {
   return __atomic_load_n(ptr, __ATOMIC_RELAXED);
 }

+inline Atomic64 Release_CompareAndSwap(volatile Atomic64* ptr,
+                                       Atomic64 old_value,
+                                       Atomic64 new_value) {
+  __atomic_compare_exchange_n(ptr, &old_value, new_value, false,
+                              __ATOMIC_RELEASE, __ATOMIC_ACQUIRE);
+  return old_value;
+}
+
 #endif // defined(__LP64__)

 }  // namespace internal

1.4) Build and Install Apache Mesos

cd /<source_root>/mesos
./bootstrap
mkdir build
cd build
../configure
make
# Install (Optional)
sudo make install

NOTE: In case of Java out of Memory issue while building, do the following modification to file /<source_root>/mesos/src/java/mesos.pom.in and run make command again.

@@ -84,6 +84,7 @@
       <plugin>
         <artifactId>maven-javadoc-plugin</artifactId>
         <configuration>
+          <maxmemory>512m</maxmemory>
           <sourcepath>@abs_top_srcdir@/src/java/src:@abs_top_builddir@/src/java/generated</sourcepath>
           <subpackages>org.apache.mesos</subpackages>
         </configuration>

Step 2: Testing (Optional)

NOTE: Test cases need to be executed as root user.

2.1) Steps to prepare images required for Apache Mesos test cases

Test cases of Apache Mesos uses the following images: alpine, mesosphere/alpine-expect, mesosphere/inky, mesosphere/test-executor, tnachen/test-executor, haosdent/https-server, zhq527725/https-server, and zhq527725/whiteout. Dockerfiles for each of these images is given below. Make sure all the images are present on your machine before you start testing.

Dockerfile for image mesosphere/alpine-expect

FROM s390x/debian

RUN apt-get update
RUN apt-get install -y expect

Dockerfile for image mesosphere/inky

FROM s390x/busybox

CMD ["inky"]
ENTRYPOINT ["echo"]

Dockerfile for both the images mesosphere/test-executor and tnachen/test-executor

FROM s390x/debian
RUN apt-get update
RUN apt-get install -y wget tar git make libnss-myhostname
WORKDIR /root
RUN wget https://storage.googleapis.com/golang/go1.7.1.linux-s390x.tar.gz
RUN tar -xzf go1.7.1.linux-s390x.tar.gz
ENV PATH $PATH:/root/go/bin
ENV GOROOT /root/go
ENV GOPATH /root
ENV CC gcc
RUN mkdir -p src/github.com/mesos/mesos-go
WORKDIR /root/src
RUN git clone https://github.com/tnachen/go-mesos.git github.com/mesos/mesos-go
WORKDIR /root/src/github.com/mesos/mesos-go/examples/
RUN go get github.com/tools/godep
ENV PATH $PATH:/root/bin
RUN godep restore && go install ./...
RUN sed -i 's/RACE := -race//g' Makefile
RUN sed -i 's/$(RACE)//g' Makefile
RUN make
RUN cp /root/src/github.com/mesos/mesos-go/examples/_output/executor /bin/test-executor

Dockerfile for image zhq527725/whiteout

FROM s390x/busybox
RUN mkdir -p /dir1/dir2 && touch /dir1/file1 && touch /dir1/dir2/file2
RUN rm -rf /dir1/file1 && rm -rf /dir1/dir2 && mkdir /dir1/dir2 && touch /dir1/dir2/file3

Steps for building both haosdent/https-server and zhq527725/https-server images

For haosdent/https-server, download files as given below

git clone https://github.com/haosdent/https-server
cd https-server

For zhq527725/https-server, download files as given below

git clone https://github.com/qianzhangxa/https-server.git
cd https-server

Replace the contents of Dockerfile in each of the https-server folders with the following

FROM s390x/debian
RUN apt-get update
RUN apt-get install -y wget tar xz-utils gcc make patch tcl-tls libssl-dev
RUN wget https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tar.xz
RUN tar -xvf Python-2.7.13.tar.xz
WORKDIR /Python-2.7.13
RUN ./configure --prefix=/usr/ --exec-prefix=/usr/
RUN make
RUN make install
WORKDIR /tmp
ADD https_server.py ./
ADD server.pem ./

Command to create image from Dockerfile is
```
docker build -t <name_of_image> .
```

Steps to save images to /opt/docker/images

mkdir -p /opt/docker/images
docker pull s390x/debian
docker tag s390x/debian debian
docker save s390x/debian >> /opt/docker/images/debian.tar
docker pull s390x/alpine
docker tag s390x/alpine alpine
docker save alpine >> /opt/docker/images/alpine.tar
docker pull s390x/hello-world
docker tag s390x/hello-world hello-world
docker save hello-world >> /opt/docker/images/hello-world.tar
# Save the mesosphere/inky image created in previous step
mkdir -p /opt/docker/images/mesosphere
docker save mesosphere/inky >> /opt/docker/images/mesosphere/inky.tar
mkdir -p /opt/docker/images/haosdent/
docker save haosdent/https-server >> /opt/docker/images/haosdent/https-server.tar
mkdir -p /opt/docker/images/zhq527725/
docker save zhq527725/https-server >> /opt/docker/images/zhq527725/https-server.tar
docker save zhq527725/whiteout >> /opt/docker/images/zhq527725/whiteout.tar

2.2) Modify the following files as given below

Modify /<source_root>/mesos/src/slave/flags.cpp

@@ -208,7 +208,7 @@
   	"path (e.g., `/tmp/docker/images`), or as an HDFS URI (*experimental*)\n"
   	"(e.g., `hdfs://localhost:8020/archives/`). Note that this option won't\n"
   	"change the default registry server for Docker containerizer.",
-  	"https://registry-1.docker.io");
+  	"/opt/docker/images");

   add(&Flags::docker_store_dir,
    	"docker_store_dir",

Modify /<source_root>/mesos/src/tests/containerizer/provisioner_docker_tests.cpp

@@ -1083,7 +1083,7 @@

   Image image;
   image.set_type(Image::DOCKER);
-  image.mutable_docker()->set_name("library/alpine@" + digest);
+  image.mutable_docker()->set_name("debian" + digest);

   ContainerInfo* container = task.mutable_container();
   container->set_type(ContainerInfo::MESOS);

Modify /<source_root>/mesos/src/tests/containerizer/runtime_isolator_tests.cpp

@@ -411,7 +411,7 @@
       v1::createCommandInfo("/bin/ls", {"ls", "-al", "/"}));

   taskInfo.mutable_container()->CopyFrom(
-      v1::createContainerInfo("library/alpine"));
+      v1::createContainerInfo("debian"));

   Future<Event::Update> updateStarting;
   Future<Event::Update> updateRunning;

Modify /<source_root>/mesos/src/tests/health_check_tests.cpp

@@ -1393,7 +1393,7 @@

  // Use Netcat to launch a HTTP server.
  const string command = strings::format(
-      "nc -lk -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
+      "nc -ll -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
       testPort).get();

   TaskInfo task = createTask(offers.get()[0], command);
@@ -1586,7 +1586,7 @@

   // Use Netcat to launch a HTTP server.
   const string command = strings::format(
-      "nc -lk -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
+      "nc -ll -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
       testPort).get();

   TaskInfo task = createTask(offers.get()[0], command);

Modify /<source_root>/mesos/src/tests/containerizer/docker_containerizer_tests.cpp

@@ -3961,7 +3961,7 @@
   // Launch a HTTP server until SIGTERM is received, then sleep for
   // 15 seconds to let the health check fail.
   const string command = strings::format(
-      "trap \"sleep 15\" SIGTERM && nc -lk -p %u -e echo",
+      "trap \"sleep 15\" SIGTERM && nc -ll -p %u -e echo",
       testPort).get();
 #endif // __WINDOWS__

2.3) Run test cases

cd /<source_root>/mesos/build
make check -k

NOTE:

1. In case of any test case failures, execute the failed test case individually as given below

cd /<source_root>/mesos/build
./bin/mesos-tests.sh --gtest_filter=<test_case>

2. If the docker containers on your machine takes longer time to release the resources after exiting, there are chances of following 2 test case failures: DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping, DockerTest.ROOT_DOCKER_CheckPortResource

3. In case of the following test case failures: CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen, execute swapoff -a command on your machine.

4. Make sure /etc/hosts file have entry with following details: <ipaddress> <Fully Qualified Domain Name> <hostname>

5.ContainerizerTest.ROOT_CGROUPS_BalloonFramework test case fails if 5432 port is already in use. Port can be changed in file /<source_root>/mesos/src/tests/balloon_framework_test.sh

6. Test cases CniIsolatorTest.ROOT_*, CniIsolatorPortMapperTest.ROOT_IPTABLES_NC_PortMapper, NetworkParam/DefaultExecutorCniTest.ROOT_VerifyContainerIP and DefaultContainerDNSInfo/DefaultContainerDNSCniTest.ROOT_VerifyDefaultDNS has been known to fail in SLES for both s390x and x86. https://issues.apache.org/jira/browse/MESOS-8364

Step 3: Run Apache Mesos

3.1) Start master

cd /<source_root>/mesos/build
sudo ./bin/mesos-master.sh --ip=<ip_address> --work_dir=/var/lib/mesos

NOTE: Ensure work directory exists and has required permissions

3.2) Start slave

cd /<source_root>/mesos/build
sudo ./bin/mesos-agent.sh --master=<ip_address>:5050 --work_dir=/var/lib/mesos

3.3) Web UI

Open http://\<ip_address\>:5050 in your browser.

Reference:

http://mesos.apache.org/

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Building Apache Mesos

Building Apache Mesos

Step 1: Building and Installing Apache Mesos

1.1) Build using script

1.2) Install dependencies

1.3) Download source code and apply patch as given below

1.4) Build and Install Apache Mesos

Step 2: Testing (Optional)

2.1) Steps to prepare images required for Apache Mesos test cases

2.2) Modify the following files as given below

2.3) Run test cases

Step 3: Run Apache Mesos

3.1) Start master

3.2) Start slave

3.3) Web UI

Reference:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!