Skip to content

Building Apache Mesos

Jump to bottom
Cindy Lee edited this page May 28, 2018 · 28 revisions

Building Apache Mesos

The instructions provided below specify the steps to build Apache Mesos version 1.5.0 on Linux on IBM Z for the following distributions:

  • RHEL (7.3, 7.4)
  • SLES (12 SP2, 12 SP3)
  • Ubuntu (16.04, 18.04)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified

  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it

Step 1: Building and Installing Apache Mesos

1.1) Install dependencies

  • RHEL (7.3, 7.4)

    • With IBM SDK

      sudo yum install make git tar wget java-1.8.0-ibm-devel.s390x gcc gcc-c++ patch libzip-devel zlib-devel libcurl-devel apr apr-util apr-devel subversion subversion-devel cyrus-sasl-md5 cyrus-sasl-devel python-devel which autoconf automake libtool bzip2 unzip
      Set environment variables
      export JAVA_HOME=/usr/lib/jvm/java-1.8.0-ibm
export PATH=$JAVA_HOME/bin:$PATH

    • With OpenJDK

      sudo yum install make git tar wget java-1.8.0-openjdk-devel gcc gcc-c++ patch libzip-devel zlib-devel libcurl-devel apr apr-util apr-devel subversion subversion-devel cyrus-sasl-md5 cyrus-sasl-devel python-devel which autoconf automake libtool bzip2 unzip
      Set environment variables
      export JAVA_HOME=/usr/lib/jvm/java-1.8.0
export PATH=$JAVA_HOME/bin:$PATH

  • SLES (12 SP2, 12 SP3)

    • With IBM SDK

      sudo zypper install wget tar gcc gcc-c++ git patch java-1_8_0-ibm java-1_8_0-ibm-devel libzypp-devel libapr1 libapr1-devel subversion subversion-devel cyrus-sasl-devel cyrus-sasl-crammd5 python-devel libclang autoconf automake libtool bzip2 unzip
      Set environment variables
      export JAVA_HOME=/usr/lib64/jvm/java-1.8.0-ibm
export PATH=$JAVA_HOME/bin:$PATH

    • With OpenJDK

      sudo zypper install wget tar gcc gcc-c++ git patch java-1_8_0-openjdk-devel libzypp-devel libapr1 libapr1-devel subversion subversion-devel cyrus-sasl-devel cyrus-sasl-crammd5 python-devel libclang autoconf automake libtool bzip2 unzip
      Set environment variables
      export JAVA_HOME=/usr/lib64/jvm/java-1.8.0
export PATH=$JAVA_HOME/bin:$PATH

  • Ubuntu (16.04, 18.04)

    • With IBM SDK

      sudo apt-get update
sudo apt-get install tar wget git build-essential python-dev python-six python-virtualenv libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev libssl-dev autoconf automake libtool bzip2 unzip

      Download IBM Java 8 sdk binary from IBM Java 8 and follow the instructions as per given in the link. Update JAVA_HOME and PATH accordingly.

    • With OpenJDK

      sudo apt-get update
sudo apt-get install tar wget git build-essential python-dev python-six python-virtualenv openjdk-8-jdk libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev libssl-dev autoconf automake libtool bzip2 unzip
      Set environment variables
      export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-s390x/
export PATH=$JAVA_HOME/bin:$PATH
export JAVA_TOOL_OPTIONS='-Xmx2048M'	#Only for OpenJDK

  • Install Maven using the following steps (on RHEL and SLES only)

    cd /<source_root>/
wget http://www-us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
tar zxvf apache-maven-3.3.9-bin.tar.gz
export PATH=/<source_root>/apache-maven-3.3.9/bin:$PATH

  • Docker is a prerequisite for Apache Mesos test cases. Instructions for installing Docker can be found here.

  • Set environment variables only for IBM SDK:

    export JVM_DIR=$JAVA_HOME/jre/lib/s390x/default
export JAVA_TEST_LDFLAGS="-L$JVM_DIR -R$JVM_DIR -Wl,-ljvm -ldl"
export JAVA_JVM_LIBRARY=$JAVA_HOME/jre/lib/s390x/default/libjvm.so
1.2) Download source code and apply patch as given below
cd /<source_root>/
git clone https://github.com/apache/mesos
cd mesos/
git checkout 1.5.0

Append the following lines to file /<source_root>/mesos/3rdparty/protobuf-3.5.0.patch:

diff --git a/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h b/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h
index 0b0b06c..075c406 100644
--- a/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h
+++ b/src/google/protobuf/stubs/atomicops_internals_generic_gcc.h
@@ -146,6 +146,14 @@ inline Atomic64 NoBarrier_Load(volatile const Atomic64* ptr) {
   return __atomic_load_n(ptr, __ATOMIC_RELAXED);
 }

+inline Atomic64 Release_CompareAndSwap(volatile Atomic64* ptr,
+                                       Atomic64 old_value,
+                                       Atomic64 new_value) {
+  __atomic_compare_exchange_n(ptr, &old_value, new_value, false,
+                              __ATOMIC_RELEASE, __ATOMIC_ACQUIRE);
+  return old_value;
+}
+
 #endif // defined(__LP64__)

 }  // namespace internal
1.3) Build and Install Apache Mesos
cd /<source_root>/mesos
./bootstrap
mkdir build
cd build
../configure
make
# Install (Optional)
sudo make install

NOTE: In case of Java out of Memory issue while building, do the following modification to file /<source_root>/mesos/src/java/mesos.pom.in and run make command again.

@@ -131,6 +131,7 @@
       <plugin>
         <artifactId>maven-javadoc-plugin</artifactId>
         <configuration>
+          <maxmemory>512m</maxmemory>
           <sourcepath>@abs_top_srcdir@/src/java/src:@abs_top_builddir@/src/java/generated</sourcepath>
           <subpackages>org.apache.mesos</subpackages>

Step 2: Testing (Optional)

NOTE: Test cases need to be executed as root user.

2.1) Steps to prepare images required for Apache Mesos test cases

Test cases of Apache Mesos uses the following images: alpine, mesosphere/alpine-expect, mesosphere/inky, mesosphere/test-executor, tnachen/test-executor, haosdent/https-server, zhq527725/https-server, and zhq527725/whiteout. Dockerfiles for each of these images is given below. Make sure all the images are present on your machine before you start testing.

  • Dockerfile for image mesosphere/alpine-expect

    FROM s390x/debian

RUN apt-get update
RUN apt-get install -y expect

  • Dockerfile for image mesosphere/inky

    FROM s390x/busybox

CMD ["inky"]
ENTRYPOINT ["echo"]

  • Dockerfile for both the images mesosphere/test-executor and tnachen/test-executor

    FROM s390x/debian
RUN apt-get update
RUN apt-get install -y wget tar git make libnss-myhostname
WORKDIR /root
RUN wget https://storage.googleapis.com/golang/go1.7.1.linux-s390x.tar.gz
RUN tar -xzf go1.7.1.linux-s390x.tar.gz
ENV PATH $PATH:/root/go/bin
ENV GOROOT /root/go
ENV GOPATH /root
ENV CC gcc
RUN mkdir -p src/github.com/mesos/mesos-go
WORKDIR /root/src
RUN git clone https://github.com/tnachen/go-mesos.git github.com/mesos/mesos-go
WORKDIR /root/src/github.com/mesos/mesos-go/examples/
RUN go get github.com/tools/godep
ENV PATH $PATH:/root/bin
RUN godep restore && go install ./...
RUN sed -i 's/RACE := -race//g' Makefile
RUN sed -i 's/$(RACE)//g' Makefile
RUN make
RUN cp /root/src/github.com/mesos/mesos-go/examples/_output/executor /bin/test-executor

  • Dockerfile for image zhq527725/whiteout

    FROM s390x/busybox
RUN mkdir -p /dir1/dir2 && touch /dir1/file1 && touch /dir1/dir2/file2
RUN rm -rf /dir1/file1 && rm -rf /dir1/dir2 && mkdir /dir1/dir2 && touch /dir1/dir2/file3

  • Steps for building both haosdent/https-server and zhq527725/https-server images

    • For haosdent/https-server, download files as given below

      git clone https://github.com/haosdent/https-server
cd https-server

    • For zhq527725/https-server, download files as given below

      git clone https://github.com/qianzhangxa/https-server.git
cd https-server

    • Replace the contents of Dockerfile in each of the https-server folders with the following

      FROM s390x/debian
RUN apt-get update
RUN apt-get install -y wget tar xz-utils gcc make patch tcl-tls libssl-dev
RUN wget https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tar.xz
RUN tar -xvf Python-2.7.13.tar.xz
WORKDIR /Python-2.7.13
RUN ./configure --prefix=/usr/ --exec-prefix=/usr/
RUN make
RUN make install
WORKDIR /tmp
ADD https_server.py ./
ADD server.pem ./

  • Command to create image from Dockerfile is

    docker build -t <name_of_image> .

  • Steps to save images to /opt/docker/images

    mkdir -p /opt/docker/images
docker pull s390x/debian
docker tag s390x/debian debian
docker save s390x/debian >> /opt/docker/images/debian.tar
docker pull s390x/alpine
docker tag s390x/alpine alpine
docker save alpine >> /opt/docker/images/alpine.tar
docker pull s390x/hello-world
docker tag s390x/hello-world hello-world
docker save hello-world >> /opt/docker/images/hello-world.tar
# Save the mesosphere/inky image created in previous step
mkdir -p /opt/docker/images/mesosphere
docker save mesosphere/inky >> /opt/docker/images/mesosphere/inky.tar
mkdir -p /opt/docker/images/haosdent/
docker save haosdent/https-server >> /opt/docker/images/haosdent/https-server.tar
mkdir -p /opt/docker/images/zhq527725/
docker save zhq527725/https-server >> /opt/docker/images/zhq527725/https-server.tar
docker save zhq527725/whiteout >> /opt/docker/images/zhq527725/whiteout.tar
2.2) Modify the following files as given below
  • Modify /<source_root>/mesos/src/slave/flags.cpp 
    @@ -205,7 +205,7 @@ mesos::internal::slave::Flags::Flags()
     "or a local path (i.e: `/tmp/docker/images`) in which Docker image\n"
     "archives (result of `docker save`) are stored. Note that this option\n"
     "won't change the default registry server for Docker containerizer.",
-      "https://registry-1.docker.io");
+      "/opt/docker/images");

 add(&Flags::docker_store_dir,
     "docker_store_dir",
  • Modify /<source_root>/mesos/src/tests/containerizer/provisioner_docker_tests.cpp 
    @@ -920,7 +920,7 @@ TEST_F(ProvisionerDockerTest, ROOT_INTERNET_CURL_ImageDigest)

 Image image;
 image.set_type(Image::DOCKER);
-  image.mutable_docker()->set_name("library/alpine@" + digest);
+  image.mutable_docker()->set_name("debian" + digest);

 ContainerInfo* container = task.mutable_container();
 container->set_type(ContainerInfo::MESOS);
  • Modify /<source_root>/mesos/src/tests/containerizer/runtime_isolator_tests.cpp 
    @@ -411,7 +411,7 @@ TEST_F(DockerRuntimeIsolatorTest, ROOT_INTERNET_CURL_NestedSimpleCommand)
      v1::createCommandInfo("/bin/ls", {"ls", "-al", "/"}));

  taskInfo.mutable_container()->CopyFrom(
-      v1::createContainerInfo("library/alpine"));
+      v1::createContainerInfo("debian"));

  Future<Event::Update> updateStarting;
  Future<Event::Update> updateRunning;
  • Modify /<source_root>/mesos/src/tests/health_check_tests.cpp 
    @@ -1686,7 +1686,7 @@ TEST_F(HealthCheckTest, ROOT_INTERNET_CURL_HealthyTaskViaHTTPWithContainerImage)

   // Use Netcat to launch a HTTP server.
   const string command = strings::format(
-      "nc -lk -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
+      "nc -ll -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
       testPort).get();

   TaskInfo task = createTask(offers.get()[0], command);
@@ -1879,7 +1879,7 @@ TEST_F(HealthCheckTest, ROOT_INTERNET_CURL_HealthyTaskViaTCPWithContainerImage)

   // Use Netcat to launch a HTTP server.
   const string command = strings::format(
-      "nc -lk -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
+      "nc -ll -p %u -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"",
       testPort).get();

   TaskInfo task = createTask(offers.get()[0], command);
@@ -2290,7 +2290,7 @@ TEST_P_TEMP_DISABLED_ON_WINDOWS(
   // Use Netcat to launch a HTTP server.
   TaskInfo task = createTask(
       offers.get()[0],
-      "nc -lk -p 80 -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"");
+      "nc -ll -p 80 -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"");

   // TODO(tnachen): Use local image to test if possible.
   ContainerInfo containerInfo;
@@ -2544,7 +2544,7 @@ TEST_P_TEMP_DISABLED_ON_WINDOWS(
   // Use Netcat to launch a HTTP server.
   TaskInfo task = createTask(
       offers.get()[0],
-      "nc -lk -p 80 -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"");
+      "nc -ll -p 80 -e echo -e \"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\"");

   // TODO(tnachen): Use local image to test if possible.
   ContainerInfo containerInfo;
  • Modify /<source_root>/mesos/src/tests/containerizer/docker_containerizer_tests.cpp 
    @@ -4049,7 +4049,7 @@ TEST_F(DockerContainerizerTest, ROOT_DOCKER_NoTransitionFromKillingToRunning)
   // Launch a HTTP server until SIGTERM is received, then sleep for
   // 15 seconds to let the health check fail.
   const string command = strings::format(
-      "trap \"sleep 15\" SIGTERM && nc -lk -p %u -e echo",
+      "trap \"sleep 15\" SIGTERM && nc -ll -p %u -e echo",
       testPort).get();

   TaskInfo task = createTask(offers->front(), command);
2.3) Run test cases
cd /<source_root>/mesos/build
make check -k

NOTE:

1. In case of any test case failures, execute the failed test case individually as given below

cd /<source_root>/mesos/build
./bin/mesos-tests.sh --gtest_filter=<test_case>

2. If the docker containers on your machine takes longer time to release the resources after exiting, there are chances of following 2 test case failures: DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping, DockerTest.ROOT_DOCKER_CheckPortResource

3. In case of the following test case failures: CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen, CgroupsAnyHierarchyMemoryPressureTest.ROOT_IncreaseRSS execute swapoff -a command on your machine.

4. If many test case failures related to LinuxFilesystemIsolatorTest occurs, there will be folders with patterns like dRmDeG, p2Bc9U, mesos-ALxExf accumulated in /tmp folder. Please delete such folders and execute the failed test cases individually.

5. If you find test case failure related to slave showing following error message, this could be because of slow response of docker, try to restart docker service and try executing these test case individually.

    Value of: (response).get().status
    Actual: "503 Service Unavailable"
    Expected: OK().status
    Which is: "200 OK"
    Agent has not finished recovery

6. Make sure /etc/hosts file have entry with following details: <ipaddress> <Fully Qualified Domain Name> <hostname>

7.ContainerizerTest.ROOT_CGROUPS_BalloonFramework test case fails if 5432 port is already in use. Port can be changed in file /<source_root>/mesos/src/tests/balloon_framework_test.sh

8. There are chances that CgroupsAnyHierarchyMemoryPressureTest.ROOT_IncreaseRSS test case might fail. https://issues.apache.org/jira/browse/MESOS-3160

9. There are chances that FaultToleranceTest.FrameworkReregister test case might fail. Use patch from https://issues.apache.org/jira/browse/MESOS-7029

10. Test case bool/UserContainerLoggerTest.ROOT_LOGROTATE_RotateWithSwitchUserTrueOrFalse/0 might failed. This is a known issue: https://issues.apache.org/jira/browse/MESOS-6823

Step 3: Run Apache Mesos

3.1) Start master
cd /<source_root>/mesos/build
sudo ./bin/mesos-master.sh --ip=<ip_address> --work_dir=/var/lib/mesos

NOTE: Ensure work directory exists and has required permissions

3.2) Start slave
cd /<source_root>/mesos/build
sudo ./bin/mesos-agent.sh --master=<ip_address>:5050 --work_dir=/var/lib/mesos
3.3) Web UI

Open http://\<ip_address\>:5050 in your browser.

Reference:

http://mesos.apache.org/

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally