Building Apache Storm

Apache Storm

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Apache Storm binaries are available and can be downloaded from here.

Note: Apache Storm(v2.2.0) was verified at the time of creation of these instructions

Step 1: Install dependencies

export SOURCE_ROOT=/<source_root>/

• RHEL (7.8, 7.9)

  sudo yum install -y snappy snappy-devel bzip2 bzip2-devel curl java-1.8.0-openjdk-devel gcc-c++ make which zlib-devel procps tar wget python xz diffutils
  

  • Build GCC 7.3.0:

    cd $SOURCE_ROOT
    mkdir gcc
    cd gcc
    wget https://ftpmirror.gnu.org/gcc/gcc-7.3.0/gcc-7.3.0.tar.xz
    tar -xf gcc-7.3.0.tar.xz
    cd gcc-7.3.0
    ./contrib/download_prerequisites
    mkdir objdir
    cd objdir
    ../configure --prefix=/opt/gcc --enable-languages=c,c++ --with-arch=zEC12 --with-long-double-128 \
           --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu                  \
           --enable-threads=posix --with-system-zlib --disable-multilib
    make -j 8
    sudo make install
    sudo ln -sf /opt/gcc/bin/gcc /usr/bin/gcc
    sudo ln -sf /opt/gcc/bin/g++ /usr/bin/g++
    sudo ln -sf /opt/gcc/bin/g++ /usr/bin/c++
    export PATH=/opt/gcc/bin:"$PATH"
    export LD_LIBRARY_PATH=/opt/gcc/lib64:"$LD_LIBRARY_PATH"
    export C_INCLUDE_PATH=/opt/gcc/lib/gcc/s390x-linux-gnu/7.3.0/include
    export CPLUS_INCLUDE_PATH=/opt/gcc/lib/gcc/s390x-linux-gnu/7.3.0/include
    sudo ln -sf /opt/gcc/lib64/libstdc++.so.6.0.24 /lib64/libstdc++.so.6
    sudo ln -sf /opt/gcc/lib64/libatomic.so.1 /lib64/libatomic.so.1
    

• RHEL (8.2, 8.3, 8.4)

  sudo yum install -y snappy snappy-devel bzip2 bzip2-devel java-1.8.0-openjdk-devel curl gcc-c++ make which zlib-devel procps tar wget python3 diffutils xz
  sudo ln /usr/bin/python3 /usr/bin/python

• SLES (12 SP5)

  sudo zypper install -y unzip snappy-devel libzip2 bzip2 curl gcc7 gcc7-c++ make which zlib-devel tar wget python
  sudo ln -sf /usr/bin/gcc-7 /usr/bin/gcc
  sudo ln -sf /usr/bin/g++-7 /usr/bin/g++
  sudo ln -sf /usr/bin/gcc /usr/bin/cc
  

• SLES (15 SP2)

  sudo zypper install -y unzip snappy-devel libzip5 bzip2 curl gcc-c++ make which zlib-devel tar wget python gzip

• Ubuntu (18.04, 20.04, 21.04)

  sudo apt update
  sudo apt-get install -y libsnappy-dev libbz2-dev curl gcc-7 g++-7 make libz-dev tar wget lbzip2 python gnupg2
  sudo ln -sf /usr/bin/python2 /usr/bin/python  # For Ubuntu (20.04, 21.04) only
  sudo rm -rf /usr/bin/gcc /usr/bin/g++ /usr/bin/cc
  sudo ln -sf /usr/bin/gcc-7 /usr/bin/gcc
  sudo ln -sf /usr/bin/g++-7 /usr/bin/g++
  sudo ln -sf /usr/bin/gcc /usr/bin/cc

Step 2: Set environment variables

export JAVA_HOME=<path to java>
export PATH=$JAVA_HOME/bin:$PATH

Step 3: Create rocksdbjni-5.18.4.jar for s390x

• Build and Create Rocksdb Jar

  cd $SOURCE_ROOT
  git clone git://github.com/facebook/rocksdb.git
  cd rocksdb
  git checkout v5.18.4
  sed -i -e "s/-march=native/-march=z196/" build_tools/build_detect_platform
  sed -i -e "s/-momit-leaf-frame-pointer/-DDUMBDUMMY/" Makefile
  PORTABLE=1 make shared_lib
  make rocksdbjava

Step 4: Setting up a Storm cluster

• Run Apache Zookeeper

  Apache ZooKeeper binaries are available and can be downloaded from here.

  To start the server, extract binary tar to $SOURCE_ROOT and follow steps given below.

  cd $SOURCE_ROOT/<apache_zookeeper_binary_folder>/
  mv conf/zoo_sample.cfg conf/zoo.cfg
  
  bin/zkServer.sh start
  bin/zkCli.sh

  Note:

  • If encountering error message 'Starting zookeeper ... FAILED TO START', please check whether 'ps' command is enabled in bash.
  • User can rename existing sample config file or create their own zoo.cfg.

• Run Apache Storm

  Extract binary tar to $SOURCE_ROOT and follow steps given below.

  cd $SOURCE_ROOT/<apache_storm_folder>/
  mkdir data

  • Append below lines to $SOURCE_ROOT/<apache_storm_folder>/conf/storm.yaml.

     @@ -105,3 +105,13 @@
     #        class: "org.apache.storm.metrics2.filters.RegexFilter"
     #        expression: ".*my_component.*emitted.*"
    
    +storm.zookeeper.servers:
    + - "localhost"
    +storm.local.dir: "$SOURCE_ROOT/<apache_storm_folder>/data"
    +ui.port: <port_no>
    +nimbus.seeds: ["localhost"]
    +supervisor.slots.ports:
    + - 6700
    + - 6701
    + - 6702
    + - 6703

    Note: The port is configured by ui.port. Please provide <port_no> of your choice. It is set to 8080 by default.

  • Replace Rocksdbjni Jar.

     cd $SOURCE_ROOT/rocksdb/java/target
     cp rocksdbjni-5.18.4-linux64.jar  rocksdbjni-5.18.4-linux64.jar.backup
     jar -xf rocksdbjni-5.18.4-linux64.jar
     cp librocksdbjni-linux64.so librocksdbjni-linuxs390x.so
     jar -ufv rocksdbjni-5.18.4-linux64.jar librocksdbjni-linuxs390x.so
     cp rocksdbjni-5.18.4-linux64.jar  rocksdbjni-5.18.4.jar
     mv  $SOURCE_ROOT/rocksdb/java/target/rocksdbjni-5.18.4.jar  $SOURCE_ROOT/<apache_storm_folder>/lib/rocksdbjni-5.18.4.jar

  • Run the Storm daemons.

     cd $SOURCE_ROOT/<apache_storm_folder>/
     nohup bin/storm nimbus &
     nohup bin/storm supervisor &
     nohup bin/storm ui &

    Note: The UI can be accessed by navigating your web browser to http://<ip_address>:{ui port}.

References:

• http://storm.apache.org/releases/current/index.html
• https://github.com/apache/storm/blob/v2.2.0/DEVELOPER.md