Building TensorFlow Serving

Building TensorFlow Serving

The instructions provided here specify the steps to build TensorFlow Serving version 2.9.1 on Linux on IBM Z for the following distributions:

• Ubuntu (18.04, 20.04, 22.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1: Build and Install TensorFlow Serving

1.1) Build using script

TensorFlow Serving can be built manually using STEP 1.2.

Use the following commands to build TensorFlow Serving using the build script. Please ensure wget is installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/TensorflowServing/2.9.1/build_tensorflow_serving.sh

bash build_tensorflow_serving.sh    [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 2. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

1.2) Build and Install TensorFlow 2.9.1

• Instructions for building TensorFlow 2.9.1 can be found here.

1.3) Build TensorFlow Serving

• Download source code

  cd $SOURCE_ROOT
  git clone https://github.com/tensorflow/serving
  cd serving
  git checkout 2.9.1

• Apply patches

  export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/TensorflowServing/2.9.1/patch"
  wget -O tfs_patch.diff $PATCH_URL/tfs_patch.diff
  sed -i "s?source_root?$SOURCE_ROOT?" tfs_patch.diff
  git apply tfs_patch.diff
  cd $SOURCE_ROOT/tensorflow
  wget -O tf_patch.diff $PATCH_URL/tf_patch.diff
  git apply tf_patch.diff

• Build TensorFlow Serving

  Tensorflow Serving can be built as follows:

  cd $SOURCE_ROOT/serving
  TF_SYSTEM_LIBS="boringssl,grpc" bazel --host_jvm_args="-Xms1024m" --host_jvm_args="-Xmx2048m" build  --copt=-march=native --color=yes --curses=yes --host_javabase="@local_jdk//:jdk" --define tflite_with_xnnpack=false --verbose_failures --output_filter=DONT_MATCH_ANYTHING -c opt tensorflow_serving/...

  Note: TensorFlow Serving build is resource intensive operation. If build continues to fail try increasing the swap space and reduce the number of concurrent jobs by specifying --jobs=n in the build command above, where n is the number of concurrent jobs.

  Copy binary to access it from anywhere, make sure /usr/local/bin is in $PATH. Run command:

  cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server /usr/local/bin/.
  tensorflow_model_server --version

1.4) Install TensorFlow Serving API

sudo pip3 install tensorflow-serving-api==2.9.1

Step 2: Verify TensorFlow Serving (Optional)

• Run TensorFlow Serving from command Line

   tensorflow_model_server --rest_api_port=8501 --model_name=<model_name> --model_base_path=<model_path> &
   curl -d '{"instances": [1.0, 2.0, 5.0]}'     -X POST http://localhost:8501/v1/models/<model_name>:predict

  • For example:

   export TESTDATA="$SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata"
   tensorflow_model_server --rest_api_port=8501 --model_name=half_plus_two --model_base_path=$TESTDATA/saved_model_half_plus_two_cpu &
   curl -d '{"instances": [1.0, 2.0, 5.0]}'     -X POST http://localhost:8501/v1/models/half_plus_two:predict

  Output should look like:

  {
      "predictions": [2.5, 3.0, 4.5
      ]
  }
  

Step 3: Execute Test Suite (Optional)

• Run complete testsuite

  cd $SOURCE_ROOT/serving
  TF_SYSTEM_LIBS="boringssl,grpc" bazel --host_jvm_args="-Xms1024m" --host_jvm_args="-Xmx2048m" test --copt=-march=native --test_tag_filters=-gpu,-benchmark-test -k --build_tests_only --test_output=errors --verbose_failures --define tflite_with_xnnpack=false -c opt tensorflow_serving/...

  Note: //tensorflow_serving/servables/tensorflow:tflite_interpreter_pool_test, tflite_session_test and saved_model_bundle_factory_test testcases require model files to be regenerated on s390x. For tflite, create and use new model files using (Please modify the python library path based on your local environment and python version):

  cd $SOURCE_ROOT/tensorflow
  bazel build --host_javabase="@local_jdk//:jdk" //tensorflow/lite/tools/signature:signature_def_utils
  cp -r bazel-bin/tensorflow/lite/tools/signature/* tensorflow/lite/tools/signature/
  sudo rm -rf $(python3 -c "import site; print(\"\\n\".join(site.getsitepackages()))" | head -n 1)/tensorflow/lite/tools
  sudo ln -s $SOURCE_ROOT/tensorflow/tensorflow/lite/tools $(python3 -c "import site; print(\"\\n\".join(site.getsitepackages()))" | head -n 1)/tensorflow/lite/tools
  
  sudo rm -rf /tmp/saved_model_half_plus_two*
  sudo python $SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two.py
  sudo cp /tmp/saved_model_half_plus_two_tflite/model.tflite $SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_tflite/00000123/
  sudo cp /tmp/saved_model_half_plus_two_tflite_with_sigdef/model.tflite $SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_tflite_with_sigdef/00000123/
  
  mkdir /tmp/parse_example_tflite
  python $SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata/parse_example_tflite.py
  cp /tmp/parse_example_tflite/model.tflite $SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata/parse_example_tflite/00000123/model.tflite

References:

• https://www.tensorflow.org/tfx/guide/serving
• https://github.com/tensorflow/serving