Could not regenerate tf_text_regression model on s390x BE system and tensorflow_model_server_test is failing

[Sidong-Wei]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04 on s390x
• TensorFlow Serving installed from (source or binary): Source
• TensorFlow Serving version: 2.5.1

Describe the problem

When running test suites on an s390x machine, test_tf_text in //tensorflow_serving/model_servers:tensorflow_model_server_test will fail with the following error message:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "U_INVALID_FORMAT_ERROR: Could not retrieve ICU NFKC normalizer
	 [[{{node NormalizeUTF8WithOffsets/NormalizeUTF8WithOffsetsMap}}]]"
	debug_error_string = "{"created":"@1626196655.285737838","description":"Error received from peer ipv4:127.0.0.1:41813","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"U_INVALID_FORMAT_ERROR: Could not retrieve ICU NFKC normalizer\n\t [[{{node NormalizeUTF8WithOffsets/NormalizeUTF8WithOffsetsMap}}]]","grpc_status":13}"
>

This seems to be caused by the fact that TF text is using Little Endian ICU normalization data, so I tried to manually generate this normalization data file on a Big Endian machine from ICU repo.

The next step I would like to try was to regenerate the tf_text_regrassion model for serving. However, after I run the script here, it does not seem to produce the correct model for this test.
TF serving 2.5.1 is depending on TF text 2.4.3, so I tried this script for both 2.4.3 and 2.5.0 versions TF text (because 2.5.0 is the first version it supports TF 2.5.0 - TF serving 2.5.1 is also depending on TF 2.5.0). But they either could not run or the generated model is still making the test case fail.

My question would be is there a clear way to regenerate the tf_text_regression model for a particular TF serving version (2.5.1)? Simply running the script is not working and also there is this version discrepancy here, so I would appreciate it if someone could guide me through generating this model on a local machine.

Exact Steps to Reproduce

This will reproduce the test case failure

bazel test --build_tests_only --test_output=errors --verbose_failures -c opt //tensorflow_serving/model_servers:tensorflow_model_server_test

[godot73]

The model servers are supposed to be built within a docker rather than native OS environments. Please try prepending the bazel test command with tools/run_in_docker.sh.

tools/run_in_docker.sh bazel test --build_tests_only --test_output=errors --verbose_failures -c opt //tensorflow_serving/model_servers:tensorflow_model_server_test

[UsharaniPagadala]

@Sidong-Wei
Could you please respond to the above comment of @godot73 and confirm if this issue can be closed.Thanks

[Sidong-Wei]

@UsharaniPagadala @godot73 Thanks for the reply, it is good to know that it is supposed to be run inside a docker container. However, what I was trying to figure out is how to generate the tf_text_regression model locally instead of using the pre-built one here.
In other words, I was trying to confirm that the pre-built model binary is not compatible with Big Endian machines, so I would appreciate it if you could inform me how this model is generated on your end, especially when the TF serving, TF, TF text version updates as they do not always update at the same time.

[rposts]

@UsharaniPagadala @godot73 In TFS 2.9.1 tensorflow_model_server_test did not run by default (bazel test -c opt tensorflow_serving/...). Could you please confirm if this testcase is deprecated? Thanks!

[singhniraj08]

@Sidong-Wei,

TF Text is updated to v2.9.0 in latest TF serving release 2.11.0. Ref: commit

Please try to generate the tf_text_regression model with latest TF serving release 2.11.0 and let us know if you issue has been resolved. Thank you!