High latency observed calling feature store and SageMaker service from EKS deployed Java application

[tushar-meesho]

We have recently done an integration with AWS SageMaker and FeatureStore for one of our use-case. On Cloudwatch, we could see the p99: 15-20ms for sagemaker and p99: 30-40ms for feature-store.

But we are observing the high latencies while using JAVA AWS sdk 2.0, to call to feature-store, somewhat around. p99: 150ms for sagemaker and p99: 100ms for feature-store.

We have the containerised spring-boot application running on EKS and here we are calling FS and SM.

Sample code :-

feature-store call

   private static SageMakerFeatureStoreRuntimeClient featureStoreRuntimeClient;

    @PostConstruct
    public void init() {
        ClientOverrideConfiguration clientOverrideConfiguration = ClientOverrideConfiguration.builder()
                .apiCallTimeout(Duration.of(featureStoreConfig.getApiCallAttemptTimeout(), ChronoUnit.MILLIS))
                .retryPolicy(RetryPolicy.builder().numRetries(0).build())
                .build();

        featureStoreRuntimeClient = SageMakerFeatureStoreRuntimeClient.builder()
                .overrideConfiguration(clientOverrideConfiguration)
                .credentialsProvider(DefaultCredentialsProvider.create())
                .region(Region.AP_SOUTHEAST_1).build();
 }

Sagemaker call:

private static SageMakerRuntimeClient sageMakerRuntimeClient;


@PostConstruct
public void init() {
    ClientOverrideConfiguration clientOverrideConfiguration = ClientOverrideConfiguration.builder()
            .apiCallTimeout(Duration.of(sageMakerConfig.getApiCallAttemptTimeout(), ChronoUnit.MILLIS))
            .retryPolicy(RetryPolicy.builder().numRetries(0).build())
    .build();

    sageMakerRuntimeClient = SageMakerRuntimeClient.builder()
            .overrideConfiguration(clientOverrideConfiguration)
            .credentialsProvider(DefaultCredentialsProvider.create())
            .region(Region.AP_SOUTHEAST_1)
            .build();
}

Actual SageMaker call method and instrumentation over it.

    @Override
    @Bulkhead(name = AWS_SAGE_MAKER, type = Type.THREADPOOL, fallbackMethod = GET_MODEL_INFERENCE_FALLBACK)
    @TimeLimiter(name = AWS_SAGE_MAKER, fallbackMethod = GET_MODEL_INFERENCE_FALLBACK)
    @CircuitBreaker(name = AWS_SAGE_MAKER, fallbackMethod = GET_MODEL_INFERENCE_FALLBACK)
    public CompletableFuture<InvokeEndpointResponse> getModelInference(String endpointName, String jsonBody) {
        return CompletableFuture.completedFuture(process(endpointName, jsonBody));
    }

    public InvokeEndpointResponse process(String endpointName, String jsonBody) {
        InvokeEndpointRequest invokeEndpointRequest = InvokeEndpointRequest.builder().endpointName(endpointName)
                .body(SdkBytes.fromUtf8String(jsonBody)).contentType("application/json").build();

        InvokeEndpointResponse invokeEndpointResponse = sageMakerRuntimeClient.invokeEndpoint(invokeEndpointRequest);

        return invokeEndpointResponse;
    }

Using AWS java-sdk2 for SageMaker and feature-store.

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>sagemakerruntime</artifactId>
  <version>2.20.26</version>
</dependency>

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>sagemakerfeaturestoreruntime</artifactId>
  <version>2.20.26</version>
</dependency>

On cloudwatch for Sagemaker we are using: p99 of ModelLatency + OverheadLatency metric as stated in the docs : docs.aws.amazon.com/sagemaker/latest/dg/…
.And For feature-store we are using Latency metrics as mentioned.

We are benchmarking our application latency from these metrics, but there's a significant difference in the latencies observed on the application side.

This is how we are measuring the end to end p99 latency on application level for sagemaker and feature-store: histogram_quantile(0.99, sum(rate(resilience4j_circuitbreaker_calls_seconds_bucket{name="awsSageMaker"}[1m])) by (le)) . and same query with method name awsFeatureMaker as well.

Could this be because of different VPC(EKS pods and SageMaker both are in the same region though AP_SOUTHEAST_1), or some other form of network latency, etc ? Is there a way to figure the cause of this ?