IotDataPlaneException: Forbidden exception when trying to publish to a topic with IotDataPlaneClient and CRT sync client

[statelessness]

Describe the bug

I've refactored my sdk clients in an AWS Lambda to use a single instance of CRT sync client. When trying to publish with IotDataPlaneClient to an IoT Core topic (same account and same region than the lambda), I get a Forbidden error. If I replace the http client with a different one, UrlConnection for example, the lambda can publish to the topic. So I think that the problem has nothing to do with lambda iam permissions. I've tried to assign the iot endpoint too but without luck.

Expected Behavior

The expected behaviour is that IotDataPlaneClient can work with CRT sync client.

Current Behavior

Forbidden (Service: IotDataPlane, Status Code: 403, Request ID: 79bbcf66-6692-ccbc-8dce-15d70c494213): software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException
software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException: Forbidden (Service: IotDataPlane, Status Code: 403, Request ID: 79bbcf66-6692-ccbc-8dce-15d70c494213)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
at software.amazon.awssdk.services.iotdataplane.DefaultIotDataPlaneClient.publish(DefaultIotDataPlaneClient.java:558)
at cloud.plexo.akron.erp.common.task.worker.processor.AbstractEventProcessor.publish(AbstractEventProcessor.java:175)
at cloud.plexo.akron.erp.common.task.worker.processor.AbstractEventProcessor.process(AbstractEventProcessor.java:70)
at cloud.plexo.akron.erp.common.task.worker.processor.SQSEventProcessor.lambda$process$1(SQSEventProcessor.java:47)
at java.base/java.util.Optional.ifPresent(Unknown Source)
at cloud.plexo.akron.erp.common.task.worker.processor.SQSEventProcessor.lambda$process$2(SQSEventProcessor.java:46)
at java.base/java.util.ArrayList.forEach(Unknown Source)
at cloud.plexo.akron.erp.common.task.worker.processor.SQSEventProcessor.process(SQSEventProcessor.java:41)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeConsumer(SimpleFunctionRegistry.java:1039)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:737)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:580)
at org.springframework.cloud.function.adapter.aws.FunctionInvoker.handleRequest(FunctionInvoker.java:91)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)

Reproduction Steps

@configuration
public class CommonAmazonConfig {

@Bean
IotDataPlaneClient provideIotDataPlaneClient(final SdkHttpClient sdkHttpClient) {
    return IotDataPlaneClient.builder()
                             .defaultsMode(DefaultsMode.IN_REGION)
                             .httpClient(sdkHttpClient)
                             .region(Region.of(System.getenv("AWS_REGION")))
                             .build();
}

@Bean
S3Client provideS3Client(final SdkHttpClient sdkHttpClient) {
    return S3Client.builder()
                   .defaultsMode(DefaultsMode.IN_REGION)
                   .httpClient(sdkHttpClient)
                   .region(Region.of(System.getenv("AWS_REGION")))
                   .build();
}

@Bean
SdkHttpClient provideSdkHttpClient() {
    return AwsCrtHttpClient.builder().build();
}

}

This is the part of my pom file configuring CRT stuff (CRT version is 0.29.9):

    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>iotdataplane</artifactId>
        <exclusions>
            <exclusion>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>netty-nio-client</artifactId>
            </exclusion>
            <exclusion>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>apache-client</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3</artifactId>
        <exclusions>
            <exclusion>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>netty-nio-client</artifactId>
            </exclusion>
            <exclusion>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>apache-client</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>aws-crt-client</artifactId>
        <exclusions>
            <exclusion>
                <groupId>software.amazon.awssdk.crt</groupId>
                <artifactId>aws-crt</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
        <classifier>linux-x86_64</classifier>
    </dependency>

Possible Solution

No response

Additional Information/Context

The lambda is a Spring Cloud function, which is using IotDataPlaneClient and S3Client. The workflow is processing a file from S3 and publishing a message to IoT Core topic after. The S3 client works fine and does its job, but the IotDataPlaneClient fails with error mentioned previously.

AWS Java SDK version used

2.24.0

JDK version used

Java17

Operating System and version

AWS Lambda Java17 X86_64 architecture with snapstart enabled

[joviegas]

Looks similar to #4920 , we are still working on this
To confirm can you please send the Content-length header = Length of the payload

//payloadLength =calcualatePayloadLength();
// In the request add the overrideConfiguration as below


.overrideConfiguration(o -> o.putHeader("Content-Length",  String.valueOf(payloadLength)))

[statelessness]

Hi John.

Thanks for your suggestion. I confirm you that it's the same bug. I made the following change to my code and it worked properly:

private void publish(final String topic, final MqttEventWrapper mqttEventWrapper) {
    final byte[] payload = marshallToByteArray(mqttEventWrapper);
    iotDataPlaneClient.publish(builder -> builder
                .overrideConfiguration(o -> o.putHeader("Content-Length", String.valueOf(payload.length)))
                .payload(SdkBytes.fromByteArray(payload))
                .qos(1)
                .topic(topic));
}

Best regards,

[zoewangg]

This is fixed in #4920 Could you try with the latest version?

[statelessness]

Hi Zoe.

Just tested with version 2.24.10 of AWS SDK and it's working fine now (no need to specify Content-Length header from my code).

Thanks to the team.

Best regards,

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.