Application hangs sending to SQS using docker container hosted in AWS AppRunner

[j5alive]

Describe the bug

I have an ASP.NET MVC Core application that publishes to an SQS queue. This works fine in my local development and when I run the container locally using Docker. When I publish the container to ECR and then deploy the container to my AppRunner application, my code hangs at the point I try to send a message to the queue. When calling await _sqsClient.SendMessageAsync(message);

Expected Behavior

When calling the AmazonSQSClient.SendMessageAsync() method the message is sent or an exception is raised.

Current Behavior

When calling the AmazonSQSClient.SendMessageAsync() method the code hangs. No further event logs are received and no exception is raised until the application tries to shut down, at which point a TimeoutException is raised.

04-24-2023 10:33:07 AM       Connection id "0HMQ49229A2N6", Request id "0HMQ49229A2N6:00000005": An unhandled exception was thrown by the application.
04-24-2023 10:33:07 AM fail: Microsoft.AspNetCore.Server.Kestrel[13]
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
04-24-2023 10:29:31 AM          at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
04-24-2023 10:29:31 AM          at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
04-24-2023 10:29:31 AM          at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
04-24-2023 10:29:31 AM          at QueueTest.Controllers.QueueController.Queue() in /App/Controllers/QueueController.cs:line 40
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.SQS.Internal.ValidationResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
04-24-2023 10:29:31 AM          at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
04-24-2023 10:29:31 AM          --- End of inner exception stack trace ---
04-24-2023 10:29:31 AM          at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
04-24-2023 10:29:31 AM          at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
04-24-2023 10:29:31 AM          at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
04-24-2023 10:29:31 AM          at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
04-24-2023 10:29:31 AM          at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
04-24-2023 10:29:31 AM          at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
04-24-2023 10:29:31 AM        ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
04-24-2023 10:29:31 AM       System.TimeoutException: A task was canceled.
04-24-2023 10:29:31 AM       Connection id "0HMQ49229A2MH", Request id "0HMQ49229A2MH:00000001": An unhandled exception was thrown by the application.
04-24-2023 10:29:31 AM fail: Microsoft.AspNetCore.Server.Kestrel[13]
04-24-2023 10:25:15 AM       Application is shutting down...
04-24-2023 10:25:15 AM �[40m�[32minfo�[39m�[22m�[49m: Microsoft.Hosting.Lifetime[0]
04-24-2023 10:24:13 AM       Sending message to SQS
04-24-2023 10:24:13 AM info: QueueTest.Controllers.QueueController[0]
04-24-2023 10:24:13 AM       Preparing to send message to SQS

Reproduction Steps

Create a new .NET 6 API MVC project with Docker support:
Add the AWSSDK.SQS package, version 3.7.100.117

Add the following Controller example:

using System;
using System.Net;
using System.Text.Json;
using System.Threading.Tasks;
using Amazon;
using Amazon.Runtime;
using Amazon.SQS;
using Amazon.SQS.Model;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace QueueTest.Controllers
{
	[ApiController]
	[Route("[controller]")]
	public class QueueController : ControllerBase
	{
		private readonly ILogger<QueueController> _logger;

		public QueueController(ILogger<QueueController> logger)
		{
			_logger = logger;
		}

		[HttpGet]
		public async Task Queue()
		{
			var client = new AmazonSQSClient(new BasicAWSCredentials("[access-key]", "[secret-key]"), RegionEndpoint.USEast1);
			
			_logger.LogInformation("Preparing to send message to SQS");

			var body = JsonSerializer.Serialize(new
			{
				FirstName = "John",
				LastName = "Doe"
			});

			var request = new SendMessageRequest
			{
				MessageBody = body,
				QueueUrl = "https://sqs.us-east-1.amazonaws.com/[queue-url]"
			};

			_logger.LogInformation("Sending message to SQS");

			var result = await client.SendMessageAsync(request);
			if (result.HttpStatusCode != HttpStatusCode.OK)
			{
				throw new Exception($"Failed to publish payload to SQS. Error code {result.HttpStatusCode}");
			}

			_logger.LogInformation("Message sent to SQS");
		}
	}
}

Example dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /App

# Copy everything
COPY . ./
# Restore as distinct layers
RUN dotnet restore
# Build and publish a release
RUN dotnet publish -c Release -o out /p:UseAppHost=false

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:6.0

ENV ASPNETCORE_URLS="http://+:5000"
EXPOSE 5000

WORKDIR /App
COPY --from=build-env /App/out .
ENTRYPOINT ["dotnet", "[ProjectName].dll"]

Publish to an ECR repo and then create an AppRunner application deploying from the ECR repo.

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.SQS 3.7.100.117

Targeted .NET Platform

.NET 6 (Also tried .NET 7 and 3.1 with the same issue)

Operating System and version

Container deployed through AppRunner

[ashishdhingra]

Hi @j5alive,

Good morning.

Somehow, I'm unable to reproduce the issue. Below are the reproductions steps, used us-east-1 region for all AWS resources (reference https://aws.amazon.com/developer/language/net/badges-and-training/app-runner/module-three/).

1. Created SQS queue named testapprunnerqueue
2. In Terminal/Console,

• Created folder named AppRunnerTest.
• Executed command dotnet new webapi -n HelloAppRunnerVpc --framework net6.0 to create a new WebApi project.

3. In the newly created WebApi project,

• Deleted the default controller and model classes created by template.
• Added/modified the below classes:
  QueueController.cs

using Amazon.Runtime;
using Amazon.SQS;
using Amazon.SQS.Model;
using System.Net;
using System.Text.Json;
using Microsoft.AspNetCore.Mvc;
using Amazon;
using Microsoft.Extensions.Options;

namespace HelloAppRunnerVpc.Controllers;

[ApiController]
[Route("")]
public class QueueController : ControllerBase
{
    private readonly ILogger<QueueController> _logger;
    private readonly QueueOptions _options;
    private readonly RegionEndpoint region;
    private readonly string queueUrl;

    public QueueController(ILogger<QueueController> logger, IOptions<QueueOptions> options)
    {
        _logger = logger;
        _options = options.Value;
        region = RegionEndpoint.GetBySystemName(_options.Region);
        queueUrl = _options.QueueUrl;

    }

    [HttpGet("")]
    public string GetHealthcheck()
    {
        return "Healthcheck: Healthy";
    }

    [HttpGet("Queue")]
    public async Task<string> Queue()
    {
        using (var client = new AmazonSQSClient(region))
        {

            _logger.LogInformation("Preparing to send message to SQS");

            var body = JsonSerializer.Serialize(new
            {
                FirstName = "John",
                LastName = "Doe"
            });

            var request = new SendMessageRequest
            {
                MessageBody = body,
                QueueUrl = queueUrl
            };

            _logger.LogInformation("Sending message to SQS");

            var result = await client.SendMessageAsync(request);
            if (result.HttpStatusCode != HttpStatusCode.OK)
            {
                throw new Exception($"Failed to publish payload to SQS. Error code {result.HttpStatusCode}");
            }

            _logger.LogInformation($"Message {result.MessageId} sent to SQS");

            return result.MessageId;
        }
    }
}

This class has GetHealthcheck() API action method accessible at default route so that App Runner health checks could be successful.
QueueOptions.cs

using System;
namespace HelloAppRunnerVpc
{
	public class QueueOptions
	{
        public const string Queue = "Queue";

        public string QueueUrl { get; set; } = String.Empty;
        public string Region { get; set; } = String.Empty;

    }
}

appsettings.json

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "Queue": {
    "QueueUrl": "https://sqs.us-east-1.amazonaws.com/<<account-id>>/testapprunnerqueue",
    "Region": "us-east-1"
  }
}

Ideally, this information should be read from Secrets Manager (AppRunner also helps configure Environment Variables from Secrets Manager values).
Program.cs

using HelloAppRunnerVpc;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

builder.Services.Configure<QueueOptions>(
    builder.Configuration.GetSection(QueueOptions.Queue));


var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

//app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

HelloAppRunnerVpc.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <DockerComposeProjectPath>docker-compose.dcproj</DockerComposeProjectPath>
    <UserSecretsId>da71b967-16cf-4eaa-b591-af0e26e98619</UserSecretsId>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />
    <PackageReference Include="AWSSDK.SQS" Version="3.7.100.117" />
  </ItemGroup>

</Project>

Dockerfile

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
#EXPOSE 80
#EXPOSE 443
ENV ASPNETCORE_URLS="http://+:5000"
EXPOSE 5000
    
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["HelloAppRunnerVpc.csproj", "."]
RUN dotnet restore "./HelloAppRunnerVpc.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "HelloAppRunnerVpc.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "HelloAppRunnerVpc.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "HelloAppRunnerVpc.dll"]

Here I used the default Docker file generated by Visual Studio.

4. Tested the application locally. (you may also test by running Docker container locally via docker run -d -p 5000:5000 command and hitting the endpoint http://localhost:5000/Queue)
5. Created a new ECR repository named testapprunnerecrrepo.
6. Pushed Docker image to ECR using Push commands from AWS ECR console.
7. Created a new IAM policy named TestAppRunnerSQSPolicy with the following permissions (replace <<account-id>> with your AWS account ID):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sqs:SendMessage",
            "Resource": "arn:aws:sqs:us-east-1:<<account-id>>:testapprunnerqueue"
        }
    ]
}

8. Created a new IAM role named TestAppRunnerSQS with the permissions TestAppRunnerSQSPolicy (newly created policy) and AWSAppRunnerFullAccess attached. Modified the trust policy to below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com",
                    "tasks.apprunner.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

9. Navigated to AWS AppRunner console and created a new service:

• Repository type: Container registry.
• Provider: Amazon ECR
• Click Browse and select the container you deployed in ECR.
• Deployment settings:
  • Deployment method: Automatic
  • ECR access role <>
• Configure service
  • Service Name: HelloAppRunnerVpc
  • Port: 5000
• Configure Instance Role: Expand the Security section and set Instance role to TestAppRunnerSQS.
• Create VPC Connector
  • Under Networking, select Custom VPC.
  • Under VPC connector, click Add new.
  • VPC connector name: TestAppRunnerSQS
  • VPC: select your default VPC.
  • Subnets: select all subnets.
  • Security groups: select your default security group. (Figure 2)
  • Click Add. If you get an error message that one of your subnets doesn't support App Runner services, remove it from the Subnets list and click Add again.
  • Click Next and Create & deploy.
• Wait for deployment to complete.
• Once deployment is complete, browse to Service URL. It should display health check endpoint result.
• Create a VPC Endpoint for SQS
  • In AWS VPC console, select Endpoints from the left panel and click Create endpoint.
    • Name: vpc-endpoint-sqs
    • Service category: AWS service.
    • Service: enter SQS in the search box and select com.amazonaws.us-east-1.sqs.
    • VPC: select your default VPC.
    • Subnets: select all subnets.
    • Click Create endpoint.

10. Now navigate to AppRunner Service URL /Queue endpoint (https://<<some-id>>.us-east-1.awsapprunner.com/Queue). It would return the message ID pushed to the SQS queue.
11. SQS queue should have message to be polled in AWS console.

So, for your case, please check,

• If the credentials used to access SQS queue has proper permissions.
• The Instance Role selected in App Runner has proper permissions.
• VPC endpoint for SQS is created (during testing, I forgot to create VPC endpoint and accessing the /Queue endpoint hanged for a while giving error after a while).

For best practices,

• Never had code AWS credentials in application code. This should be configured in App Runner instance role via permissions.
• Same goes for Queue URL and region endpoint. These can be stored in Secrets Manager and could be either configured as app runner environment variables or retrieved directly (assuming instance role has proper permissions and VPC endpoint is configured to access SQS service)

Thanks,
Ashish

[j5alive]

Hi @ashishdhingra

Thanks it was the VPC endpoint for SQS that was missing. I also needed to allow HTTPS traffic within my security group for the subnet as this was missing.

There are a few odd behaviours here still that were unexpected:

1. Without the VPC endpoint I would have expected the client to try and connect the SQS over the internet, which it should have been able to do (I also had it configured with an access key/secret for this to work).
2. Why does it hang when being blocked by either the missing VPC endpoint or security group missing rule? Does it forever retry?

Thanks, I've been stuck on this issue for weeks so very excited to finally get it working.

Cheers,
Jeremy

[ashishdhingra]

Hi @ashishdhingra

Thanks it was the VPC endpoint for SQS that was missing. I also needed to allow HTTPS traffic within my security group for the subnet as this was missing.

There are a few odd behaviours here still that were unexpected:

1. Without the VPC endpoint I would have expected the client to try and connect the SQS over the internet, which it should have been able to do (I also had it configured with an access key/secret for this to work).
2. Why does it hang when being blocked by either the missing VPC endpoint or security group missing rule? Does it forever retry?

Thanks, I've been stuck on this issue for weeks so very excited to finally get it working.

Cheers, Jeremy

@j5alive Thanks for the reply and good to hear that it worked for you. Here are the inputs:

• I assume you might have selected custom VPC (as I did) during AppRunner service creation. Per Enabling VPC access for outgoing traffic, You can choose to launch your environment in a custom VPC to customize networking and security settings for outgoing traffic. You can enable your AWS App Runner service to access applications that run in a private VPC from Amazon Virtual Private Cloud (Amazon VPC).. This is a VPC endpoint scenario.
• Please refer to Retries and timeouts. It should retry the specified number of times depending on the RetryMode (default Legacy).

Converting this issue into discussion for better discoverability and marking it as answered.

Thanks,
Ashish