Custom endpoint for SQS (VPC Endpoint) does not work.

[harihara-n]

Issue description

Please refer to aws/aws-sdk-ruby#2114 - this issue is the exact same as that one (for which a fix is being developed), but for Java.

QueueUrlHandler is the equivalent in Java that is causing SQS calls to not work when specifying a custom endpoint (such as an SQS VPC Endpoint). This is because, the custom endpoint is being replaced with the host in the queue URL, which happens to be sqs.<region>.amazonaws.com

[juaoose]

Would it be appropiate to look at the incoming request, if such object already has an endpoint, then just use that endpoint instead of setting it from the QueueUrl? Are there other assumptions I might be overlooking?

[debora-ito]

Hi @hhk1989, I just want to confirm the issue you're seeing with the SDK for Java, is the following code a good example of you're trying to do?

AmazonSQS sqsClient = AmazonSQSClientBuilder.standard()
    .withRegion(Regions.AP_SOUTH_1)
    .build();

SendMessageRequest sendMessageRequest = new SendMessageRequest()
    .withQueueUrl("https://vpce-<vpce-id>.sqs.us-west-2.vpce.amazonaws.com/<user-id>/<sqs-queue>")
    .withMessageBody("Hello");
SendMessageResult result = sqsClient.sendMessage(sendMessageRequest);

Could you share the stacktrace of the error?

[harihara-n]

@debora-ito apologies for the late response.

The code snippet we have would be something like the following (similar to the Ruby code in the linked issue above).

AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuild.EndpointConfiguration(
  "https://vpce-<vpce-id>.sqs.us-west-2.vpce.amazonaws.com",
   Regions.US_WEST_2
)

AmazonSQS sqsClient = AmazonSQSClientBuilder.standard()
    .withRegion(Regions.US_WEST_2)
    .withEndpointConfiguration(endpointConfiguration)
    .build();

SendMessageRequest sendMessageRequest = new SendMessageRequest()
    .withQueueUrl("https://sqs.us-west-2.amazonaws.com/<account_id>/<queue_name>")
    .withMessageBody("Hello");
SendMessageResult result = sqsClient.sendMessage(sendMessageRequest);

The issue is that the above makes a request to https://sqs.us-west-2.amazonaws.com origin, rather than the VPC Endpoint specified in the endpoint configuration.

The error we get in our applications in the DC is a 403, because the application has not whitelisted the above SQS public API path in our internal squid proxy that selectively allows only whitelisted public paths.

[debora-ito]

The issue is that the above makes a request to https://sqs.us-west-2.amazonaws.com origin, rather than the VPC Endpoint specified in the endpoint configuration.

I understand it now. We may not be able to change this behavior, but a possible solution would be specifying the vpc endpoint url directly in the queue url as I did in my code example.

Anyway, we are investigating.

[debora-ito]

I can reproduce the issue, marking as a bug.

The issue is in our backlog, pending prioritization.

[Quanzzzz]

Hi @hhk1989, after checking this issue and trying to reproduce it, I got a question about your expecting behavior of the sdk. I'm supposing that you want to custom the endpoint and send message to SQS via this endpoint, and that's indeed not currently supported in java sdk, as the host in queue url will override the client configured endpoint. However, I also tried to send the message to a queue with the VPC-Endpoint specified in the url as:

SendMessageRequest sendMessageRequest = new SendMessageRequest()
    .withQueueUrl("vpce-xxxx-yyy-us-west-2a.sqs.us-west-2.vpce.amazonaws.com/userid/sqs-queue-name")
    .withMessageBody("Hello");
SendMessageResult result = sqsClient.sendMessage(sendMessageRequest);

This ran into an exception of com.amazonaws.SdkClientException: Unable to execute HTTP request: vpce-0ab61fe684cccb125.sqs.us-west-2.amazonaws.com, which means SQS can't resolve this endpoint.
Besides, I also tried to use the vpc-e in the endpoint-url by:

aws sqs send-message --region us-east-2 --endpoint-url https://vpce-xxxx-yyy-us-west-2a.sqs.us-west-2.vpce.amazonaws.com/ --queue-url https://sqs.us-east-2.amazonaws.com/123456789012/ --message-body "Hello from Amazon SQS."

according to the tutorial of sending the sqs message from Amazon VPC (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-sending-messages-from-vpc.html), and this ran into the same exception too.

So I think cusomizing the endpointt of the client doesn't help with this issue. Since the vpc-e doesn't work when it's configured in the queue url, it wouldn't work when it's configured in the client, either.

As a result, could you let us know about your expected behavior of the sdk considering this?

[github-actions]

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.