Put SQS Message from Lambda inside a VPC (config endpoint)

[baumblatt]

Confirm by changing [ ] to [x] below:

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers

Describe the question
I want to put a message on a SQS queue from a Lambda function published on a private subnet of my VPC.

In accordant of documentations, this should work if the endpoint is configured to the VPC Endpoint for the SQS service.

This issue is related with the Ruby and Java equivalents where the point is QueueUrl is overriding the endpoint configurartion.

Typescript Sample code:

import {SQS, AWSError} from 'aws-sdk';
import {SendMessageResult} from 'aws-sdk/clients/sqs';

export const handler = async () => {
    new SQS({region: 'us-east-1', endpoint: 'https://vpce-{{id}}.sqs.us-east-1.vpce.amazonaws.com'}).sendMessage({
        QueueUrl: 'https://sqs.us-east-1.amazonaws.com/{{user}}/my-queue',
        MessageBody: 'Hello World!'
    }, (err: AWSError, data: SendMessageResult) => {
        if (err) {
            console.error(err.message);
        }
    })
}

Best regards,
Bernardo Baumblatt

[luismramirezr]

I can't just make it work. I remember that it worked some time ago using only the QueueUrl.

I verified that:

• SQS VPC Endpoint is created
• The SQS VPC Endpoint has the same SG and subnets of the lambda

The lambda always timeout trying to send the message to the queue.

[ajredniwja]

@baumblatt
Apologies for late reply,

The above code will make a call to the QueueUrl instead of the endpoint, the possible workaround would be to add the VPC endpoint url in the QueueUrl which would look like

https://vpce-{{id}}.sqs.us-east-1.amazonaws.com/{{user}}/my-queue

Marking this as a feature request and let the team work on it.

[rohit-gohri]

I have a different problem that I think might be happening because of this issue. That some of the requests are going through the VPC while rest are going through public endpoint (goes through a NAT):

^ I have filtered these graphs with only the queue and VPC endpoint by tags. Have also verified the subnet for all the instances and for the VPC endpoint.

Could it be possible that this issue would only impact one of the SQS request methods? We have tried using the VPC endpoint url in the queue url (Sep 08-09 days)

[pktippa]

@ajredniwja @luismramirezr
For me worked for me in QueueUrl

vpce-{{id}}-{{additional_text}}.sqs.us-east-1.vpce.amazonaws.com

which can be found in DNS names in "Details"

[belaaiza]

I'm having a similar issue. I'm running my lambda within localstack, so as described at this issue, I created my SQS object with no endpoint:

const awsConfig = { 
    region: configurations.awsRegion,
    logger: console
 };
const sqs: SQS = new SQS(awsConfig);
return new SQSAdapter(sqs);

And I call deleteMessage with QueueUrl set with the localstack network IP (the value from LOCALSTACK_HOSTNAME is 172.17.0.2):

 await this.sqsClient.deleteMessage({
    QueueUrl: 'http://${process.env.LOCALSTACK_HOSTNAME}:4566/000000000000/queue-name',
    ReceiptHandle: receiptHandle,
}).promise();

When I log the content of the sqs object, this is empty, and further I'm facing the following error when I call the deleteMessage: The input receipt handle is invalid. I tried to log the return of sqs.listQueues() before delete the message, and this is also empty.

When I run the lambda outside the localstack docker, with LOCALSTACK_HOSTNAME=localhost, I'm not facing this error.

Also, I'm using aws-sdk-js to access DocumentClient in the same way and at the same lambda and this is working fine, I'm having this problem only with the SQS class.

Can anyone help me?

[mikemklee]

Any updates on this issue? I am running into the same issue, trying to use the SQS SDK in an EC2 instance via ECS.

I am trying to use a VPC enpoint by doing

const SQS = new AWS.SQS({
  apiVersion: constants.aws.sqs.apiVersion,
  region: constants.aws.sqs.region,
  endpoint:  constants.aws.sqs.vpcEndpoint,
});

where vpcEndpoint is something like http://vpce-foobar.sqs.region.vpce.amazonaws.com

Any calls to the SQS SDK functions would eventually timeout.

[FraserYu]

As @pktippa mentioned, it works. thanks. 👍
The premise for this to take effect is that you created a VPC endpoint for SQS correctly. when you attach policy for your endpoint, either give it full access(for the test) or specified access(eg. make sure your SQS ARN is in the allowed resources block)

[y-ohno-p]

vpce-{{id}}-{{additional_text}}.sqs.{{region}}.vpce.amazonaws.com did not work for me.

This worked, but I am not sure if it is appropriate. (ap-northeast-1.queue.amazonaws.com is legacy endpoint.)

var AWS = require('aws-sdk');
AWS.config.update({region: 'ap-northeast-1'});
var sqs = new AWS.SQS({apiVersion: '2012-11-05', endpoint : 'https://ap-northeast-1.queue.amazonaws.com'});
var queueURL = "https://sqs.ap-northeast-1.amazonaws.com/{id}/my-queue";

exports.handler = (event) => {
  sqs.receiveMessage({ QueueUrl: queueURL }, function(err, data) {
    // something
  });
};

[ciminuv]

Is there anyone able to get it working? I am trying to send message to SQS from ECS Fargate, following are what I've tried:

• Normal DNS sqs.us-east-1.amazonaws.com as mentioned in the Tutorial: Sending a message to an Amazon SQS queue from Amazon Virtual Private Cloud.
• VPC Endpoint DNS vpce-{{id}}-{{additional_text}}.sqs.us-east-1.vpce.amazonaws.com as suggested by @pktippa.
• VPC Endpoint Subnet DNS vpce-{{id}}-{{additional_text}}-{{us-east-1b}}.sqs.us-east-1.vpce.amazonaws.com.
• The legacy endpoint https://us-east-1.queue.amazonaws.com suggested by @y-ohno-p.

All of above options end up timeout with the same error: "Error UnknownEndpoint: Inaccessible host: xxx. This service may not be available in the 'us-east-1' region.".

P/s: value of QueueUrl = endpoint above + /{{id}}/{{my-queue.fifo}}.

[danielcardenas75]

Finally! We were able to make it work. The answer that put us on track can be found here, so I think it could help someone out there struggling with this as we were.

• In our lambda function (node.js 14.x) we had:

var sqs = new AWS.SQS({apiVersion: '2012-11-05', endpoint : 'https://sqs.ca-central-1.amazonaws.com'});
var queueURL = "https://sqs.ca-central-1.amazonaws.com/{accountID}/{queueName}";
var params = {
	 DelaySeconds: 10,
	 MessageAttributes: {
		 "our_object_id": {
			 DataType: "String",
			 StringValue: our_object_id
		 }
	 },
	 MessageBody: "Testing message",
	 QueueUrl: queueURL
 };
const response = await sqs.sendMessage(params).promise();

• The lambda function was in its own subnet in the VPC and in the same region

• The VPC endpoint interface (in our case, com.amazonaws.ca-central-1.sqs) uses the same subnet as the lambda function

• Policy in the VPC endpoint interface:

{
    "Id": "Policy{someID}",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt{someID}",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SendMessageBatch"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:ca-central-1:{accountID}:{queueName}",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::{accountID}:role/{lambdaRoleName}"
                ]
            }
        }
    ]
}

• Security group's inbound/outbound rules in the VPC endpoint interface:

Inbound: HTTPS(TCP)/443 <-- 0.0.0.0/0
Outbound: All traffic(All)/All --> 0.0.0.0/0

Be aware that we are not security experts, so this configuration probably violates several security rules for a production environment, but for the moment we are in dev/prototype mode, so it didn't matter. We'll harden the configuration once we move to production environment (if funds are approved :) )

[ciminuv]

it was my inbound network configuration, I am not sure why it needs inbound traffic but thank danielcardenas75 for pointing it out 🙇

[waleedasad]

@danielcardenas75 Thanks for your detail answer, I just have one query, what Endpoint you are using exactly with queue url? DNS name given by VPC endpoint i.e "vpce-{{id}}-{{additional_text}}.sqs.us-east-1.vpce.amazonaws.com"?

[danielcardenas75]

Hi @waleedasad, I think you are asking about this value:
var queueURL = "https://sqs.ca-central-1.amazonaws.com/{accountID}/{queueName}";
The first part (in our case, "sqs.ca-central-1.amazonaws.com") comes from the VPC SQS interface endpoint details tab, at the bottom of that page you will find a "Private DNS Name" value. To that value I added "/{accountID}/{queueName}" and that became our queue URL value.
I remember trying to make it work with some variations of the "DNS names" like the "vpce-{{id}}-{{additional_text}}.sqs.us-east-1.vpce.amazonaws.com" you mention, but it didn't work.
I hope it helps!

[bohana]

One note we observed when Lambda is a Python function using boto3: it appears boto3 does not use the endpoint names expected by lambda when inside a VPC.

• Lambda expects: https://sqs.<region>.amazonaws.com
• By default, Boto uses: https://<region>.queue.amazonaws.com

To see which URL was created for the Boto resource object, you can inspect the sqs.meta.client.meta.endpoint_url property.

When connecting to SQS with the default URL from a VPC, the lambda call would just time out.

You can override this with the endpoint_url argument in the boto3.resource() call like so:

sqs = boto3.resource('sqs', endpoint_url='https://sqs.eu-west-1.amazonaws.com', region_name='eu-west-1')

HTH.

[metealp]

Finally! We were able to make it work. The answer that put us on track can be found here, so I think it could help someone out there struggling with this as we were.

• In our lambda function (node.js 14.x) we had:

var sqs = new AWS.SQS({apiVersion: '2012-11-05', endpoint : 'https://sqs.ca-central-1.amazonaws.com'});
var queueURL = "https://sqs.ca-central-1.amazonaws.com/{accountID}/{queueName}";
var params = {
	 DelaySeconds: 10,
	 MessageAttributes: {
		 "our_object_id": {
			 DataType: "String",
			 StringValue: our_object_id
		 }
	 },
	 MessageBody: "Testing message",
	 QueueUrl: queueURL
 };
const response = await sqs.sendMessage(params).promise();

• The lambda function was in its own subnet in the VPC and in the same region
• The VPC endpoint interface (in our case, com.amazonaws.ca-central-1.sqs) uses the same subnet as the lambda function
• Policy in the VPC endpoint interface:

{
    "Id": "Policy{someID}",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt{someID}",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SendMessageBatch"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:ca-central-1:{accountID}:{queueName}",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::{accountID}:role/{lambdaRoleName}"
                ]
            }
        }
    ]
}

• Security group's inbound/outbound rules in the VPC endpoint interface:

Inbound: HTTPS(TCP)/443 <-- 0.0.0.0/0 Outbound: All traffic(All)/All --> 0.0.0.0/0

Be aware that we are not security experts, so this configuration probably violates several security rules for a production environment, but for the moment we are in dev/prototype mode, so it didn't matter. We'll harden the configuration once we move to production environment (if funds are approved :) )

this is worked for me. Crucial step was setting Inbound rules, default one was http. thanks a lot

[busla]

One note we observed when Lambda is a Python function using boto3: it appears boto3 does not use the endpoint names expected by lambda when inside a VPC.

* Lambda expects: `https://sqs.<region>.amazonaws.com`

* By default, Boto uses: `https://<region>.queue.amazonaws.com`

To see which URL was created for the Boto resource object, you can inspect the sqs.meta.client.meta.endpoint_url property.

When connecting to SQS with the default URL from a VPC, the lambda call would just time out.

You can override this with the endpoint_url argument in the boto3.resource() call like so:

sqs = boto3.resource('sqs', endpoint_url='https://sqs.eu-west-1.amazonaws.com', region_name='eu-west-1')

HTH.

this worked for me, thanks 🙏

[EduardMcfly]

My solution

I created a VPCEndpoint

or CloudFormation

  SQSVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref VPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.sqs
      SecurityGroupIds:
        - !Ref LambdaSecurityGroup
      SubnetIds:
        - !Ref SubnetAPrivate
        - !Ref SubnetBPrivate
      VpcEndpointType: Interface
      

Outputs:
  .......
  SQSVPCEndpointDNS:
    Description: Endpoint de SQS
    Value:
      !Select [
        '1',
        !Split [':', !Select ['0', !GetAtt SQSVPCEndpoint.DnsEntries]],
      ]

---

I have 3 DNS names

I use any DNS, example:

vpce-04ab98b90de87d58e-xxxxxxxx.sqs.us-east-1.vpce.amazonaws.com

In this case I work with Node.js

import { SQS } from 'aws-sdk';

const sqs = new SQS();

sqs
  .sendMessage({
    QueueUrl:
      'https://vpce-04ab98b90de87d58e-xxxxxx.sqs.us-east-1.vpce.amazonaws.com/<account-id>/<queue>',
    MessageBody: JSON.stringify({ message: 'Hola' }),
  })
  .promise()
  .then(result => {
    console.log(result);
  });

[kellertk]

Hello! Please also see https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html for more detailed configuration. You will need to migrate to v3 of the SDK to use this.