Feature Request: Add SQS option to SNS event

[jlhood]

Description:

SAM offers simplified syntax for SNS -> Lambda and with #261 supports adding subscription filter policies. However, a common use case is SNS -> SQS -> Lambda. SAM offers simplified syntax for SQS -> Lambda, but you have to use standard CloudFormation to setup the SNS -> SQS connection and subscription filter policy, which can be verbose.

SAM could add a lot of value by adding an option to the existing SNS event type to create an SQS queue.

Here's an example that uses simple default settings for the queue:

Type: SNS
Properties:
  Topic: arn:aws:sns:us-east-1:123456789012:my_topic
  # new property that creates an SQS queue between the topic and Lambda function
  # When only the boolean is specified, SAM uses the following defaults:
  # AWS::SQS::Queue logicalId: <function logicalId><SNS event key>Queue
  # AWS::SQS::QueuePolicy logicalId: <function logicalId><SNS event key>QueuePolicy
  # BatchSize: default behavior - don't add to `AWS::Lambda::EventSourceMapping` resource so default value is used.
  # Enabled: default behavior - don't add to `AWS::Lambda::EventSourceMapping` resource so default value is used.
  SqsSubscription: true
  FilterPolicy:
    store: 
      - example_corp
    price_usd: 
      - numeric: 
          - ">="
          - 100

Here's an example that specifies additional options for the queue:

Type: SNS
Properties:
  Topic: arn:aws:sns:us-east-1:123456789012:my_topic
  # new property that creates an SQS queue between the topic and Lambda function
  # Default values can be overridden by passing an object instead of a boolean. Supported keys:
  # QueuePolicyLogicalId: overrides default logicalId naming of the AWS::SQS::QueuePolicy resource.
  # QueueArn: allows the user to specify their own queue instead of having SAM create one on their behalf, allowing them to specify non-default properties on the queue.
  # BatchSize: if specified, add BatchSize property with given value to `AWS::Lambda::EventSourceMapping` resource.
  # Enabled: if specified, add BatchSize property with given value to `AWS::Lambda::EventSourceMapping` resource.
  SqsSubscription:
    QueuePolicyLogicalId: CustomQueuePolicyLogicalId
    QueueArn: !GetAtt MyCustomQueue.Arn
    BatchSize: 5
    Enabled: false
  FilterPolicy:
    store: 
      - example_corp
    price_usd: 
      - numeric: 
          - ">="
          - 100

Implementation Note: When SAM expands this into CloudFormation, the AWS::SQS::QueuePolicy resource should look like this:

  ExampleQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref ExampleQueue
      PolicyDocument:
        Version: '2012-10-17'
        Id: ExampleQueuePolicy
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - sns.amazonaws.com
            Action:
              - sqs:SendMessage
            Resource:
              - !GetAtt ExampleQueue.Arn
            Condition:
              ArnEquals:
                aws:SourceArn: <SNS Topic ARN>

[otaviofff]

That would be really helpful indeed.

[jlhood]

Happy to work with anyone interested in picking this up!

[Buffer0x7cd]

@jlhood I would like to work on this issue.

[jlhood]

@Buffer0x7cd Awesome, thank you so much! Before implementing, we need to finalize the SAM spec. The draft I created above just has a simple UseSQS boolean, but we'll want a way to surface some of the options of the SQS event source like BatchSize and Enabled. I'll update the spec in the issue description with exactly what we want to support, then I'll post a comment so you'll know when it's ready to start implementation.

[jlhood]

@brettstack @keetonian I've updated the issue description with the proposed SAM spec for this feature. Can you both review?

[brettstack]

Do we have precedent for overriding generated logical resource id like QueuePolicyLogicalId? Do we need it?

What's a better name than UseSQS - [Forward|Push|Send]ToSqs? SqsSubscriptions?

Can we allow an array of queues also?

[jlhood]

Thanks for the feedback, @brettstack!

Do we have precedent for overriding generated logical resource id like QueuePolicyLogicalId? Do we need it?

This would be something new. Maybe this is a little selfish, but in the Event Fork Pipelines apps we just released today, I'm specifying the Queue and QueuePolicy explicitly. Once this SAM feature is released, I'd love to convert over to using it since it makes the template even easier to read. However, without being able to override the logical id, I don't have a way to use the SAM feature while keeping my apps backwards-compatible. 😞 Allowing overrides of logical ids would allow customers to simplify existing templates as new SAM features are added while maintaining backwards-compatibility. However, I won't fight too hard if you really don't want to add it.

What's a better name than UseSQS

Yeah I'm not loving the key name, but I couldn't think of anything better. None of your suggestions are jumping out to me as obviously better. Some others I can think of: EnableSQS or just SQS. I'm kind of liking just SQS, honestly. SQS: true and SQS: <additional properties> seems concise and readable. What do you think?

Can we allow an array of queues also?

You can already achieve this with multiple Events entries. Accepting a list could make the syntax slightly shorter, but I think it adds complexity since then there'd be two ways of doing this within the same resource.

[brettstack]

Given that use case I'm okay with adding the logical resource id name override.

The wording in docs is around "subscribing" an SQS Queue to an SNS Topic, so I'd like to get that word in there to make it clear what it is you achieve by specifying this property. SqsSubscription, SubscribeSqs (variation: replace Sqs with Queue or SqsQueue).

Multiple Event entries wouldn't be as succinct.

[keetonian]

I like SqsSubscription

[jlhood]

Thanks. Updated it to SqsSubscription in the top-level RFC comment.

[jlhood]

@Buffer0x7cd I've updated the top-level issue comment with the finalized spec for this feature, so it should be ready to implement once you're finished with your other PR. Let me know if you have any questions. Thanks!