Skip to content

Fix DestinationConfig in streaming event sources #2215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 15, 2021
Merged

Fix DestinationConfig in streaming event sources #2215

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions samtranslator/model/eventsources/pull.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,9 @@ def to_cloudformation(self, **kwargs):

destination_config_policy = None
if self.DestinationConfig:
if self.DestinationConfig.get("OnFailure") is None:
raise InvalidEventException(self.logical_id, "'OnFailure' is a required field for 'DestinationConfig'")
Copy link
Contributor

@marekaiv marekaiv Nov 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs for DestinationConfig show that OnFailure is not a required field. I'm not sure why someone would create DestinationConfig without its only possible field so I think the docs are incorrect. Either way, we should make the code and the docs consistent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a discrepancy between doc and code. In addition to the one you mentioned above, the doc is also missing to show Type as required under OnFailure. I'll request a doc update to address these inconsistencies.


# `Type` property is for sam to attach the right policies
destination_type = self.DestinationConfig.get("OnFailure").get("Type")

Expand All @@ -120,10 +123,6 @@ def to_cloudformation(self, **kwargs):
# the values 'SQS' and 'SNS' are allowed. No intrinsics are allowed
if destination_type not in ["SQS", "SNS"]:
raise InvalidEventException(self.logical_id, "The only valid values for 'Type' are 'SQS' and 'SNS'")
if self.DestinationConfig.get("OnFailure") is None:
raise InvalidEventException(
self.logical_id, "'OnFailure' is a required field for " "'DestinationConfig'"
)
if destination_type == "SQS":
queue_arn = self.DestinationConfig.get("OnFailure").get("Destination")
destination_config_policy = IAMRolePolicies().sqs_send_message_role_policy(
Expand All @@ -134,6 +133,7 @@ def to_cloudformation(self, **kwargs):
destination_config_policy = IAMRolePolicies().sns_publish_role_policy(
sns_topic_arn, self.logical_id
)

lambda_eventsourcemapping.DestinationConfig = self.DestinationConfig

if "role" in kwargs:
Expand Down
45 changes: 45 additions & 0 deletions tests/translator/input/error_function_with_invalid_stream_eventsource_dest_type.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
MyBatchingWindowParam:
Type: Number
Default: 45
Description: parameter for batching window in seconds

Resources:
MyFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
InlineCode: |
exports.handler = async (event) => {
return {
statusCode: 200,
body: JSON.stringify(event),
headers: {}
}
}
Runtime: nodejs12.x
Policies:
- SQSSendMessagePolicy:
QueueName: !GetAtt MySqsQueue.QueueName
Events:
StreamEvent:
Type: Kinesis
Properties:
Stream: !GetAtt KinesisStream.Arn
MaximumBatchingWindowInSeconds: !Ref MyBatchingWindowParam
StartingPosition: LATEST
DestinationConfig:
OnFailure:
Type: INVALID_VALID
Destination: !Ref MySnsTopic

KinesisStream:
Type: AWS::Kinesis::Stream
Properties:
ShardCount: 1

MySnsTopic:
Type: AWS::SNS::Topic
45 changes: 45 additions & 0 deletions ...ator/input/error_function_with_missing_on_failure_in_stream_event_destination_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
MyBatchingWindowParam:
Type: Number
Default: 45
Description: parameter for batching window in seconds

Resources:
MyFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
InlineCode: |
exports.handler = async (event) => {
return {
statusCode: 200,
body: JSON.stringify(event),
headers: {}
}
}
Runtime: nodejs12.x
Policies:
- SQSSendMessagePolicy:
QueueName: !GetAtt MySqsQueue.QueueName
Events:
StreamEvent:
Type: Kinesis
Properties:
Stream: !GetAtt KinesisStream.Arn
MaximumBatchingWindowInSeconds: !Ref MyBatchingWindowParam
StartingPosition: LATEST
DestinationConfig:
InvalidConfig:
Type: SNS
Destination: !Ref MySnsTopic

KinesisStream:
Type: AWS::Kinesis::Stream
Properties:
ShardCount: 1

MySnsTopic:
Type: AWS::SNS::Topic
16 changes: 16 additions & 0 deletions tests/translator/input/function_with_event_source_mapping.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,22 @@ Resources:
OnFailure:
Type: SQS
Destination: !GetAtt MySqsQueue.Arn
StreamEventWithoutDestinationConfigType:
Type: Kinesis
Properties:
Stream: !GetAtt KinesisStream1.Arn
MaximumBatchingWindowInSeconds: !Ref MyBatchingWindowParam
StartingPosition: LATEST
DestinationConfig:
OnFailure:
Destination: !Ref MySnsTopic
StreamEventWithEmptyDestinationConfig:
Type: Kinesis
Properties:
Stream: !GetAtt KinesisStream1.Arn
MaximumBatchingWindowInSeconds: !Ref MyBatchingWindowParam
StartingPosition: LATEST
DestinationConfig:

KinesisStream:
Type: AWS::Kinesis::Stream
Expand Down
8 changes: 8 additions & 0 deletions tests/translator/output/aws-cn/error_function_with_invalid_stream_eventsource_dest_type.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"errors": [
{
"errorMessage": "Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. The only valid values for 'Type' are 'SQS' and 'SNS'"
}
],
"errorMessage": "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. The only valid values for 'Type' are 'SQS' and 'SNS'"
}
8 changes: 8 additions & 0 deletions ...put/aws-cn/error_function_with_missing_on_failure_in_stream_event_destination_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"errors": [
{
"errorMessage": "Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. 'OnFailure' is a required field for 'DestinationConfig'"
}
],
"errorMessage": "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. 'OnFailure' is a required field for 'DestinationConfig'"
}
43 changes: 43 additions & 0 deletions tests/translator/output/aws-cn/function_with_event_source_mapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,49 @@
}
}
},
"MyFunctionForBatchingExampleStreamEventWithoutDestinationConfigType": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"MaximumBatchingWindowInSeconds": {
"Ref": "MyBatchingWindowParam"
},
"EventSourceArn": {
"Fn::GetAtt": [
"KinesisStream1",
"Arn"
]
},
"FunctionName": {
"Ref": "MyFunctionForBatchingExample"
},
"StartingPosition": "LATEST",
"DestinationConfig": {
"OnFailure": {
"Destination": {
"Ref": "MySnsTopic"
}
}
}
}
},
"MyFunctionForBatchingExampleStreamEventWithEmptyDestinationConfig": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"MaximumBatchingWindowInSeconds": {
"Ref": "MyBatchingWindowParam"
},
"EventSourceArn": {
"Fn::GetAtt": [
"KinesisStream1",
"Arn"
]
},
"FunctionName": {
"Ref": "MyFunctionForBatchingExample"
},
"StartingPosition": "LATEST"
}
},
"KinesisStream": {
"Type": "AWS::Kinesis::Stream",
"Properties": {
Expand Down
8 changes: 8 additions & 0 deletions ...ranslator/output/aws-us-gov/error_function_with_invalid_stream_eventsource_dest_type.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"errors": [
{
"errorMessage": "Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. The only valid values for 'Type' are 'SQS' and 'SNS'"
}
],
"errorMessage": "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. The only valid values for 'Type' are 'SQS' and 'SNS'"
}
8 changes: 8 additions & 0 deletions ...aws-us-gov/error_function_with_missing_on_failure_in_stream_event_destination_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"errors": [
{
"errorMessage": "Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. 'OnFailure' is a required field for 'DestinationConfig'"
}
],
"errorMessage": "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. 'OnFailure' is a required field for 'DestinationConfig'"
}
43 changes: 43 additions & 0 deletions tests/translator/output/aws-us-gov/function_with_event_source_mapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,49 @@
}
}
},
"MyFunctionForBatchingExampleStreamEventWithoutDestinationConfigType": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"MaximumBatchingWindowInSeconds": {
"Ref": "MyBatchingWindowParam"
},
"EventSourceArn": {
"Fn::GetAtt": [
"KinesisStream1",
"Arn"
]
},
"FunctionName": {
"Ref": "MyFunctionForBatchingExample"
},
"StartingPosition": "LATEST",
"DestinationConfig": {
"OnFailure": {
"Destination": {
"Ref": "MySnsTopic"
}
}
}
}
},
"MyFunctionForBatchingExampleStreamEventWithEmptyDestinationConfig": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"MaximumBatchingWindowInSeconds": {
"Ref": "MyBatchingWindowParam"
},
"EventSourceArn": {
"Fn::GetAtt": [
"KinesisStream1",
"Arn"
]
},
"FunctionName": {
"Ref": "MyFunctionForBatchingExample"
},
"StartingPosition": "LATEST"
}
},
"KinesisStream": {
"Type": "AWS::Kinesis::Stream",
"Properties": {
Expand Down
8 changes: 8 additions & 0 deletions tests/translator/output/error_function_with_invalid_stream_eventsource_dest_type.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"errors": [
{
"errorMessage": "Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. The only valid values for 'Type' are 'SQS' and 'SNS'"
}
],
"errorMessage": "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. The only valid values for 'Type' are 'SQS' and 'SNS'"
}
8 changes: 8 additions & 0 deletions ...tor/output/error_function_with_missing_on_failure_in_stream_event_destination_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"errors": [
{
"errorMessage": "Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. 'OnFailure' is a required field for 'DestinationConfig'"
}
],
"errorMessage": "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyFunction] is invalid. Event with id [MyFunctionStreamEvent] is invalid. 'OnFailure' is a required field for 'DestinationConfig'"
}
43 changes: 43 additions & 0 deletions tests/translator/output/function_with_event_source_mapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,49 @@
}
}
},
"MyFunctionForBatchingExampleStreamEventWithoutDestinationConfigType": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"MaximumBatchingWindowInSeconds": {
"Ref": "MyBatchingWindowParam"
},
"EventSourceArn": {
"Fn::GetAtt": [
"KinesisStream1",
"Arn"
]
},
"FunctionName": {
"Ref": "MyFunctionForBatchingExample"
},
"StartingPosition": "LATEST",
"DestinationConfig": {
"OnFailure": {
"Destination": {
"Ref": "MySnsTopic"
}
}
}
}
},
"MyFunctionForBatchingExampleStreamEventWithEmptyDestinationConfig": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"MaximumBatchingWindowInSeconds": {
"Ref": "MyBatchingWindowParam"
},
"EventSourceArn": {
"Fn::GetAtt": [
"KinesisStream1",
"Arn"
]
},
"FunctionName": {
"Ref": "MyFunctionForBatchingExample"
},
"StartingPosition": "LATEST"
}
},
"KinesisStream": {
"Type": "AWS::Kinesis::Stream",
"Properties": {
Expand Down