Skip to content

SAM v1.16.0 Release: Virtual Sharding and Stream failure processing support for streaming events

Choose a tag to compare

View all tags
@ShreyaGangishetty ShreyaGangishetty released this 26 Nov 16:34
· 216 commits to master since this release
v1.16.0
27b5934
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

SAM v1.16.0 Release: Virtual Sharding and Stream failure processing support for streaming events

This release adds support for ParallelizationFactor, MaximumRetryAttempts, BisectBatchOnFunctionError, MaximumRecordAgeInSeconds, and DestinationConfig properties for Kinesis and DynamoDB event types.

ParallelizationFactor property can be set to increase concurrent Lambda invocations for each shard, which by default is 1. This allows for faster stream processing without the need to over-scale the number of shards, while still guaranteeing order of records processed.

Lambda functions can skip retrying a batch of records when it has reached the value set in the MaximumRetryAttempts property, which can be configured from 0 to 10,000.

Lambda functions can skip processing a data record when it has reached the value set in MaximumRecordAgeInSeconds property, which can be configured from 60 seconds to 7 days.

Lambda functions can continue processing a shard even when it returns an error. When a data record reaches the Maximum Retry Attempts or Maximum Record Age, you can send its metadata like shard ID and stream ARN to an SQS queue or SNS topic by setting that configuration in DestinationConfig

BisectBatchOnFunctionError allows a customer to have retried invocations contain a smaller number of records. With Bisect on Function Error enabled, Lambda splits the impacted batch of records into two when a function returns an error, and retries them separately. This allows you to easily separate the malformed data record from the rest of the batch, and process the rest of data records successfully.

Resources:
  StreamProcessor:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs10.x
      CodeUri: .
      Events:
        Stream:
          Type: Kinesis
          Properties:
            Stream: !GetAtt Stream.Arn
            ParallelizationFactor: 8
            MaximumRetryAttempts: 100
            BisectBatchOnFunctionError: true
            MaximumRecordAgeInSeconds: 604800
            DestinationConfig:
              OnFailure:
                Destination: !GetAtt MySqsQueue.Arn

Changelog

  1. (#1261 ) feat: support virtual sharding and stream failure processing
Assets 2
Loading