Skip to content

feat(jmespath): new built-in envelopes to unwrap S3 events #2169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 1, 2023
Merged

feat(jmespath): new built-in envelopes to unwrap S3 events #2169

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ef81b2f
feat(jmespath): adding new envelopes + docs + small fix
leandrodamascena Apr 26, 2023
0de8370
feat(jmespath): text
leandrodamascena Apr 27, 2023
b13a6f9
Merge branch 'develop' into jmespath/envelopes
leandrodamascena Apr 27, 2023
7528d07
chore: addressing Ruben's feedback
leandrodamascena Apr 27, 2023
a5152dc
feat(jmespath): addressing Heitor's feedback
leandrodamascena Apr 28, 2023
ef54196
chore: break message in two sentences.
heitorlessa May 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions aws_lambda_powertools/utilities/jmespath_utils/envelopes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,8 @@
CLOUDWATCH_EVENTS_SCHEDULED = EVENTBRIDGE
KINESIS_DATA_STREAM = "Records[*].kinesis.powertools_json(powertools_base64(data))"
CLOUDWATCH_LOGS = "awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]"
S3_SNS_SQS = "Records[*].powertools_json(body).powertools_json(Message).Records[0]"
S3_RAWSNS_SQS = "Records[*].powertools_json(body).Records[0]"
S3_SNS_KINESIS_FIREHOSE = "records[*].powertools_json(powertools_base64(data)).powertools_json(Message).Records[0]"
S3_RAWSNS_KINESIS_FIREHOSE = "records[*].powertools_json(powertools_base64(data)).Records[0]"
S3_EVENTBRIDGE_SQS = "Records[*].powertools_json(body).detail"
28 changes: 18 additions & 10 deletions docs/utilities/jmespath_functions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,24 @@ We provide built-in envelopes for popular AWS Lambda event sources to easily dec

These are all built-in envelopes you can use along with their expression as a reference:

| Envelope | JMESPath expression |
| --------------------------------- | ------------------------------------------------------------- |
| **`API_GATEWAY_REST`** | `powertools_json(body)` |
| **`API_GATEWAY_HTTP`** | `API_GATEWAY_REST` |
| **`SQS`** | `Records[*].powertools_json(body)` |
| **`SNS`** | `Records[0].Sns.Message | powertools_json(@)` |
| **`EVENTBRIDGE`** | `detail` |
| **`CLOUDWATCH_EVENTS_SCHEDULED`** | `EVENTBRIDGE` |
| **`KINESIS_DATA_STREAM`** | `Records[*].kinesis.powertools_json(powertools_base64(data))` |
| **`CLOUDWATCH_LOGS`** | `awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]` |
| Envelope | JMESPath expression |
| --------------------------------- | ------------------------------------------------------------------------------------------ |
| **`API_GATEWAY_HTTP`** | `powertools_json(body)` |
| **`API_GATEWAY_REST`** | `powertools_json(body)` |
| **`CLOUDWATCH_EVENTS_SCHEDULED`** | `detail` |
| **`CLOUDWATCH_LOGS`** | `awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]` |
| **`EVENTBRIDGE`** | `detail` |
| **`KINESIS_DATA_STREAM`** | `Records[*].kinesis.powertools_json(powertools_base64(data))` |
| **`S3_EVENTBRIDGE_SQS`** | `Records[*].powertools_json(body).detail` |
| **`S3_RAWSNS_KINESIS_FIREHOSE`** | `records[*].powertools_json(powertools_base64(data)).Records[0]` |
| **`S3_RAWSNS_SQS`** | `Records[*].powertools_json(body).Records[0]` |
| **`S3_SNS_KINESIS_FIREHOSE`** | `records[*].powertools_json(powertools_base64(data)).powertools_json(Message).Records[0]` |
| **`S3_SNS_SQS`** | `Records[*].powertools_json(body).powertools_json(Message).Records[0]` |
| **`SNS`** | `Records[0].Sns.Message | powertools_json(@)` |
| **`SQS`** | `Records[*].powertools_json(body)` |

???+ tip
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Niiiice!! You can make it slightly better by:

  • Adding a title so ESL readers know upfront if they should care or not e.g., "Using SNS?"
  • Educate the reader that if they do need SNS event metadata they don't need to enable it

Write, review, cut the unnecessary words without losing its original intent, and ship it <3

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Added the title
  • I thought of the following mental model to improve this text, and see if it makes sense.
    • Benefits - Reduce payload size
    • How to do - enabling raw message
    • why? - architectures that require sending event notifications from S3 to SNS but not require SNS metadata

If you're using architectures that involve sending event notifications from S3 to SNS and then to SQS or Kinesis Data Firehose, consider enabling [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html){target="_blank"} option in Amazon SNS. This can help reduce the payload by stripping the SNS metadata.

## Advanced

Expand Down
20 changes: 10 additions & 10 deletions docs/utilities/validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,16 +141,16 @@ We provide built-in envelopes to easily extract the payload from popular event s

Here is a handy table with built-in envelopes along with their JMESPath expressions in case you want to build your own.

| Envelope name | JMESPath expression |
| ------------------------------- | ------------------------------------------------------------- |
| **API_GATEWAY_REST** | "powertools_json(body)" |
| **API_GATEWAY_HTTP** | "powertools_json(body)" |
| **SQS** | "Records[*].powertools_json(body)" |
| **SNS** | "Records[0].Sns.Message | powertools_json(@)" |
| **EVENTBRIDGE** | "detail" |
| **CLOUDWATCH_EVENTS_SCHEDULED** | "detail" |
| **KINESIS_DATA_STREAM** | "Records[*].kinesis.powertools_json(powertools_base64(data))" |
| **CLOUDWATCH_LOGS** | "awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]" |
| Envelope | JMESPath expression |
| --------------------------------- | ------------------------------------------------------------------------ |
| **`API_GATEWAY_HTTP`** | `powertools_json(body)` |
| **`API_GATEWAY_REST`** | `powertools_json(body)` |
| **`CLOUDWATCH_EVENTS_SCHEDULED`** | `detail` |
| **`CLOUDWATCH_LOGS`** | `awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]` |
| **`EVENTBRIDGE`** | `detail` |
| **`KINESIS_DATA_STREAM`** | `Records[*].kinesis.powertools_json(powertools_base64(data))` |
| **`SNS`** | `Records[0].Sns.Message | powertools_json(@)` |
| **`SQS`** | `Records[*].powertools_json(body)` |

## Advanced

Expand Down
10 changes: 7 additions & 3 deletions examples/jmespath_functions/src/extract_data_from_builtin_envelope.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.jmespath_utils import (
envelopes,
extract_data_from_envelope,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

logger = Logger()


def handler(event: dict, context: LambdaContext) -> dict:
payload = extract_data_from_envelope(data=event, envelope=envelopes.SQS)
customer_id = payload.get("customerId") # now deserialized
records = extract_data_from_envelope(data=event, envelope=envelopes.SQS)
for record in records: # records is a list
logger.info(record.get("customerId")) # now deserialized

return {"customer_id": customer_id, "message": "success", "statusCode": 200}
return {"message": "success", "statusCode": 200}