Skip to content

chore(docs): improve idempotency documentation #1655

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/snippets/idempotency/makeIdempotentJmes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ const createSubscriptionPayment = async (

// Extract the idempotency key from the request headers
const config = new IdempotencyConfig({
eventKeyJmesPath: 'headers."X-Idempotency-Key"',
eventKeyJmesPath: 'body',
});

export const handler = makeIdempotent(
Expand Down
76 changes: 45 additions & 31 deletions docs/utilities/idempotency.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,12 +52,22 @@ classDiagram

## Getting started

### IAM Permissions
### Installation
Install the library in your project
```shell
npm i @aws-lambda-powertools/idempotency @aws-sdk/client-dynamodb
```

While we support Amazon DynamoDB as a persistance layer out of the box, you need to bring your own AWS SDK for JavaScript v3 DynamoDB client.

Your Lambda function IAM Role must have `dynamodb:GetItem`, `dynamodb:PutItem`, `dynamodb:UpdateItem` and `dynamodb:DeleteItem` IAM permissions before using this feature.

???+ note
If you're using one of our examples: [AWS Serverless Application Model (SAM)](#required-resources) or [Terraform](#required-resources) the required permissions are already included.
This utility supports **[AWS SDK for JavaScript v3](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/){target="_blank"} only**. If you are using the `nodejs18.x` runtime, the AWS SDK for JavaScript v3 is already installed and you can install only the utility.


### IAM Permissions

Your Lambda function IAM Role must have `dynamodb:GetItem`, `dynamodb:PutItem`, `dynamodb:UpdateItem` and `dynamodb:DeleteItem` IAM permissions before using this feature. If you're using one of our examples: [AWS Serverless Application Model (SAM)](#required-resources) or [Terraform](#required-resources) the required permissions are already included.

### Required resources

Expand All @@ -69,10 +79,10 @@ As of now, Amazon DynamoDB is the only supported persistent storage layer, so yo

If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencelayer), this is the expected default configuration:

| Configuration | Value | Notes |
| ------------------ | ------------ | ----------------------------------------------------------------------------------- |
| Partition key | `id` |
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console |
| Configuration | Default value | Notes |
| ------------------ |:--------------|-----------------------------------------------------------------------------------------|
| Partition key | `id` | The id of each idempotency record which a combination of `functionName#hashOfPayload`. |
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console. |

???+ tip "Tip: You can share a single state table for all functions"
You can reuse the same DynamoDB table to store idempotency state. We add the Lambda function name in addition to the idempotency key as a hash key.
Expand Down Expand Up @@ -212,26 +222,26 @@ If you're not [changing the default configuration for the DynamoDB persistence l

You can quickly start by initializing the `DynamoDBPersistenceLayer` class and using it with the `makeIdempotent` function wrapper on your Lambda handler.

???+ note
In this example, the entire Lambda handler is treated as a single idempotent operation. If your Lambda handler can cause multiple side effects, or you're only interested in making a specific logic idempotent, use the `makeIdempotent` high-order function only on the function that needs to be idempotent.

!!! tip "See [Choosing a payload subset for idempotency](#choosing-a-payload-subset-for-idempotency) for more elaborate use cases."

=== "index.ts"

```typescript hl_lines="2-3 21 35-38"
--8<-- "docs/snippets/idempotency/makeIdempotentBase.ts"
```

=== "Types"
=== "types.ts"

```typescript
--8<-- "docs/snippets/idempotency/types.ts::13"
```

After processing this request successfully, a second request containing the exact same payload above will now return the same response, ensuring our customer isn't charged twice.

!!! question "New to idempotency concept? Please review our [Terminology](#terminology) section if you haven't yet."

???+ note
In this example, the entire Lambda handler is treated as a single idempotent operation. If your Lambda handler can cause multiple side effects, or you're only interested in making a specific logic idempotent, use the `makeIdempotent` high-order function only on the function that needs to be idempotent.

See [Choosing a payload subset for idempotency](#choosing-a-payload-subset-for-idempotency) for more elaborate use cases.


You can also use the `makeIdempotent` function wrapper on any function that returns a response to make it idempotent. This is useful when you want to make a specific logic idempotent, for example when your Lambda handler performs multiple side effects and you only want to make a specific one idempotent.

Expand All @@ -240,21 +250,21 @@ You can also use the `makeIdempotent` function wrapper on any function that retu

When using `makeIdempotent` on arbitrary functions, you can tell us which argument in your function signature has the data we should use via **`dataIndexArgument`**. If you don't specify this argument, we'll use the first argument in the function signature.

???+ note
The function in the example below has two arguments, note that while wrapping it with the `makeIdempotent` high-order function, we specify the `dataIndexArgument` as `1` to tell the decorator that the second argument is the one that contains the data we should use to make the function idempotent. Remember that arguments are zero-indexed, so the first argument is `0`, the second is `1`, and so on.

=== "index.ts"

```typescript hl_lines="22 34-38"
--8<-- "docs/snippets/idempotency/makeIdempotentAnyFunction.ts"
```

=== "Types"
=== "types.ts"

```typescript
--8<-- "docs/snippets/idempotency/types.ts::13"
```

The function this example has two arguments, note that while wrapping it with the `makeIdempotent` high-order function, we specify the `dataIndexArgument` as `1` to tell the decorator that the second argument is the one that contains the data we should use to make the function idempotent. Remember that arguments are zero-indexed, so the first argument is `0`, the second is `1`, and so on.


### MakeHandlerIdempotent Middy middleware

!!! tip "A note about Middy"
Expand All @@ -269,28 +279,25 @@ If you are using [Middy](https://middy.js.org){target="_blank"} as your middlewa
--8<-- "docs/snippets/idempotency/makeHandlerIdempotent.ts"
```

=== "Types"
=== "types.ts"

```typescript
--8<-- "docs/snippets/idempotency/types.ts::13"
```

### Choosing a payload subset for idempotency

???+ tip "Tip: Dealing with always changing payloads"
When dealing with a more elaborate payload, where parts of the payload always change, you should use the **`eventKeyJmesPath`** parameter.

Use [`IdempotencyConfig`](#customizing-the-default-behavior) to instruct the idempotent decorator to only use a portion of your payload to verify whether a request is idempotent, and therefore it should not be retried.
Use [`IdempotencyConfig`](#customizing-the-default-behavior) to instruct the idempotent decorator to only use a portion of your payload to verify whether a request is idempotent, and therefore it should not be retried. When dealing with a more elaborate payload, where parts of the payload always change, you should use the **`eventKeyJmesPath`** parameter.

> **Payment scenario**
**Payment scenario**

In this example, we have a Lambda handler that creates a payment for a user subscribing to a product. We want to ensure that we don't accidentally charge our customer by subscribing them more than once.

Imagine the function executes successfully, but the client never receives the response due to a connection issue. It is safe to retry in this instance, as the idempotent decorator will return a previously saved response.

**What we want here** is to instruct Idempotency to use the `user` and `productId` fields from our incoming payload as our idempotency key. If we were to treat the entire request as our idempotency key, a simple HTTP header or timestamp change would cause our customer to be charged twice.

???+ tip "Deserializing JSON strings in payloads for increased accuracy."
???+ warning "Deserializing JSON strings in payloads for increased accuracy."
The payload extracted by the `eventKeyJmesPath` is treated as a string by default. This means there could be differences in whitespace even when the JSON payload itself is identical.

=== "index.ts"
Expand Down Expand Up @@ -334,26 +341,26 @@ Imagine the function executes successfully, but the client never receives the re
}
```

=== "Types"
=== "types.ts"

```typescript
--8<-- "docs/snippets/idempotency/types.ts::13"
```

### Lambda timeouts

???+ note
This is automatically done when you wrap your Lambda handler with the [makeIdempotent](#makeIdempotent-function-wrapper) function wrapper, or use the [`makeHandlerIdempotent`](#makeHandlerIdempotent-middy-middleware) Middy middleware.

To prevent against extended failed retries when a [Lambda function times out](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-verify-invocation-timeouts/), Powertools for AWS calculates and includes the remaining invocation available time as part of the idempotency record.

To prevent against extended failed retries when a [Lambda function times out](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-verify-invocation-timeouts/), Powertools for AWS Lambda calculates and includes the remaining invocation available time as part of the idempotency record.
This is automatically done when you wrap your Lambda handler with the [makeIdempotent](#makeIdempotent-function-wrapper) function wrapper, or use the [`makeHandlerIdempotent`](#makeHandlerIdempotent-middy-middleware) Middy middleware.

???+ example
If a second invocation happens **after** this timestamp, and the record is marked as `INPROGRESS`, we will execute the invocation again as if it was in the `EXPIRED` state (e.g, `expire_seconds` field elapsed).

This means that if an invocation expired during execution, it will be quickly executed again on the next retry.

???+ important
If you are only using the [makeIdempotent function wrapper](#makeIdempotent-function-wrapper) to guard isolated parts of your code, you must use `registerLambdaContext` available in the [idempotency config object](#customizing-the-default-behavior) to benefit from this protection.
If you are only using the [makeIdempotent function wrapper](#makeIdempotent-function-wrapper) to guard isolated parts of your code outside of your handler, you must use `registerLambdaContext` available in the [idempotency config object](#customizing-the-default-behavior) to benefit from this protection.

Here is an example on how you register the Lambda context in your handler:

Expand All @@ -371,6 +378,7 @@ This means that new invocations will execute your code again despite having the
<center>
```mermaid
sequenceDiagram
autonumber
participant Client
participant Lambda
participant Persistence Layer
Expand Down Expand Up @@ -410,6 +418,7 @@ The following sequence diagrams explain how the Idempotency feature behaves unde
<center>
```mermaid
sequenceDiagram
autonumber
participant Client
participant Lambda
participant Persistence Layer
Expand Down Expand Up @@ -444,6 +453,7 @@ sequenceDiagram
<center>
```mermaid
sequenceDiagram
autonumber
participant Client
participant Lambda
participant Persistence Layer
Expand Down Expand Up @@ -474,6 +484,7 @@ sequenceDiagram
<center>
```mermaid
sequenceDiagram
autonumber
participant Client
participant Lambda
participant Persistence Layer
Expand Down Expand Up @@ -509,6 +520,7 @@ sequenceDiagram
<center>
```mermaid
sequenceDiagram
autonumber
participant Client
participant Lambda
participant Persistence Layer
Expand Down Expand Up @@ -537,6 +549,7 @@ sequenceDiagram
<center>
```mermaid
sequenceDiagram
autonumber
participant Client
participant Lambda
participant Persistence Layer
Expand Down Expand Up @@ -570,6 +583,7 @@ sequenceDiagram
<center>
```mermaid
sequenceDiagram
autonumber
participant Client
participant Lambda
participant Persistence Layer
Expand Down Expand Up @@ -794,4 +808,4 @@ The example function above would cause data to be stored in DynamoDB like this:
## Extra resources

If you're interested in a deep dive on how Amazon uses idempotency when building our APIs, check out
[this article](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/){target="_blank"}.
[this article](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/){target="_blank"}.