chore(v2): Split powertools-idempotency module to sub-modules and add redis implementation #1513
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -35,7 +35,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |||||
| ... | ||||||
| <dependency> | ||||||
| <groupId>software.amazon.lambda</groupId> | ||||||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||||||
| <version>{{ powertools.version }}</version> | ||||||
| </dependency> | ||||||
| ... | ||||||
|
|
@@ -56,7 +56,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |||||
| <aspectLibraries> | ||||||
| <aspectLibrary> | ||||||
| <groupId>software.amazon.lambda</groupId> | ||||||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||||||
| </aspectLibrary> | ||||||
| </aspectLibraries> | ||||||
| </configuration> | ||||||
|
|
@@ -80,7 +80,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |||||
| ... | ||||||
| <dependency> | ||||||
| <groupId>software.amazon.lambda</groupId> | ||||||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||||||
| <version>{{ powertools.version }}</version> | ||||||
| </dependency> | ||||||
| ... | ||||||
|
|
@@ -101,7 +101,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |||||
| <aspectLibraries> | ||||||
| <aspectLibrary> | ||||||
| <groupId>software.amazon.lambda</groupId> | ||||||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||||||
| </aspectLibrary> | ||||||
| </aspectLibraries> | ||||||
| </configuration> | ||||||
|
|
@@ -131,7 +131,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |||||
| } | ||||||
|
|
||||||
| dependencies { | ||||||
| aspect 'software.amazon.lambda:powertools-idempotency-dynamodb:{{ powertools.version }}' | ||||||
| } | ||||||
|
|
||||||
| sourceCompatibility = 11 // or higher | ||||||
|
|
@@ -151,18 +151,20 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |||||
| } | ||||||
|
|
||||||
| dependencies { | ||||||
| aspect 'software.amazon.lambda:powertools-idempotency-dynamodb:{{ powertools.version }}' | ||||||
| } | ||||||
|
|
||||||
| sourceCompatibility = 1.8 | ||||||
| targetCompatibility = 1.8 | ||||||
| ``` | ||||||
|
|
||||||
| ### Required resources | ||||||
| Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your Lambda functions will need read and write access to it. | ||||||
| As of now, [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) and [Redis](https://redis.io/) are the supported persistnce layers. | ||||||
|
|
||||||
| #### Using Amazon DynamoDB as persistent storage layer | ||||||
|
There was a problem hiding this comment. Bit verbose - maybe "Using DynamoDB" is enough? |
||||||
|
|
||||||
| If you are using Amazon DynamoDB you'll need to create a table. | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| **Default table configuration** | ||||||
|
|
||||||
|
|
@@ -215,6 +217,70 @@ Resources: | |||||
| see 1WCU and 1RCU. Review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/) to | ||||||
| estimate the cost. | ||||||
|
|
||||||
| #### Using Redis as persistent storage layer | ||||||
|
|
||||||
|
|
||||||
| ##### Redis resources | ||||||
|
|
||||||
| You need an existing Redis service before setting up Redis as persistent storage layer provider. You can also use Redis compatible services like [Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis/) as persistent storage layer provider. | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| !!! tip "Tip:No existing Redis service?" | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| If you don't have an existing Redis service, we recommend using DynamoDB as persistent storage layer provider. | ||||||
|
There was a problem hiding this comment.
Suggested change
.... or some words that go to the "why is this a sensible default"
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| If you are using Redis you'll need to provide the Redis host, port, user and password as AWS Lambda environment variables. | ||||||
|
There was a problem hiding this comment. I think this should be configurable; I don't think we should force people to put passwords in env vars. Long term it would be cool if we had IAM integration here, short term enough to be able to provide the password somehow, having retrieved it e.g. using the (apologies if you can already do this - i haven't got to the code yet!) There was a problem hiding this comment. For IAM Integration, I am not sure, this would be an extra possibility, since Amazon ElasticCache is the only possible Redis deployment that people might use. There was a problem hiding this comment. I agree, and don't think either of these things needs to happen here. The whole redis rabbit whole we started down for a separate impl to prove the abstraction out; we have that without this other bits! There was a problem hiding this comment. yes, having password as an environment variable is not a good practice. It should be something the user passes in the "connection string", so that he can retrieve it from secrets manager |
||||||
| If you want to connect to a Redis cluster instead of a Standalone server, you need to enable Redis cluster mode by setting an AWS Lambda | ||||||
| environment variable `REDIS_CLUSTER_MODE` to `true` | ||||||
| In the following example, you can see a SAM template for deploying an AWS Lambda function by specifying the required environment variables. | ||||||
| If you provide a [custom Redis client](#Customizing Redis client) you can omit the environment variables declaration. | ||||||
|
|
||||||
| !!! warning "Warning: Avoid including a plain text secret directly in your template" | ||||||
| This can infer security implications | ||||||
|
|
||||||
| !!! warning "Warning: Large responses with Redis persistence layer" | ||||||
| When using this utility with Redis your function's responses must be smaller than 512MB. | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| Persisting larger items cannot might cause exceptions. | ||||||
|
There was a problem hiding this comment. problem in this sentence... |
||||||
|
|
||||||
| ```yaml hl_lines="9-13" title="AWS Serverless Application Model (SAM) example" | ||||||
| Resources: | ||||||
| IdempotencyFunction: | ||||||
| Type: AWS::Serverless::Function | ||||||
| Properties: | ||||||
| CodeUri: Function | ||||||
| Handler: helloworld.App::handleRequest | ||||||
| Environment: | ||||||
| Variables: | ||||||
| REDIS_HOST: %redis-host% | ||||||
| REDIS_PORT: %redis-port% | ||||||
| REDIS_USER: %redis-user% | ||||||
| REDIS_SECRET: %redis-secret% | ||||||
|
There was a problem hiding this comment. See how Python managed the connection: https://github.com/aws-powertools/powertools-lambda-python/pull/2567/files#diff-9aa1f6d28c9806e244b0bf7a7263e02efb02cf4986da20d1b8fc6bb931097884R48 There was a problem hiding this comment. Hello Java folks! It's awesome that you're adding support for Redis in Idempotency!! My 2 cents here. Two widespread use cases when using Redis are: 1 – Customers who want to use SSL connections to encrypt data in transit. I saw Jedis supports Do you think it is possible to add these 2 parameters in this implementation? For cases where the client wants to create a very specific connection with TLS, customizations, and others, we do the same as you: the customer can bring their own client. Thanks 🚀 There was a problem hiding this comment. Thanks @leandrodamascena for the feedback. I think it makes sense. There was a problem hiding this comment. With 0c1cc38 the option to provide a custom JedisClientConfig is added. |
||||||
| REDIS_CLUSTER_MODE: "true" | ||||||
| ``` | ||||||
| ##### VPC Access | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| Your AWS Lambda Function must be able to reach the Redis endpoint before using it for idempotency persistent storage layer. In most cases you will need to [configure VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html) for your AWS Lambda Function. Using a public accessible Redis is not recommended. | ||||||
|
|
||||||
| !!! tip "Amazon ElastiCache for Redis as persistent storage layer provider" | ||||||
| If you intend to use Amazon ElastiCache for Redis for idempotency persistent storage layer, you can also reference [This AWS Tutorial](https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html). | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| !!! warning "Amazon ElastiCache Serverless not supported" | ||||||
| [Amazon ElastiCache Serverless](https://aws.amazon.com/elasticache/features/#Serverless) is not supported for now. | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| !!! warning "Check network connectivity to Redis server" | ||||||
| Make sure that your AWS Lambda function can connect to your Redis server. | ||||||
|
There was a problem hiding this comment. not sure it is super useful, or explain how? but better would be to explain the security group config (if any), or any other networking config. |
||||||
|
|
||||||
| ```yaml hl_lines="9-12" title="AWS Serverless Application Model (SAM) example" | ||||||
| Resources: | ||||||
| IdempotencyFunction: | ||||||
| Type: AWS::Serverless::Function | ||||||
| Properties: | ||||||
| CodeUri: Function | ||||||
| Handler: helloworld.App::handleRequest | ||||||
| VpcConfig: # (1)! | ||||||
| SecurityGroupIds: | ||||||
| - sg-{your_sg_id} | ||||||
|
There was a problem hiding this comment. probably deserve more explanations: SG of elasticache / SG of the function? what would be the link between the two SGs (port 6379, 6380 ?) |
||||||
| SubnetIds: | ||||||
| - subnet-{your_subnet_id_1} | ||||||
| - subnet-{your_subnet_id_2} | ||||||
| ``` | ||||||
| 1. Replace the Security Group ID and Subnet ID to match your Redis' VPC setting. | ||||||
|
|
||||||
| ### Idempotent annotation | ||||||
|
|
||||||
| You can quickly start by initializing the `DynamoDBPersistenceStore` and using it with the `@Idempotent` annotation on your Lambda handler. | ||||||
|
|
@@ -635,6 +701,29 @@ When using DynamoDB as a persistence layer, you can alter the attribute names by | |||||
| | **SortKeyAttr** | | | Sort key of the table (if table is configured with a sort key). | | ||||||
| | **StaticPkValue** | | `idempotency#{LAMBDA_FUNCTION_NAME}` | Static value to use as the partition key. Only used when **SortKeyAttr** is set. | | ||||||
|
|
||||||
| #### RedisPersistenceStore | ||||||
|
|
||||||
| The redis persistence store has as a prerequisite to install a Redis datastore(https://redis.io/docs/about/) in either Standalone or Cluster mode. | ||||||
|
|
||||||
| We are using [Redis hashes](https://redis.io/docs/data-types/hashes/) to store the idempotency fields and values. | ||||||
| There are some predefined fields that you can see listed in the following table. The predefined fields have some default values. | ||||||
|
|
||||||
|
|
||||||
| You can alter the field names by passing these parameters when initializing the persistence layer: | ||||||
|
|
||||||
| | Parameter | Required | Default | Description | | ||||||
| |--------------------|----------|--------------------------------------|--------------------------------------------------------------------------------------------------------| | ||||||
| | **KeyPrefixName** | Y | `idempotency` | The redis hash key prefix | | ||||||
| | **KeyAttr** | Y | `id` | The redis hash key field name | | ||||||
| | **ExpiryAttr** | | `expiration` | Unix timestamp of when record expires | | ||||||
| | **StatusAttr** | | `status` | Stores status of the Lambda execution during and after invocation | | ||||||
| | **DataAttr** | | `data` | Stores results of successfully idempotent methods | | ||||||
| | **ValidationAttr** | | `validation` | Hashed representation of the parts of the event used for validation | | ||||||
|
|
||||||
|
|
||||||
| !!! Tip "Tip: You can share the same prefix and key for all functions" | ||||||
| You can reuse the same prefix and key to store idempotency state. We add your function name in addition to the idempotency key as a hash key. | ||||||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| ## Advanced | ||||||
|
|
||||||
| ### Customizing the default behavior | ||||||
|
|
@@ -884,6 +973,39 @@ When creating the `DynamoDBPersistenceStore`, you can set a custom [`DynamoDbCli | |||||
| .build(); | ||||||
| ``` | ||||||
|
|
||||||
| ### Customizing Redis client | ||||||
|
|
||||||
| The `RedisPersistenceStore` uses the JedisPooled java client to connect to the Redis Server. | ||||||
|
|
||||||
| When creating the `RedisPersistenceStore`, you can set a custom [`JedisPooled`](https://www.javadoc.io/doc/redis.clients/jedis/latest/redis/clients/jedis/JedisPooled.html) client: | ||||||
|
There was a problem hiding this comment. link at first usage, move it above. |
||||||
|
|
||||||
| === "Custom JedisPooled with connection timeout" | ||||||
|
|
||||||
| ```java hl_lines="2-6 11" | ||||||
| public App() { | ||||||
| JedisPooled jedisPooled = new JedisPooled(new HostAndPort("host",6789), DefaultJedisClientConfig.builder() | ||||||
| .user("user") | ||||||
| .password("secret") | ||||||
| .connectionTimeoutMillis(3000) | ||||||
| .build()) | ||||||
|
|
||||||
| Idempotency.config().withPersistenceStore( | ||||||
| RedisPersistenceStore.builder() | ||||||
| .withKeyPrefixName("items-idempotency") | ||||||
| .withJedisPooled(jedisPooled) | ||||||
| .build() | ||||||
| ).configure(); | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| !!! info "Default configuration is the following:" | ||||||
|
There was a problem hiding this comment. What do you mean by "default configuration", it is what we use as default ? If that's the case, does it make sense? Not sure anyone is using the user "default" with no password... I think we should ask for all the required fields to create a JedisClient ourself:
@scottgerring wdyt ? There was a problem hiding this comment. Default here means, as provided by the jedis client default config. Actually I have to update the powertools documentation, since user/pass, by default, in JedisClient are null. There was a problem hiding this comment. Found also this extract here from redis.conf
|
||||||
|
|
||||||
| ```java | ||||||
| DefaultJedisClientConfig.builder() | ||||||
| .user(System.getenv(Constants.REDIS_USER)) | ||||||
| .password(System.getenv(Constants.REDIS_SECRET)) | ||||||
|
There was a problem hiding this comment. we cannot do this, sorry |
||||||
| .build(); | ||||||
| ``` | ||||||
|
|
||||||
| ### Using a DynamoDB table with a composite primary key | ||||||
|
|
||||||
| When using a composite primary key table (hash+range key), use `SortKeyAttr` parameter when initializing your persistence store. | ||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -381,7 +381,6 @@ | |
| </configuration> | ||
| <executions> | ||
| <execution> | ||
| <goals> | ||
| <goal>compile</goal> | ||
| <goal>test-compile</goal> | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
|
|
||
| <parent> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>e2e-test-handlers-parent</artifactId> | ||
| <version>1.0.0</version> | ||
| </parent> | ||
|
|
||
| <artifactId>e2e-test-handler-idempotency-dynamodb</artifactId> | ||
| <packaging>jar</packaging> | ||
| <name>A Lambda function using Powertools for AWS Lambda (Java) idempotency</name> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-logging</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.logging.log4j</groupId> | ||
| <artifactId>log4j-slf4j2-impl</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.amazonaws</groupId> | ||
| <artifactId>aws-lambda-java-events</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.aspectj</groupId> | ||
| <artifactId>aspectjrt</artifactId> | ||
| </dependency> | ||
| </dependencies> | ||
|
|
||
| <build> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>dev.aspectj</groupId> | ||
| <artifactId>aspectj-maven-plugin</artifactId> | ||
| <configuration> | ||
| <source>${maven.compiler.source}</source> | ||
| <target>${maven.compiler.target}</target> | ||
| <complianceLevel>${maven.compiler.target}</complianceLevel> | ||
| <aspectLibraries> | ||
| <aspectLibrary> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-core</artifactId> | ||
| </aspectLibrary> | ||
| <aspectLibrary> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-logging</artifactId> | ||
| </aspectLibrary> | ||
| </aspectLibraries> | ||
| </configuration> | ||
| <executions> | ||
| <execution> | ||
| <goals> | ||
| <goal>compile</goal> | ||
| </goals> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-shade-plugin</artifactId> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
| </project> |
Uh oh!
There was an error while loading. Please reload this page.