Lambda Extensions #376

calavera · 2021-11-29T19:31:46Z

Issue #, if available:

Fixes #261

Description of changes:

This PR introduces two new creates into this project to be able to implement Lambda Extensions in Rust. The work is not completed, and I wanted to share it with you all now that I have functional examples to get people's feedback on the direction I'm taking.

The first crate is lambda-runtime-client. I've extracted this crate from the current runtime. The idea with this is to be able to reuse clients for different APIs. This brings some guarantees about client behaviors, like having a User-Agent header by default.

The second crate is lambda-extension. This crate implements the Lambda Extensions API, which is the main point of this PR. I've included two examples in this new crate so I can get feedback about the API exposed to developers that want to create extensions in Rust.

Basic Example
Example that subscribes to specific events

There are a few things missing to get this PR in a mergeable state, mostly documentation, but I'll appreciate your input sooner rather than later so I can make these changes better.

By submitting this pull request

I confirm that my contribution is made under the terms of the Apache 2.0 license.
I confirm that I've made a best effort attempt to update all relevant documentation.

Extract the HTTP client logic into a new crate. This allows other crates to share the same logic to interact with the Runtime API. Signed-off-by: David Calavera <[email protected]>

This new crate encapsulates the logic to create Lambda Extensions. It includes reference examples. Signed-off-by: David Calavera <[email protected]>

nmoutschen · 2021-11-29T19:47:01Z

Thanks a lot for taking the time to create this PR @calavera ! I'll take a look at it tomorrow. 😄

lambda-extension/Cargo.toml

nmoutschen

Hey @calavera!

Thanks a lot for the time and efforts in creating this PR! Overall it's pretty solid and I'm for the idea of splitting the API client into its separate crate.

In term of potential improvements before merging this, there are three main points for me:

I'm not too keen on adding a dependency on async_trait, as it would diverge from how the Handler trait works at the moment.
I think we should push error handling under the hood as much as possible by catching the Errs and sending the right API call on behalf of the developer.
Right now, lambda-extension listens to both INVOKE and SHUTDOWN events. While doing an Ok(()) is not a big deal, I think it would be nice to be able to configure what events to listen to, and then get an enum as payload.

Cargo.toml

lambda-extension/Cargo.toml

lambda-extension/examples/basic.rs

lambda-extension/src/requests.rs

lambda-extension/examples/error_handling.rs

lambda-extension/src/lib.rs

Once upon a time, there was a crate called `lambda-runtime-client`. We don't want people to mistake this new crate with the old one. Signed-off-by: David Calavera <[email protected]>

- Remove async_trait dependency. - Use a similar handler api than the runtime. - Allow to register the extension for only certain events. Signed-off-by: David Calavera <[email protected]>

calavera · 2021-12-02T23:56:35Z

@nmoutschen I've updated this PR to address your comments. Can you take a look at the examples now and let me know what you think?

nmoutschen

Thanks a lot for the update @calavera!

I really like the changes you made! The main change I think would help here before I can approve is on the Extension trait signature, by making self mutable to be in line with recent changes in this repository (but not published yet).

I also have some other small remarks/nitpicks, but up to you to decide if you agree with me or not.

lambda-extension/src/lib.rs

lambda-extension/examples/basic.rs

lambda-extension/examples/custom_events.rs

- Remove extension ID from call signature. - Make extension mutable. - Add example showing how to implement an extension with a struct. Signed-off-by: David Calavera <[email protected]>

nmoutschen

LGTM from an implementation POV!

lambda-extension/examples/custom_trait_implementation.rs

Signed-off-by: David Calavera <[email protected]>

calavera · 2021-12-04T01:42:41Z

@nmoutschen I've added documentation where it was necessary and opened the PR for review. Thanks a lot for your feedback.

Signed-off-by: David Calavera <[email protected]>

calavera · 2021-12-04T23:03:05Z

I still have to update the README with information about this new crate.

Signed-off-by: David Calavera <[email protected]>

calavera · 2021-12-07T00:14:06Z

@nmoutschen I was not sure where to add information to the main readme, so I opted for creating two specific readmes inside the new crate directories. Let me know what you think.

nmoutschen

LGTM! Just 2 small details on libc for various OS versions, and on how users probably won't use the runtime client directly.

lambda-extension/README.md

lambda-runtime-api-client/README.md

Signed-off-by: David Calavera <[email protected]>

calavera · 2021-12-07T18:17:00Z

@nmoutschen I decided to remove the paragraph about targets explicitly. I think it was confusing and hard to explain. I'd expect that people that want to write extensions are already familiar with functions, and how the runtime works. It's still worth mentioning that the target must match the architecture that functions run on.

calavera · 2021-12-10T16:33:43Z

@nmoutschen what are the next steps to get this PR merged?

nmoutschen · 2021-12-10T16:48:47Z

Hey @calavera !

Not much on your side. I wanted to run a complete test by building an extension and running on Lambda before merging this just to make sure. I'm on sick leave at the moment, but I should be able to do it next week.

calavera · 2021-12-10T17:04:20Z

Not much on your side. I wanted to run a complete test by building an extension and running on Lambda before merging this just to make sure. I'm on sick leave at the moment, but I should be able to do it next week.

Thanks for letting me know, I wasn't sure if there was anything missing. Take care

nmoutschen · 2021-12-10T21:06:59Z

lambda-extension/src/lib.rs

+            Some(name) => name.into(),
+            None => {
+                let args: Vec<String> = std::env::args().collect();
+                args[0].clone()


After doing some tests on Lambda, this causes an issue with the Lambda Extensions API. Because this value contains slashes (e.g. /opt/extension/my-extension), the Lambda Extension API will fail to process it correctly, and assumes that this is an internal extension.

Because of this, it will fail to register the extension if it contains/starts with (not sure which one) a slash.

However, taking the last part of the path works. E.g. instead of /opt/extension/my-extension, my-extension works.

(Note: this looks like an odd behavior and is not documented, so I'm raising this with the Lambda team, but would still be better to make that change)

good catch! I've updated this code to cleanup the executable name.

nmoutschen · 2021-12-10T21:09:05Z

Hey @calavera ! I just found a bug when running a sample Lambda function with a layer, but otherwise everything else works when I changed the full extension path for its basename. Once this is fixed, I'll do another pass, but we should be good to merge.

Cleanup the path from the executable when it takes the name from arg[0]. Signed-off-by: David Calavera <[email protected]>

calavera · 2021-12-11T02:33:59Z

@nmoutschen I pushed a new commit that should fixed that problem 🙌

nmoutschen · 2021-12-11T10:32:38Z

All good now! Merged the PR! Thank again for your contribution here! 🙌

calavera added 2 commits November 29, 2021 11:19

Create new Runtime Client crate

046d380

Extract the HTTP client logic into a new crate. This allows other crates to share the same logic to interact with the Runtime API. Signed-off-by: David Calavera <[email protected]>

Add Lambda Extension crate

19e1ceb

This new crate encapsulates the logic to create Lambda Extensions. It includes reference examples. Signed-off-by: David Calavera <[email protected]>

rimutaka reviewed Nov 29, 2021

View reviewed changes

lambda-extension/Cargo.toml Outdated Show resolved Hide resolved

nmoutschen suggested changes Nov 30, 2021

View reviewed changes

Rename client crate to avoid confusion.

9d46537

Once upon a time, there was a crate called `lambda-runtime-client`. We don't want people to mistake this new crate with the old one. Signed-off-by: David Calavera <[email protected]>

calavera force-pushed the extensions branch from f3bc34b to 9d46537 Compare December 1, 2021 18:48

Modify user API.

a7161be

- Remove async_trait dependency. - Use a similar handler api than the runtime. - Allow to register the extension for only certain events. Signed-off-by: David Calavera <[email protected]>

nmoutschen suggested changes Dec 3, 2021

View reviewed changes

lambda-extension/src/lib.rs Outdated Show resolved Hide resolved

lambda-extension/examples/basic.rs Show resolved Hide resolved

lambda-extension/examples/custom_events.rs Show resolved Hide resolved

Cleanup user API.

af95f36

- Remove extension ID from call signature. - Make extension mutable. - Add example showing how to implement an extension with a struct. Signed-off-by: David Calavera <[email protected]>

nmoutschen approved these changes Dec 3, 2021

View reviewed changes

lambda-extension/examples/custom_trait_implementation.rs Outdated Show resolved Hide resolved

Add documentation and cleanup code.

c5b423a

Signed-off-by: David Calavera <[email protected]>

calavera marked this pull request as ready for review December 4, 2021 01:35

Make custom trait example more useful.

23b3631

Signed-off-by: David Calavera <[email protected]>

Fix formatting.

36cdbd8

Signed-off-by: David Calavera <[email protected]>

calavera added 2 commits December 6, 2021 15:33

Remove unused dependencies.

6110926

Signed-off-by: David Calavera <[email protected]>

Add README files for the new crates.

f607efa

Signed-off-by: David Calavera <[email protected]>

nmoutschen approved these changes Dec 7, 2021

View reviewed changes

lambda-extension/README.md Outdated Show resolved Hide resolved

lambda-runtime-api-client/README.md Show resolved Hide resolved

Update readme files.

bbde541

Signed-off-by: David Calavera <[email protected]>

nmoutschen reviewed Dec 10, 2021

View reviewed changes

nmoutschen mentioned this pull request Dec 10, 2021

Integration tests on Lambda #377

Closed

Fix extension name

981b7ce

Cleanup the path from the executable when it takes the name from arg[0]. Signed-off-by: David Calavera <[email protected]>

nmoutschen merged commit 5092de1 into awslabs:master Dec 11, 2021

calavera deleted the extensions branch December 12, 2021 03:37

nmoutschen added the 0.5 label Feb 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lambda Extensions #376

Lambda Extensions #376

calavera commented Nov 29, 2021 •

edited

Loading

nmoutschen commented Nov 29, 2021

nmoutschen left a comment

calavera commented Dec 2, 2021

nmoutschen left a comment •

edited

Loading

nmoutschen left a comment

calavera commented Dec 4, 2021

calavera commented Dec 4, 2021

calavera commented Dec 7, 2021

nmoutschen left a comment

calavera commented Dec 7, 2021 •

edited

Loading

calavera commented Dec 10, 2021

nmoutschen commented Dec 10, 2021

calavera commented Dec 10, 2021

nmoutschen Dec 10, 2021 •

edited

Loading

calavera Dec 11, 2021

nmoutschen commented Dec 10, 2021 •

edited

Loading

calavera commented Dec 11, 2021

nmoutschen commented Dec 11, 2021

Lambda Extensions #376

Lambda Extensions #376

Conversation

calavera commented Nov 29, 2021 • edited Loading

nmoutschen commented Nov 29, 2021

nmoutschen left a comment

Choose a reason for hiding this comment

calavera commented Dec 2, 2021

nmoutschen left a comment • edited Loading

Choose a reason for hiding this comment

nmoutschen left a comment

Choose a reason for hiding this comment

calavera commented Dec 4, 2021

calavera commented Dec 4, 2021

calavera commented Dec 7, 2021

nmoutschen left a comment

Choose a reason for hiding this comment

calavera commented Dec 7, 2021 • edited Loading

calavera commented Dec 10, 2021

nmoutschen commented Dec 10, 2021

calavera commented Dec 10, 2021

nmoutschen Dec 10, 2021 • edited Loading

Choose a reason for hiding this comment

calavera Dec 11, 2021

Choose a reason for hiding this comment

nmoutschen commented Dec 10, 2021 • edited Loading

calavera commented Dec 11, 2021

nmoutschen commented Dec 11, 2021

calavera commented Nov 29, 2021 •

edited

Loading

nmoutschen left a comment •

edited

Loading

calavera commented Dec 7, 2021 •

edited

Loading

nmoutschen Dec 10, 2021 •

edited

Loading

nmoutschen commented Dec 10, 2021 •

edited

Loading