Skip to content

adapt lambda-http to new lambda runtime api #217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
May 8, 2020

Conversation

softprops
Copy link
Contributor

@softprops softprops commented Apr 21, 2020

Issue #, if available:

Description of changes:

Kicking the tires on reviving lambda-http crate. This is not yet finalized but I wanted to communicate progress is being made. Basic things work with the new much more simplified lambda runtime api. Still trying to figure out what good looks like.

  • make compile
  • tests pass
  • update examples
  • update dependencies
  • adapt to new apigw http api events
  • define a proc macro for http mains

By submitting this pull request

  • I confirm that my contribution is made under the terms of the Apache 2.0 license.
  • I confirm that I've made a best effort attempt to update all relevant documentation.

fn run(&mut self, event: Request, ctx: Context) -> Result<R, HandlerError>;
/// Functions serving as ALB and API Gateway REST and HTTP API handlers must conform to this type.
///
/// This can be viewed as a `lambda::Handler` constained to `http` crate `Request` and `Response` types
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking for input on what constitutes agreeable here. The general idea is that I'd like to lambda_http::Handler that mentally maps to lambda::Handler but whose input and output's are http crate type based. Since those types are not (de)serializable. We can't use lambda::Handler directly. I also like the idea of have an interface that pins output types to input types, something that would come in handy for triggers with contracts that define a well defined input/output shapes.

I'd like to keep the same level of flexibility has lambda::Handler in that you can define your own or "just use a closure".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking for input on what constitutes agreeable here.

I think what you proposed is a the most viable approach, but I need to give this a more thorough review. The only alternative I can think of that might work doing the duck-typed dispatch that the write! macro does under the hood: https://doc.rust-lang.org/std/macro.write.html

I also like the idea of have an interface that pins output types to input types, something that would come in handy for triggers with contracts that define a well defined input/output shapes.

That's interesting, but I'm not really picturing how this might work. It's not urgent, but I'd love to hear more details.

fn call(&mut self, event: Request) -> Self::Fut;

/// Consumes this Handler into an Adapter type which implements `lambda::Hander`
fn to_adapter(self) -> Adapter<Self> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a bit annoying but I found I couldn't just implement lambda::Handler for any lambda_http::Handler directly. I needed to add a wrapping locally defined type. http://smallcultfollowing.com/babysteps/blog/2015/01/14/little-orphan-impls/ explains why. In particular the bits about "covered" types. The local Adapter type is "covering" handler impls. Open to suggestions on alternatives

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note. Yeah, this sits at a bad intersection of trait coherence, specialization, and what rustc would might allow. It's a pickle!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a change that makes this slightly more transparent.

@@ -15,7 +15,7 @@ use crate::body::Body;
/// Representation of API Gateway response
#[derive(Serialize, Debug)]
#[serde(rename_all = "camelCase")]
pub(crate) struct LambdaResponse {
pub struct LambdaResponse {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because handler exports a wrapper type, these now need to be pub

fn run(&mut self, event: Request, ctx: Context) -> Result<R, HandlerError>;
/// Functions serving as ALB and API Gateway REST and HTTP API handlers must conform to this type.
///
/// This can be viewed as a `lambda::Handler` constained to `http` crate `Request` and `Response` types
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking for input on what constitutes agreeable here.

I think what you proposed is a the most viable approach, but I need to give this a more thorough review. The only alternative I can think of that might work doing the duck-typed dispatch that the write! macro does under the hood: https://doc.rust-lang.org/std/macro.write.html

I also like the idea of have an interface that pins output types to input types, something that would come in handy for triggers with contracts that define a well defined input/output shapes.

That's interesting, but I'm not really picturing how this might work. It's not urgent, but I'd love to hear more details.

fn call(&mut self, event: Request) -> Self::Fut;

/// Consumes this Handler into an Adapter type which implements `lambda::Hander`
fn to_adapter(self) -> Adapter<Self> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note. Yeah, this sits at a bad intersection of trait coherence, specialization, and what rustc would might allow. It's a pickle!

@@ -1,6 +1,5 @@
//! ALB and API Gateway extension methods for `http::Request` types

use failure::Fail;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!

//! Enriches the `lambda_runtime` crate with [http](https://github.com/hyperium/http)
//! types targeting ALB and API Gateway proxy events.
//! Enriches the `lambda` crate with [http](https://github.com/hyperium/http)
//! types targeting ALB, API Gateway REST and HTTTP API proxy events.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//! types targeting ALB, API Gateway REST and HTTTP API proxy events.
//! types targeting ALB, API Gateway REST and HTTP API proxy events.

Note that I don't think your changes handle the schema of the new HTTP APIs yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getting there...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! My bad.

Comment on lines 26 to 31
#[derive(Debug, Fail)]
#[derive(Debug)]
pub enum PayloadError {
/// Returned when `application/json` bodies fail to deserialize a payload
#[fail(display = "failed to parse payload from application/json")]
//#[fail(display = "failed to parse payload from application/json")]
Json(serde_json::Error),
/// Returned when `application/x-www-form-urlencoded` bodies fail to deserialize a payload
#[fail(display = "failed to parse payload application/x-www-form-urlencoded")]
//#[fail(display = "failed to parse payload application/x-www-form-urlencoded")]
WwwFormUrlEncoded(SerdeError),
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few notes:

  • Does this implement std::error::Error? If not, do you want to implement it by hand or do you want to thiserror, whose removal would be a non-breaking change?
  • Mark this enum as #[non_exhaustive]?
  • Add a PartialEq implementation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently re-evaluating what I actually need. Left this commented out as a highlighted reminder for a todo.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went impl'ing std::error::Error since at this point that only really means impl fmt::Display .

At this point this enum is exhaustive. It's an error that represents not being about to deserialize a type given a application/json or application/x-www-form-urlencoded request body

Looks like I don't get an easy derive(PartialEq) since serde_json::error::Error doesn't impl it. Let me know if you really want that.

Let me know if you want to see other changes here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went impl'ing std::error::Error since at this point that only really means impl fmt::Display.

Sounds good!

At this point this enum is exhaustive. It's an error that represents not being about to deserialize a type given a application/json or application/x-www-form-urlencoded request body

My concern is that a non-#[non_exhaustive] enum would become a forwards compatibility hazard if API Gateway (or some other networking product) added support for non-HTTP 1.1 protocols like HTTP/2 or HTTP/3. By saying this error is exhaustive, we'd need to cut a breaking change to support a new protocol. It's not the end of the world, but if it means that customers are slightly better prepared for breaking changes, I'll weakly press for #[non_exhaustive].

Looks like I don't get an easy derive(PartialEq) since serde_json::error::Error doesn't impl it. Let me know if you really want that.

Oh, bummer. Yeah, not worth it then.

serde_urlencoded = "0.5"
serde_json = "^1"
serde_urlencoded = "0.6"
tokio = "^0.2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't though through this yet, but—what parts of Tokio need to be pulled into lambda_http as a project dependency?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list just got resorted. I don't think there are any actually. I'll double check and remove if so. I only needed a dev-dependency for #[tokio::main] bits but I don't think we're actually referencing tokio types directly yet which shines a light on some nice design choices for the new api. Thanks for that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only needed a dev-dependency for #[tokio::main] bits but I don't think we're actually referencing tokio types directly yet which shines a light on some nice design choices for the new api. Thanks for that.

Glad you like it :)

@davidbarsky
Copy link
Contributor

Thanks so much for doing this. I posted a review/questions kicking the tires on your tire kicking :)

@softprops
Copy link
Contributor Author

I posted a review/questions kicking the tires on your tire kicking

Thanks. I'll keep this in draft until I'm ready so expect more change. I'm just using the draft to communicate progress being made. It's still baking in bread in the oven.

@davidbarsky
Copy link
Contributor

Sounds good, I'll hold off on making additional comments until this is moved out of a draft state.

#[serde(untagged)]
pub enum RequestContext {
/// Api Gateway request context
pub enum LambdaRequest<'a> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing the strategy here. previously there was one struct made to look look like both apigw rest api proxy and alb events. With the addition of http api events that approach was starting to look long in the tooth and untenable. Going back to the drawing board on this answering "what is this even?" question. I came up with roughly

type LambdaRequest = ApiGatewayRequest | ApiGatewayV2Request | AlbRequest;

Unionizing types rather than masquerading types had some tradeoffs and some useful benefits. Updating/evolving one variant doesn't require additional consideration in how another gets deserialized. Transforming 3 similar variants incurs some boiler plate but I could add some helpers to make that less onerous. In retrospect this feels like a better approach given the tradeoffs.

@softprops softprops marked this pull request as ready for review April 25, 2020 02:15
@softprops
Copy link
Contributor Author

Ok @davidbarsky. I'm pretty happy with this now. I also tested a few deployments to check for regressions. Let me know what you think.

@@ -0,0 +1,13 @@
[package]
name = "lambda-http-attributes"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think this functionality could be a part of lambda-attributes? In tracing, the #[instrument] macro can instrument std::future::Future/async functions if another crate (tracing_futures) is in the dependency graph. See here: https://docs.rs/tracing/0.1.13/tracing/#spans-1.

I'd like it if we can do something similar here to reduce how many crates customers might need and how many crates we need to release.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way this works now is that the attribute crates are transient dependencies users don't have to manually manage these.

I have some GitHub workflows for publishing crates easy as git pushing a tag for automating releases I could share with this project.

I was trying to find a way to do this with lambda attributes but couldn't find a way do to it without adding a unnatural and cyclical dependency on lambda-http in the lambda crate.

One idea I haven't excersized is compilation flags set by the lambda and lambda-http crates respectively. Thoughts on that approach?

I really like the procmacro-main pattern but it's notably a bit boilerplatey to set up which makes it seems like the yet-another-crate-to-publish feelings are strong. I can sympathize with that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a look at what the tracing crate does. It seems it uses "asyncness' as a cue to generate code using tracing_futures It's not actually aware if tracing_futures is on the dependency graph or not. There's a little less luxury here.

I'll see if I can use the type as the first argument to main as a cue to generate lambda http references then fallback on the cargo feature flag if not.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a look at what the tracing crate does. It seems it uses "asyncness' as a cue to generate code using tracing_futures It's not actually aware if tracing_futures is on the dependency graph or not. There's a little less luxury here.

Ah yeah, I forgot how we (I?) wrote that.

I'll see if I can use the type as the first argument to main as a cue to generate lambda http references then fallback on the cargo feature flag if not.

Trying to figure out a type in a macro expansion isn't easy, least of all because I ran tried to resolve poor interactions with the async_trait and tracing's #[instrument]. You'd need something like specialization to properly distinguish between values in a macro, and short of that, autoref specialization: https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.md. I'm not sure it's worth tackling now, but if you come up with solution, please, I'm all ears!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized that procmaco doesn't really know about types, only syntax so distinguishing a type seems less feasible. I'll try a cargo feature flag something like a feature for raw handlers and http handlers.

[features]
raw = []
http = []
defaults = [raw]
[dependencies]
lambda-attributes = { path = "../lambda-attributes", version = "0.1", default_features = false, features = [http] }

then do #[cfg_attr(feature = "raw")] vs #[cfg_attr(feature = "http")] checks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hit a small snag here I wanted to share early feedback on. I added the features mentioned above to lambda-attributes then in its impl

if cfg!(feature = "http") {
 // handler code for http events
} else {
 // handler code for raw events
}

I'm able to successfully run cargo test -p lambda and cargo test -p lambda_http separately but when running cargo test the block for lambda_http gets emitted from the proc-macro presumably because cargo pins one version of the lambda-attribute dependency with one set of features for all crates in your workspace. This seems like a bug in cargo or it could be a feature. Either way its a bit problematic for the cargo feature flag approach.

error[E0433]: failed to resolve: use of undeclared type or module `lambda_http`
 --> lambda/examples/hello.rs:6:1
  |
6 | #[lambda]
  | ^^^^^^^^^ use of undeclared type or module `lambda_http`

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this behavior relates to this (open) cargo issue rust-lang/cargo#4463. I expect workspaces projects for rust lambdas be common with a mix of http handlers and some coordinating (sqs, s3, ect) handlers in the same project which makes me weary of the cargo feature flag approach.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a way out that I actually like better using item attr args that could set the stage for other strictly typed handlers

#[lambda] does what it does today

#[lambda(http)] emits http handler code

Copy link
Contributor

@davidbarsky davidbarsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments; by no means exhaustive. I'll add some more other stuff.

@@ -3,7 +3,7 @@ name = "lambda_http"
version = "0.1.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bump to 0.2.0-beta.1?

Ok(())
}

fn my_handler(e: Request, c: Context) -> Result<impl IntoResponse, HandlerError> {
Ok(match e.query_string_parameters().get("first_name") {
async fn func(event: Request) -> Result<impl IntoResponse, Error> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question—why an impl IntoResponse rather than http::Response`?

(More for customers than for me.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rust convenience and extensibility pattern.

For some common cases you'll likely just want to yield a string or serde json value. IntoResponse is a convince here, similar the new http apis interface for implicit json responses. In other cases where you want a more application specific way of structuring responses for consistency, you can just return your own type and impl into response to avoid the conversion inside the hander. Side effect, it makes that a bit wiring more unit testable.

In short make the simple cases easy and complex cases possible.

Concretely in this case there are two match arms one for the happy path which returns a string and one for the user error case which builds a response with an error status. The common interface for the result type becomes some type impl IntoResponse.

}
}
}
impl Error for PayloadError {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implement Error::source?

Comment on lines 32 to 34
//! For the simple cases you may not need much if any bootstrapping. To make life simpler
//! you can add an `#[lambda_http]` attribute to your `main` function and `lambda::run` machinery
//! will be wired in for you.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//! For the simple cases you may not need much if any bootstrapping. To make life simpler
//! you can add an `#[lambda_http]` attribute to your `main` function and `lambda::run` machinery
//! will be wired in for you.
//! Adding an `#[lambda_http]` attribute to a `#[tokio::run]`-decorated `main` function will setup and run the Lambda function.

Comment on lines 153 to 157
#[derive(Deserialize, Debug, Clone)]
#[serde(untagged)]
pub enum RequestContext {
/// API Gateway v2 request context
ApiGatewayV2(ApiGatewayV2RequestContext),
/// API Gateway request context
ApiGateway(ApiGatewayRequestContext),
/// ALB request context
Alb(AlbRequestContext),
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mark this as #[non_exhaustive].

Comment on lines +151 to +147
/// Event request context as an enumeration of request contexts
/// for both ALB and API Gateway and HTTP API events
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Event request context as an enumeration of request contexts
/// for both ALB and API Gateway and HTTP API events
/// Support invocations from Application Load Balancers and API Gateway. Supported API Gateway invocations
/// include both [REST](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-rest-api.html) and [HTTP](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api.html) APIs.

@@ -0,0 +1,39 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For later: let's create a JSON schema based off this file (https://app.quicktype.io/ is great) and fuzz/quickcheck the handler based off the schema?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting you mentioned that. I was actually think about starting a fuzzer project for lambdas, a cli, similar to Sam's templated events but random and standalone. But dully noted for unit testing as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(note: this isn't a blocker for this PR; tracking this in an issue is fine.)

One of the reasons the simulated.rs module exists is to support this use-case without something like SAM. I haven't spoken to the SAM team about this, however.

@dbasedow
Copy link

dbasedow commented May 5, 2020

I think cookies should be an Option<...> in LambdaRequest::ApiGatewayV2 in my tests the lambda event did not contain that key and i got a "data did not match any variant of untagged enum LambdaRequest" error.

Great work on the PR. It's exactly what I was looking for!

@softprops
Copy link
Contributor Author

I think cookies should be an Option<...> in LambdaRequest::ApiGatewayV2 in my tests the lambda event did not contain that key and i got a "data did not match any variant of untagged enum LambdaRequest" error

Nice catch. I deployed a lambda with the v2 payload enabled and grabbed an example of an "echo" of its event payload then added a unit test to make sure the minimal case is covered in 94b1824

@softprops
Copy link
Contributor Author

rebased on master and found some changed to the Handler::Err which is now Handler::Error. Working through accommodating those

@softprops softprops force-pushed the revive-lambda-http branch from 94b1824 to 95e7ccd Compare May 6, 2020 23:37
@softprops
Copy link
Contributor Author

@davidbarsky I think I addressed the comments above. Let me know if there are any more blockers.

a few side question/comment which should maybe be a separate gh issue.

I've love to see the versions of theses crates keep a shared version making it easier for users to manage. It's subjective but the cost of publishing all crates when on changes is less than that of a user's mental overhead of keeping track of the combinations of versions of crates all published from the same project.

What are you're thoughts on making all versions the same in this project.

@softprops
Copy link
Contributor Author

Is there any objective reason the minimum supported version in ci can't be stable? I noticed a ci failure when I introduces a matches! call introduced in 1.42

@davidbarsky
Copy link
Contributor

@davidbarsky I think I addressed the comments above. Let me know if there are any more blockers.

No, nothing that should block merging of this PR, I think. We can capture it in followups.

I've love to see the versions of theses crates keep a shared version making it easier for users to manage. It's subjective but the cost of publishing all crates when on changes is less than that of a user's mental overhead of keeping track of the combinations of versions of crates all published from the same project.
What are your thoughts on making all versions the same in this project.

Yeah, I think I'd be in favor of publishing everything/re-exporting everything through the Lambda crate and enabling subcrates through feature flags. I'd diverge from Tokio and have everything enabled by default, though. I think a blocker to this would be to collapse the two macro crates into a single one and try to have a generic Into bound on the handler definition, if that makes sense.

Is there any objective reason the minimum supported version in ci can't be stable? I noticed a ci failure when I introduces a matches! call introduced in 1.42?

I didn't know that was a thing! Let's make our MSRV the current stable Rust release until we hit 1.0. Our versioning policy should allow for bug fix releases to rely on a recent Rust release, but we should document this in a prominent location.

@davidbarsky
Copy link
Contributor

I'll merge this in and we catch any fallout of issues later on.

@davidbarsky davidbarsky merged commit 4463351 into awslabs:master May 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants