-
Notifications
You must be signed in to change notification settings - Fork 361
adapt lambda-http to new lambda runtime api #217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
lambda-http/src/lib.rs
Outdated
fn run(&mut self, event: Request, ctx: Context) -> Result<R, HandlerError>; | ||
/// Functions serving as ALB and API Gateway REST and HTTP API handlers must conform to this type. | ||
/// | ||
/// This can be viewed as a `lambda::Handler` constained to `http` crate `Request` and `Response` types |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking for input on what constitutes agreeable here. The general idea is that I'd like to lambda_http::Handler
that mentally maps to lambda::Handler
but whose input and output's are http
crate type based. Since those types are not (de)serializable. We can't use lambda::Handler
directly. I also like the idea of have an interface that pins output types to input types, something that would come in handy for triggers with contracts that define a well defined input/output shapes.
I'd like to keep the same level of flexibility has lambda::Handler
in that you can define your own or "just use a closure".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking for input on what constitutes agreeable here.
I think what you proposed is a the most viable approach, but I need to give this a more thorough review. The only alternative I can think of that might work doing the duck-typed dispatch that the write!
macro does under the hood: https://doc.rust-lang.org/std/macro.write.html
I also like the idea of have an interface that pins output types to input types, something that would come in handy for triggers with contracts that define a well defined input/output shapes.
That's interesting, but I'm not really picturing how this might work. It's not urgent, but I'd love to hear more details.
lambda-http/src/lib.rs
Outdated
fn call(&mut self, event: Request) -> Self::Fut; | ||
|
||
/// Consumes this Handler into an Adapter type which implements `lambda::Hander` | ||
fn to_adapter(self) -> Adapter<Self> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was a bit annoying but I found I couldn't just implement lambda::Handler
for any lambda_http::Handler
directly. I needed to add a wrapping locally defined type. http://smallcultfollowing.com/babysteps/blog/2015/01/14/little-orphan-impls/ explains why. In particular the bits about "covered" types. The local Adapter
type is "covering" handler impls. Open to suggestions on alternatives
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the note. Yeah, this sits at a bad intersection of trait coherence, specialization, and what rustc would might allow. It's a pickle!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pushed a change that makes this slightly more transparent.
@@ -15,7 +15,7 @@ use crate::body::Body; | |||
/// Representation of API Gateway response | |||
#[derive(Serialize, Debug)] | |||
#[serde(rename_all = "camelCase")] | |||
pub(crate) struct LambdaResponse { | |||
pub struct LambdaResponse { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
because handler
exports a wrapper type, these now need to be pub
lambda-http/src/lib.rs
Outdated
fn run(&mut self, event: Request, ctx: Context) -> Result<R, HandlerError>; | ||
/// Functions serving as ALB and API Gateway REST and HTTP API handlers must conform to this type. | ||
/// | ||
/// This can be viewed as a `lambda::Handler` constained to `http` crate `Request` and `Response` types |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking for input on what constitutes agreeable here.
I think what you proposed is a the most viable approach, but I need to give this a more thorough review. The only alternative I can think of that might work doing the duck-typed dispatch that the write!
macro does under the hood: https://doc.rust-lang.org/std/macro.write.html
I also like the idea of have an interface that pins output types to input types, something that would come in handy for triggers with contracts that define a well defined input/output shapes.
That's interesting, but I'm not really picturing how this might work. It's not urgent, but I'd love to hear more details.
lambda-http/src/lib.rs
Outdated
fn call(&mut self, event: Request) -> Self::Fut; | ||
|
||
/// Consumes this Handler into an Adapter type which implements `lambda::Hander` | ||
fn to_adapter(self) -> Adapter<Self> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the note. Yeah, this sits at a bad intersection of trait coherence, specialization, and what rustc would might allow. It's a pickle!
@@ -1,6 +1,5 @@ | |||
//! ALB and API Gateway extension methods for `http::Request` types | |||
|
|||
use failure::Fail; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yay!
lambda-http/src/lib.rs
Outdated
//! Enriches the `lambda_runtime` crate with [http](https://github.com/hyperium/http) | ||
//! types targeting ALB and API Gateway proxy events. | ||
//! Enriches the `lambda` crate with [http](https://github.com/hyperium/http) | ||
//! types targeting ALB, API Gateway REST and HTTTP API proxy events. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
//! types targeting ALB, API Gateway REST and HTTTP API proxy events. | |
//! types targeting ALB, API Gateway REST and HTTP API proxy events. |
Note that I don't think your changes handle the schema of the new HTTP APIs yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
getting there...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops! My bad.
lambda-http/src/ext.rs
Outdated
#[derive(Debug, Fail)] | ||
#[derive(Debug)] | ||
pub enum PayloadError { | ||
/// Returned when `application/json` bodies fail to deserialize a payload | ||
#[fail(display = "failed to parse payload from application/json")] | ||
//#[fail(display = "failed to parse payload from application/json")] | ||
Json(serde_json::Error), | ||
/// Returned when `application/x-www-form-urlencoded` bodies fail to deserialize a payload | ||
#[fail(display = "failed to parse payload application/x-www-form-urlencoded")] | ||
//#[fail(display = "failed to parse payload application/x-www-form-urlencoded")] | ||
WwwFormUrlEncoded(SerdeError), | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few notes:
- Does this implement
std::error::Error
? If not, do you want to implement it by hand or do you want tothiserror
, whose removal would be a non-breaking change? - Mark this enum as
#[non_exhaustive]
? - Add a
PartialEq
implementation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently re-evaluating what I actually need. Left this commented out as a highlighted reminder for a todo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gotcha, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went impl'ing std::error::Error
since at this point that only really means impl fmt::Display
.
At this point this enum is exhaustive. It's an error that represents not being about to deserialize a type given a application/json
or application/x-www-form-urlencoded
request body
Looks like I don't get an easy derive(PartialEq)
since serde_json::error::Error
doesn't impl it. Let me know if you really want that.
Let me know if you want to see other changes here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went impl'ing
std::error::Error
since at this point that only really means implfmt::Display
.
Sounds good!
At this point this enum is exhaustive. It's an error that represents not being about to deserialize a type given a application/json or application/x-www-form-urlencoded request body
My concern is that a non-#[non_exhaustive]
enum would become a forwards compatibility hazard if API Gateway (or some other networking product) added support for non-HTTP 1.1 protocols like HTTP/2 or HTTP/3. By saying this error is exhaustive, we'd need to cut a breaking change to support a new protocol. It's not the end of the world, but if it means that customers are slightly better prepared for breaking changes, I'll weakly press for #[non_exhaustive]
.
Looks like I don't get an easy derive(PartialEq) since serde_json::error::Error doesn't impl it. Let me know if you really want that.
Oh, bummer. Yeah, not worth it then.
lambda-http/Cargo.toml
Outdated
serde_urlencoded = "0.5" | ||
serde_json = "^1" | ||
serde_urlencoded = "0.6" | ||
tokio = "^0.2" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't though through this yet, but—what parts of Tokio need to be pulled into lambda_http
as a project dependency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This list just got resorted. I don't think there are any actually. I'll double check and remove if so. I only needed a dev-dependency for #[tokio::main]
bits but I don't think we're actually referencing tokio types directly yet which shines a light on some nice design choices for the new api. Thanks for that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I only needed a dev-dependency for #[tokio::main] bits but I don't think we're actually referencing tokio types directly yet which shines a light on some nice design choices for the new api. Thanks for that.
Glad you like it :)
Thanks so much for doing this. I posted a review/questions kicking the tires on your tire kicking :) |
Thanks. I'll keep this in draft until I'm ready so expect more change. I'm just using the draft to communicate progress being made. It's still baking in bread in the oven. |
Sounds good, I'll hold off on making additional comments until this is moved out of a draft state. |
#[serde(untagged)] | ||
pub enum RequestContext { | ||
/// Api Gateway request context | ||
pub enum LambdaRequest<'a> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changing the strategy here. previously there was one struct made to look look like both apigw rest api proxy and alb events. With the addition of http api events that approach was starting to look long in the tooth and untenable. Going back to the drawing board on this answering "what is this even?" question. I came up with roughly
type LambdaRequest = ApiGatewayRequest | ApiGatewayV2Request | AlbRequest;
Unionizing types rather than masquerading types had some tradeoffs and some useful benefits. Updating/evolving one variant doesn't require additional consideration in how another gets deserialized. Transforming 3 similar variants incurs some boiler plate but I could add some helpers to make that less onerous. In retrospect this feels like a better approach given the tradeoffs.
Ok @davidbarsky. I'm pretty happy with this now. I also tested a few deployments to check for regressions. Let me know what you think. |
lambda-http-attributes/Cargo.toml
Outdated
@@ -0,0 +1,13 @@ | |||
[package] | |||
name = "lambda-http-attributes" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think this functionality could be a part of lambda-attributes
? In tracing
, the #[instrument]
macro can instrument std::future::Future/async functions if another crate (tracing_futures
) is in the dependency graph. See here: https://docs.rs/tracing/0.1.13/tracing/#spans-1.
I'd like it if we can do something similar here to reduce how many crates customers might need and how many crates we need to release.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The way this works now is that the attribute crates are transient dependencies users don't have to manually manage these.
I have some GitHub workflows for publishing crates easy as git pushing a tag for automating releases I could share with this project.
I was trying to find a way to do this with lambda attributes but couldn't find a way do to it without adding a unnatural and cyclical dependency on lambda-http in the lambda crate.
One idea I haven't excersized is compilation flags set by the lambda and lambda-http crates respectively. Thoughts on that approach?
I really like the procmacro-main pattern but it's notably a bit boilerplatey to set up which makes it seems like the yet-another-crate-to-publish feelings are strong. I can sympathize with that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I took a look at what the tracing
crate does. It seems it uses "asyncness' as a cue to generate code using tracing_futures It's not actually aware if tracing_futures is on the dependency graph or not. There's a little less luxury here.
I'll see if I can use the type as the first argument to main as a cue to generate lambda http references then fallback on the cargo feature flag if not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I took a look at what the tracing crate does. It seems it uses "asyncness' as a cue to generate code using tracing_futures It's not actually aware if tracing_futures is on the dependency graph or not. There's a little less luxury here.
Ah yeah, I forgot how we (I?) wrote that.
I'll see if I can use the type as the first argument to main as a cue to generate lambda http references then fallback on the cargo feature flag if not.
Trying to figure out a type in a macro expansion isn't easy, least of all because I ran tried to resolve poor interactions with the async_trait
and tracing's #[instrument]
. You'd need something like specialization to properly distinguish between values in a macro, and short of that, autoref specialization: https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.md. I'm not sure it's worth tackling now, but if you come up with solution, please, I'm all ears!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I realized that procmaco doesn't really know about types, only syntax so distinguishing a type seems less feasible. I'll try a cargo feature flag something like a feature for raw handlers and http handlers.
[features]
raw = []
http = []
defaults = [raw]
[dependencies]
lambda-attributes = { path = "../lambda-attributes", version = "0.1", default_features = false, features = [http] }
then do #[cfg_attr(feature = "raw")]
vs #[cfg_attr(feature = "http")]
checks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hit a small snag here I wanted to share early feedback on. I added the features mentioned above to lambda-attributes then in its impl
if cfg!(feature = "http") {
// handler code for http events
} else {
// handler code for raw events
}
I'm able to successfully run cargo test -p lambda
and cargo test -p lambda_http
separately but when running cargo test
the block for lambda_http gets emitted from the proc-macro presumably because cargo pins one version of the lambda-attribute dependency with one set of features for all crates in your workspace. This seems like a bug in cargo or it could be a feature. Either way its a bit problematic for the cargo feature flag approach.
error[E0433]: failed to resolve: use of undeclared type or module `lambda_http`
--> lambda/examples/hello.rs:6:1
|
6 | #[lambda]
| ^^^^^^^^^ use of undeclared type or module `lambda_http`
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this behavior relates to this (open) cargo issue rust-lang/cargo#4463. I expect workspaces projects for rust lambdas be common with a mix of http handlers and some coordinating (sqs, s3, ect) handlers in the same project which makes me weary of the cargo feature flag approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I found a way out that I actually like better using item attr args that could set the stage for other strictly typed handlers
#[lambda]
does what it does today
#[lambda(http)]
emits http handler code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments; by no means exhaustive. I'll add some more other stuff.
lambda-http/Cargo.toml
Outdated
@@ -3,7 +3,7 @@ name = "lambda_http" | |||
version = "0.1.1" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bump to 0.2.0-beta.1
?
Ok(()) | ||
} | ||
|
||
fn my_handler(e: Request, c: Context) -> Result<impl IntoResponse, HandlerError> { | ||
Ok(match e.query_string_parameters().get("first_name") { | ||
async fn func(event: Request) -> Result<impl IntoResponse, Error> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question—why an impl IntoResponse
rather than http::Response`?
(More for customers than for me.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rust convenience and extensibility pattern.
For some common cases you'll likely just want to yield a string or serde json value. IntoResponse is a convince here, similar the new http apis interface for implicit json responses. In other cases where you want a more application specific way of structuring responses for consistency, you can just return your own type and impl into response to avoid the conversion inside the hander. Side effect, it makes that a bit wiring more unit testable.
In short make the simple cases easy and complex cases possible.
Concretely in this case there are two match arms one for the happy path which returns a string and one for the user error case which builds a response with an error status. The common interface for the result type becomes some type impl IntoResponse.
lambda-http/src/ext.rs
Outdated
} | ||
} | ||
} | ||
impl Error for PayloadError {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implement Error::source
?
lambda-http/src/lib.rs
Outdated
//! For the simple cases you may not need much if any bootstrapping. To make life simpler | ||
//! you can add an `#[lambda_http]` attribute to your `main` function and `lambda::run` machinery | ||
//! will be wired in for you. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
//! For the simple cases you may not need much if any bootstrapping. To make life simpler | |
//! you can add an `#[lambda_http]` attribute to your `main` function and `lambda::run` machinery | |
//! will be wired in for you. | |
//! Adding an `#[lambda_http]` attribute to a `#[tokio::run]`-decorated `main` function will setup and run the Lambda function. |
lambda-http/src/request.rs
Outdated
#[derive(Deserialize, Debug, Clone)] | ||
#[serde(untagged)] | ||
pub enum RequestContext { | ||
/// API Gateway v2 request context | ||
ApiGatewayV2(ApiGatewayV2RequestContext), | ||
/// API Gateway request context | ||
ApiGateway(ApiGatewayRequestContext), | ||
/// ALB request context | ||
Alb(AlbRequestContext), | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mark this as #[non_exhaustive]
.
/// Event request context as an enumeration of request contexts | ||
/// for both ALB and API Gateway and HTTP API events |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// Event request context as an enumeration of request contexts | |
/// for both ALB and API Gateway and HTTP API events | |
/// Support invocations from Application Load Balancers and API Gateway. Supported API Gateway invocations | |
/// include both [REST](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-rest-api.html) and [HTTP](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api.html) APIs. |
@@ -0,0 +1,39 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For later: let's create a JSON schema based off this file (https://app.quicktype.io/ is great) and fuzz/quickcheck the handler based off the schema?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting you mentioned that. I was actually think about starting a fuzzer project for lambdas, a cli, similar to Sam's templated events but random and standalone. But dully noted for unit testing as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(note: this isn't a blocker for this PR; tracking this in an issue is fine.)
One of the reasons the simulated.rs
module exists is to support this use-case without something like SAM. I haven't spoken to the SAM team about this, however.
I think Great work on the PR. It's exactly what I was looking for! |
Nice catch. I deployed a lambda with the v2 payload enabled and grabbed an example of an "echo" of its event payload then added a unit test to make sure the minimal case is covered in 94b1824 |
Co-authored-by: David Barsky <[email protected]>
Co-authored-by: David Barsky <[email protected]>
Co-authored-by: David Barsky <[email protected]>
Co-authored-by: David Barsky <[email protected]>
Co-authored-by: David Barsky <[email protected]>
Co-authored-by: David Barsky <[email protected]>
rebased on master and found some changed to the Handler::Err which is now Handler::Error. Working through accommodating those |
94b1824
to
95e7ccd
Compare
@davidbarsky I think I addressed the comments above. Let me know if there are any more blockers. a few side question/comment which should maybe be a separate gh issue. I've love to see the versions of theses crates keep a shared version making it easier for users to manage. It's subjective but the cost of publishing all crates when on changes is less than that of a user's mental overhead of keeping track of the combinations of versions of crates all published from the same project. What are you're thoughts on making all versions the same in this project. |
Is there any objective reason the minimum supported version in ci can't be |
No, nothing that should block merging of this PR, I think. We can capture it in followups.
Yeah, I think I'd be in favor of publishing everything/re-exporting everything through the Lambda crate and enabling subcrates through feature flags. I'd diverge from Tokio and have everything enabled by default, though. I think a blocker to this would be to collapse the two macro crates into a single one and try to have a generic
I didn't know that was a thing! Let's make our MSRV the current stable Rust release until we hit 1.0. Our versioning policy should allow for bug fix releases to rely on a recent Rust release, but we should document this in a prominent location. |
I'll merge this in and we catch any fallout of issues later on. |
Issue #, if available:
Description of changes:
Kicking the tires on reviving lambda-http crate. This is not yet finalized but I wanted to communicate progress is being made. Basic things work with the new much more simplified lambda runtime api. Still trying to figure out what good looks like.
By submitting this pull request