-
Notifications
You must be signed in to change notification settings - Fork 403
Require LengthLimitedRead
for structs that always drain reader
#3640
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Require LengthLimitedRead
for structs that always drain reader
#3640
Conversation
👋 Thanks for assigning @arik-so as a reviewer! |
6f66ebb
to
47916c1
Compare
Looking for concept ACKs before fixing fuzz! |
Also do we also want to do the same for |
47916c1
to
792fcf0
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3640 +/- ##
==========================================
+ Coverage 89.24% 90.37% +1.12%
==========================================
Files 155 155
Lines 119269 127304 +8035
Branches 119269 127304 +8035
==========================================
+ Hits 106442 115050 +8608
+ Misses 10231 9722 -509
+ Partials 2596 2532 -64 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this makes a ton of sense, and not seeing any issues with any of the commits (though the whitespace one is also in 3620)
👋 The first review has been submitted! Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer. |
nit: might point out in the last commit message that the remaining structs are the ones not covered by impl_writeable_msg |
36698bb
to
b95f2ce
Compare
nit: squash the formatting commit in between getting the second ACK and merging. |
🔔 1st Reminder Hey @TheBlueMatt! This PR has been waiting for your review. |
lightning/src/util/ser.rs
Outdated
/// Reads a `Self` in from the given [`LengthRead`]. | ||
fn read_from_fixed_length_buffer<R: LengthRead>(reader: &mut R) -> Result<Self, DecodeError>; | ||
/// Reads a `Self` in from the given [`FixedLengthReader`]. | ||
fn read_from_fixed_length_buffer<'a, R: Read>( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should either use a trait here, rather than forcing a FixedLengthReader
, or we should remove LengthRead
entirely (which looks ~unused now?). It'd be kinda nice to use a trait, I think, but maybe you thought it wouldn't be strict enough? I guess we could kinda just rename LengthRead
as LengthLimiedRead
and describe the semantics?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haha I thought you said several times offline that LengthRead
as-is wasn't sufficient so I went with the stricter option... I'll move forward with just renaming the current trait and adding docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Errr, I meant LengthReadable
isn't sufficient. I don't think we have to fix it by forcing a LengthLimitingReader
, but doing so is fine, I just wanted to highlight that we are leaving LengthRead
unused here, and should remove it, but also that we could use that with suffucient method/trait renames.
fuzz/src/chanmon_consistency.rs
Outdated
@@ -1101,7 +1101,9 @@ pub fn do_test<Out: Output>(data: &[u8], underlying_out: Out, anchors: bool) { | |||
// update_fail_htlc as we do when we reject a payment. | |||
let mut msg_ser = update_add.encode(); | |||
msg_ser[1000] ^= 0xff; | |||
let new_msg = UpdateAddHTLC::read(&mut Cursor::new(&msg_ser)).unwrap(); | |||
let mut cursor = Cursor::new(&msg_ser); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: we don't actually need a Cursor
, we can just &mut &msg_ser[..]
. Same elsewhere in this commit. Would be nice to slowly phase out Cursor
as we touch it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I started making this change, had to implement LengthLimitedRead
for &[u8]
. It ended up being weird because the Read
implementation for a slice advances the slice to point to the yet-unread part, which means that LengthLimitedRead::total_bytes
will be inaccurate after any part of the struct is read...
I'm not sure this is the way to go, when reading a TrampolinePacket
for example it "successfully" read but due to this subtlety it didn't read the whole packet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I almost prefer it that way? Like, having a "how many bytes are in the buffer" call is weird cause the caller may have already read many bytes. "How many bytes are left in the buffer" is a way more sensible question to ask, IMO.
@@ -1033,8 +1034,10 @@ impl OfferContents { | |||
} | |||
} | |||
|
|||
impl Readable for Offer { | |||
fn read<R: io::Read>(reader: &mut R) -> Result<Self, DecodeError> { | |||
impl LengthReadable for Offer { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we need to do this for all the other bolt 12 structs too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did this and a few other places in ser.rs
.
FixedLengthReadable
for structs that always drain readerLengthLimitedRead
for structs that always drain reader
b95f2ce
to
db17178
Compare
Will fix the build and a few more instances where |
3ddcf9c
to
c6ee09b
Compare
2f4c0de
to
6d13257
Compare
See prior two commits. When deserializing objects via this macro, there is no length prefix so the deser code will read the provided reader until it runs out of bytes. Readable is not an appropriate trait for this situation because it should only be used for structs that are prefixed with a length and know when to stop reading. LengthReadable instead requires that the caller supply only the bytes that are reserved for this struct.
Easier to review the previous commit this way.
Continuing the work of the last few commits, here we require that some LN gossip messages be read from a length limiting reader. The messages' updates are separated into their own commit because they definitely should use a length limited reader but they also require breaking out of the ser macros. We can't use the macros because each message contains a non-TLV field that will always consume the rest of the reader, while the ser macros only support reading non-TLV fields that know when to stop reading.
When deserializing lightning messages, many consume the reader until it is out of bytes, or could in the future if new fields or TLVs are added. Therefore, it's safer if they are only provided with readers that limit how much of the byte stream will be read. Many of the relevant LN messages were updated in a prior commit when we transitioned impl_writeable_msg to require length limiting readers, here we finish the job.
When reading a raw LSP message specifically, it will consume the provided reader until it is out of bytes. Readable is not an appropriate trait for this situation, and over the past few commits we've been transitioning any struct that always reads-to-end to require a length limited reader. This is less error-prone because there is no risk of a bug being introduced because a custom message read more data than it was supposed to from the provided reader.
Continuing the work over the past few commits, we want to transition any structs that always consume the entire provided reader when being deserialized to require a length-limiting reader. To do this we actually require all the offers' underlying ser wrappers to be read from a length-limiting reader as well, since these wrappers will always read-to-end in general. Some of the ser macros needed updating for this, which is fine because all TLV values were already read from a length-limiting reader.
6d13257
to
94f8952
Compare
Got rid of an unused lifetime (surprised there's no warning for that): diff --git a/lightning/src/ln/wire.rs b/lightning/src/ln/wire.rs
index 20e1ca789..1bb7c7448 100644
--- a/lightning/src/ln/wire.rs
+++ b/lightning/src/ln/wire.rs
@@ -238,7 +238,7 @@ impl<T: core::fmt::Debug + Type + TestEq> Message<T> {
/// # Errors
///
/// Returns an error if the message payload could not be decoded as the specified type.
-pub(crate) fn read<'a, R: LengthLimitedRead, T, H: core::ops::Deref>(
+pub(crate) fn read<R: LengthLimitedRead, T, H: core::ops::Deref>(
buffer: &mut R, custom_reader: H,
) -> Result<Message<T>, (msgs::DecodeError, Option<u16>)>
where
@@ -249,7 +249,7 @@ where
do_read(buffer, message_type, custom_reader).map_err(|e| (e, Some(message_type)))
}
-fn do_read<'a, R: LengthLimitedRead, T, H: core::ops::Deref>(
+fn do_read<R: LengthLimitedRead, T, H: core::ops::Deref>(
buffer: &mut R, message_type: u16, custom_reader: H,
) -> Result<Message<T>, msgs::DecodeError>
where |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should this commit remain or be squashed out pre-merge?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assume you're referring to the rustfmt
one? No reason to squash, IMO. Anything that helps a reviewer now helps a reviewer trying to understand the code in three years and it doesn't violate our "each commit should, on its own, build and pass tests" rule.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks great! Is it feasible to have a version of impl_writeable that uses LengthReadable so some of those structs can be reverted to using a macro call for the serialization implementation?
I felt weird using |
fair enough! |
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This is a temporary workaround to avoid introducing a big refactoring right now. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This is a temporary workaround to avoid introducing a big refactoring right now. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This is a temporary workaround to avoid introducing a big refactoring right now. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This is a temporary workaround to avoid introducing a big refactoring right now. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This is a temporary workaround to avoid introducing a big refactoring right now. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This is a temporary workaround to avoid introducing a big refactoring right now. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This changes are needed since [1] is landed on mainline. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need some temporary workaround to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This changes are needed since [1] is landed on mainline. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need this change to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This changes are needed since [1] is landed on mainline. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
When introducing the LengthLimitedRead trait in [1], for some serialization macros we need this change to avoid that we have the following compilation error error[E0277]: the trait bound `Bolt12Invoice: Readable` is not satisfied --> lightning/src/util/ser_macros.rs:1243:35 | 1243 | Ok($st::$tuple_variant_name($crate::util::ser::Readable::read(reader)?)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Readable` is not implemented for `Bolt12Invoice` | ::: lightning/src/events/mod.rs:2463:1 | 2463 | / impl_writeable_tlv_based_enum_legacy!(PaidInvoice, 2464 | | ; 2465 | | (0, Bolt12Invoice), 2466 | | (2, StaticInvoice) 2467 | | ); | |_- in this macro invocation | = help: the following other types implement trait `Readable`: () (A, B) (A, B, C) (A, B, C, D) (A, B, C, D, E) (A, B, C, D, E, F) (A, B, C, D, E, F, G) AnnouncementSigsState and 200 others = note: this error originates in the macro `impl_writeable_tlv_based_enum_legacy` (in Nightly builds, run with -Z macro-backtrace for more info) This changes are needed since [1] is landed on mainline. [1] lightningdevkit#3640 Co-Developed-by: @valentinewallace Suggested-by: @valentinewallace Signed-off-by: Vincenzo Palazzo <[email protected]>
Closes #3292