Panic in canonical_request.rs due to invalid S3 object metadata

[mlafeldt]

Bug Report

This code will panick here when executed in Lambda.

Wild guess: The same code worked perfectly fine yesterday, so I'm assuming that something has changed on the service side (?).

Here's a full backtrace from CloudWatch logs:

START RequestId: 26286f27-4dce-4928-915a-562730b70753 Version: $LATEST
--
  | [2021-09-22T10:06:17Z DEBUG get_strip] Got input: Input { date: None }
  | [2021-09-22T10:06:18Z INFO get_strip] Scraping done: Comic { date: 2021-09-22, title: "Manual Isn’t Done", image_url: "https://assets.amuniversal.com/02478e60f252013976ad005056a9545d", strip_url: "https://dilbert.com/strip/2021-09-22" }
  | [2021-09-22T10:06:18Z INFO get_strip] Downloading strip from https://dilbert.com/strip/2021-09-22 ...
  | [2021-09-22T10:06:19Z INFO get_strip] Uploading strip to bucket dilbert-feed-dev-bucket83908e77-1vf62q8omk5bf ...
  | [2021-09-22T10:06:19Z INFO aws_config::meta::region] load_region; provider=EnvironmentVariableRegionProvider { env: Env(Real) }
  | [2021-09-22T10:06:19Z INFO tracing::span] build_profile_provider
  | [2021-09-22T10:06:19Z INFO smithy_http_tower::parse_response] send_operation
  | [2021-09-22T10:06:19Z INFO smithy_http_tower::parse_response] send_operation; operation="PutObject"
  | [2021-09-22T10:06:19Z INFO smithy_http_tower::parse_response] send_operation; service="s3"
  | [2021-09-22T10:06:19Z INFO aws_config::meta::credentials::chain] load_credentials; provider=Environment
  | [2021-09-22T10:06:19Z INFO aws_config::meta::credentials::chain] loaded credentials provider=Environment
  | thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ToStrError { _priv: () }', /Users/mathias/.cargo/git/checkouts/aws-sdk-rust-be00ef1d3f696844/e80f074/sdk/aws-sigv4/src/http_request/canonical_request.rs:226:46
  | stack backtrace:
  | 0: 0x563b6e8fb780 - std::backtrace_rs::backtrace::libunwind::trace::ha0ad43e8a952bfe7
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
  | 1: 0x563b6e8fb780 - std::backtrace_rs::backtrace::trace_unsynchronized::h6830419c0c4130dc
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
  | 2: 0x563b6e8fb780 - std::sys_common::backtrace::_print_fmt::h8f3516631ffa1ef5
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:67:5
  | 3: 0x563b6e8fb780 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he1640d5f0d93f618
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:46:22
  | 4: 0x563b6e91d05c - core::fmt::write::h88012e1f01caeebf
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/fmt/mod.rs:1115:17
  | 5: 0x563b6e8f62c5 - std::io::Write::write_fmt::h360fa85b30182555
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/io/mod.rs:1665:15
  | 6: 0x563b6e8fd7bb - std::sys_common::backtrace::_print::ha1f00492f406a015
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:49:5
  | 7: 0x563b6e8fd7bb - std::sys_common::backtrace::print::hd54561b13feb6af3
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:36:9
  | 8: 0x563b6e8fd7bb - std::panicking::default_hook::{{closure}}::h84fe124cd0864662
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:208:50
  | 9: 0x563b6e8fd291 - std::panicking::default_hook::h5a8e74a76ce290a7
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:225:9
  | 10: 0x563b6e8fde84 - std::panicking::rust_panic_with_hook::h67c812a4fe9d4c91
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:622:17
  | 11: 0x563b6e8fd967 - std::panicking::begin_panic_handler::{{closure}}::h33f9c1b96af300d7
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:519:13
  | 12: 0x563b6e8fbc7c - std::sys_common::backtrace::__rust_end_short_backtrace::h51bae64be5921f0e
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:141:18
  | 13: 0x563b6e8fd8c9 - rust_begin_unwind
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
  | 14: 0x563b6e2e6851 - core::panicking::panic_fmt::h12a3a3c256485fca
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
  | 15: 0x563b6e2e6943 - core::result::unwrap_failed::h2d8d0952e3250de9
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/result.rs:1599:5
  | 16: 0x563b6e6ef7fb - <aws_sigv4::http_request::canonical_request::CanonicalRequest as core::fmt::Display>::fmt::h46a709cf319760e6
  | 17: 0x563b6e6e6cfd - aws_sigv4::http_request::calculate_signing_headers::h30efe68df85bf526
  | 18: 0x563b6e6e2a73 - aws_sig_auth::signer::SigV4Signer::sign::h920b045b6fcdc830
  | 19: 0x563b6e6ea382 - smithy_http::operation::Request::augment::h7536a466479b0bd6
  | 20: 0x563b6e6e24a8 - <aws_sig_auth::middleware::SigV4SigningStage as smithy_http::middleware::MapRequest>::apply::h89af41e18ee6e9d8
  | 21: 0x563b6e36f865 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h6ae758d27b198c4f
  | 22: 0x563b6e36f0f1 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h62fbeaa533a52620
  | 23: 0x563b6e3216c6 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h8217a25c490a1d0a
  | 24: 0x563b6e309279 - <tower::retry::future::ResponseFuture<P,S,Request> as core::future::future::Future>::poll::ha79b487b0a42282b
  | 25: 0x563b6e38586d - get_strip::handler::{{closure}}::h1f2fccc62b894e87
  | 26: 0x563b6e3787d4 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hc5663257c0d4217f
  | 27: 0x563b6e39ac8f - tokio::park::thread::CachedParkThread::block_on::ha630e586e1ddc076
  | 28: 0x563b6e39af24 - tokio::runtime::thread_pool::ThreadPool::block_on::h8f5e136b25cd397f
  | 29: 0x563b6e330a00 - tokio::runtime::Runtime::block_on::hd0513b31c553b720
  | 30: 0x563b6e386f5c - get_strip::main::h54889401df0fdcaa
  | 31: 0x563b6e391d03 - std::sys_common::backtrace::__rust_begin_short_backtrace::hbff95396b4b78db3
  | 32: 0x563b6e31514d - std::rt::lang_start::{{closure}}::h8def548e847b6767
  | 33: 0x563b6e8fe48a - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::ha9408abe89f69dc4
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:259:13
  | 34: 0x563b6e8fe48a - std::panicking::try::do_call::h5b0cc9e9102acb65
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
  | 35: 0x563b6e8fe48a - std::panicking::try::hddc7f8229138b3ba
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
  | 36: 0x563b6e8fe48a - std::panic::catch_unwind::hfa401ff8bab2986e
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
  | 37: 0x563b6e8fe48a - std::rt::lang_start_internal::{{closure}}::h8163422320d11405
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:48
  | 38: 0x563b6e8fe48a - std::panicking::try::do_call::hc742cc7bb4f0fb20
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
  | 39: 0x563b6e8fe48a - std::panicking::try::ha37d8d2dd1acf7d3
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
  | 40: 0x563b6e8fe48a - std::panic::catch_unwind::h8a5381d5ecf437bc
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
  | 41: 0x563b6e8fe48a - std::rt::lang_start_internal::h7e2cee8c90d4a4d3
  | at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:20
  | 42: 0x563b6e387082 - main
  | 43: 0x7f10c99e70ba - __libc_start_main
  | 44: 0x563b6e2e7021 - <unknown>
  | at /Volumes/build/.build/src/glibc-2.12.1/csu/../sysdeps/x86_64/elf/start.S:103
  | END RequestId: 26286f27-4dce-4928-915a-562730b70753
  | REPORT RequestId: 26286f27-4dce-4928-915a-562730b70753 Duration: 3174.53 ms Billed Duration: 3247 ms Memory Size: 128 MB Max Memory Used: 57 MB Init Duration: 72.12 ms XRAY TraceId: 1-614b0019-2264ef356cea07027f2bd9fc SegmentId: 02822c204d17fae7 Sampled: true
  | RequestId: 26286f27-4dce-4928-915a-562730b70753 Error: Runtime exited with error: exit status 101 Runtime.ExitError

Version

aws-rust-sdk v0.0.18-alpha

Platform

x86_64-unknown-linux-gnu

AWS Services

Lambda

[mlafeldt]

I just found out that the panic is caused by the specific file being uploaded to S3 via PutObject. For all other files, it has worked fine so far.

[mlafeldt]

I was able to reproduce it locally with this code:

#[tokio::main]
async fn main() {
    let http_client = reqwest::Client::new();

    let image = http_client
        .get("https://assets.amuniversal.com/02478e60f252013976ad005056a9545d")
        .send()
        .await
        .unwrap()
        .error_for_status()
        .unwrap()
        .bytes()
        .await
        .unwrap();

    aws_sdk_s3::Client::new(&aws_config::load_from_env().await)
        .put_object()
        .bucket("dilbert-feed-dev-bucket83908e77-1vf62q8omk5bf")
        .key("strips/2021-09-22.gif")
        .body(aws_sdk_s3::ByteStream::from(image))
        .content_type("image/gif")
        .metadata("title", "Manual Isn’t Done") // panicks here
        .send()
        .await
        .unwrap();
}

Looks like I need to escape Manual Isn’t Done somehow for this to not panic.

[rcoh]

wow this is a great find.

So the bug is that during canonical request construction, I misunderstood the behavior of HeaderValue::to_str: It will actually only return Ok(...) if the string is valid ascii. This is a bug in canonical request construction, and we'll get it fixed! Thanks for the report.

In the meantime as a workaround, you could replace that smart quote ’ with a regular quote: '

[rcoh]

And just so you know, when you actually query this metadata that is non-ascii, s3 will give you a fairly odd reply:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html
x-amz-meta-nonascii: =?UTF-8?B?w4PChE3Dg8KEWsODwpXDg8KRIFMz?=.

[mlafeldt]

This is how the Go SDK stores "Manual Isn’t Done" without any problem:

Retrieval also works fine. It would be cool if the Rust SDK would do the same encoding:

When using non US-ASCII characters in your metadata values, the provided Unicode string is examined for non US-ASCII characters. If the string contains only US-ASCII characters, it is presented as is. If the string contains non US-ASCII characters, it is first character-encoded using UTF-8 and then encoded into US-ASCII.

[mlafeldt]

Actually, I think this is how S3 stores metadata independent of the SDK. All we need to do is to pass the string correctly?

[jdisanti]

The fix for this has landed in v0.0.19-alpha. Thank you!