Retry after HTTP2 GOAWAY errors

[ozgrakkurt]

Describe the bug

[2023-02-13T11:10:36Z WARN aws_smithy_client::hyper_ext] unrecognized error from Hyper. If this error should be retried, please file an issue. err=http2 error: connection error received: not a result of an error: connection error received: not a result of an error (hyper::Error(Http2, Error { kind: GoAway(b"", NO_ERROR, Remote) }))

Expected Behavior

Should retry

Current Behavior

Error out

Reproduction Steps

none

Possible Solution

No response

Additional Information/Context

No response

Version

aws-smithy-client = "0.54.2"

Environment details (OS name and version, etc.)

docker ubuntu:latest

Logs

[2023-02-13T11:10:36Z WARN aws_smithy_client::hyper_ext] unrecognized error from Hyper. If this error should be retried, please file an issue. err=http2 error: connection error received: not a result of an error: connection error received: not a result of an error (hyper::Error(Http2, Error { kind: GoAway(b"", NO_ERROR, Remote) }))

[ozgrakkurt]

golang/go#18639

hyperium/h2#83

[Velfi]

@ozgrakkurt Thanks for submitting this. Are you actually encountering this error in production right now?

[ozgrakkurt]

Hey, yes. We get this regularly

[vac-adb]

I can reproduce it easily with the following code.
Note that 60s sleep duration between calls is important (there was no issue with 10s sleep).

Version
aws-config = "0.55.1"
aws-sdk-cognitoidentityprovider = "0.26.0"

use aws_sdk_cognitoidentityprovider as cognitoidentityprovider;
use tokio;
use std::thread;
use std::time::Duration;

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let config = aws_config::load_from_env().await;
    let client = cognitoidentityprovider::Client::new(&config);
    let username = "fake_user";
    let password ="fake_password";
    let client_id = "fake_client";
    for i in 1..4 {
        let cognito_result = client.initiate_auth().set_client_id(Some(client_id.to_string())).auth_flow(cognitoidentityprovider::types::AuthFlowType::UserPasswordAuth).auth_parameters("USERNAME", username).auth_parameters("PASSWORD", password).send().await;
        println!("Result {:}: {:?}", i, cognito_result);
        thread::sleep(Duration::from_secs(60));
    }
}

Results:

1. ✅ Err(ServiceError(ServiceError { ... { message: Some("User pool client fake_client does not exist.")
2. ❌ Err(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: hyper::Error(Http2, Error { kind: GoAway(b"", NO_ERROR, Remote) }), connection: Unknown } }))
3. ✅ Err(ServiceError(ServiceError { ... { message: Some("User pool client fake_client does not exist.")

[rcoh]

ah perfect! thanks for the reproducer. it's a quick fix but I wasn't able to reproduce it in tests so I wasn't able to figure out if my fix actually worked. We'll get this prioritized and fixed—or we're happy to accept a PR, the relevant code is here:
https://github.com/awslabs/smithy-rs/blob/main/rust-runtime/aws-smithy-client/src/hyper_ext.rs#LL207C1-L224C2

If we correctly classify that error is kind: Io it should be fixed

[hxk1633]

I get a similar error when trying to invoke dynamodb in Rust AWS SDK:
DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: NoMatchingAuthScheme, connection: Unknown } })
Not sure what this error means...

[rcoh]

This is a different error—no matching auth scheme. Can you open a separate issue and include some logs?

…

On Tue, Sep 12, 2023, 11:04 AM Harrison Kaiser ***@***.***> wrote: I get a similar error when trying to invoke dynamodb in Rust AWS SDK: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: NoMatchingAuthScheme, connection: Unknown } }) Not sure what this error means... — Reply to this email directly, view it on GitHub <#738 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADYKZ4SEETLOGKTYJ5O76DX2B2XRANCNFSM6AAAAAAU2EGWYA> . You are receiving this because you commented.Message ID: ***@***.***>