CurlHttpClient missing Server Name Indication when doing SSL handshake (Client Hello)

[PoundKey]

Hello AWS C++ SDK Team,

We're using the CurlHttpClient to communicate with our ADFS server, and when establishing the SSL connection, we realized that "Client Hello" does not include Server Name Indication in the header extension section.

Server Name Indication (Read more about SNI) is important when establishing SSL connection with the ADFS server.

And on OSX and Windows this piece of information is included in the Client Hello header.

A simple test application that I write to test and http client can be found on the link below, along with some Wireshark log and environment information.

HttpClientTest Source

#include <iostream>
#include <aws/core/Aws.h>
#include <aws/core/utils/StringUtils.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/core/http/HttpRequest.h>
#include <aws/core/http/HttpResponse.h>

using namespace Aws;
using namespace Aws::Client;
using namespace Aws::Utils;
using namespace Aws::Http;

namespace 
{
    static const char* ADFS_LOG_TAG = "PGOAdfsCredentialsProvider";

    // Additional configuration
    static const int MAX_REDIRECT_LIMIT = 10;
}


int main(int argc, char* argv[]) {
    std::cout << "Initializing HTTPClient..." << std::endl;

    SDKOptions options;
    InitAPI(options);

    ClientConfiguration config;
    
    config.followRedirects = false;
    config.verifySSL = false;

    std::shared_ptr<HttpClient> client = CreateHttpClient(config);
    String uri;
    
    if (argc >= 2) 
    {
 	uri = "https://" + String(argv[1]) + ":443/adfs/ls/IdpInitiatedSignOn.aspx?loginToRp=urn:amazon:webservices";
        
    } 
    else 
    {
	uri = "https://server.adfs.internal:443/adfs/ls/IdpInitiatedSignOn.aspx?loginToRp=urn:amazon:webservices";
    }
    
    std::cout << "Connecting to: " + uri << std::endl;

    std::shared_ptr<HttpRequest> request = CreateHttpRequest(
        URI(uri),
        HttpMethod::HTTP_GET,
        Stream::DefaultResponseStreamFactoryMethod);

    std::cout << "Sent HTTP GET request." << std::endl;
    std::shared_ptr<HttpResponse> response = client->MakeRequest(*request);

    if (!response || response->GetResponseCode() != HttpResponseCode::OK)
    {
        String message;
        if (response)
        {
            message = "Response code: " + std::to_string(static_cast<int>(response->GetResponseCode()));
        }

        std::cout << "[GET] Failed to send request to ADFS. " << message << std::endl;
    }

    std::cout << "Finished testing HttpCLient." << std::endl;
}

Thanks for the help in advance

[JonathanHenson]

what version of libcurl are you using? SNI support was added in 7.18.1.

Also what TLS stack is your libcurl compiled and linked against? OpenSSL, GnuTLS, libNSS etc....

[PoundKey]

Hello,

This is the environment that I used to compile the CPP SDK:

Compiler requirement: GCC 4.9

AWS C++ SDK version: 1.0.103
OpenSSL version: 1.0.2j
LibCurl version: 7.44.0
Zlib version: 1.2.8

I have also tried using the default OpenSSL, LibCurl and ZLIB specified in build_external.cmake by adding -DBUILD_OPENSSL -DBUILD_CURL and -DBUILD_ZLIB flags when compiling the SDK. Same issue can be reproduced (Missing SNI)

Thanks.

[JonathanHenson]

are you specifying a host header?

…

Sent from my iPhone

On Jun 28, 2017, at 2:26 PM, CX ***@***.***> wrote: Hello, This is the environment that I used to compile the CPP SDK: Compiler requirement: GCC 4.9 AWS C++ SDK version: 1.0.103 OpenSSL version: 1.0.2j LibCurl version: 7.44.0 Zlib version: 1.2.8 I have also tried using the default OpenSSL, LibCurl and ZLIB specified in build_external.cmake by adding -DBUILD_OPENSSL -DBUILD_CURL and -DBUILD_ZLIB flags when compiling the SDK. Same issue can be reproduced (Missing SNI) Thanks. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[PoundKey]

Hello Jonathan,

Could you elaborate more on "specifying a host header"?
Do you mean when sending out the HTTP request?

Thanks.

[JonathanHenson]

My guess is that config.verifySSL = false; is turning off SNI on that platform.

If you run curl directly do you see the SNI information being exchanged?

[PoundKey]

Hello Jonathan,

Yep, runing CURL directly on a Ubuntu/CentOS machine will work.
(OpenSSL version: 1.0.2j, LibCurl version: 7.44.0, Zlib version: 1.2.8)

config.verifySSL is turned off because the server we're trying to GET/POST against is sometimes self-signed and in this case users would still be able to interact with the server.

I can try turn it on and give you another update soon.
It just that the same code and configuration can be ran on OSX platform without any problem, whereas both Linux and OSX should be using the same CurlHttpClient. This is a bit confusing.

Thanks.

[JonathanHenson]

curl is not the same on any platform as any other platform. So comparing behavior between platforms when curl is involved would be misleading. We need to narrow this down to your particular platform and configuration. Also are you sure the libcurl is linked against openssl versus nss or gnutls?

…

Sent from my iPhone

On Jul 10, 2017, at 9:55 AM, CX ***@***.***> wrote: Hello Jonathan, Yep, runing CURL directly on a Ubuntu/CentOS machine will work. (OpenSSL version: 1.0.2j, LibCurl version: 7.44.0, Zlib version: 1.2.8) config.verifySSL is turned off because the server we're trying to GET/POST against is sometimes self-signed and in this case users would still be able to interact with the server. I can try turn it on and give you another update soon. It just that the same code and configuration can be ran on OSX platform without any problem, whereas both Linux and OSX should be using the same CurlHttpClient. This is a bit confusing. Thanks. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[PoundKey]

That I am pretty sure. We statically linked aws-sdk and libcurl against OpenSSL 1.0.2j.

[PoundKey]

Also, there might be a bug in the WinHttpSyncHttpClient.
File link: HERE

Line #99

if (!WinHttpSetOption(GetOpenHandle(), WINHTTP_OPTION_SECURITY_FLAGS, &flags, sizeof(flags))) {
     AWS_LOGSTREAM_FATAL(GetLogTag(), "Failed to turn ssl cert ca verification off.");
}

WINHTTP_OPTION_SECURITY_FLAGS cannot be set using the Session handle returned from GetOpenHandle(). According this the official API HERE. In this case, the code block should always return true.

It can be set using the RequestHandle, which is similar to function void* WinHttpSyncHttpClient::OpenRequest Line# 149

if (!WinHttpSetOption(hHttpRequest, WINHTTP_OPTION_DISABLE_FEATURE, &requestFlags, sizeof(requestFlags))) {
   AWS_LOGSTREAM_FATAL(GetLogTag(), "Failed to turn off redirects!");
}

Thanks

[PoundKey]

Hello Jonathan,

Basically CurlHttpClient cannot be used for any website that requires HTTPS and server certificate verification due to this SNI issue, unless that website has a fallback default server certificate (In this case missing SNI in the request header is fine).
We tried write our own code, linking against the same libcurl, openssl and libz and SNI is set properly in the request header.

config.verifySSL being turned on or off should not not be related to this issue.

I think if you write a simple code that makes a HTTPS GET request to any any HTTPS server, this issue can be reproduced again.

Thanks

[JonathanHenson]

can you show what you are doing differently with libcurl to make sni work?

…

Sent from my iPhone

On Aug 3, 2017, at 4:49 PM, CX ***@***.***> wrote: Hello Jonathan, Basically CurlHttpClient cannot be used for any website that requires HTTPS and server certificate verification due to this SNI issue. We tried write our own code, linking against the same libcurl, openssl and libz and SNI is set properly in the request header. config.verifySSL being turned on or off should not not be related to this issue. I think if you write a simple code that makes a HTTPS GET request to any any HTTPS server, this issue can be reproduced again. Thanks — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[PoundKey]

Hello Jonathan,

After further investigation, we realized that this really has to do with the way that we build Libcurl incorrectly. (SNI is only supported in curl 7.18.1 and after).

Sorry for the false positive alert, and thank you for your help along the way.

This ticket can be closed.

Thanks.