[S3Crt] Unable to do getobject when explicit 'Aws::S3Crt::ClientConfiguration::ca_path' is set to default path

[csi-amolpawar]

Describe the bug

Unable to do getobject when explicit 'Aws::S3Crt::ClientConfiguration::ca_path' is set to default path

Expected Behavior

Get Object should work as expected when ca_path is set explicitly.

When we don't set ca_path explicitly, it work fine.

Current Behavior

Receives the error message GetObject error:TLS (SSL) negotiation failed (aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE)

Reproduction Steps

The issue is easily reproducible with below code snippet

#include <iostream>
#include <string>
#include <openssl/crypto.h>
#include <aws/core/Aws.h>
#include <aws/core/utils/memory/stl/AWSStringStream.h>
#include <aws/core/utils/logging/CRTLogSystem.h>
#include <aws/s3-crt/S3CrtClient.h>
#include <aws/s3-crt/model/GetObjectRequest.h>

static const char ALLOCATION_TAG[] = "s3-crt-getobject-public";

std::string get_default_openssl_dir()
{
  const std::string OPENSSLDIR_KEY("OPENSSLDIR: ");

  auto ssl_dir = std::string(SSLeay_version(SSLEAY_DIR));
  auto found = ssl_dir.find(OPENSSLDIR_KEY);
  if(found != std::string::npos)
  {
    ssl_dir = ssl_dir.substr(OPENSSLDIR_KEY.size());
    if(auto s = ssl_dir.size(); ssl_dir.at(0) == '"' && ssl_dir.at(s - 1) == '"')
      ssl_dir = ssl_dir.substr(1, s -2);
  }
  return ssl_dir;
}

int main(int argc, char* argv[])
{
  Aws::SDKOptions options;

  Aws::String ca_path = get_default_openssl_dir();

  options.httpOptions.initAndCleanupCurl = false;
  options.cryptoOptions.initAndCleanupOpenSSL = false;
  options.ioOptions.tlsConnectionOptions_create_fn = [=]() {
    Aws::Crt::Io::TlsContextOptions tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitDefaultClient();
    tlsCtxOptions.SetVerifyPeer(true);
    Aws::Crt::Io::TlsContext tlsContext(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);
    return Aws::MakeShared<Aws::Crt::Io::TlsConnectionOptions>(ALLOCATION_TAG, tlsContext.NewConnectionOptions());
  };

  options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
  options.loggingOptions.crt_logger_create_fn = []() {
    return Aws::MakeShared<Aws::Utils::Logging::DefaultCRTLogSystem>(
      ALLOCATION_TAG, Aws::Utils::Logging::LogLevel::Trace);
  };

  Aws::InitAPI(options);
  {    
    Aws::S3Crt::ClientConfiguration config;
    config.region = Aws::Region::US_EAST_1;
    config.caPath = Aws::String(ca_path);
    Aws::S3Crt::S3CrtClient s3CrtClient(config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never);
    Aws::String bucket("my-tests");
    Aws::String objectKey("test/my_object");

    Aws::S3Crt::Model::GetObjectRequest request;
    request.SetBucket(bucket);
    request.SetKey(objectKey);
    
    if(auto outcome = s3CrtClient.GetObject(request); outcome.IsSuccess())
      std::cout << outcome.GetResult().GetBody().rdbuf() << std::endl;
    else
      std::cerr << "GetObject error:" << outcome.GetError().GetMessage() << std::endl;
  }
  Aws::ShutdownAPI(options);
  return 0;
}

Please note that get_default_openssl_dir() is evaluate to /etc/pki/tls

Possible Solution

NA

Additional Information/Context

Build Command:
g++ -std=c++20 -o <output> test_s3_crt_ca_path.cpp -I${AWS_INSTALL_PATH}/include -L${AWS_INSTALL_PATH}/lib64 -lcurl -lssl -lpthread -lcrypto -laws-cpp-sdk-s3-crt -laws-cpp-sdk-core

AWS CPP SDK version used

AWS SDK for C++ 1.11.351

Compiler and Version used

g++ (GCC) 13.2.0

Operating System and version

Red Hat Enterprise Linux 9.4 (Plow)

[jmklix]

Thanks for the detailed repro code and explanation. Looking into what might be causing this to fail.

Here is what I'm seeing in the logs

[ERROR] 2024-06-27 21:25:17.139 http-connection [140737202796288] static: Client connection failed with error 1029 (AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE).
[WARN] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: Failed to obtain new connection from http layer, error 1029(TLS (SSL) negotiation failed)
[DEBUG] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: Failing excess connection acquisition with error code 1029
[DEBUG] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=1
[WARN] 2024-06-27 21:25:17.140 connection-manager [140737202796288] id=0x9dd080: Failed to complete connection acquisition with error_code 1029(TLS (SSL) negotiation failed)
[ERROR] 2024-06-27 21:25:17.140 S3Endpoint [140737202796288] id=0xb0a170: Could not acquire connection due to error code 1029 (TLS (SSL) negotiation failed)

[jmklix]

It looks like config.caPath might not be finding the correct file. For now you should be able to use config.caFile to workaround this:

        Aws::String ca_file = ca_path + "/cert.pem";
        config.caFile = Aws::String(ca_file);

[csi-amolpawar]

@jmklix It works fine when config.ca_file is set. See

[root@bc281d03816c aws]# ./aws_s3_crt_w_ca_file
ca_file=/etc/pki/tls/cert.pem
object_content= 'This is a test file'

Previously, we were setting only ca_path to default system path and was working as expected. See code snippet below

auto tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitDefaultClient();
  tlsCtxOptions.SetVerifyPeer(!cfg->get<bool>(CTX_SKIP_VERIFY_SSL));
  tlsCtxOptions.OverrideDefaultTrustStore(ca_path.c_str(), "");
...

auto tlsContext = Aws::Crt::Io::TlsContext(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);

config.tlsConnectionOptions = Aws::MakeShared<Aws::Crt::Io::TlsConnectionOptions>("libcsi-io-s3-crt", tlsContext.NewConnectionOptions());

and should be expected to work only setting directly in Aws::S3Crt::ClientConfiguration::ca_path

[DmitriyMusatkin]

can you print out contents of your "/etc/pki/tls" folder?

For context, s3 crt uses s2n on linux for tls support (https://github.com/aws/s2n-tls) and s2n just uses libcrypto's X509_STORE_load_locations under the covers to load all the certs. So somehow libcrypto is not able to load certs from that path. Im not super familiar with where RHEL expects to store certs, but initial guess is that certs are in a slightly different folder or process somehow is not able to access them

[csi-amolpawar]

Please see below

@DmitriyMusatkin

[DmitriyMusatkin]

And cert.pem has the root CA that covers s3 certs? Is it default RHEL's pem or is it custom one?

I did a quick test on my AL2 machine and ca_path seems to work fine.

[csi-amolpawar]

With the same CA certs, S3 is working fine.

Aws::InitAPI(options);
{    
   Aws::Client::ClientConfiguration config;
   config.region = Aws::Region::US_EAST_1;
   config.verifySSL = true;
   config.caPath = get_default_openssl_dir();
   std::cout << "using ca_path:" << config.caPath << std::endl;

   Aws::S3::S3Client s3Client(config);
   Aws::String bucket = "my_bucket";
   Aws::String objectKey = "test/my_object";
   Aws::S3::Model::GetObjectRequest request;
   request.SetBucket(bucket);
   request.SetKey(objectKey);
   Aws::S3::Model::GetObjectOutcome outcome = s3Client.GetObject(request);
   if(outcome.IsSuccess())
     std::cout << outcome.GetResult().GetBody().rdbuf() << std::endl
   else
     std::cerr << "GetObject error:" << outcome.GetError().GetMessage() << std::endl;
}
Aws::ShutdownAPI(options);

Also note that this behaviour has been observed since we upgraded to 1.11.313 from 1.11.100.

@DmitriyMusatkin

[csi-amolpawar]

And cert.pem has the root CA that covers s3 certs? Is it default RHEL's pem or is it custom one?

Yes. This is root CA and default RHEL's PEM.

We have custom self signed certs which we can not use.

[DmitriyMusatkin]

Openssl requires a very specific format for how CAPath should be specified. That format is described here https://www.openssl.org/docs/man3.0/man3/X509_LOOKUP_hash_dir.html under "Hashed Directory Method". So if directory contains just a single cert bundle, Openssl will not be able to use certs from that bundle. Since SDK is basically just wrapping that Openssl behavior, SDK requires the same directory format.

I've tested 1.11.100 and a version before that and could not get it working. As far as i cant tell this has been the behavior for a really long time and there has not been any changes to it in the 1.11.x timeframe.

Also as far as i can tell Curl built with Openssl will have exactly same behavior. So regular s3 client should have the same requirement. And from mine quick testing it does. Curl docs do mention that non-openssl based builds might have a different behavior, but i checked a couple other openssl like libs and they all seem to require same directory format.

[csi-amolpawar]

@DmitriyMusatkin thanks for the update.

Yes, that behaviour for config.ca_path irrespective of version.
Also we were using other approach to set it (see #3007 (comment)).

We were optimised these code by setting Aws::Client::ClientConfiguration::ca_path

[DmitriyMusatkin]

So to summarize, when you were setting ca path through
tlsCtxOptions.OverrideDefaultTrustStore(ca_path.c_str(), "");
it was working, but when you moved to using
config.caPath = ca_path;
it stopped working.

So after some digging i think i know what the issue is.
There is a bug that was introduced around 1.11.97 that cause tls connection options set through tlsConnectionOptions_create_fn to be ignored. This is the pr where this was introduced #2530. Dont think it was on purpose to remove the global tls connection options.

After the PR above, it now checks tls connection options on ClientConfig and then ca_file/ca_path ignoring the globally configured tls connection options. Something we should definitely fix.

So i think whats happening in your case is that when you are providing a global tls connection options callback it currently gets ignored and none of the options you provide are applied. It worked because default options were used. But with setting config.ca_path it no longer works because ca_path is not in expected format.

[DmitriyMusatkin]

#3037 restored behavior to respect connection options specified globally. which would make your old sample work as before.
For caPath as mentioned directory needs to respect openssl cert dir format for it to work and there is nothing to fix.
Change will be available in the next release.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.