S3 Multipart Upload: s3w+https, s3w+http, s3w works in 1.22.1 fails 1.23

[cariaso]

while similar to
#477

I'm seeing what I believe to be a regression.
I have a very automated build procedure.
This command line

samtools view --fai-reference /path/GRCh38.fna -o s3://bucket/path/my.bam /path/src.cram /path/src.cram.crai

is successful under

samtools 1.22.1
Using htslib 1.22.1
Copyright (C) 2025 Genome Research Ltd.

Samtools compilation details:
Features: build=configure curses=no
CC: gcc
CPPFLAGS:
CFLAGS: -Wall -g -O2
LDFLAGS:
HTSDIR: htslib-1.22.1
LIBS:
CURSES_LIB:

HTSlib compilation details:
Features: build=configure libcurl=yes S3=yes GCS=yes libdeflate=no lzma=yes bzip2=yes plugins=no htscodecs=1.6.4
CC: gcc
CPPFLAGS:
CFLAGS: -Wall -g -O2 -fvisibility=hidden
LDFLAGS: -fvisibility=hidden

HTSlib URL scheme handlers present:
built-in: file, data, preload
S3 Multipart Upload: s3w+https, s3w+http, s3w
Amazon S3: s3+https, s3, s3+http
Google Cloud Storage: gs+http, gs+https, gs
libcurl: ftps, ftp, http, https
crypt4gh-needed: crypt4gh
mem: mem

but fails when built as

samtools 1.23-3-g893eed1
Using htslib 1.23-2-g6dd0d7d0
Copyright (C) 2025 Genome Research Ltd.

Samtools compilation details:
Features: build=configure curses=yes
CC: gcc
CPPFLAGS:
CFLAGS: -Wall -g -O2
LDFLAGS:
HTSDIR: ../htslib
LIBS:
CURSES_LIB: -lncursesw

HTSlib compilation details:
Features: build=configure libcurl=yes S3=yes GCS=yes libdeflate=yes lzma=yes bzip2=yes plugins=yes plugin-path=/usr/libexec/htslib: htscodecs=1.6.5
CC: gcc
CPPFLAGS:
CFLAGS: -Wall -g -O2 -fvisibility=hidden
LDFLAGS: -fvisibility=hidden -rdynamic

HTSlib URL scheme handlers present:
built-in: file, data, preload
Amazon S3: s3+https, s3, s3+http
Google Cloud Storage: gs+http, gs+https, gs
libcurl: ftps, ftp, http, https
crypt4gh-needed: crypt4gh
mem: mem

with the error

[E::hts_open_format] Failed to open file "s3://bucket/path/my.bam" : Resource temporarily unavailable
samtools view: failed to open "s3://bucket/path/my.bam" for writing: Resource temporarily unavailable

The likely suspect is the presence / absence of

S3 Multipart Upload:         s3w+https, s3w+http, s3w

but my build procedure hasn't changed except for the version number. Does anything about the 1.23 release explain what I'm no longer seeing?

[daviesrob]

The S3 code had a major rewrite for 1.23, mostly on the reading side, but the writing part was merged into the main S3 plug-in so the separate s3w is no longer needed. Apart from being moved, not many changes happened on the writing side so it should still work.

It would be useful if you could rerun your tests with --verbosity=8. That will make libcurl dump out all of the http transactions, which will hopefully give some clues about where the problem is occurring.

[cariaso]

that narrows it down considerably.

[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "crypt4gh-needed"
[D::init_add_plugin] Loaded "libcurl"
[D::init_add_plugin] Loaded "gcs"
[D::init_add_plugin] Loaded "s3"
hfile_s3: s3_open_v4 url https://bucket/path/my.bam
* Host bucket:443 was resolved.
* IPv6: (none)
* IPv4: 52.219.233.172, 3.5.130.0, 3.5.133.231, 52.219.179.92, 3.5.129.247, 52.219.92.228, 3.5.130.83, 52.219.179.100
*   Trying 52.219.233.172:443...
* ALPN: curl offers h2,http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=*.s3.us-east-2.amazonaws.com
*  start date: Nov 14 00:00:00 2025 GMT
*  expire date: Oct 30 23:59:59 2026 GMT
*  subjectAltName: host "bucket.s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M04
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to bucket.s3.amazonaws.com (52.219.233.172) port 443
* using HTTP/1.x
> POST /path/my.bam?uploads HTTP/1.1
Host: buckets3.amazonaws.com
User-Agent: htslib/1.23 libcurl/8.11.1
Accept: */*
x-amz-date: 20260108T161544Z
x-amz-content-sha256: 1234567890987654321234567890987654321
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

* Request completely sent off
< HTTP/1.1 403 Forbidden
< x-amz-request-id: 12345678909876
< x-amz-id-2: qwertyuioplkjhgfdsazxcvbnm0987654321qwertyuiop=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 08 Jan 2026 16:15:43 GMT
< Server: AmazonS3
<
* Connection #0 to host bucket.s3.amazonaws.com left intact
hfile_s3: S3 error 403: AccessDenied
Anonymous users cannot initiate multipart uploads.  Please authenticate.
[E::hts_open_format] Failed to open file "s3://bucket/path/my.bam" : Resource temporarily unavailable
samtools view: failed to open "s3://bucket/path/my.bam" for writing: Resource temporarily unavailable

however, I've now made a few attempts at different ways of passing alternative credentials, and I've not yet found a solution.

This is a private bucket so perhaps these are relevant.

#898
#776
#839

What I've tried so far.

long term I've mostly relied on an
"AWS IAM Role for an instance (EC2)"
which is most attractive due to it's auto rotation, and the non-need for credentials on disk.

There are cases where I instead rely on a different set of longer term credentials written in ~/.aws/credentials and then I do my samtools calls with the env var AWS_DEFAULT_PROFILE set to the chosen credential name. Doing that does improve things a bit, but does not resolve the issue.

...
* Connected to bucket.s3.amazonaws.com (3.5.130.200) port 443
* using HTTP/1.x
> POST /path/my.bam?uploads HTTP/1.1
Host: bucket.s3.amazonaws.com
User-Agent: htslib/1.23 libcurl/8.11.1
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AK23456789876543/20260108/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=1234567890876543234567890987654
x-amz-date: 20260108T163844Z
x-amz-content-sha256: 0987654321234567898765432345678765434567876543
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

* Request completely sent off
< HTTP/1.1 200 OK
< x-amz-id-2: 1234567890poiuytrewqasdfghjkmnbvcxzaq23456789oiuytrewsdfgh
< x-amz-request-id: KQWERTYUIUYTREWERTYUIUY
< Date: Thu, 08 Jan 2026 16:38:45 GMT
< x-amz-abort-date: Fri, 16 Jan 2026 00:00:00 GMT
< x-amz-abort-rule-id: delete-incomplete-mpu-7days-runs
< x-amz-server-side-encryption: AES256
< Transfer-Encoding: chunked
< Server: AmazonS3
<
* Connection #1 to host bucket.s3.amazonaws.com left intact
hfile_s3: warning, unable to report full S3 error status.
[E::hts_open_format] Failed to open file "s3://bucket/path/my.bam" : Resource temporarily unavailable
samtools view: failed to open "s3://bucket/path/my.bam" for writing: Resource temporarily unavailable

Notice the Authorization: AWS4-HMAC-SHA256 Credential=AK23456789876543/20260108/us-east-2/s3/
which wasn't present on the prior traffic.

As per
#839
this is likely to be important, but it remains insufficient.

So it seems the new code does not respect AWS IAM Roles.

I'd hope to see that functionality restored, but if there is a credential based way that should work, I'd like to hear more about it.

maybe helpful related:
https://how.wtf/aws-sigv4-requests-with-curl.html
https://gist.github.com/outofcoffee/1dd9c515e7e64c134a41de7916ddf055

[daviesrob]

The S3 plug-in has never directly supported IAM credentials. Currently you need to run a script like the example in the documentation to obtain the short-lived credentials and write them into a file that the plug-in can use.

Your second example is interesting as it appears that AWS returned a 200 response, but somehow the plug-in got into a code path that's supposed to deal with errors. We'll have to do some investigation, and maybe add a bit more logging, to see how it got there.