Downloads fail without error when a node in the cluster goes down

[hashworks]

When one uses a load balancer with health checks to access a Minio cluster, downtimes of single nodes should not affect users too much or at the very least provide error reports to them. Currently the web client / console does not do that: Failed downloads are marked as a successful download if nodes fail during the transfer.

Expected Behavior

Minimum: The download fails and reports an error message.
Best: The web client retries the failed parts for a documented amount of time and fails otherwise.

Current Behavior

The web client marks the download as successful and stores the smaller, unfinished file without an error. Users are not informed about the file corruption.

I downloaded the same file two times:

The first download succeded, but during the second download I stopped a cluster node that was providing parts, which did not result in an error and provided a cut-off / corrupted file:

 % lsd -l ~/Downloads/output_file*
.rw-rw-r-- hashworks hashworks  10 GB Mon Aug 21 22:00:51 2023  /home/hashworks/Downloads/output_file
.rw-rw-r-- hashworks hashworks 1.3 GB Mon Aug 21 22:10:42 2023  /home/hashworks/Downloads/output_file(1)

Steps to Reproduce

1. Set up a small Minio cluster, f.e. three nodes, access it using a load balancer with health checks
2. Upload a larger file
3. Try to download the file using the web client / console
4. Verify that a specific node is used to provide parts of the file (f.e. using nethogs)
5. Kill minio on said node
6. Note the successful download, but the insufficient file size

Context

Other tools handle this better:

• mc reports an error and fails the download (EOF)
• rclone retries the failed parts and finishes the download using the degraded cluster

Your Environment

• MinIO version used (minio --version): RELEASE.2023-08-09T23-30-22Z (commit-id=eb55034dfe5ef82449796c83e3126b245c5aee05)
• Server setup and configuration: 3x(Minio + nginx reverse proxy) + Hetzner Load Balancer
• Operating System and version (uname -a): Linux test1.example.net 6.1.0-11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08) x86_64 GNU/Linux
• Web browser: Firefox 116.0.3

[cesnietor]

We'll try to handle the error for large files when we are not using direct download (files bigger than 50Gb use direct download).

[cesnietor]

I was able to reproduce it,
We are using XMLHttpRequest and we could set a timeout but that would fail the entire request even if it's downloading just fine, not only if it hangs. (timeout)

Possible solution, create a unique ID per download and check with a global function whether the request is hanging inside onreadystatechange. A downside would be that it would be stored in memory.

On a side note, after having the file hanging for a couple of minutes, the object gets downloaded eventually so it might change the server in the backend. So we need to see if there's another timeout in the backend that we could use.