Swarm queries fail with `Cannot find manifest file for data file` error

[hodgesrm]

Describe the bug
Swarm queries fail on to a problem accessing manifest data. The same query works on the initiator. This appears to be due to some sort of corruption of the Iceberg files on S3.

When it works, you get an error like the following on the initiator:

2025.07.08 00:59:36.621970 [ 84 ] {} <Error> TCPHandler: Code: 36. DB::Exception: Cannot find manifest file for data file: btc/transactions/data/1751086330717-0730f2009f6fc6e8111b14b0e7c0b5a83007226d1b1564f2231d822a4973f777.parquet: While executing IcebergS3(_table_function.icebergS3Cluster)Source. (BAD_ARGUMENTS)

The full log message shows the following underlying error:

2025.07.08 00:59:36.621679 [ 84 ] {0ab0bd55-38ce-4388-8e2c-fdc6fd3fff6b} <Error> executeQuery: Code: 36. DB::Exception: Cannot find manifest file for data file: btc/transactions/data/1751086330717-0730f2009f6fc6e8111b14b0e7c0b5a83007226d1b1564f2231d822a4973f777.parquet: While executing IcebergS3(_table_function.icebergS3Cluster)Source. (BAD_ARGUMENTS) (version 25.3.3.20139.altinityantalya.20139 (official build)) (from [::1]:39160) (query 1, line 1) (in query: SELECT __table1.date AS date, sum(__table1.output_count) AS `sum(output_count)` FROM icebergS3Cluster('swarm', 's3://rhodges-ice-rest-catalog-demo/btc/transactions', 'ASIAWKGGVET5C6U2HRQS', '[HIDDEN]', 'IQoJb3JpZ2luX2VjEHkaCXVzLXdlc3QtMiJGMEQCIHSYUkGfhq1ARjBiIfhvX4Y/DZedgctaJ8BdJrqrCe8ZAiB5u6r+lyLB3EO4h6QK/nlneKNM2lESbcY4wrtqGbhpACqKBQiC//////////8BEAIaDDQzNDIwODMxODcxNCIM68xWjXSta7SIcRwUKt4EqG3rsdM7x3vILvYeHhbu6j6HAM1kFVG2CXRSMuaZ08Qi+F0nUMMU1lmUnRhLUH3A6oGYb2AwvwFNuESfi1TwA7UIA+IemH9b6NVcX6awlVaB5B9o2yYqkQ9C6NNye2IZfvlMcckjfYZP//Qm7/oImkyajr2tTcpruKNwKnw0VGS61Iz4O2R7ka90u+bepMxaHBjGPc0n4gJ71ZcWduT/KZE7VV7+BlyKuANkCwy+CeuyvnM6yJrrBlZ+XP1KdX7Rx4NupQ3ddMUjjgOr1v3zjGlHB5ESpNz+r/48tUM+xBjRHFwUpoC4buN9MKudWrHqGMt3w8abq6hcRbpT2TFEZj/aUh2XMipR/32AoAlRlHDwVQxTmCziMYk5mICd0rKr3rgBMtbL8/Xj80UAIVewONbOHT+n/vx4//Tge1//ATJgjl6hip3vSOwXZcSq91PMXdw1KuLe8ndQs0alzj5lpdupjTf44oElGi1yy6BOvHFTL4R1AxZnhYVqsZFpiBcxOyoUWYGWuMNZysSptGq5+lb46iHe9a1zt8S4xUc45FtgQojSxtxYtyEYBLRhSAxEmc5erz6RZlr40W8DKq90NYd9K/yB4qOPntVncPHrBNdoH1tg6f9yigQSANzLSf76AjnJMXxSaD59P04TyQt4rbpuWs543HonzvMDINLHa3yJ0YlX+3d2kx3IzHEi3sibfyIeGVUt7vebruYLyFQ/hgSF1+JmKJqPN//4cxurn3X4QOJb+oVZDrb3ySXCo3jc3xRY0ITyYvLzz4Gu2XevoIsF2jMAcBWxs1etSol+MI7SscMGOpsB7nb7z5uSGF1pb5Ye1NZeZkWt3P6iPxBOci1W5MD1BcO+CaA5868xKOqAmINJx2qtke+ScGjjcysH9g5BIm+gDlnWC4zmUyf4oQV5B6Mvt0YCgumy/q+ncgqOtrkYbcTFctzpFTlWw/obCFS/1Y8sAPP2x1j/8X3OCYVk5G965xyeUS9ZxMKBflEeK54nsvOuxtSijk0dUUHOums=', 'Parquet', '`hash` Nullable(String), `version` Nullable(Int64), `size` Nullable(Int64), `block_hash` Nullable(String), `block_number` Nullable(Int64), `index` Nullable(Int64), `virtual_size` Nullable(Int64), `lock_time` Nullable(Int64), `input_count` Nullable(Int64), `output_count` Nullable(Int64), `is_coinbase` Nullable(Bool), `output_value` Nullable(Float64), `outputs` Array(Tuple(address Nullable(String), index Nullable(Int64), required_signatures Nullable(Int64), script_asm Nullable(String), script_hex Nullable(String), type Nullable(String), value Nullable(Float64))), `block_timestamp` Nullable(DateTime64(6, \'UTC\')), `date` Nullable(String), `last_modified` Nullable(DateTime64(6, \'UTC\')), `fee` Nullable(Float64), `input_value` Nullable(Float64), `inputs` Array(Tuple(address Nullable(String), index Nullable(Int64), required_signatures Nullable(Int64), script_asm Nullable(String), script_hex Nullable(String), sequence Nullable(Int64), spent_output_index Nullable(Int64), spent_transaction_hash Nullable(String), txinwitness Array(Nullable(String)), type Nullable(String), value Nullable(Float64)))') AS __table1 WHERE __table1.date = '2025-02-01' GROUP BY __table1.date ORDER BY __table1.date ASC SETTINGS use_hive_partitioning = 1, input_format_parquet_use_metadata_cache = 1, enable_filesystem_cache = 1, use_iceberg_metadata_files_cache = 1), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4d923b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f9df4c
2. DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x0000000009fb3feb
3. DB::IcebergMetadata::getSchemaVersionByFileIfOutdated(String) const @ 0x0000000011f273c5                             4. DB::IcebergMetadata::getInitialSchemaByPath(String const&) const @ 0x0000000011f2bb1f
4. 5. DB::DataLakeConfiguration<DB::StorageS3Configuration, DB::IcebergMetadata>::getInitialSchemaByPath(String const&) const @ 0x000000001134fad3
6. DB::StorageObjectStorageSource::createReader(unsigned long, std::shared_ptr<DB::IObjectIterator> const&, std::shared_ptr<DB::StorageObjectStorage::Configuration> const&, std::shared_ptr<DB::IObjectStorage> const&, DB::ReadFromFormatInfo&, std::optional<DB::FormatSettings> const&, std::shared_ptr<DB::KeyCondition const> const&, std::shared_ptr<DB::Context const> const&, DB::SchemaCache*, std::shared_ptr<Poco::Logger> const&, unsigned long, unsigned long, bool) @ 0x0000000011e8b5b0
7. DB::StorageObjectStorageSource::generate() @ 0x0000000011e89793
8. DB::ISource::tryGenerate() @ 0x0000000014d9731e
9. DB::ISource::work() @ 0x0000000014d96f27
10. DB::ExecutionThreadContext::executeTask() @ 0x0000000014db2b16
11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000014da6604
12. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreadsImpl()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000014da8d3f
13. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000f60f3db
14. void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000f615f02
15. ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000f60c70f
16. void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000f6139da
17. ? @ 0x00007f7fc3f48ac3
18. ? @ 0x00007f7fc3fda850

To Reproduce
Steps to reproduce the behavior on AWS EKS:

1. Set up initiator and swarm servers using manifest files at https://github.com/Altinity/antalya-examples/tree/main/kubernetes/manifests
2. Add ice catalog using scripts at https://github.com/Altinity/ice/tree/master/examples/eks
3. Load tables using ice commands. Examples:

ice insert nyc.taxis -p https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet
ice insert btc.transactions -p --s3-no-sign-request --s3-region=us-east-2 s3://aws-public-blockchain/v1.0/btc/transactions/date=2025-0*/*/*.parquet

4. (Do the above multiple times and ensure a few things fail so that files are left on S3 without being fully loaded.)
5. Run a query that uses the swarm.

SELECT date,  sum(output_count)
FROM ice.`btc.transactions`
WHERE date = '2025-02-01'
GROUP BY date ORDER BY date ASC
SETTINGS use_hive_partitioning = 1, object_storage_cluster = 'swarm', input_format_parquet_use_metadata_cache = 1, enable_filesystem_cache = 1, use_iceberg_metadata_files_cache = 1

Expected behavior
This query should return date and sum value. It does so on the initiator.

Screenshots
On the swarm you get the error described above.

Key information
Provide relevant runtime details.

• Antalya build: 25.3.3.20139.altinityantalya.20139 (container)
• Iceberg catalog: Ice 0.4.0
• Environment: AWS EKS 1.3.2 and native S3 storage

Additional context
You can "fix" the problem by deleting tables and cleaning off S3 storage fully, then reloading. E.g.:

ice delete-table foo.bar
ice delete-table foo.baz
 2010  aws s3 rm --recursive s3://rhodges-ice-rest-catalog-demo/

Another clue: once things are broken, they seem to break across multiple tables.

The current hypothesis is that there is an edge case where Iceberg files get corrupted in a way that queries working through S3 storage fail. However, I tried a few obvious repros and they didn't seem to help.

1. Manually kill large uploads.
2. Delete and reload tables in Ice catalog using ice delete-table without cleaning up extra files (e.g., from previous step).
3. Reload the same parquet files after a failure.

[alsugiliazova]

I got the same exception on initiator node when I have two tables under one namespace and select from the second table.

Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: Received from clickhouse3:9000. DB::Exception: Cannot find manifest file for data file: data/data/symbol_partition=Eve/00000-1-0f785ef5-0a35-4bac-9003-2a29c41016c4.parquet: While executing IcebergS3(_table_function.icebergS3Cluster)Source. (BAD_ARGUMENTS) 
(query: SELECT * FROM datalake_760ea642_5cb8_11f0_8ec9_e12f95dcac7e.`iceberg_760ea641_5cb8_11f0_8ec9_e12f95dcac7e.table_2_760ea644_5cb8_11f0_8ec9_e12f95dcac7e` ORDER BY tuple() FORMAT Values)

I can create simply python script with repro.

[alsugiliazova]

The same exception message appears in scenario with table recreation.

1. Create Iceberg table and DataLakeCatalog database
2. Add data
3. Drop Iceberg table
4. Recreate with same name
5. Add new data
6. Try to read from ClickHouse

I get on 6th step:

Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: Received from clickhouse2:9000. DB::Exception: Cannot find manifest file for data file: data/data/symbol_partition=David/00000-0-258b884f-6763-4970-ba18-96928e8d9341.parquet: While executing IcebergS3(_table_function.icebergS3Cluster)Source. (BAD_ARGUMENTS)