Skip to content

Conversation

ianton-ru
Copy link

@ianton-ru ianton-ru commented Jan 16, 2025

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Alternative syntax for object storage cluster functions.
Setting object_storage_max_nodes to limit used hosts in object storage cluster family functions (s3Cluster, hdfsCluster, etc).

Documentation entry for user-facing changes

Alternative syntax for object storage cluster functions. Allows to use simple function name (s3, azureBlobStorage, etc) and call simple or cluster variant depends of object_storage_cluster setting.
SELECT * FROM s3(...) SETTINGS object_storage_cluster='cluster'
is equal to
SELECT * FROM s3Cluster('cluster', ...)

Overrided s3, azureBlobStorage, hdfs, icebergS3, icebergAzure, icebergHDFS, deltaLake, hudi table functions.

Setting object_storage_max_nodes to limit used hosts in object storage cluster family functions (s3Cluster, hdfsCluster, etc).
Default value 0 - unlimited.
Sometimes with large clusters makes sense to limit number of hosts involved in distributed request to object storage.
This setting allow to do it.
Now it chooses random hosts from cluster, in common case it should make more or less equable host loading.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing)

All builds in Builds_1 and Builds_2 stages are always mandatory
and will run independently of the checks below:

  • Allow: All Required Checks
  • Allow: Stateless tests
  • Allow: Stateful tests
  • Allow: Integration Tests
  • Allow: Performance tests
  • Allow: All Builds
  • Allow: batch 1, 2 for multi-batch jobs
  • Allow: batch 3, 4, 5, 6 for multi-batch jobs

  • Exclude: Style check
  • Exclude: Fast test
  • Exclude: All with ASAN
  • Exclude: All with TSAN, MSAN, UBSAN, Coverage
  • Exclude: All with aarch64
  • Exclude: All with release
  • Exclude: All with debug

  • Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, BuzzHouse, etc.)
  • Exclude: AST fuzzers

  • Do not test
  • Woolen Wolfdog
  • Upload binaries for special builds
  • Disable merge-commit
  • Disable CI cache

@alex-zaitsev
Copy link
Member

@ianton-ru, name suggestion: object_storage_cluster, object_storage_max_nodes

@Enmk Enmk force-pushed the project-antalya-24.12.2-cluster-alt-syntax branch from 89b24d5 to dbcffed Compare January 21, 2025 13:19
@altinity-robot
Copy link
Collaborator

altinity-robot commented Jan 21, 2025

This is an automated comment for commit c88f3d3 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check nameDescriptionStatus
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests❌ failure
Sign aarch64There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS❌ error
Sign releaseThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS❌ error
Stateful testsRuns stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc❌ failure
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc❌ failure
Stress testRuns stateless functional tests concurrently from several clients to detect concurrency-related errors❌ failure
Successful checks
Check nameDescriptionStatus
BuildsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Compatibility checkChecks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help✅ success
Docker keeper imageThe check to build and optionally push the mentioned image to docker hub✅ success
Docker server imageThe check to build and optionally push the mentioned image to docker hub✅ success
Install packagesChecks that the built packages are installable in a clear environment✅ success
Ready for releaseThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success

@Enmk Enmk force-pushed the project-antalya-24.12.2-cluster-alt-syntax branch from dbcffed to c88f3d3 Compare January 31, 2025 16:25
@Enmk Enmk merged commit 9fb30a3 into project-antalya-24.12.2 Feb 3, 2025
182 of 248 checks passed
@ianton-ru ianton-ru mentioned this pull request Apr 4, 2025
ianton-ru pushed a commit that referenced this pull request May 23, 2025
…-alt-syntax

Alternative syntax for object storage cluster functions
ianton-ru added a commit that referenced this pull request Jul 17, 2025
…-alt-syntax

Alternative syntax for object storage cluster functions
ianton-ru added a commit that referenced this pull request Aug 5, 2025
…-alt-syntax

Alternative syntax for object storage cluster functions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants