[Nebius] Nebius Object Storage support. #4838

SalikovAlex · 2025-02-27T14:19:13Z

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
pytest tests/smoke_tests/test_mount_and_storage.py::test_nebius_storage_mounts --nebius
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests:
pytest tests/smoke_tests/test_mount_and_storage.py::test_nebius_storage_mounts --nebius
pytest tests/smoke_tests/test_mount_and_storage.py::TestStorageWithCredentials::test_externally_created_bucket_mount_without_source --nebius

sync

cblmemo

Thanks for adding this @SalikovAlex ! This is exciting. Left some discussions ;)

sky/adaptors/nebius.py

tests/test_yamls/test_nebius_storage_mounting.yaml

sky/data/storage.py

cblmemo

Thanks for the update @SalikovAlex ! The PR mostly looks good to me. Left some discussions - after that it should be ready to go!

Could you also help run the related smoke test on nebius?

sky/adaptors/nebius.py

cblmemo · 2025-03-11T20:22:03Z

sky/cloud_stores.py

+        # To increase parallelism, modify max_concurrent_requests in your
+        # aws config file (Default path: ~/.aws/config).
+        endpoint_url = nebius.create_endpoint()
+        if 'nebius://' in source:


If it starts with s3://, isn't it be treated as S3 storage and route to the S3Store class?

sky/clouds/nebius.py

sky/data/data_utils.py

tests/smoke_tests/test_mount_and_storage.py

sky/data/storage.py

tests/smoke_tests/test_mount_and_storage.py

SalikovAlex · 2025-03-12T19:14:58Z

Looks like you are right.
I am trying to guess while in R2 we are using

if 'r2://' in source:
    source = source.replace('r2://', 's3://')

cblmemo

Thanks @SalikovAlex for fixing this! Left final comments. After addressing those, could you help running the related smoke test on your side?

sky/data/data_utils.py

cblmemo · 2025-03-13T18:43:44Z

sky/cloud_stores.py

+        # To increase parallelism, modify max_concurrent_requests in your
+        # aws config file (Default path: ~/.aws/config).
+        endpoint_url = nebius.create_endpoint()
+        if 'nebius://' in source:


I think here we also assume it can only start with `nebius://`

tests/smoke_tests/test_mount_and_storage.py

SalikovAlex · 2025-03-14T15:47:09Z

pytest tests/smoke_tests/test_mount_and_storage.py --nebius All tests are green

cblmemo · 2025-03-14T20:39:15Z

/quicktest-core

cblmemo · 2025-03-14T20:40:35Z

/smoke-test -k storage

cblmemo

Thanks for adding this @SalikovAlex ! This is amazing. It mostly looks great to me, besides the quicktest-core failing. Though this does not seems like caused by this PR. @zpoint can you help confirming if the backward compatibility test is failing on master as well?

zpoint · 2025-03-20T00:38:32Z

The pip issue is fixed in #4939. You can merge master and try again.

This commit introduces support for Nebius object storage, enabling users to integrate Nebius storage with various functionalities such as file mounting, syncing, and cloud transfers. It includes necessary adaptors, utility methods, and updates to the storage framework to handle Nebius-specific configurations and endpoints.

Implemented new Nebius storage mounting functionality, including COPY and MOUNT modes, with tests. Updated cloud_stores.py for AWS CLI compatibility and added a YAML template for Nebius storage configurations. Removed outdated Nebius test case in favor of the new approach.

This commit introduces comprehensive Nebius support, making it accessible for S3-compatible operations including bucket creation, deletion, and mounting. It removes reliance on the AWS_SHARED_CREDENTIALS_FILE environment variable, streamlines Nebius-specific configurations, and adds associated unit test parameters to validate functionality across storage operations.

Remove redundant code, streamline imports, and enhance error messaging. Adjust documentation for better accuracy and update function annotations. These changes improve maintainability and readability of the Nebius adaptor module.

Simplified Nebius AWS CLI installation by reusing the S3CloudStorage configuration for consistency. Removed unnecessary debug print in `run_upload_cli` to reduce console noise. Minor formatting adjustment in test YAML file.

Clean up and improve code readability, including string formatting and conditionals. Introduce `_TIMEOUT_TO_PROPAGATES` to handle timeout while verifying Nebius bucket deletions. Update comments to reflect the corrected usage on Nebius servers.

Removed unused variable from subprocess call to clean up code. Updated timeout error to include the bucket name for more detailed and helpful error reporting.

Updated the helper method to assign a default region when no region is specified, ensuring compatibility with Nebius Object Storage. This change avoids potential errors caused by missing region values.

Replace 'nebius://' with 's3://' in source paths to ensure compatibility with AWS CLI commands. This allows seamless integration of Nebius storage endpoints.

* gemma3 * Update gemma3.yaml to specify exact versions for transformers and vllm installations; add readiness probe configuration in service section. Update README.md to correct command option from 'deepseek' to 'gemma-3'. * Remove outdated command option from README.md for clarity. * update readme for serving * Update README.md to correct the number of nodes in the service launch command and update the link to the appropriate SkyPilot YAML file.

Update README and documentation to include Gemma 3 model and example. Added Gemma 3 to the news section in README.md and updated the models index in documentation.

…pilot-org#4934) * Update volume mounting docs * Update volume mounting docs * Update volume mounting docs * Add nested tabset, restructure optional steps * Move volume mounting docs * Update volume mounting docs * Reorder * casing * Comments * fix * reduce links

* remove serve and backcompact * ignore buildkite yaml file

* Update benefits for client-server * update * Update docs/source/reference/api-server/api-server.rst Co-authored-by: Zongheng Yang <[email protected]> --------- Co-authored-by: Zongheng Yang <[email protected]>

* fix flaky for test_cancel_launch_and_exec_async * comma * use generic_cloud * new line format

Signed-off-by: Aylei <[email protected]>

…ypilot-org#4935) * working codepath * remove prints and an assert * make into classes * minor changes * update codepath comment * lint * slight reformat * review feedback * autoscale_detecror -> autoscaler * unnest regions_with_offering logic * short circuit on unsupported autoscaler * formalize context name validation, add exception handling for cluster info request * account for TPUs * code hardening * remove AUTOSCALER_TO_LABEL_FORMATTER in favor of expanding AUTOSCALER_TYPE_TO_AUTOSCALER * more debug logs, review feedbacks * final review comments addressed

…org#4974) * fix annotation "kubernetes.io/ingress.class" is deprecated Signed-off-by: Ajay-Satish-01 <[email protected]> * fix: ingress spec based on version --------- Signed-off-by: Ajay-Satish-01 <[email protected]>

fix dense cli

* [API server] attach setup of controllers Signed-off-by: Aylei <[email protected]> * lint Signed-off-by: Aylei <[email protected]> * Address review comments Signed-off-by: Aylei <[email protected]> --------- Signed-off-by: Aylei <[email protected]>

…g#4963) * add support for zsh * fix for bashrc after testing

Add kubernetes key

* [k8s] GKE support for TPU V6 * gke t6 support * remove wrong check

…ilot-org#4965) * fix * longer timeout

…fix flacky of test_job_queue_with_docker (skypilot-org#4955) * different param to different steps * longer time to sleep

* [API server] cleanup executor on shutdown Signed-off-by: Aylei <[email protected]> * refine Signed-off-by: Aylei <[email protected]> * just raise impossible exceptions Signed-off-by: Aylei <[email protected]> * Update sky/utils/subprocess_utils.py Co-authored-by: Zhanghao Wu <[email protected]> --------- Signed-off-by: Aylei <[email protected]> Co-authored-by: Zhanghao Wu <[email protected]>

…#4973) * LRU cache for can_create_new_instance_of_type * request scope

* backward compat * fix * backcompat update * generate pipeline * bug fix * remove deactivate * robust backcompact test * fix * more log * bug fix * subprocess run with bash * bug fix * update template * fix flaky * limit concurrency * pip install uv * fix * low resource * fix * bump python version to 3.10 * recreate env * import order

* independent storage check * formatting * granular perms * _is_storage_cloud_enabled uses storage check * UX improvement * remove debug logs * fix local test * sky check no regression * no sky check regression, managed jobs work * api backwards compatibility * define globally minimal perms for gcp * review feedback * continue from except

…equired (skypilot-org#4991) don't require tpu support for serve:gcp if tpu support is not required

* review comments * use .get() where it makes sense

…work (skypilot-org#4978) * [Serve] BugFix: `any_of` field order issue cause version bump to not work * upd

* initial code for batched inference * Refactor batch inference scripts and configuration files for improved consistency and clarity. Removed unused bucket name generation and monitoring service launch from `batch_compute_vectors.py`. Updated `compute_text_vectors.yaml` and `monitor_progress.yaml` to use a unified bucket name. Revised README to reflect changes in embedding generation focus and performance highlights. * formattting * Update batch inference configuration files for consistency. Renamed `compute_text_vectors` and `monitor_progress` to include `batch-inference` prefix. Revised README to enhance clarity on monitoring progress and accessing the service. * Update README for batch inference: replaced local image links with external URLs, corrected endpoint variable in monitoring instructions, and added a new image for enhanced visual representation. * Update README for batch inference: corrected image URLs to include file extensions and added a new section for further learning resources. * Enhance README for batch inference: updated the section on computing embeddings to include details about the Amazon reviews dataset and clarified the use of the `Alibaba-NLP/gte-Qwen2-7B-instruct` model for generating embeddings. * update banner

zpoint · 2025-03-20T10:19:36Z

/quicktest-core

cblmemo · 2025-03-20T17:07:19Z

Thanks @SalikovAlex for adding this! LGTM. Merging now :))

SalikovAlex added 4 commits January 16, 2025 16:43

Merge pull request #2 from skypilot-org/master

238f03d

sync

Merge branch 'skypilot-org:master' into master

c89b5fc

Merge remote-tracking branch 'origin/master'

d72537d

Merge remote-tracking branch 'origin/master'

afc6b5b

SalikovAlex marked this pull request as draft February 27, 2025 14:19

SalikovAlex marked this pull request as ready for review February 28, 2025 19:47

Michaelvll requested a review from cblmemo February 28, 2025 21:58

Merge remote-tracking branch 'origin/master'

9142ab3

cblmemo reviewed Mar 3, 2025

View reviewed changes

cblmemo reviewed Mar 11, 2025

View reviewed changes

cblmemo reviewed Mar 12, 2025

View reviewed changes

sky/data/storage.py Outdated Show resolved Hide resolved

tests/smoke_tests/test_mount_and_storage.py Outdated Show resolved Hide resolved

cblmemo reviewed Mar 13, 2025

View reviewed changes

cblmemo approved these changes Mar 19, 2025

View reviewed changes

SalikovAlex added 12 commits March 20, 2025 09:34

format

14f73af

fix

3e4a5ae

typo

97a1c58

Refactor Nebius adaptor and improve clarity.

7237234

Remove redundant code, streamline imports, and enhance error messaging. Adjust documentation for better accuracy and update function annotations. These changes improve maintainability and readability of the Nebius adaptor module.

Refactor Nebius storage setup and clean up debug print.

03ca228

Simplified Nebius AWS CLI installation by reusing the S3CloudStorage configuration for consistency. Removed unnecessary debug print in `run_upload_cli` to reduce console noise. Minor formatting adjustment in test YAML file.

Refactor subprocess call and improve timeout error messaging.

2e0a3bd

Removed unused variable from subprocess call to clean up code. Updated timeout error to include the bucket name for more detailed and helpful error reporting.

Set default region for Nebius Object Storage if none provided

561856c

Updated the helper method to assign a default region when no region is specified, ensuring compatibility with Nebius Object Storage. This change avoids potential errors caused by missing region values.

Support Nebius URLs in file sync commands

d412fa3

Replace 'nebius://' with 's3://' in source paths to ensure compatibility with AWS CLI commands. This allows seamless integration of Nebius storage endpoints.

KeplerC and others added 25 commits March 20, 2025 09:36

[Doc] Gemma3 doc update (skypilot-org#4948)

ca4e915

Update README and documentation to include Gemma 3 model and example. Added Gemma 3 to the news section in README.md and updated the models index in documentation.

[Test] Simplified buildkite agent queue (skypilot-org#4932)

a90f657

* remove serve and backcompact * ignore buildkite yaml file

[Docs] Update benefits for client-server (skypilot-org#4945)

f6216ec

* Update benefits for client-server * update * Update docs/source/reference/api-server/api-server.rst Co-authored-by: Zongheng Yang <[email protected]> --------- Co-authored-by: Zongheng Yang <[email protected]>

Fix flaky for test_cancel_launch_and_exec_async (skypilot-org#4966)

0d91fc4

* fix flaky for test_cancel_launch_and_exec_async * comma * use generic_cloud * new line format

[Docs] fix typo in gemma3 example (skypilot-org#4971)

040c776

Signed-off-by: Aylei <[email protected]>

fix incorrect vcpu/mem checks for GKE autoscaler (skypilot-org#4972)

96980a1

[UX] Fix dense cli for resources not enough (skypilot-org#4962)

08bf988

fix dense cli

[Test] Add support for missing bashrc file in zsh shells (skypilot-or…

6915045

…g#4963) * add support for zsh * fix for bashrc after testing

[Docs] Fix NFS mounting docs for k8s (skypilot-org#4951)

e16cc33

Add kubernetes key

[k8s] GKE support for TPU V6 (skypilot-org#4986)

cf1ca0c

* [k8s] GKE support for TPU V6 * gke t6 support * remove wrong check

Fix test_managed_jobs_storage failure on azure in master branch (skyp…

34728d4

…ilot-org#4965) * fix * longer timeout

[Test]Separate different param into different steps on buildkite and …

0614967

…fix flacky of test_job_queue_with_docker (skypilot-org#4955) * different param to different steps * longer time to sleep

[k8s] LRU cache for GKE can_create_new_instance_of_type (skypilot-org…

0fd9771

…#4973) * LRU cache for can_create_new_instance_of_type * request scope

[GCP] Don't require TPU support for serve:gcp if TPU support is not r…

4192d88

…equired (skypilot-org#4991) don't require tpu support for serve:gcp if tpu support is not required

followup to skypilot-org#4935 (skypilot-org#4989)

2ad110b

* review comments * use .get() where it makes sense

[Serve] BugFix: any_of field order issue cause version bump to not …

7c5c29e

…work (skypilot-org#4978) * [Serve] BugFix: `any_of` field order issue cause version bump to not work * upd

SalikovAlex force-pushed the nebiusstorage branch from 9454a41 to ed366c0 Compare March 20, 2025 09:37

Merge branch 'master' into nebiusstorage

b98b853

cblmemo merged commit f727408 into skypilot-org:master Mar 20, 2025
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Nebius] Nebius Object Storage support. #4838

[Nebius] Nebius Object Storage support. #4838

SalikovAlex commented Feb 27, 2025 •

edited

Loading

cblmemo left a comment

cblmemo left a comment

cblmemo Mar 11, 2025

SalikovAlex commented Mar 12, 2025

cblmemo left a comment

cblmemo Mar 13, 2025

SalikovAlex commented Mar 14, 2025 •

edited

Loading

cblmemo commented Mar 14, 2025

cblmemo commented Mar 14, 2025

cblmemo left a comment

zpoint commented Mar 20, 2025

zpoint commented Mar 20, 2025

cblmemo commented Mar 20, 2025

[Nebius] Nebius Object Storage support. #4838

[Nebius] Nebius Object Storage support. #4838

Conversation

SalikovAlex commented Feb 27, 2025 • edited Loading

cblmemo left a comment

Choose a reason for hiding this comment

cblmemo left a comment

Choose a reason for hiding this comment

cblmemo Mar 11, 2025

Choose a reason for hiding this comment

SalikovAlex commented Mar 12, 2025

cblmemo left a comment

Choose a reason for hiding this comment

cblmemo Mar 13, 2025

Choose a reason for hiding this comment

SalikovAlex commented Mar 14, 2025 • edited Loading

cblmemo commented Mar 14, 2025

cblmemo commented Mar 14, 2025

cblmemo left a comment

Choose a reason for hiding this comment

zpoint commented Mar 20, 2025

zpoint commented Mar 20, 2025

cblmemo commented Mar 20, 2025

SalikovAlex commented Feb 27, 2025 •

edited

Loading

SalikovAlex commented Mar 14, 2025 •

edited

Loading