[GCP] Add retry for transient error during launching GCP clusters #2669
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Michaelvll
changed the title
concretevitamin
approved these changes
Oct 6, 2023
| if 'Head node fetch timed out' in stderr: | ||
| # Example: click.exceptions.ClickException: Head node fetch | ||
| # timed out. Failed to create head node. | ||
| # This is a transient error, but we have retried in need_ray_up |
There was a problem hiding this comment.
nit/Q: is this a transient error from the cloud, or due to certain image sizes being too large (which then makes the timeout in ray flaky)?
There was a problem hiding this comment.
It is likely the issue is with the cloud, as ray will get through the waiting loop, once the VM goes into PROVISIONING state and has the tag set, while the image setup should probably happen in the next state STAGING. https://cloud.google.com/compute/docs/instances/instance-life-cycle
I have seen multiple times when I sky launch --gpus A100:8, our program gets the IP address for the VM, while the VM just got deleted by GCP on the console.
concretevitamin
approved these changes
Oct 6, 2023
| if 'Head node fetch timed out' in stderr: | ||
| # Example: click.exceptions.ClickException: Head node fetch | ||
| # timed out. Failed to create head node. | ||
| # This is a transient error, but we have retried in need_ray_up |
jc9123
pushed a commit
to jc9123/skypilot
that referenced
this pull request
Oct 11, 2023
…ypilot-org#2669) * Add retry for flaky error during launching GCP clusters * handle error * format * Do not log out stderr * Add retry for gcloud crash * fix retry return code
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GCP generats some flaky error while launching causing #2666. We should add retry for the
ray upduring launching as well as the blocklist update for errors.Tested (run the relevant ones):
bash format.shsky launch --cloud gcp --gpus A100:8 --image-id projects/deeplearning-platform-release/global/images/pytorch-2-0-gpu-v20230822-ubuntu-2004-py310 --down --retry-until-uppytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_namebash tests/backward_comaptibility_tests.sh