Update wait_ready() behavior (or add a separate check for dashboard accessibility)

[Maxusmusti]

WHY

Currently, there is a gap between when the RayCluster itself is ready (and cluster.wait_ready() completes), and when the dashboard is accessible (happens a couple seconds later at most). This means that CI or scripts where job submission is performed immediately after wait_ready() has a small chance to fail, depending on how long the dashboard takes to be ready.

WHAT

Either wait_ready has to wait for the dashboard to be ready, or we can add a second function that waits for dashboard availability. Anything that can be used in CI/scripts to block until job submission is possible.

[ChristianZaccaria]

@Maxusmusti Hi Mustafa, I created a PR for this issue, however, I'm not entirely sure how to test it. I brought up the cluster with the wait_ready, where immediately afterwards a job gets submitted, but never seems to fail? Is there another way I should be testing it?

Thanks in advance!

[Maxusmusti]

Sorry, just saw this. I'll check out the PR, thanks!

[Maxusmusti]

@ChristianZaccaria the only other thing I would do is check in with @KPostOffice to make sure that this update will work alongside his dashboard authentication updates, and if not, sync on what changes need to be made on either end to make the two updates compatible

[ChristianZaccaria]

@ChristianZaccaria the only other thing I would do is check in with @KPostOffice to make sure that this update will work alongside his dashboard authentication updates, and if not, sync on what changes need to be made on either end to make the two updates compatible

Great call, I will sync in with him asap. Thanks @Maxusmusti !

[ChristianZaccaria]

@ChristianZaccaria the only other thing I would do is check in with @KPostOffice to make sure that this update will work alongside his dashboard authentication updates, and if not, sync on what changes need to be made on either end to make the two updates compatible

Great call, I will sync in with him asap. Thanks @Maxusmusti !

@Maxusmusti I just synced with Kevin, he says he will make the changes on his end to make it work with the changes here. Let me know if there is anything you would like me to change or improve on my PR.