Skip to content

Create shallow canary marker #240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 13, 2023
Merged

Create shallow canary marker #240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions test/canary/scripts/run_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ pushd $E2E_DIR
if [[ $SERVICE_REGION =~ ^(eu-north-1|eu-west-3)$ ]]; then
# If select_regions_1 true we run the notebook_instance test
pytest_args+=(-m "canary or select_regions_1")
elif [[ $SERVICE_REGION =~ ^(eu-south-2|ap-southeast-3|me-central-1|eu-central-2)$ ]]; then
pytest_args+=(-m "shallow_canary")
else
pytest_args+=(-m "canary")
fi
Expand Down
1 change: 1 addition & 0 deletions test/e2e/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ def pytest_configure(config):
)
config.addinivalue_line("markers", "slow: mark test as slow to run")
config.addinivalue_line("markers", "select_regions_1: mark test to only run if in select region")
config.addinivalue_line("markers", "shallow_canary: mark test to run in shallow canary tests")


def pytest_collection_modifyitems(config, items):
Expand Down
13 changes: 13 additions & 0 deletions test/e2e/replacement_values.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"ap-southeast-3": "951798379941.dkr.ecr.ap-southeast-3.amazonaws.com",
"me-central-1": "272398656194.dkr.ecr.me-central-1.amazonaws.com",
"eu-central-2": "680994064768.dkr.ecr.eu-central-2.amazonaws.com",
"af-south-1": "510948584623.dkr.ecr.af-south-1.amazonaws.com",
}

# https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/image_uri_config/debugger.json
Expand All @@ -73,6 +74,11 @@
"sa-east-1": "818342061345.dkr.ecr.sa-east-1.amazonaws.com",
"eu-south-1": "563282790590.dkr.ecr.eu-south-1.amazonaws.com",
"ap-northeast-3": "479947661362.dkr.ecr.ap-northeast-3.amazonaws.com",
"af-south-1": "314341159256.dkr.ecr.af-south-1.amazonaws.com",
"eu-south-2": "",
"ap-southeast-3": "",
"me-central-1": "",
"eu-central-2": "",
}

# https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html
Expand Down Expand Up @@ -103,6 +109,7 @@
"ap-southeast-3": "951798379941.dkr.ecr.ap-southeast-3.amazonaws.com",
"me-central-1": "272398656194.dkr.ecr.me-central-1.amazonaws.com",
"eu-central-2": "680994064768.dkr.ecr.eu-central-2.amazonaws.com",
"af-south-1": "455444449433.dkr.ecr.af-south-1.amazonaws.com",
}


Expand Down Expand Up @@ -163,6 +170,9 @@
"us-gov-west-1": "362178532790.dkr.ecr.us-gov-west-1.amazonaws.com",
"ap-northeast-3": "990339680094.dkr.ecr.ap-northeast-3.amazonaws.com",
"ap-southeast-3": "669540362728.dkr.ecr.ap-southeast-3.amazonaws.com",
"eu-south-2": "",
"me-central-1": "",
"eu-central-2": "",
}

# https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-configure-processing-jobs.html#clarify-processing-job-configure-container
Expand All @@ -189,6 +199,9 @@
"eu-south-1": "638885417683.dkr.ecr.eu-south-1.amazonaws.com",
"ap-northeast-3": "912233562940.dkr.ecr.ap-northeast-3.amazonaws.com",
"ap-southeast-3": "705930551576.dkr.ecr.ap-southeast-3.amazonaws.com",
"eu-south-2": "",
"me-central-1": "",
"eu-central-2": "",
}

ENDPOINT_INSTANCE_TYPES = {
Expand Down
1 change: 1 addition & 0 deletions test/e2e/tests/test_endpoint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ def faulty_config(name_suffix, single_container_model):


@service_marker
@pytest.mark.shallow_canary
Copy link
Member

@ryansteakley ryansteakley Apr 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This marker means if we run pytest with this marker then run this test. In your shell-script you are still running these tests with the regions in this ^(eu-south-2|ap-southeast-3|me-central-1|eu-central-2) Correct if I'm wrong but isn't the goal of this pr to not run these tests in those regions since image is missing?

Copy link
Contributor Author

@rd-pong rd-pong Apr 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to run shallow canary, which includes training and hosting, in eu-south-2, ap-southeast-3, me-central-1, eu-central-2. Training and hosting does not require missing images such as xgboost, debugger and clarify.

@pytest.mark.canary
class TestEndpoint:
def create_endpoint_test(self, xgboost_endpoint):
Expand Down
1 change: 1 addition & 0 deletions test/e2e/tests/test_endpoint_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ def single_variant_config():


@service_marker
@pytest.mark.shallow_canary
@pytest.mark.canary
class TestEndpointConfig:
def test_create_endpoint_config(self, single_variant_config):
Expand Down
1 change: 1 addition & 0 deletions test/e2e/tests/test_trainingjob.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ def test_stopped(self, xgboost_training_job):
training_job_desc = get_sagemaker_training_job(training_job_name)
assert training_job_desc["TrainingJobStatus"] in cfg.LIST_JOB_STATUS_STOPPED

@pytest.mark.shallow_canary
@pytest.mark.canary
def test_completed(self, xgboost_training_job):
(reference, resource) = xgboost_training_job
Expand Down