Skip to content

Add batch pipeline to GitHub actions workflow #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brightsparc merged 10 commits into from
Sep 30, 2021
Merged

Add batch pipeline to GitHub actions workflow #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
773a6a2
Create lambda on fly in batch pipline, and change actions to pass rol…
brightsparc Sep 30, 2021
decc718
Update github actions to allow read-only sagemaker permissions. Adde…
brightsparc Sep 30, 2021
b75ee65
Default env at workflow instead of secret, set role based on creds
brightsparc Sep 30, 2021
2440b00
Update to pass function arn that is created in CFN
brightsparc Sep 30, 2021
1238ba9
Adding lambda permissions, and enable batch prod, change build pipeli…
brightsparc Sep 30, 2021
02f49d1
Update SC stack to add create lambda permissions for SM functions
brightsparc Sep 30, 2021
7700a46
Move lambda inline to SM pipeline
brightsparc Sep 30, 2021
6dffc23
Update to run batch prod in main branch only
brightsparc Sep 30, 2021
ea2bfb0
Update to only do on push or pull request for main
brightsparc Sep 30, 2021
7d446f3
Renaming pipeline name to get around hung SM pipeline
brightsparc Sep 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
226 changes: 172 additions & 54 deletions .github/workflows/build-deploy-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,22 @@

name: Build and Deploy

on: [push, pull_request] # Optionally filter on branch
on:
# Trigger the workflow on push or pull request,
# but only for the main branch
push:
branches:
- main
pull_request:
branches:
- main
release:
types:
- created

env:
AWS_REGION: us-east-1
PROJECT_NAME: ${{ github.event.repository.name }}

jobs:
build:
Expand All @@ -20,25 +35,15 @@ jobs:
- name: Checkout
uses: actions/checkout@v2

- name: Config Environment
id: env-name
env:
PROJECT_NAME: ${{ github.event.repository.name }}
run: |
echo "Project name: $PROJECT_NAME"
echo "::set-output name=project_name::$PROJECT_NAME"

- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: "3.8"
architecture: "x64"

- name: Setup Node
uses: actions/setup-node@v2
with:
node-version: "12"
architecture: "x64"
cache: npm

- name: Install Requirements
Expand All @@ -52,18 +57,16 @@ jobs:
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}
role-to-assume: ${{ secrets.AWS_SAGEMAKER_ROLE }}
aws-region: ${{ env.AWS_REGION }}
role-duration-seconds: 1200

- name: Build Pipeline
id: build-pipeline
env:
SAGEMAKER_PROJECT_NAME: ${{ steps.env-name.outputs.project_name }}
SAGEMAKER_PIPELINE_NAME: ${{ steps.env-name.outputs.project_name }}-pipeline
SAGEMAKER_PIPELINE_DESCRIPTION: "SageMaker pipeline created from GitHub actions"
SAGEMAKER_PIPELINE_ROLE_ARN: ${{ secrets.AWS_SAGEMAKER_ROLE }}
AWS_REGION: ${{ secrets.AWS_REGION }}
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
SAGEMAKER_PIPELINE_NAME: ${{ env.PROJECT_NAME }}-pipeline
SAGEMAKER_PIPELINE_DESCRIPTION: "Drift detection model build pipeline created from GitHub actions"
SAGEMAKER_PIPELINE_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
run: |
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
echo "Project id: $SAGEMAKER_PROJECT_ID"
Expand All @@ -76,15 +79,14 @@ jobs:
run: cat drift-pipeline.yml

- name: Create CFN Pipeline
id: deploy-pipeline
uses: aws-actions/aws-cloudformation-github-deploy@v1
with:
name: sagemaker-${{ steps.build-pipeline.outputs.pipeline_name }}
name: sagemaker-${{ env.PROJECT_NAME }}-pipeline
template: ./build_pipeline/drift-pipeline.yml # Need to specify working-directory
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
no-fail-on-empty-changeset: "1"

- name: Start Pipeline
id: start-pipeline # TODO: Run python code that waits for pipeline to complete
run: aws sagemaker start-pipeline-execution --pipeline-name ${{ steps.build-pipeline.outputs.pipeline_name }} --pipeline-parameters Name=InputSource,Value=GitHubAction#${{ github.run_number }}

- name: Upload template
Expand All @@ -93,6 +95,145 @@ jobs:
name: drift-pipeline
path: ./build_pipeline/drift-pipeline.yml

batch_staging:
needs: build
name: Batch to staging
runs-on: ubuntu-latest
environment:
name: batch-staging
defaults:
run:
shell: bash
working-directory: ./batch_pipeline
steps:
- name: Checkout
uses: actions/checkout@v2

- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: "3.8"

- name: Setup Node
uses: actions/setup-node@v2
with:
node-version: "12"
cache: npm

- name: Install Requirements
run: |
npm install -g aws-cdk # Install cdk
pip install --requirement requirements.txt

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
id: creds
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
role-duration-seconds: 1200

- name: Build Templates
id: build-templates
env:
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
SAGEMAKER_PIPELINE_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
LAMBDA_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
run: |
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
echo "Project id: $SAGEMAKER_PROJECT_ID"
export ARTIFACT_BUCKET=sagemaker-project-$SAGEMAKER_PROJECT_ID-$AWS_REGION
echo "Artifact Bucket: $ARTIFACT_BUCKET"
npx cdk synth drift-batch-staging --path-metadata false --asset-metadata=false > drift-batch-staging.yml

- name: Print template
run: cat drift-batch-staging.yml

- name: Deploy Staging
uses: aws-actions/aws-cloudformation-github-deploy@v1
with:
name: sagemaker-${{ env.PROJECT_NAME }}-batch-staging
template: ./batch_pipeline/drift-batch-staging.yml # Need to specify working-directory
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
no-fail-on-empty-changeset: "1"

- name: Upload template
uses: actions/upload-artifact@v2
with:
name: drift-batch-staging
path: ./batch_pipeline/drift-batch-staging.yml

batch_prod:
needs: batch_staging
name: Batch to prod
if: ${{ github.ref == 'refs/heads/main' }} # Filter to only run on main branch
runs-on: ubuntu-latest
environment:
name: batch-prod
defaults:
run:
shell: bash
working-directory: ./batch_pipeline
steps:
- name: Checkout
uses: actions/checkout@v2

- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: "3.8"

- name: Setup Node
uses: actions/setup-node@v2
with:
node-version: "12"
cache: npm

- name: Install Requirements
run: |
npm install -g aws-cdk # Install cdk
pip install --requirement requirements.txt

- name: Configure AWS Credentials
id: creds
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
role-duration-seconds: 1200

- name: Build Templates
id: build-templates
env:
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
SAGEMAKER_PIPELINE_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
LAMBDA_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
run: |
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
echo "Project id: $SAGEMAKER_PROJECT_ID"
export ARTIFACT_BUCKET=sagemaker-project-$SAGEMAKER_PROJECT_ID-$AWS_REGION
echo "Artifact Bucket: $ARTIFACT_BUCKET"
npx cdk synth drift-batch-prod --path-metadata false --asset-metadata=false > drift-batch-prod.yml

- name: Print Template
run: cat drift-batch-prod.yml

- name: Deploy Prod
uses: aws-actions/aws-cloudformation-github-deploy@v1
with:
name: sagemaker-${{ env.PROJECT_NAME }}-batch-prod
template: ./batch_pipeline/drift-batch-prod.yml # Need to specify working-directory
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
no-fail-on-empty-changeset: "1"

- name: Upload template
uses: actions/upload-artifact@v2
with:
name: drift-batch-prod
path: ./batch_pipeline/drift-batch-prod.yml

deploy_staging:
needs: build
name: Deploy to staging
Expand All @@ -107,25 +248,15 @@ jobs:
- name: Checkout
uses: actions/checkout@v2

- name: Config Environment
id: env-name
env:
PROJECT_NAME: ${{ github.event.repository.name }}
run: |
echo "Project name: $PROJECT_NAME"
echo "::set-output name=project_name::$PROJECT_NAME"

- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: "3.8"
architecture: "x64"

- name: Setup Node
uses: actions/setup-node@v2
with:
node-version: "12"
architecture: "x64"
cache: npm

- name: Install Requirements
Expand All @@ -139,16 +270,14 @@ jobs:
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}
role-to-assume: ${{ secrets.AWS_SAGEMAKER_ROLE }}
aws-region: ${{ env.AWS_REGION }}
role-duration-seconds: 1200

- name: Build Templates
id: build-templates
env:
SAGEMAKER_PROJECT_NAME: ${{ steps.env-name.outputs.project_name }}
SAGEMAKER_EXECUTION_ROLE_ARN: ${{ secrets.AWS_SAGEMAKER_ROLE }}
AWS_REGION: ${{ secrets.AWS_REGION }}
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
SAGEMAKER_EXECUTION_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
run: |
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
echo "Project id: $SAGEMAKER_PROJECT_ID"
Expand All @@ -160,11 +289,11 @@ jobs:
run: cat drift-deploy-staging.yml

- name: Deploy Staging
id: deploy-pipeline
uses: aws-actions/aws-cloudformation-github-deploy@v1
with:
name: sagemaker-${{ steps.env-name.outputs.project_name }}-deploy-staging
name: sagemaker-${{ env.PROJECT_NAME }}-deploy-staging
template: ./deployment_pipeline/drift-deploy-staging.yml # Need to specify working-directory
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
no-fail-on-empty-changeset: "1"

- name: Upload template
Expand All @@ -188,25 +317,15 @@ jobs:
- name: Checkout
uses: actions/checkout@v2

- name: Config Environment
id: env-name
env:
PROJECT_NAME: ${{ github.event.repository.name }}
run: |
echo "Project name: $PROJECT_NAME"
echo "::set-output name=project_name::$PROJECT_NAME"

- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: "3.8"
architecture: "x64"

- name: Setup Node
uses: actions/setup-node@v2
with:
node-version: "12"
architecture: "x64"
cache: npm

- name: Install Requirements
Expand All @@ -220,16 +339,14 @@ jobs:
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}
role-to-assume: ${{ secrets.AWS_SAGEMAKER_ROLE }}
aws-region: ${{ env.AWS_REGION }}
role-duration-seconds: 1200

- name: Build Templates
id: build-templates
env:
SAGEMAKER_PROJECT_NAME: ${{ steps.env-name.outputs.project_name }}
SAGEMAKER_EXECUTION_ROLE_ARN: ${{ secrets.AWS_SAGEMAKER_ROLE }}
AWS_REGION: ${{ secrets.AWS_REGION }}
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
SAGEMAKER_EXECUTION_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
run: |
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
echo "Project id: $SAGEMAKER_PROJECT_ID"
Expand All @@ -244,8 +361,9 @@ jobs:
id: deploy-pipeline
uses: aws-actions/aws-cloudformation-github-deploy@v1
with:
name: sagemaker-${{ steps.env-name.outputs.project_name }}-deploy-prod
name: sagemaker-${{ env.PROJECT_NAME }}-deploy-prod
template: ./deployment_pipeline/drift-deploy-prod.yml # Need to specify working-directory
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
no-fail-on-empty-changeset: "1"

- name: Upload template
Expand Down
31 changes: 20 additions & 11 deletions ACTIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,28 @@ aws secretsmanager get-secret-value \
--output text
```

The GitHub Actions workflow has three stages: Model build, Deploy to staging, and Deploy to production. Each has their on [environment](https://docs.github.com/en/actions/reference/environments) which secrets and optional protection rules.
1. `development` This is the environment in which runs your Model Build and starts the SageMaker pipeline execution, and on completion it will publish a model to the Registry. It is recommend you run this on `pull_request` and `push`.
1. `staging` This second stage deploys your Staging endpoint. It is recommend you run this on commit to the `development` branch and configure a *protection rule* to continue after you have approved the model in the SageMaker Model Registry.
1. `prod` This final stage deploys you Production Endpoint. It is recommend you this on commit the `main` branch with a *protection rule* to require approval after Staging endpoint has been tested.
### Jobs

For each of the environments you will require setting up the following secrets.
1. Create a secret named `AWS_REGION` defaulted to region `us-east-1`
1. Create a secret named `AWS_ACCESS_KEY_ID` containing the `AccessKeyId` value returned above.
1. Create a secret named `AWS_SECRET_ACCESS_KEY` containing in the `SecretAccessKey` value returned above.
1. Create a secret named `AWS_SAGEMAKER_ROLE` containing the `SageMakerRoleArn` output in the setup stack.
This sample includes a `Build and Deploy` [GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions) workflow with contains the following jobs
1. The `Build Model` job will create or update a SageMaker Model Build Pipeline using AWS CloudFormation.
2. The `Batch` jobs will create or update a SageMaker Pipeline to run Batch Scoring for Staging and Production environments.
3. The `Deploy` jobs will deploy SageMaker Endpoints to Staging and Production environments.

If you configure *protection rules* for you environments, you will need to click **Review deployments** to approve the next stage as show below:
These jobs are configured to run against a specific [environment](https://docs.github.com/en/actions/reference/environments) which contains both secrets and optional *protection rules*.

![Execution Role](docs/github-actions-workflow.png)

When the workflow successfully completes, you will have both Endpoints deployed in staging and production, with drift detection enable which will trigger re-training on drift.
### Environments

1. `development` environment in which runs your `Build Model` job and starts the SageMaker pipeline execution. On completion this pipeline will publish a model to the Registry. It is recommend you run this on `pull_request` and `push` events.
2. `staging` and `batch-staging` environments will enable you to run the `Batch` and `Deploy` jobs respectively in staging.
* You should configure a *protection rule* for data science team so this job is delayed until the latest model has been approved in the SageMaker model registry.
3. `prod` and `batch-prod` environments will enable you to run the `Batch` and `Deploy` jobs respectively in production.
* You should configure a *protection rule* for your operations team which will approve this only once they are happy that the staging environment has been tested.

For each of the environments you will require setting up the following secrets.
1. Create a secret named `AWS_ACCESS_KEY_ID` containing the `AccessKeyId` value returned above.
1. Create a secret named `AWS_SECRET_ACCESS_KEY` containing in the `SecretAccessKey` value returned above.
1. Create a secret named `AWS_SAGEMAKER_ROLE` containing the ARN for the `AmazonSageMakerServiceCatalogProductsUseRole` in your account.

When the workflow successfully completes, drift detection is configured to trigger re-training on drift detection in the production batch pipeline or real-time endpoint.
Loading