Skip to content

Commit 6e87745

Browse files
igoragoliigoragoli
authored
Improve and simplify the microbenchmarks CI setup (#7571)
- [x] Tackle all TODOs on `microbenchmarks.yml` before merging. - [ ] Merge [refactor: simplify dd-trace-dotnet microbenchmarks](DataDog/benchmarking-platform#200) before merging. ## Summary of changes - Add a dd-octo-sts policy allowing GitLab runners to access dd-trace-dotnet contents. - Improve and simplify microbenchmarks and macrobenchmarks CI setup. - Rename the AMI creation job for `build-dd-trace-dotnet-microbenchmarks-ami`. - Use `dd-octo-sts` for generating GitHub tokens. - Simplify and reduce the number of CI variables, standardizing on the `BP_INFRA_*` prefix. - Make instance cleanup conditional on a `CLEANUP` variable. This allows us to SSH/RDP into instances after benchmarks are run, if necessary. Related changes on benchmarking-platform: [refactor: simplify dd-trace-dotnet microbenchmarks](DataDog/benchmarking-platform#200) ## Reason for change https://datadoghq.atlassian.net/browse/APMSP-2282 and https://datadoghq.atlassian.net/browse/APMSP-1908. ## Implementation details ## Test coverage Benchmark run on the CI: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1154611844 ## Other details <!-- Fixes #{issue} --> <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. -->
1 parent 8a8fa99 commit 6e87745

File tree

3 files changed

+116
-68
lines changed

3 files changed

+116
-68
lines changed

‎.github/chainguard/gitlab.github-access.read-contents.sts.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
issuer: https://gitlab.ddbuild.io
2+
3+
subject_pattern: "project_path:DataDog/apm-reliability/dd-trace-dotnet:ref_type:(branch|tag):ref:.*"
4+
5+
permissions:
6+
contents: read

‎.gitlab/benchmarks/macrobenchmarks.yml‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -467,6 +467,13 @@ profiler_cpu_timer_create-arm64:
467467
- platform/artifacts/
468468
expire_in: 3 months
469469
variables:
470+
AWS_REGION: "us-east-1"
471+
472+
# Branch containing 1. scripts to launch Windows benchmarks on ephemeral
473+
# instances (to be used by GitLab CI runners) and 2. scripts to run Windows
474+
# benchmarks (to be used by the ephemeral instances).
475+
BP_INFRA_BENCHMARKING_PLATFORM_BRANCH: "dd-trace-dotnet/macro"
476+
470477
# Whether to cleanup ephemeral instances after benchmarks are run
471478
CLEANUP: "true"
472479

@@ -489,14 +496,17 @@ profiler_cpu_timer_create-arm64:
489496
script:
490497
- source build-id.txt
491498
- echo "Building for the following build https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=$buildId&view=results"
492-
- export BP_INFRA_BENCHMARKING_PLATFORM_BRANCH=dd-trace-dotnet/macro
493499
- git clone --branch $BP_INFRA_BENCHMARKING_PLATFORM_BRANCH https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform platform && cd platform
494500
- ./ephemeral-infra/run-windows-benchmarks.sh
495501
after_script:
496502
- |
497-
bp-infra cleanup --provision ./platform/ephemeral-infra/provisions/macrobenchmark-ephemeral-instance.yaml \
498-
--region "${AWS_REGION}" \
499-
--bypass-stack-destroy
503+
if [ "$CLEANUP" == "true" ]; then
504+
bp-infra cleanup --provision ./platform/ephemeral-infra/provisions/macrobenchmark-ephemeral-instance.yaml \
505+
--region "${AWS_REGION}" \
506+
--bypass-stack-destroy
507+
else
508+
echo "'CLEANUP' is set to 'false'. Will not cleanup."
509+
fi
500510
501511
baseline-win:
502512
extends: .benchmarks-win

‎.gitlab/benchmarks/microbenchmarks.yml‎

Lines changed: 96 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -1,101 +1,133 @@
1-
.setup:
2-
script:
3-
- mkdir -p ~/.aws
4-
- /app/bp-infra/tools/fetch-ssm-parameter.sh $AWS_EPHEMERAL_INFRA_PROFILE_SSM_PARAMETER > ~/.aws/config || exit $?
5-
- export AWS_PROFILE=ephemeral-infra-ci
6-
- export BP_INFRA_KEY_PAIR_NAME=$(cat ~/.aws/key-pair-name.txt)
7-
- export BP_INFRA_KEY_PAIR_PRIVATE_KEY_PATH=~/.aws/key-pair-private-key.pem
1+
.dd-octo-sts-setup:
2+
before_script:
3+
- |
4+
set +e
5+
error_output=$({ dd-octo-sts token --scope DataDog/dd-trace-dotnet --policy gitlab.github-access.read-contents > "/tmp/github-token"; } 2>&1)
6+
exit_code=$?
7+
if [ $exit_code -ne 0 ]; then
8+
echo "ERROR: Failed to retrieve a GitHub token with dd-octo-sts gitlab.github-access.read-contents policy."
9+
echo "Original error: $error_output"
10+
echo "Continuing execution anyway..."
11+
fi
12+
set -e
813
914
stages:
10-
- infra-update
15+
- build
1116
- benchmarks
1217

13-
update-bp-infra:
14-
stage: infra-update
18+
build-dd-trace-dotnet-microbenchmarks-ami:
19+
stage: build
20+
tags: ["arch:amd64"]
1521
timeout: 3h
16-
tags: ["arch:amd64"]
1722
allow_failure: true
18-
# Image created in the following job https://gitlab.ddbuild.io/DataDog/benchmarking-platform-tools/-/jobs/869830045
23+
when: manual
1924
image: registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:dd-trace-dotnet-micro
25+
id_tokens:
26+
DDOCTOSTS_ID_TOKEN:
27+
aud: dd-octo-sts
28+
variables:
29+
AWS_REGION: "us-east-1"
30+
31+
# Branch containing a provision for building the AMI
32+
BP_INFRA_BENCHMARKING_PLATFORM_BRANCH: "dd-trace-dotnet/micro"
33+
34+
PROVISION_FILE: "platform/ephemeral-infra/ami.yaml"
2035

36+
# Where AMI creation artifacts will be stored
37+
BP_INFRA_ARTIFACTS_BUCKET_NAME: "windows-benchmarking-results-us-east-1"
38+
39+
# Whether to cleanup instances after building the AMI, since the AMI is
40+
# based on an instance that is created in this job
41+
CLEANUP: "true"
42+
before_script:
43+
- !reference [.dd-octo-sts-setup, before_script]
2144
script:
22-
- git clone --branch dd-trace-dotnet/micro https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform platform
23-
- mkdir -p ~/.aws
24-
- /app/bp-infra/tools/fetch-ssm-parameter.sh $AWS_EPHEMERAL_INFRA_PROFILE_SSM_PARAMETER >> ~/.aws/config || exit $?
25-
- aws ssm get-parameter --region "$AWS_REGION" --name "ci.${CI_PROJECT_NAME}.ephemeral-infra-ci.windows-benchmarking-key-pair-name" --with-decryption --query "Parameter.Value" --out text >> ~/.aws/key-pair-name.txt
26-
- aws ssm get-parameter --region "$AWS_REGION" --name "ci.${CI_PROJECT_NAME}.ephemeral-infra-ci.windows-benchmarking-key-private-key" --with-decryption --query "Parameter.Value" --out text >> ~/.aws/key-pair-private-key.pem
27-
- export AWS_PROFILE=ephemeral-infra-ci
28-
- export BP_INFRA_KEY_PAIR_NAME=$(cat ~/.aws/key-pair-name.txt)
29-
- export BP_INFRA_KEY_PAIR_PRIVATE_KEY_PATH=~/.aws/key-pair-private-key.pem
30-
- bp-infra launch --provision ./platform/ephemeral-infra/base-instance.yaml --region "${AWS_REGION}" --bypass-stack-destroy
45+
- git clone --branch $BP_INFRA_BENCHMARKING_PLATFORM_BRANCH https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform platform
46+
- echo "GITHUB_TOKEN=$(cat /tmp/github-token)" > .env
47+
- CLEANUP_ARG=$([[ "$CLEANUP" == "false" ]] && echo "--no-cleanup" || echo "")
48+
- |
49+
bp-infra launch --region "${AWS_REGION}" --os "windows" \
50+
--provision "${PROVISION_FILE}" \
51+
--bypass-stack-destroy \
52+
--env .env \
53+
$CLEANUP_ARG
3154
after_script:
32-
- !reference [.setup, script]
55+
# Makes sure the instance is cleaned up.
56+
# Note: This does not clean up the created AMI.
3357
- |
34-
bp-infra cleanup --provision ./platform/ephemeral-infra/base-instance.yaml \
35-
--region "${AWS_REGION}" \
36-
--bypass-stack-destroy
37-
38-
rules:
39-
- when: manual
40-
variables:
41-
AWS_REGION: "us-east-1"
42-
CLEANUP: "false"
43-
AWS_EPHEMERAL_INFRA_PROFILE_SSM_PARAMETER: "ci.dd-trace-dotnet.ephemeral-infra-ci.dd-trace-dotnet-profile"
44-
AWS_EPHEMERAL_INFRA_PROFILE_NAME: "ephemeral-infra-ci"
45-
AWS_EPHEMERAL_INFRA_ARTIFACTS_BUCKET_URI: "s3://windows-benchmarking-results/$CI_PROJECT_NAME/$CI_COMMIT_REF_NAME/$CI_JOB_ID"
46-
AWS_EPHEMERAL_INFRA_REGION: "us-east-1"
58+
if [ "$CLEANUP" == "true" ]; then
59+
bp-infra cleanup --region "${AWS_REGION}" --os "windows" \
60+
--provision "${PROVISION_FILE}" \
61+
--bypass-stack-destroy
62+
else
63+
echo "'CLEANUP' is set to 'false'. Will not cleanup."
64+
fi
4765
4866
run-benchmarks:
4967
stage: benchmarks
5068
tags: ["arch:amd64"]
5169
timeout: 2h
5270
# Image created in the following job https://gitlab.ddbuild.io/DataDog/benchmarking-platform-tools/-/jobs/869830045
5371
image: registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:dd-trace-dotnet-micro
72+
id_tokens:
73+
DDOCTOSTS_ID_TOKEN:
74+
aud: dd-octo-sts
75+
rules:
76+
- when: on_success
77+
variables:
78+
AWS_REGION: "us-east-1"
79+
80+
# Branch containing 1. scripts to launch Windows benchmarks on ephemeral
81+
# instances (to be used by GitLab CI runners) and 2. scripts to run Windows
82+
# benchmarks (to be used by the ephemeral instances).
83+
BP_INFRA_BENCHMARKING_PLATFORM_BRANCH: "dd-trace-dotnet/micro"
5484

85+
# Where benchmarking results will be stored
86+
BP_INFRA_ARTIFACTS_BUCKET_NAME: "windows-benchmarking-results-us-east-1"
87+
88+
# Whether to cleanup ephemeral instances after benchmarks are run
89+
CLEANUP: "true"
90+
91+
before_script:
92+
- !reference [.dd-octo-sts-setup, before_script]
5593
script:
56-
- git clone --branch dd-trace-dotnet/micro https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform platform
57-
- AWS_REGION=${AWS_REGION} ./platform/steps/launch-instance.sh
94+
- git clone --branch $BP_INFRA_BENCHMARKING_PLATFORM_BRANCH https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform platform
95+
- ./platform/steps/run-windows-benchmarks.sh
5896
after_script:
97+
# Future improvement: Ideally, should be in a script.
5998
- |
60-
bp-infra cleanup --provision ./platform/ephemeral-infra/ephemeral-instance-main.yaml \
61-
--region "${AWS_REGION}" \
62-
--bypass-stack-destroy
99+
if [ "$CLEANUP" == "true" ]; then
100+
bp-infra cleanup --provision ./platform/ephemeral-infra/instance.yaml \
101+
--region "${AWS_REGION}" \
102+
--bypass-stack-destroy
103+
else
104+
echo "'CLEANUP' is set to 'false'. Will not cleanup."
105+
fi
63106
- ./platform/steps/post-pr-comment.sh
64-
# Temporarily commented out pending issue resolution with sending files to backend
107+
# TODO: Uncomment this when the issue with sending files to backend is resolved
65108
# - ./platform/steps/upload-to-bp-ui.sh
66109

67-
rules:
68-
- when: on_success
69-
variables:
70-
AWS_REGION: "us-east-1"
71-
110+
# TODO: Remove if unnecessary
72111
upload-to-bp-ui:
73112
stage: benchmarks
74113
tags: ["arch:amd64"]
75114
timeout: 1h
76-
# Image created in the following job https://gitlab.ddbuild.io/DataDog/benchmarking-platform-tools/-/jobs/869830045
77-
image: registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:dotnet-microbenchmarks
78-
79-
script:
80-
- git clone --branch fayssal/test-micro-delivery https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform platform
81-
# - ./platform/steps/launch-instance.sh
82-
# - ./platform/steps/post-pr-comment.sh
83-
# Temporarely commented out pending issue resolution with sending files to backend
84-
- ./platform/steps/upload-to-bp-ui.sh
85-
115+
image: registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:dd-trace-dotnet-micro
116+
when: manual
86117
artifacts:
87118
name: "artifacts"
88119
when: always
89120
paths:
90121
- candidate-results/
91122
expire_in: 3 months
92-
93-
rules:
94-
- when: manual
95123
variables:
96124
AWS_REGION: "us-east-1"
97-
CLEANUP: "false"
98-
AWS_EPHEMERAL_INFRA_PROFILE_SSM_PARAMETER: "ci.dd-trace-dotnet.ephemeral-infra-ci.dd-trace-dotnet-profile"
99-
AWS_EPHEMERAL_INFRA_PROFILE_NAME: "ephemeral-infra-ci"
100-
AWS_EPHEMERAL_INFRA_ARTIFACTS_BUCKET_URI: "s3://windows-benchmarking-results/$CI_PROJECT_NAME/$CI_COMMIT_REF_NAME/$CI_JOB_ID"
101-
AWS_EPHEMERAL_INFRA_REGION: "us-east-1"
125+
126+
# Here, we don't include BP_INFRA since we don't use BP_INFRA in this job.
127+
BENCHMARKING_PLATFORM_BRANCH: "dd-trace-dotnet/micro"
128+
129+
# Where to fetch results from
130+
BP_INFRA_ARTIFACTS_BUCKET_NAME: "windows-benchmarking-results-us-east-1"
131+
script:
132+
- git clone --branch $BENCHMARKING_PLATFORM_BRANCH https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform platform
133+
- ./platform/steps/upload-to-bp-ui.sh

0 commit comments

Comments
 (0)