Skip to content

updated buildkite pipeline generation #65574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 11, 2023
Merged

updated buildkite pipeline generation #65574

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .ci/generate-buildkite-pipeline-premerge
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,8 +239,8 @@ if [[ "${linux_projects}" != "" ]]; then
- label: ':linux: Linux x64'
artifact_paths:
- artifacts/**/*
- '*_result.json'
- 'build/monolithic-linux/test-results.xml'
- *_result.json
- build/test-results.xml
agents: ${LINUX_AGENTS}
retry:
automatic:
Expand All @@ -262,8 +262,8 @@ if [[ "${windows_projects}" != "" ]]; then
- label: ':windows: Windows x64'
artifact_paths:
- artifacts/**/*
- '*_result.json'
- 'build/monolithic-windows/test-results.xml'
- *_result.json
- build/test-results.xml
agents: ${WINDOWS_AGENTS}
retry:
automatic:
Expand Down
46 changes: 29 additions & 17 deletions .ci/generate-buildkite-pipeline-scheduled
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,18 @@
# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
#

set -eu
set -o pipefail

# Filter rules for generic windows tests
: ${WINDOWS_AGENTS:='{"queue": "windows"}'}
# Filter rules for generic linux tests
: ${LINUX_AGENTS:='{"queue": "linux"}'}
# Set by buildkite
: ${BUILDKITE_MESSAGE:=}
: ${BUILDKITE_COMMIT:=}
: ${BUILDKITE_BRANCH:=}

cat <<EOF
steps:
- trigger: "libcxx-ci"
Expand All @@ -28,42 +40,42 @@ steps:
commit: "${BUILDKITE_COMMIT}"
branch: "${BUILDKITE_BRANCH}"

- label: ':linux: x64 Debian'
- label: ':linux: Linux x64'
artifact_paths:
- '*_result.json'
- 'build/monolithic-linux/test-results.xml'
agents:
queue: 'linux'
- artifacts/**/*
- '*_result.json'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quoting is inconsistent, I believe all of these don't need quotes
(other files as well)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

- 'build/test-results.xml'
agents: ${LINUX_AGENTS}
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: 255
limit: 2 # Forced agent shutdown
- exit_status: 255 # Forced agent shutdown
limit: 2
timeout_in_minutes: 120
env:
CC: 'clang'
CXX: 'clang++'
commands:
- './.ci/monolithic-linux.sh "bolt;clang-tools-extra;compiler-rt;flang;libc;libclc;lld;llvm;mlir;polly;pstl" "check-all"'
- ./.ci/monolithic-linux.sh "bolt;clang;clang-tools-extra;compiler-rt;flang;libc;libclc;lld;llvm;mlir;polly;pstl" "check-all"

- label: ':windows: x64 Windows'
- label: ':windows: Windows x64'
artifact_paths:
- '*_result.json'
- 'build/monolithic-windows/test-results.xml'
agents:
queue: 'windows'
- artifacts/**/*
- '*_result.json'
- 'build/test-results.xml'
agents: ${WINDOWS_AGENTS}
retry:
automatic:
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: 255
limit: 2 # Forced agent shutdown
- exit_status: 255 # Forced agent shutdown
limit: 2
timeout_in_minutes: 150
env:
CC: 'cl'
CXX: 'cl'
LD: 'link'
commands:
- 'C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat -arch=amd64 -host_arch=amd64'
- 'bash .ci/monolithic-windows.sh "clang-tools-extra;flang;libclc;lld;llvm;mlir;polly;pstl" "check-all"'
- C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat -arch=amd64 -host_arch=amd64
- bash .ci/monolithic-windows.sh "clang;clang-tools-extra;flang;libclc;lld;llvm;mlir;polly;pstl" "check-all"
13 changes: 9 additions & 4 deletions .ci/monolithic-linux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,19 @@ set -ex
set -o pipefail

MONOREPO_ROOT="${MONOREPO_ROOT:="$(git rev-parse --show-toplevel)"}"
BUILD_DIR="${BUILD_DIR:=${MONOREPO_ROOT}/build/monolithic-linux}"

BUILD_DIR="${BUILD_DIR:=${MONOREPO_ROOT}/build}"
rm -rf ${BUILD_DIR}

ccache --zero-stats
ccache --show-config

if [[ -n "${CLEAR_CACHE:-}" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how can you set this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can do that when creating a build in buildkite UI or triggering through API with "environment" varaibles. It's not available for "normal" users

echo "clearing cache"
ccache --clear
fi

function show-stats {
ccache --print-stats
mkdir -p artifacts
ccache --print-stats > artifacts/ccache_stats.txt
}
trap show-stats EXIT

Expand Down
12 changes: 9 additions & 3 deletions .ci/monolithic-windows.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,19 @@ set -ex
set -o pipefail

MONOREPO_ROOT="${MONOREPO_ROOT:="$(git rev-parse --show-toplevel)"}"
BUILD_DIR="${BUILD_DIR:=${MONOREPO_ROOT}/build/monolithic-windows}"
BUILD_DIR="${BUILD_DIR:=${MONOREPO_ROOT}/build}"

rm -rf ${BUILD_DIR}

if [[ -n "${CLEAR_CACHE:-}" ]]; then
echo "clearing sccache"
rm -rf "$SCCACHE_DIR"
fi

sccache --zero-stats
function show-stats {
sccache --show-stats
mkdir -p artifacts
sccache --show-stats >> artifacts/sccache_stats.txt
}
trap show-stats EXIT

Expand All @@ -45,4 +51,4 @@ cmake -S ${MONOREPO_ROOT}/llvm -B ${BUILD_DIR} \
-D CMAKE_CXX_COMPILER_LAUNCHER=sccache

echo "--- ninja"
ninja -C ${BUILD_DIR} ${targets}
ninja -C "${BUILD_DIR}" "${targets}"