updated buildkite pipeline generation #65574

metaflow · 2023-09-07T06:44:44Z

fixed build for linux (clang was missing)
removed /monolithic-.. from build directory - it does not add anything and makes path longer for windows which is not great;
added env-based configuration to control cache and agent targeting;
print (s)ccache stats to file not to pullute normal log.

aeubanks

the commit title doesn't make sense, can you make it something like "Various fixes to buildkite pipeline generation"

aeubanks · 2023-09-07T20:42:29Z

.ci/monolithic-linux.sh

 rm -rf ${BUILD_DIR}

 ccache --zero-stats
-ccache --show-config
+
+if [[ -n "${CLEAR_CACHE:-}" ]]; then


how can you set this?

you can do that when creating a build in buildkite UI or triggering through API with "environment" varaibles. It's not available for "normal" users

aeubanks · 2023-09-07T20:43:00Z

.ci/generate-buildkite-pipeline-scheduled

-      - 'C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat -arch=amd64 -host_arch=amd64'
-      - 'bash .ci/monolithic-windows.sh "clang-tools-extra;flang;libclc;lld;llvm;mlir;polly;pstl" "check-all"'
+    - C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat -arch=amd64 -host_arch=amd64
+    - bash .ci/monolithic-windows.sh "clang-tools-extra;flang;libclc;lld;llvm;mlir;polly;pstl;clang" "check-all"


make alphabetical just to be consistent and look nice

aeubanks · 2023-09-07T20:45:02Z

.ci/generate-buildkite-pipeline-scheduled

-    agents:
-      queue: 'linux'
+    - artifacts/**/*
+    - '*_result.json'


quoting is inconsistent, I believe all of these don't need quotes
(other files as well)

- fixed build for linux (clang was missing) - removed /monolithic-.. from build directory - it does not add anything and makes path longer for windows which is not great; - added env-based configuration to control cache and agent targeting; - print (s)ccache stats to file not to pullute normal log.

metaflow · 2023-09-11T11:20:30Z

Test build https://buildkite.com/llvm-project/experimental-premerge-checks/builds/60

joker-eph · 2023-09-12T09:16:35Z

This seems broken: https://buildkite.com/llvm-project/github-pull-requests/builds/1350#018a8848-f70e-47f4-8aec-0bb48c49a7ca

$ trap 'kill -- $' INT TERM QUIT; .ci/generate-buildkite-pipeline-premerge \| buildkite-agent pipeline upload
  | 2023-09-12 07:26:50 INFO   Reading pipeline config from STDIN
  | 2023-09-12 07:26:50 INFO   Updating BUILDKITE_COMMIT to "6de56a7dc4ff114eb36f42f1dbe8c63e557f061b"
  | Files modified:
  | mlir/include/mlir/Dialect/Affine/IR/AffineOps.h
  | mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
  | mlir/include/mlir/IR/SymbolInterfaces.td
  | mlir/lib/Dialect/Affine/IR/AffineOps.cpp
  | mlir/test/Dialect/Affine/invalid.mlir
  | mlir/test/Dialect/Affine/load-store-invalid.mlir
  | Directories modified:
  | mlir
  | 2023-09-12 07:26:50 FATAL  Pipeline parsing of "(stdin)" failed: line 5: did not find expected alphabetic or numeric character
  | 🚨 Error: The command exited with status 1

This seems to match the - '*_result.json' replaced with - *_result.json

metaflow · 2023-09-12T10:17:59Z

thanks for the heads up @joker-eph ! That should be fixed now but you need to rebase your PR to see the change. I guess noop commit update should do

metaflow · 2023-09-12T10:18:57Z

ah you've merged it already 😄

ldionne · 2023-09-12T11:51:01Z

@metaflow The reason for not adding clang to the monolithic pipeline is that clang now has its own pipeline like libc++. After your patch we will be building and testing clang twice for no reason. Can you remove it again?

metaflow · 2023-09-12T12:53:58Z

@ldionne w/o clang flang will not compile. Do you also check flang and all dependencies of clang in clang pipeline?

ldionne · 2023-09-12T18:35:57Z

I think it's fine to add clang to the set of projects in LLVM_ENABLE_PROJECTS for dependency purposes, but we could avoid calling check-clang. I guess it's moot cause we're using check-all right now.

We don't check flang in the Clang pipeline because the Clang pipeline's primary responsibility should be to test Clang, not other downstream projects.

joker-eph · 2023-09-12T20:21:36Z

I think it's fine to add clang to the set of projects in LLVM_ENABLE_PROJECTS for dependency purposes, but we could avoid calling check-clang. I guess it's moot cause we're using check-all right now.

FYI, the script for premerge generates invocations like:

'./.ci/monolithic-linux.sh "clang;flang;llvm;mlir" "check-flang check-mlir"'

So even though clang is enabled, it shouldn't be tested I think? (we don't want to test LLVM either when it isn't touched)

ldionne · 2023-09-12T23:18:52Z

The scheduled build script does call check-all, but you're right about the script that generates pre-commit CI pipelines.

.ci/monolithic-windows.sh

- fixed build for linux (clang was missing) - removed /monolithic-.. from build directory - it does not add anything and makes path longer for windows which is not great; - added env-based configuration to control cache and agent targeting; - print (s)ccache stats to file not to pullute normal log.

metaflow requested a review from ldionne September 7, 2023 06:45

metaflow added the infrastructure Bugs about LLVM infrastructure label Sep 7, 2023

metaflow requested a review from aeubanks September 7, 2023 06:47

metaflow force-pushed the ci-gen branch from 5e35b73 to 66e04bd Compare September 7, 2023 06:48

metaflow changed the title ~~Scheduled bildkite pipeline generation~~ Scheduled buildkite pipeline generation Sep 7, 2023

aeubanks reviewed Sep 7, 2023

View reviewed changes

metaflow force-pushed the ci-gen branch from 66e04bd to 428fae3 Compare September 11, 2023 10:00

metaflow changed the title ~~Scheduled buildkite pipeline generation~~ updated buildkite pipeline generation Sep 11, 2023

metaflow merged commit 2484678 into llvm:main Sep 11, 2023

metaflow deleted the ci-gen branch September 11, 2023 11:20

gchatelet mentioned this pull request Sep 11, 2023

[libc] Add type_traits tests #65956

Merged

michaelrj-google mentioned this pull request Sep 12, 2023

[libc] Move long double table option to new config #66151

Merged

vzakhari mentioned this pull request Sep 12, 2023

internap proc trampolines #66156

Closed

mtvec reviewed Sep 13, 2023

View reviewed changes

.ci/monolithic-windows.sh Show resolved Hide resolved

metaflow mentioned this pull request Sep 13, 2023

[ci] builkite don't escape windows targets #66192

Merged

updated buildkite pipeline generation #65574

updated buildkite pipeline generation #65574

Uh oh!

Conversation

metaflow commented Sep 7, 2023

Uh oh!

aeubanks left a comment

Choose a reason for hiding this comment

Uh oh!

aeubanks Sep 7, 2023

Choose a reason for hiding this comment

Uh oh!

metaflow Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

aeubanks Sep 7, 2023

Choose a reason for hiding this comment

Uh oh!

metaflow Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

aeubanks Sep 7, 2023

Choose a reason for hiding this comment

Uh oh!

metaflow Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

metaflow commented Sep 11, 2023

Uh oh!

joker-eph commented Sep 12, 2023

Uh oh!

metaflow commented Sep 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

metaflow commented Sep 12, 2023

Uh oh!

ldionne commented Sep 12, 2023

Uh oh!

metaflow commented Sep 12, 2023

Uh oh!

ldionne commented Sep 12, 2023

Uh oh!

joker-eph commented Sep 12, 2023

Uh oh!

ldionne commented Sep 12, 2023

Uh oh!

Uh oh!

Uh oh!

metaflow commented Sep 12, 2023 •

edited

Loading