[CI] Remove dependency on bot user for uploading benchmark CI results #20333

ianayl · 2025-10-09T20:37:11Z

This PR:

moves benchmarking CI data to benchmark-ci-tests branch instead of intel/llvm-ci-perf-results
removes the need for an additional bot user when pushing data (as well as the need to periodically update its tokens)

Test run: https://github.com/intel/llvm/actions/runs/18386952310/job/52388325821 (job failed due to "regression"; issue should be addressed with the merge of #20277

This reverts commit 866b64e.

This reverts commit 7ce4420.

.github/workflows/sycl-linux-run-tests.yml

devops/actions/run-tests/benchmark/action.yml

.github/workflows/sycl-docs.yml

devops/actions/run-tests/benchmark/action.yml

lukaszstolarczuk · 2025-10-13T10:53:53Z

.github/workflows/sycl-docs.yml

        # Update benchmarking dashboard configuration
        cat << EOF > benchmarks/config.js
-        remoteDataUrl = 'https://raw.githubusercontent.com/intel/llvm-ci-perf-results/refs/heads/unify-ci/';
+        remoteDataUrl = 'https://raw.githubusercontent.com/intel/llvm/refs/heads/sycl-benchmark-ci-results/';


my note on this change - while it seems convenient, I can see on my PC, that repo intel/llvm-ci-perf-results is ~600MB - this will be extra MBs added to intel/llvm repo, which is already quite big - just to consider

@ianayl would it be possible to compress the benchmarking data before pushing it to intel/llvm ?

In theory yes, but only the archived data. I think we'll need to have a conversation on this: it might even be reasonable to delete horribly outdated data.

We could also try moving archived data to intel/llvm-ci-perf-results, but alas that'll mean we need to keep an updated bot user token again for the repository.

Do github repositories have a max storage quota?

Do github repositories have a max storage quota?

https://docs.github.com/en/repositories/creating-and-managing-repositories/repository-limits#repository-size

we'll need to have a conversation on this

I think we should have that conversation before merging this PR. Currently, the size of intel/llvm-ci-perf-results is ~600MB, but, if unchecked the size will increase further.

In theory yes, but only the archived data. I think we'll need to have a conversation on this: it might even be reasonable to delete horribly outdated data.

Compressing results files gives ~190MB out of 320M, so I think it is better to just remove outdated data. We have data starting at March'25.

How many months of data do wee need to keep. 6? In this case we will have data staring from 15th April today. The rest will be in git history, hopefully well compressed.

We could also try moving archived data to intel/llvm-ci-perf-results, but alas that'll mean we need to keep an updated bot user token again for the repository.

If we go this way, we can archive e.g. once every second week, or something, and make a PR instead of pushing

Unfortunately, today is the last day of my internship, I will not have time to implement this.

I will aim to get this PR to be merge-ready today, but someone else will need to make a new PR that deletes old data

…hmark-ci-no-botuser

ianayl · 2025-10-31T23:30:20Z

Unfortunately, my internship is ending; I won't have access to the CI anymore in order to properly test and finish PR.

Leftover tasks:

another review over this PR is due
if changes are necessary, changes can be made to the ianayl/benchmark-ci-no-botuser branch within intel/llvm. I will probably not have access to this branch for much longer 😅
a mechanism still needs to be implemented for deleting old data in sycl-benchmark-ci-results branch, although this could be a separate PR
before the PR is merged: please make sure that sycl-benchmark-ci-results matches that of intel/llvm-ci-perf-results; the trees have diverged due to testing in this PR, and a force push is necessary

sarnex · 2025-11-03T15:16:50Z

@intel/llvm-reviewers-benchmarking Since Ian is no longer working on this, are there any flags to merging it as-is? It's unclear to me if any unaddressed feedback is blocking. From my POV CI seems fine.

lukaszstolarczuk

I reviewed the current code and it LGTM 👍

Perhaps the last Ian 's point should be verified - I'm not sure if any work should be done to make sure results are up-to-date...

sarnex · 2025-11-03T15:53:30Z

@lukaszstolarczuk Thanks! Do you have any suggestions on who is qualified to check that? I would do it but I don't know anything about the benchmarking code.

uditagarwal97 · 2025-11-03T16:10:42Z

I can verify that

sarnex · 2025-11-03T16:18:20Z

@uditagarwal97 Thanks!

uditagarwal97 · 2025-11-03T16:44:26Z

I've force pushed https://github.com/intel/llvm-ci-perf-results/commits/unify-ci/ into https://github.com/intel/llvm/commits/sycl-benchmark-ci-results/

sarnex · 2025-11-03T16:46:30Z

@uditagarwal97 Sorry does that mean this PR is safe to merge?

uditagarwal97 · 2025-11-03T16:47:36Z

@uditagarwal97 Sorry does that mean this PR is safe to merge?

Yes, I think we should be good to merge.

sarnex · 2025-11-03T16:48:28Z

Thank you!

ianayl added 10 commits October 3, 2025 12:33

test pushing to branch

df8990c

give write permissions

7ce4420

amend perms

866b64e

Revert "amend perms"

b518e18

This reverts commit 866b64e.

Revert "give write permissions"

2109b73

This reverts commit 7ce4420.

test perms

6e73cf7

typo

e20975f

more perms

3bea805

test fail

418bd1d

set to correct repo

1539867

ianayl requested a review from a team as a code owner October 9, 2025 20:37

ianayl temporarily deployed to WindowsCILock October 9, 2025 20:37 — with GitHub Actions Inactive

ianayl requested a review from a team October 9, 2025 20:37

aelovikov-intel requested a review from uditagarwal97 October 9, 2025 20:38

sarnex reviewed Oct 9, 2025

View reviewed changes

.github/workflows/sycl-linux-run-tests.yml Outdated Show resolved Hide resolved

.github/workflows/sycl-linux-run-tests.yml Outdated Show resolved Hide resolved

devops/actions/run-tests/benchmark/action.yml Show resolved Hide resolved

ianayl temporarily deployed to WindowsCILock October 9, 2025 20:57 — with GitHub Actions Inactive

uditagarwal97 reviewed Oct 9, 2025

View reviewed changes

.github/workflows/sycl-docs.yml Outdated Show resolved Hide resolved

uditagarwal97 reviewed Oct 9, 2025

View reviewed changes

devops/actions/run-tests/benchmark/action.yml Outdated Show resolved Hide resolved

switch branch

8e1f1b8

ianayl temporarily deployed to WindowsCILock October 9, 2025 23:51 — with GitHub Actions Inactive

ianayl temporarily deployed to WindowsCILock October 10, 2025 00:13 — with GitHub Actions Inactive

try removing manually passing secrets

66c4b5c

ianayl temporarily deployed to WindowsCILock October 10, 2025 20:40 — with GitHub Actions Inactive

ianayl temporarily deployed to WindowsCILock October 10, 2025 21:00 — with GitHub Actions Inactive

lukaszstolarczuk reviewed Oct 13, 2025

View reviewed changes

fix typo

baf64ef

ianayl had a problem deploying to WindowsCILock October 14, 2025 16:03 — with GitHub Actions Error

ianayl had a problem deploying to WindowsCILock October 31, 2025 05:08 — with GitHub Actions Error

ianayl had a problem deploying to WindowsCILock October 31, 2025 05:09 — with GitHub Actions Error

fix const

fad9bae

ianayl temporarily deployed to WindowsCILock October 31, 2025 05:13 — with GitHub Actions Inactive

ianayl temporarily deployed to WindowsCILock October 31, 2025 05:37 — with GitHub Actions Inactive

ianayl temporarily deployed to WindowsCILock October 31, 2025 05:38 — with GitHub Actions Inactive

ianayl added 2 commits October 31, 2025 14:13

fix perms

0c4e7e2

Merge branch 'sycl' of https://github.com/intel/llvm into ianayl/benc…

3b260f6

…hmark-ci-no-botuser

ianayl temporarily deployed to WindowsCILock October 31, 2025 21:16 — with GitHub Actions Inactive

ianayl temporarily deployed to WindowsCILock October 31, 2025 21:41 — with GitHub Actions Inactive

ianayl temporarily deployed to WindowsCILock October 31, 2025 22:00 — with GitHub Actions Inactive

lukaszstolarczuk approved these changes Nov 3, 2025

View reviewed changes

uditagarwal97 approved these changes Nov 3, 2025

View reviewed changes

sarnex approved these changes Nov 3, 2025

View reviewed changes

sarnex merged commit 33c11a6 into sycl Nov 3, 2025
50 of 53 checks passed

uditagarwal97 deleted the ianayl/benchmark-ci-no-botuser branch November 3, 2025 16:48

lukaszstolarczuk mentioned this pull request Nov 4, 2025

[CI][Benchmarks] Fix pushing results to new branch #20544

Merged

[CI] Remove dependency on bot user for uploading benchmark CI results #20333

[CI] Remove dependency on bot user for uploading benchmark CI results #20333

Uh oh!

Conversation

ianayl commented Oct 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukaszstolarczuk Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

uditagarwal97 Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianayl Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

uditagarwal97 Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

lslusarczyk Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

lukaszstolarczuk Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

ianayl Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

ianayl commented Oct 31, 2025

Uh oh!

sarnex commented Nov 3, 2025

Uh oh!

lukaszstolarczuk left a comment

Choose a reason for hiding this comment

Uh oh!

sarnex commented Nov 3, 2025

Uh oh!

uditagarwal97 commented Nov 3, 2025

Uh oh!

sarnex commented Nov 3, 2025

Uh oh!

uditagarwal97 commented Nov 3, 2025

Uh oh!

sarnex commented Nov 3, 2025

Uh oh!

uditagarwal97 commented Nov 3, 2025

Uh oh!

sarnex commented Nov 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

uditagarwal97 Oct 14, 2025 •

edited

Loading