Remove continue-on-error for benchmarks in CI #464
Merged
+11
−3
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'd like to remove the continue-on-error for benchmarks on TruffleRuby in CI, otherwise any regression there will most likely be missed. A concrete example is #375 didn't notice it broke all benchmarks on TruffleRuby due to the continue-on-error. I fixed that #442.
If it fails again, and it's not an issue of the code in this repository then please @-mention me (@eregon) and:
--excludeslist.continue-on-error: ${{ matrix.ruby == 'truffleruby' }}in thename: Test run_benchmarks.rbstep.Is that reasonable, WDYT?
These benchmarks run with
ruby-version: truffleruby, so the latest TruffleRuby 33 release, so that's more stable than e.g. usingtruffleruby-head. That also contains an important fix for Bundler install, which prevented to make this PR before.Independently of this I will run the same benchmarks in truffleruby/truffleruby's CI to make sure we don't regress there on
master, and that way new TruffleRuby releases shouldn't break ruby-bench's CI.