Skip to content

Report Add Table New Fails in PR & Fix Tests Not Correctly Marked as Broken #763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 39 commits into from
May 12, 2025
Merged

Report Add Table New Fails in PR & Fix Tests Not Correctly Marked as Broken #763

merged 39 commits into from
May 12, 2025

Conversation

strtgbb
Copy link
Collaborator

@strtgbb strtgbb commented May 2, 2025 •
edited
Loading

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changes:

  • When run against a PR workflow the report now has a table "New Fails in PR"
  • Build report is now linked at the top of the workflow report
  • If a previously failed test passes on rerun, it will no longer show as failed in the report
  • Improved line wrapping of test names in report when in a narrow window
  • Non-PR report now states the branch
  • Grype scan step summaries now have the full image name
  • RunConfig prints a step summary with the report location, a preview becomes available after the first test job completes
  • Many more unstable and broken tests added to broken_tests.json
  • All integration tests now have 8 batches, previously some test were not running due to hitting timeouts
  • Skipped integration tests are properly detected instead of being marked ERROR
  • Broken test marking now works in integration flaky check
  • Fixed bug in broken marking broken integration tests
  • Stress tests have azurite re-enabled and threads reduced
  • Several failing integration tests now have reasonable timeouts

Issues:

  • This is untested with regression fails, but they should be supported.
  • Some fails that should be marked as broken seem to not be, and end up in the table. Fixed.

Skip Jobs

  • Skip All Regression

@strtgbb strtgbb force-pushed the report_new_fails_in_pr branch from 2b83a9c to 2582a4b Compare May 5, 2025 13:37
@strtgbb strtgbb changed the title Report Add Table New Fails in PR Report Add Table New Fails in PR & Fix Tests Not Correctly Marked as Broken May 6, 2025
strtgbb added 19 commits May 6, 2025 09:00
@strtgbb
increase integration tests timeout
f5c8f34
@strtgbb
try moving broken test handling a bit later
9c74856
@strtgbb
fix typo
5891153
@strtgbb
log broken test handler actions verbosely to a file
8c362c8
@strtgbb
add report link to RunConfig output
5ccee71
@strtgbb
more links
3bca15e
@strtgbb
log not broken tests too
56deffb
@strtgbb
Add image to grype scan summary
d7a1d6a
@strtgbb
cross out all current fails - create a clean baseline
bffdc54
@strtgbb
MAX_RETRY=2 takes too long in integration, change to 2
50aefaf
@strtgbb
fix release report
75b66a4
@strtgbb
Remove 20 fail limit on test groups and log more in broken test handler
8e2dac1
@strtgbb
increase integration batches and log more in broken test handler
e5d5920
@strtgbb
use twice as many batches for integration
539955c
@strtgbb
fix root cause of broken tests issue, add type hints
0542d3b
@strtgbb
move _handle_broken_tests a bit earlier
87a3b65
@strtgbb
fix
f373cb7
@strtgbb
tuning batch sizes
c1d8e5b
@strtgbb
fix skipped test detection
d07b9a5
Enmk
Enmk reviewed May 8, 2025
View reviewed changes
@strtgbb strtgbb requested review from Enmk and MyroTk May 9, 2025 02:45
@Enmk Enmk merged commit 0217bf4 into antalya May 12, 2025
167 of 170 checks passed
strtgbb added a commit that referenced this pull request May 12, 2025
@strtgbb
Merge pull request #763 from Altinity/report_new_fails_in_pr
5b25b00 
Report Add Table New Fails in PR & Fix Tests Not Correctly Marked as Broken
@strtgbb strtgbb added the cicd Improvements and fixes to the CICD process label May 23, 2025
@strtgbb strtgbb deleted the report_new_fails_in_pr branch June 6, 2025 13:56
Enmk added a commit that referenced this pull request Aug 6, 2025
@Enmk
Fied some of the Integration tests
04fcc75 
By reverting those back to upstream/25.6 variant.
Those are related to the features that were not merged yet and hence fail:
 - integration/test_s3_assume_role/test.py - was added by #688
 - integration/test_mask_sensitive_info/test.py - was added by #675
 - integration/test_s3_cache_locality/test.py - was added by #763
Enmk added a commit that referenced this pull request Aug 6, 2025
@Enmk
Fied some of the Integration tests
b76d3b0 
By reverting those back to upstream/25.6 variant.
Those are related to the features that were not merged yet and hence fail:
 - integration/test_s3_assume_role/test.py - was added by #688
 - integration/test_mask_sensitive_info/test.py - was added by #675
 - integration/test_s3_cache_locality/test.py - was added by #763
Enmk added a commit that referenced this pull request Aug 6, 2025
@Enmk
Fixed some of the Integration tests
68fefe3 
By reverting those back to upstream/25.6 variant.
Those are related to the features that were not merged yet and hence fail:
 - integration/test_s3_assume_role/test.py - was added by #688
 - integration/test_mask_sensitive_info/test.py - was added by #675
 - integration/test_s3_cache_locality/test.py - was added by #763
@Enmk Enmk mentioned this pull request Aug 6, 2025
Merged
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Enmk Enmk Awaiting requested review from Enmk

@MyroTk MyroTk Awaiting requested review from MyroTk

Assignees

No one assigned

Labels

cicd Improvements and fixes to the CICD process

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@strtgbb @Enmk