-
Notifications
You must be signed in to change notification settings - Fork 274
Added new solver-specific tags for SMT regression tests #5985
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added new solver-specific tags for SMT regression tests #5985
Conversation
848beac
to
53e320c
Compare
53e320c
to
269e768
Compare
Note that I have added additional targets to the |
Codecov Report
@@ Coverage Diff @@
## develop #5985 +/- ##
========================================
Coverage 75.12% 75.12%
========================================
Files 1435 1435
Lines 156301 156301
========================================
Hits 117416 117416
Misses 38885 38885 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is more distinctions than I would make but ... if you are happy to work with these then, cool.
As per the proposal described in diffblue#5982, this PR introduces new tags that can be used to label tests that are known to be broken with specific solvers. It also updates the Makefile targets to use these solver-specific tags.
269e768
to
afc385f
Compare
Thanks for taking a look @martin-cs. I removed the cvc4 tags for now since we don't run cvc4 on CI. I will keep #5982 open for further discussion and we can add more tags later if we need to. |
As per the proposal described in #5982 and offline discussions with @tautschnig, this PR introduces new tags that can be used to label tests that are known to be broken with specific solvers. It also updates the
Makefile
targets to use these solver-specific tags.This PR would unblock #5973. The regression test there was tagged
broken-smt-backend
and has been fixed to work with Z3 and CVC4 (and other solvers that support quantifiers), but currently there is no viable fix for CPROVER SMT2. Once this PR is merged, I would label that regression test asbroken-cprover-smt-backend
instead.Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).My commit message includes data points confirming performance improvements (if claimed).White-space or formatting changes outside the feature-related changed lines are in commits of their own.