Skip to content

flaky and unhelpful incremental compilation test harness #22510

@andrewrk

Description

@andrewrk

There are inconsistent CI failures happening with incr-check that look like this:

+- test-incremental
   +- incr-check 'add_decl_namespaced' failure
warning: failed to delete tree 'tmp_942a625ed3a64fa8': FileBusy
error: expected compile errors but compilation incorrectly succeeded
error: the following command exited with code 1 (expected exited with code 0):
C:\Users\ci\george3\_work\zig\zig\build-release\zig-local-cache\o\0d8e9bdeac5505a2367bdc5e61cb7692\incr-check.exe C:\Users\ci\george3\_work\zig\zig\build-release\stage3-release\bin\zig.exe C:\Users\ci\george3\_work\zig\zig\test\incremental\add_decl_namespaced --zig-lib-dir C:\Users\ci\george3\_work\zig\zig\lib\ 

There are multiple issues here:

  • error: expected compile errors but compilation incorrectly succeeded - this is a flaky test. it sometimes passes, sometimes fails.
  • warning: failed to delete tree 'tmp_942a625ed3a64fa8': FileBusy what's going on here? this should not happen
  • the error fails to mention which target configuration is being tested. better yet the test harness should make that be a parameter to incr-check so that it's easy to repeat only that target from the CLI based on the printed failed command.
  • the error fails to mention which update it was on when the problem happened
  • there is no way to disable a target or disable a case, so I am now going to delete the flaky tests
  • Windows coverage has been disabled in 3f2cf1c

This issue can be closed when all the bullet points above are addressed, and the test cases which have been deleted disabled have been restored. See commit messages mentioning this issue number to find cases which have been deleted disabled.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugObserved behavior contradicts documented or intended behaviorincremental compilationProblem occurs only when reusing compiler state.testingThis issue is related to testing the compiler, standard library, or other parts of Zig.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions