Commit 6807817
[build] Fix unit tests semantics on VSTS (#274)
Our desired build behavior is as follows:
- **Green Build**: Everything compiles, and all unit tests pass.
- **Red Build**: Everything *doesn't* compile.
- **Orange/Unstable Build**: Everything compiles, but there is a
unit test failure.
The rationale for supporting Unstable builds is that oftentimes the
CI system won't upload build artifacts if the build fails, and even
if there are unit test failures, the build artifacts may otherwise
be suitable for testing other bug fixes.
(For example, the xamarin-android Jenkins job was recently failing
because the network tests were hitting a TLS 1.2 site which had
expired SSL certificates. This wasn't a bug in the source code or
the unit tests, but the test infrastructure. This is not a Bad build.)
Unfortunately, the Java.Interop tests on VSTS were not following this
ideal, as unit test failures were resulting in a *green* (!) build.
This happened because the `RunTests` target set
`ContinueOnError="True"` on the `<Exec/>` for `nunit-console.exe`,
and VSTS was only using `msbuild` exit status to determine
success/failure. (Even though VSTS *found test failures* in the
NUnit outputs, it still flagged the *overall* build as "good", which
are *not* the semantics we wanted.)
To get our desired behavior on VSTS, we have to:
- Make sure our call to NUnit returns a failing exit code
- Set the "Continue on Error" option in VSTS
- Add a step at the end of our build definition in VSTS to fail the
build if any issues occurred
To make this work, we have to change our `<Exec />` task usage to use
`ContinueOnError="ErrorAndContinue"`. (We *want* it to continue on
error, so that *all* unit tests are executed, *even if* failures are
detected.)
Finally, add `Resource.designer.cs` to `.gitignore` to match the
xamarin-android repo.1 parent 5e39db3 commit 6807817
2 files changed
+3
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
0 commit comments