Skip to content

Commit 6807817

Browse files
jonathanpeppersjonpryor
authored andcommitted
[build] Fix unit tests semantics on VSTS (#274)
Our desired build behavior is as follows: - **Green Build**: Everything compiles, and all unit tests pass. - **Red Build**: Everything *doesn't* compile. - **Orange/Unstable Build**: Everything compiles, but there is a unit test failure. The rationale for supporting Unstable builds is that oftentimes the CI system won't upload build artifacts if the build fails, and even if there are unit test failures, the build artifacts may otherwise be suitable for testing other bug fixes. (For example, the xamarin-android Jenkins job was recently failing because the network tests were hitting a TLS 1.2 site which had expired SSL certificates. This wasn't a bug in the source code or the unit tests, but the test infrastructure. This is not a Bad build.) Unfortunately, the Java.Interop tests on VSTS were not following this ideal, as unit test failures were resulting in a *green* (!) build. This happened because the `RunTests` target set `ContinueOnError="True"` on the `<Exec/>` for `nunit-console.exe`, and VSTS was only using `msbuild` exit status to determine success/failure. (Even though VSTS *found test failures* in the NUnit outputs, it still flagged the *overall* build as "good", which are *not* the semantics we wanted.) To get our desired behavior on VSTS, we have to: - Make sure our call to NUnit returns a failing exit code - Set the "Continue on Error" option in VSTS - Add a step at the end of our build definition in VSTS to fail the build if any issues occurred To make this work, we have to change our `<Exec />` task usage to use `ContinueOnError="ErrorAndContinue"`. (We *want* it to continue on error, so that *all* unit tests are executed, *even if* failures are detected.) Finally, add `Resource.designer.cs` to `.gitignore` to match the xamarin-android repo.
1 parent 5e39db3 commit 6807817

File tree

2 files changed

+3
-2
lines changed

2 files changed

+3
-2
lines changed

.gitignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,4 +8,5 @@ TestResult-*.xml
88
xa-gendarme.html
99
packages
1010
.vs/
11-
*.userprefs
11+
*.userprefs
12+
Resource.designer.cs

build-tools/scripts/RunNUnitTests.targets

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
<Exec
3131
Command="$(_NUnit) $(NUNIT_EXTRA) %(_TestAssembly.Identity) $(_Run) --noshadow --result=&quot;TestResult-%(Filename).xml&quot; --output=&quot;bin\Test$(Configuration)\TestOutput-%(Filename).txt&quot;"
3232
WorkingDirectory="$(_TopDir)"
33-
ContinueOnError="True"
33+
ContinueOnError="ErrorAndContinue"
3434
/>
3535
</Target>
3636
</Project>

0 commit comments

Comments
 (0)