Failed tests don't output `message`

[ErikSchierboom]

Tests that have failed don't output a message property which is essential for students using the online editor to understand what exactly has failed.

[guygastineau]

Hi @ErikSchierboom , thank you for the Issue. Is there a spec/schema for the JSON returned by the runners? The code in exercise.ss is quite simple, and after #22 gets merged I think we can update the output-results function to match the spec quite easily. Well, obviously we could do this first even, but I would like to get that fix through soon as it is causing a lot of passing solutions to fail for students.

[ErikSchierboom]

@guygastineau Yes, there is such a spec! You can find it here: https://exercism.org/docs/building/tooling/test-runners/interface#h-output-format Let me know if you have any questions.

[guygastineau]

Thank you, Erik. I found those docs from #8 yesterday.

In order to have meaningful messages for failing tests we need to keep track of the results of tests. I have added this to some exercises locally, and it works, but deploying it will be painful given our test library is inlined in all exercise tests. I thought this would become a problem when it was done initially, but it was done anyway. I will work on isolating our test library as an r6rs library form that is included in a hidden directory for each exercise that is then called by the actual test file. This will make it so we can update the test library easily for all exercises, and we will be able to keep this repo's version of the test library in sync too!

After that is in place I will be able to update the JSON reporter for this repo. Luckily, this provides an opportunity to make the reporting for local test running better too :)

[ErikSchierboom]

Sounds great!

[guygastineau]

This is technically fixed for tests using the new test-util.ss, but until exercism/scheme#262 is merged the legacy behavior will be preserved; I figure we should leave this issue open until that is resolved.