Fix test runner

.dockerignore

@@ -1 +1,2 @@
-test/output
+.github/
+tests/

.github/workflows/main.yml

@@ -1,31 +1,20 @@
----
 name: CI
 
 on:
   pull_request:
     branches:
-      - master
-    paths-ignore:
-      - '.gitignore'
-      - 'LICENSE'
-      - '**.md'
+      - main
   push:
     branches:
-      - master
-    paths-ignore:
-      - '.gitignore'
-      - 'LICENSE'
-      - '**.md'
+      - main
 
 jobs:
   build:
-    name: Test Runner
+    name: Tests
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v1
+      - name: Checkout code
+        uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f # 2.3.4
 
-      - name: Build Docker Image
-        run: docker build -f Dockerfile -t generic-test-runner .
-
-      - name: Run Tests
-        run: docker run -w "/opt/test-runner" --entrypoint "bin/run-all.sh" generic-test-runner
+      - name: Run Tests in Docker
+        run: bin/run-tests-in-docker.sh

.gitignore

@@ -1,6 +1,7 @@
-test/output/
+tests/*/results.json
 .DS_Store
 **/.build
 **/Packages
 **/Package.resolved
 **/*.xcodeproj
+bin/TestRunner

Dockerfile

@@ -15,4 +15,3 @@ COPY --from=builder /.build/release/TestRunner bin/
 ENV NAME RUNALL
 
 ENTRYPOINT ["./bin/run.sh"]
-# ENTRYPOINT ["bin/TestRunner", "--help"]

README.md

@@ -1,42 +1,51 @@
 # Exercism Swift Test Runner
 
-The Docker image to automatically run tests on Swiftsolutions submitted to [Exercism].
+The Docker image to automatically run tests on Swift solutions submitted to [Exercism].
 
-## Running the Tests
+## Run the test runner
 
-To run all of the tests, do the following:
+To run the tests of an arbitrary exercise, do the following:
 
 1. Open a terminal in the project's root
-2. Run `bin/run-all.sh test test/output`
+2. Run `./bin/run.sh <exercise-slug> <solution-dir> <output-dir>`
 
-These are [golden tests][golden] that compare the `results.json` generated by running the current state of the code against the "known good" `test/<test-name>/results.json`. All files created during the test run itself are discarded.
+Once the test runner has finished, its results will be written to `<output-dir>/results.json`.
 
-When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new `test/<test-name>/results.json` file.
+## Run the test runner on an exercise using Docker
 
-## Generating `results.json` for an Exercise
+_This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment._
 
-To run the tests of an arbitrary exercise, do the following:
+To run the tests of an arbitrary exercise using the Docker image, do the following:
 
 1. Open a terminal in the project's root
-2. Run `./bin/run.sh <exercise-slug> <path/to/solution/dir/> </path/to/output/dir/>`<br/>E.g. `bin/run.sh multiple-tests-single-fail test/ test/output`
+2. Run `./bin/run-in-docker.sh <exercise-slug> <solution-dir> <output-dir>`
 
-## Generating `results.json` for an Exercise with Docker
+Once the test runner has finished, its results will be written to `<output-dir>/results.json`.
 
-_This script is provided for testing purposes._
+## Run the tests
 
-Note that due to the current test scripts state, which uses a simple diff to compare snapshots, you will have to use a different test solution directory, here `dockerTest`. This directory uses modified golden tests that match the path information reported in XCTest's errors and keep the paths from tripping up the diff check. 
-
-To run the tests of an arbitrary exercise using the Docker container, do the following:
+To run the tests to verify the behavior of the test runner, do the following:
 
 1. Open a terminal in the project's root
-2. Run `./bin/run-in-docker.sh <exercise-slug> <./relative/path/to/solution/dir/> <./relative/path/to/output/dir/>`<br/>E.g. `bin/run-in-docker.sh multiple-tests-single-fail ./dockerTest ./dockerTest/output`
+2. Run `./bin/run-tests.sh`
+
+These are [golden tests][golden] that compare the `results.json` generated by running the current state of the code against the "known good" `tests/<test-name>/results.json`. All files created during the test run itself are discarded.
 
-**Note**: the solution and output directory's **MUST** be relative paths!
+When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new `tests/<test-name>/results.json` file.
 
-To run all of the tests using the Docker container, do the following:
+## Run the tests using Docker
+
+_This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment._
+
+To run the tests to verify the behavior of the test runner using the Docker image, do the following:
 
 1. Open a terminal in the project's root
-2. Run `bin/run-all-in-docker.sh`
+2. Run `./bin/run-tests-in-docker.sh`
+
+These are [golden tests][golden] that compare the `results.json` generated by running the current state of the code against the "known good" `tests/<test-name>/results.json`. All files created during the test run itself are discarded.
+
+When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new `tests/<test-name>/results.json` file.
 
+[test-runners]: https://github.com/exercism/automated-tests/blob/master/docs/introduction.md
 [golden]: https://ro-che.info/articles/2017-12-04-golden-tests
 [exercism]: https://exercism.io

bin/run-in-docker.sh

@@ -2,38 +2,42 @@
 set -e
 
 # Synopsis:
-# Test runner for run.sh in a docker container
-# Takes the same arguments as run.sh (EXCEPT THAT SOLUTION AND OUTPUT PATH ARE RELATIVE)
-# Builds the Dockerfile
-# Runs the docker image passing along the initial arguments
+# Run the test runner on a solution using the test runner Docker image.
+# The test runner Docker image is built automatically.
 
 # Arguments:
 # $1: exercise slug
-# $2: **RELATIVE** path to solution folder (with trailing slash)
-# $3: **RELATIVE** path to output directory (with trailing slash)
+# $2: absolute path to solution folder
+# $3: absolute path to output directory
 
 # Output:
 # Writes the test results to a results.json file in the passed-in output directory.
-# The test results are formatted according to the specifications at https://github.com/exercism/automated-tests/blob/master/docs/interface.md
+# The test results are formatted according to the specifications at https://github.com/exercism/docs/blob/main/building/tooling/test-runners/interface.md
 
 # Example:
-# ./run-in-docker.sh two-fer ./relative/path/to/two-fer/solution/folder/ ./relative/path/to/output/directory/
+# ./bin/run-in-docker.sh two-fer /absolute/path/to/two-fer/solution/folder/ /absolute/path/to/output/directory/
 
-# If arguments not provided, print usage and exit
+# If any required arguments is missing, print the usage and exit
 if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
-    echo "usage: run-in-docker.sh exercise-slug ./relative/path/to/solution/folder/ ./relative/path/to/output/directory/"
+    echo "usage: ./bin/run-in-docker.sh exercise-slug /absolute/path/to/solution/folder/ /absolute/path/to/output/directory/"
     exit 1
 fi
 
-# build docker image
-docker build --rm --no-cache -t swift-test-runner .
+SLUG="$1"
+INPUT_DIR="${2%/}"
+OUTPUT_DIR="${3%/}"
+
+# Create the output directory if it doesn't exist
+mkdir -p "${OUTPUT_DIR}"
 
-# Create output directory if it doesn't exist
-output_dir="$3"
-mkdir -p "$output_dir"
+# build docker image
+docker build --rm -t exercism/swift-test-runner .
 
 # run image passing the arguments
+# TODO: support --read-only flag
 docker run \
-    --mount type=bind,src=$PWD/$2,dst=/solution \
-    --mount type=bind,src=$PWD/$output_dir,dst=/output \
-    swift-test-runner $1 /solution/ /output/
+    --network none \
+    --mount type=bind,src="${INPUT_DIR}",dst=/solution \
+    --mount type=bind,src="${OUTPUT_DIR}",dst=/output \
+    --mount type=volume,dst=/tmp \
+    exercism/swift-test-runner $SLUG /solution/ /output/

bin/run-tests-in-docker.sh

@@ -0,0 +1,26 @@
+#! /bin/bash -e
+
+# Synopsis:
+# Test the test runner Docker image by running it against a predefined set of 
+# solutions with an expected output.
+# The test runner Docker image is built automatically.
+
+# Output:
+# Outputs the diff of the expected test results against the actual test results
+# generated by the test runner Docker image.
+
+# Example:
+# ./bin/run-tests-in-docker.sh
+
+# Build the Docker image
+docker build --rm -t exercism/swift-test-runner .
+
+# Run the Docker image using the settings mimicking the production environment
+# TODO: support --read-only flag
+docker run \
+    --network none \
+    --mount type=bind,src="${PWD}/tests",dst=/opt/test-runner/tests \
+    --mount type=volume,dst=/tmp \
+    --workdir /opt/test-runner \
+    --entrypoint /opt/test-runner/bin/run-tests.sh \
+    exercism/swift-test-runner

bin/run-tests.sh

@@ -0,0 +1,36 @@
+#! /bin/sh
+
+# Synopsis:
+# Test the test runner by running it against a predefined set of solutions 
+# with an expected output.
+
+# Output:
+# Outputs the diff of the expected test results against the actual test results
+# generated by the test runner.
+
+# Example:
+# ./bin/run-tests.sh
+
+exit_code=0
+
+# Iterate over all test directories
+for test_dir in tests/*; do
+    test_dir_name=$(basename "${test_dir}")
+    test_dir_path=$(realpath "${test_dir}")
+    results_file_path="${test_dir_path}/results.json"
+    expected_results_file_path="${test_dir_path}/expected_results.json"
+
+    bin/run.sh "${test_dir_name}" "${test_dir_path}" "${test_dir_path}"
+
+    # Normalize the results file
+    sed -i -e "s~\\\/~/~g" -e "s~${test_dir_path}~/solution~g" "${results_file_path}"
+
+    echo "${test_dir_name}: comparing results.json to expected_results.json"
+    diff "${results_file_path}" "${expected_results_file_path}"
+
+    if [ $? -ne 0 ]; then
+        exit_code=1
+    fi
+done
+
+exit ${exit_code}

bin/run.sh

@@ -1,40 +1,29 @@
-#! /bin/sh
-set -e
+#!/usr/bin/env bash
 
-# If arguments not provided, print usage and exit
+# Synopsis:
+# Run the test runner on a solution.
+
+# Arguments:
+# $1: exercise slug
+# $2: absolute path to solution folder
+# $3: absolute path to output directory
+
+# Output:
+# Writes the test results to a results.json file in the passed-in output directory.
+# The test results are formatted according to the specifications at https://github.com/exercism/docs/blob/main/building/tooling/test-runners/interface.md
+
+# Example:
+# ./bin/run.sh two-fer /absolute/path/to/two-fer/solution/folder/ /absolute/path/to/output/directory/
+
+# If any required arguments is missing, print the usage and exit
 if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
-    echo "usage: run.sh exercise-slug ./relative/path/to/solution/folder/ ./relative/path/to/output/directory/"
+    echo "usage: ./bin/run.sh exercise-slug /absolute/path/to/two-fer/solution/folder/ /absolute/path/to/output/directory/"
     exit 1
 fi
 
 SLUG="$1"
-INPUT_DIR="$2"
-OUTPUT_DIR="$3"
-
+INPUT_DIR="${2%/}"
+OUTPUT_DIR="${3%/}"
 BASEDIR=$(dirname "$0")
 
-# echo "$SLUG: testing..."
-# echo "$1"
-# echo "$2"
-# echo "$3"
-# echo "-------------"
-RUNALL=true "${BASEDIR}"/TestRunner --slug "${SLUG}" --solution-directory "${INPUT_DIR}/${SLUG}" --output-directory "${OUTPUT_DIR}" --swift-location $(which swift) --build-directory "/tmp/"
-
-#echo "$SLUG: processing test output in $INPUT_DIR..."
-## PLACEHOLDER - OPTIONAL: Your language may support outputting results
-##   in the correct format
-#
-# Create $OUTPUT_DIR if it doesn't exist
-[ -d "$OUTPUT_DIR" ] || mkdir -p "$OUTPUT_DIR"
-#
-#echo "$SLUG: copying processed results to $OUTPUT_DIR..."
-## PLACEHOLDER - OPTIONAL: Your language may support placing results
-##   directly in $OUTPUT_DIR
-#cp "${INPUT_DIR}/results.json" "$OUTPUT_DIR"
-
-echo "$SLUG: comparing ${OUTPUT_DIR}/results"
-diff "${INPUT_DIR}/${SLUG}/results.json" "${OUTPUT_DIR}/results.json"
-
-echo "$SLUG: OK\n-------------\n"
-
-
+RUNALL=true "${BASEDIR}"/TestRunner --slug "${SLUG}" --solution-directory "${INPUT_DIR}" --output-directory "${OUTPUT_DIR}" --swift-location $(which swift) --build-directory "/tmp"