add features test ... command

[joshspicer]

https://github.com/github/codespaces/issues/8002

Adds devcontainer features test

devcontainer features test

Test features

Options:
      --help               Show help                                                                           [boolean]
      --version            Show version number                                                                 [boolean]
  -i, --base-image         Base Image                                                 [string] [default: "ubuntu:focal"]
  -f, --features           Feature(s) to test as space-separated parameters. Omit to auto-detect all features in
                           collection directory.  Cannot be combined with '-s'.                                  [array]
  -s, --scenarios          Path to scenario test directory (containing scenarios.json).  Cannot be combined with '-f'.
                                                                                                                [string]
  -u, --remote-user        Remote user                                                        [string] [default: "root"]
  -c, --collection-folder  Path to folder containing 'src' and 'test' sub-folders.               [string] [default: "."]
      --log-level          Log level.                              [choices: "info", "debug", "trace"] [default: "info"]
  -q, --quiet              Quiets output                                                      [boolean] [default: false]

(Folks with visibility can check out https://github.com/devcontainers/features/tree/main/test and https://github.com/devcontainers/features/tree/main/test-scenarios)

Example usage

[joshspicer]

example running in CI: https://github.com/devcontainers/features/runs/6396166078?check_suite_focus=true#step:3:1

[stuartleeks]

There's some work underway that @chrmarti and I have been discussing to enable dev container images to be build with inline cache data. This is to enable dev container image caching to work with container-features in environments such as CI where intermediate images from previous builds aren't available.

To enable this, we plan to combine the container features into a single, multi-stage build step with main image build, so I'm not sure how well that would work with this command?

An alternative might be a helper function that takes the feature details and generates the devcontainer.json for you, and then call dev-containers-cli build?

[joshspicer]

Having a devcontainer-cli features test ... command is definitely something we want to promote features authoring and adoption.

To enable this, we plan to combine the container features into a single, multi-stage build step with main image build, so I'm not sure how well that would work with this command?

I don't think it'll really change this? Right now this is essentially just doing a devcontainer-cli up ... with an auto-generated devcontainer.json. So long as we can do a devcontainer-cli up with a feature-filled devcontainer.json this technique should still work. (And looking forward to the multi-stage build!). To run the test scripts, it's basically just a devcontainer-cli exec.

I'd love your feedback @chrmarti on the best way to "hook into" building a devcontainer from a CLI command (you can see i'm using the launch(..) command to build and start the container, but open to suggestions on doing this better.)

[stuartleeks]

Ah, I'd misunderstood the appraoch - sorry!

[chrmarti]

Let's use a Log object like other commands.

[joshspicer]

I'm finding that the Log is printing out a lot more information than I want (timestamp, header, etc..). Also, I can't easily color the message.

What reason do you have for using Log in this case?

[chrmarti]

You could probably inline launch and use the parameter objects that gets from resolve to call runRemoteCommand directly. Maybe check if that makes things clearer.

[joshspicer]

I tried to do this and found that I was duplicating a lot of code from doExec, and it was overall not working as well ($PATH wasn't accessible). I very likely was doing something wrong, but leaving this here for now. Happy to chat more about this though

(and would be nice to find a more standardized pattern. to invoke one CLI command from another)

[joshspicer]

eg output, testing two features go and dotnet against base image ubuntu:focal:

@joshspicer ➜ /workspaces/cli (jospicer/devcontainer-features-test ✗) $ ./devcontainer.js features test -f go -f dotnet -c ~/example

┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
|    dev container 'features' |   
│     Testing v0.3.0          │
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

>  baseImage:         ubuntu:focal
>  Target Folder:     /home/node/example
>  features:          go, dotnet
>  workspaceFolder:   /tmp/vsch/container-features-test/1653498316110

⏳ Building test container...

[1 ms] @devcontainers/cli 0.3.0.
[224 ms] Start: Run: docker buildx build --load --build-context dev_containers_feature_content_source=/tmp/vsch/container-features/0.3.0-1653498316364 --build-arg _DEV_CONTAINERS_BASE_IMAGE=ubuntu:focal --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp -t vsc-1653498316110-8876ba1b135628075a1cea53a9f66029-features -f /tmp/vsch/container-features/0.3.0-1653498316364/Dockerfile.extended /tmp/__dev-containers-build-empty
[+] Building 133.2s (14/14) FINISHED                                            
 => [internal] load build definition from Dockerfile.extended              0.1s
 => => transferring dockerfile: 762B                                       0.0s
 => [internal] load .dockerignore                                          0.2s
 => => transferring context: 2B                                            0.0s
 => resolve image config for docker.io/docker/dockerfile:1.4               0.1s
 => CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:443aab4c  0.0s
 => [internal] load build definition from Dockerfile.extended              0.0s
 => [internal] load .dockerignore                                          0.0s
 => [internal] load metadata for docker.io/library/ubuntu:focal            0.0s
 => [context dev_containers_feature_content_source] load .dockerignore     0.1s
 => => transferring dev_containers_feature_content_source: 2B              0.0s
 => [context dev_containers_feature_content_source] load from client       0.1s
 => => transferring dev_containers_feature_content_source: 29.99kB         0.0s
 => CACHED [stage-0 1/4] FROM docker.io/library/ubuntu:focal               0.0s
 => [stage-0 2/4] COPY --from=dev_containers_feature_content_source . /tm  0.2s
 => [stage-0 3/4] RUN cd /tmp/build-features/go_1 && export $(cat devco  101.1s
 => [stage-0 4/4] RUN cd /tmp/build-features/dotnet_2 && export $(cat de  24.5s
 => exporting to image                                                     6.5s 
 => => exporting layers                                                    6.5s 
 => => writing image sha256:9c6df1c9f4b46ef3eff83a7472016d552f6b61279647d  0.0s 
 => => naming to docker.io/library/vsc-1653498316110-8876ba1b135628075a1c  0.0s 
[133652 ms] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=/tmp/vsch/container-features-test/1653498316110,target=/workspaces/1653498316110 -l devcontainer.local_folder=/tmp/vsch/container-features-test/1653498316110 --entrypoint /bin/sh vsc-1653498316110-8876ba1b135628075a1cea53a9f66029-features -c echo Container started
Container started

🚀 Launched container.
>  containerId:          710daf4a6a71c146abe02415424cf1a9f3ecd70bdac2b6e7dea4661c8456a6a0
>  remoteUser:           root

🏃 Starting test(s)...

🧪 Executing 'go' test...


🔄 Testing 'version'

go version go1.18.2 linux/amd64
✅  Passed 'version'!
🧪 Executing 'dotnet' test...


🔄 Testing 'run dotnet info'

.NET SDK (reflecting any global.json):
 Version:   6.0.300
 Commit:    8473146e7d

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         ubuntu.20.04-x64
 Base Path:   /usr/share/dotnet/sdk/6.0.300/

Host (useful for support):
  Version: 6.0.5
  Commit:  70ae3df4a6

.NET SDKs installed:
  6.0.300 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download
✅  Passed 'run dotnet info'!


🔄 Testing 'dotnet sdks'

6.0.300 [/usr/share/dotnet/sdk]
✅  Passed 'dotnet sdks'!

✅ All tests passed!

[joshspicer]

I would love if we refactored the code a bit to make the provision and exec command easier to call internally. I couldn't seem to find a good blend of "easy to use" without moving or copying a bunch of code :)

[chrmarti]

Reapplied on main as a single commit in #81.