Skip to content

Add critest to Makefile and Docker test image #563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 18, 2022
Merged

Add critest to Makefile and Docker test image #563

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,8 +256,12 @@ TEST_BRIDGED_TAP_BIN?=$(BINPATH)/test-bridged-tap
$(TEST_BRIDGED_TAP_BIN): $(shell find internal/cmd/test-bridged-tap -name *.go) $(GOMOD) $(GOSUM)
go build -o $@ $(CURDIR)/internal/cmd/test-bridged-tap

LOOPBACK_BIN?=$(BINPATH)/loopback
$(LOOPBACK_BIN):
GOBIN=$(dir $@) GO111MODULE=off go get -u github.com/containernetworking/plugins/plugins/main/loopback
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to pin the version of loopback? I believe go get is going to fetch the latest version.

Copy link
Contributor Author

@ginglis13 ginglis13 Feb 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, go get will get version @ latest . I was following how we currently get/install other container networking plugins:

firecracker-containerd/Makefile

Lines 235 to 249 in 3c25750

BRIDGE_BIN?=$(BINPATH)/bridge
$(BRIDGE_BIN):
GOBIN=$(dir $@) GO111MODULE=off go get -u github.com/containernetworking/plugins/plugins/main/bridge
PTP_BIN?=$(BINPATH)/ptp
$(PTP_BIN):
GOBIN=$(dir $@) GO111MODULE=off go get -u github.com/containernetworking/plugins/plugins/main/ptp
HOSTLOCAL_BIN?=$(BINPATH)/host-local
$(HOSTLOCAL_BIN):
GOBIN=$(dir $@) GO111MODULE=off go get -u github.com/containernetworking/plugins/plugins/ipam/host-local
FIREWALL_BIN?=$(BINPATH)/firewall
$(FIREWALL_BIN):
GOBIN=$(dir $@) GO111MODULE=off go get -u github.com/containernetworking/plugins/plugins/meta/firewall

Do we want to be able to pin a version for each of these as well? Can take that up as a separate issue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh okay. I'm fine doing that in a separate PR in this case.


.PHONY: cni-bins
cni-bins: $(BRIDGE_BIN) $(PTP_BIN) $(HOSTLOCAL_BIN) $(FIREWALL_BIN) $(TC_REDIRECT_TAP_BIN)
cni-bins: $(BRIDGE_BIN) $(PTP_BIN) $(HOSTLOCAL_BIN) $(FIREWALL_BIN) $(TC_REDIRECT_TAP_BIN) $(LOOPBACK_BIN)

.PHONY: test-cni-bins
test-cni-bins: $(TEST_BRIDGED_TAP_BIN)
Expand All @@ -269,6 +273,7 @@ install-cni-bins: cni-bins $(CNI_BIN_ROOT)
install -D -o root -g root -m755 -t $(CNI_BIN_ROOT) $(HOSTLOCAL_BIN)
install -D -o root -g root -m755 -t $(CNI_BIN_ROOT) $(FIREWALL_BIN)
install -D -o root -g root -m755 -t $(CNI_BIN_ROOT) $(TC_REDIRECT_TAP_BIN)
install -D -o root -g root -m755 -t $(CNI_BIN_ROOT) $(LOOPBACK_BIN)

.PHONY: install-test-cni-bins
install-test-cni-bins: test-cni-bins $(CNI_BIN_ROOT)
Expand Down
18 changes: 18 additions & 0 deletions runtime/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,24 @@ perf-test:
$(FIRECRACKER_CONTAINERD_TEST_IMAGE):$(DOCKER_IMAGE_TAG) \
"go test -timeout 0 -v -count=1 -run \"^$(PERF_TESTNAME)$$\""

.PHONY: critest
critest:
$(CURDIR)/../tools/thinpool.sh reset "$(FICD_DM_POOL)"
docker run --rm -it \
--privileged \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it work with --network=none?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as I have right now, it does not work with --network=none because critest need to pull an image for creating Pod sandboxes. Would it be best to pull this sandbox image into FIRECRACKER_CONTAINERD_TEST_IMAGE , then ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. It would be better to keep that as is. Technically we could pull some images beforehand, but that may make critest less useful, assuming we might break the pulling part someday.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok yes and just more on that, the sandbox image could be predownloading but networking would still have to be enable as there are tests for pulling images too (e.g. [It] public image with tag should be pulled and removed)

So for now I think we'll have to enable networking for this test.

--ipc=host \
--volume /dev:/dev \
--volume /run/udev/control:/run/udev/control \
--volume $(CURDIR)/..:/src \
--volume $(GO_CACHE_VOLUME_NAME):/go \
--env FICD_DM_VOLUME_GROUP=$(FICD_DM_VOLUME_GROUP) \
--env FICD_DM_POOL=$(FICD_DM_POOL) \
--env GOPROXY=direct \
--env GOSUMDB=off \
--workdir="/src/runtime" \
$(FIRECRACKER_CONTAINERD_TEST_IMAGE):$(DOCKER_IMAGE_TAG) \
"critest -runtime-endpoint unix:///run/firecracker-containerd/containerd.sock"

clean:
- rm -f containerd-shim-aws-firecracker

Expand Down
14 changes: 13 additions & 1 deletion tools/docker/Dockerfile.integ-test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,18 @@ RUN mkdir -p \
/var/run/firecracker-containerd \
/src \
/srv/firecracker_containerd_tests \
${FICD_LOG_DIR}
${FICD_LOG_DIR} \
/etc/cni/net.d


# Pull the images the tests need into the content store so we don't need internet
# access during the tests themselves. This runs as a seperate step before the other
# installs so we can minimize re-runs of the time-expensive downloading of images.
COPY tools/docker/config.toml /etc/containerd/config.toml
COPY tools/docker/do_not_edit_for_firecracker-control.config.json /etc/containerd/firecracker-runtime.json
COPY tools/docker/critest/10-mynet.conf /etc/cni/net.d
COPY tools/docker/critest/99-loopback.conf /etc/cni/net.d

RUN --mount=type=bind,source=firecracker-control/cmd/containerd,target=/src \
make -C /src install && \
ln -sv /usr/local/bin/firecracker-containerd /usr/local/bin/containerd && \
Expand All @@ -40,6 +45,13 @@ RUN containerd 2>/dev/null & \
ctr --address /run/firecracker-containerd/containerd.sock content fetch docker.io/library/alpine:3.10.1 >/dev/null && \
ctr --address /run/firecracker-containerd/containerd.sock content fetch docker.io/mlabbe/iperf3:3.6-r0 >/dev/null

# Install critest
ENV VERSION="v1.23.0"
RUN wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/critest-$VERSION-linux-amd64.tar.gz && \
tar zxvf critest-$VERSION-linux-amd64.tar.gz -C /tmp/ && \
install -D -o root -g root -m755 -t /usr/local/bin /tmp/critest && \
rm -f critest-$VERSION-linux-amd64.tar.gz

# Install everything we need in this image. Due to the bind-mount, if the host has already
# up-to-date versions of everything built, this step will be a very quick copy
RUN --mount=type=bind,target=/src make -C /src \
Expand Down
2 changes: 1 addition & 1 deletion tools/docker/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
imports = ["/etc/containerd/snapshotter/*.toml"]
imports = ["/etc/containerd/snapshotter/*.toml", "/etc/containerd/cri/*.toml"]
[grpc]
address = "/run/firecracker-containerd/containerd.sock"
15 changes: 15 additions & 0 deletions tools/docker/critest/10-mynet.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"cniVersion": "0.2.0",
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.22.0.0/16",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
5 changes: 5 additions & 0 deletions tools/docker/critest/99-loopback.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"cniVersion": "0.2.0",
"name": "lo",
"type": "loopback"
}
42 changes: 42 additions & 0 deletions tools/docker/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ set -e
chmod a+rwx ${FICD_LOG_DIR}

mkdir -p /etc/containerd/snapshotter
mkdir -p /etc/containerd/cri

if [[ -z "$FICD_DM_VOLUME_GROUP" ]]; then
pool_name="${FICD_DM_POOL}"
Expand All @@ -19,6 +20,47 @@ version = 2
base_image_size = "1024MB"
EOF

cat > /etc/containerd/cri/criconfig.toml <<EOF
version = 2
[plugins]
# The 'plugins."io.containerd.grpc.v1.cri"' table contains all of the server options.
[plugins."io.containerd.grpc.v1.cri"]

# 'plugins."io.containerd.grpc.v1.cri".containerd' contains config related to containerd
[plugins."io.containerd.grpc.v1.cri".containerd]

# snapshotter is the snapshotter used by containerd.
snapshotter = "devmapper"

# default_runtime_name is the default runtime name to use.
default_runtime_name = "containerd-shim-aws-firecracker"

# 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes' is a map from CRI RuntimeHandler strings, which specify types
# of runtime configurations, to the matching configurations.
# In this example, 'runc' is the RuntimeHandler string to match.
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.containerd-shim-aws-firecracker]
# runtime_type is the runtime type to use in containerd.
# The default value is "io.containerd.runc.v2" since containerd 1.4.
# The default value was "io.containerd.runc.v1" in containerd 1.3, "io.containerd.runtime.v1.linux" in prior releases.
runtime_type = "aws.firecracker"


# conf_dir is the directory in which the admin places a CNI conf.
# this allows a different CNI conf for the network stack when a different runtime is being used.
cni_conf_dir = "/etc/cni/net.d"

# 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
[plugins."io.containerd.grpc.v1.cri".cni]
# bin_dir is the directory in which the binaries for the plugin is kept.
bin_dir = "/opt/cni/bin"

# conf_dir is the directory in which the admin places a CNI conf.
conf_dir = "/etc/cni/net.d"

[debug]
level = "debug"
EOF

touch ${FICD_CONTAINERD_OUTFILE}
chmod a+rw ${FICD_CONTAINERD_OUTFILE}
/usr/local/bin/containerd --log-level debug &>> ${FICD_CONTAINERD_OUTFILE} &
Expand Down