Basic scorecard test not working #5232
Description
Bug Report
What did you do?
I am trying to implement the basic-spec-check test for operator-sdk scorecard on v0.18.2. Please note that for my use case, operator-sdk version v0.18.2 is a requirement; I am unable to use a newer version at this time.
I am testing my scorecard implementation on the following cluster specs:
[root@my-oc-cluster]# operator-sdk version
operator-sdk version: "v0.18.2", commit: "f059b5e17447b0bbcef50846859519340c17ffad", kubernetes version: "v1.18.2", go version: "go1.13.10 linux/amd64"
In order to run scorecard, I created .osdk-scorecard.yaml in the root with the following info:
scorecard:
output: text
bundle: deploy/olm-catalog/<my>-orchestrator
plugins:
- basic:
cr-manifest:
- "deploy/crds/<my-orchestrator>.com_v1alpha1_installation_cr.yaml"
Then, in order to run the scorecard test, I installed operator-sdk v0.18.2 and ran the scorecard test. Note this is a fresh cluster, so no CRDs exists on the cluster at the moment.
operator-sdk scorecard -o text --selector=test=basic-check-spec-test
After running the above, however, the following error is returned:
[root@my-oc-cluster]# operator-sdk scorecard -o text --selector=test=basic-check-spec-test
INFO[0000] Using config file: /root/<my-bundle>/.osdk-scorecard.yaml
ERRO[0061] Plugin `Basic Tests` failed with error (failed to create namespaced resources: failed to get proxyPod: timed out waiting for the condition)
Basic Tests
CR:
Labels:
Errors:
plugin error
Log:
failed to create namespaced resources: failed to get proxyPod: timed out waiting for the condition:
Logs:
I figured maybe this might be happening because I was using the old osdk-scorecard.yaml method to run scorecard, instead of the newer config.yaml method. To attempt all possibilities, I tried the newer config.yaml method as well, but ran into a separate (different) panic error.
I ran the below command:
operator-sdk scorecard --config=tests/scorecard/config.yaml -o text --selector=test=basic-check-spec-test
and got the following panic error:
[root@my-oc-cluster]# operator-sdk scorecard --config=tests/scorecard/config.yaml -o text --selector=test=basic-check-spec-test
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0xb5bb87]
goroutine 1 [running]:
github.com/spf13/viper.(*Viper).realKey(0x0, 0x223c555, 0x6, 0x6, 0x7fea060fed98)
pkg/mod/github.com/spf13/[email protected]/viper.go:1166 +0x37
github.com/spf13/viper.(*Viper).Set(0x0, 0x223c555, 0x6, 0x1e87fa0, 0xc0003a1ad0)
pkg/mod/github.com/spf13/[email protected]/viper.go:1208 +0x72
github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard.initConfig(0x1f0, 0xc0006fba18, 0x40c1c4)
src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard/cmd.go:138 +0x2d1
github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard.buildScorecardConfig(0xc0006d52c0)
src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard/cmd.go:174 +0x37
github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard.NewCmd.func1(0xc0006d1080, 0xc0003b3600, 0x0, 0x4, 0x0, 0x0)
src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard/cmd.go:62 +0x56
github.com/spf13/cobra.(*Command).execute(0xc0006d1080, 0xc0003b35c0, 0x4, 0x4, 0xc0006d1080, 0xc0003b35c0)
pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0xc0006bedc0, 0x2603f60, 0xc0006d2c00, 0x0)
pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
pkg/mod/github.com/spf13/[email protected]/command.go:887
github.com/operator-framework/operator-sdk/cmd/operator-sdk/cli.RunLegacy(0xc0000ae000, 0x0)
src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/cli/legacy.go:49 +0x11b
main.main()
src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:43 +0x92
As you can see, neither method worked properly.
What did you expect to see?
I expect to see the scorecard basic-spec-check test run successfully.
What did you see instead? Under which circumstances?
Unfortunately, as described above, neither of the two possible methods to invoke the scorecard test worked properly.
Environment
Operator type:
/language go
Kubernetes cluster type:
Openshift$ operator-sdk version
operator-sdk version: "v0.18.2", commit: "f059b5e17447b0bbcef50846859519340c17ffad", kubernetes version: "v1.18.2", go version: "go1.13.10 linux/amd64"
$ go version (if language is Go)
go version go1.15.14 linux/amd64
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1-5-g76a04fc", GitCommit:"c66c03f3012a10f16eb86fdce6330433adf6c9ee", GitTreeState:"clean", BuildDate:"2021-02-13T03:54:59Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0+bd9e442", GitCommit:"bd9e4421804c212e6ac7ee074050096f08dda543", GitTreeState:"clean", BuildDate:"2021-02-11T23:05:38Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Possible Solution
N/A
Additional context
N/A