How to migrate to modules for operator project that uses dep? #1810

jsanda · 2019-08-09T20:23:26Z

Type of question

building with operator-sdk

Question

I have a project that was that currently uses dep and all dependencies are declared in Go.toml. I am trying to migrate to Go modules but not clear on how exactly to do this. Are there documented steps for this?

#1566 mentions that new projects should use modules, but the PR does not mention what to do for existing projects that already use a vendor directory.

The text was updated successfully, but these errors were encountered:

jmrodri · 2019-08-10T02:10:27Z

@jsanda Hey John, thanks for the question. I looked and didn't see anything either. I'll see what can be done.

jsanda · 2019-08-10T02:53:52Z

Hi @jmrodri. Nice to hear from you and thanks for helping out :)

The project is currently using v0.7.0 of operator-sdk. I was working on upgrading to v0.9.0. This resulted in some changes to the project's CircleCI configuration which we might be able to circumvent by migrating to modules. I have run into various dependency/build issues when I trying moving to module at v0.7.0 as well as at v0.9.0. I suspect that some of those bumps might be unavoidable so any guidance is greatly appreciated.

jsanda · 2019-08-10T17:18:29Z

Let me share what I m hitting in case it is of any help:

$ export GO111MODULE=on 
$ go mod init
go: creating new go.mod: module github.com/Orange-OpenSource/cassandra-k8s-operator
go: copying requirements from Gopkg.lock

$ operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [db:[v1alpha1], ]
F0810 13:07:52.075636   69813 deepcopy.go:885] Hit an unsupported type invalid type for invalid type, from github.com/Orange-OpenSource/cassandra-k8s-operator/pkg/apis/db/v1alpha1.CassandraCluster
goroutine 1 [running]:
k8s.io/klog.stacks(0xc002f14100, 0xc00030a480, 0xc5, 0x115)
	pkg/mod/k8s.io/[email protected]/klog.go:855 +0xb1
k8s.io/klog.(*loggingT).output(0x3ffc080, 0xc000000003, 0xc000329c00, 0x3e3ba28, 0xb, 0x375, 0x0)
	pkg/mod/k8s.io/[email protected]/klog.go:806 +0x2d9
k8s.io/klog.(*loggingT).printf(0x3ffc080, 0x3, 0x29c3d82, 0x2a, 0xc002484e88, 0x3, 0x3)
	pkg/mod/k8s.io/[email protected]/klog.go:705 +0x14e
k8s.io/klog.Fatalf(...)
	pkg/mod/k8s.io/[email protected]/klog.go:1256
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).doStruct(0xc002f00080, 0xc002eb0160, 0xc002eca730)
	pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:885 +0x7bf
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).generateFor(0xc002f00080, 0xc002eb0160, 0xc002eca730)
	pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:695 +0xc5
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).GenerateType(0xc002f00080, 0xc002dff380, 0xc002eb0160, 0x2c99d40, 0xc002f02360, 0x0, 0x0)
	pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:608 +0xe86
k8s.io/gengo/generator.(*Context).executeBody(0xc002dff380, 0x2c968c0, 0xc002ef2650, 0x2d0f5a0, 0xc002f00080, 0x60, 0x10)
	pkg/mod/k8s.io/[email protected]/generator/execute.go:304 +0x11d
k8s.io/gengo/generator.(*Context).ExecutePackage(0xc002dff1a0, 0xc0002f2ba0, 0x24, 0x2ce7ae0, 0xc002ebff80, 0x0, 0x0)
	pkg/mod/k8s.io/[email protected]/generator/execute.go:265 +0xbf7
k8s.io/gengo/generator.(*Context).ExecutePackages(0xc002dff1a0, 0xc0002f2ba0, 0x24, 0xc002ef07f0, 0x1, 0x1, 0x0, 0xc00043b290)
	pkg/mod/k8s.io/[email protected]/generator/execute.go:51 +0xc5
k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc0005685a0, 0xc002485810, 0x296a2c4, 0x6, 0x2a5f9d8, 0xc00014a820, 0x16)
	pkg/mod/k8s.io/[email protected]/args/args.go:194 +0x2d7
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.deepcopyGen(0xc0000c7900, 0x3a, 0xc00034d3a0, 0x1, 0x1, 0x18, 0xc0006cb9b0)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:94 +0x4d2
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen.func1(0xc0000c7900, 0x3a, 0xc0000c7900, 0x3a)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:53 +0x50
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.generateWithHeaderFile(0xc000281c60, 0x0, 0x0)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/genutil.go:104 +0x178
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen(0xc0005b0c80, 0x0)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:54 +0x4a5
github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate.k8sFunc(0xc0005b0c80, 0x401fc80, 0x0, 0x0, 0x0, 0x0)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate/k8s.go:56 +0x156
github.com/spf13/cobra.(*Command).execute(0xc0005b0c80, 0x401fc80, 0x0, 0x0, 0xc0005b0c80, 0x401fc80)
	pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002fb680, 0x2c9b2e0, 0xc0002f7b00, 0x0)
	pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
	pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:85 +0x50f

jsanda · 2019-08-10T19:51:52Z

kubernetes/kubernetes#79149 may be relevant.

camilamacedo86 · 2019-08-11T10:03:09Z

HI @jsanda,

It https://blog.callr.tech/migrating-from-dep-to-go-1.11-modules/ may help you with.

joelanford · 2019-08-13T18:00:06Z

@jsanda I'd typically recommend using the operator-sdk print-deps command, but that seems to be broken for go modules at the moment.

EDIT: you should be able to use operator-sdk print-deps --as-file which should output the v0.9.0 go.mod file on stdout.

Instead you may be able to piece things together using our go.mod file template that operator-sdk new scaffolds.

operator-sdk/internal/pkg/scaffold/go_mod.go

Lines 40 to 92 in 560208d

    
           require ( 
        
           	contrib.go.opencensus.io/exporter/ocagent v0.4.9 // indirect 
        
           	github.com/Azure/go-autorest v11.5.2+incompatible // indirect 
        
           	github.com/appscode/jsonpatch v0.0.0-20190108182946-7c0e3b262f30 // indirect 
        
           	github.com/coreos/prometheus-operator v0.26.0 // indirect 
        
           	github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect 
        
           	github.com/emicklei/go-restful v2.8.1+incompatible // indirect 
        
           	github.com/go-logr/logr v0.1.0 // indirect 
        
           	github.com/go-logr/zapr v0.1.0 // indirect 
        
           	github.com/go-openapi/spec v0.18.0 // indirect 
        
           	github.com/golang/groupcache v0.0.0-20180924190550-6f2cf27854a4 // indirect 
        
           	github.com/golang/mock v1.2.0 // indirect 
        
           	github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c // indirect 
        
           	github.com/google/uuid v1.0.0 // indirect 
        
           	github.com/googleapis/gnostic v0.2.0 // indirect 
        
           	github.com/gophercloud/gophercloud v0.0.0-20190318015731-ff9851476e98 // indirect 
        
           	github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect 
        
           	github.com/grpc-ecosystem/grpc-gateway v1.8.5 // indirect 
        
           	github.com/imdario/mergo v0.3.6 // indirect 
        
           	github.com/operator-framework/operator-sdk master 
        
           	github.com/pborman/uuid v0.0.0-20180906182336-adf5a7427709 // indirect 
        
           	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect 
        
           	github.com/spf13/pflag v1.0.3 
        
           	go.opencensus.io v0.19.2 // indirect 
        
           	go.uber.org/atomic v1.3.2 // indirect 
        
           	go.uber.org/multierr v1.1.0 // indirect 
        
           	go.uber.org/zap v1.9.1 // indirect 
        
           	golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2 // indirect 
        
           	k8s.io/apimachinery v0.0.0-20190221213512-86fb29eff628 
        
           	k8s.io/client-go v2.0.0-alpha.0.0.20181126152608-d082d5923d3c+incompatible 
        
           	k8s.io/kube-openapi v0.0.0-20190603182131-db7b694dc208 // indirect 
        
           	k8s.io/kube-state-metrics v1.6.0 // indirect 
        
           	sigs.k8s.io/controller-runtime v0.1.12 
        
           	sigs.k8s.io/controller-tools v0.1.10 
        
           	sigs.k8s.io/testing_frameworks v0.1.0 // indirect 
        
           ) 
        
           // Pinned to kubernetes-1.13.4 
        
           replace ( 
        
           	k8s.io/api => k8s.io/api v0.0.0-20190222213804-5cb15d344471 
        
           	k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.0.0-20190228180357-d002e88f6236 
        
           	k8s.io/apimachinery => k8s.io/apimachinery v0.0.0-20190221213512-86fb29eff628 
        
           	k8s.io/client-go => k8s.io/client-go v0.0.0-20190228174230-b40b2a5939e4 
        
           ) 
        
           replace ( 
        
           	github.com/coreos/prometheus-operator => github.com/coreos/prometheus-operator v0.29.0 
        
           	sigs.k8s.io/controller-runtime => sigs.k8s.io/controller-runtime v0.1.12 
        
           	sigs.k8s.io/controller-tools => sigs.k8s.io/controller-tools v0.1.11-0.20190411181648-9d55346c2bde 
        
           	k8s.io/kube-state-metrics => k8s.io/kube-state-metrics v1.6.0 
        
           ) 
        
           replace github.com/operator-framework/operator-sdk => github.com/operator-framework/operator-sdk v0.9.0

jsanda · 2019-08-14T01:39:52Z

Thanks for the reference @camilamacedo86. It was helpful.

Since I am upgrading a project on v0.7.0 I decided to create the example project at https://github.com/operator-framework/operator-sdk/tree/v0.7.0 and upgrade from it to see what happens. I was able to reproduce the error, but I think it was because I needed to run go mod tidy.

I then went back through the example app a second time and successfully upgraded. Here is a summary of my steps:

Create operator project, operator-sdk new app-operator
Generate api, operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService
Add controller, operator-sdk add controller --api-version=app.example.com/v1alpha1 --kind=AppService
Upgrade to v0.9.0 of operator-skd and confirm by running operator-sdk version
Update version operator-sdk in Go.toml
go mod init
go mod tidy
rm -rf vendor/
rm -f Gopkg.lock Gopkg.toml
operator-sdk generate k8s

The first time I went through those steps but left out go mod init and the project failed to build after switching to modules and upgrading to v0.9.0.

I still have some errors in my project to resolve, but I think it is just a matter of sorting out some dependencies.

@joelanford thanks for the tip! Should help with resolving my errors.

estroz · 2019-08-15T16:47:31Z

@jsanda #1819 allows you to force dep manager type using --dep-manager, so you can migrate your project to the correct dep manager:

$ operator-sdk print-deps --dep-manager modules > go.mod
$ rm Gopkg.toml Gopkg.lock
$ export GO111MODULES=on
$ operator-sdk build my-image

Note this only works with master until we release again, and any project v0.10+ as deps have changed (slightly) over the last few releases. It may still work with a v0.9.0 operator though.

jmrodri added the kind/documentation Categorizes issue or PR as related to documentation. label Aug 10, 2019

jsanda mentioned this issue Aug 10, 2019

upgrade operator-sdk to v0.9.0 Orange-OpenSource/casskop#88

Merged

joelanford self-assigned this Aug 13, 2019

estroz mentioned this issue Aug 13, 2019

*: print-deps prints file of user's choice or by top-level file; remove --as-file #1819

Merged

jsanda closed this as completed Aug 14, 2019

jpkrohling mentioned this issue Aug 22, 2019

Deepcopy generation fails due to "unsupported type invalid type for invalid type" #1854

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to migrate to modules for operator project that uses dep? #1810

How to migrate to modules for operator project that uses dep? #1810

jsanda commented Aug 9, 2019 •

edited

Loading

jmrodri commented Aug 10, 2019

jsanda commented Aug 10, 2019

jsanda commented Aug 10, 2019

jsanda commented Aug 10, 2019

camilamacedo86 commented Aug 11, 2019

joelanford commented Aug 13, 2019 •

edited

Loading

jsanda commented Aug 14, 2019

estroz commented Aug 15, 2019 •

edited

Loading

How to migrate to modules for operator project that uses dep? #1810

How to migrate to modules for operator project that uses dep? #1810

Comments

jsanda commented Aug 9, 2019 • edited Loading

Type of question

Question

jmrodri commented Aug 10, 2019

jsanda commented Aug 10, 2019

jsanda commented Aug 10, 2019

jsanda commented Aug 10, 2019

camilamacedo86 commented Aug 11, 2019

joelanford commented Aug 13, 2019 • edited Loading

jsanda commented Aug 14, 2019

estroz commented Aug 15, 2019 • edited Loading

jsanda commented Aug 9, 2019 •

edited

Loading

joelanford commented Aug 13, 2019 •

edited

Loading

estroz commented Aug 15, 2019 •

edited

Loading