Cloud-init 18.4 is needed on CentOS 7 for templating used by CAPA and CAPV

[akutz]

/kind feature

Describe the solution you'd like
@detiber informed me that in v1alpha2 the fork (18.3-52-gc5f78957-1~bddeb~18.04.1) of cloud-init will no longer be used by the bootstrap controller. Instead the decision was made to use the runcmd. I would be thrilled as I originally questioned the value of the kubeadm module over the runcmd, except I've come to appreciate Jason's fork since it is required on CentOS 7. Cloud-init on CentOS 7 is slow to update to newer versions, and the most recent version on CentOS 7.6.1810 is cloud-init 18.2.

Since cloud-init 18.4+ is required for JINJA templating, used by both CAPA and CAPV and presumably the new bootstrap controller, CentOS 7 will be incompatible with v1alpha2.

And before someone says "But Jason's fork is based on cloud-init 18.3, and it works with templating," yes, yes it does. But that's the result of a terribly happy accident. Jason just so happened to have forked cloud-init off of master between 18.3 and 18.4 after templating was merged into the master branch.

The sites rpm2find and pkgs.org have lists of the RPMs for cloud-init, and it's unclear if any of the newer packages are compatible with CentOS 7.

For machine images based on CentOS 7 to work with v1alpha2, someone needs to either fork cloud-init 18.4+ (ideally 19.1, the latest release) for CentOS 7, or document how to install cloud-init 18.4+ on CentOS 7.

[akutz]

Hi @ncdc,

I filed this as a feature, but it could also be seen as a regression from v1alpha1. I can refile it if you like.

[neolit123]

while re-inventing the wheel is probably something we should not be doing, it seems to me cloud-init can be problematic. i had concerns about it, as a bystander, back when CAP* decided to go for it, but didn't have anything concrete to bring up.

are there plans to move away from cloud-init completely or search for alternatives (i don't think there are that many)?

[akutz]

Hi @neolit123,

I think cloud-init is fine. It's the distro that lags behind. And the fact is, any alternative is going to introduce an entirely new set of problems. Cloud-init is pervasive and works well. It's an issue of CentOS, and I'm about to post a Dockerfile that builds the RPM we need.

[akutz]

FWIW, I have a lot of experience with cloud-init and Ignition, and cloud-init is very mature and works very well. Anything that spans most Linux distributions is going to encounter occasional issues across versions of a given distribution.

[neolit123]

Anything that spans most Linux distributions is going to encounter occasional issues across versions of a given distribution.

that is true. i common solution is to use the lowest version possible that conforms all distros, but if a new feature is really required a fork will be needed.

[akutz]

if a new feature is really required a fork will be needed.

Except it's not new per se as @detiber's fork in v1alpha1 already has it. This is kind of a regression IMO. Anyway, I'm producing a Dockerfile that can create the RPM for CentOS from the cloud-init repo. At least that way the provenance is ensured.

[akutz]

Hi @neolit123,

This branch of CAPV includes a Dockerfile that can be used to build cloud-init for CentOS 7:

$ docker run -it --rm \
  -v "$(pwd):/build" \
  gcr.io/cluster-api-provider-vsphere/extra/cloud-init/centos \
  /bin/bash -c 'make rpm && cp *.rpm /build'
python2 ./packages/brpm --distro=redhat
Archived the code in u'/tmp/rpmbuilddlXYQk/rpmbuild/SOURCES/cloud-init-19.1-24-g07b17236.tar.gz'
Created spec file at '/tmp/rpmbuilddlXYQk/rpmbuild/SPECS/cloud-init.spec'
Running 'rpmbuild' in '/tmp/rpmbuilddlXYQk/rpmbuild'
Wrote out redhat package '/cloud-init/cloud-init-19.1+24.g07b17236-1.el7.noarch.rpm'
Wrote out redhat package '/cloud-init/cloud-init-19.1+24.g07b17236-1.el7.src.rpm'

That will create two RPMs for cloud-init in the current, local directory:

$ ls *.rpm
-rw-r--r--  1 akutz  staff   881K Jul 17 18:51 cloud-init-19.1+24.g07b17236-1.el7.noarch.rpm
-rw-r--r--  1 akutz  staff   1.0M Jul 17 18:51 cloud-init-19.1+24.g07b17236-1.el7.src.rpm

Anyway, we can potentially use something like this to create a package for CentOS if we cannot find a standard one.

[akutz]

Please bear in mind that I haven't had time to test this yet. I'll do that later tonight or tomorrow morning.

[moshloop]

I think there are multiple problems with this:

• Using a forked/self-built package as critical as cloud-init has implications for the security and supportability of the resulting images. I am pretty sure doing this will invalidate any support contracts with the OS vendor e.g. Canonical, RedHat, Amazon (for Amazon Linux), By forking the package we also to take responsibility for fixing CVE's within 24 - 48 hours of release.

• Using the least common set of features has implications for the different operating systems and versions this provider can support, especially if there are no E2E tests - Ignition based OS's (while unsupported today, have a fairly simple path to adding support if the least common denomination is used)

Templating can be handled in multiple different ways without implications on supportability and security:

• bash script
• go helper app
• adding templating supporting directly into kubeadm

[akutz]

I’m not building from a fork. It’s built using the cloud init process. They just don’t provide a package.

Additionally, cloud init is a mature, scalable, tested, and extensible solution, and providing an alternative is not as easy as you make it sound. As you said, this is a critical piece of the bootstrap process and should not be removed just because it needs to be built.

Finally, the templating aspect is only possible because of the data generated by cloud init. If it is missing then the data isn’t there in the first place. I can host a call to explain how it works for those interested.

[michaelgugino]

My expectation is we support the major distributions without having to override system packages. It would be most helpful to have a clear proposal outlining the things we need cloud-init to do, and then evaluate what might be better alternatives (whether using features supported by all cloud-inits, or having cloud-init just execute some helper script/bin).

[moshloop]

As far as I know, each distribution creates their own "fork" of cloud-init with config specific to them, so by building from upstream directly, we are deviating from the norm.

CVE's that are under security embargo would also first be patched first into the forks, and only once the embargo is lifted patched upstream.

Instead of cloud-init generating data in the pre-network phase (so that it can be used to configure the network), interfaces can be brought up post the cloud-init networking phase in runcmd.

[akutz]

Instead of cloud-init generating data in the pre-network phase (so that it can be used to configure the network), interfaces can be brought up post the cloud-init networking phase in runcmd.

No it cannot. Has to happen in the local stage. I can host a call to describe the workflow for anyone interested.

Additionally, let’s please focus this issue on cloud init 18.4+ for CentOS. Discussing alternate bootstrap methods should be its own issue.

[moshloop]

Has to happen in the local stage

Why ? I know it is definitely possible as that was the approach I used to bring up multiple interfaces with different VLAN's / MTU's which netplan 2 doesn't handle well.