question: how does it work?

[streamnsight]

Hello,

Awesome project!

I have a question before spending the time using it:
I deployed k8s on CoreOs before: one would setup a cluster, which uses etcd for discovery: you need 3 nodes+ to get started. Flannel then uses etcd for communication in the cluster, and then docker runs using flannel as its networking layer. Once docker runs then k8s can use it to deploy / orchestrate containers.

So i'm not sure to understand how this project works: you're running k8s inside docker, that means docker needs to be up before you even setup etcd, flannel, docker and then k8s on top.
Are you running docker in docker, with the lower level docker some priviledge container using the host networking layer?

[luxas]

Hello @streamnsight!
I can clearly understand your confusion, it took me some month to get understanding fully in this topic.
This is how kubernetes-on-arm works:

First time only

• Write the sd card
• On first boot, run kube-config install. That installs everything Kubernetes needs.
• Reboot

Every other case:

• The node starts up
• If Kubernetes is enabled by kube-config earlier, everything starts up by automatically
  • Otherwise, run kube-config enable-master or kube-config enable-worker, depending on whether if you have setup it before
• Step one: system-docker starts up. This is a system docker daemon, which runs services that the main docker daemon needs.
• Step two: etcd starts up on system-docker. It uses the /var/lib/etcd directory as data store.
• Step three: flannel starts up on system-docker. It uses etcd as data store for the FLANNEL_SUBNET. Outputs a subnet file to /var/lib/flannel/subnet.env.
• Step four: Now the main docker daemon is started. It uses the flannel subnet file.
• Step five: The k8s-(master|worker) service is started. These services contains two containers at the moment: kubelet and kube-proxy. kube-proxy just manages the network, separately of the other things.
• Step six: kubelet starts up and if no master services is running (and it is a master) it will start the three master containers: apiserver, controller-manager and scheduler.

After that Kubernetes is up and running
Upstream docs: https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker-multinode.md

What is hyperkube?
It's Kubernetes all-in-one binary. It includes kubelet, kube-proxy, apiserver, controller-manager and scheduler.
So, instead of maintaining five different binaries (and docker images) one is much easier.

Why do one want this system? It's so complicated!
This issue discusses it: kubernetes/kubernetes#13424 (comment)
This is a very flexible system, package the binaries and other required things into one docker image, and you don't have to think about which OS the container then lands on.

It's also very easy to tear down:
kube-config disable stops Kubernetes
docker rmi $(docker images -q) removes all docker images and includes the kubernetes images.

What is the difference between kube-config and kubectl?
The difference is that kubectl manages the cluster, which pods you run... Basicly everything that Kubernetes has to offer.

kube-config is my own utility for installing Kubernetes on ARM, it starts and stops Kubernetes on a node, makes it possible to build the binaries yourself, outputs some machine info and installs the required packages for running Kubernetes.

Hope this helps! 😄

[yorele]

Thanks!
my only remaining confusion is with etcd: you say it uses /var/lib/etcd for storage but don't you need 3 nodes to even get it started?

Other question about building: all this happens on the Pi, right? Have you looked at cross compiling binaries on another machine to make things faster?
Can I up the version numbers in the version file to get latest k8s? or is there known issues? (i'd like to use v1.1.1)

Thanks

[luxas]

Hello @yorele

No, etcd doesn't have to be clustered, it works fine with just one instance (it elects itself as leader)

Yes, everything builds on the Pi
The only way I think could make it faster is to use a Scaleway machine.
AFAIK, it's not possible to cross-compile dynamically. This makes sense because of cAdvisor: kubernetes/kubernetes#8772

Yes, there are known issues with v1.1.1 and higher when building on non-amd64 platforms: kubernetes/kubernetes#17060
But, it shouldn't be hard to patch that on a separate patch branch.
And yeah, you only have to edit the value in images/version.sh

I'm working on a quite big release just now, v0.6.0, here I described some of the new features: #2 (comment)

[streamnsight]

do i understand that this recent PR to k8s kubernetes/kubernetes#13901 (comment) is meant to take over some of your deployment steps?

[luxas]

@streamnsight Yes and no.
When this gets merged, it won't affect this project at all.
However, I always strive to share as much as possible code with core Kubernetes.
So maybe something will be common.

But you are free to use any deployment strategy, of course :)

[luxas]

@streamnsight Just be aware that master is equal to v0.5.8, while dev the branch is where I push new (sometimes unstable, used for testing) code. When I consider dev ready, it will be merged to master as v0.6.0

[streamnsight]

@luxas
Seems like it should be possible to compile on x86 or x64 using qemu, even in docker
Check out these 2 projects:
https://github.com/dweinstein/dockerfile-qemu-arm-chroot
https://github.com/moul/docker-binfmt-register
The second one is the easiest to get started with qemu in docker, but you'll need to put you data in a volume as explained in the first one.
I got my ARM images to run on my ubuntu x64 machine in a few minutes.

However, at this point, compiling kubernetes does not work because of a bug in go 1.5, but it seems like it should be fixed in 1.5.2 (see golang/go#13024)

[luxas]

Thanks for the links.
I will investigate it.
Yes, I agree that it would be nice if one could compile this on amd64 machines.

On 16 Nov 2015, at 20:54, streamnsight notifications@github.com wrote:

@luxas
Seems like it should be possible to compile on x86 or x64 using qemu, even in docker
Check out these 2 projects:
https://github.com/dweinstein/dockerfile-qemu-arm-chroot
https://github.com/moul/docker-binfmt-register
The second one is the easiest to get started with qemu in docker, but you'll need to put you data in a volume as explained in the first one.
I got my ARM images to run on my ubuntu x64 machine in a few minutes.

However, at this point, compiling kubernetes does not work because of a bug in go 1.5, but it seems like it should be fixed in 1.5.2 (see golang/go#13024)

—
Reply to this email directly or view it on GitHub.

[luxas]

@yorele Now I've released v0.6.0 which features v1.1.2 k8s as you requested.

[luxas]

@streamnsight Now I've made it possible to compile Kubernetes for ARM on a amd64 machine.
https://gist.github.com/luxas/e319b314f22b18243b1d

Thanks for the links you gave me earlier. They helped.

I'll close this now. If you have additional questions, just ask :)

[streamnsight]

awesome... thanks!

[lavvy]

Awesome Christmas gift

awesome... thanks!

—
Reply to this email directly or view it on GitHub
#16 (comment)
.