images: Use docker buildx for kube-cross images and consolidate go-based image Makefiles

[justaugustus]

What type of PR is this?

/kind feature cleanup
/area dependency

What this PR does / why we need it:

• images: Update env to k8s-testimages/gcb-docker-gcloud:v20200824-5d057db

• kube-cross: Enable building via docker buildx

  Use go-runner Makefile pattern for multi-arch

• images: Use shell for loops to "simplify" platform building/pushing

• images: Standardize Makefiles for build images

  Here we create a few "common" Makefiles for image building:

  • images/Makefile.common-image
  • images/build/Makefile.build-image

  Makefile.common-image takes the targets from the go-runner Makefile and
  makes them reusable for both go-runner and kube-cross.

  Makefile.build-image is intended to contain information specific to
  images that will be built in the k8s-staging-build-image GCP project.
  Right now, it only contains the target Google Container Registry.

  images/build/{go-runner,kube-cross}/Makefile now only contains variables
  specific to the image and includes the aforementioned "common" Makefiles
  via the 'include' directive.

  (This is borrowed from how k-sigs/kind handles their image building.)

• go-runner: Drop extraneous pre-build step in Dockerfile

  This seems like it might be causing build failures on s390x.

• images: Fix image manifests annotating 'os/arch' instead of 'arch'

  Use make functions for string substitution instead of sed.
  Frankly, the sed was hard for me for comprehend.

  For amending manifests to the final manifest list, we were previously
  pushing images with the full platform name:

  e.g., "linux/amd64" (OS/architecture) instead of "amd64" (just arch)

  This commit properly strips the OS during manifest list creation.

  (The same issue was fixed for image tags in a previous commit.)

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Accompanying k/test-infra PR is here: kubernetes/test-infra#19176

Does this PR introduce a user-facing change?

- images: Update env to k8s-testimages/gcb-docker-gcloud:v20200824-5d057db

- kube-cross: Enable building via docker buildx

  Use go-runner Makefile pattern for multi-arch
  
- images: Use shell for loops to "simplify" platform building/pushing

- images: Standardize Makefiles for build images
  
  Here we create a few "common" Makefiles for image building:
  - images/Makefile.common-image
  - images/build/Makefile.build-image
  
  Makefile.common-image takes the targets from the go-runner Makefile and
  makes them reusable for both go-runner and kube-cross.
  
  Makefile.build-image is intended to contain information specific to
  images that will be built in the k8s-staging-build-image GCP project.
  Right now, it only contains the target Google Container Registry.
  
  images/build/{go-runner,kube-cross}/Makefile now only contains variables
  specific to the image and includes the aforementioned "common" Makefiles
  via the 'include' directive.
  
  (This is borrowed from how k-sigs/kind handles their image building.)

- go-runner: Drop extraneous pre-build step in Dockerfile
  
  This seems like it might be causing build failures on s390x.

- images: Fix image manifests annotating 'os/arch' instead of 'arch'
  
  Use make functions for string substitution instead of sed.
  Frankly, the sed was hard for me for comprehend.
  
  For amending manifests to the final manifest list, we were previously
  pushing images with the full platform name:
  
  e.g., "linux/amd64" (OS/architecture) instead of "amd64" (just arch)
  
  This commit properly strips the OS during manifest list creation.
  
  (The same issue was fixed for image tags in a previous commit.)

[dims]

/uncc

please add me back when needed.

[justaugustus]

kube-cross single-arch make container works against 7a03bdf.

[justaugustus]

/hold cancel
/assign @dims @saschagrunert @cpanato @hasheddan

This is some cleanup work I'd like to land today ahead of building the go1.15.2 images (which are still building upstream).
cc: @kubernetes/release-engineering

[justaugustus]

/hold need to land a k/test-infra PR first to update the kube-cross build dir. I'll get that up shortly.

[cpanato]

/lgtm
/hold for others to review if needed

[xmudrii]

/lgtm
/hold
for other folks to take a look

[saschagrunert]

/lgtm

[hasheddan]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cpanato, hasheddan, justaugustus, saschagrunert, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justaugustus,saschagrunert]
• images/build/OWNERS [justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justaugustus]

/hold cancel
(kubernetes/test-infra#19176 has merged.)

[mkumatag]

I don't see any of the non-amd64 archs, so wondering whether we really enabled the build for other architecture.

root@ip9-114-192-229:~/k8s_ws/src/k8s.io/kubernetes# manifest-tool inspect k8s.gcr.io/build-image/kube-cross:v1.15.1-2
Name:   k8s.gcr.io/build-image/kube-cross:v1.15.1-2 (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:70e89c3637e40560006cf0cb5939e5fcb88be4db1b547f01f97beffba8caf657
 * Contains 1 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:e8f9ad3015bcf7d2055a362caaa17050a2e8ee6dc3946794ceb704d08defabdc
1  Mfst Length: 3487
1     Platform:
1           -      OS: linux
1           - OS Vers:
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant:
1           - Feature:
1     # Layers: 15
         layer 1: digest = sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7
         layer 2: digest = sha256:c958d65b3090aefea91284d018b2a86530a3c8174b72616c4e76993c696a5797
         layer 3: digest = sha256:edaf0a6b092f5673ec05b40edb606ce58881b2f40494251117d31805225ef064
         layer 4: digest = sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab
         layer 5: digest = sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551
         layer 6: digest = sha256:2c05b70ce67bc243d51420f193483abb3770f85c73566323f6c28027f3436246
         layer 7: digest = sha256:e82ad91b19e9572c9d257303f89252ae6c51d727736d5748f8b6700af2f572a4
         layer 8: digest = sha256:7071d2629af9c4ff405959e7af9bde5f2e370c3edfa69c48e583f0e87a25581e
         layer 9: digest = sha256:1a251bb545e22943616ca5d596d3b6ab75c915f4a033997ff9bd8f456ebfa7fe
         layer 10: digest = sha256:31ef8af6d16405e737eb1f2f0fcde7f2c69ec39f553b5eeae24b95bab0c1edf5
         layer 11: digest = sha256:0f4f85a9f6cf652b79ef5d8e581d4ac61bbd21f1deafeef479ab1b5e6acdc70c
         layer 12: digest = sha256:06b05205e60a9dcfbb7aa0896eef0885d66ec3b0307223fe2757acec3121f1a8
         layer 13: digest = sha256:869778081ebc00f1ee82786ac705cbf0fb944abeee8a51b60ad3a8ba404e0b47
         layer 14: digest = sha256:47ed7561362d0fc643986b334ebf7aaff0fc79a017332a70cd004f00135da13f
         layer 15: digest = sha256:23dc6a647c78a8c3b0b72b9a117d94e543a0d8b5f1d8316971a22dc957ed8d29

root@ip9-114-192-229:~/k8s_ws/src/k8s.io/kubernetes#

[justaugustus]

@mkumatag -- we haven't published multi-arch images for kube-cross yet. This was merely the next step in the process. There are some hardcoded linux/amd64 references that I need to unfurl first.