*: automate releases with goreleaser #4034

estroz · 2020-10-14T17:46:27Z

Description of the change:

.goreleaser.yml: goreleaser config file
.ci/gpg: GPG public key, encrypted private subkey (used for signing), and script to decrypt private subkey in CI
website/scripts: scripts to update versions in release commits and branches at build time
website/content/en/docs: release and installation guide simplifications, related docs cleanup
*: script and Makefile updates to support automated releases

Motivation for the change: This PR adds goreleaser and tooling for automated releases in CI, agnostic of CI platform (albeit with some environment configuration). Any Github user with write capabilities to the SDK repo can start a release by doing git tag vX.Y.Z && git push --tags.

Closes #4114

Checklist

If the pull request includes user-facing changes, extra documentation is required:

Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
Add or update relevant sections of the docs website in website/content/en/docs

estroz · 2020-10-20T21:06:42Z

release/Makefile

+
+.PHONY: changelog
+changelog: check_tag ## Generate the changelog.
+	mkdir -p changelog/generated


@joelanford goreleaser needs a file containing release notes if we're going to use a custom changelog generator, so the idea is to create a new changelog file per release under changelog/generated, much like we do with migration guides. This file will be committed in the release commit prior to running goreleaser, much as we do with CHANGELOG.md now.

This file will be committed in the release commit prior to running goreleaser, much as we do with CHANGELOG.md now.

Sounds good while we have the need for a pre-release commit.

Thinking more in the future, if we're able to get rid of the pre-release commit would it be possible to generate it on the fly during the tag CI run just before running goreleaser? It might take some non-trivial changes to our current changelog fragment and generation process, but I think that would be on the table if we could get to the point of not needing pre-release commits.

if we're able to get rid of the pre-release commit would it be possible to generate it on the fly during the tag CI run just before running goreleaser

We could get rid of committing the changelog entirely, if that's desirable, right now (I think it is since we publish release notes).

I take ^ back since we still need to remove fragments in the pre-release commit.

Yeah, maybe let's make a separate issue to investigate options for changelog/release notes generation without a pre-release commit.

.goreleaser.yml

joelanford · 2020-10-20T21:36:53Z

.goreleaser.yml

+
+# We don't use archives, so skip creating them.
+archives:
+  - format: binary


I think we may need the following to make the binaries have the same naming scheme we've been using:

Suggested change

- format: binary

- format: binary

replacements:

amd64: x86_64

darwin: apple-darwin

linux: linux-gnu

arm64: aarch64

name_template: '{{.Binary}}-{{.Env.GIT_VERSION}}-{{.Arch}}-{{.Os}}'

One caveat: I noticed that our arm64 binary release on github use aarch64 and our images in quay use arm64. They have the same meaning, just a different name. But I think at some point we might want to align those if that's possible as a non-breaking change.

Is it a breaking change to modify the released binary names? If not I'd prefer to use those defined by Golang going forward.

Also removing version from binary names removes one more doc we have to change in the release commit (there are other ways around the latter issue, but this is the simplest).

I'm okay using the Golang arch and os for the binary names.

release/Makefile

estroz · 2020-10-22T05:17:21Z

I need to test a full release on my operator-sdk fork before merging this, and add a PR description.

estroz · 2020-10-22T17:28:35Z

netlify.toml

+  command = """
+git submodule update -f --init themes/docsy && \
+npm install postcss-cli autoprefixer@^9.0.0 && \
+./scripts/set_version.sh && \


@asmacdo this script updates version_menu = "Releases" to version_menu = "vX.Y" at build time if the branch matches vX.Y.x. This should render the updated menu as it currently is now, but without us having to commit this change, right?

website/content/en/docs/installation/install-operator-sdk.md

estroz · 2020-11-03T21:24:01Z

Successful release: https://travis-ci.org/github/estroz/operator-sdk/builds/741229073
Published release from job (using my Github key): https://github.com/estroz/operator-sdk/releases/tag/v0.33.0

estroz · 2020-11-03T22:20:35Z

/cc @asmacdo @rashmigottipati @theishshah

camilamacedo86 · 2020-11-05T14:06:53Z

.travis.yml

@@ -3,7 +3,7 @@ dist: xenial

 language: go
 go:
- 1.15.x
+- 1.15.2


why make it fixed to 1.15.2?

Reproducible builds, see #4034 (comment). I am reconsidering this decision though.

For doing builds, I think pinning makes sense. CI should use the latest one so that if a specific one causes a build failure we catch it in the PR before we do a release. I would hate to see a release fail because of an updated go compiler (though I suspect it would be rare, not something I'd want to debug when it happens).

tl;dr +1 to pinning

camilamacedo86 · 2020-11-05T14:07:43Z

.travis.yml

+  - name: check
+    if: type == pull_request
+  - name: test
+    if: type == pull_request


Will we no longer run the check/test on master?

Correct. We already run PR tests, so testing master post-merge is redundant.

I only think that we need to run the test/check when we push the release tag. I mean to generate the tag. Otherwise, +1 for me. However, in the latest release meeting, this idea was not accepted. @joelanford @jmrodri are you ok with as well

I'm okay with not running them on master anymore.

website/content/en/docs/installation/install-operator-sdk.md

.ci/gpg/create-keyring.sh

joelanford · 2020-11-05T14:25:56Z

tools/scripts/fetch

+      [[ -z "$ver" ]] || (echo "cannot set goreleaser version" && exit 1)
+      ver="${DEST}/goreleaser"
+      ver_cmd="command -v ${DEST}/goreleaser"
+      fetch_cmd="curl -sfL https://install.goreleaser.com/github.com/goreleaser/goreleaser.sh | sh -s -- -b ${DEST}"


The goreleaser install script allows specifying a tag:

Suggested change

[[ -z "$ver" ]] || (echo "cannot set goreleaser version" && exit 1)

ver="${DEST}/goreleaser"

ver_cmd="command -v ${DEST}/goreleaser"

fetch_cmd="curl -sfL https://install.goreleaser.com/github.com/goreleaser/goreleaser.sh | sh -s -- -b ${DEST}"

ver_cmd="${DEST}/goreleaser --version 2>/dev/null | grep version | cut -d' ' -f3"

fetch_cmd="curl -sSfL https://install.goreleaser.com/github.com/goreleaser/goreleaser.sh | sh -s -- -b \"${DEST}\" -d \"v${ver}\""

camilamacedo86 · 2020-11-05T21:40:16Z

website/content/en/docs/contribution-guidelines/developer-guide.md

-```sh
-$ make cli-doc
-```
-


website/content/en/docs/contribution-guidelines/releasing.md

.ci/gpg/create-keyring.sh

jmrodri

The configuration for goreleaser and travis seem sane. Pinning golang makes sense. I like the GPG stuff, kudos for that. The Makefile looks good too.

My biggest request come with the release document. I think we should put the instructions directly in each step and not link to other parts of the doc which makes it hard to keep track of what step you're own. And I was wondering if we could add the git add step of the prerelease to the makefile which is one less command the release person has to run.

jmrodri · 2020-11-18T20:25:14Z

.ci/gpg/create-keyring.sh

+echo -e "\nDecrypting secret key..."
+{
+  # $GPG_PASSWORD is taken from the script's env (injected by Travis CI).
+  echo $GPG_PASSWORD | gpg --decrypt \
+    --pinentry-mode loopback --batch \
+    --passphrase-fd 0 \
+    --output "${SECRING_AUTO}" \
+    "${SECRING_AUTO}".gpg ; \
+} || { err_exit "Failed to decrypt secret key." ; }
+echo "Success!"


Exactly what we used to do in RHN back in the day when we stored our encrypted keys in CVS. I approve.

jmrodri · 2020-11-18T20:37:22Z

.goreleaser.yml

+    asmflags: *build-asmflags
+    gcflags: *build-gcflags
+    ldflags: *build-ldflags
+    targets: *build-targets


I like this reuse of the flags and the fact that if we ever need to change one just for one of the builds, we still have the flexibility to update the flags in this section.

jmrodri · 2020-11-18T20:45:41Z

.goreleaser.yml

+
+# We don't use archives, so skip creating them.
+archives:
+  - format: binary


I'm okay using the Golang arch and os for the binary names.

jmrodri · 2020-11-18T20:47:31Z

.travis.yml

@@ -3,7 +3,7 @@ dist: xenial

 language: go
 go:
- 1.15.x
+- 1.15.2


For doing builds, I think pinning makes sense. CI should use the latest one so that if a specific one causes a build failure we catch it in the PR before we do a release. I would hate to see a release fail because of an updated go compiler (though I suspect it would be rare, not something I'd want to debug when it happens).

jmrodri · 2020-11-18T20:47:41Z

.travis.yml

@@ -3,7 +3,7 @@ dist: xenial

 language: go
 go:
- 1.15.x
+- 1.15.2


tl;dr +1 to pinning

jmrodri · 2020-11-18T22:35:14Z

website/content/en/docs/contribution-guidelines/releasing.md

+git add --all
+git commit -m "Release v1.3.1"
+git push -u origin release-v1.3.1


Can we do the add as part of the prerelease target, and then printout the git status or git diff from the Makefile.
That way I just have to review, then commit and push.

Will add in a follow-up.

website/content/en/docs/contribution-guidelines/releasing.md

agnostic of CI platform (albeit with some environment configuration). Any Github user with write capabilities to the SDK repo can start a release. .ci/gpg: GPG public key, encrypted private subkey (used for signing), and script to decrypt private subkey in CI website/scripts: scripts to update versions in release commits and branches at build time .goreleaser.yml: goreleaser config file *: script and Makefile updates to support automated releases docs: release and installation guide simplifications, related docs cleanup

estroz · 2020-11-30T21:31:31Z

We've effectively moved away from travis, so some of the CI hookup stuff is no longer valid. I plan on making a follow-up for porting the release job to GH actions.

jmrodri

/lgtm

This commit adds tooling for automated releases in CI using goreleaser, agnostic of CI platform (albeit with some environment configuration). Any Github user with write capabilities to the SDK repo can start a release. .ci/gpg: GPG public key, encrypted private subkey (used for signing), and script to decrypt private subkey in CI website/scripts: scripts to update versions in release commits and branches at build time .goreleaser.yml: goreleaser config file *: script and Makefile updates to support automated releases docs: release and installation guide simplifications, related docs cleanup Signed-off-by: reinvantveer <[email protected]>

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 14, 2020

openshift-ci-robot requested review from jberkhahn and marc-obrien October 14, 2020 17:46

estroz mentioned this pull request Oct 14, 2020

Use multi-stage Dockerfiles for all images #3984

Closed

estroz force-pushed the feature/goreleaser branch from 461b978 to 69c5e53 Compare October 14, 2020 20:03

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 15, 2020

estroz force-pushed the feature/goreleaser branch from 69c5e53 to 17e7412 Compare October 19, 2020 17:50

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 19, 2020

estroz mentioned this pull request Oct 20, 2020

*: refactor Dockerfiles and Makefiles #4069

Merged

2 tasks

estroz commented Oct 20, 2020

View reviewed changes

joelanford reviewed Oct 20, 2020

View reviewed changes

estroz force-pushed the feature/goreleaser branch from ded4754 to b11186c Compare October 21, 2020 00:38

joelanford reviewed Oct 21, 2020

View reviewed changes

release/Makefile Outdated Show resolved Hide resolved

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 21, 2020

estroz force-pushed the feature/goreleaser branch from 1f7afbb to 9fb0e22 Compare October 22, 2020 02:28

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 22, 2020

estroz force-pushed the feature/goreleaser branch 3 times, most recently from 3cf5db6 to 337ef12 Compare October 22, 2020 03:02

estroz commented Oct 22, 2020

View reviewed changes

website/content/en/docs/installation/install-operator-sdk.md Outdated Show resolved Hide resolved

estroz force-pushed the feature/goreleaser branch 3 times, most recently from 15d249d to f94d600 Compare October 30, 2020 20:55

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2020

estroz force-pushed the feature/goreleaser branch from f94d600 to 33ece13 Compare October 30, 2020 20:58

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2020

estroz force-pushed the feature/goreleaser branch 2 times, most recently from 85d5cbc to 1280207 Compare October 31, 2020 00:45

estroz force-pushed the feature/goreleaser branch from 1280207 to 9e4d2dd Compare October 31, 2020 19:14

estroz force-pushed the feature/goreleaser branch from 9e4d2dd to 423f6f7 Compare November 3, 2020 21:51

openshift-ci-robot requested review from asmacdo, rashmigottipati and theishshah November 3, 2020 22:20