Partial CVE automation

[swapneils]

Summary

This script restructures some scripts, e.g. generate_changelog.sh, and adds a python3 script publish_cve_update.py which automates the process of checking the remote AL2 repository and (if needed) generating a commit to release a new latest version. The commit is made in a new local branch named patch-automation-X where X is the same date extension that would be used for the CVE update.

Note that this change verifies whether there have been any commits to aws-for-fluent-bit or upstream-to-fluent-bit since the last release (will change to fluent-bit upstream later), and if so attempts to infer change descriptions for these changes. If a CVE release has the string Compared to the previous release, this release adds:, even if the section is empty, please check both this repository and its upstream to ensure no unintended changes are being released.

If a release is not a CVE release, the operator will need to modify the version number in linux.version, AWS_FOR_FLUENT_BIT_VERSION, and changelog.md.

This script also pushes the new branch to the origin remote of your local cloned repository. Note that we validate the attempt to switch branches and said action is idempotent, so this should not ever result in pushing to origin/mainline unless some other process switches the branch partway through this script.

Usage steps (after merging this PR):

1. Go to the root directory of your clone of aws-for-fluent-bit
2. Switch to the mainline branch
3. Run python3 ./scripts/publish_cve_update.py
4. Monitor the script's output for shell script failures. If these occur, the local environment or the script will need to be debugged.
5. You should end up in the mainline branch, with a new patch-automation-X branch existing in your local clone of the repository as well as the remote origin.

Issue #, if available:

Testing

make debug succeeded: N/A
Integ tests succeeded: N/A
New tests cover the changes: N/A

Description for the changelog

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[ShelbyZ]

Some initial comments, may need some additional time on the py script

[sparrc]

Note that this change verifies whether there have been any commits to aws-for-fluent-bit or upstream-to-fluent-bit since the last release (will change to fluent-bit upstream later), and if so attempts to infer change descriptions for these changes. If a CVE release has the string Compared to the previous release, this release adds:, even if the section is empty, please check both this repository and its upstream to ensure no unintended changes are being released.

For the purposes of CVE-only releases, I'm not sure if we should be doing this much logic and parsing. It seems to me that at a high level we should simply do something like:

1. Checkout most recent tagged commit (ex. v2.32.5.20250305).
2. Update base Amazon Linux container version.
3. Update release/tag version (ex. v2.32.5.20250324)
4. Run release (maybe from a branch to avoid picking up unwanted changes to mainline).

Is this something that was considered? I am wondering if we should separate the "regular" releases that infer change descriptions et al into a separate change from the CVE-only releases?

[swapneils]

Is this something that was considered? I am wondering if we should separate the "regular" releases that infer change descriptions et al into a separate change from the CVE-only releases?

For CVE releases:

The dependency on checking the commit history at all isn't one we can remove, since for CVE releases we need to know we're not accidentally releasing a feature change.

Given that we're going to parse the commits regardless, the additional logic to guess what those commits mean improves debuggability if we are accidentally releasing change commits, especially if we're running this script via automations and just double-checking the results. The investment to add these components is only writing the code, so it doesn't add any additional ops burden either.

For feature releases:

I don't think we will fully automate feature releases, at least at this stage of the pipeline. In addition to the changelog modifications, they also need to bump the correct component of our semantic versioning, and manual checks through the relevant repos are also useful as a final pre-integ-test spot-check to avoid publishing bad changes.

If we take partial automation of feature releases as the goal, there seems no reason to separate them into a separate script, as the automateable parts of the process are generally shared with CVE releases. The more advanced parts of the commit parsing are the only components essential to feature releases but not CVE releases, and as mentioned above they do improve the CVE release experience as well, so inclined to keep them here.

Note: Even if we do make separate script endpoints for feature vs CVE releases, still inclined for this logic to stay in generate_changelog.sh and be referenced by both automations, to avoid branching logic or duplicating generate_changelog.sh for each release type.