Ensure rollouts are completed before starting a new upgrade.

[thetechnick]

Interruptions during bundle reconciliation could lead to operator-controller not completing the current running rollout and instead pick up the next available release from the catalog.

Not rolling out a release completely, may lead to inconsistent results and unexpected behavior.

;tldr: Binary restarts and transient errors should not trigger re-resolve bundle images when a rollout is still in flight.

[joelanford]

Scenario:

• v1 was previously successfully installed
• v2 failed to upgrade
• v3 available in the catalog, upgrades from v1

Discussion in WG meeting:

• If the upgrade to v2 fails because of an ephemeral error AND the catalog contains no upgrade edges from v1, we should try to complete the rollout of the failed release.
• If the upgrade fails because of a "blocking" error OR the catalog contains a "better" upgrade edge from v1, we should discard the failed release, re-resolve and proceed with an upgrade using the newly resolved bundle.
• It may be necessary for extension authors to produce a patch version on a non-latest minor in order to help upgrades progress.

Need to be careful returning nil errors while we are in this retrying state (in either of the above cases).

[tmshort]

@thetechnick to confirm, you're referring to pushing out the deployment objects, not necessarily that Deplpyments/SS/RS/DS have all their replicas up and running?

[LalatenduMohanty]

@thetechnick The PR has merged now, so please test it and reopen the issue if you still see the issue.

[everettraven]

Reopening this issue as we identified the fix that was merged in #1349 builds on the Helm library Get call and that call is insufficient for making an appropriate decision as to what the latest successfully deployed release is.

We identified in a sync call that we should instead:

• Add a List method to the helm-operator-plugins ActionInterface that allows us to use the Helm library List functionality. This allows filtering releases by specific criteria (like status == Deployed), and sorting
• Use the new List method in operator-controllers InstalledBundleGetter implementation to only return the most recent release that has status == Deployed
  • NOTE: Using List would make it such that OLM v1 won't upgrade from a "partial upgrade" UNLESS there is another preferred upgrade path on a future resolution attempt.

[LalatenduMohanty]

Here is PR in helm-operator-plugins repo to get more information about the state of release operator-framework/helm-operator-plugins#397

[LalatenduMohanty]

#1379 is the PR which uses operator-framework/helm-operator-plugins#397.