Decide whether we want to reconcile all operator resources at once vs. one at a time

[oceanc80]

Background:

• Historically, LM considered all operators in a namespace at a single time rather than upgrading operators individually. This has prevented us from hitting nasty version deadlock edge-cases, as described here.
• Unfortunately, customers frequently hit cases where one operator is unable to upgrade, thereby preventing any operator in the namespace from upgrading.
• We should consider how each approach may affect circular dependencies: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/dependency-resolution.md#example-version-deadlock
  -- There are currently a few operators that introduce circular dependencies, one of which is ODF, MCG, OCS.

Goals:

• Consider our past approaches and how they can best be applied to our new operator singleton design.

Acceptance Criteria:

• We have documented whether or not Operator CRs are reconciled in parallel or serially.

[exdx]

Kubernetes is a level-based system, so my recommendation would be to apply changes one step at a time. Basically, you go gradually from the existing state to the desired state, one operator install at a time.

For example, this mimics how autoscaling works in kubernetes. Say you have 2 pods and you want 5. Instead of the autoscaler trying to spin up 3 at once and then reporting status, it spins up the 3rd...then 4th...then 5th. This is the way kube APIs should behave on the backend IMO.

[dtfranz]

We discussed this topic last Friday and determined there's no way we can guarantee that Operator Controller is operating deterministically and/or optimally unless it is completely aware of the entire "state of the world". If we process each operator install request serially, we may run into cases where changing the order of a set of operator installs would change the resulting state of the cluster. We may also run into cases where a new set of Operator installs will fail because the resolver did not have enough information from the get-go to find an acceptable solution for the set.

See our discussion here for more detail as well as more examples stated better than I can summarize here: https://www.youtube.com/watch?v=put7ywnwh_s

The following issues were created:
operator-framework/operator-controller#89
operator-framework/operator-controller#90
operator-framework/operator-controller#88