[self-hosted] Add officially support service type `ClusterIP` for proxy

[corneliusludmann]

With #10537, we support setting the service type of the proxy via an experimental config option in the installer. We see, that we need to support this officially (internal discussion).

For official support, we need:

• Proposed documentation #10983
• Move loadbalancer-specific annotations to loadbalancer-only #11025
• Move installer config value out of experimental to the main section. #11008
• Add a config option in KOTS. #11009
• Write a docs page that explains the usage. #11011
• Make sure it will be tested (could be: unit tests, part of an explicit dimension in the automated self-hosted tests, part of a reference architecture that will be tested by the automated tests, etc). #11033

[lucasvaltl]

Replaces #9981

[szab100]

@corneliusludmann @lucasvaltl When implementing this, can we move adding these two hard-coded annotations to inside the IF(type == LB) {..} stmt above or remove the hard-coded entries completely and just document in which cases should these be added to ProxyConfig.ServiceAnnotations?

Generally, these should only be used when 'proxy' is a load-balancer. When it's NodePort or ClusterIP, the 'external-dns' annotation on 'proxy' can interfere with other ingress / load-balancer resources that can also be using this annotation with same host values. This creates a race condition in the external dns system, causing the LB to be unreachable.

[mrsimonemms]

@corneliusludmann should this also remove the ClusterIP from the experimental section?

[mrsimonemms]

In "blocked" until #11011 is merged when July release is published