Decision: Mapping of OpenSearch node types to roles

[siegfriedweber]

OpenSearch node types

An OpenSearch node (instance) can take one or several of the following node types:

• Cluster manager
• Cluster manager eligible
• Data
• Ingest
• Coordinating
• Dynamic
• Warm
• Search

see https://docs.opensearch.org/docs/latest/tuning-your-cluster/#nodes

A typical cluster only contains some node types and some nodes consist of several types, e.g.

• 3 nodes of type ["Cluster manager"]
• 8 nodes of type ["Data", "Ingest", "Coordinating"]

Proposed solution

Every node type is modelled as a separate role, but extra types per role can be defined on role or role-group level, e.g.

---
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
metadata:
  name: opensearch
spec:
  clusterManager: # implicit node type "Cluster manager"
    roleGroups:
      default:
        replicas: 3
  data: # implicit node type "Data"
    extraNodeTypes: # set the node types of the role-groups to ["Data", "Ingest", "Coordinating"]
      - ingest
      - coordinating
    roleGroups:
      hot:
        replicas: 2
      warm:
        extraNodeTypes: # set the node type of the role-group to ["Data", "Ingest", "Coordinating", "Warm"]
          - warm
        replicas: 6

Advantages:

• Role and role-group configurations are similar to the other operators.

Disadvantages:

• In a typical setup, extraNodeTypes must be defined.

Alternative solution

There is only the role "nodes" with the implicit types ["Cluster manager eligible", "Data", "Ingest", "Coordinating']. The node types can be overriden at the role-group level:

---
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
metadata:
  name: opensearch
spec:
  nodes: # implicit node types ["Cluster manager eligible", "Data", "Ingest", "Coordinating']
    roleGroups:
      clusterManager:
        nodeTypes: # overrides the implicit node types
          - clusterManager
        replicas: 3
      dataHot:
        nodeTypes: # overrides the implicit node types
          - data
          - ingest
          - coordinating
        replicas: 2
      dataWarm:
        nodeTypes: # overrides the implicit node types
          - data
          - ingest
          - coordinating
          - warm
        replicas: 6

Advantages:

• Simple clusters work out of the box without the need to configure node types.
• The configuration is closer to the Helm chart where node types are configured via node.roles.

Disadvantages:

• The role/role-group configuration is different to other operators.
• Configurations at the role level are not as useful as in the proposed solution.
• Role-groups should be named more carefully, e.g. the role-group names of data nodes should contain the term "data".

[nightkr]

• Cluster manager
• Cluster manager eligible

These are one role from our perspective, AFAICT? Just a weird way to state that one node is elected manager at a time.

A typical cluster only contains some node types and some nodes consist of several types, e.g.

How much benefit is there to giving a single OS node multiple roles, vs running one OS node per role on the same K8s node? I assume this is mostly about reducing per-JVM overhead?

Every node type is modelled as a separate role, but extra types per role can be defined on role or role-group level, e.g.

Not a huge fan of this tbh, this leads to questions like "is there a difference between these two variants?":

data:
  default:
    extraNodeTypes: [warm]
---
warm:
  default:
    extraNodeTypes: [data]

# implicit node types ["Cluster manager eligible", "Data", "Ingest", "Coordinating']

Ehhhh...

[siegfriedweber]

• Cluster manager
• Cluster manager eligible

These are one role from our perspective, AFAICT? Just a weird way to state that one node is elected manager at a time.

This can be discussed. I would not offer a role "Cluster manager eligible" because this is more of an attribute of a node than a real role.

A typical cluster only contains some node types and some nodes consist of several types, e.g.

How much benefit is there to giving a single OS node multiple roles, vs running one OS node per role on the same K8s node? I assume this is mostly about reducing per-JVM overhead?

If a role is needed, but is only used slightly, it makes sense in my opinion to assign this role to an existing node instead of deploying a dedicated node. That reduces resources and costs. Furthermore, if OpenSearch allows it, we should not prevent it.

Every node type is modelled as a separate role, but extra types per role can be defined on role or role-group level, e.g.

Not a huge fan of this tbh, this leads to questions like "is there a difference between these two variants?":

data:
  default:
    extraNodeTypes: [warm]
---
warm:
  default:
    extraNodeTypes: [data]

No, there is no difference. In the first case, the pods are named opensearch-data-default-, in the second case opensearch-warm-default-, so I would prefer the first one.

# implicit node types ["Cluster manager eligible", "Data", "Ingest", "Coordinating']

Ehhhh...

These are the default types in OpenSearch, if no types are defined. With sensible defaults, this is a minimum working cluster definition:

---
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
metadata:
  name: opensearch
spec:
  nodes:
    roleGroups:
      default:
        replicas: 1

I should have written:

spec:
  nodes:
    nodeTypes: ... # defaults to ["Cluster manager eligible", "Data", "Ingest", "Coordinating']

[siegfriedweber]

@nightkr, did I understand you correctly, that your preferred option is to offer one role where the node types have to be defined explicitly?

[razvan]

What does it mean for "data" to be implicit? Do I have to explicitly exclude it from a group if I want just a cluster manager without any data on it?

The alternative solution seems a more natural fit for the OS model IMO but maybe with fewer implicit node types if possible.

[maltesander]

For the alternative solution, just an impl question: Do they all share the same configs? Or do we need to split between node types?

[adwk67]

I think it is important to allow both

1. defining OS nodes that combine types
2. defining one OS node type per role (or rolegroup)

so that the user can use 1. for e.g. small setups where resource allocation is not so criticial, and 2. for e.g. cases where scaling up can be done to a specific node type (without having to also scale up its accompanying attributes/services/node types). I think both solutions allow for both 1. and 2., although I find the proposed one a bit cleaner.

[siegfriedweber]

What does it mean for "data" to be implicit? Do I have to explicitly exclude it from a group if I want just a cluster manager without any data on it?

A pure cluster manager can be defined in the proposed solution as follows:

---
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
metadata:
  name: opensearch
spec:
  clusterManager:
    roleGroups:
      default:
        replicas: 1

In the alternative solution, the default node types are overriden, not added:

---
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
metadata:
  name: opensearch
spec:
  nodes:
    roleGroups:
      clusterManager:
        nodeTypes:
          - clusterManager
        replicas: 1

[siegfriedweber]

Do they all share the same configs? Or do we need to split between node types?

They share the same config options but it is hard to define sensible defaults that make sense for all role types. For instance, a cluster manager does not require much disk space, but a data node does.

With the proposed solution, the user probably must set the volume size only on the data role, but with the alternative solution, the volume size must be set on all role-groups.

[nightkr]

@nightkr, did I understand you correctly, that your preferred option is to offer one role where the node types have to be defined explicitly?

I think so. If OS doesn't differentiate between a "primary" and "secondary" role of a node then we probably shouldn't either.

On some level I would prefer to only support a single subrole per (OS) node, and say "just run multiple pods then". But I'm also pretty sure that would devolve into the "secondary subroles" option anyway.

No, there is no difference. In the first case, the pods are named opensearch-data-default-, in the second case opensearch-warm-default-, so I would prefer the first one.

Ideally, yes. But if you allow two different ways to do something then differences are inevitably going to start creeping in... Case in point: while I was writing this reply you posted #8 (comment), documenting one such proposed difference!

Which.. I emphasize with, but.. I also think this helps highlight that our usual role/fragment setup just isn't a great match for this use case. If we do want to do "native" multi-role then we'd want to let each subrole mix in its own defaults, which we don't really have the infra for at all at the moment. That also raises questions like "what happens if multiple subroles provide different defaults for the same value?" (though we could just forbid that through testing).

If a role is needed, but is only used slightly, it makes sense in my opinion to assign this role to an existing node instead of deploying a dedicated node. That reduces resources and costs. Furthermore, if OpenSearch allows it, we should not prevent it.

My question was.. by how much? If we're talking about saving 2GiB per subrole then it makes a lot more sense than if we're talking about 20MiB per subrole.

This can be discussed. I would not offer a role "Cluster manager eligible" because this is more of an attribute of a node than a real role.

Other way around; we can only offer (what OS calls) "Cluster manager eligible", and let OS decide which one becomes the actual "Cluster manager"? Though yes, on our side we should just call that subrole clusterManager (just like we don't differentiate between primary and standby HDFS namenodes).

[siegfriedweber]

Comment for voting

Please read the commets above for the advantages and disadvantages of each option.

Option 1 wins if there are more votes than for option 2a and 2b together.

Option 1 😄

Use a role per node type and allow adding extra types:

spec:
  clusterManager: ... # implicit node type "Cluster manager"
  data: # implicit node type "Data"
    extraNodeTypes: # add the node types ["Ingest", "Coordinating"] to all role-groups in this role
      - ingest
      - coordinating
    roleGroups:
      default:
        extraNodeTypes: # add the role-groups ["Warm"] to the role-group
          - warm

Option 2a 🎉

Use a neutral role nodes and use the OpenSearch defaults at the role level:

spec:
  nodes:
    nodeTypes: ... # defaults to ["Cluster manager eligible", "Data", "Ingest", "Coordinating"]
    roleGroups:
      clusterManager:
        nodeTypes: # overrides the role types set at the role level
          - clusterManager
      data:
        nodeTypes: # overrides the role types set at the role level
          - data
          - ingest
          - coordinating

Option 2b 🚀

Use a neutral role nodes and set no defaults:

spec:
  nodes:
    nodeTypes: ... # defaults to []
    roleGroups:
      clusterManager:
        nodeTypes: # overrides the role types set at the role level
          - clusterManager
      data:
        nodeTypes: # overrides the role types set at the role level
          - data
          - ingest
          - coordinating

Option 3 😕

I have no preference.

[Techassi]

Sorry for coming in rather late, but I miss the discussion of using separate CRDs instead of roles and role groups. I'd really hate to see them ruled out right from the start.

Also, the currently most voted option (2a) looks like an "override hell".

I currently don't have any concrete plan how separate CRDs could look like, but I can come up with one if needed.

[siegfriedweber]

@Techassi and I talked about other options. We have not found a good way to split the node type configuration into several CRDs. We also talked about removing one layer (role or role-group) because the role layer is unnecessary if there is only one role. In principal, removing the role layer would result in a better CRD, especially if the top-level configs like clusterConfig, roleConfig and role.config are merged, but the downside would be that the common operator documentation (e.g. https://docs.stackable.tech/home/stable/concepts/overrides/) would not be valid for the OpenSearch operator.

[siegfriedweber]

So the option 2b won.