Replace helm chart set-up with SDP opensearch-operator

[adwk67]

An alternative to the original Helm chart set-up - using the SDP operator (and original OS image, not our one) instead - is adjacent to the Helm setup folder.

See this folder.

This is a work in progress and will be used to list outstanding tasks.

Progress

• 🟢 Deploy opensearch cluster using operator
• 🟢 Test dashboard endpoint
• 🟢 Run benchmark (v. slow when run locally, even with only 2% of percolate data set: also slow when run from original setup)
• 🟢 Check integration of performance analyser
• 🟢 Check integration of prometheus exporter
• 🟢 Check deployment of security admin
• 🟢 Check upgrade of security admin
• 🟢 extend data-nodes test
• 🟢 scale-up-and-down test (not yet checked-in, shards not returnable due to OpenSearch Security not initialized)

Benchmark output

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] [Test Execution ID]: 5405a7e1-529c-4625-bb6c-147d62e037cb
[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Downloading workload data file: queries-2.json.bz2 (121.1 kB total size)   [100.0%]
[INFO] Decompressing workload data from [/stackable/.osb/benchmarks/data/percolator/queries-2.json.bz2] to [/stackable/.osb/benchmarks/data/percolator/queries-2.json] (resulting size: [0.10] GB) ... [OK]
[INFO] Downloading workload data file: queries-2.json.offset (661 bytes to...     [100.0%]
[INFO] Executing test with workload [percolator], test_procedure [append-no-conflicts] and provision_config_instance ['external'] with version [3.0.0].

[WARNING] merges_total_time is 11 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 286 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 1007 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 57 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index                                                                  [100% done]
Running refresh-after-index                                                    [100% done]
...

TODOs

• security admin fails: the CN in the certificate is not listed as an admin user
• security admin: ensure initialization scripts complete successfully

Part of #1

[adwk67]

🟢 run-securityadmin

Will connect to opensearch-nodes-cluster-manager-headless.sos.svc.cluster.local:9200 ... done
Connected as "CN=generated certificate for pod"
ERR: "CN=generated certificate for pod" is not an admin user
Seems you use a node certificate. This is not permitted, you have to use a client certificate and register it as admin_dn in opensearch.yml

Setting this

plugins.security.authcz.admin_dn: "CN=generated certificate for pod"

in the opensearch cluster resource results in matching certifcates and thus a valid connection...

...but the script still fails

Security Admin v7
Will connect to opensearch-nodes-cluster-manager-headless.sos.svc.cluster.local:9200 ... done
Connected as "CN=generated certificate for pod"
ERR: Seems you use a node certificate which is also an admin certificate
     That may have worked with older OpenSearch Security versions but it indicates
     a configuration error and is therefore forbidden now.
OpenSearch Version: 2.19.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch
Clusterstate: GREEN
Number of nodes: 6
Number of data nodes: 3
.opendistro_security index already exists, so we do not need to create one.
Populate config from /stackable/opensearch/config/opensearch-security
Will update '/config' with config/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with config/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with config/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with config/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with config/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with config/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with config/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with config/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with config/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with config/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
FAIL: 6 nodes reported failures. Failure is /{"_nodes":{"total":6,"successful":0,"failed":6,"failures":[{"type":"failed_node_exception","reason":"Failed node [onpMMv7BRGiGuYNo4fz3lA]","node_id":"onpMMv7BRGiGuYNo4fz3lA","caused_by":{"type":"exception","reason":"java.lang.NullPointerException: Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null"}}},{"type":"failed_node_exception","reason":"Failed node [Y2Z3Z6o1QkSALyn7u1jLbw]","node_id":"Y2Z3Z6o1QkSALyn7u1jLbw","caused_by":{"type":"exception","reason":"java.lang.NullPointerException: Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null"}}},{"type":"failed_node_exception","reason":"Failed node [BwdQyhCzRV-xFNRQK9bVpw]","node_id":"BwdQyhCzRV-xFNRQK9bVpw","caused_by":{"type":"exception","reason":"java.lang.NullPointerException: Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null"}}},{"type":"failed_node_exception","reason":"Failed node [YIyypn8NRuG2cQh8ypcEYw]","node_id":"YIyypn8NRuG2cQh8ypcEYw","caused_by":{"type":"exception","reason":"java.lang.NullPointerException: Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null"}}},{"type":"failed_node_exception","reason":"Failed node [qhp552WHT_-1b3j7J11MTQ]","node_id":"qhp552WHT_-1b3j7J11MTQ","caused_by":{"type":"exception","reason":"java.lang.NullPointerException: Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null"}}},{"type":"failed_node_exception","reason":"Failed node [qrJBHPxRRHOQyUJ8YoAbeA]","node_id":"qrJBHPxRRHOQyUJ8YoAbeA","caused_by":{"type":"exception","reason":"java.lang.NullPointerException: Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.security.securityconf.impl.CType.toLCString()\" because \"events[i]\" is null"}}}]},"cluster_name":"opensearch","configupdate_response":{"nodes":{},"node_size":0,"has_failures":true,"failures_size":6}}
FAIL: Expected 6 nodes to return response, but got 0
Done with failures
Stream closed EOF for sos/run-securityadmin-zrw6f (run-securityadmin)

Update

when using the SDP 3.1.0 OpenSearch image this error disappears.

Successful message

Security Admin v7
Will connect to opensearch-nodes-cluster-manager-headless.sos.svc.cluster.local:9200 ... done
Connected as "CN=generated certificate for pod"
ERR: Seems you use a node certificate which is also an admin certificate
     That may have worked with older OpenSearch Security versions but it indicates
     a configuration error and is therefore forbidden now.
OpenSearch Version: 3.1.0
...
Populate config from /stackable/opensearch-3.1.0-stackable0.0.0-dev/config/opensearch-security
Will update '/config' with config/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with config/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with config/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with config/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with config/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with config/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with config/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/audit' with config/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with config/opensearch-security/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
Done with success

[adwk67]

🟢 update-security-config

Works as per this commit.