Merge remote-tracking branch 'origin/main' into adr/authorization-abstraction-layer

Author: fhennig

.github/workflows/build.yml

@@ -8,13 +8,13 @@ jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@755da8c3cf115ac066823e79a1e1788f8940201b # v3.2.0
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 #tag=v4.1.1
         with:
           submodules: true
           fetch-depth: 0
-      - uses: actions/setup-node@1a4442cacd436585916779262731d5b162bc6ec7 # v3
+      - uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 #tag=v4.0.2
         with:
-          node-version: '16'
+          node-version: '20'
           cache: 'npm'
       - run: npm ci
       - run: make ANTORAFLAGS=--fetch

modules/ROOT/nav2.adoc

@@ -3,4 +3,5 @@
 * xref:release-notes.adoc[Release notes]
 * xref:product-information.adoc[]
 * xref:policies.adoc[]
-* xref:licenses.adoc[Licenses]
+* xref:licenses.adoc[Licenses]
+* xref:export.adoc[Export Control]

modules/ROOT/pages/export.adoc

@@ -0,0 +1,16 @@
+= Export Control
+
+== USA
+
+The USA adopts the https://en.wikipedia.org/wiki/Export_Administration_Regulations[Export Administration Regulations (EAR)] as the primary regulation to control exports.
+
+All of our products are outside of the scope of EAR because they fall under the _publicly available_ exemption and do not contain non-standard cryptography.
+
+NOTE: That means E-Mail notifications to the NSA and BIS are not required and are therefore not published here.
+
+In particular:
+
+* We are exempt under https://www.ecfr.gov/current/title-15/subtitle-B/chapter-VII/subchapter-C/part-734/section-734.3[EAR 734.3(b)] because we publish according to https://www.ecfr.gov/current/title-15/subtitle-B/chapter-VII/subchapter-C/part-734/section-734.7[EAR 734.7]
+* We are exempt under https://www.ecfr.gov/current/title-15/subtitle-B/chapter-VII/subchapter-C/part-742/section-742.15[EAR 742.15(b)] as our software includes only https://ecfr.io/Title-15/Section-772.1[standard encryption]
+
+The Stackable Data Platform is open source, and the source code of all the components can be found on https://github.com/stackabletech/[GitHub].

modules/concepts/examples/authenticationclass-keycloak.yaml

@@ -0,0 +1,16 @@
+apiVersion: authentication.stackable.tech/v1alpha1
+kind: AuthenticationClass
+metadata:
+  name: keycloak
+spec:
+  provider:
+    oidc:
+      hostname: my.keycloak.server        # <1>
+      port: 8080                          # <2>
+      rootPath: /realms/master            # <3>
+      scopes:                             # <4>
+      - email
+      - openid
+      - profile
+      principalClaim: preferred_username  # <5>
+      providerHint: Keycloak              # <6>

modules/concepts/nav.adoc

@@ -21,4 +21,5 @@
 *** xref:operations/pod_placement.adoc[]
 *** xref:operations/graceful_shutdown.adoc[]
 ** Observability
+*** xref:labels.adoc[]
 *** xref:logging.adoc[]

modules/concepts/pages/authentication.adoc

@@ -14,6 +14,7 @@ Multiple operators use this CRD as a way to express and configure the authentica
 The following authentication providers are supported:
 
 * <<LDAP>>: Authenticate users using an LDAP server.
+* <<OIDC>>: Authenticate users using an OpenID connect provider.
 * <<TLS>>: Authenticate users with client TLS certificates.
 * <<Static>>: Authenticate users against a static list of users and passwords in a simple Kubernetes Secret.
 
@@ -36,6 +37,25 @@ image::image$authentication-overview.drawio.svg[]
 
 NOTE: Learn more in the xref:tutorials:authentication_with_openldap.adoc[OpenLDAP tutorial] and get a full overview of all the properties in the {crd-docs}/authentication.stackable.tech/authenticationclass/v1alpha1/#spec-provider-ldap[AuthenticationClass LDAP provider CRD reference].
 
+[#OIDC]
+=== OpenID Connect
+
+An OIDC provider like https://www.keycloak.org/[Keycloak {external-link-icon}^] could be configured as follows:
+
+[source,yaml]
+----
+include::example$authenticationclass-keycloak.yaml[]
+----
+
+<1> Hostname of the identity provider.
+<2> Port of the identity provider. If TLS is used defaults to 443, otherwise to 80.
+<3> Root HTTP path of the identity provider. Defaults to `/`.
+<4> Scopes to request from your identity provider. It is recommended to request the `openid`, `email`, and `profile` scopes.
+<5> If a product extracts some sort of "effective user" that is represented by a string internally, this config determines which claim is used to extract that string.
+<6> This is a hint about which identity provider is used by the AuthenticationClass.
+
+NOTE: Get a full overview of all the properties in the {crd-docs}/authentication.stackable.tech/authenticationclass/v1alpha1/#spec-provider-oidc[AuthenticationClass OIDC provider CRD reference].
+
 [#tls]
 === TLS
 The `TLS` provider configures a product to authenticate users using TLS certificates.

modules/concepts/pages/index.adoc

@@ -32,4 +32,5 @@ It covers xref:operations/cluster_operations.adoc[starting, stopping and restart
 
 == Observability
 
-Learn abouth xref:logging.adoc[].
+Learn about which xref:labels.adoc[labels] are attached to the operators and the stacklets and the resources that get created by the operator.
+Also learn about xref:logging.adoc[].

modules/concepts/pages/labels.adoc

@@ -0,0 +1,58 @@
+= Labels
+
+Labels are key/value pairs in the metadata of Kubernetes objects that add identifying information to the object.
+They do not have direct semantic implications but can be used to organize and manage resources.
+The `stackablectl` tool, the cockpit and the Helm Charts add labels to the resources that are part of a Stacklet, and the operators add labels to the resources they create.
+
+== Resource labels added by the operators
+
+Every resource created by a Stackable operator is labelled with a common set of labels.
+Some of these labels are https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/[recommended] to use by the Kubernetes documentation.
+The following labels are added to resources created by our operators:
+
+- `app.kubernetes.io/name` - The name of the product, i.e. `druid` or `zookeeper`.
+- `app.kubernetes.io/version` - The version of the product and Stackable version, i.e. `3.3.4-stackable23.11.0`
+- `app.kubernetes.io/instance` - The name of the Stacklet.
+- `app.kubernetes.io/component` - This is the xref:concepts:roles-and-role-groups.adoc[role] that this resource belongs to.
+- `app.kubernetes.io/role-group` - The name of the xref:concepts:roles-and-role-groups.adoc[role group] that a resource belongs to.
+- `app.kubernetes.io/managed-by` - Which software manages this resource? This will be the operator, i.e. `airflow-operator`.
+
+also this:
+
+- `stackable.tech/vendor` with value `Stackable`
+
+== Labels added by tools
+
+The resources associated with an operator installation are also labeled (or can be).
+
+=== stackablectl
+
+==== Installing operators and releases
+
+When installing operators and releases, `stackablectl` will always label the Deployment, ConfigMap, ClusterRole etc. with
+
+* `stackable.tech/vendor=Stackable`
+
+==== Stacks and demos
+
+When installing a stack or a demo, `stackablectl` adds additional labels to identify the parts of the stack or demo:
+
+* `stackable-tech/stack` with the value being the Stack name.
+* `stackable-tech/demo` with the value being the Demo name.
+* `stackable.tech/managed-by=stackablectl`
+
+These labels are attached to the demo and stack manifests, which include Secret, ConfigMap or ClusterRole.
+
+NOTE: The Stacklets themselves and external dependencies are not yet labeled with the Stack or Demo that they belong to.
+
+=== Helm
+
+The Helm Charts for the Stackable operators support specifying labels in the `values.yaml` file.
+All resources deployed by the Helm Chart (Deployment, ClusterRole, Configmap, etc.) will have the labels attached when the operator is deployed.
+The default set of labels includes:
+
+* `stackable.tech/vendor=Stackable`
+
+== Further reading
+
+Have a look at https://docs.stackable.tech/home/nightly/contributor/adr/adr031-resource-labels[] if you want to find out about the design decisions for our labels.

modules/concepts/pages/logging.adoc

@@ -57,32 +57,45 @@ As part of the logging configuration, individual modules can be configured with
 
 [source,yaml]
 ----
-vectorAggregatorConfigMapName: vector-aggregator-discovery  // <1>
-...
-  logging:                   // <2>
-    enableVectorAgent: true  // <3>
-    containers:              // <4>
-      main-container:
-        console:             // <5>
-          level: DEBUG
-        file:
-          level: INFO
-        loggers:             // <6>
-          ROOT:
-            level: INFO
-          my.module:
-            level: DEBUG
-          some.chatty.module:
-            level: NONE
-      sidecar-container:
-        ...
+spec:
+  clusterConfig:
+    vectorAggregatorConfigMapName: vector-aggregator-discovery  // <1>
+  myRole:
+    config:                        // <2>
+      logging:
+        enableVectorAgent: true
+        containers:
+          main-container:
+            console:
+              level: INFO
+    myRoleGroup:
+      config:                      // <2>
+        logging:                   // <3>
+          enableVectorAgent: true  // <4>
+          containers:              // <5>
+            main-container:
+              console:             // <6>
+                level: DEBUG
+              file:
+                level: INFO
+              loggers:             // <7>
+                ROOT:
+                  level: INFO
+                my.module:
+                  level: DEBUG
+                some.chatty.module:
+                  level: NONE
+            sidecar-container:
+              console:
+                level: ERROR 
 ----
 <1> The discovery ConfigMap of the Vector aggregator to publish the logs to. This is set at cluster level.
-<2> The logging configuration fragment, can be set at role or role group level.
-<3> Enable the Vector agent to aggregate logs.
-<4> Logging is defined for each container.
-<5> Console and file appenders can have different log level thresholds.
-<6> Setting log levels for individual modules is also possible.
+<2> The role or role group config containing the logging configuration.
+<3> The logging configuration fragment, can be set at role or role group level.
+<4> Enable the Vector agent to aggregate logs.
+<5> Logging is defined for each container.
+<6> Console and file appenders can have different log level thresholds.
+<7> Setting log levels for individual modules is also possible.
 
 **Log levels per module** - In the `loggers` section the log levels for each module are configured.
 Note the `ROOT` module.

modules/concepts/pages/product-image-selection.adoc

@@ -50,28 +50,23 @@ What this means is, that for example the Stackable Operator for Apache HBase wil
 |===
 
 
-However, since the last digit of the Stackable version is considered to be a patchlevel indicator, operators will be compatible with all images from the same release line.
+However, since the last digit of the Stackable version is considered to be a patch level indicator, operators will be compatible with all images from the same release line.
 So an operator of version _23.4.x_ will be compatible with all images of version _23.4.y_.
-This is intended to allow shorter update cycles for users, when new image versions are released that may contain security fixes - should the user so choose.
-
+This is intended to allow shorter update cycles for users, when new image versions are released that may contain security fixes.
 
 The following paragraphs explain the available settings and how they work.
 
 At the bottom of this page in the <<_common_scenarios, common scenarios>> section some common update scenarios are explained as examples.
 
 == Stackable provided images
 
-If your Kubernetes cluster has internet access, the easiest way is to use the publicly available Images from the https://docker.stackable.tech/[Image registry hosted by Stackable].
+If your Kubernetes cluster has internet access, the easiest way is to use the publicly available images from the https://docker.stackable.tech/[Image registry hosted by Stackable].
 If the Kubernetes cluster does not have internet access, a xref:_custom_docker_registry[] or xref:_custom_images[] can be used.
 
 Currently, you need to specify the product version. This can be found on the xref:operators:supported_versions.adoc[list of supported product versions] or on the website of the product itself.
 This requirement might be relaxed in the future, as every platform release will ship wth a recommended product versions, which will be used by default.
 
-Additionally, you can specify the Stackable version: As we need to make changes to the Images from time to time (e.g. security updates), we also have to version them using the Stackable version. An image gets released for every version of the SDP.
-There are two variants you can choose from:
-
-1. Fixed version, e.g. `23.7.0`. This image will never change.
-2. Release line, e.g. `23.7`. This will be a floating tag pointing to the latest patch release of the SDP release line. It will contain the latest security patches, but will also change over time.
+Additionally, as images should be updated from time to time (e.g. new base image, security updates), a Stackable version can be provided. An image with the Stackable version `23.7.0` is fixed and will never change. Security updates within a release line will result in patch version bumps in the Stackable version to e.g. `23.7.1`.
 
 If you don't specify the Stackable version, the operator will use its own version, e.g. `23.7.0`.
 When using a nightly operator or a `pr` version, it will use the nightly `0.0.0-dev` image.
@@ -137,7 +132,7 @@ When deriving images from official Stackable images this will mean updating the
 The recommended process here is:
 
 ** Tag clusters as "do not reconcile" (see xref:operations/cluster_operations.adoc[])
-** Update Stackable plattform
+** Update Stackable platform
 ** Change custom images in cluster specifications
 ** Remove "do not reconcile flag"
 
@@ -156,34 +151,8 @@ spec:
     productVersion: 3.3.1
 ----
 
-### Quick updates of images
-Sometimes it can be useful to decouple operators upgrades from the image versions to allow using updated images as soons as Stackable releases them.
-This can significantly shorten turnaround times when reacting to security vulnerabilities for example.
-
-For this scenario the Stackable version can be set to the release line, without including the patch level indicator.
-This will cause the operator to always use the most current image that it is compatible with when starting products.
-
-[NOTE]
-====
-This behavior can result in _mixed_ clusters running on different image versions of the product.
-This should not create any issues, since the contained product binaries are exactly the same, but is worth knowing.
-
-A rolling restart of the product would clean this mixed state up.
-====
-
-#### Config
-[source,yaml]
-----
-spec:
-  image:
-    productVersion: 3.3.1
-    stackableVersion: 23.4
-----
-
-
-
-#### Custom images / pinned images
-When a setup requires the utmost stability and it is preferrable for things to break, rather than run with a different image version that for example has not been certified.
+### Custom images / pinned images
+When a setup requires the utmost stability, and it is preferable for things to break, rather than run with a different image version that for example has not been certified.
 Or when a user requires custom libraries / code in the images they run and build their own images derived from official Stackable images, this is the only possible way to do this.
 
 Please see the warnings in <<customimages, custom images section>> above for how to upgrade in this scenario.