From 0a346abc77bcf9a1b192cc807e4c186b87de1f50 Mon Sep 17 00:00:00 2001 From: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com> Date: Wed, 1 Nov 2023 17:47:41 +0100 Subject: [PATCH 1/7] Glossary --- modules/concepts/glossary.adoc | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 modules/concepts/glossary.adoc diff --git a/modules/concepts/glossary.adoc b/modules/concepts/glossary.adoc new file mode 100644 index 000000000..5fd7394c3 --- /dev/null +++ b/modules/concepts/glossary.adoc @@ -0,0 +1,8 @@ += Glossary + +== Stacklet + +A custom resource for a product and all objects that are owned by it. + +A product custom resource defines different aspects of a product installation like the kind of product to install (ex. Airflow), the secrets it needs to connect to other systems, the amount of storage it needs and so on. An operator translates this custom resource in a multitude of Kubernetes objects like services, stateful sets, persistent volume claims and so on. The set of objects that are created as a result of this operation is called a `stacklet`. + From 41ca76a71a5386f1b8289631a7df41bee54cad0a Mon Sep 17 00:00:00 2001 From: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com> Date: Wed, 1 Nov 2023 17:49:57 +0100 Subject: [PATCH 2/7] Update concepts nav --- modules/concepts/nav.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/concepts/nav.adoc b/modules/concepts/nav.adoc index a2cc144e7..ade87fade 100644 --- a/modules/concepts/nav.adoc +++ b/modules/concepts/nav.adoc @@ -17,3 +17,4 @@ *** xref:operations/pod_disruptions.adoc[] *** xref:operations/pod_placement.adoc[] *** xref:operations/graceful_shutdown.adoc[] +* xref:concepts:glossary.adoc[] From 2782959ea0840ea00f829d85e430f620dc893098 Mon Sep 17 00:00:00 2001 From: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com> Date: Thu, 2 Nov 2023 11:34:26 +0100 Subject: [PATCH 3/7] Move to the correct location (hopefully) --- modules/concepts/{ => pages}/glossary.adoc | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename modules/concepts/{ => pages}/glossary.adoc (100%) diff --git a/modules/concepts/glossary.adoc b/modules/concepts/pages/glossary.adoc similarity index 100% rename from modules/concepts/glossary.adoc rename to modules/concepts/pages/glossary.adoc From 607ffa60ffe7522b867f1c8b9b0ce480e40cad27 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Tue, 26 Dec 2023 02:10:27 +0100 Subject: [PATCH 4/7] moved the glossary, added some entries, added styling and linking --- modules/concepts/nav.adoc | 1 - modules/concepts/pages/glossary.adoc | 8 ---- modules/reference/nav.adoc | 4 +- modules/reference/pages/glossary.adoc | 57 +++++++++++++++++++++++++++ 4 files changed, 60 insertions(+), 10 deletions(-) delete mode 100644 modules/concepts/pages/glossary.adoc create mode 100644 modules/reference/pages/glossary.adoc diff --git a/modules/concepts/nav.adoc b/modules/concepts/nav.adoc index 22b5daa6d..00496354a 100644 --- a/modules/concepts/nav.adoc +++ b/modules/concepts/nav.adoc @@ -17,4 +17,3 @@ *** xref:operations/pod_disruptions.adoc[] *** xref:operations/pod_placement.adoc[] *** xref:operations/graceful_shutdown.adoc[] -* xref:concepts:glossary.adoc[] diff --git a/modules/concepts/pages/glossary.adoc b/modules/concepts/pages/glossary.adoc deleted file mode 100644 index 5fd7394c3..000000000 --- a/modules/concepts/pages/glossary.adoc +++ /dev/null @@ -1,8 +0,0 @@ -= Glossary - -== Stacklet - -A custom resource for a product and all objects that are owned by it. - -A product custom resource defines different aspects of a product installation like the kind of product to install (ex. Airflow), the secrets it needs to connect to other systems, the amount of storage it needs and so on. An operator translates this custom resource in a multitude of Kubernetes objects like services, stateful sets, persistent volume claims and so on. The set of objects that are created as a result of this operation is called a `stacklet`. - diff --git a/modules/reference/nav.adoc b/modules/reference/nav.adoc index a3897a721..137226402 100644 --- a/modules/reference/nav.adoc +++ b/modules/reference/nav.adoc @@ -1 +1,3 @@ -- {crd-docs}[CRD Reference {external-link-icon}^] \ No newline at end of file +* Reference +** {crd-docs}[CRD Reference {external-link-icon}^] +** xref:glossary.adoc[] diff --git a/modules/reference/pages/glossary.adoc b/modules/reference/pages/glossary.adoc new file mode 100644 index 000000000..1c4ccff08 --- /dev/null +++ b/modules/reference/pages/glossary.adoc @@ -0,0 +1,57 @@ += Glossary +:li: pass:[] + +// refined styling for the glossary +++++ + +++++ + +// syntax explanation: +// - the [[...]] creates an anchor at the list item +// - The <<...>> creates a link to the achor +// - The {li} references the pass that you can find at the top of the document, it is a FontAwesome icon +// - The ...:: is the syntax for a definition list item +// - The {empty} followed by the + means that there is no "normal" list item, instead there is a block +// - The period followed by text is the summary of a details block that is collapsed +// - Inside the "====" is the details of the details block. +// +// Please create new entries the same way! This allows easy linking to glossary items. +// Antora generates dl, dt and dd tages for the definition list, which is great because these are +// semantic HTML tags. + +[[role]]Role <>:: {empty} ++ +.A <> is made up of multiple roles, which refer to the different processes that make up the service. +[%collapsible] +==== +For example HDFS consists of 3 roles: Name nodes, journal nodes and data nodes. +Learn more about xref:concepts:roles-and-role-groups.adoc[]. +==== + +[[role-group]]Role group <>:: {empty} ++ +.A <> is made up of multiple role groups. +[%collapsible] +==== +A role group can override configuration set at role level, allowing for different configurations for sets of processes. +Learn more about xref:concepts:roles-and-role-groups.adoc[]. +==== + +[[stacklet]]Stacklet <>:: {empty} ++ +.A deployed service or product instance that is managed by a Stackable operator. +[%collapsible] +==== +A Stacklet is defined by a custom resource like AirflowCluster or DruidCluster. +The term refers to the defining resource and all the resources that belong to it, which are created by the operator, such as StatefulSets, Services, Secrets and ConfigMaps. +All objects together are the Stacklet. +==== \ No newline at end of file From 3702744bfc73b1d76fb43acf772c84e1b0c4478d Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Tue, 2 Jan 2024 10:05:09 +0100 Subject: [PATCH 5/7] Open blocks by default --- modules/reference/pages/glossary.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/reference/pages/glossary.adoc b/modules/reference/pages/glossary.adoc index 1c4ccff08..4cf84099d 100644 --- a/modules/reference/pages/glossary.adoc +++ b/modules/reference/pages/glossary.adoc @@ -31,7 +31,7 @@ dt { [[role]]Role <>:: {empty} + .A <> is made up of multiple roles, which refer to the different processes that make up the service. -[%collapsible] +[%collapsible%open] ==== For example HDFS consists of 3 roles: Name nodes, journal nodes and data nodes. Learn more about xref:concepts:roles-and-role-groups.adoc[]. @@ -40,7 +40,7 @@ Learn more about xref:concepts:roles-and-role-groups.adoc[]. [[role-group]]Role group <>:: {empty} + .A <> is made up of multiple role groups. -[%collapsible] +[%collapsible%open] ==== A role group can override configuration set at role level, allowing for different configurations for sets of processes. Learn more about xref:concepts:roles-and-role-groups.adoc[]. @@ -49,7 +49,7 @@ Learn more about xref:concepts:roles-and-role-groups.adoc[]. [[stacklet]]Stacklet <>:: {empty} + .A deployed service or product instance that is managed by a Stackable operator. -[%collapsible] +[%collapsible%open] ==== A Stacklet is defined by a custom resource like AirflowCluster or DruidCluster. The term refers to the defining resource and all the resources that belong to it, which are created by the operator, such as StatefulSets, Services, Secrets and ConfigMaps. From 8dccd4f0c6fafa12553de52e533af1ddc5f8c392 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Tue, 2 Jan 2024 09:41:12 +0100 Subject: [PATCH 6/7] concepts reorganized and new overview page (#325) * Added a bunch of images * More stuff * use different domain in local playbook to make fonts work * build from local branch * temporary branch ref * fixed refs * Update local-antora-playbook.yml * reorganized menu * Update modules/concepts/pages/overview.adoc Co-authored-by: Sebastian Bernauer * fixed illlustration * Added more links --------- Co-authored-by: Sebastian Bernauer --- .../concepts/images/common_objects.drawio.svg | 4 ++ modules/concepts/images/deployment.drawio.svg | 4 ++ modules/concepts/images/discovery.drawio.svg | 4 ++ .../images/operator_overview.drawio.svg | 4 ++ modules/concepts/images/overview.drawio.svg | 4 ++ modules/concepts/nav.adoc | 31 +++++---- modules/concepts/pages/index.adoc | 34 ++++++++- modules/concepts/pages/overview.adoc | 69 +++++++++++++++++++ modules/reference/nav.adoc | 1 + .../pages/duration.adoc | 1 + 10 files changed, 142 insertions(+), 14 deletions(-) create mode 100644 modules/concepts/images/common_objects.drawio.svg create mode 100644 modules/concepts/images/deployment.drawio.svg create mode 100644 modules/concepts/images/discovery.drawio.svg create mode 100644 modules/concepts/images/operator_overview.drawio.svg create mode 100644 modules/concepts/images/overview.drawio.svg create mode 100644 modules/concepts/pages/overview.adoc rename modules/{concepts => reference}/pages/duration.adoc (97%) diff --git a/modules/concepts/images/common_objects.drawio.svg b/modules/concepts/images/common_objects.drawio.svg new file mode 100644 index 000000000..0b2b1b54f --- /dev/null +++ b/modules/concepts/images/common_objects.drawio.svg @@ -0,0 +1,4 @@ + + + +
DruidCluster
DruidCluster
TrinoCluster
TrinoCluster
S3Bucket
S3Bucket
AuthenticationClass
AuthenticationClassText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/images/deployment.drawio.svg b/modules/concepts/images/deployment.drawio.svg new file mode 100644 index 000000000..67470e226 --- /dev/null +++ b/modules/concepts/images/deployment.drawio.svg @@ -0,0 +1,4 @@ + + + +
Operator
Operator
Kubernetes cluster
Kubernetes cluster
Docker repository
Docker repository
Resource
Resource
Stackable CLI
Stackable CLI
Manifests
ManifestsText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/images/discovery.drawio.svg b/modules/concepts/images/discovery.drawio.svg new file mode 100644 index 000000000..eb3b069a7 --- /dev/null +++ b/modules/concepts/images/discovery.drawio.svg @@ -0,0 +1,4 @@ + + + +
ZookeeperCluster
ZookeeperClust...
zk-cluster
zk-cluster
Stackable Operator for Apache ZooKeeper
Stackable Operator for...
ConfigMap
ConfigMap
zk-cluster
zk-cluster
HdfsCluster
HdfsCluster
hdfs-cluster
hdfs-cluster
Stackable Operator for Apache HDFS
Stackable Operator for...
ConfigMap
ConfigMap
hdfs-cluster
hdfs-cluster
DruidCluster
DruidCluster
druid-cluster
druid-cluster
Stackable Operator for Apache Druid
Stackable Operator for...
ConfigMap
ConfigMap
druid-cluster
druid-clusterText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/images/operator_overview.drawio.svg b/modules/concepts/images/operator_overview.drawio.svg new file mode 100644 index 000000000..7e42c0ce4 --- /dev/null +++ b/modules/concepts/images/operator_overview.drawio.svg @@ -0,0 +1,4 @@ + + + +
Product Operator
Product Operator
Statefulset
Statefulset
Service
Service
Pod
Pod
Discovery ConfigMap
Discovery ConfigM...
ConfigMap
ConfigMap
ProductCluster
ProductCluster
create
create
read
read
per Role
per Role
Legend
Legend
Operator
Operator
Resource
Resource
Custom
Resource
Custom...Text is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/images/overview.drawio.svg b/modules/concepts/images/overview.drawio.svg new file mode 100644 index 000000000..912d630d7 --- /dev/null +++ b/modules/concepts/images/overview.drawio.svg @@ -0,0 +1,4 @@ + + + +
Operators
Operators
Custom resources
Custom resources
Resources
ResourcesText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/nav.adoc b/modules/concepts/nav.adoc index 00496354a..afb8ff313 100644 --- a/modules/concepts/nav.adoc +++ b/modules/concepts/nav.adoc @@ -1,19 +1,24 @@ * xref:concepts:index.adoc[] -** xref:roles-and-role-groups.adoc[] -** xref:service-exposition.adoc[] -** xref:service_discovery.adoc[] -** xref:logging.adoc[] -** xref:authentication.adoc[] -** xref:opa.adoc[] -** xref:product_image_selection.adoc[] -** xref:resources.adoc[] -** xref:s3.adoc[] -** xref:pvc.adoc[] -** xref:tls_server_verification.adoc[] -** xref:overrides.adoc[] -** xref:duration.adoc[] +** xref:overview.adoc[Overview] +** Common configuration mechanisms +*** xref:roles-and-role-groups.adoc[] +*** xref:product_image_selection.adoc[] +*** xref:overrides.adoc[Advanced: overrides] +** Resources +*** xref:resources.adoc[] +*** xref:s3.adoc[] +*** xref:pvc.adoc[] +** Connectivity +*** xref:service_discovery.adoc[] +*** xref:service-exposition.adoc[] +** Security +*** xref:authentication.adoc[] +*** xref:opa.adoc[] +*** xref:tls_server_verification.adoc[] ** xref:operations/index.adoc[] *** xref:operations/cluster_operations.adoc[] *** xref:operations/pod_disruptions.adoc[] *** xref:operations/pod_placement.adoc[] *** xref:operations/graceful_shutdown.adoc[] +** Observability +*** xref:logging.adoc[] diff --git a/modules/concepts/pages/index.adoc b/modules/concepts/pages/index.adoc index 2c7c6105d..f45e27a51 100644 --- a/modules/concepts/pages/index.adoc +++ b/modules/concepts/pages/index.adoc @@ -1,3 +1,35 @@ = Concepts -This section of the documentation is intended to be read to gain a deeper understanding of the bigger picture and architectural design of the platform. +== Overview + +The xref:overview.adoc[Platform overview] is a good starting point to understand the Stackable Data Platform covering the overall architecture, deployement and configuration. + +== General configuration mechanisms + +Learn about xref:roles-and-role-groups.adoc[], how product image selection works. +There is also the common xref:overrides.adoc[override] mechanism for configuration settings, although this tool should be used with care! + +== Resources + +Learn about how xref:resources.adoc[] are configured; this covers CPU, memory and storage. +Learn about how xref:s3.adoc[] are configured across the platform. + +== Connectivity + +Many Platform components depend on other components or expose functionality that you can connect to. +This connectivity is achived with xref:service-discovery.adoc[service discovery ConfigMaps]. +To access your Stackable operated products from outside the Kuberenetes cluster learn more about xref:service-exposition.adoc[]. + +== Security + +Security aspects include xref:authentication.adoc[authenticating] users when services are accessed and subsequently xref:opa.adoc[authorizing] access to operations and data inside of the services. +It also includes xref:tls-server-verification.adoc[]. + +== Operations + +The xref:operations/index.adoc[operations] section is directed at platform maintainers. +It covers xref:operations/cluster_operations.adoc[starting, stopping and restarts] of products, xref:operations/graceful_shutdown.adoc[] and other topics related to maintenance and ensuring stability of the platform operation. + +== Observability + +Learn abouth xref:logging.adoc[]. diff --git a/modules/concepts/pages/overview.adoc b/modules/concepts/pages/overview.adoc new file mode 100644 index 000000000..845ef984a --- /dev/null +++ b/modules/concepts/pages/overview.adoc @@ -0,0 +1,69 @@ += Stackable Data Platform explained + +The Stackable Data Platform (SDP) is built on Kubernetes. +Its core is a collection of Kubernetes Operators and custom resources which are designed to work together. + +image::overview.drawio.svg[] + +The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services. + +The operators are deployed with xref:management:stackablectl:index.adoc[] (the Stackable CLI tool) and product instances are created by deploying manifests into Kubernetes. + +Aspects like SQL database configuration, xref:resources.adoc[storage configuration] or xref:authentication.adoc[authentication] and xref:opa.adoc[authorization] work the same way across all operators. +Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization. + +[#operators] +== Operators + +The Operators form the core of the Stackable platform. There is one operator for every supported product, as well as a few supporting operators. All Stackable Operators are built on top of a common framework, so they look and behave in a similar way. + +Every Operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid). +It reads this resource and creates kubernetes resources in accordance with the product CR. + +image::operator_overview.drawio.svg[] + +The diagram above shows the custom resource in green. It contains all the configuration needed to create a product instance. This includes which services the product should connect to, with how many replicas it should operate and how meany resources it should use, among other things. + +[#discovery] +=== Discovery + +The operator also creates a xref:service-discovery.adoc[**discovery ConfigMap**] for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources. + +image::discovery.drawio.svg[] + +For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service. + +You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref:service-discovery.adoc[]. + +[#roles] +=== Roles + +Almost all products that Stackable supports need multiple different processes to run. Because they are often still the same software but running with different parameters, Stackable calls them _roles_. For example HDFS has three roles: DataNode, NameNode and JournalNode. + +All roles are configured together in the custom resource for the product, but they each get their own StatefulSet, ConfigMaps and Service. + +Learn more about roles: xref:roles-and-role-groups.adoc[] + +[#deployment] +== Deployment + +All operators and products run as containers in a xref:home::kubernetes.adoc[Kubernetes cluster]. The operators are deployed with stackablectl (the Stackable CLI) or Helm. + +image::deployment.drawio.svg[] + +To deploy a product instance, a product resource needs to be created in Kubernetes, this is usually done by passing a YAML manifest file to kubernetes with `kubectl apply -f `. The manifest file contains the configuration of how the product should operate. +The operators read the product resources and create the according Kubernetes resources. + +=== Stackable command line interface + +The Stackable command line interface is called _stackablectl_. It knows about the Stackable platform releases and can install sets of operators from a specific release. It is also possible to deploy stacks of product instances that are already wired together. + +== Common configuration of common objects + +Besides the products themselves, there are also related objects, such as S3 buckets or LDAP configuration. + +image::common_objects.drawio.svg[] + +These objects can be reused by all operators that support this feature. The S3 bucket only needs to be described once, and then it can be referenced in all products that support reading and/or writing from/to S3. Learn more about S3 configuration: xref:s3.adoc[]. + +Similarly for the OpenPolicyAgent (OPA). Configuring it looks the same across all products. Learn more: xref:opa.adoc[]. \ No newline at end of file diff --git a/modules/reference/nav.adoc b/modules/reference/nav.adoc index 137226402..f23327915 100644 --- a/modules/reference/nav.adoc +++ b/modules/reference/nav.adoc @@ -1,3 +1,4 @@ * Reference ** {crd-docs}[CRD Reference {external-link-icon}^] ** xref:glossary.adoc[] +** xref:duration.adoc[] diff --git a/modules/concepts/pages/duration.adoc b/modules/reference/pages/duration.adoc similarity index 97% rename from modules/concepts/pages/duration.adoc rename to modules/reference/pages/duration.adoc index 528420c52..62620fc8b 100644 --- a/modules/concepts/pages/duration.adoc +++ b/modules/reference/pages/duration.adoc @@ -1,4 +1,5 @@ = Duration format +:page-aliases: concepts:duration.adoc :rust-duration-max: https://doc.rust-lang.org/std/time/struct.Duration.html#associatedconstant.MAX :go-std-time: https://cs.opensource.google/go/go/+/refs/tags/go1.21.2:src/time/format.go;l=1589 From b46687c220707af95d6343a4782940bd3dacdbf1 Mon Sep 17 00:00:00 2001 From: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com> Date: Tue, 2 Jan 2024 10:18:05 +0100 Subject: [PATCH 7/7] fix: merge breakage --- modules/reference/nav.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/nav.adoc b/modules/reference/nav.adoc index e795d5634..f23327915 100644 --- a/modules/reference/nav.adoc +++ b/modules/reference/nav.adoc @@ -1,4 +1,4 @@ * Reference ** {crd-docs}[CRD Reference {external-link-icon}^] ** xref:glossary.adoc[] -** xref:dura +** xref:duration.adoc[]