From e6841dccf4d3fe3b1eaed6457cdc77a72663515a Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Wed, 23 Nov 2022 11:12:27 +0100 Subject: [PATCH 01/11] Added a bunch of images --- modules/concepts/images/deployment.drawio.svg | 4 ++ modules/concepts/images/discovery.drawio.svg | 4 ++ .../images/operator_overview.drawio.svg | 4 ++ modules/concepts/images/overview.drawio.svg | 4 ++ modules/concepts/nav.adoc | 4 +- modules/concepts/pages/index.adoc | 50 ++++++++++++++++++- 6 files changed, 66 insertions(+), 4 deletions(-) create mode 100644 modules/concepts/images/deployment.drawio.svg create mode 100644 modules/concepts/images/discovery.drawio.svg create mode 100644 modules/concepts/images/operator_overview.drawio.svg create mode 100644 modules/concepts/images/overview.drawio.svg diff --git a/modules/concepts/images/deployment.drawio.svg b/modules/concepts/images/deployment.drawio.svg new file mode 100644 index 000000000..67470e226 --- /dev/null +++ b/modules/concepts/images/deployment.drawio.svg @@ -0,0 +1,4 @@ + + + +
Operator
Operator
Kubernetes cluster
Kubernetes cluster
Docker repository
Docker repository
Resource
Resource
Stackable CLI
Stackable CLI
Manifests
ManifestsText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/images/discovery.drawio.svg b/modules/concepts/images/discovery.drawio.svg new file mode 100644 index 000000000..eb3b069a7 --- /dev/null +++ b/modules/concepts/images/discovery.drawio.svg @@ -0,0 +1,4 @@ + + + +
ZookeeperCluster
ZookeeperClust...
zk-cluster
zk-cluster
Stackable Operator for Apache ZooKeeper
Stackable Operator for...
ConfigMap
ConfigMap
zk-cluster
zk-cluster
HdfsCluster
HdfsCluster
hdfs-cluster
hdfs-cluster
Stackable Operator for Apache HDFS
Stackable Operator for...
ConfigMap
ConfigMap
hdfs-cluster
hdfs-cluster
DruidCluster
DruidCluster
druid-cluster
druid-cluster
Stackable Operator for Apache Druid
Stackable Operator for...
ConfigMap
ConfigMap
druid-cluster
druid-clusterText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/images/operator_overview.drawio.svg b/modules/concepts/images/operator_overview.drawio.svg new file mode 100644 index 000000000..7e42c0ce4 --- /dev/null +++ b/modules/concepts/images/operator_overview.drawio.svg @@ -0,0 +1,4 @@ + + + +
Product Operator
Product Operator
Statefulset
Statefulset
Service
Service
Pod
Pod
Discovery ConfigMap
Discovery ConfigM...
ConfigMap
ConfigMap
ProductCluster
ProductCluster
create
create
read
read
per Role
per Role
Legend
Legend
Operator
Operator
Resource
Resource
Custom
Resource
Custom...Text is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/images/overview.drawio.svg b/modules/concepts/images/overview.drawio.svg new file mode 100644 index 000000000..912d630d7 --- /dev/null +++ b/modules/concepts/images/overview.drawio.svg @@ -0,0 +1,4 @@ + + + +
Operators
Operators
Custom resources
Custom resources
Resources
ResourcesText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/nav.adoc b/modules/concepts/nav.adoc index d74b862de..2796bf52e 100644 --- a/modules/concepts/nav.adoc +++ b/modules/concepts/nav.adoc @@ -1,7 +1,7 @@ -* xref:concepts:index.adoc[] +* xref:concepts:index.adoc[Platform Explanation] ** xref:roles-and-role-groups.adoc[] +** xref:resources.adoc[] ** xref:service_discovery.adoc[] ** xref:opa.adoc[] ** xref:s3.adoc[] ** xref:pvc.adoc[] -** xref:resources.adoc[] diff --git a/modules/concepts/pages/index.adoc b/modules/concepts/pages/index.adoc index 2c7c6105d..31ef889a4 100644 --- a/modules/concepts/pages/index.adoc +++ b/modules/concepts/pages/index.adoc @@ -1,3 +1,49 @@ -= Concepts += Stackable Data Platform explained -This section of the documentation is intended to be read to gain a deeper understanding of the bigger picture and architectural design of the platform. +The Stackable Data Platform (SDP) is built on Kubernetes. +Its core is a collection of Kubernetes Operators and custom resources which are designed to work together. + +They are _conceptually similar_ in that they are deployed and used the same way. +Per supported product there is an operator. +Every operator has at its core a custom resource (CR) which defines a product instance. + +Aspects like SQL database configuration, storage configuration or authentication and authorization work the same way across all operators. +Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization. + +image::overview.drawio.svg[] + +== Operator Overview + +All Stackable Operators work very similar, so they are easy to use. + +Every Operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid). +It reads this resource and creates kubernetes resources in accordance with the product CR. + +image::operator_overview.drawio.svg[] + +=== Resource lifecycle + +When do resources get created and destroyed? + +What about reosurces like the DruidConnection? + +=== Discovery + +Products find each other with **discovery ConfigMaps**. The ConfigMap has the same name as the product instance resource. + +image::discovery.drawio.svg[] + +=== Roles + +== Deployment + +The operators are deployed with stackablectl - the Stackable CLI - or Helm, into the Kubernetes cluster, using docker images. +Product cluster manfests are created by the user and put into the Kubernetes cluster. +The operators read the product resources and create the according Kubernetes resources. +They also pull product images from the docker repository. + +image::deployment.drawio.svg[] + +=== Stackable command line interface + +The Stackable command line interface is called stackablectl. It knows about the Stackable platform releases and can install sets of operators from a specific release. It is also possible to deploy stacks of product instances that are already wired together. \ No newline at end of file From 2372148e90ea6782987852d0a65b73d7450f9c8f Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Wed, 23 Nov 2022 14:46:28 +0100 Subject: [PATCH 02/11] More stuff --- .../concepts/images/common_objects.drawio.svg | 4 ++ modules/concepts/pages/index.adoc | 55 +++++++++++++------ 2 files changed, 41 insertions(+), 18 deletions(-) create mode 100644 modules/concepts/images/common_objects.drawio.svg diff --git a/modules/concepts/images/common_objects.drawio.svg b/modules/concepts/images/common_objects.drawio.svg new file mode 100644 index 000000000..720b32bed --- /dev/null +++ b/modules/concepts/images/common_objects.drawio.svg @@ -0,0 +1,4 @@ + + + +
DruidCluster
DruidCluster
TrinoCluster
TrinoCluster
S3Bucket
S3Bucket
LdapServer
LdapServerText is not SVG - cannot display \ No newline at end of file diff --git a/modules/concepts/pages/index.adoc b/modules/concepts/pages/index.adoc index 31ef889a4..24faadac3 100644 --- a/modules/concepts/pages/index.adoc +++ b/modules/concepts/pages/index.adoc @@ -3,47 +3,66 @@ The Stackable Data Platform (SDP) is built on Kubernetes. Its core is a collection of Kubernetes Operators and custom resources which are designed to work together. -They are _conceptually similar_ in that they are deployed and used the same way. -Per supported product there is an operator. -Every operator has at its core a custom resource (CR) which defines a product instance. +image::overview.drawio.svg[] + +The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services. + +The operators are deployed with the Stackable CLI tool and product instances are created by deploying manifests into Kubernetes. Aspects like SQL database configuration, storage configuration or authentication and authorization work the same way across all operators. Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization. -image::overview.drawio.svg[] - -== Operator Overview +== Operators -All Stackable Operators work very similar, so they are easy to use. +The Operators form the core of the Stackable platform. There is one operator for every supported product, as well as a few supporting operators. All Stackable Operators are built on top of a common framework, so they look and behave in a similar way. Every Operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid). It reads this resource and creates kubernetes resources in accordance with the product CR. image::operator_overview.drawio.svg[] -=== Resource lifecycle - -When do resources get created and destroyed? - -What about reosurces like the DruidConnection? +The diagram above shows the custom resource in green. It contains all the configuration needed to create a product instance. This includes which services the product should connect to, with how many replicas it should operate and how meany resources it should use, among other things. +[#discovery] === Discovery -Products find each other with **discovery ConfigMaps**. The ConfigMap has the same name as the product instance resource. +The operator also creates a **discovery ConfigMap** for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources. image::discovery.drawio.svg[] +For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service. + +You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref::service_discovery.adoc[]. + +[#roles] === Roles +Almost all products that Stackable supports need multiple different processes to run. Because they are often still the same software but running with different parameters, Stackable calls them _roles_. For example HDFS has three roles: DataNode, NameNode and JournalNode. + +All roles are configured together in the custom resource for the product, but they each get their own StatefulSet, ConfigMaps and Service. + +Learn more about roles: xref::roles-and-role-groups.adoc[] + +[#deployment] == Deployment -The operators are deployed with stackablectl - the Stackable CLI - or Helm, into the Kubernetes cluster, using docker images. -Product cluster manfests are created by the user and put into the Kubernetes cluster. -The operators read the product resources and create the according Kubernetes resources. -They also pull product images from the docker repository. +All operators and products run as containers in a Kubernetes cluster. The operators are deployed with stackablectl (the Stackable CLI) or Helm. image::deployment.drawio.svg[] +To deploy a product instance, a product resource needs to be created in Kubernetes, this is usually done by passing a YAML manifest file to kubernetes with `kubectl apply -f`. The manifest file contains the configuration of how the product should operate. +The operators read the product resources and create the according Kubernetes resources. + === Stackable command line interface -The Stackable command line interface is called stackablectl. It knows about the Stackable platform releases and can install sets of operators from a specific release. It is also possible to deploy stacks of product instances that are already wired together. \ No newline at end of file +The Stackable command line interface is called _stackablectl_. It knows about the Stackable platform releases and can install sets of operators from a specific release. It is also possible to deploy stacks of product instances that are already wired together. + +== Common configuration of common objects + +Besides the products themselves, there are also related objects, such as S3 buckets or LDAP configuration. + +image::common_objects.drawio.svg[] + +These objects can be reused by all operators that support this feature. The S3 bucket only needs to be described once, and then it can be referenced in all products that support reading and/or writing from/to S3. Learn more about S3 configuration: xref::s3.adoc[]. + +Similarly for the OpenPolicyAgent (OPA). Configuring it looks the same across all products. Learn more: xref::opa.adoc[]. \ No newline at end of file From 3f55dad96b69d1fa6e0bef6a69443fe5a3341121 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Thu, 24 Nov 2022 14:06:09 +0100 Subject: [PATCH 03/11] use different domain in local playbook to make fonts work --- local-antora-playbook.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 54a833ce7..ce61d12d6 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -1,7 +1,7 @@ --- site: title: Stackable Documentation - url: https://docs.stackable.tech + url: https://repo.stackable.tech start_page: home::index.adoc robots: allow keys: From 5e6cdd1229b13531fcdf8e723666cda5fe61bc8c Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Mon, 27 Mar 2023 13:27:15 +0200 Subject: [PATCH 04/11] build from local branch --- antora-playbook.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/antora-playbook.yml b/antora-playbook.yml index 1644df57d..7d178fb6d 100644 --- a/antora-playbook.yml +++ b/antora-playbook.yml @@ -15,7 +15,7 @@ urls: content: sources: - - url: https://github.com/stackabletech/documentation.git + - url: ./ branches: - HEAD - release/23.1 From 10ad12647266ee81a56876b7bddfc23e0991fdff Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Mon, 27 Mar 2023 13:32:01 +0200 Subject: [PATCH 05/11] temporary branch ref --- antora-playbook.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/antora-playbook.yml b/antora-playbook.yml index 7d178fb6d..1b6fec2da 100644 --- a/antora-playbook.yml +++ b/antora-playbook.yml @@ -15,9 +15,9 @@ urls: content: sources: - - url: ./ + - url: https://github.com/stackabletech/documentation.git branches: - - HEAD + - operator-concept - release/23.1 - release/22.11 - release/22.09 From 9ea6ef03dbdbad3b8861c7f8d13e47516bd098d7 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Mon, 27 Mar 2023 13:46:55 +0200 Subject: [PATCH 06/11] fixed refs --- modules/concepts/pages/opa.adoc | 2 +- modules/concepts/pages/s3.adoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/concepts/pages/opa.adoc b/modules/concepts/pages/opa.adoc index 546766c3b..66cb78721 100644 --- a/modules/concepts/pages/opa.adoc +++ b/modules/concepts/pages/opa.adoc @@ -83,6 +83,6 @@ Read more about the xref:opa:index.adoc[]. Read more about product integration w * xref:trino:usage_guide/security.adoc#_authorization[Trino] * xref:kafka:usage.adoc[Kafka] -* xref:druid:usage.adoc#_using_open_policy_agent_opa_for_authorization[Druid] +* xref:druid:usage-guide/security.adoc#authorization[Druid] You can also have a look at the xref:contributor:opa_configuration.adoc[implementation guidelines for OPA authorizers] or learn more about the xref:service_discovery.adoc[service discovery mechanism] used across the platform. diff --git a/modules/concepts/pages/s3.adoc b/modules/concepts/pages/s3.adoc index a6cfb3b9c..c95d6ecf0 100644 --- a/modules/concepts/pages/s3.adoc +++ b/modules/concepts/pages/s3.adoc @@ -3,7 +3,7 @@ // -------------- Intro ---------------- Many of the tools on the Stackable platform integrate with S3 storage in some way. -For example Druid can xref:druid:usage.adoc#_s3_for_ingestion[ingest data from S3] and also xref:druid:usage.adoc##_s3_deep_storage[use S3 as a backend for deep storage], Spark can use an xref:spark-k8s:usage.adoc#_s3_bucket_specification[S3 bucket] to store application files and data. +For example Druid can xref:druid:usage-guide/ingestion.adoc#s3[ingest data from S3] and also xref:druid:usage-guide/deep-storage.adoc#s3[use S3 as a backend for deep storage], Spark can use an xref:spark-k8s:usage.adoc#_s3_bucket_specification[S3 bucket] to store application files and data. == S3Connection and S3Bucket // introducing the objects From 73c640d9b647d15ff417bba0f31d1b949a7a6ab4 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Mon, 25 Dec 2023 23:22:04 +0100 Subject: [PATCH 07/11] Update local-antora-playbook.yml --- local-antora-playbook.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index f42717c8a..178ef98ee 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -1,7 +1,7 @@ --- site: title: Stackable Documentation - url: https://repo.stackable.tech + url: https://docs.stackable.tech start_page: home::index.adoc robots: allow keys: From d79fc3ec7d03a9e6423d2da5eea0fa230d794721 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Tue, 26 Dec 2023 00:44:12 +0100 Subject: [PATCH 08/11] reorganized menu --- modules/concepts/nav.adoc | 34 +++++---- modules/concepts/pages/index.adoc | 75 ++++++------------- modules/concepts/pages/overview.adoc | 68 +++++++++++++++++ modules/reference/nav.adoc | 4 +- .../pages/duration.adoc | 1 + 5 files changed, 112 insertions(+), 70 deletions(-) create mode 100644 modules/concepts/pages/overview.adoc rename modules/{concepts => reference}/pages/duration.adoc (97%) diff --git a/modules/concepts/nav.adoc b/modules/concepts/nav.adoc index 3dfea7496..afb8ff313 100644 --- a/modules/concepts/nav.adoc +++ b/modules/concepts/nav.adoc @@ -1,20 +1,24 @@ -* xref:concepts:index.adoc[Platform Explanation] -** xref:roles-and-role-groups.adoc[] -** xref:resources.adoc[] -** xref:service-exposition.adoc[] -** xref:service_discovery.adoc[] -** xref:logging.adoc[] -** xref:authentication.adoc[] -** xref:opa.adoc[] -** xref:product_image_selection.adoc[] -** xref:pvc.adoc[] -** xref:s3.adoc[] -** xref:pvc.adoc[] -** xref:tls_server_verification.adoc[] -** xref:overrides.adoc[] -** xref:duration.adoc[] +* xref:concepts:index.adoc[] +** xref:overview.adoc[Overview] +** Common configuration mechanisms +*** xref:roles-and-role-groups.adoc[] +*** xref:product_image_selection.adoc[] +*** xref:overrides.adoc[Advanced: overrides] +** Resources +*** xref:resources.adoc[] +*** xref:s3.adoc[] +*** xref:pvc.adoc[] +** Connectivity +*** xref:service_discovery.adoc[] +*** xref:service-exposition.adoc[] +** Security +*** xref:authentication.adoc[] +*** xref:opa.adoc[] +*** xref:tls_server_verification.adoc[] ** xref:operations/index.adoc[] *** xref:operations/cluster_operations.adoc[] *** xref:operations/pod_disruptions.adoc[] *** xref:operations/pod_placement.adoc[] *** xref:operations/graceful_shutdown.adoc[] +** Observability +*** xref:logging.adoc[] diff --git a/modules/concepts/pages/index.adoc b/modules/concepts/pages/index.adoc index 24faadac3..f45e27a51 100644 --- a/modules/concepts/pages/index.adoc +++ b/modules/concepts/pages/index.adoc @@ -1,68 +1,35 @@ -= Stackable Data Platform explained += Concepts -The Stackable Data Platform (SDP) is built on Kubernetes. -Its core is a collection of Kubernetes Operators and custom resources which are designed to work together. +== Overview -image::overview.drawio.svg[] +The xref:overview.adoc[Platform overview] is a good starting point to understand the Stackable Data Platform covering the overall architecture, deployement and configuration. -The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services. +== General configuration mechanisms -The operators are deployed with the Stackable CLI tool and product instances are created by deploying manifests into Kubernetes. +Learn about xref:roles-and-role-groups.adoc[], how product image selection works. +There is also the common xref:overrides.adoc[override] mechanism for configuration settings, although this tool should be used with care! -Aspects like SQL database configuration, storage configuration or authentication and authorization work the same way across all operators. -Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization. +== Resources -== Operators +Learn about how xref:resources.adoc[] are configured; this covers CPU, memory and storage. +Learn about how xref:s3.adoc[] are configured across the platform. -The Operators form the core of the Stackable platform. There is one operator for every supported product, as well as a few supporting operators. All Stackable Operators are built on top of a common framework, so they look and behave in a similar way. +== Connectivity -Every Operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid). -It reads this resource and creates kubernetes resources in accordance with the product CR. +Many Platform components depend on other components or expose functionality that you can connect to. +This connectivity is achived with xref:service-discovery.adoc[service discovery ConfigMaps]. +To access your Stackable operated products from outside the Kuberenetes cluster learn more about xref:service-exposition.adoc[]. -image::operator_overview.drawio.svg[] +== Security -The diagram above shows the custom resource in green. It contains all the configuration needed to create a product instance. This includes which services the product should connect to, with how many replicas it should operate and how meany resources it should use, among other things. +Security aspects include xref:authentication.adoc[authenticating] users when services are accessed and subsequently xref:opa.adoc[authorizing] access to operations and data inside of the services. +It also includes xref:tls-server-verification.adoc[]. -[#discovery] -=== Discovery +== Operations -The operator also creates a **discovery ConfigMap** for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources. +The xref:operations/index.adoc[operations] section is directed at platform maintainers. +It covers xref:operations/cluster_operations.adoc[starting, stopping and restarts] of products, xref:operations/graceful_shutdown.adoc[] and other topics related to maintenance and ensuring stability of the platform operation. -image::discovery.drawio.svg[] +== Observability -For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service. - -You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref::service_discovery.adoc[]. - -[#roles] -=== Roles - -Almost all products that Stackable supports need multiple different processes to run. Because they are often still the same software but running with different parameters, Stackable calls them _roles_. For example HDFS has three roles: DataNode, NameNode and JournalNode. - -All roles are configured together in the custom resource for the product, but they each get their own StatefulSet, ConfigMaps and Service. - -Learn more about roles: xref::roles-and-role-groups.adoc[] - -[#deployment] -== Deployment - -All operators and products run as containers in a Kubernetes cluster. The operators are deployed with stackablectl (the Stackable CLI) or Helm. - -image::deployment.drawio.svg[] - -To deploy a product instance, a product resource needs to be created in Kubernetes, this is usually done by passing a YAML manifest file to kubernetes with `kubectl apply -f`. The manifest file contains the configuration of how the product should operate. -The operators read the product resources and create the according Kubernetes resources. - -=== Stackable command line interface - -The Stackable command line interface is called _stackablectl_. It knows about the Stackable platform releases and can install sets of operators from a specific release. It is also possible to deploy stacks of product instances that are already wired together. - -== Common configuration of common objects - -Besides the products themselves, there are also related objects, such as S3 buckets or LDAP configuration. - -image::common_objects.drawio.svg[] - -These objects can be reused by all operators that support this feature. The S3 bucket only needs to be described once, and then it can be referenced in all products that support reading and/or writing from/to S3. Learn more about S3 configuration: xref::s3.adoc[]. - -Similarly for the OpenPolicyAgent (OPA). Configuring it looks the same across all products. Learn more: xref::opa.adoc[]. \ No newline at end of file +Learn abouth xref:logging.adoc[]. diff --git a/modules/concepts/pages/overview.adoc b/modules/concepts/pages/overview.adoc new file mode 100644 index 000000000..24faadac3 --- /dev/null +++ b/modules/concepts/pages/overview.adoc @@ -0,0 +1,68 @@ += Stackable Data Platform explained + +The Stackable Data Platform (SDP) is built on Kubernetes. +Its core is a collection of Kubernetes Operators and custom resources which are designed to work together. + +image::overview.drawio.svg[] + +The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services. + +The operators are deployed with the Stackable CLI tool and product instances are created by deploying manifests into Kubernetes. + +Aspects like SQL database configuration, storage configuration or authentication and authorization work the same way across all operators. +Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization. + +== Operators + +The Operators form the core of the Stackable platform. There is one operator for every supported product, as well as a few supporting operators. All Stackable Operators are built on top of a common framework, so they look and behave in a similar way. + +Every Operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid). +It reads this resource and creates kubernetes resources in accordance with the product CR. + +image::operator_overview.drawio.svg[] + +The diagram above shows the custom resource in green. It contains all the configuration needed to create a product instance. This includes which services the product should connect to, with how many replicas it should operate and how meany resources it should use, among other things. + +[#discovery] +=== Discovery + +The operator also creates a **discovery ConfigMap** for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources. + +image::discovery.drawio.svg[] + +For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service. + +You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref::service_discovery.adoc[]. + +[#roles] +=== Roles + +Almost all products that Stackable supports need multiple different processes to run. Because they are often still the same software but running with different parameters, Stackable calls them _roles_. For example HDFS has three roles: DataNode, NameNode and JournalNode. + +All roles are configured together in the custom resource for the product, but they each get their own StatefulSet, ConfigMaps and Service. + +Learn more about roles: xref::roles-and-role-groups.adoc[] + +[#deployment] +== Deployment + +All operators and products run as containers in a Kubernetes cluster. The operators are deployed with stackablectl (the Stackable CLI) or Helm. + +image::deployment.drawio.svg[] + +To deploy a product instance, a product resource needs to be created in Kubernetes, this is usually done by passing a YAML manifest file to kubernetes with `kubectl apply -f`. The manifest file contains the configuration of how the product should operate. +The operators read the product resources and create the according Kubernetes resources. + +=== Stackable command line interface + +The Stackable command line interface is called _stackablectl_. It knows about the Stackable platform releases and can install sets of operators from a specific release. It is also possible to deploy stacks of product instances that are already wired together. + +== Common configuration of common objects + +Besides the products themselves, there are also related objects, such as S3 buckets or LDAP configuration. + +image::common_objects.drawio.svg[] + +These objects can be reused by all operators that support this feature. The S3 bucket only needs to be described once, and then it can be referenced in all products that support reading and/or writing from/to S3. Learn more about S3 configuration: xref::s3.adoc[]. + +Similarly for the OpenPolicyAgent (OPA). Configuring it looks the same across all products. Learn more: xref::opa.adoc[]. \ No newline at end of file diff --git a/modules/reference/nav.adoc b/modules/reference/nav.adoc index a3897a721..4642c5bba 100644 --- a/modules/reference/nav.adoc +++ b/modules/reference/nav.adoc @@ -1 +1,3 @@ -- {crd-docs}[CRD Reference {external-link-icon}^] \ No newline at end of file +* Reference +** {crd-docs}[CRD Reference {external-link-icon}^] +** xref:duration.adoc[] \ No newline at end of file diff --git a/modules/concepts/pages/duration.adoc b/modules/reference/pages/duration.adoc similarity index 97% rename from modules/concepts/pages/duration.adoc rename to modules/reference/pages/duration.adoc index 528420c52..62620fc8b 100644 --- a/modules/concepts/pages/duration.adoc +++ b/modules/reference/pages/duration.adoc @@ -1,4 +1,5 @@ = Duration format +:page-aliases: concepts:duration.adoc :rust-duration-max: https://doc.rust-lang.org/std/time/struct.Duration.html#associatedconstant.MAX :go-std-time: https://cs.opensource.google/go/go/+/refs/tags/go1.21.2:src/time/format.go;l=1589 From d6dd9f1d6ddfe942c98154ba856cdc7a1c3562b6 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Fri, 29 Dec 2023 11:53:01 +0100 Subject: [PATCH 09/11] Update modules/concepts/pages/overview.adoc Co-authored-by: Sebastian Bernauer --- modules/concepts/pages/overview.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/concepts/pages/overview.adoc b/modules/concepts/pages/overview.adoc index 24faadac3..78df6d45b 100644 --- a/modules/concepts/pages/overview.adoc +++ b/modules/concepts/pages/overview.adoc @@ -50,7 +50,7 @@ All operators and products run as containers in a Kubernetes cluster. The operat image::deployment.drawio.svg[] -To deploy a product instance, a product resource needs to be created in Kubernetes, this is usually done by passing a YAML manifest file to kubernetes with `kubectl apply -f`. The manifest file contains the configuration of how the product should operate. +To deploy a product instance, a product resource needs to be created in Kubernetes, this is usually done by passing a YAML manifest file to kubernetes with `kubectl apply -f `. The manifest file contains the configuration of how the product should operate. The operators read the product resources and create the according Kubernetes resources. === Stackable command line interface From 612f106c7aeda488f8d28dd2b00c0795c8a2789d Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Fri, 29 Dec 2023 11:56:43 +0100 Subject: [PATCH 10/11] fixed illlustration --- modules/concepts/images/common_objects.drawio.svg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/concepts/images/common_objects.drawio.svg b/modules/concepts/images/common_objects.drawio.svg index 720b32bed..0b2b1b54f 100644 --- a/modules/concepts/images/common_objects.drawio.svg +++ b/modules/concepts/images/common_objects.drawio.svg @@ -1,4 +1,4 @@ - + -
DruidCluster
DruidCluster
TrinoCluster
TrinoCluster
S3Bucket
S3Bucket
LdapServer
LdapServerText is not SVG - cannot display \ No newline at end of file +
DruidCluster
DruidCluster
TrinoCluster
TrinoCluster
S3Bucket
S3Bucket
AuthenticationClass
AuthenticationClassText is not SVG - cannot display \ No newline at end of file From fb51d7217ea36a63838f8d800d8d5d450e013e0e Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Fri, 29 Dec 2023 12:11:13 +0100 Subject: [PATCH 11/11] Added more links --- modules/concepts/pages/overview.adoc | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/modules/concepts/pages/overview.adoc b/modules/concepts/pages/overview.adoc index 78df6d45b..845ef984a 100644 --- a/modules/concepts/pages/overview.adoc +++ b/modules/concepts/pages/overview.adoc @@ -7,11 +7,12 @@ image::overview.drawio.svg[] The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services. -The operators are deployed with the Stackable CLI tool and product instances are created by deploying manifests into Kubernetes. +The operators are deployed with xref:management:stackablectl:index.adoc[] (the Stackable CLI tool) and product instances are created by deploying manifests into Kubernetes. -Aspects like SQL database configuration, storage configuration or authentication and authorization work the same way across all operators. +Aspects like SQL database configuration, xref:resources.adoc[storage configuration] or xref:authentication.adoc[authentication] and xref:opa.adoc[authorization] work the same way across all operators. Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization. +[#operators] == Operators The Operators form the core of the Stackable platform. There is one operator for every supported product, as well as a few supporting operators. All Stackable Operators are built on top of a common framework, so they look and behave in a similar way. @@ -26,13 +27,13 @@ The diagram above shows the custom resource in green. It contains all the config [#discovery] === Discovery -The operator also creates a **discovery ConfigMap** for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources. +The operator also creates a xref:service-discovery.adoc[**discovery ConfigMap**] for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources. image::discovery.drawio.svg[] For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service. -You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref::service_discovery.adoc[]. +You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref:service-discovery.adoc[]. [#roles] === Roles @@ -41,12 +42,12 @@ Almost all products that Stackable supports need multiple different processes to All roles are configured together in the custom resource for the product, but they each get their own StatefulSet, ConfigMaps and Service. -Learn more about roles: xref::roles-and-role-groups.adoc[] +Learn more about roles: xref:roles-and-role-groups.adoc[] [#deployment] == Deployment -All operators and products run as containers in a Kubernetes cluster. The operators are deployed with stackablectl (the Stackable CLI) or Helm. +All operators and products run as containers in a xref:home::kubernetes.adoc[Kubernetes cluster]. The operators are deployed with stackablectl (the Stackable CLI) or Helm. image::deployment.drawio.svg[] @@ -63,6 +64,6 @@ Besides the products themselves, there are also related objects, such as S3 buck image::common_objects.drawio.svg[] -These objects can be reused by all operators that support this feature. The S3 bucket only needs to be described once, and then it can be referenced in all products that support reading and/or writing from/to S3. Learn more about S3 configuration: xref::s3.adoc[]. +These objects can be reused by all operators that support this feature. The S3 bucket only needs to be described once, and then it can be referenced in all products that support reading and/or writing from/to S3. Learn more about S3 configuration: xref:s3.adoc[]. -Similarly for the OpenPolicyAgent (OPA). Configuring it looks the same across all products. Learn more: xref::opa.adoc[]. \ No newline at end of file +Similarly for the OpenPolicyAgent (OPA). Configuring it looks the same across all products. Learn more: xref:opa.adoc[]. \ No newline at end of file