Skip to content

Commit 6abbe14

Browse files
fhennigsbernauer
fhennig
and
sbernauer
authored
concepts reorganized and new overview page (#325)
* Added a bunch of images * More stuff * use different domain in local playbook to make fonts work * build from local branch * temporary branch ref * fixed refs * Update local-antora-playbook.yml * reorganized menu * Update modules/concepts/pages/overview.adoc Co-authored-by: Sebastian Bernauer <[email protected]> * fixed illlustration * Added more links --------- Co-authored-by: Sebastian Bernauer <[email protected]>
1 parent 3e9905f commit 6abbe14

10 files changed

+144
-15
lines changed

‎modules/concepts/images/common_objects.drawio.svg

+4
Loading

‎modules/concepts/images/deployment.drawio.svg

+4
Loading

‎modules/concepts/images/discovery.drawio.svg

+4
Loading

‎modules/concepts/images/operator_overview.drawio.svg

+4
Loading

‎modules/concepts/images/overview.drawio.svg

+4
Loading

‎modules/concepts/nav.adoc

+18-13
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,24 @@
11
* xref:concepts:index.adoc[]
2-
** xref:roles-and-role-groups.adoc[]
3-
** xref:service-exposition.adoc[]
4-
** xref:service_discovery.adoc[]
5-
** xref:logging.adoc[]
6-
** xref:authentication.adoc[]
7-
** xref:opa.adoc[]
8-
** xref:product_image_selection.adoc[]
9-
** xref:resources.adoc[]
10-
** xref:s3.adoc[]
11-
** xref:pvc.adoc[]
12-
** xref:tls_server_verification.adoc[]
13-
** xref:overrides.adoc[]
14-
** xref:duration.adoc[]
2+
** xref:overview.adoc[Overview]
3+
** Common configuration mechanisms
4+
*** xref:roles-and-role-groups.adoc[]
5+
*** xref:product_image_selection.adoc[]
6+
*** xref:overrides.adoc[Advanced: overrides]
7+
** Resources
8+
*** xref:resources.adoc[]
9+
*** xref:s3.adoc[]
10+
*** xref:pvc.adoc[]
11+
** Connectivity
12+
*** xref:service_discovery.adoc[]
13+
*** xref:service-exposition.adoc[]
14+
** Security
15+
*** xref:authentication.adoc[]
16+
*** xref:opa.adoc[]
17+
*** xref:tls_server_verification.adoc[]
1518
** xref:operations/index.adoc[]
1619
*** xref:operations/cluster_operations.adoc[]
1720
*** xref:operations/pod_disruptions.adoc[]
1821
*** xref:operations/pod_placement.adoc[]
1922
*** xref:operations/graceful_shutdown.adoc[]
23+
** Observability
24+
*** xref:logging.adoc[]

‎modules/concepts/pages/index.adoc

+33-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,35 @@
11
= Concepts
22

3-
This section of the documentation is intended to be read to gain a deeper understanding of the bigger picture and architectural design of the platform.
3+
== Overview
4+
5+
The xref:overview.adoc[Platform overview] is a good starting point to understand the Stackable Data Platform covering the overall architecture, deployement and configuration.
6+
7+
== General configuration mechanisms
8+
9+
Learn about xref:roles-and-role-groups.adoc[], how product image selection works.
10+
There is also the common xref:overrides.adoc[override] mechanism for configuration settings, although this tool should be used with care!
11+
12+
== Resources
13+
14+
Learn about how xref:resources.adoc[] are configured; this covers CPU, memory and storage.
15+
Learn about how xref:s3.adoc[] are configured across the platform.
16+
17+
== Connectivity
18+
19+
Many Platform components depend on other components or expose functionality that you can connect to.
20+
This connectivity is achived with xref:service-discovery.adoc[service discovery ConfigMaps].
21+
To access your Stackable operated products from outside the Kuberenetes cluster learn more about xref:service-exposition.adoc[].
22+
23+
== Security
24+
25+
Security aspects include xref:authentication.adoc[authenticating] users when services are accessed and subsequently xref:opa.adoc[authorizing] access to operations and data inside of the services.
26+
It also includes xref:tls-server-verification.adoc[].
27+
28+
== Operations
29+
30+
The xref:operations/index.adoc[operations] section is directed at platform maintainers.
31+
It covers xref:operations/cluster_operations.adoc[starting, stopping and restarts] of products, xref:operations/graceful_shutdown.adoc[] and other topics related to maintenance and ensuring stability of the platform operation.
32+
33+
== Observability
34+
35+
Learn abouth xref:logging.adoc[].

‎modules/concepts/pages/overview.adoc

+69
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
= Stackable Data Platform explained
2+
3+
The Stackable Data Platform (SDP) is built on Kubernetes.
4+
Its core is a collection of Kubernetes Operators and custom resources which are designed to work together.
5+
6+
image::overview.drawio.svg[]
7+
8+
The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services.
9+
10+
The operators are deployed with xref:management:stackablectl:index.adoc[] (the Stackable CLI tool) and product instances are created by deploying manifests into Kubernetes.
11+
12+
Aspects like SQL database configuration, xref:resources.adoc[storage configuration] or xref:authentication.adoc[authentication] and xref:opa.adoc[authorization] work the same way across all operators.
13+
Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization.
14+
15+
[#operators]
16+
== Operators
17+
18+
The Operators form the core of the Stackable platform. There is one operator for every supported product, as well as a few supporting operators. All Stackable Operators are built on top of a common framework, so they look and behave in a similar way.
19+
20+
Every Operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid).
21+
It reads this resource and creates kubernetes resources in accordance with the product CR.
22+
23+
image::operator_overview.drawio.svg[]
24+
25+
The diagram above shows the custom resource in green. It contains all the configuration needed to create a product instance. This includes which services the product should connect to, with how many replicas it should operate and how meany resources it should use, among other things.
26+
27+
[#discovery]
28+
=== Discovery
29+
30+
The operator also creates a xref:service-discovery.adoc[**discovery ConfigMap**] for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources.
31+
32+
image::discovery.drawio.svg[]
33+
34+
For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service.
35+
36+
You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref:service-discovery.adoc[].
37+
38+
[#roles]
39+
=== Roles
40+
41+
Almost all products that Stackable supports need multiple different processes to run. Because they are often still the same software but running with different parameters, Stackable calls them _roles_. For example HDFS has three roles: DataNode, NameNode and JournalNode.
42+
43+
All roles are configured together in the custom resource for the product, but they each get their own StatefulSet, ConfigMaps and Service.
44+
45+
Learn more about roles: xref:roles-and-role-groups.adoc[]
46+
47+
[#deployment]
48+
== Deployment
49+
50+
All operators and products run as containers in a xref:home::kubernetes.adoc[Kubernetes cluster]. The operators are deployed with stackablectl (the Stackable CLI) or Helm.
51+
52+
image::deployment.drawio.svg[]
53+
54+
To deploy a product instance, a product resource needs to be created in Kubernetes, this is usually done by passing a YAML manifest file to kubernetes with `kubectl apply -f <file.yaml>`. The manifest file contains the configuration of how the product should operate.
55+
The operators read the product resources and create the according Kubernetes resources.
56+
57+
=== Stackable command line interface
58+
59+
The Stackable command line interface is called _stackablectl_. It knows about the Stackable platform releases and can install sets of operators from a specific release. It is also possible to deploy stacks of product instances that are already wired together.
60+
61+
== Common configuration of common objects
62+
63+
Besides the products themselves, there are also related objects, such as S3 buckets or LDAP configuration.
64+
65+
image::common_objects.drawio.svg[]
66+
67+
These objects can be reused by all operators that support this feature. The S3 bucket only needs to be described once, and then it can be referenced in all products that support reading and/or writing from/to S3. Learn more about S3 configuration: xref:s3.adoc[].
68+
69+
Similarly for the OpenPolicyAgent (OPA). Configuring it looks the same across all products. Learn more: xref:opa.adoc[].

‎modules/reference/nav.adoc

+3-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1-
- {crd-docs}[CRD Reference {external-link-icon}^]
1+
* Reference
2+
** {crd-docs}[CRD Reference {external-link-icon}^]
3+
** xref:duration.adoc[]

‎modules/concepts/pages/duration.adoc renamed to ‎modules/reference/pages/duration.adoc

+1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
= Duration format
2+
:page-aliases: concepts:duration.adoc
23

34
:rust-duration-max: https://doc.rust-lang.org/std/time/struct.Duration.html#associatedconstant.MAX
45
:go-std-time: https://cs.opensource.google/go/go/+/refs/tags/go1.21.2:src/time/format.go;l=1589

0 commit comments

Comments
 (0)