Skip to content

Feat: add sharded mongodb #196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions catalog/mongodb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
### 1. 简介
MongoDB 是一种文档数据库,它所具备的可扩展性和灵活性可以满足您对查询和索引的需求

### 2. 产品特性

#### 文档存储
用类似 JSON 的格式存储数据,灵活多变,适合各种数据结构。

#### 高性能
读写速度快,支持嵌套文档和数组,减少连接操作,查询更快。

#### 高可用
通过副本集自动复制数据,一台服务器出问题,其他服务器可以接管,保证不停机。

#### 可扩展
通过分片技术,数据分布在多台机器上,轻松应对数据增长和流量增加。

#### 强大查询
提供丰富的查询语言,支持各种复杂查询和实时数据分析。
131 changes: 131 additions & 0 deletions catalog/mongodb/apps/mongodb.app/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
### 1. 简介

MongoDB 是一种文档数据库,它所具备的可扩展性和灵活性可以满足您对查询和索引的需求

### 基本概念

1. 文档(Document)
- MongoDB 存储数据的基本单位,类似于 JSON 对象,包含键值对。

2. 集合(Collection)
- 文档的容器,类似于关系数据库中的表。

3. 数据库(Database)
- 集合的物理容器,一个 MongoDB 服务器可以包含多个数据库。

4. 字段(Field)
- 文档中的键值对,用于存储具体的数据。

5. 索引(Index)
- 用于提高查询性能,支持多种类型的索引。

6. 查询(Query)
- 用于从 MongoDB 中检索数据的指令,支持丰富的查询语言。

7. 聚合(Aggregation)
- 用于处理数据记录并返回计算结果,提供聚合管道和 map-reduce 两种聚合框架。

8. 副本集(Replica Set)
- 一组 MongoDB 服务器,维护相同的数据副本,提供冗余和高可用性。

9. 分片(Sharding)
- 将数据分布在多个服务器上的过程,用于处理大量数据和高吞吐量的应用。


### 2. 快速开始


#### 使用 Python
```python
# Connect to MongoDB server
mongosh "mongodb://localhost:27017" -u "root" -p "root.password"

# show databases
show dbs

# Select database
use mydatabase

# Create (Insert)
db.mycollection.insertOne({
"name": "Alice",
"age": 30,
"email": "[email protected]"
})

# Read (Find)
db.mycollection.find()

# Update
db.mycollection.updateOne(
{ "name": "Alice" },
{ $set: { "age": 31 } }
)

# Delete
db.mycollection.deleteOne({ "name": "Alice" })

# Drop collection
db.mycollection.drop()

# Drop database
db.dropDatabase()

```

#### 使用命令行
进入MongoDB Pod 容器

```bash
# Connect to MongoDB server
mongosh "mongodb://localhost:27017" -u "root" -p "root.password"

# show databases
show dbs

# Select database
use mydatabase

# Create (Insert)
db.mycollection.insertOne({
"name": "Alice",
"age": 30,
"email": "[email protected]"
})

# Read (Find)
db.mycollection.find()

# Update
db.mycollection.updateOne(
{ "name": "Alice" },
{ $set: { "age": 31 } }
)

# Delete
db.mycollection.deleteOne({ "name": "Alice" })

# Drop collection
db.mycollection.drop()

# Drop database
db.dropDatabase()

```


### 3. FAQ

1. 如何减少 Shard 的数量?
请参考 https://www.mongodb.com/docs/manual/tutorial/remove-shards-from-cluster/
由于 shard 是 k8s satefulset 负载, 因此需要移除序号最大的 shard , 如 mongodb-sharded-shard-<max-number>
然后在KDP调整 shard 数量

2. 生产环境中, 如何配置组件数量?
https://www.mongodb.com/docs/manual/core/sharded-cluster-components/#production-configuration



https://www.mongodb.com/docs/manual/faq/


13 changes: 13 additions & 0 deletions catalog/mongodb/apps/mongodb.app/app.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: bdc.kdp.io/v1alpha1
kind: Application
metadata:
annotations:
app.core.bdos/catalog: mongodb
labels:
app: mongodb
app.core.bdos/type: system
spec:
name: mongodb
type: mongodb
properties:
shards: 1
124 changes: 124 additions & 0 deletions catalog/mongodb/apps/mongodb.app/i18n/en/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
### 1. Introduction
MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need.

### Basic Concepts

1. Document
- The fundamental unit of data storage in MongoDB, similar to a JSON object, consisting of key-value pairs.

1. Collection
- A container for documents, analogous to a table in relational databases.

1. Database
- A physical container for collections, with a MongoDB server capable of hosting multiple databases.

1. Field
- A key-value pair within a document, used to store specific data.

1. Index
- Used to enhance query performance, supporting various types of indexes.

1. Query
- Instructions for retrieving data from MongoDB, featuring a rich query language.

1. Aggregation
- Operations for processing data records and returning computed results, offering both aggregation pipelines and map-reduce frameworks.

1. Replica Set
- A group of MongoDB servers that maintain identical data copies, providing redundancy and high availability.

1. Sharding
- The process of distributing data across multiple servers, designed to handle large volumes of data and high throughput applications.

### 2. Quick Start

#### Using Python
```python
# Connect to MongoDB server
mongosh "mongodb://localhost:27017" -u "root" -p "root.password"

# show databases
show dbs

# Select database
use mydatabase

# Create (Insert)
db.mycollection.insertOne({
"name": "Alice",
"age": 30,
"email": "[email protected]"
})

# Read (Find)
db.mycollection.find()

# Update
db.mycollection.updateOne(
{ "name": "Alice" },
{ $set: { "age": 31 } }
)

# Delete
db.mycollection.deleteOne({ "name": "Alice" })

# Drop collection
db.mycollection.drop()

# Drop database
db.dropDatabase()

```

#### Using Command Line
Execute the following commands in the MongoDB Pod:

```bash
# Connect to MongoDB server
mongosh "mongodb://localhost:27017" -u "root" -p "root.password"

# show databases
show dbs

# Select database
use mydatabase

# Create (Insert)
db.mycollection.insertOne({
"name": "Alice",
"age": 30,
"email": "[email protected]"
})

# Read (Find)
db.mycollection.find()

# Update
db.mycollection.updateOne(
{ "name": "Alice" },
{ $set: { "age": 31 } }
)

# Delete
db.mycollection.deleteOne({ "name": "Alice" })

# Drop collection
db.mycollection.drop()

# Drop database
db.dropDatabase()

```


### 3. FAQ

1. **How to Reduce the Number of Shards?**
- Refer to the official MongoDB documentation on [removing shards from a cluster](https://www.mongodb.com/docs/manual/tutorial/remove-shards-from-cluster/).
- Since shards are Kubernetes StatefulSet workloads, you need to remove the shard with the highest number, such as `mongodb-sharded-shard-<max-number>`.
- Then adjust the number of shards in KDP (Kubernetes Deployment Platform).

2. **How to Configure Component Numbers in a Production Environment?**
- Consult the MongoDB documentation on [production configuration for sharded cluster components](https://www.mongodb.com/docs/manual/core/sharded-cluster-components/#production-configuration).

Additional FAQs can be found at the [MongoDB FAQ](https://www.mongodb.com/docs/manual/faq/).
13 changes: 13 additions & 0 deletions catalog/mongodb/apps/mongodb.app/metadata.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
version: 7.0.12
alias: MongoDB
description: MongoDB 是一种文档数据库,它所具备的可扩展性和灵活性可以满足您对查询和索引的需求
isGlobal: false
i18n:
en:
description: >-
MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need.
dashboard:
- id: mongodb
name: MongoDB 监控面板
i18n:
en: MongoDB Dashboard
19 changes: 19 additions & 0 deletions catalog/mongodb/i18n/en/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
### 1. Introduction
MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need.

### 2. Product Features

#### Document Storage
Stores data in a JSON-like format, offering flexibility and adaptability for various data structures.

#### High Performance
Fast read and write speeds, supports nested documents and arrays, reducing join operations and speeding up queries.

#### High Availability
Automatically replicates data through replica sets; if one server fails, others can take over, ensuring continuous operation without downtime.

#### Scalability
Distributes data across multiple machines using sharding technology, easily handling data growth and increased traffic.

#### Powerful Querying
Offers a rich query language, supporting various complex queries and real-time data analysis.
8 changes: 8 additions & 0 deletions catalog/mongodb/metadata.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
name: MongoDB
category: 系统/存储系统
description: MongoDB 是一种文档数据库,它所具备的可扩展性和灵活性可以满足您对查询和索引的需求
i18n:
en:
category: system.storage
description: >-
MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need.
Loading
Loading