Skip to content

Commit f43d09e

Browse files
amotlamotl
committed
Prometheus: Refresh usage guide based on Docker Compose
1 parent 91d3c42 commit f43d09e

File tree

7 files changed

+133
-159
lines changed

7 files changed

+133
-159
lines changed

‎docs/howto/index.md‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Instructions how to get tasks done with CrateDB.
1111
:::{rubric} 2025
1212
:::
1313

14+
- {ref}`prometheus-usage`
1415
- {ref}`influxdb-usage`
1516
- {ref}`amqp-usage`
1617
- {ref}`mqtt-usage`
@@ -62,11 +63,6 @@ Instructions how to get tasks done with CrateDB.
6263
- {ref}`terraform-usage`
6364
- {ref}`trino-usage`
6465

65-
:::{rubric} 2022
66-
:::
67-
68-
- {ref}`prometheus-usage`
69-
7066

7167
:::{note}
7268
You can expect the more recent documents to be more up-to-date than previous

‎docs/integrate/prometheus/compose.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Service composition file for Docker Compose or Podman Compose
2+
# https://cratedb.com/docs/guide/integrate/prometheus/
3+
4+
services:
5+
6+
cratedb:
7+
image: "docker.io/crate:latest"
8+
command: >
9+
crate -Cdiscovery.type=single-node
10+
ports:
11+
- "4200:4200"
12+
- "5432:5432"
13+
volumes:
14+
- cratedb-data:/data
15+
restart: unless-stopped
16+
17+
cratedb-ddl:
18+
image: docker.io/crate/crate:latest
19+
command: sh -c "crash --hosts http://cratedb:4200/ -c 'SELECT 1'; crash --hosts http://cratedb:4200/ < /var/ddl.sql"
20+
volumes:
21+
- ./ddl.sql:/var/ddl.sql
22+
depends_on:
23+
- cratedb
24+
25+
prometheus:
26+
image: "docker.io/prom/prometheus:latest"
27+
volumes:
28+
- ./prometheus.yml:/etc/prometheus/prometheus.yml
29+
ports:
30+
- "9090:9090"
31+
restart: unless-stopped
32+
33+
cratedb-prometheus-adapter:
34+
image: ghcr.io/crate/cratedb-prometheus-adapter:0.5.8
35+
command: -config.file /etc/cratedb-prometheus-adapter.yaml
36+
ports:
37+
- "9268:9268/tcp"
38+
volumes:
39+
- ./cratedb-prometheus-adapter.yaml:/etc/cratedb-prometheus-adapter.yaml
40+
depends_on:
41+
- cratedb-ddl
42+
- prometheus
43+
restart: unless-stopped
44+
45+
volumes:
46+
cratedb-data:

‎docs/integrate/prometheus/cratedb-prometheus-adapter.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# https://cratedb.com/docs/guide/integrate/prometheus/
2+
3+
cratedb_endpoints:
4+
- host: "cratedb" # Host to connect to (default: "localhost").
5+
port: 5432 # Port to connect to (default: 5432).
6+
user: "crate" # Username to use (default: "crate")
7+
password: "" # Password to use (default: "").
8+
schema: "testdrive" # Schema to use (default: "").
9+
max_connections: 0 # The maximum number of concurrent connections (default: runtime.NumCPU()).
10+
# It will get forwarded to pgx's `pool_max_conns`, and determines
11+
# the maximum number of connections in the connection pool for
12+
# both connection pools (read and write).
13+
read_pool_size_max: 0 # Configure the maximum pool size for read operations individually.
14+
# (default: runtime.NumCPU())
15+
write_pool_size_max: 0 # Configure the maximum pool size for write operations individually.
16+
# (default: runtime.NumCPU())
17+
connect_timeout: 10 # TCP connect timeout (seconds) (default: 10).
18+
# It has the same meaning as libpq's `connect_timeout`.
19+
read_timeout: 5 # Query context timeout for read queries (seconds) (default: 5).
20+
write_timeout: 5 # Query context timeout for write queries (seconds) (default: 5).
21+
enable_tls: false # Whether to connect using TLS (default: false).
22+
allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).

‎docs/integrate/prometheus/ddl.sql‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
-- https://cratedb.com/docs/guide/integrate/prometheus/
2+
3+
CREATE TABLE IF NOT EXISTS "testdrive"."metrics" (
4+
"timestamp" TIMESTAMP,
5+
"labels_hash" STRING,
6+
"labels" OBJECT(DYNAMIC),
7+
"value" DOUBLE,
8+
"valueRaw" LONG,
9+
"day__generated" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', "timestamp"),
10+
PRIMARY KEY ("timestamp", "labels_hash", "day__generated")
11+
) PARTITIONED BY ("day__generated");

‎docs/integrate/prometheus/index.md‎

Lines changed: 3 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@
77
```{div} .clearfix
88
```
99

10-
11-
## About
10+
:::{rubric} About
11+
:::
1212

1313
[Prometheus] is an open-source systems monitoring and alerting toolkit
1414
for collecting metrics data from applications and infrastructures.
@@ -37,8 +37,6 @@ Prometheus's main features are:
3737
::::
3838

3939

40-
## Learn
41-
4240
:::{rubric} Remote Endpoints and Storage
4341
:::
4442

@@ -57,15 +55,10 @@ massively scale-out Prometheus.
5755

5856
::::{grid}
5957

60-
:::{grid-item-card} Blog: Getting started
61-
:link: https://cratedb.com/blog/getting-started-prometheus-cratedb-long-term-storage
62-
:link-type: url
63-
Getting started with Prometheus and CrateDB for long-term storage.
64-
:::
65-
6658
:::{grid-item-card} Store Prometheus long-term metrics into CrateDB
6759
:link: prometheus-usage
6860
:link-type: ref
61+
:columns: 6
6962
Set up CrateDB as a long-term metrics store for Prometheus using Docker Compose.
7063
:::
7164

@@ -99,13 +92,6 @@ usage guide.
9992
::::
10093

10194

102-
:::{rubric} Repositories
103-
:::
104-
105-
[CrateDB] •
106-
[Prometheus] •
107-
[CrateDB Prometheus Adapter]
108-
10995

11096
```{seealso}
11197
[CrateDB and Prometheus]

‎docs/integrate/prometheus/prometheus.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# https://cratedb.com/docs/guide/integrate/prometheus/
2+
3+
global:
4+
scrape_interval: 15s # By default, scrape targets every 15 seconds.
5+
6+
# Attach these labels to any time series or alerts when communicating with
7+
# external systems (federation, remote storage, Alertmanager).
8+
external_labels:
9+
monitor: 'codelab-monitor'
10+
11+
# A scrape configuration containing exactly one endpoint to scrape:
12+
# Here it's Prometheus itself.
13+
scrape_configs:
14+
# The job name is added as a label `job=<job_name>` to any time-series scraped from this config.
15+
- job_name: 'prometheus'
16+
17+
# Override the global default and scrape targets from this job every 5 seconds.
18+
scrape_interval: 5s
19+
20+
static_configs:
21+
- targets: ['prometheus:9090']
22+
23+
remote_write:
24+
- url: http://cratedb-prometheus-adapter:9268/write
25+
remote_read:
26+
- url: http://cratedb-prometheus-adapter:9268/read

‎docs/integrate/prometheus/usage.md‎

Lines changed: 24 additions & 137 deletions
Original file line numberDiff line numberDiff line change
@@ -2,149 +2,39 @@
22
# Store Prometheus long-term metrics into CrateDB
33

44
This usage guide shows how to use Docker Compose to run the services
5-
CrateDB, Prometheus, and the CrateDB Prometheus Adapter.
6-
7-
## Motivation
8-
9-
[Prometheus] is a monitoring system that collects metrics from applications and
10-
infrastructure. It focuses on ingesting large volumes of concise time‑series
11-
events—timestamped points with key‑value labels.
12-
13-
This data helps you track a system’s state and trajectory. Many Prometheus users
14-
want to retain it long term.
15-
16-
[CrateDB] helps here. With the [CrateDB Prometheus Adapter], you can store metrics
17-
in CrateDB and use its fast ingestion and query performance to scale Prometheus.
18-
19-
## Set up Docker Compose
20-
21-
Run CrateDB, Prometheus, and the CrateDB Prometheus Adapter as [Docker containers].
22-
Use [Docker Compose] to centralize container management so you can build and run
23-
all containers with one command and define their connections in a YAML file.
24-
25-
Install Docker by following the [Docker installation guide].
26-
27-
Create a directory on your machine to host the configuration files.
28-
Create three YAML files with your preferred editor and save them with the `.yml` extension.
29-
30-
### Create `docker-compose.yml`
31-
32-
Create `docker-compose.yml` to configure the three containers.
33-
Define services for CrateDB, Prometheus, and the adapter. Mount `config.yml`
34-
and `prometheus.yml` into the adapter and Prometheus containers, respectively.
35-
You will create these files in the following steps.
36-
```yaml
37-
services:
38-
cratedb:
39-
image: "docker.io/crate:latest"
40-
command: >
41-
crate -Cdiscovery.type=single-node
42-
ports:
43-
- "4200:4200"
44-
- "5432:5432"
45-
volumes:
46-
- cratedb-data:/data
47-
restart: unless-stopped
48-
prometheus:
49-
image: "docker.io/prom/prometheus:latest"
50-
volumes:
51-
- ./prometheus.yml:/etc/prometheus/prometheus.yml
52-
ports:
53-
- "9090:9090"
54-
restart: unless-stopped
55-
cratedb-prometheus-adapter:
56-
image: "ghcr.io/crate/cratedb-prometheus-adapter:latest"
57-
volumes:
58-
- ./config.yml:/etc/cratedb-prometheus-adapter/config.yml
59-
ports:
60-
- "9268:9268"
61-
depends_on:
62-
- cratedb
63-
- prometheus
64-
restart: unless-stopped
65-
volumes:
66-
cratedb-data:
67-
```
68-
69-
### Create `prometheus.yml`
70-
71-
Next, create `prometheus.yml` following the [Prometheus documentation].
72-
This file defines the services that Prometheus scrapes.
73-
74-
To keep it simple, monitor Prometheus itself.
75-
76-
To forward samples to CrateDB, set `remote_write` and `remote_read` to the adapter URL (see the [CrateDB Prometheus Adapter setup](https://github.com/crate/cratedb-prometheus-adapter)).
5+
CrateDB, Prometheus, and the [CrateDB Prometheus Adapter].
776

78-
Because the adapter runs in Docker, use the adapter service name from `docker-compose.yml` (`cratedb-prometheus-adapter`) instead of `localhost` in the URLs.
7+
## Prerequisites
798

80-
Use the following `prometheus.yml`:
81-
```yaml
82-
global:
83-
scrape_interval: 15s # By default, scrape targets every 15 seconds.
9+
Use Docker or Podman to run all components. This approach works consistently
10+
across Linux, macOS, and Windows.
8411

85-
# Attach these labels to any time series or alerts when communicating with
86-
# external systems (federation, remote storage, Alertmanager).
87-
external_labels:
88-
monitor: 'codelab-monitor'
12+
### Files
8913

90-
# A scrape configuration containing exactly one endpoint to scrape:
91-
# Here it's Prometheus itself.
92-
scrape_configs:
93-
# The job name is added as a label `job=<job_name>` to any time-series scraped from this config.
94-
- job_name: 'prometheus'
14+
First, download and save all required files to your machine.
15+
- {download}`compose.yaml`
16+
- {download}`cratedb-prometheus-adapter.yaml`
17+
- {download}`ddl.sql`
18+
- {download}`prometheus.yml`
9519

96-
# Override the global default and scrape targets from this job every 5 seconds.
97-
scrape_interval: 5s
20+
### Services
9821

99-
static_configs:
100-
- targets: ['prometheus:9090']
22+
Start services using Docker Compose or Podman Compose.
23+
If you use Podman, replace `docker` with `podman` (or enable the podman‑docker
24+
compatibility shim) and run `podman compose up`.
10125

102-
remote_write:
103-
- url: http://cratedb-prometheus-adapter:9268/write
104-
remote_read:
105-
- url: http://cratedb-prometheus-adapter:9268/read
106-
```
107-
### Create `config.yml`
108-
109-
Finally, create `config.yml` following the [CrateDB Prometheus Adapter].
110-
This file defines the CrateDB endpoints that the adapter writes to.
111-
112-
Set the host to `cratedb` (the service name in `docker-compose.yml`) instead of `localhost`. Keep the remaining variables at their defaults.
113-
```yaml
114-
cratedb_endpoints:
115-
- host: "cratedb" # Host to connect to (default: "localhost")
116-
port: 5432 # Port to connect to (default: 5432).
117-
user: "crate" # Username to use (default: "crate")
118-
password: "" # Password to use (default: "").
119-
schema: "" # Schema to use (default: "").
120-
connect_timeout: 10 # TCP connect timeout (seconds) (default: 10).
121-
max_connections: 5 # The maximum number of concurrent connections (default: 5).
122-
enable_tls: false # Whether to connect using TLS (default: false).
123-
allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).
26+
```shell
27+
docker compose up
12428
```
125-
Place `docker-compose.yml`, `config.yml`, and `prometheus.yml` in the same directory on your machine.
12629

127-
## Start the services
30+
## Explore data
12831

129-
Finally, navigate to your CrateDB–Prometheus directory and start Docker Compose:
32+
CrateDB stores the metrics in the designated table, ready for inspection and analysis.
13033
```shell
131-
cd /Users/Path/To/Directory/CrateDB-Prometheus
132-
docker-compose up
34+
docker compose exec cratedb crash -c "SELECT * FROM testdrive.metrics ORDER BY timestamp LIMIT 5;"
13335
```
13436

135-
After Docker Compose starts, open the CrateDB Admin UI at `http://localhost:4200`
136-
and create a `metrics` table to store metrics gathered by Prometheus.
137-
```sql
138-
CREATE TABLE "metrics" (
139-
"timestamp" TIMESTAMP,
140-
"labels_hash" STRING,
141-
"labels" OBJECT(DYNAMIC),
142-
"value" DOUBLE,
143-
"valueRaw" LONG,
144-
"day__generated" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', "timestamp"),
145-
PRIMARY KEY ("timestamp", "labels_hash", "day__generated")
146-
) PARTITIONED BY ("day__generated")
147-
```
37+
## Screenshots
14838

14939
Navigate to `http://localhost:9090` to open the Prometheus UI. Go to **Status** → **Targets**.
15040

@@ -156,23 +46,20 @@ Confirm that Prometheus monitors itself.
15646

15747
Return to the CrateDB Admin UI and select the `metrics` table you created.
15848

159-
After a few minutes, Prometheus will have gathered hundreds of thousands of data points.
49+
After a few minutes, Prometheus will have gathered a few thousands of data points.
16050

16151
![CrateDB Admin UI showing the populated metrics table](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/22e8c7d5a90ec9240a4cb4269774e143759aa92e.jpeg)
16252

16353
Use CrateDB’s query engine to analyze and visualize this data with tools
164-
like {ref}`grafana`.
165-
166-
Explore these related resources:
167-
168-
* [Visualizing time‑series data with Grafana and CrateDB](https://cratedb.com/blog/visualizing-time-series-data-with-grafana-and-cratedb)
169-
* [Monitoring CrateDB with Prometheus and Grafana](https://cratedb.com/blog/monitoring-cratedb-with-prometheus-and-grafana)
54+
like {ref}`grafana`, see also
55+
[Monitoring a self-managed CrateDB cluster with Prometheus and Grafana].
17056

17157

17258
[CrateDB]: https://cratedb.com/database
17359
[CrateDB Prometheus Adapter]: https://github.com/crate/cratedb-prometheus-adapter
17460
[Docker Compose]: https://docs.docker.com/compose/
17561
[Docker containers]: https://www.docker.com/resources/what-container
17662
[Docker installation guide]: https://docs.docker.com/get-docker/
63+
[Monitoring a self-managed CrateDB cluster with Prometheus and Grafana]: https://community.cratedb.com/t/monitoring-a-self-managed-cratedb-cluster-with-prometheus-and-grafana/1236
17764
[Prometheus]: https://prometheus.io/docs/introduction/overview/
17865
[Prometheus documentation]: https://prometheus.io/docs/prometheus/latest/getting_started/

0 commit comments

Comments
 (0)