Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions .github/workflows/docker-publish-master.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Docker Publish - update.redisearch branch
name: Docker Publish - Latest

on:
push:
Expand All @@ -11,7 +11,7 @@ on:

env:
REGISTRY: docker.io
IMAGE_NAME: filipe958/vector-db-benchmark
IMAGE_NAME: redis/vector-db-benchmark

jobs:
docker-publish:
Expand Down Expand Up @@ -61,8 +61,6 @@ jobs:
images: ${{ env.IMAGE_NAME }}
tags: |
type=raw,value=latest
type=raw,value=update-redisearch-{{sha}}
type=raw,value=update-redisearch-{{date 'YYYYMMDD-HHmmss'}}

- name: Build and push Docker image
uses: docker/build-push-action@v5
Expand Down Expand Up @@ -95,4 +93,4 @@ jobs:
echo "docker run --rm ${{ env.IMAGE_NAME }}:latest run.py --help" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/filipe958/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/redis/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY
7 changes: 2 additions & 5 deletions .github/workflows/docker-publish-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:

env:
REGISTRY: docker.io
IMAGE_NAME: filipe958/vector-db-benchmark
IMAGE_NAME: redis/vector-db-benchmark

jobs:
docker-publish:
Expand Down Expand Up @@ -57,9 +57,6 @@ jobs:
tags: |
type=ref,event=tag
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=raw,value=latest,enable={{is_default_branch}}

- name: Build and push Docker image
uses: docker/build-push-action@v5
Expand Down Expand Up @@ -110,5 +107,5 @@ jobs:
echo "docker run --rm ${{ env.IMAGE_NAME }}:latest run.py --help" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/filipe958/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/redis/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY
echo "🔒 [Security Scan Results](https://github.com/${{ github.repository }}/security/code-scanning)" >> $GITHUB_STEP_SUMMARY
176 changes: 176 additions & 0 deletions DOCKER_README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
# Redis Vector Database Benchmark

A comprehensive benchmarking tool for vector databases, including Redis (both RediSearch and Vector Sets), Weaviate, Milvus, Qdrant, OpenSearch, Postgres, and others...

In a one-liner cli tool you can get this and much more:

```
docker run --rm --network=host redis/vector-db-benchmark:latest run.py --host localhost --engines vectorsets-fp32-default --datasets glove-100-angular --parallels 100
(...)
================================================================================
BENCHMARK RESULTS SUMMARY
Experiment: vectorsets-fp32-default - glove-100-angular
================================================================================

Precision vs Performance Trade-off:
--------------------------------------------------
Precision QPS P50 (ms) P95 (ms)
--------------------------------------------------
0.86 1408.3 61.877 107.548
0.80 2136.3 38.722 69.102
0.72 2954.3 25.820 48.072
0.68 3566.5 20.229 38.581

QPS vs Precision Trade-off - vectorsets-fp32-default - glove-100-angular (up and to the right is better):

3566 │●
│ ●
2594 │
│ ●
1621 │ ●
648 │
0 │
└────────────────────────────────────────────────────────────
0.680 0.726 0.772 0.817
Precision (0.0 = 0%, 1.0 = 100%)
================================================================================

```

## Quick Start

```bash
# Pull the latest image
docker pull redis/vector-db-benchmark:latest

# Run with help
docker run --rm redis/vector-db-benchmark:latest run.py --help

# Check available datasets
docker run --rm redis/vector-db-benchmark:latest run.py --describe datasets

# Basic Redis benchmark (requires local Redis)
docker run --rm -v $(pwd)/results:/app/results --network=host \
redis/vector-db-benchmark:latest \
run.py --host localhost --engines redis-default-simple --dataset random-100
```

## Features

- **42+ Datasets**: Pre-configured datasets from 25 to 1B+ vectors
- **Multiple Engines**: Redis, Qdrant, Weaviate, Milvus, and more
- **Real-time Monitoring**: Live performance metrics during benchmarks
- **Precision Analysis**: Detailed accuracy vs performance trade-offs
- **Easy Discovery**: `--describe` commands for datasets and engines

## Available Tags

- `latest` - Latest development build from update.redisearch branch

## Redis quick start

### Redis 8.2 with RediSearch
```bash
# Start Redis 8.2 with built-in vector support
docker run -d --name redis-test -p 6379:6379 redis:8.2-rc1-bookworm

# Run benchmark
docker run --rm -v $(pwd)/results:/app/results --network=host \
redis/vector-db-benchmark:latest \
run.py --host localhost --engines redis-default-simple --dataset glove-25-angular
```


## Common Usage Patterns

### Explore Available Options
```bash
# List all datasets
docker run --rm redis/vector-db-benchmark:latest run.py --describe datasets

# List all engines
docker run --rm redis/vector-db-benchmark:latest run.py --describe engines
```

### Run Benchmarks
```bash
# Quick test with small dataset
docker run --rm -v $(pwd)/results:/app/results --network=host \
redis/vector-db-benchmark:latest \
run.py --host localhost --engines redis-default-simple --dataset random-100

# Comprehensive benchmark with multiple configurations
docker run --rm -v $(pwd)/results:/app/results --network=host \
redis/vector-db-benchmark:latest \
run.py --host localhost --engines "*redis*" --dataset glove-25-angular

# With Redis authentication
docker run --rm -v $(pwd)/results:/app/results --network=host \
-e REDIS_AUTH=mypassword -e REDIS_USER=myuser \
redis/vector-db-benchmark:latest \
run.py --host localhost --engines redis-default-simple --dataset random-100
```

### Results Analysis
```bash
# View precision summary
jq '.precision_summary' results/*-summary.json

# View detailed results
jq '.search' results/*-summary.json
```

## Volume Mounts

- `/app/results` - Benchmark results (JSON files)
- `/app/datasets` - Dataset storage (optional, auto-downloaded)

## Environment Variables

- `REDIS_HOST` - Redis server hostname (default: localhost)
- `REDIS_PORT` - Redis server port (default: 6379)
- `REDIS_AUTH` - Redis password (default: None)
- `REDIS_USER` - Redis username (default: None)
- `REDIS_CLUSTER` - Enable Redis cluster mode (default: 0)

## Performance Tips

1. **Use `--network=host`** for best performance with local Redis
2. **Mount results volume** to persist benchmark data
3. **Start with small datasets** (random-100, glove-25-angular) for testing
4. **Use wildcard patterns** to test multiple configurations: `--engines "*-m-16-*"`

## Example Output

```json
{
"precision_summary": {
"0.91": {
"qps": 1924.5,
"p50": 49.828,
"p95": 58.427
},
"0.94": {
"qps": 1819.9,
"p50": 51.68,
"p95": 66.83
}
}
}
```

## Support

- **GitHub**: [redis-performance/vector-db-benchmark](https://github.com/redis-performance/vector-db-benchmark)
- **Issues**: Report bugs and feature requests on GitHub
- **Documentation**: Full documentation available in the repository

## License

This project is licensed under the MIT License - see the repository for details.
27 changes: 12 additions & 15 deletions DOCKER_SETUP.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Docker Setup and Publishing Guide

This guide explains how to set up Docker publishing for the `vector-db-benchmark` project to Docker Hub repository `filipe958/vector-db-benchmark`.
This guide explains how to set up Docker publishing for the `vector-db-benchmark` project to Docker Hub repository `redis/vector-db-benchmark`.

## 🔐 Required GitHub Secrets

Expand Down Expand Up @@ -57,11 +57,8 @@ Once secrets are configured, Docker images will be automatically published:

### Example Tags for Release v1.2.3
```
filipe958/vector-db-benchmark:v1.2.3
filipe958/vector-db-benchmark:1.2.3
filipe958/vector-db-benchmark:1.2
filipe958/vector-db-benchmark:1
filipe958/vector-db-benchmark:latest
redis/vector-db-benchmark:v1.2.3
redis/vector-db-benchmark:latest
```

## 🛠️ Manual Building and Publishing
Expand Down Expand Up @@ -100,32 +97,32 @@ export DOCKER_PASSWORD=your_access_token
### Pull and Run
```bash
# Latest version
docker pull filipe958/vector-db-benchmark:latest
docker run --rm filipe958/vector-db-benchmark:latest run.py --help
docker pull redis/vector-db-benchmark:latest
docker run --rm redis/vector-db-benchmark:latest run.py --help

# Specific version
docker pull filipe958/vector-db-benchmark:v1.2.3
docker run --rm filipe958/vector-db-benchmark:v1.2.3 run.py --help
docker pull redis/vector-db-benchmark:v1.2.3
docker run --rm redis/vector-db-benchmark:v1.2.3 run.py --help
```

### Example Usage
```bash
# Basic Redis benchmark
docker run --rm --network=host filipe958/vector-db-benchmark:latest \
docker run --rm --network=host redis/vector-db-benchmark:latest \
run.py --host localhost --engines redis --dataset random-100 --experiment redis-default-simple

# With custom Redis host
docker run --rm filipe958/vector-db-benchmark:latest \
docker run --rm redis/vector-db-benchmark:latest \
run.py --host redis-server --engines redis --dataset random-100 --experiment redis-default-simple

# With results output (mount current directory)
docker run --rm -v $(pwd)/results:/app/results --network=host \
filipe958/vector-db-benchmark:latest \
redis/vector-db-benchmark:latest \
run.py --host localhost --engines redis --dataset random-100 --experiment redis-default-simple

# Using with Redis container
docker run -d --name redis-test -p 6379:6379 redis:8.2-rc1-bookworm
docker run --rm --network=host filipe958/vector-db-benchmark:latest \
docker run --rm --network=host redis/vector-db-benchmark:latest \
run.py --host localhost --engines redis --experiment redis-default-simple
docker stop redis-test && docker rm redis-test
```
Expand All @@ -145,7 +142,7 @@ docker stop redis-test && docker rm redis-test
- Prevents merging PRs with broken Docker builds

### Docker Hub
- View images at: https://hub.docker.com/r/filipe958/vector-db-benchmark
- View images at: https://hub.docker.com/r/redis/vector-db-benchmark
- Check image sizes and platforms
- Review vulnerability scan results

Expand Down
Loading