Skip to content

Commit 4186364

Browse files
imabhichowimabhichow
committed
python-benchmark
1 parent 50ac00e commit 4186364

File tree

15 files changed

+2730
-0
lines changed

15 files changed

+2730
-0
lines changed

‎db-esdk-performance-testing/benchmarks/python/README.md‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
# DB-ESDK Performance Benchmark - Python
2+
3+
This directory contains the Python implementation of the AWS Database Encryption SDK (DB-ESDK) performance benchmark suite.
4+
5+
## Overview
6+
7+
The Python benchmark provides comprehensive performance testing for the DB-ESDK Python runtime, measuring:
8+
9+
- **Throughput**: Operations per second and bytes per second using ItemEncryptor operations
10+
- **Latency**: Encrypt, decrypt, and end-to-end timing for encrypted operations
11+
- **Memory Usage**: Peak memory consumption and efficiency
12+
- **Concurrency**: Multi-threaded performance scaling
13+
- **Statistical Analysis**: P50, P95, P99 latency percentiles
14+
15+
## Prerequisites
16+
17+
- Python 3.11 or higher
18+
- Poetry package manager
19+
20+
## Setup
21+
22+
### Install Poetry
23+
24+
```bash
25+
# Install Poetry (if not already installed)
26+
curl -sSL https://install.python-poetry.org | python3 -
27+
28+
# Or using pip
29+
pip install poetry
30+
```
31+
32+
### Install Dependencies
33+
34+
```bash
35+
# Install all dependencies including dev dependencies
36+
poetry install
37+
38+
# Install only production dependencies
39+
poetry install --no-dev
40+
```
41+
42+
## Running Benchmarks
43+
44+
### Quick Test
45+
46+
```bash
47+
# Using Poetry
48+
poetry run esdk-benchmark --quick
49+
50+
# Using tox (recommended for isolated environment)
51+
tox -e benchmark
52+
53+
# Using module execution
54+
poetry run python -m esdk_benchmark --quick
55+
56+
# Direct script execution
57+
poetry run python src/esdk_benchmark/program.py --quick
58+
```
59+
60+
### Full Benchmark Suite
61+
62+
```bash
63+
# Using Poetry
64+
poetry run esdk-benchmark
65+
66+
# Using tox (recommended for isolated environment)
67+
tox -e benchmark-full
68+
69+
# Using module execution
70+
poetry run python -m esdk_benchmark
71+
72+
# Direct script execution
73+
poetry run python src/esdk_benchmark/program.py
74+
```
75+
76+
### Custom Configuration
77+
78+
```bash
79+
# Specify custom config and output paths
80+
poetry run esdk-benchmark \
81+
--config /path/to/config.yaml \
82+
--output /path/to/results.json
83+
```
84+
85+
## Command Line Options
86+
87+
- `--config, -c`: Path to test configuration file (default: `../../../config/test-scenarios.yaml`)
88+
- `--output, -o`: Path to output results file (default: `../../../results/raw-data/python_results.json`)
89+
- `--quick, -q`: Run quick test with reduced iterations
90+
- `--help, -h`: Show help message
91+
92+
## Configuration
93+
94+
The benchmark uses a YAML configuration file to define test parameters:
95+
96+
```yaml
97+
data_sizes:
98+
small: [1024, 5120, 10240]
99+
medium: [102400, 512000, 1048576]
100+
large: [10485760, 52428800, 104857600]
101+
102+
iterations:
103+
warmup: 5
104+
measurement: 10
105+
106+
concurrency_levels: [1, 2, 4, 8]
107+
```
108+
109+
## Output Format
110+
111+
Results are saved in JSON format with the following structure:
112+
113+
```json
114+
{
115+
"metadata": {
116+
"language": "python",
117+
"timestamp": "2025-09-05T15:30:00Z",
118+
"python_version": "3.11.5",
119+
"platform": "Darwin-23.1.0-arm64-arm-64bit",
120+
"cpu_count": 8,
121+
"total_memory_gb": 16.0,
122+
"total_tests": 45
123+
},
124+
"results": [
125+
{
126+
"test_name": "throughput",
127+
"language": "python",
128+
"data_size": 1024,
129+
"concurrency": 1,
130+
"put_latency_ms": 0.85,
131+
"get_latency_ms": 0.72,
132+
"end_to_end_latency_ms": 1.57,
133+
"ops_per_second": 636.94,
134+
"bytes_per_second": 652224.0,
135+
"peak_memory_mb": 0.0,
136+
"memory_efficiency_ratio": 0.0,
137+
"p50_latency": 1.55,
138+
"p95_latency": 1.89,
139+
"p99_latency": 2.12,
140+
"timestamp": "2025-09-05T15:30:15Z",
141+
"python_version": "3.11.5",
142+
"cpu_count": 8,
143+
"total_memory_gb": 16.0
144+
}
145+
]
146+
}
147+
```
148+
149+
## Key Features
150+
151+
### DB-ESDK Integration
152+
153+
- Uses AWS Database Encryption SDK for DynamoDB with transparent encryption
154+
- Configures attribute actions (ENCRYPT_AND_SIGN, SIGN_ONLY, DO_NOTHING)
155+
- Tests ItemEncryptor operations with client-side encryption
156+
- Uses Raw AES keyring for consistent performance testing
157+
158+
### ItemEncryptor Operations
159+
160+
- Performs encrypt_python_item operations using Python dict format
161+
- Measures decrypt_python_item operations for consistency
162+
- Tests realistic workloads with encryption overhead
163+
- Supports multiple data formats (Python dict, DynamoDB JSON, DBESDK shapes)
164+
165+
### Performance Metrics
166+
167+
- **Throughput Tests**: Measures ops/sec and bytes/sec for ItemEncryptor operations
168+
- **Memory Tests**: Tracks peak memory usage during encrypted operations using psutil
169+
- **Concurrency Tests**: Evaluates multi-threaded performance scaling with ThreadPoolExecutor
170+
- **Latency Analysis**: P50, P95, P99 percentiles for operation timing
171+
172+
## Project Structure
173+
174+
```
175+
python/
176+
├── README.md # This file
177+
├── pyproject.toml # Poetry configuration and dependencies
178+
├── tox.ini # Tox configuration for testing
179+
├── src/
180+
│ └── esdk_benchmark/
181+
│ ├── __init__.py # Package initialization
182+
│ ├── __main__.py # Module execution entry point
183+
│ ├── program.py # Main program and CLI
184+
│ ├── benchmark.py # Core benchmark implementation
185+
│ ├── models.py # Data models and configuration
186+
│ └── tests.py # Individual test implementations
187+
├── tests/ # Test suite
188+
│ ├── __init__.py
189+
│ └── test_benchmark.py
190+
└── run_benchmark.py # Convenience runner script
191+
```

‎db-esdk-performance-testing/benchmarks/python/pyproject.toml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
[build-system]
2+
requires = ["poetry-core<2.0.0"]
3+
build-backend = "poetry.core.masonry.api"
4+
5+
[tool.poetry]
6+
name = "esdk-benchmark-python"
7+
version = "0.1.0"
8+
description = "Performance tests DB-ESDK Python"
9+
authors = ["AWS Crypto Tools <[email protected]>"]
10+
readme = "README.md"
11+
packages = [{include = "esdk_benchmark", from = "src"}]
12+
13+
[tool.poetry.dependencies]
14+
python = "^3.11.0"
15+
aws-dbesdk-dynamodb = { path = "../../../DynamoDbEncryption/runtimes/python", develop = false, extras = ["legacy-ddbec"]}
16+
boto3 = ">=1.26.0"
17+
PyYAML = "^6.0"
18+
pydantic = "^2.0.0"
19+
tqdm = "^4.66.0"
20+
psutil = "^5.9.0"
21+
numpy = "^1.24.0"
22+
23+
[tool.poetry.group.dev.dependencies]
24+
pytest = "^7.4.0"
25+
pytest-cov = "^4.1.0"
26+
black = "^23.0.0"
27+
flake8 = "^6.0.0"
28+
mypy = "^1.5.0"
29+
memory-profiler = "^0.60.0"
30+
tox = "^4.0.0"
31+
32+
[tool.poetry.scripts]
33+
esdk-benchmark = "esdk_benchmark.program:main"
34+
35+
[tool.black]
36+
line-length = 88
37+
target-version = ['py311']
38+
include = '\.pyi?$'
39+
exclude = '''
40+
/(
41+
\.eggs
42+
| \.git
43+
| \.hg
44+
| \.mypy_cache
45+
| \.tox
46+
| \.venv
47+
| _build
48+
| buck-out
49+
| build
50+
| dist
51+
)/
52+
'''
53+
54+
[tool.mypy]
55+
python_version = "3.11"
56+
warn_return_any = true
57+
warn_unused_configs = true
58+
disallow_untyped_defs = true
59+
disallow_incomplete_defs = true
60+
check_untyped_defs = true
61+
disallow_untyped_decorators = true
62+
no_implicit_optional = true
63+
warn_redundant_casts = true
64+
warn_unused_ignores = true
65+
warn_no_return = true
66+
warn_unreachable = true
67+
strict_equality = true
68+
69+
[[tool.mypy.overrides]]
70+
module = [
71+
"aws_cryptographic_material_providers.*",
72+
"aws_dbesdk_dynamodb.*",
73+
"tqdm",
74+
"psutil",
75+
]
76+
ignore_missing_imports = true
77+
78+
[tool.pytest.ini_options]
79+
minversion = "7.0"
80+
addopts = "-ra -q --strict-markers --strict-config"
81+
testpaths = ["tests"]
82+
pythonpath = ["src"]
83+
markers = [
84+
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
85+
"integration: marks tests as integration tests",
86+
]
87+
88+
[tool.coverage.run]
89+
source = ["src"]
90+
omit = [
91+
"*/tests/*",
92+
"*/test_*",
93+
]
94+
95+
[tool.coverage.report]
96+
exclude_lines = [
97+
"pragma: no cover",
98+
"def __repr__",
99+
"if self.debug:",
100+
"if settings.DEBUG",
101+
"raise AssertionError",
102+
"raise NotImplementedError",
103+
"if 0:",
104+
"if __name__ == .__main__.:",
105+
"class .*\\bProtocol\\):",
106+
"@(abc\\.)?abstractmethod",
107+
]
108+
109+
[tool.flake8]
110+
max-line-length = 88
111+
extend-ignore = ["E203", "W503"]
112+
max-complexity = 10
113+
per-file-ignores = [
114+
"__init__.py:F401",
115+
"tests/*:S101,D103",
116+
]

‎db-esdk-performance-testing/benchmarks/python/run_benchmark.py‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
#!/usr/bin/env python3
2+
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
import os
6+
import sys
7+
import subprocess
8+
from pathlib import Path
9+
10+
11+
def main():
12+
script_dir = Path(__file__).parent.absolute()
13+
os.chdir(script_dir)
14+
15+
print("=== DB-ESDK Python Performance Benchmark ===")
16+
print(f"Working directory: {script_dir}")
17+
18+
# Check virtual environment
19+
in_venv = hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix)
20+
if not in_venv:
21+
print("WARNING: Not running in a virtual environment.")
22+
print("It's recommended to create and activate a virtual environment:")
23+
print(" python -m venv benchmark-env")
24+
print(" source benchmark-env/bin/activate # (macOS/Linux)")
25+
print(" benchmark-env\\Scripts\\activate # (Windows)")
26+
print()
27+
28+
# Check Poetry availability
29+
try:
30+
subprocess.run(["poetry", "--version"], check=True, capture_output=True)
31+
print("Found Poetry package manager")
32+
except (subprocess.CalledProcessError, FileNotFoundError):
33+
print("Poetry not found. Please install Poetry first:")
34+
print(" curl -sSL https://install.python-poetry.org | python3 -")
35+
print(" # or")
36+
print(" pip install poetry")
37+
return 1
38+
39+
# Check package installation
40+
try:
41+
result = subprocess.run(["poetry", "run", "python", "-c", "import esdk_benchmark; print(esdk_benchmark.__file__)"],
42+
check=True, capture_output=True, text=True, cwd=script_dir)
43+
print(f"Found esdk_benchmark package at: {result.stdout.strip()}")
44+
except subprocess.CalledProcessError:
45+
print("Installing benchmark package with Poetry...")
46+
try:
47+
subprocess.run(["poetry", "install"], check=True, cwd=script_dir) # Install dependencies via Poetry
48+
print("Package installed successfully.")
49+
except subprocess.CalledProcessError as e:
50+
print(f"Failed to install package: {e}")
51+
print("Please install manually with: poetry install")
52+
return 1
53+
54+
args = sys.argv[1:]
55+
56+
if not args:
57+
print("\nRunning benchmark with default settings...")
58+
print("Use --help to see all available options.")
59+
print("Use --quick for a faster test run.")
60+
print()
61+
62+
# Execute benchmark via Poetry
63+
try:
64+
cmd = ["poetry", "run", "esdk-benchmark"] + args
65+
print(f"Running: {' '.join(cmd)}")
66+
result = subprocess.run(cmd, cwd=script_dir)
67+
return result.returncode
68+
except Exception as e:
69+
print(f"Benchmark execution failed: {e}")
70+
import traceback
71+
traceback.print_exc()
72+
return 1
73+
74+
75+
if __name__ == "__main__":
76+
sys.exit(main())

‎db-esdk-performance-testing/benchmarks/python/src/esdk_benchmark/__init__.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
"""
5+
AWS Database Encryption SDK (DB-ESDK) Performance Benchmark Suite - Python Implementation
6+
7+
This package provides comprehensive performance testing for the DB-ESDK Python runtime,
8+
measuring throughput, latency, memory usage, and concurrency scaling.
9+
"""
10+
11+
__version__ = "1.0.0"
12+
__author__ = "AWS Cryptography Team"

0 commit comments

Comments
 (0)