[Epic] 🗜️ Performance - Brotli/Zstd/GZip Response Compression

[crivetimihai]

🗜️ Performance - Brotli/Zstd/GZip Response Compression

Goal

Implement response compression middleware to reduce bandwidth usage and improve client load times:

1. Add starlette-compress middleware supporting Brotli, Zstd, and GZip compression
2. Auto-negotiate compression algorithm based on client Accept-Encoding headers
3. Configure optimal compression settings balancing CPU usage with compression ratio
4. Add response headers (Vary: Accept-Encoding) for proper cache behavior
5. Test compression across different content types and sizes

This reduces bandwidth by 30-70% for text-based responses (JSON, HTML, CSS, JS) and improves page load times, especially for clients on slower connections.

Why Now?

Response compression is a fundamental web performance optimization with immediate impact:

1. Bandwidth Reduction: 30-70% reduction in data transfer for JSON/HTML responses
2. Faster Load Times: Smaller payloads = faster page loads, especially on mobile networks
3. Better User Experience: Responsive UI, particularly for admin interface and large API responses
4. Cost Savings: Reduced bandwidth usage lowers hosting costs
5. Modern Standards: Brotli supported by 96%+ of browsers, outperforms gzip by 15-20%
6. Zero Client Changes: Transparent to API clients, works automatically

---

📖 User Stories

US-1: API Client - Receive Compressed JSON Responses

As an API Client
I want to receive compressed JSON responses
So that API calls are faster and use less bandwidth

Acceptance Criteria:

Given I am an API client that supports Brotli compression
When I send a request to GET /tools with header "Accept-Encoding: br"
Then the response should be compressed with Brotli
And the response header "Content-Encoding: br" should be present
And the response header "Vary: Accept-Encoding" should be present
And the response size should be 50-70% smaller than uncompressed

Given I am an API client that only supports gzip
When I send a request to GET /tools with header "Accept-Encoding: gzip"
Then the response should be compressed with gzip
And the response header "Content-Encoding: gzip" should be present
And the response size should be 40-60% smaller than uncompressed

Given I am an API client that doesn't support compression
When I send a request to GET /tools without Accept-Encoding header
Then the response should be uncompressed
And no Content-Encoding header should be present

Technical Requirements:

• Install starlette-compress middleware
• Configure compression middleware in main.py
• Set minimum_size=500 (don't compress tiny responses)
• Support Brotli, Zstd, and GZip algorithms
• Add Vary: Accept-Encoding header for cache compatibility

US-2: Admin UI User - Faster Page Loads with Compression

As an Admin UI User
I want the admin interface to load quickly
So that I can work efficiently without waiting for large data sets

Acceptance Criteria:

Given I am viewing the Admin UI tools page with 100+ tools
When the page loads
Then all HTML, CSS, JS, and JSON responses should be compressed
And the page should load 30-50% faster than without compression
And browser DevTools Network tab should show "Content-Encoding: br" or "gzip"

Given I am on a slow mobile connection
When I navigate between admin pages
Then compression should reduce data transfer
And pages should remain responsive

Technical Requirements:

• Compression applies to all text-based content (HTML, CSS, JS, JSON)
• Browser automatically decompresses responses
• No JavaScript changes required

US-3: DevOps Engineer - Configure Compression Settings

As a DevOps Engineer
I want to configure compression settings via environment variables
So that I can tune performance for my deployment

Acceptance Criteria:

Given I want to optimize for maximum compression
When I set COMPRESSION_LEVEL=9 in environment
Then responses should be compressed with maximum compression
And CPU usage should be higher but responses smaller

Given I want to optimize for speed
When I set COMPRESSION_LEVEL=1 in environment
Then responses should be compressed with minimum compression
And CPU usage should be lower but responses larger

Given I want to disable compression for debugging
When I set COMPRESSION_ENABLED=false in environment
Then no responses should be compressed
And Content-Encoding headers should not be present

Technical Requirements:

• Environment variables: COMPRESSION_ENABLED, COMPRESSION_LEVEL, COMPRESSION_MINIMUM_SIZE
• Default values documented in .env.example
• Settings applied at application startup

---

🏗 Architecture

Middleware Stack Order

graph TD
    A[Client Request] --> B[CompressionMiddleware]
    B --> C[SecurityHeadersMiddleware]
    C --> D[CORSMiddleware]
    D --> E[Application Logic]
    E --> F[Response]
    F --> D2[CORS Headers]
    D2 --> C2[Security Headers]
    C2 --> B2[Compress Response]
    B2 --> G[Compressed Response to Client]

Loading

Compression Algorithm Selection

graph TD
    A[Client Request] --> B{Check Accept-Encoding}
    B -->|Contains 'br'| C[Use Brotli]
    B -->|Contains 'zstd'| D[Use Zstd]
    B -->|Contains 'gzip'| E[Use GZip]
    B -->|No compression| F[Uncompressed]
    C --> G{Response Size > 500 bytes?}
    D --> G
    E --> G
    G -->|Yes| H[Compress]
    G -->|No| I[Don't Compress]
    H --> J[Add Content-Encoding header]
    J --> K[Send Response]
    I --> K
    F --> K

Loading

Configuration

# mcpgateway/main.py

from starlette_compress import CompressionMiddleware

# Add compression middleware (should be added early in middleware stack)
app.add_middleware(
    CompressionMiddleware,
    minimum_size=500,        # Only compress responses > 500 bytes
    gzip_level=6,           # Balanced gzip compression (1-9)
    brotli_quality=4,       # Balanced Brotli compression (0-11)
    zstd_level=3,           # Fast Zstd compression (1-22)
)

---

📋 Implementation Tasks

Phase 1: Dependencies & Setup ✅

• Add starlette-compress Dependency
  • Add starlette-compress>=1.0.0 to pyproject.toml dependencies section
  • Run make install-dev to install the package
  • Verify installation with python -c "import starlette_compress; print(starlette_compress.__version__)"

Phase 2: Middleware Configuration ✅

• Configure Compression Middleware

  • Import CompressionMiddleware in mcpgateway/main.py
  • Add middleware configuration after line 878 (before CORS middleware for proper ordering)
  • Configure compression settings:
    • minimum_size=500
    • gzip_level=6 (balanced)
    • brotli_quality=4 (balanced)
    • zstd_level=3 (fast)
  • Add detailed comments explaining each setting

• Verify Header Support

  • Verify Vary: Accept-Encoding header is automatically added
  • Verify Content-Encoding header shows correct algorithm
  • Verify Content-Length is recalculated after compression

Phase 3: Testing ✅

• Test Brotli Compression

  • Test with curl: curl -H "Accept-Encoding: br" http://localhost:8000/tools -v
  • Verify response has Content-Encoding: br header
  • Measure compressed vs uncompressed size

• Test GZip Compression

  • Test with curl: curl -H "Accept-Encoding: gzip" http://localhost:8000/tools -v
  • Verify response has Content-Encoding: gzip header
  • Measure compression ratio

• Test Zstd Compression

  • Test with curl: curl -H "Accept-Encoding: zstd" http://localhost:8000/tools -v
  • Verify response has Content-Encoding: zstd header
  • Measure compression ratio

• Test Minimum Size Threshold

  • Test with small response (<500 bytes): GET /health
  • Verify no compression is applied
  • Test with large response (>500 bytes): GET /tools
  • Verify compression is applied

• Browser Testing

  • Open admin UI in Chrome/Firefox
  • Check Network tab for Content-Encoding headers
  • Verify responses are automatically decompressed
  • Test with different network throttling settings

Phase 4: Performance Measurement ✅

• Measure Compression Ratios

  • Measure uncompressed response size for GET /tools with 100+ tools
  • Measure Brotli compressed size
  • Measure GZip compressed size
  • Measure Zstd compressed size
  • Calculate and document compression ratios

• Measure CPU Impact

  • Run load test without compression: wrk -t4 -c100 -d30s http://localhost:8000/tools
  • Record CPU usage percentage
  • Run load test with compression enabled
  • Record CPU usage percentage
  • Calculate CPU overhead percentage

Phase 5: Documentation ✅

• Update Project Documentation

  • Update CLAUDE.md with compression middleware configuration
  • Document compression settings and trade-offs
  • Add troubleshooting section for compression issues

• Update Configuration Documentation

  • Add compression configuration to .env.example if made configurable
  • Document compression levels and their trade-offs
  • Add example curl commands for testing compression

Phase 6: Quality Assurance ✅

• Code Quality

  • Run make autoflake isort black to format code
  • Run make flake8 and fix any issues
  • Run make pylint and address warnings
  • Pass make verify checks

• Testing

  • Add unit tests for compression configuration
  • Add integration tests for compression middleware
  • Verify all existing tests still pass

---

✅ Success Criteria

• starlette-compress installed and imported successfully
• Compression middleware configured in main.py with optimal settings
• Brotli compression working for clients that support it (96%+ of browsers)
• GZip compression working as universal fallback
• Zstd compression working for modern clients
• Vary: Accept-Encoding header present in all compressed responses
• Small responses (<500 bytes) not compressed to avoid overhead
• Compression ratio of 30-70% achieved for JSON/HTML responses
• CPU overhead acceptable (<20% increase)
• No breaking changes to API behavior
• Browser testing confirms automatic decompression works
• Documentation updated with configuration examples

---

🏁 Definition of Done

• Dependency added to pyproject.toml and installed
• CompressionMiddleware configured in main.py
• Compression settings optimized (minimum_size, quality levels)
• Vary header verified in responses
• All three compression algorithms tested (Brotli, GZip, Zstd)
• Compression ratios measured and documented (30-70% reduction)
• CPU overhead measured and acceptable
• Browser testing completed (Chrome, Firefox, Safari)
• Minimum size threshold verified working
• Code passes make verify checks
• Documentation updated (CLAUDE.md, comments)
• No regression in existing tests
• Ready for production deployment

---

📝 Additional Notes

🔹 Compression Algorithm Priority:

• Brotli: Best compression ratio (15-20% better than gzip), supported by 96%+ browsers
• Zstd: Fast compression with good ratio, growing support in modern browsers
• GZip: Universal fallback, 99%+ browser support

🔹 Compression Level Guidelines:

• Gzip: Level 6 (balanced) - lower = faster/larger, higher = slower/smaller
• Brotli: Quality 4 (balanced) - 0-3 fast, 4-9 balanced, 10-11 max compression
• Zstd: Level 3 (fast) - 1-3 fast, 4-9 balanced, 10+ slow

🔹 What Gets Compressed:

• ✅ JSON API responses (largest benefit)
• ✅ HTML pages (admin UI)
• ✅ CSS stylesheets
• ✅ JavaScript files
• ✅ SVG images
• ✅ Text-based content types

🔹 What Doesn't Get Compressed:

• ❌ Already compressed formats (PNG, JPG, GIF, WOFF2, MP4, ZIP)
• ❌ Small responses (<500 bytes) - compression overhead not worth it
• ❌ Streaming responses (handled separately if needed)

🔹 Performance Considerations:

• Compression is CPU-intensive but typically worth the trade-off
• Brotli offers best compression but higher CPU cost
• GZip is fastest with good compression
• Pre-compression for static assets is more efficient (done in Epic 4)

---

🔗 Related Issues

• Part of Performance Optimization initiative
• Related to Epic: Performance - HTTP/2 & Keep-Alive Transport
• Related to Epic: Performance - Static Asset Caching & CDN (pre-compressed assets)
• Complements caching strategies (smaller cached values)

---

📚 References

• starlette-compress GitHub
• starlette-compress PyPI
• Brotli Compression RFC 7932
• Zstandard Compression
• MDN: Content-Encoding
• MDN: Accept-Encoding
• Can I Use Brotli