Performance testing #1204
Merged
Performance testing #1204
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Mihai Criveti <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>
- Consolidate 5 documentation files into 2:
- README.md: practical quick start guide (309 lines)
- PERFORMANCE_STRATEGY.md: comprehensive deep dive (2,116 lines)
- Delete: QUICK_REFERENCE.md, SERVER_PROFILES_GUIDE.md, README_AUTOMATION.md,
IMPLEMENTATION_STATUS.md, FINAL_SUMMARY.md
- Result: 35% reduction in documentation (3,742 → 2,425 lines)
- Fix regex escape issue in report_generator.py:
- Use lambda function in re.sub() to avoid backslash interpretation
- Fixes: re.error with JSON Unicode sequences (\uXXXX)
- Location: utils/report_generator.py:767
Signed-off-by: Mihai Criveti <[email protected]>
Change INFO level log in token_scoping middleware to DEBUG to avoid performance degradation during load testing. Logging every request at INFO level was causing: - Massive disk I/O bottleneck - CPU overhead for log formatting - Lock contention on log files With 50 concurrent requests, this reduced RPS from expected 500+ to just 6.67. This log statement triggers for every request to endpoints without resource IDs (like /rpc, /health, etc.), making it unsuitable for INFO level. Signed-off-by: Mihai Criveti <[email protected]>
1. Change default LOG_LEVEL from INFO to ERROR - Reduces unnecessary log I/O overhead - config.py:629: Changed default from INFO to ERROR 2. Add DISABLE_ACCESS_LOG environment variable support - run-gunicorn.sh: Added conditional access logging - When true, routes logs to /dev/null (massive performance gain) - docker-compose.yml: Added DISABLE_ACCESS_LOG=true 3. Performance impact analysis: - uvicorn.access logs EVERY request at INFO level - With 50 concurrent requests: ~6.67 RPS (bottlenecked by log I/O) - Expected after fix: 500+ RPS 4. Related middleware logging fix (previous commit) - token_scoping.py: Changed INFO to DEBUG for high-frequency logs The combination of excessive access logging and INFO-level middleware logging was causing ~75x performance degradation during load testing. Signed-off-by: Mihai Criveti <[email protected]>
Critical bug fix: LoggingService.__init__() hardcoded self._level = LogLevel.INFO, which was never updated from settings.log_level. This caused uvicorn.access logger to always log at INFO level regardless of LOG_LEVEL environment variable. This caused massive performance degradation (~75x slower) during high concurrency because every HTTP request was logged at INFO level, creating severe I/O bottleneck. Fix: Read self._level from settings.log_level at start of initialize() method, BEFORE configuring uvicorn loggers. Now uvicorn.access correctly respects LOG_LEVEL. Expected performance improvement: 7 RPS → 500+ RPS with LOG_LEVEL=ERROR. Signed-off-by: Mihai Criveti <[email protected]>
- Change LOG_LEVEL default from INFO to ERROR - Add DISABLE_ACCESS_LOG configuration option - Document performance impact of access logging - Add production recommendations for logging settings These changes reflect the 251x performance improvement achieved by optimizing logging configuration (7 RPS → 1810 RPS). Signed-off-by: Mihai Criveti <[email protected]>
Problem: ^C was not working to interrupt performance tests, forcing users to wait for timeout or manually kill processes. Root cause: 1. Scripts had trap handlers but cleanup functions exited with code 0 2. No signal handling in run-advanced.sh 3. Background processes weren't being killed Changes: - run-configurable.sh: Exit with 130 (SIGINT code), kill background jobs - run-advanced.sh: Add signal trap and cleanup handler - Both: Properly propagate SIGINT to all child processes Now Ctrl+C immediately stops tests and cleans up partial results. Signed-off-by: Mihai Criveti <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>
The JWT payload's 'teams' field contains full team objects (dicts) with
structure {id, name, slug, is_personal, role}, but the token scoping
middleware was treating them as simple string IDs.
This caused SQLAlchemy/psycopg2 errors when trying to use dict objects
in SQL queries:
ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'dict'
Changes:
- _check_team_membership(): Extract team ID from dict or use string
directly for backward compatibility
- _check_resource_team_ownership(): Normalize token_teams at start,
extracting team IDs from dict objects before using in comparisons
This maintains backward compatibility with older tokens that may have
simple string team IDs while supporting the current dict format.
Fixes internal server error when calling /tools and other team-scoped
endpoints with modern JWT tokens.
Signed-off-by: Mihai Criveti <[email protected]>
Add detailed manual testing documentation for MCP Gateway API with: - Prerequisites and setup instructions (curl, jq, hey) - Quick start test script covering all major endpoints - Individual endpoint tests (health, auth, tools, servers, resources, prompts, profile) - Performance testing with hey load testing tool - Benchmarking scripts for measuring throughput - Expected performance baselines (1500-2000 RPS for authenticated endpoints) - Troubleshooting guide (token expiration, validation) - Advanced scenarios (continuous monitoring, deployment verification, load testing) - Integration with existing automated test suite All commands are CLI-ready and tested. Complements automated tests in tests/performance/ with human-readable examples for development and debugging workflows. Location: tests/performance/MANUAL_TESTING.md (458 lines, 12KB) Signed-off-by: Mihai Criveti <[email protected]>
Fix remaining pylint warnings: - token_scoping.py: Disable too-many-return-statements warning for complex authorization logic - support_bundle_service.py: Convert README string to f-string format (C0209) - support_bundle_service.py: Disable import-outside-toplevel for optional psutil import - admin.py: Disable import-outside-toplevel for support bundle import All fixes maintain existing functionality while improving code style compliance. Pylint rating: 10.00/10 Signed-off-by: Mihai Criveti <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>
madhav165
approved these changes
Oct 10, 2025
p4yl04d3r
pushed a commit
to p4yl04d3r/mcp-context-forge
that referenced
this pull request
Nov 19, 2025
* Performance testing: Signed-off-by: Mihai Criveti <[email protected]> * Performance testing Signed-off-by: Mihai Criveti <[email protected]> * Performance testing Signed-off-by: Mihai Criveti <[email protected]> * refactor: consolidate performance testing docs and fix report generator - Consolidate 5 documentation files into 2: - README.md: practical quick start guide (309 lines) - PERFORMANCE_STRATEGY.md: comprehensive deep dive (2,116 lines) - Delete: QUICK_REFERENCE.md, SERVER_PROFILES_GUIDE.md, README_AUTOMATION.md, IMPLEMENTATION_STATUS.md, FINAL_SUMMARY.md - Result: 35% reduction in documentation (3,742 → 2,425 lines) - Fix regex escape issue in report_generator.py: - Use lambda function in re.sub() to avoid backslash interpretation - Fixes: re.error with JSON Unicode sequences (\uXXXX) - Location: utils/report_generator.py:767 Signed-off-by: Mihai Criveti <[email protected]> * perf: reduce token scoping middleware logging to DEBUG level Change INFO level log in token_scoping middleware to DEBUG to avoid performance degradation during load testing. Logging every request at INFO level was causing: - Massive disk I/O bottleneck - CPU overhead for log formatting - Lock contention on log files With 50 concurrent requests, this reduced RPS from expected 500+ to just 6.67. This log statement triggers for every request to endpoints without resource IDs (like /rpc, /health, etc.), making it unsuitable for INFO level. Signed-off-by: Mihai Criveti <[email protected]> * perf: optimize logging for production performance 1. Change default LOG_LEVEL from INFO to ERROR - Reduces unnecessary log I/O overhead - config.py:629: Changed default from INFO to ERROR 2. Add DISABLE_ACCESS_LOG environment variable support - run-gunicorn.sh: Added conditional access logging - When true, routes logs to /dev/null (massive performance gain) - docker-compose.yml: Added DISABLE_ACCESS_LOG=true 3. Performance impact analysis: - uvicorn.access logs EVERY request at INFO level - With 50 concurrent requests: ~6.67 RPS (bottlenecked by log I/O) - Expected after fix: 500+ RPS 4. Related middleware logging fix (previous commit) - token_scoping.py: Changed INFO to DEBUG for high-frequency logs The combination of excessive access logging and INFO-level middleware logging was causing ~75x performance degradation during load testing. Signed-off-by: Mihai Criveti <[email protected]> * fix: initialize LoggingService._level from settings Critical bug fix: LoggingService.__init__() hardcoded self._level = LogLevel.INFO, which was never updated from settings.log_level. This caused uvicorn.access logger to always log at INFO level regardless of LOG_LEVEL environment variable. This caused massive performance degradation (~75x slower) during high concurrency because every HTTP request was logged at INFO level, creating severe I/O bottleneck. Fix: Read self._level from settings.log_level at start of initialize() method, BEFORE configuring uvicorn loggers. Now uvicorn.access correctly respects LOG_LEVEL. Expected performance improvement: 7 RPS → 500+ RPS with LOG_LEVEL=ERROR. Signed-off-by: Mihai Criveti <[email protected]> * docs: update .env.example with performance-optimized logging defaults - Change LOG_LEVEL default from INFO to ERROR - Add DISABLE_ACCESS_LOG configuration option - Document performance impact of access logging - Add production recommendations for logging settings These changes reflect the 251x performance improvement achieved by optimizing logging configuration (7 RPS → 1810 RPS). Signed-off-by: Mihai Criveti <[email protected]> * fix: enable Ctrl+C (SIGINT) handling in performance test scripts Problem: ^C was not working to interrupt performance tests, forcing users to wait for timeout or manually kill processes. Root cause: 1. Scripts had trap handlers but cleanup functions exited with code 0 2. No signal handling in run-advanced.sh 3. Background processes weren't being killed Changes: - run-configurable.sh: Exit with 130 (SIGINT code), kill background jobs - run-advanced.sh: Add signal trap and cleanup handler - Both: Properly propagate SIGINT to all child processes Now Ctrl+C immediately stops tests and cleans up partial results. Signed-off-by: Mihai Criveti <[email protected]> * Disable some plugins Signed-off-by: Mihai Criveti <[email protected]> * fix: handle team dict objects in JWT token validation The JWT payload's 'teams' field contains full team objects (dicts) with structure {id, name, slug, is_personal, role}, but the token scoping middleware was treating them as simple string IDs. This caused SQLAlchemy/psycopg2 errors when trying to use dict objects in SQL queries: ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'dict' Changes: - _check_team_membership(): Extract team ID from dict or use string directly for backward compatibility - _check_resource_team_ownership(): Normalize token_teams at start, extracting team IDs from dict objects before using in comparisons This maintains backward compatibility with older tokens that may have simple string team IDs while supporting the current dict format. Fixes internal server error when calling /tools and other team-scoped endpoints with modern JWT tokens. Signed-off-by: Mihai Criveti <[email protected]> * docs: add comprehensive manual API testing guide Add detailed manual testing documentation for MCP Gateway API with: - Prerequisites and setup instructions (curl, jq, hey) - Quick start test script covering all major endpoints - Individual endpoint tests (health, auth, tools, servers, resources, prompts, profile) - Performance testing with hey load testing tool - Benchmarking scripts for measuring throughput - Expected performance baselines (1500-2000 RPS for authenticated endpoints) - Troubleshooting guide (token expiration, validation) - Advanced scenarios (continuous monitoring, deployment verification, load testing) - Integration with existing automated test suite All commands are CLI-ready and tested. Complements automated tests in tests/performance/ with human-readable examples for development and debugging workflows. Location: tests/performance/MANUAL_TESTING.md (458 lines, 12KB) Signed-off-by: Mihai Criveti <[email protected]> * style: fix pylint issues for 10.00/10 rating Fix remaining pylint warnings: - token_scoping.py: Disable too-many-return-statements warning for complex authorization logic - support_bundle_service.py: Convert README string to f-string format (C0209) - support_bundle_service.py: Disable import-outside-toplevel for optional psutil import - admin.py: Disable import-outside-toplevel for support bundle import All fixes maintain existing functionality while improving code style compliance. Pylint rating: 10.00/10 Signed-off-by: Mihai Criveti <[email protected]> * lint Signed-off-by: Mihai Criveti <[email protected]> * Add benchmark for prompts Signed-off-by: Mihai Criveti <[email protected]> * Add benchmark for prompts Signed-off-by: Mihai Criveti <[email protected]> --------- Signed-off-by: Mihai Criveti <[email protected]> Signed-off-by: p4yl04d3r <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1203