fix: automatically handle container exits and reconnect clients #2634
+425
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #2270 When containers exit or are removed, clients (Cursor, Cline) lose connection and the tool stops working. This PR implements automatic handling of container exits with client config updates to force reconnection. ## Changes ### Transport Layer - Updated handleContainerExit() in http.go and stdio.go to log clear warnings - Made TransparentProxy.Stop() idempotent to prevent double-close panics - Added logging to distinguish between container exit vs removal ### Runner Layer - Check if workload exists in runtime (not just status files) to determine if container was removed or just exited - If container removed: clean up client config and exit gracefully - If container exited: signal restart needed to workload manager ### Workload Manager - Automatic restart with exponential backoff (5s, 10s, 20s, 40s, 60s) - Up to 10 retry attempts (~10 minutes) - Remove from client config before restart, re-add after successful restart - This forces clients to see the config change and reconnect with fresh sessions ## Behavior **Container exits (crash)**: - Detect exit → Remove from ~/.cursor/mcp.json → Wait with backoff → Restart container → Re-add to mcp.json → Client reconnects ✅ **Container removed (docker rm / thv delete)**: - Detect removal → Remove from ~/.cursor/mcp.json → Exit gracefully → Tool vanishes from client ✅ ## Testing Tested with Cursor on macOS: - Container kill: Automatic restart and reconnection works - Container remove: Clean removal from client config
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2634 +/- ##
==========================================
- Coverage 55.47% 55.30% -0.18%
==========================================
Files 312 312
Lines 29714 29810 +96
==========================================
+ Hits 16485 16486 +1
- Misses 11789 11882 +93
- Partials 1440 1442 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Adds unit tests for: - HTTPTransport.ShouldRestart() - restart logic for container exits vs removals - StdioTransport.ShouldRestart() - restart logic for container exits vs removals - TransparentProxy.Stop() - idempotent stop functionality - DefaultManager.RunWorkload() - container exit handling These tests cover the new automatic container exit detection and restart functionality, improving code coverage for the PR.
Contributor
Author
|
This may be related to https://discord.com/channels/1184987096302239844/1439759480069034036/1440007583145394297 |
Fixes test failures caused by corrupted module cache in CI environment. The 'go clean -modcache' step ensures a clean state before downloading dependencies, preventing 'dial tcp: lookup proxy.golang.org' errors during test compilation. Related to #2634
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2270
Problem
When containers exit or are removed, clients (Cursor, Cline, VSCode) lose connection and tools stop working. The issue noted: "When a pod dies, we have to change the config (just touching the file doesn't work. we need to actually change the config)"
Solution
This PR implements automatic container exit handling with client config updates to force reconnection.
Key Changes
1. Automatic Container Exit Detection
TransparentProxy.Stop()idempotent to prevent panics2. Smart Workload Existence Check
thv lsas source of truth for whether to restart or clean up3. Automatic Restart with Exponential Backoff
4. Client Config Management (The Core Fix!)
~/.cursor/mcp.json~/.cursor/mcp.json5. Clean Removal Handling
docker rm,thv delete): permanently removes from client configBehavior
Container exits (crash):
Container removed intentionally:
Files Changed
pkg/transport/http.go- Container exit handling for HTTP transportpkg/transport/stdio.go- Container exit handling for stdio transportpkg/transport/proxy/transparent/transparent_proxy.go- Idempotent Stop()pkg/runner/runner.go- Workload existence check and client config cleanuppkg/workloads/manager.go- Automatic restart with backoff + config refreshdocs/fixes/container-exit-handling.md- Documentation (new)Testing
✅ Container kill → Automatic restart → Client reconnects
✅ Container remove → Clean config removal → Tool vanishes
✅ Multiple failures → Exponential backoff works
✅ No panics on double-close
Tested with Cursor on macOS.