[CU-86b7p1xgv] Add --local-federated flag to bypass Explorer federation bottleneck #130
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…on bottleneck When concurrent requests to federated questions overwhelm the Explorer service, circuit breakers and connection timeouts cause failures. This happens because Explorer's federation layer becomes a bottleneck when coordinating responses from multiple publishers with queuing systems. This change adds a --local-federated flag that queries each collection directly in parallel, bypassing the Explorer federation layer. This allows workloads to proceed while the long-term solution (converting Explorer's federated API to use Data Connect-style streaming responses) is implemented. The flag maintains full compatibility with existing behavior and output format, making it a safe workaround for high-concurrency scenarios. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
… correctly The initial implementation only returned the first page of results from each collection, which would miss data when results span multiple pages. This is critical because Data Connect APIs return empty pages with next_page_url links until data is ready, then continue pagination until all results are delivered. This fix implements proper pagination handling by: - Making the initial POST request to start the query - Following next_page_url links with GET requests until exhausted - Aggregating all data across all pages before returning - Preventing infinite loops by tracking visited URLs Without this fix, users would get incomplete results when querying collections that return paginated responses, making the --local-federated flag unreliable for production use. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…eature - Add 29 tests across LocalFederatedQuestionQueryResultLoader, ExplorerClient, and CLI - Test Data Connect pagination handling, parallel collection queries, error scenarios - Test CLI --local-federated flag integration and parameter parsing - Fix ClientError constructor to use trace_context parameter - Ensure compatibility with existing utilities and result formats 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
--local-federatedflag toexplorer questions askcommand to bypass federation bottleneckProblem
When concurrent requests to federated questions overwhelm the Explorer service, circuit breakers and connection timeouts cause failures. This happens because Explorer's federation layer becomes a bottleneck when coordinating responses from multiple publishers with queuing systems.
Solution
The
--local-federatedflag enables direct parallel queries to each collection, bypassing the Explorer federation layer. This allows workloads to proceed while the long-term solution (converting Explorer's federated API to use Data Connect-style streaming responses) is implemented.Implementation Details
/api/collections/{slug}/questions/{questionId}/queryTesting
Example Usage
dnastack explorer questions ask \ --question-name allele-frequency \ --param position=64314563 \ --param chromosome=chr15 \ --collections "yFAS-2e200380-a0da-464e-8554-663fa485965c,Lu0K-cd1cdf5a-1cb0-4b47-bf52-d365f928a1b4" \ --local-federatedNotes
🤖 Generated with Claude Code