feat(core): add continuous screenshot capturing feature and enhance context handling #1258
Conversation
✅ Deploy Preview for midscene ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Pull Request Overview
This PR adds a continuous screenshot capturing feature that automatically captures screenshots at regular intervals and enhances context handling to support multiple screenshots. The feature enables better visualization and analysis of user interactions over time.
- Adds continuous screenshot configuration with interval and count limits
- Integrates screenshot lists throughout the AI model inspection and agent workflow
- Updates visualization components to display multiple screenshots with overlap handling
Reviewed Changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/types.ts | Adds screenshot list support to UIContext and continuous screenshot configuration to AgentOpt |
| packages/core/src/agent/agent.ts | Implements continuous screenshot capturing logic with timer management and context merging |
| packages/core/src/agent/utils.ts | Enhances context parser to handle screenshot lists with deduplication and size limits |
| packages/core/src/agent/tasks.ts | Updates task executor to extract and record multiple screenshots |
| packages/core/src/ai-model/inspect.ts | Modifies AI inspection functions to include additional screenshots in prompts |
| packages/core/src/yaml.ts | Adds continuous screenshot configuration to YAML script config |
| packages/core/src/yaml/player.ts | Updates script player to pass screenshot list options to AI methods |
| packages/core/tests/utils.ts | Updates test utilities to include screenshot list in fake insights |
| packages/visualizer/src/utils/replay-scripts.ts | Adds function to extract screenshot sequences from context for animation scripts |
| packages/web-integration/src/puppeteer/agent-launcher.ts | Passes continuous screenshot configuration to agent |
| packages/cli/src/create-yaml-player.ts | Forwards continuous screenshot configuration from YAML to agent |
| packages/cli/tests/midscene_scripts/online/video-player.yaml | Example YAML configuration using continuous screenshot feature |
| apps/report/src/components/timeline/index.tsx | Updates timeline visualization to handle multiple screenshots with overlap positioning |
| apps/report/src/components/global-hover-preview/index.tsx | Enhances hover preview to display all available screenshots |
| apps/report/src/components/detail-panel/index.tsx | Updates detail panel to show all screenshots with proper labeling |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| if (screenshotBase64List.length) { | ||
| const uniqueScreenshots = new Set<string>(); | ||
| for (const item of screenshotBase64List) { | ||
| if (!item) continue; | ||
| if (uniqueScreenshots.has(item)) { | ||
| uniqueScreenshots.delete(item); | ||
| } | ||
| uniqueScreenshots.add(item); | ||
| } | ||
| screenshotBase64List = Array.from(uniqueScreenshots); | ||
| } |
There was a problem hiding this comment.
The deduplication logic is unnecessarily complex. The current implementation deletes and re-adds items that already exist in the Set, which is redundant. A Set naturally handles uniqueness, so you can simply add all items without checking if they exist first.
Copilot uses AI. Check for mistakes.
| private mergeScreenshotLists(...lists: (string[] | undefined)[]): string[] { | ||
| const { maxCount } = this.getContinuousScreenshotConfig(); | ||
| const unique = new Set<string>(); | ||
| for (const list of lists) { | ||
| if (!list) continue; | ||
| for (const item of list) { | ||
| if (!item) continue; | ||
| if (unique.has(item)) { | ||
| unique.delete(item); | ||
| } | ||
| unique.add(item); | ||
| } | ||
| } | ||
| if (!unique.size) { | ||
| return []; | ||
| } | ||
| const merged = Array.from(unique); | ||
| if (merged.length > maxCount) { | ||
| return merged.slice(merged.length - maxCount); | ||
| } | ||
| return merged; | ||
| } |
There was a problem hiding this comment.
The deduplication logic is unnecessarily complex. The current implementation deletes and re-adds items that already exist in the Set, which is redundant since Sets naturally handle uniqueness. Simply adding items without the has() check would be more efficient and cleaner.
Copilot uses AI. Check for mistakes.
1 participant
No description provided.