Feat/hierarchical workflow tracing

[serglom21]

Summary

This PR transforms the GitHub Actions workflow tracing from individual job transactions to a single workflow transaction containing nested job and step spans, providing better visibility into workflow timing and structure.

Problem that I am attempting to solve

Before: Individual job transactions created a flat structure in Sentry, making it difficult to understand workflow timing and relationships.
After: Single workflow transaction with proper hierarchical spans showing:

• Workflow-level timing and status
• Individual job spans as children
• Step spans as children of their respective jobs

Key Changes

Core Implementation:

• WorkflowTracer: New class for creating single workflow transactions with nested spans
• WorkflowJobCollector: Collects jobs from workflow runs and sends workflow-level traces
• Enhanced WebAppHandler: Integrates job collection and workflow tracing

Trace Structure:

workflow: Multi-Job Test (transaction)
├─ security-scan (job span)
│  ├─ Set up job (step span)
│  ├─ Checkout code (step span)
│  └─ Run security scan (step span)
├─ performance-tests (job span)
│  ├─ Set up job (step span)
│  ├─ Setup performance test environment (step span)
│  └─ Run performance tests (step span)
└─ backend-tests (job span)
   ├─ Set up job (step span)
   ├─ Setup Python (step span)
   └─ Run backend unit tests (step span)

Files Changed

• src/workflow_tracer.py - New WorkflowTracer implementation
• src/web_app_handler.py - Enhanced with WorkflowJobCollector
• src/enhanced_web_app_handler.py - Alternative handler implementation

Before

After

[seer-by-sentry]

Potential bug: A hard-coded job count check if len(self.workflow_jobs[run_id]) >= 5 prevents workflows with fewer than 5 jobs from ever being processed, leading to data loss.

• Description: The workflow processing logic is only triggered for workflows with 5 or more jobs due to a hard-coded check: if len(self.workflow_jobs[run_id]) >= 5. Workflows with fewer jobs will never meet this condition, and their tracing data will be permanently lost. The analysis notes that the repository's own CI workflow has only 4 jobs and would be ignored by this logic. A comment, For testing, we'll wait for 5 jobs, suggests this was intended for testing but affects production functionality. There is no fallback mechanism to process these smaller workflows.

• Suggested fix: Remove the hard-coded job count check. Instead, implement a more robust mechanism to determine workflow completion, such as using the _is_workflow_complete method which exists but is currently unused, or by waiting for a signal that all jobs for a given run have been received.
  _{severity: 0.8, confidence: 0.95}

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[seer-by-sentry]

Potential bug: An unhandled exception in the _process_workflow_immediately timer callback will cause the thread to fail silently, preventing cleanup and leading to a resource leak.

• Description: The _process_workflow_immediately method, executed by a threading.Timer, lacks exception handling. If an exception occurs within this method or in methods it calls like _send_workflow_trace (e.g., due to network issues or malformed data, as seen in Sentry issue 6767327085), the timer thread will terminate silently. This prevents the execution of critical cleanup logic in the finally block, such as removing the workflow's data from self.workflow_jobs and self.workflow_timers, and adding the run_id to self.processed_workflows. This results in a permanent resource leak and leaves the workflow in an inconsistent state.

• Suggested fix: Wrap the contents of the _process_workflow_immediately method in a try...except block. In the except block, log the exception to ensure failures are not silent. This will allow the program to handle errors gracefully without causing resource leaks or state corruption.
  _{severity: 0.7, confidence: 0.9}

_{Did we get this right? 👍 / 👎 to inform future reviews.}