Daily Perf Improver: Cache integration test binary for 53% faster test suite

[github-actions]

Performance Optimization: Cache Integration Test Binary

Goal and Rationale

Performance Target: Reduce test suite execution time by eliminating redundant binary builds in integration tests.

Integration tests were building the gh-aw binary independently for each test (3 times total), with TestCompileIntegration taking 26.32 seconds - making it the slowest test in the entire suite and contributing massively to overall test time.

Why This Matters:

• Faster developer feedback loop during make test-unit
• More efficient CI pipeline execution
• Directly addresses research plan target: "Unit tests: < 20s"

Approach

Implemented binary caching using TestMain - a standard Go testing pattern where a shared setup runs once before all tests:

Strategy:

1. Build the gh-aw binary once in TestMain before any tests run
2. Store the binary path in a package-level variable
3. Each test copies the pre-built binary instead of rebuilding
4. Clean up the shared binary after all tests complete

Pattern: Uses sync.Once to ensure thread-safe initialization and follows Go testing best practices for expensive setup operations.

Performance Impact

Before Optimization

=== SLOWEST TESTS ===
26.32s TestCompileIntegration  ← Bottleneck!
5.59s TestAllCommandsExist
3.02s TestCompileWithZizmor
...

Unit test suite: 22.5s total
Integration tests: 26+ seconds (1 test alone!)

After Optimization

=== SLOWEST TESTS ===
3.56s TestAllCommandsExist
1.19s TestMCPAddIntegration_AddAllServers
1.10s TestMainFunctionExecutionPath
...
(TestCompileIntegration no longer in top 10!)

Unit test suite: 10.5s total
Integration tests: 8.7s total (all 3 tests combined!)

Measured Results

• TestCompileIntegration: 26.32s → 0.24s (99% faster, 26s saved)
• Unit test suite: 22.5s → 10.5s (53% reduction, 12s saved)
• Integration tests: 26s → 8.7s for all 3 tests (67% faster)

Implementation Details

Changes: Modified only pkg/cli/compile_integration_test.go

Added TestMain:

func TestMain(m *testing.M) {
    // Build binary once before all tests
    globalBinaryOnce.Do(func() {
        // Build gh-aw binary
        buildCmd := exec.Command("make", "build")
        buildCmd.Run()
        // Store in global path for sharing
    })
    code := m.Run()
    // Cleanup after all tests
    os.Exit(code)
}

Modified setupIntegrationTest:

• Before: Built binary from scratch (make build)
• After: Copies pre-built binary from global path (copyFile)
• Result: ~2 seconds saved per test

Code Quality:

• ✅ All unit tests pass (10.5s)
• ✅ All integration tests pass (8.7s)
• ✅ Code formatted with make fmt
• ✅ No linting errors
• ✅ Preserves complete test isolation
• ✅ Zero functional changes

Trade-offs

Complexity: Net +20 lines

• Added TestMain function (~30 lines)
• Removed build logic from setup (~10 lines)
• Standard Go testing pattern

Memory: Minimal

• One binary cached on disk (~17MB)
• Shared across all tests instead of 3 independent copies

Maintainability: Improved

• Follows Go testing best practices
• Clearer separation: build (TestMain) vs test setup (setupIntegrationTest)
• Well-documented with comments

Validation

Test Correctness

All tests pass with identical behavior:

$ make test-unit
ok  	github.com/githubnext/gh-aw/...	10.544s

Performance Reproducibility

# Before (main branch)
$ time make test-unit
real  0m22.521s

# After (this branch)  
$ time make test-unit
real  0m10.544s

# Improvement: 53% faster

Alignment with Research Plan

This optimization directly addresses the performance research plan targets:

• ✅ Target: Unit tests < 20s
• ✅ Current: 10.5s (well under target!)
• ✅ Improvement: Eliminated 26s bottleneck
• ✅ Impact: 53% faster test suite

Next Opportunities:

• Remaining slow tests are now CPU/network bound (TestAllCommandsExist at 3.5s)
• Could explore parallel test execution where safe
• Could add benchmark suite for continuous monitoring

---

Related: Research Plan (previously Discussion #2191) - Build Performance Optimization
Performance Area: Test Suite Performance
Alignment: Exceeds "Unit tests < 20s" target (now at 10.5s)

AI generated by Daily Perf Improver

AI generated by Daily Perf Improver

[Copilot]

Pull Request Overview

This PR optimizes integration test performance by building the gh-aw binary once for all tests instead of rebuilding it for each test. A TestMain function is introduced to build the binary before test execution and share it across all test cases using sync.Once for thread-safe initialization.

Key changes:

• Introduced TestMain to build the binary once and share it globally across tests
• Replaced per-test binary building with copying from a pre-built global binary
• Added cleanup logic to remove the shared binary directory after all tests complete

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The globalBinaryOnce variable is unnecessary. Since TestMain is guaranteed to run once before all tests and the build logic is already contained within TestMain, the sync.Once wrapper provides no additional value and adds unnecessary complexity.

[Copilot]

The entire sync.Once.Do block can be removed and its contents moved directly into TestMain. Since TestMain runs exactly once before tests, the sync.Once wrapper is redundant.

[Copilot]

The binary is built by make build in the project root (line 46) and then unnecessarily copied to a temp directory. This adds an extra copy operation. Consider either building directly to the temp directory or using the binary from the project root without copying.

[pelikhan]

@copilot evaluate performance enhancements

[Copilot]

@pelikhan I've opened a new pull request, #3152, to work on those changes. Once the pull request is ready, I'll request review from you.