Firefox fails intermittently for test

[jsjoeio]

Describe the bug
Firefox fails intermittently for basic test.

To Reproduce
Steps to reproduce the behavior:

1. Install code-server:
   1. yarn global add code-server or
   2. More installation options
2. Run code-server: code-server
3. git clone https://github.com/jsjoeio/jest-playwright-repro.git
4. git checkout firefox-issue
5. yarn
6. sh run-multiple-test.sh which runs yarn test 10 times

Expected behavior
The test passes every time (10/10)

Screenshots

Expected

Actual

$ jest
FAIL browser: firefox ./browser.test.ts
  ● Test suite failed to run

    browser.close: Protocol error (Browser.removeBrowserContext): Browser closed.
    ==================== Browser output: ====================
    <launching> /Users/jp/Library/Caches/ms-playwright/firefox-1238/firefox/Nightly.app/Contents/MacOS/firefox -no-remote -headless -profile /var/folders/c7/ns6lqzvx3z10mq647y5g9lqc0000gn/T/playwright_firefoxdev_profile-pmj59e -juggler-pipe -silent
    <launched> pid=73607
    [pid=73607][err] *** You are running in headless mode.
    [pid=73607][out]
    [pid=73607][out] Juggler listening to the pipe
    [pid=73607][out] console.warn: SearchSettings: "get: No settings file exists, new profile?" (new Error("", "(unknown module)"))
    [pid=73607][out] Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=0.517666) [GFX1-]: Receive IPC close with reason=AbnormalShutdown
    [pid=73607][err] Exiting due to channel error.
    [pid=73607][err] Exiting due to channel error.

      at Object.captureStackTrace (node_modules/playwright/lib/utils/stackTrace.js:48:19)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        3.713 s
Ran all test suites.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
yarn run v1.22.10

c60a0355-02ed-d34c-9da0-c43c1d65400c.mp4

Desktop (please complete the following information):

• OS: macOS 11.2.2 (20D80)
• Playwright version: 1.10.0
• jest-playwright version: 1.5.1

Jest configuration (Either in the package.json > jest or in the jest.config.js):

module.exports = {
 preset: "jest-playwright-preset",
 transform: {
   "^.+\\.ts$": "ts-jest",
 },
 testEnvironmentOptions: {
   "jest-playwright": {
       browsers: ["firefox"],
       contextOptions: {
         recordVideo: {
           dir: "./videos",
         },
     },
 },
}

Additional context
Running in an isolated index.js works as expected (no errors):

const { firefox } = require("playwright")

// 1. Install code-server `yarn add global code-server` and run locally: `code-server`
// 2. node index.js

;(async () => {
  const browser = await firefox.launch()
  const context = await browser.newContext()
  const page = await context.newPage()

  await page.goto("http://localhost:8080")
  const title = await page.title()

  if (title === "code-server login") {
    console.log("It worked!")
  } else {
    console.log("It failed.")
  }
})()

Sometimes 5/10 pass. Other times 0/10. It's almost always different.

Possibly related?:

• chore: firefox crashes upstream + keyboard shortcuts changed microsoft/playwright#5721
• https://bugzilla.redhat.com/show_bug.cgi?id=1860451

[mxschmitt]

Hey @jsjoeio, thanks for reporting, I was able to track the issue down to a minimum reproducible and opened an issue on Playwright side. We could implement a workaround by having:

await browser.close().catch(() => {})

instead of:

await browser.close()

so it does not throw unhandled promise rejection warnings here until they have a fix ready.

[jsjoeio]

Woo! 🎉 Glad it was reproducible. And super impressed you found it so quickly.

I subscribed to the issue in playwright so feel free to close this one if you want.