Recommend Playwright rather than Cypress

[mercmobily]

About Cypress:

• They have recently locked out third party panels
• They have disrupted existing users

Given the latest controversies about Cypress, I recommend we change the default to Playright since it's a much better tool, it's actually free, and its experiencing massive (deserved) growth.

I am not affiliated to Playright in any way.

If you are for the change, I will be happy to submit a PR and also change the default testing for create-vue (vuejs/create-vue#418)

[Yordan-Ramchev]

I would just attach this screenshot to support @mercmobily

[lmiller1990]

One concern is Playwright's Component Testing solution is still in experimental - not sure we want to push something that's not considered production ready.

It may be worth considering offering both.

[mercmobily]

At the moment Vue already offers both. This ticket is about what's offered by default, with --with-test, which is dependant on what the documentation recommends by default. Right now, Playwright is in the "Other Options" section.

I totally understand why though. I asked the Playwright guys what the plan is in terms of marking component testing ready for production.

[pavelfeldman]

@mercmobily thanks for reaching out in microsoft/playwright#29037.

We (Playwright) were just discussing the component testing and reached a conceptual agreement to make it a part of the official story. It is a big commitment for us, so we don't take it lightly. So far we've been validating our variation of the code splitting where tests run in Node and components run in the browser. At this point we feel like it works well for the users, so we'd like to move forward.

Next steps

Our policy allows for breaking changes when switching from experimental to stable, so if we (you) feel strongly about the exposed APIs or use cases, we have that flexibility. Porting a substantial test suite from another component testing framework to Playwright and iterating on it to make it feel awesome sounds like a next logical step. Any pointers to a reference suite would be appreciated. If we all are happy with the outcome, we can move forward.

Deployment

Playwright Testing is on a release train that updates monthly with the new browser versions. Vite and Vue versioning is orthogonal to it, so we'll need to figure out the deployment model that would be manageable and would not explode our testing matrix beyond reason.

Invitation to co-own

Playwright is a small team with the common code ownership. We don't have anyone dedicated to component testing. I started it, then @sand4rt contributed a lot of nice changes, now I am revisiting it again. All this is to say that we would want to invite Vuejs team members to work on it together with us, share the ownership over the Vue component as well as influence the component testing altogether. We can't do a great job without you - we lack domain expertise in the component domain. We want the right hooks to be used by the testing story, etc.

All this is unrelated to making Playwright the default, more about aligning with you on the plans and inviting to collaborate.

[mercmobily]

@lmiller1990 I think you are the best person to answer this. However, I just wanted to put my hands up as a person willing to help with this. By "help" I mean that I would be happy to be somebody who is comfortable with vue-test-utils 2 source code (sorry, only version 2, not 1) and will be happy to help with issues relating vue 3 <-> Playwright integration.
I will just need a little bit of man holding to start with -- but I am happy to start right away.

To keep things practical:

Porting a substantial test suite from another component testing framework to Playwright and iterating on it to make it feel awesome sounds like a next logical step

@lmiller1990 How can I help you here? Do you have something in mind that will fit this question nicely?

About this point:

Playwright Testing is on a release train that updates monthly with the new browser versions. Vite and Vue versioning is orthogonal to it, so we'll need to figure out the deployment model that would be manageable and would not explode our testing matrix beyond reason.

@pavelfeldman Question: How do you deal with this and React?

Invitation to co-own

Again this is a direct question for @lmiller1990. However, as I mentioned above, I would be happy to be the one helping out here in terms of integration of Playwright and Vue 3.

[Akryum]

tests run in Node and components run in the browser.

I'm worried that this isn't gonna work in practice. There is a lot more to it around a component that needs to be setup for it to run: router, stores, plugins, API mocks... which can all be different in each test. The currently proposed solutions in the docs feel it's too much work compared to how it's done in vitest or Cypress. IMO in most cases the downsides of this architecture are too many compared to the upsides.

---

HMR/Hot-reloading is also a must when writing the tests or doing TDD and afaik it is not supported in Playwright CT but is supported in vitest and Cypress.

[sand4rt]

There is a lot more to it around a component that needs to be setup for it to run: router, stores, plugins, API mocks

There is support for this ATM, or do you see problems with it?:

// playwright/index.ts
import { beforeMount, afterMount } from '@playwright/experimental-ct-vue/hooks';
import { router } from '../src/router';
import { createTestingPinia } from '@pinia/testing';
import type { StoreState } from 'pinia';
import type { useStore } from '../src/store';

export type HooksConfig = {
  store?: StoreState<ReturnType<typeof useStore>>;
  enableRouting?: boolean;
}

beforeMount<HooksConfig>(async ({ app, hooksConfig }) => {
  if (hooksConfig?.enableRouting)
    app.use(router); // <--- enables the router
    
  if (hooksConfig?.store)
    createTestingPinia({ initialState: hooksConfig?.store }); // <--- enables the store
    
  // plugins work the same (using app.use)
});

import { type HooksConfig } from '../playwright';

test('override store on a per test basis', async ({ mount }) => {
  await mount<HooksConfig>(Component, {
    hooksConfig: {
      store: { boop: 'boop' }
    }
  });
});

test('component with router', async ({ mount }) => {
  await mount<HooksConfig>(Component, { 
    hooksConfig: { enableRouting: true }
  });
});

The currently proposed solutions in the docs feel it's too much work compared to how it's done in vitest or Cypress

Yeah, agree, this is also my main concern, but I think there might be ways to improve this.

HMR/Hot-reloading is also a must when writing the tests or doing TDD and afaik it is not supported in Playwright CT but is supported in vitest and Cypress.

Also agree here, there are outstanding issues to improve on this tho: microsoft/playwright#21960 and microsoft/playwright#14748

[mercmobily]

@Akryum Possibly dumb question, but... in terms of router, stores, plugins, API mocks, etc., why doesn't the problem show up when using Cypress?

Also, I think it would be beneficial to have a clear list of issues that are still outstanding and that would effectively make Cypress/Vitest a better choice to test Vue components/apps. This ticket would depend on them getting closed.

I am happy to write a list of them here, and finalise it, so that we know where Playwright testing stands compared to more established paths.

[Akryum]

@mercmobily No, we don't have this problem with Cypress since tests are run in the browser.

[pavelfeldman]

I'm worried that this isn't gonna work in practice. There is a lot more to it around a component that needs to be setup for it to run: router, stores, plugins, API mocks... which can all be different in each test.

All these are indeed supported. We can improve test ergonomics though and that's exactly where we would need your feedback.

HMR/Hot-reloading is also a must when writing the tests or doing TDD and afaik it is not supported in Playwright CT but is supported in vitest and Cypress.

This is good feedback as it is something we can support in the interactive (UI) mode.

IMO in most cases the downsides of this architecture are too many compared to the upsides.

The other side of the scale is:

• parallelism
• tracing
• accurate click: (if component is swapped on mouse hover / down programmatic click (cypress) won't work
• event order: programmatic click (cypress) gives wrong event order task/microtask-wise

---

Overall, thank you for your feedback!

If the attitude towards the initiative is lukewarm, we can slowly work on improving the support and steer it in the right direction, while keeping it experimental for Vue!

[pavelfeldman]

HMR/Hot-reloading is also a must when writing the tests or doing TDD and afaik it is not supported in Playwright CT but is supported in vitest and Cypress.

FYI, I'm collecting the feedback on the shape of the fix for this here: microsoft/playwright#14748 (comment)

[lmiller1990]

FYI... I work on Vue and Cypress for fun, neither is a paid job - opinions here are my own, with no prejudice. I also think Playwright is a great tool in many aspects, performance and parallelism is very impressive.

I agree with @Akryum that hot reload and the such are extremely useful for any component development. The fact Cypress is browser based and uses a dev server works out very well for component development. This is also what makes Storybook so popular, you can just start it up and tinker with your components. microsoft/playwright#14748 seems like the issue to follow regarding this.

As discussed, Playwright CT is still experimental - great to hear from @pavelfeldman that the team is looking to make it part of the core offering, always like to see innovation in the component development ecosystem. I don't think we (Vue docs) should be defaulting to a tool that is still in the experimental stage.

Right now the default is plain old Test Utils, mainly since it's part of the core Vue ecosystem, so we can guarantee it works and is supported with all versions of Vue. Why not just keep this as the default, and include both PW and Cypress as options for "visual testing" or "browser based testing". We could give unbiased recommendations, like "Cypress is browser based, which is great for XX, but Playwright's architecture supports YY".

---

Re: @mercmobily - both Cypress and Playwright wrap the Vue Test Utils library, I don't think there is much to be done between Playwright <> Test Utils, is there? What specific issues do you see that need addressing around the integration? I know the code bases very well, so I can definitely point you to where things are and how they work if needed.

[pavelfeldman]

I agree with @Akryum that hot reload and the such are extremely useful for any component development.

This is now working on ToT Playwright (microsoft/playwright#14748 (comment)).

[lmiller1990]

pavelfeldman

Great work, can't wait to give this a try!

Btw, what does ToT stand for?

I think both products are great, I'd like to see them both featured in our docs. I think something simple describing each tool with a link to the respective tool onboarding docs would be appropriate.