[Bug] Unreadable virtual file added by yarn install

[MLoughry]

Describe the bug

I was invetigating using @berry/pnpify@0.0.5 in my repo. However, when I ran yarn, it added a file .yarn/virtual/@berry-pnpify-virtual-46907f4d0e2d3a80455eb5b30214ba5e256b13da685f0c8279698b23320cbfdc that the Windows file system could not open, and which git could not add.

PS E:\VSO\typescript-repro> git add .
error: readlink(".yarn/virtual/@berry-pnpify-virtual-46907f4d0e2d3a80455eb5b30214ba5e256b13da685f0c8279698b23320cbfdc"): Function not implemented
error: unable to index file .yarn/virtual/@berry-pnpify-virtual-46907f4d0e2d3a80455eb5b30214ba5e256b13da685f0c8279698b23320cbfdc
fatal: adding files failed

To Reproduce

See my exploration repo on the pnpify branch. Running yarn from a Windows machine from commit 5ae2746d00dd9942ba066de931ddacca46f66c6e seems to generate this file, and you can see the results from there.

Environment if relevant (please complete the following information):

• OS: Windows 10
• Node version 12.4.0
• Yarn version v2.0.0 (unless there's a more granular version than given by yarn --version)

[MLoughry]

I was about to open a separate bug about yarn pnpify tsc not working in v0.0.5, but it looks like it may be the same underlying issue:

PS E:\VSO\typescript-repro> yarn build      
E:\VSO\typescript-repro\.pnp.js:3418
        throw firstError;
        ^

Error: ENOENT: no such file or directory, stat 'E:\VSO\typescript-repro\.yarn\virtual\@berry-pnpify-virtual-46907f4d0e2d3a80455eb5b30214ba5e256b13da685f0c8279698b23320cbfdc'
    at Object.statSync (fs.js:856:3)
    at NodeFS.statSync (E:\VSO\typescript-repro\.pnp.js:893:28)
    at new ZipFS (E:\VSO\typescript-repro\.pnp.js:1719:46)
    at ZipOpenFS.getZipSync (E:\VSO\typescript-repro\.pnp.js:3216:50)
    at ZipOpenFS.makeCallSync (E:\VSO\typescript-repro\.pnp.js:3153:21)
    at ZipOpenFS.existsSync (E:\VSO\typescript-repro\.pnp.js:2825:21)
    at E:\VSO\typescript-repro\.pnp.js:3668:36
    at Array.find (<anonymous>)
    at applyNodeExtensionResolution (E:\VSO\typescript-repro\.pnp.js:3666:18)
    at resolveUnqualified (E:\VSO\typescript-repro\.pnp.js:3917:31) {
  errno: -4058,
  syscall: 'stat',
  code: 'ENOENT',
  path: 'E:\\VSO\\typescript-repro\\.yarn\\virtual\\@berry-pnpify-virtual-46907f4d0e2d3a80455eb5b30214ba5e256b13da685f0c8279698b23320cbfdc'
}

[arcanis]

I can't check right now, but I kinda remember that git on windows needs a settings to properly install symlinks (otherwise they are raw files), can you check that? If that's the problem we'll want to document it.

[MLoughry]

So, I found this link on the subject: https://github.com/git-for-windows/git/wiki/Symbolic-Links.

When I change core.symlinks in the git config, I can yarn install, but only in an administrator prompt; otherwise I get the following error:

PS D:\vso\typescript-repro> yarn install
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 0.01s
➤ YN0000: ┌ Fetch step
➤ YN0001: │ Error: @berry/pnpify@npm:0.0.5: EPERM: operation not permitted, symlink '..\cache\@berry-pnpify-npm-0.0.5-7a2cdd755d6b5a5da331a93d9034e4e45d8543fd8e8e2d5fd2a527eeffdb5709.zip' -> 'D:\vso\typescript-repro\.yarn\virtual\@berry-pnpify-virtual-46907f4d0e2d3a80455eb5b30214ba5e256b13da685f0c8279698b23320cbfdc'
➤ YN0000: └ Completed in 0.09s
➤ YN0000: Failed with errors in 0.11s

In either case, I still get the same resulting error from running yarn pnpify tsc, whether I have admin privileges or not:

PS D:\vso\typescript-repro> yarn pnpify tsc
D:\vso\typescript-repro\.pnp.js:3418
        throw firstError;
        ^

Error: ENOENT: no such file or directory, stat 'D:\vso\typescript-repro\.yarn\virtual\@berry-pnpify-virtual-46907f4d0e2d3a80455eb5b30214ba5e256b13da685f0c8279698b23320cbfdc'
    at Object.statSync (fs.js:871:3)
    at NodeFS.statSync (D:\vso\typescript-repro\.pnp.js:893:28)
    at new ZipFS (D:\vso\typescript-repro\.pnp.js:1719:46)
    at ZipOpenFS.getZipSync (D:\vso\typescript-repro\.pnp.js:3216:50)
    at ZipOpenFS.makeCallSync (D:\vso\typescript-repro\.pnp.js:3153:21)
    at ZipOpenFS.existsSync (D:\vso\typescript-repro\.pnp.js:2825:21)
    at extensions.map.find.candidateFile (D:\vso\typescript-repro\.pnp.js:3668:36)
    at Array.find (<anonymous>)
    at applyNodeExtensionResolution (D:\vso\typescript-repro\.pnp.js:3666:18)
    at resolveUnqualified (D:\vso\typescript-repro\.pnp.js:3917:31)

[arcanis]

1. I think you don't need the administrator mode if you have enabled the developer mode
2. A Windows-related symlink bug got fixed in Fixes the virtual link type #281; make sure you're on an updated branch (you might need to remove the .yarn/virtual folder and run yarn install again)

[MLoughry]

It looks like you were right about #281. Re-running yarn policies set-version berry to download the latest version seemed to fix the issue, and yarn pnpify tsc now works as expected.

Lastly, as I continue exploring the viability of migrating our project (Outlook Web) over to using this, I'm wondering how stable you consider berry, and/or if there's any timeline for an official v2 release. I haven't been able to find any official documentation on either point.

[arcanis]

Developer mode is described here: https://www.howtogeek.com/292914/what-is-developer-mode-in-windows-10/. That said, we should implement some checks and throw helpful errors when we detect this situation; it's a very common pitfall, and is pretty hard to debug.

---

Regarding stability, I would say I find Berry much more stable than the v1 codebase, even today. At the very least, we understand every bit of code around here and would be able to say what is a bug and at what level, and what the fix should look like. The same can't be said from the v1 codebase, which has various terra incognita where components interact together in impredictable ways, causing subtle issues whose causes aren't always fully understood, leading us to fix the symptoms rather than the real issue.

In term of practical features, Berry aims to provides a better user experience (especially when working with workspaces), zero-installs (which would cut down significantly the CI time and improve their stability), and various smaller improvements (peer dependencies in yarn link, portals, dependenciesMeta, ...). Lots of those cannot be brought to the v1 without large rearchitecturings.

The main blockers I see for a release are:

• Some fan-favorite commands are missing (for example upgrade-interactive).

• The documentation needs to be finished (for example Implement package search bar on docs website #270, or Adds initial documentation for plugins #289)

• Some packages from the ecosystem shipped compatibility fixes in their majors, and I'd like to sync our release so that migration is easier. Even if those fixes are usually good things regardless of whether projects are installed using PnP or not, it wouldn't be a good experience for our average users to find out that their stack is broken in subtle ways:

  • For example ESLint had some problems linked to shared config; they overhauled this system in ESLint 6, so we needed to wait for that. I now need to test it again, and add it to our e2e pipeline.

  • Similarly, Webpack 5 will include builtin PnP support, which would certainly help quite a bit any migration - all packages like Storybook, CRA, etc, wouldn't need to add the PnP plugin themselves. So there's a good argument to wait for Webpack 5 to be released.

  • TypeScript having to be run behind pnpify is a bit annoying. Although we won't block the release on it, it would be awesome to find a compromise to add native PnP support, rather than having to hack our way around it (Add new moduleResolution option: yarn-pnp microsoft/TypeScript#28289).

  • The Angular CLI has a few compat issues that @bgotink is looking into ([Case Study] Angular CLI #266). The Vue CLI had some as well that @lbogdan investigated; we now need to submit the fixes upstream.

Note that the compat problems that are discovered from time to time are usually very simple to fix (a require.resolve somewhere, a new peer dependency somewhere else ...), but it can be very time consuming for a small team like ours to write the fixes ourselves, especially when we don't use the projects to begin with (I literally never used Vue or Angular 2 myself, for example). So any help is appreciated 😃

Overall, I think we'll target an official release by the end of year, although I would recommend our most advanced users to start using the v2 when they have the chance. You'd get the improvements, we'd get a higher confidence regarding our compat with the ecosystem.