Support per volume case sensitivity

[gluxon]

🔎 Search Terms

"tsserver", "cache invalidation", "new file", "case sensitive", "IntelliSense", "inferred project", "mac os", "watch", "tsserver restart"

🕗 Version & Regression Information

This bug is reproducible between TypeScript 5.1.6 and 5.3.0-nightly (commit b555727).

I can test versions prior to 5.1.6 if necessary.

⏯ Playground Link

This bug is specific to a local editing environment and file system quirks. A playground would not be able reproduce this.

🪜 Steps to Reproduce

A few developers on my team have noticed that IntelliSense operations intermittently do not work in newly created files in VS Code. IntelliSense works in the new file after restarting TSServer.

Breakpoint debugging TSServer, I believe there's a file system caching bug related to case sensitivity. The bug can be reproduced consistently on macOS.

1. Open a directory in VS Code with a tsconfig.json file on a case-sensitive volume.
2. Ensure the TypeScript version is set to "Use VS Code's Version". The problem only reproduces when tsserver.js is on a case-insensitive volume, which is usually true for /Applications/Visual Studio Code.app/Contents/Resources/app/extensions/node_modules/typescript/lib/tsserver.js.
3. Create a new file with at least one capital letter in its name. (e.g. Foo.tsx).
4. Observe that IntelliSense operations performed within the new Foo.tsx file do not have information on other files of the same tsconfig.json program.

🙁 Actual behavior

I've created a small repo to help reproduce the bug. There isn't anything special about the repo. It's more where the repo is cloned to and the file system case sensitivity that matters. https://github.com/gluxon/tsserver-new-file-intellisense-bug

Here's a video of the bug.

1. The newly created bar.ts file is able to see IntelliSense information from the rest of the project.
2. The newly created Baz.ts file is not.

bug.mov

Looking over the TSServer logs for the video above, we can see that bar.ts is matched with the right project. The Baz.ts file is instead mapped to /dev/null/inferredProject2*.

Info 743  [15:52:29.973] Open files: 
Info 743  [15:52:29.973] 	FileName: /Volumes/Code/tsserver-new-file-intellisense-bug/foo.ts ProjectRootPath: /Volumes/Code/tsserver-new-file-intellisense-bug
Info 743  [15:52:29.973] 		Projects: /Volumes/Code/tsserver-new-file-intellisense-bug/tsconfig.json
Info 743  [15:52:29.973] 	FileName: /Volumes/Code/tsserver-new-file-intellisense-bug/bar.ts ProjectRootPath: /Volumes/Code/tsserver-new-file-intellisense-bug
Info 743  [15:52:29.973] 		Projects: /Volumes/Code/tsserver-new-file-intellisense-bug/tsconfig.json
Info 743  [15:52:29.973] 	FileName: /Volumes/Code/tsserver-new-file-intellisense-bug/Baz.ts ProjectRootPath: /Volumes/Code/tsserver-new-file-intellisense-bug
Info 743  [15:52:29.973] 		Projects: /dev/null/inferredProject2*

🙂 Expected behavior

TSServer always associates new files with the tsconfig.json file that includes it instead of an inferred project.

---

🔬 Self-Debugging

We noticed that this problem does not reproduce when tsserver.js is loaded from the same case-sensitive file system. It only happens when tsserver.js is loaded from a case-insensitive file system. The main way of changing where tsserver.js was loaded from was through typescript.tsdk in .vscode/settings.json.

With that toggle, we were able to reliably control whether or not the problem reproduces and compare TSServer's internal state for the successful and failing cases. Skip to the bottom for the conclusion.

Program Loading and Caching

Starting at a high-level, we've confirmed in a debugger a few important pieces of information:

1. The TypeScript Program object's rootFilesMap records what files are part of a tsconfig.json. The new file is not added to the expected program's rootFilesMap in the failing case, but is added in the successful case. It makes sense why TSServer associates the newly created file to an inferred project given this.
2. The rootFiles and rootFilesMap fields are computed through getFileNamesFromConfigSpecs(...). This function does not return the newly created file in the failing case (but does in the successful case).
3. The getFileNamesFromConfigSpecs function calls host.readDirectory(...). This readDirectory call does not return the newly created file in the failing case, but does in the successful case.

The host object in getFileNamesFromConfigSpecs is an instance of the CachedDirectoryStructureHost

Debugging CachedDirectoryStructureHost.

The CachedDirectoryStructureHost object is kept up to date through addOrDeleteFileOrDirectory in src/compiler/watchUtilities.ts. We believe the fundamental difference between the failing and successful cases is the evaluation of this line

TypeScript/src/compiler/watchUtilities.ts

Line 315 in 913f65c

fileExists: host.fileExists(fileOrDirectoryPath),

Failing Case

• The fileOrDirectoryPath const evaluates to /volumes/code/tsserver-new-file-intellisense-bug/baz.ts
• fsQueryResult.fileExists subsequently becomes false.

Note that the entire fileOrDirectoryPath string has been been shifted to be lowercase.

Successful Case

• The fileOrDirectoryPath const evaluates to /Volumes/Code/tsserver-new-file-intellisense-bug/Baz.ts
• fsQueryResult.fileExists subsequently becomes true.

In this scenario, the casing of the new file is preserved.

Why is fsQueryResult.fileExists important?

The value of fsQueryResult.fileExists is passed to updateFilesOfFileSystemEntry.

TypeScript/src/compiler/watchUtilities.ts

Line 324 in a0e0104

updateFilesOfFileSystemEntry(parentResult, baseName, fsQueryResult.fileExists);

Once updateFilesOfFileSystemEntry runs, this changes whether or not the CachedDirectoryStructureHost sees the newly created file. I'm able to verify this with a breakpoint on this line, and evaluating the results of readDirectory in the debug console.

TypeScript/src/compiler/watchUtilities.ts

Line 326 in a0e0104

return fsQueryResult;

In the successful case

❯ this.readDirectory("/Volumes/Code/tsserver-new-file-intellisense-bug").includes("/Volumes/Code/tsserver-new-file-intellisense-bug/Baz.ts")
false

// ...updateFilesOfFileSystemEntry(...)

❯ this.readDirectory("/Volumes/Code/tsserver-new-file-intellisense-bug").includes("/Volumes/Code/tsserver-new-file-intellisense-bug/Baz.ts")
true

In the failing case, this always evaluates to false, even after the updateFilesOfFileSystemEntry call.

❯ this.readDirectory("/Volumes/Code/tsserver-new-file-intellisense-bug").includes("/Volumes/Code/tsserver-new-file-intellisense-bug/Baz.ts")
false

Ultimately, I think the problem is before the call to updateFilesOfFileSystemEntry problem though. I would guess updateFilesOfFileSystemEntry should be passed a value of fsQueryResult.fileExists that's true regardless of file system case sensitivity.

How does TypeScript decide when to lowercase?

The fileOrDirectoryPath may or may not be lowercased through:

TypeScript/src/server/editorServices.ts

Line 1505 in a0e0104

const fileOrDirectoryPath = this.toPath(fileOrDirectory);

TypeScript/src/server/editorServices.ts

Lines 1066 to 1068 in a0e0104

	toPath(fileName: string) {
	return toPath(fileName, this.currentDirectory, this.toCanonicalFileName);
	}

The this.toCanonicalFileName function performs lower-casing based on this.host.useCaseSensitiveFileNames.

TypeScript/src/compiler/core.ts

Line 2604 in a0e0104

export function createGetCanonicalFileName(useCaseSensitiveFileNames: boolean): GetCanonicalFileName {

TypeScript/src/server/editorServices.ts

Line 1028 in a0e0104

this.toCanonicalFileName = createGetCanonicalFileName(this.host.useCaseSensitiveFileNames);

There's likely an assumption to unwind related to this.host.useCaseSensitiveFileNames. I'm not sure if this is the TypeScript team's preferred fix, but I suspect we need to compute a different toCanonicalFileName function per-project or per-watched folder since the case sensitivity of an opened project folder is independent to where tsserver.js loads from.

Conclusion

The CachedDirectoryStructureHost has a bug where it does not update its cachedReadDirectoryResult internal variable if (1) newly created files have a capital letter and (2) tsserver.js is spawned from a file system volume that's case-insensitive.

[gluxon]

Thanks in advance to the TypeScript team for looking at this! Hope my initial debugging is helpful.

I think it narrows down the problem succinctly. I stopped short of recommending an exact fix since I'm still unsure what's best.

[gluxon]

Was curious how this.host.useCaseSensitiveFileNames is computed. Found isFileSystemCaseSensitive().

I like the // If this file exists under a different case, we must be case-insensitive trick here.

TypeScript/src/compiler/sys.ts

Lines 1731 to 1738 in b555727

	function isFileSystemCaseSensitive(): boolean {
	// win32\win64 are case insensitive platforms
	if (platform === "win32" || platform === "win64") {
	return false;
	}
	// If this file exists under a different case, we must be case-insensitve.
	return !fileExists(swapCase(__filename));
	}

---

I see this gets difficult to unwind. TypeScript associates the concept of file watching and case sensitivity together with a "host", but that host can be reused across volumes that have different case sensitivity settings. I think that leaves us with a few options for a bug fix, none of which are easy.

1. Unwind the host and case sensitivity association.
   

   TypeScript/src/compiler/sys.ts

   Lines 1393 to 1396 in b555727

   	export interface System {
   	args: string[];
   	newLine: string;
   	useCaseSensitiveFileNames: boolean;

2. Unwind the host and file watching association.
   

   TypeScript/src/server/types.ts

   Lines 20 to 22 in b555727

   	export interface ServerHost extends System {
   	watchFile(path: string, callback: FileWatcherCallback, pollingInterval?: number, options?: WatchOptions): FileWatcher;
   	watchDirectory(path: string, callback: DirectoryWatcherCallback, recursive?: boolean, options?: WatchOptions): FileWatcher;

3. Create new host objects for each project.

[sheetalkamat]

As per debugging, currently we determine case sensitivity is system wide and determined from location of the tsc/tsserver/typescript file and its volume. We use case sensitivity in so many places to determine if files are canonically same, watches, symlinks and many more that this change would need to handle so many different types of hosts we have, and in so many parts that we would need to wait for more data and figuring out if there are many more users facing this.

We would need to make useCaseSensitiveFileNames a function that determines based on file name
Handle getCanonicalFileName and toPath scenarios all over code.

[gluxon]

Thanks for taking a look @sheetalkamat. I agree that this would be a large refactor to fix, and the amount of work here may be disproportionate to the value a bug fix would provide. The Awaiting More Feedback label makes sense.

I would add however that this bug is deeply intricate. I do suspect many users hit this bug, but won't know the right search terms to discover this particular issue and voice their feedback. We may not get any feedback as a result despite this bug affecting every user that clones their git repo to a case sensitive volume.

[sheetalkamat]

I have added few more search terms and probably will keep on eye on issues reported esp on Mac os for this and link them to this one for awareness.

In the meantime, do you mind sharing tsserver log from this so that I can see if we can somehow unblock it a little bit easily without having to work on the whole thing. It would give inconsistent errors probably and things might not work as is but at least hoping that some of the places we are using "canonical paths" for existence/read checking can be moved to "file names" queried.

[gluxon]

Thanks Sheetal. I appreciate the help adding search terms and making this easier for others to find in the future. 🙂

In the meantime, do you mind sharing tsserver log from this so that I can see if we can somehow unblock it a little bit easily without having to work on the whole thing. It would give inconsistent errors probably and things might not work as is but at least hoping that some of the places we are using "canonical paths" for existence/read checking can be moved to "file names" queried.

Can do! I generated this tsserver.log from

1. Cloning https://github.com/gluxon/tsserver-new-file-intellisense-bug to /Volumes/git.
2. I created a new Bar.ts file and attempted to import the foo identifier.

This works when using the project's typescript.tsdk instance, but fails when using VS Code's tsserver instance since the later is on a different volume. The log is using VS Code's tsserver.

https://github.com/gluxon/tsserver-new-file-intellisense-bug/blob/main/sample-logs/2023-10-31%20tsserver.log

[sheetalkamat]

@gluxon you would want to try nightly and see how it fares. The fix does not go all the way to fix the issue but tries to use fileNames instead of canonical file names for host methods.