Add support for source generators in Razor compiler

[captainsafia]

• Update the version of Microsoft.CodeAnalysis.CSharp that we consume otherwise we get some downstream assembly resolution issues in the version of Microsoft.AspNetCore.Razor.Extensions that we use in the SDK repo.
• Add IVT info for M.NET.Sdk.Razor.SourceGenerators assembly. I'm working around this for now by giving the source generators package an rzc assembly name over in the sdk repo.
• Read from source document in project item. I'm working around this by implementing a Read method that returns a text stream from the string of a SourceText object but that's not ideal.

Part of #26902

[pranavkm]

I think the new compiler introduces some additional nullability warnings that you would need to address. But otherwise this looks good.

FYI @NTaylorMullen

[NTaylorMullen]

Hmm, this is concerning. The intent of RazorProjectItem is to be decoupled from a SourceDocument entirely. Why do you need this here? Typically we just re-create a project item.

[pranavkm]

One of the things that helped with perf was having a RazorSourceLineCollection that was backed by the Roslyn SourceText document (because that's what we're provided in a source generator). Here's what that looks like: https://github.com/dotnet/aspnetcore/pull/28807/files#diff-04e8fa11176d9e8cdfa5d0edeeb3ba7c1c6d20683201775277a3215a1fcb1a3f.

If I'm not mistaken, there isn't an API that lets you affect how a RazorProjectItem gets turned in to a RazorSourceDocument. We could create one where the default implementation reads from the stream and produces a source document and the source generator configures it with a custom provider. This one felt easier to implement as we did not have to introduce yet another concept, but you have more context here, so open to suggestions.

[NTaylorMullen]

Why not just re-create a source document?

[captainsafia]

This and the change above are designed to work in conjunction with some work we've done to support source generators.

We have a new SourceGeneratorProjectItem (ref) type that gets constructed from the list of files that Roslyn provides in a generator's compilation context (ref).

These additional files are provided as SourceTexts. We have a SourceTextRazorSourceDocument type (ref) that we create from the SourceText.

Here we are updating the ReadFrom method to directly use the SourceDocument that we compute from the SourceText provided by Roslyn instead of reading in the file contents from a stream.

You can view a hacky workaround I used to get around this here by reading the string of the SourceText into a stream but this isn't ideal.

Hope this explanation was clear. Thoughts?

[NTaylorMullen]

Thanks for the clarification, definitely helps piece it all together a bit more. It's been a while since I've dug through the code but why can't we have the project item read directly from the Roslyn source text?

[captainsafia]

We did create a ProjectItem implementation that reads directly from the Roslyn source text (ref). But currently, the project engine computes the source document (ref) from a project item by reading from a stream (ref).

We can try implementing an overload of ReadFrom that takes in a SourceGeneratorProjectItem

public static RazorSourceDocument ReadFrom(SourceGeneratorProjectItem projectItem)
{
   return new SourceGeneratorSourceDocument(projectItem.AdditionalText); // or similiar
}

But that would add source generator-specific logic in the compiler.

An alternative is to create some sort of abstraction where in a custom ProjectItem can define how source documents should be read from it similar to @pranavkm's recommendation above.

[NTaylorMullen]

An alternative is to create some sort of abstraction where in a custom ProjectItem can define how source documents should be read from it similar to @pranavkm's recommendation above.

Ya I think that'd be better. It'd be a more defined route of translating from one type to another.

[captainsafia]

Yep! Will create an issue to track this. We want to get the initial E2E with source generators working then refine things.

[captainsafia]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[dougbu]

@captainsafia what in particular would you like me to review❔ If it's just the CODEOWNERS file, no objections from me

/fyi dotnet/install-scripts#136 is blocking this PR at the moment

[dougbu]

Turns out we have to make some updates to the PublicShipped.API.txt files under both the Component and Ignitor project since we compile include source files from the components project in the ignitor project.

Changing the PublicAPI.Shipped.txt files outside an X.Y.0 release is never the right idea. @JunTaoLuo is scrambling to fix up the places where we've made this mistake. Please don't make it worse.

If necessary, change the PublicAPI.Unshipped.txt files to remove changed APIs. The syntax is to copy the line from PublicAPI.Shipped.txt (again, not changing that file) into PublicAPI.Unshipped.txt and add a *Removed* prefix.

[pranavkm]

@dougbu in this case it's changing the nullability annotation on a shipped API. We've made a number of those so far - aspnet/Announcements#444.

[dougbu]

We've made a number of those so far - aspnet/Announcements#444.

I know. The problem isn't the nullabiltity changes but how they were handled in the API baseline files. PublicAPI.Shipped.txt files should change only immediately after X.Y.0 ships.