Merging ref assemblies feature into master

build/NuGetAdditionalFiles/Microsoft.Net.Compilers.props

@@ -23,6 +23,8 @@
              AssemblyFile="$(MSBuildThisFileDirectory)..\tools\Microsoft.Build.Tasks.CodeAnalysis.dll" />
   <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Vbc"
              AssemblyFile="$(MSBuildThisFileDirectory)..\tools\Microsoft.Build.Tasks.CodeAnalysis.dll" />
+  <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.CopyRefAssembly"
+             AssemblyFile="$(MSBuildThisFileDirectory)..\tools\Microsoft.Build.Tasks.CodeAnalysis.dll" />
   <PropertyGroup>
     <!-- By default don't use the compiler server in Visual Studio. -->
     <UseSharedCompilation Condition="'$(UseSharedCompilation)' == ''">false</UseSharedCompilation>

build/Targets/Versions.props

@@ -15,7 +15,8 @@
     <!-- The release moniker for our packages.  Developers should use "dev" and official builds pick the branch
          moniker listed below -->
     <RoslynNuGetMoniker Condition="'$(RoslynNuGetMoniker)' == ''">dev</RoslynNuGetMoniker>
-    <RoslynNuGetMoniker Condition="'$(OfficialBuild)' == 'true'">beta2</RoslynNuGetMoniker>
+    <!-- PROTOTYPE(refout) This needs to be reverted before merging to master -->
+    <RoslynNuGetMoniker Condition="'$(OfficialBuild)' == 'true'">refout</RoslynNuGetMoniker>
     <!-- This is the base of the NuGet versioning for prerelease packages -->
     <NuGetPreReleaseVersion>$(RoslynFileVersionBase)-$(RoslynNuGetMoniker)</NuGetPreReleaseVersion>
 

docs/compilers/CSharp/CommandLine.md

@@ -4,6 +4,7 @@
 | ---- | ---- |
 | **OUTPUT FILES** |
 | `/out:`*file* | Specify output file name (default: base name of file with main class or first file)
+| `/refout:`*file* | Specify the reference assembly's output file name
 | `/target:exe` |  Build a console executable (default) (Short form: `/t:exe`)
 | `/target:winexe` | Build a Windows executable (Short form: `/t:winexe` )
 | `/target:library` | Build a library (Short form: `/t:library`)
@@ -32,6 +33,7 @@
 | `/debug`:{`full`|`pdbonly`|`portable`} | Specify debugging type (`full` is default, and  enables attaching a debugger to a running program. `portable` is a cross-platform format)
 | `/optimize`{`+`|`-`} | Enable optimizations (Short form: `/o`)
 | `/deterministic` | Produce a deterministic assembly (including module version GUID and timestamp)
+| `/refonly | Produce a reference assembly, instead of a full assembly, as the primary output 
 | **ERRORS AND WARNINGS**
 | `/warnaserror`{`+`|`-`} | Report all warnings as errors
 | `/warnaserror`{`+`|`-`}`:`*warn list* | Report specific warnings as errors

docs/compilers/Visual Basic/CommandLine.md

@@ -4,6 +4,7 @@
 | ---- | ---- |
 | **OUTPUT FILE**
 | `/out:`*file* | Specifies the output file name.
+| `/refout:`*file* | Specify the reference assembly's output file name
 | `/target:exe` | Create a console application (default). (Short form: `/t`)
 | `/target:winexe` | Create a Windows application.
 | `/target:library` | Create a library assembly.
@@ -34,6 +35,7 @@
 | `/debug:portable` | Emit debugging information in the portable format.
 | `/debug:pdbonly` | Emit PDB file only.
 | `/deterministic` | Produce a deterministic assembly (including module version GUID and timestamp)
+| `/refonly | Produce a reference assembly, instead of a full assembly, as the primary output 
 | **ERRORS AND WARNINGS**
 | `/nowarn` | Disable all warnings.
 | `/nowarn:`*number_list* | Disable a list of individual warnings.

docs/features/refout.md

@@ -0,0 +1,78 @@
+# Reference assemblies
+
+Reference assemblies are metadata-only assemblies with the minimum amount of metadata to preserve the compile-time behavior of consumers (diagnostics may be affected, though).
+The compiler may choose to remove more metadata in later versions, if it is determined to be safe (ie. respects the principle above).
+
+## Scenarios
+There are 4 scenarios:
+
+1. The traditional one, where an assembly is emitted as the primary output (`/out` command-line parameter, or `peStream` parameter in `Compilation.Emit` APIs).
+2. The IDE scenario, where the metadata-only assembly is emitted (via `Emit` API), still as the primary output. Later on, the IDE is interested to get metadata-only assemblies even when there are errors in the compilation.
+3. The CoreFX scenario, where only the ref assembly is emitted, still as the primary output (`/refonly` command-line parameter) 
+4. The MSBuild scenario, which is the new scenario, where both a real assembly is emitted as the primary output, and a ref assembly is emitted as the secondary output (`/refout` command-line parameter, or `metadataPeStream` parameter in `Emit`).
+
+
+## Definition of ref assemblies
+Metadata-only assembly have their method bodies replaced with a single `throw null` body, but include all members except anonymous types. The reason for using `throw null` bodies (as opposed to no bodies) is so that PEVerify could run and pass (thus validating the completeness of the metadata).
+Ref assemblies will include an assembly-level `ReferenceAssembly` attribute. This attribute may be specified in source (then we won't need to synthesize it). Because of this attribute, runtimes will refuse to load ref assemblies for execution (but they can still be loaded Reflection-only mode).
+Ref assemblies further remove metadata (private members) from metadata-only assemblies:
+
+- A ref assembly will only have references for what it needs in the API surface. The real assembly may have additional references related to specific implementations. For instance, the ref assembly for `class C { private void M() { dynamic d = 1; ... } }` will not reference any types required for `dynamic`.
+- Private function-members (methods, properties and events) will be removed. If there are no `InternalsVisibleTo` attributes, do the same for internal function-members
+- But all types (including private or nested types) must be kept in ref assemblies. All attributes must be kept (even internal ones).
+- All virtual methods will be kept. Explicit interface implementations will be kept.
+- All fields of a struct will be kept. (This is a candidate for post-C#-7.1 refinement)
+
+## API changes
+
+### Command-line
+Two mutually exclusive command-line parameters will be added to `csc.exe` and `vbc.exe`:
+- `/refout`
+- `/refonly`
+
+The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below). The filename for the ref assembly should generally match that of the primary assembly (we will warn when they don't match). The recommended convention (used by MSBuild) is to place the ref assembly in a "ref/" sub-folder relative to the primary assembly.
+
+The `/refonly` parameter is a flag that indicates that a ref assembly should be output instead of an implementation assembly, as the primary output.
+The `/refonly` parameter is not allowed together with the `/refout` parameter, as it doesn't make sense to have both the primary and secondary outputs be ref assemblies. Also, the `/refonly` parameter silently disables outputting PDBs, as ref assemblies cannot be executed. 
+The `/refonly` parameter translates to `EmitMetadataOnly` being `true`, and `IncludePrivateMembers` being `false` in the `Emit` API (see details below).
+Neither `/refonly` nor `/refout` are permitted with net modules (`/target:module`, `/addmodule` options).
+
+The compilation from the command-line will either produce both assemblies (implementation and ref) or neither. There is no "partial success" scenario.
+When the compiler produces documentation, it is un-affected by either the `/refonly` or `/refout` parameters. This may change in the future.
+The main purpose of the `/refout` option is to speed up incremental build scenarios. The current implementation for this flag can produce a ref assembly with more metadata than `/refonly` (for instance, anonymous types). This is a candidate for post-C#-7.1 refinement.
+
+### CscTask/CoreCompile
+The `CoreCompile` target will support a new output, called `IntermediateRefAssembly`, which parallels the existing `IntermediateAssembly`.
+The `Csc` task will support a new output, called `OutputRefAssembly`, which parallels the existing `OutputAssembly`.
+Both of those basically map to the `/refout` command-line parameter.
+
+An additional task, called `CopyRefAssembly`, will be provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match (by comparing their MVIDs, see details below), then the destination file is left untouched.
+
+As a side-note, `CopyRefAssembly` uses the same assembly resolution/redirection trick as `Csc` and `Vbc`, to avoid type loading problems with `System.IO.FileSystem`.
+
+### CodeAnalysis APIs
+It is already possible to produce metadata-only assemblies by using `EmitOptions.EmitMetadataOnly`, which is used in IDE scenarios with cross-language dependencies.
+The compiler will be updated to honour the `EmitOptions.IncludePrivateMembers` flag as well. When combined with `EmitMetadataOnly` or a `metadataPeStream` in `Emit`, a ref assembly will be produced.
+The diagnostic check for emitting methods lacking a body (`void M();`) will be filtered from declaration diagnostics, so that code will successfully emit with `EmitMetadataOnly`.
+Later on, the `EmitOptions.TolerateErrors` flag will allow emitting error types as well.
+`Emit` is also modified to produce a new PE section called ".mvid" containing a copy of the MVID, when producing ref assemblies. This makes it easy for `CopyRefAssembly` to extract and compare MVIDs from ref assemblies.
+
+Going back to the 4 driving scenarios:
+1. For a regular compilation, `EmitMetadataOnly` is left to `false` and no `metadataPeStream` is passed into `Emit`.
+2. For the IDE scenario, `EmitMetadataOnly` is set to `true`, but `IncludePrivateMembers` is left to `true`.
+3. For the CoreFX scenario, ref assembly source code is used as input, `EmitMetadataOnly` is set to `true`, and `IncludePrivateMembers` is set to `false`.
+4. For the MSBuild scenario, `EmitMetadataOnly` is left to `false`, a `metadataPeStream` is passed in and `IncludePrivateMembers` is set to `false`.
+
+## Future
+As mentioned above, there may be further refinements after C# 7.1:
+- Further reduce the metadata in ref assemblies produced by `/refout`, to match those produced by `/refonly`.
+- Controlling internals (producing public ref assemblies)
+- Produce ref assemblies even when there are errors outside method bodies (emitting error types when `EmitOptions.TolerateErrors` is set)
+- When the compiler produces documentation, the contents produced could be filtered down to match the APIs that go into the primary output. In other words, the documentation could be filtered down when using the `/refonly` parameter.
+
+## Open questions
+
+## Related issues
+- Produce ref assemblies from command-line and msbuild (https://github.com/dotnet/roslyn/issues/2184)
+- Refine what is in reference assemblies and what diagnostics prevent generating one (https://github.com/dotnet/roslyn/issues/17612)
+- [Are private members part of the API surface?](http://blog.paranoidcoding.com/2016/02/15/are-private-members-api-surface.html)

src/Compilers/CSharp/Portable/CSharpCodeAnalysis.csproj

@@ -272,6 +272,7 @@
     <Compile Include="Emitter\EditAndContinue\EmitHelpers.cs" />
     <Compile Include="Emitter\EditAndContinue\PEDeltaAssemblyBuilder.cs" />
     <Compile Include="Emitter\EditAndContinue\CSharpSymbolMatcher.cs" />
+    <Compile Include="Emitter\Model\SourceAssemblySymbolAdapter.cs" />
     <Compile Include="Emitter\Model\ArrayTypeSymbolAdapter.cs" />
     <Compile Include="Emitter\Model\AssemblyReference.cs" />
     <Compile Include="Emitter\Model\AttributeDataAdapter.cs" />

src/Compilers/CSharp/Portable/CSharpResources.resx

@@ -2297,6 +2297,12 @@ If such a class is used as a base class and if the deriving class defines a dest
   <data name="ERR_Merge_conflict_marker_encountered" xml:space="preserve">
     <value>Merge conflict marker encountered</value>
   </data>
+  <data name="ERR_NoRefOutWhenRefOnly" xml:space="preserve">
+    <value>Do not use refout when using refonly.</value>
+  </data>
+  <data name="ERR_NoNetModuleOutputWhenRefOutOrRefOnly" xml:space="preserve">
+    <value>Cannot compile net modules when using /refout or /refonly.</value>
+  </data>
   <data name="ERR_OvlOperatorExpected" xml:space="preserve">
     <value>Overloadable operator expected</value>
   </data>
@@ -4405,6 +4411,7 @@ To remove the warning, you can use /reference instead (set the Embed Interop Typ
  /target:winmdobj              Build a Windows Runtime intermediate file that 
                                is consumed by WinMDExp (Short form: /t:winmdobj)
  /doc:&lt;file&gt;                   XML Documentation file to generate
+ /refout:&lt;file&gt;                Reference assembly output to generate
  /platform:&lt;string&gt;            Limit which platforms this code can run on: x86,
                                Itanium, x64, arm, anycpu32bitpreferred, or 
                                anycpu. The default is anycpu.
@@ -4448,6 +4455,7 @@ To remove the warning, you can use /reference instead (set the Embed Interop Typ
  /optimize[+|-]                Enable optimizations (Short form: /o)
  /deterministic                Produce a deterministic assembly
                                (including module version GUID and timestamp)
+ /refonly                      Produce a reference assembly in place of the main output
  /instrument:TestCoverage      Produce an assembly instrumented to collect
                                coverage information
  /sourcelink:&lt;file&gt;            Source link info to embed into Portable PDB.

src/Compilers/CSharp/Portable/CommandLine/CSharpCommandLineParser.cs

@@ -67,6 +67,8 @@ internal sealed override CommandLineArguments CommonParse(IEnumerable<string> ar
             string outputDirectory = baseDirectory;
             ImmutableArray<KeyValuePair<string, string>> pathMap = ImmutableArray<KeyValuePair<string, string>>.Empty;
             string outputFileName = null;
+            string outputRefFilePath = null;
+            bool refOnly = false;
             string documentationPath = null;
             string errorLogPath = null;
             bool parseDocumentationComments = false; //Don't just null check documentationFileName because we want to do this even if the file name is invalid.
@@ -381,6 +383,7 @@ internal sealed override CommandLineArguments CommonParse(IEnumerable<string> ar
                             }
 
                             continue;
+
                         case "out":
                             if (string.IsNullOrWhiteSpace(value))
                             {
@@ -393,6 +396,26 @@ internal sealed override CommandLineArguments CommonParse(IEnumerable<string> ar
 
                             continue;
 
+                        case "refout":
+                            value = RemoveQuotesAndSlashes(value);
+                            if (string.IsNullOrEmpty(value))
+                            {
+                                AddDiagnostic(diagnostics, ErrorCode.ERR_NoFileSpec, arg);
+                            }
+                            else
+                            {
+                                outputRefFilePath = ParseGenericPathToFile(value, diagnostics, baseDirectory);
+                            }
+
+                            continue;
+
+                        case "refonly":
+                            if (value != null)
+                                break;
+
+                            refOnly = true;
+                            continue;
+
                         case "t":
                         case "target":
                             if (value == null)
@@ -1150,6 +1173,16 @@ internal sealed override CommandLineArguments CommonParse(IEnumerable<string> ar
                 diagnosticOptions[o.Key] = o.Value;
             }
 
+            if (refOnly && outputRefFilePath != null)
+            {
+                AddDiagnostic(diagnostics, diagnosticOptions, ErrorCode.ERR_NoRefOutWhenRefOnly);
+            }
+
+            if (outputKind == OutputKind.NetModule && (refOnly || outputRefFilePath != null))
+            {
+                AddDiagnostic(diagnostics, diagnosticOptions, ErrorCode.ERR_NoNetModuleOutputWhenRefOutOrRefOnly);
+            }
+
             if (!IsScriptRunner && !sourceFilesSpecified && (outputKind.IsNetModule() || !resourcesOrModulesSpecified))
             {
                 AddDiagnostic(diagnostics, diagnosticOptions, ErrorCode.WRN_NoSources);
@@ -1264,7 +1297,8 @@ internal sealed override CommandLineArguments CommonParse(IEnumerable<string> ar
 
             var emitOptions = new EmitOptions
             (
-                metadataOnly: false,
+                metadataOnly: refOnly,
+                includePrivateMembers: !refOnly && outputRefFilePath == null,
                 debugInformationFormat: debugInformationFormat,
                 pdbFilePath: null, // to be determined later
                 outputNameOverride: null, // to be determined later
@@ -1290,8 +1324,9 @@ internal sealed override CommandLineArguments CommonParse(IEnumerable<string> ar
                 Utf8Output = utf8output,
                 CompilationName = compilationName,
                 OutputFileName = outputFileName,
+                OutputRefFilePath = outputRefFilePath,
                 PdbPath = pdbPath,
-                EmitPdb = emitPdb,
+                EmitPdb = emitPdb && !refOnly, // silently ignore emitPdb when refOnly is set
                 SourceLink = sourceLink,
                 RuleSetPath = ruleSetPath,
                 OutputDirectory = outputDirectory,