Skip to content

(GH-799) Revive docfx scripts #800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

glennsarti
Copy link
Contributor

Previously the document generation scripts for docfx were broken e.g. Unable to
download docfx from github due to TLS. This commit;

  • Updates the BuildDocs script to properly clean and download docfx from Github
  • Updates the docfx.json file to use the correct naming (cwd -> src) and to
    use the correct project files
  • Updates gitignore to ignore docfx working directory
  • Adds two build tasks (BuildDocs and ServeDocs) to build and serve
    autogenerated documentation
  • Adds additional configuration switch for Invoke-Build for legacy applications,
    like docfx, which use MSBuild as a library but it needs the dotNet Core
    versions of MSBuild instead of VS 2015/2017

@glennsarti glennsarti requested a review from rjmholt as a code owner November 26, 2018 04:17
@glennsarti
Copy link
Contributor Author

Note that this does not actually fix docfx, but at least makes it so it can be executed.

Building with the 1.x docfx gives

Build succeeded with warning.
Warning: Error opening project C:/source/PowerShellEditorServices/src/PowerShellEditorServices/PowerShellEditorServices.csproj: The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.. Ignored.
Warning: Error opening project C:/source/PowerShellEditorServices/src/PowerShellEditorServices.Protocol/PowerShellEditorServices.Protocol.csproj: The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.. Ignored.
Warning: Error opening project C:/source/PowerShellEditorServices/src/PowerShellEditorServices.Channel.WebSocket/PowerShellEditorServices.Channel.WebSocket.csproj: The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.. Ignored.
Warning: Error opening project C:/source/PowerShellEditorServices/src/PowerShellEditorServices.Host/PowerShellEditorServices.Host.csproj: The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.. Ignored.
Warning: Error opening project C:/source/PowerShellEditorServices/src/PowerShellEditorServices.VSCode/PowerShellEditorServices.VSCode.csproj: The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.. Ignored.
Warning: No metadata is generated for .
        6 Warning(s)
        0 Error(s)

While the 2.x docfx errors with

[18-11-26 03:58:56.852]Info:Completed in 5989.4747 milliseconds


Build succeeded with warning.
[18-11-26 03:58:56.861]Warning:[ExtractMetadata](C:/source/PowerShellEditorServices/src/PowerShellEditorServices.Channel.WebSocket/PowerShellEditorServices.Channel.WebSocket.csproj)Workspace failed with: [Failure] Msbuild failed when processing the file 'C:\source\PowerShellEditorServices\src\PowerShellEditorServices.Channel.WebSocket\PowerShellEditorServices.Channel.WebSocket.csproj' with message: C:\Program Files\dotnet\sdk\2.1.403\Roslyn\Microsoft.CSharp.Core.targets: (52, 5): The "Microsoft.CodeAnalysis.BuildTasks.Csc" task could not be loaded from the assembly C:\Program Files\dotnet\sdk\2.1.403\Roslyn\Microsoft.Build.Tasks.CodeAnalysis.dll. Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask.
[18-11-26 03:58:56.862]Warning:[ExtractMetadata]Project 'C:\source\PowerShellEditorServices\src\PowerShellEditorServices.Host\PowerShellEditorServices.Host.csproj' does not contain any documents.
[18-11-26 03:58:56.862]Warning:[ExtractMetadata]Project 'C:\source\PowerShellEditorServices\src\PowerShellEditorServices\PowerShellEditorServices.csproj' does not contain any documents.
[18-11-26 03:58:56.862]Warning:[ExtractMetadata]Project 'C:\source\PowerShellEditorServices\src\PowerShellEditorServices.Protocol\PowerShellEditorServices.Protocol.csproj' does not contain any documents.
[18-11-26 03:58:56.862]Warning:[ExtractMetadata]Project 'C:\source\PowerShellEditorServices\src\PowerShellEditorServices.VSCode\PowerShellEditorServices.VSCode.csproj' does not contain any documents.
[18-11-26 03:58:56.862]Warning:[ExtractMetadata]Invalid cref value "!:ChannelBase" found in triple-slash-comments for WebsocketClientChannel defined in ../src/PowerShellEditorServices.Channel.WebSocket/WebsocketClientChannel.cs Line 20, ignored.
[18-11-26 03:58:56.862]Warning:[ExtractMetadata]Invalid cref value "!:ChannelBase" found in triple-slash-comments for WebSocketServerChannel defined in ../src/PowerShellEditorServices.Channel.WebSocket/WebsocketServerChannel.cs Line 22, ignored.
[18-11-26 03:58:56.863]Warning:[ExtractMetadata]Invalid cref value "!:WebSocketConnection" found in triple-slash-comments for Dispatch defined in ../src/PowerShellEditorServices.Channel.WebSocket/WebsocketServerChannel.cs Line 57, ignored.
[18-11-26 03:58:56.863]Warning:[ExtractMetadata]Invalid cref value "!:MessageDispatcher" found in triple-slash-comments for Dispatch defined in ../src/PowerShellEditorServices.Channel.WebSocket/WebsocketServerChannel.cs Line 57, ignored.
[18-11-26 03:58:56.863]Warning:[ExtractMetadata]Invalid cref value "!:LanguageServer" found in triple-slash-comments for LanguageServerWebSocketConnection defined in ../src/PowerShellEditorServices.Channel.WebSocket/WebsocketServerChannel.cs Line 138, ignored.
[18-11-26 03:58:56.863]Warning:[ExtractMetadata]Invalid cref value "!:DebugAdapter" found in triple-slash-comments for DebugAdapterWebSocketConnection defined in ../src/PowerShellEditorServices.Channel.WebSocket/WebsocketServerChannel.cs Line 149, ignored.
        11 Warning(s)
        0 Error(s)
[18-11-26 03:58:57.275]Info:6 plug-in(s) loaded.
[18-11-26 03:58:57.334]Info:No files are found with glob pattern apidoc/**.md, excluding obj/**,_site/**, under directory "C:\source\PowerShellEditorServices\docs"
[18-11-26 03:58:57.360]Info:Markdown engine is dfm
[18-11-26 03:58:57.778]Info:[BuildCore.Build Document]Max parallelism is 8.
[18-11-26 03:58:57.846]Info:[BuildCore.Build Document.Prepare.CreateIncrementalBuildContext]Build strategy: IsIncrementalBuild
[18-11-26 03:58:58.390]Info:[BuildCore.Build Document.Load.TocDocumentProcessor](metadata/api/toc.yml)For git repo <C:/Source/PowerShellEditorServices>, can't find remote branch in this repo and fallback to use local branch [make-docfx-work]: fatal: no upstream configured for branch 'make-docfx-work'

[18-11-26 03:58:58.397]Info:[BuildCore.Build Document.Load.TocDocumentProcessor](toc.yml)For git repo <C:/Source/PowerShellEditorServices>, can't find remote branch in this repo and fallback to use local branch [make-docfx-work]: fatal: no upstream configured for branch 'make-docfx-work'

[18-11-26 03:58:58.405]Info:[BuildCore.Build Document.Load.TocDocumentProcessor](guide/toc.md)For git repo <C:/Source/PowerShellEditorServices>, can't find remote branch in this repo and fallback to use local branch [make-docfx-work]: fatal: no upstream configured for branch 'make-docfx-work'

[18-11-26 03:58:58.517]Info:[BuildCore.Build Document.CompilePhaseHandlerWithIncremental.ManagedReferenceDocumentProcessor]Building 0 file(s) in ManagedReferenceDocumentProcessor(BuildManagedReferenceDocument=>ValidateManagedReferenceDocumentMetadata=>ApplyOverwriteDocumentForMref=>FillReferenceInformation)...
[18-11-26 03:58:58.520]Info:[BuildCore.Build Document.CompilePhaseHandlerWithIncremental.TocDocumentProcessor]Building 3 file(s) in TocDocumentProcessor(BuildTocDocument)...
[18-11-26 03:58:58.520]Info:[BuildCore.Build Document.CompilePhaseHandlerWithIncremental.ResourceDocumentProcessor]Building 3 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
[18-11-26 03:58:58.526]Info:[BuildCore.Build Document.CompilePhaseHandlerWithIncremental.ConceptualDocumentProcessor]Building 0 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>CountWord=>ValidateConceptualDocumentMetadata)...
[18-11-26 03:58:58.545]Warning:[BuildCore.Build Document.CompilePhaseHandlerWithIncremental.TocDocumentProcessor.Prebuild.BuildTocDocument](toc.yml)Unable to find either toc.yml or toc.md inside api/. Make sure the file is included in config file docfx.json!
[18-11-26 03:58:58.752]Info:[BuildCore.Build Document.LinkPhaseHandlerWithIncremental.UpdateContext]0 external references found in 1 xref maps.
[18-11-26 03:58:58.863]Info:[BuildCore.Build Document.LinkPhaseHandlerWithIncremental.Apply Templates]Applying templates to 6 model(s)...
[18-11-26 03:58:59.401]Info:[BuildCore.Build Document]XRef map exported.
[18-11-26 03:58:59.454]Info:Completed building documents in 2165.3796 milliseconds.
[18-11-26 03:58:59.463]Error:System.AggregateException: One or more errors occurred. ---> Microsoft.DocAsCode.Build.Engine.Incrementals.BuildCacheException: Last incremental post processor outputs should contain guide/using_the_host_process.html.
   at Microsoft.DocAsCode.Build.Engine.PostProcessorsHandlerWithIncremental.<CopyToCurrentCache>b__8_4(OutputFileInfo item)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body)
   at Microsoft.DocAsCode.Build.Engine.PostProcessorsHandlerWithIncremental.CopyToCurrentCache(List`1 increItems)
...

Copy link
Contributor

@rjmholt rjmholt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to try running this before approving, but it looks really good to me :) Thanks @glennsarti!


Write-Host "`n### Using dotnet SDK at path $SDKVersionPath" -ForegroundColor Green
} else {
Throw "Unable to find any SDKs in path $SDKPath"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep Throw as lowercase just since all the other keywords we use are lowercase.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@@ -11,18 +11,47 @@ $docsRepoPath = [System.IO.Path]::GetFullPath("$PSScriptRoot\..\docs\_repo")
# Ensure the tools path exists
mkdir $toolsPath -Force | Out-Null

if (![System.IO.File]::Exists($docfxBinPath)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not Test-Path?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(That will be more tolerant of xplat paths)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise worth a comment as to why not

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe wrap with a function that resolves the path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copying the code style of

if (![System.IO.File]::Exists($docfxExePath)) {

I defer to having a consistent style (even if wrong) than inconsistent. Do you want me to change both?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think change both

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed both.

# 2.1 support seems to be slated for docfx 3.x release - https://github.com/dotnet/docfx/projects/1
#
# For now use the 1.x series of docfx
(new-object net.webclient).DownloadFile('https://github.com/dotnet/docfx/releases/download/v1.9.4/docfx.zip', "$docfxZipPath")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for not using Invoke-WebRequest?

Copy link
Contributor Author

@glennsarti glennsarti Nov 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea. This is just a whitespace and version update of the original code from @daviwil 3 years ago. https://github.com/PowerShell/PowerShellEditorServices/blame/f847e2fd64f89edcc089803dd03775409e511344/scripts/BuildDocs.ps1#L17

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, up to you, but Invoke-WebRequest would be my preference here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the age, it was probably due to IWR being SOO slow compared to WebClient

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there were (are) issues with IWR in the Windows PowerShell and early versions of PowerShell Core. Do we still need to build on a system running PS 5.1?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Changed to IWR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah didn't know about issues. Early versions of PSCore won't be a problem since they're out of support in a few months. WinPS I have no idea, but performance shouldn't be an issue for official builds at least.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested on WinPS (I haven't made the change PSCore yet ⏲).

@glennsarti glennsarti force-pushed the make-docfx-work branch 2 times, most recently from e7e0485 to 4ce1676 Compare November 28, 2018 02:08
@glennsarti
Copy link
Contributor Author

glennsarti commented Nov 28, 2018

I want to try running this before approving, but it looks really good to me

@rjmholt A reminder that this doesn't fix the documentation problem, this just gets us to a point where we can start trying to fix it.

So IMO, cross platform support of the documentation generation isn't a priority right now.

@rjmholt
Copy link
Contributor

rjmholt commented Nov 28, 2018

@rjmholt A reminder that this doesn't fix the documentation problem, this just gets us to a point where we can start trying to fix it.

So IMO, cross platform support of the documentation generation isn't a priority right now.

Understood. It should run on Windows though, right?

Previously the document generation scripts for docfx were broken e.g. Unable to
download docfx from github due to TLS.  This commit;
* Updates the BuildDocs script to properly clean and download docfx from Github
* Updates the docfx.json file to use the correct naming (cwd -> src) and to
  use the correct project files
* Updates gitignore to ignore docfx working directory
* Adds two build tasks (BuildDocs and ServeDocs) to build and serve
  autogenerated documentation
* Adds additional configuration switch for Invoke-Build for legacy applications,
  like docfx, which use MSBuild as a library but it needs the dotNet Core
  versions of MSBuild instead of VS 2015/2017
This commit removes redundant code from the build scripts.
@glennsarti
Copy link
Contributor Author

glennsarti commented Nov 28, 2018

Understood. It should run on Windows though, right?

Yup! PowerShell 6 (and probably Windows PowerShell)

I still haven't figured out the magical environment for docfx to work yet. But that's for another PR..

@rjmholt
Copy link
Contributor

rjmholt commented Nov 28, 2018

I'll see if I can track down someone to fill me in on docfx.

I know the PowerShell team is standardising on markdown instead, but I'm guessing docfx has a more structured website integration story.

@glennsarti
Copy link
Contributor Author

Actually that's a really good point. What's https://github.com/PowerShell/PowerShell using for automated docs creation?

@rjmholt
Copy link
Contributor

rjmholt commented Nov 28, 2018

PowerShell's docs are kept in the PowerShell-Docs repo. That stores all the docs as markdown files and builds them with platyPS, but I think it's reasonably specific to PowerShell — not meant for .NET code.

@rjmholt
Copy link
Contributor

rjmholt commented Nov 28, 2018

Here's an example: MicrosoftDocs/PowerShell-Docs#3238

@glennsarti
Copy link
Contributor Author

glennsarti commented Nov 28, 2018

I meant the documentation of C# classes of PowerShell e.g. Where does this information go?

@rjmholt
Copy link
Contributor

rjmholt commented Nov 28, 2018

Microsoft.PowerShell.LocalAccounts isn't a great example, since it's not in PowerShell Core. But something like this comes from our GitHub code.

I've spoken to @SteveL-MSFT and @joeyaiello and the way this works is the PowerShell SDK gets published on nuget and the documentation XML gets pulled from the package and semi-automatically is published to docs.microsoft.com.

After discussing with them, publishing to nuget seems to be the way we want to go. It will take a little bit longer, but should make API documentation much easier to discover and keep it up to date once we integrate nuget publishing into our release process.

@rjmholt
Copy link
Contributor

rjmholt commented Nov 28, 2018

I've reopened #41 for this.

@sdwheeler
Copy link

@rjmholt The docs for the dotnet classes in PowerShell are in MicrosoftDocs/powershell-docs-sdk-dotnet. This content is autogenerated from the nuget and XML. This content must be updated in the source code (/// comments) and new nupkg/XML created. Changes made in this repo get overwritten by the autogeneration processes.

@rjmholt
Copy link
Contributor

rjmholt commented Dec 17, 2018

The docs for the dotnet classes in PowerShell are in MicrosoftDocs/powershell-docs-sdk-dotnet. This content is autogenerated from the nuget and XML. This content must be updated in the source code (/// comments) and new nupkg/XML created. Changes made in this repo get overwritten by the autogeneration processes.

Yes understood. Thanks @sdwheeler

@andyleejordan
Copy link
Member

Ahh, this is very out of date, I need to close it, sorry! Thanks anyway for the work. I am looking into setting up a GitHub Action to run docfx and update our documentation automatically, referencing this will be helpful 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants