[PSCore Align] Adopt build.ps1 build concept

[TylerLeonhardt]

PowerShell Core has this concept of a build.psm1 file that contains a bunch of functions for building and testing PSCore: https://github.com/PowerShell/PowerShell/blob/master/build.psm1

We should align with those processes so that it's consistent across PowerShell repos.

We can still continue to use InvokeBuild, but we can wrap the Invoke-Build * with the same entrypoints that PSCore has.

[TravisEz13]

We should discuss what is actually required here.

[MartinSGill]

I've just been trying to build using vscode-powershell.build.ps1 and got a load of errors ; because I actually just ran the script. So I'm very interested in improvements to the build.

Does this change mean removing the dependency on InvokeBuild?

build.psm1 (powershell/powershell) exports the following methods.

• Clear-PSRepo
• Compress-TestContent
• Convert-TxtResourceToXml
• Find-Dotnet
• Get-EnvironmentInformation
• Get-PackageVersionAsMajorMinorBuildRevision
• Get-PesterTag
• Get-PSCommitId
• Get-PSLatestTag
• Get-PSOptions
• Get-PSOutput
• Get-PSVersion
• Get-RedHatPackageManager
• Install-Dotnet
• log
• logerror
• New-MSIPackage
• New-PSOptions
• precheck
• Publish-NuGetFeed
• Publish-PSTestTools
• Restore-GitModule
• Restore-PSModule
• Restore-PSModuleToBuild
• Restore-PSPester
• Set-PSOptions
• Show-PSPesterError
• Start-BuildNativeUnixBinaries
• Start-BuildNativeWindowsBinaries
• Start-CrossGen
• Start-DevPowerShell
• Start-NativeExecution
• Start-PSBootstrap
• Start-PSBuild
• Start-PSPester
• Start-PSxUnit
• Start-ResGen
• Start-TypeGen
• Start-UnelevatedProcess
• Sync-PSTags
• Test-PSPesterResults
• Test-Win10SDK
• Test-XUnitTestResults
• Use-MSBuild

Which of these constitutes an "entrypoint" that should be implemented? None of these look like standard/obvious commands to call; as I discovered when I wasn't able to figure out how to build PSCore without reading the docs/appveyor.yml.

I feel the following points ought to be considered:

1. build.psm1 - Had to read docs/appveyor.yml to figure out how to build it.
2. The existing build.ps1 is also not intuitive to use. You need to run it with Invoke-Build, instead of just calling it as you would a normal script.
3. One advantage of a framework like PSake, MSBuild, Maven, et. al. is they make it easy to figure out how to build; especially as there are defaults and they enforce an interface. See a .proj, call msbuild, see a .pom call maven etc.

[MartinSGill]

Since msbuild is already being used in most projects, why not standardise on that?

vscode-powershell.proj
powershell.proj
powershell-editor-services.proj
etc.

This defines the targets and doesn't need to be more than a wrapper around powershell scripts.

[rkeithhill]

@tylerl0706 While I agree that consistency is nice PowerShell Core might not be the best "baseline" for every other PowerShell project. It is quite unique and quite a large build.

It seems to me we have this "grouping" of PowerShell projects:

• PowerShell modules - mostly script modules e.g. PowerShellGet, Plaster. InvokeBuild/psake work fine here
• PowerShell modules - binary (or binary & script) e.g. PSReadLine, PSScriptAnalyzer. This requires invoking msbuild to build C# code.
• VSCode extension - this "build" is largely driven by NPM but can be orchestrated by InvokeBuild or psake.
• PS Core - large, fairly complex custom build system.

So I don't think it makes sense to try to apply the same "custom" build system to other, much simpler to build projects. OTOH it might be nice to perhaps have a common way to start the build no matter what tool is used to build the project. Maybe that could be build.ps1 or <project-name>.build.ps1?

BTW the latest version of psake now automatically recognizes a file called psakefile.ps1. So in a directory with a psakefile.ps1 file in it, you can simply run invoke-psake and it will run the default task in that file - which is typically build.

@MartinSGill vscode-powershell.build.ps1 is an InvokeBuild recipe. You have to pass that into Invoke-Build. That said, I never invoke that script directly. I invoke it from within VSCode via the task runner. If you open the extension's root folder in VSCode (after you have git cloned PSES into a peer folder), run task Install and then task Build or just press F5 to start debugging which will execute the BuildAll task. BuildAll builds PSES as well.

[MartinSGill]

@rkeithhill While using VSCode is nice and hardly an outlandish suggestion given that it's a vscode extension, I've always taken the view that a build system that cannot give you the most common build scenario with a single command is incomplete; be that command "maven", "msbuild", "gradle", Invoke-Build or whatever else.

I also feel that easy discovery is important. Powershell made this deliberate choice with the cmdlet naming convention and standardised verbs. The commands might get a bit long winded at times, but this type of design decision is what attracted me to powershell in the first place.

In many ways first impressions count, and the first impression anyone who thinks of contributing code to a project has is how easily they can get it to build.

Powershell doesn't really have a recognisable build system yet, so simply seeing a *.csproj or *.pom file like you would in a C# or Java project to identify build system isn't an option.

• vscode-powershell.build.ps1 looks as though it should be run to build the project. Yet it throws an error if you try; and not a very helpful one.
• build.psm1 doesn't look like it should be run, it looks like something else uses it.
• a psakefile.ps1, while it still looks like you should run it, at least provides a good clue, just as a Rakefile does for ruby, excepting of course that rakefiles don't have an extension, despite being ruby source.
• Personal experience tends me towards Invoke-Build over psake, but invoke-build doesn't have such an obvious build file.

I think the jury is still out on the preferred build system for powershell related projects, nothing seems to have established itself as the defacto standard yet. Whatever the various powershell projects align on will have a big part in establishing such a defacto standard.

[rkeithhill]

@MartinSGill I agree with just about everything you're saying. Where I work - doing cross-plat C++ development - we have settled upon using build.ps1 at the root of the repo. Any dev can walk up to a system and run build.ps1 and it will generate the cmake cache and the build the project. Many times devs only want the cmake cache gen'd and will load the generated SLN into Visual Studio. In that case, they can run build.ps1 -CMakeCacheGenOnly. If they want to do any of this with a "clean" they just add -Clean.

We have quite a bit of stuff that needs to be installed & configured for dev/build machines: app installs, PS module installs, Conan, NPM & Artifactory config. I've started to implement that in build.ps1 via the switch -BootstrapBuildEnv. This is similar to the PS Core's Start-PSBootstrap command but our build's are still simple enough to handle this with a parameter, rather than a separate command.

I could see VSCode having a build.ps1 file that simply runs:

#requires -Modules InvokeBuild
Invoke-Build Restore
Invoke-Build Build # or maybe BuildAll

For an example of this see: https://github.com/PowerShell/Plaster/blob/master/build.ps1 While you're there, check out:
https://github.com/PowerShell/Plaster/blob/master/build.settings.ps1 and
https://github.com/PowerShell/Plaster/blob/master/build.psake.ps1

At one point, I had this idea that PowerShell needs a project system like the C# project system - where NOBODY worries about writing custom MSBuild targets files to embed resources or strong name or these days - emit a NuGet pkg. You simply provide the correct properties for the MSBuild targets file for C#. Personally I still like this idea for PowerShell. I can see an equivalent "targets" file for PowerShell modules that can handle various phases (some optional) such as: Clean, CompileSources (C#), GenerateHelp (platyPS), CopyArtifactsToOutDir (signing here doesn't generate Git changes), Analyze (PSSA), RunTests (Pester), SignFiles, GenerateModuleCatalog. And as a manual step - PublishModule.

It is interesting to see various build tools trying to replicate MSBuild but with syntax specific for that language e.g. rake, psake, cake, InvokeBuild, etc. It really makes me wish MSBuild had a way to allow for DSLs to use it. Short of that, I wonder if you could create a PowerShell friendly syntax that you could then render to a temp MSBuild file (kind of like MSBuild does for SLN files) and then use MSBuild to build it? Hmm....

[nightroman]

You can make Invoke-Build scripts directly invokable easily

Project.build.ps1

param(
    [Parameter(Position=0)]
    $Tasks,
    #... other script parameters
)

if ($MyInvocation.ScriptName -notlike '*Invoke-Build.ps1') {
    Invoke-Build $Tasks $MyInvocation.MyCommand.Path @PSBoundParameters
    return
}

# The usual build script stuff
task ...

See the full topic Tasks/Direct.

And you can extend this idea in order to add automatic bootstrapping to such a script.
See the topic Tasks/Paket.
This sample uses paket but you can use other package managers in the same way.

As a result, a user may just clone a project and run Project.build.ps1 directly.
This triggers bootstrapping and then invokes Build by just downloaded Invoke-Build.

[nightroman]

As far as this project now uses Invoke-Build, let me suggest a tweak. The project appveyor.yml fixes IB version to 3.2.1, quite old. Please use the latest 4.2.0 (nothing breaking for the project script). You will get colored Invoke-Build output in AppVeyor and some improved error logging, also done for AppVeyor. See PSReadLine AppVeyor log and its log with an error.

[TylerLeonhardt]

Hey everyone - so I want to first be absolutely clear here, I had no intention of removing InvokeBuild from the build of vscode-powershell or pses.

The reason behind this issue is because @SteveL-MSFT is now the lead of this project and pses (and I'm a dev on these) and since he also owns PSCore, we wanted to try to align the projects together in order to have a consistent build experience so that anyone can jump on to these projects and run the same commands to build/clean/bootstrap/etc.

Now, about your comments so far, I really love the brainstorming going on here. I like @rkeithhill's example build.ps1:

I could see VSCode having a build.ps1 file that simply runs:

#requires -Modules InvokeBuild
Invoke-Build Restore
Invoke-Build Build # or maybe BuildAll

and think that @nightroman's automatic bootstrapping is something we should investigate.

One thing I'd like to achieve in this issue is that those that are use to the existing build process for vscode-powershell and pses should not have to change their ways just because we want to align with PSCore. Keith's example is a perfect example of this as folks could call the build.ps1 or call Invoke-Build themselves.

[TylerLeonhardt]

Note to self: don't open a vague issue and then go on holiday 😅

[rkeithhill]

No worries. It's been a good discussion.

[TylerLeonhardt]

Agreed! I'd like to hear @SteveL-MSFT's thoughts when he gets back from vacation.

[MartinSGill]

Looking forward to seeing where this goes.

[SteveL-MSFT]

I think there's been some great discussion here. My objective is to simply have consistency across PowerShell Open Source projects so that contributors (including my team members) don't have to learn a different system for each repo. Minimally for the repos I own, but ideally we can set a pattern that any PowerShell project should adopt. I'm most familiar with what we've done with PSCore6 and agree that it's a bit unique, but I think conceptually, a new contributor should be able to:

• easily understand how to on-board (CONTRIBUTING.md)
• easily boot strap the engineer toolchain
• easily build the product
• easily run tests

However, there's also some logistical work that needs to be done to make sure all of the Microsoft repos are in Compliance. For example, before we sign binaries/packages with Microsoft's private key, we need to ensure that it was built on a secure system at Microsoft (AppVeyor/Travis builds aren't sufficient for release). So with multiple projects in multiple repos, it's better to have similar ways to build across our repos to make it easier to on-board to our internal compliance tools rather than incurring onboarding costs for each repo as they have distinct ways of doing builds/releases.

I'm fine with having 2 similar but different ways of doing things for small projects vs big projects if it improves engineering efficiency and keeps things simpler.

[rkeithhill]

Cool. I propose that the vast majority of simpler, module-based PowerShell projects use a more-or-less standardized repo root level build.ps1 script. The file should have a shebang:

#!/usr/bin/env pwsh

And be committed to Git with (so you can fire it off from Bash or another shell):

git update-index --chmod=+x .\build.ps1

The script when run without any parameters would ideally "build" the module - whatever that means. It could have an optional -Clean parameter to do a clean before building. Another optional feature would be a -BootstrapBuildEnv parameter that would attempt to install the required dev tools - up to a point. I wouldn't install VS for instance. This latter support is challenging when it comes to cross-platform development and varying pkg mgrs. - apt, yum, choco, etc.

Whether that script chains down to a tool like psake or InvokeBuild (or MSBuild) shouldn't matter as much for a person just starting dev work on a project.

Anyway, just a few thoughts to continue the discussion. :-)

[SteveL-MSFT]

The standardization on build.ps1 makes sense to me to abstract the actual build tools. If you're using shebang (which I like), you would need to use pwsh instead of powershell :)

I think it's fine for -BootstrapBuildEnv to just do checks and output messages on what is missing rather than automatically installing (which can be addressed over time).

[rkeithhill]

Oops, copied that from an old repo. Yeah, should be pwsh. Fixed. :-)

[nightroman]

Some thoughts on build.ps1 with InvokeBuild or psake and bootstrapping,
to ensure consistent and reproducible build environments.

Install-Module in the bootstrapping part should use RequiredVersion with
some tested version. Major updates in build tools may break build scripts.
Even minor updates may break due to bugs or subtle incompatible changes.

Import-Module in the building part should use RequiredVersion, too, the
same as used for Install-Module. Otherwise, i.e. if Import-Module is
omitted (automatic import is often assumed) or its RequiredVersion is
omitted, then some of the existing module versions on a particular
machine may be imported, not always the same as in Install-Module.

[nightroman]

Build tools may be installed to and imported from a local temporary directory (say, packages). This way does not pollute standard module directories.

[TravisEz13]

Using Invoke-Build directly seems far more flexible to me.

[rkeithhill]

I guess what I'm shooting for is more of a PowerShell community standard and as good as InvokeBuild is, not everybody is using. There's a decent contingent of folks using psake. And where I work, our central build process is a combination of CMake & Conan. Still, we have a single build.ps1 script that kicks off the build for these projects as well. This is similar to OSS projects that have a build.bat or build.cmd that kicks off make, cmake, msbuild, or whatever else.

[nightroman]

a single build.ps1 script that kicks off the build for these projects as well.

This was exactly a reason why InvokeBuild did not choose this name for its default script.

[TylerLeonhardt]

Great discussion everyone! Compiling our thoughts... and copy and pasting a bunch, here's what I have:

Module-based PowerShell projects should* use a repo root level build.ps1 script.

Tasks

• The script when run without any parameters would "build" the module.
• optional -Clean parameter can be available.
• optional -BootstrapBuildEnv parameter that would do checks and output messages on what is missing from your dev tools

Contents

What the build.ps1 does under-the-hood is under the owners' control 🙂 - InvokeBuild, psake, msbuild

However, the file should have a shebang:

#!/usr/bin/env pwsh

And be committed to Git with (so you can fire it off from Bash or another shell):

git update-index --chmod=+x .\build.ps1

[TylerLeonhardt]

vscode-powershell & PowerShellEditorServices can follow this.

Their build.ps1's will just leverage the InvokeBuild tasks that they are already using.

"Build" would run Invoke-Build Build
-Clean would run Invoke-Build Clean
-BootstrapBuildEnv may need a little bit of extra work for vscode-powershell to check for node and such but should be trivial.

Thoughts?

[SteveL-MSFT]

Sounds good. We can stage this so that -BootstrapBuildEnv can come as separate PR.
For running tests, is it simply Invoke-Pester from the root of the repo?

[nightroman]

@TravisEz13

Using Invoke-Build directly seems far more flexible to me.

If a project and its build.ps1 use InvokeBuild then you can continue to call
Invoke-Build directly, so you do not miss anything. But for many users it is
more intuitive to clone a project and run something like build.x.

As far as I understand build.ps1 will be responsible for bootstrapping.
Thus, a build script does not have to care of this and becomes cleaner.

---

The approach of using helper scripts build.x and hiding used build engines in
the top level commands is rather popular.

FAKE based projects provide build.cmd and build.sh in addition to the
actual F# build scripts build.fs. See FAKE.

Another popular build engine Cake provides build.ps1 in addition to the
actual build script build.cake, see Cake.

[MartinSGill]

Looks good to me. I've already been using a build.ps1 wrapper for my c# / npm projects, so this suits me really well.

We could add a -test switch to build.ps1, and indeed if you also add a publish/package option then the build script would do 90% of everything most developers will ever need to do, and all without them having to gain any underlying knowledge of the build/test toolchain.

Another good aspect of this is that you'll get powershell completion for the primary build tasks, even if the underlying tool(s) do not have such support.

I'd add a requirement/specification that build.ps1 should only be a facade and not itself contain any build logic, to ensure build logic isn't spread around all over the place.

Longer term: keeping it simple should make it possible to easily generate these build files as part of Plaster/Yeoman templates, or create a library resource of them for people to just quickly select the correct one for their preferred build system.

[TylerLeonhardt]

Here's the build.ps1 PR for PowerShellEditorServices:
PowerShell/PowerShellEditorServices#623

[TylerLeonhardt]

And here's the PR for vscode-powershell:
#1199