Set up CI with Azure Pipelines

[azure-pipelines]

Massive summary:

• Changed \ to / in build.proj and Repo.props to support cross-plat
  • Backslashes work fine for anything running through MSBuild, but Exec targets would create invalid paths for external commands
• build.proj changes
  • Made Build target primarily only build the product
    • Separated out validation to make them encapsulated.
  • Made Test target primarily do testing
    • It does build the pre-built Azure.PowerShell.sln if need be. In ADO, this only builds the .Test.csproj files. The modules are downloaded via the Build job.
  • Changed Azure.PowerShell.sln to be built into the artifacts folder
    • This is consistent with our build process producing artifacts into the artifacts folder.
    • This allows us to download (from artifacts) the .sln for the Test job
  • Made Azure.PowerShell.sln project additions per dotnet CLI call
    • There was a substantial issue with Windows were the command length was too long, even if using relative paths. This change guarantees it will never issue a command that is too long. It does, however, make the build slower by about 30 seconds. But, we need it for the ADO Windows workers.
  • Changed Build target to use dotnet build instead of dotnet publish
    • publish makes sense if you are running the .dlls cross-plat. However, we don't run our .dlls directly (except for StaticAnalysis). PowerShell Core runs our .dlls, so publishing is not necessary.
    • This allows us to build the modules and .Test.csproj projects in the Build target. Thus, Build target actually verifies that everything builds, as it should.
    • I had to add an extra step to dotnet publish the StaticAnalysis project to allow it to run cross-plat correctly. Otherwise, it won't work.
    • This change removed significant copying/removing of files. Thus, speeding up the build process.
  • Removed ContinueOnError from Exec tasks
    • ContinueOnError doesn't exist for Exec tasks. The error from the command ran is always returned unless it is suppressed via IgnoreErrorCode
    • https://docs.microsoft.com/en-us/visualstudio/msbuild/exec-task?view=vs-2017
  • Removed duplicate NewOutputTypeIndex and NewHelpIndex calls
  • Made CleanupBuild.ps1 ran only when CodeSign is true
    • This is because we don't need to clean up on normal build. Cleaning up the DLLs only makes sense when publishing the modules. For build/test, this is not necessary.
  • Moved BinScope and PoliCheck execution to OnPremChecks target
    • This allows us to easily separate this validation.
    • This simply needs to be added to the Jenkins build targets
    • Also, Jenkins will only need to do /t:Clean,Build,OnPremChecks now. Other machines (local or ADO), cannot do these checks anyway.
  • Created GenerateHelp target
    • This target specifically does help generation (-Help.xml files) and runs StaticAnalysis with a new help-only flag (--help-only or -h).
    • This is ran as a separate GenerateHelp job on ADO in parallel with Test and StaticAnalysis
  • Made Full target run Clean;Build;OnPremChecks;GenerateHelp;StaticAnalysis;Test
    • These still contain the conditonals required to run the tasks. Meaning, if you use SkipHelp, it won't run GenerateHelp, etc.
  • Removed DependencyAnalysis and DependencyAnalysisNetcore targets. Instead, created StaticAnalysis target.
    • The first target was for net452 which we don't build anymore.
    • The second was doing extra work (rebuilding StaticAnalysis)
    • The new StaticAnalysis target runs StaticAnalysis with -s (skip help) always, since GenerateHelp does help analysis
    • This new target also does CheckAssemblies.ps1 which is additional static analysis outside of the project
  • Test target changes
    • The Test target was running BinScope and PoliCheck (for some reason). I removed this since it was previously handled in the Build target, but is now handled in the OnPremChecks target.
    • The Test target no longer re-created Azure.PowerShell.sln. Instead, it uses the one created by Build target. This means that Test cannot run indepedent of Build (which makes sense).
  • Removed duplicate UpdateModules.ps1 calls in the build process (called once in Build and once in Publish)
  • Made all calls to PowerShell use PowerShell Core
  • Moved CodeSign.targets back into build.proj as they shared properties/logic, and depended on each other
• Moved after.Azure.PowerShell.sln.targets to build folder
  • This is copied from this folder during the build into the artifacts folder, since Azure.PowerShell.sln is built into there
• ADO pipeline additions
  • Created 3 pipeline files at repo root for each OS
  • Created pipeline templates into the build folder. These are accessed by the pipelines.
  • The pipelines are the same for each OS except Windows has a flag for installing PowerShell Core. This pipeline step only takes ~1 minute during the build and is required for Windows. Other OSs skip this step via a condition in install-pwsh-steps.yml.
• Updated developer guide with additional GenerateHelp and StaticAnalysis target information
• Added Mono.Posix.NETStandard.dll to Accounts
  • This file is a dependency in Microsoft.WindowsAzure.Storage.DataMovement.dll and should have been failing static analysis previously. However, we had issues running static analysis properly on Travis. Now that it is running properly on linux and macOS via ADO, this issue was found.
  • Additionally, this package has platform-dependent builds in it. Logically, we would want to actually use the correct platform DLL for Accounts. However, we can only ship a single DLL. I've used the base netstandard2.0 DLL from that nupkg and have had no issues.
• 6 tests were skipped
  • 4 are failing for linux and macOS
  • 2 are failing for macOS only
• Fixed an issue in Kusto resources
  • It looks like there must have been a copy/paste or merge issue at some point.
  • I appropriately named the duplicated resource key to match the pattern they used.
• Removed '.\Microsoft.WindowsAzure.Storage.dll' from Az.Storage module's RequiredAssemblies list
  • This shouldn't be there as this DLL is part of Accounts. This could have potentially caused issues.
  • I also needed to add Mono.Posix.NETStandard.dll to Storage as it uses Microsoft.WindowsAzure.Storage.DataMovement.dll
• Fixed incorrect path to Repo.props in the BatchModelGenerator tool
• Output/artifacts changes
  • To keep things easy for ADO, I've made some minor adjustments to outputs to the artifacts folder
  • The StaticAnalysis tool is built into artifacts/StaticAnalysis
  • The results from StaticAnalysis (both via StaticAnalysis and GenerateHelp targets) go into artifacts/StaticAnalysisResults
  • The results from Test are now in artifacts/TestResults
  • I'd like to keep this consistency where 'results' of things are in a suffix Results folder.
• Fixed bug with CheckPoliCheck.ps1 not checking only the built modules (it was checking everything in artifacts)
• GenerateHelp.ps1 changes
  • Adjustments to GenerateHelp.ps1 to properly access the Exceptions folder in tools instead of the one output by the StaticAnalysis build
  • This script was duplicating validation. It was checking the entire repo instead of the artifacts. This was fixed.
  • Removed the temporary ValidateHelpIssues.csv from Exceptions if validation suceeded
  • Added NoHelpIssues file to artifacts/StaticAnalysisResults on success
    • This allows for ADO artifacts aggregation on successful builds. Linux/macOS cannot have empty folders.
• Updated DependencyAnalyzer to consider System.Management.Automation as a framework dll and removed System.Configuration.ConfigurationManager
  • System.Configuration.ConfigurationManager is a nupkg that RecoveryServices relies on. So, this cannot be marked as framework since it is in their .psd1.
• StaticAnalysis changes
  • Fixed bug where StaticAnalysis would crash if the reports directory path supplied did not exist
  • Created --help-only flag
  • Fixed issue where StaticAnalysis was incorrectly identifying the assembly's directory. This is needed so it can access the Exceptions folder that is built into that directory.
  • StaticAnalysis build goes to artifacts\StaticAnalysis\$(Configuration). StaticAnalysis publishing goes to artifacts\StaticAnalysis
    • This allows scripts to run the published version, and the built version simply stays at a lower depth.
• No longer made InternalsVisibleTo exposed by signed DLLs
• Removed non-NetStandard/Core versions of StaticAnalysis and Tools.Common projects
• Allowed modules to publish to the same directory as they build to
• Fixed bug in NewOutputTypeIndex.ps1 looking for Az.Profile.psd1 (which doesn't exist)
• Made Version controller build into artifacts\VersionController
• Updated HelpGeneration.psm1 to use friendly verbs so that there is no longer a warning about this in every build

[msftclas]

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

❌ azure-pipelines[bot] sign now
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[adxsdkps]

Can one of the admins verify this patch?

[MiYanni]

@azuresdkci add to whitelist

[markcowl]

failures

[MiYanni]

Check-in: https://azuresdkci.westus2.cloudapp.azure.com/view/PowerShell/job/ado-powershell-temp/7/
Sign: https://azuresdkci.westus2.cloudapp.azure.com/view/PowerShell/job/ado-psnetsign-esrp-temp/15/

Addressed