Skip to content

Do less copying / zipping for Helix #40134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 15, 2022

Conversation

dougbu
Copy link
Contributor

@dougbu dougbu commented Feb 11, 2022

  • use .dotnet/ folders instead of App.Ref/App.Runtime packages for dotnet-cli layouts
    • no need to unzip files into another layout
    • fail fast or emit message when layouts don't exist
  • set $(TestDependsOnAspNetPackages) to false by default
    • no need to copy all packages into publish/ folders of most test projects
      • upload very few times per queue (2 today, maybe 4 in the future), not for every work item
      • reduce the size of most work item payloads, saving bandwidth and time
      • reduction should not cause problems, even around major version and TFM changes (see .dotnet/ folders)
    • add $(TestDependsOnAspNetAppPackages) for the App.UnitTests case; just 2 packages needed there
  • get App.UnitTests working in local builds
    • grab RuntimeList.xml file from App.Runtime's obj/ folder
    • find needed packages whether running locally or on Helix agents; should always exist

nits:

  • remove test infrastructure we don't need anymore
    • use one property for shared Fx and targeting pack versions; always the same
    • remove $env:ASPNET_RUNTIME_PATH
  • rename silly ...ListListsContains... tests
  • remove extra $(TestDependsOnAspNetXyz) settings to their new default values
  • add more comments

@dougbu dougbu added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Feb 11, 2022
@dougbu dougbu requested review from HaoK and a team February 11, 2022 01:58
@dougbu
Copy link
Contributor Author

dougbu commented Feb 11, 2022

/fyi

  • Not currently separating Publish and doing that in outer builds. The separation would add needless complexity
    unless up-to-date checks are expensive for inner (per-queue) builds after the first one (despite matching global
    properties).
  • Not moving package uploads into a Helix correlation payload because the requirements are per-work item (test project).

<PropertyGroup>
<TargetFramework>$(DefaultNetCoreTargetFramework)</TargetFramework>
<RootNamespace>Microsoft.AspNetCore</RootNamespace>
<VerifyAncmBinary Condition="'$(TargetOsName)' == 'win' AND '$(TargetArchitecture)' != 'arm'">true</VerifyAncmBinary>
<TestDependsOnAspNetPackages>true</TestDependsOnAspNetPackages>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to consider is to just stop running these special app tests on helix if these are now the only ones that require all this custom packaging, and just run these on the build agents once (say the windows test job). Might save us a bunch of grief in the long term maintaining this stuff if we no longer need it for any of the other normal tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to consider is to just stop running these special app tests on helix if these are now the only ones that require all this custom packaging, and just run these on the build agents once (say the windows test job). Might save us a bunch of grief in the long term maintaining this stuff if we no longer need it for any of the other normal tests

There's now very little infrastructure for this and any move away from Helix is a regression. In addition, the project template tests need more stuff than these tests.

Copy link
Member

@HaoK HaoK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me, less magic and less setup that we have to do for almost all the tests!

@@ -79,9 +78,12 @@
<HelixCorrelationPayload Include="$(HelixTestConfigurationFilePath)" AsArchive="false" />
</ItemGroup>

<Target Name="IncludeAspNetRuntime" BeforeTargets="Gather"
Condition="'$(DoNotRequireSharedFxHelix)' != 'true' OR
EXISTS('$(RepoRoot)artifacts\packages\$(Configuration)\Shipping\Microsoft.AspNetCore.App.Runtime.$(TargetRuntimeIdentifier).$(SharedFxVersion).nupkg')">
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BrennanConroy I realize this line didn't work despite my insistence on including it in your $(DoNotRequireSharedFxHelix) work. $(SharedFxVersion) was never set which meant RunHelix.ps1 couldn't successfully run App.Unitest or the project template tests ☹️

@dougbu dougbu force-pushed the dougbu/app.unittest.fixes branch 2 times, most recently from 359dbcc to 3d76c90 Compare February 13, 2022 01:44
- use .dotnet/ folders instead of App.Ref/App.Runtime packages for dotnet-cli layouts
  - no need to unzip files into another layout
  - fail fast or emit message when layouts don't exist
- set `$(TestDependsOnAspNetPackages)` to `false` by default
  - no need to copy all packages into publish/ folders of most test projects
    - upload very few times per queue (2 today, maybe 4 in the future), not for _every_ work item
    - reduce the size of most work item payloads, saving bandwidth and time
    - reduction should not cause problems, even around major version and TFM changes (see .dotnet/ folders)
  - add `$(TestDependsOnAspNetAppPackages)` for the App.UnitTests case; just 2 packages needed there
- get App.UnitTests working in local builds
  - grab RuntimeList.xml file from App.Runtime's obj/ folder
  - find needed packages whether running locally or on Helix agents; should always exist

nits:
- remove test infrastructure we don't need anymore
  - use one property for shared Fx and targeting pack versions; always the same
  - remove `$env:ASPNET_RUNTIME_PATH`
- rename silly ...ListListsContains... tests
- remove extra `$(TestDependsOnAspNetXyz)` settings to their new default values
- add more comments
- use `$(ArtifactsShippingPackagesDir)` in Helix.targets
- use `[MSBuild]::NormalizeDirectory(...)` and `$([System.IO.Path]::Combine(...)` in helix.proj
- explain a bit more about missing .dotnet/ layouts
@dougbu dougbu force-pushed the dougbu/app.unittest.fixes branch from 3d76c90 to 3032d6c Compare February 14, 2022 18:39
@dougbu dougbu merged commit 4dfc0a7 into dotnet:main Feb 15, 2022
@dougbu dougbu deleted the dougbu/app.unittest.fixes branch February 15, 2022 06:03
@ghost ghost added this to the 7.0-preview2 milestone Feb 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants