Skip to content

[SBRP] Segfault during official build, and when building with low memory #1687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dagood opened this issue Jul 24, 2020 · 3 comments
Closed

[SBRP] Segfault during official build, and when building with low memory #1687

dagood opened this issue Jul 24, 2020 · 3 comments
Labels
area-sbrp Source build reference packages untriaged

Comments

@dagood
Copy link
Member

dagood commented Jul 24, 2020

When I build soruce-build-reference-packages cd5a8c6f with low memory, it fails with this end of log:

...
"/work/src/referencePackages/buildReferencePackages.proj" (default target) (1) ->
(Build target) ->
  /work/src/referencePackages/buildReferencePackages.proj(55,5): error MSB3073: The command "$__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/ref
erencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Ext
ensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649
,CS3001,CS3002,CS3003,CS3009" exited with code 139.

    1305 Warning(s)
    1 Error(s)

There's a segfault in an Exec task. From the binlog (also shows up on console):

Exec
    Assembly = Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
    CommandLineArguments = $__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009
    Parameters
        Command = $__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009
    Microsoft (R) Build Engine version 16.3.2+e481bbf88 for .NET Core
    Copyright (C) Microsoft Corporation. All rights reserved.
    /work/.dotnet/sdk/3.0.101/MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/work/.dotnet/sdk/3.0.101/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/work/.dotnet/sdk/3.0.101/dotnet.dll -maxcpucount -property:PackageOutputPath=/work/artifacts/reference-packages/ /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ -restore -target:pack -verbosity:m /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009 /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj
      Restore completed in 71.42 ms for /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj.
      System.Reflection.Extensions -> /work/artifacts/bin/system.reflection.extensions/4.3.0/ref/netstandard1.0/System.Reflection.Extensions.dll
      Successfully created package '/work/artifacts/reference-packages/System.Reflection.Extensions.4.3.0.nupkg'.
    Warnings
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5048: ...
    /tmp/tmp9059cebb54e44b989a9188feed9b95e1.exec.cmd: line 2: 48710 Segmentation fault      (core dumped) $__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009
    Errors
        /work/src/referencePackages/buildReferencePackages.proj(55,5): error MSB3073: The command "$__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009" exited with code 139. [/work/src/referencePackages/buildReferencePackages.proj]

This same error has been showing up as flakiness in the SBRP official build.

When I run the build locally, I see an error like this in `dmesg | grep -C10 'out of memory':

ory values in pages):
[10633484.918482] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[10633484.918483] [2331245]  1000 2331245     1005      691    40960      121             0 bash
[10633484.918485] [2331280]  1000 2331280   842737    12282   499712     4282             0 dotnet
[10633484.918486] [2331298]  1000 2331298   894790    10661   483328     6906             0 dotnet
[10633484.918487] [2331320]  1000 2331320      596      145    40960       24             0 sh
[10633484.918488] [2331322]  1000 2331322   842819    13750   499712     4119             0 dotnet
[10633484.918489] [2331340]  1000 2331340   932950    23921   606208     1298             0 dotnet
[10633484.918489] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=24768a8d1b80e0c40481a7aeca8cd63844eb99c524c2af1ee236ac3dcca2de8b,mems_allowed=0,oom_memcg=/d
ocker/24768a8d1b80e0c40481a7aeca8cd63844eb99c524c2af1ee236ac3dcca2de8b,task_memcg=/docker/24768a8d1b80e0c40481a7aeca8cd63844eb99c524c2af1ee236ac3dcca2de8b,task=dotnet,pid
=2331340,uid=1000
[10633484.918525] Memory cgroup out of memory: Killed process 2331340 (dotnet) total-vm:3731800kB, anon-rss:43584kB, file-rss:52100kB, shmem-rss:0kB, UID:1000 pgtables:59
2kB oom_score_adj:0
[10633484.921746] oom_reaper: reaped process 2331340 (dotnet), now anon-rss:4kB, file-rss:0kB, shmem-rss:0kB

The inner binlog appears 100% successful, the error appears to be "outside" the inner MSBuild but still "inside" the Exec.

(I don't know if this is the exact OOM kill that caused the error--I ran some other simulations to see what it does and don't know how to differentiate them. The build takes long enough that I didn't want to just rerun it.)

A few things we can try:

  • Remove libraries that we're 100% confident no future builds will use to reduce overall footprint.
  • Investigate situations that might be using more memory than necessary.
    • I wonder if calling so many Execs with inner builds might be causing some particularly bad behavior in the binlog logger. I want to try this out with the 5.0 changes (no more Execs) to see if it still happens there, and play with the limit.
  • Upgrade the distro version.
  • Upgrade the .NET Core version.
  • Disable binlogs. (😢 please, no.)

I haven't tried to actually get a core dump, not sure if it'll actually have anything valuable in this situation. Maybe it would let MSBuild improve its error logging or handle this without a segfault in a future version.

I used --memory 512MB on my docker run command to repro the SBRP error.
@dagood dagood added the area-infra Source-build infrastructure and reporting label Jul 24, 2020
@dagood dagood mentioned this issue Jul 24, 2020
Merged
@dagood
Copy link
Member Author

dagood commented Jul 27, 2020

  • I want to try this out with the 5.0 changes (no more Execs) to see if it still happens there, and play with the limit.

The dev branch starts to fail somewhere between 1.5 to 2 GB. (Without binlogs enabled: default in that branch. Not that I know if this affects it.)

@dseefeld
Copy link
Contributor

Perhaps adding something to log memory usage to see how much memory the build is consuming at different stages to help identify memory requirements for the build.

@dagood dagood mentioned this issue Nov 9, 2020
Closed
@MichaelSimons MichaelSimons added area-sbrp Source build reference packages and removed area-infra Source-build infrastructure and reporting labels Nov 16, 2023
@MichaelSimons
Copy link
Member

[Triage] Closing - We haven't heard of any reports over the past several years of this being an issue.

@MichaelSimons MichaelSimons closed this as not planned Won't fix, can't repro, duplicate, stale Jul 25, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in .NET Source Build Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-sbrp Source build reference packages untriaged
Projects
.NET Source Build
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dseefeld @MichaelSimons @dagood