crossgen2 test failures in ArrayLdtokenTests

[GrabYourPitchforks]

We're seeing some transient test failures in crossgen2. See for example:

• https://github.com/dotnet/runtime/pull/50392/checks
• https://github.com/dotnet/runtime/runs/2240696791

Code generation failed for method '[crossgen2smoke]Program.ArrayLdtokenTests()'

  Starting:    readytorun.crossgen2.XUnitWrapper
    readytorun/crossgen2/crossgen2smoke/crossgen2smoke.sh [FAIL]
      usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
             cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ... target_directory
      rm: /private/tmp/helix/working/B733099D/w/A0EB08E7/e/readytorun/crossgen2/crossgen2smoke/composite-r2r.dll.rsp: No such file or directory
      Unhandled exception. System.AggregateException: One or more errors occurred. (Code generation failed for method '[crossgen2smoke]Program.ArrayLdtokenTests()')
       ---> ILCompiler.CodeGenerationFailedException: Code generation failed for method '[crossgen2smoke]Program.ArrayLdtokenTests()'
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at ILCompiler.ReadyToRunCompilationModuleGroupBase.VersionsWithType(TypeDesc typeDesc)
         at ILCompiler.DependencyAnalysis.ReadyToRun.ModuleTokenResolver.AddModuleTokenForType(TypeDesc type, ModuleToken token)
         at Internal.JitInterface.CorInfoImpl.resolveToken(CORINFO_RESOLVED_TOKEN& pResolvedToken)
         at Internal.JitInterface.CorInfoImpl._resolveToken(IntPtr thisHandle, IntPtr* ppException, CORINFO_RESOLVED_TOKEN* pResolvedToken)
         --- End of inner exception stack trace ---
         at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(IMethodNode methodCodeNodeNeedingCode, MethodIL methodIL)
         at Internal.JitInterface.CorInfoImpl.CompileMethod(MethodWithGCInfo methodCodeNodeNeedingCode, Logger logger)
         at ILCompiler.ReadyToRunCodegenCompilation.<ComputeDependencyNodeDependencies>b__36_0(DependencyNodeCore`1 dependency)
         at System.Threading.Tasks.Parallel.<>c__DisplayClass33_0`2.<ForEachWorker>b__0(Int32 i)
         at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
         at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
         at System.Threading.Tasks.TaskReplicator.Replica.Execute()
         --- End of inner exception stack trace ---
         at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
         at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
         at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
         at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IList`1 list, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
         at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
         at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body)
         at ILCompiler.ReadyToRunCodegenCompilation.ComputeDependencyNodeDependencies(List`1 obj)
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes()
         at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile)
         at ILCompiler.Program.Run(String[] args)
         at ILCompiler.Program.Main(String[] args)

[VincentBu]

Failed again in runtime 20210331.87 and runtime-coreclr outerloop 20210331.4

Failed test:

CoreCLR Linux x64 Checked no_tiered_compilation @ Ubuntu.1804.Amd64.Open
 -readytorun/crossgen2/crossgen2smoke/crossgen2smoke.sh

Error message:

cp: target '/home/helixbot/work/BCD20A13/p/lib*.dylib' is not a directory
rm: cannot remove '/home/helixbot/work/BCD20A13/w/A38B0927/e/readytorun/crossgen2/crossgen2smoke/composite-r2r.dll.rsp': No such file or directory
Unhandled exception. System.AggregateException: One or more errors occurred. (Code generation failed for method '[crossgen2smoke]Program.ArrayLdtokenTests()')
 ---> ILCompiler.CodeGenerationFailedException: Code generation failed for method '[crossgen2smoke]Program.ArrayLdtokenTests()'
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
 at ILCompiler.ReadyToRunCompilationModuleGroupBase.VersionsWithType(TypeDesc typeDesc)
 at ILCompiler.DependencyAnalysis.ReadyToRun.ModuleTokenResolver.AddModuleTokenForType(TypeDesc type, ModuleToken token)
 at Internal.JitInterface.CorInfoImpl.resolveToken(CORINFO_RESOLVED_TOKEN& pResolvedToken)
 at Internal.JitInterface.CorInfoImpl._resolveToken(IntPtr thisHandle, IntPtr* ppException, CORINFO_RESOLVED_TOKEN* pResolvedToken)
 --- End of inner exception stack trace ---
 at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(IMethodNode methodCodeNodeNeedingCode, MethodIL methodIL)
 at Internal.JitInterface.CorInfoImpl.CompileMethod(MethodWithGCInfo methodCodeNodeNeedingCode, Logger logger)
 at ILCompiler.ReadyToRunCodegenCompilation.<ComputeDependencyNodeDependencies>b__36_0(DependencyNodeCore1 dependency)
 at System.Threading.Tasks.Parallel.<>c__DisplayClass33_02.<ForEachWorker>b__0(Int32 i)
 at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location ---
 at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
 at System.Threading.Tasks.TaskReplicator.Replica1.ExecuteAction(Boolean& yieldedBeforeCompletion)
 at System.Threading.Tasks.TaskReplicator.Replica.Execute()
 --- End of inner exception stack trace ---
 at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction1 action, ParallelOptions options, Boolean stopOnFirstFailure)
 at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally)
--- End of stack trace from previous location ---
 at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
 at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally)
 at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IList1 list, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally)
 at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally)
 at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body)
 at ILCompiler.ReadyToRunCodegenCompilation.ComputeDependencyNodeDependencies(List1 obj)
 at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ComputeMarkedNodes()
 at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile)
 at ILCompiler.Program.Run(String[] args)
 at ILCo

Stack trace
   at readytorun_crossgen2._crossgen2smoke_crossgen2smoke_._crossgen2smoke_crossgen2smoke_sh()

[jkotas]

@trylek It looks like a failure introduce by #48557. Could you please take a look ? It is causing many PRs to fail.

[jkotas]

Fixed by #50539