Update helix installation retry logic

[JunTaoLuo]

Follow up to #29842 to fix partial installation issues.

The problem was that if the process timed out when the SDK was partially installed, upon retry, the dotnet-install script will see the partial installation and consider the SDK already installed and exit with success. This PR changes it so that upon failure (timeout or non-zero exit code) of either SDK or runtime installation, both will be re-installed as part of the retry.

[jkotalik]

Looks like it is looping 5 times for install https://helixri8s23ayyeko0k025g8.blob.core.windows.net/dotnet-aspnetcore-refs-pull-30022-merge-dc8f7ac3f9be488598/Microsoft.AspNetCore.Components.Web.Tests--net6.0/console.b7933ca8.log?sv=2019-07-07&se=2021-05-10T03%3A21%3A02Z&sr=c&sp=rl&sig=h8UXasvVb%2FZBKG39q0wMgf9i8qlHXi1rPPUOpq4Y2ic%3D

[JunTaoLuo]

Because it's trying to install from a location without proper credentials... hmmm

Failed to install dotnet from custom location ****.
WARNING: InstallDotNet . eng\common\tools.ps1; InstallDotNet C:\h\w\BE9709EC\w\C12E0A35\e\sdk1002\x64 
6.0.100-preview.1.21073.4 x64 dotnet $true '****' '' $true failed with 
exit code 1

[jkotalik]

Lot of open api tests failures: https://dev.azure.com/dnceng/public/_build/results?buildId=984687&view=ms.vss-test-web.build-test-results-tab&runId=30847640&resultId=123469&paneView=debug

Also seems to be another failure to run test, but I can't quite tell why: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-aspnetcore-refs-pull-30022-merge-8591e2e205bb4b4195/Microsoft.AspNetCore.Components.Web.Tests--net6.0/console.fb3f49d7.log?sv=2019-07-07&se=2021-03-01T05%3A54%3A50Z&sr=c&sp=rl&sig=u0luYC6dZARykXQ3YWPE3cHo%2BZp%2FpxunIWlAkUuhmzM%3D

[JunTaoLuo]

Weird, I wonder if the OpenAPI tests are just flaky:

Threw error: Adding package NSwag.ApiDescription.Client to /var/folders/f9/g1yw0c115c1_bffkqgrd9v8h0000gy/T/dotnet-tool-tests/cf68c41e2bab4eda8f798b363b63a3bc took longer than 20 seconds.

For the Components.Web test failing to start:

Found .NET SDK, but did not find dotnet.dll at [/private/tmp/helix/working/A5A70973/w/9D7308ED/e/.dotnet32726/sdk/6.0.100-preview.1.21073.4/dotnet.dll]

which is weird because the installation was a success

InstallDotNet /private/tmp/helix/working/A5A70973/w/9D7308ED/e/.dotnet32726 6.0.100-preview.1.21073.4 '' '' true
Downloading 'https://dot.net/v1/dotnet-install.sh'
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: Downloading primary link https://dotnetcli.azureedge.net/dotnet/Sdk/6.0.100-preview.1.21073.4/dotnet-sdk-6.0.100-preview.1.21073.4-osx-x64.tar.gz
dotnet-install: Extracting zip from https://dotnetcli.azureedge.net/dotnet/Sdk/6.0.100-preview.1.21073.4/dotnet-sdk-6.0.100-preview.1.21073.4-osx-x64.tar.gz
dotnet-install: Adding to current process PATH: `/private/tmp/helix/working/A5A70973/w/9D7308ED/e/.dotnet32726`. Note: This change will be visible only when sourcing script.
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://docs.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
dotnet-install: Installation finished successfully.

I'm going to hit retry and see if these issues are reproducible.

[jkotalik]

I'm going to hit retry and see if these issues are reproducible.

The Components.Web makes me think that the issue isn't resolved yet. That still is a similar error to what we were seeing before.

[JunTaoLuo]

Hmm... I'm not sure. There was no retry, both installation scripts completed successfully on the first try...

[JunTaoLuo]

Hmm, this PR should help with the reliability of the tests quite a bit so I'll merge right now, but I'll keep an eye out for the Components.Web types of errors.

[dougbu]

I remember asking about a PR that switched to pr: none somewhere but we shouldn't be running this pipeline for all PRs