Investigate flaky blazor templates

[HaoK]

For: #23992

[BrennanConroy]

We should probably re-enable the tests for 5.0

[HaoK]

Okay I'll file a PR with just the revert of the skip tests for 5.0

[HaoK]

Looks like its only disabled in master, since the fix came from 3.1 https://github.com/dotnet/aspnetcore/blob/release/5.0/src/Shared/E2ETesting/E2ETesting.props looks fine so we are good in 5.0

[HaoK]

Yeah this is why i was confused since the tests were running in 5.0 PRs when I was investigating

[HaoK]

@javiercn is it normal to sometimes get empty screenshots from the selenium captures? The Templates.Test.BlazorWasmTemplateTest.BlazorWasmStandaloneTemplate_IndividualAuth_Works png appears to be blank to me :(

[HaoK]

Note these tests seem to all fail together, i.e. from https://dev.azure.com/dnceng/public/_build/results?buildId=849186&view=ms.vss-test-web.build-test-results-tab&runId=27060606&resultId=100004&paneView=debug they all pass for two builds, then all of them fail, which seems like something might be getting borked at the infrastructure level

[BrennanConroy]

Possible issue

dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1]
              Failed to authenticate HTTPS connection.
              System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
               ---> System.ComponentModel.Win32Exception (0x80090327): An unknown error occurred while processing the certificate.
                 --- End of inner exception stack trace ---
                 at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
                 at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)

[HaoK]

Ah that could explain it, @Tratcher any ideas what might cause this cert exception? Or how to dig deeper into this? Its happen fairly reliability, in that I hit it within 3 builds today

[Tratcher]

https://docs.microsoft.com/en-us/windows/win32/secauthn/schannel-error-codes-for-tls-and-ssl-alerts
SEC_E_CERT_UNKNOWN
0x80090327

Which could be any of these:
TLS1_ALERT_BAD_CERTIFICATE
TLS1_ALERT_UNSUPPORTED_CERT
TLS1_ALERT_CERTIFICATE_UNKNOWN

Are these tests relying on the dev cert or are they using a specific test cert?

[HaoK]

These are just the template tests so the dev cert, they just do new template build and run without any extra operations I don't think.

[HaoK]

Could this mean the dev cert wasn't installed properly some % of the time?

[Tratcher]

Seems likely, or it's being messed up by another test. Tests using the dev cert should:

• Check it before starting the test
• Not run in parallel in case they try to fix the cert in parallel

Or, we should make these tests not use the dev cert.

[HaoK]

Do we have any tests that currently check if the dev cert is valid? Aka is there any code I can steal for that?

[Tratcher]

@javiercn ?

[HaoK]

Rebased version continuing in #27948