Fix chrome/selenium tests #25330

JunTaoLuo · 2020-08-27T22:11:20Z

Fixes #25322

pranavkm

#yolo

@javiercn

dougbu · 2020-08-27T22:15:49Z

🤞

dougbu · 2020-08-27T22:18:07Z

Just noticed the base branch. This is test-only and, if possible, we should have the Selenium tests running for 5.0

JunTaoLuo · 2020-08-27T22:30:15Z

I was planning on trying things out in master before back-porting the fixes to release/5.0.

captainsafia · 2020-08-27T22:43:59Z

eng/Versions.props

    <SeleniumWebDriverMicrosoftDriverPackageVersion>17.17134.0</SeleniumWebDriverMicrosoftDriverPackageVersion>
-    <SeleniumWebDriverChromeDriverPackageVersion>2.43.0</SeleniumWebDriverChromeDriverPackageVersion>
-    <SeleniumWebDriverPackageVersion>3.12.1</SeleniumWebDriverPackageVersion>
+    <SeleniumWebDriverChromeDriverPackageVersion>85.0.4183.8300</SeleniumWebDriverChromeDriverPackageVersion>


This is a pretty big jump. Did something change with the way the package is versioned?

Yup, they moved to use versions matching Chrome versions. There as a jump from v.2.46.0 to v.71.0.3578.137 a while back.

In addition, we were almost 2 years out-of-date in using v2.43.0.

I see. That makes sense.

How does this mesh with the fact that we don't control what version of Chrome is downloaded with the installer (AFAIK)?

It should mean we can be proactive. Whenever a dev notices their Chrome installation wants to be restarted, they can check whether it's a new major version and update this particular package version. Or, @sebastienros could add a reminder to check the Selenium versions in his ops rotation invites every few (10?) rotations.

I could, I am afraid I will forget ;) I would suggest to update the operations book, and let's say "if your rotation happens on the first Tuesday of the month then you should ..."

Mondays are frequently skipped due to holidays

Good idea. May also need something in https://github.com/dotnet/aspnetcore-internal/blob/master/.github/ISSUE_TEMPLATE/Build_OPS_handoff.md @JunTaoLuo what do you think❔

dougbu · 2020-08-27T23:24:35Z

I was planning on trying things out in master before back-porting the fixes to release/5.0.

Makes sense.

Looks like you have at least one breaking change to handle. But, it's not that bad considering how far out of date we were.

Pilchie · 2020-08-27T23:28:19Z

👀

dougbu · 2020-08-28T00:09:16Z

@JunTaoLuo I suggest copying #25323 into the 'release/3.1' branch if possible (@Pilchie and @wtgodbe is that branch open for test-only changes at this point❔). Eventually we'll want this fix there too because it also uses Selenium and our InstallGoogleChrome.ps1 script. E.g. some of the errors in my attempted validation of #25217 mention cannot find Chrome binary.

@captainsafia other validation problems w/ #25217 require your #25324 fix i.e. everyone will need to include your commit in their 3.1 PRs or validations are just busted. Sooner it can go in (too), the better. But, again, only if branch is open.

SteveSandersonMS

This looks great. Thanks for figuring this out, @JunTaoLuo!

wtgodbe · 2020-08-28T15:04:39Z

(@wtgodbe is that branch open for test-only changes at this point❔)

3.1 isn't open for anything until we ship 3.1.8

JunTaoLuo · 2020-08-28T21:27:35Z

Sounds like there's no rush to unblock 3.1 right now. I'll just continue with making fixes here and cherry-pick them to release/5.0 and 3.1 when it's done.

JunTaoLuo · 2020-08-29T01:52:17Z

I've updated the APIs but I'll work with @ajaybhargavb's to test it on Monday.

dougbu · 2020-08-30T19:07:47Z

@JunTaoLuo how far are we from having a commit we can cherry-pick into 3.1 PRs to see if they're ready to go❔

BrennanConroy · 2020-09-01T15:11:59Z

This PR is failing the exact same way as before, unable to launch chrome and hanging.

ajaybhargavb · 2020-09-01T19:15:46Z

This PR is failing the exact same way as before, unable to launch chrome and hanging.

Looks like it is still trying to use the old chrome driver (2.43.xxx) instead of the new 85.0.xxx. I wonder where it is picking that from. cc @JunTaoLuo

 OpenQA.Selenium.WebDriverException : unknown error: cannot find Chrome binary\n  (Driver info: chromedriver=2.43.600210 (68dcf5eebde37173d4027fa8635e332711d2874a),platform=Windows NT 10.0.17763 x86_64)

ajaybhargavb · 2020-09-01T23:23:04Z

@JunTaoLuo's latest commit fixed the Chrome not being found issue but it is now throwing a null-ref,

System.NullReferenceException : Object reference not set to an instance of an object.
   at OpenQA.Selenium.Remote.HttpCommandExecutor.HttpRequestInfo..ctor(Uri serverUri, Command commandToExecute, CommandInfo commandInfo)
   at OpenQA.Selenium.Remote.HttpCommandExecutor.Execute(Command commandToExecute)
   at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.Remote.RemoteLogs.GetLog(String logKind)
   at OpenQA.Selenium.WebDriverExtensions.GetBrowserLogs(IWebDriver driver, LogLevel level) in /_/src/Shared/E2ETesting/WebDriverExtensions.cs:line 20
   at Microsoft.AspNetCore.E2ETesting.WaitAssert.WaitAssertCore[TResult](IWebDriver driver, Func`1 assertion, TimeSpan timeout) in /_/src/Shared/E2ETesting/WaitAssert.cs:line 120
   at Microsoft.AspNetCore.E2ETesting.WaitAssert.Exists(IWebDriver driver, By finder, TimeSpan timeout) in /_/src/Shared/E2ETesting/WaitAssert.cs:line 64
   at Microsoft.AspNetCore.E2ETesting.WaitAssert.Exists(IWebDriver driver, By finder) in /_/src/Shared/E2ETesting/WaitAssert.cs:line 51
   at Templates.Test.BlazorWasmTemplateTest.TestBasicNavigation(String appName, Boolean usesAuth, Boolean skipFetchData) in /_/src/ProjectTemplates/BlazorTemplates.Tests/BlazorWasmTemplateTest.cs:line 525
   at Templates.Test.BlazorWasmTemplateTest.BlazorWasmStandaloneTemplate_Works() in /_/src/ProjectTemplates/BlazorTemplates.Tests/BlazorWasmTemplateTest.cs:line 73

Also only a subset of the tests failed. @captainsafia @pranavkm could these be test specific issues?

JunTaoLuo · 2020-09-02T07:39:35Z

Turns out that getting browser logs is broken: SeleniumHQ/selenium#7342. Seems like the issue is fixed in 4.0.0-alpha3 so I'll try that version and see what happens.

JunTaoLuo · 2020-09-02T17:12:06Z

Okay this got even further but now I'm seeing actual test failures in blazor templates and component e2e tests.

JunTaoLuo · 2020-09-02T21:55:05Z

Hey @javiercn, you've been recommended to take a look at the test failures. Can you investigate and see what's causing the test failures. Note that this is blocking us from turning the E2ETests on in 5.0-rc1

javiercn · 2020-09-02T21:59:01Z

src/Shared/E2ETesting/BrowserFixture.cs

+                    break;
+                default:
+                    throw new InvalidOperationException($"Browser name {sauce.BrowserName} not recognized");
+            }


Why do we need this? We ended up not going this route @ajaybhargavb isn't that the case?

We only need chrome I believe

We only need Chrome for local and CI tests. But we use other browsers when running in SauceLabs. We need to pass in the correct options to SauceLabs in that case. DesiredCapabilities used to be enough but that is now obsoleted in the newer version.

@ajaybhargavb but do we actually run in SauceLabs? I thought we discarded that idea. That's why I'm asking.

We still want to. What we discarded was the idea to do real device testing on SauceLabs.

JunTaoLuo · 2020-09-02T22:24:54Z

~~FYI, let's backport this fix to 5.0-rc2 only once we are done fixing it in master, turns out it's too late for RC1.~~

Actually I'll defer to @mkArtakMSFT in terms of assigning an investigator and which release this goes into.

Current status:

I'm seeing several Component.E2ETests and BlazorWasm template test failures with different Assert errors. It's unclear to me if these are caused by issues with the tests, issues with the product or something else.

For example BlazorWasmStandaloneTemplate_Works fails without any error messages in its console log whereas BlazorWasmHostedPwaTemplate_Works fails at the same TestBasicNavigation step but there are messages in the console log saying An unknown error occurred while processing the certificate. At this point I think someone familiar with what these browser based tests are trying to do should investigate since I can't find a common pattern in the failures that point to an issue with the infrastructure.

dougbu · 2020-09-02T22:53:58Z

FYI, let's backport this fix to 5.0-rc2 only once we are done fixing it in master, turns out it's too late for RC1.

Agreed but we also want to fix Selenium tests in release/3.1. The same-site fixes in @BrennanConroy and @captainsafia's #25324 aren't enough. Would rather not back-port #25323 into a servicing branch due to the loss of code coverage.

halter73 · 2020-09-09T01:03:37Z

@mkArtakMSFT Do we still need to assign an investigator for this?

dougbu · 2020-09-09T02:17:53Z

Do we still need to assign an investigator for this?

We urgently need an investigator because we're seeing the same issues in release/3.1 and that has a short timeline. @javiercn do you have time to help w/ the WASM template tests❔

JunTaoLuo · 2020-09-09T02:20:09Z

I'll take a look tonight and see if I can find anything or at least narrow down the issue we're seeing. Also I'm curious if #25689 might contain some of the fixes that'll get this working.

dougbu · 2020-09-09T02:26:56Z

I'm curious if #25689 might contain some of the fixes that'll get this working.

@SteveSandersonMS❔❔

JunTaoLuo · 2020-09-09T08:22:21Z

Seems like those didn't affect the test failures seen in this PR. I'll try to pin down the cause a little more clearly tomorrow.

SteveSandersonMS · 2020-09-09T13:19:39Z

I'm curious if #25689 might contain some of the fixes that'll get this working.

@SteveSandersonMS❔❔

I hope so! The changes in #25689 are enough to make all the Components E2E tests pass for me locally. I don't know about the templates tests though - that's a separate thing.

JunTaoLuo · 2020-09-09T17:39:23Z

It didn't seem to fix the Components.E2ETests actually, I'm going to look into why today.

…25323)" This reverts commit 332f150.

JunTaoLuo · 2020-09-10T17:56:35Z

Turns out the browser logs are not working. I might just disable the specific tests that checks logs to unblock the rest of the tests enabled by this PR while I investigate.

JunTaoLuo · 2020-09-11T23:25:20Z

@dougbu @mkArtakMSFT I suppose we want these fixes in rc2 as well?

Pilchie · 2020-09-11T23:26:41Z

@dougbu Doug Bunting FTE @mkArtakMSFT Artak Mkrtchyan FTE I suppose we want these fixes in rc2 as well?

Yes, please!

dougbu · 2020-09-11T23:37:44Z

And, release/3.1 if possible ❕

* Revert "Disable failing/hanging tests due to Chrome/Selenium issue (#25323)" This reverts commit 332f150. * Update Selenium to latest * Update API * Try specifying a version * Update Selenium to 4.0.0-beta5 * Disable browser log tests * Fix components e2e tests and disable blazor standalone template test

* Fix chrome/selenium tests (#25330) * Revert "Disable failing/hanging tests due to Chrome/Selenium issue (#25323)" This reverts commit 332f150. * Update Selenium to latest * Update API * Try specifying a version * Update Selenium to 4.0.0-beta5 * Disable browser log tests * Fix components e2e tests and disable blazor standalone template test * Disable tests using browser log * Disable template test * Avoid using .NET formatted strings in tests * Annotate BasicTestApp suggesting that it needs the all globalization data * Culture specific formatting relies on the ICU data carried by the OS. This causes issues in our tests if WebAssembly carries a different set than the OS. Instead updating these tests to use hardcoded strings. * Additionally fixing an issue where some projects in the solution were using tasks from the .dotnet SDK rather than the local copy of the SDK. This was causing issues building locally. Co-authored-by: Pranav K <[email protected]>

JunTaoLuo requested review from dougbu, SteveSandersonMS and a team as code owners August 27, 2020 22:11

JunTaoLuo added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Aug 27, 2020

pranavkm approved these changes Aug 27, 2020

View reviewed changes

captainsafia reviewed Aug 27, 2020

View reviewed changes

SteveSandersonMS approved these changes Aug 28, 2020

View reviewed changes

JunTaoLuo force-pushed the johluo/reenable-chrome-tests branch from b34bd79 to 150fd53 Compare September 1, 2020 21:54

javiercn reviewed Sep 2, 2020

View reviewed changes

JunTaoLuo assigned mkArtakMSFT Sep 2, 2020

dougbu mentioned this pull request Sep 8, 2020

Set SameSiteMode for cookies in authentication tests (#25281) #25324

Merged

John Luo added 5 commits September 10, 2020 10:01

Revert "Disable failing/hanging tests due to Chrome/Selenium issue (#…

dd42fdc

…25323)" This reverts commit 332f150.

Update Selenium to latest

3e0f62e

Update API

e0cb381

Try specifying a version

7d71792

Update Selenium to 4.0.0-beta5

a24de39

JunTaoLuo force-pushed the johluo/reenable-chrome-tests branch from d50f283 to a24de39 Compare September 10, 2020 17:02

JunTaoLuo mentioned this pull request Sep 11, 2020

Re-enable Components tests that verifies browser logs #25803

Closed

John Luo added 2 commits September 10, 2020 21:25

Disable browser log tests

24e98d7

Fix components e2e tests and disable blazor standalone template test

b5b63bd

JunTaoLuo merged commit 91c8c9b into master Sep 11, 2020

JunTaoLuo deleted the johluo/reenable-chrome-tests branch September 11, 2020 23:24

JunTaoLuo mentioned this pull request Sep 12, 2020

[5.0.0-rc2] Backport Fix chrome/selenium tests (#25330) #25840

Merged

Fix chrome/selenium tests #25330

Fix chrome/selenium tests #25330

Uh oh!

Conversation

JunTaoLuo commented Aug 27, 2020

Uh oh!

pranavkm left a comment

Choose a reason for hiding this comment

Uh oh!

dougbu commented Aug 27, 2020

Uh oh!

dougbu commented Aug 27, 2020

Uh oh!

JunTaoLuo commented Aug 27, 2020

Uh oh!

captainsafia Aug 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dougbu commented Aug 27, 2020

Uh oh!

Pilchie commented Aug 27, 2020

Uh oh!

dougbu commented Aug 28, 2020

Uh oh!

SteveSandersonMS left a comment

Choose a reason for hiding this comment

Uh oh!

wtgodbe commented Aug 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JunTaoLuo commented Aug 28, 2020

Uh oh!

JunTaoLuo commented Aug 29, 2020

Uh oh!

dougbu commented Aug 30, 2020

Uh oh!

BrennanConroy commented Sep 1, 2020

Uh oh!

ajaybhargavb commented Sep 1, 2020

Uh oh!

ajaybhargavb commented Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JunTaoLuo commented Sep 2, 2020

Uh oh!

JunTaoLuo commented Sep 2, 2020

Uh oh!

JunTaoLuo commented Sep 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JunTaoLuo commented Sep 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dougbu commented Sep 2, 2020

Uh oh!

halter73 commented Sep 9, 2020

Uh oh!

dougbu commented Sep 9, 2020

Uh oh!

JunTaoLuo commented Sep 9, 2020

Uh oh!

captainsafia Aug 27, 2020 •

edited

Loading

wtgodbe commented Aug 28, 2020 •

edited

Loading

ajaybhargavb commented Sep 1, 2020 •

edited

Loading

JunTaoLuo commented Sep 2, 2020 •

edited

Loading

JunTaoLuo commented Sep 2, 2020 •

edited

Loading