Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e2e: Fix broken tests due to Customer Home AB test rollout. #39653

Merged
merged 3 commits into from
Feb 25, 2020

Conversation

kwight
Copy link
Contributor

@kwight kwight commented Feb 25, 2020

End-to-end tests that rely on landing on stats page are breaking, due to a rollout by AB test of the new Customer Home for all users. Instead of older users landing on stats, they are now sent to Customer Home like newer users, breaking all e2e sequences checking for a stats page before continuing.

This PR removes the assertion that checks that a user lands in either Stats or Customer Home after clicking on My Sites with a more generic check that verifies that the My Sites layout is loaded.

@kwight kwight requested a review from a team as a code owner February 25, 2020 01:04
@matticbot
Copy link
Contributor

@kwight kwight self-assigned this Feb 25, 2020
@kwight kwight added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. [Status] Needs e2e Testing [Status] Needs Jetpack e2e Testing Runs the full suite of Jetpack e2e automated tests against this PR using calypso.live [Status] Needs e2e Testing horizon Run e2e tests against horizon env [Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot labels Feb 25, 2020
@matticbot
Copy link
Contributor

This PR does not affect the size of JS and CSS bundles shipped to the user's browser.

Generated by performance advisor bot at iscalypsofastyet.com.

@mmtr mmtr requested a review from a team February 25, 2020 10:20
@mmtr
Copy link
Member

mmtr commented Feb 25, 2020

There are some failing builds but I think they are unrelated to these changes:

  • wp-e2e-test-full-jetpack: tests passed but CircleCI failed to report
  • wp-e2e-tests-horizon-*: all tests passed except ones checking signup which I believe cannot be tested on Horizon.
  • test-e2e-canary-ie: all tests passed except one checking Gutenberg. Seems the test cannot dismiss the Welcome to the Block Editor modal (maybe caused by Block Editor: Welcome guide cannot be dismissed with Do Not Track enabled. #39641?).

@kwight kwight merged commit 145d16f into master Feb 25, 2020
@kwight kwight deleted the fix/e2e-home-abtest branch February 25, 2020 21:17
@matticbot matticbot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Feb 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Status] Needs e2e Testing Gutenberg Edge Run e2e tests with a special site that is running the latest Gutenberg plugin snapshot [Status] Needs e2e Testing horizon Run e2e tests against horizon env [Status] Needs Jetpack e2e Testing Runs the full suite of Jetpack e2e automated tests against this PR using calypso.live
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants