Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify variations pull requests testing process #39689

Closed
goodov opened this issue Jul 11, 2024 · 7 comments · Fixed by brave/brave-core#24604
Closed

Simplify variations pull requests testing process #39689

goodov opened this issue Jul 11, 2024 · 7 comments · Fixed by brave/brave-core#24604
Assignees
@goodov
Labels
dev-concern griffin OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality QA Pass - Android ARM QA Pass - Android Tab QA Pass - iPad QA Pass - iPhone QA Pass-Linux QA Pass-macOS QA Pass-Win64 QA/Test-All-Platforms QA/Test-Plan-Specified QA/Yes release-notes/exclude
Milestone
1.68.x - Release

Comments

@goodov
Copy link
Member

goodov commented Jul 11, 2024 •
edited
Loading

Description

Add support for --variations-pr=<pull_number> command line flag that will automatically set required flags to fetch the test seed generated as a part of https://github.com/brave/brave-variations pull request.

Related #33654
@goodov goodov added dev-concern OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality labels Jul 11, 2024
@goodov goodov self-assigned this Jul 11, 2024
@goodov goodov mentioned this issue Jul 11, 2024
Merged
24 tasks
@goodov goodov added the griffin label Jul 11, 2024
@brave-builds brave-builds added this to the 1.69.x - Nightly milestone Jul 12, 2024
@brave-builds brave-builds mentioned this issue Jul 12, 2024
Merged
7 tasks
@goodov
Copy link
Member Author

goodov commented Jul 12, 2024 •
edited
Loading

Test cases

Only one desktop platform can be tested here (mac/linux/windows), don't need to test all of them. Android and iOS should be tested both.

By default https://variations.brave.com/seed is used

  1. Launch browser
  2. Open brave://version
  3. Expect 
    --variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
    is present in "Command Line" field.
  4. Expect production variations are listed in "Active Variations" field.

Setting a test seed via --variations-pr command line flag

For the test we will use a test seed from this draft PR: brave/brave-variations#1123
The seed contains a single study BraveAdsAdEventStudy enabled on all platforms.

  1. Launch browser with --variations-pr=1123 parameter (on Android use debug menu to set command line; on iOS set "Variations PR" parameter in Brave-core debug menu).
  2. Restart browser
  3. Wait 10 seconds
  4. Restart browser
  5. Open brave://version
  6. Expect 
    --accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
    is present in "Command Line" field.
  7. Expect "Seed Details > Version" from this comment is present in "Active Variations" field.

Ensure https://variations.brave.com/seed is restored

  1. Remove the parameter from the command line/debug menu.
  2. Restart browser
  3. Wait 10 second
  4. Restart browser
  5. Expect 
    --variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
    is present in "Command Line" field.
  6. Expect these lines are NOT present in "Command Line" field: 
    --accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
  7. Expect production variations are listed in "Active Variations" field.

@goodov goodov mentioned this issue Jul 15, 2024
Closed
7 tasks
@stephendonner stephendonner added QA/Test-Plan-Specified QA/In-Progress Indicates that QA is currently in progress for that particular issue and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Jul 15, 2024
@stephendonner stephendonner mentioned this issue Jul 15, 2024
Closed
6 tasks
@kjozwiak
Copy link
Member

The above requires 1.68.124 or higher for 1.68.x verification 👍 @brave/qa-team please use brave/brave-core#24604 (comment) or brave/brave-core#24604 (comment) as the template. Would recommend using brave/brave-core#24604 (comment) as it has several other cases for Desktop like an upgrade case that should also be checked.

@stephendonner stephendonner added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Jul 18, 2024
@stephendonner
Copy link

stephendonner commented Jul 18, 2024 •
edited
Loading

Verification PASSED using

Brave | 1.68.124 Chromium: 127.0.6533.57 (Official Build) (x86_64)
-- | --
Revision | d048917c488542dc56e67059011b2cf9d6043bb9
OS | macOS Version 11.7.10 (Build 20G1427)

Test Case #1 - (universal executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases:

  • launched 1.68.124 Chromium: 127.0.6533.57 via the standalone executable and visited brave://version
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version
Example Example Example
Screen Shot 2024-07-18 at 11 05 29 AM Screen Shot 2024-07-18 at 11 06 30 AM Screen Shot 2024-07-18 at 11 06 47 AM
  • restarted 1.68.124 Chromium: 127.0.6533.57 and ensured that the studies/variations were being pulled/applied
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version
Example Example Example
Screen Shot 2024-07-18 at 11 08 12 AM Screen Shot 2024-07-18 at 11 08 29 AM Screen Shot 2024-07-18 at 11 08 42 AM

Test Case #2 - (x64 executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases:

  • launched 1.68.124 Chromium: 127.0.6533.57 via the x64 online executable and visited brave://version
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that the studies/variations were being pulled/applied via brave://version
Example Example Example
Screen Shot 2024-07-18 at 11 16 42 AM Screen Shot 2024-07-18 at 11 17 06 AM Screen Shot 2024-07-18 at 11 17 13 AM

Test Case #3 - Setting test seed using --variations-pr= - PASSED

STR/Cases:

  • launched 1.68.124 Chromium: 127.0.6533.57 using --enable-logging=stderr --variations-pr=1123
  • ensured that there's no variations being displayed via brave://version (requires a restart to apply the pulled variation)
  • re-launched using --enable-logging=stderr --variations-pr=1123
  • ensured that the following parameters/switches are present/visible under Command Line
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
  • ensured that pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled is being displayed under Active variations: via brave://version
Example Example
Screen Shot 2024-07-18 at 11 20 52 AM Screen Shot 2024-07-18 at 11 21 39 AM

Test Case #4 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases:

  • after running through Test Case #2 above, removed --variations-pr=1123 and started 1.68.124 Chromium: 127.0.6533.57 using --enable-logging=stderr.
  • ensured that the following two parameters/switches are visible under Command Line via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
  • ensured that the following parameters/switches are not visible/present via Command Line under brave://version
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
Example Example
Screen Shot 2024-07-18 at 11 25 32 AM Screen Shot 2024-07-18 at 11 25 45 AM

Test Case #5 - Upgrade from 1.67.134 Chromium: 126.0.6478.186 -> 1.68.124 Chromium: 127.0.6533.57 - PASSED

STR/Cases:

  • installed 1.67.134 Chromium: 126.0.6478.186 and relaunched so the studies are applied/visible via brave://version
  • upgraded to 1.68.124 Chromium: 127.0.6533.57 via brave://settings/help by launching with ./Brave\ Browser --update-feed-url=https://updates.bravesoftware.com/sparkle/Brave-Browser/test/appcast.xml
  • ensured that once updated, the variations/studies are still visible/being applied via brave://version
1.67.134 Chromium: 126.0.6478.186 update 1.68.124 Chromium: 127.0.6533.57
Screen Shot 2024-07-18 at 11 58 35 AM Screen Shot 2024-07-18 at 11 59 23 AM Screen Shot 2024-07-18 at 12 00 18 PM

@stephendonner stephendonner added QA Pass-macOS and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Jul 18, 2024
@btlechowski
Copy link

Verified with

Brave 1.68.124 Chromium: 127.0.6533.57 (Official Build) (64-bit)
Revision d048917c488542dc56e67059011b2cf9d6043bb9
OS Linux

Test Case #1 - (offline executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases:

  • launched 1.69.x via the standalone executable and visited brave://version
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version

image

  • restarted 1.69.x and ensured that the studies/variations were being pulled/applied
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version

image

Test Case #2 - (online using https://variations.brave.com/seed as default) - PASSED

STR/Cases:

  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that the studies/variations were being pulled/applied via brave://version

image

Test Case #3 - Setting test seed using --variations-pr= - PASSED

STR/Cases:

  • launched 1.69.x using --enable-logging=stderr --variations-pr=1123
  • ensured that there's no variations being displayed via brave://version (requires a restart to apply the pulled variation)
  • re-launched using brave.exe --enable-logging=stderr --variations-pr=1123
  • ensured that the following parameters/switches are present/visible under Command Line
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
  • ensured that pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled is being displayed under Active variations: via brave://version

image

Test Case #4 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases:

  • after running through Test Case #2 above, removed --variations-pr=1123 and started 1.69.111 Chromium: 127.0.6533.43 using brave.exe --enable-logging=stderr.
  • ensured that the following two parameters/switches are visible under Command Line via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
  • ensured that the following parameters/switches are not visible/present via Command Line under brave://version
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling

image

Test Case #5 - Upgrade from 1.69.102 Chromium: 127.0.6533.43 -> 1.69.110 Chromium: 127.0.6533.43 - PASSED

STR/Cases:

  • installed 1.69.102 Chromium: 127.0.6533.43 and relaunched so the studies are applied/visible via brave://version
  • upgraded to 1.69.110 Chromium: 127.0.6533.43 via brave://settings/help
  • ensured that once updated, the variations/studies are still visible/being applied via brave://version

image

@MadhaviSeelam
Copy link

MadhaviSeelam commented Jul 18, 2024 •
edited
Loading

Verification Passed using

Brave | 1.68.124 Chromium: 127.0.6533.57 (Official Build) (64-bit)
-- | --
Revision | d048917c488542dc56e67059011b2cf9d6043bb9
OS | Windows 11 Version 23H2 (Build 22631.3880)

Test Case #1 - (offline executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases:

  • launched 1.68.124 Chromium: 127.0.6533.57 via the standalone executable and visited brave://version
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version
Example Example Example
image image image
  • restarted 1.68.124 Chromium: 127.0.6533.57 and ensured that the studies/variations were being pulled/applied
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version
Example Example Example
image image image

Test Case #2 - (online stub executable using https://variations.brave.com/seed as default) - PASSED

STR/Cases:

  • launched 1.68.124 Chromium: 127.0.6533.57 via the stub online executable and visited brave://version
  • ensured that --variations-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that --variations-insecure-server-url=https://variations.brave.com/seed is visible via brave://version
  • ensured that the studies/variations were being pulled/applied via brave://version
Example Example Example
image image image

Test Case #3 - Setting test seed using --variations-pr= - PASSED

STR/Cases:

  • launched 1.68.124 Chromium: 127.0.6533.57 using brave.exe --enable-logging=stderr --variations-pr=1123
  • ensured that there's no variations being displayed via brave://version (requires a restart to apply the pulled variation)
  • re-launched using brave.exe --enable-logging=stderr --variations-pr=1123
  • ensured that the following parameters/switches are present/visible under Command Line
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
  • ensured that pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled is being displayed under Active variations: via brave://version
Example Example
image image

Test Case #4 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases:

  • after running through Test Case #2 above, removed --variations-pr=1123 and started 1.68.124 Chromium: 127.0.6533.57 using brave.exe --enable-logging=stderr.
  • ensured that the following two parameters/switches are visible under Command Line via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
  • ensured that the following parameters/switches are not visible/present via Command Line under brave://version
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
Example Example
image image

Test Case #5 - Upgrade from 1.67.134 Chromium: 126.0.6478.186 -> 1.68.124 Chromium: 127.0.6533.57 - PASSED

STR/Cases:

  • installed 1.67.134 Chromium: 126.0.6478.186 and relaunched so the studies are applied/visible via brave://version
  • upgraded to 1.68.124 Chromium: 127.0.6533.57 via brave://settings/help
  • ensured that once updated, the variations/studies are still visible/being applied via brave://version
1.67.134 Chromium: 126.0.6478.186 1.68.124 Chromium: 127.0.6533.57
image image

@kjozwiak
Copy link
Member

kjozwiak commented Jul 23, 2024 •
edited
Loading

Verification PASSED on Pixel 6 Pro running Android 15 using the following build(s):

Brave | 1.68.125 Chromium: 127.0.6533.57 (Official Build) (64-bit)
--- | ---
Revision | 14ff4d467fb3eec4e6fc0a2f3e2d3f7b4c8cb788
OS | Android 15; Build/AP31.240617.009; 35; REL

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases:

  • installed/launched 1.68.125 Chromium: 127.0.6533.57
  • visited brave://version and ensured that the following parameters/switches were being used via Command Line
  • ensured that production variations/studies were being displayed via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example Example
Screenshot_20240723-014318 Screenshot_20240723-014313 Screenshot_20240723-014256

Test Case #2 - Setting test seed using --variations-pr= - PASSED

Prerequisite: ensured that the network on the device has been disabled before setting --variations-pr=

STR/Cases:

  • installed/launched 1.68.125 Chromium: 127.0.6533.57
  • added --variations-pr=1123viaCommand Line Stringunder theQA Developer` settings
  • closed Brave and re-enabled the devices network/WiFi
  • visited brave://version and ensured there was nothing being listed under Variations via brave://version
  • restarted 1.68.125 Chromium: 127.0.6533.57 once again after waiting ~10s
  • ensured that the following parameters/switches are present/visible under Command Line
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
Example Example Example Example Example Example
Screenshot_20240723-014749 Screenshot_20240723-014823 Screenshot_20240723-014908 Screenshot_20240723-014924 Screenshot_20240723-014941 Screenshot_20240723-014959
  • ensured that pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled is being displayed under Active variations: via brave://version

Screenshot_20240723-015009

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases:

  • after running through Test Case #2 above, removed --variations-pr=1123 from the Command Line String under the QA Developer settings and restarted 1.69.113 Chromium: 127.0.6533.43
  • waited ~10s and restarted 1.69.113 Chromium: 127.0.6533.43 once again
  • ensured that the following two parameters/switches are visible under Command Line via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example Example
Screenshot_20240723-015328 Screenshot_20240723-015443 Screenshot_20240723-015512 Screenshot_20240723-015531
  • ensured that the following parameters/switches are not visible/present via Command Line under brave://version
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
Example Example
Screenshot_20240723-015645 Screenshot_20240723-015633

Verification PASSED on Samsung Galaxy Tab S8 Ultra running Android 14 using the following build(s):

Brave | 1.68.125 Chromium: 127.0.6533.57 (Official Build) (64-bit)
--- | ---
Revision | 14ff4d467fb3eec4e6fc0a2f3e2d3f7b4c8cb788
OS | Android 14; Build/UP1A.231005.007; 34; REL

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases:

  • installed/launched 1.68.125 Chromium: 127.0.6533.57
  • visited brave://version and ensured that the following parameters/switches were being used via Command Line
  • ensured that production variations/studies were being displayed via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example
Screenshot_20240723_020438_Brave Screenshot_20240723_020531_Brave Screenshot_20240723_020548_Brave

Test Case #2 - Setting test seed using --variations-pr= - PASSED

Prerequisite: ensured that the network on the device has been disabled before setting --variations-pr=

STR/Cases:

  • installed/launched 1.68.125 Chromium: 127.0.6533.57
  • added --variations-pr=1123viaCommand Line Stringunder theQA Developer` settings
  • closed Brave and re-enabled the devices network/WiFi
  • visited brave://version and ensured there was nothing being listed under Variations via brave://version
  • restarted 1.68.125 Chromium: 127.0.6533.57 once again after waiting ~10s
  • ensured that the following parameters/switches are present/visible under Command Line
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
Example Example Example Example Example Example
Screenshot_20240723_020929_Brave Screenshot_20240723_021012_Brave Screenshot_20240723_021113_Brave Screenshot_20240723_021125_Brave Screenshot_20240723_021141_Brave Screenshot_20240723_021213_Brave
  • ensured that pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled is being displayed under Active variations: via brave://version

Screenshot_20240723_021223_Brave

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases:

  • after running through Test Case #2 above, removed --variations-pr=1123 from the Command Line String under the QA Developer settings and restarted 1.69.113 Chromium: 127.0.6533.43
  • waited ~10s and restarted 1.69.113 Chromium: 127.0.6533.43 once again
  • ensured that the following two parameters/switches are visible under Command Line via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example Example
Screenshot_20240723_021505_Brave Screenshot_20240723_021551_Brave Screenshot_20240723_021608_Brave Screenshot_20240723_021626_Brave
  • ensured that the following parameters/switches are not visible/present via Command Line under brave://version
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
Example Example
Screenshot_20240723_021658_Brave Screenshot_20240723_021713_Brave

@kjozwiak
Copy link
Member

kjozwiak commented Jul 29, 2024 •
edited
Loading

Verification PASSED on iPhone 11 running iOS 17.5.1 using the following build(s):

Brave | 1.68.130 Chromium: 127.0.6533.73 (Official Build) stable (64-bit) 
--- | ---
Revision | 841d137ffede...
OS | iOS

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases:

  • installed 1.68.130 Chromium: 127.0.6533.73 via TF
  • visited brave://version and ensured that there's no studies/variations being listed
  • restarted 1.68.130 Chromium: 127.0.6533.73 once again
  • visited brave://version and ensured that the following parameters/switches were being used via Command Line
  • ensured that production variations/studies were being displayed via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example
IMG_0417 IMG_0419 IMG_0418

Test Case #2 - Setting test seed using --variations-pr= - PASSED (using provided switch)

Prerequisite: ensured that the network on the device has been disabled before setting --variations-pr=

STR/Cases:

  • installed 1.68.130 Chromium: 127.0.6533.73 via TF
  • enabled the Variations PR switch under BraveCore Switches and entered 1123 as the value
  • closed Brave and re-enabled the devices network/WiFi
  • visited brave://version and ensured there was nothing being listed under Variations via brave://version
  • restarted 1.68.130 Chromium: 127.0.6533.73 once again after waiting ~10s
  • ensured that the following parameters/switches are present/visible under Command Line
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
Example Example Example Example Example Example
IMG_0420 IMG_0421 IMG_0422 IMG_0423 IMG_0424 IMG_0425
  • ensured that pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled is being displayed under Active variations: via brave://version

IMG_0426

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases:

  • after running through Test Case #2 above, removed --variations-pr=1123 from the Command Line String under the QA Developer settings and restarted 1.68.130 Chromium: 127.0.6533.73
  • waited ~10s and restarted 1.68.130 Chromium: 127.0.6533.73 once again
  • ensured that the following two parameters/switches are visible under Command Line via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example Example Example
IMG_0427 IMG_0428 IMG_0429 IMG_0430 IMG_0431
  • ensured that the following parameters/switches are not visible/present via Command Line under brave://version
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
Example Example
IMG_0432 IMG_0433

Verification PASSED on iPad Air (3rd Gen) running iOS 17.6 using the following build(s):

Brave | 1.68.130 Chromium: 127.0.6533.73 (Official Build) stable (64-bit) 
--- | ---
Revision | 841d137ffede0c1166265873663e6ebc56f93aa6
OS | iOS

Test Case #1 - using https://variations.brave.com/seed as default - PASSED

STR/Cases:

  • installed 1.68.130 Chromium: 127.0.6533.73 via TF
  • visited brave://version and ensured that there's no studies/variations being listed
  • restarted 1.68.130 Chromium: 127.0.6533.73 once again
  • visited brave://version and ensured that the following parameters/switches were being used via Command Line
  • ensured that production variations/studies were being displayed via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example
IMG_0412 IMG_0413 IMG_0414

Test Case #2 - Setting test seed using --variations-pr= - PASSED (using provided switch)

Prerequisite: ensured that the network on the device has been disabled before setting --variations-pr=

STR/Cases:

  • installed 1.68.130 Chromium: 127.0.6533.73 via TF
  • enabled the Variations PR switch under BraveCore Switches and entered 1123 as the value
  • closed Brave and re-enabled the devices network/WiFi
  • visited brave://version and ensured there was nothing being listed under Variations via brave://version
  • restarted 1.68.130 Chromium: 127.0.6533.73 once again after waiting ~10s
  • ensured that the following parameters/switches are present/visible under Command Line
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
--variations-server-url=https://griffin.brave.com/pull/1123/seed
--variations-insecure-server-url=https://griffin.brave.com/pull/1123/seed
Example Example Example Example Example Example
IMG_0417 IMG_0418 IMG_0419 IMG_0415 IMG_0416 IMG_0420
  • ensured that pull/1123@1082ba20390784d33de2b4e6275434deef6c4633 BraveAdsAdEventStudy:Enabled is being displayed under Active variations: via brave://version

IMG_0421

Test Case #3 - ensuring https://variations.brave.com/seed is being restored - PASSED

STR/Cases:

  • after running through Test Case #2 above, removed --variations-pr=1123 from the Command Line String under the QA Developer settings and restarted 1.68.130 Chromium: 127.0.6533.73
  • waited ~10s and restarted 1.68.130 Chromium: 127.0.6533.73 once again
  • ensured that the following two parameters/switches are visible under Command Line via brave://version
--variations-server-url=https://variations.brave.com/seed
--variations-insecure-server-url=https://variations.brave.com/seed
Example Example Example Example Example
IMG_0422 IMG_0423 IMG_0424 IMG_0425 IMG_0426
  • ensured that the following parameters/switches are not visible/present via Command Line under brave://version
--accept-empty-variations-seed-signature
--disable-variations-seed-fetch-throttling
Example Example
IMG_0427 IMG_0428

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@goodov goodov

Labels
dev-concern griffin OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality QA Pass - Android ARM QA Pass - Android Tab QA Pass - iPad QA Pass - iPhone QA Pass-Linux QA Pass-macOS QA Pass-Win64 QA/Test-All-Platforms QA/Test-Plan-Specified QA/Yes release-notes/exclude
Projects
None yet
Milestone
1.68.x - Release
Development

Successfully merging a pull request may close this issue.

Support brave-variations test seed testing process brave/brave-core
6 participants
@stephendonner @kjozwiak @goodov @btlechowski @brave-builds @MadhaviSeelam