Set Leo default model to Llama 3 8b (25% rollout) #1225

nvonpentz · 2024-10-15T18:57:18Z

Related to brave/brave-core#21398
Blocked on deploying https://github.com/brave/aichat-ops/pull/359 to prod

This sets the default model for 25% of free users to chat-basic, which corresponds to Llama 3 8b.

The intention is to start this to progressively roll out this change (25%, 50%, 75%, 100%), so we can ensure our single instance of Llama 3 can handle the increase in traffic. Based on these estimations, one instance of llama 3 (one gpu) should have higher token throughput than our single instance of mixtral (four gpus), so we expect this to work with two llama instances which will be added in https://github.com/brave/aichat-ops/pull/359.

cc @petemill @LorenzoMinto

Note:

Chromium version 122.0.6261.57 was selected because that was the first chromium version when 1.63.x was released (see https://bravesoftware.slack.com/archives/C04PX1BUN/p1708629893634639), which is when AI Chat: introduce freemium model concept brave-core#21398 went in.
I have included all platforms because we want to make this change across all platforms, however this does differ from the BraveAIChatEnabledStudy which applies only to desktop

github-actions · 2024-10-15T18:57:33Z

✅ Test Seed Generated Successfully

To apply the test seed:

Desktop: Launch the browser with --variations-pr=1225.
Android: Set the command line to --variations-pr=1225 in debug menu, restart the browser.
iOS: Set Variations PR to 1225 in Brave Core Switches debug menu, restart the browser.
Wait 5-10 seconds to fetch the seed.
Restart the browser to apply the seed.
Ensure Active Variations section at brave://version starts with the expected seed version (see below).

Seed Details

Parameter	Value
Version	`pull/1225@29f327ca26bca5d1900801196a3db1b3b478798d`
Uploaded	Mon, 28 Oct 2024 12:29:09 GMT
PR commit	`646922f`
Base commit	`a3ff5bd`
Merge commit	`29f327c`
Serial number	`a00f6077f3eac1b09f5be572757f6ba0`

petemill · 2024-10-15T21:18:35Z

seed/seed.json

+                    "BETA",
+                    "NIGHTLY"
+                ],
+                "min_version": "122.0.6261.57",


looks like in this file the versioning is [chrome_major].[brave_version], e.g 122.1.63.0?

I updated to use 122.1.63.161 as the min version of the default model change. 1.63.161 seems to be the first 1.63 release. This also adds in a little buffer because brave/brave-browser#34721 was uplifted into 1.62.x.

I specified 122.1.63.160 as the new max version for the previous setting. This may not correlate to an actual release, it's just one patch version behind 122.1.63.161.

petemill · 2024-10-15T21:24:21Z

I tested running with --enable-features=AIChat:default_model\/chat-basic and all seems fine

kjozwiak · 2024-10-22T01:33:31Z

@petemill will grab this tomorrow/verify and push it into production 👍

seed/seed.json

…on't overlap

Use [chrome_major].[brave_version] format, not just the chromium version.

kjozwiak · 2024-10-28T16:12:50Z

`Windows x64` verification(s):

Verification PASSED on Win 11 x64 using the following build(s):

Brave | 1.71.118 Chromium: 130.0.6723.70 (Official Build) (64-bit)
-- | --
Revision | d29310b051bde8ba041cd7b4edcb41ce951be12c
OS | Windows 11 Version 23H2 (Build 22631.4317)

`Release` - `BraveAIChatDefaultModelStudy:DefaultLlama`

launched/re-launched 1.71.118 Chromium: 130.0.6723.70 using brave.exe --enable-logging=stderr --variations-pr=1225
ensured that pull/1225@02be91adb07839f2668690152f397b0a9ec4c0c2 via brave://version under Active variations
ensured that BraveAIChatDefaultModelStudy:DefaultLlama via brave://version
ensured that Llama 3.1 8B is selected as the default model via brave://settings/leo-assistant
ensured that Llama 3.1 8B is selected as the default model via the side panel
ensured that Llama 3.1 8B is being used when using Leo when BraveAIChatDefaultModelStudy:DefaultLlama

`Example`	`Example`	`Example`	`Example`

`Release` - `BraveAIChatDefaultModelStudy:DefaultMixtral`

launched/re-launched 1.71.118 Chromium: 130.0.6723.70 using brave.exe --enable-logging=stderr --variations-pr=1225
ensured that pull/1225@02be91adb07839f2668690152f397b0a9ec4c0c2 via brave://version under Active variations
ensured that BraveAIChatDefaultModelStudy:DefaultMixtral via brave://version
ensured that Mixtral is selected as the default model via brave://settings/leo-assistant
ensured that Mixtral is selected as the default model via the side panel
ensured that Mixtral is being used when using Leo when BraveAIChatDefaultModelStudy:DefaultMixtral

`Example`	`Example`	`Example`	`Example`

`Android` verification(s):

Verification PASSED on Pixel 6 running Android 15 using the following build(s):

Brave | 1.71.118 Chromium: 130.0.6723.70 (Official Build) (64-bit)
--- | ---
Revision | d29310b051bde8ba041cd7b4edcb41ce951be12c
OS | Android 15; Build/AP41.240823.009; 35; REL

`Release` - `BraveAIChatDefaultModelStudy:DefaultLlama`

launched 1.71.118 Chromium: 130.0.6723.70 and added --variations-pr=1225 into the command line via QA dev tools
re-launched 1.71.118 Chromium: 130.0.6723.70 for the second time
ensured that pull/1225@02be91adb07839f2668690152f397b0a9ec4c0c2 via brave://version under Active variations
ensured that BraveAIChatDefaultModelStudy:DefaultLlama via brave://version
ensured that Llama 3.1 8B is selected as the default model via Settings -> Leo -> Default model for new conversations
ensured that Llama 3.1 8B is selected as the default model via Leo panel
ensured that Llama 3.1 8B is being used when using Leo when BraveAIChatDefaultModelStudy:DefaultLlama

`Example`	`Example`	`Example`	`Example`	`Example`

`Release` - `BraveAIChatDefaultModelStudy:DefaultMixtral`

launched 1.71.118 Chromium: 130.0.6723.70 and added --variations-pr=1225 into the command line via QA dev tools
re-launched 1.71.118 Chromium: 130.0.6723.70 for the second time
ensured that pull/1225@02be91adb07839f2668690152f397b0a9ec4c0c2 via brave://version under Active variations
ensured that BraveAIChatDefaultModelStudy:DefaultMixtral via brave://version
ensured that Mixtral is selected as the default model via Settings -> Leo -> Default model for new conversations
ensured that Mixtral is selected as the default model via Leo panel
ensured that Mixtral is being used when using Leo when BraveAIChatDefaultModelStudy:DefaultMixtral

`Example`	`Example`	`Example`	`Example`	`Example`

`iOS` verification(s):

Verification PASSED on iPhone 11 running iOS 18 using the following build(s):

Brave | 1.70.133 Chromium: 129.0.6668.100 (Official Build) stable (64-bit) 
--- | ---
Revision | 53596529e7d7...
OS | iOS

`Release` - `BraveAIChatDefaultModelStudy:DefaultLlama`

launched 1.70.133 Chromium: 129.0.6668.100 and added 1225 into the Variations PR switch under BraveCore Switches
re-launched 1.70.133 Chromium: 129.0.6668.100 for the second time
ensured that pull/1225@02be91adb07839f2668690152f397b0a9ec4c0c2 via brave://version under Active variations
ensured that BraveAIChatDefaultModelStudy:DefaultLlama via brave://version
ensured that Llama 3.1 8B is selected as the default model via Settings -> Leo -> Default model
ensured that Llama 3.1 8B is selected as the default model via Leo panel
ensured that Llama 3.1 8B is being used when using Leo when BraveAIChatDefaultModelStudy:DefaultLlama

`Example`	`Example`	`Example`	`Example`	`Example`	`Example`

`Release` - `BraveAIChatDefaultModelStudy:DefaultMixtral`

launched 1.70.133 Chromium: 129.0.6668.100 and added 1225 into the Variations PR switch under BraveCore Switches
re-launched 1.70.133 Chromium: 129.0.6668.100 for the second time
ensured that pull/1225@02be91adb07839f2668690152f397b0a9ec4c0c2 via brave://version under Active variations
ensured that BraveAIChatDefaultModelStudy:DefaultMixtral via brave://version
ensured that Mixtral is selected as the default model via Settings -> Leo -> Default model
ensured that Mixtral is selected as the default model via Leo panel
ensured that Mixtral is being used when using Leo when BraveAIChatDefaultModelStudy:DefaultMixtral

`Example`	`Example`	`Example`	`Example`	`Example`	`Example`

kjozwiak

Uplift into production approved after deliberating with @brave/uplift-approvers.

Follows on #1225 Co-authored-by: Nick von Pentz <[email protected]>

petemill reviewed Oct 15, 2024

View reviewed changes

nvonpentz marked this pull request as ready for review October 17, 2024 16:38

nvonpentz requested a review from a team as a code owner October 17, 2024 16:38

nvonpentz requested a review from petemill October 17, 2024 16:39

petemill approved these changes Oct 22, 2024

View reviewed changes

petemill reviewed Oct 25, 2024

View reviewed changes

seed/seed.json Outdated Show resolved Hide resolved

nvonpentz and others added 5 commits October 28, 2024 19:25

Set Leo default model to Llama 3 8b (25% rollout)

51dead3

Specify a max_version for the older BraveAIChatEnabledStudy so they d…

a8fee42

…on't overlap

Update min / max versions

2d09ef3

Use [chrome_major].[brave_version] format, not just the chromium version.

fix parameter format

6790c22

Use updated study storage format.

646922f

goodov force-pushed the leo-default-model branch from 9d410b5 to 646922f Compare October 28, 2024 12:27

kjozwiak approved these changes Oct 28, 2024

View reviewed changes

kjozwiak added this pull request to the merge queue Oct 28, 2024

Merged via the queue into main with commit 619d91e Oct 28, 2024
6 checks passed

kjozwiak deleted the leo-default-model branch October 28, 2024 18:22

nvonpentz mentioned this pull request Oct 30, 2024

Rollout Llama as default model to 50% of users #1249

Merged

github-merge-queue bot pushed a commit that referenced this pull request Oct 30, 2024

Rollout Llama as default model to 50% of users (#1249)

9c9b546

Follows on #1225 Co-authored-by: Nick von Pentz <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set Leo default model to Llama 3 8b (25% rollout) #1225

Set Leo default model to Llama 3 8b (25% rollout) #1225

nvonpentz commented Oct 15, 2024 •

edited

Loading

github-actions bot commented Oct 15, 2024 •

edited

Loading

petemill Oct 15, 2024

nvonpentz Oct 17, 2024

petemill commented Oct 15, 2024

kjozwiak commented Oct 22, 2024

kjozwiak commented Oct 28, 2024 •

edited

Loading

kjozwiak left a comment

Set Leo default model to Llama 3 8b (25% rollout) #1225

Set Leo default model to Llama 3 8b (25% rollout) #1225

Conversation

nvonpentz commented Oct 15, 2024 • edited Loading

github-actions bot commented Oct 15, 2024 • edited Loading

✅ Test Seed Generated Successfully

Seed Details

petemill Oct 15, 2024

Choose a reason for hiding this comment

nvonpentz Oct 17, 2024

Choose a reason for hiding this comment

petemill commented Oct 15, 2024

kjozwiak commented Oct 22, 2024

kjozwiak commented Oct 28, 2024 • edited Loading

Windows x64 verification(s):

Release - BraveAIChatDefaultModelStudy:DefaultLlama

Release - BraveAIChatDefaultModelStudy:DefaultMixtral

Android verification(s):

Release - BraveAIChatDefaultModelStudy:DefaultLlama

Release - BraveAIChatDefaultModelStudy:DefaultMixtral

iOS verification(s):

Release - BraveAIChatDefaultModelStudy:DefaultLlama

Release - BraveAIChatDefaultModelStudy:DefaultMixtral

kjozwiak left a comment

Choose a reason for hiding this comment

nvonpentz commented Oct 15, 2024 •

edited

Loading

github-actions bot commented Oct 15, 2024 •

edited

Loading

kjozwiak commented Oct 28, 2024 •

edited

Loading

`Windows x64` verification(s):

`Release` - `BraveAIChatDefaultModelStudy:DefaultLlama`

`Release` - `BraveAIChatDefaultModelStudy:DefaultMixtral`

`Android` verification(s):

`Release` - `BraveAIChatDefaultModelStudy:DefaultLlama`

`Release` - `BraveAIChatDefaultModelStudy:DefaultMixtral`

`iOS` verification(s):

`Release` - `BraveAIChatDefaultModelStudy:DefaultLlama`

`Release` - `BraveAIChatDefaultModelStudy:DefaultMixtral`