Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Language Selection for Review #354

Merged
merged 3 commits into from
Jan 12, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion action.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,10 @@ jobs:
- name: Code Review GPT
uses: mattzcarey/[email protected]
with:
GITHUB_TOKEN: ${{ github.token }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
MODEL: 'gpt-4o'
GITHUB_TOKEN: ${{ github.token }}
REVIEW_LANGUAGE: 'English'
```

### Workflow yml option 2: Add a code review bot
Expand Down Expand Up @@ -60,7 +61,10 @@ jobs:
- name: Code Review GPT
uses: mattzcarey/[email protected]
with:
GITHUB_TOKEN: ${{ github.token }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
MODEL: 'gpt-4o'
REVIEW_LANGUAGE: 'English'
MODEL: 'gpt-4o-mini' # reference: https://platform.openai.com/settings/organization/limits
GITHUB_TOKEN: ${{ github.token }}
```
7 changes: 6 additions & 1 deletion action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ inputs:
description: 'The GPT model to use'
required: true
default: 'gpt-4o'
REVIEW_LANGUAGE:
description: 'Target natural language for translation'
required: false
default: 'English'
OPENAI_API_KEY:
description: 'OpenAI API Key'
required: true
Expand All @@ -27,10 +31,11 @@ runs:

- name: Run Code Review GPT
shell: bash
run: npx code-review-gpt review --ci=github --model=$MODEL
run: npx code-review-gpt review --ci=github --model=$MODEL --reviewLanguage=$REVIEW_LANGUAGE

env:
MODEL: ${{ inputs.MODEL }}
REVIEW_LANGUAGE: ${{ inputs.REVIEW_LANGUAGE }}
OPENAI_API_KEY: ${{ inputs.OPENAI_API_KEY }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
GITHUB_TOKEN: ${{ inputs.GITHUB_TOKEN }}
Expand Down
2 changes: 2 additions & 0 deletions packages/code-review-gpt/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ You can now run `code-review-gpt review` in the root directory of any git-enable

- `--reviewType` - Used with the 'review' command. The options are --reviewType=("changed" | "full" | "costOptimized). Defaults to "changed" if no option is specified. Specifies whether the review is for the full file or just the changed lines. costOptimized limits the context surrounding the changed lines to 5 lines.

- `--reviewLanguage` - Specifies the target natural language for translation, such as "German", "Spanish", or "French". Use this parameter to set the language in which the utility will provide output.

- `--remote` - Used with the 'review' command. Usage `--remote=mattzcarey/code-review-gpt#96`. Review a remote GitHub Pull Request.

- `--commentPerFile` - Used when the `--ci` flag is set. Defaults to false. It enables the bot to comment the feedback on a file-by-file basis.
Expand Down
4 changes: 4 additions & 0 deletions packages/code-review-gpt/src/args.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ export const getYargs = async (): Promise<ReviewArgs> => {
type: "string",
default: "changed",
})
.option("reviewLanguage", {
description: "Specifies the target natural language for translation",
type: "string",
})
.option("remote", {
description: "The identifier of a remote Pull Request to review",
type: "string",
Expand Down
1 change: 1 addition & 0 deletions packages/code-review-gpt/src/common/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ export type ReviewArgs = {
commentPerFile: boolean;
model: string;
reviewType: string;
reviewLanguage: string | undefined;
org: string | undefined;
remote: string | undefined;
provider: string;
Expand Down
4 changes: 3 additions & 1 deletion packages/code-review-gpt/src/review/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ export const review = async (
const reviewType = yargs.reviewType;
const organization = yargs.org;
const provider = yargs.provider;
const reviewLanguage = yargs.reviewLanguage;

const filteredFiles = filterFiles(files);

Expand All @@ -54,7 +55,8 @@ export const review = async (
const prompts = constructPromptsArray(
filteredFiles,
maxPromptLength,
reviewType
reviewType,
reviewLanguage
);

logger.debug(`Prompts used:\n ${prompts.toString()}`);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ import { instructionPrompt } from "../prompts";
export const constructPromptsArray = (
files: ReviewFile[],
maxPromptLength: number,
reviewType: string
reviewType: string,
reviewLanguage = 'English',
): string[] => {
const maxPromptPayloadLength = maxPromptLength - instructionPrompt.length;
let promptPayloads: PromptFile[][];
Expand All @@ -33,10 +34,9 @@ export const constructPromptsArray = (
);
}

const languageToInstructionPrompt = instructionPrompt.replace(
"{Language}",
getLanguageName(files[0].fileName) //assume the first file is representative of the language
);
const languageToInstructionPrompt = instructionPrompt
.replace("{ProgrammingLanguage}", getLanguageName(files[0].fileName))
.replace("{ReviewLanguage}", reviewLanguage);

const prompts = promptPayloads.map((payload) => {
return languageToInstructionPrompt + JSON.stringify(payload);
Expand Down
37 changes: 2 additions & 35 deletions packages/code-review-gpt/src/review/prompt/prompts.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
export const instructionPrompt = `You are an expert {Language} developer, your task is to review a set of pull requests.
export const instructionPrompt = `You are an expert {ProgrammingLanguage} developer, your task is to review a set of pull requests.
You are given a list of filenames and their partial contents, but note that you might not have the full context of the code.

Only review lines of code which have been changed (added or removed) in the pull request. The code looks similar to the output of a git diff command. Lines which have been removed are prefixed with a minus (-) and lines which have been added are prefixed with a plus (+). Other lines are added to provide context but should be ignored in the review.
Expand All @@ -11,7 +11,7 @@ Do not comment on breaking functions down into smaller, more manageable function

Use markdown formatting for the feedback details. Also do not include the filename or risk level in the feedback details.

Ensure the feedback details are brief, concise, accurate. If there are multiple similar issues, only comment on the most critical.
Ensure the feedback details are brief, concise, accurate, and in {ReviewLanguage}. If there are multiple similar issues, only comment on the most critical.

Include brief example code snippets in the feedback details for your suggested changes when you're confident your suggestions are improvements. Use the same programming language as the file under review.
If there are multiple improvements you suggest in the feedback details, use an ordered list to indicate the priority of the changes.
Expand All @@ -38,36 +38,3 @@ You are a senior developer and have just reviewed a pull request. This was your
{feedback}
Please summarise the review using 3 emojis.
`;

export const demoPrompt = `You are an senior developer, your task is to review a code snippet.
Note that you do not have the full context of the code.

Begin your review by evaluating the code using a risk score similar to a LOGAF score but measured from 1 to 5, where 1 is the lowest risk to the code base if the code is merged and 5 is the highest risk which would likely break something or be unsafe.

In your feedback, focus on highlighting potential bugs, improving readability if it is a problem, making code cleaner, and maximising the performance of the programming language. Flag any API keys or secrets present in the code in plain text immediately as highest risk. Rate the changes based on SOLID principles if applicable.

Do not comment on breaking functions down into smaller, more manageable functions unless it is a huge problem. Also be aware that there will be libraries and techniques used which you are not familiar with, so do not comment on those unless you are confident that there is a problem.

Use markdown formatting for the feedback details. Also do not include the risk level in the feedback details.

Ensure the feedback details are brief, concise, accurate. If there are multiple similar issues, only comment on the most critical.

Include brief example code snippets in the feedback details for your suggested changes when you're confident your suggestions are improvements. Use the same programming language as the file under review.
If there are multiple improvements you suggest in the feedback details, use an ordered list to indicate the priority of the changes.

Format the response in a valid JSON format as a list of feedbacks, where the value is an object containing the risk score ("riskScore") and the feedback ("details"). Also add the filename ("filename") which will always be "demo code". The schema of the JSON feedback object must be:
{
"fileName": {
"type": "string"
},
"riskScore": {
"type": "number"
},
"details": {
"type": "string"
}
}

The code to review is provided below:

`;
5 changes: 3 additions & 2 deletions packages/code-review-gpt/src/test/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import { loadOrGenerateCodeSnippets } from "./load/loadTestCodeSnippets";
import { runTests } from "./run/runTest";

export const test = async (
{ ci, model, reviewType }: ReviewArgs,
{ ci, model, reviewType, reviewLanguage }: ReviewArgs,
openAIApiKey: string
): Promise<void> => {
const maxPromptLength = getMaxPromptLength(model);
Expand Down Expand Up @@ -40,12 +40,13 @@ export const test = async (

// Run the review on the code snippets and compare the results to the expected results.
const testSummary = await runTests(
openAIApiKey,
testCasesWithSnippets,
model,
maxPromptLength,
vectorStore,
reviewType,
openAIApiKey
reviewLanguage,
);

if (ci === PlatformOptions.GITHUB) {
Expand Down
18 changes: 14 additions & 4 deletions packages/code-review-gpt/src/test/run/runTest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,24 @@ import {

/**
* Run a single test case.
* @param openAIApiKey Open AI API Key.
* @param testCase The test case.
* @param modelName The name of the model.
* @param maxPromptLength The maximum prompt length.
* @param vectorStore The vector store.
* @param reviewType The review type.
* @param reviewLanguage Target natural language for translation.
* @returns The test result.
*/
const runTest = async (
openAIApiKey: string,
testCase: TestCase,
modelName: string,
maxPromptLength: number,
vectorStore: MemoryVectorStore,
reviewType: string,
openAIApiKey: string
reviewLanguage?: string,
// eslint-disable-next-line max-params
): Promise<testResult> => {
if (!testCase.snippet) {
throw new Error(`Test case ${testCase.name} does not have a snippet.`);
Expand All @@ -38,7 +42,8 @@ const runTest = async (
const prompts = constructPromptsArray(
[testCase.snippet],
maxPromptLength,
reviewType
reviewType,
reviewLanguage
);

const { markdownReport: reviewResponse } = await askAI(
Expand Down Expand Up @@ -74,20 +79,24 @@ const runTest = async (

/**
* Run all the test cases.
* @param openAIApiKey Open AI API Key.
* @param testCases The test cases.
* @param modelName The name of the model.
* @param maxPromptLength The maximum prompt length.
* @param vectorStore The vector store.
* @param reviewType The review type.
* @param reviewLanguage Target natural language for translation.
* @returns The test results.
*/
export const runTests = async (
openAIApiKey: string,
testCases: TestCase[],
modelName: string,
maxPromptLength: number,
vectorStore: MemoryVectorStore,
reviewType: string,
openAIApiKey: string
reviewLanguage?: string,
// eslint-disable-next-line max-params
): Promise<string> => {
if (testCases.length === 0) {
return "No test cases found.";
Expand All @@ -101,12 +110,13 @@ export const runTests = async (
for (const testCase of testCases) {
try {
const result = await runTest(
openAIApiKey,
testCase,
modelName,
maxPromptLength,
vectorStore,
reviewType,
openAIApiKey
reviewLanguage
);
testResults[testCase.name] = result;
} catch (error) {
Expand Down