diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index c6cbf82..d161478 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -25,7 +25,7 @@ jobs: id: test timeout-minutes: 1 run: | - ./scripts/test.sh + python -m pip install colorama && ./scripts/test.py - name: Perm issue fix if: success() || failure() diff --git a/Dockerfile b/Dockerfile index afaf3f2..76ae85b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -64,4 +64,8 @@ RUN ../configure --prefix=$RISCV --host=riscv64-unknown-elf --with-arch=rv32imfd RUN make RUN make install +# Install Python packages +RUN pip install --no-cache-dir --upgrade pip && \ + pip install --no-cache-dir colorama + ENTRYPOINT [ "/bin/bash" ] diff --git a/compiler_tests/default/test_RETURN.c b/compiler_tests/default/test_RETURN.c deleted file mode 100644 index f37a955..0000000 --- a/compiler_tests/default/test_RETURN.c +++ /dev/null @@ -1,4 +0,0 @@ -int f() -{ - return 10; -} diff --git a/compiler_tests/default/test_RETURN_driver.c b/compiler_tests/default/test_RETURN_driver.c deleted file mode 100644 index ae7e7c7..0000000 --- a/compiler_tests/default/test_RETURN_driver.c +++ /dev/null @@ -1,7 +0,0 @@ - -int f(); - -int main() -{ - return !( 10==f() ); -} diff --git a/docs/assembler_directives.md b/docs/assembler_directives.md index 31b392f..8c90162 100644 --- a/docs/assembler_directives.md +++ b/docs/assembler_directives.md @@ -4,7 +4,11 @@ Assembler directives The linked guide explains in details all available directives, but fortunately you only need a very small subset to start with and even the more advanced features only require a few additional directives. While [Godbolt](https://godbolt.org/z/vMMnWbsff) emits some directives, to see all of them (more than you actually need) you are advised to run: -```riscv64-unknown-elf-gcc -std=c90 -pedantic -ansi -O0 -march=rv32imfd -mabi=ilp32d -S [source-file.c] -o [dest-file.s]```. +```console +> user@host:langproc-cw# riscv64-unknown-elf-gcc -std=c90 -pedantic -ansi -O0 -march=rv32imfd -mabi=ilp32d -S [source-file.c] -o [dest-file.s] +``` + +In the [`scripts/test.py`](../scripts/test.py) script, when running testcases, there is the aforementioned call to the compiler and the compiled test programs can be found as `.gcc.s`. Your compiler may not produce the exact same assembly as GCC, so it is not advisable to blindly attempt to replicate the GCC output. Instead, the purpose of the `.gcc.s` files is to assist in debugging issues within your own compiler. The below picture offers a quick walk-through of a very simple program with detailed annotations describing the meaning behind the included directives. Some of them a crucial (e.g. section specifiers, labels, data emitting) while others not so much (e.g. file attributes, compiler identifier, symbol types) - you will get a feel for them during the development of the compiler. Most importantly, you only need to set the correct section and provide function directives as long as you deal with local variables. **In other words, you can postpone studying this document in details until you decide to deal with global variables.** diff --git a/docs/basic_compiler.md b/docs/basic_compiler.md index a9f3123..af6b507 100644 --- a/docs/basic_compiler.md +++ b/docs/basic_compiler.md @@ -3,8 +3,7 @@ Basic compiler For the first time ever, you are provided with a basic compiler that can lex, parse and generate code for only the following program: ``` -int f() -{ +int f() { return 5; } ``` @@ -13,14 +12,14 @@ Having a somewhat functioning compiler should allow you to get started with deve The provided basic compiler is able to traverse the following AST related to the above program. In order to expand its capabilities, you should develop the parser and the corresponding code generation at the same time - do not try to fully implement one before the other. -![int_main_return_tree](./int_main_return_tree.png) +![int_main_return_tree](./int_main_return_5_tree.png) The lexer and parser are loosely based on the "official" grammar covered [here](https://www.lysator.liu.se/c/ANSI-C-grammar-l.html) and [here](https://www.lysator.liu.se/c/ANSI-C-grammar-y.html) respectively. While they should suffice for a significant portions of features, you might need to improve them to implement the more advanced ones. If you find the grammar too complicated to understand, it is also perfectly fine to create your own simple grammar and build upon it as you add more features. Two versions of the grammar have been provided: -- [parser.y](../src/parser.y) contains a stripped down version of the grammar which contains only the elements from the full grammar that are necessary to parse the above program. **This is the grammar used when you run ./test.sh**. +- [parser.y](../src/parser.y) contains a stripped down version of the grammar which contains only the elements from the full grammar that are necessary to parse the above program. **This is the grammar used when you run ./scripts/test.py**. - [parser_full.y.example](../src/parser_full.y.example) is the same as `parser.y` but also contains the rest of the C90 grammar not used in the above program. Once you understand how the stripped version works, you have two options: you can extend `parser.y` at your own pace and add pieces of the C90 grammar to it as you implement them; alternatively, you may paste the contents of `parser_full.y.example` into `parser.y` allowing you to parse everything but this may make the file much harder to navigate. Note that you don't have to use any of the provided grammar examples (or even flex and bison) if you are comfortable with doing something else. However, be warned that TAs will likely only be able to provided limited support for such alternatives. diff --git a/docs/c_compiler.md b/docs/c_compiler.md index 0193c44..2c64f06 100644 --- a/docs/c_compiler.md +++ b/docs/c_compiler.md @@ -14,37 +14,39 @@ If you wish to use C++, then a basic framework for building your compiler has be Source files can be found in the [./src](../src) folder and header files can be found in the [./include](../include) folder. -If you have Python installed, you can test your compiler by running +You can test your compiler by running [`scripts/test.py`](../scripts/test.py) from the top of this repo; the -output should look as follows (note: the progress bar will be coloured). -Full usage guide is found in the file header. +output should look as follows (note: the progress bar and results will be coloured): ```console -> scripts/test.py +> user@host:langproc-cw# scripts/test.py > -make: Entering directory '/home/saturn691/projects/university/iac/langproc-cw' -make: 'bin/c_compiler' is up to date. -make: Leaving directory '/home/saturn691/projects/university/iac/langproc-cw' Running Tests [################################################################] -Pass: 1 | Fail: 87 | Remaining: 0 +Pass: 1 | Fail: 85 | Remaining: 0 See logs for more details (use -v for verbose output). ->> Test Summary: 1 Passed, 87 Failed +>> Test Summary: 1 Passed, 85 Failed +``` + +Full usage guide of [`scripts/test.py`](../scripts/test.py) is found in the file header or after running: +```console +> user@host:langproc-cw# scripts/test.py --help ``` -If Python is not installed, you can also test your compiler against the provided +If for any reason you run into issues with the Python script, you can also test your compiler against the provided test-suite by running [`scripts/test.sh`](../scripts/test.sh) from the top of this repo; the output should look as follows: ```console -> scripts/test.sh +> user@host:langproc-cw# scripts/test.sh > compiler_tests/_example/example.c > Pass compiler_tests/array/declare_global.c > Fail: simulation did not exit with exit-code 0 ... +Passing 1/86 tests ``` By default, the first [`_example/example.c`](../compiler_tests/_example/example.c) test should be passing. @@ -56,11 +58,15 @@ Program build and execution Your program should be built by running the following command in the top-level directory of your repo: - make bin/c_compiler +```console +> user@host:langproc-cw# make bin/c_compiler +``` The compilation function is invoked using the flag `-S`, with the source file and output file specified on the command line: - bin/c_compiler -S [source-file.c] -o [dest-file.s] +```console +> user@host:langproc-cw# bin/c_compiler -S [source-file.c] -o [dest-file.s] +``` You can assume that the command-line (CLI) arguments will always be in this order, and that there will be no spaces in source or destination paths. @@ -157,28 +163,45 @@ It should be possible to assemble and link this code against a C run-time, and h For instance, suppose I have a file called `test_program.c` that contains: - int f() { return 5; } +``` +int f() { + return 5; +} +``` and another file called `test_program_driver.c` that contains: - int f(); - int main() { return !( 5 == f() ); } +``` +int f(); + +int main() { + return !( 5 == f() ); +} +``` I run the compiler on the test program, like so: - bin/c_compiler -S test_program.c -o test_program.s +```console +> user@host:langproc-cw# bin/c_compiler -S test_program.c -o test_program.s +``` I then use GCC to assemble the generated assembly program (`test_program.s`), like so: - riscv64-unknown-elf-gcc -march=rv32imfd -mabi=ilp32d -o test_program.o -c test_program.s +```console +> user@host:langproc-cw# riscv64-unknown-elf-gcc -march=rv32imfd -mabi=ilp32d -o test_program.o -c test_program.s +``` I then use GCC to link the generated object file (`test_program.o`) with the driver program (`test_program_driver.c`), to produce an executable (`test_program`), like so: - riscv64-unknown-elf-gcc -march=rv32imfd -mabi=ilp32d -static -o test_program test_program.o test_program_driver.c +```console +> user@host:langproc-cw# riscv64-unknown-elf-gcc -march=rv32imfd -mabi=ilp32d -static -o test_program test_program.o test_program_driver.c +``` I then use spike to simulate the executable on RISC-V, like so: - spike pk test_program +```console +> user@host:langproc-cw# spike pk test_program +``` This command should produce the exit code `0`. diff --git a/docs/coverage.md b/docs/coverage.md index d29fedc..651633f 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -7,18 +7,18 @@ This will generate a webpage `coverage/index.html` with a listing of all the sou ![Index.html screenshot](./coverage_example.png) -It can also be used automatically on all test files by running `COVERAGE=1 ./test.sh`. +It can also be used automatically on all test files by running: `./scripts/test.py --coverage` or using the old test script: `COVERAGE=1 ./test.sh`. ## Viewing the coverage webpage -You can view the webpage in your browser by navigating to the coverage directory and running the following command: - -```python3 -m http.server``` - -For example: +You can view the webpage in your browser by navigating to the coverage directory and running a Python HTTP server: ```console -user@host:/workspaces/langproc-cw# cd coverage -user@host:/workspaces/langproc-cw/coverage# python3 -m http.server +> user@host:langproc-cw# cd coverage +> user@host:langproc-cw/coverage# python3 -m http.server +> Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ... -``` \ No newline at end of file +``` + +VS Code should prompt you to open the webpage in your browser. +You can also manually do so by navigating to [http://0.0.0.0:8000](`http://0.0.0.0:8000`). diff --git a/docs/int_main_return_5_tree.png b/docs/int_main_return_5_tree.png new file mode 100644 index 0000000..09ee9b2 Binary files /dev/null and b/docs/int_main_return_5_tree.png differ diff --git a/docs/int_main_return_tree.png b/docs/int_main_return_tree.png deleted file mode 100644 index f4986c1..0000000 Binary files a/docs/int_main_return_tree.png and /dev/null differ diff --git a/scripts/test.py b/scripts/test.py index d3d1f19..9dbe5a4 100755 --- a/scripts/test.py +++ b/scripts/test.py @@ -7,14 +7,14 @@ This script will also generate a JUnit XML file, which can be used to integrate with CI/CD pipelines. -Usage: test.py [-h] [-m] [-v] [--version] [dir] +Usage: test.py [-h] [-m] [-s] [--version] [--no_clean | --coverage] [dir] Example usage: scripts/test.py compiler_tests/_example This will print out a progress bar and only run the example tests. The output would be placed into bin/output/_example/example/. -For more information, run scripts/test.py -h +For more information, run scripts/test.py --help """ @@ -22,24 +22,31 @@ __author__ = "William Huynh (@saturn691)" -import argparse import os +import argparse import shutil import subprocess +import re import queue from pathlib import Path from concurrent.futures import ThreadPoolExecutor, as_completed +from typing import List, Optional +from colorama import Fore, init, just_fix_windows_console +just_fix_windows_console() # Make Windows behave as standard terminals +init(autoreset=True) # No need to reset style to default after each style changing call # "File" will suggest the absolute path to the file, including the extension. SCRIPT_LOCATION = Path(__file__).resolve().parent PROJECT_LOCATION = SCRIPT_LOCATION.joinpath("..").resolve() OUTPUT_FOLDER = PROJECT_LOCATION.joinpath("bin/output").resolve() -J_UNIT_OUTPUT_FILE = PROJECT_LOCATION.joinpath( - "bin/junit_results.xml").resolve() +J_UNIT_OUTPUT_FILE = PROJECT_LOCATION.joinpath("bin/junit_results.xml").resolve() COMPILER_TEST_FOLDER = PROJECT_LOCATION.joinpath("compiler_tests").resolve() COMPILER_FILE = PROJECT_LOCATION.joinpath("bin/c_compiler").resolve() +COVERAGE_FOLDER = PROJECT_LOCATION.joinpath("coverage").resolve() +BUILD_TIMEOUT = 60 # seconds +RUN_TIMEOUT = 15 # seconds class ProgressBar: """ @@ -50,10 +57,11 @@ class ProgressBar: - total_tests: the length of the progress bar. """ - def __init__(self, total_tests): + def __init__(self, total_tests: int, silent: bool = False): self.total_tests = total_tests self.passed = 0 self.failed = 0 + self.silent = silent _, max_line_length = os.popen("stty size", "r").read().split() self.max_line_length = min( @@ -63,8 +71,13 @@ def __init__(self, total_tests): # Initialize the lines for the progress bar and stats print("Running Tests [" + " " * self.max_line_length + "]") - print("Pass: 0 | Fail: 0 | Remaining: {}".format(total_tests)) - print("See logs for more details (use -v for verbose output).") + print( + Fore.GREEN + "Pass: 0 | " + + Fore.RED + "Fail: 0 | " + + Fore.RESET + f"Remaining: {total_tests:2}" + ) + if not self.silent: + print("See logs for more details (use -s to disable verbose output).") # Initialize the progress bar self.update() @@ -89,18 +102,24 @@ def update(self): remaining = self.max_line_length - prop_passed - prop_failed - progress_bar += '\033[92m#\033[0m' * prop_passed # Green - progress_bar += '\033[91m#\033[0m' * prop_failed # Red - progress_bar += ' ' * remaining # Empty space + progress_bar += Fore.GREEN + '#' * prop_passed # Green + progress_bar += Fore.RED + '#' * prop_failed # Red + progress_bar += Fore.RESET + ' ' * remaining # Empty space - # Move the cursor up 3 lines, to the beginning of the progress bar - print("\033[3A\r", end='') + # Move the cursor up 2 or 3 lines, to the beginning of the progress bar + lines_to_move_cursor = 2 if self.silent else 3 + print(f"\033[{lines_to_move_cursor}A\r", end='') print("Running Tests [{}]".format(progress_bar)) + # Space is left there intentionally to flush out the command line - print("Pass: {:2} | Fail: {:2} | Remaining: {:2} ".format( - self.passed, self.failed, remaining_tests)) - print("See logs for more details (use -v for verbose output).") + print( + Fore.GREEN + f"Pass: {self.passed:2} | " + + Fore.RED + f"Fail: {self.failed:2} | " + + Fore.RESET + f"Remaining: {remaining_tests:2}" + ) + if not self.silent: + print("See logs for more details (use -s to disable verbose output).") def test_passed(self): self.passed += 1 @@ -129,12 +148,70 @@ def fail_testcase( init_xml_message + xml_message)) +def run_subprocess( + cmd: List[str], + timeout: int, + env: Optional[dict] = None, + log_queue: Optional[queue.Queue] = None, + init_message: Optional[str] = None, + path: Optional[str] = None, + silent: bool = False, +) -> int: + """ + Simple wrapper for subprocess.run(...) with common arguments and error handling + """ + if silent: + assert not log_queue, "You can only silent subprocesses that do not redirect stdout/stderr" + stdout = subprocess.DEVNULL + stderr = subprocess.DEVNULL + else: + # None means that stdout and stderr are handled by parent, i.e., they go to console by default + stdout = None + stderr = None + + try: + if not log_queue: + subprocess.run( + cmd, + env=env, + stdout=stdout, + stderr=stderr, + timeout=timeout, + check=True + ) + else: + subprocess.run( + cmd, + env=env, + stderr=open(f"{path}.stderr.log", "w"), + stdout=open(f"{path}.stdout.log", "w"), + timeout=timeout, + check=True + ) + except subprocess.CalledProcessError as e: + if not log_queue: + print(f"{e.cmd} failed with return code {e.returncode}") + else: + assert (init_message and path) + fail_testcase( + init_message, + Fore.RED + f"Fail: see {path}.stderr.log and {path}.stdout.log", + log_queue + ) + return e.returncode + except subprocess.TimeoutExpired as e: + print(f"{e.cmd} took more than {e.timeout}") + return 5 + + return 0 + + def run_test(driver: Path, log_queue: queue.Queue) -> int: """ Run an instance of a test case. Returns: - 1 if passed, 0 otherwise. This is to increment the pass counter. + 0 if passed, non-0 otherwise. This is to increment the pass counter. """ # Replaces example_driver.c -> example.c @@ -153,113 +230,97 @@ def run_test(driver: Path, log_queue: queue.Queue) -> int: # Ensure the directory exists. log_path.parent.mkdir(parents=True, exist_ok=True) - init_message = (str(to_assemble) + "\n", - f'\n') + to_assemble_str = str(to_assemble) + init_message = (to_assemble_str + "\n", + f'\n') for suffix in [".s", ".o", ""]: log_path.with_suffix(suffix).unlink(missing_ok=True) + # Modifying environment to combat errors on memory leak + custom_env = os.environ.copy() + custom_env["ASAN_OPTIONS"] = "exitcode=0" + # Compile - compiler_result = subprocess.run( - [ - COMPILER_FILE, - "-S", str(to_assemble), - "-o", f"{log_path}.s", - ], - stderr=open(f"{log_path}.compiler.stderr.log", "w"), - stdout=open(f"{log_path}.compiler.stdout.log", "w") + return_code = run_subprocess( + cmd=[COMPILER_FILE, "-S", to_assemble_str, "-o", f"{log_path}.s"], + timeout=RUN_TIMEOUT, + env=custom_env, + log_queue=log_queue, + init_message=init_message, + path=f"{log_path}.compiler", ) - - if compiler_result.returncode != 0: - fail_testcase( - init_message, - f"Fail: see {log_path}.compiler.stderr.log " - f"and {log_path}.compiler.stdout.log", - log_queue - ) - return 0 + if return_code != 0: + return return_code # GCC Reference Output - gcc_result = subprocess.run( - [ - "riscv64-unknown-elf-gcc", - "-std=c90", - "-pedantic", - "-ansi", - "-O0", - "-march=rv32imfd", - "-mabi=ilp32d", - "-o", f"{log_path}.gcc.s", - "-S", str(to_assemble) - ] + return_code = run_subprocess( + cmd=[ + "riscv64-unknown-elf-gcc", + "-std=c90", + "-pedantic", + "-ansi", + "-O0", + "-march=rv32imfd", + "-mabi=ilp32d", + "-o", f"{log_path}.gcc.s", + "-S", to_assemble_str + ], + timeout=RUN_TIMEOUT, ) + if return_code != 0: + return return_code # Assemble - assembler_result = subprocess.run( - [ - "riscv64-unknown-elf-gcc", - "-march=rv32imfd", "-mabi=ilp32d", - "-o", f"{log_path}.o", - "-c", f"{log_path}.s" - ], - stderr=open(f"{log_path}.assembler.stderr.log", "w"), - stdout=open(f"{log_path}.assembler.stdout.log", "w") + return_code = run_subprocess( + cmd=[ + "riscv64-unknown-elf-gcc", + "-march=rv32imfd", "-mabi=ilp32d", + "-o", f"{log_path}.o", + "-c", f"{log_path}.s" + ], + timeout=RUN_TIMEOUT, + log_queue=log_queue, + init_message=init_message, + path=f"{log_path}.assembler", ) - - if assembler_result.returncode != 0: - fail_testcase( - init_message, - f"Fail: see {log_path}.assembler.stderr.log " - f"and {log_path}.assembler.stdout.log", - log_queue - ) - return 0 + if return_code != 0: + return return_code # Link - linker_result = subprocess.run( - [ - "riscv64-unknown-elf-gcc", - "-march=rv32imfd", "-mabi=ilp32d", "-static", - "-o", f"{log_path}", - f"{log_path}.o", str(driver) - ], - stderr=open(f"{log_path}.linker.stderr.log", "w"), - stdout=open(f"{log_path}.linker.stdout.log", "w") + return_code = run_subprocess( + cmd=[ + "riscv64-unknown-elf-gcc", + "-march=rv32imfd", "-mabi=ilp32d", "-static", + "-o", f"{log_path}", + f"{log_path}.o", str(driver) + ], + timeout=RUN_TIMEOUT, + log_queue=log_queue, + init_message=init_message, + path=f"{log_path}.linker", ) - - if linker_result.returncode != 0: - fail_testcase( - init_message, - f"Fail: see {log_path}.linker.stderr.log " - f"and {log_path}.linker.stdout.log", - log_queue - ) - return 0 + if return_code != 0: + return return_code # Simulate - try: - simulation_result = subprocess.run( - ["spike", "pk", log_path], - stdout=open(f"{log_path}.simulation.log", "w"), - timeout=3 - ) - except subprocess.TimeoutExpired: - print("The subprocess timed out.") - simulation_result = subprocess.CompletedProcess(args=[], returncode=1) - - if simulation_result.returncode != 0: - fail_testcase( - init_message, - f"Fail: simulation did not exit with exitcode 0", - log_queue - ) - return 0 - else: - init_print_message, init_xml_message = init_message - log_queue.put((init_print_message + "\t> Pass", - init_xml_message + "\n")) + return_code = run_subprocess( + cmd=["spike", "pk", log_path], + timeout=RUN_TIMEOUT, + log_queue=log_queue, + init_message=init_message, + path=f"{log_path}.simulation", + ) + if return_code != 0: + return return_code - return 1 + init_print_message, init_xml_message = init_message + log_queue.put(( + f"{init_print_message}\t> " + Fore.GREEN + "Pass", + f"{init_xml_message}\n" + )) + + return 0 def empty_log_queue( @@ -267,22 +328,27 @@ def empty_log_queue( verbose: bool = False, progress_bar: ProgressBar = None ): + """ + Empty log queue while logs are sent to XML file and terminal/progress bar + """ while not log_queue.empty(): print_msg, xml_message = log_queue.get() + processed_msg = re.sub(r"(/workspaces/).+?/([compiler_tests|bin])", r"\1\2", print_msg) if verbose: print(print_msg) else: - if "Pass" in print_msg: + if "Pass" in processed_msg: progress_bar.test_passed() - elif "Fail" in print_msg: + elif "Fail" in processed_msg: progress_bar.test_failed() with open(J_UNIT_OUTPUT_FILE, "a") as xml_file: xml_file.write(xml_message) -def main(): + +def main() -> int: parser = argparse.ArgumentParser() parser.add_argument( "dir", @@ -292,7 +358,6 @@ def main(): help="(Optional) paths to the compiler test folders. Use this to select " "certain tests. Leave blank to run all tests." ) - parser.add_argument( "-m", "--multithreading", action="store_true", @@ -301,10 +366,10 @@ def main(): "but order is not guaranteed. Should only be used for speed." ) parser.add_argument( - "-v", "--verbose", + "-s", "--short", action="store_true", default=False, - help="Enable verbose output into the terminal. Note that all logs will " + help="Disable verbose output into the terminal. Note that all logs will " "be stored automatically into log files regardless of this option." ) parser.add_argument( @@ -312,6 +377,22 @@ def main(): action="version", version=f"BetterTesting {__version__}" ) + # Coverage cannot be perfomed without rebuilding the compiler + group = parser.add_mutually_exclusive_group(required=False) + group.add_argument( + "--no_clean", + action="store_true", + default=False, + help="Do no clean the repository before testing. This will make it " + "faster but it can be safer to clean if you have any compilation issues." + ) + group.add_argument( + "--coverage", + action="store_true", + default=False, + help="Run with coverage if you want to know which part of your code is " + "executed when running your compiler. See docs/coverage.md" + ) args = parser.parse_args() try: @@ -321,7 +402,32 @@ def main(): Path(OUTPUT_FOLDER).mkdir(parents=True, exist_ok=True) - subprocess.run(["make", "-C", PROJECT_LOCATION, "bin/c_compiler"]) + # Clean the repo + if not args.no_clean: + return_code = run_subprocess( + cmd=["make", "-C", PROJECT_LOCATION, "clean"], + timeout=BUILD_TIMEOUT, + silent=args.short, + ) + if return_code != 0: + return return_code + + # Run coverage if needed + if args.coverage: + shutil.rmtree(COVERAGE_FOLDER, ignore_errors=True) + cmd = ["make", "-C", PROJECT_LOCATION, "with_coverage"] + + # Otherwise run the main make command for building and testing + else: + cmd = ["make", "-C", PROJECT_LOCATION, "bin/c_compiler"] + + return_code = run_subprocess( + cmd=cmd, + timeout=BUILD_TIMEOUT, + silent=args.short, + ) + if return_code != 0: + return return_code with open(J_UNIT_OUTPUT_FILE, "w") as f: f.write('\n') @@ -331,7 +437,13 @@ def main(): drivers = sorted(drivers, key=lambda p: (p.parent.name, p.name)) log_queue = queue.Queue() results = [] - progress_bar = ProgressBar(len(drivers)) + + try: + progress_bar = ProgressBar(len(drivers), silent=args.short) + except ValueError as e: # Error comes from TTY when running in Github Actions + assert not args.short,\ + "Progress bar cannot be initialised, so program has to be verbose" + progress_bar = None if args.multithreading: with ThreadPoolExecutor() as executor: @@ -340,22 +452,37 @@ def main(): for future in as_completed(futures): results.append(future.result()) - empty_log_queue(log_queue, args.verbose, progress_bar) + empty_log_queue(log_queue, not args.short, progress_bar) else: for driver in drivers: result = run_test(driver, log_queue) results.append(result) - empty_log_queue(log_queue, args.verbose, progress_bar) + empty_log_queue(log_queue, not args.short, progress_bar) - passing = sum(results) + passing = sum([1 if result == 0 else 0 for result in results]) total = len(drivers) with open(J_UNIT_OUTPUT_FILE, "a") as f: f.write('\n') - print("\n>> Test Summary: {} Passed, {} Failed".format( - passing, total-passing)) + if not args.short: + print( + "\n>> Test Summary: " + + Fore.GREEN + f"{passing} Passed, " + Fore.RED + f"{total-passing} Failed" + ) + + # Run coverage if needed + if args.coverage: + return_code = run_subprocess( + cmd=["make", "-C", PROJECT_LOCATION, "coverage"], + timeout=BUILD_TIMEOUT, + silent=args.short, + ) + if return_code != 0: + return return_code + + return 0 if __name__ == "__main__": diff --git a/test.sh b/test.sh old mode 100755 new mode 100644 index 69823a2..6b906ac --- a/test.sh +++ b/test.sh @@ -1,3 +1,3 @@ #!/bin/bash -. scripts/test.sh +./scripts/test.py $@