Skip to content

Benchmark

Jump to bottom
yacoubii edited this page Sep 3, 2024 · 1 revision

Evaluation and benchmark are available in two different jobs:

  • benchmark: performs normal benchmark and gives out the results.
  • benchmark-test: performs evaluation based on benchmark results.

Full CI/CD pipeline is described as follows:

image.png

Benchmark and evaluation

Files:

  • Benchmark dependencies installation script: scripts/benchmark/benchmark-dependencies.sh
  • Benchmark bash script: scripts/benchmark/classic-benchmark.sh
  • Benchmark evaluation script: scripts/benchmark/eval-benchmark.sh
  • Benchmark environment variables: scripts/benchmark/benchmark.env.sh
  • Benchmark interface script: script/benchmark.sh

Benchmark can be launched by executing the following command:

  • ./scripts/benchmark.sh "classic": classic headless benchmark
    • ./scripts/benchmark.sh "classic" "visual": classic visual benchmark
    • ./scripts/benchmark.sh "classic" "headless": classic headless benchmark
  • ./scripts/benchmark.sh "evaluation": headless evaluation based on benchmark
    • ./scripts/benchmark.sh "evaluation" "visual": visual evaluation based on benchmark
    • ./scripts/benchmark.sh "evaluation" "headless": headless evaluation based on benchmark

Using yarn:

  • yarn benchmark -- classic: launches benchmark
  • yarn benchmark -- evaluation: launches evaluation based on benchmark results

Benchmark

  • Three files will be outputted:

    • <filename>.results: contains benchmark result.

    image.png

    • <filename>.context: contains benchmark execution context.

    image.png

    • <filename>.data: contains palindrome.js configuration as well as used data.

    image.png

  • Cleaned benchmark so it only displays the results and the total time spent in a table.

image.png

Evaluation

Approach 1: Remote expected values

  • Deployed an Express web server (https://palindrome-production.onrender.com/api/requirements) just to simulate the values that are coming from "production". This web server return just a json object that contains expected values: {"basicFps":10,"basicMs":1000,"workersFps":15,"workersMs":100} .
  • These values will be compared to the values found by benchmark.
  • This approach needs EXPECTED_VALUES_SOURCE=remote and EXPECTED_VALUES_REMOTE_SOURCE_URL=https://palindrome-production.onrender.com/api/requirements\ to be exported.

Approach 2: Median computation

  • Launch n times the benchmark test and compute the median values.
  • Median results are stored in a file (MEDIAN_VALUES_OUTPUT_FILE).
  • n+1 benchmark launch is the actual test. Its results will be compared to the median values.
  • MEDIAN_ITERATIONS needs to be exported and defined as the number of iterations to calculate median.
  • RECALCULATE_MEDIAN needs to be set to false if there is no need to recalculate median. Previous median results will be used.

Benchmark results

Successful execution:

image.png

Unsuccessful execution:

image.png

Environment variables reference

Variable Description Type Options
PALINDROME_BENCH_PALINDROME_TYPE Choose palindrome type (dev: with sidebar, basic: without sidebar) String dev | basic
PALINDROME_BENCH_IS_BENCHMARK Activate or deactivate benchmark Boolean

-
PALINDROME_BENCH_DURATION Benchmark duration in minutes for each version (default and web workers) Number

-
PALINDROME_BENCH_USE_CASE_NAME The name of the use case to benchmark String benchLoadTest or any other user defined usecase.
PALINDROME_BENCH_WORKERS_RESOURCES_LEVEL Percentage of the resources allowed for web workers to use Number

-
PALINDROME_BENCH_OUTPUT

The name of the file that will contain the results. Three files should be outputted:

  • <filename>.results: contains benchmark result.
  • <filename>.context: contains benchmark execution context.
  • <filename>.data: contains palindrome.js configuration as well as used data.
 String

-
PALINDROME_BENCH_DL_DIRECTORY The directory name of Firefox downloads folder. String

-
PALINDROME_BENCH_GPU Enable or disable GPU usage Boolean

-
PALINDROME_BENCH_HEADLESS Enable or disable headless mode Boolean

-
PALINDROME_BENCH_WEBSERVER Execute Palindrome.js with or without web server. In case of true, parcel will be used, otherwise the build HTML file will be used. Boolean

-
PALINDROME_BENCH_MEDIAN_ITERATIONS Number of iterations before choosing the median based on iterations results Number

-
PALINDROME_BENCH_EXPECTED_VALUES_SOURCE On evaluation, expected values can be chosen locally (based on median computation) or remotely via a remote data source. String local | remote
PALINDROME_BENCH_EXPECTED_VALUES_REMOTE_SOURCE_URL In case of remote expected values then a remote data source URL should be defined. String

-
PALINDROME_BENCH_MEDIAN_VALUES_OUTPUT_FILE The file that will contain the median computation results String

-
PALINDROME_BENCH_RECALCULATE_MEDIAN In case of an already computed median, it is possible to use the old median by using the old generated file. Boolean true | false
PALINDROME_BENCH_BROWSER The browser on which the benchmark will run String firefox | chromium

Notes:

  • Chromium should not be used with GPU based benchmark. Please use Firefox instead.
  • Firefox should not be used with CPU based benchmark. Please use Chromium instead.
  • Headless Chromium can only be used within Gitlab CI/CD pipeline. For headless execution, please use Firefox.
Clone this wiki locally