Use continues benchmark to track long term performance

[ringsaturn]

https://github.com/marketplace/actions/continuous-benchmark could save each commit’s benchmark results to a file, and come with a nice HTML page to view trends, I believe it will be useful for performance improvements.

[jannikmi]

https://github.com/benchmark-action/github-action-benchmark/blob/master/examples/pytest/README.md

[jannikmi]

AFAIK the benchmark will run on GitHub actions for every build. One question that comes to my mind is if the performance is really comparable when the runtime environment (computational setup, CPU, RAM etc.) of the benchmark cannot be controlled?
@ringsaturn do you know about this?

[ringsaturn]

There are two main variables

Hardware

Sometimes the performance could be very different(maybe the CPU load caused), take tzf as example https://ringsaturn.github.io/tzf/ the performance will regression after many benchmark has runned.

That’s part we can’t control. Just run many times and wait the trends came to the regression.

Project based env

Sometimes we need to test under multi env, like OS/verison, and that we can set in GA.

An internal project uploads different env benchmark data to different folder, such as Go’s version:

      - name: Continuous Benchmark
        uses: benchmark-action/[email protected]
        with:
          name: Go Benchmark
          tool: 'go'
          output-file-path: ${{ matrix.go }}_benchmark_result.txt
          github-token: ${{ secrets.TOKEN }}
          gh-repository: 'github.com/xxxxx/yyyyy'
          auto-push: true
          alert-threshold: '200%'
          comment-on-alert: true
          fail-on-alert: false
          gh-pages-branch: "benchmark"
          benchmark-data-dir-path: "benchmark/${{ matrix.go }}/"
          alert-comment-cc-users: '@ringsaturn'

The save path need a unique path and then we can see diffent benchmark results:

.
└── benchmark
    ├── 1.17
    │   ├── data.js
    │   └── index.html
    ├── 1.18
    │   ├── data.js
    │   └── index.html
    └── 1.19
        ├── data.js
        └── index.html

If we need a page to view, we can add index.html to gh-pages branch, and then show a list of benchmark sets.

[ringsaturn]

Another problem I know is that if PR isn’t opened by owner, secret token couldn’t be accessed like ringsaturn/tz-benchmark#7

[ringsaturn]

Here is a screenshot from our internal project. A few days ago we add a performance improvement feature, which is obviously.

[jannikmi]

For future reference: the scripts folder already contains scripts for time benchmarks