This document explains how to benchmark ASP.NET Core changes in the ASP.NET Core perf lab. Currently, only members of the .NET team have the permissions necessary to trigger performance runs on the official infrastructure.
It describes how to trigger benchmarks using the Crank CLI and the Pull-Request bot.
This is the recommended method as it is quick, flexible and doesn't require a pre-existing pull-request.
The Crank dotnet tool is required to start benchmarks remotely on the available performance infrastructure.
Install Crank with the following command:
dotnet tool install Microsoft.Crank.Controller --version "0.2.0-*" --global
For more information about Crank please refer to the Crank GitHub repository.
- In the PowerBI dashboard select a scenario that you would like to benchmark and the environment to use.
- At the bottom of the page there is a table that lists the command lines that were used to execute this benchmark. Pick one.
- Run the command in a shell. This will start the benchmark remotely on the same target machines as the charts.
This command line contains the specific version of the runtimes and SDK that were used, which makes it deterministic. This means that if you want to investigate a regression you can pick two points on a chart and re-run these to confirm the regression. You can also change each version to understand if the regression is coming from ASP.NET or the .NET Runtime.
Using the same Crank command line you can also test the impact of local changes.
- Build your changes in
release
for the desired architecture/os (if it matters). - Add
--application.options.outputFiles c:\build\release
and replace the folder by the one containing your changes. - Run the updated Crank command line
Use this technique when you want to benchmark a community contributed PR or when you don't have the changes available locally.
Here is an example of a PR that was benchmarked using this bot.
- In the pull-request, add
/benchmark
in the comments. - A text similar to this will be displayed (it might take up to 10 minutes):
Crank Pull Request Bot
/benchmark <benchmark[,...]> <profile[,...]> <component,[...]> <arguments>
Benchmarks:
plaintext
: TechEmpower Plaintext Scenario - ASP.NET Platform implementationjson
: TechEmpower JSON Scenario - ASP.NET Platform implementationfortunes
: TechEmpower Fortunes Scenario - ASP.NET Platform implementationyarp
: YARP - http-http with 10 bytesmvcjsoninput2k
: Sends 2Kb Json Body to an MVC controller
Profiles:
aspnet-perf-lin
: INTEL/Linux 12 Coresaspnet-perf-win
: INTEL/Windows 12 Coresaspnet-citrine-lin
: INTEL/Linux 28 Coresaspnet-citrine-win
: INTEL/Windows 28 Coresaspnet-citrine-ampere
: Ampere/Linux 80 Coresaspnet-citrine-amd
: AMD/Linux 48 Cores
Components:
kestrel
(Microsoft.AspNetCore.Server.Kestrelsrc\Servers\Kestrel\build.cmd
)mvc
(Microsoft.AspNetCore.Mvcsrc\Mvc\build.cmd
)routing
(Microsoft.AspNetCore.Routingsrc\Http\build.cmd
)
See prbenchmarks.aspnetcore.config.yml for more details.
Arguments: any additional arguments to pass through to crank, e.g. --variable name=value
Where:
- Benchmark is the application that will be stressed.
- Profile is the environment where the application will run on.
- Component is the ASP.NET project under test that will be built and contains your changes.
Important
Any changes outside of the "component" projects will not be benchmarked.
Create a new comment with the correct benchmarks, profiles, and components to build. For instance:
/benchmark json aspnet-citrine-lin kestrel
Once the benchmark has started (can take up to 10 minutes) the bot adds a comment to the PR. Then, when the benchmark is finished the results are added in a separate comment for both your changes and the source branch so you can compare the metrics you care about.
For more information please look the Benchmarks repository where we maintain most of the benchmarks, and the Crank repository.