Quickstart draft

doc/quickstart.md

@@ -0,0 +1,93 @@
+# A Quickstart Guide to Proper Benchmarking with BenchExec
+
+This guide provides a brief summary of instructions to set up reliable
+benchmark measurements using BenchExec and important points to consider. It is
+meant for users who either want to set up benchmarking from scratch or already
+have a simple setup using, e.g., `time`, `timeout`, `taskset`, `ulimit`, etc.
+and will guide you how to use `runexec`, a simple "drop-in" replacement for
+these tools.
+
+## Guiding Example
+
+As an example, suppose that we want to benchmark the executable `program` with
+arguments `--foo` and `--bar` on the input files `input_1.in` to `input_9.in`.
+To measure the runtime of the tool, one may run
+```
+$ /usr/bin/time program --foo input_1.in
+$ /usr/bin/time program --bar input_1.in
+$ /usr/bin/time program --foo input_2.in
+...
+```
+etc. and note the results. In case resource limitations are required (e.g. 
+limiting to 1 CPU and 60 sec of wallclock time), the call might be
+changed to
+```
+$ taskset -c 0 timeout 60s /usr/bin/time program ...
+```
+and the list of calls might be organized by a script.
+
+If your current setup looks like this or you are thinking about a similar
+setup, we strongly recommend following this guide for a much more reliable
+process.
+
+## Benchmarking with BenchExec
+
+### Step 1. Install BenchExec
+
+Depending on your distribution, the setup steps vary slightly.
+On Debian/Ubuntu and similar, you can
+```
+sudo add-apt-repository ppa:sosy-lab/benchmarking
+sudo apt update && sudo apt install benchexec
+```
+otherwise, try the pip-based setup
+```
+pip3 install --user benchexec[systemd] coloredlogs
+```
+
+TODO Docker
+
+See [INSTALL.md](the installation guide) for further details and
+troubleshooting. You should be able to run the command
+`python3 -m benchexec.check_cgroups` without issues.
+
+### Step 2. Consider the Setup of Your Benchmark
+
+Consider and document the benchmarking process beforehand. In particular, think
+about which executions you want to measure, what resource limits should be
+placed on the benchmarked tool(s), such as CPU time, CPU count, memory, etc.
+Also consider how timeouts should be treated.
+
+In case you want to execute multiple benchmarks in parallel, think about how to
+deal with shared resources (e.g. the memory bus).
+
+For more complicated setups, please also refer to the
+[benchmarking setup guide](benchmarking.md).
+
+### Step 3. Gather Measurements using runexec
+
+Using the example from above, suppose that we want to limit the process to 60s
+wall time, 1 GB of memory, and cpu core 0. Then, simply run
+```
+$ runexec --quiet --walltimelimit 60s --memlimit 1GB --cores 0 --output output_1_foo.log \
+  program --foo input_1.in
+```
+This executes `program --foo input_1.in` and prints measurements to standard
+output, such as walltime, cputime, memory, I/O, etc. in a simple to read and
+parse format. The output of program is redirected to `output_1_foo.log`.
+
+The tool `runexec` offers several other features, run `runexec --help` for
+further information or refer to the [documentation](runexec.md).
+
+## Further Resources
+
+BenchExec provides much more beyond this core feature of reliable measurements.
+
+The tool `benchexec` provides ways to specify the complete set of benchmarks
+and allows you to run thousands of tool / input combinations with a single
+command as well as gather, aggregate, and display all the data at the same
+time. See [the documentation](benchexec.md) for further details.
+
+Additionally, `runexec` can also be accessed through a simple
+[Python API](runexec.md#integration-into-other-benchmarking-frameworks) with
+which you can integrate it programmatically in your framework.