Skip to content

Latest commit

 

History

History
255 lines (195 loc) · 9.64 KB

quickstart-cmake.md

File metadata and controls

255 lines (195 loc) · 9.64 KB
Raw

Quickstart with CMake

This tutorial describes how to set up your development environment with CMake as the build system, and how to get a simple fuzz test up and running. We recommend this tutorial if you're new to FuzzTest or if you need a quick refresher.

Definitions

The library can work in several different modes.

  • Unit test mode (default): FUZZ_TESTs are built without coverage instrumentation and run with random inputs for short period of time along with regular unit TESTs.

  • Fuzzing mode (controlled by -DFUZZTEST_FUZZING_MODE=on): FUZZ_TESTs are built with coverage instrumentation and run individually and indefinitely or for specified amount of time.

  • Compatibility mode (controlled by -DFUZZTEST_COMPATIBILITY_MODE=libfuzzer): like Fuzzing mode, but an external engine is used instead FuzzTest's built in engine. Currently libFuzzer is supported.

    Warning: compatibility mode is experimental and does not guarantee full FuzzTest features.

Overall rules

We provide a default CMake option for unittest mode, the instructions are straightforward and work like googletest.

To run in a fuzztest mode, you need to make sure all targets are compiled in a special way.

  • -g -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -UNDEBUG
  • -fsanitize-coverage=inline-8bit-counters -fsanitize-coverage=trace-cmp
  • -fsanitize=address -DADDRESS_SANITIZER (optionally enabling sanitizer)

The FuzzTest framework doesn't need to be built with these flags, only the target project.

For all projects to have correct flags set, use a CMake macro fuzztest_setup_fuzzing_flags().

For a concrete example, see the root CMakeLists.txt. First, we build framework without additional flags and then build the rest of translation units with coverage flags. Otherwise we might get into recursive issues, i.e. fuzztest framework will be fuzzing itself.

Easiest way to make sure you build everything correctly is to invoke add_subdirectory first in your project on fuzztest.

Prerequisites

To use FuzzTest, you'll need:

  • A Linux-based operating system.
  • Clang. GCC is not yet supported. Unittest mode mode should work but it's not yet tested.
  • CMake.

Set up a CMake workspace

First, create a directory where your project will reside:

$ mkdir first_fuzz_project && cd first_fuzz_project
$ git clone https://github.com/google/fuzztest.git

Create a CMakeLists.txt.

cmake_minimum_required(VERSION 3.19)
project(first_fuzz_project)

# GoogleTest requires at least C++17
set(CMAKE_CXX_STANDARD 17)

add_subdirectory(fuzztest)

FuzzTest CMake already handles its dependencies on its own: RE2, Abseil. We highly recommend using GoogleTest as well.

Create and run a fuzz test

With the CMake workspace set up, you can start using FuzzTest. Let's create a trivial example to make sure everything runs correctly.

Create a file named first_fuzz_test.cc in the directory first_fuzz with the following contents:

#include "fuzztest/fuzztest.h"
#include "gtest/gtest.h"

TEST(MyTestSuite, OnePlustTwoIsTwoPlusOne) {
  EXPECT_EQ(1 + 2, 2 + 1);
}

void IntegerAdditionCommutes(int a, int b) {
  EXPECT_EQ(a + b, b + a);
}
FUZZ_TEST(MyTestSuite, IntegerAdditionCommutes);

The file contains two tests in the test suite named MyTestSuite. The first one is a unit test named OnePlustTwoIsTwoPlusOne asserting that integer addition commutes for two specific values.

The second test is a fuzz test that captures the commutative property of integer addition more generally. The test consists of a property function with a suggestive name IntegerAdditionCommutes, and the test registration using the macro FUZZ_TEST. The property function asserts the commutative property for two generic integer parameters.

To build and run the tests, add to the file CMakeLists.txt the following content:

...

enable_testing()

include(GoogleTest)

add_executable(
  first_fuzz_test
  first_fuzz_test.cc
)

# If you have other dependencies than FuzzTest, link them:
# target_link_libraries(first_fuzz_test PRIVATE other_deps)

link_fuzztest(first_fuzz_test)
gtest_discover_tests(first_fuzz_test)

This defines a C++ test binary and links it with the FuzzTest and GoogleTest libraries, as well as the version of the default GoogleTest main that supports FuzzTest.

Unit test mode

There are several ways to run the tests: in the unit test mode and in the fuzzing mode (see above). The unit test mode runs both test for a short time without sanitizer and coverage instrumentation:

$ mkdir build && cd build
$ CC=clang CXX=clang++ cmake \
  -DCMAKE_BUILD_TYPE=RelWithDebug ..
$ cmake --build .
$ ./first_fuzz_test

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from MyTestSuite
[ RUN      ] MyTestSuite.OnePlustTwoIsTwoPlusOne
[       OK ] MyTestSuite.OnePlustTwoIsTwoPlusOne (0 ms)
[ RUN      ] MyTestSuite.IntegerAdditionCommutes
[       OK ] MyTestSuite.IntegerAdditionCommutes (13 ms)
[----------] 2 tests from MyTestSuite (13 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (13 ms total)
[  PASSED  ] 2 tests.

Fuzzing mode

The fuzzing mode runs a single specified fuzz test with sanitizer and coverage instrumentation. It keeps running the test with different input values until it finds a crash or it is explicitly terminated by the user.

Before add_executable(first_fuzz_test ...), add the specified flags:

fuzztest_setup_fuzzing_flags()

Then run:

$ CC=clang CXX=clang++ cmake \
  -DCMAKE_BUILD_TYPE=RelWithDebug \
  -DFUZZTEST_FUZZING_MODE=on ..
$ cmake --build .
$ ./first_fuzz_test --fuzz=MyTestSuite.IntegerAdditionCommutes

[.] Sanitizer coverage enabled. Counter map size: 21290, Cmp map size: 262144
Note: Google Test filter = MyTestSuite.IntegerAdditionCommutes
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MyTestSuite
[ RUN      ] MyTestSuite.IntegerAdditionCommutes
[*] Corpus size:     1 | Edges covered:    131 | Fuzzing time:    504.798us | Total runs:  1.00e+00 | Runs/secs:     1
[*] Corpus size:     2 | Edges covered:    133 | Fuzzing time:    934.176us | Total runs:  3.00e+00 | Runs/secs:     3
[*] Corpus size:     3 | Edges covered:    134 | Fuzzing time:   2.384383ms | Total runs:  5.30e+01 | Runs/secs:    53
[*] Corpus size:     4 | Edges covered:    137 | Fuzzing time:   2.732274ms | Total runs:  5.40e+01 | Runs/secs:    54
[*] Corpus size:     5 | Edges covered:    137 | Fuzzing time:   7.275553ms | Total runs:  2.48e+02 | Runs/secs:   248
[*] Corpus size:     6 | Edges covered:    137 | Fuzzing time:   21.97155ms | Total runs:  9.12e+02 | Runs/secs:   912
[*] Corpus size:     7 | Edges covered:    137 | Fuzzing time: 166.721522ms | Total runs:  8.49e+03 | Runs/secs:  8491
[*] Corpus size:     8 | Edges covered:    146 | Fuzzing time: 500.398497ms | Total runs:  2.77e+04 | Runs/secs: 27741
[*] Corpus size:     9 | Edges covered:    146 | Fuzzing time: 500.821386ms | Total runs:  2.77e+04 | Runs/secs: 27742
[*] Corpus size:    10 | Edges covered:    147 | Fuzzing time: 2.597553524s | Total runs:  1.75e+05 | Runs/secs: 87476
^C

Congratulations! You're now all set for fuzzing with FuzzTest.

Compatibility mode

Warning: compatibility mode is experimental and does not guarantee full FuzzTest features.

$ CC=clang CXX=clang++ cmake \
  -DCMAKE_BUILD_TYPE=RelWithDebug \
  -DFUZZTEST_COMPATIBILITY_MODE=libfuzzer ..
$ cmake --build .
$ ./first_fuzz_test --fuzz=MyTestSuite.IntegerAdditionCommutes

Note: Google Test filter = MyTestSuite.IntegerAdditionCommutes
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MyTestSuite
[ RUN      ] MyTestSuite.IntegerAdditionCommutes
FUZZTEST_PRNG_SEED=OffQXb8u5_vZtH4-7wgVOLu_HNAhPIbLz7CFF13u3nk
INFO: found LLVMFuzzerCustomMutator (0x55edfe61cac0). Disabling -len_control by default.
INFO: libFuzzer ignores flags that start with '--'
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1109364873
INFO: Loaded 1 modules   (18692 inline 8-bit counters): 18692 [0x55edfe975548, 0x55edfe979e4c), 
INFO: Loaded 1 PC tables (18692 PCs): 18692 [0x55edfe979e50,0x55edfe9c2e90), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 17 ft: 18 corp: 1/1b exec/s: 0 rss: 38Mb
#3	NEW    cov: 104 ft: 108 corp: 2/66b lim: 4096 exec/s: 0 rss: 38Mb L: 65/65 MS: 1 Custom-
#4	NEW    cov: 105 ft: 112 corp: 3/121b lim: 4096 exec/s: 0 rss: 38Mb L: 55/65 MS: 1 Custom-
#5	NEW    cov: 105 ft: 113 corp: 4/196b lim: 4096 exec/s: 0 rss: 38Mb L: 75/75 MS: 1 Custom-
#16	REDUCE cov: 105 ft: 113 corp: 4/195b lim: 4096 exec/s: 0 rss: 38Mb L: 54/75 MS: 1 Custom-
#41	NEW    cov: 105 ft: 114 corp: 5/247b lim: 4096 exec/s: 0 rss: 38Mb L: 52/75 MS: 5 Custom-Custom-Custom-Custom-Custom-
#65	NEW    cov: 106 ft: 115 corp: 6/312b lim: 4096 exec/s: 0 rss: 38Mb L: 65/75 MS: 4 Custom-Custom-Custom-Custom-
#69	NEW    cov: 108 ft: 119 corp: 7/365b lim: 4096 exec/s: 0 rss: 38Mb L: 53/75 MS: 4 Custom-Custom-Custom-Custom-

Next steps

  • This tutorial covered the basic setup of the build environment with CMake. To learn more about the FuzzTest framework and how to use it, check out the Codelab.
  • Explore the rest of the documentation.