How can I generate a profraw file based on the output of my custom fuzzer?

[thealberto]

Hi all,
I'm trying to get the code coverage of a simple test fuzzer but I cannot obtain a `.profraw" file.

I have enabled -fprofile-instr-generate and -fcoverage-mapping in the BUILD.gn file.

I have tried the following so far

➜  src git:(ccbeb246d2fd) autoninja -C out/asan vpx_custom_fuzzer -j15 -v && ./out/asan/vpx_custom_fuzzer --fuzz=
ninja: Entering directory `out/asan'
[0/1] ../../buildtools/linux64/gn --root=../.. -q --regeneration gen .
[1/2] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/third_party/libvpx/vpx_custom_fuzzer/vpx_custom_fuzzer.o.d -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -DMEMORY_TOOL_REPLACES_ALLOCATOR -DADDRESS_SANITIZER -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -DCR_CLANG_REVISION=\"llvmorg-18-init-12938-geb1d5065-1\" -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -DCR_LIBCXX_REVISION=4ab8957af3274befed723defd5333873478ca7c6 -DCR_SYSROOT_KEY=20230611T210420Z-2 -DSANITIZER_COVERAGE -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_56 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_56 -DBASE_USE_PERFETTO_CLIENT_LIBRARY=1 -DUNIT_TEST -DGTEST_API_= -DGTEST_HAS_POSIX_RE=0 -DGTEST_LANG_CXX11=1 -DGTEST_HAS_TR1_TUPLE=0 -DGTEST_HAS_ABSL=1 -I../.. -Igen -I../../buildtools/third_party/libc++ -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/libvpx/source/libvpx -I../../third_party/libvpx/source/config/linux/x64 -I../../third_party/fuzztest/src -I../../third_party/abseil-cpp -I../../third_party/re2/src -I../../third_party/googletest/custom -I../../third_party/googletest/src/googletest/include -fprofile-instr-generate -fcoverage-mapping -Wall -Wextra -Wimplicit-fallthrough -Wextra-semi -Wunreachable-code-aggressive -Wthread-safety -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wenum-compare-conditional -Wno-ignored-pragma-optimize -Wno-deprecated-builtins -Wno-bitfield-constant-conversion -Wno-deprecated-this-capture -Wno-invalid-offsetof -Wno-vla-extension -Wno-thread-safety-reference-return -Wshadow -Werror -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing -fstack-protector -funwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -mllvm -split-threshold-for-reg-with-hint=0 -ffp-contract=off -fcomplete-member-pointers -m64 -msse3 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -O2 -fdata-sections -ffunction-sections -fno-unique-section-names -fno-math-errno -fno-omit-frame-pointer -gdwarf-4 -g1 -gcolumn-info -fsanitize-coverage=inline-8bit-counters -mllvm -sanitizer-coverage-prune-blocks=1 -fsanitize=address -fno-sanitize-address-globals-dead-stripping -fvisibility=hidden -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ref-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang check-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang check-raw-ptr-to-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-fix-crbug-1449812 -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=/third_party/dawn/ -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Xclang -plugin-arg-find-bad-constructs -Xclang check-raw-ptr-fields -Xclang -plugin-arg-find-bad-constructs -Xclang check-raw-ref-fields -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -isystem../../build/linux/debian_bullseye_amd64-sysroot/usr/include/glib-2.0 -isystem../../build/linux/debian_bullseye_amd64-sysroot/usr/lib/x86_64-linux-gnu/glib-2.0/include -Wno-sign-compare -Wno-unused-private-field -Wno-unreachable-code-return -Wno-unused-but-set-variable -Wno-shadow -Wno-unused-const-variable -Wno-unused-function -msse4.2 -std=c++20 -Wno-trigraphs -gsimple-template-names -fno-exceptions -fno-rtti -nostdinc++ -isystem../../third_party/libc++/src/include -isystem../../third_party/libc++abi/src/include --sysroot=../../build/linux/debian_bullseye_amd64-sysroot -fvisibility-inlines-hidden -c ../../third_party/libvpx/tests/fuzzer/vpx_custom_fuzzer.cc -o obj/third_party/libvpx/vpx_custom_fuzzer/vpx_custom_fuzzer.o
[2/2] "python3" "../../build/toolchain/gcc_link_wrapper.py" --output="./vpx_custom_fuzzer" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -Werror -fuse-ld=lld -Wl,--fatal-warnings -Wl,--build-id -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--color-diagnostics -Wl,--undefined-version -Wl,--no-call-graph-profile-sort -m64 -no-canonical-prefixes -Wl,-O2 -Wl,--gc-sections -nostdlib++ --sysroot=../../build/linux/debian_bullseye_amd64-sysroot -fsanitize=address -fsanitize-coverage=inline-8bit-counters -rdynamic -pie -Wl,--disable-new-dtags -fsanitize=address -fsanitize-coverage=inline-8bit-counters -Wl,-u_sanitizer_options_link_helper -o "./vpx_custom_fuzzer" -Wl,--start-group @"./vpx_custom_fuzzer.rsp"  -Wl,--end-group  ../../third_party/llvm-build/Release+Asserts/lib/clang/18/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 obj/third_party/rust/cxx/v1/lib/libcxx-cxx-1.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libprofiler_builtins.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_abort.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_std.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunicode_width.rlib local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest.rlib
[.] Sanitizer coverage enabled. Counter map size: 149258, Cmp map size: 262144
Note: Google Test filter = CustomFuzzTest.encode
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CustomFuzzTest
[ RUN      ] CustomFuzzTest.encode
FUZZTEST_PRNG_SEED=1nCr7HCbkv-a4GLeBHEo2jIX1QRrEXacbh6otlnKVwE
[*] Corpus size:     1 | Edges covered:     11 | Fuzzing time:        245.714us | Total runs:  1.00e+00 | Runs/secs:  4069 | Max stack usage:        0
^C
[.] Fuzzing was terminated.

=================================================================
=== Fuzzing stats

Elapsed time: 11.782110656s
Total runs: 802776
Edges covered: 11
Total edges: 149258
Corpus size: 1
Max stack used: 0

[       OK ] CustomFuzzTest.encode (11782 ms)
[----------] 1 test from CustomFuzzTest (11782 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (11782 ms total)
[  PASSED  ] 1 test.
➜  src git:(ccbeb246d2fd) find . -iname "*profraw"                                                               
./third_party/dawn/third_party/dxc/test/tools/llvm-profdata/Inputs/c-general.profraw
➜  src git:(ccbeb246d2fd)

[thealberto]

Hi,
I managed to obtain the file following this documentation. The problem is that if I run the fuzzer with --fuzz it run forever even if I use --fuzz_for=3s so the profraw is never generated.

Any idea on how to solve it?

[lszekeres]

Thanks for flagging this!

You need to use either --fuzz or --fuzz_for, they are mutually exclusive.

We need to check and alert if both flags are set, or maybe we need to change the semantics so that they are not mutually exclusive.

[thealberto]

Hi @lszekeres ,
Happy to help. In my opinion they shouldn't be mutually exclusive so a user could decide which fuzztest to run and how long.

What do you think?

Thanks

[lszekeres]

Sorry, I misspoke, they can be combined:

fuzztest/fuzztest/init_fuzztest.cc

Lines 57 to 61 in 787fb76

	"Runs all fuzz tests in fuzzing mode for the specified duration. Can "
	"be combined with --" FUZZTEST_FLAG_PREFIX
	"fuzz to select a single fuzz tests, or "
	"with --" FUZZTEST_FLAG_PREFIX
	"filter to select a subset of fuzz tests. Recommended "

I initially understood that the problem was that when you use the two flags together the process doesn't stop. But I think that's not what you meant.

[thealberto]

No worries. In the end I now have coverage via the coverage.py file so

If I remember properly even with fuzz_for=3s the fuzztest didn't stop.

[thealberto]

@lszekeres ,
I had some time and I tried again.

I'm running the fuzzer in this way:

➜  src git:(ccbeb246d2fd) python3 tools/code_coverage/coverage.py vpx_custom_fuzzer \             
      -b out/asan -o out/report \
      -c 'out/asan/vpx_custom_fuzzer --fuzz= --fuzz_for=10s' -f third_party/libvpx

As output I have the following

FUZZTEST_PRNG_SEED=dYUrvZw8g1cSzbKjA6fHBn5d-C8RvzIjezvrE89-4K0
INFO: found LLVMFuzzerCustomMutator (0x556d774ddf00). Disabling -len_control by default.
INFO: libFuzzer ignores flags that start with '--'
INFO: Running with entropic power schedule (0xFF, 100).

Is it possible to pass some sort of timeout for the entire fuzzing process?

I have tried to find an option with no luck.

Thanks