22 Crash Recovery Bugs Found in PMDK Ports #1

iangneal · 2022-01-31T21:32:51Z

Summary

We found 22 crash-consistency bugs introduced in WITCHER's PMDK adaptation layer added during the process of porting data structures to us a PM allocator. These bugs are all related to the initialization phase of the data structures (i.e., during PMDK pool creation and allocation of the root/parent structures of the key-value data structures).

Since these bugs are specific to WITCHER's port, I'm filing these bugs here rather than with the maintainers of the ported applications/data structures.

`NULL` dereferences

Most of these bugs are related to the misuse of the PMDK API. In the common function init_nvmm_pool: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/pmdk.c#L5-L30

If a crash occurs during pmemobj_create: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/pmdk.c#L15

Then calls to pmemobj_open in the post-crash execution of the application will return NULL. This case is handled within init_nvmm_pool by returning NULL: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/pmdk.c#L21-L24

However, all of the benchmarks that use init_nvmm_pool do not check for the case when init_nvmm_pool returns NULL. Ergo, these benchmarks will immediately segfault when trying to . For example, in WOART: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/woart.c#L785-L789

If root_obj is NULL, a segfault will occur on access to root_obj->woat_ptr: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/woart.c#L788

Non-atomic initialization/lack of `NULL` checks

Many of the benchmarks that use init_nvmm_pool do not check for the crash when a crash occurs between pool allocation and the initial allocation of the object. Ergo, they will immediately crash upon their first read of the data structure, as the root pointer is not allocated.

For example, during WOART initialization: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/woart.c#L780-L797

If there is a crash after init_nvmm_pool and root_obj->woart_ptr = tree;, then root_obj will not be NULL but root_obj->woart_ptr will still be NULL, and init_woart will return NULL on restart: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/woart.c#L788

This will cause NULL to be passed to the art_{search,insert} functions, i.e.: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/benchmark/WORT/woart/main/main.c#L34

Since the art_* functions assume they will not be passed a NULL pointer: https://github.com/cosmoss-vt/witcher/blob/ad69038cdcd4ac20f1bde38ebf7e6d9fd6999b36/third_party/WORT/src/woart/woart.c#L293-L295

This will ultimately result in a segfault on recovery.

Specific Bugs by Location/Benchmark

Here's a list of all the bugs by their source code locations. All of these bugs fall into the previously-outlined groups, so the per-bug explanations are left fairly terse.