The Address Sanitizer Finds Bugs in TokuFT

This blog describes how the Address Sanitizer found bugs in the TokuFT (now PerconaFT) storage library.  TokuFT is the storage library used by the TokuDB for MySQL and TokuMX for MongoDB products.  TokuFT is currently tested with valgrind's memcheck tool.  However, memcheck is very slow compared to native execution, so memcheck is not always used for large tests.  This leads to missed bugs.

The Address Sanitizer is a memory error detector which detects memory leaks, out of bounds memory access, and several other types of memory related bugs.  Recent versions of the clang and gcc compilers have integrated the Address Sanitizer into the compiler.  The Address Sanitizer claims an order of magnitude increase in speed compared to valgrind's memcheck.  This is interesting since it opens up execution scenarios that are problematic with valgrind given the execution time expansion.

The Address Sanitizer found bugs in the TokuFT library when running existing tests.  These bugs were not previously found because the valgrind tools were deemed too slow and were not used for these tests.

TokuFT test failures detected with the address sanitizer

TokuFT has been developed and tested for awhile.   It is always interesting to find new bugs in existing code by using new tools like the Address Sanitizer.

Memory leak in the bnc-insert-test.  This bug is found when running the basic TokuFT regression tests with the Address Sanitizer compiled into TokuFT.  For some reason, the 'bnc-insert-test' was not run with valgrind for some reason (probably execution time), so the bug was not found.

Memory leak in the context status initialization.   This bug is found by the TokuFT stress tests with the Address Sanitizer compiled into TokuFT.  The bug was not previously found because valgrind is not used by the stress tests.

Future work

Cmake file hackery.   IMO, an option should be added to the TokuFT cmake files to support the Address Sanitizer in addition to valgrind's memcheck, helgrind, and DRD tools.  Since these tools continue to improve, TokuFT should continue to support all of them.

Stress tests.  The TokuFT stress tests (and perhaps other tests) should be run with the Address Sanitizer enabled since the execution speed is claimed to only be 2x slower than native compiled code execution.

Address Sanitizer and MySQL.  I did not realize until today that MySQL 5.6 (and 5.7) include the 'WITH_ASAN' option that will compile MySQL with the Address Sanitizer enabled.  This makes testing MySQL with memory verification feasible for large tests.  I assume that since this option has been available in MySQL for awhile, it must be part of the MySQL test plan.  Is it?

Tool versions

Ubuntu 14.04

Clang 3.5

Build and test

# setup the compiler options to build with the address sanitizer
export CC=clang-3.5
export CXX=clang++-3.5
export CXXFLAGS=-fsanitize=address
export ASAN_OPTIONS=detect_odr_violation=0

# get the FT source
git clone git@github.com:percona/perconaft

# run the basic tests
mkdir perconaft-asan && chdir perconaft-asan
cmake -DCMAKE_BUILD_TYPE=Debug ../perconaft
ctest -j8 --verbose

# run the stress tests as described here.
Directions on how to run the FT stress tests