MySQL 8.0 and the thread sanitizer

MySQL 8.0 now supports the thread sanitizer.   This is good news as the thread sanitizer provides MySQL developers another tool to help find bugs in the multi-threaded MySQL server.  What happens when we build MySQL 8.0 with the thread sanitizer enabled and try to run some basic MySQL tests?  Unfortunately, no MySQL tests run since the MySQL bootstrap fails with lots of data races and other issues raised by the thread sanitizer.  When these issues are suppressed, some of the basic MySQL and InnoDB tests pass.  Some of these issues are real bugs and need to be investigated.

Both gcc 6.1 and clang 3.9 support the thread sanitizer.  Just add  the 'WITH_TSAN=ON' cmake option when configuring and building MySQL 8.0, and the MySQL code will be compiled with the thread sanitizer.

The next step after building MySQL is to run some MySQL tests.  Unfortunately, MySQL tests need to bootstrap the MySQL data directory, and there are lots of thread sanitizer issues during the bootstrap that prohibit the bootstrap to succeed.   This means that no MySQL tests can be run until these issues are fixed or suppressed.  MySQL 8.0 does not include a basic suppression file for the thread sanitizer as it does for the address sanitizer and for valgrind.  So, how many suppressions are needed to get a simple test to run?  The answer is about 4 or 5 suppressions (see below).  The problem is that the InnoDB suppression covers ANY data race in the InnoDB storage engine.  Since InnoDB is the primary storage engine for MySQL, this is not acceptable.  I hope that the InnoDB developers address this.

I used the following thread sanitizer suppressions when running the main and InnoDB MySQL test suites.  Anyone interesting in using the thread sanitizer to find bugs in MySQL 8.0 software can use these suppressions as a starting point in their investigation.  Good luck!

# ignore races in the pthread barrier implementation

# http://prohaska7.blogspot.com/2015/09/mysql-57-and-barriers.html

race:pthread_barrier

# ignore possible locking deadlock

# http://bugs.mysql.com/bug.php?id=77872

deadlock:plugin_thdvar_init

# ignore all races in the innodb storage engine.  this is overkill

# http://prohaska7.blogspot.com/2015/12/mysql-57-innodb-versus-thread-sanitizer.html

race:innobase

# ignore races in the perf schema

# http://prohaska7.blogspot.com/2015/12/mysql-57-performance-schema-versus.html

race:^pfs

race:^PFS

# ignore race on charsets_dir

# http://prohaska7.blogspot.com/2016/02/mysqlrealconnect-is-not-thread-safe.html

race:^charsets_dir$

# ignore race on THR_KEY_mysys_initialized. should be an atomic variable

race:^THR_KEY_mysys_initialized$