MySQL 5.7 Initialization Versus the Thread Sanitizer

The MySQL server is a large multi-threaded program primarily written in the C++ programming language.  Unfortunately,  data races and deadlocks in concurrent C++ programs are common.  Fortunately, the Thread Sanitizer can be used to find data races and deadlocks.  MySQL 5.7 does not support the Thread Sanitizer, but is is easy to add to the MySQL configuration using a simple build procedure.  When MySQL 5.7 is built with the Thread Sanitizer and database initialization is run, the Thread Sanitizer reports hundreds of possible data races and deadlocks in the MySQL server.

Database initialization occurs when the MySQL data is generated for a new MySQL instance.  This is done in MySQL 5.7 when running 'mysqld --initialize'.   Since the MySQL server (mysqld) is run to initialize the database and the MySQL server is compiled with the Thread Sanitizer, many possible data races and deadlocks are reported while the MySQL server is run.  This blog analyzes four of the issues:  a problem with pthread barrier destruction, a potential lock deadlock, a data race on error reporting, and a data race on the global THD count variable.

The Thread Sanitizer reports a data race inside pthread barriers when a pthread barrier is destroyed.  MySQL 5.7 only has one use of pthread barriers which is to synchronize the main thread with the initialization of a background timer notification thread.  IMO, the Thread Sanitizer reports this race because many pthread barrier implementations have bugs WRT the barrier wait and barrier destroy operations.  MySQL can either choose to not use pthread barriers or suppress this error since the MySQL code using pthread barriers is correct.  The pthread barriers blog discusses this issue in more detail.

The Thread Sanitizer reports a possible deadlock with 3 locks.   This problem was also detected by helgrind and reported in MySQL bug 77872.  Variations of this 3 way locking deadlock have been detected in the field and reported to MySQL.  The Thread Sanitizer is faster than helgrind and it reports all of the thread stacks involved in the deadlock while helgrind does not.  These two Thread Sanitizer advantages should make debugging with the Thread Sanitizer feasible. 

The Thread Sanitizer reports a data race on the log_error_verbosity global variable between the bootstrap and signal handling threads.  These two threads run concurrently.  The bootstrap thread does a racy write of the log_error_verbosity variable while the signal handling thread does a racy read of the log_error_verbosity variable.  The Thread Sanitizer reports a data race since it can not compute a happens before relationship WRT the writes and reads of the log_error_verbosity variable done by these two threads.  This looks like a bug to me.

The Thread Sanitizer reports a data race on the global thd count when shutting down the MySQL server.  The MySQL server does a racy read of the global_thd_count variable without holding a lock while most of the other reads and writes of the global_thd_count variable are done while holding a lock.  Since the MySQL code states that the global_thd_count variable is racy, this racy read is by design.  If MySQL uses a relaxed memory atomic variable to store the global_thd_count, then the Thread Sanitizer would not consider this a data race.  Otherwise, a Thread Sanitizer suppression can be used to ignore this issue.  It would be nice to annotate racy variables in the code rather than in a comment on the code or in a data race suppression rule.

In conclusion, a configuration option that supports MySQL builds with the Thread Sanitizer was added to MySQL 5.7.  Hundreds of possible data races and deadlocks were reported by the Thread Sanitizer when running MySQL initialization.  This blog only examined four of the issues that illustrate the value of the tool.  The next blogs will look into the Thread Sanitizer issues WRT the performance schema and the InnoDB storage engine.

Tools

MySQL 5.7.9
Thread Sanitizer in Clang 3.8

Ubuntu 14.04