RandomQueryGeneratorDebugging

Debugging Random Query Generator Tests

Using --debug

The --debug option turns on all sorts of checks that report on the performance of the test when it ends.

The following things are reported:

• if any grammar rules generated only queries that produced errors when executed. Such rules are likely broken;
• the different SQL errors that your queries produced, with a query count for each error. Invalid queries are sometimes unavoidable if trying to keep a grammar broad and uncomplicated, however an excessive number of invalid queries may impact on the concurrency of a test;
• report on the number of rows returned by the queries. For Optimizer-related testing, queries that return zero rows are only partially useful. They do not exercise the complete optimization and execution path and the empty result set is not very helpful when doing comparison tests. Microsoft report that a purely random workload may contain over 50% queries that return no data;
• report on the various optimizations triggered by the workload, based on the fields (in particular the Extra field) from the EXPLAIN output. Optimizations such as "Impossible WHERE noticed" happen early in the optimization cycle so the entire optimization and execution path will not be exercised by such queries.
• A report listing the queries that triggered otherwise rare optimizations as determined by EXPLAIN. Such queries can then be used to tweak the grammar to increase the chance of rare optimizations being triggered, thereby making code coverage more robust and independent of just a few isolated queries;

Using code coverage

If you run runall.pl with --lcov, the lcov and genhtml tools will be used to produce a color-coded HTML sorce tree in /tmp that displays the line covered by the runall.pl run.

To produce a coverage report, the following needs to be true:

• The tree needs to be compiled with the -g gcc switch. This is achieved most easily using the ./BUILD/*gcov* scripts.
• The server has to shut down cleanly before the end of the test, so that coverage data is written out to disk. This is achieved by using the Shutdown Reporter. Reporting coverage on crash recovery and such may not be possible.

Complete RQG command line:

 perl runall.pl \
 --lcov \
 --basedir=/home/philips/bzr/maria-5.3 \
 --grammar=conf/example/example.yy \
 --reporter=Shutdown

To quckly and dirtily make the tree easily visible, symlink /tmp from /var/www/html. If you get "Forbidden" errors, you may also need to disable SeLinux by issuing echo 0 > /selinux/enforce, which is, of course a major security risk.

Category:RandomQueryGenerator