Cluster Fixture Framework

Drill provides two ways to test. The original tests are based on the BaseTestQuery are are (or will be) described elsewhere. Limitations of this class prompted creation of a new framework, which this page describes.

• A single base class BaseTestQuery holds a large amount of functionality, making it hard to create specialized test classes. One either starts with BaseTestQuery, or must replicate that functionality. Since one often has to create specialized setups, this was a bit of a limitation.
• BaseTestQuery is very handy in that it starts an embedded Drillbit. But, it does so using a fixed set of boot options. To change the boot options as needed for some tests, one has to allow the initial Drillbit to start, then shut it down, create the new config, and restart. This is tedious when using tests for debugging.
• The set of tools provided by BaseTestQuery left some holes: data verification must be in the form of a query, it was hard to run a query and print the results for debugging, and so on.

The "cluster" fixture framework solves these issues by taking a different approach:

• Test functionality is not part of the test class hierarchy. Instead, it is something that tests can use as needed, allowing tests to use whatever test class hierarchy is needed for the job at hand.
• Allows very simply means to set up the config, session and system options needed for a tests.
• Allows starting and stoping the Drillbit as needed: one Drillbit for the entire test, or one per test case. Even allows multiple Drillbits for concurrency tests.
• Provides a wide range of tools to execute queries and inspect results. Includes not just the TestBuilder functionality from BaseTestQuery, but also a new QueryBuilder that provides additional options.

That is the background. Let's see how to use this in practice. The best place to start is with the ExampleTest class.

Simplest Case: Run Query and Print

The simplest test case is one that runs a query and prints the results to CSV. While not a true test case, this is often a handy way to get started on a project or bug fix. It also illustrates the basic needed for advanced cases.

public class ExampleTest {
  @Test
  public void firstTest() throws Exception {
    try (ClusterFixture cluster = ClusterFixture.standardCluster();
         ClientFixture client = cluster.clientFixture()) {
      client.queryBuilder().sql("SELECT * FROM `cp`.`employee.json` LIMIT 10").printCsv();
    }
  }
}

Let's look at each piece. Every test needs two critical components:

ClusterFixture cluster = ...
ClientFixture client = ...

• The cluster fixture which represents your embedded Drill cluster. Most often the "cluster" is a single Drillbit, as is the case here. But, the cluster can include multiple Drillbits (coordinated either via Zookeeper or an embededded cluster coordinator.) For now, let's use a single Drillbit.
• The client fixture which represents your Drill client application. The client fixture provides a wealth of functionality that a client may need. Here we only use the ability to run a query.

As your tests grow, you will find the need to set options: often on the server, but sometimes on the client. The two fixtures provide "builder" that help you set up both the client and server the way you want. Here, we use default builders that use the reasonable set of default options.

ClusterFixture cluster = ClusterFixture.standardCluster();
ClientFixture client = cluster.clientFixture();

We want tests to succeed, but sometimes they fail. In fact, some tests even want to test failures (that, say, the server catches configuration mistakes.) To ensure cleanup, we use the try-with-resources idiom to clean up if anything goes wrong.

    try (ClusterFixture cluster = ClusterFixture.standardCluster();
         ClientFixture client = cluster.clientFixture()) {
      // Your test here
    }

Next, we want to run a query. Drill has many ways to run a query, and many ways to process the query results. Rather than provide zillions of functions, the client fixture provides a "query builder" that lets you walk through the steps to build and run the query. In the example above, we build the query from a SQL statement, then run it synchronously and print the results to CSV.

 // Create a query builder for our Drill client
client.queryBuilder()
   // Run a SQL statement
  .sql("SELECT * FROM `cp`.`employee.json` LIMIT 10")
  // Print the results as CSV
  .printCsv();

The best thing at this point is to try the above test case. Create a new JUnit test case, copy the test case from ExampleTest (including imports) and run it as a JUnit test case. If there are any glitches, now is the time to catch them.

Setting Boot Options

By default, the cluster fixture builder sets a standard set of boot options. These include:

• The same options set on the command line in the SureFire setup in the root pom.xml file.
• Adjusts some thread counts to a smaller size to allow faster Drillbit start in tests.
• Adjusts some local directory paths.

You can see the full set of options in ClusterFixture.TEST_CONFIGURATIONS.

But, often you want to set up boot options in some special way. To do that, just use the cluster fixture builder. Suppose we want to set the slice target to 10:

  @Test
  public void secondTest() throws Exception {
    FixtureBuilder builder = ClusterFixture.builder()
        .configProperty(ExecConstants.SLICE_TARGET, 10)
        ;

    try (ClusterFixture cluster = builder.build();
         ClientFixture client = cluster.clientFixture()) {

The above uses the configProperty() method to set the config property as a name/value pair. The name can be a string. But, often it is a bit more maintainable to use the constant declaration for the property, here we use one defined in ExecConstants.

The configProperty() method has another convenience: you can pass the value as a Java value, not just as a string. For example, above we passed the value as an integer. You can also use strings, doubles and other types.

Setting System Options