Skip to content

Latest commit

 

History

History
255 lines (199 loc) · 10.7 KB

README.adoc

File metadata and controls

255 lines (199 loc) · 10.7 KB
Raw

Query Stage SDK

The Query Stage SDK is a Java framework for developing custom Fusion query stage plugins.

Plugins

A plugin is a .zip file that contains one or more query stage implementations. The file contains .jar files for stage definitions and additional dependencies. It also contains a manifest file that holds the metadata Fusion uses to run the plugin.

For build and deployment instructions, see the example plugin README.

Alternatively, it can be assembled by any other build tool as long as the file and directory structure is correct.

Query Stage

The Query Stage class is the basic contract for defining a plugin query stage.

A plugin stage class must:

For convenience, the plugin stage can extend com.lucidworks.querying.api.QueryStageBase, which already contains initialization logic and some helpful methods.

Example of a plugin query stage implementation:

@Stage(type = "simple", configClass = SimpleStageConfig.class)
public class SimpleStage extends QueryStageBase<SimpleStageConfig> {

  private static final Logger logger = LoggerFactory.getLogger(SimpleStage.class);

  @Override
  public DslQueryRequestResponse process(DslQueryRequestResponse query, Context context) {
    queryRequestResponse.getDslQueryRequest().getParams().put(config.queryParam(), config.value());
    return queryRequestResponse;
  }
}

Lifecycle

  1. Fusion creates a QueryStage instance and initializes it.

  2. Fusion begins calling the process method (see the example above) for each query passing through the query pipeline.

One stage instance can be used by Fusion for processing multiple queries and the process method can be called from multiple concurrently-running threads. Additionally, Fusion can initialize and maintain multiple stage instances with the same configuration in separate query service nodes. Therefore, it is important to make sure the plugin stage implementation is thread-safe and processing logic is stateless.

The plugin stage may throw an exception while processing a query. This does not cause any side effects; the query will simply not be processed anymore. The whole query pipeline will still be in use and Fusion will continue to call the process method for other queries. Any information about thrown exceptions can be found in the logs.

Initialization

After creation, each stage object will be initialized using the init(T config, Fusion fusion) method. This allows the stage to create any needed internal structure and validate the stage configuration.

Note that initialization occurs immediately after the stage configuration has been saved in Fusion. After this the stage instance can be maintained and re-used by Fusion for extensive periods of time even if no queries pass through the stage. You should be mindful of this fact when making decisions on resource allocations.

Query-Response processing

After the stage has been initialized, Fusion will start sending queries to the stage for processing by invoking process(DslQueryRequestResponse query, Context context) on every incoming query.

Note that multiple threads can call the process method concurrently, therefore its implementation must be thread-safe.

Logging

Query Stage SDK is using the slf4j logging API. The following example demonstrates how logging can be implemented inside a plugin stage:

public class MyStage extends QueryStageBase<MyStageConfig> {

  private static final Logger logger = LoggerFactory.getLogger(MyStage.class);

  @Override
  public DslQueryRequestResponse process(DslQueryRequestResponse query, Context context) {
    // ......
    logger.info("Processing query '{}'", query);
    // ......
  }
}

Metrics

Fusion is using Prometheus and Grafana for enhanced metrics collection and querying. Query Stage plugins can add their own custom metrics to the list of default metrics already generated by the Fusion querying service.

To be able to publish custom metrics the Prometheus client library must be added to the plugin’s dependencies:

provided 'io.prometheus:simpleclient_dropwizard:0.7.0'

After that, the Prometheus client can be used to record custom metrics. More information on the Prometheus Java client API can be found in the Prometheus documentation.

The following code demonstrates how to capture external request time metrics using the Prometheus client API.

public class SampleStage {

  private static final Histogram EXTERNAL_REQUEST_TIME = Histogram.build()
      .help("Time to execute external query request.")
      .name("external_query_request_time")
      .labelNames("request_url")
      .register();

    @Override
    public DslQueryRequestResponse process(DslQueryRequestResponse query, Context context) {
      // ......
      Histogram.Timer externalQueryTimer = EXTERNAL_REQUEST_TIME
          .labels(requestUrl)
          .startTimer();
      try {
        // perform external request...
      } finally {
        externalQueryTimer.observeDuration();
      }
      // ......
    }
}

Query stage configuration

The QueryStageConfig defines configuration options specific to a particular query stage instance. These options will be available to the end user via the Fusion UI and API. The plugin config class must extend com.lucidworks.querying.config.QueryStageConfig and be annotated with @RootSchema.

By adding @Property and type annotations to your stage configuration interface methods, you can define metadata and type constraints for your plugin configuration fields. This is very similar to Fusion’s connector configuration schema. For more detailed information on the configuration and schema capabilities, see Java Connector Development.

Here is an example of a simple stage configuration schema definition:

@RootSchema(
    title = "Simple",
    description = "Simple Query Stage"
)
public interface SimpleStageConfig extends QueryStageConfig {

    @Property(
            title = "Time allowed",
            description = "The amount of time allowed for a search to complete.",
            required = true
    )
    @NumberSchema()
    long timeAllowed();
}

Exposed Fusion APIs

SDK-based plugins are capable of communicating with other Fusion components via the Fusion object. This object is passed to the stage during the initialization phase.

RestCall

The RestCall API provides access to the Fusion REST API.

Blobs

The Blobs API is a specialized API for interaction with the Fusion Blob Store API

Data structures

DslQueryRequestResponse

DslQueryRequestResponse is a representation of a query request and response. In Fusion, both the query request and response are treated and tracked as one entity. This is the class that is passed through to the process method and may be manipulated to inspect/update the request and/or response.

DslQueryRequest

DslQueryRequest is a representation of a query request. You can access this via the QueryRequestAndResponse class. It contains information about the query parameters, the headers, as well as the HTTP method called.

DslQueryResponse

DslQueryResponse is a representation of a query response. You can access this via the QueryRequestAndResponse class. It contains information about the documents, the facets, and any highlighted terms.