A multiple linear regression plugin for Elasticsearch

Linear regression model has been a mainstay of statistics and machine learning in the past decades and remains one of the most important tools in context of supervised learning algorithms. It’s a powerful technique for prediction of the value of a dependent variable y (called response variable) given the values of another independent variables x = (x1, x2,…​,xC) (called explanatory variables) based on a training data set. Prediction of the response variable with respect to the input values for the explanatory variables is described by the linear hypothesis function h(x) with

This plugin enhances Elasticsearch’s query engine by two new aggregations, which utilize the index data during search as training data for estimating a linear regression model in order to expose information like prediction of a value for the target variable, anomaly detection and measuring the accuracy or rather predictiveness of the model. Estimation is performed regarding the OLS (ordinary least-squares) approach over the search result set.

Aggregations

Both aggregations are numeric aggregations that estimate the linear regression coefficients based on document results of a search query. Each search result document is handled as an observation and the numerical fields as variables (explanatory and response) for the linear model.

Aggregation for prediction

The linreg_predict aggregation computes the predicted outcome for the response variable regarding the estimated model with respect to a set of given input values for the explanatory variables.

value	The predicted value for the response variable computed using the estimated linear hypothesis function h(x) with x given by C input values for the explanatory variables x = [x1, x2,…​,xC].
coefficients	Estimated coefficients of the linear linear hypothesis function h(x).

Assuming the data consists of documents representing sold house prices with features like number of bedrooms, bathrooms and size etc. we can let predict or validate the price for our house in Morro Bay with 2000 square feet, 4 bedrooms and 2 bathrooms by:

/houses/_search?size=0
{
    "query": {
        "match" : {
            "location" : "Morro Bay"
        }
    },
    "aggs": {
        "house_prices": {
            "linreg_predict": {
                "fields": ["size", "bedrooms", "bathrooms", "price"],
                "inputs": [2000, 4, 2]
            }
        }
    }
}

1. fields instructs this aggregation to use for the linear regression model the house feature fields size, bedrooms and bathrooms as explanatory variables and the price field as the response variable. The size of the fields array is C + 1 with C entries for the explanatory variables and one entry for the response variable.

2. inputs passes the feature values of our house we like to predict the price for. The numeric input values have to be passed in array form in the order corresponding to the features listed in the fields attribute. The size of the inputs array is C equivalent to the number of the explanatory variables.

And the following may be the response with the estimated price of around $ 581,458 for our house:

{
    ...
    "aggregations": {
        "my_house_price": {
            "value": 581458.3087492324,
            "coefficients": [
                227990.63952712028,
                248.92285661317254,
                -68297.7720278421,
                64406.52205356777
            ]
        }
    }
}

Aggregation for linear regression statistics

The linreg_stats aggregation computes statistics for the estimated linear regression model.

rss	Residual sum of squares as a measure of the discrepancy between the data and the estimated model. The lower the rss number, the smaller the error of the prediction, and the better the model.
mse	Mean squared error or rather rss divided by the number of documents consumed for model estimation.
r2	Coefficient of determination, denoted R², as a statistical measure of how well the regression model approximates the real data points. R² ranges from 0 to 1, where 1 indicates that the estimated hypothesis function perfectly fits the data. (Available since 5.5.1.2)
coefficients	Estimated coefficients of the linear linear hypothesis function h(x).

Assuming the data consists of documents representing house prices we can compute statistics for the estimated best fitting linear hypothesis function which predicts house prices based on number of bedrooms, bathrooms and size with

/houses/_search?size=0
{
    "aggs": {
        "house_prices": {
            "linreg_stats": {
                "fields": ["bedrooms", "bathrooms", "size", "price"]
            }
        }
    }
}

The aggregation type is linreg_stats and the fields setting defines the set of fields (as an array) to be used for building the linear model. The first one to many fields stand for the explanatory variables and the last for the response variable. The above request returns the following response:

{
    ...
    "aggregations": {
        "house_prices": {
            "rss": 49523788338938.75,
            "mse": 63410740510.80505,
            "r2": 0.4788369924642064,
            "coefficients": [
                47553.1873756476,
                -100544.07258945837,
                45981.15827544975,
                309.6013051477474
            ]
        }
    }
}

Data conditions

Due to algorithmic constraints both aggregations result an empty response, if

• the search result size is less or equal than the number of indicated explanatory variables,

• values of the explanatory variables in the search result set is linearly dependent (that means that a column can be written as a linear combination of the other columns).

Algorithm

This implementation is based on a new parallel, single-pass OLS estimation algorithm for multiple linear regression (not yet published). By aggregating over the data only once and in parallel the algorithm is ideally suited for large-scale, distributed data sets and in this respect surpasses the majority of existing multi-pass analytical OLS estimators or iterative optimization algorithms.

The overall complexity of the implemented algorithm to estimate the regression coefficients is O(N C² + C³), where N denotes the size of the training data set (the number of documents in the search result set) and C the number of the indicated explanatory variables (fields).

Installation

Elasticsearch 5.x

For installing this plugin please choose first the proper version under the compatible matrix which matches your Elasticsearch version and use the download link for the following command.

./bin/elasticsearch-plugin install https://github.com/scaleborn/elasticsearch-linear-regression/releases/download/5.5.2.1/elasticsearch-linear-regression-5.5.2.1.zip

The plugin will be installed under the name "linear-regression". Do not forget to restart the node after installing.

Table 1. Compatibility matrix

Plugin version	Elasticsearch version	Release date
5.5.2.1	5.5.2	Aug 29, 2017
5.5.1.2	5.5.1	Aug 29, 2017
5.5.1.1	5.5.1	Jul 27, 2017
5.5.0.1	5.5.0	Jul 18, 2017
5.3.0.2	5.3.0	Jul 16, 2017
5.3.0.1	5.3.0	Jun 30, 2017

Examples

Predicting house prices

The idea is very simple. We have data in our Elasticsearch index representing sold house prices in our region with some features like square footage of the house, # of bathrooms, # of bedrooms etc. Now we want to find out which price we have to pay for a house of our dreams.

In this example we use test data from: http://wiki.csc.calpoly.edu/datasets/attachment/wiki/Houses/RealEstate.csv?format=raw

To import the data into Elasticsearch we use logstash and this pipeline config house-prices-import.conf:

./bin/logstash -f house-prices-import.conf

The indexed documents will have this form:

{
  "_index": "houses",
  "_type": "prices",
  "_id": "AV0zjVhTomRh2LZNgmfJ",
  "_source": {
      "bathrooms": 3,
      "bedrooms": 4,
      "size": 4168,
      "mls": "140077",
      "price": 1100000,
      "location": "Morro Bay",
      "price_sq_ft": 263.92,
      "status": "Short Sale"
  }
}

We can now query the index for houses in "Morro Bay" and let predict the price for our dream house with respect to the desired features like 3 bedrooms, 2 bathrooms and at least 2000 square feet:

/houses/_search?size=0
{
    "query": {
        "match" : {
            "location" : "Morro Bay"
        }
    },
    "aggs": {
        "dream_house_price": {
            "linreg_predict": {
                "fields": ["size", "bedrooms", "bathrooms", "price"],
                "inputs": [2000, 3, 2]
            }
        }
    }
}

Regarding the following prediction response we have to expect about $ 650,000 to pay for the desired house in "Morro Bay".

{
    "aggregations": {
        "dream_house_price": {
            "value": 649918.0709489314,
            "coefficients": [
                228318.6161854365,
                249.02340193904183,
                -68314.4830871133,
                64248.05007337558
            ]
        }
    }
}

By using sub aggregations we are able to find out the estimated prices per location:

/houses/_search?size=0
{
    "aggs": {
        "locations": {
            "terms": {
                "field": "location.keyword",
                "size": 15
            },
            "aggs": {
                "dream_house_price": {
                    "linreg_predict": {
                        "fields": ["size", "bedrooms", "bathrooms", "price"],
                        "inputs": [2000, 3, 2]
                    }
                }
            }
        }
    }
}

The response uncovers that "Arroyo Grande" would be the most expensive region for our dream house:

{
    "aggregations": {
        "locations": {
            "buckets": [
                {
                    "key": "Santa Maria-Orcutt",
                    "doc_count": 265,
                    "dream_house_price": {
                        "value": 256251.9105297585,
                        "coefficients": [
                            26437.192829649313,
                            81.19071633227178,
                            6825.9128627023265,
                            23477.773223729317
                        ]
                    }
                },
                {
                    "key": "Paso Robles",
                    "doc_count": 85,
                    "dream_house_price": {
                        "value": 365620.0386191703,
                        "coefficients": [
                            42958.257094706176,
                            151.7000907380368,
                            6486.477078139843,
                            -98.91559301451247
                        ]
                    }
                },
                ...
                {
                    "key": " Arroyo Grande",
                    "doc_count": 12,
                    "dream_house_price": {
                        "value": 1140196.791331573,
                        "coefficients": [
                            728566.7474390095,
                            1956.6474540196602,
                            -706891.620925945,
                            -690495.0006844609
                        ]
                    }
                }
                ...
            ]
        }
    }
}

License

Licensed under the Apache License 2.0.