Releases: pdphilip/laravel-elasticsearch
v4.5.0
This release is for compatibility with both Laravel 10 and 11, and marks a version bump to 4.5.0.
New features
1. Bypass field map validation for queries that require keyword mapping
Keyword type queries are checked by default and will select the keyword sub-mapping if it is found; however, this invokes an extra query to check the mapping first.
You can now disable this by setting options.bypass_map_validation = true
'elasticsearch' => [
......
'options' => [
'bypass_map_validation' => env('ES_OPT_BYPASS_MAP_VALIDATION', false),
.....
],
.....
],2. Adjustable chunk size for bulk insert (default 1000)
When performing bulk inserts, the default is 1000 at a time.
You can now adjust this by setting options.insert_chunk_size to your desired amount.
'elasticsearch' => [
......
'options' => [
'insert_chunk_size' => env('ES_OPT_INSERT_CHUNK_SIZE', 1000),
.....
],
.....
],Updated connection config
'elasticsearch' => [
'driver' => 'elasticsearch',
'auth_type' => env('ES_AUTH_TYPE', 'http'), //http or cloud
'hosts' => explode(',', env('ES_HOSTS', 'http://localhost:9200')),
'username' => env('ES_USERNAME', ''),
'password' => env('ES_PASSWORD', ''),
'cloud_id' => env('ES_CLOUD_ID', ''),
'api_id' => env('ES_API_ID', ''),
'api_key' => env('ES_API_KEY', ''),
'ssl_cert' => env('ES_SSL_CA', ''),
'ssl' => [
'cert' => env('ES_SSL_CERT', ''),
'cert_password' => env('ES_SSL_CERT_PASSWORD', ''),
'key' => env('ES_SSL_KEY', ''),
'key_password' => env('ES_SSL_KEY_PASSWORD', ''),
],
'index_prefix' => env('ES_INDEX_PREFIX', false),
'options' => [
'bypass_map_validation' => env('ES_OPT_BYPASS_MAP_VALIDATION', false),
'insert_chunk_size' => env('ES_OPT_INSERT_CHUNK_SIZE', 1000),
'logging' => env('ES_OPT_LOGGING', false),
'allow_id_sort' => env('ES_OPT_ID_SORTABLE', false),
'ssl_verification' => env('ES_OPT_VERIFY_SSL', true),
'retires' => env('ES_OPT_RETRIES', null),
'meta_header' => env('ES_OPT_META_HEADERS', true),
],
'error_log_index' => env('ES_ERROR_INDEX', false),
],3. Removed redundant methods, new exceptions and code clean by @use-the-fork via #48
Full Changelog: v4.4.1...v4.5.0
Contributors
Assets 2
v4.4.1
v4.4.0
This release introduces compatibility with both Laravel 10 and 11, and marks a version bump to 4.4.0.
While there are no breaking changes, the connection class has been fully rebuilt by @use-the-fork. This foundational improvement justifies the version increment and sets the stage for further enhancements.
What's Changed
- Refract/connection by @use-the-fork in #46
Full Changelog: v4.3.0...v4.4.0
Contributors
Assets 2
v4.3.0
This release marks a version bump within the 4.x branch due to one breaking change in rawSearch(). This branch is committed to Laravel 10 and 11 compatibility.
Breaking Change
rawSearch($bodyParams, $returnRaw = false) has now been split into rawSearch($bodyParams) and rawDsl($bodyParams) where:
rawSearch($bodyParams)returns an ElasticCollection of resultsrawDsl($bodyParams)returns the result body as is from Elasticsearch. Equivalent to$returnRaw = truepreviously.
New features
[1] Schema field mapping call: getFieldMapping(string|array $field = '*', $raw = false)
Schema method that can be called from your model:
Product::getFieldMapping('color'); //Returns a key/value array of field/types for color
Product::getFieldMapping('color',true); //Returns the mapping for color field as is from Elasticsearch
Product::getFieldMapping(['color','name']); //Returns maapings for color and name
Product::getFieldMapping(); //returns all field mappings, same as getFieldMapping('*')or via Schema: Schema::getFieldMapping($index, $field, $raw)
Schema::getFieldMapping('products','color',true); [2] Order By Random: orderByRandom(string $column, int $seed = 1)
Uses ES's random_score to randomise ordering
Product::where('color', 'silver')->orderByRandom('created_at', rand(1, 1000))->limit(5)->get();The value of $seed will return the same results if unchanged. This is required for consistencey, ex: pagination
Bug fix
whereExact() keyword field validator has been fixed
PRs
- Refract/code coverage - Coverage up to 79% by @use-the-fork in #40
- fix(DSL): improve keyword mapping handling for nested fields by @use-the-fork in #43
- feat(elasticsearch): enhance raw search functionality by @use-the-fork in #44
Full Changelog: v4.2.0...v4.3.0
Contributors
Assets 2
v4.2.0
This release marks a version bump within the 4.x branch committed to Laravel 10 & 11 compatibility. There are no breaking changes from 4.1.x
This version introduces query-building methods to allow matching across multiple fields: ES docs
The current search builder works in isolation for full-text searching; this upgrade brings those features into the standard query builder that will run like a normal query. Meaning you can:
get(), first(), aggregate(), paginate() etc on full text search results. In time, this will replace the current search methods like: Book::term('Eric')->search();
Methods
searchTerm($term, $fields = ['*'], $options = []) - type: best_fields
searchTermMost($term, $fields = ['*'], $options = []) type: most_fields
searchTermCross($term, $fields = ['*'], $options = []) type: cross_fields
searchPhrase($phrase, $fields = ['*'], $options = []) type: phrase
searchPhrasePrefix($phrase, $fields = ['*'], $options = []) type: phrase_prefix
searchBoolPrefix($phrase, $fields = ['*'], $options = []) type: bool_prefix
Each method has a corresponding OR version, ex orSearchPhrase('Laravel Elasticsearch')
These methods will introduce an Elasticsearch score and will be ordered by score by default.
$fields: By default, all fields will be searched through; you can specify which as well as boost certain fields, ex:
People:searchTerm('John',['name^3','description^2','friends.name'])->get();$options: Allows you to set any options for the multi_match clause to use, ex:
analyzer, boost, operator, minimum_should_match, fuzziness, lenient, prefix_length, max_expansions, fuzzy_rewrite, zero_terms_query.
searchFor($value) is a convenience method that will either use term or phrase depending on the word count of $value
withHighlights($fields = [], $preTag = '<em>', $postTag = '</em>', $globalOptions = [])
- Will highlight the same way as https://elasticsearch.pdphilip.com/full-text-search#highlighting
Option helpers
asFuzzy()
- Will mark the previous clause as fuzzy
setMinShouldMatch(int $value)
- will set the option
{"minimum_should_match": $value}to the previous clause
setBoost(int $value)
- will set the option
{"boost": $value}to the previous clause
Examples
Product::searchTerm('remarkble')->asFuzzy()->withHighlights()->get();
Product::searchPhrasePrefix('coffee bea')->where('is_active',true)->paginate();
Product::searchPhrase('United States')->orSearchPhrase('United Kingdom')->sum('orders');Upgrades
find($id), findOrFail($id) and findOrNew($id) now uses the ES get call directly ie: /my_index/_doc/my_id
- With
findOrNew($id): If the $id was not found, then it will return a new model instance with the $id value as provided - Credit @use-the-fork via #41
Full Changelog: v4.1.1...v4.2.0
Contributors
Assets 2
v4.1.1
v4.1.0
V4.1.0 Release Notes
This release marks a version bump within the 4.x branch committed to Laravel 10 & 11 compatibility.
Acknowledgements
This release represents a significant advancement in the package's development, primarily driven by PR #32. Key improvements include:
- Comprehensive refactoring of the package
- Introduction of new features
- Extensive coverage with automated tests
- Enhanced compliance with coding standards and type-checking
A sincere thank you to @use-the-fork for their substantial contributions to this release. Their efforts have been instrumental in elevating the quality, reliability and maturity of this package.
Breaking Changes
There's only one breaking change:
$query->orderBy(string $column, string $direction = 'asc', string $mode = null, array $missing = '_last');
// Has been changed back to:
$query->orderBy(string $column, string $direction = 'asc');
// With a new method to add any ES sort parameter to a given column:
$query->withSort(string $col, string $key, mixed $value);- This also applies to
orderByDesc
Examples
$products = Product::orderBy('color.keyword', 'desc', null, '_first')->get();
// Is now:
$products = Product::orderBy('color.keyword', 'desc')->withSort('color.keyword', 'missing', '_first')->get();And:
$products = Product::where('is_active', true)->orderBy('order_values', 'desc', 'avg')->get();
// Is now:
$products = Product::where('is_active', true)->orderBy('order_values', 'desc')->withSort('order_values', 'mode', 'avg')->get();This change is to future-proof for the outlier sorting options that ES offers, ex:
$products = Product::withSort('price', 'unmapped_type', 'long')->get();
$products = Product::withSort('price', 'ignore_unmapped', true')->get();Inspired by #36 via @ildar-developer
Refactor
- Automated tests
- PHPStan compliance
- PINT formatted
- Updated PHPDocs for better IDE support
New Features
1. Cursor Pagination
Given the default limitations of Elasticsearch to paginate beyond 10k records, cursor pagination allows you to paginate indefinitely using search_after given a sort map.
As is the case with core Laravel's cursorPaginate , you can only page next and prev
cursorPaginate($perPage, $columns, $cursorName, $cursor)Example:
Product::orderByDesc('orders')->orderByDesc('price')->cursorPaginate(50)->withQueryString();The accuracy of this method depends on the sort, as would be the case using ES directly. If no sort is provided, it will try to sort by created_at and updated_at if those mappings are found; otherwise, it will throw a MissingOrderException.
2. Bulk Import
insert($values, $returnData = null);
insertWithoutRefresh($values, $returnData = null);These methods use Elasticsearch's Bulk API under the hood, providing efficient ways to insert multiple documents at once.
If the $returnData parameter is null or false, it will return a summary as an array:
{
"success": true,
"timed_out": false,
"took": 7725,
"total": 100000,
"created": 100000,
"modified": 0,
"failed": 0
}Otherwise, it will return an ElasticCollection of all the inserted records.
Performance and Usage:
insert($values, $returnData = null)- Performs a bulk insert and waits for the index to refresh.
- Ensures that inserted documents are immediately available for search.
- Use this when you need the inserted data to be searchable right away.
- Slower than
insertWithoutRefreshbut provides immediate consistency.
insertWithoutRefresh($values, $returnData = null)- Executes bulk inserts without waiting for the index to refresh.
- Offers a significant speed boost compared to
insert(). - The speed increase is due to skipping the index refresh operation, which can be resource-intensive.
- Inserted records may not be immediately available for search
- Use this when you need to insert large amounts of data quickly and can tolerate a slight delay in searchability.
When to use each:
- Use
insert()when:- You need immediate searchability of inserted data.
- You're inserting smaller batches of data where the performance difference is negligible.
- In user-facing applications where real-time data availability is crucial.
- Use
insertWithoutRefresh()when:- You're performing large batch imports where speed is a priority.
- In background jobs or data migration scripts where immediate searchability isn't necessary.
- You can handle a delay between insertion and searchability in your application logic.
3. ElasticCollection
Queries that return collections (get(), search(), insert()) now return them as an ElasticCollection. ElasticCollection is the same as Laravel's Eloquent Collection but with the executed query's metadata embedded in it.
$fetch = Product::where('orders', '>', 100)->limit(50)->orderByDesc('price')->get();
$shards = $fetch->getShards(); // Shards info
$dsl = $fetch->getDsl(); // The full query DSL that was used
$total = $fetch->getTotal(); // Total records in the index if there were hits
$maxScore = $fetch->getMaxScore(); // Max score of the search
$took = $fetch->getTook(); // Time taken for the search in milliseconds
$meta = $fetch->getQueryMetaAsArray(); // All the metadata in an array
return $fetch; //gives your collection of results as alwaysBug fix
- Starting a new query before an existing one has been executed was causing the connection (and queries) to get mixed up. via #36
What's Changed
- Refract/cleanup - Added Test suite / formating etc. by @use-the-fork in #32
Full Changelog: v4.0.4...v4.1.0
Contributors
Assets 2
v4.0.4
This unified release includes updates for multiple Laravel versions:
Upgrades
sum(),avg(),min()andmax()can now process multiple fields in one call
Product::where('color','blue')->avg(['orders', 'price']);
//Previously required two separate calls:
Product::where('color','blue')->avg('orders');
Product::where('color','blue')->avg('price');rawSearch(array $bodyParams, bool $returnRaw = false)- a second bool parameter will return data as is (unsanitized) if set to
true
Product::rawSearch($body, true);Bug fixes
- rawAggregation with multiple aggregations now returns all aggs
- Fixed issue when saving fields where the data didn't change threw an error
Full Changelog: v4.0.3...v4.0.4
v4.0.3 - (Unified with v3.9.3 & v3.8.3)
This unified release includes updates for multiple Laravel versions:
New Features
wherePhrasePrefix($field, $phraseWithPrefix)
Method for looking up a specific sequence of words where the last word starts with a particular prefix
Person::wherePhrasePrefix('description', 'loves es')->get();
// returns: loves espresso, loves essays, loves eskimos, etcDocs: https://elasticsearch.pdphilip.com/es-specific#where-phrase-prefix
phrase($field)
Method for searching across multiple fields for a specific phrase (sequence of words in order)
Book::phrase('United States')->orPhrase('United Kingdom')->search();
// Search for books that contain either 'United States' or 'United Kingdom', phrases like 'United Emirates' will not be included.Docs: https://elasticsearch.pdphilip.com/full-text-search#phrase-search-phrase
agg(array $functions,$field)
Optimization method that allows you to call multiple aggregation functions on a single field in one call.
Available aggregation functions: count, avg, min, max, sum, matrix.
Product::where('is_active',true)->agg(['count','avg','min','max','sum'],'sales');https://elasticsearch.pdphilip.com/aggregation#grouped-aggregations
toDsl() (or toSql())
Returns the parsed DSL query from the query builder
Product::whereIn('color', ['red', 'green'])->orderByDesc('sales')->toDsl();Returns
{
"index": "products",
"body": {
"query": {
"terms": {
"color.keyword": [
"red",
"green"
]
}
},
"_source": [
"*"
],
"sort": [
{
"sales": {
"order": "desc"
}
}
]
}
}
Docs: https://elasticsearch.pdphilip.com/es-specific#to-dsl
Tests for new features: https://github.com/pdphilip/laravel-elasticsearch-tests/blob/main/tests/EloquentTests/Update24-01Test.php
Bug fixes
- unset _meta on save. by @use-the-fork in #30
- Now throws an explicit error when trying to use
BelongsToMany()which is not supported (but can be worked around easily)
Upgrades
- Fixed error tracking index for writing ES errors to a dedicated index
// database.php
'elasticsearch' => [
'driver' => 'elasticsearch',
//......
//......
//......
'error_log_index' => env('ES_ERROR_INDEX', false),
],- White space code clean-up
New Contributors
- @use-the-fork made their first contribution in #30
Full Changelog: v4.0.2...v4.0.3
Contributors
Assets 2
v4.0.2
New Features
New numeric type mappings for IndexBlueprint
double($field)- A double-precision 64-bit IEEE 754 floating point number, restricted to finite values.byte($field)- A signed 8-bit integer with a minimum value of -128 and a maximum value of 127.halfFloat($field)- A half-precision 16-bit IEEE 754 floating point number, restricted to finite values.scaledFloat($field, $scalingFactor = 100)- A floating point number that is backed by a long, scaled by a fixed double scaling factor.unsignedLong($field)- An unsigned 64-bit integer with a minimum value of 0 and a maximum value of 264-1.
Example:
Schema::create('my_index', function (IndexBlueprint $index) {
$index->double('some_field_a');
$index->byte('some_field_b');
$index->halfFloat('some_field_c');
$index->scaledFloat('some_field_d', 100);
$index->unsignedLong('some_field_e');
});Upgrades
- Upgraded Connection class to parse the config's connection name. This allows for multiple connections or if you define your connection in the database file something other than
elasticsearch
Example with multiple connections (database.php):
'elasticsearch' => [
'driver' => 'elasticsearch',
'auth_type' => env('ES_AUTH_TYPE', 'http'), //http, cloud or api
'hosts' => explode(',', env('ES_HOSTS', 'http://localhost:9200')),
'username' => env('ES_USERNAME', ''),
'password' => env('ES_PASSWORD', ''),
'cloud_id' => env('ES_CLOUD_ID', ''),
'api_id' => env('ES_API_ID', ''),
'api_key' => env('ES_API_KEY', ''),
'ssl_cert' => env('ES_SSL_CA', ''),
'ssl' => [
'cert' => env('ES_SSL_CERT', ''),
'cert_password' => env('ES_SSL_CERT_PASSWORD', ''),
'key' => env('ES_SSL_KEY', ''),
'key_password' => env('ES_SSL_KEY_PASSWORD', ''),
],
'index_prefix' => env('ES_INDEX_PREFIX', false),
'options' => [
'allow_id_sort' => env('ES_OPT_ID_SORTABLE', false),
'ssl_verification' => env('ES_OPT_VERIFY_SSL', true),
'retires' => env('ES_OPT_RETRIES', null),
'meta_header' => env('ES_OPT_META_HEADERS', true),
],
'query_log' => [
'index' => false,
'error_only' => true,
],
],
'elasticsearch-cloud' => [
'driver' => 'elasticsearch',
'auth_type' => env('ES_CLOUD_AUTH_TYPE', 'http'), //http or cloud
'hosts' => explode(',', env('ES_CLOUD_HOSTS', 'http://localhost:9200')),
'username' => env('ES_CLOUD_USERNAME', ''),
'password' => env('ES_CLOUD_PASSWORD', ''),
'cloud_id' => env('ES_CLOUD_CLOUD_ID', ''),
'api_id' => env('ES_CLOUD_API_ID', ''),
'api_key' => env('ES_CLOUD_API_KEY', ''),
'ssl_cert' => env('ES_CLOUD_SSL_CA', ''),
'ssl' => [
'cert' => env('ES_CLOUD_SSL_CERT', ''),
'cert_password' => env('ES_CLOUD_SSL_CERT_PASSWORD', ''),
'key' => env('ES_CLOUD_SSL_KEY', ''),
'key_password' => env('ES_CLOUD_SSL_KEY_PASSWORD', ''),
],
'index_prefix' => env('ES_CLOUD_INDEX_PREFIX', false),
'options' => [
'allow_id_sort' => env('ES_CLOUD_OPT_ID_SORTABLE', false),
'ssl_verification' => env('ES_CLOUD_OPT_VERIFY_SSL', true),
'retires' => env('ES_CLOUD_OPT_RETRIES', null),
'meta_header' => env('ES_CLOUD_OPT_META_HEADERS', true),
],
'query_log' => [
'index' => false,
'error_only' => true,
],
],Examples of selecting connection:
Schema::on('elasticsearch-cloud')->create('my_index', ...... );
Product::on('elasticsearch-cloud')->get() //If $connection in Product model is not 'elasticsearch-cloud';New Contributors
Full Changelog: v4.0.1...v4.0.2
