Skip to content

Commit

Permalink
docs: minor improvements in Joining.md and related with that
Browse files Browse the repository at this point in the history
  • Loading branch information
sanikolaev committed Jan 23, 2025
1 parent 6b58319 commit 711c47d
Show file tree
Hide file tree
Showing 2 changed files with 59 additions and 25 deletions.
80 changes: 57 additions & 23 deletions manual/Searching/Joining.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ Table joins in Manticore Search enable you to combine documents from two tables

### SQL

<!--example join_sql_json_type -->

```sql
SELECT
select_expr [, select_expr] ...
Expand All @@ -25,6 +23,8 @@ join_condition: {

For more information on select options, refer to the [SELECT](../Searching/Intro.md#General-syntax) section.

<!--example join_sql_json_type -->

When joining by a value from a JSON attribute, you need to explicitly specify the value's type using the `int()` or `string()` function.

<!-- request String JSON attribute -->
Expand Down Expand Up @@ -83,7 +83,7 @@ on.type: {
```
Note, there is the `type` field in the `left` operand section which you should use when joining two tables using json attributes. The allowed values are `string` and `int`.

## Types of Joins
## Types of joins

Manticore Search supports two types of joins:

Expand Down Expand Up @@ -357,7 +357,9 @@ POST /search

<!-- end -->

### Complex Join with Faceting
### Example: Complex JOIN with faceting

Building on the previous examples, let's explore a more advanced scenario where we combine table joins with faceting. This allows us to not only retrieve joined data but also aggregate and analyze it in meaningful ways.

<!-- example basic_complex -->

Expand Down Expand Up @@ -505,7 +507,7 @@ POST /search

<!-- end -->

## Query options and match weights
## Search options and match weights

Separate options can be specified for queries in a join: for the left table and the right table. The syntax is `OPTION(<table_name>)` for SQL queries and one or more subobjects under `"options"` for JSON queries.

Expand All @@ -530,19 +532,16 @@ OPTION(customers) field_weights=(address=1500);
POST /search
{
"table": "orders",
"options":
{
"customers":
{
"field_weights":
{
"address": 15000
"options": {
"customers": {
"field_weights": {
"address": 1500
}
}
},
"join": [
{
"type": "inner",
"type": "inner",
"table": "customers",
"query": {
"query_string": "maple"
Expand Down Expand Up @@ -618,24 +617,44 @@ POST /search

## Join batching

Table joins work by accumulating a batch of matches, which are the results of the query executed on the left table. This batch is then executed as a single query on the right table.
When performing table joins, Manticore Search processes the results in batches to optimize performance and resource usage. Here's how it works:

The batch size can be adjusted using the `join_batch_size` query option. It is also configurable in the searchd section of the configuration file. The default value is 1000, and setting this option to 0 disables batching.
- **How Batching Works**:
- The query on the left table is executed first, and the results are accumulated into a batch.
- This batch is then used as input for the query on the right table, which is executed as a single operation.
- This approach minimizes the number of queries sent to the right table, improving efficiency.

A larger batch size may improve performance, but for some queries, it can lead to excessive memory consumption.
- **Configuring Batch Size**:
- The size of the batch can be adjusted using the `join_batch_size` search option.
- It is also [configurable](../../Server_settings/Searchd.md#join_batch_size) in the `searchd` section of the configuration file.
- The default batch size is `1000`, but you can increase or decrease it depending on your use case.
- Setting `join_batch_size=0` disables batching entirely, which may be useful for debugging or specific scenarios.

- **Performance Considerations**:
- A larger batch size can improve performance by reducing the number of queries executed on the right table.
- However, larger batches may consume more memory, especially for complex queries or large datasets.
- Experiment with different batch sizes to find the optimal balance between performance and resource usage.

## Join caching

Each query executed on the right table is defined by specific JOIN ON conditions, meaning that these conditions determine the result set retrieved from the right table.
To further optimize join operations, Manticore Search employs a caching mechanism for queries executed on the right table. Here's what you need to know:

- **How Caching Works**:
- Each query on the right table is defined by the `JOIN ON` conditions.
- If the same `JOIN ON` conditions are repeated across multiple queries, the results are cached and reused.
- This avoids redundant queries and speeds up subsequent join operations.

If there are only a few unique JOIN ON conditions, it may be beneficial to reuse the results instead of repeatedly executing queries on the right table. To achieve this, the result sets are stored in a cache.
- **Configuring Cache Size**:
- The size of the join cache can be configured using the [join_cache_size](../../Server_settings/Searchd.md#join_cache_size) option in the `searchd` section of the configuration file.
- The default cache size is `20MB`, but you can adjust it based on your workload and available memory.
- Setting `join_cache_size=0` disables caching entirely.

The size of this cache can be configured using the `join_cache_size` option in the searchd section of the configuration file. The default value is 20MB.
Note that each thread maintains its own cache, so you should account for the number of threads executing queries when estimating total memory usage.
- **Memory Considerations**:
- Each thread maintains its own cache, so the total memory usage depends on the number of threads and the cache size.
- Ensure your server has sufficient memory to accommodate the cache, especially for high-concurrency environments.

Setting `join_cache_size=0` disables caching.

## Caveats and Best Practices
## Caveats and best practices

When using JOINs in Manticore Search, keep the following points in mind:

Expand All @@ -656,7 +675,22 @@ When using JOINs in Manticore Search, keep the following points in mind:

4. **Filtering on aliased expressions**: You cannot use aliases for expressions involving fields from both tables in the WHERE clause.

5. **Using ANY with MVA**: When using the `ANY()` function with multi-valued attributes in JOINs, alias the multi-valued attribute from the joined table:
5. **JSON attributes**: When joining on JSON attributes, you must explicitly cast the values to the appropriate type:
```sql
-- Correct:
SELECT * FROM t1 LEFT JOIN t2 ON int(t1.json_attr.id) = t2.json_attr.id

-- Incorrect:
SELECT * FROM t1 LEFT JOIN t2 ON t1.json_attr.id = t2.json_attr.id
```

6. **NULL handling**: You can use IS NULL and IS NOT NULL conditions on joined fields:
```sql
SELECT * FROM t1 LEFT JOIN t2 ON t1.id = t2.id WHERE t2.name IS NULL
SELECT * FROM t1 LEFT JOIN t2 ON t1.id = t2.id WHERE t2.name IS NOT NULL
```

7. **Using ANY with MVA**: When using the `ANY()` function with multi-valued attributes in JOINs, alias the multi-valued attribute from the joined table:
```sql
SELECT *, t2.m AS alias
FROM t
Expand Down
4 changes: 2 additions & 2 deletions manual/Server_settings/Searchd.md
Original file line number Diff line number Diff line change
Expand Up @@ -464,7 +464,7 @@ In most cases, a "job" means one query to a single local table (plain table or a

Table joins work by accumulating a batch of matches, which are the results of the query executed on the left table. This batch is then processed as a single query on the right table.

This option allows you to adjust the batch size. The default value is 1000, and setting this option to 0 disables batching.
This option allows you to adjust the batch size. The default value is `1000`, and setting this option to `0` disables batching.

A larger batch size may improve performance; however, for some queries, it can lead to excessive memory consumption.

Expand All @@ -484,7 +484,7 @@ Each query executed on the right table is defined by specific JOIN ON conditions

If there are only a few unique JOIN ON conditions, reusing the results can be more efficient than repeatedly executing queries on the right table. To enable this, the result sets are stored in a cache.

This option allows you to configure the size of this cache. The default value is 20 MB, and setting this option to 0 disables caching.
This option allows you to configure the size of this cache. The default value is `20 MB`, and setting this option to 0 disables caching.

Note that each thread maintains its own cache, so you should account for the number of threads executing queries when estimating total memory usage.

Expand Down

0 comments on commit 711c47d

Please sign in to comment.