Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement] Add support for elasticsearch and openSearch #273

Merged
merged 5 commits into from
Nov 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 6 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,40 +2,19 @@
title: Advanced Object Search
---

# Pimcore Advanced Object Search via OpenSearch
# Pimcore Advanced Object Search via OpenSearch or Elasticsearch

Advanced Object Search bundle provides advanced object search in
Pimcore backend powered by OpenSearch.
Pimcore backend powered by search index technology (OpenSearch or Elasticsearch).

## Integration into Pimcore

### Installation and Configuration
Follow [Installation instructions](./doc/00_Installation.md).

#### Configure OpenSearch Client
OpenSearch client configuration takes place via [Pimcore OpenSearch Client Bundle](https://github.com/pimcore/opensearch-client) and has two parts.
1) Configuring an OpenSearch client.
2) Define the client to be used by advanced object search.

```yaml

# Configure an OpenSearch client
pimcore_open_search_client:
clients:
default:
hosts: ['https://opensearch:9200']
password: 'admin'
username: 'admin'
ssl_verification: false


# Define the client to be used by advanced object search
advanced_object_search:
client_name: default # default is default value here, just need to be specified when other client should be used.
```

If nothing is configured, a default client connecting to `localhost:9200` is used.

#### Configure Search Client
Setup search client configuration in your Symfony configuration files (e.g. `config.yaml`).
See [OpenSearch Client Setup](./doc/04_Opensearch.md) or [Elasticsearch Client Setup](./doc/05_Elasticsearch.md) for more information.

#### Configure Advanced Object Search
Before starting, setup at least following configuration in symfony configuration tree:
Expand Down Expand Up @@ -69,7 +48,7 @@ Following event listeners are called automatically

### Pimcore Console
Functions in Pimcore console.
- `advanced-object-search:process-update-queue` --> processes whole update queue of es search index.
- `advanced-object-search:process-update-queue` --> processes whole update queue of search index.
- `advanced-object-search:re-index` --> Reindex all data objects of given class. Does not delete index first or resets update queue.
- `advanced-object-search:update-mapping` --> Deletes and recreates mapping of given classes. Resets update queue for given class.

Expand Down
5 changes: 3 additions & 2 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@
"minimum-stability": "dev",
"require": {
"handcraftedinthealps/elasticsearch-dsl": "^8.0",
"pimcore/opensearch-client": "^1.0.0",
"pimcore/pimcore": "^11.1",
"pimcore/opensearch-client": "^1.x-dev",
"pimcore/elasticsearch-client": "^1.x-dev",
"pimcore/pimcore": "11.x-dev",
"symfony/config": "^6.2",
"symfony/console": "^6.2",
"symfony/dependency-injection": "^6.2",
Expand Down
2 changes: 2 additions & 0 deletions doc/00_Installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

This bundle is only supported on Pimcore Core Framework 11.

This bundle requires minimum version of OpenSearch 2.7. or Elasticsearch 8.0.0.

:::

## Installation
Expand Down
10 changes: 10 additions & 0 deletions doc/03_Upgrade_Notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,13 @@
### Upgrade to v6.0.0
- Removed Pimcore 10 support
- Removed Elasticsearch support and added OpenSearch support (Kept ONGR Elasticsearch library as it is compatible with OpenSearch)

### Upgrade to v6.1.0
- Added support for Elasticsearch in parallel to Opensearch. Opensearch remains the default search technology. If you are using Elasticsearch, you need to update your symfony configuration as follows:
```yml
advanced_object_search:
client_name: default
client_type: 'elasticsearch'
```
- Introduced new service alias `pimcore.advanced_object_search.search-client`. This will replace deprecated alias `pimcore.advanced_object_search.opensearch-client` which will be removed in the next major version.
The new service alias can be used to inject the search client into your services. This search client is an instance of `Pimcore\SearchClient\SearchClientInterface` which is a common interface for OpenSearch and Elasticsearch clients.
31 changes: 31 additions & 0 deletions doc/04_Opensearch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# OpenSearch Client Setup

:::info

This bundle requires minimum version of OpenSearch 2.7.

:::

Following configuration is required to set up OpenSearch. The OpenSearch client configuration takes place via [Pimcore Opensearch Client](https://github.com/pimcore/opensearch-client) and has two parts:
1) Configuring an OpenSearch client.
2) Define the client to be used by Advanced Object Search bundle.

```yaml
# Configuring an OpenSearch client
pimcore_open_search_client:
clients:
default:
hosts: ['https://opensearch:9200']
password: 'admin'
username: 'admin'
ssl_verification: false


# Define the client to be used by advanced object search
advanced_object_search:
client_name: default # default is default value here, just need to be specified when other client should be used.
```

If nothing is configured, a default client connecting to `localhost:9200` is used.

For the further configuration of the client, please refer to the [Pimcore OpenSearch Client documentation](https://github.com/pimcore/opensearch-client/blob/1.x/doc/02_Configuration.md).
31 changes: 31 additions & 0 deletions doc/05_Elasticsearch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Elasticsearch Client Setup

:::info

This bundle requires minimum version of Elasticsearch 8.0.

:::

Following configuration is required to set up Elasticsearch. The Elasticsearch client configuration takes place via [Pimcore Elasticsearch Client](https://github.com/pimcore/elasticsearch-client) and has two parts:
1) Configuring an Elasticsearch client.
2) Define the client to be used by Advanced Object Search bundle.

```yaml
# Configuring an Elasticsearch client
pimcore_elasticsearch_client:
es_clients:
default:
hosts: ['elastic:9200']
username: 'elastic'
password: 'somethingsecret'
logger_channel: 'pimcore.elasicsearch'

# Define the client to be used by advanced object search
advanced_object_search:
client_name: default # default is default value here, just need to be specified when other client should be used.
client_type: 'elasticsearch' # default is 'openSearch'
```

If nothing is configured, a default client connecting to `localhost:9200` is used.

For the further configuration of the client, please refer to the [Pimcore Elasticsearch Client documentation](https://github.com/pimcore/elasticsearch-client/blob/1.x/README.md).
2 changes: 2 additions & 0 deletions src/AdvancedObjectSearchBundle.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

namespace AdvancedObjectSearchBundle;

use Pimcore\Bundle\ElasticsearchClientBundle\PimcoreElasticsearchClientBundle;
use Pimcore\Bundle\OpenSearchClientBundle\PimcoreOpenSearchClientBundle;
use Pimcore\Bundle\SimpleBackendSearchBundle\PimcoreSimpleBackendSearchBundle;
use Pimcore\Extension\Bundle\AbstractPimcoreBundle;
Expand Down Expand Up @@ -103,6 +104,7 @@ public function getInstaller(): Installer

public static function registerDependentBundles(BundleCollection $collection): void
{
$collection->addBundle(new PimcoreElasticsearchClientBundle());
$collection->addBundle(new PimcoreOpenSearchClientBundle());
$collection->addBundle(new PimcoreSimpleBackendSearchBundle());
}
Expand Down
33 changes: 31 additions & 2 deletions src/DependencyInjection/AdvancedObjectSearchExtension.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,10 @@

namespace AdvancedObjectSearchBundle\DependencyInjection;

use AdvancedObjectSearchBundle\Enum\ClientType;
use AdvancedObjectSearchBundle\Maintenance\UpdateQueueProcessor;
use AdvancedObjectSearchBundle\Messenger\QueueHandler;
use RuntimeException;
use Symfony\Component\Config\FileLocator;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\DependencyInjection\Extension\PrependExtensionInterface;
Expand Down Expand Up @@ -64,9 +66,19 @@ public function loadInternal(array $config, ContainerBuilder $container)

$definition = $container->getDefinition(UpdateQueueProcessor::class);
$definition->setArgument('$messengerQueueActivated', $config['messenger_queue_processing']['activated']);
if ($config['client_type'] === ClientType::OPEN_SEARCH->value) {
$openSearchClientId = 'pimcore.open_search_client.' . $config['client_name'];
$container->setAlias('pimcore.advanced_object_search.opensearch-client', $openSearchClientId)
->setDeprecated(
'pimcore/advanced-object-search',
'6.1',
'The "%alias_id%" service alias is deprecated and will be removed in version 7.0. ' .
'Please use "pimcore.advanced_object_search.search-client" instead.'
);
}

$openSearchClientId = 'pimcore.open_search_client.' . $config['client_name'];
$container->setAlias('pimcore.advanced_object_search.opensearch-client', $openSearchClientId);
$clientId = $this->getDefaultSearchClientId($config);
$container->setAlias('pimcore.advanced_object_search.search-client', $clientId);
}

/**
Expand All @@ -83,4 +95,21 @@ public function prepend(ContainerBuilder $container)
$loader->load('doctrine_migrations.yml');
}
}

/**
* @throws RuntimeException
*/
private function getDefaultSearchClientId(array $indexSettings): string
{
$clientType = $indexSettings['client_type'];
$clientName = $indexSettings['client_name'];

return match ($clientType) {
ClientType::OPEN_SEARCH->value => 'pimcore.openSearch.custom_client.' . $clientName,
ClientType::ELASTIC_SEARCH->value => 'pimcore.elasticsearch.custom_client.' . $clientName,
default => throw new RuntimeException(
sprintf('Invalid client type: %s', $clientType)
)
};
}
}
8 changes: 7 additions & 1 deletion src/DependencyInjection/Configuration.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

namespace AdvancedObjectSearchBundle\DependencyInjection;

use AdvancedObjectSearchBundle\Enum\ClientType;
use Symfony\Component\Config\Definition\Builder\TreeBuilder;
use Symfony\Component\Config\Definition\ConfigurationInterface;

Expand All @@ -41,9 +42,14 @@ public function getConfigTreeBuilder()
->info('Prefix for index names')
->end()
->scalarNode('client_name')
->info('Name of OpenSearch client configuration to be used.')
->info('Name of search client configuration to be used.')
->defaultValue('default')
->end()
->enumNode('client_type')
->info('Type of search client to be used.')
->values([ClientType::OPEN_SEARCH->value, ClientType::ELASTIC_SEARCH->value])
->defaultValue(ClientType::OPEN_SEARCH->value)
->end()
->arrayNode('index_configuration')
->info('Add mapping between data object type and service implementation for field definition adapter')
->children()
Expand Down
26 changes: 26 additions & 0 deletions src/Enum/ClientType.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
<?php
declare(strict_types=1);

/**
* Pimcore
*
* This source file is available under two different licenses:
* - GNU General Public License version 3 (GPLv3)
* - Pimcore Commercial License (PCL)
* Full copyright and license information is available in
* LICENSE.md which is distributed with this source code.
*
* @copyright Copyright (c) Pimcore GmbH (http://www.pimcore.org)
* @license http://www.pimcore.org/license GPLv3 and PCL
*/

namespace AdvancedObjectSearchBundle\Enum;

/**
* @internal
*/
enum ClientType: string
{
case OPEN_SEARCH = 'openSearch';
case ELASTIC_SEARCH = 'elasticsearch';
}
78 changes: 78 additions & 0 deletions src/Factory/OpenSearch/LegacyServiceFactory.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
<?php
declare(strict_types=1);

/**
* Pimcore
*
* This source file is available under two different licenses:
* - GNU General Public License version 3 (GPLv3)
* - Pimcore Commercial License (PCL)
* Full copyright and license information is available in
* LICENSE.md which is distributed with this source code.
*
* @copyright Copyright (c) Pimcore GmbH (http://www.pimcore.org)
* @license http://www.pimcore.org/license GPLv3 and PCL
*/

namespace AdvancedObjectSearchBundle\Factory\OpenSearch;

use AdvancedObjectSearchBundle\Service;
use AdvancedObjectSearchBundle\Tools\IndexConfigService;
use OpenSearch\ClientBuilder;
use Pimcore\Bundle\ElasticsearchClientBundle\SearchClient\ElasticsearchClientInterface;
use Pimcore\Bundle\OpenSearchClientBundle\SearchClient\OpenSearchClientInterface;
use Pimcore\SearchClient\SearchClientInterface;
use Pimcore\Security\User\TokenStorageUserResolver;
use Pimcore\Translation\Translator;
use Psr\Container\ContainerInterface;
use Psr\Log\LoggerInterface;
use RuntimeException;
use Symfony\Component\EventDispatcher\EventDispatcherInterface;

/**
* @internal
* @deprecated will be removed in version 7.0
*/
final class LegacyServiceFactory
{
public function __construct(
private readonly LoggerInterface $logger,
private readonly TokenStorageUserResolver $userResolver,
private readonly EventDispatcherInterface $eventDispatcher,
private readonly Translator $translator,
private readonly IndexConfigService $indexConfigService,
private readonly SearchClientInterface $client
)
{

}

public function create(
ContainerInterface $filterLocator
): Service
{
$openSearchClient = match (true) {
$this->client instanceof OpenSearchClientInterface => $this->client->getOriginalClient(),
$this->client instanceof ElasticsearchClientInterface => ClientBuilder::create()->build(),
default => null,
};

if ($openSearchClient === null) {
throw new RuntimeException('No client found for OpenSearch');
}

$service = new Service(
$this->logger,
$this->userResolver,
$filterLocator,
$this->eventDispatcher,
$this->translator,
$this->indexConfigService,
$openSearchClient
);

$service->setSearchClientInterface($this->client);

return $service;
}
}
7 changes: 6 additions & 1 deletion src/Resources/config/services.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,18 @@ services:
tags:
- { name: monolog.logger, channel: advanced_object_search.es }

bundle.advanced_object_search.service.legacy_factory:
class: AdvancedObjectSearchBundle\Factory\OpenSearch\LegacyServiceFactory
arguments:
$client: '@pimcore.advanced_object_search.search-client'

bundle.advanced_object_search.service:
alias: AdvancedObjectSearchBundle\Service

AdvancedObjectSearchBundle\Service:
factory: ['@bundle.advanced_object_search.service.legacy_factory', 'create']
arguments:
$filterLocator: '@bundle.advanced_object_search.filter_locator'
$openSearchClient: '@pimcore.advanced_object_search.opensearch-client'
calls:
- [setCoreFieldsConfig, ['%advanced_object_search.core_fields_configuration%']]
tags:
Expand Down
Loading
Loading