Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add index mechanism for extension #5121

Merged
merged 13 commits into from
Jan 19, 2024

Conversation

guqing
Copy link
Member

@guqing guqing commented Dec 27, 2023

What type of PR is this?

/kind feature
/area core
/milestone 2.12.x

What this PR does / why we need it:

新增自定义模型索引机制

默认为所有的自定义模型都添加了以下索引:

  • metadata.name
  • metadata.labels
  • metadata.creationTimestamp
  • metadata.deletionTimestamp

how to test it?

  1. 测试应用的启动和停止
  2. 测试 Reconciler 被正确执行,如创建文章发布文章,测试删除文章的某个 label 数据启动后能被 PostReconciler 恢复(即Reconciler 被正确执行)
  3. 测试自定义模型自动生成的 list APIs
    1. 能根据 labels 正确过滤数据和分页
    2. 能根据 creationTimestamp 正确排序
    3. 测试插件启用后也能正确使用 list APIs 根据 labels 过滤数据和 creationTimestamp 排序
  4. 能正确删除数据(则表示 GcReconciler 使用索引正确)
  5. 测试在插件中为自定义模型注册索引
public class DemoPlugin extension BasePlugin {
    private final SchemeManager schemeManager;

    public MomentsPlugin(PluginContext pluginContext, SchemeManager schemeManager) {
        super(pluginContext);
        this.schemeManager = schemeManager;
    }

    @Override
    public void start() {
        schemeManager.register(Moment.class, indexSpecs -> {
            indexSpecs.add(new IndexSpec()
                .setName("spec.tags")
                .setIndexFunc(multiValueAttribute(Moment.class, moment -> {
                    var tags = moment.getSpec().getTags();
                    return tags == null ? Set.of() : tags;
                }))
            );
            indexSpecs.add(new IndexSpec()
                .setName("spec.owner")
                .setIndexFunc(simpleAttribute(Moment.class,
                    moment -> moment.getSpec().getOwner())
                )
            );
            indexSpecs.add(new IndexSpec()
                .setName("spec.releaseTime")
                .setIndexFunc(simpleAttribute(Moment.class, moment -> {
                    var releaseTime = moment.getSpec().getReleaseTime();
                    return releaseTime == null ? null : releaseTime.toString();
                }))
            );

            indexSpecs.add(new IndexSpec()
                .setName("spec.visible")
                .setIndexFunc(simpleAttribute(Moment.class, moment -> {
                    var visible = moment.getSpec().getVisible();
                    return visible == null ? null : visible.toString();
                }))
            );
        });
    }

    @Override
    public void stop() {
        // unregister scheme 即可,不需要手动删除索引
    }
}

可以正确在自动生成的 list APIs 使用 fieldSelector 来过滤 spec.slug 和排序,可以自己添加其他的 indexSpec 测试
6. 测试唯一索引并添加重复数据,期望无法添加进去

Which issue(s) this PR fixes:

Fixes #5058

Does this PR introduce a user-facing change?

新增自定义模型索引机制,优化查询效率和内存占用

@f2c-ci-robot f2c-ci-robot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Dec 27, 2023
@f2c-ci-robot f2c-ci-robot bot added this to the 2.12.x milestone Dec 27, 2023
@f2c-ci-robot f2c-ci-robot bot added the area/core Issues or PRs related to the Halo Core label Dec 27, 2023
@f2c-ci-robot f2c-ci-robot bot requested review from minliacom and wan92hen December 27, 2023 11:04
Copy link

codecov bot commented Dec 27, 2023

Codecov Report

Attention: 320 lines in your changes are missing coverage. Please review.

Comparison is base (3ebb45c) 55.98% compared to head (14fa3ef) 57.06%.

Files Patch % Lines
...n/halo/app/extension/index/query/QueryFactory.java 53.44% 26 Missing and 1 partial ⚠️
...o/app/extension/router/selector/LabelSelector.java 0.00% 25 Missing ⚠️
...n/java/run/halo/app/extension/PageRequestImpl.java 0.00% 23 Missing ⚠️
...halo/app/extension/router/selector/SetMatcher.java 0.00% 23 Missing ⚠️
...lo/app/extension/router/selector/SelectorUtil.java 0.00% 22 Missing ⚠️
...app/extension/router/selector/EqualityMatcher.java 0.00% 19 Missing ⚠️
...un/halo/app/extension/DefaultExtensionMatcher.java 28.57% 12 Missing and 3 partials ⚠️
...run/halo/app/extension/index/IndexBuilderImpl.java 53.12% 13 Missing and 2 partials ⚠️
...ension/router/selector/FieldSelectorConverter.java 0.00% 14 Missing ⚠️
...ension/router/selector/LabelSelectorConverter.java 0.00% 14 Missing ⚠️
... and 33 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5121      +/-   ##
============================================
+ Coverage     55.98%   57.06%   +1.07%     
- Complexity     3027     3333     +306     
============================================
  Files           538      584      +46     
  Lines         17973    19190    +1217     
  Branches       1319     1444     +125     
============================================
+ Hits          10062    10950     +888     
- Misses         7367     7664     +297     
- Partials        544      576      +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@guqing guqing force-pushed the feature/index branch 14 times, most recently from 7367f2d to 14deb2a Compare January 2, 2024 10:21
@guqing guqing marked this pull request as ready for review January 2, 2024 10:22
@f2c-ci-robot f2c-ci-robot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 2, 2024
@f2c-ci-robot f2c-ci-robot bot requested review from LIlGG and ruibaby January 2, 2024 10:24
@guqing guqing force-pushed the feature/index branch 3 times, most recently from 50a95d2 to 090e50d Compare January 3, 2024 08:44
@guqing guqing requested a review from JohnNiang January 16, 2024 09:39
Copy link
Member

@JohnNiang JohnNiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

目前似乎没有发现可以自定义 Indexer 的地方。

IMO,我们可以在构建索引之前任意添加 Indexer,无论是在 Halo Core 中还是在插件中。

@guqing
Copy link
Member Author

guqing commented Jan 17, 2024

目前似乎没有发现可以自定义 Indexer 的地方。

IMO,我们可以在构建索引之前任意添加 Indexer,无论是在 Halo Core 中还是在插件中。

目前自己添加 Indexer 的需求是什么呢?每个自定义模型添加 IndexSpec 后启动时就会构建索引并创建 Indexer,可以参考这个适配的 PR halo-sigs/plugin-moments#69

Copy link
Member

@JohnNiang JohnNiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@f2c-ci-robot f2c-ci-robot bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 19, 2024
Copy link
Member

@ruibaby ruibaby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@f2c-ci-robot f2c-ci-robot bot added the lgtm Indicates that a PR is ready to be merged. label Jan 19, 2024
@f2c-ci-robot f2c-ci-robot bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 19, 2024
Copy link
Member

@ruibaby ruibaby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@f2c-ci-robot f2c-ci-robot bot added the lgtm Indicates that a PR is ready to be merged. label Jan 19, 2024
Copy link

f2c-ci-robot bot commented Jan 19, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JohnNiang, ruibaby

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/core Issues or PRs related to the Halo Core kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

希望 Halo 提供内存索引功能提高数据的获取速度
3 participants