[Feature/multi_tenancy] [WIP] SdkClient interface and implementation #2430

dbwiddis · 2024-05-09T20:37:03Z

Description

Work In Progress

Creates an abstraction for index metadata, see opensearch-project/OpenSearch#13336 and initial implementation for Connector.

The near-term goal is proving out the flexibility of the interface.

Still TODO:

Update and Search APIs
Thorough exception handling and user error messages
More documentation

Issues Resolved

Part of #2358

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Daniel Widdis <[email protected]>

dhrubo-os · 2024-05-09T20:52:37Z

plugin/src/main/java/org/opensearch/ml/sdkclient/XContentClient.java

+    }
+
+    @Override
+    public CompletionStage<PutCustomResponse> putCustom(PutCustomRequest request) {


What about Update?

Coming soon to a PR near you, but prioritizing initial APIs to support multitenancy work.

dhrubo-os · 2024-05-09T20:56:15Z

plugin/src/main/java/org/opensearch/ml/sdkclient/XContentClient.java

+                new DeleteRequest(request.index(), request.id()),
+                ActionListener
+                    .wrap(
+                        r -> future.complete(new DeleteCustomResponse.Builder().id(r.getId()).deleted(r.getResult() == DELETED).build()),


What happens if the deletion fails? How are we sending the error response back to the customer?

The actionListener for the whole execution is being completed exceptionally. I have not yet gone through to more carefully identify exception possibilities and give more detailed user information. Again, this is an early-stage WIP.

dhrubo-os · 2024-05-09T23:29:45Z

common/src/main/java/org/opensearch/sdk/DeleteCustomResponse.java

+
+    /**
+     * Returns whether deletion was successful
+     * @return true if deletion was successful, false if the object was not found


what happens if the object was found but for some reason we couldn't delete.

The onFailure / exception path will be followed instead. This is the same behavior as the current DeleteResponse where you can have either DELETED (true) or NOT_FOUND (false) successful response.

dhrubo-os · 2024-05-10T00:14:06Z

I'm seeing:

/home/runner/work/ml-commons/ml-commons/common/src/main/java/org/opensearch/ml/common/connector/HttpConnector.java:43: error: HttpConnector is not abstract and does not override abstract method parse(XContentParser) in Custom
public class HttpConnector extends AbstractConnector {

Are you planning to look into this?

dbwiddis · 2024-05-10T00:26:00Z

I'm seeing:

/home/runner/work/ml-commons/ml-commons/common/src/main/java/org/opensearch/ml/common/connector/HttpConnector.java:43: error: HttpConnector is not abstract and does not override abstract method parse(XContentParser) in Custom
public class HttpConnector extends AbstractConnector {

Are you planning to look into this?

Yeah, I missed actually implementing the parse() method in my own interface that needs to wrap the (static) xContent parsing methods in Connector. I'll push a fix tomorrow.

Signed-off-by: Daniel Widdis <[email protected]>

dbwiddis added 2 commits May 9, 2024 13:25

Create SdkClient interface and XContent implementation

fb99c9a

Signed-off-by: Daniel Widdis <[email protected]>

Implement put, get, delete for Connector

d3fdbdb

Signed-off-by: Daniel Widdis <[email protected]>

dbwiddis had a problem deploying to ml-commons-cicd-env May 9, 2024 20:37 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 9, 2024 20:37 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env May 9, 2024 20:37 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 9, 2024 20:37 — with GitHub Actions Failure

dbwiddis mentioned this pull request May 9, 2024

[RFC] Data Access Object Interface for Metadata Store opensearch-project/OpenSearch#13336

Open

dhrubo-os reviewed May 9, 2024

View reviewed changes

Implement Custom interface parse method in Connector interface

79b2e1f

Signed-off-by: Daniel Widdis <[email protected]>

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 04:26 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 04:26 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 04:26 — with GitHub Actions Error

Fix test compiling. Still expect failures on exceptions/messages

b5d0e2a

Signed-off-by: Daniel Widdis <[email protected]>

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 04:38 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 04:38 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 04:39 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 04:39 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env May 10, 2024 15:19 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 07:03 — with GitHub Actions Error

dbwiddis temporarily deployed to ml-commons-cicd-env May 17, 2024 07:03 — with GitHub Actions Inactive

dbwiddis force-pushed the feature_multi_tenancy branch from da5f1f3 to 7d03c1e Compare May 17, 2024 07:45

dbwiddis temporarily deployed to ml-commons-cicd-env May 17, 2024 07:45 — with GitHub Actions Inactive

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 08:34 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 08:34 — with GitHub Actions Error

dbwiddis added 2 commits May 17, 2024 09:12

Add test classes

b984ad5

Signed-off-by: Daniel Widdis <[email protected]>

Refactor clients with supplyAsync, handle not found on get

fd22232

Signed-off-by: Daniel Widdis <[email protected]>

dbwiddis force-pushed the feature_multi_tenancy branch from 7d03c1e to fd22232 Compare May 17, 2024 16:13

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 16:13 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 16:13 — with GitHub Actions Failure

dbwiddis temporarily deployed to ml-commons-cicd-env May 17, 2024 16:13 — with GitHub Actions Inactive

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 17:02 — with GitHub Actions Error

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 17:02 — with GitHub Actions Failure

dbwiddis had a problem deploying to ml-commons-cicd-env May 17, 2024 17:02 — with GitHub Actions Error

peterzhuamazon deleted the branch opensearch-project:feature_multi_tenancy May 17, 2024 17:28

peterzhuamazon closed this May 17, 2024

dbwiddis mentioned this pull request May 17, 2024

[Feature/multi_tenancy] SdkClient interface with local and remote implementations for Connector #2459

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature/multi_tenancy] [WIP] SdkClient interface and implementation #2430

[Feature/multi_tenancy] [WIP] SdkClient interface and implementation #2430

dbwiddis commented May 9, 2024 •

edited

Loading

dhrubo-os May 9, 2024

dbwiddis May 9, 2024

dhrubo-os May 9, 2024

dbwiddis May 9, 2024

dhrubo-os May 9, 2024 •

edited

Loading

dbwiddis May 9, 2024

dhrubo-os commented May 10, 2024

dbwiddis commented May 10, 2024

[Feature/multi_tenancy] [WIP] SdkClient interface and implementation #2430

[Feature/multi_tenancy] [WIP] SdkClient interface and implementation #2430

Conversation

dbwiddis commented May 9, 2024 • edited Loading

Description

Issues Resolved

Check List

dhrubo-os May 9, 2024

Choose a reason for hiding this comment

dbwiddis May 9, 2024

Choose a reason for hiding this comment

dhrubo-os May 9, 2024

Choose a reason for hiding this comment

dbwiddis May 9, 2024

Choose a reason for hiding this comment

dhrubo-os May 9, 2024 • edited Loading

Choose a reason for hiding this comment

dbwiddis May 9, 2024

Choose a reason for hiding this comment

dhrubo-os commented May 10, 2024

dbwiddis commented May 10, 2024

dbwiddis commented May 9, 2024 •

edited

Loading

dhrubo-os May 9, 2024 •

edited

Loading