Feature store marketplace operator #441

harsh97 · 2023-11-21T04:11:18Z

An operator to assist in setting up feature store environment in tenancy via marketplace.
Steps to test:
ads opctl configure
ads init -t feature_store_marketplace
ads run -f <--generated--yaml--> -b marketplace

For more details refer documentation:

cd /ads/feature_store/docs
make livehtml
Go to this url for documentation http://127.0.0.1:8000/user_guides/setup/feature_store_operator.html

…e. Fix miscellaneous bugs

github-actions · 2023-11-21T04:46:15Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2023-11-21T06:47:06Z

📌 Cov diff with main:

📌 Overall coverage:

…e-operator' into feature/feature-store-marketplace-operator

github-actions · 2023-11-21T08:38:23Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2023-11-22T04:58:33Z

📌 Cov diff with main:

No success to gather report. 😿

📌 Overall coverage:

No success to gather report. 😿

ads/opctl/backend/marketplace/prerequisite_checker.py

github-actions · 2024-01-18T22:08:25Z

📌 Cov diff with main:

📌 Overall coverage:

guptadivyank · 2024-01-22T12:14:20Z

ads/feature_store/docs/source/transformation.rst

@@ -1,17 +1,21 @@
 Transformation
 **************

-Transformations in a feature store refers to the operations and processes applied to raw data to create, modify or derive new features that can be used as inputs for ML Models. These transformations are crucial for improving the quality, relevance and usefulness of features which in turn can enhance the performance of ml models. It is an object that represents a transformation applied on the feature group and can be a pandas transformation or spark sql transformation.


Please add the contract user must follow while authoring these transformations

Added the contract as part of the documentation

github-actions · 2024-01-23T05:25:02Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-01-23T05:59:17Z

📌 Cov diff with main:

📌 Overall coverage:

mrDzurb · 2024-01-24T00:21:55Z

ads/feature_store/docs/requirements.txt

 sphinx
 sphinxcontrib-napoleon
 sphinx_copybutton
 sphinx_code_tabs
 sphinx-autobuild
 sphinx-autorun
-oracle_ads==2.9.0rc0
+oracle_ads


Why do we need this line?

This is needed for generation of class files for feature store for class level documentation

mrDzurb · 2024-01-24T00:24:47Z

ads/feature_store/docs/source/user_guides.setup.getting_started.rst

@@ -0,0 +1,110 @@
+===============


This file will be removed?

yes. removed this file. thanks

mrDzurb · 2024-01-24T00:26:07Z

ads/opctl/backend/marketplace/helm_helper.py

+    if result.returncode == 0:
+        pass
+    else:
+        print(result.stderr)


shouldn't the logger be used instead?

done. thanks!1

ads/opctl/operator/common/backend_factory.py

ads/opctl/operator/lowcode/feature_store_marketplace/errors.py

tests/unitary/with_extras/operator/test_common_backend_factory.py

ads/opctl/operator/common/backend_factory.py

github-actions · 2024-01-24T02:15:41Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-01-24T02:47:46Z

📌 Cov diff with main:

📌 Overall coverage:

…-marketplace-operator

github-actions · 2024-01-24T03:38:25Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-01-24T04:08:24Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2024-01-24T08:34:22Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-01-24T09:05:16Z

📌 Cov diff with main:

📌 Overall coverage:

…e-operator' into feature/feature-store-marketplace-operator

github-actions · 2024-01-24T12:08:34Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-01-24T12:38:28Z

📌 Cov diff with main:

📌 Overall coverage:

qiuosier · 2024-01-24T14:38:25Z

pyproject.toml

@@ -190,10 +194,11 @@ pii = [
  "spacy==3.6.1",
 ]
 llm = [
-  "langchain>=0.0.295",
+  "langchain>=0.0.295,<=0.0.350",


If you restrict langchain<=0.0.350, you will also need langchain_community<=0.0.14 to pass the unit tests. Are you planning to remove this restriction in the future?

github-actions · 2024-01-24T15:58:14Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

qiuosier · 2024-01-24T16:05:41Z

ads/common/oci_client.py

@@ -48,12 +51,13 @@ class OCIClientFactory:
    oc.OCIClientFactory(**auth).object_storage # Creates Object storage client using instance principal authentication
    """

-    def __init__(self, config={}, signer=None, client_kwargs=None):
+    def __init__(self, config=(), signer=None, client_kwargs=None):


why using () as default here? I think we need {} as default. Otherwise there might be an error in some cases when using resource principal. The following example works in the existing ADS:

>>> import ads >>> ads.set_auth("resource_principal") >>> import oci >>> signer = signer = oci.auth.signers.get_resource_principals_signer() >>> from ads.common.oci_client import OCIClientFactory >>> OCIClientFactory(signer=signer).object_storage <oci.object_storage.object_storage_client.ObjectStorageClient object at 0x7fb04b094910>

but the same code will fail with this PR:

>>> import ads >>> ads.set_auth("resource_principal") >>> import oci >>> signer = signer = oci.auth.signers.get_resource_principals_signer() >>> from ads.common.oci_client import OCIClientFactory >>> OCIClientFactory(signer=signer).object_storage Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/conda/lib/python3.8/site-packages/ads/common/oci_client.py", line 108, in object_storage return self.create_client("object_storage") File "/opt/conda/lib/python3.8/site-packages/ads/common/oci_client.py", line 104, in create_client return client(config=self.config, signer=self.signer, **kwargs) File "/opt/conda/lib/python3.8/site-packages/oci/object_storage/object_storage_client.py", line 119, in __init__ self.base_client = BaseClient("object_storage", config, signer, object_storage_type_mapping, **base_client_init_kwargs) File "/opt/conda/lib/python3.8/site-packages/oci/base_client.py", line 304, in __init__ endpoint=config.get('endpoint'), AttributeError: 'tuple' object has no attribute 'get'

However probably it was a bad idea to put {} as default in the arguments. Maybe we should add some logic to set the default.

if not config: config = {}

good catch!!!

github-actions · 2024-01-24T16:25:18Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2024-01-24T16:26:05Z

📌 Cov diff with main:

📌 Overall coverage:

KshitizLohia

LGTM.

github-actions · 2024-01-24T16:56:51Z

📌 Cov diff with main:

📌 Overall coverage:

najiyacl and others added 17 commits November 16, 2023 17:25

bug fix

1109802

init operator

3fbcda9

scaffolding changes

22553e3

changes

f322656

Complete scaffolding

51969cf

Implement ads init and ads run for marketplace operator

dd143ea

Updating schema for helidon

ee6c6a7

Review changes

ddcff7c

Updating metastore id

10a57fa

Added a way to export helm chart from marketplace

a53d7d5

License checker

3807662

adeded release notes

d6ddf7c

adeded release notes

53bb9c3

Fs bug fix (#440)

0cb3b29

temp changes

f8a664e

updating schema yaml for marketplace

ca4c1df

Revert to old api server for marketplace e2e. Fix security token issu…

c9ac25f

…e. Fix miscellaneous bugs

harsh97 requested a review from KshitizLohia November 21, 2023 04:11

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Nov 21, 2023

amitkrprajapati and others added 2 commits November 21, 2023 10:25

Upgrade helm if chart it already exist.

fa61db9

Added changes to operator (#442)

ce76a1e

amitkrprajapati added 2 commits November 21, 2023 13:34

FIX MERGE CONFLICT

1d2e353

Merge remote-tracking branch 'origin/feature/feature-store-marketplac…

617eea8

…e-operator' into feature/feature-store-marketplace-operator

harsh97 added 2 commits November 22, 2023 10:24

Major refactoring and addition of helmers

b6e5fa5

changes to schema yaml

2659178

github-advanced-security bot found potential problems Nov 22, 2023

View reviewed changes

ads/opctl/backend/marketplace/prerequisite_checker.py Fixed Show fixed Hide fixed

ads/opctl/backend/marketplace/prerequisite_checker.py Fixed Show fixed Hide fixed

guptadivyank reviewed Jan 22, 2024

View reviewed changes

revert uneeded modifications

3f63194

mrDzurb reviewed Jan 24, 2024

View reviewed changes

remove uneeded file

b69dde3

Merge remote-tracking branch 'origin/main' into feature/feature-store…

6d4632a

…-marketplace-operator

addressed comments

8f87171

harsh97 added 2 commits January 24, 2024 17:36

Addressing review comments

50425af

Merge remote-tracking branch 'origin/feature/feature-store-marketplac…

ecb4de3

…e-operator' into feature/feature-store-marketplace-operator

qiuosier reviewed Jan 24, 2024

View reviewed changes

Fixing requirements.txt

4de3355

qiuosier reviewed Jan 24, 2024

View reviewed changes

Addressing cli client bug

24e2c1c

qiuosier approved these changes Jan 24, 2024

View reviewed changes

KshitizLohia approved these changes Jan 24, 2024

View reviewed changes

harsh97 merged commit f04a4a6 into main Jan 24, 2024
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature store marketplace operator #441

Feature store marketplace operator #441

harsh97 commented Nov 21, 2023 •

edited

Loading

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Jan 18, 2024

guptadivyank Jan 22, 2024

KshitizLohia Jan 24, 2024

github-actions bot commented Jan 23, 2024

github-actions bot commented Jan 23, 2024

mrDzurb Jan 24, 2024

KshitizLohia Jan 24, 2024

mrDzurb Jan 24, 2024

KshitizLohia Jan 24, 2024

mrDzurb Jan 24, 2024

harsh97 Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

qiuosier Jan 24, 2024

github-actions bot commented Jan 24, 2024

qiuosier Jan 24, 2024

harsh97 Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

KshitizLohia left a comment

github-actions bot commented Jan 24, 2024

Feature store marketplace operator #441

Feature store marketplace operator #441

Conversation

harsh97 commented Nov 21, 2023 • edited Loading

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Jan 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 23, 2024

github-actions bot commented Jan 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

Choose a reason for hiding this comment

github-actions bot commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

KshitizLohia left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 24, 2024

harsh97 commented Nov 21, 2023 •

edited

Loading