Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move credentials resolver to the config component #4259

Open
ElenaKhaustova opened this issue Oct 28, 2024 · 3 comments
Open

Move credentials resolver to the config component #4259

ElenaKhaustova opened this issue Oct 28, 2024 · 3 comments
Labels
Issue: Feature Request New feature or improvement to existing feature

Comments

@ElenaKhaustova
Copy link
Contributor

Description

Currently, we pass credentials to the catalog separately from the rest configuration:

credentials=conf_creds,

Then we resolve credentials at the CatalogConfigResolver level:

self._resolved_configs = self._resolve_config_credentials(config, credentials)

We suggest to move this resolution to the config component, then:

  1. We simplify CatalogConfigResolver, so it only resolves patterns and there's no confusion when using "resolve" term;
  2. All the configuration resolution will be handled in one place by config module

Context

This should be done as a preceding step to #3811

@ElenaKhaustova ElenaKhaustova added the Issue: Feature Request New feature or improvement to existing feature label Oct 28, 2024
@ElenaKhaustova ElenaKhaustova changed the title [DataCatalog]: Move credentials resolver to the config component Move credentials resolver to the config component Oct 28, 2024
@datajoely
Copy link
Contributor

Such a welcome move

@datajoely
Copy link
Contributor

datajoely commented Nov 19, 2024

Having just hit this in my current project, having something that was a more fully featured version of this would be a really welcome quality of life improvement.

def _get_local_creds(top_level_key: str, key: str) -> str:
    cred_data = OmegaConf.load("conf/local/credentials.yml")
    return cred_data[top_level_key][key]


CONFIG_LOADER_ARGS = {
    "base_env": "base",
    "default_run_env": "local",
    "custom_resolvers": {"creds": _get_local_creds},
}

In turn the YAML looks like this:

synthea_patients:
  type: ibis.TableDataset
  table_name: patients
  connection: 
    backend: postgres
    database: synthea
    user: "${creds:postgres, user}"
    password: "${creds:postgres, password}"

All in all it feels like the oc.env workflow but against our locals/credentials.yml which is a core part of the project template - the fundamental difference is that it takes the onus of integrating from Kedro's archaic credentials mechanism (from v0.0.1) away from the dataset implementation and developer.

@datajoely
Copy link
Contributor

Little hack to make my MVP resolver mask credentials in the __repr__
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Feature Request New feature or improvement to existing feature
Projects
Status: No status
Development

No branches or pull requests

2 participants