Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: discovery module with config model #241

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,11 @@ additional code to set up error handling and basic Bento service boilerplate.

`db` contains common base classes for setting up database managers.

### `discovery`

`discovery` contains models and helper functions for the Bento Discovery Configuration specification, used
in [Katsu](https://github.com/bento-platform/katsu).

### `drs`

`drs` provides utilities for fetching data and record metadata from
Expand Down
13 changes: 12 additions & 1 deletion bento_lib/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

from . import apps
from . import auth
from . import discovery
from . import drs
from . import events
from . import schemas
Expand All @@ -12,5 +13,15 @@

__version__ = metadata.version(__name__)
__all__ = [
"__version__", "apps", "auth", "drs", "events", "schemas", "search", "service_info", "streaming", "workflows"
"__version__",
"apps",
"auth",
"discovery",
"drs",
"events",
"schemas",
"search",
"service_info",
"streaming",
"workflows",
]
4 changes: 4 additions & 0 deletions bento_lib/discovery/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from . import helpers
from . import models

__all__ = ["helpers", "models"]
Empty file added bento_lib/discovery/helpers.py
Empty file.
Empty file.
22 changes: 22 additions & 0 deletions bento_lib/discovery/models/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
from pydantic import BaseModel

Check warning on line 1 in bento_lib/discovery/models/config.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/config.py#L1

Added line #L1 was not covered by tests

from .fields import DateFieldDefinition, NumberFieldDefinition, StringFieldDefinition
from .overview import OverviewSection
from .search import SearchSection

Check warning on line 5 in bento_lib/discovery/models/config.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/config.py#L3-L5

Added lines #L3 - L5 were not covered by tests

__all__ = [

Check warning on line 7 in bento_lib/discovery/models/config.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/config.py#L7

Added line #L7 was not covered by tests
"DiscoveryConfigRules",
"DiscoveryConfig",
]


class DiscoveryConfigRules(BaseModel):
count_threshold: int
max_query_parameters: int

Check warning on line 15 in bento_lib/discovery/models/config.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/config.py#L13-L15

Added lines #L13 - L15 were not covered by tests


class DiscoveryConfig(BaseModel):
overview: list[OverviewSection]
search: list[SearchSection]
fields: dict[str, DateFieldDefinition | NumberFieldDefinition | StringFieldDefinition]
rules: DiscoveryConfigRules

Check warning on line 22 in bento_lib/discovery/models/config.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/config.py#L18-L22

Added lines #L18 - L22 were not covered by tests
69 changes: 69 additions & 0 deletions bento_lib/discovery/models/fields.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
from pydantic import AliasChoices, BaseModel, Field
from typing import Literal

Check warning on line 2 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L1-L2

Added lines #L1 - L2 were not covered by tests

__all__ = [

Check warning on line 4 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L4

Added line #L4 was not covered by tests
"BaseFieldDefinition",
# string
"StringFieldConfig",
"StringFieldDefinition",
# number
"BaseNumberFieldConfig",
"ManualBinsNumberFieldConfig",
"AutoBinsNumberFieldConfig",
"NumberFieldDefinition",
# date
"DateFieldConfig",
"DateFieldDefinition",
]


class BaseFieldDefinition(BaseModel):
mapping: str
title: str # TODO: make optional and pull from Bento schema if not set
description: str # TODO: make optional and pull from Bento schema if not set
datatype: Literal["string", "number", "date"] = Field(validation_alias=AliasChoices("data_type", "datatype"))

Check warning on line 24 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L20-L24

Added lines #L20 - L24 were not covered by tests
# --- The below fields are currently valid, but need to be reworked for new search ---------------------------------
mapping_for_search_filter: str | None = None
group_by: str | None = None
group_by_value: str | None = None
value_mapping: str | None = None

Check warning on line 29 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L26-L29

Added lines #L26 - L29 were not covered by tests
# ------------------------------------------------------------------------------------------------------------------


class StringFieldConfig(BaseModel):
enum: list[str] | None

Check warning on line 34 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L33-L34

Added lines #L33 - L34 were not covered by tests


class StringFieldDefinition(BaseFieldDefinition):
datatype: Literal["string"]
config: StringFieldConfig

Check warning on line 39 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L37-L39

Added lines #L37 - L39 were not covered by tests


class BaseNumberFieldConfig(BaseModel):
units: str

Check warning on line 43 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L42-L43

Added lines #L42 - L43 were not covered by tests


class ManualBinsNumberFieldConfig(BaseNumberFieldConfig):
bins: list[int]

Check warning on line 47 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L46-L47

Added lines #L46 - L47 were not covered by tests


class AutoBinsNumberFieldConfig(BaseNumberFieldConfig):
bin_size: int
taper_left: int
taper_right: int
minimum: int
maximum: int

Check warning on line 55 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L50-L55

Added lines #L50 - L55 were not covered by tests


class NumberFieldDefinition(BaseFieldDefinition):
datatype: Literal["number"]
config: ManualBinsNumberFieldConfig | AutoBinsNumberFieldConfig

Check warning on line 60 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L58-L60

Added lines #L58 - L60 were not covered by tests


class DateFieldConfig(BaseModel):
bin_by: str # TODO

Check warning on line 64 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L63-L64

Added lines #L63 - L64 were not covered by tests


class DateFieldDefinition(BaseFieldDefinition):
datatype: Literal["date"]
config: DateFieldConfig

Check warning on line 69 in bento_lib/discovery/models/fields.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/fields.py#L67-L69

Added lines #L67 - L69 were not covered by tests
17 changes: 17 additions & 0 deletions bento_lib/discovery/models/overview.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
from pydantic import BaseModel
from typing import Literal

Check warning on line 2 in bento_lib/discovery/models/overview.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/overview.py#L1-L2

Added lines #L1 - L2 were not covered by tests

__all__ = [

Check warning on line 4 in bento_lib/discovery/models/overview.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/overview.py#L4

Added line #L4 was not covered by tests
"OverviewChart",
"OverviewSection",
]


class OverviewChart(BaseModel):
field: str
chart_type: Literal["bar", "choropleth", "histogram", "pie"]

Check warning on line 12 in bento_lib/discovery/models/overview.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/overview.py#L10-L12

Added lines #L10 - L12 were not covered by tests


class OverviewSection(BaseModel):
section_title: str
charts: list[OverviewChart]

Check warning on line 17 in bento_lib/discovery/models/overview.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/overview.py#L15-L17

Added lines #L15 - L17 were not covered by tests
8 changes: 8 additions & 0 deletions bento_lib/discovery/models/search.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from pydantic import BaseModel

Check warning on line 1 in bento_lib/discovery/models/search.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/search.py#L1

Added line #L1 was not covered by tests

__all__ = ["SearchSection"]

Check warning on line 3 in bento_lib/discovery/models/search.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/search.py#L3

Added line #L3 was not covered by tests


class SearchSection(BaseModel):
section_title: str
fields: list[str]

Check warning on line 8 in bento_lib/discovery/models/search.py

View check run for this annotation

Codecov / codecov/patch

bento_lib/discovery/models/search.py#L6-L8

Added lines #L6 - L8 were not covered by tests