Ability for dynamic configuration through config files

[virajkanwade]

It would be nice to be able to configure piccolo through runtime app configs instead of a conf file. It would help write framework specific extensions for piccolo.

E.g. flask_sqlalchemy https://flask-sqlalchemy.palletsprojects.com/en/2.x/quickstart/

from flask import Flask
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)

If you write it as follows:

def get_db_engine():
    return PostgresEngine(
        config={
            "database": app.configuration.pg_db_name,
            "user": app.configuration.pg_user,
            "password": app.configuration.pg_password,
            "host": app.configuration.pg_host,
            "port": app.configuration.pg_port,
        }
    )

async def open_database_connection_pool(application):
    try:
        engine = get_db_engine()
        await engine.start_connection_pool()
    except Exception:
        print("Unable to connect to the database")

class Task(Table, db=get_db_engine()):
    """
    An example table.
    """
    name = Varchar()
    completed = Boolean(default=False)

You can bypass piccolo_conf.py.

But then when running piccolo migrations new home it becomes a challenge, since it relies a lot on Finder looking for piccolo_conf.py.

APP_REGISTRY: AppRegistry = Finder().get_app_registry()
engine = Finder().get_engine()
app_config = Finder().get_app_config(app_name=app_name)

If instead of looking up piccolo_conf.py module everytime, if we get to a state where it is read only once and engine and app_config are available everywhere else, one day we might be able to make piccolo_conf.py optional.

[dantownsend]

Yeah, it's a tricky one.

For simple projects, you can just directly pass the database config to your Table:

DB = PostgresEngine(...)

class MyTable(db=DB):
    ...

You have to be careful that there is only one PostgresEngine instance - if it's generated inside a function, you might end up with several of them.

# This would be problematic - we need one shared PostgresEngine for the entire application, so
# we can use a single connection pool
def get_engine():
    return PostgresEngine(...)

class MyTable(db=get_engine()):
    ...

class MyOtherTable(db=get_engine()):
    ...

For a more complex project, you currently need piccolo_conf.py. It serves two purposes:

• stores the database credentials
• stores a list of Piccolo apps

At the moment, the migration code requires a piccolo_conf.py file, because it needs to know the database credentials, and which Piccolo apps are available. There's currently no way to specify which Piccolo apps are available except via piccolo_conf.py.

One thing to be aware of, is you can customise the location of piccolo_conf.py using an environment variable. So let's say you have an application, and you want to store all of the configuration in settings.py, you can configure Piccolo to import DB and APP_REGISTRY from there instead by using the PICCOLO_CONF environment variable.

PICCOLO_CONF=settings

In the future, it might be possible to have a Piccolo class, which sets the database and app settings in some global context:

from piccolo import Piccolo

Piccolo(db=PostgresEngine(...), app_config=AppConfig(...))

Then Finder would look for either piccolo_conf.py, or this global context.

[virajkanwade]

@dantownsend This is what I did for making piccolo_conf.py configurable

""" Piccolo ORM conf file """
import pkgutil

from piccolo.conf.apps import AppRegistry
from piccolo.engine.postgres import PostgresEngine

from app.configuration import load_configuration


configuration = load_configuration()

DB = PostgresEngine(
    config={
        "database": configuration.pg_db_name,
        "user": configuration.pg_user,
        "password": configuration.pg_password,
        "host": configuration.pg_host,
        "port": configuration.pg_port,
    }
)

piccolo_apps = []

for _p in pkgutil.walk_packages(['.']):
    if not _p.ispkg and _p.name.endswith('piccolo_app'):
            piccolo_apps.append(_p.name)

# if configuration.environment == 'dev':
#    piccolo_apps.append('piccolo_admin.piccolo_app')

APP_REGISTRY = AppRegistry(
    apps=piccolo_apps
)

""" Load app configuration """
import os

# https://github.com/Neoteroi/essentials-configuration
from configuration.common import Configuration, ConfigurationBuilder
from configuration.env import EnvironmentVariables
from configuration.yaml import YAMLFile

environment_name = os.environ.get("APP_ENVIRONMENT", "dev")


def load_configuration() -> Configuration:
    """Create Configuration object from config"""
    builder = ConfigurationBuilder(
        YAMLFile(os.path.join("config", "config.yaml")),
        YAMLFile(os.path.join("config", f"config.{environment_name}.yaml"), optional=True),
        EnvironmentVariables(prefix="APP_"),
    )

    return builder.build()

Please let me know if this could result in some side-effects!

Thanks

[dantownsend]

It's interesting how you automatically loaded the apps. What you've done should work OK.

[virajkanwade]

Similar to table_finder could be app_finder

def app_finder(paths: List)
    piccolo_apps = []

    for _p in pkgutil.walk_packages(paths):
        if not _p.ispkg and _p.name.endswith('piccolo_app'):
                piccolo_apps.append(_p.name)

    return piccolo_apps

[dantownsend]

It's an interesting idea. It isn't something I've thought about before.

I think people would find this useful if they had a few apps.

[virajkanwade]

piccolo_app.py

assuming

home
> tables
    > __init__.py
    > piccolo_app.py
    > piccolo_migrations
    > organization.py
    > user.py
    > blog.py
    > post.py

"""
Import all of the Tables subclasses in your app here, and register them with
the APP_CONFIG.
"""

import os
import pkgutil

from piccolo.conf.apps import AppConfig, table_finder

migrations_folder_path: str = os.path.join(
    os.path.dirname(os.path.abspath(__file__)), "piccolo_migrations"
)

os.makedirs(migrations_folder_path, exist_ok=True)

table_modules: list = []

module_parents: list = __name__.split(".")[:-1]

for _p in pkgutil.walk_packages([os.path.join(*module_parents)]):
    if not _p.ispkg and _p.name != "piccolo_app":
        table_modules.append(".".join(module_parents + [_p.name]))

APP_CONFIG = AppConfig(
    app_name="home",
    migrations_folder_path=migrations_folder_path,
    table_classes=table_finder(modules=table_modules, exclude_imported=True),
    migration_dependencies=[],
    commands=[],
)

NOTE: Caveat, will only work if above folder structure