How to configure alembic autogenerate with multiple model files

[lehoangnam040]

Describe the bug
Have a versions file that create Account. After add another Product model, Run alembic revision --autogenerate -m "..." doesn't create new table Product but drop current table Account on new ..py in versions

To Reproduce

• Project structure

.
├── devops
│   ├── db_migration
│   │   ├── alembic.ini
│   │   └── pg
│   │       ├── env.py
│   │       └── versions
│   │           ├── d65c5dec925e_create_account_table.py
├── service
│   ├── __init__.py
│   ├── databases
│   │   ├── __init__.py
│   │   ├── postgres
│   │   │   ├── __init__.py
│   │   │   ├── account.py
│   │   │   ├── product.py

• devops/db_migration/pg/env.py

myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + "/../../../")

config = context.config
if config.config_file_name is not None:
    fileConfig(config.config_file_name)

from service.databases.postgres import DB_URL, metadata
target_metadata = metadata
...

• devops/db_migration/pg/versions/d65c5dec925e_create_account_table.py

revision = 'd65c5dec925e'
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
    op.create_table('account',
    sa.Column('uid', sa.BigInteger(), autoincrement=False, nullable=False),
    ...
    )
def downgrade() -> None:
    op.drop_table('account')

• service/databases/__init__.py

DB_URL = f"postgresql://..."
metadata = sqlalchemy.MetaData()
database = databases.Database(DB_URL)

class PgBaseMeta(ormar.ModelMeta):
    database = database
    metadata = metadata

• service/databases/account.py

from . import PgBaseMeta
class PgAccount(ormar.Model):
    class Meta(PgBaseMeta):
        tablename = "account"
    uid = ormar.BigInteger(autoincrement=False, primary_key=True)
    username = ormar.String(max_length=64, unique=True)

• service/databases/product.py

from . import PgBaseMeta
class PgProduct(ormar.Model):
    class Meta(PgBaseMeta):
        tablename = "product"
    pid = ormar.BigInteger(autoincrement=False, primary_key=True)
    product_name = ormar.String(max_length=256)

• Run command ...devops/db_migration $ alembic revision --autogenerate -m "create product table" create new file devops/db_migration/pg/versions/64ec73d106d7_create_product_table.py

revision = '64ec73d106d7'
down_revision = 'd65c5dec925e'
branch_labels = None
depends_on = None
def upgrade() -> None:
    op.drop_table('account')
def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table('account',
    ...
    )

Expected behavior

• File devops/db_migration/pg/versions/64ec73d106d7_create_product_table.py should be

revision = '64ec73d106d7'
down_revision = 'd65c5dec925e'
branch_labels = None
depends_on = None
def upgrade() -> None:
    op.create_table('product')
def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_table('product',
    ...
    )

Versions (please complete the following information):

• postgresql 14
• Python version 3.11.4
• ormar version 0.12.2
• pydantic version 1.10.8
• fastapi version 0.99.1

[rorik302]

Have you try to import models to alembic env.py?

[lehoangnam040]

Have you try to import models to alembic env.py?

Thank you, it worked when I import in env.py as

from service.databases.postgres import account, product

but not worked if I import

from service.databases.postgres import *

And honestly, I don't want to modify env.py everytime I add a new model.

[WojciechPartyka]

I think you just might need to sort out your imports,
first you need module where your metadata and PgBaseMeta lies,
then in env.py import that module and import all models from all apps

#somewhere at the beggining of env.py
from app1.models import *
from app2.models import *
import module_with_base

and set target_metadata = module_with_base.PgBaseMeta.metadata
that kind of structure works fine on my side at least :D

[lehoangnam040]

and set target_metadata = module_with_base.PgBaseMeta.metadata that kind of structure works fine on my side at least :D

Thank you but it doesn't work on my side :))

[rorik302]

Have you try to import models to alembic env.py?

Thank you, it worked when I import in env.py as

from service.databases.postgres import account, product

but not worked if I import

from service.databases.postgres import *

And honestly, I don't want to modify env.py everytime I add a new model.

Create variable somewhere in project, for example:

# src/models.py
app_models = [account, product]

Then just import app_models to env.py. When you create new models just add them to models.py app_models

[Hassan-Ahmadi]

I am thinking of writing a util in my project to auto-discover the models.py in for example apps subdirectories and import them in the env.py dynamically. Do you think this can be the right way to handle this issue?

import os
import importlib

def import_all_models(app_name):
    # Relative path to the app directory
    app_directory = f"src.apps.{app_name}"

    # Construct absolute path to the app directory
    abs_path = os.path.join(os.path.dirname(__file__), 'src', 'apps', app_name)

    # Iterate over files in the app directory
    for filename in os.listdir(abs_path):
        if filename.endswith('.py') and filename != '__init__.py':
            module_name = f"{app_directory}.{filename[:-3]}"  # Remove .py extension
            importlib.import_module(module_name)  # Import the module