Merge pull request #346 from collerek/add_index

Add multi columns non-unique index and sql_nullable setting. Important performance booster for dict().

README.md

@@ -657,7 +657,8 @@ The following keyword arguments are supported on all field types.
 
 All fields are required unless one of the following is set:
 
-* `nullable` - Creates a nullable column. Sets the default to `None`.
+* `nullable` - Creates a nullable column. Sets the default to `False`. Read the fields common parameters for details.
+* `sql_nullable` - Used to set different setting for pydantic and the database. Sets the default to `nullable` value. Read the fields common parameters for details.
 * `default` - Set a default value for the field. **Not available for relation fields**
 * `server_default` - Set a default value for the field on server side (like sqlalchemy's `func.now()`). **Not available for relation fields**
 * `primary key` with `autoincrement` - When a column is set to primary key and autoincrement is set on this column.

docs/fields/common-parameters.md

@@ -22,18 +22,41 @@ Used both in sql and pydantic (changes pk field to optional for autoincrement).
 
 ## nullable
 
-`nullable`: `bool` = `not primary_key` -> defaults to False for primary key column, and True for all other. 
+`nullable`: `bool` = `False` -> defaults to False for all fields except relation fields.
+
+Automatically changed to True if user provide one of the following:
+
+* `default` value or function is provided
+* `server_default` value or function is provided
+* `autoincrement` is set on `Integer` `primary_key` field
+* **[DEPRECATED]**`pydantic_only=True` is set 
 
 Specifies if field is optional or required, used both with sql and pydantic.
 
+By default, used for both `pydantic` and `sqlalchemy` as those are the most common settings:
+
+* `nullable=False` - means database column is not null and field is required in pydantic
+* `nullable=True` - means database column is null and field is optional in pydantic
+
+If you want to set different setting for pydantic and the database see `sql_nullable` below.
+
 !!!note
     By default all `ForeignKeys` are also nullable, meaning the related `Model` is not required.
 
     If you change the `ForeignKey` column to `nullable=False`, it becomes required.
 
 
-!!!info
-    If you want to know more about how you can preload related models during queries and how the relations work read the [queries][queries] and [relations][relations] sections. 
+## sql_nullable
+
+`sql_nullable`: `bool` = `nullable` -> defaults to the value of nullable (described above). 
+
+Specifies if field is not null or allows nulls in the database only. 
+
+Use this setting in combination with `nullable` only if you want to set different options on pydantic model and in the database. 
+
+A sample usage might be i.e. making field not null in the database, but allow this field to be nullable in pydantic (i.e. with `server_default` value).
+That will prevent the updates of the field to null (as with `server_default` set you cannot insert null values already as the default value would be used)
+
 
 
 ## default

docs/index.md

@@ -660,7 +660,8 @@ The following keyword arguments are supported on all field types.
 
 All fields are required unless one of the following is set:
 
-  * `nullable` - Creates a nullable column. Sets the default to `None`.
+  * `nullable` - Creates a nullable column. Sets the default to `False`. Read the fields common parameters for details.
+  * `sql_nullable` - Used to set different setting for pydantic and the database. Sets the default to `nullable` value. Read the fields common parameters for details.
   * `default` - Set a default value for the field. **Not available for relation fields**
   * `server_default` - Set a default value for the field on server side (like sqlalchemy's `func.now()`). **Not available for relation fields**
   * `primary key` with `autoincrement` - When a column is set to primary key and autoincrement is set on this column. 

docs/models/index.md

@@ -357,11 +357,16 @@ You can overwrite this parameter by providing `Meta` class `tablename` argument.
 
 On a model level you can also set model-wise constraints on sql columns.
 
-Right now only `UniqueColumns` constraint is present. 
+Right now only `IndexColumns` and `UniqueColumns` constraints are supported. 
+
+!!!note
+        Note that both constraints should be used only if you want to set a name on constraint or want to set the index on multiple columns, otherwise `index` and `unique` properties on ormar fields are preferred.
 
 !!!tip
     To read more about columns constraints like `primary_key`, `unique`, `ForeignKey` etc. visit [fields][fields].
 
+#### UniqueColumns
+
 You can set this parameter by providing `Meta` class `constraints` argument.
 
 ```Python hl_lines="14-17"
@@ -373,6 +378,20 @@ You can set this parameter by providing `Meta` class `constraints` argument.
         To set one column as unique use [`unique`](../fields/common-parameters.md#unique) common parameter. 
         Of course you can set many columns as unique with this param but each of them will be checked separately.
 
+#### IndexColumns
+
+You can set this parameter by providing `Meta` class `constraints` argument.
+
+```Python hl_lines="14-17"
+--8<-- "../docs_src/models/docs017.py"
+```
+
+!!!note
+        Note that constraints are meant for combination of columns that should be in the index. 
+        To set one column index use [`unique`](../fields/common-parameters.md#index) common parameter. 
+        Of course, you can set many columns as indexes with this param but each of them will be a separate index.
+
+
 ### Pydantic configuration
 
 As each `ormar.Model` is also a `pydantic` model, you might want to tweak the settings of the pydantic configuration.

docs/releases.md

@@ -1,3 +1,21 @@
+# 0.10.19
+
+## ✨ Features
+
+* Add support for multi-column non-unique `IndexColumns` in `Meta.constraints` [#307](https://github.com/collerek/ormar/issues/307)
+* Add `sql_nullable` field attribute that allows to set different nullable setting for pydantic model and for underlying sql column [#308](https://github.com/collerek/ormar/issues/308)
+
+## 🐛 Fixes
+
+* Enable caching of relation map to increase performance [#337](https://github.com/collerek/ormar/issues/337)
+* Clarify and fix documentation in regard of nullable fields [#339](https://github.com/collerek/ormar/issues/339)
+
+## 💬 Other
+
+* Bump supported `databases` version to `<=5.2`.
+
+
+
 # 0.10.18
 
 ## 🐛 Fixes

docs_src/models/docs017.py

@@ -0,0 +1,21 @@
+import databases
+import sqlalchemy
+
+import ormar
+
+database = databases.Database("sqlite:///db.sqlite")
+metadata = sqlalchemy.MetaData()
+
+
+class Course(ormar.Model):
+    class Meta:
+        database = database
+        metadata = metadata
+        # define your constraints in Meta class of the model
+        # it's a list that can contain multiple constraints
+        # hera a combination of name and column will have a compound index in the db
+        constraints = [ormar.IndexColumns("name", "completed")]
+
+    id: int = ormar.Integer(primary_key=True)
+    name: str = ormar.String(max_length=100)
+    completed: bool = ormar.Boolean(default=False)

ormar/__init__.py

@@ -51,6 +51,7 @@
     Float,
     ForeignKey,
     ForeignKeyField,
+    IndexColumns,
     Integer,
     JSON,
     LargeBinary,
@@ -77,7 +78,7 @@ def __repr__(self) -> str:
 
 Undefined = UndefinedType()
 
-__version__ = "0.10.18"
+__version__ = "0.10.19"
 __all__ = [
     "Integer",
     "BigInteger",
@@ -102,6 +103,7 @@ def __repr__(self) -> str:
     "Undefined",
     "UUID",
     "UniqueColumns",
+    "IndexColumns",
     "QuerySetProtocol",
     "RelationProtocol",
     "ModelMeta",

ormar/fields/__init__.py

@@ -5,7 +5,8 @@
 Also a definition for custom CHAR based sqlalchemy UUID field
 """
 from ormar.fields.base import BaseField
-from ormar.fields.foreign_key import ForeignKey, ForeignKeyField, UniqueColumns
+from ormar.fields.constraints import IndexColumns, UniqueColumns
+from ormar.fields.foreign_key import ForeignKey, ForeignKeyField
 from ormar.fields.many_to_many import ManyToMany, ManyToManyField
 from ormar.fields.model_fields import (
     BigInteger,
@@ -36,6 +37,7 @@
     "DateTime",
     "String",
     "JSON",
+    "IndexColumns",
     "Integer",
     "Text",
     "Float",
@@ -45,7 +47,6 @@
     "ManyToMany",
     "ManyToManyField",
     "BaseField",
-    "UniqueColumns",
     "ForeignKeyField",
     "ThroughField",
     "Through",
@@ -54,4 +55,5 @@
     "DECODERS_MAP",
     "ENCODERS_MAP",
     "LargeBinary",
+    "UniqueColumns",
 ]

ormar/fields/base.py

@@ -43,6 +43,7 @@ def __init__(self, **kwargs: Any) -> None:
         self.primary_key: bool = kwargs.pop("primary_key", False)
         self.autoincrement: bool = kwargs.pop("autoincrement", False)
         self.nullable: bool = kwargs.pop("nullable", False)
+        self.sql_nullable: bool = kwargs.pop("sql_nullable", False)
         self.index: bool = kwargs.pop("index", False)
         self.unique: bool = kwargs.pop("unique", False)
         self.pydantic_only: bool = kwargs.pop("pydantic_only", False)
@@ -265,7 +266,7 @@ def get_column(self, name: str) -> sqlalchemy.Column:
                 self.column_type,
                 *self.construct_constraints(),
                 primary_key=self.primary_key,
-                nullable=self.nullable and not self.primary_key,
+                nullable=self.sql_nullable,
                 index=self.index,
                 unique=self.unique,
                 default=self.ormar_default,

ormar/fields/constraints.py

@@ -0,0 +1,22 @@
+from typing import Any
+
+from sqlalchemy import Index, UniqueConstraint
+
+
+class UniqueColumns(UniqueConstraint):
+    """
+    Subclass of sqlalchemy.UniqueConstraint.
+    Used to avoid importing anything from sqlalchemy by user.
+    """
+
+
+class IndexColumns(Index):
+    def __init__(self, *args: Any, name: str = None) -> None:
+        if not name:
+            name = "TEMPORARY_NAME"
+        super().__init__(name, *args)
+
+    """
+    Subclass of sqlalchemy.Index.
+    Used to avoid importing anything from sqlalchemy by user.
+    """

ormar/fields/foreign_key.py

@@ -18,7 +18,6 @@
 import sqlalchemy
 from pydantic import BaseModel, create_model
 from pydantic.typing import ForwardRef, evaluate_forwardref
-from sqlalchemy import UniqueConstraint
 
 import ormar  # noqa I101
 from ormar.exceptions import ModelDefinitionError, RelationshipInstanceError
@@ -160,13 +159,6 @@ def validate_not_allowed_fields(kwargs: Dict) -> None:
         )
 
 
-class UniqueColumns(UniqueConstraint):
-    """
-    Subclass of sqlalchemy.UniqueConstraint.
-    Used to avoid importing anything from sqlalchemy by user.
-    """
-
-
 @dataclass
 class ForeignKeyConstraint:
     """
@@ -242,6 +234,9 @@ def ForeignKey(  # noqa CFQ002
     skip_reverse = kwargs.pop("skip_reverse", False)
     skip_field = kwargs.pop("skip_field", False)
 
+    sql_nullable = kwargs.pop("sql_nullable", None)
+    sql_nullable = nullable if sql_nullable is None else sql_nullable
+
     validate_not_allowed_fields(kwargs)
 
     if to.__class__ == ForwardRef:
@@ -263,6 +258,7 @@ def ForeignKey(  # noqa CFQ002
         alias=name,
         name=kwargs.pop("real_name", None),
         nullable=nullable,
+        sql_nullable=sql_nullable,
         constraints=constraints,
         unique=unique,
         column_type=column_type,

ormar/fields/model_fields.py

@@ -75,6 +75,7 @@ def __new__(cls, *args: Any, **kwargs: Any) -> BaseField:  # type: ignore
         default = kwargs.pop("default", None)
         server_default = kwargs.pop("server_default", None)
         nullable = kwargs.pop("nullable", None)
+        sql_nullable = kwargs.pop("sql_nullable", None)
         pydantic_only = kwargs.pop("pydantic_only", False)
 
         primary_key = kwargs.pop("primary_key", False)
@@ -86,6 +87,13 @@ def __new__(cls, *args: Any, **kwargs: Any) -> BaseField:  # type: ignore
 
         overwrite_pydantic_type = kwargs.pop("overwrite_pydantic_type", None)
 
+        nullable = is_field_nullable(
+            nullable, default, server_default, pydantic_only
+        ) or is_auto_primary_key(primary_key, autoincrement)
+        sql_nullable = (
+            nullable if sql_nullable is None else (sql_nullable and not primary_key)
+        )
+
         namespace = dict(
             __type__=cls._type,
             __pydantic_type__=overwrite_pydantic_type
@@ -97,8 +105,8 @@ def __new__(cls, *args: Any, **kwargs: Any) -> BaseField:  # type: ignore
             primary_key=primary_key,
             default=default,
             server_default=server_default,
-            nullable=is_field_nullable(nullable, default, server_default, pydantic_only)
-            or is_auto_primary_key(primary_key, autoincrement),
+            nullable=nullable,
+            sql_nullable=sql_nullable,
             index=kwargs.pop("index", False),
             unique=kwargs.pop("unique", False),
             pydantic_only=pydantic_only,

ormar/models/helpers/models.py

@@ -75,6 +75,8 @@ def populate_default_options_values(
         if field.__type__ == bytes
     }
 
+    new_model.__relation_map__ = None
+
 
 class Connection(sqlite3.Connection):
     def __init__(self, *args: Any, **kwargs: Any) -> None:  # pragma: no cover

ormar/models/helpers/sqlalchemy.py

@@ -285,24 +285,40 @@ def populate_meta_sqlalchemy_table_if_required(meta: "ModelMeta") -> None:
 
     :param meta: Meta class of the Model without sqlalchemy table constructed
     :type meta: Model class Meta
-    :return: class with populated Meta.table
-    :rtype: Model class
     """
     if not hasattr(meta, "table") and check_for_null_type_columns_from_forward_refs(
         meta
     ):
-        for constraint in meta.constraints:
-            if isinstance(constraint, sqlalchemy.UniqueConstraint):
-                constraint.name = (
-                    f"uc_{meta.tablename}_"
-                    f'{"_".join([str(col) for col in constraint._pending_colargs])}'
-                )
+        set_constraint_names(meta=meta)
         table = sqlalchemy.Table(
             meta.tablename, meta.metadata, *meta.columns, *meta.constraints
         )
         meta.table = table
 
 
+def set_constraint_names(meta: "ModelMeta") -> None:
+    """
+    Populates the names on IndexColumn and UniqueColumns constraints.
+
+    :param meta: Meta class of the Model without sqlalchemy table constructed
+    :type meta: Model class Meta
+    """
+    for constraint in meta.constraints:
+        if isinstance(constraint, sqlalchemy.UniqueConstraint) and not constraint.name:
+            constraint.name = (
+                f"uc_{meta.tablename}_"
+                f'{"_".join([str(col) for col in constraint._pending_colargs])}'
+            )
+        elif (
+            isinstance(constraint, sqlalchemy.Index)
+            and constraint.name == "TEMPORARY_NAME"
+        ):
+            constraint.name = (
+                f"ix_{meta.tablename}_"
+                f'{"_".join([col for col in constraint._pending_colargs])}'
+            )
+
+
 def update_column_definition(
     model: Union[Type["Model"], Type["NewBaseModel"]], field: "ForeignKeyField"
 ) -> None:

ormar/models/metaclass.py

@@ -17,6 +17,7 @@
 from sqlalchemy.sql.schema import ColumnCollectionConstraint
 
 import ormar  # noqa I100
+import ormar.fields.constraints
 from ormar import ModelDefinitionError  # noqa I100
 from ormar.exceptions import ModelError
 from ormar.fields import BaseField
@@ -219,7 +220,8 @@ def update_attrs_from_base_meta(  # noqa: CCR001
                     parent_value=parent_value,
                 )
                 parent_value = [
-                    ormar.UniqueColumns(*x._pending_colargs) for x in parent_value
+                    ormar.fields.constraints.UniqueColumns(*x._pending_colargs)
+                    for x in parent_value
                 ]
             if isinstance(current_value, list):
                 current_value.extend(parent_value)