Skip to content

Commit

Permalink
Significant rewrite (#19)
Browse files Browse the repository at this point in the history 
* WIP: use nfigure for flag parsing

* breaking change: flags are no longer used by default

* breaking change in for OnMigrations* callbacks; and on some flags; WIP

* WIP

* WIP: new decorators, README about how to deploy migrations

* WIP, new decorators not yet implmented

* add missing internal file

* Change signature of Computed migrations; WIP

* reverse that: do not add migration to computed

* drop MyLogger from migration functions

* remove useless Context from Database

* replace MyLogger with three functions to create loggers

* files missing from last commit

* WIP, implementing repeatUntilNoOp, does not compile

* fixed, mysql test added too

* a tad more coverage

* a tiny bit more coverage

* more coverage

* a bit more coverage

* and even more coverage

* readme updates
  • Loading branch information
@muir
muir authored Apr 29, 2022
1 parent 58983f6 commit 5d6c207
Show file tree
Hide file tree
Showing 25 changed files with 1,264 additions and 419 deletions.
196 changes: 142 additions & 54 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,14 @@ Install:

Libschema provides a way for Go libraries to manage their own migrations.

Trying migrations to libraries supports two things: the first is source code
locality: the migrations can be next to the code that uses the tables that
the migrations address.
Trying migrations to libraries supports two things: the first is
source code locality: the migrations can be next to the code that
uses the tables that the migrations address.

The second is support for migrations in third-party libraries. This is a relatively
unexplored and unsolved problem: how can an open source (or proprietary) library
specify and maintain a database schema. Libschema hopes to start solving this problem.
The second is support for migrations in third-party libraries. This
is a relatively unexplored and unsolved problem: how can an open
source (or proprietary) library specify and maintain a database
schema. Libschema hopes to start solving this problem.

## Register and execute

Expand All @@ -40,14 +41,14 @@ database, err := lspostgres.New(logger, "main-db", schema, sqlDB)

database.Migrations("MyLibrary",
lspostgres.Script("createUserTable", `
CREATE TAGLE users (
name text,
id bigint
)`
CREATE TAGLE users (
name text,
id bigint
)`
}),
lspostgres.Script("addLastLogin", `
ALTER TABLE users
ADD COLUMN last_login timestamp
ALTER TABLE users
ADD COLUMN last_login timestamp
`
}),
)
Expand All @@ -65,25 +66,26 @@ Migrations may be SQL strings or migrations can be done in Go:

```go
database.Migrations("MyLibrary",
lspostgres.Computed("importUsers", func(_ context.Context, _ MyLogger, tx *sql.Tx) error {
lspostgres.Computed("importUsers", func(_ context.Context, _ Migration, tx *sql.Tx) error {
// code to import users here
}),
)
```

## Asynchronous migrations

The normal mode for migrations is to run the migrations synchronously when
`schema.Migrate()` is called. Asynchronous migrations are started when `schema.Migrate()`
is called but they're run in the background in a go-routine. If there are later
migrations, after the asynchronous migration, they'll force the asynchronous migration
The normal mode for migrations is to run the migrations synchronously
when `schema.Migrate()` is called. Asynchronous migrations are
started when `schema.Migrate()` is called but they're run in the
background in a go-routine. If there are later migrations, after
the asynchronous migration, they'll force the asynchronous migration
to be synchronous unless they're also asynchronous.

## Version blocking

Migrations can be tied to specific code versions so that they are not run until
conditions are met. This is done with `SkipRemainingIf`. This be used to backfill
data.
Migrations can be tied to specific code versions so that they are
not run until conditions are met. This is done with `SkipRemainingIf`.
This be used to backfill data.

```go
database.Migrations("MyLibrary",
Expand All @@ -107,9 +109,9 @@ database.Migrations("MyLibrary",

## Cross-library dependencies

Although it is best if the schema from one library is independent of the schema for
another, sometimes that's not possible, especially if you want to enforce foriegn key
constraints.
Although it is best if the schema from one library is independent
of the schema for another, sometimes that's not possible, especially
if you want to enforce foriegn key constraints.

Use `After()` to specify a cross-library dependency.

Expand All @@ -134,64 +136,150 @@ database.Migrations("orgs",

## Transactions

For databases that support transactions on metadata, all migrations will be wrapped with
a `BEGIN` and `COMMIT`. For databases that do not support transactions on metadata,
migrations will be split into individual commands and run one at a time. If only some
of the commands succeed, the migration will be marked as partially complete. If the migration
is revised, then the later parts can be re-tried as long as the earlier parts are not
modified. This does not apply to `Compute()`ed migrations.
For databases that support transactions on metadata, all migrations
will be wrapped with a `BEGIN` and `COMMIT`. For databases that
do not support transactions on metadata, migrations will be split
into individual commands and run one at a time. If only some of
the commands succeed, the migration will be marked as partially
complete. If the migration is revised, then the later parts can
be re-tried as long as the earlier parts are not modified. This
does not apply to `Compute()`ed migrations.

## Command line

Libschema adds command line flags that change the behavior of calling
`schema.Migrate()`
The `OverrideOptions` can be added as command line flags that
change the behavior of calling `schema.Migrate()`

--migrate-only Call os.Exit() after completing migrations
--migrate-database Migrate only the database named by NewDatabase
--migrate-database Migrate only one logical database (must match NewDatabase)
--migrate-dsn Override *sql.DB
--no-migrate Skip all migrations
--exit-if-migrate-needed Return error if migrations are not current
--error-if-migrate-needed Return error if there are outstanding synchronous migrations
--migrate-all-synchronously Treat asychronous migrations as synchronous

## Ordering and pull requests

Migrations are run the order that they're defined. If the set of migrations is
updated so that there are new migrations that are earlier in the table than migrations
that have already run, this is not considered an error and the new migrations will
be run anyway. This allows multiple branches of code with migrations to be merged
Migrations are run the order that they're defined. If the set of
migrations is updated so that there are new migrations that are
earlier in the table than migrations that have already run, this
is not considered an error and the new migrations will be run anyway.
This allows multiple branches of code with migrations to be merged
into a combined branch without hassle.

Migrations can have explicit dependencies and these dependencies
can cross between libraries so that one library's migrations can
depend on anothers.

## Code structure

Registering the migrations before executing them suggests using library singletons.
Library singletons can be supported by using [nserve](https://github.com/muir/nject/nserve)
or [fx](https://github.com/uber-go/fx). With nserve, migrations can be given their
own hook.
Registering the migrations before executing them is easier if using
library singletons. Library singletons can be supported by using
[nserve](https://github.com/muir/nject/nserve) or
[fx](https://github.com/uber-go/fx). With nserve, migrations can
be given their own hook.

## Driver inclusion and database support

Like database/sql, libschema requires database-specific drivers. Include
"github.com/muir/libschema/lspg" for Postgres support and "github.com/muir/libschema/lsmysql"
for Mysql support.
Like database/sql, libschema requires database-specific drivers.
Include "github.com/muir/libschema/lspostgres" for Postgres support
and "github.com/muir/libschema/lsmysql" for Mysql support.

libschema currently supports: PostgreSQL, MySQL. It is easy to add additional databases.
libschema currently supports: PostgreSQL, MySQL. It is relatively easy to add additional databases.

## Forward only

Libschema does not support reverse migrations. If you need to fix a migration, fix forward.
The history behind this is that reverse migrations are rarely the right answer for production
systems and the extra work for maintaining reverse migrations is does not have enough of a
payoff during development to be worth the effort.
Libschema does not support reverse migrations. If you need to fix
a migration, fix forward. The history behind this is that reverse
migrations are rarely the right answer for production systems and
the extra work for maintaining reverse migrations is does not have
enough of a payoff during development to be worth the effort.

One way to get the benefits of reverse migrations for development is to put enough enough
reverse migrations to reverse to the last production schema at the end of the migration
list but protected by a gateway:
One way to get the benefits of reverse migrations for development
is to put enough enough reverse migrations to reverse to the last
production schema at the end of the migration list but protected
by a gateway:

```go
libschema.SkipThisAndRemainingIf(func() bool {
return os.Getenv("LIBMIGRATE_REVERSE_TO_PROD") != "true"
}),
```

This set of reverse migrations would always be small since it would just be enough to take you
back to the current production release.
This set of reverse migrations would always be small since it would
just be enough to take you back to the current production release.

## Patterns for applying migrations

When using a migration tool like libschema there are several
reasonable patterns one can follow to apply migrations to produciton
code.

### Down-Up deploys

The simplist pattern is to deploy migrations synchronously when
rolling out updates. If you take your service down to do deploys
then your migrations do not have to be backwards compatible. This
has the huge upside of allowing your schema to eveolve easily and
avoid the build up to technical debt. For example, if you have a
column whose name is sub-optimal, you can simply rename it.

To minimimize downtime so that the downtime doesn't matter in
practice, run expensive migrations asynchronously. Asychronous
migrations are harder to define because they should be broken up
into a whole bunch of smallish transactions. The `RepeatUntilNoOp()`
decorator may be useful.

### Green-Blue deploys

When you decide to run without downtime, one consequence is that
all migrations must be backewards compatible with the deployed
code.

DDL operations that are backwards compatible include:

- adding a column, table, or view
- removing a column, table, or view that is no longer accessed
- adding a default value to a column
- remvoing a constraint
- adding a constraint as long as there are no violations and won't be any new ones

From a coding point-of-view, the simplest way to manage developing
with these restrictions is to separate the migration into a separate
code change. Any code that might be broken by the migration should
be tested with the same code change. No other changes to code
should be allowed in the same code change. Local and CI testing
should apply the migration and validate that the the existing code
isn't broken by the change in database schema.

Only after the migration has been deployed can code that uses the
migration be deployed. When using git, this can be done by having
layered side branches:
main -> migration branch -> code branch -> cleanup migration branch.

### Kubernetes and slow migrations

One issue with using libschema to deploy changes is that servers can take
a long time to come up if there are expensive migrations that need to be
deployed first. A solution for this is to use `OverrideOptions` to separate
the migrations into a separate step and run them in an
[init container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/).

To do this use the `MigrateOnly` / `--migrate-only` option on your main program
when running it in the init container.

Then use the `ErrorIfMigrateNeeded` / `--error-if-migrate-needed` option on your main
program when it starts up for normal use.

## Code Stability

Libschema is still subject to changes. Anything that is not backwards compatible
will be clearly documented and will fail in a way that does not cause hidden problems.
For example, switching from using "flag" to using OverrideOptions will trigger
an obvious breakage if you try to use a flag that no longer works.

Anticpated changes for the future:

- API tweaks
- Support for additional databases
- Support for additional logging APIs
- Support for tracing spans (per migration)
Loading

0 comments on commit 5d6c207

Please sign in to comment.