This application is a demonstration of how to validate database migrations will succeed before being run in production environments. The application uses real, anonymized production data to verify migrations work and don't fail against outlier data.
This application is based off of the reference app at pvcy/anonymize-demo.
This sample app runs a GitHub workflow to verify success when pull requests contain database migration code.
The app uses two GitHub workflows, build-and-push.yml and test-migration.yml, to coordinate everything. The first workflow, build-and-push.yml, builds and stores a container image from the /db directory which will load data from a GCS bucket at startup. The second worklow, test-migration.yml, detects when migration changes in pull requests exist and runs the database container with a full copy of production data against which migrations can be tested. The success or failure of the migration is added as a comment to the pull request.
- This app assumes there is a
pg_dumpfrom PostgreSQL stored in a Google Cloud Storage (GCS) bucket. - This is a Node app and uses Sequelize to run the database migration.
- The workflow variable
DB_BACKUP_URLneeds to be pointed at the bucket containing the backup. - This app loads the database backup into a PostgreSQL container when the workflow
test-migrationruns. If the data is exceptionally large, it may not load in time.
- Anonymize your production database using Privacy Dynamics and create a backup of it with
pg_dump. Store the backup in blob storage. In this example, GCS is used but any storage will do. - Run the PostgreSQL database in your dev environment with the command
okteto deploy --build. This will launch the three containers defined in/docker-compose.ymland load the sample data (/users-api/data/users.json) into the database container. - From the command line, run the migration with
$\users-api\src npx sequelize-cli db:migrate. The migration should pass locally. - Create a migration.