This repository contains a collection of engines that power the core stack for Prisma, most prominently Prisma Client and Prisma Migrate.
If you're looking for how to install Prisma or any of the engines, the Getting Started guide might be useful.
This document describes some of the internals of the engines, and how to build and test them.
This repository contains four engines:
- Query engine, used by the client to run database queries from Prisma Client
- Schema engine, used to create and run migrations and introspection
- Prisma Format, used to format prisma files
Additionally, the psl (Prisma Schema Language) is the library that defines how the language looks like, how it's parsed, etc.
You'll also find:
- libs, for various (small) libraries such as macros, user facing errors, various connector/database-specific libraries, etc.
- a
docker-compose.yml
file that's helpful for running tests and bringing up containers for various databases - a
flake.nix
file for bringing up all dependencies and making it easy to build the code in this repository (the use of this file andnix
is entirely optional, but can be a good and easy way to get started) - an
.envrc
file to make it easier to set everything up, including thenix shell
The API docs (cargo doc) are published on our fabulous repo page.
Prerequisites:
- Installed the latest stable version of the Rust toolchain. You can get the toolchain at rustup or the package manager of your choice.
- Linux only: OpenSSL is required to be installed.
- Installed direnv, then
direnv allow
on the repository root.- Make sure direnv is hooked into your shell
- Alternatively: Load the defined environment in
./.envrc
manually in your shell.
- For m1 users: Install Protocol Buffers
Note for nix users: it should be enough to direnv allow
.
How to build:
To build all engines, simply execute cargo build
on the repository root. This
builds non-production debug binaries. If you want to build the optimized
binaries in release mode, the command is cargo build --release
.
Depending on how you invoked cargo
in the previous step, you can find the
compiled binaries inside the repository root in the target/debug
(without
--release
) or target/release
directories (with --release
):
Prisma Component | Path to Binary |
---|---|
Query Engine | ./target/[debug|release]/query-engine |
Schema Engine | ./target/[debug|release]/migration-engine (*) |
Prisma Format | ./target/[debug|release]/prisma-fmt |
(*) Will be renamed to schema-engine in Prisma 5.
The Prisma Schema Language is a library which defines the data structures and parsing rules for prisma files, including the available database connectors. For more technical details, please check the library README.
The PSL is used throughout the schema engine, as well as prisma format. The DataModeL (DML), which is an annotated version of the PSL is also used as input for the query engine.
The Query Engine is how Prisma Client queries are executed. Here's a brief description of what it does:
- takes as inputs an annotated version of the Prisma Schema file called the DataModeL (DML),
- using the DML (specifically, the datasources and providers), it builds up a GraphQL model for queries and responses,
- runs as a server listening for GraphQL queries,
- it translates the queries to the respective native datasource(s) and returns GraphQL responses, and
- handles all connections and communication with the native databases.
When used through Prisma Client, there are two ways for the Query Engine to be executed:
- as a binary, downloaded during installation, launched at runtime;
communication happens via HTTP (
./query-engine/query-engine
) - as a native, platform-specific Node.js addon; also downloaded during
installation (
./query-engine/query-engine-node-api
)
You can also run the Query Engine as a stand-alone GraphQL server.
Warning: There is no guaranteed API stability. If using it on production please be aware the api and the query language can change any time.
Notable environment flags:
RUST_LOG_FORMAT=(devel|json)
sets the log format. By default outputsjson
.QE_LOG_LEVEL=(info|debug|trace)
sets the log level for the Query Engine. If you need Query Graph debugging logs, set it to "trace"FMT_SQL=1
enables logging formatted SQL queriesPRISMA_DML_PATH=[path_to_datamodel_file]
should point to the datamodel file location. This orPRISMA_DML
is required for the Query Engine to run.PRISMA_DML=[base64_encoded_datamodel]
an alternative way to provide a datamodel for the server.RUST_BACKTRACE=(0|1)
if set to 1, the error backtraces will be printed to the STDERR.LOG_QUERIES=[anything]
if set, the SQL queries will be written to theINFO
log. Needs the right log level enabled to be seen from the terminal.RUST_LOG=[filter]
sets the filter for the logger. Can be eithertrace
,debug
,info
,warning
orerror
, that will output ALL logs from every crate from that level. The.envrc
in this repo shows how to log different parts of the system in a more granular way.
Starting the Query Engine:
The engine can be started either with using the cargo
build tool, or
pre-building a binary and running it directly. If using cargo
, replace
whatever command that starts with ./query-engine
with cargo run --bin query-engine --
.
You can also pass --help
to find out more options to run the engine.
Running make show-metrics
will start Prometheus and Grafana with a default metrics dashboard.
Prometheus will scrape the /metrics
endpoint to collect the engine's metrics
Navigate to http://localhost:3000
to view the Grafana dashboard.
The Schema Engine does a couple of things:
- creates new migrations by comparing the prisma file with the current state of the database, in order to bring the database in sync with the prisma file
- run these migrations and keeps track of which migrations have been executed
- (re-)generate a prisma schema file starting from a live database
The engine uses:
- the prisma files, as the source of truth
- the database it connects to, for diffing and running migrations, as well as
keeping track of migrations in the
_prisma_migrations
table - the
prisma/migrations
directory which acts as a database of existing migrations
Prisma format can format prisma schema files. It also comes as a WASM module via a node package. You can read more here.
When trying to debug code, here's a few things that might be useful:
- use the language server; being able to go to definition and reason about code can make things a lot easier,
- add
dbg!()
statements to validate code paths, inspect variables, etc., - you can control the amount of logs you see, and where they come from using the
RUST_LOG
environment variable; see the documentation, - you can use the
test-cli
to test migration and introspection without having to go through theprisma
npm package.
There are two test suites for the engines: Unit tests and integration tests.
-
Unit tests: They test internal functionality of individual crates and components.
You can find them across the whole codebase, usually in
./tests
folders at the root of modules. These tests can be executed viacargo test
. Note that some of them will require theTEST_DATABASE_URL
enviornment variable set up. -
Integration tests: They run GraphQL queries against isolated instances of the Query Engine and asserts that the responses are correct.
You can find them at
./query-engine/connector-test-kit-rs
.
Prerequisites:
- Installed Rust toolchain.
- Installed Docker.
- Installed
direnv
, thendirenv allow
on the repository root.- Alternatively: Load the defined environment in
./.envrc
manually in your shell.
- Alternatively: Load the defined environment in
Setup:
There are helper make
commands to set up a test environment for a specific
database connector you want to test. The commands set up a container (if needed)
and write the .test_config
file, which is picked up by the integration
tests:
make dev-mysql
: MySQL 5.7make dev-mysql8
: MySQL 8make dev-postgres
: PostgreSQL 10make dev-sqlite
: SQLitemake dev-mongodb_5
: MongoDB 5
*On windows:
If not using WSL, make
is not available and you should just see what your
command does and do it manually. Basically this means editing the
.test_config
file and starting the needed Docker containers.
To actually get the tests working, read the contents of .envrc
. Then Edit environment variables for your account
from Windows settings, and add at least
the correct values for the following variables:
WORKSPACE_ROOT
should point to the root directory ofprisma-engines
project.PRISMA_BINARY_PATH
is usually%WORKSPACE_ROOT%\target\release\query-engine.exe
.MIGRATION_ENGINE_BINARY_PATH
should be%WORKSPACE_ROOT%\target\release\migration-engine.exe
.
Other variables may or may not be useful.
Run:
Run cargo test
in the repository root.
When rust-analzyer runs cargo check
it will lock the build directory and stop any cargo commands from running until it has completed. This makes the build process feel a lot longer. It is possible to avoid this by setting a different build path for
rust-analyzer. To avoid this. Open VSCode settings and search for Check on Save: Extra Args
. Look for the Rust-analyzer › Check On Save: Extra Args
settings and add a new directory for rust-analyzer. Something like:
--target-dir:/tmp/rust-analyzer-check
(Since July 2022). Any branch name starting with integration/
will, first, run the full test suite and, second, if passing, run the publish pipeline (build and upload engines to S3)
The journey through the pipeline is the same as a commit on the main
branch.
- It will trigger prisma/engines-wrapper and publish a new
@prisma/engines-version
npm package but on theintegration
tag. - Which triggers prisma/prisma to create a
chore(Automated Integration PR): [...]
PR with a branch name also starting withintegration/
- Since in prisma/prisma we also trigger the publish pipeline when a branch name starts with
integration/
, this will publish all prisma/prisma monorepo packages to npm on theintegration
tag. - Our ecosystem-tests tests will automatically pick up this new version and run tests, results will show in GitHub Actions
This end to end will take minimum ~1h20 to complete, but is completely automated 🤖
Notes:
- in prisma/prisma repository, we do not run tests for
integration/
branches, it is much faster and also means that there is no risk of test failing (e.g. flaky tests, snapshots) that would stop the publishing process. - in prisma/prisma-engines tests must first pass, before publishing starts. So better keep an eye on them and restart them as needed.
If you have a security issue to report, please contact us at [email protected]