From 8809db970d4f324b1935547d788b1a5ebce48f11 Mon Sep 17 00:00:00 2001 From: Doug Stevenson Date: Wed, 17 May 2023 09:25:42 -0400 Subject: [PATCH] Improve readmes --- libsql-server/README.md | 173 +++++++++-------------------- libsql-server/docs/BUILD-RUN.md | 187 ++++++++++++++++++++++++++++++++ 2 files changed, 241 insertions(+), 119 deletions(-) create mode 100644 libsql-server/docs/BUILD-RUN.md diff --git a/libsql-server/README.md b/libsql-server/README.md index a08e2bf419..f531b1d785 100644 --- a/libsql-server/README.md +++ b/libsql-server/README.md @@ -1,62 +1,43 @@ # `sqld` - a server mode for libSQL -The `sqld` ("SQL daemon") project is a server mode for [libSQL](https://libsql.org). +The `sqld` ("SQL daemon") project is a server mode for +[libSQL](https://github.com/libsql/libsql/). -Embedded SQL databases such as libSQL and SQLite are great for a lot of use cases, but sometimes you really do want to consume your database as a server. -For example, with serverless apps, fitting a database engine, as slim as it may be, might be hard, and even when it's _possible_, it might be really inconvenient, which is why we created `sqld`. - -With `sqld` you can use SQLite-like interface in your app, but have a transparent proxy translates the C API calls to PostgreSQL wire protocol to talk to a database server, which is internally libSQL. - -_Disclaimer: although you can connect to `sqld` with a PostgreSQL client and many things just work because PostgreSQL and SQLite are so similar, the goal of this project is to provide SQLite compatibility at the client. -That is, whenever there's a conflict in behaviour between SQLite and PostgreSQL, we always go with SQLite behavior. -However, if libSQL starts to provide more [PostgreSQL compatibility](https://github.com/libsql/libsql/issues/80), `sqld` will also support that._ - -## Clients - -* [TypeScript and JavaScript](https://github.com/libsql/libsql-client-ts) -* [Rust](https://github.com/libsql/libsql-client-rs) -* [Go](https://github.com/libsql/libsql-client-go) +Embedded SQL databases such as libSQL and SQLite are great for a lot of use +cases, but sometimes you really do want to consume your database as a server. +For example, with apps running on serverless infrastructure, fitting a database +engine might be difficult given the limited size of the hardware. And even when +it's _possible_, it might be really inconvenient. We created `sqld` for this use +case. ## Features -* SQLite dialect layered on top of HTTP or the PostgreSQL wire protocol. -* TypeScript/JavaScript, Rust, Go and Python clients -* SQLite-compatible API that you can drop-in with `LD_PRELOAD` in your application to switch from local database to a remote database. +* SQLite dialect layered on top of HTTP. +* SQLite-compatible API that you can drop-in with `LD_PRELOAD` in your + application to switch from local database to a remote database. * Read replica support. -* Integration with [mvSQLite](https://github.com/losfair/mvsqlite) for high availability and fault tolerance. -# SQLite extensions support - -## Roadmap - -* Client authentication and TLS -* Integration with libSQL's [bottomless storage](https://github.com/libsql/sqld/tree/main/bottomless) +* Integration with [mvSQLite](https://github.com/losfair/mvsqlite) for high + availability and fault tolerance. -## Getting Started +## Build and run -Start a server with a postgres and http listeners, writing to the local SQLite-compatible file `foo.db`: - -```console -sqld -d foo.db -p 127.0.0.1:5432 --http-listen-addr=127.0.0.1:8000 -``` - -connect to it with psql: - -```console -psql -h 127.0.0.1 -``` +Follow the [instructions](./docs/BUILD-RUN.md) to build and run `sqld` +using Homebrew, Docker, or your own Rust toolchain. -or HTTP: +## Client libraries -```console -curl -s -d '{\"statements\": [\"SELECT * from sqlite_master;\"] }' http://127.0.0.1:8000 -``` +The following client libraries enable your app to query `sqld` programmatically: -You can also inspect the local foo.db file with the `sqlite3` shell +* [TypeScript and JavaScript](https://github.com/libsql/libsql-client-ts) +* [Rust](https://github.com/libsql/libsql-client-rs) +* [Go](https://github.com/libsql/libsql-client-go) +* [Python](https://github.com/libsql/libsql-client-py) -### Loading extensions +## SQLite extensions support -Extensions need to be preloaded at startup. To do that, add all of your extensions to a directory, -and add a file called `trusted.lst` with the `sha256sum` of each file to that directory. For example: +Extensions must be preloaded at startup. To do that, add all of your extensions +to a directory, and add a file called `trusted.lst` with the `sha256sum` of each +file to that directory. For example: ```console $ cat trusted.lst @@ -74,21 +55,32 @@ ae7fff8412e4e66e7f22b9af620bd24074bc9c77da6746221a9aba9d2b38d6a6 text.so Extensions will be loaded in the order they appear on that file, so if there are dependencies between extensions make sure they are listed in the proper order. -Then start the server with the `--extensions-path` option pointing at the extension directory +Then start the server with the `--extensions-path` option pointing at the +extension directory +## Integration with S3 bottomless replication -### Integration with S3 bottomless replication +`sqld` is integrated with [bottomless replication subproject]. With bottomless +replication, the database state is continuously backed up to S3-compatible +storage. Each backup session is called a "generation" and consists of the main +database file snapshot and replicates [SQLite WAL] pages. -`sqld` is integrated with [bottomless replication subproject](https://github.com/libsql/sqld/tree/main/bottomless). With bottomless replication, the database state is continuously backed up to S3-compatible storage. Each backup session is called a "generation" and consists of the main database file snapshot and replicates [WAL](https://www.sqlite.org/wal.html) pages. +In order to enable automatic replication to S3 storage, run `sqld` with +`--enable-bottomless-replication` parameter: -In order to enable automatic replication to S3 storage, run `sqld` with `--enable-bottomless-replication` parameter: -```console +```bash sqld --http-listen-addr=127.0.0.1:8000 --enable-bottomless-replication ``` -#### Configuration -Replication needs to be able to access an S3-compatible bucket. The following environment variables can be used to configure the replication: -```sh +[bottomless replication subproject]: ./bottomless +[SQLite WAL]: https://www.sqlite.org/wal.html + +### Configuration + +Replication needs to be able to access an S3-compatible bucket. The following +environment variables can be used to configure the replication: + +```bash LIBSQL_BOTTOMLESS_BUCKET=my-bucket # Default bucket name: bottomless LIBSQL_BOTTOMLESS_ENDPOINT='http://localhost:9000' # address can be overridden for local testing, e.g. with Minio AWS_SECRET_ACCESS_KEY= # regular AWS variables are used @@ -96,78 +88,21 @@ AWS_ACCESS_KEY_ID= # ... to set up auth, regions AWS_REGION= # . ``` -#### bottomless-cli -Replicated snapshots can be inspected and managed with the official command-line interface. - -The tool can be installed via `cargo`: -```console -RUSTFLAGS='--cfg uuid_unstable' cargo install bottomless-cli -``` -For usage examples and description, refer to the official bottomless-cli documentation: https://github.com/libsql/sqld/tree/main/bottomless#cli - -## Homebrew +### bottomless-cli -You can install `sqld` through homebrew by doing: +Replicated snapshots can be inspected and managed with the official command-line +interface. -``` -brew tap libsql/sqld -brew install sqld-beta -``` - -Note that until a stable version is released, it is provided as a separate tap, with a `beta` suffix. - -## Clients - -`sqld` ships with a native Javascript driver for TypeScript and Javascript. You can find more information [here](https://www.npmjs.com/package/@libsql/client) - -## Building from Sources - -### Using Docker/Podman -The easiest way to build `sqld` is using Docker; Build dependencies will be installed in the build container and compilation will be done in a clean, isolated environment. - -```console -docker build -t libsql/sqld:latest . -``` - -After building the docker image, you can run `sqld` as follows: - -```console -docker volume create sqld-data -docker container run -d -v sqld-data:/var/lib/sqld --name sqld -P libsql/sqld:latest -docker container port sqld # View the mapped port for sqld container -``` - -The following environment variables can be used to configure the `sqld` container: -- `SQLD_DB_PATH` - Database file, defaults to `iku.db`. Absolute path can be used if you want the file in a different directory than `/var/lib/sqld` - note that the folder needs to be writable for `sqld` user (uid 666). -- `SQLD_NODE` - Node type, defaults to `primary`. Valid values are `primary`, `replica` or `standalone`. - -All other standard `sqld` environment variables work as well. Try `docker container run --rm -it libsql/sqld /bin/sqld --help` to view them. - -### Dependencies - -**Linux:** - -```console -./scripts/install-deps.sh -``` - -### Submodules - -```run -git submodule update --init --force --recursive --depth 1 -``` - -### Building +The tool can be installed via `cargo`: -```console -cargo build +```bash +RUSTFLAGS='--cfg uuid_unstable' cargo install bottomless-cli ``` -### Running tests +For usage examples and description, refer to the [bottomless-cli +documentation]. -```console -make test -``` +[bottomless-cli documentation]: ./bottomless#cli ## License diff --git a/libsql-server/docs/BUILD-RUN.md b/libsql-server/docs/BUILD-RUN.md new file mode 100644 index 0000000000..b4aaf1326b --- /dev/null +++ b/libsql-server/docs/BUILD-RUN.md @@ -0,0 +1,187 @@ +# Build and run sqld + +There are three ways to build and run sqld: + +- [Using Homebrew](#build-and-install-with-homebrew) +- [From source using Docker/Podman](#build-from-source-using-docker--podman) +- [From source using Rust](#build-from-source-using-rust) + +## Query sqld + +After building, you can query sqld using one of the provided [client +libraries](../#client-libraries). + +By default, sqld persists database data in a directory `./data.sqld`. The file +`data` is a normal SQLite 3 compatible database file. You can work with it +directly using the SQLite CLI: + +```bash +sqlite3 ./data.sqld/data +``` + +Be sure to stop sqld before using `sqlite3` like this. + +## Build and install with Homebrew + +The sqld formulae for Homebrew works with macOS, Linux (including WSL). + +### 1. Add the tap `libsql/sqld` to Homebrew + +```bash +brew tap libsql/sqld +``` + +### 2. Install the formulae `sqld-beta` + +```bash +brew install sqld-beta +``` + +This builds and installs the binary `sqld` into `$HOMEBREW_PREFIX/bin/sqld`, +which should be in your PATH. + +### 3. Verify that `sqld` works + +```bash +sqld --help +``` + +## Build from source using Docker / Podman + +To build sqld with Docker, you must have a Docker [installed] and running on +your machine with its CLI in your shell PATH. + +[installed]: https://docs.docker.com/get-docker/ + +### 1. Clone this repo + +Clone this repo using your preferred mechanism. You may want to use one of the +[sqld release tags]. + +Change to the `sqld` directory. + +### 2. Build with Docker + +Run the following to build a Docker image named "libsql/sqld" tagged with +version "latest". + +```bash +docker build -t libsql/sqld:latest . +``` + +### 3. Verify the build + +Check that sqld built successfully using its --help flag: + +```bash +docker container run \ + --rm \ + -i \ + libsql/sqld \ + /bin/sqld --help +``` + +### 4. Create a data volume + +The following will create a volume named `sqld-data` that sqld uses to persist +database files. + +```bash +docker volume create sqld-data +``` + +### 5. Run sqld in a container + +The following uses the built image to create and run a new container named +`sqld`, attaching the `sqld-data` volume to it, and exposing its port 8080 +locally: + +```bash +docker container run \ + -d \ + --name sqld \ + -v sqld-data:/var/lib/sqld \ + -p 127.0.0.1:8080:8080 \ + libsql/sqld:latest +``` + +8080 is the default port for the sqld HTTP service that handles client queries. +With this container running, you can use the URL `http://127.0.0.1:8080` or +`ws://127.0.0.1:8080` to configure one of the libSQL client SDKs for local +development. + +### 6. Configure sqld with environment variables + +In the sqld output using `--help` from step 3, you saw the names of command line +flags along with the names of environment variables (look for "env:") used to +configure the way sqld works. + +## Build from source using Rust + +To build from source, you must have a Rust development environment installed and +available in your PATH. + +Currently we only support building sqld on macOS and Linux (including WSL). We +are working native Windows build instructions. + +### 1. Setup + +Install dependencies: + +```bash +./scripts/install-deps.sh +``` + +### 2. Clone this repo + +Clone this repo using your preferred mechanism. You may want to use one of the +[sqld release tags]. + +Change to the `sqld` directory. + +Install git submodules: + +```bash +git submodule update --init --force --recursive --depth 1 +``` + +### 3. Build with cargo + +```bash +cargo build +``` + +The sqld binary will be in `./target/debug/sqld`. + +### 4. Verify the build + +Check that sqld built successfully using its --help flag: + +```bash +./target/debug/sqld --help +``` + +### 5. Run sqld with all defaults + +The following starts sqld, taking the following defaults: + +- Local files stored in the directory `./data.sqld` +- Client HTTP requests on 127.0.0.1:8080 + +```bash +./target/debug/sqld +``` + +8080 is the default port for the sqld HTTP service that handles client queries. +With this container running, you can use the URL `http://127.0.0.1:8080` or +`ws://127.0.0.1:8080` to configure one of the libSQL client SDKs for local +development. + +### 6. Run tests (optional) + +```console +make test +``` + + +[sqld release tags]: /libsql/sqld/releases \ No newline at end of file