Skip to content

Commit

Permalink
Merge pull request #38 from EchoVault/update/README.md
Browse files Browse the repository at this point in the history 
Updated README.md in preparation for new docs site && Added tests for admin module commands
  • Loading branch information
@kelvinmwinuka
kelvinmwinuka authored May 8, 2024
2 parents 362cbc3 + 270c5ef commit 6ed59ef
Show file tree
Hide file tree
Showing 12 changed files with 763 additions and 420 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/go.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,10 @@ jobs:
run: go build -v ./cmd/main.go

- name: Test
run: go test -coverprofile=coverage.out -v ./...
run: make test-unit

- name: Test for Data Race
run: go test ./... --race
run: make test-race

- name: Upload coverage reports to Codecov
uses: codecov/[email protected]
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ volumes/nodes
dist/
pkg/modules/*/aof
pkg/echovault/aof
dump.rdb
dump.rdb
**/*/testdata
12 changes: 9 additions & 3 deletions Makefile
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
build-modules:
CGO_ENABLED=$(CGO_ENABLED) CC=$(CC) GOOS=$(GOOS) GOARCH=$(GOARCH) go build -buildmode=plugin -o $(DEST)/module_set/module_set.so ./volumes/modules/module_set/module_set.go && \
CGO_ENABLED=$(CGO_ENABLED) CC=$(CC) GOOS=$(GOOS) GOARCH=$(GOARCH) go build -buildmode=plugin -o $(DEST)/module_get/module_get.so ./volumes/modules/module_get/module_get.go
CGO_ENABLED=$(CGO_ENABLED) CC=$(CC) GOOS=$(GOOS) GOARCH=$(GOARCH) go build -buildmode=plugin -o $(DEST)/module_get/module_get.so ./volumes/modules/module_get/module_get.go

build-server:
CGO_ENABLED=$(CGO_ENABLED) CC=$(CC) GOOS=$(GOOS) GOARCH=$(GOARCH) go build -o $(DEST)/server ./cmd/main.go
Expand All @@ -13,7 +13,13 @@ run:
make build && docker-compose up --build

test-unit:
go clean -testcache && go test ./... -coverprofile coverage/coverage.out
CGO_ENABLED=1 go build -buildmode=plugin -o internal/modules/admin/testdata/modules/module_set/module_set.so ./volumes/modules/module_set/module_set.go && \
CGO_ENABLED=1 go build -buildmode=plugin -o internal/modules/admin/testdata/modules/module_get/module_get.so ./volumes/modules/module_get/module_get.go && \
go clean -testcache && \
CGO_ENABLED=1 go test ./... -coverprofile coverage/coverage.out

test-race:
go clean -testcache && go test ./... --race
CGO_ENABLED=1 go build -buildmode=plugin --race -o internal/modules/admin/testdata/modules/module_set/module_set.so ./volumes/modules/module_set/module_set.go && \
CGO_ENABLED=1 go build -buildmode=plugin --race -o internal/modules/admin/testdata/modules/module_get/module_get.so ./volumes/modules/module_get/module_get.go && \
go clean -testcache && \
CGO_ENABLED=1 go test ./... --race
214 changes: 18 additions & 196 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ capability is always being worked on and improved.

# Features

Some key features offered by EchoVault include:
Features offered by EchoVault include:

1) TLS and mTLS support for multiple server and client RootCAs.
2) Replication cluster support using the RAFT algorithm.
Expand All @@ -43,19 +43,20 @@ Some key features offered by EchoVault include:
5) Sets, Sorted Sets, Hashes, Lists and more.
6) Persistence layer with Snapshots and Append-Only files.
7) Key Eviction Policies.
8) Command extension via shared object files.
9) Command extension via embedded API.

We are working hard to add more features to EchoVault to make it
much more powerful. Features in the roadmap include:

1) Sharding
2) Shared Object File Plugins
3) Streams
4) Transactions
5) Bitmap
6) HyperLogLog
7) Lua Modules
8) JSON
9) Improved Observability
2) Streams
3) Transactions
4) Bitmap
5) HyperLogLog
6) Lua Modules
7) JSON
8) Improved Observability


# Usage (Embedded)
Expand All @@ -68,25 +69,21 @@ You can access all of EchoVault's commands using an ergonomic API.

```go
func main() {
server, err := echovault.NewEchoVault(
echovault.WithConfig(config.DefaultConfig()),
echovault.WithCommands(commands.All()),
)
server, err := echovault.NewEchoVault()

if err != nil {
log.Fatal(err)
}

_, _ = server.SET("key", "Hello, world!", echovault.SETOptions{})

v, _ := server.GET("key")

_, _ = server.Set("key", "Hello, world!", echovault.SETOptions{})

v, _ := server.Get("key")
fmt.Println(v) // Hello, world!

wg := sync.WaitGroup{}

// Subscribe to multiple EchoVault channels.
readMessage := server.SUBSCRIBE("subscriber1", "channel_1", "channel_2", "channel_3")
readMessage := server.Subscribe("subscriber1", "channel_1", "channel_2", "channel_3")
wg.Add(1)
go func() {
wg.Done()
Expand All @@ -103,7 +100,7 @@ func main() {
// Simulating delay.
<-time.After(1 * time.Second)
// Publish message to each EchoVault channel.
_, _ = server.PUBLISH(fmt.Sprintf("channel_%d", i), "Hello!")
_, _ = server.Publish(fmt.Sprintf("channel_%d", i), "Hello!")
}
wg.Done()
}()
Expand All @@ -128,195 +125,20 @@ To install via homebrew, run:
Once installed, you can run the server with the following command:
`echovault --bind-addr=localhost --data-dir="path/to/persistence/directory"`

Next, [install the client via homebrew](https://github.com/EchoVault/EchoVault-CLI).

### Binaries

You can download the binaries by clicking on a release tag and downloading
the binary for your system.

Checkout the [configuration section](#configuration) for the possible configuration
flags.

# Clients

EchoVault uses RESP, which makes it compatible with existing
Redis clients.

# Development Setup

Pre-requisites:
1) Go
2) Docker
3) Docker Compose
4) x86_64-linux-musl-gcc cross-compile toolchain as the development image is built for an Alpine container

Steps:
1) Clone the repository.
2) If you're on MacOS, you can run `make build && docker-compose up --build` to build the project and spin up the development docker container.
3) If you're on another OS, you will have to use `go build` with the relevant flags for your system.

# Table of Contents
1. [Configuration](#configuration)
2. [Eviction](#eviction)
3. [Contribution](#contribution)

# Configuration

EchoVault is highly configurable. It provides the following configuration options to you:

Flag: `--config`<br/>
Type: `string/path`<br/>
Description: The file path for the server configuration. A JSON or YAML file can be used for server configuration. You can combine CLI flags and config files, but remember that config files override CLI flags. The config file will be prioritised if you have the same config option in the CLI flags and the config file.

Flag: `--port`<br/>
Type: `integer`<br/>
Description: The port on which to listen to client connections. The default is `7480`.

Flag: `--bind-addr`<br/>
Type: `string`<br/>
Description: Specify the IP address to which the listener is bound.

Flag: `--require-pass`<br/>
Type: `boolean`<br/>
Description: Determines whether the server should require a password for the default user before allowing commands. The default is `false`. If this option is provided, it must be accompanied by the `--password` config.

Flag: `--password`<br/>
Type: `string`<br/>
Description: The password used to authorize the default user to run commands. This flag should be provided alongside the `--require-pass` flag.

Flag: `--tls`<br/>
Type: `boolean`<br/>
Description: A TLS connection with a client is required. The default is `false`.

Flag: `mtls`<br/>
Type: `boolean`<br/>
Description: Require mTLS connection with client. It is useful when the client and the server need to verify each other. If `--tls` and `mtls` are provided, `--mtls` will take higher priority. The default is `false`.

Flag: `--cert-key-pair`<br/>
Type: `string`<br/>
Description: The cert/key pair used by the server to authenticate itself to the client when using TLS or mTLS. This flag can be provided multiple times with multiple cert/key pairs. This is a comma-separated string in the following format: `<path-to-cert>,<path-to-key>`,

Flag: `--client-ca`<br/>
Type: `string`<br/>
Description: The path to the RootCA that is used to verify client certs when the `--mtls` flag is provided to enable verifying the client. This flag can be passed multiple times with paths to several client RootCAs.

Flag: `--server-id`<br/>
Type: `string`<br/>
Description: If this node is part of a raft replication cluster, then this flag provides the server ID to use within the cluster configuration. This ID must be unique to all the other nodes' IDs in the cluster.

Flag: `--join-addr`<br/>
Type: `string`<br/>
Description: When adding a node to a replication cluster, this is the address and port of any cluster member. The current node will use this to request permission to join the cluster. The format of this flag is `<ip-address>:<memberlist-port>`.

Flag: `--raft-port`<br/>
Type: `integer`<br/>
Description: If starting a node in a raft replication cluster, this port will be used for communication between nodes on the raft layer. The default is `7481`.

Flag: `--memberlist-port`<br/>
Type: `integer`<br/>
Description. If starting a node in a replication cluster, this port is used for communication between nodes on the memberlist layer. The default is `7946`.

Flag: `--in-memory`<br/>
Type: `boolean`<br/>
Description: When starting a node in a raft replication cluster, this directs the raft layer to store logs and snapshots in memory. It is only recommended in test mode. The default is `false`.

Flag: `--data-dir`<br/>
Type: `string`<br/>
Description: The directory for storing Append-Only Logs, Write Ahead Logs, and Snapshots. The default is `/var/lib/echovault`

Flag: `--bootstrap-cluster`<br/>
Type: `boolean`<br/>
Description: Whether to initialize a new replication cluster with this node as the leader. The default is `false`.

Flag: `--acl-config`<br/>
Type: `string`<br/>
Description: The file path for the ACL layer config file. The ACL configuration file can be a YAML or JSON file.

Flag: `--snapshot-threshold`<br/>
Type: `integer`<br/>
Description: The number of write commands required to trigger a snapshot. The default is `1,000`

Flag: `--snapshot-interval`<br/>
Type: `string`<br/>
Description: The interval between snapshots. You can provide a parseable time format such as `30m45s` or `1h45m`. The default is 5 minutes.

Flag: `--restore-snapshot`<br/>
Type: `boolean`<br/>
Description: Determines whether to restore from a snapshot on startup. The default is `false`.

Flag: `--restore-aof`<br/>
Type: `boolean`<br/>
Description: This flag determines whether to restore from an aof file on startup. If both this flag and `--restore-snapshot` are provided, this flag will take higher priority.

Flag: `--forward-commands`<br/>
Type: `boolean`<br/>
Description: This flag allows you to send write commands to any node in the cluster. The node will forward the command to the cluster leader. When this is false, write commands can only be accepted by the leader. The default is `false`.

Flag: `--max-memory`<br/>
Type: `string`<br/>
Examples: "200mb", "8gb", "1tb"<br/>
Description: The maximum memory usage that EchoVault should observe. Once this limit is reached, the chosen key eviction strategy is triggered. The default is no limit.

Flag: `--eviction-policy`<br/>
Type: `string`<br/>
Description: This flag allows you to choose the key eviction strategy when the maximum memory is reached. The flag accepts the following options:<br/>
1) noeviction - Do not evict any keys even when max-memory is exceeded. All new write operations will be rejected. This is the default eviction strategy.
2) allkeys-lfu - Evict the least frequently used keys when max-memory is exceeded.
3) allkeys-lru - Evict the least recently used keys when max-memory is exceeded.
4) volatile-lfu - Evict the least frequently used keys with an expiration when max-memory is exceeded.
5) volatile-lru - Evict the least recently used keys with an expiration when max-memory is exceeded.
6) allkeys-random - Evict random keys until we get under the max-memory limit when max-memory is exceeded.
7) volatile-random - Evict random keys with an expiration when max-memory is exceeded.

Flag: `--eviction-sample`<br/>
Type: `integer`<br/>
Description: An integer specifying the number of keys to sample when checking for expired keys. By default, EchoVault will sample 20 keys. The sampling is repeated if the number of expired keys found exceeds 20%.

Flag: `--eviction-interval`<br/>
Type: `string`<br/>
Example: "10s", "5m30s", "100ms"<br/>
Description: The interval between each sampling of keys to evict. By default, this happens every 100 milliseconds.

# Eviction

### Memory Limit
The memory limit can be set using the `--max-memory` config flag. This flag accepts a parsable memory value (e.g 100mb, 16gb). If the limit set is 0, then no memory limit is imposed. The default value is 0.

### Passive eviction
In passive eviction, the expired key is not deleted immediately after the expiry time. The key will remain in the store until the next time it is accessed. When attempting to access an expired key, that is when the key is deleted.

### Active eviction
Echovault will run a background goroutine that samples a set of volatile keys at a given interval. Any keys that are found to be expired will be deleted. If 20% or more of the sampled keys are deleted, then the process will immediately begin again. Otherwise, wait for the given interval until the next round of sampling/eviction. The default number of keys sampled is 20, and the default interval for sampling is 100 milliseconds. These can be configured using the `--eviction-sample` and `--eviction-interval` flags.

### Eviction Policies
Eviction policy can be set using the --eviction-policy flag. The following options are available.

<b>noeviction:</b><br/>
This policy does not evict any keys. When max memory is reached, all new write commands will be rejected until keys are manually deleted by the user.

<b>allkeys-lfu:</b><br/>
With this policy, all keys are considered for eviction when the max memory is reached. When max memory is reached, the least frequently accessed keys will be evicted until the memory usage is under the memory limit.

<b>allkeys-lru:</b><br/>
This policy will consider all keys for eviction when max memory is reached. The least recently accessed keys will be deleted one by one until we are below the memory limit.

<b>allkeys-random:</b><br/>
Evict random keys until we're below the max memory limit.

<b>volatile-lfu:</b><br/>
With this policy, only keys with an associated expiry time will be evicted to adhere to the memory limit. When the memory limit is exceeded, volatile keys will be evicted starting from the least frequently used until we are below the memory limit or are out of volatile keys to evict.
# Documentation

<b>volatile-lru:</b><br/>
With this policy, only keys with an associated expiry time will be evicted to adhere to the memory limit. When the memory limit is exceeded, volatile keys will be evicted starting from the list recently used until we are below the memory limit or are out of volatile keys to evict.
https://echovault.io

<b>volatile-random:</b><br/>
Evict random volatile keys until we're below the memory limit, or we're out of volatile keys to evict.

# Contribution

Contributions are welcome! If you're interested in contributing,
feel free to clone the repository and submit a Pull Request.

Join the [discord server](https://discord.gg/JrG4kPrF8v) if you'd like to discuss your contribution and/or
be a part of the community.
Loading

0 comments on commit 6ed59ef

Please sign in to comment.