cl-naive-store

cl-naive-store is a log structured document store. Documents are loaded in-memory to facilitate fast querying. Depending on how you use the store documents will be lazy loaded and indexed. It is written completely in Common Lisp.

The “naive” comes from the fact that data is persisted as plists in files to make them human and machine readable, also there is no rocket science code.

The store was designed to be customisable, just about anything can be customised, have a look at the implementation API to get an idea of what is possible.

Status

The package is considered stable, but will keep on evolving as it is used in some commercial software.

License

cl-naive-store is released under the MIT license.

News

• The concept of a multiverse was added to the store. Previously the top level container for data was a universe (a clustering of databases). A multiverse is the top structural container for data now. A multiverse contains one or more universes. A multiverse could be viewed as a clustering of clusters of databases.
• Code underwent a majour refactoring. There where many changes but the use of cl-naive-deprecation should allow old code to run as is (compiler macros replace old code with new).
• Tests where rewritten to use cl-naive-tests and are a lot simpler to read and learn from. Test coverage was also expanded considerably.
• Code to manage your database schemas was added to naive-core. You can now use schema files to initialise and load a database.
• Speed tests where added for those that want to dig into performance metrics. Going through speed tests code will teach you a couple of tricks about how to use the library optimally.
• Documentation was updated and expanded in some places. The documentation PDF has been restructured to follow the org document structuring now. Examples and some other stuff that was missing from the PDF is now included as well.

Acknowledgements

Thank you very much to everyone that contributed over the years.

Features

Persisted

Data can be written to file for each document update, or as a batch update.

In memory

Data is loaded into memory for querying and lookups, which makes these operations fast.

Lazy Loading

Data is only loaded when needed. If you use the store correctly it means that you will only have the data that users requested up to that point in memory.

Indexed

Documents and key-values can be indexed, and a user can specify their own additional indexes as well. Queries and lookups can both be done using indexes which speeds up the retrieval of data considerably.

Sharding

Sharding is the breaking down of files into smaller files, in the case of naive-store that means that instead of one file per collection there could be many.

Sharding is done based on the actual data in collections. The user specifies which elements of a document it wants to use for sharding on a collection. If none is specified no sharding is done.

Layered Design

cl-naive-store can do a lot but you as the user decides how much of the store’s functionality you want to use for your own project.

Functionality was broken down into these packages:

• cl-naive-store.naive-core
• cl-naive-store.document-types
• cl-naive-store.naive-documents
• cl-naive-store.naive-indexed
• cl-naive-store.naive-merkle
• cl-naive-store.test

The following .asd files can be used to load different functionality:

• cl-naive-store.naive-core.asd loads the most basic functionality for cl-naive-store. Use this if you don’t any of the other extensions.
• cl-naive-store.naive-indexed.asd loads naive-core and index functionality.
• cl-naive-store.document-types.asd loads naive-core and document-type functionality.
• cl-naive-store.naive-documents.asd loads naive-core, naive-indexed, documents-types, and document functionality.
• cl-naive-store.asd loads the whole shebang.
• cl-naive-store.tests.asd loads tests

Schemas (New)

You can use schema definitions to create and load whole database structures.

Documentation

The documentation can be found in the docs folder in the repository, start with the home.org file.

Installation

*Be careful there is an older version of cl-naive-store in quicklisp that has not been removed yet. You need to clone the relevant projects to get the latest version.*

The new dependencies are not in quicklisp and wont get there in the forseeable future so you have to clone them.

git clone https://gitlab.com/naive-x/cl-getx.git
git clone https://gitlab.com/naive-x/cl-naive-tests.git
git clone https://gitlab.com/naive-x/cl-naive-deprecation.git
git clone https://gitlab.com/naive-x/cl-naive-ptrees.git
git clone https://gitlab.com/naive-x/cl-naive-store.git

If you want to use them with quicklisp you need to create symbolic links in quicklisp/local-projects/.

The rest of the dependencies can all be found in quicklisp.

(ql:quicklisp :cl-naive-store)

Examples

Examples are in the examples folder in the git repository. If those are to simplistic for you have a look at the code in the tests.

Dependencies

• bordeaux-threads
• cl-fad
• cl-murmurhash
• cl-getx
• cl-cpus
• cl-naive-deprecation
• cl-naive-ptrees
• cl-naive-sb-cover-ext (only needed when running coverage jobs)
• cl-naive-tests (for tests only)
• iron-clad
• local-time
• lparallel
• split-sequence
• uuid

Supported CL Implementations

All tests pass on SBCL an CCL

Coverage

**WARNING: Only works on SBCL!*

You can refresh the coverage artefacts by running make.

make coverage

You can view coverage artefacts in src/tests/coverage/.

The coverage.xml and coverage.txt files are produced to register coverage with gitlab.

The src/tests/coverage/sb-core/ contains the sb-cover artifacts open the home.html and navigate from there.

The coverage is aggressively trimmed to only show coverage for src/ directory.

Tests

(ql:quickload :cl-naive-store.tests)
(cl-naive-store.tests:test-all)

Alternatively you can use the make file to run tests.

Go to the tests folder

cd cl-naive-store/tests/

Run make with any of the following

• test
• test-run-tests
• run-tests-ccl
• run-tests-sbcl

For example:

make test-run-tests

You should see the following at the end.

Testsuite TEST-DEFINITIONS:

Testsuite TEST-BASIC:

Testsuite TEST-BASIC-PERSISTED:

Testsuite TEST-INDEXED:

Testsuite TEST-NAIVE-DOCUMENTS:

Testsuite TEST-SHARDING-SIMPLE:

Testsuite TEST-SHARDING-INDEXED:
Passed:       48
Failed:        0