From 5c307b378bbc0a6bddc41f8cf7cef1072c5f79fe Mon Sep 17 00:00:00 2001 From: Borislav Stanimirov Date: Sat, 23 Nov 2024 16:36:45 +0200 Subject: [PATCH] docs: update some docs with the new plugin changes, ref #204 --- doc/dict.md | 2 ++ doc/iapi.md | 5 +++++ doc/lapi.md | 7 ++----- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/doc/dict.md b/doc/dict.md index 6af94869..69273f25 100644 --- a/doc/dict.md +++ b/doc/dict.md @@ -16,3 +16,5 @@ The supported types are: * `object` - a nested key-value store * `array` - an ordered list of values * `binary` - a contiguous memory buffer + +`Dict` is currently implemented as a typedef of [`nlohmann::json`](https://github.com/nlohmann/json) diff --git a/doc/iapi.md b/doc/iapi.md index 723e2ab1..11728944 100644 --- a/doc/iapi.md +++ b/doc/iapi.md @@ -11,6 +11,10 @@ The inference API defines the communication protocol with a given inference engi "description": "A schema for the inference API of a model type", "type": "object", "properties": { + "id": { + "description": "A unique identifier of the schema", + "type": "string" + }, "description": { "description": "A human-readable description of the inference engine", "type": "string" @@ -69,6 +73,7 @@ As an example, here's how the whisper.cpp schema looked like at some point durin ```json { + "id": "whisper", "description": "Inference based on our fork of https://github.com/ggerganov/whisper.cpp", "params": { "type": "null" }, "instances": { diff --git a/doc/lapi.md b/doc/lapi.md index 9d8705c0..b3913d59 100644 --- a/doc/lapi.md +++ b/doc/lapi.md @@ -6,18 +6,18 @@ The language api is programming language specific, but all implementations share Elements of the Language API provide language-specific way to interact with [`Dict`-s](dict.md) and the [Inference API Elements](intro.md#api-elements). They include: +* An interface to load plugins * A representation of `Dict` which is used to parameters and results * This includes representations of `Dict` values like `string`, `number`, `null` * ... and cruically also `binary`, which is used for arbitrary Inference-API-specific binary data * An way to define a model description, characterized by: * Inference type: used to route to the correct inference engine * Assets: a list of assets needed for the model. For now the assets are just paths to files on the filesystem, but this is likely to change (expanded with more options) in the future. -* A model factory, which creates models based on a description. +* An interface to model loaders (wich come from the plugins) which is used to create models based on a description. * An representation of `Model` which can be used to create instances * An representation of `Instance` * ... which can be used to run `op`-s. Essentially `Dict -> Dict` functions - ## Characteristics The Language API is low level. It's meant to be wrapped in higher-level application-specific abstractions, which normally would not expose the Inference API directly (or at least not completely). @@ -31,6 +31,3 @@ Some of the low level implications are listed below * This includes automatic reference counting, such as in in C++ (via `std::shared_ptr`) or Swift, but excludes "classic" garbage collection such as in Java or Python. * Fine grain management of resources is critical. Models and instances can take up gigabytes of memory, and it's important to have control over when they're destroyed. * **Instaces keep strong refs to models**: When you have an instance and you don't need more instances of the same model, it's safe to release the model. - - -