From a46d2c26e5944dca20c5ac097100be1ba224b783 Mon Sep 17 00:00:00 2001 From: ShahanaFarooqui Date: Thu, 18 Jan 2024 11:07:45 -0800 Subject: [PATCH] doc: Schema generation instructions update This PR will generate doc/lightning-*.md file contents for rpc commands. It will get the information from consolidated doc/schemas/lightning-*.json and use it to generate markdown for RPC commands. Changelog-None. --- .../code-generation.md | 18 +- .../writing-json-schemas.md | 43 ++-- doc/developers-guide/app-development/rest.md | 2 +- .../a-day-in-the-life-of-a-plugin.md | 220 +++++++++--------- .../plugin-development/event-notifications.md | 4 +- doc/schemas/WRITING_SCHEMAS.md | 95 ++++---- 6 files changed, 192 insertions(+), 190 deletions(-) diff --git a/doc/contribute-to-core-lightning/code-generation.md b/doc/contribute-to-core-lightning/code-generation.md index 5eba77849750..2af618109396 100644 --- a/doc/contribute-to-core-lightning/code-generation.md +++ b/doc/contribute-to-core-lightning/code-generation.md @@ -3,7 +3,7 @@ title: "Code Generation" slug: "code-generation" hidden: false createdAt: "2023-04-22T12:29:01.116Z" -updatedAt: "2023-04-22T12:44:47.814Z" +updatedAt: "2024-01-18T12:44:47.814Z" --- The CLN project has a multitude of interfaces, most of which are generated from an abstract schema: @@ -13,19 +13,11 @@ The CLN project has a multitude of interfaces, most of which are generated from 1. addition of FD passing semantics to allow establishing a new connection between daemons (communication uses [socketpair](https://man7.org/linux/man-pages/man2/socketpair.2.html), so no `connect`) 2. change the message length prefix from `u16` to `u32`, allowing for messages larger than 65Kb. The CSV files are with the respective sub-daemon and also use [generate-wire.py](https://github.com/ElementsProject/lightning/blob/master/tools/generate-wire.py) to generate encoding, decoding and printing functions -- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `.request.json` for the request message, and a `.schema.json` for the response (the mismatch is historical and will eventually be addressed). During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`][msggen] tool. More on the generated stubs and utilities below. +- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `lightning-*.json` for request and response. During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`[msggen] tool. More on the generated stubs and utilities below. ## Man pages -The manpages are partially generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request schema and fills in the manpage between two markers: - -```markdown -[comment]: # (GENERATE-FROM-SCHEMA-START) -... -[comment]: # (GENERATE-FROM-SCHEMA-END) -``` - - +The manpages are generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request and response schema from `lightning-*.json` and generates markdown contents and manpages: > 📘 > @@ -51,10 +43,6 @@ The manpages are partially generated from the JSON schemas using the [`fromschem } [/block] - - - - ### `cln-rpc` We use `msggen` to generate the Rust bindings crate [`cln-rpc`](https://github.com/ElementsProject/lightning/tree/master/cln-rpc). These bindings contain the stubs for the JSON-RPC methods, as well as types for the request and response structs. The [generator code](https://github.com/ElementsProject/lightning/blob/master/contrib/msggen/msggen/gen/rust.py) maps each abstract JSON-RPC type to a Rust type, minimizing size (e.g., binary data is hex-decoded). diff --git a/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md b/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md index c2d4609275d3..f46292f7caa8 100644 --- a/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md +++ b/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md @@ -3,36 +3,24 @@ title: "Writing JSON Schemas" slug: "writing-json-schemas" hidden: false createdAt: "2023-01-25T05:46:43.718Z" -updatedAt: "2023-01-30T15:36:28.523Z" +updatedAt: "2024-01-18T15:36:28.523Z" --- -A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command responses, and also use it to generate our documentation. +A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command requests and responses, and also use it to generate our documentation. Yes, schemas are horrible to write, but they're damn useful. We can only use a subset of the full [JSON Schema Specification](https://json-schema.org/), but if you find that limiting it's probably a sign that you should simplify your JSON output. ## Updating a Schema -If you add a field, you should add it to the schema, and you must add "added": "VERSION" (where VERSION is the next release version!). +If you add a field, you should add it to the field schema, and you must add "added": "VERSION" (where VERSION is the next release version!). -Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version). They will be removed two versions later. +Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version) to the field. They will be removed two versions later. ## How to Write a Schema -Name the schema doc/schemas/`command`.schema.json: the testsuite should pick it up and check all invocations of that command against it. +Name the schema doc/schemas/lightning-`command`.json: the testsuite should pick it up and check all invocations of that command against it. +The core lightning RPC commands use custom schema specification defined in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json). -I recommend copying an existing one to start. - -You will need to put the magic lines in the manual page so `make doc-all` will fill it in for you: - -```json -[comment]: # (GENERATE-FROM-SCHEMA-START) -[comment]: # (GENERATE-FROM-SCHEMA-END) -``` - - - -If something goes wrong, try tools/fromscheme.py doc/schemas/`command`.schema.json to see how far it got before it died. - -You should always use `"additionalProperties": false`, otherwise your schema might not be covering everything. Deprecated fields simply have `"deprecated": true` in their properties, so they are allowed by omitted from the documentation. +I recommend copying an existing one to start. If something goes wrong, try tools/fromscheme.py doc/schemas/lightning-`command`.json to see how far it got before it died. You should always list all fields which are _always_ present in `"required"`. @@ -57,6 +45,23 @@ To add conditional fields: 6. Inside the "then", use `"additionalProperties": false` and place empty `{}` for all the other possible properties. 7. If you haven't covered all the possibilties with `if` statements, add an `else` with `"additionalProperties": false` which simply mentions every allowable property. This ensures that the fields can _only_ be present when conditions are met. +### Exceptions in dynamic schema generation + +- If response (`RETURN VALUE`) should not be generated dynamically and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover`. +- If only one of multiple request parameters can be provided then utilize `oneOfMany` + key with condition defining arrays. For example, `plugin` command defines it as + `"oneOfMany": [["plugin", "directory"]]` and it prints the parameter output as + `[*plugin|directory*]`. +- If request parameters are paired with other parameter and either all of them can be passed + to the command or none of them; then utilize `pairedWith` key with condition defining arrays. + For example, `delpay` command defines it as `"pairedWith": [["partid", "groupid"]]` + and it prints the parameter output as `[*partid* *groupid*]`. +- - If some of the optional request parameters are dependent upon other optional parameters, + use `dependentUpon` key where object key can be mapped with the array of dependent params. + For example, `listforwards` command has `start` and `limit` params dependent upon `index` and + it can be defined as `"dependentUpon": { "index": ["start", "limit"] }` in the json and it will + generate the markdown syntax as `[*index* [*start*] [*limit*]]`. + ### JSON Drinking Game! 1. Sip whenever you have an additional comma at the end of a sequence. diff --git a/doc/developers-guide/app-development/rest.md b/doc/developers-guide/app-development/rest.md index 1ade48d0c7d6..11aaea78f2b1 100644 --- a/doc/developers-guide/app-development/rest.md +++ b/doc/developers-guide/app-development/rest.md @@ -78,7 +78,7 @@ With the default configurations, the Swagger user interface will be available at The POST method requires `rune` header for authorization. - A new `rune` can be created via [createrune](https://docs.corelightning.org/reference/lightning-createrune) or the list of -existing runes can be retrieved with [listrunes](https://docs.corelightning.org/reference/lightning-commando-listrunes) command. +existing runes can be retrieved with [showrunes](https://docs.corelightning.org/reference/lightning-showrunes) command. Note: in version v23.08, a parameter `Nodeid` was required to be the id of the node we're talking to (see `id (pubkey)` received from [getinfo](https://docs.corelightning.org/reference/lightning-getinfo)). You can still send this for backwards compatiblity, diff --git a/doc/developers-guide/plugin-development/a-day-in-the-life-of-a-plugin.md b/doc/developers-guide/plugin-development/a-day-in-the-life-of-a-plugin.md index 414180e69723..d37989f2d8cf 100644 --- a/doc/developers-guide/plugin-development/a-day-in-the-life-of-a-plugin.md +++ b/doc/developers-guide/plugin-development/a-day-in-the-life-of-a-plugin.md @@ -38,55 +38,63 @@ The `getmanifest` method is required for all plugins and will be called on start ```json { - "options": [ - { - "name": "greeting", - "type": "string", - "default": "World", - "description": "What name should I call you?", - "deprecated": false, - "dynamic": false - } - ], - "rpcmethods": [ - { - "name": "hello", - "usage": "[name]", - "description": "Returns a personalized greeting for {greeting} (set via options)." - }, - { - "name": "gettime", - "usage": "", - "description": "Returns the current time in {timezone}", - "long_description": "Returns the current time in the timezone that is given as the only parameter.\nThis description may be quite long and is allowed to span multiple lines.", - "deprecated": false - } - ], - "subscriptions": [ - "deprecated_oneshot", - "connect", - "disconnect" - ], - "hooks": [ - { "name": "openchannel", "before": ["another_plugin"] }, - { "name": "htlc_accepted" } - ], - "featurebits": { - "node": "D0000000", - "channel": "D0000000", - "init": "0E000000", - "invoice": "00AD0000" - }, - "notifications": [ - { - "method": "mycustomnotification" - } - ], - "custommessages": [ - 11008, 11010 - ], - "nonnumericids": true, - "dynamic": true + "options": [ + { + "name": "greeting", + "type": "string", + "default": "World", + "description": "What name should I call you?", + "deprecated": false, + "dynamic": false + } + ], + "rpcmethods": [ + { + "name": "hello", + "usage": "[name]", + "description": "Returns a personalized greeting for {greeting} (set via options)." + }, + { + "name": "gettime", + "usage": "", + "description": "Returns the current time in {timezone}", + "long_description": "Returns the current time in the timezone that is given as the only parameter.\nThis description may be quite long and is allowed to span multiple lines.", + "deprecated": false + } + ], + "subscriptions": [ + "deprecated_oneshot", + "connect", + "disconnect" + ], + "hooks": [ + { + "name": "openchannel", + "before": [ + "another_plugin" + ] + }, + { + "name": "htlc_accepted" + } + ], + "featurebits": { + "node": "D0000000", + "channel": "D0000000", + "init": "0E000000", + "invoice": "00AD0000" + }, + "notifications": [ + { + "method": "mycustomnotification" + } + ], + "custommessages": [ + 11008, + 11010 + ], + "nonnumericids": true, + "dynamic": true } ``` @@ -129,38 +137,38 @@ Nota bene: if a `flag` type option is not set, it will not appear in the options Here's an example option set, as sent in response to `getmanifest` ```json - "options": [ - { - "name": "greeting", - "type": "string", - "default": "World", - "description": "What name should I call you?" - }, - { - "name": "run-hot", - "type": "flag", - "description": "If set, overclocks plugin" - }, - { - "name": "is_online", - "type": "bool", - "default": false, - "description": "Set to true if plugin can use network" - }, - { - "name": "service-port", - "type": "int", - "default": 6666, - "description": "Port to use to connect to 3rd-party service" - }, - { - "name": "number", - "type": "int", - "default": 0, - "description": "Another number to add", - "multi": true - } - ], + "options": [ + { + "name": "greeting", + "type": "string", + "default": "World", + "description": "What name should I call you?" + }, + { + "name": "run-hot", + "type": "flag", + "description": "If set, overclocks plugin" + }, + { + "name": "is_online", + "type": "bool", + "default": false, + "description": "Set to true if plugin can use network" + }, + { + "name": "service-port", + "type": "int", + "default": 6666, + "description": "Port to use to connect to 3rd-party service" + }, + { + "name": "number", + "type": "int", + "default": 0, + "description": "Another number to add", + "multi": true + } + ] ``` #### Custom notifications @@ -175,7 +183,7 @@ When forwarding a custom notification `lightningd` will wrap the payload of the "method": "mycustomnotification", "params": { "key": "value", - "message": "Hello fellow plugin!" + "message": "Hello fellow plugin!" } } ``` @@ -205,29 +213,31 @@ The `init` method is required so that `lightningd` can pass back the filled comm ```json { - "options": { - "greeting": "World", - "number": [0] - }, - "configuration": { - "lightning-dir": "/home/user/.lightning/testnet", - "rpc-file": "lightning-rpc", - "startup": true, - "network": "testnet", - "feature_set": { - "init": "02aaa2", - "node": "8000000002aaa2", - "channel": "", - "invoice": "028200" - }, - "proxy": { - "type": "ipv4", - "address": "127.0.0.1", - "port": 9050 - }, - "torv3-enabled": true, - "always_use_proxy": false - } + "options": { + "greeting": "World", + "number": [ + 0 + ] + }, + "configuration": { + "lightning-dir": "/home/user/.lightning/testnet", + "rpc-file": "lightning-rpc", + "startup": true, + "network": "testnet", + "feature_set": { + "init": "02aaa2", + "node": "8000000002aaa2", + "channel": "", + "invoice": "028200" + }, + "proxy": { + "type": "ipv4", + "address": "127.0.0.1", + "port": 9050 + }, + "torv3-enabled": true, + "always_use_proxy": false + } } ``` diff --git a/doc/developers-guide/plugin-development/event-notifications.md b/doc/developers-guide/plugin-development/event-notifications.md index 677dfd05ba44..203fb9e545e6 100644 --- a/doc/developers-guide/plugin-development/event-notifications.md +++ b/doc/developers-guide/plugin-development/event-notifications.md @@ -173,11 +173,13 @@ A notification for topic `invoice_creation` is sent every time an invoice is cre "invoice_creation": { "label": "unique-label-for-invoice", "preimage": "0000000000000000000000000000000000000000000000000000000000000000", - "amount_msat": 10000 + "msat": 10000 } } ``` +Before version `23.11` the `msat` field was a string with msat-suffix, e.g: `"10000msat"`. + ### `warning` A notification for topic `warning` is sent every time a new `BROKEN`/`UNUSUAL` level(in plugins, we use `error`/`warn`) log generated, which means an unusual/borken thing happens, such as channel failed, message resolving failed... diff --git a/doc/schemas/WRITING_SCHEMAS.md b/doc/schemas/WRITING_SCHEMAS.md index aff29fcef60e..33ade3c20650 100644 --- a/doc/schemas/WRITING_SCHEMAS.md +++ b/doc/schemas/WRITING_SCHEMAS.md @@ -12,73 +12,70 @@ your JSON output. ## Updating a Schema -If you add a field, you should add it to the schema, and you must add +If you add a field, you should add it to the field schema, and you must add "added": "VERSION" (where VERSION is the next release version!). -Similarly, if you deprecate a field, add "deprecated": "VERSION" (where -VERSION is the next release version). They will be removed two versions -later. +Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION +is the next release version) to the field. They will be removed two versions later. ## How to Write a Schema -Name the schema doc/schemas/`command`.schema.json: the testsuite should -pick it up and check all invocations of that command against it. +Name the schema doc/schemas/lightning-`command`.json: the testsuite should pick it +up and check all invocations of that command against it. The core lightning RPC +commands use custom schema specification defined in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json). -I recommend copying an existing one to start. +I recommend copying an existing one to start. If something goes wrong, try +tools/fromscheme.py doc/schemas/lightning-`command`.json to see how far it +got before it died. -You will need to put the magic lines in the manual page so `make doc-all` -will fill it in for you: +You should always list all fields which are _always_ present in `"required"`. -``` -[comment]: # (GENERATE-FROM-SCHEMA-START) -[comment]: # (GENERATE-FROM-SCHEMA-END) -``` +We extend the basic types; see [fixtures.py](https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-testing/pyln/testing/fixtures.py). -If something goes wrong, try tools/fromscheme.py -doc/schemas/`command`.schema.json to see how far it got before it died. - -You should always use `"additionalProperties": false`, otherwise -your schema might not be covering everything. Deprecated fields -simply have `"deprecated": true` in their properties, so they -are allowed by omitted from the documentation. - -You should always list all fields which are *always* present in -`"required"`. - -We extend the basic types; see [fixtures.py][fixtures]. - -[fixtures]: https://github.com/ElementsProject/lightning/blob/master/contrib/pyln-testing/pyln/testing/fixtures.py - -In addition, before committing a new schema or a new version of it, make sure that it -is well formatted. If you don't want do it by hand, use `make fmt-schema` that uses -jq under the hood. +In addition, before committing a new schema or a new version of it, make sure +that it is well formatted. If you don't want do it by hand, use `make fmt-schema` +that uses jq under the hood. ### Using Conditional Fields -Sometimes one field is only sometimes present; if you can, you should make -the schema know when it should (and should not!) be there. +Sometimes one field is only sometimes present; if you can, you should make the schema +know when it should (and should not!) be there. -There are two kinds of conditional fields expressable: fields which -are only present if another field is present, or fields only present -if another field has certain values. +There are two kinds of conditional fields expressable: fields which are only present +if another field is present, or fields only present if another field has certain values. To add conditional fields: -1. Do *not* mention them in the main "properties" section. +1. Do _not_ mention them in the main "properties" section. 2. Set `"additionalProperties": true` for the main "properties" section. -3. Add an `"allOf": [` array at the same height as `"properties"'`. Inside - this place one `if`/`then` for each conditional field. -4. If a field simply requires another field to be present, use the pattern +3. Add an `"allOf": [` array at the same height as `"properties"'`. Inside this place + one `if`/`then` for each conditional field. +4. If a field simply requires another field to be present, use the pattern `"required": [ "field" ]` inside the "if". -5. If a field requires another field value, use the pattern - `"properties": { "field": { "enum": [ "val1", "val2" ] } }` inside - the "if". -6. Inside the "then", use `"additionalProperties": false` and place - empty `{}` for all the other possible properties. -7. If you haven't covered all the possibilties with `if` statements, - add an `else` with `"additionalProperties": false` which simply - mentions every allowable property. This ensures that the fields - can *only* be present when conditions are met. +5. If a field requires another field value, use the pattern + `"properties": { "field": { "enum": [ "val1", "val2" ] } }` inside the "if". +6. Inside the "then", use `"additionalProperties": false` and place empty `{}` for + all the other possible properties. +7. If you haven't covered all the possibilties with `if` statements, add an `else` + with `"additionalProperties": false` which simply mentions every allowable property. + This ensures that the fields can _only_ be present when conditions are met. + +### Exceptions in dynamic schema generation + +- If response (`RETURN VALUE`) should not be generated dynamically and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover`. +- If only one of multiple request parameters can be provided then utilize `oneOfMany` + key with condition defining arrays. For example, `plugin` command defines it as + `"oneOfMany": [["plugin", "directory"]]` and it prints the parameter output as + `[*plugin|directory*]`. +- If request parameters are paired with other parameter and either all of them can be passed + to the command or none of them; then utilize `pairedWith` key with condition defining arrays. + For example, `delpay` command defines it as `"pairedWith": [["partid", "groupid"]]` + and it prints the parameter output as `[*partid* *groupid*]`. +- - If some of the optional request parameters are dependent upon other optional parameters, + use `dependentUpon` key where object key can be mapped with the array of dependent params. + For example, `listforwards` command has `start` and `limit` params dependent upon `index` and + it can be defined as `"dependentUpon": { "index": ["start", "limit"] }` in the json and it will + generate the markdown syntax as `[*index* [*start*] [*limit*]]`. ### JSON Drinking Game!