doc: Schema generation instructions update

This PR will generate doc/lightning-*.md file contents for rpc commands. It will get the information from consolidated doc/schemas/lightning-*.json and use it to generate markdown for RPC commands. Changelog-None.
ElementsProject · Feb 8, 2024 · a46d2c2 · a46d2c2
1 parent 4427a0e
commit a46d2c2
Show file tree

Hide file tree

Showing 6 changed files with 192 additions and 190 deletions.
diff --git a/doc/contribute-to-core-lightning/code-generation.md b/doc/contribute-to-core-lightning/code-generation.md
@@ -3,7 +3,7 @@ title: "Code Generation"
 slug: "code-generation"
 hidden: false
 createdAt: "2023-04-22T12:29:01.116Z"
-updatedAt: "2023-04-22T12:44:47.814Z"
+updatedAt: "2024-01-18T12:44:47.814Z"
 ---
 The CLN project has a multitude of interfaces, most of which are generated from an abstract schema:
 
@@ -13,19 +13,11 @@ The CLN project has a multitude of interfaces, most of which are generated from
   1. addition of FD passing semantics to allow establishing a new connection between daemons (communication uses [socketpair](https://man7.org/linux/man-pages/man2/socketpair.2.html), so no `connect`)
   2. change the message length prefix from `u16` to `u32`, allowing for messages larger than 65Kb. The CSV files are with the respective sub-daemon and also use [generate-wire.py](https://github.com/ElementsProject/lightning/blob/master/tools/generate-wire.py) to generate encoding, decoding and printing functions
 
-- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `.request.json` for the request message, and a `.schema.json` for the response (the mismatch is historical and will eventually be addressed). During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`][msggen] tool. More on the generated stubs and utilities below.
+- We describe the JSON-RPC using [JSON Schema](https://json-schema.org/) in the [`doc/schemas`](https://github.com/ElementsProject/lightning/tree/master/doc/schemas) directory. Each method has a `lightning-*.json` for request and response. During tests the `pytest` target will verify responses, however the JSON-RPC methods are _not_ generated (yet?). We do generate various client stubs for languages, using the `msggen`[msggen] tool. More on the generated stubs and utilities below.
 
 ## Man pages
 
-The manpages are partially generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request schema and fills in the manpage between two markers:
-
-```markdown
-[comment]: # (GENERATE-FROM-SCHEMA-START)
-...
-[comment]: # (GENERATE-FROM-SCHEMA-END)
-```
-
-
+The manpages are generated from the JSON schemas using the [`fromschema`](https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py) tool. It reads the request and response schema from `lightning-*.json` and generates markdown contents and manpages:
 
 > 📘 
 > 
@@ -51,10 +43,6 @@ The manpages are partially generated from the JSON schemas using the [`fromschem
 }
 [/block]
 
-
-
-
-
 ### `cln-rpc`
 
 We use `msggen` to generate the Rust bindings crate [`cln-rpc`](https://github.com/ElementsProject/lightning/tree/master/cln-rpc). These bindings contain the stubs for the JSON-RPC methods, as well as types for the request and response structs. The [generator code](https://github.com/ElementsProject/lightning/blob/master/contrib/msggen/msggen/gen/rust.py) maps each abstract JSON-RPC type to a Rust type, minimizing size (e.g., binary data is hex-decoded).

diff --git a/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md b/doc/contribute-to-core-lightning/coding-style-guidelines/writing-json-schemas.md
@@ -3,36 +3,24 @@ title: "Writing JSON Schemas"
 slug: "writing-json-schemas"
 hidden: false
 createdAt: "2023-01-25T05:46:43.718Z"
-updatedAt: "2023-01-30T15:36:28.523Z"
+updatedAt: "2024-01-18T15:36:28.523Z"
 ---
-A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command responses, and also use it to generate our documentation.
+A JSON Schema is a JSON file which defines what a structure should look like; in our case we use it in our testsuite to check that they match command requests and responses, and also use it to generate our documentation.
 
 Yes, schemas are horrible to write, but they're damn useful.  We can only use a subset of the full [JSON Schema Specification](https://json-schema.org/), but if you find that limiting it's probably a sign that you should simplify your JSON output.
 
 ## Updating a Schema
 
-If you add a field, you should add it to the schema, and you must add "added": "VERSION" (where VERSION is the next release version!).
+If you add a field, you should add it to the field schema, and you must add "added": "VERSION" (where VERSION is the next release version!).
 
-Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version).  They will be removed two versions later.
+Similarly, if you deprecate a field, add "deprecated": "VERSION" (where VERSION is the next release version) to the field.  They will be removed two versions later.
 
 ## How to Write a Schema
 
-Name the schema doc/schemas/`command`.schema.json: the testsuite should pick it up and check all invocations of that command against it.
+Name the schema doc/schemas/lightning-`command`.json: the testsuite should pick it up and check all invocations of that command against it.
+The core lightning RPC commands use custom schema specification defined in [rpc-schema-draft](https://github.com/ElementsProject/lightning/doc/rpc-schema-draft.json).
 
-I recommend copying an existing one to start.
-
-You will need to put the magic lines in the manual page so `make doc-all` will fill it in for you:
-
-```json
-[comment]: # (GENERATE-FROM-SCHEMA-START)
-[comment]: # (GENERATE-FROM-SCHEMA-END)
-```
-
-
-
-If something goes wrong, try tools/fromscheme.py doc/schemas/`command`.schema.json to see how far it got before it died.
-
-You should always use `"additionalProperties": false`, otherwise your schema might not be covering everything.  Deprecated fields simply have `"deprecated": true` in their properties, so they are allowed by omitted from the documentation.
+I recommend copying an existing one to start. If something goes wrong, try tools/fromscheme.py doc/schemas/lightning-`command`.json to see how far it got before it died.
 
 You should always list all fields which are _always_ present in `"required"`.
 
@@ -57,6 +45,23 @@ To add conditional fields:
 6. Inside the "then", use `"additionalProperties": false` and place empty `{}` for all the other possible properties.
 7. If you haven't covered all the possibilties with `if` statements, add an `else` with `"additionalProperties": false` which simply mentions every allowable property.  This ensures that the fields can _only_ be present when conditions are met.
 
+### Exceptions in dynamic schema generation
+
+- If response (`RETURN VALUE`) should not be generated dynamically and you want it to be a custom text message instead. You can use `return_value_notes` to add custom text with empty `properties`. Examples: `setpsbtversion`, `commando`, `recover`.
+- If only one of multiple request parameters can be provided then utilize `oneOfMany`
+   key with condition defining arrays. For example, `plugin` command defines it as
+   `"oneOfMany": [["plugin", "directory"]]` and it prints the parameter output as
+   `[*plugin|directory*]`.
+- If request parameters are paired with other parameter and either all of them can be passed
+   to the command or none of them; then utilize `pairedWith` key with condition defining arrays.
+   For example, `delpay` command defines it as `"pairedWith": [["partid", "groupid"]]` 
+   and it prints the parameter output as `[*partid* *groupid*]`.
+- - If some of the optional request parameters are dependent upon other optional parameters,
+   use `dependentUpon` key where object key can be mapped with the array of dependent params.
+   For example, `listforwards` command has `start` and `limit` params dependent upon `index` and
+   it can be defined as `"dependentUpon": { "index": ["start", "limit"] }` in the json and it will
+   generate the markdown syntax as `[*index* [*start*] [*limit*]]`.
+
 ### JSON Drinking Game!
 
 1. Sip whenever you have an additional comma at the end of a sequence.

diff --git a/doc/developers-guide/app-development/rest.md b/doc/developers-guide/app-development/rest.md
@@ -78,7 +78,7 @@ With the default configurations, the Swagger user interface will be available at
 The POST method requires `rune` header for authorization.
 
 - A new `rune` can be created via [createrune](https://docs.corelightning.org/reference/lightning-createrune) or the list of 
-existing runes can be retrieved with [listrunes](https://docs.corelightning.org/reference/lightning-commando-listrunes) command.
+existing runes can be retrieved with [showrunes](https://docs.corelightning.org/reference/lightning-showrunes) command.
 
 Note: in version v23.08, a parameter `Nodeid` was required to be the id of the node we're talking to (see `id (pubkey)` received 
 from [getinfo](https://docs.corelightning.org/reference/lightning-getinfo)). You can still send this for backwards compatiblity, 

diff --git a/doc/developers-guide/plugin-development/a-day-in-the-life-of-a-plugin.md b/doc/developers-guide/plugin-development/a-day-in-the-life-of-a-plugin.md
@@ -38,55 +38,63 @@ The `getmanifest` method is required for all plugins and will be called on start
 
 ```json
 {
-  "options": [
-    {
-      "name": "greeting",
-      "type": "string",
-      "default": "World",
-      "description": "What name should I call you?",
-      "deprecated": false,
-      "dynamic": false
-    }
-  ],
-  "rpcmethods": [
-    {
-      "name": "hello",
-      "usage": "[name]",
-      "description": "Returns a personalized greeting for {greeting} (set via options)."
-    },
-    {
-      "name": "gettime",
-      "usage": "",
-      "description": "Returns the current time in {timezone}",
-      "long_description": "Returns the current time in the timezone that is given as the only parameter.\nThis description may be quite long and is allowed to span multiple lines.",
-      "deprecated": false
-    }
-  ],
-  "subscriptions": [
-    "deprecated_oneshot",
-    "connect",
-    "disconnect"
-  ],
-  "hooks": [
-    { "name": "openchannel", "before": ["another_plugin"] },
-    { "name": "htlc_accepted" }
-  ],
-  "featurebits": {
-    "node": "D0000000",
-    "channel": "D0000000",
-    "init": "0E000000",
-    "invoice": "00AD0000"
-  },
-  "notifications": [
-    {
-        "method": "mycustomnotification"
-    }
-  ],
-  "custommessages": [
-    11008, 11010
-  ],
-  "nonnumericids": true,
-  "dynamic": true
+	"options": [
+		{
+			"name": "greeting",
+			"type": "string",
+			"default": "World",
+			"description": "What name should I call you?",
+			"deprecated": false,
+			"dynamic": false
+		}
+	],
+	"rpcmethods": [
+		{
+			"name": "hello",
+			"usage": "[name]",
+			"description": "Returns a personalized greeting for {greeting} (set via options)."
+		},
+		{
+			"name": "gettime",
+			"usage": "",
+			"description": "Returns the current time in {timezone}",
+			"long_description": "Returns the current time in the timezone that is given as the only parameter.\nThis description may be quite long and is allowed to span multiple lines.",
+			"deprecated": false
+		}
+	],
+	"subscriptions": [
+		"deprecated_oneshot",
+		"connect",
+		"disconnect"
+	],
+	"hooks": [
+		{
+			"name": "openchannel",
+			"before": [
+				"another_plugin"
+			]
+		},
+		{
+			"name": "htlc_accepted"
+		}
+	],
+	"featurebits": {
+		"node": "D0000000",
+		"channel": "D0000000",
+		"init": "0E000000",
+		"invoice": "00AD0000"
+	},
+	"notifications": [
+		{
+			"method": "mycustomnotification"
+		}
+	],
+	"custommessages": [
+		11008,
+		11010
+	],
+	"nonnumericids": true,
+	"dynamic": true
 }
 ```
 
@@ -129,38 +137,38 @@ Nota bene: if a `flag` type option is not set, it will not appear in the options
 Here's an example option set, as sent in response to `getmanifest`
 
 ```json
-  "options": [
-    {
-      "name": "greeting",
-      "type": "string",
-      "default": "World",
-      "description": "What name should I call you?"
-    },
-    {
-      "name": "run-hot",
-      "type": "flag",
-      "description": "If set, overclocks plugin"
-    },
-    {
-      "name": "is_online",
-      "type": "bool",
-      "default": false,
-      "description": "Set to true if plugin can use network"
-    },
-    {
-      "name": "service-port",
-      "type": "int",
-      "default": 6666,
-      "description": "Port to use to connect to 3rd-party service"
-    },
-    {
-      "name": "number",
-      "type": "int",
-      "default": 0,
-      "description": "Another number to add",
-	  "multi": true
-    }
-  ],
+	"options": [
+		{
+			"name": "greeting",
+			"type": "string",
+			"default": "World",
+			"description": "What name should I call you?"
+		},
+		{
+			"name": "run-hot",
+			"type": "flag",
+			"description": "If set, overclocks plugin"
+		},
+		{
+			"name": "is_online",
+			"type": "bool",
+			"default": false,
+			"description": "Set to true if plugin can use network"
+		},
+		{
+			"name": "service-port",
+			"type": "int",
+			"default": 6666,
+			"description": "Port to use to connect to 3rd-party service"
+		},
+		{
+			"name": "number",
+			"type": "int",
+			"default": 0,
+			"description": "Another number to add",
+			"multi": true
+		}
+	]
 ```
 
 #### Custom notifications
@@ -175,7 +183,7 @@ When forwarding a custom notification `lightningd` will wrap the payload of the
   "method": "mycustomnotification",
   "params": {
     "key": "value",
-	"message": "Hello fellow plugin!"
+	  "message": "Hello fellow plugin!"
   }
 } 
 ```
@@ -205,29 +213,31 @@ The `init` method is required so that `lightningd` can pass back the filled comm
 
 ```json
 {
-  "options": {
-    "greeting": "World",
-	"number": [0]
-  },
-  "configuration": {
-    "lightning-dir": "/home/user/.lightning/testnet",
-    "rpc-file": "lightning-rpc",
-    "startup": true,
-    "network": "testnet",
-    "feature_set": {
-        "init": "02aaa2",
-        "node": "8000000002aaa2",
-        "channel": "",
-        "invoice": "028200"
-    },
-    "proxy": {
-        "type": "ipv4",
-        "address": "127.0.0.1",
-        "port": 9050
-    },
-    "torv3-enabled": true,
-    "always_use_proxy": false
-  }
+	"options": {
+		"greeting": "World",
+		"number": [
+			0
+		]
+	},
+	"configuration": {
+		"lightning-dir": "/home/user/.lightning/testnet",
+		"rpc-file": "lightning-rpc",
+		"startup": true,
+		"network": "testnet",
+		"feature_set": {
+			"init": "02aaa2",
+			"node": "8000000002aaa2",
+			"channel": "",
+			"invoice": "028200"
+		},
+		"proxy": {
+			"type": "ipv4",
+			"address": "127.0.0.1",
+			"port": 9050
+		},
+		"torv3-enabled": true,
+		"always_use_proxy": false
+	}
 }
 ```
 

diff --git a/doc/developers-guide/plugin-development/event-notifications.md b/doc/developers-guide/plugin-development/event-notifications.md
@@ -173,11 +173,13 @@ A notification for topic `invoice_creation` is sent every time an invoice is cre
   "invoice_creation": {
     "label": "unique-label-for-invoice",
     "preimage": "0000000000000000000000000000000000000000000000000000000000000000",
-    "amount_msat": 10000
+    "msat": 10000
   }
 }
 ```
 
+Before version `23.11` the `msat` field was a string with msat-suffix, e.g: `"10000msat"`.
+
 ### `warning`
 
 A notification for topic `warning` is sent every time a new `BROKEN`/`UNUSUAL` level(in plugins, we use `error`/`warn`) log generated, which means an unusual/borken thing happens, such as channel failed, message resolving failed...