Skip to content

Commit

Permalink
docs: Updates script and fixed test to generate lightning-sql
Browse files Browse the repository at this point in the history 
- Updated doc/Makefile for generating schemas/lightning-sql.json
    - Read the lightning-sql template from schemas/lightning-sql-template.json
    - Generate sql tables data and merge it with json object read from above template
    - Save final merged object in schemas/lightning-sql.json
- Updated doc/Makefile to auto generate lightning-sql.7.md and lightning-sql.7
- Deleted schemas/lightning-sql.json and adding it into .gitignore
- Added lightning-sql specific titles in the fromschema.py script
- Fixed test_sql by changing the sequence for listpeerchannels `reestablished` field
  • Loading branch information
@ShahanaFarooqui
ShahanaFarooqui committed Mar 18, 2024
1 parent 610bc12 commit 4b8fd4d
Show file tree
Hide file tree
Showing 8 changed files with 923 additions and 275 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ tests/plugins/test_selfdisable_after_getmanifest

# Ignore generated files
devtools/features
doc/schemas/lightning-sql.json
doc/lightning*.[1578]
doc/reckless*.[1578]
*_sqlgen.[ch]
Expand Down
265 changes: 265 additions & 0 deletions contrib/msggen/msggen/schema.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29266,6 +29266,271 @@
"Main web site: <https://github.com/ElementsProject/lightning>"
]
},
"lightning-sql-template.json": {
"$schema": "../rpc-schema-draft.json",
"type": "object",
"additionalProperties": false,
"added": "v23.02",
"rpc": "sql",
"title": "Command to do complex queries on list commands",
"description": [
"The **sql** RPC command runs the given query across a sqlite3 database created from various list commands.",
"",
"When tables are accessed, it calls the below commands, so it's no faster than any other local access (though it goes to great length to cache `listnodes` and `listchannels`) which then processes the results.",
"",
"It is, however faster for remote access if the result of the query is much smaller than the list commands would be."
],
"request": {
"required": [
"query"
],
"properties": {
"query": {
"type": "string",
"description": [
"The standard sqlite3 query to run.",
"Note that queries like \"SELECT *\" are fragile, as columns will change across releases; see lightning-listsqlschemas(7)."
]
}
}
},
"response": {
"required": [
"rows"
],
"properties": {
"rows": {
"type": "array",
"items": {
"type": "array"
}
},
"warning_db_failure": {
"type": "string",
"description": [
"A message if the database encounters an error partway through."
]
}
},
"pre_return_value_notes": [
"On success, an object containing **rows** is returned. It is an array. Each array entry contains an array of values, each an integer, real number, string or *null*, depending on the sqlite3 type.",
"",
"The object may contain **warning_db_failure** if the database fails partway through its operation."
]
},
"treatment_of_types": [
"The following types are supported in schemas, and this shows how they are presented in the database. This matters: a JSON boolean is represented as an integer in the database, so a query will return 0 or 1, not true or false.",
"",
"* *hex*. A hex string.",
" * JSON: a string",
" * sqlite3: BLOB",
"",
"* *hash*/*secret*/*pubkey*/*txid*: just like *hex*.",
"",
"* *msat*/*integer*/*u64*/*u32*/*u16*/*u8*. Normal numbers.",
" * JSON: an unsigned integer",
" * sqlite3: INTEGER",
"",
"* *boolean*. True or false.",
" * JSON: literal **true** or **false**",
" * sqlite3: INTEGER",
"",
"* *number*. A floating point number (used for times in some places).",
" * JSON: number",
" * sqlite3: REAL",
"",
"* *string*. Text.",
" * JSON: string",
" * sqlite3: TEXT",
"",
"* *short_channel_id*. A short-channel-id of form 1x2x3.",
" * JSON: string",
" * sqlite3: TEXT"
],
"permitted_sqlite3_functions": [
"Writing to the database is not permitted, and limits are placed on various other query parameters.",
"",
"Additionally, only the following functions are allowed:",
"",
"* abs",
"* avg",
"* coalesce",
"* count",
"* hex",
"* quote",
"* length",
"* like",
"* lower",
"* upper",
"* min",
"* max",
"* sum",
"* total"
],
"tables": [
"Note that the first column of every table is a unique integer called `rowid`: this is used for related tables to refer to specific rows in their parent. sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key.",
""
],
"errors": [
"On failure, an error is returned."
],
"example_usage": [
"Here are some example using lightning-cli. Note that you may need to use `-o` if you use queries which contain `=` (which make lightning-cli(1) default to keyword style):",
"",
"A simple peer selection query:",
"",
"```shell",
"$ lightning-cli sql \"SELECT id FROM peers\"",
"{",
" \"rows\": [",
" [",
" \"02ba9965e3db660385bd1dd2c09dd032e0f2179a94fc5db8917b60adf0b363da00\"",
" ]",
" ]",
"}",
"```",
"",
"A statement containing using `=` needs `-o`:",
"",
"```shell",
"$ lightning-cli sql -o \"SELECT node_id,last_timestamp FROM nodes WHERE last_timestamp>=1669578892\"",
"{",
" \"rows\": [",
" [",
" \"02ba9965e3db660385bd1dd2c09dd032e0f2179a94fc5db8917b60adf0b363da00\",",
" 1669601603",
" ]",
" ]",
"}",
"```",
"",
"If you want to compare a BLOB column, `x'hex'` or `X'hex'` are needed:",
"",
"```shell",
"$ lightning-cli sql -o \"SELECT nodeid FROM nodes WHERE nodeid != x'03c9d25b6c0ce4bde5ad97d7ab83f00ae8bd3800a98ccbee36f3c3205315147de1';\"",
"{",
" \"rows\": [",
" [",
" \"0214739d625944f8fdc0da9d2ef44dbd7af58443685e494117b51410c5c3ff973a\"",
" ],",
" [",
" \"02ba9965e3db660385bd1dd2c09dd032e0f2179a94fc5db8917b60adf0b363da00\"",
" ]",
" ]",
"}",
"$ lightning-cli sql -o \"SELECT nodeid FROM nodes WHERE nodeid IN (x'03c9d25b6c0ce4bde5ad97d7ab83f00ae8bd3800a98ccbee36f3c3205315147de1', x'02ba9965e3db660385bd1dd2c09dd032e0f2179a94fc5db8917b60adf0b363da00')\"",
"{",
" \"rows\": [",
" [",
" \"02ba9965e3db660385bd1dd2c09dd032e0f2179a94fc5db8917b60adf0b363da00\"",
" ],",
" [",
" \"03c9d25b6c0ce4bde5ad97d7ab83f00ae8bd3800a98ccbee36f3c3205315147de1\"",
" ]",
" ]",
"}",
"```",
"",
"Related tables are usually referenced by JOIN:",
"",
"```shell",
"$ lightning-cli sql -o \"SELECT nodeid, alias, nodes_addresses.type, nodes_addresses.port, nodes_addresses.address FROM nodes INNER JOIN nodes_addresses ON nodes_addresses.row = nodes.rowid\"",
"{",
" \"rows\": [",
" [",
" \"02ba9965e3db660385bd1dd2c09dd032e0f2179a94fc5db8917b60adf0b363da00\",",
" \"YELLOWWATCH-22.11rc2-31-gcd7593b\",",
" \"dns\",",
" 7272,",
" \"localhost\"",
" ],",
" [",
" \"0214739d625944f8fdc0da9d2ef44dbd7af58443685e494117b51410c5c3ff973a\",",
" \"HOPPINGSQUIRREL-1rc2-31-gcd7593b\",",
" \"dns\",",
" 7171,",
" \"localhost\"",
" ]",
" ]",
"}",
"```",
"",
"Simple function usage, in this case COUNT. Strings inside arrays need \", and ' to protect them from the shell:",
"",
"```shell",
"$ lightning-cli sql 'SELECT COUNT(*) FROM nodes\"",
"{",
" \"rows\": [",
" [",
" 3",
" ]",
" ]",
"}",
"```"
],
"example_json_request": [
{
"id": "example:sql#1",
"method": "sql",
"params": [
"SELECT * FROM forwards;"
]
},
{
"id": "example:sql#2",
"method": "sql",
"params": [
"SELECT * from peerchannels_features"
]
}
],
"example_json_response": [
{
"rows": []
},
{
"rows": [
[
6,
1,
0,
"option_static_remotekey"
],
[
7,
1,
1,
"option_anchors_zero_fee_htlc_tx"
],
[
16,
11,
0,
"option_static_remotekey"
],
[
17,
11,
1,
"option_anchors_zero_fee_htlc_tx"
]
]
}
],
"author": [
"Rusty Russell <<[email protected]>> is mainly responsible."
],
"see_also": [
"lightning-listtransactions(7)",
"lightning-listchannels(7)",
"lightning-listpeers(7)",
"lightning-listnodes(7)",
"lightning-listforwards(7)"
],
"resources": [
"Main web site: <https://github.com/ElementsProject/lightning>"
]
},
"lightning-sql.json": {
"$schema": "../rpc-schema-draft.json",
"type": "object",
Expand Down
11 changes: 5 additions & 6 deletions doc/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,12 +141,11 @@ MANPAGES := $(GENERATE_MARKDOWN) \

MARKDOWN_WITH_SCHEMA := $(GENERATE_MARKDOWN:=.md)

doc/schemas/lightning-sql.json: plugins/sql
# Delete the old tables schema and generate the new ones.
@if [ "$$(jq '[.tables[] | startswith("The following tables are currently supported:")]' doc/schemas/lightning-sql.json | jq 'any')" = "true" ]; then \
jq 'del(.tables[] | select(startswith("The following tables are currently supported:")))' "$@" > "[email protected]" && mv "[email protected]" "$@"; \
fi
@plugins/sql --print-docs | jq --arg sqldata "$$(awk '{printf "%s\n", $$0}')" '.tables += [$$sqldata]' "$@" > "[email protected]" && mv "[email protected]" "$@";
# - Read the json template from schemas/lightning-sql-template.json
# - Generate the tables schema via plugins/sql
# - Merge both and generate final schemas/lightning-sql.json
doc/schemas/lightning-sql.json: doc/schemas/lightning-sql-template.json plugins/sql
@plugins/sql --print-docs | jq --arg sqldata "$$(awk '{printf "%s\n", $$0}')" '.tables += [$$sqldata]' $< > "$@.tmp" && mv "$@.tmp" "$@";

doc-all: $(MANPAGES) doc/index.rst

Expand Down
Loading

0 comments on commit 4b8fd4d

Please sign in to comment.