Skip to content

Commit

Permalink
bkpr: add two custom notifications that we listen for
Browse files Browse the repository at this point in the history
It might be nice to let the bookkeeper keep track of external accounts
as well as the internal onchain wallet? To this end, we add some new
custom notifications, which the bookkeeper will ingest and add to its
ledger.

Suggested-By: @chrisguida

Changelog-Added: PLUGINS: `bookkeeper` now listens for two custom events: `utxo_deposit` and `utxo_spend`. This allows for 3rd party plugins to send onchain coin events to the `bookkeeper`.  See the new plugins/bkpr/README.md for details on how these work!
  • Loading branch information
niftynei authored and rustyrussell committed Aug 9, 2024
1 parent cd4298d commit 0e99f2e
Show file tree
Hide file tree
Showing 6 changed files with 375 additions and 11 deletions.
77 changes: 77 additions & 0 deletions plugins/bkpr/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
The bookkeeper keeps track of coins moving through your Lightning node.

See the doc/PLUGINS.md#coin_movement section on the message that CLN emits for us to process.

// FIXME: add more detailed documenation for how bookkeeper works.


## 3rd Party Coin Movements
Bookeeper ingests 3rd party plugin notifications about on-chain movements that it should watch.

This allows for us to account for non-internal on-chain wallets in the single place, making `bookkeeper` your single source of truth for bitcoin for an organization or node-operator.

As a plugin writer, if you want to emit onchain events that the bookkeeper should track, you should emit an event with the following format:

```
{
"utxo_deposit": {
"account": "nifty's secret stash",
"transfer_from: null,
"outpoint": xxxx:x,
"amount_msat": "10000sat",
"coin_type": "bc",
"timestamp": xxxx,
"blockheight": xxx,
}
}
```

```
{
"utxo_spend": {
"account": "nifty's secret stash",
"outpoint": xxxx:x,
"spending_txid": xxxx,
"amount_msat": "10000sat",
"coin_type": "bc",
"timestamp": xxxx,
"blockheight": xxx,
}
}
```


## Withdrawing money (sending to a external account)

Sending money to an external account is a bit unintuitive in in the UTXO model that we're using to track coin moves; technically a send to an external account is a "deposit" to 3rd party's UTXO.

To account for these, `bookkeeper` expects to receive a `utxo_deposit` event for the creation of an output to a 3rd party. It's assumed that you'll issue these at transaction creation time, and that they won't be final until we receive notice of spend of the inputs in the tx that created them.

To notify that money is being sent to a 3rd party output, here's the event we'd expect.

The two keys here are the following:

- The `account` is `external`. This is a special account in `bookkeeper` and used for tracking external deposits (aka sends)
- The `transfer_from` field is set to the name of the account that is sending out the money.


```
{
"utxo_deposit": {
"account": "external",
"transfer_from": "nifty's secret stash",
"outpoint": xxxx:x,
"amount_msat": "10000sat",
"coin_type": "bc",
"timestamp": xxxx,
"blockheight": xxx,
}
}
```


## List of todos

List of things to check/work on, as a todo list.

- Transfers btw a 3rd party wallet and the internal CLN wallet? These should be registered as internal transfers and not show up in `listincome`
180 changes: 178 additions & 2 deletions plugins/bkpr/bookkeeper.c
Original file line number Diff line number Diff line change
Expand Up @@ -1519,7 +1519,8 @@ parse_and_log_chain_move(struct command *cmd,
/* Go see if there's any deposits to an external
* that are now confirmed */
/* FIXME: might need updating when we can splice? */
maybe_closeout_external_deposits(db, e);
maybe_closeout_external_deposits(db, e->spending_txid,
e->blockheight);
db_commit_transaction(db);
}

Expand Down Expand Up @@ -1674,6 +1675,173 @@ static char *parse_tags(const tal_t *ctx,
return NULL;
}

static struct command_result *json_utxo_deposit(struct command *cmd, const char *buf, const jsmntok_t *params)
{
const char *move_tag ="utxo_deposit";
struct chain_event *ev = tal(cmd, struct chain_event);
struct account *acct;
const char *err;

err = json_scan(tmpctx, buf, params,
"{payload:{utxo_deposit:{"
"account:%"
",transfer_from:%"
",outpoint:%"
",amount_msat:%"
",coin_type:%"
",timestamp:%"
",blockheight:%"
"}}}",
JSON_SCAN_TAL(tmpctx, json_strdup, &ev->acct_name),
JSON_SCAN_TAL(tmpctx, json_strdup, &ev->origin_acct),
JSON_SCAN(json_to_outpoint, &ev->outpoint),
JSON_SCAN(json_to_msat, &ev->credit),
JSON_SCAN_TAL(tmpctx, json_strdup, &ev->currency),
JSON_SCAN(json_to_u64, &ev->timestamp),
JSON_SCAN(json_to_u32, &ev->blockheight));

if (err)
plugin_err(cmd->plugin,
"`%s` payload did not scan %s: %.*s",
move_tag, err, json_tok_full_len(params),
json_tok_full(buf, params));

/* Log the thing */
db_begin_transaction(db);
acct = find_account(tmpctx, db, ev->acct_name);

if (!acct) {
acct = new_account(tmpctx, ev->acct_name, NULL);
account_add(db, acct);
}

ev->tag = "deposit";
ev->ignored = false;
ev->stealable = false;
ev->rebalance = false;
ev->debit = AMOUNT_MSAT(0);
ev->output_value = ev->credit;
ev->spending_txid = NULL;
ev->payment_id = NULL;
ev->desc = NULL;

plugin_log(cmd->plugin, LOG_DBG, "%s (%s|%s) %s -%s %"PRIu64" %d %s",
move_tag, ev->tag, ev->acct_name,
fmt_amount_msat(tmpctx, ev->credit),
fmt_amount_msat(tmpctx, ev->debit),
ev->timestamp, ev->blockheight,
fmt_bitcoin_outpoint(tmpctx, &ev->outpoint));

if (!log_chain_event(db, acct, ev)) {
db_commit_transaction(db);
/* This is not a new event, do nothing */
return notification_handled(cmd);
}

/* Can we calculate any onchain fees now? */
err = maybe_update_onchain_fees(cmd, db, &ev->outpoint.txid);
db_commit_transaction(db);
if (err)
plugin_err(cmd->plugin,
"Unable to update onchain fees %s",
err);

/* FIXME: do account close checks, when allow onchain close to externals? */
return notification_handled(cmd);;
}

static struct command_result *json_utxo_spend(struct command *cmd, const char *buf, const jsmntok_t *params)
{
const char *move_tag ="utxo_spend";
struct account *acct;
struct chain_event *ev = tal(cmd, struct chain_event);
const char *err, *acct_name;

ev->spending_txid = tal(ev, struct bitcoin_txid);
err = json_scan(tmpctx, buf, params,
"{payload:{utxo_spend:{"
"account:%"
",outpoint:%"
",spending_txid:%"
",amount_msat:%"
",coin_type:%"
",timestamp:%"
",blockheight:%"
"}}}",
JSON_SCAN_TAL(tmpctx, json_strdup, &acct_name),
JSON_SCAN(json_to_outpoint, &ev->outpoint),
JSON_SCAN(json_to_txid, ev->spending_txid),
JSON_SCAN(json_to_msat, &ev->debit),
JSON_SCAN_TAL(tmpctx, json_strdup, &ev->currency),
JSON_SCAN(json_to_u64, &ev->timestamp),
JSON_SCAN(json_to_u32, &ev->blockheight));

if (err)
plugin_err(cmd->plugin,
"`%s` payload did not scan %s: %.*s",
move_tag, err, json_tok_full_len(params),
json_tok_full(buf, params));

/* Log the thing */
db_begin_transaction(db);
acct = find_account(tmpctx, db, acct_name);

if (!acct) {
acct = new_account(tmpctx, acct_name, NULL);
account_add(db, acct);
}

ev->origin_acct = NULL;
ev->tag = "withdrawal";
ev->ignored = false;
ev->stealable = false;
ev->rebalance = false;
ev->credit = AMOUNT_MSAT(0);
ev->output_value = ev->debit;
ev->payment_id = NULL;
ev->desc = NULL;

plugin_log(cmd->plugin, LOG_DBG, "%s (%s|%s) %s -%s %"PRIu64" %d %s %s",
move_tag, ev->tag, acct_name,
fmt_amount_msat(tmpctx, ev->credit),
fmt_amount_msat(tmpctx, ev->debit),
ev->timestamp, ev->blockheight,
fmt_bitcoin_outpoint(tmpctx, &ev->outpoint),
fmt_bitcoin_txid(tmpctx, ev->spending_txid));

if (!log_chain_event(db, acct, ev)) {
db_commit_transaction(db);
/* This is not a new event, do nothing */
return notification_handled(cmd);
}

err = maybe_update_onchain_fees(cmd, db, ev->spending_txid);
if (err) {
db_commit_transaction(db);
plugin_err(cmd->plugin,
"Unable to update onchain fees %s",
err);
}

err = maybe_update_onchain_fees(cmd, db, &ev->outpoint.txid);
if (err) {
db_commit_transaction(db);
plugin_err(cmd->plugin,
"Unable to update onchain fees %s",
err);
}

/* Go see if there's any deposits to an external
* that are now confirmed */
/* FIXME: might need updating when we can splice? */
maybe_closeout_external_deposits(db, ev->spending_txid,
ev->blockheight);
db_commit_transaction(db);

/* FIXME: do account close checks, when allow onchain close to externals? */
return notification_handled(cmd);;
}

static struct command_result *json_coin_moved(struct command *cmd,
const char *buf,
const jsmntok_t *params)
Expand Down Expand Up @@ -1749,7 +1917,15 @@ const struct plugin_notification notifs[] = {
{
"balance_snapshot",
json_balance_snapshot,
}
},
{
"utxo_deposit",
json_utxo_deposit,
},
{
"utxo_spend",
json_utxo_spend,
},
};

static const struct plugin_command commands[] = {
Expand Down
11 changes: 6 additions & 5 deletions plugins/bkpr/recorder.c
Original file line number Diff line number Diff line change
Expand Up @@ -1950,11 +1950,12 @@ char *maybe_update_onchain_fees(const tal_t *ctx, struct db *db,
}

void maybe_closeout_external_deposits(struct db *db,
struct chain_event *ev)
const struct bitcoin_txid *txid,
u32 blockheight)
{
struct db_stmt *stmt;

assert(ev->spending_txid);
assert(txid);
stmt = db_prepare_v2(db, SQL("SELECT "
" e.id"
" FROM chain_events e"
Expand All @@ -1966,7 +1967,7 @@ void maybe_closeout_external_deposits(struct db *db,

/* Blockheight for unconfirmeds is zero */
db_bind_int(stmt, 0);
db_bind_txid(stmt, ev->spending_txid);
db_bind_txid(stmt, txid);
db_bind_text(stmt, EXTERNAL_ACCT);
db_query_prepared(stmt);

Expand All @@ -1979,7 +1980,7 @@ void maybe_closeout_external_deposits(struct db *db,
" blockheight = ?"
" WHERE id = ?"));

db_bind_int(update_stmt, ev->blockheight);
db_bind_int(update_stmt, blockheight);
db_bind_u64(update_stmt, id);
db_exec_prepared_v2(take(update_stmt));
}
Expand All @@ -1994,7 +1995,7 @@ bool log_chain_event(struct db *db,
struct db_stmt *stmt;

/* We're responsible for de-duping chain events! */
if (find_chain_event(e, db, acct,
if (find_chain_event(tmpctx, db, acct,
&e->outpoint, e->spending_txid,
e->tag))
return false;
Expand Down
11 changes: 7 additions & 4 deletions plugins/bkpr/recorder.h
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,9 @@ void add_payment_hash_desc(struct db *db,
*
* This method updates the blockheight on these events to the
* height an input was spent into */
void maybe_closeout_external_deposits(struct db *db, struct chain_event *ev);
void maybe_closeout_external_deposits(struct db *db,
const struct bitcoin_txid *txid,
u32 blockheight);

/* Keep track of rebalancing payments (payments paid to/from ourselves.
* Returns true if was rebalance */
Expand All @@ -224,9 +226,10 @@ void log_channel_event(struct db *db,
struct channel_event *e);

/* Log a chain event.
* Returns true if inserted, false if already exists */
* Returns true if inserted, false if already exists;
* ctx is for allocating objects onto chain_event `e` */
bool log_chain_event(struct db *db,
const struct account *acct,
struct chain_event *e);
const struct account *acct,
struct chain_event *e);

#endif /* LIGHTNING_PLUGINS_BKPR_RECORDER_H */
52 changes: 52 additions & 0 deletions tests/plugins/bookkeeper_custom_coins.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
from pyln.client import Plugin


plugin = Plugin()


UTXO_DEPOSIT_TAG = "utxo_deposit"
UTXO_SPEND_TAG = "utxo_spend"


@plugin.method("sendspend")
def emit_spend(plugin, acct, outpoint, txid, amount, **kwargs):
"""Emit a 'utxo_spend' movement
"""
utxo_spend = {
"account": acct,
"outpoint": outpoint,
"spending_txid": txid,
"amount_msat": amount,
"coin_type": "bcrt",
"timestamp": 1679955976,
"blockheight": 111,
}
plugin.notify(UTXO_SPEND_TAG, {UTXO_SPEND_TAG: utxo_spend})


@plugin.method("senddeposit")
def emit_deposit(plugin, acct, is_withdraw, outpoint, amount, **kwargs):
"""Emit a 'utxo_deposit' movement
"""
transfer_from = None

if is_withdraw:
acct = "external"
transfer_from = acct

utxo_deposit = {
"account": acct,
"transfer_from": transfer_from,
"outpoint": outpoint,
"amount_msat": amount,
"coin_type": "bcrt",
"timestamp": 1679955976,
"blockheight": 111,
}
plugin.notify(UTXO_DEPOSIT_TAG, {UTXO_DEPOSIT_TAG: utxo_deposit})


plugin.add_notification_topic(UTXO_DEPOSIT_TAG)
plugin.add_notification_topic(UTXO_SPEND_TAG)
plugin.run()
Loading

0 comments on commit 0e99f2e

Please sign in to comment.