Add time buckets and forwarded and rebalanced volume #231

ksedgwic · 2024-08-17T20:20:17Z

This PR (includes and) extends #228 ; only the final commits of this PR are unique ...

Addresses #229
Addresses #227

IMPORTANT - This PR makes a schema change and migrates data!

Abstract

This PR modifies the EarningsTracker schema to hold data in "buckets", one per day. This PR only modifies the collection and storage, the strategies and existing status continue to use earnings data from all time for now.

Motivation

As described in #227 there are several shortcomings w/ the current storage of EarningsTracker data:

Ancient data can dilute/overwhelm current data
Users would like income/expense reports for specific periods of time (especially recent periods)
The data doesn't "heal" as bugs are fixed and strategies are improved.

This PR enables time-based data earnings collection, future PR's should investigate modifying the strategies and reports to take advantage of the ability to filter by time range.

Specification

This PR updates the EarningsTracker table to have a "time bucket" column. The timestamps of incoming fee and balance events are quantized to a time bucket, currently one day.

The existing strategies and reports use the sql SUM to return the same answers that they would with the current schema.

The time and nodeid columns are indexed for efficient operations.

			CREATE TABLE IF NOT EXISTS EarningsTracker
			     ( node TEXT NOT NULL
			     , time_bucket REAL NOT NULL
			     , in_earnings INTEGER NOT NULL
			     , in_expenditures INTEGER NOT NULL
			     , out_earnings INTEGER NOT NULL
			     , out_expenditures INTEGER NOT NULL
			     , PRIMARY KEY (node, time_bucket)
			     );
			CREATE INDEX IF NOT EXISTS
			    idx_earnings_tracker_node_time ON EarningsTracker (node, time_bucket);
			CREATE INDEX IF NOT EXISTS
			    idx_earnings_tracker_time_node ON EarningsTracker (time_bucket, node);

Existing legacy data is migrated into the table with a time bucket value of 0. This allows it to be considered and used but eventually be aged out.

Two new CLI commands are added to allow inspecting the new data:

clboss-recent-earnings:
- Purpose: Returns a data structure equivalent to the
  offchain_earnings_tracker collection in clboss-status, but
  only includes recent earnings and expenditures.
- Arguments:
  - days (optional): Specifies the number of days to include in
    the report. Defaults to a fortnight (14 days) if not provided.
clboss-earnings-history:
- Purpose: Provides a daily breakdown of earnings and expenditures.
- Arguments:
  - nodeid (optional): Limits the history to a particular node if
    provided. Without this argument, the values are accumulated
    across all peers.
- Output:
  - The history consists of an array of records showing the earnings
    and expenditures for each day.
  - The history includes an initial record with a time value of 0,
    which contains any legacy earnings and expenditures collected by
    CLBOSS before daily tracking was implemented.

This allows effective feerates (PPM) to be computed for earnings and expenses.

This PR updates the schema automatically. Downgrading to previous will require manual DB migration (but is possible).

Adds two scripts which summarize earnings data. The first summarizes earning history by date:

cln@prod0:~$ ./clboss-earnings-history 
+------------+---------------+------+------------+---------------+-------+-----------+--------------+
|       Date |     Forwarded | Rate |   Earnings |    Rebalanced |  Rate |   Expense | Net Earnings |
+------------+---------------+------+------------+---------------+-------+-----------+--------------+
|     Legacy |             0 |    0 | 14_518_178 |             0 |     0 | 2_353_034 |   12_165_144 |
| 2024-08-10 |             0 |    0 |     26_029 |             0 |     0 |         0 |       26_029 |
| 2024-08-11 |             0 |    0 |     47_357 |             0 |     0 |         0 |       47_357 |
| 2024-08-12 |             0 |    0 |     30_054 |             0 |     0 |    76_458 |      -46_404 |
| 2024-08-13 |             0 |    0 |  2_798_957 |             0 |     0 |         0 |    2_798_957 |
| 2024-08-14 |             0 |    0 |     79_788 |             0 |     0 | 1_327_020 |   -1_247_232 |
| 2024-08-15 |             0 |    0 |     24_827 |             0 |     0 |    81_179 |      -56_352 |
| 2024-08-16 | 5_716_719_980 |  351 |  2_006_680 |   649_422_200 | 1,190 |   772_855 |    1_233_825 |
| 2024-08-17 | 2_178_796_021 |  394 |    857_664 |   429_497_800 |   156 |    66_936 |      790_728 |
|       ---- |     --------- | ---- |   -------- |    ---------- |  ---- |   ------- | ------------ |
|      TOTAL | 7_895_516_001 |      | 20_389_534 | 1_078_920_000 |       | 4_677_482 |   15_712_052 |
+------------+---------------+------+------------+---------------+-------+-----------+--------------+

The second summarizes the recent earnings data:

cln@prod0:~$ ./clboss-recent-earnings 1
+--------------------+---------------+-------+-----------+---------------+------+-----------+---------------+-------+---------+---------------+-------+---------+-----------+
|              Alias |  In Forwarded |  Rate |   In Earn | Out Forwarded | Rate |  Out Earn |      In Rebal |  Rate |  In Exp |     Out Rebal |  Rate | Out Exp |  Net Earn |
+--------------------+---------------+-------+-----------+---------------+------+-----------+---------------+-------+---------+---------------+-------+---------+-----------+
|          SilentBob |             0 |     0 |         0 | 3_923_897_717 |  696 | 2_732_514 |             0 |     0 |       0 |   640_091_200 | 1,203 | 769_885 | 1_962_629 |
|               Luna | 1_050_300_100 |   791 |   831_251 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |   831_251 |
|           U got IT |   841_647_852 |   958 |   806_085 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |   806_085 |
|     LQwD-Indonesia | 1_707_846_541 |   441 |   752_371 | 2_253_593_891 |   29 |    66_246 |             0 |     0 |       0 |   412_235_400 |   150 |  61_816 |   756_801 |
| NodeMcNodyface2000 |   952_854_280 |   164 |   156_665 | 1_310_011_343 |   16 |    20_830 |             0 |     0 |       0 |    26_593_400 |   304 |   8_090 |   169_405 |
|         PinsLNN_01 |   225_026_794 |   443 |    99_680 |   202_919_000 |   23 |     4_700 |             0 |     0 |       0 |             0 |     0 |       0 |   104_380 |
|        p.zax.cloud |    80_680_801 | 1,019 |    82_180 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |    82_180 |
|      Play-Asia.com |   149_076_430 |   442 |    65_818 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |    65_818 |
|   Start9 Donations |             0 |     0 |         0 |    50_000_000 |  536 |    26_775 |             0 |     0 |       0 |             0 |     0 |       0 |    26_775 |
|        silent.link |    58_717_000 |    21 |     1_230 |    48_566_440 |  120 |     5_843 |             0 |     0 |       0 |             0 |     0 |       0 |     7_073 |
|   LQwD-Japan-Osaka |    72_933_625 |    22 |     1_595 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |     1_595 |
|        pay.p2sh.co |             0 |     0 |         0 |    89_926_148 |   13 |     1_134 |             0 |     0 |       0 |             0 |     0 |       0 |     1_134 |
|    STACKER.NEWS    |    25_440_000 |    21 |       539 |    10_000_000 |   42 |       422 |             0 |     0 |       0 |             0 |     0 |       0 |       961 |
|   DeutscheBank|CLN |    21_186_250 |    41 |       869 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |       869 |
|   sats_distributor |    22_807_000 |    22 |       507 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |       507 |
|       MindlinerTre |    17_807_347 |    13 |       226 |             0 |    0 |         0 |             0 |     0 |       0 |             0 |     0 |       0 |       226 |
|            aybabtu |             0 |     0 |         0 |     6_601_462 |   12 |        79 |             0 |     0 |       0 |             0 |     0 |       0 |        79 |
|     slicksparks.ky | 2_107_096_453 |    23 |    47_635 |             0 |    0 |         0 |   346_639_000 | 1,193 | 413_391 |             0 |     0 |       0 |  -365_756 |
|       EverNukeNode |   562_095_528 |    31 |    17_693 |             0 |    0 |     5_801 |   732_281_000 |   582 | 426_400 |             0 |     0 |       0 |  -402_906 |
|              ----- |  ------------ |  ---- |   ------- | ------------- | ---- |  -------- |      -------- |  ---- |  ------ |     --------- |  ---- | ------- |  -------- |
|              TOTAL | 7_895_516_001 |   363 | 2_864_344 | 7_895_516_001 |  363 | 2_864_344 | 1_078_920_000 |   778 | 839_791 | 1_078_920_000 |   778 | 839_791 | 2_024_553 |
+--------------------+---------------+-------+-----------+---------------+------+-----------+---------------+-------+---------+---------------+-------+---------+-----------+

.gitignore

contrib/clboss-earnings-history

This makes it easier to generate test cases by using literal JSON.

This makes it easier to write test cases by comparing to expected values

…tracker A time source is needed for upcoming time buckets change.

This commit modifies the schema of EarningsTracker to allow storing and accessing earning and expenditure data in specific time ranges. All existing strategies and reports still use all data from all time so this PR should not change any balancing behavior. After we've run w/ this for a while we'll have time-based data collected and can evaluate how to improve the strategies.

…racker Addresses ([ZmnSCPxj#229]) This allows effective feerates (PPM) to be computed for earnings and expenses. This PR updates the schema automatically. Downgrading to previous will require manual DB migration (but is possible). Downgrade commands are in a comment in EarningsTracker.c

ksedgwic · 2024-09-09T18:32:08Z

I rebased onto master and improved the scripts output format

chrisguida · 2024-09-17T22:09:12Z

ok i'm running it

$ lightning-cli clboss-recent-earnings
{
   "recent": {},
   "total": {
      "in_earnings": 0,
      "in_forwarded": 0,
      "in_expenditures": 0,
      "in_rebalanced": 0,
      "out_earnings": 0,
      "out_forwarded": 0,
      "out_expenditures": 0,
      "out_rebalanced": 0
   }
}

$ lightning-cli clboss-earnings-history
{
   "history": [
      {
         "bucket_time": 0,
         "in_earnings": 0,
         "in_forwarded": 0,
         "in_expenditures": 0,
         "in_rebalanced": 0,
         "out_earnings": 0,
         "out_forwarded": 0,
         "out_expenditures": 0,
         "out_rebalanced": 0
      }
   ],
   "total": {
      "in_earnings": 0,
      "in_forwarded": 0,
      "in_expenditures": 0,
      "in_rebalanced": 0,
      "out_earnings": 0,
      "out_forwarded": 0,
      "out_expenditures": 0,
      "out_rebalanced": 0
   }
}

Any thing you need me to keep an eye on?

Any way I can generate some history?

This is a small mainnet node btw.

chrisguida · 2024-09-17T23:01:54Z

For some reason I can't get this to build on my raspi :/

chrisguida · 2024-09-18T00:53:22Z

okay it compiled on my raspi with make CXXFLAGS="-std=c++17"

any way we can add this to the Makefile?

running on my raspiblitz now! :)

ksedgwic · 2024-09-18T16:26:09Z

@chrisguida Check out the scripts in the contrib directory. They're described at the bottom of the README. They format (and augment w/ aliases) the JSON data ...

ksedgwic · 2024-09-18T16:35:39Z

Figuring out the CXXFLAGS="-std=c++17" part next ...

@chrisguida

@chrisguida reports that compilation fails on raspiblitz w/o this

ksedgwic · 2024-09-18T17:28:52Z

@chrisguida pushed a commit to configure for c++-17 ... does that fix the raspiblitz build?

chrisguida · 2024-09-19T02:44:39Z

@ksedgwic works great, thanks!! :D

chrisguida · 2024-09-20T00:26:30Z

@ksedgwic I tried the scripts in contrib/

₿ ls contrib/
clboss  clboss-earnings-history  clboss-recent-earnings  clboss-routing-stats

how are these scripts different from the rpc commands?

₿ cl help | grep clboss
clboss-dowser fromid toid
clboss-earnings-history [nodeid]
clboss-earnings-rebalancer 
clboss-externpay payment_hash
clboss-findbypopularity 
clboss-ignore-onchain [hours]
clboss-movefunds amount_msat sourcenode destnode feebudget_msat
clboss-notice-onchain 
    Cancel previous clboss-ignore-onchain and resume managing onchain funds.
clboss-recent-earnings [days]
clboss-status 
clboss-swaps 
clboss-unmanage nodeid tags

also, do you have a requirements file for these scripts?

₿ ./contrib/clboss-recent-earnings
Traceback (most recent call last):
  File "/home/bitcoin/cl-plugins-available/clboss/./contrib/clboss-recent-earnings", line 7, in <module>
    from tabulate import tabulate
ModuleNotFoundError: No module named 'tabulate'

ksedgwic · 2024-09-20T19:30:30Z

I need to add install stuff for the scripts, will do.

chrisguida · 2024-09-20T20:23:19Z

Tested, and works! missing some python deps

bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ pip install --user tabulate
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting tabulate
  Downloading https://www.piwheels.org/simple/tabulate/tabulate-0.9.0-py3-none-any.whl (35 kB)
Installing collected packages: tabulate
Successfully installed tabulate-0.9.0

[notice] A new release of pip is available: 23.3.1 -> 24.2
[notice] To update, run: python3 -m pip install --upgrade pip
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ 
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ 
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ ./clboss
clboss/                  clboss-earnings-history  clboss-recent-earnings   clboss-routing-stats     
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ ./clboss-earnings-history 
+------------+------------+-----+----------+------------+-----+---------+--------------+
|       Date |  Forwarded | PPM | Earnings | Rebalanced | PPM | Expense | Net Earnings |
+------------+------------+-----+----------+------------+-----+---------+--------------+
|     Legacy |          0 |   0 |        0 |          0 |   0 |       0 |            0 |
| 2024-09-18 | 12_396_811 |  10 |      123 |          0 |   0 |       0 |          123 |
| 2024-09-19 |          0 |   0 |        0 |          0 |   0 |       0 |            0 |
| 2024-09-20 |          0 |   0 |        0 |          0 |   0 |       0 |            0 |
|       ---- |  --------- | --- | -------- | ---------- | --- | ------- | ------------ |
|      TOTAL | 12_396_811 |     |      123 |          0 |     |       0 |          123 |
+------------+------------+-----+----------+------------+-----+---------+--------------+
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ 
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ 
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ ./clboss-recent-earnings 
+------------------+--------------+-----+---------+---------------+-----+----------+----------+-----+--------+-----------+-----+---------+----------+
|            Alias | In Forwarded | PPM | In Earn | Out Forwarded | PPM | Out Earn | In Rebal | PPM | In Exp | Out Rebal | PPM | Out Exp | Net Earn |
+------------------+--------------+-----+---------+---------------+-----+----------+----------+-----+--------+-----------+-----+---------+----------+
|       [redacted] |   12_396_811 |  10 |     123 |             0 |   0 |        0 |        0 |   0 |      0 |         0 |   0 |       0 |      123 |
|       [redacted] |            0 |   0 |       0 |    12_396_811 |  10 |      123 |        0 |   0 |      0 |         0 |   0 |       0 |      123 |
|            ----- | ------------ | --- | ------- | ------------- | --- | -------- | -------- | --- | ------ | --------- | --- | ------- | -------- |
|            TOTAL |   12_396_811 |  10 |     123 |    12_396_811 |  10 |      123 |        0 |   0 |      0 |         0 |   0 |       0 |      123 |
+------------------+--------------+-----+---------+---------------+-----+----------+----------+-----+--------+-----------+-----+---------+----------+
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ ./clboss-routing-stats 
Traceback (most recent call last):
  File "/home/bitcoin/cl-plugins-available/clboss/contrib/./clboss-routing-stats", line 16, in <module>
    from wcwidth import wcswidth
ModuleNotFoundError: No module named 'wcwidth'
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ pip install --user wcwidth
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting wcwidth
  Downloading https://www.piwheels.org/simple/wcwidth/wcwidth-0.2.13-py2.py3-none-any.whl (34 kB)
Installing collected packages: wcwidth
Successfully installed wcwidth-0.2.13

[notice] A new release of pip is available: 23.3.1 -> 24.2
[notice] To update, run: python3 -m pip install --upgrade pip
bitcoin@raspberrypi:~/cl-plugins-available/clboss/contrib $ ./clboss-routing-stats 
Alias             SCID                 to us    Age    Net msat  SPD    PeerID
[redacted]        [redacted]     862,854,842    216         123  13.3  [redacted]
[redacted]        [redacted]      28,133,574     73           0   2.7  [redacted]
[redacted]        [redacted]      13,174,053    180           0   2.3  [redacted]

contrib/README.md

ksedgwic mentioned this pull request Aug 17, 2024

Enhance EarningsTracker with time buckets and primary volume #230

Closed

ksedgwic changed the title ~~2024 08 earnings tracker volume~~ Add time buckets and forwarded and rebalanced volume Aug 17, 2024

ksedgwic force-pushed the 2024-08-earnings-tracker-volume branch 5 times, most recently from 0c974d0 to dad5753 Compare August 20, 2024 00:38

chrisguida reviewed Aug 21, 2024

View reviewed changes

.gitignore Show resolved Hide resolved

chrisguida reviewed Aug 22, 2024

View reviewed changes

contrib/clboss-earnings-history Outdated Show resolved Hide resolved

ksedgwic force-pushed the 2024-08-earnings-tracker-volume branch from dad5753 to 080468d Compare August 23, 2024 17:59

ksedgwic force-pushed the 2024-08-earnings-tracker-volume branch from 080468d to 2ee3b69 Compare September 9, 2024 17:34

ksedgwic added 9 commits September 9, 2024 10:36

factor parse_json into Jsmn::Object static method

d4ecee7

This makes it easier to generate test cases by using literal JSON.

add Jsmn::Object::operator==

72bf383

This makes it easier to write test cases by comparing to expected values

add test_earningstracker for legacy tracking

c943b79

add get_now() and mock_get_now() to EarningsTracker and test_earnings…

42c0488

…tracker A time source is needed for upcoming time buckets change.

add EarningsTracker::bucket_time quantizer and unit tests

ed7d2e5

add Either::operator<< to support debugging

ed8abad

add clboss-recent-earnings and clboss-earnings-history

77c9143

ksedgwic force-pushed the 2024-08-earnings-tracker-volume branch from 2ee3b69 to c3d679d Compare September 9, 2024 17:38

contrib: add clboss-earnings-history and clboss-recent-earnings scripts

c674e28

ksedgwic force-pushed the 2024-08-earnings-tracker-volume branch from c3d679d to c674e28 Compare September 12, 2024 03:13

configure to use c++17

8f1b44a

@chrisguida reports that compilation fails on raspiblitz w/o this

contrib: add poetry project to manange python dependencies

652b26f

chrisguida reviewed Sep 24, 2024

View reviewed changes

contrib/README.md Show resolved Hide resolved

contrib: add contrib/README.md

3b06673

ksedgwic force-pushed the 2024-08-earnings-tracker-volume branch from edc2e4c to 3b06673 Compare September 25, 2024 18:11

ksedgwic merged commit 305f049 into ZmnSCPxj:master Sep 25, 2024
1 check passed

ksedgwic mentioned this pull request Sep 25, 2024

Release v0.14.0 Tracker #237

Closed

5 tasks

ksedgwic mentioned this pull request Oct 18, 2024

Boss/Mod/EarningsTracker.cpp: Implement clboss-destroyearningsinfo command. #143

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add time buckets and forwarded and rebalanced volume #231

Add time buckets and forwarded and rebalanced volume #231

ksedgwic commented Aug 17, 2024

ksedgwic commented Sep 9, 2024

chrisguida commented Sep 17, 2024

chrisguida commented Sep 17, 2024

chrisguida commented Sep 18, 2024 •

edited

Loading

ksedgwic commented Sep 18, 2024

ksedgwic commented Sep 18, 2024

ksedgwic commented Sep 18, 2024

chrisguida commented Sep 19, 2024

chrisguida commented Sep 20, 2024

ksedgwic commented Sep 20, 2024

chrisguida commented Sep 20, 2024 •

edited

Loading

Add time buckets and forwarded and rebalanced volume #231

Add time buckets and forwarded and rebalanced volume #231

Conversation

ksedgwic commented Aug 17, 2024

Abstract

Motivation

Specification

ksedgwic commented Sep 9, 2024

chrisguida commented Sep 17, 2024

chrisguida commented Sep 17, 2024

chrisguida commented Sep 18, 2024 • edited Loading

ksedgwic commented Sep 18, 2024

ksedgwic commented Sep 18, 2024

ksedgwic commented Sep 18, 2024

chrisguida commented Sep 19, 2024

chrisguida commented Sep 20, 2024

ksedgwic commented Sep 20, 2024

chrisguida commented Sep 20, 2024 • edited Loading

chrisguida commented Sep 18, 2024 •

edited

Loading

chrisguida commented Sep 20, 2024 •

edited

Loading