pay: Remember and update channel_hints across payments

[cdecker]

Builds on top of #7487.

This PR takes the channel_hints that were previously created,
updated and then discarded as part of a payment, and instead we are
making them long-lived. This means that we share what we learn from
one payment with all later payments, allowing us to skip attempts that
will likely fail anyway, and improve the overall performance
(time-to-completion as well as success rate of payments).

The downside of this is that we are now keeping observations for
prolonged periods, meaning that these observations are no longer valid
when we use them. If used unchanged these earlier observations could
lead us to skip routes that we deem unavailable due to a prior
observation, but which is now back to being usable, which in turn
impacts our ability to complete a payment successfully. The core
contribution in this PR therefore is the time-based relaxation of
learnt constraints. It is based on a leaky-bucket approach, with a
linear refill-rate for depleted channels, such that after 2h we
consider any observation outdated.

Example

If at time t we observe a channel c with cap_c as overall
capacity, and bal_c = 0.5 * cap_c the currently available balance
(i.e., we observed a payment of bal_c + 1 fail to generate this
hint), then we have a refill rate r = cap_c / 7200, and after 1h the
channel is considered to be fully available again, i.e., we don't have
a more restrictive observation, and it is old enough that it does not
add to the structural information we already have from the gossip.

graph TD;
    CLN7487 --> CLN7494 ;
    click CLN7487 "https://github.com/ElementsProject/lightning/pull/7487"
    click CLN7494 "https://github.com/ElementsProject/lightning/pull/7494"
    style CLN7494 fill:#AAFFAA

Loading

[rustyrussell]

I rebased, removed the final commit (which was from a different PR, and already merged) and added a commit with my suggested changes.

But I'm not sure this PR is complete?

Addressed

[cdecker]

Ok, I think I addressed all the feedback:

• The channel_hint total_amount is now in msat again, saving us a couple of conversions
• The test_mutual_reconnect_race test has been re-added with a step down so a failed payment doesn't immediately cause all others to fail

Can we merge this now?