[bug]: Unexpected Pathfinding Failure After Edge Node Experiences Unhandled Error FeeInsufficient #1240
Comments
|
I think the
I wasn't able to reproduce this in an itest, likely it's dependent on exact balances at the moment of the attempt... |
|
tl;dr: the TEMPORARY_CHANNEL_FAILURE above was due to Alice generating a 100k asset invoice with <100k assets of inbound liquidity (100003 - 356 = 99647). I was able to repro that, and the fact that it makes a Taproot Assets channel inactive, however the Taproot Assets aspect of the channel is able to be recovered by disconnecting and reconnecting which allows it to receive asset payments for which it has inbound liquidity. The sats aspect of the channel appears to be irrevocably borked, however. Before making the 1 sat payment that is expected to fail, here is Carol's listchannels output (showing the sats-only channel between her and Bob): And here is Alice's listchannels output (showing the Taproot Assets channel between her and Bob): I had Alice generate an invoice for 1 asset: And then Carol attempt to pay it with sats: After this failure, all channels are still active. But Carol immediately fails any attempted payment to Alice with NO_ROUTE: All channels are still listed as active at this point. I started with a small invoice this time, and after a mission control reset Carol succeeds in paying a Taproot Assets invoice generated by Alice, but immediately fails any attempt to pay a sats invoice generated by Alice: Alice's Taproot Assets channel at this point has <100000 assets of inbound liquidity. So I'll generate an invoice for 100000 assets like I did in the OP: Carol attempting to pay that invoice (again after a lncli resetmc) reproduces the TEMPORARY_CHANNEL_FAILURE error, though that is expected as there is not enough asset liquidity for Bob to forward the payment to Alice: And again that error causes Alice and Bob's channel to go inactive, preventing Alice from generating more Taproot Assets invoices, but not more sats invoices: Though Carol can't pay that sats invoice anyways: Disconnecting and reconnecting Alice and Bob brings the channel back to active. Carol still cannot pay sats invoices, but can pay a Taproot Assets invoice of 1k assets (which Alice has plenty of inbound liquidity for): Logs attached with the proper settings: |
|
Looks like we're running into the same error as is being fixed by the forward bandwidth manager PR (#1236 and lightninglabs/lightning-terminal#912): |
|
I tried increasing |
|
Note, I do not get this issue if I try to exchange "5 sat worth" of an asset directly with a peer. This kind of makes sense though because the peer doesn't need to route (and pay the 354 sat) for the next hop. |
I tried to increase to 2000 as well (just to make sure it is well over 354*2 to pay for each hop), and that still fails. |
Just tested with the PR's you mentioned merged and I am still getting the same issues. |
|
Fixed by #1236. |
github-project-automation
bot
moved this from 🆕 New
to ✅ Done
in Taproot-Assets Project Board
Can you test with RC2 please? Perhaps something in the versioning got mixed up? |
|
here's the build I'm using: https://github.com/lightninglabs/tapdvalidation/blob/05d5cca6e6f34f5073255741666f1089b85df3e3/docker-compose.yaml#L47 which points to: https://github.com/lightninglabs/lightning-terminal/blob/1f4a0cd56422bd82410c022fe89ba8d3a7c66239/go.mod#L26 which says |
|
|
The test case I'm running is defined here: https://github.com/lightninglabs/tapdvalidation/blob/05d5cca6e6f34f5073255741666f1089b85df3e3/scripts/failed_tests And my network setup is defined here: https://github.com/lightninglabs/tapdvalidation/blob/05d5cca6e6f34f5073255741666f1089b85df3e3/scripts/init_network Instructions for rebuilding this setup can be found here: https://github.com/lightninglabs/tapdvalidation/blob/05d5cca6e6f34f5073255741666f1089b85df3e3/README.md I think my setup resembles @ryanthegentry 's. @ryanthegentry , can you confirm? |
|
|
Here is my network topology if you didn't bother to click on the links above: Some nodes are for other tests (which don't fail). |
|
Thanks. Can you please run |
|
IIUC, the case posted above is an asset to asset multi-hop test case. We can see how this differs from our existing itest for this case to find the gap: https://github.com/lightninglabs/lightning-terminal/blob/1f4a0cd56422bd82410c022fe89ba8d3a7c66239/itest/litd_custom_channels_test.go#L1981-L2003 |
The case I'm working on and I believe is the same as @ryanthegentry is a case where we have one node sending sats, an intermediate node, and then a final node accepting an asset. The topology I have above has asset channels for both, but since there are also sats in the asset channel between alice and bob, I'm just sending sats through that for this test case. I believe @ryanthegentry is doing the same thing with a sats only channel as the first hop. @ryanthegentry 's case is more specific to the problem whereas I've built a network that can run a wider range of test cases. |
Okay, so it was easier for me to run Not sure if you wanted me to do this on George instead? Because it was so much data, I only included running
and skipped
|
Actually, I need to correct myself. The case
is indeed not multi asset (it is sats to asset), which is what this issue is about, but in my file https://github.com/lightninglabs/tapdvalidation/blob/05d5cca6e6f34f5073255741666f1089b85df3e3/scripts/failed_tests I also include an asset to asset payment, which is the case
. In the example above in #1240 (comment), the asset to asset case runs second and it fails without even trying to do any HTLC because of the mission control issue. I have tried running it on it's own before nodes get blacklisted and it also gets the fee insufficient error. I think we need to forget about the asset to asset case for now and focus on the simpler sats to asset case. I'm not having this issue with asset to sats at all, so I'm hopeful if we get the sats to asset case fixed, asset to asset will work too. |
Yeah, sure, that works as well.
If what you're experiencing is the same issue as Ryan ran into, you should be able to find I'll also try to run your script suite locally in the meantime, so perhaps I can reproduce it before you're online. |
|
@ZZiigguurraatt was able to run your script and reproduce. Will look into it. I have everything I need for now, will report back. |
Did you see https://github.com/user-attachments/files/18104794/debug_started.txt linked above? |
|
Found the problem. See #1252. Basically we need to enforce that an asset invoice can only be created for an equivalent amount of 354 satoshi or more of assets. |
github-project-automation
bot
moved this from 🏗 In progress
to ✅ Done
in Taproot-Assets Project Board
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
I am testing single hop Taproot Assets LN payments in Polar. I have three litd v0.14.0-alpha.rc1 nodes (each backed by their own bitcoind v27.0 node) Alice, Bob, and Carol. Alice has minted 21M units of a Taproot Asset called USDTest with a decimal_display of 6, and opened a 1M asset channel to Bob. Bob has then opened a 1M sats channel to Carol, and is using the improved edge node config --taproot-assets.experimental.rfq.mockoracleassetsperbtc=100000000. I am testing Carol trying to pay Taproot Assets invoices generated by Alice with sats, with Bob acting as the edge node.
Before beginning this sequence of test payments, Alice's listchannels output looked like so:
Carol had 186,070 local_balance of sats, and 810,460 remote_balance of sats (owned by Bob).
I then made one successful payment where Alice invoiced 356 assets, and Carol paid it with sats:
And then one expected failure, where Alice only invoiced 1 asset, and Carol failed to find a route (bc Bob is not interested in receiving 0 + 0 sats, and forwarding out 354 sats + 1 asset).
The unexpected behavior begins here however, as Carol is no longer able to pay Alice after this. Initially Alice and Bob's Taproot Assets channel remains active, but Carol just immediately fails an attempted payment:
Assuming she blacklisted the receiving node from her mission control for some reason, I reset it and try to pay Alice's invoice again but this time am hit with a TEMPORARY_CHANNEL_FAILURE at the Taproot Assets channel:
And after this attempt, Alice no longer sees the channel as active:
I can get the channel back to active by disconnecting Alice from Bob, and then reconnecting:
But then Carol encounters the same TEMPORARY_CHANNEL_FAILURE when trying to pay Alice's invoice again:
And Alice sees the channel as inactive:
After disconnecting and reconnecting again, Bob attempts to pay Alice's invoice directly, instead of acting as an edge node, but the payment fails and the CLI hangs (i.e. needs to be manually killed):
I think the key line in this final attempt from Bob's logs is here:
And earlier in the saga during the initial failed payment, the key line in Bob's logs is here:
Your environment
Alice, Bob and Carol's litd config essentially look like this:
Alice's tapcli getinfo output:
Bob's tapcli getinfo output:
Carol's tapcli getinfo output:
Steps to reproduce
Expected behavior
Carol should be able to successfully pay Alice in step 3, and in step 4, and the Taproot Assets channel should not go inactive every time it is asked to forward a payment in steps 5 and 6.
Actual behavior
After step 2, Alice and Bob's Taproot Assets channel is incapable for forwarding a payment. It appears Bob is unable to find the channel link.
Logs attached:
alice-deadchan.log
bob-deadchan.log
carol-deadchan.log
The text was updated successfully, but these errors were encountered: