"Ran out of routes to try after XX attempts"

[simongltr]

Hey,

I open this request to track the error we have in our app since the recent events. I heard from Breez that most of the issues are behind us and that it should work again but we still have issues.

The error we get for each payment:

Payment timeout: status: Unknown, message: "Error calling method Pay:
RcError { code: Some(210), message:
"Ran out of routes to try after XX attempts: see 'paystatus'", data:
None }", details: [], metadata:
MetadataMap { headers: {"content-type": "application/grpc", "date":
"Wed, 24 Apr 2024 19:41:46 GMT",
"content-length": "0"} }

Sender's node id:

03811368ccafc98c085998a5113ad669f22cbbabc88e5a00ea52aff5ae8c824abb

Receiver's invoice

lnbc150n1pnz5juudqqpp5ru55jnh7uj2d7a6mt5mmy5uqn4v4ant39d8w776esfqt320e9upsxqrrsssp5cjhadgzczq4spyh9jn0eu547mm3pt9depqswhft0fvc4xd0mhhqq9qrsgqcqzysrzjqtypret4hcklglvtfrdt85l3exc0dctdp4qttmtcy5es3lpt6uts6jrcryfj7cp3lgqqqqlgqqqqqzsqjqva9n0jrq9regtp3xg28kyr46srqzthyxsa24d847rr2uedjr7hxhcy2vjkpv895m4hrql85l0yajsmfgl2k4du2ulxqpyklfygrn89sqnrzuv3

Our LSP is LQWD:

03232432128398d8cb31e67921ffd4d4cfb10b42dc72529a45dba6f105adfb83d4

Let me know if you need more data or informations.

Best,
Simon.

[kingonly]

@simongltr can you please try again and report if the issue is still happening?

[damiandizeo]

@simongltr can you please try again and report if the issue is still happening?

We tried again, it fails but with a different error message

Payment timeout: status: Unknown, message: "Error calling method Pay:
RpcError { code: Some(210), message:
\"Destination 032f6993f42C8a01405e33bba6080237c07a7e67f62f1d7da407ef7e390cafc606 is not
reachable directly and all routehints
were unusable.\", data: None }", details:
Il, metadata: MetadataMap { headers: {"content-type": "application/grpc",
"date": "Fri, 26 Apr 2024 11:46:23
GMT", "content-length": "0"} }

[kingonly]

@simongltr there was another fix deployed (more on Monday), could you please try again? Try to make a few payments if possible.

[simongltr]

@simongltr there was another fix deployed (more on Monday), could you please try again? Try to make a few payments if possible.

My node ID is 032f6993f42c8a01405e33bba6080237c07a7e67f62f1d7da407ef7e390cafc606

I just tried paying [email protected]and [email protected], I got respectively:
"Ran out of routes to try after 2 attempts" and "Ran out of routes to try after 1 attempts"

A few payments worked but it was to pay myself.

One of the payments to WalletOfSatoshi gave me no error but after 3 minutes of waiting time I stopped it. This one was appended to my transaction history, but not on WalletOfSatoshi's side. It's just a display bug our our side because we don't have a filter to eliminate unsuccessful payments. No preimage so no payment ...

[kingonly]

@simongltr thank you 🙏 can you please try a new node as well?

[simongltr]

I created a fresh node 02bf04e333701fa02859af1eeccecc3052a9a51b55821c5487560b30467ca59336 but I got this error during the sync:

Generic: Generic: Generic: status: Unknown, message: "transport error", details: [1, metadata: MetadataMap { headers: t}

After restarting the app the node synced successfully but I can't generate any invoice or do any action. It seems like important setup steps were skipped because of the error. For instance I have no LSP attributed to this node.

On monday I will try again with the SDK but on desktop in CLI mode, at least to try on a different environment and try to collect better data. Thank you for helping us !

[simongltr]

Tried again with another fresh node: 023b9a720f6d21c97416586f428d5b360e895350ecb5df7980ac799aa577c85eb1
-> Synced with no error, tried to generate the first invoice of 10,000 but no response, and now the node is not syncing anymore. When I restart the app it syncs, I try again the generate the invoice and no response, can't sync, ... I see no LSP attributed to me

Tried again with a fresh node after reinstalling the app: 03ab83d878e884c7533e359dd1754ffea42fe8ceb0b99baeeef7855ed61d284a31
-> same, it syncs, no LSP, can't generate an invoice

[dangeross]

@simongltr any log files from the Breez SDK would also be very helpful. With the CLI there should be a log file created in the data directory, when using the SDK the method set_log_stream can be used to capture the SDK log stream.

[simongltr]

@simongltr any log files from the Breez SDK would also be very helpful. With the CLI there should be a log file created in the data directory, when using the SDK the method set_log_stream can be used to capture the SDK log stream.

I tried with the CLI, I get the same error:

Error: Payment timeout: status: Unknown, message: "Error calling method Pay: RpcError { code: Some(210), message: \"Ran out of routes to try after 2 attempts: see `paystatus`\", data: None }", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Mon, 29 Apr 2024 14:13:17 GMT", "content-length": "0"} }

This time I can give you the logs:
sdk.log

Here is the history file:

#V2
help
sync
node_info
service_health_check
set_api_key
set_api_key <API_KEY>
service_health_check
node
node_info
connect
node_info
lsp_info
send_payment
sync
send_payment lnbc200n1pnzl2kypp5zzkh2vxss5h77qjm9sq2hsau40htv30nla3hfv3r20auuzdmja6sdqqcqzzsxqyz5vqsp5t6we92quxgq8m3z6vuwe2xmv52394fawe75n4n2cd6cwvmcrltfq9qyyssqx88uphmt06dshldftj7xggmjvuyduezkx4zycun3me9tsp5eu5s84ce3sxs4ppnshkh6kruyzzvvq3q6qsy4sfaq9vxszfcl2p6rnasplxn02u

[kingonly]

@dangeross please check the logs and you can also work with Simon to use the same seed yourself. This seems like a gossip issue, so if you come tho same conclusion, let's involve the GL team ASAP.

[simongltr]

Yeah I can give you my seed @dangeross, just give me a PGP key or keybase account I can encrypt to

[dangeross]

Thanks @simongltr, I'll look through the logs and let you know if the seed would be helpful too

[simongltr]

Just to mention it, it seems like when I was testing the CLI this morning a channel was closed.

Also @damiandizeo told me that he sees this error a lot with my seed:

ios BreezSDK Event backupStarted
ios BreezSDK Event backupFailed

[dangeross]

For node id 032f6993f42c8a01405e33bba6080237c07a7e67f62f1d7da407ef7e390cafc606

I think the reason is there are 2 pre-anchor channels, from what I can see it 2 were opened around Sept 2023 and 1 anchor channel Oct 2023. There are errors concerning the fee rate:

WARN lightning_signer::policy:78] policy failed: policy-commitment-fee-range validate_fee: feerate above maximum: 33803 > 25000
WARN lightning_signer::channel:1993] VALIDATION FAILED: Policy("validate_holder_commitment_tx: validate_commitment_tx: validate_fee: feerate above maximum: 33803 > 25000")

Channel 809595x2857x0 is pending close (pre-anchor), with 2 still open

The backup seems to be workings from this CLI, not seeing any failures

Do you have logs from your fresh nodes @simongltr ?

[simongltr]

[EDITED] This problem is solved

Do you have logs from your fresh nodes @simongltr ?

The commands I typed:

cargo run -- --data_dir /home/ubuntu/breez_data/E

sdk> connect -c /home/ubuntu/breez_data/client.crt -k /home/ubuntu/breez_data/client-key.pem
Node was connected successfully
sdk> sync
Sync finished successfully
sdk> set_api_key <KEY>
API key was set
sdk> lsp_info
Error: Generic: No LSP ID found
sdk> list_lsps
[]
sdk> receive_payment 10000000 debug
Error: Generic: No LSP ID found

The logs:
sdk.log

I tried multiple times and the issue was always a missing LDP IP

[damiandizeo]

[UPDATE]

2 mobile devices with 2 fresh nodes:

1 node created using breez cli and imported into mobile app
1 node created directly in the mobile app

Both apps with funds ~5.000 sats

Test:
Pay 50 sats to an invoice of WoS

Result:
From the node with the seed created with breez cli and imported in mobile, the payment worked fine
From the node created in mobile fails with ran out of routes error

Attach logs from mobile app with the issue:
logs.txt

[simongltr]

2 days ago @damiandizeo described (last message) a situation where a fresh node can be funded but can't pay because of a ran out of routeserror.

This node is now able to send payments, its node ID is 033070dc551a25d2bf6bf01cbf5f031cf2482c2a8c8d9be05ce5617ccc63e618ca.

This morning (around 2:44pm GMT) we successfully funded a new node with ID 028faee7d37af3aa91b74fee26507518e8293d4988a64cfd3ec8f1bae8790f8fbc by paying this invoice:

lnbc27500n1pnr8f57dqqpp52d49v0usjnaajgy7h9xv692s5l5pvjc0k3fqk53h0p6dw5y2wnjsxqrrsssp5uss29md0je8yp50ts5htm3k0n853gfty239fd37d7pqu457d3y8s9qrsgqcqzysrzjqv3jgvsjswvd3je3ueujrl756n8mzz6zm3e99xj9mwn0zpddlwpagqqqqyqqqqqqqqqqqqlgqqqqqqgqjqpze6a05fqy5ngw6nc9dalaxze6sh8ru9mzg4uum6zwdg9dn8dtrhjpp32l3jdnz663y5umedg03jsyusu0q7s9kdx4cfhq456xytuvspvx7hz5

Around 2:46pm GMT this node was not able to pay.

Around 5:22pm GMT we successfully sent a payment.

[cdecker]

I tried fetching the logs, and all I could find was a successful payment today. This is most likely still an old report that predates the inclusion of ElementsProject/lightning#7190 :-)

If the problem does not persist, shall we close this?

[simongltr]

Thank you for looking at our problem. I'm closing this issue as the situation has improved significantly compared to a few weeks ago. We are currently implementing new features in our app, so I will need to conduct thorough testing again in about a week to confirm that the issue has been resolved. If the problem persists, I will reopen the issue and provide as much data as possible.

[cdecker]

The feerange issue is tracked here: #538