[bug]: SendPayment fails with EOF error after successful payment

[ZZiigguurraatt]

I am trying to make a payment using the SendPayment (https://lightning.engineering/api-docs/api/taproot-assets/taproot-asset-channels/send-payment/) gRPC in python where alice pays bob over a direct taproot asset channel (no routing or exchanges to sats). The payment succeeds, but after that I get an EOF error from the gRPC. I'm getting this error when using litd with commit d82cb9d275ad706207eebaa86ad68edb5ef055d3 as well as tag v0.13.993-exp . Note: this is not a problem with my python gRPC because no other calls have this error and LND's SendPaymentV2 (https://lightning.engineering/api-docs/api/lnd/router/send-payment-v2/) works fine.

payment_result {
  payment_hash: "62afeac0474399f2ca8687c2deab04b5b4aeaf7f3514ab9896ec4cb39f18e8fb"
  value: 500
  creation_date: 1732645337
  payment_preimage: "765dde18d7518b67ebd7f3f2f25ae21bb12ef60bcd3438a62dd909bd07486f50"
  value_sat: 500
  value_msat: 500000
  payment_request: "lnbcrt5u1pn5vywepp5v2h74sz8gwvl9j5xslpda2cykk62atmlx522hxyka3xt88ccarasdqqcqzzsxqyz5vqrzjqwedrfsykk87jmxmy4hf93usnc3ywawqst7uclt3hz602z2965ma0aa8ecy32drt8qqqqqlgqqqqqqgq2qsp5du78r6hcay0ayklukp40d2mjyagmkxchrr40ce4vw3zktrz2ch5q9qxpqysgqx3zhe5jvssletg9a596zw06q5p687lwdzhgnddkjtsucyw8zrzdrz2ptgatj6ds0u6jqya35a4gmu65zq60rwkrzrvpp94w33ajhqycqstx4qd"
  status: SUCCEEDED
  creation_time_ns: 1732645337576337406
  htlcs {
    status: SUCCEEDED
    route {
      total_time_lock: 14923
      total_amt: 500
      hops {
        chan_id: 16251881370222592
        chan_capacity: 100000
        amt_to_forward: 500
        expiry: 14923
        amt_to_forward_msat: 500000
        pub_key: "03295e89e0eb6f166f9d285b4d00409f83c3c266a5e6ceb7e20ad26fb719d65aaa"
        tlv_payload: true
        mpp_record {
          total_amt_msat: 500000
          payment_addr: "o<q\352\370\351\037\322[\374\260j\366\253r\'Q\273\033\027\030\352\374f\254tEe\214J\305\350"
        }
      }
      total_amt_msat: 500000
      first_hop_amount_msat: 354000
      custom_channel_data: "{\"balances\":[{\"asset_id\":\"7fff4ab4747999660099d73f3ecc0c21ee7087fb33b56f803d03803c91ea27a7\",\"amount\":5}],\"rfq_id\":\"797acedae5053c8085c003b6a6f037543d7ea02a29d9faacf79177a066cd2170\"}"
    }
    attempt_time_ns: 1732645337595293695
    resolve_time_ns: 1732645337738987028
    preimage: "v]\336\030\327Q\213g\353\327\363\362\362Z\342\033\261.\366\013\31548\246-\331\t\275\007HoP"
    attempt_id: 3
  }
  payment_index: 17
  first_hop_custom_records {
    key: 65536
    value: "\000"
  }
  first_hop_custom_records {
    key: 65538
    value: "yz\316\332\345\005<\200\205\300\003\266\246\3607T=~\240*)\331\372\254\367\221w\240f\315!p"
  }
}

Traceback (most recent call last):
  File "/mini_META/scripts/run_tests", line 15, in <module>
    SendTaprootAssetReceiveTaprootAsset('alice','bob','alice','bob',5,TheAssetID)          # 1 hop
  File "/mini_META/scripts/mini_META_lib.py", line 93, in SendTaprootAssetReceiveTaprootAsset
    for response in ThePaymentResponse:
  File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
  File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "EOF"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.99.0.4:8443 {created_time:"2024-11-26T13:22:17.750675996-05:00", grpc_status:2, grpc_message:"EOF"}"
>

[Roasbeef]

Can you also give us some example code here? It could be the way you're reading the response in Python.

EOF for gRPC generally just means that the other side has hung up.

At a glance, it looks like you're iterating over a generator of the response, but this isn't a streaming call, so there's only ever one response. So perhaps on the second iteration it runs into this EOF issue?

[guggero]

I think I saw the same thing in litd today, so it's probably that we close the stream improperly after we're done (and not a Python specific issue). So you can ignore it in the sense that everything should work normally still. But we should fix it anyway.

[ZZiigguurraatt]

Here are some functions used to reproduce the issue, highest level to lowest level

• https://github.com/ZZiigguurraatt/mini_META/blob/7a8ca99e8652ee8f83a3529de404f2084cdff755/scripts/run_tests#L14-L16
• https://github.com/ZZiigguurraatt/mini_META/blob/7a8ca99e8652ee8f83a3529de404f2084cdff755/scripts/mini_META_lib.py#L91
• https://github.com/ZZiigguurraatt/lnd-grpc-client/blob/18f239f200e079dbc5e2f4d955e191490b5b0b38/lndgrpc/tapchannel.py#L28-L32

[ZZiigguurraatt]

I think I saw the same thing in litd today, so it's probably that we close the stream improperly after we're done (and not a Python specific issue). So you can ignore it in the sense that everything should work normally still. But we should fix it anyway.

I can catch the exception in python, but the way gRPC errors come through in python, they are a bit less clean than other exceptions, so it's harder to distinguish from a real error.

[ZZiigguurraatt]

Looks like we have #1217 as a possible fix.