Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

It gives me a "wrong password" after restarting Umbrel. #372

Closed
diegokolling opened this issue May 24, 2022 · 25 comments
Closed

It gives me a "wrong password" after restarting Umbrel. #372

diegokolling opened this issue May 24, 2022 · 25 comments
Labels
bug Something isn't working

Comments

@diegokolling
Copy link

After submitting an ask order this last saturday, of around 7MM for 2 weeks, 17% APR, I started seeing error messages on Interface of Lightning Terminal app on Umbrel, updated to the last version (LND 14.2 and Umbrel 0.18).

It was saying that it couldnt fetch a lot of things: My nodes Tier, my orders, and so on...

So I restarted Umbrel and it got worse: Lightning Terminal is the only app not loading properly.

It first give me messages of "error_connection_refused" before getting connected. But when it does, it does not accept my password anymore, the same as always...

All other apps working properly...

So I dont have a GUI interface for my Pool funds anymore.

  1. Whats happening to Lightning Terminal app? Why it started giving me error messages and then apparently broke even more, after restarting Umbrel?

  2. In the meanwhile, how do I check through command line my funds and/or have my funds back and Pool account closed? Can you guys be specific, since Im not that used to command line?

A little bit of more context:

I first opened my Pool account 10 days ago. I submitted various orders. For a week it was all working properly. I could access Terminal and check on them.

Since none of them got accepted, I withdraw the orders and remade a one, bigger ask proposal with all my funds (last saturday). That is when things blew up.

Thank you guys for your attention.
@diegokolling diegokolling added the bug Something isn't working label May 24, 2022
@jamaljsr
Copy link
Member

  1. Whats happening to Lightning Terminal app? Why it started giving me error messages and then apparently broke even more, after restarting Umbrel?

This is difficult to answer without more information.

  1. In the meanwhile, how do I check through command line my funds and/or have my funds back and Pool account closed? Can you guys be specific, since Im not that used to command line?

Here are steps you can follow to close your account and return the funds to your LND wallet.

  1. SSH into the Umbrel node. The password is the same one you use to login to the Umbrel website 
    ssh [email protected]

    This command works on Mac & Linux. If you are using a Windows computer, you'll need to use an app like PuTTY to SSH into the Umbrel node.

  2. Once logged in, your terminal prompt should say umbrel@umbrel:~ $
  3. Run bash inside the lightning-terminal docker container 
    docker exec -it lightning-terminal_web_1 bash
  4. When inside the container, your terminal prompt should say bash-5.1 $
  5. Run the pool CLI command to list your accounts 
    pool --rpcserver=localhost:8443 --tlscertpath=~/.lit/tls.cert --macaroonpath=~/.pool/mainnet/pool.macaroon accounts list
  6. It will output your account information, such as 
    {
 "accounts": [
 	{
 		"trader_key": "026b7447973b79e8eaf79258c104b716a8107eedeeb25e484616daf001cf24bba3",
 		"outpoint": "5b09744a8a4ab8e31eabd1e14ce7de9c434dedf13ab9fe6b27bd2ab807276f04:0",
 		"value": 5381514,
 		"available_balance": 5381514,
 		"expiration_height": 1906780,
 		"state": "EXPIRED",
 		"latest_txid": "5b09744a8a4ab8e31eabd1e14ce7de9c434dedf13ab9fe6b27bd2ab807276f04"
 	}
 ]
}
  7. The trader_key value is what you'll need for the next command.
  8. Run the pool CLI command to close this account 
    pool --rpcserver=localhost:8443 --tlscertpath=~/.lit/tls.cert --macaroonpath=~/.pool/mainnet/pool.macaroon accounts close <trader_key> <sat_per_vbyte>
    Replace <trader_key> with the value from step 6 and replace the <sat_per_vbyte> with the fee rate you'd like to use.

To quit the SSH session, just enter the exit command twice.

@diegokolling
Copy link
Author

diegokolling commented May 24, 2022 via email

@diegokolling
Copy link
Author

diegokolling commented May 24, 2022 via email

@levmi
Copy link
Contributor

levmi commented May 25, 2022 •
edited
Loading

The line breaks got borked, re-writing this.

@jamaljsr
Copy link
Member

Here are the commands you should run on your umbrel to recover your account:

# connect to the Umbrel node
ssh [email protected]

# stop the Terminal app
cd ~/umbrel
./scripts/app stop lightning-terminal

# rename the pool db file
cd ~/umbrel/app-data/lightning-terminal/data/.pool/mainnet
mv pool.db pool.db.bak

# start the Terminal app
cd ~/umbrel
./scripts/app start lightning-terminal

# run bash in the LiT docker container
docker exec -it lightning-terminal_web_1 bash

# recover the account
pool --rpcserver=localhost:8443 --tlscertpath=~/.lit/tls.cert --macaroonpath=~/.pool/mainnet/pool.macaroon accounts recover

@diegokolling
Copy link
Author

Aand, I'm back! Thank you, guys! I have my Pool account with my full balance. So, to start ask orders again, what can I do to prevent what happened? What was the post mortem? Did our case triggered any update to Pool or LIghtning Terminal itself? Would it be recommended to use the web version instead? Thank you one more time!

@jamaljsr
Copy link
Member

jamaljsr commented May 26, 2022 •
edited
Loading

Awesome! Glad you are back up and running.

The problem appears to have been a connectivity issue during the process of creating a new batch. This was likely tor related. Due to this issue, your pool client was out of sync with your lnd node. Pool thought lnd had a pending channel open, but lnd didn't know anything about this channel. So when pool restarted, it attempted to cleanup this pending channel, but lnd threw an error because the pending channel didn't exist. This is what caused your restart loop.

It is unlikely that you've have this problem again, so you can proceed to create orders normally now. We're tracking the issue in the pool repo to perform proper handling of this specific edge case if it were to happen to someone again in the future.

Thanks for providing all of that info. It was very helpful in identifying the cause of this issue.

@diegokolling
Copy link
Author

You guys are amazing. Thank you one more time!

@diegokolling diegokolling mentioned this issue Jun 14, 2022
Closed
@guggero guggero mentioned this issue Jul 4, 2022
Closed
@diegokolling
Copy link
Author

Hey guys! So, here we go again, but now with a twist: I didnt have a unexpected restart or an energy blip. I suddenly lost access to my Lightning Terminal, Pool account and so on. Before going through the recovery process (as mentioned above) of my Pool account, where I happen to loose all my data of Pool orders, I would like to know from you:

Is it possible to recover my data afterwards? Like: maybe save my data -> recover account -> import data? How would that be?

  1. Does updating to LND 15 and Umbrel 5 improve my chances of that not happening again?

Thank you guys one more time 🙏

@jamaljsr @levmi

@diegokolling
Copy link
Author

umbrel-1657555338550-debug.log

As usual, my Umbrel logs.

@diegokolling
Copy link
Author

@levmi @jamaljsr Im going to have to run through this process once again. Can I recover my Pool data afterwards?

@diegokolling
Copy link
Author

Here is my logs
umbrel-1657828433534-debug.log

@guggero
Copy link
Member

guggero commented Jul 15, 2022

There is no export/import yet. But if you still have the data, you can create a copy of the whole Pool data folder (not sure where that is located on an Umbrel, but it's the parent directory that contains the pool.db and pool.macaroon files, perhaps a search will find them).
If you then restore Umbrel and copy back that whole folder (most important are pool.db and lsat.token), you should get your account and all orders back (and you do not have to run account recovery!)

@diegokolling
Copy link
Author

Oh, very nice @guggero ! Thank you. Do you have a step by step guide to do that on Umbrel 5?

My Umbrel is running fine.

Doing that, will I be able to use Pool fully normally again? There are 12MM in Pool orders that I can’t see right now (+dozens of matched orders).

@guggero
Copy link
Member

guggero commented Jul 15, 2022

Do you have a step by step guide to do that on Umbrel 5?

No, I'm not familiar with Umbrel at all.

If you already lost your Pool data because of an Umbrel problem, then creating a backup of the empty data won't change anything. So when you say "that I can't see right now", is that just the orders but the accounts are there?
What I suggested above only works if you have intact data, then after restoring Umbrel from seed, you should get everything back as before.
But if Pool is already showing incomplete data, then you're probably out of luck.

Account recovery will always bring back your accounts (meaning your balance), but it won't restore your orders (neither the active nor the executed ones with your history).

@diegokolling
Copy link
Author

No, no, it’s not an Umbrel problem, it’s a Pool problem and has been since the beginning. It’s the third time that I have to recover my Pool account, not any other app account or Umbrel account.

Umbrel recovery from seed was never an option, my Umbrel and Lightning node are working just fine.

@jamaljsr has a written a step by step guide, as you can see above, to recover my Pool account on Umbrel 4.18.

But since @levmi asked me to upgrade to Umbrel 5, it doesn’t work anymore.

So all I want is Jamal steps re-written for the new Umbrel ASAP, since I’m about a Week without access to my 12MM open orders on Pool and it’s starting to get really frustrating.

Sorry to be as frank and direct as I can, but I couldn’t find another way to put it.

@guggero
Copy link
Member

guggero commented Jul 15, 2022

I didnt have a unexpected restart or an energy blip. I suddenly lost access to my Lightning Terminal, Pool account and so on. Before going through the recovery process (as mentioned above) of my Pool account, where I happen to loose all my data of Pool orders, I would like to know from you:

Okay, I guess we've had a classic misunderstanding. Because you wrote you wanted to recover again, I didn't take a close look at the log file, so I didn't see what the cause was and assumed something went wrong with the Umbrel upgrade, they've been known to just delete data on upgrade. So I apologize for not reading more carefully. I look at dozens of issues every day, so things might slip by sometimes.

It looks like an attempted batch execution got stuck. This could be solved with a simple command, but not in the Umbrel environment (the setup is just too complicated). So you have three options:

  1. Stop the lightning-terminal app, then send me the pool.db over Slack (same username there, but I'll be traveling over the next few days and won't be responding immediately) for me to fix the state.
  2. Run through the recovery process again, which means losing the orders and lease history. See the updated script below.
  3. Wait for us to fix the bug that caused the error in the first place, then upgrade to the new version of LiT once we do.

I'm sorry this is such an inconvenience, but both LiT and Pool are still in alpha, so there are bugs to iron out.

In case you want to go with option 2, here's the assumed way to run things on Umbrel 5 (again, I have 0 experience with that system, this is purely guessing from what I found in their documentation):

# connect to the Umbrel node
ssh [email protected]

# stop the Terminal app
~/umbrel/scripts/app compose lightning-terminal stop

# rename the pool db file
cd ~/umbrel/app-data/lightning-terminal/data/.pool/mainnet
mv pool.db pool.db.bak

# start the Terminal app
~/umbrel/scripts/app compose lightning-terminal start

# run bash in the LiT docker container
~/umbrel/scripts/app compose lightning-terminal exec lightning-terminal bash

# recover the account
pool --rpcserver=localhost:8443 --tlscertpath=~/.lit/tls.cert --macaroonpath=~/.pool/mainnet/pool.macaroon accounts recover

@diegokolling
Copy link
Author

Oh, don't worry @guggero you've been doing really great, I'm really glad you are working hard on it and I appreciate that.

I've talked with Briscoe from Umbrel on the mean time and, looking at both set of commands, we've come to believe these one below are the better set (its a mix of Jamals and yours):

See what you think:

connect to the Umbrel node

ssh [email protected]

stop the Terminal app

cd ~/umbrel
./scripts/app stop lightning-terminal

rename the pool db file

cd ~/umbrel/app-data/lightning-terminal/data/.pool/mainnet
mv pool.db pool.db.bak

start the Terminal app

cd ~/umbrel
./scripts/app start lightning-terminal

run bash in the LiT docker container

docker exec -it lightning-terminal_web_1 bash

recover the account

pool --rpcserver=localhost:8443 --tlscertpath=/.lit/tls.cert --macaroonpath=/.pool/mainnet/pool.macaroon accounts recover

@guggero guggero mentioned this issue Jul 15, 2022
Merged
1 task
@diegokolling
Copy link
Author

@guggero , good news: I have my account back after running the lines above. The point is: I firmly believe it was supposed to have 12MM, when I just have 11,783.

Where can I find what happened in the meantime? Channels leased, ammount earned and so on? From what I can tell, Ive leased around 13 times earning around 100k sats.

The data is lost on the interface, is it to be found anywhere else? To kind of audit it?

@guggero
Copy link
Member

guggero commented Jul 15, 2022

Your account was involved in a batch about 15 days ago where two channels were sold, one for 0.7MM and one for 0.3MM, which explains the 1MM difference. You should see those two channels in your channel list.

The audit information is in the database you just removed/renamed. It also exists on the server side, but there currently is no option to "download" it again. Feel free to open a feature request issue in the Pool repo for that.

@diegokolling
Copy link
Author

Hmmm nice, I will. Meanwhile, is there a way for some of you guys to do it manually, if you don’t mind, please? I can wait, no problem.

It will be important for me, tho, to keep track of my nodes return and Pool is an important part of it.

It’s the main part, actually! Since it accounts for more than 80% of my profits.

it’s been somewhere between 11-13 leasings, so no big deal I think. You’ve already spotted two. (Although I don’t know for how much they were opened for example).

@levmi
Copy link
Contributor

levmi commented Jul 15, 2022

Hey Diego, currently, there isn't an easy way for us to grab that information for you. As Oli mentioned above, for the time being, the account recovery process does result in the database being removed/renamed. As mentioned, the option of pursuing the account recovery path means that the order and lease history were removed from the client.

While I understand this might be frustrating, please take into account the alpha nature of both Pool and litd. That said, I'm really glad that Pool is driving a large portion of your revenue. We expect it to continue to grow. And, thank you for opening issues to improve this experience, the team is already discussing some of your ideas. As always, we will continue to work hard to improve the experience for all our users and we appreciate your feedback.

@diegokolling
Copy link
Author

Oh, great to hear from you Michael! Yah, Alpha software is kinda… well, it’s not that good sometimes.

But okay, than! Maybe someday I have everything back :)

I liked the idea of posting suggestions and really enjoyed the early feedback. It’s great to see!

Keep moving forward!

This was referenced Sep 11, 2022
Closed
Closed
@jamaljsr jamaljsr mentioned this issue Sep 14, 2022
Closed
@jamaljsr jamaljsr mentioned this issue Oct 24, 2022
Closed
@em1xAdmin
Copy link

Hi,
How can I perform an account recovery on a node running on Start9 Embassy?
I had an account setup with an ask offer about 2 weeks ago. Hadn't looked at it for a while and when I did I got the welcome banner with no evidence of my account, Restore from my latest backup did not help. There was no new updates installed.

@guggero
Copy link
Member

guggero commented Apr 11, 2023

You should be able to follow this guide: https://docs.lightning.engineering/lightning-network-tools/lightning-terminal/troubleshooting#pool-account-recovery
Maybe you also need to move/rename the pool.db as described in the step above.

Some arguments/flags might change on Start9 Embassy, unfortunately I'm not familiar with software and can't give you more info on that. Their support might be able to help you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@guggero @jamaljsr @diegokolling @levmi @em1xAdmin