Add data directory option to storage settings

[GBKS]

This is a follow-up issue to this comment in the PR for adding data directory selection UI during onboarding.

Idea is that we also need to show the directory later during main application use.

Here's a quick mock up of what that could look like.

I'd like to clarify the user actions here. There's opening that directory in the native finder, and changing the directory. I am curious what's involved in changing the directory. Is already downloaded data moved over? And do we need to inform users about what is happening? If this is a more involved process, then we may need a dedicated page to handle it. Otherwise it could be just a simple interaction.

[D33r-Gee]

@GBKS thanks for starting this issue... It is addressed in here bitcoin-core/gui-qml#397

I like having the address under the header will look into how to pull that off :)

[pablomartin4btc]

I'd like to clarify the user actions here. There's opening that directory in the native finder, and changing the directory. I am curious what's involved in changing the directory. Is already downloaded data moved over? And do we need to inform users about what is happening? If this is a more involved process, then we may need a dedicated page to handle it. Otherwise it could be just a simple interaction.

If we want to allow users to change the datadir yeah, we would need to move the data over if the sync process already started and yeah we'll need to inform the user about all this process is about (perhap adding a new page as you said).

Also, when you start the app with a specific datadir from the command line (e.g.: bitcoin-qt datadir="/some/datadir"), we'd need to inform it there on the same "Data directory" field, maybe we could indicate that this was passed from the command line as well, which is not persisted (bypassing the datadir that has been configured during onboarding) and only "lasts" until the app shuts down, calling the bitcoin-qt again without passing -datadir will use the datadir saved in the settings.

[GBKS]

Is it a good idea to notify people explicitly about what parameters they passed in and what the effect will be?

Here's a quick mock-up, which could probably be extended further.

Temporarily changing the data directory just for the lifetime of a single run sounds very messy. Would it move data to that directory during the run, and then the application would not find any data anymore during the next run and start from scratch?

[pablomartin4btc]

Is it a good idea to notify people explicitly about what parameters they passed in and what the effect will be?

I'd like that, it's something currently QT doesn't have, but there is a request about the same on that repo.

Here's a quick mock-up, which could probably be extended further.

I'd also indicate in the "permanent" "Storage settings" that the datadir was passed, leaving the persisted/ configured/ onboarded datadir visible but with a mark/ link to this temporary settings section (I'm thinking loud about it, it makes sense to me, but let's see if we follow this path it makes it clearer for the user or not).

Temporarily changing the data directory just for the lifetime of a single run sounds very messy.

It's something that's used in both bitcoind and bitcoin-qt, sometimes a user would like to verify some stuff without changing current env setup (e.g. release testing).

Would it move data to that directory during the run, and then the application would not find any data anymore during the next run and start from scratch?

If I follow your question correctly, yes, using -datadir or not, if there are already blocks sync-ed on the directory (or default datadir) it would continue from there, if there's nothing, it would start from scratch. If you remove data during the run, depending what, you would make the app to crash and making the data to be in a inconsistent state.

[pablomartin4btc]

Would it move data to that directory during the run, and then the application would not find any data anymore during the next run and start from scratch?

Also, adding a useful reference of the documentation about the files stored in the data directory.

[GBKS]

Thanks for those details. It sounds like we have two different user needs here.

1. A casual user just wants blocks to be stored in a different place. Maybe their hard drive got full and they bought an external drive they want to start using. So they want to permanently move data to this new place.
2. A developer needs to regularly switch between environments. An environment might be using testnet and storing blocks in the blocks_testnet folder, or using mainnet and using the blocks_mainnet folder. They manage multiple data sets and want control.

A direction here could be that the GUI is optimized for the first use case, and we rely on devs using the command line.

So if the user changes the directory in the GUI, the existing files are moved. If the directory is passed in from the command line, no files are moved. Does that make any sense? I fell like I'm overthinking or forgetting something.

I'll try to mock something up about the temporary overrides.

[pablomartin4btc]

Thanks for those details. It sounds like we have two different user needs here.

1. A casual user just wants blocks to be stored in a different place. Maybe their hard drive got full and they bought an external drive they want to start using. So they want to permanently move data to this new place.

If a user wants to move the data from where it is to another folder or drive, at the moment, it has to do a manual process, I'm not aware of such tool or feature.

2. A developer needs to regularly switch between environments. An environment might be using `testnet` and storing blocks in the `blocks_testnet` folder, or using `mainnet` and using the `blocks_mainnet` folder. They manage multiple data sets and want control.

Yes, the different chains have their own subdir within the datadir (default or custom). Please check in the datadir location doc, if there's something unclear please let me know so we can update it.

A direction here could be that the GUI is optimized for the first use case, and we rely on devs using the command line.

Not sure if it's something other bitcoin core devs want (to support?), maybe it's worth it to ask on the IRC channel #bitcoin-core-gui.

So if the user changes the directory in the GUI, the existing files are moved.

If we allow them to do it, again, not sure if that's what we want. (If the following is confusing, please ignore it, sorry for the unnecessary tech details) This path it's also specify in the config file Bitcoin-Qt.conf (in current QT you can see it in its contents something like strDataDir=/home/pablo/.bitcoin). (For reference, PR bitcoin-core/gui-qml#398 it's for us to use a different file from QT).

If the directory is passed in from the command line, no files are moved.

If a user passes -datadir="..." from the command line, the above config will be bypassed so the block files and all datadir files will be different.

Does that make any sense? I fell like I'm overthinking or forgetting something.

Yes, better to clarify everything in advance.

[GBKS]

Thanks for digging into this issue, I think I still have some wrong ideas about how the logic currently works. I'll try to get a clearer understanding.

If we want to allow users to change the datadir yeah, we would need to move the data over if the sync process already started and yeah we'll need to inform the user about all this process is about (perhap adding a new page as you said).

Based on this, I thought we could/would move data for the user, but from your other statement it sounds like that is not the case (or not possible). Can you please clarify?

I think what I am trying to resolve is how the user flow of changing the directory later would work. Let's say I synced a good amount of data, but now want to move, and I only use the app (not config or command line). If I start the application, go to settings, and change the directory, what happens? Do I need to restart? Do I need to manually move the data, or can the application do it for me? What do you think is best based on how the code/logic works at the moment?

[GBKS]

Small note via Mo to also design error states for the directory picker here and in the first-use.

[pablomartin4btc]

Small note via Mo to also design error states for the directory picker here and in the first-use.

Mo's comment.

[pablomartin4btc]

Based on this, I thought we could/would move data for the user, but from your other statement it sounds like that is not the case (or not possible). Can you please clarify?

It is possible, unless there's something I'm not aware of, just saying we need to warn the user and make the user aware of the steps, and from the dev side be prepare to handle issues related with the procedure.

I think what I am trying to resolve is how the user flow of changing the directory later would work. Let's say I synced a good amount of data, but now want to move, and I only use the app (not config or command line).
If I start the application, go to settings, and change the directory, what happens? Do I need to restart?

I'd need to test this once the wiring (settings saved/ persisted in the config file) is done, but as in current QT (not QML) I imagine, if you change the settings file manually or thru the app in order to the changes being taken into account you'd need to restart, but if you restart and there are no files there, it will start the sync from scratch.

(e.g. current QT when user changes a setting, the app advises to restart)

> Do I need to manually move the data, or can the application do it for me?

Currently, you'd need to move the data manually but you need to make sure the app is not running otherwise you can brake it. Currently the the app doesn't do it for you, I think it's doable to add the feature, not sure if it's convenient at this stage.

What do you think is best based on how the code/logic works at the moment?

Until we define if we add the migration/ moving datadir feature (scope, approach), better not to let the user change the datadir after onboarding (current QT doesn't allow it).

[GBKS]

Thanks. Let me try to summarize an approach.

V1 (of this feature)

• Users can set the directory during onboarding
• Settings shows the directory and points user to manually editing bitcoin.conf for further changes (close app, move files, edit bitcoin.conf, start app)

V2

• Users can also change the directory in settings
• App automatically stops the node, moves files over, and restarts the node (no restart required)

Does that sound right?

[pablomartin4btc]

V1 (of this feature)

* Users can set the directory during onboarding

* Settings shows the directory and points user to manually editing `bitcoin.conf` for further changes (close app, move files, edit bitcoin.conf, start app)

Yeah, for V1 we should only display the datadir selected during onboarding on the settings making it not editable/ read-only. Not sure if there's an official bitcoin core documentation already on how to migrate/ move datadir with all the steps if the user needs to, but if it is (or once it is) I'd add a link to that doc next to the displayed datadir.

V2

* Users can also change the directory in settings

* App automatically stops the node, moves files over, and restarts the node (no restart required)

Yes, I'd think so.

Does that sound right?

Yes