Automatic migration of code from message_data

[glatterf42]

Summary

For automatic migration of code from message_data, we provide the scripts in util. Most recently, @GamzeUnlu95 tried running these scripts via Git Bash on her Windows system, revealing some issues with her system, but also with these scripts. The main issue seems to be that when running the scripts manually in the shell on Windows, the settings.sh might not set the variables persistently, so the other scripts don't recognize$SRC and the rest.

Description of issues

With the system

(Git) Bash

Git Bash did, for some unknown reason, not recognize commands like python or python3 or git filter-repo. We did not take the time to figure out the exact reasons for that, but instead found an option to install a Linux shell (bash and fish) on Windows. Windows 10 (since version 2004) and 11 provide the Windows Subsystem for Linux, which can be installed from the command prompt run as administrator. As detailed in the article, we used wsl -l -o to see a list of available options and wsl --install -d Ubuntu-22.04 to install the latest available version of Ubuntu. Now, bash is available in the command prompt and opens in a home directory which seems to be well-located if called from the drive of interest (which is C:\ rather than H:\, where the command prompt opens per default; use C: to change the drive). This allows python3 to start a python shell.
Please note that we had to set up another ssh key in this Linux subsystem to access GitHub in the usual way.

Fish

Since I prefer fish over bash, we then followed this guide to install that as well. At the moment, this unfortunately leaves us with the rather cumbersome solution of

1. Opening the command prompt (searching for cmd on the search bar)
2. Starting bash
3. Starting fish

to get to the shell of our choice. No doubt this can be made more convenient, we just didn't take the time for it (yet).

git filter-repo

This still left git-filter-repo unrecognized, since it was already installed before, but not recognized anymore. So we installed it again, for which we needed to install pip first, using sudo apt-get install python3-pip. pip install git-filter-repo is then readily available.

With the scripts

The scripts were still not running after this. The main issue seems to be interaction with other scripts in the first few lines. util/reset.sh contains line 5 as . common.sh, but there is not file called common.sh in util/. Similarly, both util/1-prepare.sh and util/2-migrate.sh call in line 5 or 6 . settings.sh, which does exist, but wasn't recognized by our new setup.
As an interesting side note, this recognition was not a problem when double-clicking the scripts from the file explorer, which opened an instance of git bash and ran them, while we were unable to run them manually from within git bash.
We were able to get the files recognized by changing line 5 or 6 to ./settings.sh. However, this merely resulted in statements of the sort +SRC=message_data in the terminal, but when the further script commands tried to call $SRC, the variable would be empty. We tried running the scripts simply via ./2-migrate.sh. Again, in the interest of time we did not research the most broadly viable way to set these variables independent of the surrounding environment, but specified them manually. In fish, this is done via set SRC message_data.
Afterwards, we continued copying the commands from 2-migrate.sh (since 1-prepare.sh had succeeded in the automatic git bash call) to the shell and executed them manually without further issues.

[khaeru]

./common.sh and . common.sh are different from one another:

• ./common.sh means "Execute the file common.sh that appears in the current directory, .". The file could be a binary, or a Python, Perl, shell, etc. script. Since this happens to be a shell script, it executes in a sub-shell. This means that any variables created live within only the scope of that sub-shell, and are lost when it exits.
• . common.sh is the POSIX (not bash-specific; see here) way to include the lines from another file in the current script. Thus when those included lines define variables, those variables are in the same scope as the current script.

The latter is what is intended in these migration scripts. The name mismatch between "settings" and "common" is one of the errors I noticed doing #116.

[glatterf42]

Huh, thanks for the explanation. Now I don't understand why the shell said it couldn't find settings.sh. Maybe the script chose another directory as its working directory at the start of being executed, which indeed did not contain the file. To check this, I would need to try on your system again to study the error message, @GamzeUnlu95.