Vagrant up does not install dependencies properly.

[alcemirsantos]

I ranvagrant up after clone the repository as suggested in the README.md file and then proceeded, however after the process finished, dependencies were not present and I could not run the second step example analysis:

• Start the ID service with /vagrant/id_service/start_id_service.sh&

In the following, I added some prints of the vagrant up log

...

...

Am I missing something, or do I need to do some hacks to get it up and running?

[bockthom]

Hi @alcemirsantos, thanks for reporting this issue.
Unfortunately, the README.md as well as the Vagrantfile are outdated and have not been updated for 6 years now. In the meantime, we have fixed a lot of bugs and issues in the source code of Codeface in our Codeface-fork and also updated several dependencies to Ubuntu 22.04. However, we did not touch the Vagrantfile through all the years, so vagrant still uses some Ubuntu version between 12.04 and 16.04, but the source code of Codeface now requires Ubuntu 22.04. I guess this is the main reason for installation problems with npm as in your first screenshot and python problems in your second screenshot. Consequently, codeface is not installed and not availble in vagrant, leading to the errors in your third screenshot. However, there seem to be additional problems with the integration scripts, as "mysql: not found" in the third screenshot indicates.

To be honest, I don't exactly know what's going on in the vagrant scripts and also not what's happening in the testing and analysis scripts that are given in the README.md. For quite some time, we use Codeface only in our own, self-managed Ubuntu VMs (now Ubuntu 22.04) and installed the current version of Codeface there without any problems, calling the installation scripts directly. Also, instead of calling example and test scripts, we also call Codeface and the ID service in our own scripts directly. In this setting, everything works. (But the example scripts have never been updated, as this was previously just a fork of the original Codeface version. However, since the original fork stopped continuing last year, we started to fix several other parts in our fork, which is still ongoing work.)

Since a few weeks, we are busy with porting Codeface to python3, and we will update the README and installation scripts afterwards, to make Codeface fit for the future and perform all the upgrades and important changes that have been slept through during the last 5 years. However, this will still need a couple of weeks to be eventually finished, I guess, and we will update the README only once after the final portings have finished. I am also not sure if we will keep the Vagrant file (I personally have never used vagrant, so I am not familiar with it) or whether we will provide a Docker file instead, or just provide updated installation scripts for installing Codeface in a self-hosted virtual machine. Let's see...

After all these explanations, now back to your issue: I am very sorry for the troubles you currently encounter with vagrant and Codeface. For the moment, I can offer you two possible solutions:

• You can either setup your own (virtual or not virtual) machine with Ubuntu 22.04 and install Codeface in this machine on your own, calling the 7 installations scripts manually (in the same order as in the Vagrant file; however, you possibly have to replace libgdal20 in integration-scripts/install_codeface_R.sh by libgdal30). If you choose to do so, don't hesitate to ask me if unexpected errors occur during installations - maybe I already have a solution for them (and if not, I will definitively help you finding one).
• Or, if you can wait for a couple of days/weeks, we will update the installation scripts anyway as we are currently fixing them.

I hope that this does not restrain you from using Codeface. We are heavily working on all the upgrades that have bee neglected for a couple of years and are happy to provide you with an up-to-date version of Codeface in python3 and updated installation scripts soon. However, there will be no functional changes. If you decide to install the current version of Codeface, which is fully working on Ubuntu 22.04 with python2.7, I'll be glad to help you with any problems and can also provide you with working example scripts, depending on your use case.

[alcemirsantos]

Hi @bockthom, thanks for the quick and detailed answer.

I did use the original codeface version (from siemens repo) while ago in Passau.
I also noticed the long time without changes in the files, however I was misguided by the Joblin's study homepage reference, which makes me think the vagrant way would still work.

I never wrote a Vagrantfile as well, but I will dig a bit more on its update of the vagrant file, because I still think is a good solution for those not in a Linux environment.

If you could provide a docker file would also be of great help. I can wait a few weeks.

[bockthom]

I did use the original codeface version (from siemens repo) while ago in Passau. I also noticed the long time without changes in the files, however I was misguided by the Joblin's study homepage reference, which makes me think the vagrant way would still work.

Yeah, that's indeed a little bit confusing. Our study homepage already refers to this repo (the se-sic repo) with all the updated files as we have used the updated version of Codeface for the study, but we just used it in a local installation and did not update vagrant and README. Nevertheless, we will also cleanup and update the README as soon as we have fixed the installation processes.

I never wrote a Vagrantfile as well, but I will dig a bit more on its update of the vagrant file, because I still think is a good solution for those not in a Linux environment.
If you could provide a docker file would also be of great help. I can wait a few weeks.

We will probably discontinue the vagrant file, but we aim at providing a Docker file for automatized installation for those who are not in a Linux environment. I'll let you know as soon as we have finished working on it.

[alcemirsantos]

@bockthom I tried to install dependencies in local VM of Ubuntu 22.04 by running the installation scripts in this folder with no success. For instance, I faced several problems running Rscript packages.r due to unavailable versions and timeouts in the downloads.

Would it be possible to make an image of your local installation and make it available meanwhile you finish the docker files? If not, is there an update list of dependencies and architecture diagram, so we could try to build a docker-compose ourselves?

From this thread and the current docs I can enumerate the following dependencies for the old codeface:

1. Ubuntu 22.04;
2. Nginx (which version?)
3. Node.js (which version?)
4. MySQL (which version?)
5. R (which version?)
6. Graphviz (which version?)
7. cppstats (which version?)
8. Python (which version?)
9. Git (which version?)

The current codeface version in the main branch still hold this dependencies? Any other major dependency?

[bockthom]

The "current docs" have not been kept up to date by the former maintainers. Hence, I would not rely on them, they definitively won't work. We will remove them soon. For most of the dependencies, you should be able to use the recent version (exception is R and python, but I will talk about that below in detail.) (@nlschn: we need to replace the outdated docs in the docs directory by recent ones. Please remind me in our next meeting if I forget to talk about it.)

In contrast to the docs, the actual installation scripts in the integration-scripts directory on branch infosaar-updates are (almost) up to date and still work (I tried them last week with the most recent versions of all the dependencies). That is, you can just use the integration scripts with a few minor adjustments for Ubuntu 22.04 to install the current Codeface version which works with python2.7 (python3 is on the way...).
@alcemirsantos Please have a look at the following "installation guide" and let me know if this is convenient enough for you. If not, we can also update these scripts in the repository to contain the changes described below, but I am not sure if it is worth the effort, maybe the following description is already enough. Hope this helps already.

Installation guide for Codeface using python 2.7 from branch se-sic/infosaar-updates:

To install Codeface, run the following scripts in the following order. However, you need to make a number of adjustments to some of the scripts beforehand, see below:

./integration-scripts/install_repositories.sh (*)
./integration-scripts/install_common.sh (**)
./integration-scripts/install_codeface_R.sh (***)
./integration-scripts/install_codeface_node.sh (****)
./integration-scripts/install_codeface_python.sh (*****)
./integration-scripts/install_cppstats.sh
./integration-scripts/install_setup_database.sh

Step 1: Prepare for R installation:

(*) install_repositories.sh: R-Version: Instead of the outdated version checks in this script in lines 9-21 for old Ubuntu versions, I would recommend to replace lines 9-21 by the following two lines:

deb https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/
wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | sudo tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc

This allows installing R versions between 4.0 and 4.3. We have tested the old version of Codeface with R version 4.1.2. Installing R 4.1.2 is a little bit cumbersome, but I will guide you on that below.

Step 2: Install common system packages:

(**) install_common.sh: Add few more dependencies in lines 10-18: cmake, libmagick++-dev, python2-dev, python-pip, libharfbuzz-dev libfribidi-dev

Step 3: Install R, R packages, and a few more dependencies:

(***) install_codeface_R.sh: Replace "libgdal20" by "libgdal30" in line 21.
If you would like to make sure to install R 4.1.2 (as I am not 100% sure whether Codeface works with 4.3.0 out of the box), I would recommend to install the following instead of lines 7-16:

sudo apt install r-base-core=4.1.2-1ubuntu2
sudo apt install r-base-dev=4.1.2-1ubuntu2
sudo apt-mark hold r-base-core r-base-dev
sudo R CMD javareconf
sudo apt install r-cran-littler=0.3.15-1
sudo apt install r-cran-lattice=0.20-45-1
sudo apt install r-cran-zoo
sudo apt install r-cran-xts
sudo apt install r-cran-xtable
sudo apt install r-cran-reshape
sudo apt install r-cran-stringr
sudo apt install r-cran-scales
sudo apt install r-cran-rmysql
sudo apt install r-cran-rcurl
sudo apt install r-cran-nlme=3.1.155-1
sudo apt install r-cran-matrix=1.4-0-1
sudo apt install r-cran-mgcv=1.8-39-1
sudo apt install r-cran-rjson
sudo apt install r-cran-testthat
sudo apt install libx11-dev libssl-dev libssh2-1-dev libudunits2-dev

Afterwards run lines 18-25 as they are in install_codeface_R.sh (after replacing "libgdal20" by "libgdal30" in line 21).

Step 4: Install nodejs id-service

(****) install_codeface_node.sh: replace the two lines 9 and 10 (sudo mkdir node_modules; sudo chown vagrant.vagrant node_modules) by the single line mkdir node_modules to avoid permission problems.

Step 5: Install python packages and Codeface

(*****) install_codeface_python.sh: Replace "pip" in lines 7-9 by "pip2". As this script calls "setup.py", you need to make a few adjustments to setup.py first. You can simply replace the whole content in setup.py by what follows:

setup(name='codeface',
      version='0.2.0',
      description='Codeface: Socio-Technical Analysis of Software Development',
      author='Wolfgang Mauerer',
      author_email='[email protected]',
      url='https://github.com/siemens/codeface',
      packages=find_packages(exclude=['experiments']),
      package_data={'codeface': ['R/*.r', 'R/cluster/*.r', 'perl/*.pl']},
      entry_points={'console_scripts': ['codeface = codeface.cli:main']},
      install_requires=['progressbar', 'VCS', 'python-ctags3','PyYAML==3.12',
                        'mysqlclient<2', 'jira==1.0.14', 'ftfy<5', 'requests==2.18.4', 'Pygments==2.1.3',
                        'certifi==2018.1.18']
      )

Then calling "./integration-scripts/install_codeface_python.sh" should work.

Step 6: Install cppstats:

Using the remaining integration scripts, cppstats will be installed in version 0.9.4 via
"./integration-scripts/install_cppstats.sh"

Step 7: Setup database:

using "./integration-scripts/setup_databases.sh" sets the database up.

I hope I have not forgotten about anything here. Afterwards the installation should be finished. You can try "codeface --help" to see whether it works or not. If not, please let me know. I will help you with that, just send me an error message then and I will have a look at it.

Start Codeface:

If successfully installed, you can run codeface. Before doing so, you need to start the ID service:

cd "id_service"
nodejs id_service.js ${CFCONF} "info" &
IDSERVICE=$!
cd ..

To run the commit analysis:

codeface -j 1 -l "devinfo" run --recreate -c ${CFCONF} -p ${CSCONF} ${RESULTS} ${REPOS} > ${LOGS}/codeface_run.log 2>&1

To run the mail analysis:

codeface -j 1 -l "devinfo" ml -c ${CFCONF} -p ${CSCONF} "${RESULTS}" "${MAILINGLISTS}" > ${LOGS}/codeface_ml.log 2>&1

using the variables as described in the following:

CFCONF="codeface.conf" 
CSCONF="/path/to/the/casestudy.conf" (see examples in the conf directory)
REPOS="/path/to/the/repository/you/want/to/analyze"
MAILINGLISTS="/path/to/mboxfiles" (needed for the mailing list analysis, can be downloaded from gmane via nntp2mbox, for instance)
RESULTS="/path/to/results"
LOGS="/path/to/logs"

After Codeface has finished running, don't forget to terminate the ID service: kill $IDSERVICE

[alcemirsantos]

On Step 3, while running Rscript packages.R, I faced the following errors:

package from url: https://cran.r-project.org/src/contrib/Archive/BH/BH_1.75.0-0.tar.gz

package from url: https://cran.r-project.org/src/contrib/Archive/proxy/proxy_0.4-16.tar.gz

package from url: https://cran.r-project.org/src/contrib/Archive/BH/BH_1.75.0-0.tar.gz

...and it stops right there. I can not move any further.

I checked the version and I am using:

r --version
r ('littler') version 0.3.15
using GNU R Version 4.1.2 (2021-11-01)

[bockthom]

On Step 3, while running Rscript packages.R, I faced the following errors:

package from url: https://cran.r-project.org/src/contrib/Archive/BH/BH_1.75.0-0.tar.gz

This problem looks weird. When I call it here in Germany, it works. I don't know why the download fails, maybe there is some geographically located problem. Maybe you can try another CRAN mirror to download this package? And replace the URL of the BH installation in "packages.r" in line 64? Here you can find a list of mirrors, maybe it helps to try a Brazilian mirror: https://cran-r.c3sl.ufpr.br/src/contrib/Archive/BH/BH_1.75.0-0.tar.gz (edit: I originally had pasted the wrong link here)

package from url: https://cran.r-project.org/src/contrib/Archive/proxy/proxy_0.4-16.tar.gz

This error looks like a typical R 4.3.0 error. This kind of error does usually not appear prior to R 4.3.0, as they turned such a warning into an error in R 4.3.0. Are you sure that you really use R 4.1.2? Did you install a more recent R version before installing R 4.1.2? Maybe there are some packages installations / dependencies etc. left from a more recent version? Could you run R -e '.libPaths()' please and check whether there are some paths to a more recent R version? In this case, it might help to remove all R libraries first (sudo rm -rf for each of the outdated paths listed by the previous command) and start to reinstall R in the needed version cleanly afterwards (but, of course, this will take some time to reinstall all the R packages then...)

I even checked the version of gcc and I have used exactly the same version, so I expect the installation to work exactly as on my machine.

I checked the version and I am using:

r --version
r ('littler') version 0.3.15
using GNU R Version 4.1.2 (2021-11-01)

Even if you have installed 4.1.2 and use 4.1.2, there might be remnants from a more recent R version that has been installed before, as indicated above.

[alcemirsantos]

I tried to remove R completely and reinstall the 4.1.2. Then, I had to change some versions of he tools in the packages.r. Changed python to pyhon2 somewhere else. Finally I managed to pass through step 3 and tried the next steps.

Step 4: Installation of node

It went like this:

Step 7: Database setup

It went lik this:

Codeface --help output

id_service failing?

Although it seems codeface has installed, I believe starting it failed, right?

[bockthom]

I tried to remove R completely and reinstall the 4.1.2. Then, I had to change some versions of he tools in the packages.r. Changed python to pyhon2 somewhere else. Finally I managed to pass through step 3 and tried the next steps.

Great! And apologies for all the inconveniences. (We will definitively improve the documentation for the installation in the updated version of Codeface to prevent all these headaches.)

id_service failing?

Although it seems codeface has installed, I believe starting it failed, right?

Your Codeface installation has finished correctly, but the id service failed starting because of using an undefined environment variable in your call:

nodejs id_service.js ${CFCONF} "info"

I have used an evironment variable ${CFCONF} here for better configurability, but in your case, the path to the database configuration file could also be just hard-coded. In my exemplary statement above, I have used the following variables, which need to be defined before starting codeface and its id service (and before entering the id_service directory, as the paths are relative to Codeface's main directory):

CFCONF="codeface.conf" 
CSCONF="/path/to/the/casestudy.conf" (see examples in the conf directory)
REPOS="/path/to/the/repository/you/want/to/analyze"
MAILINGLISTS="/path/to/mboxfiles" (needed for the mailing list analysis, can be downloaded from gmane via nntp2mbox, for instance)
RESULTS="/path/to/results"
LOGS="/path/to/logs"

I should have put these code block before the id-service starting call in my installation guide above. Sorry for the confusion. So, for CSCONF you should just use the "codeface.conf" that comes with codeface. And for all the other variables, please set the paths as you like.

One last comment: We just encountered that the current Codeface version needs R package markovchain in a specific version, which is not part of the installation guide I posted above. So, when you start Codeface it will currently run into an error regarding markovchain. We will let you know very soon which version of markovchain you need and how you can install this specific version - and then everything should work with your Codeface installation; this should be the last step that is missing in the installation. Sorry for the inconvenience again, we will comment on this last piece here ASAP.

May I ask which of the output files of Codeface you actually need for your purposes? If you just need the commit and mail data that codeface writes to the database (which is the only data from Codeface that we have used in Joblin's paper on hierarchical and hybrid structures), you could also comment out the steps that need markovchain (lines 116-147 in codeface/project.py) and use codeface-extraction afterwards to get the commit/mail data from the database as csv files by calling python codeface-extraction/run-extraction.py -c ${CFCONF} -p ${CSCONF} ${RESULTS} > ${LOGS}/codeface_extraction.log 2>&1 (If you have additional questions regarding the data extraction, don't hesitate to ask. This will also be documented in Codeface in more detail after our update anyway... )

[alcemirsantos]

I installed the markovchain 0.8.6 since the 0.9.3 depends on R 4.3.0.

EDIT: I will open a pull request so you all can see what I have updated in my scripts so far.