WIP: Use HPC for CI

[jakob-fritz]

No description provided.

[brownbaerchen]

Do you need to pip install in the CI? User installation with pip is persistent, afaik. Is the gitlab not associated with any user?
I would prefer to put the module load commands in the job scripts. But if you need to repeat the pip installs every time, it's better here.

[jakob-fritz]

I'm not sure, if we need pip install each time. The jobs are run as a predefined user (the one who triggers the CI in Gitlab). As mirroring is done with a personal access token, that user is impersonated and the CI will always run as this specific user. So if pip install is persistent, it will be available in all runs, as these runs are executed as the same user

Regarding module load: We can put them in the script. I find it easier/nicer if the scripts are as short as possible. Therefore, I moved the module load into the YAML-file. Furthermore, these steps (as module load or pip install) are executed on a login-node. The content of the sh-file is executed on a compute-node. In term of quota, it is "cheaper" if we don't spend compute-time for module load (although it does not take too long).

If you want to, feel free to move the pip install and module load into the script. If I shall do that, feel free to ping me!

[brownbaerchen]

Could we also have a directory where all job scripts can post their output? It would be neat to allow multiple job scripts. Not really needed, though.

[jakob-fritz]

Yes sure we can have a directory for the outputs. Each job in Gitlab spawns in a separate directory. Therefore, multiple jobs (from Gitlab) won't put files in the same directory. If multiple tasks from slurm are spawned from a single Job in Gitlab, having this directory might make sense.

[jakob-fritz]

For the job mirror to gitlab to work, the information needs to be provided in the Github-Repo.
For the required variables (and where to get them) see the readme of the used action

[brownbaerchen]

How do we continue? If I understand correctly, we need to add a personal access token from GitLab as a secret to this repository. Does it make sense that @pancetta sets up the GitLab repo now? He has to add the secret anyhow.

[jakob-fritz]

Yes, probably it is best, if @pancetta does this. I can also create the repo (or use an existing one from me), but pancetta needs to add the secret in the end anyway.

As said: The steps are listed above. If questions occur, I am willing to help. In person only in the next year, as this year, I'm out of office

[pancetta]

OK, the repository is https://gitlab.jsc.fz-juelich.de/atml-ati/pysdc and the secret is under GITLAB_SECRET.

[pancetta]

Note that the gitlab repository is empty, so I could not allow force-push. I tried with wildcard *, but I don't know if that works. @jakob-fritz and @brownbaerchen are now maintainers of the gitlab repository, so you should be able to play around with this.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.04%. Comparing base (249741b) to head (376b44a).
Report is 25 commits behind head on create_gitlab_ci.

❗ Current head 376b44a differs from pull request most recent head 0cec1f3. Consider uploading reports for the commit 0cec1f3 to get more accurate results

Additional details and impacted files

@@                Coverage Diff                @@
##           create_gitlab_ci     #386   +/-   ##
=================================================
  Coverage             74.04%   74.04%           
=================================================
  Files                   274      274           
  Lines                 23153    23153           
=================================================
  Hits                  17143    17143           
  Misses                 6010     6010           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.