reorganise NorESM input data

[jgriesfeller]

In conjunction with INES WP5 there is the topic of reorganising the NorESM input data. So far @matsbn , @MichaelSchulzMETNO , @tylov and @jgriesfeller have mainly discussed this and would like to open the discussion to a wider community. No decisions have been made yet.

Currently the basic idea is system where only a certain number of people can write (and delete) to a central input data "repository". This would essentially be a stripped down noresm group on the sigma2 systems (why is mentioned further down).

In addition, the inputdata folder on Betzy would act as the central repository that should hold all worth to keep inputdata and as such act as the inputdata master for NorESM and all its installations on the different HPCs. We would also need to define process to ingest new input data into the central repository on Betzy. For the discussions I would like to to refer to this as the inputdata-repo folder.

In order to keep things for the normal users simple, we suggest to link all current input data into another folder (to be called inputdata in the discussion), so that users can still add input data the current way and create input data for their runs as they do it right now. The model will use the inputdata folder.
Furthermore, the inputdata-repo will be distributed from betzy to the other HPCs currently running NorESM on a regular basis (e.g. weekly).

In principle we should have another unix group to handle the inputdata writing, but as of now sigma2 only allows one unix group per project. So unless we want only one user to handle the writing (there's also no role users @Sigma2, right?), we need to use the groups we have for the different NorESM based projects.

@matsbn, @MichaelSchulzMETNO and @tylov I hope I have summarised our discussions correctly, if not, please correct me.

[jmaerz]

Hi, being rather new to NorESM and being currently more on the development than on the application side, may I ask, how the versioning of input data (as in linking to certain input versions of data) is currently done and are there thoughts to change it with the reorganisation?

[MichaelSchulzMETNO]

I think the reorganisation should elaborate an answer to the following questions, documented in the FAQ and input data section of the NorESM documentation :

1. What is the master copy of the input data ?
2. How can I efficiently use the input data for a model on one of our HPC systems?
3. Is there a back up of the master copy attached to it?
   3b) Which version of the input data have I used for my model run?
4. How can I add and use my own input data?
   4b) How can I share my input data with other users?
5. Are the default input data from CESM copied into the master directory?
6. Who is deciding on what is in the master data copy?

[jmaerz]

Hi @MichaelSchulzMETNO , thanks for summarizing the relevant questions - some of them I had in mind when asking about planned changes with regards to the versioning. So, from your response, I grasp that this is still under (early) debate. Potentially another question could be how to best/whether to ensure that certain code version is linked to certain default input data (which not necessarily are the most recent).