How many maximum processor we can put on SE?

[monsieuralok]

@gold2718 @tto061 @oyvindseland @DirkOlivie Hi, I was thinking how many maximum processors we can put on SE in CAM? EXample :-
ne30np4.pg3 48600 elements
ne30np4.pg4 86400 elements
ne60np4.pg3 194400 elements

How many maximum processors we can set and how are those calculated? Are they dependents on also vertical levels like as in FV? if yes, How? Thanks a lot Alok

[gold2718]

The maximum scaling of CAM-SE is purely on the number of spectral elements (not column numbers as you used). For uniform grids, that can easily be calculated based on the first number (30 for ne30, 60 for ne60). If we call that number X, the maximum number of MPI tasks is:

X * X * 6

So for ne30, this is 5400 and for ne60, it is 21600.

To avoid idle processors, lower task counts should be the maximum task count divided by an integer (which will be the number of elements on each task). The maximum number of tasks does not depend on the use of CSLAM, the underlying physics grid, or the number of levels.

For refined meshes, you have to retrieve the number of spectral elements from the mesh definition file.

[monsieuralok]

@gold2718 Thanks a lot. Now, it is more clearer than before. But, I was thinking again why people in NCAR is not setting optimal number of processors on CAM? Example for this case as I was trying to reproduce:- https://github.com/NCAR/amwg_dev/blob/f.cam6_3_110.FMTHIST_v0c.ne30.tuningF.001/ it uses 2160 PES while optimal number should be 1800 or 2700? I did most benchmark using 2160 so, I guess that is not a good number.

Also, if I am correct SE grid is a surface mesh ? Then, What makes it different from FV that it is not parallelized along vertical/level direction ?

[gold2718]

For the CAM experiment you mention, you would have to ask them to get a full answer, however, there is always a tradeoff between efficiency and machine cost and in this case, they seem to have selected a job size (60 full Cheyenne nodes) that optimizes their run. Larger task counts tend to spend more time in the queue so maybe this is the sweet spot for them.

Final tuning usually depends quite a bit on the target machine. For instance, on Betzy, 5400 is not an even number of nodes so you have to find another setting that optimizes cost and throughput.

[gold2718]

The SE grid is not necessarily the same as the surface mesh (i.e., the grid that communicates with the surface models). For NE30np4, this is the same as the surface, however, for NE30np4.pg3, the surface mesh is the pg3 grid. SE does not parallelize across the vertical dimension (one of many many differences between the two dycores).

[monsieuralok]

@gold2718 Thanks a lot, now it clears a lot of my questions going in my mind.