IAP AGCM (CAMv5.1) "physics buffer" and CCPP metadata

[grantfirl]

I'm trying to help make the IAP AGCM (which is heavily based off of CAMv5.1) a CCPP host model. It uses what they call a "physics buffer" which is basically a giant array that contains data for all blocks (or chunks as they call it) for all variables that are used internally in the physics suite. It can contain what we have been calling interstitial variables (those that are ephemeral and only need to pass between physics during one timestep) and those that in FV3 and SCM are part of persistent DDTs, depending on what is used as a variable's "scope" when it is added to the array. I should note that not ALL variables are in the physics buffer. Some, like what are in FV3's statein and stateout exist in similar DDTs, for which adding metadata isn't a problem.

Currently, at some point during initialization of the model, the variables that are needed by a physics suite (that are not already part of the aforementioned DDTs) are added to the physics buffer. Then, during the run phase, within a loop that is going over all blocks/chunks, data is retrieved from the physics buffer for a given block/chunk index prior to calling the physics scheme (that then operates on only that chunk of columns).

Has anyone thought of an elegant way to handle this with CCPP metadata? For the purpose of the CCPP, do I need to turn what it is in the physics buffer into a DDT (that holds data for one chunk) like is done in FV3? I seem to remember there being discussion during CCPP's early days about models with "registries" (which it sounds like the physics buffer is an implementation of) and that one could loop through a registry adding variables to what used to be "cdata", but that doesn't really exist anymore (for most intents-and-purposes). I don't remember whether that looping was to be handled by the framework or the host -- probably the host.

Regardless, I'll keep on thinking of a way to make this work, but in the meantime, if any other CCPP experts have already thought of a good solution in the past, I'm all ears.

-Grant

[grantfirl]

I should add that I understand that this is likely a moot point if one were to use capgen because of its ability to allocate memory for physics variables, but that doesn't exist in ccpp_prebuild, which is what I have to use today.

[climbfuji]

As far as I understand this, there is no need to use DDTs to use blocking. Can you post an example for how an exemplary variable is allocated and how the chunks look like that get passed to the physics? I'd be interested to see how the dimensions look like. Thanks!

[gold2718]

As someone who has worked with the "physics buffer" (pbuf) for many years, I have to say that I think it is incompatible with the CCPP concept. pbuf fields are created at run time; If they are not created, it is different than an unallocated variable -- it simply does not exist.
The only thing I can think of is to define the metadata for all the fields you think you need. At init time, check to see if they all exist (using pbuf_get_field), and quit if any are missing.

[grantfirl]

@climbfuji Here is an example for you.

The pbuf variable is an array of DDTs of type pbuf_fld. The pbuf_fld DDT looks like:

type, public :: pbuf_fld
      character(len=16)                       :: name
      character(len=16)                       :: scope
      integer                                 :: fdim, mdim, ldim
      real(r8), pointer, dimension(:,:,:,:,:) :: fld_ptr
      type(var_desc_t)                        :: vardesc
      type(var_desc_t), pointer               :: vardesc_list(:)   ! used to seperate variables of > 4MB (required for netcdf)
      type(io_desc_t), pointer                :: iodesc
   end type pbuf_fld

where the flt_ptr field has dimensions of (fdim,horizontal_loop_extent,mdim,all blocks/chunks,ldim) where fdim,mdim,ldim are given to a variable when it is added to pbuf. For 2D variables, fdim=mdim=ldim = 1. For 3D variables fdim=ldim=1, but mdim=vertical_dimension. I think that ldim is used for a time level dimension when it is used.

Fields get added to the pbuf array during a "register" subroutine for each process-level physics driver that is called at initialization time. Some fields are dependent on the chosen scheme. Others get added for all available schemes. At run time, in each process-level physics driver, variables are retrieved from the buffer into a local variable pointer, e.g.

evapcsh => pbuf(nevapr_shcu_idx)%fld_ptr(1,1:pcols,1:pver,lchnk,1)

that is then passed into the actual scheme subroutine.

So, I see a couple of issues. The first is what Steve said, that pbuf is filled at runtime and may not have all variables. The second is the blocking. The way that I understand it for ccpp_prebuild (and I could be wrong) is that the TYPEDEFS_NEW_METADATA dictionary in ccpp_prebuild_config is used to associate DDT instances with a string that will allow the autogenerated caps to access the memory associated with a variable. E.g., for FV3, it will associate a diag DDT instance to the correct block via GFS_data(cdata%blk_no)%Intdiag. I don't see how that would work e.g. with pbuf(nevapr_shcu_idx)%fld_ptr(1,1:pcols,1:pver,cdata%blk_no,1).

The only way that I'm thinking might work is to write some code that turns the pbuf into something that is more easily referenced for the CCPP metadata/caps and to include all variables that might be used (although perhaps we can also make use of the active metadata attribute in this case to not allocate a superset).

[grantfirl]

Another option that might work, but would not be portable at all and would not allow the model to run any existing CCPP physics (which is probably a non-starter, but might help to meet the project's temporary deliverable), is to have metadata for the variable indices in the pbuf array only and then to pass in the entire pbuf, the necessary variable indices, and the block number to each scheme that uses pbuf variables. Yuck.