Update varobs and cx writers for visibility #221

ReubenHill · 2024-08-15T15:32:18Z

The changes made in this PR are necessary for the assimilation of visibility by VAR. There are three key changes:

The surface namelist has been updated to include model fields for visibility, surface total water (qt2) and aerosol (specifically with stash code 90, called "Total Aerosol (for Vis)").
The cx writer has been updated to allow surface total water and aerosol to be output. These are used by VAR, alongside other model variables, to calculate background visibility.
The varobs writer will now write ObsValue/horizonalVisibility values to VarField_logvis.

Updating the cx writer for outputting aerosol was non-trivial:

There are multiple aerosol stash codes - 90 is the correct one to use for the calculation of background visibility in VAR.
The field is defined over all 70 levels, but OPS only outputs the level closest to the surface: see the comments in src/opsinputs/opsinputs_fill_mod.F90 for details. This required special case code.

@adammaycock this was the code you were helping me to debug last month: the issue was an incorrect stash code causing the necessary select case to be skipped over.

@ss421 I have now confirmed that I need the following for VAR to run, so please don't exclude them from FieldNameCompare.xlsx:

qt2 - surface total water - stash code 3255
aerosol - Total Aerosol (for Vis) - stash code 90

NOTE: the ukv surface KGO will need to be remade once this is merged.

VAR expects aerosol (for visibility) to be called logm.

A search using the stash code reveals that the field using the correct stash code is named aerosol rather than logm.

ctgh

Very nice! Does this require any modifications to sith KGO output?

src/opsinputs/opsinputs_fill_mod.F90

ReubenHill · 2024-08-16T08:13:37Z

Very nice! Does this require any modifications to sith KGO output?

I don't think so: don't KGOs only use the NetCDF output from the jopa background task? This ought not to effect that.

ctgh · 2024-08-16T08:28:45Z

I believe they check ODB, Varobs and CX output. It might be worth running it just to be on the safe side.

ReubenHill · 2024-08-16T10:46:10Z

I believe they check ODB, Varobs and CX output. It might be worth running it just to be on the safe side.

Yes you are correct, the kgos will need to be remade. Unless it's very straightforward, it's probably only worth doing once the jjdocs PR is ready since that will also need new kgos.

ctgh

Thanks for making the change. The ctest failure is fixed in #222

matthewrmshin

Ran OK in my environment. No failures detected for mo-bundle main build + ctest, and sith/malak kgo runs (apart from what's expected).

ReubenHill · 2024-08-16T15:26:07Z

OFFICIAL I guess we don't have a UKV surface KGO set up then. Please merge if you are happy. OFFICIAL

…

________________________________ From: Matt Shin ***@***.***> Sent: 16 August 2024 4:16 PM To: MetOffice/opsinputs ***@***.***> Cc: Reuben Nixon-Hill ***@***.***>; Assign ***@***.***> Subject: Re: [MetOffice/opsinputs] Update varobs and cx writers for visibility (PR #221) This email was received from an external source. Always check sender details, links & attachments. @matthewrmshin approved this pull request. Ran OK in my environment. No failures detected for mo-bundle main build + ctest, and sith/malak kgo runs. — Reply to this email directly, view it on GitHub<#221 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB4I73KNSWJ5T3DO565QRXDZRYJWVAVCNFSM6AAAAABMSNKOV6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDENBSHE2TGNRRGY>. You are receiving this because you were assigned.Message ID: ***@***.***>

matthewrmshin · 2024-08-16T15:33:55Z

OFFICIAL Hi Reuben, I just realised my typing was cut off halfway. I actually meant that there were no unexpected KGO issues. I've now amended my comment in the PR. Sorry for the confusion. Best, Matt --- Dr Matt Shin Expert Scientific Software Engineer Met Office FitzRoy Road Exeter EX1 3PB United Kingdom Tel: +44 (0)330135 1724 ***@***.*** http://www.metoffice.gov.uk/ OFFICIAL

…

________________________________ From: Reuben W. Nixon-Hill ***@***.***> Sent: 16 August 2024 16:26 To: MetOffice/opsinputs ***@***.***> Cc: Matthew Shin ***@***.***>; Mention ***@***.***> Subject: Re: [MetOffice/opsinputs] Update varobs and cx writers for visibility (PR #221) This email was received from an external source. Always check sender details, links & attachments. OFFICIAL I guess we don't have a UKV surface KGO set up then. Please merge if you are happy. OFFICIAL

________________________________ From: Matt Shin ***@***.***> Sent: 16 August 2024 4:16 PM To: MetOffice/opsinputs ***@***.***> Cc: Reuben Nixon-Hill ***@***.***>; Assign ***@***.***> Subject: Re: [MetOffice/opsinputs] Update varobs and cx writers for visibility (PR #221) This email was received from an external source. Always check sender details, links & attachments. @matthewrmshin approved this pull request. Ran OK in my environment. No failures detected for mo-bundle main build + ctest, and sith/malak kgo runs. — Reply to this email directly, view it on GitHub<#221 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB4I73KNSWJ5T3DO565QRXDZRYJWVAVCNFSM6AAAAABMSNKOV6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDENBSHE2TGNRRGY>. You are receiving this because you were assigned.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub<#221 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAG52RI6HYFYIO53FAAZ3NLZRYK2NAVCNFSM6AAAAABMSNKOV6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJTG4YTIMZVG4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

src/opsinputs/opsinputs_fill_mod.F90

This reverts commit b4efa39. Either the syntax is wrong or this is not allowed.

matthewrmshin · 2024-09-04T11:42:28Z

This change appears to require KGO changes for various Surface.* files for sith kgo_glu and kgo_ukv as well as malak kgo_glu_jopa_jada and kgo_glu_jopa_var - so not just the UKV ones. Is this expected?

ReubenHill · 2024-09-04T13:54:43Z

This change appears to require KGO changes for various Surface.* files for sith kgo_glu and kgo_ukv as well as malak kgo_glu_jopa_jada and kgo_glu_jopa_var - so not just the UKV ones. Is this expected?

The varobs writer now outputs visibility - since these are in the list of observed variables for both global and ukv yamls, I therefore expect both sets of varobs files to change. The changes here should not effect global surface cx files, though I can imagine that a change to a varobs file might somehow necessitate a change to a cx file. What exact differences do you see?

matthewrmshin · 2024-09-04T14:04:43Z

Differences reported for these files:

malak:

kgo_glu_jopa_jada:
- glu_jopa_varobs/Surface.varobs
kgo_glu_jopa_var:
- glu_jopa_varobs/Surface.varobs

sith:

kgo_glu:
- glu_jopa_varobs/Surface.varobs
- glu_jopa_varobs_screen/Surface.varobs
kgo_ukv:
- ukv_jopa_var_cx/Surface.cx
- ukv_jopa_varobs/Surface.varobs

See outputs of these suites of mine: http://fcm1/cylc-review/suites?user=frsn&names=*oi221-kgo_glu*+*oi221-kgo_ukv

ReubenHill · 2024-09-05T13:06:16Z

Differences reported for these files:

malak:

kgo_glu_jopa_jada:

glu_jopa_varobs/Surface.varobs

kgo_glu_jopa_var:

glu_jopa_varobs/Surface.varobs

sith:

kgo_glu:

glu_jopa_varobs/Surface.varobs

glu_jopa_varobs_screen/Surface.varobs

kgo_ukv:

ukv_jopa_var_cx/Surface.cx

ukv_jopa_varobs/Surface.varobs

See outputs of these suites of mine: http://fcm1/cylc-review/suites?user=frsn&names=*oi221-kgo_glu*+*oi221-kgo_ukv

These are as expected. Whilst the global varobs files now contain visibility (as the OPS ones always have), the global JOPA VAR task job.stats outputs before and after are identical - i.e. the presence of visibility in the global surface JOPA output has no impact on VAR. So please go ahead and recreate the KGOs.

ReubenHill added 14 commits August 15, 2024 13:00

Output visibility in ukv surface cx files

44094a2

Test cxwriter for visibility

67d7c77

Implement varfield logvis for horizontal visibility

b0c0d2e

Add aerosol and qt2 to cx output Surface.nl

6dafc8e

Add and test aerosol and total water cx fields

a4df030

Rename aerosol cx field logm

1bb0249

VAR expects aerosol (for visibility) to be called logm.

remove changes to OPS

0e94e4a

Use correct OPS aerosol stash code name

aedc80f

rename logm aerosol again

e9ecdb5

A search using the stash code reveals that the field using the correct stash code is named aerosol rather than logm.

WIP: Working aerosol output! tests may fail.

e28295b

Remove print statements

1102379

Comment and correct aerosol behaviour

de4350e

Allow for geovals of fewer than 70 levels

3669f7a

Update tests since aerosol has multiple levels

6481513

ReubenHill requested review from adammaycock, ctgh, DJDavies2, matthewrmshin, mikecooke77 and yaswant as code owners August 15, 2024 15:32

ctgh reviewed Aug 16, 2024

View reviewed changes

src/opsinputs/opsinputs_fill_mod.F90 Outdated Show resolved Hide resolved

tidy special case

b0f5598

matthewrmshin assigned ReubenHill Aug 16, 2024

ctgh approved these changes Aug 16, 2024

View reviewed changes

Merge branch 'develop' into feature/ukv_surface_add_vis

dfae86a

matthewrmshin approved these changes Aug 16, 2024

View reviewed changes

matthewrmshin added the do not merge label Aug 16, 2024

DJDavies2 reviewed Aug 18, 2024

View reviewed changes

src/opsinputs/opsinputs_fill_mod.F90 Show resolved Hide resolved

DJDavies2 approved these changes Aug 18, 2024

View reviewed changes

ReubenHill added 2 commits August 19, 2024 10:03

simplify level querying

b4efa39

Revert "simplify level querying"

23a7e6a

This reverts commit b4efa39. Either the syntax is wrong or this is not allowed.

matthewrmshin added waiting for kgo update waiting for kgo update ready to merge and removed do not merge waiting for kgo update waiting for kgo update labels Sep 5, 2024

matthewrmshin merged commit 2d6b182 into develop Sep 6, 2024
7 of 9 checks passed

matthewrmshin deleted the feature/ukv_surface_add_vis branch September 6, 2024 14:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update varobs and cx writers for visibility #221

Update varobs and cx writers for visibility #221

ReubenHill commented Aug 15, 2024 •

edited

Loading

ctgh left a comment

ReubenHill commented Aug 16, 2024

ctgh commented Aug 16, 2024

ReubenHill commented Aug 16, 2024 •

edited

Loading

ctgh left a comment •

edited

Loading

matthewrmshin left a comment •

edited

Loading

ReubenHill commented Aug 16, 2024 via email

matthewrmshin commented Aug 16, 2024 via email

matthewrmshin commented Sep 4, 2024

ReubenHill commented Sep 4, 2024

matthewrmshin commented Sep 4, 2024

ReubenHill commented Sep 5, 2024

Update varobs and cx writers for visibility #221

Update varobs and cx writers for visibility #221

Conversation

ReubenHill commented Aug 15, 2024 • edited Loading

ctgh left a comment

Choose a reason for hiding this comment

ReubenHill commented Aug 16, 2024

ctgh commented Aug 16, 2024

ReubenHill commented Aug 16, 2024 • edited Loading

ctgh left a comment • edited Loading

Choose a reason for hiding this comment

matthewrmshin left a comment • edited Loading

Choose a reason for hiding this comment

ReubenHill commented Aug 16, 2024 via email

matthewrmshin commented Aug 16, 2024 via email

matthewrmshin commented Sep 4, 2024

ReubenHill commented Sep 4, 2024

matthewrmshin commented Sep 4, 2024

ReubenHill commented Sep 5, 2024

ReubenHill commented Aug 15, 2024 •

edited

Loading

ReubenHill commented Aug 16, 2024 •

edited

Loading

ctgh left a comment •

edited

Loading

matthewrmshin left a comment •

edited

Loading