Add CX and varobs for Surfacecloud #175
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| &CXControlNL | ||
| ! Quantities for which H(x) is not calculated so GeoVaLs are used to produced CX columns. | ||
| ! - multi-level | ||
| ! 4 = theta, 10 = q, 12 = qcf, 254 = qcl, 407 = p, 9201 = cloud_layer | ||
| ! - single-level | ||
| ! 1 = pstar, 31 = seaice, 33 = orog, 9217 = CloudAmount | ||
| CxFields=1,4,10,12,31,33,254,407,9201,9217 | ||
| / |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| &VarobsControlNL | ||
| Varfields=15 | ||
| / |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -963,6 +963,10 @@ subroutine opsinputs_varobswriter_populateobservations( & | |
| ! TODO(someone): handle this varfield | ||
| ! call Ops_Alloc(Ob % Header % AMSUb_Temp, "AMSUb_Temp", Ob % Header % NumObsLocal, Ob % AMSUb_Temp) | ||
| case (VarField_cloud) | ||
| call opsinputs_fill_fillelementtype2dfromnormalvariablewithlevels( & | ||
| Ob % Header % Cloud, "Cloud", Ob % Header % NumObsLocal, Ob % Cloud, & | ||
| ObsSpace, self % modlevs, "cloudAmount_", "DerivedObsValue", self % GeoVaLsAreTopToBottom, & | ||
| "cloudAmount_", "DerivedObsError") | ||
|
There was a problem hiding this comment. I don't think we should be writing out from DerivedObsError here (as mentioned in https://github.com/JCSDA-internal/ufo/pull/2957). Also can I check why there is a trailing underscore after cloudAmount? There was a problem hiding this comment. OK, thanks James. There was a problem hiding this comment. Ah I see, that makes sense regarding the underscores, thanks for clarifying. We would want to replace DerivedObsError with ObsErrorData I think |
||
| ! TODO(someone): handle this varfield | ||
| ! call Ops_Alloc(Ob % Header % Cloud, "Cloud", Ob % Header % NumObsLocal, Ob % Cloud) | ||
| case (VarField_rainrate) | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| window begin: 2018-01-01T00:00:00Z | ||
| window end: 2018-01-01T01:00:00Z | ||
|
|
||
| observations: | ||
| - obs space: | ||
| name: SurfaceCloud | ||
| obsdatain: | ||
| engine: | ||
| type: H5File | ||
| obsfile: Data/dummy.nc4 | ||
| simulated variables: [dummy] | ||
| geovals: | ||
| filename: Data/015_UpperAirCxField_cloud_layer.nc4 | ||
| obs filters: | ||
| # Set the flag of observations with missing values to "pass": we want to check if these | ||
| # values are encoded correctly in the Cx file. | ||
| - filter: Reset Flags to Pass | ||
| flags_to_reset: [10, 15] # missing, Hfailed | ||
| # Reject observation 3: we want to check if it is omitted from the Cx file, as expected. | ||
| - filter: Domain Check | ||
| where: | ||
| - variable: | ||
| name: MetaData/latitude | ||
| minvalue: 0.0 | ||
| - filter: Cx Writer | ||
| namelist_directory: testinput/CxWriterNamelists_015_UpperAirCxField_cloud_layer | ||
| reject_obs_with_any_variable_failing_qc: true | ||
| general_mode: debug | ||
| IC_PLevels: 5 | ||
| - filter: Cx Checker | ||
| expected_surface_variables: [] | ||
| expected_upper_air_variables: ["15"] # IndexCxcloud_layer | ||
| expected_main_table_columns: | ||
| - # batch 1 | ||
| - ["1.10", "1.20", "1.30"] # column 1 | ||
| - ["2.10", "**********", "2.30"] # column 2 (the asterisks represent a missing float) | ||
| - ["4.10", "4.20", "4.30"] # column 3 | ||
| HofX: ObsValue # just a placeholder -- not used, but needed to force calls to postFilter. | ||
| benchmarkFlag: 1000 # just to keep the ObsFilters test happy | ||
| flaggedBenchmark: 0 |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| window begin: 2018-01-01T00:00:00Z | ||
| window end: 2018-01-01T02:00:00Z | ||
|
|
||
| observations: | ||
| - obs space: | ||
| name: SurfaceCloud | ||
| obsdatain: | ||
| engine: | ||
| type: H5File | ||
| obsfile: Data/015_VarField_cloud.nc4 | ||
| simulated variables: [dummy] | ||
| channels: 1-3 | ||
| obs filters: | ||
| # Set the flag of observations with missing values to "pass": we want to check if these | ||
| # values are encoded correctly in the VarObsFile. | ||
| - filter: Reset Flags to Pass | ||
| flags_to_reset: [10, 15] # missing, Hfailed | ||
| # Reject observation 3: we want to check if it is omitted from the VarObs file, as expected. | ||
| - filter: Domain Check | ||
| where: | ||
| - variable: | ||
| name: MetaData/latitude | ||
| minvalue: 0.0 | ||
| - filter: VarObs Writer | ||
| namelist_directory: ../etc/ukv/varobs | ||
| general_mode: debug | ||
| IC_PLevels: 3 | ||
| geovals_are_top_to_bottom: false | ||
| - filter: VarObs Checker | ||
| expected_main_table_columns: | ||
| # In the arrays below, rows denote locations and columns channels. | ||
| field: ["15", "15", "15", | ||
| "15", "15", "15", | ||
| "15", "15", "15", | ||
| "15", "15", "15"] | ||
| ob value: ["4.10000" ,"5.10000","6.10000", | ||
| "-1073741824.00000","5.20000","6.20000", | ||
| "4.30000" ,"5.30000","6.30000", | ||
| "4.40000" ,"5.40000","6.40000"] | ||
| lat: ["21.00000", "21.00000", "21.00000", | ||
| "22.00000", "22.00000", "22.00000", | ||
| "-23.00000", "-23.00000", "-23.00000", | ||
| "24.00000", "24.00000", "24.00000"] | ||
| lon: ["31.00000", "31.00000", "31.00000", | ||
| "32.00000", "32.00000", "32.00000", | ||
| "33.00000", "33.00000", "33.00000", | ||
| "34.00000", "34.00000", "34.00000"] | ||
| time: ["-3540.00000", "-3540.00000", "-3540.00000", | ||
| "-3480.00000", "-3480.00000", "-3480.00000", | ||
| "-3420.00000", "-3420.00000", "-3420.00000", | ||
| "-3360.00000", "-3360.00000", "-3360.00000"] | ||
| Callsign: ["station_1", "station_1", "station_1", | ||
| "station_2", "station_2", "station_2", | ||
| "station_3", "station_3", "station_3", | ||
| "station_4", "station_4", "station_4"] | ||
| HofX: ObsValue # just a placeholder -- not used, but needed to force calls to postFilter. | ||
| benchmarkFlag: 1000 # just to keep the ObsFilters test happy | ||
| flaggedBenchmark: 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| &CXControlNL | ||
| CxFields=9201 | ||
| / |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| &VarobsControlNL | ||
| Varfields=15 | ||
| / |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| window begin: 2018-01-01T00:00:00Z | ||
| window end: 2018-01-01T01:00:00Z | ||
|
|
||
| observations: | ||
| - obs space: | ||
| name: SurfaceCloud | ||
| obsdatain: | ||
| engine: | ||
| type: H5File | ||
| obsfile: Data/dummy.nc4 | ||
| simulated variables: [dummy] | ||
| geovals: | ||
| filename: Data/cx_ukvnamelist_surfacecloud.nc4 | ||
| obs filters: | ||
| # Set the flag of observations with missing values to "pass": we want to check if these | ||
| # values are encoded correctly in the Cx file. | ||
| - filter: Reset Flags to Pass | ||
| flags_to_reset: [10] # missing | ||
| # Reject observation 3: we want to check if it is omitted from the Cx file, as expected. | ||
| - filter: Domain Check | ||
| where: | ||
| - variable: | ||
| name: MetaData/latitude | ||
| minvalue: 0.0 | ||
| - filter: Cx Writer | ||
| namelist_directory: ../etc/ukv/cx | ||
| reject_obs_with_any_variable_failing_qc: true | ||
| general_mode: debug | ||
| geovals_are_top_to_bottom: false | ||
| - filter: Cx Checker | ||
| expected_surface_variables: ["1","2","17","24"] | ||
| expected_upper_air_variables: ["1","5","11","15","29","30"] | ||
| expected_main_table_columns: | ||
| - # observation 3 is rejected by the tests above hence only 3 (1,2,4) columns | ||
| - ["7.10","17.10","27.10","37.10","1.30","1.20","1.10","11.30", | ||
| "11.20","11.10","21.30","21.20","21.10","31.30","31.20","31.10", | ||
| "41.30","41.20","41.10","51.30","51.20","51.10"] | ||
| - ["**********","**********","**********","**********","2.30","**********","2.10","12.30", | ||
| "**********","12.10","22.30","**********","22.10","32.30","**********","32.10","42.30", | ||
| "**********","42.10","52.30","**********","52.10"] | ||
| - ["7.40","17.40","27.40","37.40","4.30","4.20","4.10","14.30", | ||
| "14.20","14.10","24.30","24.20","24.10","34.30","34.20","34.10", | ||
| "44.30","44.20","44.10","54.30","54.20","54.10"] | ||
| HofX: ObsValue # just a placeholder -- not used""but needed to force calls to postFilter. | ||
| benchmarkFlag: 1000 # just to keep the ObsFilters test happy | ||
| flaggedBenchmark: 0 |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| window begin: 2018-01-01T00:00:00Z | ||
| window end: 2018-01-01T02:00:00Z | ||
|
|
||
| observations: | ||
| - obs space: | ||
| name: SurfaceCloud | ||
| obsdatain: | ||
| engine: | ||
| type: H5File | ||
| obsfile: Data/varobs_ukvnamelist_surfacecloud.nc4 | ||
| simulated variables: [stationPressure] | ||
| observed variables: [stationPressure] | ||
| derived variables: [cloudAmount] | ||
| obs filters: | ||
| # Set the flag of observations with missing values to "pass": we want to check if these | ||
| # values are encoded correctly in the VarObsFile. | ||
| - filter: Reset Flags to Pass | ||
| flags_to_reset: [10] # missing | ||
| # Blacklist all missing entries in the original profiles (i.e. observation 2) | ||
| # This must be run after the Reset Flags to Pass filter, | ||
| # which sets the flags of any missing values in the original profiles | ||
| # to 'pass'. Therefore this filter is run with the 'defer to post' option set to true. | ||
| - filter: BlackList | ||
| where: | ||
| - variable: | ||
| name: MetaData/latitude | ||
| is_not_defined: | ||
| defer to post: true | ||
| - filter: VarObs Writer | ||
| namelist_directory: ../etc/ukv/varobs | ||
| general_mode: debug | ||
| IC_PLevels: 3 | ||
| reject_obs_with_all_variables_failing_qc: true | ||
| - filter: VarObs Checker | ||
| expected_main_table_columns: | ||
| # Rows are locations, columns are filter variables | ||
| # The blacklist prevents the missing ob from being written out | ||
| field: [15,15,15, | ||
| 15,15,15, | ||
| 15,15,15, | ||
| 15,15,15] | ||
| ob value: ["1.30000","1.20000","1.10000", | ||
| "2.30000","-1073741824.00000","2.10000", | ||
| "3.30000","3.20000","3.10000", | ||
| "4.30000","4.20000","4.10000"] | ||
| lat: ["7.10000","7.10000","7.10000", | ||
| "-1073741824.00000","-1073741824.00000","-1073741824.00000", | ||
| "7.30000","7.30000","7.30000", | ||
| "7.40000","7.40000","7.40000"] | ||
| lon: ["17.10000","17.10000","17.10000", | ||
| "-1073741824.00000","-1073741824.00000","-1073741824.00000", | ||
| "17.30000","17.30000","17.30000", | ||
| "17.40000","17.40000","17.40000"] | ||
| HofX: ObsValue # just a placeholder -- not used, but needed to force calls to postFilter. | ||
| benchmarkFlag: 1000 # just to keep the ObsFilters test happy | ||
| flaggedBenchmark: 0 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should the name
Cloudmatch up with the name used in JCSDA-internal/ioda#1062?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not necessarily, but maybe.
The way it works in OPS is that
Cloudorcloudis the field that goes into the varobs, i.e. this one, it is the cloud fraction on model levels calculated in the SurfaceCloud routine, whereas the CX field is what is calledlevel_cloudand I think is the one Fabien has added there. I'm not 100% sure what their interaction should be in JOPA, whether they should have the same name or not.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this needs to match with the name proposed in https://github.com/JCSDA-internal/ioda/pull/1062, i.e.
cloudAmountinstead ofCloud.However I am happy if you want to commit this as is, and then handle the variable name change with separate PR to ioda, ufo, opsinputs, etc?