Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using IMERG data for observations in PCP-combine (could not set PYTHON interpreter manually) #2865

Open
jiangleads opened this issue Jan 16, 2025 Discussed in #1009 · 1 comment
Open

Using IMERG data for observations in PCP-combine (could not set PYTHON interpreter manually) #2865

jiangleads opened this issue Jan 16, 2025 Discussed in #1009 · 1 comment

Comments

@jiangleads
Copy link

jiangleads commented Jan 16, 2025 •
edited
Loading

Hello, I need to use PCP-combine to calculate GPM IMERG precipitation data over a certain period.
I tried running the example from this link:
https://metplus.readthedocs.io/en/latest/generated/met_tool_wrapper/PCPCombine/PCPCombine_python_embedding.html,
but I encountered the following errors:

DEBUG 4: Reading temporary Python dataplane file: /home/lzjiang/myconf/data/output/tmp/tmp_met_nc_3306477_0
Traceback (most recent call last):
File "/usr/local/met/11.1.0/gcc/13.2.0/share/met/python/met/dataplane.py", line 3, in
import numpy as np
ModuleNotFoundError: No module named 'numpy'
WARNING:
WARNING: tmp_nc_dataplane() -> an error occurred importing module "met.dataplane"
WARNING:
ERROR :
ERROR : get_field() -> can't get data plane from file "PYTHON_NUMPY"
ERROR :
01/16 06:23:46.646Z metplus.fdb34a11 INFO: Finished running /usr/local/met/11.1.0/gcc/13.2.0/bin/pcp_combine - took 0:00:04.413060

It seems that I didn't correctly call the required Python version.

The MET has been installed with python-3.12 which didn't have numpy library. I use module load command to load the MET tools.
module load met/11.1.0/gcc/13.2.0
Since the embeded python of MET didn't have the libraies, I would like to use the python of my conda env which had installed the needed libraries. I had activate the env of my own. conda activate myenv

Although I have already referred to this post , and made some adjustments, it doesn't seem to work.

  1. I had added MET_PYTHON_EXE variable to the PCPCombine_python_embedding.conf, And I had exported the environmental variable MET_PYTHON_EXE as well.
  2. I had tried to call the python interpreter explicitly, ~/miniconda3/envs/myenv/bin/python3 ~/pkgs/metplus/ush/run_metplus.py -c $PWD/PCPCombine_python_embedding.conf -c $PWD/user_system.conf
  3. In order to avoid calling a wrong python, I had also set a alias of python and python3.
alias python=/home/lzjiang/miniconda3/envs/myenv/bin/python
alias python3=/home/lzjiang/miniconda3/envs/myenv/bin/python3

Unfortunately, none of these seem to make sense. Same error as mentioned earlier. Could anyone kindly offer some help? Thanks in advance!

(myenv) lzjiang@nimbus myconf $ module load met/11.1.0/gcc 
(myenv) lzjiang@nimbus myconf $ /usr/local/met/11.1.0/gcc/13.2.0/bin/pcp_combine -add PYTHON_NUMPY 'name="/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02";' -name "APCP" /home/lzjiang/myconf/data/output/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20180102_1300_A06h -v 5
DEBUG 1: Start pcp_combine by lzjiang(1196101016) at 2025-01-16 17:12:02Z  cmd: /usr/local/met/11.1.0/gcc/13.2.0/bin/pcp_combine -add PYTHON_NUMPY name="/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02"; -name APCP /home/lzjiang/myconf/data/output/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20180102_1300_A06h -v 5 
DEBUG 2: Since the "-field" command line option was not used, parsing the command line arguments a list of files, each followed by a configuration string.
DEBUG 2: Performing derivation command (sum) for 1 files.
DEBUG 1: Reading data (name="/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02";) from input file: PYTHON_NUMPY
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_Python_Numpy".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_Python_Numpy".
DEBUG 3: Running user's python script (/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5).
DEBUG 3: Initializing MET compile time python instance: /usr/local/python/3.12.0/gcc/13.2.0/bin/python3
Traceback (most recent call last):
  File "/usr/local/met/11.1.0/gcc/13.2.0/share/met/python/met/dataplane.py", line 3, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
WARNING: 
WARNING: straight_python_dataplane() -> an error occurred importing module "/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5"
WARNING: 
ERROR  : 
ERROR  : get_field() -> can't get data plane from file "PYTHON_NUMPY"
ERROR  : 
(myenv) lzjiang@nimbus myconf $ echo $MET_PYTHON_EXE
(myenv) lzjiang@nimbus myconf $ export MET_PYTHON_EXE=/home/lzjiang/miniconda3/envs/myenv/bin/python
(myenv) lzjiang@nimbus myconf $ echo $MET_PYTHON_EXE
/home/lzjiang/miniconda3/envs/myenv/bin/python
(myenv) lzjiang@nimbus myconf $ /usr/local/met/11.1.0/gcc/13.2.0/bin/pcp_combine -add PYTHON_NUMPY 'name="/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02";' -name "APCP" /home/lzjiang/myconf/data/output/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20180102_1300_A06h -v 5
DEBUG 1: Start pcp_combine by lzjiang(1196101016) at 2025-01-16 17:16:03Z  cmd: /usr/local/met/11.1.0/gcc/13.2.0/bin/pcp_combine -add PYTHON_NUMPY name="/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02"; -name APCP /home/lzjiang/myconf/data/output/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20180102_1300_A06h -v 5 
DEBUG 2: Since the "-field" command line option was not used, parsing the command line arguments a list of files, each followed by a configuration string.
DEBUG 2: Performing derivation command (sum) for 1 files.
DEBUG 1: Reading data (name="/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02";) from input file: PYTHON_NUMPY
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_Python_Numpy".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_Python_Numpy".
DEBUG 3: Calling /home/lzjiang/miniconda3/envs/myenv/bin/python to run user's python script (/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5).
DEBUG 4: Writing temporary Python dataplane file:
DEBUG 4: 	/home/lzjiang/miniconda3/envs/myenv/bin/python /usr/local/met/11.1.0/gcc/13.2.0/share/met/python/pyembed/write_tmp_dataplane.py /tmp/tmp_met_nc_3328041_0 /home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5 /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02
Python Script:	'/usr/local/met/11.1.0/gcc/13.2.0/share/met/python/pyembed/write_tmp_dataplane.py'
User Command:	'/home/lzjiang/pkgs/metplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5 /home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg IRprecipitation 2018010213 02'
Temporary File:	'/tmp/tmp_met_nc_3328041_0'
Input Directory      : '/home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg'
Variable Name        : 'IRprecipitation'
Valid Time           : datetime.datetime(2018, 1, 2, 13, 0)
Accumulation (hr)    : 2.0
======================
Processing Valid Time: datetime.datetime(2018, 1, 2, 13, 0)
Reading File Name    : '/home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg/3B-HHR.MS.MRG.3IMERG.20180102-S123000-E125959.0750.V06B.HDF5'
Data Shape and Type  :(1800, 3600), dtype('float64')
Running Accum (hr)   : 0.5
======================
Processing Valid Time: datetime.datetime(2018, 1, 2, 12, 30)
Reading File Name    : '/home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg/3B-HHR.MS.MRG.3IMERG.20180102-S120000-E122959.0720.V06B.HDF5'
Data Shape and Type  :(1800, 3600), dtype('float64')
Running Accum (hr)   : 1.0
======================
Processing Valid Time: datetime.datetime(2018, 1, 2, 12, 0)
Reading File Name    : '/home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg/3B-HHR.MS.MRG.3IMERG.20180102-S113000-E115959.0690.V06B.HDF5'
Data Shape and Type  :(1800, 3600), dtype('float64')
Running Accum (hr)   : 1.5
======================
Processing Valid Time: datetime.datetime(2018, 1, 2, 11, 30)
Reading File Name    : '/home/lzjiang/pkgs/METplus-6.0.0_DATA/met_test/new/imerg/3B-HHR.MS.MRG.3IMERG.20180102-S110000-E112959.0660.V06B.HDF5'
Data Shape and Type  :(1800, 3600), dtype('float64')
Running Accum (hr)   : 2.0
======================
Valid Data    : 4302756 of 6480000 (66.0%)
Range of Data : 0.0 to 60.941396713256836
Attributes    : {'name': 'IRprecipitation', 'long_name': 'IRprecipitation', 'level': 'Surface', 'units': 'mm', 'grid': {'name': 'IMERG-Grid', 'type': 'LatLon', 'lat_ll': -89.94999694824219, 'lon_ll': -179.9499969482422, 'delta_lat': 0.1, 'delta_lon': 0.1, 'Nlat': 1800, 'Nlon': 3600}, 'valid': '20180102_130000', 'init': '20180102_130000', 'lead': '00', 'accum': '02'}
DEBUG 3: Initializing MET compile time python instance: /usr/local/python/3.12.0/gcc/13.2.0/bin/python3
DEBUG 4: Reading temporary Python dataplane file: /tmp/tmp_met_nc_3328041_0
Traceback (most recent call last):
  File "/usr/local/met/11.1.0/gcc/13.2.0/share/met/python/met/dataplane.py", line 3, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
WARNING: 
WARNING: tmp_nc_dataplane() -> an error occurred importing module "met.dataplane"
WARNING: 
ERROR  : 
ERROR  : get_field() -> can't get data plane from file "PYTHON_NUMPY"
ERROR  :
@jiangleads
Copy link
Author

jiangleads commented Jan 17, 2025 •
edited
Loading

Update Jan 17 2025

After changed into a Apptainer and using user customed python interpreter, the problem solved. Strange question.
here are my commands history.

git clone https://github.com/apptainer/apptainer.git
cd apptainer
export APPTAINER_INSTALL_DIR=$HOME/apptainer
mkdir -p $APPTAINER_INSTALL_DIR
./mconfig --prefix=$APPTAINER_INSTALL_DIR
cd builddir/
make && make install
export PATH=$APPTAINER_INSTALL_DIR/bin:$PATH
export LD_LIBRARY_PATH=$APPTAINER_INSTALL_DIR/lib:$LD_LIBRARY_PATH

apptainer pull docker://dtcenter/metplus-dev:main_v6.0

cd ~/pkgs/apptainer/builddir/
mkdir input_data output_data

apptainer build --sandbox my_sandbox metplus-dev_main_v6.0.sif 

export APPTAINER_BIND="./input_data:/data/input/METplus_Data,./output_data:/data/output"

vi PCPCombine_python_embedding.conf

apptainer exec my_sandbox /metplus/METplus/ush/run_metplus.py PCPCombine_python_embedding.conf

output

(myenv) lzjiang@nimbus builddir $ apptainer exec --writable my_sandbox /metplus/METplus/ush/run_metplus.py PCPCombine_python_embedding.conf 
Starting METplus v6.0.0
Parsing config file: /metplus/METplus/metplus/parm/metplus_config/defaults.conf
Parsing config file: /home/lzjiang/pkgs/apptainer/builddir/PCPCombine_python_embedding.conf
Logging to /data/output/logs/metplus.log.20250117212434
01/17 21:24:34.704Z metplus.f6bc4e5c INFO: Running METplus v6.0.0 as user lzjiang(1196101016) with command: /metplus/METplus/ush/run_metplus.py PCPCombine_python_embedding.conf
01/17 21:24:34.705Z metplus.f6bc4e5c INFO: Log file: /data/output/logs/metplus.log.20250117212434
01/17 21:24:34.705Z metplus.f6bc4e5c INFO: METplus Base: /metplus/METplus
01/17 21:24:34.705Z metplus.f6bc4e5c INFO: Final Conf: /data/output/metplus_final.conf.20250117212434
01/17 21:24:34.705Z metplus.f6bc4e5c INFO: Config Input: /metplus/METplus/metplus/parm/metplus_config/defaults.conf
01/17 21:24:34.705Z metplus.f6bc4e5c INFO: Config Input: /home/lzjiang/pkgs/apptainer/builddir/PCPCombine_python_embedding.conf
01/17 21:24:34.709Z metplus.f6bc4e5c INFO: Running wrapper: PCPCombine
01/17 21:24:34.711Z metplus.f6bc4e5c INFO: ****************************************
01/17 21:24:34.711Z metplus.f6bc4e5c INFO: * Running METplus PCPCombineWrapper
01/17 21:24:34.711Z metplus.f6bc4e5c INFO: *  at valid time: 2023-07-28 11:00
01/17 21:24:34.711Z metplus.f6bc4e5c INFO: ****************************************
01/17 21:24:34.711Z metplus.f6bc4e5c INFO: Processing OBS data
01/17 21:25:49.203Z metplus.f6bc4e5c INFO: Processing forecast lead 0 hours
01/17 21:25:49.204Z metplus.f6bc4e5c (pcp_combine_wrapper.py:608) WARNING: OBS_PCP_COMBINE_LOOKBACK is not set. Using A01 from OBS_VAR1_LEVELS. It is recommended that you explicitly set the output accumulation.
01/17 21:25:49.205Z metplus.f6bc4e5c (pcp_combine_wrapper.py:308) WARNING: OBS_PCP_COMBINE_OUTPUT_NAME is not set. Using APCP from OBS_VAR1_NAME.
01/17 21:25:49.206Z metplus.f6bc4e5c INFO: COMMAND: /usr/local/bin/pcp_combine -add PYTHON_NUMPY 'name="/metplus/METplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /data/input/METplus_Data/gpmdata precipitation 2023072904 01";' -name "APCP" 

.....

/data/output/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20230729_0400_A01h -v 2
01/17 21:25:54.421Z metplus.f6bc4e5c INFO: Finished running /usr/local/bin/pcp_combine - took 0:00:05.210432
01/17 21:25:54.421Z metplus.f6bc4e5c INFO: ****************************************
01/17 21:25:54.421Z metplus.f6bc4e5c INFO: * Running METplus PCPCombineWrapper
01/17 21:25:54.421Z metplus.f6bc4e5c INFO: *  at valid time: 2023-07-29 05:00
01/17 21:25:54.421Z metplus.f6bc4e5c INFO: ****************************************
01/17 21:25:54.421Z metplus.f6bc4e5c INFO: Processing OBS data
01/17 21:25:54.422Z metplus.f6bc4e5c INFO: Processing forecast lead 0 hours
01/17 21:25:54.422Z metplus.f6bc4e5c (pcp_combine_wrapper.py:608) WARNING: OBS_PCP_COMBINE_LOOKBACK is not set. Using A01 from OBS_VAR1_LEVELS. It is recommended that you explicitly set the output accumulation.
01/17 21:25:54.423Z metplus.f6bc4e5c (pcp_combine_wrapper.py:308) WARNING: OBS_PCP_COMBINE_OUTPUT_NAME is not set. Using APCP from OBS_VAR1_NAME.
01/17 21:25:54.425Z metplus.f6bc4e5c INFO: COMMAND: /usr/local/bin/pcp_combine -add PYTHON_NUMPY 'name="/metplus/METplus/metplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py /data/input/METplus_Data/gpmdata precipitation 2023072905 01";' -name "APCP" /data/output/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed/IMERG.20230729_0500_A01h -v 2
01/17 21:25:59.641Z metplus.f6bc4e5c INFO: Finished running /usr/local/bin/pcp_combine - took 0:00:05.210739
01/17 21:25:59.646Z metplus.f6bc4e5c INFO: Overwrite final conf here: /data/output/metplus_final.conf.20250117212434
01/17 21:25:59.649Z metplus.f6bc4e5c INFO: METplus took 0:01:25.649378 to run.
01/17 21:25:59.649Z metplus.f6bc4e5c INFO: Check the log file for more information: /data/output/logs/metplus.log.20250117212434
01/17 21:25:59.649Z metplus.f6bc4e5c INFO: METplus has successfully finished running as user lzjiang(1196101016).

PS: A customized version of Python outside the container was used

PCPCombine_python_embedding.conf file

[config]

# Documentation for this use case can be found at
# https://metplus.readthedocs.io/en/latest/generated/met_tool_wrapper/PCPCombine/PCPCombine_python_embedding.html

# For additional information, please see the METplus Users Guide.
# https://metplus.readthedocs.io/en/latest/Users_Guide

###
# Processes to run
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list
###

PROCESS_LIST = PcpCombine


###
# Time Info
# LOOP_BY options are INIT, VALID, RETRO, and REALTIME
# If set to INIT or RETRO:
#   INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set
# If set to VALID or REALTIME:
#   VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set
# LEAD_SEQ is the list of forecast leads to process
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control
###

LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d%H%M
VALID_BEG=202307281100
VALID_END=202307290500
VALID_INCREMENT=1H

LEAD_SEQ = 0


###
# File I/O
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info
###

OBS_PCP_COMBINE_RUN = True

#OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}/met_test/new/imerg
OBS_PCP_COMBINE_INPUT_DIR = {INPUT_BASE}/gpmdata
OBS_PCP_COMBINE_INPUT_TEMPLATE = PYTHON_NUMPY

OBS_PCP_COMBINE_OUTPUT_DIR = {OUTPUT_BASE}/met_tool_wrapper/PCPCombine/PCPCombine_combine_py_embed
OBS_PCP_COMBINE_OUTPUT_TEMPLATE = IMERG.{valid?fmt=%Y%m%d_%H%M}_A{level?fmt=%2H}h


###
# PCPCombine Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#pcpcombine
###

#LOG_PCP_COMBINE_VERBOSITY = 2

OBS_PCP_COMBINE_METHOD = ADD

OBS_VAR1_NAME = APCP
#OBS_VAR1_LEVELS = A06
OBS_VAR1_LEVELS = A01

OBS_PCP_COMBINE_INPUT_DATATYPE = PYTHON_NUMPY
OBS_PCP_COMBINE_INPUT_ACCUMS = 01
OBS_PCP_COMBINE_INPUT_NAMES = {PARM_BASE}/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py {OBS_PCP_COMBINE_INPUT_DIR} precipitation {valid?fmt=%Y%m%d%H} 01

#OBS_PCP_COMBINE_INPUT_THRESH =
#OBS_PCP_COMBINE_VLD_THRESH =

[user_env_vars]
# uncomment and change this to the path of a version of python that has the h5py package installed
MET_PYTHON_EXE=/home/lzjiang/miniconda3/envs/myenv/bin/python3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jiangleads