Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Fix _pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed (VS code debugger only?) #235

Open
tomvothecoder opened this issue Dec 18, 2023 · 0 comments
Open

[Bug]: Fix _pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed (VS code debugger only?) #235

tomvothecoder opened this issue Dec 18, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@tomvothecoder
Copy link
Collaborator

tomvothecoder commented Dec 18, 2023 •
edited
Loading

What happened?

The multiprocessing API can't pickle the _cmor.CMORError module.

What did you expect to happen? Are there are possible answers you came across?

I very vaguely recall getting past this issue before. It might be related to VS Code's debugger/interactive console and requires installing another dependency fix (Jupyter dependency?).

We need to figure out how to pickle CMOR errors with multiprocessing. This error does not appear when using --serial.

Minimal Complete Verifiable Example (MVCE)

from e3sm_to_cmip.__main__ import E3SMtoCMIP

cmortables_dir = "/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables"
input_dir = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test_unified_1.9.2rc1/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr"
dest_cmip = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test_unified_1.9.2rc1/v2.LR.historical_0201/post/atm/180x360_aave/cmip_ts/monthly"
output_path = "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/qa/12-18-23-zppy-seg"

args = [
    "--output-path",
    output_path,
    "--var-list",
    "pr, tas, rsds, rlds, rsus",
    "--realm",
    "atm",
    "--input-path",
    input_dir,
    "--user-metadata",
    "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/qa/12-18-23-zppy-seg/default_metadata.json",
    "--num-proc",
    "12",
    "--tables-path",
    cmortables_dir,
    "--serial",
]

run = E3SMtoCMIP(args)

run.run()

Relevant log output

2023-12-18 18:12:43,900 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-12-18 18:12:43,900 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-12-18 18:12:43,900_900:INFO:__init__:--------------------------------------
2023-12-18 18:12:43,901 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-12-18 18:12:43,901 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-12-18 18:12:43,901_901:INFO:__init__:| E3SM to CMIP Configuration
2023-12-18 18:12:43,902 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-12-18 18:12:43,902 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-12-18 18:12:43,902_902:INFO:__init__:--------------------------------------
2023-12-18 18:12:43,903 [INFO]: __main__.py(__init__:145) >>     * var_list='['pr', 'tas', 'rsds', 'rlds', 'rsus']'
2023-12-18 18:12:43,903 [INFO]: __main__.py(__init__:145) >>     * var_list='['pr', 'tas', 'rsds', 'rlds', 'rsus']'
2023-12-18 18:12:43,903_903:INFO:__init__:    * var_list='['pr', 'tas', 'rsds', 'rlds', 'rsus']'
2023-12-18 18:12:43,904 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test_unified_1.9.2rc1/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr'
2023-12-18 18:12:43,904 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test_unified_1.9.2rc1/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr'
2023-12-18 18:12:43,904_904:INFO:__init__:    * input_path='/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test_unified_1.9.2rc1/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr'
2023-12-18 18:12:43,905 [INFO]: __main__.py(__init__:147) >>     * output_path='/home/ac.tvo/E3SM-Project/e3sm_to_cmip/qa/12-18-23-zppy-seg'
2023-12-18 18:12:43,905 [INFO]: __main__.py(__init__:147) >>     * output_path='/home/ac.tvo/E3SM-Project/e3sm_to_cmip/qa/12-18-23-zppy-seg'
2023-12-18 18:12:43,905_905:INFO:__init__:    * output_path='/home/ac.tvo/E3SM-Project/e3sm_to_cmip/qa/12-18-23-zppy-seg'
2023-12-18 18:12:43,905 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-12-18 18:12:43,905 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-12-18 18:12:43,905_905:INFO:__init__:    * precheck_path='None'
2023-12-18 18:12:43,906 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-12-18 18:12:43,906 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-12-18 18:12:43,906_906:INFO:__init__:    * freq='mon'
2023-12-18 18:12:43,907 [INFO]: __main__.py(__init__:150) >>     * realm='atm'
2023-12-18 18:12:43,907 [INFO]: __main__.py(__init__:150) >>     * realm='atm'
2023-12-18 18:12:43,907_907:INFO:__init__:    * realm='atm'
2023-12-18 18:12:43,908 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231218_181243_897215
2023-12-18 18:12:43,908 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231218_181243_897215
2023-12-18 18:12:43,908_908:INFO:__init__:    * Writing log output file to: logs/20231218_181243_897215
2023-12-18 18:12:46,784 [WARNING]: utils.py(derive_handlers:218) >> No handlers could be derived for the variables: ['rsds', 'rlds', 'rsus']. Make sure the input E3SM datasets have the variables needed derivation.
2023-12-18 18:12:46,784 [WARNING]: utils.py(derive_handlers:218) >> No handlers could be derived for the variables: ['rsds', 'rlds', 'rsus']. Make sure the input E3SM datasets have the variables needed derivation.
2023-12-18 18:12:46,784_784:WARNING:derive_handlers:No handlers could be derived for the variables: ['rsds', 'rlds', 'rsus']. Make sure the input E3SM datasets have the variables needed derivation.
2023-12-18 18:12:46,785 [INFO]: __main__.py(_get_handlers:220) >> --------------------------------------
2023-12-18 18:12:46,785 [INFO]: __main__.py(_get_handlers:220) >> --------------------------------------
2023-12-18 18:12:46,785_785:INFO:_get_handlers:--------------------------------------
2023-12-18 18:12:46,787 [INFO]: __main__.py(_get_handlers:221) >> | Derived CMIP6 Variable Handlers
2023-12-18 18:12:46,787 [INFO]: __main__.py(_get_handlers:221) >> | Derived CMIP6 Variable Handlers
2023-12-18 18:12:46,787_787:INFO:_get_handlers:| Derived CMIP6 Variable Handlers
2023-12-18 18:12:46,787 [INFO]: __main__.py(_get_handlers:222) >> --------------------------------------
2023-12-18 18:12:46,787 [INFO]: __main__.py(_get_handlers:222) >> --------------------------------------
2023-12-18 18:12:46,787_787:INFO:_get_handlers:--------------------------------------
2023-12-18 18:12:46,788 [INFO]: __main__.py(_get_handlers:224) >>     * 'pr' -> ['PRECC', 'PRECL']
2023-12-18 18:12:46,788 [INFO]: __main__.py(_get_handlers:224) >>     * 'pr' -> ['PRECC', 'PRECL']
2023-12-18 18:12:46,788_788:INFO:_get_handlers:    * 'pr' -> ['PRECC', 'PRECL']
2023-12-18 18:12:46,789 [INFO]: __main__.py(_get_handlers:224) >>     * 'tas' -> ['TREFHT']
2023-12-18 18:12:46,789 [INFO]: __main__.py(_get_handlers:224) >>     * 'tas' -> ['TREFHT']
2023-12-18 18:12:46,789_789:INFO:_get_handlers:    * 'tas' -> ['TREFHT']
2023-12-18 18:12:46,806 [INFO]: __main__.py(_run:737) >> --------------------------------------
2023-12-18 18:12:46,806 [INFO]: __main__.py(_run:737) >> --------------------------------------
2023-12-18 18:12:46,806_806:INFO:_run:--------------------------------------
2023-12-18 18:12:46,807 [INFO]: __main__.py(_run:738) >> | Running E3SM to CMIP in Parallel
2023-12-18 18:12:46,807 [INFO]: __main__.py(_run:738) >> | Running E3SM to CMIP in Parallel
2023-12-18 18:12:46,807_807:INFO:_run:| Running E3SM to CMIP in Parallel
2023-12-18 18:12:46,808 [INFO]: __main__.py(_run:739) >> --------------------------------------
2023-12-18 18:12:46,808 [INFO]: __main__.py(_run:739) >> --------------------------------------
2023-12-18 18:12:46,808_808:INFO:_run:--------------------------------------
2023-12-18 18:12:46,947 [INFO]: handler.py(cmorize:209) >> pr: Starting CMORizing
2023-12-18 18:12:46,947 [INFO]: handler.py(cmorize:209) >> pr: Starting CMORizing
  0%|          | 0/2 [00:00<?, ?it/s]2023-12-18 18:12:46,949 [INFO]: handler.py(cmorize:209) >> tas: Starting CMORizing
2023-12-18 18:12:46,947_947:INFO:cmorize:pr: Starting CMORizing
2023-12-18 18:12:46,949 [INFO]: handler.py(cmorize:209) >> tas: Starting CMORizing
2023-12-18 18:12:46,949_949:INFO:cmorize:tas: Starting CMORizing
2023-12-18 18:12:47,185 [INFO]: handler.py(_setup_cmor_module:323) >> tas: CMOR setup complete
2023-12-18 18:12:47,186 [INFO]: handler.py(_setup_cmor_module:323) >> pr: CMOR setup complete
2023-12-18 18:12:47,185 [INFO]: handler.py(_setup_cmor_module:323) >> tas: CMOR setup complete
2023-12-18 18:12:47,186 [INFO]: handler.py(_setup_cmor_module:323) >> pr: CMOR setup complete
2023-12-18 18:12:47,185_185:INFO:_setup_cmor_module:tas: CMOR setup complete
2023-12-18 18:12:47,186_186:INFO:_setup_cmor_module:pr: CMOR setup complete
2023-12-18 18:12:47,188 [INFO]: handler.py(cmorize:239) >> tas: loading E3SM variables dict_keys(['TREFHT'])
2023-12-18 18:12:47,188 [INFO]: handler.py(cmorize:239) >> pr: loading E3SM variables dict_keys(['PRECC', 'PRECL'])
2023-12-18 18:12:47,188 [INFO]: handler.py(cmorize:239) >> tas: loading E3SM variables dict_keys(['TREFHT'])
2023-12-18 18:12:47,188 [INFO]: handler.py(cmorize:239) >> pr: loading E3SM variables dict_keys(['PRECC', 'PRECL'])
2023-12-18 18:12:47,188_188:INFO:cmorize:tas: loading E3SM variables dict_keys(['TREFHT'])
2023-12-18 18:12:47,188_188:INFO:cmorize:pr: loading E3SM variables dict_keys(['PRECC', 'PRECL'])
2023-12-18 18:12:49,786 [INFO]: handler.py(cmorize:248) >> tas: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:49,786 [INFO]: handler.py(cmorize:248) >> tas: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:49,786_786:INFO:cmorize:tas: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:49,803 [INFO]: handler.py(_cmor_write:648) >> tas: time span 0.0 - 730.0
2023-12-18 18:12:49,803 [INFO]: handler.py(_cmor_write:648) >> tas: time span 0.0 - 730.0
2023-12-18 18:12:49,803_803:INFO:_cmor_write:tas: time span 0.0 - 730.0
2023-12-18 18:12:49,804 [INFO]: handler.py(_cmor_write:652) >> tas: Writing variable to file...
2023-12-18 18:12:49,804 [INFO]: handler.py(_cmor_write:652) >> tas: Writing variable to file...
2023-12-18 18:12:49,804_804:INFO:_cmor_write:tas: Writing variable to file...
2023-12-18 18:12:49,823 [INFO]: handler.py(cmorize:248) >> pr: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:49,823 [INFO]: handler.py(cmorize:248) >> pr: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:49,823_823:INFO:cmorize:pr: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:49,843 [INFO]: handler.py(_cmor_write:648) >> pr: time span 0.0 - 730.0
2023-12-18 18:12:49,843 [INFO]: handler.py(_cmor_write:648) >> pr: time span 0.0 - 730.0
2023-12-18 18:12:49,843_843:INFO:_cmor_write:pr: time span 0.0 - 730.0
2023-12-18 18:12:49,845 [INFO]: handler.py(_cmor_write:652) >> pr: Writing variable to file...
2023-12-18 18:12:49,845 [INFO]: handler.py(_cmor_write:652) >> pr: Writing variable to file...
2023-12-18 18:12:49,845_845:INFO:_cmor_write:pr: Writing variable to file...
2023-12-18 18:12:50,012 [INFO]: handler.py(cmorize:239) >> tas: loading E3SM variables dict_keys(['TREFHT'])
2023-12-18 18:12:50,012 [INFO]: handler.py(cmorize:239) >> tas: loading E3SM variables dict_keys(['TREFHT'])
2023-12-18 18:12:50,012_012:INFO:cmorize:tas: loading E3SM variables dict_keys(['TREFHT'])
2023-12-18 18:12:50,034 [INFO]: handler.py(cmorize:248) >> tas: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:50,034 [INFO]: handler.py(cmorize:248) >> tas: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:50,034_034:INFO:cmorize:tas: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:50,048 [INFO]: handler.py(_cmor_write:648) >> tas: time span 730.0 - 1460.0
2023-12-18 18:12:50,048 [INFO]: handler.py(_cmor_write:648) >> tas: time span 730.0 - 1460.0
2023-12-18 18:12:50,048_048:INFO:_cmor_write:tas: time span 730.0 - 1460.0
2023-12-18 18:12:50,049 [INFO]: handler.py(_cmor_write:652) >> tas: Writing variable to file...
2023-12-18 18:12:50,049 [INFO]: handler.py(_cmor_write:652) >> tas: Writing variable to file...
2023-12-18 18:12:50,049_049:INFO:_cmor_write:tas: Writing variable to file...
2023-12-18 18:12:50,108 [INFO]: handler.py(cmorize:239) >> pr: loading E3SM variables dict_keys(['PRECC', 'PRECL'])
2023-12-18 18:12:50,108 [INFO]: handler.py(cmorize:239) >> pr: loading E3SM variables dict_keys(['PRECC', 'PRECL'])
2023-12-18 18:12:50,108_108:INFO:cmorize:pr: loading E3SM variables dict_keys(['PRECC', 'PRECL'])
  File "/gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
          ^^^^^^^^^^^^
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
 50%|█████     | 1/2 [00:03<00:03,  3.17s/it]  File "/gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
          ^^^^^^^^^^^^
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
          ^^^^^^^^^^^^
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
100%|██████████| 2/2 [00:03<00:00,  1.58s/it]2023-12-18 18:12:50,164 [INFO]: handler.py(cmorize:248) >> pr: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:50,164 [INFO]: handler.py(cmorize:248) >> pr: creating CMOR variable with CMOR axis objects.
2023-12-18 18:12:50,164_164:INFO:cmorize:pr: creating CMOR variable with CMOR axis objects.
Process ForkProcess-1:
_cmor.CMORError: Problem with 'cmor.axis'. Please check the logfile (if defined).

During handling of the above exception, another exception occurred:

_pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/process.py", line 264, in _process_worker
    _sendback_result(result_queue, call_item.work_id, exception=exc,
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/concurrent/futures/process.py", line 221, in _sendback_result
    result_queue.put(_ResultItem(work_id, exception=exc,
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/multiprocessing/queues.py", line 371, in put
    obj = _ForkingPickler.dumps(obj)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.tvo/mambaforge/envs/e3sm_to_cmip_dev_master/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed

2023-12-18 18:12:50,283 [INFO]: __main__.py(_run_parallel:930) >> 0 of 2 handlers complete
2023-12-18 18:12:50,283 [INFO]: __main__.py(_run_parallel:930) >> 0 of 2 handlers complete
2023-12-18 18:12:50,283_283:INFO:_run_parallel:0 of 2 handlers complete
2023-12-18 18:12:50,284 [ERROR]: __main__.py(_run_parallel:934) >> tas, pr failed to complete
2023-12-18 18:12:50,284 [ERROR]: __main__.py(_run_parallel:934) >> tas, pr failed to complete
2023-12-18 18:12:50,284_284:ERROR:_run_parallel:tas, pr failed to complete
2023-12-18 18:12:50,284 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 2 handlers complete
2023-12-18 18:12:50,284 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 2 handlers complete
2023-12-18 18:12:50,284_284:ERROR:_run_parallel:0 of 2 handlers complete
A process in the process pool was terminated abruptly while the future was running or pending.
A process in the process pool was terminated abruptly while the future was running or pending.

Anything else we need to know?

No response

Environment

v1.11.0 and master branch
@tomvothecoder tomvothecoder added the bug Something isn't working label Dec 18, 2023
@tomvothecoder tomvothecoder changed the title [Bug]: Fix _pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed [Bug]: Fix _pickle.PicklingError: Can't pickle <class '_cmor.CMORError'>: import of module '_cmor' failed (VS code debugger only?) Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
E3SM to CMIP Development
Status: To do
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tomvothecoder