Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TypeError: unhashable type: 'list' in Loader #391

Open
smeisler opened this issue Dec 3, 2022 · 6 comments
Open

TypeError: unhashable type: 'list' in Loader #391

smeisler opened this issue Dec 3, 2022 · 6 comments

Comments

@smeisler
Copy link
Contributor

smeisler commented Dec 3, 2022

Environment

  • fitlins version: 0.11.0

Expected Behavior

The model should be returning run-level and session-average contrast maps for a set of conditions in each subject. In some subjects, the model runs fine, but in others, I get the error described below. Nothing obvious seems to differentiate the subjects who do and do not get this error. Of note, all subjects come from the same dataset and were preprocessed with the same fMRIPrep command / version without errors.

Observed Behavior

221203-09:59:04,271 nipype.workflow INFO:
	 [Node] Setting-up "fitlins_wf.loader" in "/om2/scratch/tmp/smeisler/summit_fmriprep/sub-summit035/fitlins_wf/loader".
221203-09:59:04,345 nipype.workflow INFO:
	 [Node] Executing "loader" <fitlins.interfaces.bids.LoadBIDSModel>
/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py:707: UserWarning: Use 'Contrasts' not 'Conditions' or 'ConditionList' to specifyDummyContrasts. Renaming to 'Contrasts' for now.
  warnings.warn(
221203-09:59:18,686 nipype.workflow INFO:
	 [Node] Finished "loader", elapsed time 14.320313s.
221203-09:59:18,687 nipype.workflow WARNING:
	 Storing result file without outputs
221203-09:59:18,697 nipype.workflow WARNING:
	 [Node] Error on "fitlins_wf.loader" (/om2/scratch/tmp/smeisler/summit_fmriprep/sub-summit035/fitlins_wf/loader)
221203-09:59:19,718 nipype.workflow ERROR:
	 Node loader failed to run on host node041.
221203-09:59:19,718 nipype.workflow ERROR:
	 Saving crash info to /om2/scratch/tmp/smeisler/summit_fmriprep/sub-summit035/crash-20221203-095919-smeisler-loader-4e02757b-5af6-4ba6-ac7e-a1c5ed601874.txt
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node loader.

Traceback:
	Traceback (most recent call last):
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 248, in _run_interface
	    self._results['all_specs'] = self._load_graph(runtime, graph)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 282, in _load_graph
	    self._load_graph(runtime, graph, child.destination, outputs, **child.filter)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 256, in _load_graph
	    specs = node.run(inputs, group_by=node.group_by, **filters)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py", line 457, in run
	    groups = self._build_groups(all_objects, group_by)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py", line 349, in _build_groups
	    metadata_vars = reduce(pd.DataFrame.merge, collections)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/frame.py", line 9190, in merge
	    return merge(
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 121, in merge
	    return op.get_result()
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 715, in get_result
	    join_index, left_indexer, right_indexer = self._get_join_info()
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 966, in _get_join_info
	    (left_indexer, right_indexer) = self._get_join_indexers()
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 940, in _get_join_indexers
	    return get_join_indexers(
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 1481, in get_join_indexers
	    zipped = zip(*mapped)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 1478, in <genexpr>
	    _factorize_keys(left_keys[n], right_keys[n], sort=sort, how=how)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 2186, in _factorize_keys
	    llab = rizer.factorize(lk)
	  File "pandas/_libs/hashtable.pyx", line 111, in pandas._libs.hashtable.ObjectFactorizer.factorize
	  File "pandas/_libs/hashtable_class_helper.pxi", line 5404, in pandas._libs.hashtable.PyObjectHashTable.get_labels
	  File "pandas/_libs/hashtable_class_helper.pxi", line 5310, in pandas._libs.hashtable.PyObjectHashTable._unique
	TypeError: unhashable type: 'list'

Model

{
   "Name":"lettersound-model-6MP5ACompCor",
   "BIDSModelVersion":"1.0.0",
   "Description":"",
   "Input":{
      "task":["lettersound"]
   },
   "Nodes":[
      {
         "Level":"run",
         "Name":"runLettersound6MP5ACompCor",
         "GroupBy":[
            "subject",
	    "session",
	    "run"
         ],
	"Transformations":{
            "Transformer":"pybids-transforms-v1",
            "Instructions":[
               {
                  "Name":"Factor",
                  "Input":[
                     "trial_type"
                  ]
               },
               {
                  "Name":"Convolve",
                  "Input":[
                     "trial_type.1",
                     "trial_type.2",
                     "trial_type.3",
                     "trial_type.4"
                  ],
                  "Model":"spm"
               }
            ]
         },
         "Model":{
            "X":[
               "trial_type.1",
               "trial_type.2",
               "trial_type.3",
               "trial_type.4",
	       "non_steady_state*",
	       "trans_x",
               "trans_y",
               "trans_z",
               "rot_x",
               "rot_y",
               "rot_z",
	       "a_comp_cor_00",
	       "a_comp_cor_01",
	       "a_comp_cor_02",
	       "a_comp_cor_03",
	       "a_comp_cor_04",
	       "cosine*",
               1
            ],
           "Type":"glm"
         },
         "Contrasts":[
            {
               "Name":"congruent_gt_incongruent",
               "ConditionList":[
                  "trial_type.3",
                  "trial_type.4"
               ],
               "Weights":[
                  1,
                  -1
               ],
               "Test":"t"
            },
	    {
               "Name":"look_gt_listen",
               "ConditionList":[
                  "trial_type.1",
                  "trial_type.2"
               ],
               "Weights":[
                  1,
                  -1
               ],
               "Test":"t"
            },
	    {
               "Name":"multi_gt_look",
               "ConditionList":[
	          "trial_type.3",
                  "trial_type.4",
		  "trial_type.1"
               ],
               "Weights":[
                  0.5,
		  0.5,
                  -1
               ],
               "Test":"t"
            },
	    {
               "Name":"multi_gt_listen",
               "ConditionList":[
	          "trial_type.3",
                  "trial_type.4",
		  "trial_type.2"
               ],
               "Weights":[
                  0.5,
		  0.5,
                  -1
               ],
               "Test":"t"
            }
         ]
      },
     {
    "Level":"session",
	"Name":"sessionLettersound6MP5ACompCor",
	"GroupBy":["session",
		"subject",
		"contrast"],
	"Model":{
            "X":[1],
            "Type":"meta"
            },
	"DummyContrasts": {"Test": "t"}
     	}
   ],
   "Edges":[
	{
      "Source": "runLettersound6MP5ACompCor",
      "Destination": "sessionLettersound6MP5ACompCor"
    }
   ]
}

Steps to Reproduce

  1. Run fMRIPrep v22.0.0 with --output-spaces MNI152NLin2009cAsym:res-2 --cifti-output 91k flags
  2. Run above model with fitlins 0.11.0 using --space $space --drop-missing -s $smoothing --desc-label "$desc_label" --estimator nilearn flags
@effigies
Copy link
Collaborator

effigies commented Dec 3, 2022

Probably list metadata. It looks like it's at the run level. Are you comfortable running the debug shell? You should be able to step up the call stack until you see the variables that are trying to be merged in get_collections.

@smeisler
Copy link
Contributor Author

smeisler commented Dec 5, 2022

I am not sure how to do this, is there documentation somewhere?

@effigies
Copy link
Collaborator

effigies commented Dec 5, 2022

Bad news is that your crashfile format is txt, not pklz, so you just get that traceback instead of something executable with nipypecli. The easiest thing to do is to cd to that directory (/om2/scratch/tmp/smeisler/summit_fmriprep/sub-summit035/fitlins_wf/loader), singularity shell into the container, open up a Python shell, then do something like:

>>> import pdb
>>> from fitlins.interfaces.bids import LoadBIDSModel
>>> lbm = LoadBIDSModel()
>>> # Here you'll be going off of what's in _report/report.rst
>>> lbm.inputs.x = X
>>> lbm.inputs.y = Y
>>> ... 
>>> pdb.run("lbm.run()")

@smeisler
Copy link
Contributor Author

smeisler commented Dec 7, 2022 •
edited
Loading

Here is what I tried:

  • Open the shell's neuro environment Python /opt/miniconda-latest/envs/neuro/bin/python
  • Import the modules successfully and create the lbm object as described
  • Define model variables directly copied from the report RST file
model={'Name': 'lettersound-model-6MP5ACompCor', 'BIDSModelVersion': '1.0.0', 'Description': '', 'Input': {'task': ['lettersound']}, 'Nodes': [{'Level': 'run', 'Name': 'runLettersound6MP5ACompCor', 'GroupBy': ['subject', 'session', 'run'], 'Transformations': {'Transformer': 'pybids-transforms-v1', 'Instructions': [{'Name': 'Factor', 'Input': ['trial_type']}, {'Name': 'Convolve', 'Input': ['trial_type.1', 'trial_type.2', 'trial_type.3', 'trial_type.4'], 'Model': 'spm'}]}, 'Model': {'X': ['trial_type.1', 'trial_type.2', 'trial_type.3', 'trial_type.4', 'non_steady_state*', 'trans_x', 'trans_y', 'trans_z', 'rot_x', 'rot_y', 'rot_z', 'a_comp_cor_*', 'cosine*', 1], 'Type': 'glm'}, 'DummyContrasts': {'Conditions': ['trial_type.1', 'trial_type.2'], 'Test': 't'}, 'Contrasts': [{'Name': 'congruent_gt_incongruent', 'ConditionList': ['trial_type.3', 'trial_type.4'], 'Weights': [1, -1], 'Test': 't'}, {'Name': 'look_gt_listen', 'ConditionList': ['trial_type.1', 'trial_type.2'], 'Weights': [1, -1], 'Test': 't'}, {'Name': 'multi_gt_look', 'ConditionList': ['trial_type.3', 'trial_type.4', 'trial_type.1'], 'Weights': [0.5, 0.5, -1], 'Test': 't'}, {'Name': 'multi_gt_listen', 'ConditionList': ['trial_type.3', 'trial_type.4', 'trial_type.2'], 'Weights': [0.5, 0.5, -1], 'Test': 't'}]}, {'Level': 'session', 'Name': 'sessionLettersound6MP5ACompCor', 'GroupBy': ['session', 'subject', 'contrast'], 'Model': {'X': [1], 'Type': 'meta'}, 'DummyContrasts': {'Test': 't'}}], 'Edges': [{'Source': 'runLettersound6MP5ACompCor', 'Destination': 'sessionLettersound6MP5ACompCor'}]}

selectors={'desc': None, 'space': 'fsLR', 'subject': ['summit035']}

database_path='/om2/scratch/tmp/smeisler/summit_fmriprep/sub-summit035/dbcache'
  • Add these to the LBM object (lbm.inputs has no x or y field)
lbm.inputs.model=model
lbm.inputs.selectors=selectors
lbm.inputs.database_path=database_path
  • Run the model with pdb.run("lbm.run()")
>>> pdb.run("lbm.run()")
> <string>(1)<module>()
(Pdb)

When I run continue while in the debugger, I get the original error.
When I run lbm.run() outside the debugger, I get the original error.
When I try to step through the function line by line, here is where the original error came up:

> <string>(1)<module>()
(Pdb) step
--Call--
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(361)run()
-> def run(self, cwd=None, ignore_exception=None, **inputs):
(Pdb) step
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(379)run()
-> rtc = RuntimeContext(
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(380)run()
-> resource_monitor=config.resource_monitor and self.resource_monitor,
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(382)run()
-> if ignore_exception is not None
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(381)run()
-> ignore_exception=ignore_exception
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(383)run()
-> else self.ignore_exception,
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(379)run()
-> rtc = RuntimeContext(
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(386)run()
-> with indirectory(cwd or os.getcwd()):
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(387)run()
-> self.inputs.trait_set(**inputs)
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(388)run()
-> self._check_mandatory_inputs()
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(389)run()
-> self._check_version_requirements(self.inputs)
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(391)run()
-> with rtc(self, cwd=cwd, redirect_x=self._redirect_x) as runtime:
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(394)run()
-> inputs = self.inputs.get_traitsfree()
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(395)run()
-> outputs = None
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(397)run()
-> runtime = self._pre_run_hook(runtime)
(Pdb) next
> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py(398)run()
-> runtime = self._run_interface(runtime)
(Pdb) next
TypeError: unhashable type: 'list'

@effigies
Copy link
Collaborator

effigies commented Dec 7, 2022

So when you get the error with continue, you don't get something like:

> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/reshape/merge.py(2186)_factorize_keys()
-> llab = rizer.factorize(lk)
(Pdb)

If you do see that, then you should be able to go up until you see

> /opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py(349)_build_groups()
-> metadata_vars = reduce(pd.DataFrame.merge, collections)
(pdb)

And look at collections.

@adelavega
Copy link
Collaborator

What is happening is that pandas is trying to merge the meta-data of various "groups" (possibly in this case its runs). However, if the metadata includes nested columns, such that the entry is a list (and the dtype of the column is object), then the merge won't work.

So it's hard to say specifically what is causing the issue here, but lists as meta-data is BIDS valid. This seems related to: bids-standard/pybids#683

Although now the issue is cropping up w/ meta-data in pandas dfs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@effigies @adelavega @smeisler