ImportError: /root/anaconda3/envs/test2/lib/python3.10/site-packages/awq_inference_engine.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

[Jackson-cj]

I try to use TheBloke/OpenHermes-2.5-Mistral-7B-AWQ,but to happen the error,ask for help,thanks.

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0
PyTorch version: 2.4.1

[Jackson-cj]

use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
Traceback (most recent call last):
File "/home/PAPERCLIP-Hubble-main/summarize.py", line 15, in
from awq import AutoAWQForCausalLM
File "/root/anaconda3/envs/test2/lib/python3.10/site-packages/awq/init.py", line 2, in
from awq.models.auto import AutoAWQForCausalLM
File "/root/anaconda3/envs/test2/lib/python3.10/site-packages/awq/models/init.py", line 1, in
from .mpt import MptAWQForCausalLM
File "/root/anaconda3/envs/test2/lib/python3.10/site-packages/awq/models/mpt.py", line 1, in
from .base import BaseAWQForCausalLM
File "/root/anaconda3/envs/test2/lib/python3.10/site-packages/awq/models/base.py", line 16, in
from awq.modules.linear import WQLinear_GEMM, WQLinear_GEMV
File "/root/anaconda3/envs/test2/lib/python3.10/site-packages/awq/modules/linear.py", line 4, in
import awq_inference_engine # with CUDA kernels
ImportError: /root/anaconda3/envs/test2/lib/python3.10/site-packages/awq_inference_engine.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

(test2) root@6a60bca8419a:/home/PAPERCLIP-Hubble-main# python
Python 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import awp
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'awp'
import awq
/root/anaconda3/envs/test2/lib/python3.10/site-packages/transformers/utils/generic.py:441: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/root/anaconda3/envs/test2/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/root/anaconda3/envs/test2/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
Traceback (most recent call last):
File "", line 1, in
File "/root/anaconda3/envs/test2/lib/python3.10/site-packages/awq/init.py", line