[Bug] internlm2_5-20b-q0f16-MLC模型对话胡言乱语

[l241025097]

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

1. 编译
   mlc_llm compile /path/to/internlm2_5-20b-q0f16-MLC/mlc-chat-config.json
   --device metal
   --overrides "max_batch_size=128;prefill_chunk_size=4096"
   -o /path/to/internlm2_5-20b-q0f16-MLC/libs/internlm2_5-20b-q0f16-MLC-metal.so

2. 运行
   mlc_llm chat
   /path/to/internlm2_5-20b-q0f16-MLC
   --model-lib /path/to/internlm2_5-20b-q0f16-MLC/libs/internlm2_5-20b-q0f16-MLC-metal.so

Expected behavior

能够正常回复“你好”。

Environment

Platform: arm64
Operating system: MacOS 15.1.1
Device: metal
How you installed MLC-LLM : pip, source
How you installed TVM-Unity: pip, source
Python version: 3.11
TVM Unity Hash Tag:
python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))"
[11:43:36] /Users/lyn/Documents/python/learn/mlc_llm/modules/tvm-unity/src/target/llvm/llvm_instance.cc:226: Error: Using LLVM 19.1.4 with -mcpu=apple-latest is not valid in -mtriple=arm64-apple-macos, using default -mcpu=generic
[11:43:36] /Users/lyn/Documents/python/learn/mlc_llm/modules/tvm-unity/src/target/llvm/llvm_instance.cc:226: Error: Using LLVM 19.1.4 with -mcpu=apple-latest is not valid in -mtriple=arm64-apple-macos, using default -mcpu=generic
[11:43:36] /Users/lyn/Documents/python/learn/mlc_llm/modules/tvm-unity/src/target/llvm/llvm_instance.cc:226: Error: Using LLVM 19.1.4 with -mcpu=apple-latest is not valid in -mtriple=arm64-apple-macos, using default -mcpu=generic
CLML Target Version: 3
USE_NVTX: OFF
USE_GTEST: AUTO
SUMMARIZE: OFF
TVM_DEBUG_WITH_ABI_CHANGE: OFF
USE_IOS_RPC: OFF
USE_MSC: OFF
USE_ETHOSU: OFF
CUDA_VERSION: NOT-FOUND
USE_LIBBACKTRACE: AUTO
DLPACK_PATH: 3rdparty/dlpack/include
USE_TENSORRT_CODEGEN: OFF
USE_OPENCL_EXTN_QCOM: NOT-FOUND
USE_TARGET_ONNX: OFF
USE_AOT_EXECUTOR: ON
BUILD_DUMMY_LIBTVM: OFF
USE_CUDNN: OFF
USE_TENSORRT_RUNTIME: OFF
USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF
USE_THRUST: OFF
USE_CCACHE: AUTO
USE_ARM_COMPUTE_LIB: OFF
USE_CPP_RTVM: OFF
USE_OPENCL_GTEST: /path/to/opencl/gtest
TVM_LOG_BEFORE_THROW: OFF
USE_MKL: OFF
USE_PT_TVMDSOOP: OFF
MLIR_VERSION: NOT-FOUND
USE_CLML: OFF
USE_STACKVM_RUNTIME: OFF
USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF
ROCM_PATH: /opt/rocm
USE_DNNL: OFF
USE_MSCCL: OFF
USE_NNAPI_RUNTIME: OFF
USE_VITIS_AI: OFF
USE_MLIR: OFF
USE_RCCL: OFF
USE_LLVM: llvm-config
USE_VERILATOR: OFF
USE_TF_TVMDSOOP: OFF
USE_THREADS: ON
USE_MSVC_MT: OFF
BACKTRACE_ON_SEGFAULT: OFF
USE_GRAPH_EXECUTOR: ON
USE_NCCL: OFF
USE_ROCBLAS: OFF
GIT_COMMIT_HASH: 30f97b0df3a0078ac5e6be1e8ad50eadcc2dff43
USE_VULKAN: OFF
USE_RUST_EXT: OFF
USE_CUTLASS: OFF
USE_CPP_RPC: OFF
USE_HEXAGON: OFF
USE_CUSTOM_LOGGING: OFF
USE_UMA: OFF
USE_FALLBACK_STL_MAP: OFF
USE_SORT: ON
USE_RTTI: ON
GIT_COMMIT_TIME: 2024-11-15 11:16:12 -0500
USE_HIPBLAS: OFF
USE_HEXAGON_SDK: /path/to/sdk
USE_BLAS: none
USE_ETHOSN: OFF
USE_LIBTORCH: OFF
USE_RANDOM: ON
USE_CUDA: OFF
USE_COREML: OFF
USE_AMX: OFF
BUILD_STATIC_RUNTIME: OFF
USE_CMSISNN: OFF
USE_KHRONOS_SPIRV: OFF
USE_CLML_GRAPH_EXECUTOR: OFF
USE_TFLITE: OFF
USE_HEXAGON_GTEST: /path/to/hexagon/gtest
PICOJSON_PATH: 3rdparty/picojson
USE_OPENCL_ENABLE_HOST_PTR: OFF
INSTALL_DEV: OFF
USE_PROFILER: ON
USE_NNPACK: OFF
LLVM_VERSION: 19.1.4
USE_MRVL: OFF
USE_OPENCL: OFF
COMPILER_RT_PATH: 3rdparty/compiler-rt
USE_NNAPI_CODEGEN: OFF
RANG_PATH: 3rdparty/rang/include
USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF
USE_OPENMP: none
USE_BNNS: OFF
USE_FLASHINFER: OFF
USE_CUBLAS: OFF
USE_METAL: ON
USE_MICRO_STANDALONE_RUNTIME: OFF
USE_HEXAGON_EXTERNAL_LIBS: OFF
USE_ALTERNATIVE_LINKER: AUTO
USE_BYODT_POSIT: OFF
USE_NVSHMEM: OFF
USE_HEXAGON_RPC: OFF
USE_MICRO: OFF
DMLC_PATH: 3rdparty/dmlc-core/include
INDEX_DEFAULT_I64: ON
USE_RELAY_DEBUG: OFF
USE_RPC: ON
USE_TENSORFLOW_PATH: none
TVM_CLML_VERSION:
USE_MIOPEN: OFF
USE_ROCM: OFF
USE_PAPI: OFF
USE_CURAND: OFF
TVM_CXX_COMPILER_PATH: /Library/Developer/CommandLineTools/usr/bin/c++
HIDE_PRIVATE_SYMBOLS: ON

Additional context

[HoiM]

Same problem when I run llava-1.5-7b-hf with q4f16_0 or q4f16_1

[l241025097]

模型是mlc的huggingface仓库下载的，测试发现所有的internlm2.5模型都是这样的情况。

[MasterJH5574]

Hi @l241025097, we found that the internlm2_5-20b-q0f16-MLC model we delivered earlier is the base model and is not instruction fine-tuned and that's likely the reason why it does give reasonable response. We have re-delivered that chat model https://huggingface.co/mlc-ai/internlm2_5-20b-chat-q0f16-MLC. Would you like to try again?

[l241025097]

@MasterJH5574 非常感谢，马上去试试。

[l241025097]

internlm2_5-7b-chat模型还是没法正常工作

internlm2_5-20b-chat模型启动chat时报错：
启动代码：
mlc_llm chat /workspace/models/internlm2_5-20b-chat-q0f16-MLC --overrides "tensor_parallel_shards=2;prefill_chunk_size=1024"
@MasterJH5574

[MasterJH5574]

@l241025097 The first issue is due to a bug, which we fixed in #3058. Please upgrade the nightly package in 20h and try again.

[l241025097]

我重新编译安装了新的版本，不知道版本是否正确？ 但是执行mlc_llm chat internlm2_5-20b-chat-q0f16-MLC时失败，报：Cannot find any tokenizer under /xxx 错误。