Issue with Gemma 7b

[CraterCrater]

Describe the bug

I tried loading Gemma 7b, and finally got the following long string of random words instead of a response after the model was loaded.

To Reproduce

Steps to reproduce the behavior:
start AI Playground
Go to Answer tab
Select Gemma 7b model from the list of models,
type in a query, (hello,)
get the following long string of nonsense

Expected behavior

I expected to get a response from the LLM

Screenshots

[ai-backend]: 2024-11-16 18:56:46,907 - INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.

• Running on http://127.0.0.1:59000
  2024-11-16 18:56:46,907 - INFO - Press CTRL+C to quit

[ai-backend]: 2024-11-16 18:56:47,162 - INFO - 127.0.0.1 - - [16/Nov/2024 18:56:47] "POST /api/init HTTP/1.1" 200 -

[ai-backend]: 2024-11-16 18:56:47,167 - INFO - 127.0.0.1 - - [16/Nov/2024 18:56:47] "POST /api/getGraphics HTTP/1.1" 200 -

[ai-backend]: 2024-11-16 18:57:03,637 - INFO - 127.0.0.1 - - [16/Nov/2024 18:57:03] "POST /api/checkModelExist HTTP/1.1" 200 -

[ai-backend]: 2024-11-16 18:57:03,757 - INFO - 127.0.0.1 - - [16/Nov/2024 18:57:03] "POST /api/llm/chat HTTP/1.1" 200 -

[ai-backend]: Gemma's activation function should be approximate GeLU and not exact GeLU.
Changing the activation function to gelu_pytorch_tanh.if you want to use the legacy gelu, edit the model.config to set hidden_activation=gelu instead of hidden_act. See huggingface/transformers#29402 for more details.

Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|ΓûêΓûêΓûî | 1/4 [00:01<00:04, 1.37s/it]
Loading checkpoint shards: 50%|ΓûêΓûêΓûêΓûêΓûê | 2/4 [00:02<00:02, 1.50s/it]
Loading checkpoint shards: 75%|ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûî | 3/4 [00:04<00:01, 1.53s/it]
Loading checkpoint shards: 100%|ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûê| 4/4 [00:05<00:00, 1.34s/it]

[ai-backend]: 2024-11-16 18:57:09,288 - INFO - Converting the current model to sym_int4 format......

[ai-backend]: C:\Users\rober\AppData\Local\Programs\AI Playground\resources\env\Lib\site-packages\torch\nn\init.py:452: UserWarning: Initializing zero-element tensors is a no-op
warnings.warn("Initializing zero-element tensors is a no-op")

[ai-backend]: 2024-11-16 18:57:21,046 - INFO - got prompt: [{'question': 'hello', 'answer': ''}]

[ai-backend]:
No chat template is defined for this tokenizer - using a default chat template that implements the ChatML format (without BOS/EOS tokens!). If the default is not appropriate for your model, please set tokenizer.chat_template to an appropriate template. See https://huggingface.co/docs/transformers/main/chat_templating for more information.

[ai-backend]: C:\Users\rober\AppData\Local\Programs\AI Playground\resources\env\Lib\site-packages\intel_extension_for_pytorch\xpu\amp_init_.py:14: UserWarning: is_autocast_xpu_enabled is deprecated. Please use torch.is_autocast_enabled('xpu') instead.
warnings.warn(

[ai-backend]: 2024-11-16 18:58:02,951 - INFO -
----------inference finish----------
num_tokens : 1025
total_time : 41.9050 s
overall tokens/s : 24.4601
2nd+ token/s : 25.4566
first_token_latency : 1.6797 s
after_token_latency : 0.0393 s

[ai-backend]: load llm model google/gemma-7b_tmp finish. cost 15.302s
nadru ofre suscepti apprehen eto suscepti apprehen eto scrat suscepti scrat encomp scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder
[ai-backend]: 2024-11-16 18:58:03,096 - INFO - got prompt: [{'question': 'Create me a short descriptive title for the following conversation in a maximum of 20 characters. Don't use unnecessary words like 'Conversation about': \n\n[{"question":"hello","answer":" nadru ofre suscepti apprehen eto suscepti apprehen eto scrat suscepti scrat encomp scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat scrat syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp syp inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder inder"}]', 'answer': ''}]

Environment (please complete the following information):

OS: Windows 11 Pro
GPU: Intel Arc A770 16G
CPU: i9-10850K
Version: v1.22.1-beta

Additional context

Add any other context about the problem here.

[dochlme]

i am having the same issue.

[otichy]

Same issue here.

[Jesselrj]

Same issue