Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ollama with zluda on 7780hs 780m AMD #13

Open
kenhuang opened this issue Apr 22, 2024 · 5 comments
Open

ollama with zluda on 7780hs 780m AMD #13

kenhuang opened this issue Apr 22, 2024 · 5 comments

Comments

@kenhuang
Copy link

confirm working using zluda + stable diffusion via GPU here

C:\Users\ken>d:\ZLUDA\zluda.exe -- ollama.exe serve
time=2024-04-22T20:03:03.214+10:00 level=INFO source=images.go:817 msg="total blobs: 20"
time=2024-04-22T20:03:03.216+10:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-22T20:03:03.217+10:00 level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)"
time=2024-04-22T20:03:03.220+10:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\ken\AppData\Local\Temp\ollama3073857994\runners
time=2024-04-22T20:03:03.428+10:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11.3 rocm_v5.7]"
[GIN] 2024/04/22 - 20:03:08 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2024/04/22 - 20:03:08 | 200 |       535.6µs |       127.0.0.1 | POST     "/api/show"
time=2024-04-22T20:03:08.550+10:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-22T20:03:08.551+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-22T20:03:08.583+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll]"
time=2024-04-22T20:03:08.622+10:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-22T20:03:08.622+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:03:08.846+10:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.8"
time=2024-04-22T20:03:08.846+10:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-22T20:03:08.846+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-22T20:03:08.878+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll]"
time=2024-04-22T20:03:08.879+10:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-22T20:03:08.879+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:03:08.879+10:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.8"
time=2024-04-22T20:03:08.879+10:00 level=INFO source=server.go:127 msg="offload to gpu" reallayers=33 layers=33 required="5222.5 MiB" used="5222.5 MiB" available="22085.4 MiB" kv="1024.0 MiB" fulloffload="164.0 MiB" partialoffload="193.0 MiB"
time=2024-04-22T20:03:08.879+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:03:08.885+10:00 level=INFO source=server.go:264 msg="starting llama server" cmd="C:\\Users\\ken\\AppData\\Local\\Temp\\ollama3073857994\\runners\\cuda_v11.3\\ollama_llama_server.exe --model C:\\Users\\ken\\.ollama\\models\\blobs\\sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --port 63294"
time=2024-04-22T20:03:08.911+10:00 level=INFO source=server.go:389 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2603,"msg":"logging to file is disabled.","tid":"17584","timestamp":1713780189}
{"build":2679,"commit":"7593639","function":"wmain","level":"INFO","line":2820,"msg":"build info","tid":"17584","timestamp":1713780189}
{"function":"wmain","level":"INFO","line":2827,"msg":"system info","n_threads":8,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"17584","timestamp":1713780189,"total_threads":16}
llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from C:\Users\ken\.ollama\models\blobs\sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 2
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {% if messages[0]['role'] == 'system'...
llama_model_loader: - kv  22:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW)
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
  Device 0: AMD Radeon 780M Graphics [ZLUDA], compute capability 8.8, VMM: no
llm_load_tensors: ggml ctx size =    0.22 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    70.31 MiB
llm_load_tensors:      CUDA0 buffer size =  3577.56 MiB
..................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  1024.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.14 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   164.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    12.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 2
CUDA error: CUBLAS_STATUS_NOT_INITIALIZED
  current device: 0, in function cublas_handle at C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda/common.cuh:526
  cublasCreate_v2(&cublas_handles[device])
GGML_ASSERT: C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda.cu:60: !"CUDA error"
time=2024-04-22T20:03:13.722+10:00 level=ERROR source=routes.go:120 msg="error loading llama server" error="llama runner process no longer running: 3221226505 CUDA error\""
[GIN] 2024/04/22 - 20:03:13 | 500 |     5.562023s |       127.0.0.1 | POST     "/api/generate"
^C
C:\Users\ken>d:\ZLUDA\zluda.exe -- ollama.exe serve
time=2024-04-22T20:11:47.288+10:00 level=INFO source=images.go:817 msg="total blobs: 20"
time=2024-04-22T20:11:47.289+10:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-22T20:11:47.290+10:00 level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)"
time=2024-04-22T20:11:47.292+10:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\ken\AppData\Local\Temp\ollama1878556081\runners
time=2024-04-22T20:11:47.483+10:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx2 cuda_v11.3 rocm_v5.7 cpu cpu_avx]"
[GIN] 2024/04/22 - 20:11:52 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2024/04/22 - 20:11:52 | 200 |       530.1µs |       127.0.0.1 | POST     "/api/show"
time=2024-04-22T20:11:52.991+10:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-22T20:11:52.991+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-22T20:11:53.032+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll]"
time=2024-04-22T20:11:53.058+10:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-22T20:11:53.058+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:11:53.281+10:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.8"
time=2024-04-22T20:11:53.281+10:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-22T20:11:53.281+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-22T20:11:53.313+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll]"
time=2024-04-22T20:11:53.314+10:00 level=INFO source=gpu.go:126 msg="Nvidia GPU detected via cudart"
time=2024-04-22T20:11:53.314+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:11:53.315+10:00 level=INFO source=gpu.go:202 msg="[cudart] CUDART CUDA Compute Capability detected: 8.8"
time=2024-04-22T20:11:53.315+10:00 level=INFO source=server.go:127 msg="offload to gpu" reallayers=33 layers=33 required="5222.5 MiB" used="5222.5 MiB" available="22085.4 MiB" kv="1024.0 MiB" fulloffload="164.0 MiB" partialoffload="193.0 MiB"
time=2024-04-22T20:11:53.315+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:11:53.320+10:00 level=INFO source=server.go:264 msg="starting llama server" cmd="C:\\Users\\ken\\AppData\\Local\\Temp\\ollama1878556081\\runners\\cuda_v11.3\\ollama_llama_server.exe --model C:\\Users\\ken\\.ollama\\models\\blobs\\sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --port 63561"
time=2024-04-22T20:11:53.345+10:00 level=INFO source=server.go:389 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2603,"msg":"logging to file is disabled.","tid":"1600","timestamp":1713780713}
{"build":2679,"commit":"7593639","function":"wmain","level":"INFO","line":2820,"msg":"build info","tid":"1600","timestamp":1713780713}
{"function":"wmain","level":"INFO","line":2827,"msg":"system info","n_threads":8,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"1600","timestamp":1713780713,"total_threads":16}
llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from C:\Users\ken\.ollama\models\blobs\sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 2
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {% if messages[0]['role'] == 'system'...
llama_model_loader: - kv  22:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW)
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
  Device 0: AMD Radeon 780M Graphics [ZLUDA], compute capability 8.8, VMM: no
llm_load_tensors: ggml ctx size =    0.22 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    70.31 MiB
llm_load_tensors:      CUDA0 buffer size =  3577.56 MiB
..................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  1024.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.14 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   164.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    12.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 2
CUDA error: CUBLAS_STATUS_NOT_INITIALIZED
  current device: 0, in function cublas_handle at C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda/common.cuh:526
  cublasCreate_v2(&cublas_handles[device])
GGML_ASSERT: C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda.cu:60: !"CUDA error"
time=2024-04-22T20:12:00.555+10:00 level=ERROR source=routes.go:120 msg="error loading llama server" error="llama runner process no longer running: 3221226505 CUDA error\""
[GIN] 2024/04/22 - 20:12:00 | 500 |    7.9525007s |       127.0.0.1 | POST     "/api/generate"
@kenhuang
Copy link
Author

error without zluda

C:\Users\ken>ollama.exe serve
time=2024-04-22T20:19:08.641+10:00 level=INFO source=images.go:817 msg="total blobs: 20"
time=2024-04-22T20:19:08.642+10:00 level=INFO source=images.go:824 msg="total unused blobs removed: 0"
time=2024-04-22T20:19:08.643+10:00 level=INFO source=routes.go:1143 msg="Listening on [::]:11434 (version 0.1.32)"
time=2024-04-22T20:19:08.654+10:00 level=INFO source=payload.go:28 msg="extracting embedded files" dir=C:\Users\ken\AppData\Local\Temp\ollama570398800\runners
time=2024-04-22T20:19:08.850+10:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu_avx2 cuda_v11.3 rocm_v5.7 cpu cpu_avx]"
[GIN] 2024/04/22 - 20:19:14 | 200 |            0s |       127.0.0.1 | HEAD     "/"
[GIN] 2024/04/22 - 20:19:14 | 200 |      1.0618ms |       127.0.0.1 | POST     "/api/show"
time=2024-04-22T20:19:14.572+10:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-22T20:19:14.572+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-22T20:19:14.611+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll]"
time=2024-04-22T20:19:14.612+10:00 level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll: your nvidia driver is too old or missing, please upgrade to run ollama"
time=2024-04-22T20:19:14.612+10:00 level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll: your nvidia driver is too old or missing, please upgrade to run ollama"
time=2024-04-22T20:19:14.612+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library nvml.dll"
time=2024-04-22T20:19:14.646+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: []"
time=2024-04-22T20:19:14.646+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:19:14.663+10:00 level=INFO source=amd_windows.go:40 msg="AMD Driver: 50732000"
time=2024-04-22T20:19:14.663+10:00 level=INFO source=amd_windows.go:69 msg="detected 1 hip devices"
time=2024-04-22T20:19:14.663+10:00 level=INFO source=amd_windows.go:87 msg="[0] Name: AMD Radeon 780M Graphics"
time=2024-04-22T20:19:14.663+10:00 level=INFO source=amd_windows.go:90 msg="[0] GcnArchName: gfx1103"
time=2024-04-22T20:19:14.900+10:00 level=INFO source=amd_windows.go:117 msg="[0] Total Mem: 23158226944"
time=2024-04-22T20:19:14.900+10:00 level=INFO source=amd_windows.go:118 msg="[0] Free Mem:  23309320192"
time=2024-04-22T20:19:14.900+10:00 level=INFO source=assets.go:123 msg="Updating PATH to C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\rocm;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Recovery\\OEM\\Backup\\;C:\\Program Files\\dotnet\\;C:\\Program Files\\Git\\cmd;C:\\Program Files\\Git LFS;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files\\Docker\\Docker\\resources\\bin;C:\\Program Files\\nodejs\\;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Recovery\\OEM\\Backup\\;C:\\Program Files\\dotnet\\;C:\\Program Files\\Git\\cmd;C:\\Program Files\\Git LFS;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files\\Docker\\Docker\\resources\\bin;C:\\Program Files\\nodejs\\;C:\\Users\\ken\\AppData\\Local\\Programs\\Python\\Python310\\Scripts\\;C:\\Users\\ken\\AppData\\Local\\Programs\\Python\\Python310\\;C:\\Users\\ken\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\ken\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;C:\\Users\\ken\\AppData\\Roaming\\npm;C:\\Program Files\\AMD\\ROCm\\5.7\\bin;;"
time=2024-04-22T20:19:14.962+10:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"
time=2024-04-22T20:19:14.962+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library cudart64_*.dll"
time=2024-04-22T20:19:14.997+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: [C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll]"
time=2024-04-22T20:19:14.998+10:00 level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library C:\\Users\\ken\\AppData\\Local\\Programs\\Ollama\\cudart64_110.dll: your nvidia driver is too old or missing, please upgrade to run ollama"
time=2024-04-22T20:19:14.998+10:00 level=INFO source=gpu.go:343 msg="Unable to load cudart CUDA management library C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\cudart64_60.dll: your nvidia driver is too old or missing, please upgrade to run ollama"
time=2024-04-22T20:19:14.999+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library nvml.dll"
time=2024-04-22T20:19:15.035+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: []"
time=2024-04-22T20:19:15.035+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:19:15.046+10:00 level=INFO source=amd_windows.go:40 msg="AMD Driver: 50732000"
time=2024-04-22T20:19:15.046+10:00 level=INFO source=amd_windows.go:69 msg="detected 1 hip devices"
time=2024-04-22T20:19:15.046+10:00 level=INFO source=amd_windows.go:87 msg="[0] Name: AMD Radeon 780M Graphics"
time=2024-04-22T20:19:15.047+10:00 level=INFO source=amd_windows.go:90 msg="[0] GcnArchName: gfx1103"
time=2024-04-22T20:19:15.268+10:00 level=INFO source=amd_windows.go:117 msg="[0] Total Mem: 23158226944"
time=2024-04-22T20:19:15.268+10:00 level=INFO source=amd_windows.go:118 msg="[0] Free Mem:  23309320192"
time=2024-04-22T20:19:15.323+10:00 level=INFO source=server.go:127 msg="offload to gpu" reallayers=33 layers=33 required="5222.5 MiB" used="5222.5 MiB" available="22229.5 MiB" kv="1024.0 MiB" fulloffload="164.0 MiB" partialoffload="193.0 MiB"
time=2024-04-22T20:19:15.323+10:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-22T20:19:15.330+10:00 level=INFO source=server.go:264 msg="starting llama server" cmd="C:\\Users\\ken\\AppData\\Local\\Temp\\ollama570398800\\runners\\rocm_v5.7\\ollama_llama_server.exe --model C:\\Users\\ken\\.ollama\\models\\blobs\\sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --port 63771"
time=2024-04-22T20:19:15.350+10:00 level=INFO source=server.go:389 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2603,"msg":"logging to file is disabled.","tid":"18916","timestamp":1713781155}
{"build":2679,"commit":"7593639","function":"wmain","level":"INFO","line":2820,"msg":"build info","tid":"18916","timestamp":1713781155}
{"function":"wmain","level":"INFO","line":2827,"msg":"system info","n_threads":8,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"18916","timestamp":1713781155,"total_threads":16}
llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from C:\Users\ken\.ollama\models\blobs\sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 2
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {% if messages[0]['role'] == 'system'...
llama_model_loader: - kv  22:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW)
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon 780M Graphics, compute capability 11.0, VMM: no
llm_load_tensors: ggml ctx size =    0.22 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  3577.56 MiB
llm_load_tensors:        CPU buffer size =    70.31 MiB
..................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =  1024.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =     0.14 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =   164.00 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =    12.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 2
ggml_cuda_compute_forward: RMS_NORM failed
CUDA error: invalid device function
  current device: 0, in function ggml_cuda_compute_forward at C:/a/ollama/ollama/llm/llama.cpp/ggml-cuda.cu:2212
  err
GGML_ASSERT: C:/a/ollama/ollama/llm/llama.cpp/ggml-cuda.cu:60: !"CUDA error"
time=2024-04-22T20:19:19.352+10:00 level=ERROR source=routes.go:120 msg="error loading llama server" error="llama runner process no longer running: 3221226505 CUDA error: invalid device function\r\n  current device: 0, in function ggml_cuda_compute_forward at C:/a/ollama/ollama/llm/llama.cpp/ggml-cuda.cu:2212\r\n  err\r\nGGML_ASSERT: C:/a/ollama/ollama/llm/llama.cpp/ggml-cuda.cu:60: !\"CUDA error\""
[GIN] 2024/04/22 - 20:19:19 | 500 |    5.1720784s |       127.0.0.1 | POST     "/api/generate"

@4thanks
Copy link

4thanks commented Apr 30, 2024

我按这个测试了ollama_windows_10_rx6600xt_zluda，amd显卡能成功运行并加速推理，你也试试。

@kenhuang
Copy link
Author

Wow, confirm latest release works, did follow instruction copy cublas.dll to Ollama folder and renamed, 50% performance bump via APU
IMG_2442
IMG_2443

@kenhuang
Copy link
Author

Try use 5700xt getting errors, maybe related to my ROCm hack, it’s also not on support list. Happy with the APU working 🎉
IMG_2445
IMG_2444
IMG_2446

@4thanks
Copy link

4thanks commented May 1, 2024

Try use 5700xt getting errors, maybe related to my ROCm hack, it’s also not on support list. Happy with the APU working 🎉 !

我也是5700xt，zluda有支持并调用gpu，可以试试ROCmLibs_Testing.7z那个包，我用的没问题。

This was referenced May 11, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kenhuang @4thanks