Does it support NVIDIA H20?

[thomZ1]

Recently, I'm testing the inference performance of H20.
H20 requires CUDA 12.3, but SGlang environment is CUDA 12.1.
Does SGlang support NVIDIA H20?

[zhyncs]

We support CUDA 12.4 docker pull lmsysorg/sglang:v0.2.10-cu124

[thomZ1]

thx

[thomZ1]

I try to use this image lmsysorg/sglang:v0.2.10-cu124

still unable to server any model on H20

get this error

root@VM-0-8-centos:/mnt/cfs_standard/User/zhangyi/Inference/H20# CUDA_VISIBLE_DEVICES=1  python3 -m sglang.launch_server --model-path /mnt/cfs_standard/Model/Llama3/Meta-Llama-3-8B-Instruct   --port 8002
server_args=ServerArgs(model_path='/mnt/cfs_standard/Model/Llama3/Meta-Llama-3-8B-Instruct', tokenizer_path='/mnt/cfs_standard/Model/Llama3/Meta-Llama-3-8B-Instruct', tokenizer_mode='auto', load_format='auto', dtype='auto', trust_remote_code=False, context_length=None, quantization=None, served_model_name='/mnt/cfs_standard/Model/Llama3/Meta-Llama-3-8B-Instruct', chat_template=None, host='127.0.0.1', port=8002, additional_ports=[8003, 8004, 8005, 8006], mem_fraction_static=0.88, max_prefill_tokens=None, max_running_requests=None, max_num_reqs=None, max_total_tokens=None, schedule_policy='lpm', schedule_conservativeness=1.0, tp_size=1, stream_interval=1, random_seed=52157490, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, api_key=None, file_storage_pth='SGlang_storage', dp_size=1, load_balance_method='round_robin', chunked_prefill_size=None, disable_flashinfer=False, disable_flashinfer_sampling=False, disable_radix_cache=False, disable_regex_jump_forward=False, disable_cuda_graph=False, disable_disk_cache=False, enable_torch_compile=False, enable_p2p_check=False, enable_mla=False, attention_reduce_in_fp32=False, efficient_weight_load=False, nccl_init_addr=None, nnodes=1, node_rank=None)
[gpu=0] Init nccl begin.
[gpu=0] Load weight begin. avail mem=94.69 GB
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:03,  1.33s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:02<00:02,  1.36s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:03<00:00,  1.09it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00,  1.14s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00,  1.15s/it]

[gpu=0] Load weight end. type=LlamaForCausalLM, dtype=torch.bfloat16, avail mem=79.66 GB
[gpu=0] Memory pool end. avail mem=11.12 GB
[gpu=0] Capture cuda graph begin. This can take up to several minutes.

[thomZ1]

[gpu=0] Load weight end. type=LlamaForCausalLM, dtype=torch.bfloat16, avail mem=79.66 GB
[gpu=0] Memory pool end. avail mem=11.12 GB
[gpu=0] Capture cuda graph begin. This can take up to several minutes.
^CProcess Process-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/sgl-workspace/sglang/python/sglang/launch_server.py", line 14, in <module>
    launch_server(server_args)
  File "/sgl-workspace/sglang/python/sglang/srt/server.py", line 279, in launch_server
    controller_init_state = pipe_controller_reader.recv()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/managers/detokenizer_manager.py", line 152, in start_detokenizer_process
    loop.run_until_complete(manager.handle_loop())
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run
  File "uvloop/handles/poll.pyx", line 216, in uvloop.loop.__on_uvpoll_event
  File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
  File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run
  File "uvloop/loop.pyx", line 397, in uvloop.loop.Loop._read_from_self
  File "uvloop/loop.pyx", line 402, in uvloop.loop.Loop._invoke_signals
  File "uvloop/loop.pyx", line 377, in uvloop.loop.Loop._ceval_process_signals
KeyboardInterrupt

[zhyncs]

Why you use Ctrl-C?

[zhyncs]

Capture cuda graph begin. This can take up to several minutes.

Be patient and wait for server startup to complete instead of Ctrl C

[nstl-zyb]

Me too. After 30-40min it`s still here "[gpu=0] Capture cuda graph begin. This can take up to several minutes." , can not complete.

[liunn5]

我也遇到这个问题了，您那解决了吗