speed up cuda test (#3284) #5290

Job	Run time
determine-arm64-runner	1s
Build source distribution	15s
Build wheels for cp311-manylinux_x86_64	5m 22s
Build wheels for cp311-manylinux_x86_64	5m 25s
Build wheels for cp311-macosx_x86_64	5m 54s
Build wheels for cp311-macosx_arm64	3m 2s
Build wheels for cp311-win_amd64	6m 39s
Build wheels for cp310-manylinux_aarch64	4m 26s
build_pypi_index	10s
build_docker (12)	2m 17s
build_docker (_cu11, 11)	2m 20s
upload_pypi	0s
deploy_pypi_index	0s
Pass testing build wheels	3s
	35m 54s

Provide feedback