Fix max nbor size related issues #4548

Job	Run time
Build source distribution	18s
determine-arm64-runner	1s
Build wheels for cp311-manylinux_x86_64	5m 12s
Build wheels for cp311-manylinux_x86_64	5m 27s
Build wheels for cp311-macosx_x86_64	6m 47s
Build wheels for cp311-macosx_arm64	5m 38s
Build wheels for cp311-win_amd64	6m 23s
Build wheels for cp310-manylinux_aarch64	4m 29s
build_pypi_index	6s
build_docker (12)	2m 56s
build_docker (_cu11, 11)	2m 46s
upload_pypi	0s
deploy_pypi_index	0s
Pass testing build wheels	3s
	40m 6s

Provide feedback