[gpu] strict driver and cuda version assignment #1275
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gpu/install_gpu_driver.sh * exclusively using .run file installation method when available * build nccl from source * cache build artifacts from kernel driver and nccl * Tested more CUDA minor versions * gathering CUDA and driver version from URLs if passed * Printing warnings when combination provided is known to fail * waiting on apt lock when it exists * wrapping expensive functions in completion checks to reduce re-run time * fixed a problem with ops agent not installing ; using venv * Installing gcc-12 on ubuntu22 to fix kernel driver FTBFS * setting better spark defaults * skipping proxy setup if http-proxy metadata not set * added function to check secure-boot and os version compatability gpu/manual-test-runner.sh * order commands correctly gpu/test_gpu.py * clearer test skipping logic * added instructions on how to test pyspark
…ment ; added better description of what is in the runfile
… less well supported 11.8
…torch and tensorflow test functions
… the signing key used
This was referenced Dec 16, 2024
|
/gcbrun |
|
/gcbrun |
|
There are too many changes in this single PR. Can the PR be broken into smaller logical chunks so that it can be reviewed better |
|
yeah, there has been a lot of change. I'll re-order the functions a little. This should reduce the delta significantly. Testing my change before commit and push... |
…y creation script intended to be removed in 70f37b6
cjac
changed the title
|
/gcbrun |
|
blocked on #1283 |
cjac
changed the title
cjac
commented
Dec 25, 2024
cjac
added a commit
to LLC-Technologies-Collier/initialization-actions
that referenced
this pull request
Jan 3, 2025
cjac
added a commit
to LLC-Technologies-Collier/initialization-actions
that referenced
this pull request
Jan 6, 2025
|
Referencing a related discussion: #1269 (comment). |
cjac
added a commit
to LLC-Technologies-Collier/initialization-actions
that referenced
this pull request
Jan 9, 2025
cjac
added a commit
to LLC-Technologies-Collier/initialization-actions
that referenced
this pull request
Jan 10, 2025
gpu/test_gpu.py: * using tests from GoogleCloudDataproc#1275 gpu/verify_pyspark.py: * new test file ; will probably be moved to mlvm templates/gpu/install_gpu_driver.sh.in: * this action template includes only the code unique to this action
cjac
added a commit
to LLC-Technologies-Collier/initialization-actions
that referenced
this pull request
Jan 10, 2025
gpu/test_gpu.py: * using tests from GoogleCloudDataproc#1275 gpu/verify_pyspark.py: * new test file ; will probably be moved to mlvm templates/gpu/install_gpu_driver.sh.in: * this action template includes only the code unique to this action
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves Issues
gpu/install_gpu_driver.sh
Driver version defaults to version in driver .run file if specified
CUDA version defaults to version in cuda .run file if specified
exclusively using .run file installation method for cuda and driver installation
build nccl from source, since that is the only mechanism which supports all Dataproc OSs
wrap expensive functions in completion checks to reduce re-run time when testing manually
cache build results in GCS
waiting on apt lock when it exists
only install build dependencies if build is necessary
fix problem with ops agent not installing ; using venv
Installing gcc-12 on ubuntu22 to fix kernel driver FTBFS
mark task completion by creating a file rather than setting a variable
added functions to check and report secure-boot and os version details
gpu/manual-test-runner.sh
gpu/test_gpu.py