[template] create templates for use in generating actions #1282
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cjac
commented
Dec 29, 2024
templates/generate-action.pl
Outdated
There was a problem hiding this comment.
Hello @davorg - As I was reading through the literature, bringing myself back up to speed on the state of the art of template toolkits, I saw that there was a book with one of my friends' names on it that I had glanced at many times over the last couple of decades. I did not realize until just a few days ago that the dlc who was in charge of desk allocation in my cube farm when I started was the same dlc who wrote the book on this particular subject.
Anyway, I've been thinking of you and our peers as I've been hacking away at this installer. If you felt like looking things over and picking some nits, I'd love to hear your feedback. I hope your holidays are merry and all that!
There was a problem hiding this comment.
@shlomif oh, hey I see that you are actively participating in Template.pm development. I'm not doing a lot with it in this repository; everything is pretty straightforward, I think. If you had some spare time to take a peek at the new templates/ directory in this repo, and especially the templates/generate-action.pl, it might be fun to chat about it. I hope your holidays went well!
There was a problem hiding this comment.
@cjac: hi! Where can I find the templates directory? Please give a url.
There was a problem hiding this comment.
@shlomif thanks for your prompt response! That URL is
https://github.com/LLC-Technologies-Collier/initialization-actions/tree/template-gpu-20241219
|
/gcbrun |
1 similar comment
|
/gcbrun |
cjac
force-pushed
the
template-gpu-20241219
branch
from
8d28938 to
e511a6e
Compare
|
/gcbrun |
1 similar comment
|
/gcbrun |
cjac
commented
Jan 2, 2025
templates/spark-rapids/mig.sh.in
Outdated
| # --metadata=ENABLE_MIG can be used to enable or disable MIG. The default is to enable it. | ||
| # The script does a reboot to fully enable MIG and then configures the MIG device based on the | ||
| # user specified MIG_CGI profiles specified via: --metadata=^:^MIG_CGI='9,9'. If MIG_CGI | ||
| # is not specified it assumes it's using an A100 and configures 2 instances with profile id 9. |
templates/spark-rapids/mig.sh.in
Outdated
| # | ||
| # This script should be specified in --metadata=startup-script-url= option and | ||
| # --metadata=ENABLE_MIG can be used to enable or disable MIG. The default is to enable it. | ||
| # The script does a reboot to fully enable MIG and then configures the MIG device based on the |
There was a problem hiding this comment.
does not ever reboot, and neither should you
|
/gcbrun |
2 similar comments
|
/gcbrun |
|
/gcbrun |
|
using the test suite I just cleaned up for #1275 |
|
/gcbrun |
|
2.1-debian11 failure: |
|
/gcbrun |
3 similar comments
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
well that's good news, then. |
|
/gcbrun |
|
/gcbrun |
rapids.sh.in into a function install_conda_packages * removed redundant yarn service restarts in rapids.sh.in * added conda prep and exit handlers
templates/dask/util_functions, templates/gpu/install_gpu_driver.sh.in, templates/gpu/util_functions, templates/rapids/rapids.sh.in, templates/spark-rapids/spark-rapids.sh.in: * cleaned up definition of RAPIDS_RUNTIME ; default to SPARK and use DASK only for dask-rapids templates/dask/util_functions, templates/gpu/util_functions, templates/common/util_functions: * added utility functions to check whether a phase has been complete, mark a phase complete and mark a phase as incomplete templates/dask/util_functions: * conda environment is now archived from the environment directory rather than from / templates/rapids/rapids.sh.in: * Now executing gpu installer logic before installing dask-rapids * now exiting if rapids runtime is not DASK
* increased minimum memory threshold for ram disk * moved apt_add_repo and friends to common/install_functions templates/dask/util_functions: * validating conda tarball before caching to gcs templates/generate-action.pl: * improved usage documentation a little templates/gpu/install_functions * using /opt/conda/miniconda3/bin/python3 instead of /usr/bin/ for venv pre-install
cjac
force-pushed
the
template-gpu-20241219
branch
from
019f562 to
119f1b1
Compare
* increase wait time for scheduler to come online * reduce noise from tar -t templates/gpu/yarn_functions, templates/gpu/install_functions: * protect many functions from running without attached accelerator templates/gpu/install_gpu_driver.sh.in * set +e in exit handler templates/gpu/spark_functions: * re-factor new function into this template templates/spark-rapids/spark-rapids.sh.in * removed redundant call to configure_gpu_script * set +e in exit handler
…ing the readonly operations
* include version in action generator
cjac
force-pushed
the
template-gpu-20241219
branch
from
763e1ff to
900c10a
Compare
cjac
commented
Jan 10, 2025
templates/generate-action.pl
Outdated
There was a problem hiding this comment.
@dlc - Can I get a review of the templates/ directory in this repository, please? I tried to keep it simple for the initial implementation, but if you have any advice about how we can further reduce duplication, I'd be all ears. I'm thinking about picking up your book and getting into the minutia, but the PR will be closed far before then, I hope!
cjac
force-pushed
the
template-gpu-20241219
branch
from
cea2aa3 to
2afff45
Compare
cjac
force-pushed
the
template-gpu-20241219
branch
from
2afff45 to
aa792c3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR should resolve #1276 and is an attempt at better solving the problem space of #1030
I believe that #1259 could be implemented easier using this change.
The mlvm/mlvm.sh action is generated using the templates defined here. There is also code to generate other actions in the templates committed with this PR, but only this action is generated and tested here. See other PRs for each individual action.