Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicitly select for CUDA-suffixed dependencies in rapids-make-pip-dependencies #365

Merged
merged 12 commits into from
Jul 23, 2024
Merged

Explicitly select for CUDA-suffixed dependencies in rapids-make-pip-dependencies #365

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8cd4ba2
select CUDA-suffixed dependencies for pip env
jameslamb Jul 8, 2024
c8bffd4
change more places
jameslamb Jul 22, 2024
7fa1e1f
Merge branch 'branch-24.08' of github.com:rapidsai/devcontainers into…
jameslamb Jul 22, 2024
072f5e5
expand comment
jameslamb Jul 22, 2024
5c93127
one more place
jameslamb Jul 22, 2024
52f567d
bump version
jameslamb Jul 22, 2024
23179ed
Merge branch 'more-rapids-build-backend' of github.com:jameslamb/devc…
jameslamb Jul 22, 2024
c40f39f
move matrix entry to argument
jameslamb Jul 22, 2024
9e15d70
always keep cuda, arch, and py in the rapids-dependency-file-generato…
jameslamb Jul 22, 2024
0df12df
Update features/src/rapids-build-utils/opt/rapids-build-utils/bin/mak…
jameslamb Jul 23, 2024
378b87b
Merge branch 'branch-24.08' of github.com:rapidsai/devcontainers into…
jameslamb Jul 23, 2024
3df408c
update make-conda-dependencies too
jameslamb Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion features/src/rapids-build-utils/devcontainer-feature.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "NVIDIA RAPIDS devcontainer build utilities",
"id": "rapids-build-utils",
"version": "24.8.14",
"version": "24.8.15",
"description": "A feature to install the RAPIDS devcontainer build utilities",
"containerEnv": {
"BASH_ENV": "/etc/bash.bash_env"
Expand Down
63 changes: 45 additions & 18 deletions features/src/rapids-build-utils/opt/rapids-build-utils/bin/make-pip-dependencies.sh
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also make these changes in make-conda-dependencies.sh?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I think that makes sense. I'd originally not done that when I was just going to hard-code this behavior as defaults inside make-pip-dependencies (since conda lists should never care about CUDA suffixes), but now that we're exposing --matrix-entry as an argument, I agree it should be supported in make-conda-dependencies too.

Just pushed that change.

Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@
# -e,--exclude <file> Path(s) to requirement files of packages to exclude.
# -i,--include <file> Path(s) to requirement files of packages to include.
# -k,--key <key> Only include the key(s)
# --matrix-entry <entry> Matrix entries, in the form 'key=value' to be added to the '--matrix' arg
# of rapids-dependency-file-generator.
# (can be passed multiple times)
# @_include_value_options rapids-list-repos -h | tail -n+2 | head -n-3;
# --repo <repo> Only include dependencies for repo(s).
# (default: all repositories)
Expand Down Expand Up @@ -59,6 +62,30 @@ make_pip_dependencies() {
local python_version="${PYTHON_VERSION:-$(python3 --version 2>&1 | cut -d' ' -f2)}";
python_version="$(cut -d'.' -f3 --complement <<< "${python_version}")";

# Why default to cuda_suffixed=true?
#
# Projects that depend on different pip libraries across different CUDA versions
# (e.g. 'cudf' only depending on 'pynvjitlink' from CUDA 12.0 onwards), split up their
# dependency lists with 'cuda_suffixed={true,false}'.
#
# Here we want the suffixed versions (like 'pynvjitlink-cu12').
#
# It's ok for other RAPIDS libraries to end up in this list (like 'rmm-cu12')... in builds
# where those are also being built in the devcontainer, they'll be filtered out via
# inclusion in the 'pip_noinstall' list below.
local -a _matrix_selectors=(
arch="$(uname -m)"
cuda="${cuda_version}"
cuda_suffixed=true
py="${python_version}"
);

# add extra arguments (if there are conflicts, e.g. 'py=3.10;py=3.11', it's fine... the last one will win)
local ent; for ent in "${matrix_entry[@]}"; do
_matrix_selectors+=("${ent}")
done
jameslamb marked this conversation as resolved.
Show resolved Hide resolved
local -r matrix_selectors=$(IFS=";"; echo "${_matrix_selectors[*]}")

local pip_reqs_txts=();

eval "$(rapids-list-repos "${OPTS[@]}")";
Expand All @@ -81,12 +108,12 @@ make_pip_dependencies() {
for ((keyi=0; keyi < ${#repo_keys[@]}; keyi+=1)); do
local file="/tmp/${!repo_name}.${repo_keys[$keyi]}.requirements.txt";
pip_reqs_txts+=("${file}");
generate_requirements \
"${file}" \
--file-key "${repo_keys[$keyi]}" \
--output requirements \
--config ~/"${!repo_path}/dependencies.yaml" \
--matrix "arch=$(uname -m);cuda=${cuda_version};py=${python_version}" \
generate_requirements \
"${file}" \
--file-key "${repo_keys[$keyi]}" \
--output requirements \
--config ~/"${!repo_path}/dependencies.yaml" \
--matrix "${matrix_selectors}" \
;
done

Expand All @@ -103,12 +130,12 @@ make_pip_dependencies() {
for ((keyi=0; keyi < ${#repo_keys[@]}; keyi+=1)); do
local file="/tmp/${!repo_name}.lib${!cpp_name}.${repo_keys[$keyi]}.requirements.txt";
pip_reqs_txts+=("${file}");
generate_requirements \
"${file}" \
--file-key "${repo_keys[$keyi]}" \
--output requirements \
--config ~/"${!repo_path}/dependencies.yaml" \
--matrix "arch=$(uname -m);cuda=${cuda_version};py=${python_version}" \
generate_requirements \
"${file}" \
--file-key "${repo_keys[$keyi]}" \
--output requirements \
--config ~/"${!repo_path}/dependencies.yaml" \
--matrix "${matrix_selectors}" \
;
done
done
Expand All @@ -126,12 +153,12 @@ make_pip_dependencies() {
for ((keyi=0; keyi < ${#repo_keys[@]}; keyi+=1)); do
local file="/tmp/${!repo_name}.${!py_name}.${repo_keys[$keyi]}.requirements.txt";
pip_reqs_txts+=("${file}");
generate_requirements \
"${file}" \
--file-key "${repo_keys[$keyi]}" \
--output requirements \
--config ~/"${!repo_path}/dependencies.yaml" \
--matrix "arch=$(uname -m);cuda=${cuda_version};py=${python_version}" \
generate_requirements \
"${file}" \
--file-key "${repo_keys[$keyi]}" \
--output requirements \
--config ~/"${!repo_path}/dependencies.yaml" \
--matrix "${matrix_selectors}" \
;
done
done
Expand Down
Loading