Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update feedstock to use cirun-openstack-cpu-large with Cirun #21

Closed

Conversation

conda-forge-admin
Copy link
Contributor

@conda-forge-admin conda-forge-admin commented Feb 23, 2024

Note that only builds triggered by maintainers of the feedstock (and core)
who have accepted the terms of service and privacy policy will run
on Github actions via Cirun.

Also, note that rerendering with Github actions as CI provider must be done
locally in the future for this feedstock.

Closes #19

automatic conda-forge administrator and others added 2 commits February 23, 2024 10:27
@conda-forge-webservices
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@jaimergp
Copy link
Member

@RaulPPelaez, push an empty commit here to test the Cirun triggers. Thanks!

@RaulPPelaez
Copy link
Contributor

Seems to be running. Should I merge #19 here or the other way around?

@jaimergp
Copy link
Member

Whatever you prefer, this PR is here for your convenience. Pick one and iterate there :)

@RaulPPelaez
Copy link
Contributor

RaulPPelaez commented Feb 23, 2024

Also, note that rerendering with Github actions as CI provider must be done
locally in the future for this feedstock.

@jaimergp could you clarify what this means? I cannot longer call
@conda-forge-admin, please rerender
in this feedstock?

Also I cannot manage to make the CI run in #19 so I will move the commits here...

Copy link
Contributor

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/cutlass-feedstock/actions/runs/8018330283.

Copy link
Contributor

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/cutlass-feedstock/actions/runs/8018331935.

@h-vetinari
Copy link
Member

Running into:

[ 72%] Linking CXX shared library lib/libcutlass.so
$BUILD_PREFIX/x86_64-conda-linux-gnu/bin/ld: cannot find -lcuda_driver: No such file or directory

I'm not sure how linking the driver would have worked on azure (though we also haven't been able to build anything >=v3.2), but by itself, this looks like we might have to use the gpu runner after all?

@jaimergp @jakirkham

@RaulPPelaez
Copy link
Contributor

I believe it should be enough to add cuda-driver-dev as a dependency. Lets try

@jaimergp
Copy link
Member

@jaimergp could you clarify what this means? I cannot longer call

Correct. GH prevents workflows from modifying other workflows in a PR for security reasons. Since cirun requires GHA workflows and rerendering happens via GHA jobs under the hood... we hit that limitation. We might come up with a workaround, but for now this is what we have. You'll need to rerender locally with conda-smithy rerender.

@RaulPPelaez
Copy link
Contributor

Adding cuda driver was not enough. I can see it installed, but the same error arises.

@h-vetinari
Copy link
Member

I believe it should be enough to add cuda-driver-dev as a dependency. Lets try

Neither cuda-driver-dev nor cuda-driver-dev_linux-64 contain libcuda_driver.so; IIUC, the actual driver is the only piece of the puzzle that we aren't able to distribute. CC @jakirkham @leofang @adibbley

If so, then I think we do need cirun-openstack-gpu-large after all (unless there's an option for cutlass not to directly link to the driver of course).

@leofang
Copy link
Member

leofang commented Feb 27, 2024

To the best of my knowledge, I am unaware of such thing as libcuda_driver.so. Doesn't seem to come from NVIDIA?

@leofang
Copy link
Member

leofang commented Feb 27, 2024

Is it coming from the CMake FindCUDAToolkit module?
https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html#cuda-driver-library

@leofang
Copy link
Member

leofang commented Feb 27, 2024

Haven't looked into how cutlass's CMake works, but this preprocessing log seems fishy

-- CUDART: Not Found
-- CUDA Driver: Not Found
-- NVRTC: Not Found

@@ -26,6 +25,7 @@ requirements:
host:
- cudnn
- libcublas-dev # [(cuda_compiler_version or "").startswith("12")]
- cuda-driver-dev # [(cuda_compiler_version or "").startswith("12")]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be correct to me, if you need the header and to link to libcuda.so.

@h-vetinari
Copy link
Member

Thanks for your input @leofang!

To the best of my knowledge, I am unaware of such thing as libcuda_driver.so. Doesn't seem to come from NVIDIA?

Well, cutlass if from NVIDIA, so in some way yes. Looking at the source upstream now, I see that cutlass does a sort of CMake-wrapper around the driver library.

It looks like we need to teach the search just above to look in the right place for finding libcuda.so, then the rest should work.

@RaulPPelaez
Copy link
Contributor

Actually I am with @leofang here, wth is libcuda_driver.so? I cannot find references to it online.

Maybe setting CUDA_TOOLKIT_ROOT_DIR to CONDA_PREFIX?

@h-vetinari
Copy link
Member

Actually I am with @leofang here, wth is libcuda_driver.so? I cannot find references to it online.

Click the first link in my comment

@leofang
Copy link
Member

leofang commented Mar 21, 2024

Again, NVIDIA does not ship libcuda_driver.so. If this is from CUTLASS CMake, you need to ask CUTLASS folks about how to patch/fix it.

@leofang
Copy link
Member

leofang commented May 17, 2024

Thanks very much for all the efforts and sorry for the struggling. Now that we restored cutlass as a header-only package (#22) and also released 3.3.0 (#24), this PR is no longer needed. Let me proceed and close it.

@leofang leofang closed this May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants