Support simulation of 4090 cards

[Enoch2090]

Since 4090 really has an edge on inferencing with smaller models - is it possible to add data for 4090 cards? Thanks!

[AgrawalAmey]

Hi @Enoch2090, thanks a lot of your interest in the project. We definitely want to extend support to as many devices as possible. Unfortunately, I don't have access to 4090 cards, but if you are willing to contribute we could share all the profiling instructions. Thanks again!

[teds-lin]

Hi @Enoch2090, thanks a lot of your interest in the project. We definitely want to extend support to as many devices as possible. Unfortunately, I don't have access to 4090 cards, but if you are willing to contribute we could share all the profiling instructions. Thanks again!

Based on my current understanding, one of the key points for adding new GPU card support is collecting profiling data. Will you publish the relevant code and implementation steps? Thank!

[AgrawalAmey]

@teds-lin yes, the profiling code is already in the repo. We will add appropriate documentation soon. Thanks!

[Enoch2090]

@AgrawalAmey Thanks for sharing! Would also be great if the document can point out approximately how much data profiled (or equivalently, how many GPU hours) is needed to produce accurate results.

[AgrawalAmey]

@teds-lin @Enoch2090 We have added the documentation to add profiling data for new models/devices. Adding a new model should take less than 15ish mins. Adding a new device across all 6 models should take less than a couple of hours. Please have a look https://github.com/microsoft/vidur/blob/main/vidur/profiling/README.md and let us know if you have any questions.

[lhpp1314]

@AgrawalAmey The profiling link https://github.com/microsoft/vidur/blob/main/vidur/profiling/README.md is not available now. Can you give a new one?

[nitinkedia7]

Hi @lhpp1314, the Supported Models section in the project README has the updated link for profiling. Please use the central readme to find links to other documentation.