-
-
Notifications
You must be signed in to change notification settings - Fork 289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Running humaneval against llama-3-8b-instruct exl2 quant results in a silent OOM when samples per task > 7 #496
Comments
Also running However I narrowed it down to humaneval relying on signal.setitimer which is for UNIX systems only: I guess no humnaneval results grading for us Windows folks unless going down the WSL2 road XD |
That looks like a valid result, so I assume you're right about the evaluation not working under Windows. Perhaps you could run it in WSL. As for where it fails while running the test, I would suspect it's a bug in the You should be able to use Mistral with a tokenizer.json file instead of tokenizer.model, which would cause ExLlama to rely on |
You say OOM.. can you actually see memory usage go up in task manager or whatever as it's creating the jobs? It might be some sort of memory leak in PyTorch then. |
I've been experiencing some possibly related errors in an unrelated project, using the HF tokenizer for Sentencepiece models. I get inexplicable segfaults when I run more than about 80,000 encoding operations in a row, and this is without any ExLlama code. I don't seem to be getting the same errors when using non-Sentencepiece models. I think it might be related since most of what happens in the loop that fails for you is just calling the tokenizer thousands of times.
|
Same specs, same issue |
Hi @turboderp , I've created a PR with a workoraund for this issue. I hope you'll find some time to have a look, cheers! |
Evaluation runs to the finish when -spt is set to 7 or less though (Windows 11, 64GB RAM, RTX 4070Ti Super16GB VRAM, ExllamaV2 0.1.4 from the dev branch). This happens with https://huggingface.co/turboderp/Llama-3-8B-Instruct-exl2/tree/6.0bpw quant.
Windows Event Viewer shows this:
However evaluating Mistral works just fine (Tested against this quant: https://huggingface.co/turboderp/Mistral-7B-v0.2-exl2/tree/6.0bpw)
Here's what I tried:
List of installed packages in my system:
The text was updated successfully, but these errors were encountered: