Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Possibly system specific] Wild (12% vs 20%) run-to-run swings in multiple-cpp reported scores #258

Open
alat-rights opened this issue Jul 18, 2024 · 1 comment
Open

[Possibly system specific] Wild (12% vs 20%) run-to-run swings in multiple-cpp reported scores #258

alat-rights opened this issue Jul 18, 2024 · 1 comment

Comments

@alat-rights
Copy link

alat-rights commented Jul 18, 2024 •
edited
Loading

This issue is currently blocking for me, so I am looking into it myself. I would really appreciate any potential help with investigating/isolating this issue. I'm happy to do everything I can to help reproduce.

Full logs for two consecutive runs here. The first run got pass@1 of 12%, the second run got pass@1 of 20%. They were ran on the same system with the same invocation.

System Information
Ubuntu Linux
Lambda Labs Instance: 1x A10 (24 GB PCIe) (30 CPU cores, 205.4 GB RAM, 1.5 TB SSD)
@alat-rights alat-rights changed the title Wild (12% vs 20%) run-to-run swings in multiple-cpp reported scores [Possibly system specific] Wild (12% vs 20%) run-to-run swings in multiple-cpp reported scores Jul 19, 2024
@alat-rights
Copy link
Author

On a fresh instance, I'm consistently getting 38.5%. I suspect the problem was specific to my system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@alat-rights