Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run nightly benchmarks in memory-constrained fashion? #269

Open
jpountz opened this issue May 22, 2024 · 2 comments
Open

Run nightly benchmarks in memory-constrained fashion? #269

jpountz opened this issue May 22, 2024 · 2 comments

Comments

@jpountz
Copy link
Collaborator

jpountz commented May 22, 2024

We're currently working on better parallelizing I/O for Lucene queries. To be able to track improvements (and regressions!), it would be nice if we could run nightly benchmarks with a page cache size that is only a fraction of the total index size.

@rmuir
Copy link
Collaborator

rmuir commented May 22, 2024

For now we can use ramhog for this if needed. In the future we could specify cgroupv2 limits in the systemd service/timer for all the benchmarks, or run particular benchmarks with systemd-run, specifying lower limits.

@mikemccand
Copy link
Owner

@rmuir also suggested using fincore --output-all /path/to/index/* to monitor how many hot/cold pages we see in the index while/after benchmarking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants