Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow threads on Index creation #121

Merged
merged 1 commit into from
Dec 11, 2024
Merged

Allow threads on Index creation #121

merged 1 commit into from
Dec 11, 2024

Conversation

torymur
Copy link
Contributor

@torymur torymur commented Dec 11, 2024

Closes #113

GIL prevents several threads from executing Python bytecode in parallel, so if multiple threads are created, only one thread can execute Python bytecode at any given time.

Release the GIL

To temporarily release the GIL for the cpu-heavy index creation step pyo3's allow_threads seems to be enough to allow other Python threads to run and to get roughly x2+ improvement in multithreaded case, where number of threads is equal to the number of physical cores.

Compare current situation on main branch without allow_threads, tested on my local machine with 6 threads (6 physical cores):
on_main

To these benchmarks with allow_threads on index creation and different number of threads:
diff num cores

Of course, amount of threads will still affect the results due to GIL's overall overhead, here as expected: less threads, better results.

Also, separately confirmed equal thread usage with top:
parallel2

GIL's switch interval affects too

CPU-bound tasks are especially limited by the GIL since they require continuous execution of Python bytecode and default GIL's switch interval is 5ms. To show this effect I added a benchmark with increased interval to 5 secs (the same 6 threads):
custom switch interval

These results are much closer to the one-threaded case, than the ones with default GIL's switch interval.

@torymur torymur changed the title Allow threads on Index init Allow threads on Index creation Dec 11, 2024
@torymur torymur added the enhancement New feature or request label Dec 11, 2024
@rlouf rlouf requested review from umut-sahin and 414owen December 11, 2024 13:33
@rlouf rlouf merged commit 31ab9f1 into main Dec 11, 2024
8 checks passed
@rlouf rlouf deleted the parallel-bench-113 branch December 11, 2024 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate the performance of outlines-core in a concurrent environment
2 participants