-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
speeding up create_raw_dataset.py #56
Comments
segfault was happening due to numba |
Spent some time applying one of the multiprocessing package but the results weren't that good |
When writing a dataset, process function should also take in sample (AudioClipExample) and use sample.audio_data when metadata.path does not exist (https://github.com/castorini/howl/blob/master/howl/data/dataset/serialize.py#L67-L72) |
A simple way to speed this up is to call out to ffmpeg in |
create_raw_dataset.py takes quite a long time to generate datasets.
I thinking multi-threading AudioDatasetMetadataWriter write will do the job.
Also, this process terminates with segfault
The text was updated successfully, but these errors were encountered: