Medialib memory leak?

[Parakovsky]

When I try to use the following example (previously reindexing the database) my script eat all of the memory and eventually crash the system. Take note that the database is pretty large about 30 megabytes it indexed south of 1 terrabyte of data. Is there anyway I can improve it? Stuff like beets which also in a manual requires much less resources, a margin, to perform the same type of operation, but it obviously has it's own challenges. Here is the script from a cookbook example, if you want me I can put my scanned json file somewhere for you to test at home. The artist name is obv different but I don't think that's the issue. Thank you for your job still awesome software!

m = medialib(persistency="/tmp/medialib.json", "~/Music/")
l = m.find(artist_contains="Brassens")
l = list.shuffle(l)
output(playlist.list(l))

[Parakovsky]

When I try to use the following example (previously reindexing the database) my script eat all of the memory and eventually crash the system. Take note that the database is pretty large about 30 megabytes it indexed south of 1 terrabyte of data. Is there anyway I can improve it? Stuff like beets which also in a manual requires much less resources, a margin, to perform the same type of operation, but it obviously has it's own challenges. Here is the script from a cookbook example, if you want me I can put my scanned json file somewhere for you to test at home. The artist name is obv different but I don't think that's the issue. Thank you for your job still awesome software!

m = medialib(persistency="/tmp/medialib.json", "~/Music/") l = m.find(artist_contains="Brassens") l = list.shuffle(l) output(playlist.list(l))

I am using latest version on Archlinux and my computer have 15 gb of ram. If that's mattered.

[toots]

That seems like a lot of memory though. How many files do you have in the libraries after it loads?

[toots]

I would suspect that something is getting indexed from the metadata that shouldn't be like a cover or some large binary blob.

[smimram]

For the record, we should already remove some memory-consuming metadata:

    # Sanitize
    m = metadata.cover.remove(m)
    m = list.assoc.filter(fun (k, _) -> not list.mem(k, ["priv", "rva2"]), m)

[toots]

Hi! For a large database, I would indeed suggest looking at something that is not stored in-memory. The media library is all loaded in memory so it will, indeed, occupy a large amount of it when indexing large set of files.

[Parakovsky]

Thanks. Then we shouldn't bother about it right now! Have a great day!

[Parakovsky]

Good morning Romain, let me know if you need my assistance in experimenting to improve this function, I run my own private radio with occasional DJ sets and generating a custom playlists based on metadata is what I was trying to achieve currently. Have a great day.

[toots]

Yes sure, I think if you could send me a selection of tracks to [email protected] for further inspection that could help!

[Parakovsky]

Will do after i finish that next Kardashians episode!

…

On Wed, Nov 15, 2023, 09:03 Romain Beauxis ***@***.***> wrote: Yes sure, I think if you could send me a selection of tracks to ***@***.*** for further inspection that could help! — Reply to this email directly, view it on GitHub <#3534 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJCGU5SKFJFK2BTW4YFZUUDYERLJLAVCNFSM6AAAAAA7LL4Z3WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TKNZTGAZTS> . You are receiving this because you authored the thread.Message ID: ***@***.***>