Slow Provider Sync

[rfporter]

I have noticed that the time to sync my music library with Music Assistant is abnormally long when compared with other music servers.

I have tried switching off the fetching of metadata and images from other external sources, but it is still very slow.

For example, when syncing from Gonic, it takes in excess of 8 hours to fully sync. For comparison, sync from Gonic to Symfonium takes a whopping 1 minute.

Does Music Assistant have a wait 1s between adding tracks command hiding somewhere?

[OzGav]

@khers

[khers]

Does Music Assistant have a wait 1s between adding tracks command hiding somewhere?

Yes, I write all my code this way, don't you? Seriously, I'm not sure what kind of question this is.

Please provide some details around how this is running. What hardware is MA running on, how big is the Gonic library? What is the internet connectivity of the MA instance?

For comparison, I am using a mid-tier i5 based NUC with HAOS with supposed gigabit down stream (it's comcast so more like 400-600mbps) and I have a modest library (~12k tracks on ~1200 albums from 1100 artists) and my sync takes about 15 minutes initially and a few seconds after that.

[rfporter]

Apologies @khers. Didn't mean to offend. I come from an embedded/PLC/Instrument Driver background. Waits are commonplace and often inserted during testing and forgotten.

The Gonic library is 24k tracks, 1700 albums, 2400 artists. The initial sync takes over 8 hours.
Both Gonic and MA are running on a Synology DS920+ in separate docker containers.
During the sync, CPU usage is less than 5%.
Internet connectivity is on the slow side (100mbps). But I have disabled metadata retrieval from online metadata providers (from the MA metadata controller settings).

[khers]

No offense taken, I just was amused by the question.

[rfporter]

I tried again on my Home Assistant server, which happens to have an SSD. Much faster sync. About 40 minutes for 24k tracks.

I think that the issue is coming from high disk IO during the sync. The DS920+ has slow HDDs, and Music Assistant is making lots of small transactions with the Sqlite database.

So I will change my question to: Has anyone else experienced very slow sync when MA's database is stored on an HDD?

[rfporter]

So Gonic appears to use WAL: https://github.com/sentriz/gonic/blob/master/db/db.go
See their DefaultOptions() function.

To enable this on MA would it be as simple as adding:
await self.execute("PRAGMA journal_mode=WAL;")

to DatabaseConnection.setup()?

[khers]

This is a @marcelveldt question.

[marcelveldt]

We have everything configured on the safe side - performance is not important, stability and low cpu and memory consumption is.
Why? Because in 99% of the cases, MA is configured to run as an add-on to Home Assistant and we do not want some provider sync to be consuming all resources and hinder the actual HA process.

That said, if you see improvements to speed with still keeping our main goals in mind, feel free to PR changes.

I'm not an sqlite expert, but doesn't WAL mode mean that there is a huge risk of data loss if/when the power gets shutdown unexpected ? Now you can argue if that is a real problem as there is not really user data stored.

[khers]

I read the WAL documentation and I don't think it fits our use case. Fundamentally, MA is a read heavy work load, except for the initial sync and the key part of enabling WAL is ensuring frequent enough checkpoints that read performance is not impacted. This would require a full understanding of the IO interactions with the DB and then to redesign that around checkpoints. I don't think it is impossible but I don't have the time to take it on.

[rfporter]

Probably not worth going down the route of optimizing at this stage in the project. But there is definitely some room for improvement here. The initial sync is very very long if your database is on a HDD. Other projects like Gonic and LMS, which do essentially the same thing (build a sqlite database of music meta data), have managed to significantly reduce this sync time.

For now I will just sync on tmpfs then copy the db to my slow HDD. I'm testing out different providers and ID3 tag formats to find the best fit. Fast sync gets me the results within minutes instead of hours.

At least I can rest assured that there is no 1s wait between adding tracks ;)