Drastically increase performance by rethinking the SDK store

[krille-chan]

This is a refactoring proposal about the Matrix Dart SDK to improve the start-up performance and memory usage of apps using the Matrix Dart SDK.

What is the current state?

On init the SDK loads all rooms from the store into memory. But also a lot of States which are needed to display a nice room list in the UI. This includes:

• Room state, room members and room type (from account data) to calculate the name of the room
• Room state or room members to have an avatar url
• The last event and the room member event of the sender to display a last message preview
• The timestamp of the last event
• The unread count
• If the room is pinned or muted
• Room type (is it a DM room? Special type?)

So on app start the SDK loads a lot of stuff from the store to display the room list:

• All room objects
• All account data objects
• All room account data objects
• A lot of room state objects

And this needs some time and scales badly. Also this may need too much memory.

Ideas

1. We make the database of the SDK non-nullable
2. We switch to FluffyBox: feat: Implement FluffyBox database #1541
3. We do not preload account data, room account data or room states at all and make all getters for this a future, while the store is responsible for a reasonable caching (FluffyBox can do this)
4. We make all getters to account data, room account data, room state, device keys a Future<Type>
5. We store additional properties per room in a new RoomListItemSummary object:
   • Room displayname String
   • Room avatar URL URI
   • Room last message preview String
   • Room last activity timestamp DateTime
   • Is DM room bool
   • Is pinned bool
   • Is marked as unread bool (also triggered to be toggled like "hasNewMessages")

This is then indeed duplicated in the database but it means that to start the app we only need to load all room objects from the rooms table of the database which should work much much faster.

This way we would include some logic on the one side but also can remove them on the other side. We wouldn't need other getters anymore which currently search inside of state events to calculate stuff for us.

If a specific app needs additional information, this could become tricky though. I would suggest that we make the RoomListItemSummary object expandable in a way that it contains a custom map for additional information and an API so that the SDK consumer can inject a dart function to react on sync to update this map. For example could the room type be an additional parameter which then updates by a dart functionality written by the SDK consumer.

Pros

App would start much faster and consume less memory. The overall performance will probably drastically improve.

Cons

We would have some duplicated data where we always have the risk that the can become out of sync. More specific use cases like the need for the room type could become more complicated. Accessing other parameters always as a Future could also produce problems.

[techno-disaster]

Other suggestions added from internal meet:

• provide a local box for clients consuming the sdk to use it for misc storage
• having separate timelines to avoid arduous filtering for visible items

[MichalNemec]

#1601 probably related

[predictron-cloud]

Extremely desirable fix! Will it also fix, that the Fluffychat becomes laggy if the user have a lot of huge rooms (like Matrix HQ etc.)?

Also maybe "Room topic String" can be also added to the RoomListItemSummary (or if the "I would suggest that we make the RoomListItemSummary object expandable in a way that it contains a custom map for additional information and an API so that the SDK consumer can inject a dart function to react on sync to update this map." is implemented, it can be done with it)