Cache Refactor

[rossneilson]

Intro

By far the most complex part of DXvote is the cache. While it is very novel and impressive its also still an initial implementation and we can do a lot to improve it. At the end of the day we want to be using the cache as little as possible but this will require adding contract storage and so we have to use what we have now which means relying on event storage so we have to improve the cache.

For anyone unfamiliar you can read an overview here: https://dxdocs.eth.limo/docs/Products/dxgov/DXvote/tehCache/

Whiteboard from discussions

First tasks

First we discussed some of the simpler parts of cache, reputation and permissions. These are relatively simple and not super critical to what we show in the UI so its a good candidate for initial refactors allowing us to architect how we want the code to look. Currently all of the code is in one or two files (cacheService) and so the first aim will be splitting this up to make it more reusable and just more readable.
With reputation and permissions in particular we can also experiment with loading in the background. Currently all loading is done up front so the user cannot see or interact before the entire UI has all of its data, where a lot of current frustrations stem from. So instead we can do what we do in guilds where we load data in the background and show loading in specific components, something we already have good support for.

More complex parts

There is then the other parts of cache which are more difficult to improve. We read controller events to find new scheme registrations meaning we would have to look here to find new schemes and their proposals (something that is unlikely to happen often and could in future be a background task).
Then getting the proposal data from schemes and voting machine events is what is currently very complex and time consuming. In initial refactors we will focus more on just cleaning up code, making it more maintainable. What will improve this part of the code is just upgrading to the new wallet schemes as they are far simpler as we do not need to fetch raw events and can instead directly access contract storage for proposal specific data. We will make improvements and separate as much as we can the legacy from the new code so that we keep legacy code the same, only improving new wallet scheme code.

State management

This will be a big decision as we currently use mobx and it is not ideal. There are a few ways we could go here, we are going to be splitting cache up more and so vanilla react contexts are maybe not enough as things get more complex. I have bad experience with redux but I have heard it has gotten far better with hooks. We can maybe use swr in some way to automate some of the caching on client side. But we have also discussed solutions like indexie using index db in browser which could be a very fitting solution since we basically have a client side database and could even import and export from indexed db and of course we get persistent browser caching.

Testing

This is complex logic and it needs to be tested to improve develop understanding of the code alongside of course just cleaning it up and documenting.

Raw events

Augusto explained the difference between raw events and normal events. He may be better clarifying this but from my understanding:

• Raw events (getRawEvents) are much less targetted and involves more processing on our side but they also pass back a data object which for many legacy schemes is required.
• Normal events (getEvents) is more targetted on indexed data for specific topics and therefor should be what we aim to use (I believe it also is better supported by RPCs with less chance of being deprecated).

Conclusions

This is something we absolutely need to focus on as we adapt DXvote into the new UI in preparation for v1.1 the cache must be improved also. Of course, we need to make optimisation to how we load the data but a lot of focus also needs to go into improving readability, structure, documentation and developer understanding. When someone types in chat asking why the cache isn't loading (hopefully less frequently after this) it should be able to be debugged by everyone in the team and even new developers.

So going forward I think its best we use the same approach as we did for UI refactors. We will have more mobbing sessions for us all to talk through our decisions to build a checklist of changes and guidelines for the refactor. In these sessions we can decide structure, state solutions and take turns building a PR with well documented changes and steps. We will take the example of reputation as its core logic is simplest.