Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decrease RAM requirements #88

Open
glen3b opened this issue Jan 24, 2019 · 3 comments
Open

Decrease RAM requirements #88

glen3b opened this issue Jan 24, 2019 · 3 comments

Comments

@glen3b
Copy link
Owner

glen3b commented Jan 24, 2019

Score data is very redundant in RAM. We should work on reducing the bot's RAM usage through various means, including improving this.

@glen3b
Copy link
Owner Author

glen3b commented Jan 24, 2019

Possibility: use a relational database

@glen3b
Copy link
Owner Author

glen3b commented Jan 29, 2019

Another possible consideration: binary serialization and caching.

A relational database makes a lot of sense. I'll have to rework how I do "expensive" queries but that's OK.

@glen3b
Copy link
Owner Author

glen3b commented Jan 29, 2019

Thoughts on relational database:

  • Remove the "cheap query" score provider metadata property in favor of...
  • Adding these two functions
        Task<IEnumerable<TResult>> QuerySummaryAsync<TResult>(Func<System.Linq.IQueryable<ScoreboardSummaryEntry>, System.Linq.IQueryable<TResult>> executeQuery);
        Task<IEnumerable<TResult>> QueryDetailsAsync<TResult>(Func<System.Linq.IQueryable<ScoreboardDetails>, System.Linq.IQueryable<TResult>> executeQuery);

This way we can hopefully get the benefits of an IQueryable against a database while abstracting away the async call (which is not in the BCL but is ORM-dependent), and this should work against our existing score provider infrastructure. Assuming this actually works, the biggest issue I can think of is handling aggregate queries. I think the async aggregate calls in EF Core are in EF Core (i.e. not BCL), and I'd rather not have this interface dependent on EF Core. This leaves a few options:

  • We could add IScoreRetrievalService interface methods separately for aggregates, e.g.
        Task<int> QueryDetailsMeanAsync<TResult>(Func<System.Linq.IQueryable<ScoreboardDetails>, System.Linq.IQueryable<int>> selectorQuery);
        Task<int> QueryDetailsSumAsync<TResult>(Func<System.Linq.IQueryable<ScoreboardDetails>, System.Linq.IQueryable<int>> selectorQuery);
        Task<double> QueryDetailsMeanAsync<TResult>(Func<System.Linq.IQueryable<ScoreboardDetails>, System.Linq.IQueryable<double>> selectorQuery);
        Task<double> QueryDetailsSumAsync<TResult>(Func<System.Linq.IQueryable<ScoreboardDetails>, System.Linq.IQueryable<double>> selectorQuery);
        // etc

I think this would get out of hand.

  • Let the aggregates evaluate C#-side. It certainly avoids a hassle, for some cases (e.g. histograms) we have to do this anyway, and at our scale I don't think the performance impact would be that great.
  • Add some interface for aggregate queries that we expose through the library and implement on the database provider side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant