Improve cache clearing

[abitofevrything]

Description

Rewrites the caching code to not hold on to outdated objects and empty caches.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist:

• Ran dart analyze or make analyze and fixed all issues
• Ran dart format --set-exit-if-changed -l 160 ./lib or make format and fixed all issues
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have checked my changes haven't lowered code coverage