[Bug]: Ensure stashes are always newer than the base bloom filters

[willdurand]

What happened?

Firefox downloads base bloom filters and stashes from the Remote Settings collection. An assumption when we worked on #15014 was that it would probably fine to have stashes older than the base bloom filters, which - in retrospective - makes little sense. Stashes are supposed to always be newer than the base filters. We cannot really change this assumption since we would break the hard blocklist for all clients.

That being said, this is mostly working except for unblocked add-ons when:

1. a version was hard blocked in a stash S
2. this stash S is newer than the oldest bloom filter (e.g. soft filter)
3. enough hard block changes are made, including the version being unblocked, which triggers a new hard filter
4. since the soft filter is older than the stash S, we keep it in the collection

In this case, Firefox will download 2 base filters and the stash S. Firefox will see the hard block in the stash S and mark the version as hard blocked. Firefox will check the new hard filter but it only looks for blocked versions. Firefox will not unblock the version.

This is likely only a problem for a version that is blocked in a stash, and later unblocked in its base filter. Blocks changing from one type to the other aren't affected by this.

In short, AMO must ensure that the Remote Settings collection only contains stash records newer than any filters, always.

Now what?

At the moment, we have two options that the team wants to explore:

1. we always regenerate the base bloom filters when one of the two block types has reached the threshold, so that we can remove all the stash records
2. when a base filter is regenerated for a block type, we remove all the stash records and we build a new stash for the other block type (that is the difference between its last base bloom filter and "now")

What did you expect to happen?

Not that.

Is there an existing issue for this?

• I have searched the existing issues

┆Issue is synchronized with this Jira Task

[willdurand]

Meeting notes: https://docs.google.com/document/d/1qnP8VwB7TRpOuS-HEZ7ZpPLLKkL2rcd3M_cuPib9jHI/edit?usp=sharing

[diox]

We have several possible solutions that need to be evaluated to figure out if these really solve our problem and what are the drawbacks, and sync up with the rest of the team to make sure everyone agrees.