Add support for hard/soft bloom filter base + stash

[KevinMind]

Fixes: mozilla/addons#15014
Relates to: mozilla/addons#15155

Description

Adds logic to generate and write bloom filters for both soft and hard blocked addons. Additionally this PR introduces logic to determine whether we should update one or both bloom filters and or a stash as multiple possible outcomes are possible now. Finally, we handle cleaning up files on a more granular level from both the local storage and remote settings.

Context

Now when we run the upload_mlbf_to_remote_settings cron job we will check for both hard and soft blocked items. It is possible to:

• do nothing
• upload a hard block filter only
• upload a hard block filter and a stash (for soft blocks)
• upload a soft block filter
• upload a soft block filter and a stash (for hard blocks)
• upload both filters

This adds a bit of complexity we need to address.

Additionally, instead of deleting all records from remote settings, we need to check for the current set of block filters and only delete records older than the older of the two.

Finally, since it is also possible to run the cron when no updates have occurred, we can safely delete mlbf cache files when that happens as there is no benefit from diffing an empty array.

Testing

This is gonna suck to test. First some preparation work.

Setup

• Setup a local remote server here
• Set the base replace threshold to a low number (so you can trigger re-uploading of filters without creating a bunch of blocks)
• enable enable-soft-blocking and blocklist_mlbf_submit waffle switch

src/olympia/constants/blocklist.py

BASE_REPLACE_THRESHOLD = 1

See the test scenarios

from olympia.blocklist.models import BlockType
from olympia.amo.tests import addon_factory, block_factory, version_factory

def _blocked_addon(block_type=BlockType.BLOCKED, **kwargs):
    addon = addon_factory(**kwargs)
    block = block_factory(
        guid=addon.guid, updated_by=user, block_type=block_type
    )
    return addon, block

user = UserProfile.objects.first()

Now you can call the _blocked_addon method to create an addon with block/version of the specified type.

Ex:

_blocked_addon(block_type=BlockType.BLOCKED)
_blocked_addon(block_type=BlockType.BLOCKED)
_blocked_addon(block_type=BlockType.SOFT_BLOCKED)

If you run the cron job now, you'd expect a blocked filter and a stash with the soft blocked version added.

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[willdurand]

Fixes: mozilla/addons#15014
Fixes: mozilla/addons#15166
Rleates to: mozilla/addons#15155

we should almost never fix two issues with a single PR, so please fix mozilla/addons#15166 in a different PR.

[KevinMind]

Fixes: mozilla/addons#15014
Fixes: mozilla/addons#15166
Rleates to: mozilla/addons#15155

we should almost never fix two issues with a single PR, so please fix mozilla/addons#15166 in a different PR.

I've never heard of this rule and regularly do this. Why should that be a rule?

[willdurand]

I am done for today, but that makes it much easier to see what's going on, thanks.

To be continued...

[willdurand]

Some more feedback. This is coming together.

[willdurand]

r+wc, thanks!

[willdurand]

(note: it'd be good if you could clear the commit message before merging)