Use internal data object for MLBF to accommodate soft/hard block versions

[KevinMind]

Relates to: mozilla/addons#15014

Description

Adds the concept of a data property to MLBF. This allows storing of both soft/hard blocks without actually modifying how stashes/filters are generated

Context

This PR doesn't change the underlying behaviour of the bloomfilter but modifies the data structure used for storing data in the MLBF class. Adding data for soft blocked versions and then handling the creating of filters or stashes based on that data will be much easier after this PR.

Specifically this PR ensures that data is loaded consistently via storage or database and that all data is loaded on instantiation. This ensures filters produced contain valid data.

Testing

You can create a blockfilter locally and ensure it has correct data.

./manage.py export_blocklist

Expect a directory to be created in ./storage/mlbf matching either the parameter you specified for id or a timestamp.

Expect two files cache.json containing the blocked/not_blocked versions for that data type (in this case always hard blocked) and filter where the suffix matches the data type (in the future it could be blocked or soft_blocked)

Expect the strings in the blocked property of cache.json to match the versions in the query above if you search for hard blocks.

Scenario 2: create a stash

Now you need to add a new block, using the django shell.
Next create another filter using the same command above.

Now, open a django shell and we can create a stash (make sure mlbf is the latest filter and previous is the first filter you created)

mlbf = MLBF.load_from_storage(<second_id>)
previouis = MLBF.load_from_storage(<first_id>)

Verify you can diff the filters and see what has changed

mlbf.blocks_changed_since_previous(previous_mlbf=previous)

The value should return 1 if you only added one additional block.
You can also pass None for previous and get back the number of blocks there are currently (2)

Now you can generate a stash

mlbf.generate_and_write_stash(previous_mlbf=previous)

This should create a new directory matching the id of the mlbf and should contain 2 files cache.json and now stash.json.

The cache.json should contain the new block you created.

{"unblocked": [], "blocked": ["{0f1fbd83-d0fe-42ef-b6ce-365860d6ca10}:23609.517.8654.43211"]}

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[eviljeff]

🤔

[eviljeff]

looks good for now - I'm assuming they'll be a fair bit of reworking needed once we're building an additional soft-blocked bloom filter, and including details of both hard and soft in the stashes.