[Task]: Generate separate bloomfilter for soft-blocks

[diox]

Description

Depends on #15012

Once the backend for soft-blocking is done, we want to be able to generate a separate bloomfilter for soft-blocks and make it have its own set of stashes. The "original" bloomfilter and stashes for blocks should then only consider hard-blocked version as blocked.

NOTE: This ticket does not have stashes in scope. Those will be implemented in #15050

Acceptance Criteria

Milestones/checkpoints

Preview Give feedback

1. Original bloomfilter should only consider hard-blocked versions as blocked
2. Separate bloomfilter should be available, only considering soft-blocked versions as blocked

Scenarios

The following are some of the edge cases that we ran into during development. All of these should be run in dev/stage to ensure we produce the expected stash/filter records in remote settings. Some scenarios are the sum of other scenarios.

Prerequisites:

• open the admin
• open zadmin configs
• create a config blocklist_base_replace_threshold and set the value to a low number e.g. 2

For each scenario:

• create the necessary block additions/deletions/updates
• open the admin, navigate to blocks
• click "Upload MLBF" button to trigger the cron job to update the blocklist
• Verify the expectations in remote settings

1. add enough hard blocks to recreate hard block filter

• Expect a new bloomfilter-base filter in remote settings
• Expect any pre-existing filters of the same attachment_type to be deleted
• If the previous filter was the oldest, then expect any stashes older than it to be deleted

2. add enough hard block to create a stash, but not a filter

• Expect a new stash record in remote settings including the newly blocked versions

Note

'stash_time' should match the time at which we ran the cron

{
   'stash': {
      'blocked': ['{eb478b77-4d0e-47b3-b1a6-e99293dbac45}:8608.50922.39325.18898'],
      'unblocked': [],
      'softblocked': []
   },
   'key_format': '{guid}:{version}',
   'stash_time': 1732641041453,
   'id': '8f14e38e-605d-439f-ae1c-15537fd799a4',
   'last_modified': 1732641041544
}

3. add enough soft blocks to recreate a soft block filter

• Expect a new softblocks-bloomfilter-base filter in remote settings
• Expect any pre-existing filters of the same attachment_type to be deleted
• If the previous filter was the oldest, then expect any stashes older than it to be deleted

4. add enough soft blocks to create a stash, but not a filter

• Expect a new stash record in remote settings including the newly softblocked versions

Note

'stash_time' should match the time at which we ran the cron

{
   'stash': {
      'blocked': [],
      'unblocked': [],
      'softblocked': ['{eb478b77-4d0e-47b3-b1a6-e99293dbac45}:8608.50922.39325.18898']
   },
   'key_format': '{guid}:{version}',
   'stash_time': 1732641041453,
   'id': '8f14e38e-605d-439f-ae1c-15537fd799a4',
   'last_modified': 1732641041544
}

5. Add enough hard blocks to create a new filter + add not enough soft blocks to create a new filter (hard filter + soft stash)

For the hard block:

• Expect a new bloomfilter-base filter in remote settings
• Expect any pre-existing filters of the same attachment_type to be deleted
• If the previous filter was the oldest, then expect any stashes older than it to be deleted

For the soft block:

• Expect a new stash record in remote settings including the newly softblocked versions
• The stash_time and generation_time of the filter and stash should match

6. Add not enough hard blocks to create a new filter + add enough soft blocks to create a filter (hard stash + soft filter)

For the hard block:

• Expect a new stash record in remote settings including the newly blocked versions
• The stash_time and generation_time of the two records should match

For the soft block:

• Expect a new softblocks-bloomfilter-base filter in remote settings
• Expect any pre-existing filters of the same attachment_type to be deleted
• If the previous filter was the oldest, then expect any stashes older than it to be deleted

7. soften not enough hard blocks to create a new soft filter (soft stash)

• Expect a new stash record in remote settings including the now softblocked versions

Note

You don't expect the block to be included in the notblocked list, only the softblocked

8. soften not enough hard blocks to create a new filter + add not enough soft blocks to create a new filter (soft stash)

• Expect a new stash record in remote settings including the newly softblocked version and the softened version

9. harden not enough soft blocks to create a new soft filter (hard stash)

• Expect a new stash record in remote settings including the now softblocked versions
• Expect empty soft and unblocked lists

10. harden not enough soft blocks to create a new soft filter + add not enough hard block to create a new filter (hard stash)

• Expect a new stash record in remote settings including the now hard blocked versions and the newly soft blocked versions
• Expect empty soft block and unblock lists

11. Delete a block, update the block list, and recreate the same block

STR:

• create a block (of either block type)
• run generate mlbf
• expect: stash with block in the selected block type
• delete the block
• run generate mlbf
• expect: stash with block in notblocked list
• recreate the block for the same guid (same or different type)
• run generate mlbf
• expect: stash with block in the type you specified the second time

Note

This should work for any combination of hard and soft blocks assuming you do not reach either threshold for recreating the given filter

Checks

• If the issue is ready to work on, I have removed the "needs:info" label.
• If I have identified that the work is specific to a repository, I have removed "repository:addons-server" or "repository:addons-frontend"

┆Issue is synchronized with this Jira Task

[willdurand]

@diox while the base (full) bloom filter should be a distinct attachment, we'd want to have a single stash with both the hard and soft blocks. Something like this:

{
  "stash": {
    "blocked": [
      "[email protected]:1.1",
      "[email protected]:1.2"
    ],
   "softblocked": [
      "[email protected]:1.3"
   ],
    "unblocked": [
      "[email protected]:1.0"
    ]
  },
  "key_format": "{guid}:{version}",
  "stash_time": 1587990900999,
}

[KevinMind]

@willdurand do we have an understanding of the data structure expected by FX? Does it match the json structure you've presented above?

[willdurand]

Yeah, it does match the json structure above. blocked and unblocked are required. softblocked is not required, but if it is specified, then it must be an array.

[ioanarusiczki]

I tried on dev today got to scenario 5 (had some issues about dicts, lang packs that needed to be clarified so I interrupted this)

All looked good until scenario 5. I am not sure why I've a guid at 3 [email protected]:4.4.3 (dictionary)

config set to 2, I've made
3 blocks for
{fbe76d50-e79a-4262-8649-b102e6c14b05}
{76ce19b2-9155-44ca-bc9a-b301e3cad2a0}
{32e77121-7799-4d29-a8a4-96b06204ccaa}

2 restrictions for
{20a53c5d-b1b9-4098-956b-cdb1ec429992}
{ff4177a2-f967-469f-a9e0-ccb4fe21ae89}

https://remote-settings-dev.allizom.org/v1/buckets/blocklists/collections/addons-bloomfilters/records now

later edit it's #15214

[willdurand]

later edit it's #15214

Yes, and FTR the collection looks like this:

{
  "data": [
    {
      "schema": 1732887589439,
      "attachment": {
        "hash": "5ffacca4d39196f792956f7733ecc421a4b916126572fefdd169de34c85fff75",
        "size": 1845,
        "filename": "filter.bin",
        "location": "staging/addons-bloomfilters/fcda007e-3a9f-4b4e-a4b6-82c034987905.bin",
        "mimetype": "application/octet-stream"
      },
      "key_format": "{guid}:{version}",
      "attachment_type": "bloomfilter-base",
      "generation_time": 1732892587553,
      "id": "707dc060-839c-4212-941b-d39285f8a5ae",
      "last_modified": 1732892593067
    },
    {
      "stash": {
        "blocked": [],
        "unblocked": [],
        "softblocked": [
          "{20a53c5d-b1b9-4098-956b-cdb1ec429992}:2.0",
          "{ff4177a2-f967-469f-a9e0-ccb4fe21ae89}:2.0"
        ]
      },
      "schema": 1732892592675,
      "key_format": "{guid}:{version}",
      "stash_time": 1732892587553,
      "id": "59585b7a-6d22-4df0-89a0-74ada37006fa",
      "last_modified": 1732892593064
    },
    {
      "schema": 1732882989019,
      "attachment": {
        "hash": "f9c5e9d82d880709c79d466a7b4dca68cb369922f425f883a053834f74dc8c47",
        "size": 2334,
        "filename": "filter.bin",
        "location": "staging/addons-bloomfilters/3fb3e734-6c3b-409c-be60-b9eab44994a7.bin",
        "mimetype": "application/octet-stream"
      },
      "key_format": "{guid}:{version}",
      "attachment_type": "softblocks-bloomfilter-base",
      "generation_time": 1732887584214,
      "id": "c16628a5-fe0e-4fff-a3c6-848126cea627",
      "last_modified": 1732887589318
    },
    {
      "stash": {
        "blocked": [
          "[email protected]:4.4.3"
        ],
        "unblocked": [],
        "softblocked": []
      },
      "schema": 1732887589012,
      "key_format": "{guid}:{version}",
      "stash_time": 1732887584214,
      "id": "2a1df6c6-7f27-43a0-aafe-2e95f3d5206d",
      "last_modified": 1732887589315
    }
  ]
}

[ioanarusiczki]

I am not sure of any of the test results I got today, and confused about some scenarios

7. soften a hard block (not enough to create a new soft filter)

Expect (2) for the block you hardened

When I soften a hard block shouldn't I be expecting(4) instead of (2) ?

9. harden a soft block (enough to create a new soft filter)

Expect (4) for the block you softened

I think here's expecting(2) ?

11. Delete a block, update the block list, and recreate the same block

What's the exact str?
I tried like this:
delete the block
generate mlbf
block again the same guid
generate mlbf

or it's

delete a block/restriction
block/restrict again the same guid
generate mlbf ?

[KevinMind]

@ioanarusiczki I updated the description. I hope this helps make the scenarios more clear. I was trying to save space/time with the addition logic but I think that led to mistakes and confusion.

[ioanarusiczki]

@KevinMind

Tried yesterday 1-4
I've tried again scenarios from 5-10
11 has been discussed.

I think it should be alright. What got me confused was that sometimes there were old stashes not deleted (thing in which I've run again today, but it's a known issue)

I'm going to set back to 5000 the config on dev.
Extensions and themes installed in browser were blocked , restricted , softened, hardened or block deleted.
Lang packs and dicts - not in scope (these have a signature issue)

I keep having remote settings errors yet the changes made on the extensions are there after the sync. -> maybe I should ping again someone from remote settings

[KevinMind]

cc @willdurand I'm not sure if these signature errors are in scope for us or if they are maybe a part of the changes you had mentioned in remote settings?

[willdurand]

These errors seem to be unrelated.