[Attention block] relative positional embedding

[vgrau98]

Fixes #7356

Description

Add relative positinoal embedding in attention block as described in https://arxiv.org/pdf/2112.01526.pdf
Largely inspired by https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py

Can be useful for #6357

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[marksgraham]

I wonder if this is an example of the point we should start splitting out different types of attention block into their own blocks? Thoughts @KumoLiu @ericspod ?

Note this also will have some merge conflicts as it stands with the local window pr

[ericspod]

@marksgraham @KumoLiu I feel with the generative merging in progress we should think about refactoring how these blocks work in general to try and separate these differing features into separate classes. We can perhaps have a single attention block class that is customisable with features, but if not we can do multiple classes but doing this cleanly will probably involve refactoring the existing things that have been merged from GenerativeModels. This PR adds something good however so we should keep this open but postpone until we have the other things merged perhaps.

[vgrau98]

Following @ericspod suggestions:
Use factory pattern to implement relative positional embedding in self attention block (use LayerFactory for now but open to any suggestion to better organise code). Anyone can extend the RelPosEmbedding feature, or add new transformer/attention features

[KumoLiu]

Thanks for the quick update, looks cleaner now. Only have a few concern inline.

[ericspod]

I think this is looking much better. Would it be possible to merge into gen-ai-dev instead of dev so we can continue the generative refactoring? @KumoLiu @marksgraham any other feedback?

[vgrau98]

I think this is looking much better. Would it be possible to merge into gen-ai-dev instead of dev so we can continue the generative refactoring? @KumoLiu @marksgraham any other feedback?

target branch updated to gen-ai-dev

[KumoLiu]

Hi @vgrau98, could you please try git rebase --onto <newparent> <oldparent> <until> to rebase your commit onto the gen-ai-dev, which may avoid merging the commit added from the dev branch (causing DCO error)?
https://womanonrails.com/git-rebase-onto

[vgrau98]

Hi @vgrau98, could you please try git rebase --onto <newparent> <oldparent> <until> to rebase your commit onto the gen-ai-dev, which may avoid merging the commit added from the dev branch (causing DCO error)? https://womanonrails.com/git-rebase-onto

should be ok now

[KumoLiu]

/build

[KumoLiu]

Hi @vgrau98, could you please check-pick the commit here after it merged which caused a CI error and could not merge?
Thanks.

[vgrau98]

Hi @vgrau98, could you please check-pick the commit here after it merged which caused a CI error and could not merge? Thanks.

Is it still relevant? The bug seems to have been resolved according to your comments on PR #7391

[KumoLiu]

Yes, I'm creating another PR to fix the error caused by the upgrade of flake8-bugbear
https://github.com/Project-MONAI/MONAI/actions/runs/7544331654/job/20537411217#step:2:2601

You can either cherry-pick that one or make the same change in this PR.

[KumoLiu]

/build