Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring of RTL/HLS component integration #928

Merged
merged 336 commits into from
Mar 27, 2024
Merged

Conversation

auphelia
Copy link
Collaborator

@auphelia auphelia commented Nov 28, 2023

With the growth of more and more RTL variants of common FINN hardware building blocks, there is a need to better organize the integration of these components. The purpose of this PR is to make this refactoring visible to the community because it will introduce significant changes in the compiler.

The FINN compiler was developed with the assumption that the hardware blocks corresponding to the neural network layers are developed based on HLS. Although we do not want to abolish this HLS implementation at this time, it has become apparent over the years that for certain modules it makes sense to implement them in RTL. This allows us greater control over the resulting hardware and we can make optimal use of FPGA resources.
To make it easier for RTL and HLS components to both be integrated into the FINN compiler and for users to better make use of one variant or the other, a new custom op class hierarchy will be introduced.

The following steps will be needed for the refactoring:

  • Convert previous HLSCustomOp class to HWCustomOp class and backend mixin classes (HLSBackend and RTLBackend)
  • Introduce new hardware layer intermediate node representation for each node, e.g. FMPadding
  • Introduce backend specific nodes for each layer, e.g. FMPadding_hls and FMPadding_rtl
  • Add transformation to convert from standard and custom ONNX layers to hardware layer representation (will replace previous FINN compiler step "Conversion to HLS layers")
  • Add transformation to convert intermediate hardware layer into either HLS or RTL variant, based on availability (some layers will only have HLS or RTL variants) and other criteria (e.g bit width, fpga parts, ...)
  • Introduce new builder steps
    • for conversion to hardware layers
    • for specialization to HLS or RTL variants
  • Refactor existing builder steps, documentation, tutorials/notebooks, ...

auphelia and others added 29 commits January 30, 2024 17:17
Signed-off-by: aziz bahri <[email protected]>
auphelia and others added 25 commits March 21, 2024 16:21
[Thresholding RTL] Prepend dummy threshold for narrow range quantization
VVAU renaming and disabled resType selection in convert-to-hw
[Tests] Force HLS components for special case cnv-2-2 on u250 and pyn…
@auphelia auphelia marked this pull request as ready for review March 27, 2024 11:59
aziz bahri and others added 2 commits March 27, 2024 14:44
@auphelia auphelia merged commit 68b1a6d into dev Mar 27, 2024
2 checks passed
@auphelia auphelia deleted the refactor/rtl_integration branch March 29, 2024 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants