Benchmark metrics: model FLOPs utilization for @onefact benchmark?

[jaanli]

Hi! My team at @onefact is excited to benchmark this awesome project for our RAG tasks.

quick question: how can we accurately estimate the any-to-any MFU metric please? Because it is any-to-any, we need a good sense of the pre-training mix dataset statistics (median length of tokens, etc).

We also need to run it on edge devices. Have you figured out how to distill it for iPhone pro max 15 devices? WASMEdge and WebAssembly have been working well for us; feel free to email me: [email protected].