Issue in Python vs Mojo benchmark for Matrix multiplication

[armgabrielyan]

In the Matrix multiplication notebook, I am getting very weird results when benchmarking "Python implementation" and "Importing the Python implementation to Mojo". The naive "Python implementation" results in $0.002352014789024577 \text{ GFLOP/s}$ and the "Importing the Python implementation to Mojo" results in $79137.811320754714 \text{ GFLOP/s}$. This is a $33646817.05661153$ (more than $33$ million 🤯) speedup over Python. Despite sounding very impressive, this looks very unrealistic.

Please note that this is not reproducible for other Mojo implementations. Also, I have changed matrix dimensions from $512$ to $128$ and that is the only change.

"Python implementation"

"Importing the Python implementation to Mojo"

Upon further inspection, I noticed that in the benchmark_matmul_untyped function the call to the matmul_untyped function is wrapped in a try-except block. I checked and it looks like the matmul_untyped function raises an error most of the time during the benchmark, which affects the results. Nevertheless, I am not able to reproduce this issue outside the benchmark_matmul_untyped benchmarking function.

Could you please help me figure this out?

[mojodojodev]

This is because in the second screenshot an error is being thrown, so it goes to the except block and runs pass, which takes almost no time. Something inside the matmul_untyped def is causing it to fail, but that isn't included in the screenshot.

[armgabrielyan]

This was taken from Matmul notebook and I did not make any changes there. I believe that the issue is still reproducible if this portion of code is not changed.

[mojodojodev]

I’ve run through the notebook a number of times and it hasn’t been an issue, and tried again yesterday and it worked fine. Perhaps there was just some bug with the kernel when you ran it. If it happens again try running the function once outside the benchmark to see what’s causing it to fail for you.