Fix issues with macOS in CI

[lebrice]

Fix some weird issue that seem to be related to using setup-python with cache: poetry on MacOS, for example here: https://github.com/mila-iqia/milatools/actions/runs/8834122225/job/24257059736?pr=112#step:4:53

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 44.98%. Comparing base (3a1237b) to head (eac9490).
Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #120       +/-   ##
===========================================
- Coverage   65.80%   44.98%   -20.82%     
===========================================
  Files          12       11        -1     
  Lines        1778     1776        -2     
===========================================
- Hits         1170      799      -371     
- Misses        608      977      +369     

Flag	Coverage Δ
integrationtests	?
unittests	44.98% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.