WIP: Added implementation of feature_permutation

[andersbogsnes]

Working on issue #59

[codecov]

Codecov Report

Merging #91 into develop will decrease coverage by 2.09%.
The diff coverage is 14.28%.

@@            Coverage Diff             @@
##           develop      #91     +/-   ##
==========================================
- Coverage    97.41%   95.31%   -2.1%     
==========================================
  Files            7        7             
  Lines          541      555     +14     
  Branches        69       71      +2     
==========================================
+ Hits           527      529      +2     
- Misses           7       19     +12     
  Partials         7        7

Impacted Files	Coverage Δ
src/ml_tooling/metrics.py	77.35% <14.28%> (-22.65%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3653b8b...a81de62. Read the comment docs.

[thomasfrederikhoeck]

This implementation can be multi-threaded. I can give it at shot at a later point.

Leaving this link for later:

TeamHG-Memex/eli5#244

[thomasfrederikhoeck]

As this is similar to ELI5's PermutationImportance I suggest the following features that it has:

• Multiple permutation of the same feature to improve estimate of importance.

• Option for Cross-Calidation if model is refitted. This can be used to show how important the features are for generalization.

[thomasfrederikhoeck]

Isn't y training/test target?

[andersbogsnes]

Yeah, changed the implementation without updating docs - still have some work to do on this feature 😄

[andersbogsnes]

I'll have a look at their implementation - see if there's some cleverness we can use