Large Memory Footprint

[JAM-Beni]

Using the Python version of this library: https://pypi.org/project/phevaluator/, there is a large memory footprint of 399.9 MiB when importing evaluate_cards.

Here is the code I used in order to determine this:

from memory_profiler import profile

@profile
def import_phevaluator():
    from phevaluator import evaluate_cards

if __name__ == "__main__":
    import_phevaluator()

and the output:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     4     22.8 MiB     22.8 MiB           1   @profile
     5                                         def import_phevaluator():
     6                                             # The import statement you want to profile
     7    422.7 MiB    399.9 MiB           1       from phevaluator import evaluate_cards

This was causing me to get memory issues on my deployment which is how I noticed the problem in the first place.

Now, I'm not sure why as I believe it's only supposed to be ~100kb for a 7 card evaluation. Is this meant to be on top of the already memory used for initialization? I'm assuming it might be because it's loading most of the tables, but from what I saw it's not importing any PLO which I believe is the bulk of the memory usage.

Please let me know if this is expected and recommendations to reduce this given I only need the standard 7 card evaluation.

[HenryRLee]

I only did memory optimization in C/C++. In C/C++, 7-card evaluation indeed costs 144kb memory. Actually, most of memory usage are indeed constant tables initialized at the global memory region.

But Python is completely different, as it uses an interpreter instead of compiling the code to assembly, it's hard to maintain the same memory usage as in C/C++. An alternative solution is to compile the C library (the pheval7 library) and let your Python code link to that C library. That should achieve the optimal memory usage, i.e. 144kb.

Having said that, hundreds of Mb is way too big. I believe that's probably also loading the tables for Omaha evaluation. You may try removing those Omaha related methods and tables (maybe just do it in __init__.py), see how much memory you can reduce. I am pretty sure it would not be as good as 144kb though.

[JAM-Beni]

Oh yes, that seems to have done the trick. It's now down quite significantly:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     3     22.9 MiB     22.9 MiB           1   @profile
     4                                         def import_phevaluator():
     5     24.2 MiB      1.3 MiB           1       from phevaluator import evaluate_cards

Would you require my support to try and create a PR to amend the import code?

[HenryRLee]

It would be nice if you can fix the import code. Thank you!

[JAM-Beni]

I will spend some time tomorrow making an attempt.

[HenryRLee]

No need to rush. Thanks a lot!

[JAM-Beni]

I ended up doing it before going to bed, see #87

The tests that were written failed for me from the start, wasn't sure if that was a me problem so I left it alone. I'll continue discussion from now on in the PR