Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SYSTEMDS-3534 Cache for build phase in bin and recode encoder #2036

Open
wants to merge 99 commits into
base: main
Choose a base branch
from
Open

SYSTEMDS-3534 Cache for build phase in bin and recode encoder #2036

wants to merge 99 commits into from

Commits on Jun 2, 2024

  1. scaffold for cache class

    @Mayaryin
    Mayaryin committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    26e6a05 View commit details
    Browse the repository at this point in the history

  2. Add put and get methods to the cacheMap

    @ingunnaf
    ingunnaf committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    1c71262 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Changing the cache to a singleton class, and adding the check of the … 

    …cache to the build class
    @ingunnaf
    ingunnaf committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    4e80a9a View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2024

  1. test data, spec and dml script

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    fb1ae4d View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/feature/cache' into feature/cache

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    b71820c View commit details
    Browse the repository at this point in the history

  3. toString

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    2be43fd View commit details
    Browse the repository at this point in the history

  4. debug printouts

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    addda62 View commit details
    Browse the repository at this point in the history

  5. specs for testing

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    3c02b1e View commit details
    Browse the repository at this point in the history

  6. criteo dataset

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    eb4acd5 View commit details
    Browse the repository at this point in the history

  7. debug printouts

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    8af3d26 View commit details
    Browse the repository at this point in the history

  8. test dml script

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    074e00c View commit details
    Browse the repository at this point in the history

  9. added hash function

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    67ee253 View commit details
    Browse the repository at this point in the history

  10. first cache test

    @Mayaryin
    Mayaryin committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    1452aba View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2024

  1. [SYSTEMDS-3534] - added debug logs

    @Mayaryin
    Mayaryin committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    2c308b9 View commit details
    Browse the repository at this point in the history

  2. [SYSTEMDS-3534] - first test for recode encoder

    @Mayaryin
    Mayaryin committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    9d0deb8 View commit details
    Browse the repository at this point in the history

  3. Create class to store bin boundaries

    @ingunnaf
    ingunnaf committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    525db76 View commit details
    Browse the repository at this point in the history

  4. Extend CacheKey to include binMethod so we can cache build result bas… 

    …ed on binMethod
    @ingunnaf
    ingunnaf committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    f21d4bd View commit details
    Browse the repository at this point in the history

  5. Cache the build result from the binning

    @ingunnaf
    ingunnaf committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    29d0e4c View commit details
    Browse the repository at this point in the history

  6. throw exception if someone tries to create a cachekey with bin encodi… 

    …ng without specifying bin method
    @ingunnaf
    ingunnaf committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    e1c8733 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. [SYSTEMDS-3534] - added debug logs

    @Mayaryin
    Mayaryin committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    4f59ac4 View commit details
    Browse the repository at this point in the history

  2. [SYSTEMDS-3534] - extended to test caching in binning encoder

    @Mayaryin
    Mayaryin committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    d71c4be View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Merge branch 'apache:main' into feature/cache

    @Mayaryin
    Mayaryin authored Jun 25, 2024
    Configuration menu
    Copy the full SHA
    1769f72 View commit details
    Browse the repository at this point in the history

  2. [SYSTEMDS-3534] - changed log level in test for demonstration purposes

    @Mayaryin
    Mayaryin committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    716a10d View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. [SYSTEMDS-3534] - renamed CacheKey to EncodeCacheKey

    @Mayaryin
    Mayaryin committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    55c6a18 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. [SYSTEMDS-3534] - Created scaffold and ToDos for the EncodeBuildCache

    @Mayaryin
    Mayaryin committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    6d2238c View commit details
    Browse the repository at this point in the history

  2. [SYSTEMDS-3534] - adapted changes in encoder classes

    @Mayaryin
    Mayaryin committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    48ab3ea View commit details
    Browse the repository at this point in the history

  3. [SYSTEMDS-3534] - added size computation

    @Mayaryin
    Mayaryin committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    e8558d4 View commit details
    Browse the repository at this point in the history

  4. Implement put logic in the cache

    @ingunnaf
    ingunnaf committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    47a3596 View commit details
    Browse the repository at this point in the history

  5. Implement put logic in the cache

    @ingunnaf
    ingunnaf committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    8399b6a View commit details
    Browse the repository at this point in the history

  6. Remove duplicate function, rename the new synchronized getEncodeBuild… 

    …Cache
    @ingunnaf
    ingunnaf committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    01d05ed View commit details
    Browse the repository at this point in the history

  7. Remove and add key from queue to put recently accessed item at the en… 

    …d of the queue
    @ingunnaf
    ingunnaf committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    3df8e11 View commit details
    Browse the repository at this point in the history

  8. Only use the cache in Recode and Bin if it is enabled

    @ingunnaf
    ingunnaf committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    ca80e24 View commit details
    Browse the repository at this point in the history

  9. Remove Caffeine from the pom.xml

    @ingunnaf
    ingunnaf committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    be862a4 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. Use cacheLimit instead and remove duplicate computation of the limit

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    aedd8da View commit details
    Browse the repository at this point in the history

  2. Move if-else with binmethod to separate method and make logic turning… 

    … on or off the cache cleaner
    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    8b661ff View commit details
    Browse the repository at this point in the history

  3. Set input values to final in BinMinsMaxs

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    5033d59 View commit details
    Browse the repository at this point in the history

  4. Set the put and get methods in cache to synchronized

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    4a6ea8c View commit details
    Browse the repository at this point in the history

  5. Add a setSize and do the size computation there instead, and make get… 

    …Size just return the size
    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    310c4b9 View commit details
    Browse the repository at this point in the history

  6. Set this._size instead of _size

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    f807f1a View commit details
    Browse the repository at this point in the history

  7. Rename setSize to computeSize

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    ed84f76 View commit details
    Browse the repository at this point in the history

  8. [SYSTEMDS-3534] - wrapper class for rcdMap, including its estimated size

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    bdb2b71 View commit details
    Browse the repository at this point in the history

  9. [SYSTEMDS-3534] - added getter for cache entry size

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    de34a44 View commit details
    Browse the repository at this point in the history

  10. [SYSTEMDS-3534] - removed timestamp, removed raw use of EncodeCacheEntry

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    291f811 View commit details
    Browse the repository at this point in the history

  11. [SYSTEMDS-3534] - added sizeEstimation to build method

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    d888cf8 View commit details
    Browse the repository at this point in the history

  12. [SYSTEMDS-3534] - removed raw definition of class parameter map

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    6342740 View commit details
    Browse the repository at this point in the history

  13. [SYSTEMDS-3534] - revert to raw use

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    6b6445f View commit details
    Browse the repository at this point in the history

  14. [SYSTEMDS-3534] - added single colum test for encode cache

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    208ab76 View commit details
    Browse the repository at this point in the history

  15. Use DMLConfig and ConfigurationManager to toggle cache on and off

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    e612f45 View commit details
    Browse the repository at this point in the history

  16. [SYSTEMDS-3534] - added cache policy parameter to EncodeBuildCache an… 

    …d moved eviction logic to seperate method. Pulls policy from cache config. added switch statement for extendability purposes
    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    b186e4c View commit details
    Browse the repository at this point in the history

  17. Merge remote-tracking branch 'origin/feature/cache' into feature/cache

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    b6e0742 View commit details
    Browse the repository at this point in the history

  18. use a factory method to create binminsmaxs instead

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    1be8855 View commit details
    Browse the repository at this point in the history

  19. add test

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    142cc62 View commit details
    Browse the repository at this point in the history

  20. Merge branch 'feature/cache' of https://github.com/Mayaryin/systemds-… 

    …feature-3534 into feature/cache
    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    51111e1 View commit details
    Browse the repository at this point in the history

  21. [SYSTEMDS-3534] - Scaffold for unit tests

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    4aba1f2 View commit details
    Browse the repository at this point in the history

  22. add error message to binminmax if lenght of binmins and binmax are no… 

    …t the same
    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    5c384d9 View commit details
    Browse the repository at this point in the history

  23. Merge branch 'feature/cache' of https://github.com/Mayaryin/systemds-… 

    …feature-3534 into feature/cache
    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    daa8092 View commit details
    Browse the repository at this point in the history

  24. [SYSTEMDS-3534] - removed unused code

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    0d4d32a View commit details
    Browse the repository at this point in the history

  25. Merge remote-tracking branch 'origin/feature/cache' into feature/cache

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    f98f27f View commit details
    Browse the repository at this point in the history

  26. [SYSTEMDS-3534] - added cache flag default

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    8542441 View commit details
    Browse the repository at this point in the history

  27. [SYSTEMDS-3534] - added to string method

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    1dfc194 View commit details
    Browse the repository at this point in the history

  28. [SYSTEMDS-3534] - added to string method

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    0f80a85 View commit details
    Browse the repository at this point in the history

  29. Read memory fraction from the global config file, use the cache confi… 

    …g class as intermediary to make testing easier
    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    0e33e4e View commit details
    Browse the repository at this point in the history

  30. Merge branch 'feature/cache' of https://github.com/Mayaryin/systemds-… 

    …feature-3534 into feature/cache
    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    c698714 View commit details
    Browse the repository at this point in the history

  31. [SYSTEMDS-3534] - added to string method

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    240c514 View commit details
    Browse the repository at this point in the history

  32. [SYSTEMDS-3534] - moved evition logic out of the get method

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    40dfc31 View commit details
    Browse the repository at this point in the history

  33. [SYSTEMDS-3534] - scaffold for unit testing

    @Mayaryin
    Mayaryin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    b2f0774 View commit details
    Browse the repository at this point in the history

  34. Add licenses

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    7eff634 View commit details
    Browse the repository at this point in the history

  35. Add licenses, remove binminsmaxtest

    @ingunnaf
    ingunnaf committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    2d6cba8 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. [SYSTEMDS-3534] - remove feature dir

    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    8a79a47 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/feature/cache' into feature/cache

    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    cdbea46 View commit details
    Browse the repository at this point in the history

  3. [SYSTEMDS-3534] - remove feature dir

    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    ebf653c View commit details
    Browse the repository at this point in the history

  4. [SYSTEMDS-3534] - remove config flags from dml config, hardcode setti… 

    …ngs in cache config
    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    66bd6b2 View commit details
    Browse the repository at this point in the history

  5. [SYSTEMDS-3534] - cleanup

    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    f5ef6ab View commit details
    Browse the repository at this point in the history

  6. [SYSTEMDS-3534] - added methods for testing, removed runtime exceptio… 

    …n for uncachable build results
    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    c8de628 View commit details
    Browse the repository at this point in the history

  7. [SYSTEMDS-3534] - added unit tests for encode cache

    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    be6730c View commit details
    Browse the repository at this point in the history

  8. [SYSTEMDS-3534] - changed log level to debug

    @Mayaryin
    Mayaryin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    a58e883 View commit details
    Browse the repository at this point in the history

  9. Remove explicit synchronization of put and get methods in the cache a… 

    …s this is already implemented in the ConcurrentHashmap
    @ingunnaf
    ingunnaf committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    e4566b8 View commit details
    Browse the repository at this point in the history

  10. Rename BinMinxMaxs to BinBoundaries

    @ingunnaf
    ingunnaf committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    74d5e3f View commit details
    Browse the repository at this point in the history

  11. Run tests with 1, 2, 4 and 8 threads instead

    @ingunnaf
    ingunnaf committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    5ee8b9d View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2024

  1. Assert that the resulting matrix block is equal independent of the nu… 

    …mber of threads
    @ingunnaf
    ingunnaf committed Jul 7, 2024
    Configuration menu
    Copy the full SHA
    1a0101a View commit details
    Browse the repository at this point in the history

  2. Remove multi threads from the normal encodecache test as a separate c… 

    …lass was added
    @ingunnaf
    ingunnaf committed Jul 7, 2024
    Configuration menu
    Copy the full SHA
    bd9daec View commit details
    Browse the repository at this point in the history

  3. Typos

    @ingunnaf
    ingunnaf committed Jul 7, 2024
    Configuration menu
    Copy the full SHA
    8551f5c View commit details
    Browse the repository at this point in the history

  4. Revert renaming

    @ingunnaf
    ingunnaf committed Jul 7, 2024
    Configuration menu
    Copy the full SHA
    811fc2c View commit details
    Browse the repository at this point in the history

  5. Remove these from template config as these will only be exposed to us… 

    …er later
    @ingunnaf
    ingunnaf committed Jul 7, 2024
    Configuration menu
    Copy the full SHA
    b80ad32 View commit details
    Browse the repository at this point in the history

  6. Remove performance test from multi thread class

    @ingunnaf
    ingunnaf committed Jul 7, 2024
    Configuration menu
    Copy the full SHA
    2271c73 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. [SYSTEMDS-3534] - clear chache after setup

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    1924e64 View commit details
    Browse the repository at this point in the history

  2. [SYSTEMDS-3534] - added performance test multithread

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    9aedd3b View commit details
    Browse the repository at this point in the history

  3. [SYSTEMDS-3534] - make cache toggle setter static

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    5772386 View commit details
    Browse the repository at this point in the history

  4. [SYSTEMDS-3534] - refactored unit test

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    730a8fc View commit details
    Browse the repository at this point in the history

  5. [SYSTEMDS-3534] - added static utils class for tests

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    0a12778 View commit details
    Browse the repository at this point in the history

  6. [SYSTEMDS-3534] - cleanup

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    475e53b View commit details
    Browse the repository at this point in the history

  7. [SYSTEMDS-3534] - deleted redundant test

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    9387bf6 View commit details
    Browse the repository at this point in the history

  8. [SYSTEMDS-3534] - refactored tests

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    b80b227 View commit details
    Browse the repository at this point in the history

  9. [SYSTEMDS-3534] - added test to compare build times in isolation

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    a4b8bd3 View commit details
    Browse the repository at this point in the history

  10. [SYSTEMDS-3534] - moved consistencs test to seperate class to avoid c… 

    …onfusion about thread numbers
    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    16cdb0a View commit details
    Browse the repository at this point in the history

  11. [SYSTEMDS-3534] - removed redundant test

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    6866450 View commit details
    Browse the repository at this point in the history

  12. [SYSTEMDS-3534] - move matrix comparison to utils class

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    56723e3 View commit details
    Browse the repository at this point in the history

  13. [SYSTEMDS-3534] - clean up and minor changes

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    1825ef6 View commit details
    Browse the repository at this point in the history

  14. [SYSTEMDS-3534] - add license

    @Mayaryin
    Mayaryin committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    d8607b2 View commit details
    Browse the repository at this point in the history