Plan 08 Jan

[sachinn12]

• Reproduce results from issue Reproduce the best results of the DeepMistake Team for LSCDiscovery #50, close issue
• Reproduce results from WiC Unit and Integration Test #45, close issue, add integration test for xl-lexeme if it dosen't exist
• Add integration tests for LSCD graded with apd for resampled datasets and close issue Add resampled datasets to benchmark #54
• Verify unification of deepmistake caches and close issue unify caches for deepmistake models #42, re-run model multiple times and verify results are consistent, verify WiC task with deepmistake model #32 is not applying anymore and close
• Do multirun (example: https://github.com/ChangeIsKey/LSCDBenchmark/blob/main/experiments/warmup_wic.sh), reproduce issue path error on the hydra result collector #41, The warning message after running with deepmistake model #38, solve and close
• Add possibility of COMPARE evaluation, close issue lscd_compare task #34, this may have been implemented with Reproduce the best results of the DeepMistake Team for LSCDiscovery #50
• Fix inconsistency in missing parameters behaviour  #6
• Solve no working examples #31
• Fix documentation and close Update read the docs  #56, solve DeepMistake models for predictions between 1-4 do not run #9
• Unit test for cuda installation write unit test to check cuda installation #28
• Change result export change empty score field into np.nan in result file  #27
• Fix truncation_indices function might be probamatic  #26, add python version to readme.md #25, bash: task/[email protected]=apd_compare_all: No such file or directory #24, bash: task.model.wic.ckpt=bert-base-german-cased: command not found... #23, the parameter of 'task/wic/[email protected]_metric' is missing #22, Error after running: 'python main.py' #21, Errors After Annotation and Evaluation Phase #18, Unexpected Crashing during Installation #16
• Remove non functional code from tests/ and experiments/
• Add missing datasets Add missing datasets #58
• Write minimal version of experiments script, Experiments February #57

push after each point with commit link