Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BREAKING: v2.0.0 #1433

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

BREAKING: v2.0.0 #1433

wants to merge 16 commits into from

Commits on Nov 11, 2024

  1. Configuration menu
    Copy the full SHA
    e2520df View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Consolidate Retrieval/Reranking/Instruction Variants (#1359)

    * update
    
    * merged retrieval; working
    
    * update tasks; working multilingual
    
    * everything working except instructions
    
    * working instructions; just need cleanup
    
    * add metadata for all but MindSmall
    
    * faster evaluation; mindsmall can compute in reasonable time
    
    * fix bad merge of docs
    
    * lint
    
    * fix test
    
    * qa
    
    * updated mindsmall
    
    * lint
    
    * fix debug
    
    * Update mteb/abstasks/dataloaders.py
    
    Co-authored-by: Roman Solomatin <[email protected]>
    
    * lint
    
    ---------
    
    Co-authored-by: Roman Solomatin <[email protected]>
    orionw and Samoed authored Nov 13, 2024
    Configuration menu
    Copy the full SHA
    9c58518 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Configuration menu
    Copy the full SHA
    2a8a370 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    dea2b77 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    23d6cb2 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8868cd4 View commit details
    Browse the repository at this point in the history
  5. fix: Merge main into v2 (#1454)

    * fix: Count unique texts, data leaks in calculate metrics (#1438)
    
    * add more stat
    
    * add more stat
    
    * update statistics
    
    * fix: update task metadata to allow for null (#1448)
    
    * Update tasks table
    
    * 1.19.5
    
    Automatically generated by python-semantic-release
    
    * base
    
    * sync with main
    
    ---------
    
    Co-authored-by: Kenneth Enevoldsen <[email protected]>
    Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
    Co-authored-by: github-actions <[email protected]>
    4 people authored Nov 14, 2024
    Configuration menu
    Copy the full SHA
    5470c88 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. feat: enable codecarbon by default (#1428)

    * enable codecarbon by default
    
    * lint
    
    * update flag
    
    * add allow_multiple_runs param
    
    * make lint
    
    * add warning
    
    * lint
    
    * negate the flag
    
    ---------
    
    Co-authored-by: Isaac Chung <[email protected]>
    Samoed and isaac-chung authored Nov 15, 2024
    Configuration menu
    Copy the full SHA
    70a3ff2 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2024

  1. Add decriptive stat almost to all datasets (#1466)

    * run tasks
    
    * remove test script
    
    * lint
    
    * remove cache
    
    * fix sickbrsts
    
    * fix tests
    
    * add datasets
    Samoed authored Nov 18, 2024
    Configuration menu
    Copy the full SHA
    0e9b6fd View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2024

  1. fix: Fix test for empty descriptive tasks (#1413)

    * fix test
    
    * skip mock
    
    * add message to assert
    
    * fix test
    
    * lint
    
    * fix tests
    
    * upd tests
    
    * update descriptive stats files
    
    * add stat to speed
    Samoed authored Nov 19, 2024
    Configuration menu
    Copy the full SHA
    0a5bedb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6da2a1a View commit details
    Browse the repository at this point in the history
  3. feat: Multilingual retrieval loader (#1473)

    * multilingual loader
    
    * lint
    Samoed authored Nov 19, 2024
    Configuration menu
    Copy the full SHA
    a27de33 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. fix: add citations to ModelMeta (#1477)

    * add citations
    
    * fix typo
    Samoed authored Nov 21, 2024
    Configuration menu
    Copy the full SHA
    0df0210 View commit details
    Browse the repository at this point in the history
  2. Add descriptive stats to mising tasks and add number of qrels (#1476)

    * add code for comupting number of qrels
    
    * add stats fever hotpotqa msmarco topiocqa
    
    * miracl mrtidy
    
    * multilongdoc  miracl reranking
    
    * add multi eurlex
    
    * fix tests for descriptive stats
    
    * fix tests
    
    ---------
    
    Co-authored-by: Roman Solomatin <[email protected]>
    imenelydiaker and Samoed authored Nov 21, 2024
    Configuration menu
    Copy the full SHA
    0abe1a0 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2024

  1. 1475 add descriptive stats to all tasks v2 (#1482)

    * add code for comupting number of qrels
    
    * BibleNLPBitextMining descriptive stats added
    
    * SwissJudgementClassification descriptive stats added
    
    * VoyageMMarcoReranking descriptive stats added
    
    * WebLINXCandidatesReranking descriptive stats added
    
    * MultiEURLEXMultilabelClassification descriptive stats added
    
    * MIRACLReranking descriptive stats added
    
    * MindSmallReranking descriptive stats added
    
    * updated test_TaskMetadata
    
    * fix test
    
    ---------
    
    Co-authored-by: Imene Kerboua <[email protected]>
    Co-authored-by: Imene Kerboua <[email protected]>
    Co-authored-by: Roman Solomatin <[email protected]>
    4 people authored Nov 23, 2024
    Configuration menu
    Copy the full SHA
    a7a5214 View commit details
    Browse the repository at this point in the history
  2. fix: Fix BrightRetrieval calculate stats (#1484)

    * fix bright loader
    
    * lint
    
    * fix comment
    Samoed authored Nov 23, 2024
    Configuration menu
    Copy the full SHA
    99247b2 View commit details
    Browse the repository at this point in the history