Unable to load merged model for MTEB evaluation

[sandeep-krutrim]

I have trained a model using supervised contrastive. I saved the model using -

l2v.save('/llm2vec_models/final_merged_model', merge_before_save=True, save_config=True)

Now when I try to run mteb_eval.py using -
!python experiments/mteb_eval.py --model_name model_name \ --task_name STS16 \ --task_to_instructions_fp test_configs/mteb/task_to_instructions.json \ --output_dir results

I get the error -

`WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name sentence-transformers/model_name. Creating a new one with mean pooling.
Traceback (most recent call last):
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
response.raise_for_status()
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/sentence-transformers/model_name/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/utils/hub.py", line 402, in cached_file
resolved_file = hf_hub_download(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download
return _hf_hub_download_to_cache_dir(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1325, in _hf_hub_download_to_cache_dir
_raise_on_head_call_error(head_call_error, force_download, local_files_only)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1823, in _raise_on_head_call_error
raise head_call_error
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error
metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata
r = _request_wrapper(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper
response = _request_wrapper(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 396, in _request_wrapper
hf_raise_for_status(response)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 352, in hf_raise_for_status
raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-6690ade6-4a6e9ced7aa95d064f357ee5;f3ba928a-c379-4be1-8ae2-6748cf28a965)

Repository Not Found for url: https://huggingface.co/sentence-transformers/model_name/resolve/main/config.json.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/sandeep.pandey/august15/llm2vec/experiments/mteb_eval.py", line 28, in
model = mteb.get_model(args.model_name, **model_kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/mteb/models/init.py", line 42, in get_model
meta = get_model_meta(model_name, revision)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/mteb/models/init.py", line 77, in get_model_meta
model = SentenceTransformer(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 299, in init
modules = self._load_auto_model(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 1324, in _load_auto_model
transformer_model = Transformer(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/models/Transformer.py", line 53, in init
config = AutoConfig.from_pretrained(model_name_or_path, **config_args, cache_dir=cache_dir)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 965, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict
resolved_config_file = cached_file(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/utils/hub.py", line 425, in cached_file
raise EnvironmentError(
OSError: sentence-transformers/model_name is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with huggingface-cli login or by passing token=<your_token>

[vaibhavad]

Hi @sandeep-krutrim,

Currently the evaluation only supports the models that we have released on huggingface, supporting custom models will require modifying the mteb package, specifically llm2vec_models.py file.

[stefanhgm]

Hi @sandeep-krutrim,

is there any progress in evaluating on MTEB for locally trained LLM2Vec models? Specifically, I would like to train a version of Llama 3.1 and get evaluation results on MTEB.

In case there is not, @vaibhavad could you maybe provide some more specific pointers to what would need to be changed in the llm2vec_models.py file to evaluate models trained according to your Readme? Or is there any other way that allows a way to evaluate custom trained models?

Thank you!

[vaibhavad]

I have added support to evaluate locally trained models. Please check out mteb_eval_custom.py.