Possible to train Llama 3.1?

[mosh98]

Hi,

I tried training llama 3.1 with run_mntp.py but get an obsucre error

AttributeError: 'LlamaBiModel' object has no attribute 'rotary_emb'

What is that about ?

[bzantium]

you can check this: 03382c3

[mosh98]

hmm still not sure what to do...

[stefanhgm]

Hi everyone,

@bzantium thanks for pointing us to the commit. I added the respective lines and used a more recent version of transformers to make it work. MNTP training for Llama 3.1 seems to work now for me. However, I failed to do the MTEB evaluation locally so far, see #123.

Did you make any progress in training Llama 3.1 for LLM2Vec?

[mosh98]

@bzantium Thanks i was able to get the embeddings after adding in the lines, haven't been able to train it yet through MNTP but i'll keep on trying

[andupotorac]

@stefanhgm Once Llama 3.1 (I presume the 8B parameters model) is trained, can you use it for generating images the way ELLA uses t5, with better prompt adherence?

[stefanhgm]

@andupotorac I am not familiar with the ELLA project, but you could use the model to create embeddings just as with the other LLM2Vec models.

However, the eval on MTEB currently hangs #135

[andupotorac]

Thanks, I will keep an eye on it as well.