Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fine tune a large language model for psychology domain (DOXA AI competition) #55

Open
woodthom2 opened this issue Oct 24, 2024 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@woodthom2
Copy link
Contributor

Description

We need to fine tune an LLM for mental health data. We have training and evaluation data available.

We are running a competition on the online platform DOXA AI where you can win up to 500 GBP in vouchers (1st place prize). Check it out here: https://harmonydata.ac.uk/doxa/

How to get started?

Create an account on DOXA AI and run the example notebook. This will download the training data.

If you would like some tips on how to train an LLM, I recommend this Hugging Face tutorial

Rationale

We have noticed for a while that the similarity score that Harmony gives back could be improved. For example, items to do with "sleep" are often grouped together (because of the data that the off the shelf LLMs such as SentenceTransformers are trained on) while a psychologist would consider them to be different. Read a blog post about how we evaluated the models.

Even the OpenAI models make the same mistakes as the Hugging Face models due to lack of domain understanding. For example, a large number of psychological conditions are sleep related and the LLMs tend to group things together when anything is related to sleep, while a human would consider them entirely different for the point of view of harmonisation of items in a study ("I sleep deeply" vs "I have trouble sleeping" vs "I have little time to sleep"), and similarly "child is bullied" vs "child bullies others" are another false positive. That's why we have the initiative to fine tune an LLM for this domain.

@woodthom2 woodthom2 added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed labels Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant