Using LLMs to improve user experience with packages/documentation

[adamkucharski]

One of the things that came up during the Summit is how to help users interact with our documentation and packages. Specifically, if someone wants to do a particular task, what's the fastest, easiest way of getting them the advice they need?

Andree and I - in discussion with Yale Fox of Applied Science - have been exploring the use of large language models to interpret user queries and provide answers based on Epiverse-TRACE documentation, using the OpenAI API. The basic ChatGPT doesn't know anything about our packages, given its training period, and can hallucinate or miss domain-specific nuances that are outlined in our vignettes.

One challenge is the token limit in API request length (4096 tokens for the standard GPT-3.5 model, or about 16000 characters) - so we can't simply put all our documentation in the prompt along with the user question.

Instead, we've been developing a Shiny implementation of that makes use of vector embeddings as well as generative models, e.g. Document Q&A with LangChain. The basic pipeline is as follows:

• For each package, load all function documentation (i.e. all .R files in R folder) and vignettes (i.e. all .Rmd files in this folder) as raw strings, then split strings into 'chunks' (e.g. around 2000 characters long). Then use the OpenAI embedding API to convert each chunk into a vector embedding representing the semantic characteristics of the chunk. This is all done as pre-processing.
• Embed the user question.
• Compare the user question to our embedding database and identify the top semantic matches (e.g. 6 closest by cosine similarity metric). Pick the best matching package based on which one dominates these matches.
• Use the top matching 'chunks' for that package as the context in a prompt that is then passed to GPT-3.5 along with the user query. GPT is instructed to only use this context to answer the question, and return output as markdown.
• Parse markdown in shiny backend as display as formatted HTML alongside the package recommendation.

Some examples that work nicely to illustrate functionality:
"Calculate case fatality risk over time"
"How do I calculate the parameters of a Gamma distribution from the mean and variance?"
"How can I calculate force of infection over time from serological data?"
"How can I clean my data?"

Full repo here: https://github.com/adamkucharski/epiverse-llm - happy to move over to main repo when/if better developed. The pre-processing embedding code is in R_not_run/generate_doc_embeddings.R using the functions in helper_functions.R. Andree has also put together some useful training questions that we need to incorporate as well.

The advantage of above pipeline is it gives a lot of control over what is displayed (i.e. minimises risk of hallucinations or broken generative code). There's also potential to introduce additional chain steps (e.g. summarise key aspects of chunks before passing to the GPT model), and expand to other packages in the ecosystem.

Obviously would eventually be more powerful for users if we could generate whole scripts that could then run (e.g. like https://rtutor.ai/) - but this would have much larger potential for errors to be introduced. Keen to hear thoughts/ideas!

[adamkucharski]

Repository moved to here so can add issues and iterate: https://github.com/epiverse-trace/llm-guidance/tree/main

main is currently under active development, with package search app here: https://kucharski.shinyapps.io/package_search/

[adamkucharski]

Flagging to @chartgerink, as might be relevant to follow up work. One thing in particular we found with initial prototype testing is people often search for single terms, which can be sensitive to how documents embedded. E.g. 'reproduction number' would be more likely to point to superspreading in our implementation (because term appears a lot in documentation), rather than a tool like EpiNow2, which follows best practice for what they're really trying to do (i.e. calculate R).

In the end, we focused more on developing the howto guides, partly as it provided a nicer way to skim through worked examples for common tasks, but also it could be a useful resource to embed and point to in response to task-based questions.