-
Notifications
You must be signed in to change notification settings - Fork 1
/
README.Rmd
153 lines (111 loc) · 9.32 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
---
output:
github_document:
html_preview: true
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# GPTscreenR
<!-- badges: start -->
[](https://lifecycle.r-lib.org/articles/stages.html#experimental)
[](https://github.com/wilkox/GPTscreenR/actions/workflows/R-CMD-check.yaml)
<!-- badges: end -->
GPTscreenR is an R package that uses [OpenAI's GPT large language model](https://platform.openai.com/docs/guides/gpt) to screen titles and abstracts for [scoping reviews](https://en.wikipedia.org/wiki/Systematic_review#Scoping_reviews).
## Installation
GPTscreenR requires the [lemur package](https://github.com/wilkox/lemur). You can install the development version of lemur from [GitHub](https://github.com/) with:
```{r install_lemur, eval=FALSE}
devtools::install_github("wilkox/lemur")
```
You can install the development version of GPTscreenR from [GitHub](https://github.com/) with:
```{r install, eval=FALSE}
devtools::install_github("wilkox/GPTscreenR")
```
```{r load, echo = FALSE, message = FALSE}
library(GPTscreenR)
library(cli)
```
GPTscreenR is not currently available on CRAN.
## OpenAI API key
Using GPTscreenR requires an OpenAI API key, which you must store in an R environmental variable called `OPENAI_API_KEY`. OpenAI charges fees to use their API. To obtain and set up an API key:
1. [Create an OpenAI account](https://platform.openai.com/signup?launch), if you don't have one already.
2. Once you have created and signed in to your account, go to the [View API Keys page](https://platform.openai.com/account/api-keys).
3. Click on the 'Create new secret key' button.
4. Give your new key a name, for example 'GPTscreenR', and click 'Create secret key'.
5. The page will display your new secret API key. You must copy the key now, as it will never be displayed again.
6. Store your API key in an R environmental variable by adding the following line to your `.Renviron` file: `OPENAI_API_KEY=<your secret API key goes here>`. There are a few different ways to edit this file:
a. On any operating system, you can use the function `usethis::edit_r_environ()` from [the usethis package](https://usethis.r-lib.org) to open and edit your `.Renviron` file.
b. On macOS or Linux, you can create or edit a file in your home directory called `~/.Renviron`.
c. You can also make an `.Renviron` file for a particular project. I strongly recommend that you *do not* check this file into a version control system like git, as this means other people with access to the repository (including the public, if the repository is publicly accessible) can see your secret key.
To check that you have set the environmental variable successfully, open a new R session and load GPTscreenR with `library(GPTscreenR)`. You should see the following message:
``✔ The `OPENAI_API_KEY` environmental variable is set``
If instead you see:
``✖ The `OPENAI_API_KEY` environmental variable is not set``
this means the environmental variable has not been set correctly. Note that this does not check whether or not you have stored a valid key.
## Screening sources
'Source' is the term used in GPTscreenR for any published scholarly work, such as an article, review, or letter, which might be considered for inclusion in a scoping review. There are three steps to use GPTscreenR to help screen source titles and abstracts for inclusion. First, you must write a description of your review, so that GPT knows what sorts of sources should be included and excluded. Second, you must prepare a data frame of your sources including their titles and abstracts. Third, you must ask GPT to screen each source against your review description.
### Review description
Begin by writing a description of your review and storing it in a variable. This can be as simple as a few sentences describing the aims of your review. However, the more detailed and specific your review description, the better GPT is likely to perform in screening sources, and if you are using GPT to screen sources in parallel with human reviewers it should be given the same guidelines that are given to the humans. If you have specific inclusion and exclusion criteria, for example for certain source types or languages, these should be included.
The GPTscreenR package provides a function `review_description()` to help in compiling a review description using the [Population, Concept, and Context (PCC) framework](https://journals.lww.com/jbisrir/Fulltext/2020/10000/Updated_methodological_guidance_for_the_conduct_of.4.aspx) for scoping reviews. Here is an example of using `review_description()` to generate a PCC review description:
```{r generate_review_description}
alpaca_review_description <- review_description(
objective = "This scoping review will examine the existing primary research
on the role of therapy alpacas in enhancing the mental health and emotional
well-being of elderly residential aged care facility residents",
population = "Elderly people living in residential aged care facilities.
'Elderly' is defined as 65 years of age or older.",
context = "Residential aged care facilities that have used therapy alpacas as
part of their programme of care for residents. 'Residential aged care
facility' is defined as a residential setting intended as a long-term place
of residence for elderly people which includes provision of support for
activities of daily living (e.g. meal preparation, bathing, housekeeping) and
nursing support (e.g. medication management). Such facilities will also
typically offer other structured programmes and facilities to provide
entertainment, diversion, and wellbeing. It excludes other residential
settings intended for elderly people that do no provide daily living or
nursing supports (e.g. independent living villages) or that are not long-term
(e.g. hospitals or hospices).",
concept = "The impact of therapy alpaca programmes on stress reduction,
emotional well-being, mental health, overall life satisfaction, or similar
outcomes for residents."
)
```
### Data frame of sources
Your sources should be stored in a data frame that includes columns named 'title' and 'abstract'. For example:
```{r example_sources, eval = FALSE}
print(alpaca_sources)
```
```{r print_example_sources, echo = FALSE}
knitr::kable(rbind(alpaca_sources[7, ], data.frame(title = "…", abstract = "…", include = "…")), row.names = FALSE)
```
### Screening sources against your review description
The main function of GPTscreenR is `screen_sources()`. This takes the data frame of package sources and the review description, and uses GPT to screen the source titles and abstracts. It returns a data frame that includes GPT's recommendation on whether to include or exclude the source. By default, `screen_sources()` will create a file called `sources_cache.rds` to cache the results of screening, which allows screening to be split across multiple sessions and to recover from interruptions.
```{r screen_sources, eval = FALSE}
alpaca_results <- screen_sources(alpaca_sources, alpaca_review_description)
print(alpaca_results)
```
```{r print_screen_sources, echo = FALSE}
df <- alpaca_results[7, ]
df$GPT_conversation <- "`GPT_mssg`"
df <- rbind(df, data.frame(title = "…", abstract = "…", include = "…", GPT_conversation = "…", GPT_recommendation = "…"))
row.names(df) <- NULL
knitr::kable(df)
```
The output of `screen_sources()` is a data frame identical to the data frame of sources but with two additional columns. 'GPT_conversation' is a [list-column](https://jennybc.github.io/purrr-tutorial/ls13_list-columns.html), each element of which is a `GPT_mssg` object containing the full conversation between the GPTscreenR packages and GPT. This can be useful for understanding how GPT came to a particular recommendation. The `GPT_recommendation` column contains either 'INCLUDE' or 'EXCLUDE', representing GPT's recommendation for each source.
## LLM service and model
By default, GPTscreenR uses the OpenAI chat API with the `gpt-4o` model. Alternative LLM services, including the OpenAI assistants API and local LLMs with ollama, can be selected using the `service` argument to `screen()`. The model can be set with the `model` argument to `screen()`.
## Please consider contributing your scoping review data
If you've used GPTscreenR to help screen sources for a scoping review, please consider contributing some data to help monitor GPTscreenR's performance in real-world screening and guide future improvements to the package. All you need to contribute are:
- The review description
- The list of sources that were screened at the title and abstract level, including at least:
- The title
- The abstract
- The consensus decision by human reviewers to include or exclude the source
In recognition of your contribution, a citation to your review will be added to the GPTscreenR README. Of course, you are not expected to release data publicly until your review is published.
If you have any questions, or to contribute data now, send an email to [[email protected]](mailto:[email protected]) or [open a new issue](https://github.com/wilkox/GPTscreenR/issues/new).