From f10c62de7dc0cd6e9940aa7cbb72cec7f2c19bd5 Mon Sep 17 00:00:00 2001 From: James McMahon Date: Wed, 13 Sep 2023 17:08:53 +0100 Subject: [PATCH 1/4] Add a new vignette on the CHI functions --- .gitignore | 1 + DESCRIPTION | 4 + NEWS.md | 2 + vignettes/.gitignore | 2 + vignettes/chi-operations.Rmd | 244 +++++++++++++++++++++++++++++++++++ 5 files changed, 253 insertions(+) create mode 100644 vignettes/.gitignore create mode 100644 vignettes/chi-operations.Rmd diff --git a/.gitignore b/.gitignore index 234f028..0d7f03b 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ .RData .Ruserdata docs +inst/doc diff --git a/DESCRIPTION b/DESCRIPTION index 8a8deba..66ade97 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -41,7 +41,10 @@ Imports: utils Suggests: covr, + ggplot2, here, + knitr, + rmarkdown, spelling, testthat (>= 3.0.0) RdMacros: @@ -51,3 +54,4 @@ Encoding: UTF-8 Language: en-GB LazyData: true RoxygenNote: 7.2.3 +VignetteBuilder: knitr diff --git a/NEWS.md b/NEWS.md index fb67176..ba6a8c6 100644 --- a/NEWS.md +++ b/NEWS.md @@ -8,6 +8,8 @@ - The installation instructions in the README have been updated. +- A new article has been added to the documentation - [Working with CHI numbers](https://public-health-scotland.github.io/phsmethods/articles/chi-operations.html). + # phsmethods 0.2.2 (2022-11-14) - Improved `chi_check()` to make it more efficient and run faster. diff --git a/vignettes/.gitignore b/vignettes/.gitignore new file mode 100644 index 0000000..097b241 --- /dev/null +++ b/vignettes/.gitignore @@ -0,0 +1,2 @@ +*.html +*.R diff --git a/vignettes/chi-operations.Rmd b/vignettes/chi-operations.Rmd new file mode 100644 index 0000000..cac2a1c --- /dev/null +++ b/vignettes/chi-operations.Rmd @@ -0,0 +1,244 @@ +--- +title: "Working with CHI numbers" +output: rmarkdown::html_vignette +author: James McMahon +vignette: > + %\VignetteIndexEntry{Working with CHI numbers} + %\VignetteEncoding{UTF-8} + %\VignetteEngine{knitr::rmarkdown} +editor_options: + chunk_output_type: console +--- + +```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) +``` + +## Checking and correcting CHI numbers + +The first thing you should do when working with CHI numbers of unknown quality is to check their validity. phsmethods provides a function to easily do this: `chi_check()` - This function expects a character vector (1 or more), it will then return a value for each CHI number letting you know if it's valid, and if it isn't what the issue is. + +```{r} +chi_numbers <- c( + "0211165794", + "9999999999", + "402070763", + "00402070763", + "0101010000", + "Missing CHI", + NA, + "" +) + +library(phsmethods) + +chi_check(chi_numbers) +``` + +### Cleaning up bad CHI numbers + +Usually, we will have the CHI as a variable in some data. + +```{r, message=FALSE} +library(dplyr) + +data <- tibble(chi = c( + "0211165794", + "9999999999", + "402070763", + "00402070763", + "0101010000", + "Missing CHI", + NA, + "" +)) +``` + +It looks like one of the CHI numbers '402070763' might have just lost a leading zero, this is a common occurrence if the data has passed through Excel at some point. We can fix this specific issue with `chi_pad()`. + +```{r} +fixed_data <- data %>% + mutate(chi = chi_pad(chi)) + +checked_data <- fixed_data %>% + mutate(valid_chi = chi_check(chi)) + +checked_data +``` + +On a larger dataset, it might be useful to get a count of the issues rather than seeing them per CHI. + +```{r} +fixed_data %>% + count(valid_chi = chi_check(chi), sort = TRUE) +``` + +Now we have this knowledge we have a few options. Which option we take will depend on the type and purpose of the analysis as well as how many CHI numbers have issues. + +1. Go back to the data source/provider and try to fix the erroneous CHI numbers. +2. Set them as NA so we keep the rest of the data. +3. Filter the data with invalid CHIs out completely. + +```{r} +fixed_data %>% + mutate(chi = if_else(chi_check(chi) != "Valid CHI", NA_character_, chi)) + +fixed_data %>% + filter(chi_check(chi) == "Valid CHI") +``` + +## Inferring data from a CHI number + +In an ideal world, we would always have supplementary data such as Date of Birth, Age and Sex alongside the CHI number, however, we often work with data where we only have the CHI number and the other demographic variables are either completely missing or incomplete. + +Once we have checked and (if necessary) padded the CHI numbers, we can then try and extract some information from them. + +#### The structure of a CHI number + +As explained in this [Wikipedia article](https://en.wikipedia.org/wiki/National_Health_Service_Central_Register#Community_Health_Index) a CHI number is constructed as follows: \* The first 6 digits are the patient's Date of Birth in the format `DDMMYY`. \* Digits 7 and 8 are random. \* The 9th number indicates the patient's sex - odd for male, even for female. \* The final, 10th, digit is a (Modulus-11) 'check digit' - This helps guard against transcription errors, for example, if someone makes a typo it is very unlikely that the check digit will still be valid. + +### Extracting sex from CHI + +With `sex_from_chi()` we can extract the infer and extract the patient's sex. By default, the function will first check the CHI for validity and will return `NA` if a CHI is invalid. + +If you have already checked the CHI in a previous step it can be useful to use `chi_check = FALSE` as this will be faster. + +```{r} +data <- tibble( + chi = c("0101011237", "0211165794", "0402070763", "0101336489", "1904851231", "2902960018") +) + +# Confirm all of the CHIs are valid +count(data, chi_check(chi)) + +data_sex <- data %>% + mutate(sex = sex_from_chi(chi, chi_check = FALSE)) +data_sex +``` + +By default sex will be returned as an integer with '1' representing 'Male' and '2' representing 'Female', this is consistent with the [coding of sex](https://www.ndc.scot.nhs.uk/Dictionary-A-Z/Definitions/index.asp?Search=S&ID=1277&Title=Sex) in other PHS datasets. + +We can have sex returned as a factor using `as_factor = TRUE`, which by default will have levels of '1' and '2' and labels of 'Male' and 'Female' which can be useful, particularly when visualising the data. + +```{r} +data_sex <- data_sex %>% + mutate(sex_factor = sex_from_chi(chi, as_factor = TRUE)) + +data_sex +``` + +```{r} +library(ggplot2) + +data_sex %>% + ggplot(aes(y = "", fill = sex_factor)) + + geom_bar() + + coord_polar() + + labs(title = "Count of Male vs Female", x = "", y = "") + + scale_fill_brewer("Sex (from CHI)", type = "qual") + + theme_minimal() +``` + +### Extracting Date of Birth from CHI + +It is usually not possible to definitively infer a patient's Date of Birth from the CHI number, this is because the CHI only contains 2 digits for the year. Looking at the first 6 digits of a CHI number '010120' could be '1 January 1920' or '1 January 2020'. However, with some extra context, we can usually eliminate one of the possibilities, for example in 2023 we know that any CHI numbers of the form 'DDMM24' etc. must mean 1924 since it can't be 2024. + +The function `dob_from_chi()` will try to extract the Date of Birth and will return `NA` if the date is ambiguous. + +```{r} +data_dob <- data %>% + mutate(dob = dob_from_chi(chi)) + +data_dob +``` + +We will need to provide some more context to be able to work out the still missing dates. Often we will be working with historical data, for instance, if we know the data is from 2015 we know the patients must have been born earlier than that. We can use the `min_date` and `max_date` arguments to provide this context. + +- `min_date` will usually be some information from the data, it is the latest possible date that the CHI could have been born. +- `max_date` will default to today's date, it will usually be some common sense date about the latest date you'd expect. For instance, if working with childhood vaccine data you could use `Sys.Date() - lubridate::years(16)`, it to imply you don't expect anyone older than 16 as of today's date. + +```{r} +# Expect no one born after 2015-12-31 +data %>% + mutate(dob = dob_from_chi(chi, max_date = as.Date("2015-12-31"))) + +# Expect no one born before 1999-12-31 i.e. 16 years before our data started. +data %>% + mutate(dob = dob_from_chi( + chi, + max_date = as.Date("2015-12-31"), + min_date = as.Date("2015-12-31") - lubridate::years(16) + )) +``` + +Usually, we will have event dates e.g. an admission date alongside the data and this can be used instead of, or in conjunction with a fixed date. + +```{r} +data <- data %>% + mutate(event_date = as.Date(c( + "2015-01-01", + "2014-01-01", + "2013-01-01", + "2012-01-01", + "2011-01-01", + "2010-01-01" + ))) + +# Using the event date as the maximum date +data %>% + mutate(dob = dob_from_chi(chi, max_date = event_date)) + +# Setting a 'fixed' minimum date as well as using the event date +data_dob <- data %>% + mutate(dob = dob_from_chi( + chi, + max_date = event_date, + min_date = as.Date("1915-01-01") + )) + +data_dob +``` + +### Extracting age from CHI + +The function `age_from_chi()` provides a simpler interface for just extracting a patient's age from the CHI number. In the background, it uses `dob_from_chi()` but allows you to specify `min_age` and `max_age`, which are usually conceptually simpler than trying to work out dates. We do lose some amount of fine control here though, so it will sometimes be necessary to use `dob_from_chi()` and then `age_calculate()`. + +Note that age is calculated at *today's* date unless otherwise specified with the `ref_date` argument. + +```{r} +data %>% + mutate(age = age_from_chi(chi)) + +# Work out age at a fixed date +data %>% + mutate(age = age_from_chi(chi, ref_date = as.Date("2016-01-01"))) + +# Work out age at a relative date +data %>% + mutate(age = age_from_chi(chi, ref_date = event_date)) +``` + +We will get different results depending on which context we supply. + +```{r} +data %>% + mutate(age = age_from_chi(chi, ref_date = event_date, max_age = 18)) + +data %>% + mutate(age = age_from_chi( + chi, + ref_date = event_date, + min_age = 60, + max_age = 120) + ) + +data %>% + mutate(age = age_from_chi( + chi, + min_age = 60, + max_age = 120) + ) +``` From 4a33ad065598a9ebbf21859da64cbb978dc8d3d1 Mon Sep 17 00:00:00 2001 From: James McMahon Date: Wed, 13 Sep 2023 17:09:22 +0100 Subject: [PATCH 2/4] Update the workflows so they will run on a PR to dev or development --- .github/workflows/R-CMD-check.yaml | 10 ++++++++-- .github/workflows/lint.yaml | 10 ++++++++-- .github/workflows/test-coverage.yaml | 8 ++++++-- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/.github/workflows/R-CMD-check.yaml b/.github/workflows/R-CMD-check.yaml index 9fb4ca8..5b5ee01 100644 --- a/.github/workflows/R-CMD-check.yaml +++ b/.github/workflows/R-CMD-check.yaml @@ -2,9 +2,15 @@ # Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help on: push: - branches: [main, master, dev] + branches: + - 'main' + - 'master' + - 'dev**' pull_request: - branches: [main, master, dev] + branches: + - 'main' + - 'master' + - 'dev**' name: R-CMD-check diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index c200b81..92e929b 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -2,9 +2,15 @@ # Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help on: push: - branches: [main, master, dev] + branches: + - 'main' + - 'master' + - 'dev**' pull_request: - branches: [main, master, dev] + branches: + - 'main' + - 'master' + - 'dev**' name: lint diff --git a/.github/workflows/test-coverage.yaml b/.github/workflows/test-coverage.yaml index 1365b61..6f5abde 100644 --- a/.github/workflows/test-coverage.yaml +++ b/.github/workflows/test-coverage.yaml @@ -2,9 +2,13 @@ # Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help on: push: - branches: [main, master, dev] + branches: + - 'main' + - 'master' pull_request: - branches: [main, master, dev] + - 'main' + - 'master' + - 'dev**' name: test-coverage From fa3f30032329fffa04fada433a5a1c2d73a22362 Mon Sep 17 00:00:00 2001 From: Moohan Date: Wed, 13 Sep 2023 16:12:13 +0000 Subject: [PATCH 3/4] Style code --- vignettes/chi-operations.Rmd | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/vignettes/chi-operations.Rmd b/vignettes/chi-operations.Rmd index cac2a1c..2ddd666 100644 --- a/vignettes/chi-operations.Rmd +++ b/vignettes/chi-operations.Rmd @@ -229,16 +229,16 @@ data %>% data %>% mutate(age = age_from_chi( - chi, - ref_date = event_date, - min_age = 60, - max_age = 120) - ) + chi, + ref_date = event_date, + min_age = 60, + max_age = 120 + )) data %>% mutate(age = age_from_chi( - chi, - min_age = 60, - max_age = 120) - ) + chi, + min_age = 60, + max_age = 120 + )) ``` From 0c4eb4fdb9fa827138b8c4628a207a256f7e24eb Mon Sep 17 00:00:00 2001 From: James McMahon Date: Tue, 21 Nov 2023 16:40:33 +0000 Subject: [PATCH 4/4] Update test-coverage.yaml --- .github/workflows/test-coverage.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/test-coverage.yaml b/.github/workflows/test-coverage.yaml index 79b902d..18a1973 100644 --- a/.github/workflows/test-coverage.yaml +++ b/.github/workflows/test-coverage.yaml @@ -7,6 +7,10 @@ on: - 'master' - 'dev**' pull_request: + branches: + - 'main' + - 'master' + - 'dev**' name: test-coverage