diff --git a/DESCRIPTION b/DESCRIPTION index 025a441..37dbd85 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: LightLogR Title: Process Data from Wearable Light Loggers and Optical Radiation Dosimeters -Version: 0.4.0 +Version: 0.4.1 Authors@R: c( person("Johannes", "Zauner", email = "johannes.zauner@tum.de", role = c("aut", "cre"), @@ -48,7 +48,7 @@ Imports: tidyr, utils Depends: - R (>= 2.10) + R (>= 4.3) LazyData: true Suggests: covr, diff --git a/NEWS.md b/NEWS.md index 6275a1a..76f5f95 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,13 @@ +# LightLogR 0.4.1 + +* added support for OcuWEAR devices + +* added support for MotionWatch 8 devices #32 + +* added support for LIMO devices + +* added support for GENEActiv devices, when data was preprocessed with the [`GGIR`](https://cran.r-project.org/web/packages/GGIR/vignettes/GGIR.html) package. The function `import$GENEActiv_GGIR()` takes the `GGIR` output and imports it with LightLogR naming schemes. #27 + # LightLogR 0.4.0 "Nautical dawn" * release on CRAN! diff --git a/R/import_LL.R b/R/import_LL.R index eafe9cf..790bf1b 100644 --- a/R/import_LL.R +++ b/R/import_LL.R @@ -35,9 +35,10 @@ #' * `manual.id`: If this argument is not `NULL`, and no `Id` column is part #' of the `dataset`, this `character` scalar will be used. **We discourage the #' use of this arguments when importing more than one file** +#' * `silent`: If set to `TRUE`, the function will not print a summary message of the import or plot the overview. Default is `FALSE`. #' * `locale`: The locale controls defaults that vary from place to place. -#' * `dst_adjustment`: If a file crosses daylight savings time, but the device does not adjust time stamps accordingly, you can set this argument to `TRUE`, to apply this shift manually. It is selective, so it will only be done in files that cross between DST and standard time. Default is `FALSE`. Uses `dst_change_handler()` to do the adjustment. Look there for more infos. It is not equipped to handle two jumps in one file (so back and forth between DST and standard time), but will work fine if jums occur in separate files. -#' * `auto.plot`: a logical on whether to call [gg_overview()] after import. Default is `TRUE`. +#' * `dst_adjustment`: If a file crosses daylight savings time, but the device does not adjust time stamps accordingly, you can set this argument to `TRUE`, to apply this shift manually. It is selective, so it will only be done in files that cross between DST and standard time. Default is `FALSE`. Uses [dst_change_handler()] to do the adjustment. Look there for more infos. It is not equipped to handle two jumps in one file (so back and forth between DST and standard time), but will work fine if jums occur in separate files. +#' * `auto.plot`: a logical on whether to call [gg_overview()] after import. Default is `TRUE`. But is set to `FALSE` if the argument `silent` is set to `TRUE`. #' * `...`: supply additional arguments to the \pkg{readr} import functions, like `na`. Might also be used to supply arguments to the specific import functions, like `column_names` for `Actiwatch_Spectrum` devices. Those devices will always throw a helpful error message if you forget to supply the necessary arguments. #' If the `Id` column is already part of the `dataset` it will just use this #' column. If the column is not present it will add this column and fill it @@ -184,6 +185,36 @@ #' Model: Kronowise #' #' Implemented: July 2024 +#' +#' ## GENEActiv with GGIR preprocessing +#' +#' Manufacturer: Activeinsights +#' +#' Model: GENEActiv +#' +#' **Note:** This import function takes GENEActiv data that was preprocessed through the [GGIR](https://cran.r-project.org/web/packages/GGIR/vignettes/GGIR.html) package. By default, `GGIR` aggregates light data into intervals of 15 minutes. This can be set by the `windows` argument in GGIR, which is set to 900 seconds by default. To import the preprocessed data with `LightLogR`, the `filename` argument requires a path to the parent directory of the GGIR output folders, specifically the `meta` folder, which contains the light exposure data. Multiple `filename`s can be specified, each of which needs to be a path to a different GGIR parent directory. GGIR exports can contain data from multiple participants, these will always be imported fully by providing the parent directory. Use the `pattern` argument to extract sensible `Id`s from the *.RData* filenames within the *meta/basic/* folder. +#' +#' ## MotionWatch 8 +#' +#' Manufacturer: CamNtech +#' +#' Implemented: September 2024 +#' +#' ## LIMO +#' +#' Manufacturer: ENTPE +#' +#' Implemented: September 2024 +#' +#' LIMO exports `LIGHT` data and `IMU` (inertia measurements, also UV) in separate files. Both can be read in with this function, but not at the same time. Please decide what type of data you need and provide the respective filenames. +#' +#' ## OcuWEAR +#' +#' Manufacturer: Ocutune +#' +#' Implemented: September 2024 +#' +#' OcuWEAR data contains spectral data. Due to the format of the data file, the spectrum is not directly part of the tibble, but rather a list column of tibbles within the imported data, containing a `Wavelength` (nm) and `Intensity` (mW/m^2) column. #' #' @section Examples: #' @@ -339,8 +370,8 @@ imports <- function(device, print_n = print_n #how many rows to print for observation intervals ) - #if autoplot is TRUE, make a plot - if(auto.plot) { + #if autoplot is TRUE & silent is FALSE, make a plot + if(auto.plot & !silent) { data %>% gg_overview() %>% print() } #return the file @@ -354,7 +385,7 @@ imports <- function(device, # Import functions ------------------------------------------------------- #source the import expressions -source("R/import_expressions.R") +source("R/import_expressions.R", local = TRUE) #' Import Datasets from supported devices #' diff --git a/R/import_expressions.R b/R/import_expressions.R index 0510746..2b5ce24 100644 --- a/R/import_expressions.R +++ b/R/import_expressions.R @@ -417,7 +417,125 @@ import_expr <- list( lubridate::as_datetime(time_stamp, tz = "UTC"), tz), .before = 1 ) } - ) + ), + #GENEActiv GGIR + GENEActiv_GGIR = rlang::expr({ + data <- purrr::map(filename, \(x){ + #define the subfolder structure + subfolder <- "/meta/basic" + full_path <- paste0(x, subfolder) + #get all the files from the subfolder + files <- list.files(full_path, pattern = ".RData$", full.names = TRUE) + #collect all the data from the files + data <- purrr::map(files, \(x) { + #create a new environment and load the data into + env <- new.env() + load(x, envir = env) + #collect the dataframe with the data + data <- env$M$metalong + #changes to the dataframe to make it compatible with LightLogR naming schemes + data <- + data %>% + dplyr::mutate( + file.name = x) %>% + dplyr::mutate( + Datetime = lubridate::ymd_hms(timestamp) %>% + lubridate::with_tz(tz), .before = 1) + }) %>% purrr::list_rbind() + #return the function with the data frame + return(data) + }) %>% purrr::list_rbind() + }), + #OcuWEAR + OcuWEAR = rlang::expr({ + #import the data + data <- + suppressWarnings( + readr::read_csv(filename, + n_max = n_max, + id = "file.name", + locale = locale, + name_repair = "universal_quiet", + show_col_types = FALSE, + ...) + ) + #do some basic renaming + data <- data %>% + dplyr::rename(Datetime = DateTime) %>% + dplyr::mutate( + MEDI = Melanopic, .after = Datetime, + Datetime = Datetime %>% + lubridate::force_tz(tzone = tz) + ) + #special handling of the Spectrum column to convert it to a list + data <- data %>% + dplyr::mutate( + Spectrum = Spectrum %>% + stringr::str_remove_all("\\[|\\]") %>% + stringr::str_split(", ") %>% + purrr::map(\(x) x %>% + dplyr::case_when( + is.na(.) ~ character(401), .default = . + ) %>% as.numeric() %>% + tibble::enframe(name = "Wavelength", value = "Intensity") %>% + dplyr::mutate(Wavelength = Wavelength+379) + ) + ) + }), + #MotionWatch8 + MotionWatch8 = rlang::expr({ + column_names <- c("Date", "Time", "Activity", "Light") + data <- + purrr::map( + filename, + \(x) { + rows_to_skip <- detect_starting_row(x, + locale = locale, + column_names = column_names, + n_max = 1000) + df <- suppressMessages( + readr::read_csv2( + x, + skip = rows_to_skip, + locale=locale, + id = "file.name", + show_col_types = FALSE, + col_types = "ctid", + name_repair = "universal", + ... + ) + ) + + df <- df %>% + dplyr::mutate( + Datetime = + lubridate::parse_date_time( + paste(Date, Time), orders = "dmyHMS", tz = tz + ), + .before = Date + ) + + }) %>% purrr::list_rbind() + + }), + #LIMO + LIMO = rlang::expr({ + data <- + readr::read_csv(filename, + n_max = n_max, + skip = 1, + id = "file.name", + locale = locale, + show_col_types = FALSE, + name_repair = "universal_quiet", + ... + ) + data <- data %>% + dplyr::rename(Datetime = date.time.iso.) %>% + dplyr::mutate( + Datetime = lubridate::force_tz(Datetime, tz = tz) + ) + }) ) #order the list by their names diff --git a/README.Rmd b/README.Rmd index 76254ca..8f2703c 100644 --- a/README.Rmd +++ b/README.Rmd @@ -46,9 +46,7 @@ To come: - Integration of data into a unified database for cross-study analyses -##### Please note that LightLogR is work in progress! If you are interested in the project and want to know more, you can subscribe to the [LightLogR mailing list](https://lists.lrz.de/mailman/listinfo/lightlogr-users). If you find a bug, please open an issue on the [GitHub repository](https://github.com/tscnlab/LightLogR/issues). - -##### To maximize LightLogRs utility, we want to hear from you! What features would you like to see, what are common issues you face when working with wearable data, and what kind of analysis are you performing? Let us know in the [LightLogR community survey](https://de.surveymonkey.com/r/3LL9NKQ)! +##### Please note that LightLogR is work in progress! If you are interested in the project and want to know more, you can subscribe to the [LightLogR mailing list](https://lists.lrz.de/mailman/listinfo/lightlogr-users). If you find a bug or would like to see new or improved features, please open an issue on the [GitHub repository](https://github.com/tscnlab/LightLogR/issues). Have a look at the **Example** section down below to get started, or dive into the [Articles](https://tscnlab.github.io/LightLogR/articles/index.html) to get more in depth information about how to work with the package and generate images such as the one above, import data, visualization, and metric calculation. @@ -152,6 +150,8 @@ LLdata %>% gg_overview() ![](man/figures/gg_overview2.png){width="60%"} ::: +*note:* the above example can not be executed through copy and paste, as it requires a large dataset not included in the package. It is available, however, in the article on [Import & cleaning](https://tscnlab.github.io/LightLogR/articles/Import.html). + ### Visualize Once imported, **LightLogR** allows you conveniently visualize the data. diff --git a/README.md b/README.md index d8bf681..a739cc0 100644 --- a/README.md +++ b/README.md @@ -58,9 +58,7 @@ To come: - Integration of data into a unified database for cross-study analyses -##### Please note that LightLogR is work in progress! If you are interested in the project and want to know more, you can subscribe to the [LightLogR mailing list](https://lists.lrz.de/mailman/listinfo/lightlogr-users). If you find a bug, please open an issue on the [GitHub repository](https://github.com/tscnlab/LightLogR/issues). - -##### To maximize LightLogRs utility, we want to hear from you! What features would you like to see, what are common issues you face when working with wearable data, and what kind of analysis are you performing? Let us know in the [LightLogR community survey](https://de.surveymonkey.com/r/3LL9NKQ)! (open until 18 August 2024)! +##### Please note that LightLogR is work in progress! If you are interested in the project and want to know more, you can subscribe to the [LightLogR mailing list](https://lists.lrz.de/mailman/listinfo/lightlogr-users). If you find a bug or would like to see new or improved features, please open an issue on the [GitHub repository](https://github.com/tscnlab/LightLogR/issues). Have a look at the **Example** section down below to get started, or dive into the @@ -261,6 +259,11 @@ missing, based on the measurement epochs found in the data. +*note:* the above example can not be executed through copy and paste, as +it requires a large dataset not included in the package. It is +available, however, in the article on [Import & +cleaning](https://tscnlab.github.io/LightLogR/articles/Import.html). + ### Visualize Once imported, **LightLogR** allows you conveniently visualize the data. diff --git a/man/import_Dataset.Rd b/man/import_Dataset.Rd index 880a0f9..1ac18f2 100644 --- a/man/import_Dataset.Rd +++ b/man/import_Dataset.Rd @@ -6,7 +6,7 @@ \alias{import} \title{Import a light logger dataset or related data} \format{ -An object of class \code{list} of length 14. +An object of class \code{list} of length 18. } \usage{ import_Dataset(device, ...) @@ -57,9 +57,10 @@ without file extension. \item \code{manual.id}: If this argument is not \code{NULL}, and no \code{Id} column is part of the \code{dataset}, this \code{character} scalar will be used. \strong{We discourage the use of this arguments when importing more than one file} +\item \code{silent}: If set to \code{TRUE}, the function will not print a summary message of the import or plot the overview. Default is \code{FALSE}. \item \code{locale}: The locale controls defaults that vary from place to place. -\item \code{dst_adjustment}: If a file crosses daylight savings time, but the device does not adjust time stamps accordingly, you can set this argument to \code{TRUE}, to apply this shift manually. It is selective, so it will only be done in files that cross between DST and standard time. Default is \code{FALSE}. Uses \code{dst_change_handler()} to do the adjustment. Look there for more infos. It is not equipped to handle two jumps in one file (so back and forth between DST and standard time), but will work fine if jums occur in separate files. -\item \code{auto.plot}: a logical on whether to call \code{\link[=gg_overview]{gg_overview()}} after import. Default is \code{TRUE}. +\item \code{dst_adjustment}: If a file crosses daylight savings time, but the device does not adjust time stamps accordingly, you can set this argument to \code{TRUE}, to apply this shift manually. It is selective, so it will only be done in files that cross between DST and standard time. Default is \code{FALSE}. Uses \code{\link[=dst_change_handler]{dst_change_handler()}} to do the adjustment. Look there for more infos. It is not equipped to handle two jumps in one file (so back and forth between DST and standard time), but will work fine if jums occur in separate files. +\item \code{auto.plot}: a logical on whether to call \code{\link[=gg_overview]{gg_overview()}} after import. Default is \code{TRUE}. But is set to \code{FALSE} if the argument \code{silent} is set to \code{TRUE}. \item \code{...}: supply additional arguments to the \pkg{readr} import functions, like \code{na}. Might also be used to supply arguments to the specific import functions, like \code{column_names} for \code{Actiwatch_Spectrum} devices. Those devices will always throw a helpful error message if you forget to supply the necessary arguments. If the \code{Id} column is already part of the \code{dataset} it will just use this column. If the column is not present it will add this column and fill it @@ -78,9 +79,10 @@ continuously as the package is maintained. \if{html}{\out{
}}\preformatted{supported_devices() #> [1] "ActLumus" "ActTrust" "Actiwatch_Spectrum" #> [4] "Actiwatch_Spectrum_de" "Circadian_Eye" "DeLux" -#> [7] "Kronowise" "LYS" "LiDo" -#> [10] "LightWatcher" "Speccy" "SpectraWear" -#> [13] "VEET" "nanoLambda" +#> [7] "GENEActiv_GGIR" "Kronowise" "LIMO" +#> [10] "LYS" "LiDo" "LightWatcher" +#> [13] "MotionWatch8" "OcuWEAR" "Speccy" +#> [16] "SpectraWear" "VEET" "nanoLambda" }\if{html}{\out{
}} \subsection{ActLumus}{ @@ -215,6 +217,40 @@ Model: Kronowise Implemented: July 2024 } + +\subsection{GENEActiv with GGIR preprocessing}{ + +Manufacturer: Activeinsights + +Model: GENEActiv + +\strong{Note:} This import function takes GENEActiv data that was preprocessed through the \href{https://cran.r-project.org/web/packages/GGIR/vignettes/GGIR.html}{GGIR} package. By default, \code{GGIR} aggregates light data into intervals of 15 minutes. This can be set by the \code{windows} argument in GGIR, which is set to 900 seconds by default. To import the preprocessed data with \code{LightLogR}, the \code{filename} argument requires a path to the parent directory of the GGIR output folders, specifically the \code{meta} folder, which contains the light exposure data. Multiple \code{filename}s can be specified, each of which needs to be a path to a different GGIR parent directory. GGIR exports can contain data from multiple participants, these will always be imported fully by providing the parent directory. Use the \code{pattern} argument to extract sensible \code{Id}s from the \emph{.RData} filenames within the \emph{meta/basic/} folder. +} + +\subsection{MotionWatch 8}{ + +Manufacturer: CamNtech + +Implemented: September 2024 +} + +\subsection{LIMO}{ + +Manufacturer: ENTPE + +Implemented: September 2024 + +LIMO exports \code{LIGHT} data and \code{IMU} (inertia measurements, also UV) in separate files. Both can be read in with this function, but not at the same time. Please decide what type of data you need and provide the respective filenames. +} + +\subsection{OcuWEAR}{ + +Manufacturer: Ocutune + +Implemented: September 2024 + +OcuWEAR data contains spectral data. Due to the format of the data file, the spectrum is not directly part of the tibble, but rather a list column of tibbles within the imported data, containing a \code{Wavelength} (nm) and \code{Intensity} (mW/m^2) column. +} } \section{Examples}{ diff --git a/tests/testthat/test-import_LL.R b/tests/testthat/test-import_LL.R new file mode 100644 index 0000000..ebb17d1 --- /dev/null +++ b/tests/testthat/test-import_LL.R @@ -0,0 +1,52 @@ +test_that("import works", { + path <- + system.file( + "extdata", + package = "LightLogR" + ) + filename <- "205_actlumus_Log_1020_20230904101707532.txt.zip" + tz <- "UTC" + pattern <- "^(\\d{3})" + data <- import$ActLumus(filename, path, tz = tz, auto.id = pattern, silent = TRUE) + import_cols <- c("Id", + "file.name", + "Datetime", + "MS", + "EVENT", + "TEMPERATURE", + "EXT.TEMPERATURE", + "ORIENTATION", + "PIM", + "PIMn", + "TAT", + "TATn", + "ZCM", + "ZCMn", + "LIGHT", + "AMB.LIGHT", + "RED.LIGHT", + "GREEN.LIGHT", + "BLUE.LIGHT", + "IR.LIGHT", + "UVA.LIGHT", + "UVB.LIGHT", + "STATE", + "CAP_SENS_1", + "CAP_SENS_2", + "F1", + "F2", + "F3", + "F4", + "F5", + "F6", + "F7", + "F8", + "MEDI", + "CLEAR") + #check if the data has the correct dimensions + expect_equal(dim(data), c(61016, 35)) + #check if the data has the correct column names + expect_equal(names(data), import_cols) + #check if the function extracted the correct name from the filepath and whether there is only one + expect_equal(unique(data$Id) %>% as.character(), "205") +}) diff --git a/vignettes/articles/Import.Rmd b/vignettes/articles/Import.Rmd index 8de34f3..65f6a91 100644 --- a/vignettes/articles/Import.Rmd +++ b/vignettes/articles/Import.Rmd @@ -81,11 +81,15 @@ data <- import$ActLumus(files, tz = tz, auto.id = pattern, print_n=33) There are several possibilities, why the import is slow. The most common reasons are: - The data files are simply large. With short measurement intervals, many participants, and long measurement periods, files for a single study can easily reach several gigabytes. This takes some time to import and is ok. + - The data files contain many gaps. During import, `LightLogR` checks for and visualizes gaps in the data. Especially large datasets with small intervals contain many gaps, which can slow down the import process. + - The device model you are importing from has non-consistent data structures. Some devices have a varying number of rows before the actual data starts. This means a small portion of every file has to be read in and the correct starting row has to be found. This can slow down the import process if you have many files. If you are experiencing slow imports, you can try the following: -- Only read in part of your datasets, or split your dataset into several peaces, that each gets loaded in separately. You can combine them afterwards with `join_datasets()`. + +- Only read in part of your datasets, or split your dataset into several pieces, that each gets loaded in separately. You can combine them afterwards with `join_datasets()`. + - If you have many gaps in your data, you can set `auto.plot = FALSE` in the import function. This will eliminate the call to `gg_overview()`, which calculates and visualizes the gaps in the data. # Data cleaning #1 @@ -93,14 +97,14 @@ If you are experiencing slow imports, you can try the following: Before we can dive into the analysis part, we need to make sure we have a clean dataset. The import summary shows us two problems with the data: - two files have data that crosses daylight saving time (DST) changes. Because the ActLumus device does not adjust for DST, we need to correct for this. + - Multiple Ids have single datapoints at the beginning of the dataset with gaps before actual data collection starts. These are test measurements to check equipment, but must be removed from the dataset. Let us first deal with the DST change. LightLogR has an in-built function to correct for this during import. We thus will re-import the data, but make the import silent as to not clutter the output. ```{r, dst change} data <- - import$ActLumus(files, tz = tz, auto.id = pattern, dst_adjustment = TRUE, - auto.plot = FALSE, silent = TRUE) + import$ActLumus(files, tz = tz, auto.id = pattern, dst_adjustment = TRUE, silent = TRUE) ``` The second problem requires the filtering of certain Ids. The `filter_Datetime_multiple()` function is ideal for this. We can provide a length (1 week), starting from the end of data collection and backwards. The variable `arguments` provide variable arguments to the filter function, they have to be provided in list form and expressions have to be quoted through`quote()`. Fixed arguments, like the length and`length_from_start\` are provided as named arguments and only have to be specified once, as they are the same for all Ids.