Skip to content

Latest commit

 

History

History
195 lines (155 loc) · 6.09 KB

README.md

File metadata and controls

195 lines (155 loc) · 6.09 KB
Raw

butcher

Codecov test coverage R-CMD-check

Overview

Modeling pipelines in R occasionally result in fitted model objects that take up too much memory. There are two main culprits:

  1. Heavy dependencies on formulas and closures that capture the enclosing environment in the modeling process; and
  2. Lack of selectivity in the construction of the model object itself.

As a result, fitted model objects carry over components that are often redundant and not required for post-fit estimation activities. butcher makes it easy to axe parts of the fitted output that are no longer needed, without sacrificing much functionality from the original model object.

Installation

Install the released version from CRAN:

install.packages("butcher")

Or install the development version from GitHub:

# install.packages("devtools")
devtools::install_github("tidymodels/butcher")

Butchering

To make the most of your memory available, this package provides five S3 generics for you to remove parts of a model object:

  • axe_call(): To remove the call object.
  • axe_ctrl(): To remove controls associated with training.
  • axe_data(): To remove the original training data.
  • axe_env(): To remove environments.
  • axe_fitted(): To remove fitted values.

As an example, we wrap a lm model:

library(butcher)
our_model <- function() {
  some_junk_in_the_environment <- runif(1e6) # we didn't know about
  lm(mpg ~ ., data = mtcars) 
}

The lm that exists in our modeling pipeline is:

library(lobstr)
obj_size(our_model())
#> 8,022,440 B

When, in fact, it should only require:

small_lm <- lm(mpg ~ ., data = mtcars) 
obj_size(small_lm)
#> 22,224 B

To understand which part of our original model object is taking up the most memory, we leverage the weigh() function:

big_lm <- our_model()
butcher::weigh(big_lm)
#> # A tibble: 25 × 2
#>    object            size
#>    <chr>            <dbl>
#>  1 terms         8.01    
#>  2 qr.qr         0.00666 
#>  3 residuals     0.00286 
#>  4 fitted.values 0.00286 
#>  5 effects       0.0014  
#>  6 coefficients  0.00109 
#>  7 call          0.000728
#>  8 model.mpg     0.000304
#>  9 model.cyl     0.000304
#> 10 model.disp    0.000304
#> # … with 15 more rows

The problem here is in the terms component of our big_lm. Because of how lm is implemented in the stats package, the environment (in which our model was made) was also carried along in the fitted output. To remove this (mostly) extraneous component, we can use axe_env():

cleaned_lm <- butcher::axe_env(big_lm, verbose = TRUE)

Comparing it against our small_lm, we’ll find:

butcher::weigh(cleaned_lm)
#> # A tibble: 25 × 2
#>    object            size
#>    <chr>            <dbl>
#>  1 terms         0.00789 
#>  2 qr.qr         0.00666 
#>  3 residuals     0.00286 
#>  4 fitted.values 0.00286 
#>  5 effects       0.0014  
#>  6 coefficients  0.00109 
#>  7 call          0.000728
#>  8 model.mpg     0.000304
#>  9 model.cyl     0.000304
#> 10 model.disp    0.000304
#> # … with 15 more rows

…it now takes the same memory on disk as small_lm:

butcher::weigh(small_lm)
#> # A tibble: 25 × 2
#>    object            size
#>    <chr>            <dbl>
#>  1 terms         0.00781 
#>  2 qr.qr         0.00666 
#>  3 residuals     0.00286 
#>  4 fitted.values 0.00286 
#>  5 effects       0.0014  
#>  6 coefficients  0.00109 
#>  7 call          0.000728
#>  8 model.mpg     0.000304
#>  9 model.cyl     0.000304
#> 10 model.disp    0.000304
#> # … with 15 more rows

Axing the environment is not the only functionality of butcher. We can also remove call, ctrl, data and fitted_values, or simply run butcher() to execute all of these axing functions at once. Any kind of axing on the object will append a butchered class to the current model object class(es) as well as a new attribute named butcher_disabled that lists any post-fit estimation functions that are disabled as a result.

Model Object Coverage

Check out the vignette("available-axe-methods") to see butcher’s current coverage. If you are working with a new model object that could benefit from any kind of axing, we would love for you to make a pull request! You can visit the vignette("adding-models-to-butcher") for more guidelines, but in short, to contribute a set of axe methods:

  1. Run new_model_butcher(model_class = "your_object", package_name = "your_package")
  2. Use butcher helper functions butcher::weigh() and butcher::locate() to decide what to axe
  3. Finalize edits to R/your_object.R and tests/testthat/test-your_object.R
  4. Make a pull request!

Contributing

This project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.