debt.Rmd

# Intellectual Debt {#debt}

```{r setup, include=FALSE}
source("etc/common.R")
```

We have accumulated some intellectual debt in the previous lessons,
and we should clear this burden from our conscience before we go on to new topics.

## Learning Objectives

- Explain what the formula operator `~` was created for and what other uses it has.
- Describe and use `.`, `.x`, `.y, `..1`, `..2`, and other convenience parameters.
- Define copy-on-modify and explain its use in R.

## Why shouldn't I use `setwd`?

Because [reasons][bryan-setwd].

**But…**

No.
Use the [here package][here-package] instead to create paths that are relative to your current location:

```{r how-here-works}
print(glue('here by itself: {here()}'))
print(glue('here("book.bib"): {here("book.bib")}'))
print(glue('here("etc", "common.R"): {here("etc", "common.R")}'))
```

## What the hell are factors?

Another feature of R that doesn't have an exact analog in Python is [factors](glossary.html#factor).
In statistics, a factor is a categorical variable such as "flavor",
which can be "vanilla", "chocolate", "strawberry", or "mustard".
Factors can be represented as strings,
but storing the same string many times wastes space and is inefficient
(since comparing strings takes longer than comparing numbers).
R therefore stores each string once and gives it with a numeric key,
so that internally, "mustard" is the number 4 in the lookup table for "flavor",
but is presented as "mustard" rather than 4.

This is useful, but brings with it some problems:

1.  On the statistical side,
    it encourages people to put messy reality into tidy but misleading boxes.
    For example, it's unfortunately still common for forms to require people to identify themselves
    as either "male" or "female",
    which is [scientifically](https://www.quora.com/Scientifically-how-many-sexes-genders-are-there)
    [incorrect](https://www.joshuakennon.com/the-six-common-biological-sexes-in-humans/).
    Similarly, census forms that ask questions about racial or ethnic identity often leave people scratching their heads,
    since they don't belong to any of the categories offered.
2.  On the computational side,
    some functions in R automatically convert strings to factors by default.
    This makes sense when working with statistical data—in most cases,
    a column in which the same strings are repeated many times is categorical—but
    it is usually not the right choice in other situations.
    This has surprised enough people the years that the tidyverse goes the other way
    and only creates factors when asked to.

Let's work through a small example.
Suppose we've read a CSV file and wound up with this table:

```{r person-flavor-ranking}
raw <- tribble(
  ~person, ~flavor, ~ranking,
  "Lhawang", "strawberry", 1.7,
  "Lhawang", "chocolate",  2.5,
  "Lhawang", "mustard",    0.2,
  "Khadee",  "strawberry", 2.1,
  "Khadee", "chocolate",   2.4,
  "Khadee", "vanilla",     3.9,
  "Haddad", "strawberry",  1.8,
  "Haddad", "vanilla",     2.1
)
raw
```

Let's aggregate using flavor values so that we can check our factor-based aggregating later:

```{r aggregate-flavor-values}
raw %>%
  group_by(flavor) %>%
  summarize(number = n(), average = mean(ranking))
```

It probably doesn't make sense to turn `person` into factors,
since names are actually character strings,
but `flavor` is a good candidate:

```{r convert-flavor-to-factor}
raw <- mutate_at(raw, vars(flavor), as.factor)
raw
```

We can still aggregate as we did before:

```{r aggregate-with-factor}
raw %>%
  group_by(flavor) %>%
  summarize(number = n(), average = mean(ranking))
```

We can also impose an ordering on the factor's elements:

```{r order-with-factor}
raw <- raw %>%
  mutate(flavor = fct_relevel(flavor, "chocolate", "strawberry", "vanilla", "mustard"))
raw
```

This changes the order in which they are displayed after grouping:

```{r order-after-grouping}
raw %>%
  group_by(flavor) %>%
  summarize(number = n(), average = mean(ranking))
```

And also changes the order of bars in a bar chart:

```{r simple_bar_chart}
raw %>%
  group_by(flavor) %>%
  summarize(number = n(), average = mean(ranking)) %>%
  ggplot(mapping = aes(x = flavor, y = average)) +
  geom_col()
```

To learn more about how factors work and how to use them when analyzing categorical data,
please see [this paper](https://peerj.com/preprints/3163/) by McNamara and Horton.

## How do I refer to various arguments in a pipeline?

When we put a function in a pipeline using `%>%`,
that operator calls the function with the incoming data as the first argument,
so `data %>% func(arg)` is the same as `func(data, arg)`.
This is fine when we want the incoming data to be the first argument,
but what if we want it to be second?  Or third?

One possibility is to save the result so far in a temporary variable
and then start a second pipe:

```{r create-temps}
data <- tribble(
  ~left, ~right,
  1,     NA,
  2,     20
)

empties <- data %>%
  pmap_lgl(function(...) {
    args <- list(...)
    any(is.na(args))
  })

data %>%
  transmute(id = row_number()) %>%
  filter(empties) %>%
  pull(id)
```

This builds a logical vector `empties` with as many entries as `data` has rows,
then filters data according to which of the entries in the vector are `TRUE`.

A better practice is to use the parameter name `.`,
which means "the incoming data".
In some functions (e.g., a two-argument function being used in `map`)
we can also use `.x` and `.y` for the first and second arguments,
and for more arguments,
we can use `..1`, `..2`, and so on (with two dots at the front):

```{r using-dot}
data %>%
  pmap_lgl(function(...) {
    args <- list(...)
    any(is.na(args))
  }) %>%
  tibble(empty = .) %>%
  mutate(id = row_number()) %>%
  filter(empty) %>%
  pull(id)
```

In this model,
we create the logical vector,
then turn it into a tibble with one column called `empty`
(which is what `empty = .` does in `tibble`'s constructor).
After that,
we add another column with row numbers,
filter,
and pull out the row numbers.

And while we're here:
`row_number` doesn't do what its name suggests.
We're better off using `rowid_to_column`:

```{r rowid-to-column}
data %>%
  rowid_to_column()
```

## I thought you said that R encouraged functional programming?

I did.
Here is a function that reads a file and returns one of its columns:

```{r define-function, message=FALSE}
col_from_file <- function(filename, colname) {
  dat <- readr::read_csv(filename)
  dat[colname]
}

person_filename <- here::here("data", "person.csv")
col_from_file(person_filename, "family_name")
```

Note that the column name *must* be passed as a quoted string;
Chapter \@ref(nse) will show us how to pass unquoted column names.

We might occasionally want to allow the user to specify
what values in the file are to be considered NAs.
This small addition allows us to do that,
while keeping the empty string and the string `"NA"` as defaults:

```{r default-value, message=FALSE}
col_from_file <- function(filename, colname, na = c("", "NA")) {
  dat <- readr::read_csv(filename, na = na)
  dat[colname]
}

col_from_file(person_filename, "family_name", c("Dyer"))
```

We can also allow the user to specify any number of columns
by capturing "extra" parameters in `...`
and passing that value directly to `dplyr::select`:

```{r quote-multi-column, message=FALSE}
cols_from_file <- function(filename, ..., na = c("", "NA")) {
  readr::read_csv(filename, na = na) %>%
    dplyr::select(...)
}

cols_from_file(person_filename, personal_name, family_name)
```

Now that we can create functions,
we can use the tools in the `purrr` library to wield them.
`purrr::map` applies a function to each value in a vector in turn
and returns a list:

```{r purrr-map}
is_long_name <- function(name) {
  stringr::str_length(name) > 4
}

person <- read_csv(here::here("data", "person.csv"))
purrr::map(person$family_name, is_long_name)
```

For small calculations,
we will define the function where it is used—this is sometimes called
an [anonymous function](glossary.html#anonymous-function)
since it isn't given a name.
We will also use `purrr::map_lgl`
so that the result of the call is a logical vector rather than a list.
Similarly-named functions will give us numbers, character strings, and so on:

```{r anonymous-function}
purrr::map_lgl(person$family_name,
               function(name) stringr::str_length(name) > 4)
```

Little functions like this are so common
that `purrr` allows us to use write them as formulas using the `~ operator
with `.x` as a shorthand for the value from the vector being processed:

```{r}
purrr::map_chr(person$family_name, ~ stringr::str_to_upper(.x))
```

Other functions in `purrr` let us work on two vectors at once:

```{r}
purrr::map2_chr(person$personal_name,
                person$family_name,
                ~ stringr::str_c(.y, .x, sep = '_'))
```

If we need to collapse the result to a single value
(e.g., to use in `if`)
we have `purrr::some` and `purrr::every`:

```{r}
purrr::every(person$personal_name, ~ .x > 'M')
```

### Modify specific elements of a list:

```{r}
purrr::modify_at(person$personal_name, c(2, 4), stringr::str_to_upper)
```

*Use `modify_if` to upper-case names that are greater than "M".*

### Create an acronym:

```{r}
purrr::reduce(person$personal_name, ~stringr::str_c(.x, stringr::str_sub(.y, 1, 1)), .init = "")
```

*Explain why using `stringr::str_c(stringr::str_sub(.x, 1, 1), stringr::str_sub(.y, 1, 1))` doesn't work.*

### Create intermediate values:

```{r}
purrr::accumulate(person$personal_name, ~stringr::str_c(.x, stringr::str_sub(.y, 1, 1)), .init = "")
```

*Modify this so that the initial empty string isn't in the final result.*

## How does R give the appearance of immutable data?

Another feature of R that can surprise the unwary is its use of [copy-on-modify](glossary.html#copy-on-modify)
to make data appear [immutable](glossary.html#immutable)
(a jargon term meaning "cannot be changed after creation").
If two or more variables refer to the same data
and that data is updated via one variable,
R automatically makes a copy of the data so that the other variable's value doesn't change.
Here's a simple example:

```{r immutable-vec}
first <- c("red", "green", "blue")
second <- first
print(glue("before modification, first is {paste(first, collapse='-')} and second is {paste(second, collapse='-')}"))
first[[1]] <- "sulphurous"
print(glue("after modification, first is {paste(first, collapse='-')} and second is {paste(second, collapse='-')}"))
```

This is true of nested structures as well:

```{r immutable-tibble}
first <- tribble(
  ~left, ~right,
  101,   202,
  303,   404)
second <- first
first$left[[1]] <- 999
print("first after modification")
first
print("second after modification")
second
```

In this case,
the entire `left` column of `first` has been replaced:
tibbles (and data frames) are stored as lists of vectors,
so changing any value in a column triggers construction of a new column vector.

We can watch this happen using the `tracemem` function,
which shows us where objects live in the computer's memory:

```{r pryr}
first <- tribble(
  ~left, ~right,
  101,   202,
  303,   404
)
tracemem(first)
first$left[[1]] <- 999
untracemem(first)
```

This rather cryptic output tell us the address of the tibble,
then notifies us of changes to the tibble and its contents.
We can accomplish something a little more readable using `pryr::address`
(i.e., the `address` function from the pryr package):

```{r address}
left <- first$left # alias
print(glue("left column is initially at {pryr::address(left)}"))
first$left[[2]] <- 888
print(glue("after modification, the original column is still at {pryr::address(left)}"))
temp <- first$left # another alias
print(glue("but the first column is at {pryr::address(temp)}"))
```

(We need to use the [alias](glossary.html#alias) `temp` because `address(first$left)` doesn't work:
the argument to `address` needs to be a variable name.)

R's copy-on-modify semantics is particularly important when writing functions.
If we modify an argument inside a function,
that modification isn't visible to the caller,
so even functions that appear to modify structures usually don't.
("Usually", because there are exceptions, but we must stray off the path to find them.)

## What else should I worry about?

Ralph Waldo Emerson once wrote, "A foolish consistency is the hobgoblin of little minds."
Here, then, are few of the hobgoblins I've encountered on my journey through R.

### The `order` function

The function `order` generates indices to pull values into place rather than push them,
i.e.,
`order(x)[i]` is the index in `x` of the element that belongs at location `i`.
For example:

```{r order-func}
bases <- c("g", "c", "t", "a")
order(bases)
```

shows that the value at location 4 (the `"a"`) belongs in the first spot of the vector;
it does *not* mean that the value in the first location (the `"g"`) belongs in location 4.
This convention means that `something[order(something)]` does the right thing:

```{r}
bases[order(bases)]
```

### One of a set of values

The function `one_of` is a handy way to specify several values for matching
without complicated Boolean conditionals.
For example,
`gather(data, key = "year", value = "cases", one_of(c("1999", "2000")))`
collects data for the years 1999 and 2000.

### `|` and `&` are not the same as `||` and `&&`

Let's try some experiments:

```{r}
TRUE_TRUE <- c(TRUE, TRUE)
TRUE_FALSE <- c(TRUE, FALSE)
FALSE_TRUE <- c(FALSE, TRUE)
print(glue("TRUE_TRUE &  TRUE_FALSE: {paste(TRUE_TRUE &  TRUE_FALSE, collapse = ' ')}"))
print(glue("TRUE_TRUE &  FALSE_TRUE: {paste(TRUE_TRUE &  FALSE_TRUE, collapse = ' ')}"))
print(glue("TRUE_TRUE && TRUE_FALSE: {paste(TRUE_TRUE && TRUE_FALSE, collapse = ' ')}"))
print(glue("TRUE_TRUE && FALSE_TRUE: {paste(TRUE_TRUE && FALSE_TRUE, collapse = ' ')}"))
```

The difference is that `&` always returns a vector result after doing element-by-element conjunction,
while `&&` returns a scalar result.
This means that `&` is almost always what we want to use when working with data.

### Functions and columns

There is a function called `n`.
It's not the same thing as a column called `n`.
I only made this mistake a dozen times.

```{r func-col-n}
data <- tribble(
  ~a, ~n,
  1,  10,
  2,  20
)
data %>% summarize(total = sum(n))
```

```{r}
data %>% summarize(total = sum(n()))
```

## Key Points
```{r keypoints, child="keypoints/debt.md"}
```

```{r links, child="etc/links.md"}
```