Chapter 16 Exercises.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ef3d8e84",
   "metadata": {},
   "outputs": [],
   "source": [
    "suppressPackageStartupMessages(library(rstanarm))\n",
    "suppressPackageStartupMessages(library(ggformula))\n",
    "library(tibble)\n",
    "suppressPackageStartupMessages(library(glue))\n",
    "suppressPackageStartupMessages(library(dplyr))\n",
    "suppressPackageStartupMessages(library(modelr))\n",
    "suppressPackageStartupMessages(library(rstan))\n",
    "library(stringr)\n",
    "library(tidyr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d1a4ff5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the maximum number of columns and rows to display\n",
    "options(repr.matrix.max.cols=150, repr.matrix.max.rows=200)\n",
    "# Set the default plot size\n",
    "options(repr.plot.width=18, repr.plot.height=12)\n",
    "rstan_options(auto_write=TRUE)\n",
    "options(mc.cores = parallel::detectCores())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "159859d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "download_if_missing <- function(filename, url) {\n",
    "    if (!file.exists(filename)) {\n",
    "        dir.create(dirname(filename), showWarnings=FALSE, recursive=TRUE)\n",
    "        download.file(url, destfile = filename, method=\"curl\")\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b1eb12c",
   "metadata": {},
   "source": [
    "# Sample size calculations for estimating a proportion\n",
    "\n",
    "## Sample size for a main effect\n",
    "\n",
    "How large a sample survey would be required to estimate, to within a standard error of &pm;3%, the proportion of the U.S. population who support the death penalty?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "483ba901",
   "metadata": {},
   "source": [
    "## Sample size for an interaction\n",
    "\n",
    "About 14% of the U.S. population is Latino.\n",
    "How large would a national sample of Americans have to be in order to estimate, to within a standard error of &pm;3%, the proportion of Latinos in the United States who support the death penalty?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f010f2e9",
   "metadata": {},
   "source": [
    "## Sample size with increasing precision\n",
    "\n",
    "How large would a national sample of Americans have to be in order to estimate, to within a standard error of &pm;1%, the proportion who are Latino?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc961875",
   "metadata": {},
   "source": [
    "# Sample size calculation for estimating a difference\n",
    "\n",
    "Consider an election with two major candidates, A and B, and a minor candidate, C, who are believed to have support of approximately 45%, 35%, and 20% in the population.\n",
    "A poll is conducted with the goal of estimating the difference in support between candidates A and B.\n",
    "How large a sample would you estimate is needed to estimate this difference to within a standard error of 5 percentage points.\n",
    "(Hint: consider an outcome variable that is codes as +1, -1, and 0 for supporters of A, B, and C, respectively)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5f2eba5",
   "metadata": {},
   "source": [
    "# Power\n",
    "\n",
    "Following Figure 16.3, determine the power (The probability of getting an estimate that is \"statistically significantly\" different from zero at the 5% level) of a study where the true effect size is X standard errors from zero.\n",
    "Answer for the following values of X: 0, 1, 2, and 3."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f0003f5",
   "metadata": {},
   "source": [
    "# Power, type M error, and type S error\n",
    "\n",
    "Consider the experiment shown in Figure 16.1 where the true effect could not realistically be more than 2 percentage points and it is estimated with a standard error of 8.1 percentage points.\n",
    "\n",
    "## Power at 2% effect size\n",
    "\n",
    "Assuming the estimate is unbiased and normally distributed and the true effect size is 2 percentage points, use simulation to answer the following questions:\n",
    "\n",
    "* What is the power of this study?\n",
    "* What is the type M error rate?\n",
    "* What is the type S error rate?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39d284be",
   "metadata": {},
   "source": [
    "## Power\n",
    "\n",
    "Assuming the estimate is unbiased and normally distributed and the true effect size is *no more than* 2 percentage points in absolute value, what can you say about the power, type M error rate, and type S error rate?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a1f4680",
   "metadata": {},
   "source": [
    "# Design analysis for an experiment\n",
    "\n",
    "You conduct an experiment in which half the people get a special get-out-the-vote message and others do not. Then you follow up after the election with a random sample of 500 people to see if they voted.\n",
    "\n",
    "## Estimating standard error\n",
    "\n",
    "What will be the standard error of your estimate of effect size? Figure this out making reasonable assumptions about voter turnout and the true effect size?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57dfe25a",
   "metadata": {},
   "source": [
    "## Impact of assumptions\n",
    "\n",
    "Check how sensititve your standard error calculation is to your assumptions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dfcff26",
   "metadata": {},
   "source": [
    "## Conducting research\n",
    "\n",
    "For a range of plausible effect sizes, consider conclusions from this study, in light of the statistical significance filter. As a reasearcher, how can you avoid this problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08028d76",
   "metadata": {},
   "source": [
    "# Design analysis with pre-treatment information\n",
    "\n",
    "A new teaching method is hoped to increase scores by 5 points of a certain standardizes test.\n",
    "An experiment is performed on $n$ students, where half get this intervention and half get the control.\n",
    "Suppose that the standard deviation of test scored in the population is 20 points.\n",
    "Further suppose that a pre-test is available which has a correlation of 0.8 with the post-test under the constrol condition.\n",
    "What will be the standard error of the estimated treatment effect based on a fitted regression, assuming that the treatment effect is constant and independent of the value of the pre-test?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f6e9ec4",
   "metadata": {},
   "source": [
    "# Decline effect\n",
    "\n",
    "After a study is publiched on the effect of some treatment or intervention, it is common for the efstimated effect in future studies to be lower.\n",
    "Give five reasons why you might expect this to happen."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31fe4482",
   "metadata": {},
   "source": [
    "# Effect size and sample size\n",
    "\n",
    "Consider a toxin that can be tested on animals at different doses.\n",
    "Suppose a typical exposure level for humans is 1 (in some units), and at this level the toxin is hypothesized to introduce a risk of 0.01% of death per person.\n",
    "\n",
    "## Sample with effect size\n",
    "\n",
    "Consider different animal studies, each time assuming a linear dose-response relation (that is, 0.01% risk of death per animal per unit of the toxin), with does of 1, 100, and 10&thinsp;000.\n",
    "At each of these exposure levels, what is the sample size needed to have 80% power of detecting the effect?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d37d53f",
   "metadata": {},
   "source": [
    "## Sample with logarithmic response\n",
    "\n",
    "This time assume that response is a logged function of dose and redo the calculations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "481e234a",
   "metadata": {},
   "source": [
    "# Cluster sampling with equal-sized clusters\n",
    "\n",
    "A survey is being planned with the goal of interviewing $n$ people in some number $J$ of clusters.\n",
    "For simplicity, assume simple random sampling of clusters and a simple random sample of site $n/J$ (appropriately rounded) within each sampled cluster.\n",
    "\n",
    "Consider inferences for the proportion of Yes responses in the population for some question of interest.\n",
    "The estimate will simply be the average response for the $n$ people in the sample.\n",
    "Suppose that the true proportion of Yes responses is not too far from 0.5 and that the standard deviation amoung the mean responses of clusters is 0.1.\n",
    "\n",
    "## Sample error with cluster size\n",
    "\n",
    "Suppose the total sample size is $n=1000$.\n",
    "What is the standard error for the sample average if $J=1000$?\n",
    "What if $J = 100, 10, 1$?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7de246bc",
   "metadata": {},
   "source": [
    "## Optimal number of clusters\n",
    "\n",
    "Suppose the cost of the survey is \\\\$50 per interview, plus \\\\$500 per cluster.\n",
    "Further suppose that the goal is to estimate the proportion of Yes responses in the population with a standard error of no more than 2%.\n",
    "What values of $n$ and $J$ will achieve this at the lowest cost?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c87607c8",
   "metadata": {},
   "source": [
    "# Simulation for design analysis\n",
    "\n",
    "The folder [ElectricCompany](https://github.com/avehtari/ROS-Examples/tree/master/ElectricCompany/) contains data from the Electric Company experiment analyzed in Chapter 19.\n",
    "Suppose you wanted to perform a new experiment under similar conditions, but for simplicity just for second graders, with the goal of having 80% power to find a statistically significant result (at the 5% level) in grade 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d26f4c68",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename <- \"./data/ElectricCompany/electric.csv\"\n",
    "\n",
    "download_if_missing(filename,\n",
    "                    'https://raw.githubusercontent.com/avehtari/ROS-Examples/master/ElectricCompany/data/electric.csv')\n",
    "\n",
    "electric <- read.csv(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "caea8e87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"dataframe\">\n",
       "<caption>A data.frame: 6 × 7</caption>\n",
       "<thead>\n",
       "\t<tr><th></th><th scope=col>X</th><th scope=col>post_test</th><th scope=col>pre_test</th><th scope=col>grade</th><th scope=col>treatment</th><th scope=col>supp</th><th scope=col>pair_id</th></tr>\n",
       "\t<tr><th></th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;int&gt;</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>1</td><td>48.9</td><td>13.8</td><td>1</td><td>1</td><td>1</td><td>1</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>2</td><td>70.5</td><td>16.5</td><td>1</td><td>1</td><td>0</td><td>2</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>3</td><td>89.7</td><td>18.5</td><td>1</td><td>1</td><td>1</td><td>3</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>4</td><td>44.2</td><td> 8.8</td><td>1</td><td>1</td><td>0</td><td>4</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>5</td><td>77.5</td><td>15.3</td><td>1</td><td>1</td><td>1</td><td>5</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>6</td><td>84.7</td><td>15.0</td><td>1</td><td>1</td><td>0</td><td>6</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "A data.frame: 6 × 7\n",
       "\\begin{tabular}{r|lllllll}\n",
       "  & X & post\\_test & pre\\_test & grade & treatment & supp & pair\\_id\\\\\n",
       "  & <int> & <dbl> & <dbl> & <int> & <int> & <int> & <int>\\\\\n",
       "\\hline\n",
       "\t1 & 1 & 48.9 & 13.8 & 1 & 1 & 1 & 1\\\\\n",
       "\t2 & 2 & 70.5 & 16.5 & 1 & 1 & 0 & 2\\\\\n",
       "\t3 & 3 & 89.7 & 18.5 & 1 & 1 & 1 & 3\\\\\n",
       "\t4 & 4 & 44.2 &  8.8 & 1 & 1 & 0 & 4\\\\\n",
       "\t5 & 5 & 77.5 & 15.3 & 1 & 1 & 1 & 5\\\\\n",
       "\t6 & 6 & 84.7 & 15.0 & 1 & 1 & 0 & 6\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "A data.frame: 6 × 7\n",
       "\n",
       "| <!--/--> | X &lt;int&gt; | post_test &lt;dbl&gt; | pre_test &lt;dbl&gt; | grade &lt;int&gt; | treatment &lt;int&gt; | supp &lt;int&gt; | pair_id &lt;int&gt; |\n",
       "|---|---|---|---|---|---|---|---|\n",
       "| 1 | 1 | 48.9 | 13.8 | 1 | 1 | 1 | 1 |\n",
       "| 2 | 2 | 70.5 | 16.5 | 1 | 1 | 0 | 2 |\n",
       "| 3 | 3 | 89.7 | 18.5 | 1 | 1 | 1 | 3 |\n",
       "| 4 | 4 | 44.2 |  8.8 | 1 | 1 | 0 | 4 |\n",
       "| 5 | 5 | 77.5 | 15.3 | 1 | 1 | 1 | 5 |\n",
       "| 6 | 6 | 84.7 | 15.0 | 1 | 1 | 0 | 6 |\n",
       "\n"
      ],
      "text/plain": [
       "  X post_test pre_test grade treatment supp pair_id\n",
       "1 1 48.9      13.8     1     1         1    1      \n",
       "2 2 70.5      16.5     1     1         0    2      \n",
       "3 3 89.7      18.5     1     1         1    3      \n",
       "4 4 44.2       8.8     1     1         0    4      \n",
       "5 5 77.5      15.3     1     1         1    5      \n",
       "6 6 84.7      15.0     1     1         0    6      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "electric %>% head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8bc7042",
   "metadata": {},
   "source": [
    "## Assumptions\n",
    "\n",
    "State clearly the assumptions you are making for your design calculations.\n",
    "(Hint: you can set the numerical values for these assumptions based on the analysis of the existing Electric Company data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df2b9b8d",
   "metadata": {},
   "source": [
    "## Sample size for average comparisons\n",
    "\n",
    "Suppose that the data will be analyzsed by simply comparing the average scores for the treated classrooms to the average scores for the controls.\n",
    "How many classrooms would be needed for 80% power?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6d942b",
   "metadata": {},
   "source": [
    "## Sample size for gain\n",
    "\n",
    "Repeat the analysis, but supposing that the new data will be analyzed by comparin gthe average gain scored for the treated classrooms to the average gain scores of the controls."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45d1ffdb",
   "metadata": {},
   "source": [
    "## Sample size for regression\n",
    "\n",
    "Repeat, but supposing that the new data will be analyzed by regression, adjusting for pre-test scores as well as the treatment indicator."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd119a7a",
   "metadata": {},
   "source": [
    "# Optimal design\n",
    "\n",
    "## Power at minimal cost\n",
    "\n",
    "Suppose that the zinc study described in Section 16.3 would cost \\\\$150 for each treated child and \\\\$100 for each control.\n",
    "Under the assumptions given in that section, determine the number of control and treated children needed to attain 80% power at minimal total cost.\n",
    "You will need to set up a loop of simulations as illustrated for the example in the text.\n",
    "Asssume that the number of measurements per child is fixed at $K=7$ (that is, measuring every two months for a year)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c97c85",
   "metadata": {},
   "source": [
    "## Impact of changing design parameter\n",
    "\n",
    "Make a generalization of Figure 16.1 with several lines corresponding to different values of the design parameter $K$, the number of measurements for each child."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fac34a10",
   "metadata": {},
   "source": [
    "# Experiment with pre-treatment information\n",
    "\n",
    "An intervention is hoped to increase voter turnout in a local election from 20% to 25%.\n",
    "\n",
    "## Sample size for estimating treatment effect\n",
    "\n",
    "In a simple randomized experiment, how large a sample size would be needed so that the standard error of the estimated treatement effect is less than 2 percentage points?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3659bcf6",
   "metadata": {},
   "source": [
    "## Impact of a pre-treatment indicator\n",
    "\n",
    "Now suppose that previous voter turnout was known for all participants in the experiment.\n",
    "Make a reasonable assumption about the correlation between turnout in two successibe elections.\n",
    "Uner this assumption, how much would the standard error decrease if previous voter turnout was included as a pre-treatment predictor in a regression to estimate the treatment effect?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0ef00bb",
   "metadata": {},
   "source": [
    "# Sample size calculations for main effects and interactions\n",
    "\n",
    "In causal inference, it is often important to study varying treatment effects: for example, a treatment could be more effective for men than for women, or for healthy than for unhealthy patients.\n",
    "Suppose a study is designed to have 80% power to detect a main effect at a 95% confidence level.\n",
    "Further suppose that interactions of interest are half the size of main effects.\n",
    "\n",
    "## Power for interaction\n",
    "\n",
    "What is its power for detecting an interaction , comparing men to women (say) in a study that is half men and half women?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f439a2e",
   "metadata": {},
   "source": [
    "## Type M error\n",
    "\n",
    "Suppose 1000 studies of this size are performed.\n",
    "How many of the studies would you expect to report a statistically signficiant\" interaction?\n",
    "Of these, what is the expectation of the ratio of the estimated effect size to actual effect size?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.6.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}