Comparing the Wald Method to Bootstrapping for a Population Proportion and Why Resampling Methods Belong in Elementary Statistics Courses

Introduction

Almost all elementary statistics textbooks are broken up and ordered in the following way: Part 1: Descriptive Statistics, Part 2: Probability, and Part 3: Inferential Statistics (Bluman, 2017; Larson, 2022; Levin et al., 2016; Witte & Witte, 2017). The most powerful takeaway of an elementary statistics course is the inferential statistics portion found in Part 3. Generally speaking, students do well in Part 1, poorly in Part 2, and passable in Part 3. Probability is the necessary link between Parts 1 and 3 with the key takeaway being an understanding and familiarity with sampling distributions and how they apply towards statistical inference. With that said, probability is the most challenging part of the course. Resampling methods bridge Parts 1 and 3 without the challenges of learning all the details of probability. Once students learn about relative frequency histograms from Part 1, the thought is that they can more intuitively and easily understand sampling distributions produced by resampling methods (Chance et al., 2017; Tintle et al., 2018). The resampling approach is very direct and does not require a long sequence of classical probability, sampling distributions, and central limit theorem. A variety of current textbooks have taken the resampling approach (Lock et al., 2021; Tintle et al., 2020). Not only are resampling methods easier to understand, they have revolutionized statistics and are commonly used in practice (Bai & Pan, 2008; Bradley Efron’s Profile | Stanford Profiles, n.d.; Latpate et al., 2021; Yu, 2001).

The aim of this study is to compare the bootstrapping percentile method for estimating one sample proportion to methods traditionally taught in elementary statistics courses. In particular, for bootstrapping the percentile method will be compared to the Wald method, the plus 4 approximation of Agresti and Coull’s method, and the Clopper-Pearson “exact” method. The latter is not commonly introduced in elementary statistics textbooks but will serve as a comparison for small sample and edge cases where $\widehat{p}$ is close to zero or one. All simulation, graphs, and tables in this study were produced using R.

Bootstrapping

The general idea of bootstrapping is to construct a sampling distribution based on the sample obtained. Once a sample is obtained, a computer is used to resample from the original sample with replacement. These resamples are referred to as bootstrap samples. For each bootstrap sample, the sample statistic of interest is calculated. This is called the bootstrap estimate. This procedure is repeated usually at least 1,000 times. Finally, the bootstrap estimates are sorted and used to construct a confidence interval.

Notation

${\widehat{\theta}}^{*}$ is the bootstrapped estimate of $\theta$ and $\widehat{\theta}$ is the original sample estimate of $\theta$. Similarly, ${\widehat{SE}}^{*}$ is an estimator of the standard error of ${\widehat{\theta}}^{*}$ and is calculated for each bootstrap sample. $\widehat{SE}$ is an estimator of the standard error of $\widehat{\theta}$ and is calculated from the original sample.

Percentile Method

There is a variety of ways to form confidence intervals by bootstrapping. The percentile method is the most straightforward and easiest to understand. Obtain R many bootstrap samples (usually at least 1,000) from the original sample and obtain the bootstrap estimates of interest. Then sort the bootstrap estimates and form an interval by using the cutoffs for the $\frac{\alpha}{2}$ and $1 - \frac{\alpha}{2}$ percentiles.

For instance, suppose you resample 1,000 times, compute 1,000 bootstrap estimates, and sort them in ascending order. Then you will have the following sorted array of estimates

$$\left\lbrack {\widehat{\theta}}_{1}^{*},\ {\widehat{\theta}}_{2}^{*},\ {\widehat{\theta}}_{3}^{*},\ldots,\ {\widehat{\theta}}_{998}^{*},\ {\widehat{\theta}}_{999}^{*},\ {\widehat{\theta}}_{1000}^{*} \right\rbrack$$

where ${\widehat{\theta}}_{1}^{*}, \leq {\widehat{\theta}}_{2}^{*} \leq {\widehat{\theta}}_{3}^{*} \leq \ldots \leq {\widehat{\theta}}_{1000}^{*}$. Now suppose you would like to compute a 95% confidence interval to estimate $\theta$. You would form it by using ${\widehat{\theta}}_{25}^{*}$ as the lower limit and ${\widehat{\theta}}_{975}^{*}$ as the upper limit. In general, the percentile confidence interval looks like the following

$$\left( {\widehat{\theta}}_{\frac{\alpha}{2}}^{*},\ {\widehat{\theta}}_{1 - \frac{\alpha}{2}}^{*} \right)$$

For the purposes of this paper $\theta = p = x/N$, is the population proportion, and $\widehat{\theta} = \widehat{p} = X/n$, is the sample proportion, where $X\sim binomial(n,\ p)$. The following are classical probability methods used in elementary statistics courses.

Wald

This is the standard classical approach to interval estimation for the population proportion $p$.

$$\left( \widehat{p} - z_{1 - \frac{\alpha}{2}}\sqrt{\frac{\widehat{p}\left( 1 - \widehat{p} \right)}{n}},\ \widehat{p} + z_{1 - \frac{\alpha}{2}}\sqrt{\frac{\widehat{p}\left( 1 - \widehat{p} \right)}{n}} \right)$$

Adjusted Wald (Plus 4 approximation of Agresti Coull method for 95% confidence)

$${\widehat{p}}_{a} = \frac{x + 2}{n + 4}$$

$$\left( {\widehat{p}}_{a} - 1.96\sqrt{\frac{{\widehat{p}}_{a}\left( 1 - {\widehat{p}}_{a} \right)}{n}},\ {\widehat{p}}_{a} + 1.96\sqrt{\frac{{\widehat{p}}_{a}\left( 1 - {\widehat{p}}_{a} \right)}{n}} \right)$$

The Clopper-Pearson “exact” method is usually not taught in an elementary statistics course, but it is shown here and will be used as a baseline for comparison in cases where $n$ is small and/or $\widehat{p}$ is close to zero or one.

$$\left( 1 + \frac{n - x + 1}{xF_{\frac{\alpha}{2},\ 2x,\ 2(n - x + 1)}},\ 1 + \frac{n - x}{(x + 1)F_{1 - \frac{\alpha}{2},\ 2(x + 1),\ 2(n - x)}} \right)$$

Methods

Samples from binomial distributions with varying sample sizes from $n = 5$ to $n = 1000$ and from $p = 0.01$ to $p = 0.99$ were simulated 1,000 times for each combination of $n$ and $p$. The tables in Appendix A show all levels used. Then, confidence intervals were constructed by the various methods listed. For each method, the number of times $p$ was between the lower and upper bound of the interval was counted and divided by 1,000 to determine the coverage probability. For the bootstrap method, resampling was repeated 1,000 times. All confidence intervals were constructed with 95% confidence. All simulations were done using R.

Results

Coverage probability plots and tables (Appendix A) are presented to summarize the results. In the coverage probability plots, we see that the Wald and bootstrap methods mirror each other while the adjusted Wald and exact methods mirror each other as well. The Wald method performs poorly in the edge cases when $p$ goes to 0 or 1 across all levels of $n$ with improvements as $n$ gets larger. The bootstrap method does this as well, but it stabilizes over all levels of $p$ when $n \geq 500$, giving consistent results around 95% coverage probability. Overall, the bootstrap percentile method performed at least as well as the Wald method when considering coverage probabilities. Where the bootstrap percentile method performed slightly better than the Wald method was for smaller sample sizes. In particular for sample sizes $n = 5$ through $n = 50$, we see that the bootstrap method performs a little better than the Wald method (Figures 1 through 4). Another important consideration is the average interval width. In terms of average interval width, the bootstrap and Wald methods performed nearly identically. For larger sample sizes, we see that the variability in interval widths was larger for the bootstrap method. Each cell in the tables found in Appendix A has, from left to right, coverage probability, average interval width, and the standard deviation of the interval width.

Conclusion

The bootstrap percentile method is very easy to understand and performs at least as well as the Wald method. The only time the Wald method had any advantage over the bootstrap was that it had better consistency (smaller standard deviation) in interval width when $n \geq 250$. What the Wald method gave up to the bootstrap method in that range for $n$ was coverage probability in the edge cases for $p$. Resampling methods such as bootstrapping are commonly used and rely on very basic mathematics (Bai & Pan, 2008; Bradley Efron’s Profile | Stanford Profiles, n.d.; Latpate et al., 2021; Yu, 2001). One of the advantages that they bring to an elementary statistics course is a low barrier to understanding and to potentially improve student understanding of concepts and even retention (Tintle et al., 2018). Once a student gets through descriptive statistics, he or she does not need to know anything about probability and sampling distributions to comprehend the percentile method for bootstrapping confidence intervals. Furthermore, Recommendation 5: Use technology to explore concepts and analyze data from the GAISE College Report that states technology should aid student learning through the use of simulation (Chance et al., 2017). Another benefit of resampling methods is that the visualization of the sampling distribution becomes more concrete when it is obtained from data as opposed to an abstract distribution from mathematical statistics. The demystifying of sampling distributions can help students more intuitively understand important concepts such p-values. We also note that the performance of the bootstrap method in every relevant measure is comparable, if not better, than the Wald method. By adjusting the Wald method with the addition of 2 successes and 2 failures (plus 4 method) we see drastic improvement in the Wald method as noted by Agresti and Coull in their seminal 1998 paper, but students still need to spend weeks on learning probability and sampling distributions. In addition, adding 2 successes and 2 failures can seem arbitrary and mysterious to students as to why it makes such an improvement. At the very least, it is time that elementary statistics courses begin including resampling methods so students can gain familiarity with what is commonly used in practice (Bai & Pan, 2008; Bradley Efron’s Profile | Stanford Profiles, n.d.; Latpate et al., 2021; Yu, 2001).