Introduction to Statistical Thinking
(With R, Without Calculus)
Benjamin Yakir, The Hebrew University
March, 2011
2
In memory of my father, Moshe Yakir, and the family he lost.
ii
Preface
The target audience for this book is college students who are required to learn
statistics, students with little background in mathematics and often no motivation to learn more. It is assumed that the students do have basic skills in using
computers and have access to one. Moreover, it is assumed that the students
are willing to actively follow the discussion in the text, to practice, and more
importantly, to think.
Teaching statistics is a challenge. Teaching it to students who are required
to learn the subject as part of their curriculum, is an art mastered by few. In
the past I have tried to master this art and failed. In desperation, I wrote this
book.
This book uses the basic structure of generic introduction to statistics course.
However, in some ways I have chosen to diverge from the traditional approach.
One divergence is the introduction of R as part of the learning process. Many
have used statistical packages or spreadsheets as tools for teaching statistics.
Others have used R in advanced courses. I am not aware of attempts to use
R in introductory level courses. Indeed, mastering R requires much investment
of time and energy that may be distracting and counterproductive for learning
more fundamental issues. Yet, I believe that if one restricts the application of
R to a limited number of commands, the benefits that R provides outweigh the
difficulties that R engenders.
Another departure from the standard approach is the treatment of probability as part of the course. In this book I do not attempt to teach probability
as a subject matter, but only specific elements of it which I feel are essential
for understanding statistics. Hence, KolmogorovàAxioms are out as well as
attempts to prove basic theorems and a Balls and Urns type of discussion. On
the other hand, emphasis is given to the notion of a random variable and, in
that context, the sample space.
The first part of the book deals with descriptive statistics and provides probability concepts that are required for the interpretation of statistical inference.
Statistical inference is the subject of the second part of the book.
The first chapter is a short introduction to statistics and probability. Students are required to have access to R right from the start. Instructions regarding
the installation of R on a PC are provided.
The second chapter deals with data structures and variation. Chapter 3
provides numerical and graphical tools for presenting and summarizing the distribution of data.
The fundamentals of probability are treated in Chapters 4 to 7. The concept
of a random variable is presented in Chapter 4 and examples of special types of
random variables are discussed in Chapter 5. Chapter 6 deals with the Normal
iii
iv
PREFACE
random variable. Chapter 7 introduces sampling distribution and presents the
Central Limit Theorem and the Law of Large Numbers. Chapter 8 summarizes
the material of the first seven chapters and discusses it in the statistical context.
Chapter 9 starts the second part of the book and the discussion of statistical inference. It provides an overview of the topics that are presented in the
subsequent chapter. The material of the first half is revisited.
Chapters 10 to 12 introduce the basic tools of statistical inference, namely
point estimation, estimation with a confidence interval, and the testing of statistical hypothesis. All these concepts are demonstrated in the context of a single
measurements.
Chapters 13 to 15 discuss inference that involve the comparison of two measurements. The context where these comparisons are carried out is that of
regression that relates the distribution of a response to an explanatory variable.
In Chapter 13 the response is numeric and the explanatory variable is a factor
with two levels. In Chapter 14 both the response and the explanatory variable
are numeric and in Chapter 15 the response in a factor with two levels.
Chapter 16 ends the book with the analysis of two case studies. These
analyses require the application of the tools that are presented throughout the
book.
This book was originally written for a pair of courses in the University of the
People. As such, each part was restricted to 8 chapters. Due to lack of space,
some important material, especially the concepts of correlation and statistical
independence were omitted. In future versions of the book I hope to fill this
gap.
Large portions of this book, mainly in the first chapters and some of the
quizzes, are based on material from the online book ïllaborative Statistics¢y Barbara Illowsky and Susan Dean (Connexions, March 2, 2010. http://
cnx.org/content/col10522/1.37/). Most of the material was edited by this
author, who is the only person responsible for any errors that where introduced
in the process of editing.
Case studies that are presented in the second part of the book are taken
from Rice Virtual Lab in Statistics can be found in their Case Studies section.
The responsibility for mistakes in the analysis of the data, if such mistakes are
found, are my own.
I would like to thank my mother Ruth who, apart from giving birth, feeding
and educating me, has also helped to improve the pedagogical structure of this
text. I would like to thank also Gary Engstrom for correcting many of the
mistakes in English that I made.
This book is an open source and may be used by anyone who wishes to do so.
(Under the conditions of the Creative Commons Attribution License (CC-BY
3.0).))
Jerusalem, March 2011
Benjamin Yakir
Contents
Preface
iii
I
1
Introduction to Statistics
1 Introduction
1.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . .
1.2 Why Learn Statistics? . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6 The R Programming Environment . . . . . . . . . . . . . . . . .
1.6.1 Some Basic R Commands . . . . . . . . . . . . . . . . . .
1.7 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
3
3
4
5
6
7
7
10
13
2 Sampling and Data Structures
15
2.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 15
2.2 The Sampled Data . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.1 Variation in Data . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.2 Variation in Samples . . . . . . . . . . . . . . . . . . . . . 16
2.2.3 Frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.4 Critical Evaluation . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Reading Data into R . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.1 Saving the File and Setting the Working Directory . . . . 19
2.3.2 Reading a CSV File into R . . . . . . . . . . . . . . . . . . 23
2.3.3 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3 Descriptive Statistics
29
3.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 29
3.2 Displaying Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2.1 Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.2 Box Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 Measures of the Center of Data . . . . . . . . . . . . . . . . . . . 35
3.3.1 Skewness, the Mean and the Median . . . . . . . . . . . . 36
3.4 Measures of the Spread of Data . . . . . . . . . . . . . . . . . . . 38
v
vi
CONTENTS
3.5
3.6
Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
45
4 Probability
47
4.1 Student Learning Objective . . . . . . . . . . . . . . . . . . . . . 47
4.2 Different Forms of Variability . . . . . . . . . . . . . . . . . . . . 47
4.3 A Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.4.1 Sample Space and Distribution . . . . . . . . . . . . . . . 54
4.4.2 Expectation and Standard Deviation . . . . . . . . . . . . 56
4.5 Probability and Statistics . . . . . . . . . . . . . . . . . . . . . . 59
4.6 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5 Random Variables
65
5.1 Student Learning Objective . . . . . . . . . . . . . . . . . . . . . 65
5.2 Discrete Random Variables . . . . . . . . . . . . . . . . . . . . . 65
5.2.1 The Binomial Random Variable . . . . . . . . . . . . . . . 66
5.2.2 The Poisson Random Variable . . . . . . . . . . . . . . . 71
5.3 Continuous Random Variable . . . . . . . . . . . . . . . . . . . . 74
5.3.1 The Uniform Random Variable . . . . . . . . . . . . . . . 75
5.3.2 The Exponential Random Variable . . . . . . . . . . . . . 79
5.4 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6 The Normal Random Variable
87
6.1 Student Learning Objective . . . . . . . . . . . . . . . . . . . . . 87
6.2 The Normal Random Variable . . . . . . . . . . . . . . . . . . . . 87
6.2.1 The Normal Distribution . . . . . . . . . . . . . . . . . . 88
6.2.2 The Standard Normal Distribution . . . . . . . . . . . . . 90
6.2.3 Computing Percentiles . . . . . . . . . . . . . . . . . . . . 92
6.2.4 Outliers and the Normal Distribution . . . . . . . . . . . 94
6.3 Approximation of the Binomial Distribution . . . . . . . . . . . . 96
6.3.1 Approximate Binomial Probabilities and Percentiles . . . 96
6.3.2 Continuity Corrections . . . . . . . . . . . . . . . . . . . . 97
6.4 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7 The Sampling Distribution
105
7.1 Student Learning Objective . . . . . . . . . . . . . . . . . . . . . 105
7.2 The Sampling Distribution . . . . . . . . . . . . . . . . . . . . . 105
7.2.1 A Random Sample . . . . . . . . . . . . . . . . . . . . . . 106
7.2.2 Sampling From a Population . . . . . . . . . . . . . . . . 107
7.2.3 Theoretical Models . . . . . . . . . . . . . . . . . . . . . . 112
7.3 Law of Large Numbers and Central Limit Theorem . . . . . . . . 115
7.3.1 The Law of Large Numbers . . . . . . . . . . . . . . . . . 115
7.3.2 The Central Limit Theorem (CLT) . . . . . . . . . . . . . 116
7.3.3 Applying the Central Limit Theorem . . . . . . . . . . . . 119
7.4 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
CONTENTS
vii
8 Overview and Integration
125
8.1 Student Learning Objective . . . . . . . . . . . . . . . . . . . . . 125
8.2 An Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
8.3 Integrated Applications . . . . . . . . . . . . . . . . . . . . . . . 127
8.3.1 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
8.3.2 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
8.3.3 Example 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8.3.4 Example 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
8.3.5 Example 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
II
Statistical Inference
137
9 Introduction to Statistical Inference
139
9.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 139
9.2 Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
9.3 The Cars Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . 141
9.4 The Sampling Distribution . . . . . . . . . . . . . . . . . . . . . 144
9.4.1 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
9.4.2 The Sampling Distribution . . . . . . . . . . . . . . . . . 145
9.4.3 Theoretical Distributions of Observations . . . . . . . . . 146
9.4.4 Sampling Distribution of Statistics . . . . . . . . . . . . . 147
9.4.5 The Normal Approximation . . . . . . . . . . . . . . . . . 148
9.4.6 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . 149
9.5 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
9.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
10 Point Estimation
159
10.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 159
10.2 Estimating Parameters . . . . . . . . . . . . . . . . . . . . . . . . 159
10.3 Estimation of the Expectation . . . . . . . . . . . . . . . . . . . . 160
10.3.1 The Accuracy of the Sample Average . . . . . . . . . . . 161
10.3.2 Comparing Estimators . . . . . . . . . . . . . . . . . . . . 164
10.4 Variance and Standard Deviation . . . . . . . . . . . . . . . . . . 166
10.5 Estimation of Other Parameters . . . . . . . . . . . . . . . . . . 171
10.6 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
10.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
11 Confidence Intervals
181
11.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 181
11.2 Intervals for Mean and Proportion . . . . . . . . . . . . . . . . . 181
11.2.1 Examples of Confidence Intervals . . . . . . . . . . . . . . 182
11.2.2 Confidence Intervals for the Mean . . . . . . . . . . . . . 183
11.2.3 Confidence Intervals for a Proportion . . . . . . . . . . . 187
11.3 Intervals for Normal Measurements . . . . . . . . . . . . . . . . . 188
11.3.1 Confidence Intervals for a Normal Mean . . . . . . . . . . 190
11.3.2 Confidence Intervals for a Normal Variance . . . . . . . . 192
11.4 Choosing the Sample Size . . . . . . . . . . . . . . . . . . . . . . 195
11.5 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
11.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
viii
CONTENTS
12 Testing Hypothesis
203
12.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 203
12.2 The Theory of Hypothesis Testing . . . . . . . . . . . . . . . . . 203
12.2.1 An Example of Hypothesis Testing . . . . . . . . . . . . . 204
12.2.2 The Structure of a Statistical Test of Hypotheses . . . . . 205
12.2.3 Error Types and Error Probabilities . . . . . . . . . . . . 208
12.2.4 p-Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
12.3 Testing Hypothesis on Expectation . . . . . . . . . . . . . . . . . 211
12.4 Testing Hypothesis on Proportion . . . . . . . . . . . . . . . . . . 218
12.5 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
12.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
13 Comparing Two Samples
227
13.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 227
13.2 Comparing Two Distributions . . . . . . . . . . . . . . . . . . . . 227
13.3 Comparing the Sample Means . . . . . . . . . . . . . . . . . . . . 229
13.3.1 An Example of a Comparison of Means . . . . . . . . . . 229
13.3.2 Confidence Interval for the Difference . . . . . . . . . . . 232
13.3.3 The t-Test for Two Means . . . . . . . . . . . . . . . . . . 235
13.4 Comparing Sample Variances . . . . . . . . . . . . . . . . . . . . 237
13.5 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
13.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
14 Linear Regression
247
14.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 247
14.2 Points and Lines . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
14.2.1 The Scatter Plot . . . . . . . . . . . . . . . . . . . . . . . 248
14.2.2 Linear Equation . . . . . . . . . . . . . . . . . . . . . . . 251
14.3 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
14.3.1 Fitting the Regression Line . . . . . . . . . . . . . . . . . 253
14.3.2 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
14.4 R-squared and the Variance of Residuals . . . . . . . . . . . . . . 260
14.5 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
14.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
15 A Bernoulli Response
281
15.1 Student Learning Objectives . . . . . . . . . . . . . . . . . . . . . 281
15.2 Comparing Sample Proportions . . . . . . . . . . . . . . . . . . . 282
15.3 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 285
15.4 Solved Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
16 Case Studies
299
16.1 Student Learning Objective . . . . . . . . . . . . . . . . . . . . . 299
16.2 A Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
16.3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
16.3.1 Physicians eactions to the Size of a Patient . . . . . . . 300
16.3.2 Physical Strength and Job Performance . . . . . . . . . . 306
16.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
16.4.1 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . 313
16.4.2 Discussion in the Forum . . . . . . . . . . . . . . . . . . . 314
Part I
Introduction to Statistics
1
Chapter 1
Introduction
1.1
Student Learning Objectives
This chapter introduces the basic concepts of statistics. Special attention is
given to concepts that are used in the first part of this book, the part that
deals with graphical and numeric statistical ways to describe data (descriptive
statistics) as well as mathematical theory of probability that enables statisticians
to draw conclusions from data.
The course applies the widely used freeware programming environment for
statistical analysis, known as R. In this chapter we will discuss the installation
of the program and present very basic features of that system.
By the end of this chapter, the student should be able to:
ecognize key terms in statistics and probability.
nstall the R program on an accessible computer.
earn and apply a few basic operations of the computational system R.
1.2
Why Learn Statistics?
You are probably asking yourself the question, èen and where will I use
statistics? If you read any newspaper or watch television, or use the Internet,
you will see statistical information. There are statistics about crime, sports,
education, politics, and real estate. Typically, when you read a newspaper
article or watch a news program on television, you are given sample information.
With this information, you may make a decision about the correctness of a
statement, claim, or ¡ct Statistical methods can help you make the ¥st
educated guessŠSince you will undoubtedly be given statistical information at some point in
your life, you need to know some techniques to analyze the information thoughtfully. Think about buying a house or managing a budget. Think about your
chosen profession. The fields of economics, business, psychology, education, biology, law, computer science, police science, and early childhood development
require at least one course in statistics.
3
CHAPTER 1. INTRODUCTION
2
0
1
y = Frequency
3
4
4
5
5.5
6
6.5
7
8
9
x = Time
Figure 1.1: Frequency of Average Time (in Hours) Spent Sleeping per Night
Included in this chapter are the basic ideas and words of probability and
statistics. In the process of learning the first part of the book, and more so in
the second part of the book, you will understand that statistics and probability
work together.
1.3
Statistics
The science of statistics deals with the collection, analysis, interpretation, and
presentation of data. We see and use data in our everyday lives. To be able
to use data correctly is essential to many professions and is in your own best
self-interest.
For example, assume the average time (in hours, to the nearest half-hour) a
group of people sleep per night has been recorded. Consider the following data:
5, 5.5, 6, 6, 6, 6.5, 6.5, 6.5, 6.5, 7, 7, 8, 8, 9 .
In Figure 1.1 this data is presented in a graphical form (called a bar plot). A bar
plot consists of a number axis (the x-axis) and bars (vertical lines) positioned
1.4. PROBABILITY
5
above the number axis. The length of each bar corresponds to the number
of data points that obtain the given numerical value. In the given plot the
frequency of average time (in hours) spent sleeping per night is presented with
hours of sleep on the horizontal x-axis and frequency on vertical y-axis.
Think of the following questions:
ould the bar plot constructed from data collected from a different group
of people look the same as or different from the example? Why?
f one would have carried the same example in a different group with the
same size and age as the one used for the example, do you think the results
would be the same? Why or why not?
here does the data appear to cluster? How could you interpret the
clustering?
The questions above ask you to analyze and interpret your data. With this
example, you have begun your study of statistics.
In this course, you will learn how to organize and summarize data. Organizing and summarizing data is called descriptive statistics. Two ways to
summarize data are by graphing and by numbers (for example, finding an average). In the second part of the book you will also learn how to use formal
methods for drawing conclusions from ïod$ata. The formal methods are
called inferential statistics. Statistical inference uses probabilistic concepts to
determine if conclusions drawn are reliable or not.
Effective interpretation of data is based on good procedures for producing
data and thoughtful examination of the data. In the process of learning how
to interpret data you will probably encounter what may seem to be too many
mathematical formulae that describe these procedures. However, you should
always remember that the goal of statistics is not to perform numerous calculations using the formulae, but to gain an understanding of your data. The
calculations can be done using a calculator or a computer. The understanding
must come from you. If you can thoroughly grasp the basics of statistics, you
can be more confident in the decisions you make in life.
1.4
Probability
Probability is the mathematical theory used to study uncertainty. It provides
tools for the formalization and quantification of the notion of uncertainty. In
particular, it deals with the chance of an event occurring. For example, if the
different potential outcomes of an experiment are equally likely to occur then
the probability of each outcome is taken to be the reciprocal of the number of
potential outcomes. As an illustration, consider tossing a fair coin. There are
two possible outcomes ! head or a tail !nd the probability of each outcome
is 1/2.
If you toss a fair coin 4 times, the outcomes may not necessarily be 2 heads
and 2 tails. However, if you toss the same coin 4,000 times, the outcomes will
be close to 2,000 heads and 2,000 tails. It is very unlikely to obtain more than
2,060 tails and it is similarly unlikely to obtain less than 1,940 tails. This is
consistent with the expected theoretical probability of heads in any one toss.
Even though the outcomes of a few repetitions are uncertain, there is a regular
6
CHAPTER 1. INTRODUCTION
pattern of outcomes when the number of repetitions is large. Statistics exploits
this pattern regularity in order to make extrapolations from the observed sample
to the entire population.
The theory of probability began with the study of games of chance such as
poker. Today, probability is used to predict the likelihood of an earthquake, of
rain, or whether you will get an T in this course. Doctors use probability
to determine the chance of a vaccination causing the disease the vaccination is
supposed to prevent. A stockbroker uses probability to determine the rate of
return on a clientàinvestments. You might use probability to decide to buy a
lottery ticket or not.
Although probability is instrumental for the development of the theory