There is more to statistics than
appears in your textbooks

Faculty Seminar

Oct 14, 2001

The purpose of this page is to serve as an outline for what I will cover in Faculty Seminar on Friday. It is going to be more of an outline than most pages I create.

Before I forget, all of this is covered in much more depth by going to http://www.uvm.edu/~dhowell/StatPages/Resampling/Resampling.html
- You can even download a pretty good piece of software that will perform the tests I describe here and much more.

Randomness

Parametric Tests usually rely on two critical assumptions--random selection and random assignment
We assume that the participants in our experiment are randomly drawn from some larger population.
- Because they are random samples, we can estimate the population parameters--e.g. m_i
  - In theory, it doesn't make sense to estimate the parameters of a population if you do not have a random sample from that population.
- Then we can ask if two or more population parameters are equal.
  - This leads us to test the null hypothesis: H₀: m_i = m_j
Because we have random assignment, we can conclude that any difference we find is due to the variable that we manipulated, rather than to some other extraneous variable.
So both random sampling and random assignment are fundamental characteristics of a valid experiment, even though, in practice, we often violate both of those.
We are speaking here of an approach put forth by Neyman and Pearson in 1928, and one reason to call it a "parametric" approach is that it uses random samples to estimate population parameters.

An Alternative View

Fisher (1930's) put forth an alternative approach, which is philosophically much different.
- Most people, including Fisher (?), think of Fisher as the originator of most of the parametric statistics we love.
- His approach was actually quite different.
Fisher didn't worry about random sampling because he didn't worry about populations.
Fisher focused on the data we collect, rather than the populations from which they came.
- Whereas most parametric statistics see the raw data as only a "window" on the population, Fisher saw the raw data as the only thing we know.
Fisher did assume that cases were randomly assigned to conditions, and for him this was the major assumption.
Fisher's "null hypothesis"
- "The observation we have for each case would be the same regardless of which treatment it received.
- Notice that there is nothing about m_i = m_j here. Nor is anything said about populations or parameters.
- This is a very general null hypothesis.
- Because it doesn't say anything about the population(s), it doesn't say anything about the shape of those populations, or whether two or more populations have the same variance.

Different views lead to different procedures

The Neyman-Pearson approach leads to estimating population parameters, and then testing their equality.
- This, in turn, requires random sampling from one or more populations, and estimation of parameters.
- It also requires underlying assumptions about normality if we want to test those parameters.
Fisher's view does not say anything about parameters, so does not require random sampling.
- His approach does require random assignment, which is the basis for the test.
- This approach assumes that if the treatment makes no difference, a particular observation was equally like to come from one group as another.
- This implies that any shuffling of the data among groups would produce results that would be equally likely under the null.

The Normal Theory--Parametric Approach

We assume two (or more) normally distributed populations, sampled at random, and we imagine what these population parameters would be.
We estimate the parameters from our data
We compute a t statistic on the assumption that the data are normally distributed.

Notice that we imagine these populations, and we imagine the results of sampling randomly from them. Then, if they are normal, we can work them into a t test.

The Randomization/Permutation Approach

We assume that we have a pool of cases which have been assigned at random to treatments.
We do not assume that the scores associated with those cases are normally distributed, that they have equal variances, etc.
We set up the null hypothesis that the treatment does not have an effect on the scores.
- This comes down to the null hypothesis that a person's score would be the same regardless of the treatment to which he or she was assigned.
- If that is true, that any arrangement of the data is as likely to have occurred as any other.
What we will do is to make up all possible arrangements of the data between groups, and for each arrangement we will calculate a statistic--e.g. the difference between means, or a t statistic calculated on those means.
We will then determine how likely we are to get a difference (or a t) as great as our if all arrangements are equally likely.

An example--Ludbrook and Dudley (1998)

This is an artificial example (unfortunately) that Ludbrook and Dudley developed to make a point. I think that data are perfectly believable--they aren't weirdly distorted data.
Imagine that we take 12 of our friends (i.e. not randomly sampled) and ask seven of them to eat significant amounts of fish for the next year, while the others are asked to eat only meat. At the end of the year we record their cholesterol levels (or differences in cholesterol level over the year, if you prefer). (The units are those given by Ludbrook and Dudley, and look odd for cholesterol.)
The data
- Fish eaters 5.42 5.86 6.16 6.55 6.80 7.00 7.11
- Meat eaters 6.51 7.56 7.61 7.84 11.50
  
  mean difference = 6.414 - 8.204 = -1.79
  median differnce = 6.55 - 7.61 = -1.06
  I could use either as my test statistic (perhaps with different outcomes), but I am going to use the means partly for convenience.
Notice that there is one unusually high score in the meat eater group, but this doesn't seem surprising.
If dietary differences have no influence on cholesterol level (the null hypothesis), then any one of those numbers would be just as likely to appear in the Fish eating group as in the Meat eating group.
- This is commonly called "exchangeability."
- Thus the following data would be just as likely as the data we had:
  - Fish eaters 5.42 6.16 6.51 6.55 7.00 7.11 7.61
  - Meat eaters 5.86 6.80 7.56 7.84 11.50
    
    mean difference = 6.623 - 7.912 = -1.289
    median difference = 6.55 - 7.56 = -1.01
- And, the following data are also just as likely to occur under the null
  - Fish eaters 5.42 5.86 6.16 6.51 6.55 6.80 7.00
  - Meat eaters 7.11 7.56 7.61 7.84 11.50
    
    (Notice that I have put the lowest scores in the Fish group.)
    
    mean difference = 6.329 - 8.324 = -1.995
    median difference = 6.51 - 7.61 = -1.10
One approach would be to calculate all possible arrangements of the observations across groups, and calculate the difference in means for each arrangement.
- There are 792 possible combinations
  - That is the number of ways of selecting 5 scores out of twelve (with the other 7 in the other group)
    - = 12!/(5! * 7!)
- For each of these 792 combinations, we would calculate the means and the difference between the means.
  - This gives us the kinds of differences we would expect under the null. This sampling distribution, copied from Ludbrook and Dudley, is
  - Notice the dark areas--these are the results that are more extreme than the ones that we obtained.
  - If you take all 792 arrangements, only 7/792 = .0088 = .88% of the outcomes are as extreme or more extreme than the ones we one we obtained.
    - Put another way, if the null hypothesis were true, the probability is only .0088 that we would obtain results as extreme as those we actually obtained.
    - Put another way, if the null hypothesis were true, we are very unlikely to obtain results like those we obtained.
    - We will reject H₀. We will conclude that eating fish leads to lower cholesterol than eating meat.
What does this tell us?
- We have said nothing here about populations of scores.
- We have said nothing about parameters
- We have not tested the traditional null hypothesis of H₀: m_i = m_j
- Our test was derived entirely by asking how these particular data could have come out if treatments had no differential effect.
- Notice that our conclusion is that diets make a difference, not that one diet leads to a higher mean cholesterol count than another.

Monte Carlo Solutions

The previous example was a simple one because we only had 5 scores in one group and 7 in the other.
- This meant that there were only 792 possible arrangements, and it is possible to calculate the results of all possible arrangements.
What if we had 24 friends, and had 10 and 14 scores in the Fish and Meat groups, respectively?
- This would require 24!/(10! * 14!) = 1,961,256 arrangements, and that is still a pretty small study.
- With 20 and 25 scores, this would require 45!/(20! * 25!) = 3,169,870,830,126 arrangements, which would be impractical, even for a supercomputer.
One very reasonable approach would be to take a random sample of all possible arrangements, and base our conclusions on these.
- For example, we could randomly rearrange the data 10,000 times, each time calculating the means, mean difference, or some other statistic, plot those 10,000 values as a reference distribution, and calculate our probability relative to that distribution.
- The result would be so close as to make no important difference.
- And it is easy to calculate 10,000 samples.
My program using Ludbrook and Dudley's data.

Notice:
- I have plotted t instead of the difference in means.
  - This is just a different metric, but it leads to exactly the same conclusion.
  - I chose t because I wanted a common metric that I could apply across a bunch of different kinds of experimental designs.
  - This is not a traditional t test--it just shares a statistic with it.
- The distribution of t is bimodal, whereas the sampling distribution of t for parametric tests is nearly normal.
  - This is the result of that one extreme score--when it gets randomly assigned to the Fish group, it makes the mean difference positive.
  - This does not adversely affect the legitimacy of the test.
- The probability value that results = .00855, which is very close to the "true" value of .0088
- The printout shows the value of t for our sample data (t = -2.345), and if we compared this to the usual sampling distribution of t, the two-tailed probability would be .041 if we assumed that the two populations had equal variances, and .105 if we did not. These are a lot larger than .0088.

What can we conclude?

With our randomization test we can conclude that the probability is only .0088 that we would get a result as extreme as this if the treatments had the same effect on cholesterol.
- We cannot conclude that the means are significantly different, because we didn't test that null.
- However, the most likely difference between the groups is that the "locations" are different.
- We could have used some other statistic, such as differences in medians, or differences in variance.
  - We would simply change the statistic we compute, not the rest of the test.
We cannot conclude that this result generalizes from this sample to the country as a whole, or the world.
- But we couldn't conclude that if we ran a traditional t test either.
- This is due to the fact that we did not randomly sample our participants.
The critical feature here was random assignment.

Where do we go from here?

There is nothing to stop us from applying this approach to almost any other experimental design that we can think of.
- We can test the mean of one sample.
- We can test the means of two independent samples.
- We can run a one-way analysis of variance.
- We can use this to look at correlations.
- There is some doubt about whether we can develop an analog to the factorial analysis of variance, and that is a loss.

What about intact groups?

I have made the point that the test hinges on random assignment. What if we want to compare males and females on cholesterol level?
We cannot do this on statistical grounds, but we can run our test and make the case for a difference by verbal argument.
- These leads me off in a direction that I would love to go, but don't have the time.
- This is the same issue that faces the parametric statistician, although the role of random assignment isn't as obvious in that case, because the way the test is constructed doesn't depend directly on it.

If these tests are so great, why don't we hear more about them?

For a long time they were very impractical, because the calculations were overwhelming.
- However, Fisher did once carry out such a test with about 32,000 rearrangements, but that was just to prove a point--and the first time he got it wrong.
While we gain something by avoiding assumptions, we also lose something,
- With a parametric test, we know that a significant difference means that the means are different, because our assumptions rule out all other possible differences.
- With a randomization test, we can only really conclude that something has happened as a result of the treatment.
Many designs (e.g. factorial designs with an interaction) are not readily amenable to this approach.
People generally don't know about the approach.
Actually, we do use these tests, except, for ease of calculation, we apply them to data that have been ranked.
- Such things as the Wilcoxon tests, the Mann-Whitney test, etc. are exactly what I have described, except performed on the ranked data.
Interesting, Kempthorne (a very highly regarded statistician in the 1950's) argued, along with Fisher, that the fact that the standard t test and the randomization test would come to similar conclusions could be taken as support for the t test.
- I learned yesterday that Kempthorne was actually a great fan of Fisher, and his comments in this regard may actually be a political statement as much as a statistical one.

What else is there out there that relates to "resampling?"

Bootstrapping

Here we take our raw data, treat them as a "pseudopopulation," draw repeated samples with replacement from that pseudopopulation, and plot the resulting sampling distribution.
I don't have time to go there, but I would like to some time.
My program will do it.

Last revised: 09/13/2001

There is more to statistics than appears in your textbooks