Overview of Randomization Tests

I want to discuss randomization procedures for data analysis, but I have worked myself into somewhat of a mess. Years ago I wrote a set of programs in Visual Basic that worked very well. (They were quite pretty, too.) I then created a set of pages based on those programs. More recently I have approached the whole problem from the point of view of R. These pages come in different forms. Some of them, such as philosophy.html, are generic and do not depend on any particular program. They are like the first couple of chapters in a textbook, which layout the underlying issues. The rest of them are directed toward a specific problem--e.g. comparing the means of two groups, running an analysis of variance, and so on. Where I had a page discussing procedures using Visual Basic, and another page covering roughly the same material using R, I am combining the two. Often you will see a discussion about comparing two groups where the underlying program is first, Visual Basic, followed by R code to do the same thing in R. I tried working with two separate pages, but that became unwieldy. There is also a page about downloading R and understanding the basic commands, and another page about where to find the Visual Basic programs and how to install them.

Before elaborating on specific procedures, I need to say something, actually quite a lot, on the characteristics of randomization tests.

Randomization tests differ from parametric tests in almost every respect.


I need to say something about exchangeability. It applies to the null hypothesis sampling distribution--in other words, data are exchangeable under the null. Phil Good, for example, is a big fan of this term. He would argue that if the scores in one group have a higher variance than the other, then the data are not exchangeable and the test is not valid. BUT, if the hypothesis being tested is that treatments have no effect on scores, then under the null hypothesis why would one set of scores have a higher variance other than by chance? The problem is that we have to select the statistic to test with care. We normally test means, or their equivalent, but we also need to consider variances, for example, because that is another way in which the treatment groups could differ. If we are focussing on means, then we have to assume exchangeability including variance. But we need to be specific. So much for that little hobby horse of mine.

Cliff Lunneborg has written an excellent discussion of randomization tests. (Unfortunately, Cliff Lunneborg has died, and the paper is no longer available at his web site. By some good fortune, I happen to have copies of those pages. You can download files at "Paper-One and Paper Two. I consider these required reading to fully understand the underlying issues behind randomization tests. Lunneborg writes extremely well, but (and?) he chooses his words very carefully. Don't read this when you are too tired to do anything else--you have to be alert.

Structure of these pages

I have broken these many pages down into three sections. The first deals with the pages that are basically designed to explain the logic and structure of resampling tests. I have labled these "Background Material" for obvious reasons. I then move to what I have called "Randomization Tests." These are pages that deal with specific tests, such as comparing two group means. Here I have combined my pages on Visual Basic programs and those on programs written in R. Generally the Visual Basic section serves better to describe the tests and their logic, while the sectionon R is primarily aimed at providing the code for similar tests in R. In R I have deliberately pulled coding steps apart, when I could have shortened the code by using more complex commands. I did this to make it easier for you to follow what I am doing. After the randomization tests I provide a section on bootstrapping. This is a bit shorter simply because topics on bootstrapping are more easily covered and are fewer. Again I have tried to provide programs in both Visual Basic and R.

Background Material


Edgington, E. & Onghena, P. (2007) Randomization tests, New York, Chapman & Hall.

Efron, B. & Tibshirani, R. J. (1993) An introduction to the bootstrap. New York: Chapman and Hall.

Lunneborg, C. E. (2000) Random assignment of available cases: Let the inference fit the design. http://faculty.washington.edu/lunnebor/Australia/randomiz.pdf


David C. Howell
University of Vermont