Superficially, a test on medians is similar to a test on means. But there are fundamental differences. With the two previous randomization tests (two independent samples and two paired samples), we used the t computed between the means of two groups to measure differences. We don't have a t that we can use with medians, because we don't have a useful way of calculating the standard error, which is needed to compute a t.
One alternative to this might have been to take the obtained median difference as my statistic, and to count the number of randomizations that obtained a median difference as great as the one in our obtained data. That was not the approach that I took when I wrote the program included in Resampling.exe, although I may go back to that in the near future.
What I did was actually more in line with what we will do with bootstrapping. I calculate confidence limits on the median difference under the null hypothesis, and will then rejected the null hypothesis if the obtained difference in medians falls outside these limits. The advantage of this approach is that it gives us confidence limits, which are nice to have. On the other hand, the limits that it produces are probably not the ones that really interest us. These are the limits we would obtain when the null hypothesis is true, they are not limits on the true difference between medians. In fact, I am not sure how we would get limits on the true difference between medians with a randomization test. The disadvantage of the approach that I have taken is that it will show that you can reject the null if the obtained difference falls outside the interval, but it will not give you a p value on the probability of such a difference under the null.
As I say, I may change the approach. But until I do, I need to describe the one I used.
I will use the same example that I will use when comparing two medians with bootstrapping. This is a study by Merrell on maze performance of mice raised in the daily presence of music by Mozart or Anthrax. The data are available at Merrell.dat.
I used Merrell's data, which recorded the times each mouse took to navigate a maze. In the Mozart condition the median was 119 seconds, while in the Anthrax condition the median was 306 seconds. That is a difference of 187 seconds, which is approximately 3 minutes.
I created 5000 randomizations of the 24 + 24 = 48 observations. For each permutation the first 24 scores were assigned to the Mozart condition, and the last 24 observations to the Anthrax condition. I then calculated the median under each condition, and then the median difference. Across 5000 resamples, this gave me the sampling distribution of the difference between two medians, from which I calculated the limits of the confidence interval by what Efron calls the "percentile method." This method takes the lower and upper limits and the 2.5% and 97.5% points of the sampling distribution. As I explain elsewhere, I am not particularly comfortable with this approach, but it is a standard approach and, when the distribution is symmetric, as this one usually is, it produces nearly the same limits as other approaches.
The results of this approach can be seen in the following figure.
As you can see, the 95% confidence interval on the difference in medians is -107 to +109. You would expect the interval to be nearly symmetrical about 0, and it is. You can also see the obtained median difference was 187, which clearly falls outside the interval. Thus we can reject the null hypothesis, and conclude that the median response time of the Anthrax condition is significantly greater than the median response time of the Mozart condition.
Notice that this sampling distribution is noticeably more sparse than the distributions we have seen for means. That is because of the limited possibilities for a median. It must either be one of the values in the original sample, or half way between two original values.
It is useful to compare this result against the one we will find when we use the bootstrapping approach. The two will be visibly different because the approach is different. With bootstrapping, we are able to set the confidence interval on the true difference in medians, whereas with randomization we are forced to set the confidence limits on the difference under the null. From the above figure I can tell you that my 95% confidence interval on the median when the null is true, is -107 to 109. (Actually, that is a pretty strange sounding statement, and I need to think a lot about it.) Because 187 is not included in that interval, we can reject the null. On the other hand, from the figure below I can tell you that the 95% confidence interval on the true difference between medians is 146 to 228. Because this interval does not include 0, we can also reject the null. Notice how different these two approaches are.
Last revised: 06/28/2001