I was asked in class about the "correct" way to report statistical values and their probabilities under the null. There is no one correct way to do so, but I have included below the relevant section out of the APA Publication Manual (4th edition, pp 17-18). (While I was at it I included a short section on Effect Size. That is all that the manual says about effect size, but it is important that you look at the paper by Wilkinson et al. (1999) for what they say on this topic.).
Statistical significance. Two types of probabilities associated with the significance of inferential statistical tests are reported. One refers to the a priori probability you have selected as an acceptable level of falsely rejecting a given null hypothesis. This probability, called the alpha level, is the probability of a Type I error in hypothesis testing. Commonly used alpha levels are .05 and .01 Before you begin to report specific results, you should routinely state the particular alpha level you selected for the statistical tests you conducted:An alpha level of .05 was used for all statistical tests.
If you do not make a general statement about the alpha level, specify the alpha level when reporting each result.
The other kind of probability refers to the a posteriori likelihood of obtaining a result that is as extreme as or more extreme than the actual value of the statistic you obtained, assuming that the null hypothesis is true. For example, given a true null hypothesis, the probability of obtaining the particular value of the statistic you computed might be .008. Many statistical packages now provide these exact values. You can report this distinct piece of information in addition to specifying whether you rejected or failed to reject the null hypothesis using the specified alpha level.
With an alpha level of .05, the effect of age was statistically significant, F(1, 123) = 7.27, p = .008.
or
The effect of age was not statistically significant, F(1, 123) = 2.45, p = .12.
The second example should be used only if you have included a general statement about the alpha level earlier in your article.
If you do not wish to report the exact probability, you can report the commonly used probability value that is nearest to it:
With an alpha level of .05, the effect of age was statistically significant, F(1, 123) = 7.27, p < .01.
or
The effect of age was not statistically significant, F(1, 123) = 2.45, p > .10.
Effect size and strength of relationship. Neither of the two types of probability values reflects the importance (magnitude) of an effect or the strength of a relationship because both probability values depend on sample size. You can estimate the magnitude of the effect or the strength of the relationship with a number of measures that do not depend on sample size. Common measures are r2, h2, w2, R2, f2 Cramer's V, Kendall's W Cohen's d and k, Goodman and Kruskal's l and g, Jacobson and Truax's (1991) proposed measures of clinical significance, and the multivariate Roy's q and the Pillai-Bartlett V.
You are encouraged to provide effect-size information, although in most cases such measures are readily obtainable whenever the test statistics (e.g., t and F) and sample sizes (or degrees of freedom) are reported. For example, given an F ratio based on u1 and u2 degrees of freedom, the proportion of variance accounted for by the associated effect (h2, as the generalization of r2) can be determined as u1F/( u1F+ u2 ).
Last revised: 01/28/02