Steps in Hypothesis Testing
Content
Steps in Hypothesis Testing (1 of 5)
The basic logic of hypothesis testing has been presented somewhat informally in the sections on "Ruling out chance as an explanation" and the "Null hypothesis." In this section the logic will be presented in more detail and more formally. 1. The first step in hypothesis testing is to specify the null hypothesis (H0) and the alternative hypothesis (H1). If the research concerns whether one method of presenting pictorial stimuli leads to better recognition than another, the null hypothesis would most likely be that there is no difference between methods (H0: μ1 - μ2 = 0). The alternative hypothesis would be H1: μ1 ≠ μ2. If the research concerned the correlation between grades and SAT scores, the null hypothesis would most likely be that there is no correlation (H0: ρ= 0). The alternative hypothesis would be H1: ρ ≠ 0. 2. The next step is to select a significance level. Typically the 0.05 or the 0.01 level is used. 3. The third step is to calculate a statistic analogous to the parameter specified by the null hypothesis. If the null hypothesis were defined by the parameter μ1- μ2, then the statistic M1 - M2 would be computed. 4. The fourth step is to calculate the probability value (often called the p value). The p value is the probability of obtaining a statistic as different or more different from the parameter specified in the null hypothesis as the statistic computed from the data. The calculations are made assuming that the null hypothesis is true. (click here for a concrete example) 5. The probability value computed in Step 4 is compared with the significance level chosen in Step 2. If the probability is less than or equal to the significance level, then the null hypothesis is rejected; if the probability is greater than the significance level then the null hypothesis is not rejected. When the null hypothesis is rejected, the outcome is said to be "statistically significant" when the null hypothesis is not rejected then the outcome is said be "not statistically significant." 6. If the outcome is statistically significant, then the null hypothesis is rejected in favor of the alternative hypothesis. If the rejected null hypothesis were that μ1- μ2 = 0, then the alternative hypothesis would be that μ1≠ μ2. If M1 were greater than M2 then the researcher would naturally conclude that μ1 ≥ μ2. (Click here to see why you can conclude more than μ1 ≠ μ2) The final step is to describe the result and the statistical conclusion in an understandable way. Be sure to present the descriptive statistics as well as whether the effect was significant or not. For example, a significant difference between a group that received a drug and a control group might be described as follow: Subjects in the drug group scored significantly higher (M = 23) than did subjects in the control group (M = 17), t(18) = 2.4, p = 0.027.
The statement that "t(18) =2.4" has to do with how the probability value (p) was calculated. A small minority of researchers might object to two aspects of this wording. First, some believe that the significance level rather than the probability level should be reported. The argument for reporting the probability value is presented in another section. Second, since the alternative hypothesis was stated as µ1 ≠ µ2, some might argue that it can only be concluded that the population means differ and not that the population mean for the drug group is higher than the population mean for the control group. This argument is misguided. Intuitively, there are strong reasons for inferring that the direction of the difference in the population is the same as the difference in the sample. There is also a more formal argument. A non significant effect might be described as follows: Although subjects in the drug group scored higher (M = 23) than did subjects in the control group, (M = 20), the difference between means was not significant, t(18) = 1.4, p = 0.179. It would not have been correct to say that there was no difference between the performance of the two groups. There was a difference. It is just that the difference was not large enough to rule out chance as an explanation of the difference. It would also have been incorrect to imply that there is no difference in the population. Be sure not to accept the null hypothesis. At this point you may wish to see a concrete example of using these seven steps in hypothesis testing. If so, jump to the section on "Tests of μ, σ known."
Why the Null Hypothesis is Not Accepted (1 of 5)
A null hypothesis is not accepted just because it is not rejected. Data not sufficient to show convincingly that a difference between means is not zero do not prove that the difference is zero. Such data may even suggest that the null hypothesis is false but not be strong enough to make a convincing case that the null hypothesis is false. For example, if the probability value were 0.15, then one would not be ready to present one's case that the null hypothesis is false to the (properly) skeptical scientific community. More convincing data would be needed to do that. However, there would be no basis to conclude that the null hypothesis is true. It may or may not be true, there just is not strong enough evidence to reject it. Not even in cases where there is no evidence that the null hypothesis is false is it valid to conclude the null hypothesis is true. If the null hypothesis is that µ1 - µ2 is zero then the
hypothesis is that the difference is exactly zero. No experiment can distinguish between the case of no difference between means and an extremely small difference between means. If data are consistent with the null hypothesis, they are also consistent with other similar hypotheses. Thus, if the data do not provide a basis for rejecting the null hypothesis that µ1- µ2 = 0 then they almost certainly will not provide a basis for rejecting the hypothesis that µ1- µ2 = 0.001. The data are consistent with both hypotheses. When the null hypothesis is not rejected then it is legitimate to conclude that the data are consistent with the null hypothesis. It is not legitimate to conclude that the data support the acceptance of the null hypothesis since the data are consistent with other hypotheses as well. In some respects, rejecting the null hypothesis is comparable to a jury finding a defendant guilty. In both cases, the evidence is convincing beyond a reasonable doubt. Failing to reject the null hypothesis is comparable to a finding of not guilty. The defendant is not declared innocent. There is just not enough evidence to be convincing beyond a reasonable doubt. In the judicial system, a decision has to be made and the defendant is set free. In science, no decision has to be made immediately. More experiments are conducted. One experiment might provide data sufficient to reject the null hypothesis, although no experiment can demonstrate that the null hypothesis is true. Where does this leave the researcher who wishes to argue that a variable does not have an effect? If the null hypothesis cannot be accepted, even in principle, then what type of statistical evidence can be used to support the hypothesis that a variable does not have an effect. The answer lies in relaxing the claim a little and arguing not that a variable has no effect whatsoever but that it has, at most, a negligible effect. This can be done by constructing a confidence interval around the parameter value. Consider a researcher interested in the possible effectiveness of a new psychotherapeutic drug. The researcher conducted an experiment comparing a drug-treatment group to a control group and found no significant difference between them. Although the experimenter cannot claim the drug has no effect, he or she can estimate the size of the effect using a confidence interval. If µ1 were the population mean for the drug group and µ2 were the population mean for the control group, then the confidence interval would be on the parameter µ1 - µ2. Assume the experiment measured "well being" on a 50 point scale (with higher scores representing more well being) that has a standard deviation of 10. Further assume the 99% confidence interval computed from the experimental data was: -0.5 ≤ µ1- µ2 ≤ 1 This says that one can be confident that the mean "true" drug treatment effect is somewhere between -0.5 and 1. If it were -0.5 then the drug would, on average, be slightly detrimental; if it were 1 then the drug would, on average, be slightly
beneficial. But, how much benefit is an average improvement of 1? Naturally that is a question that involves characteristics of the measurement scale. But, since 1 is only 0.10 standard deviations, it can be presumed to be a small effect. The overlap between two distributions whose means differ by 0.10 standard deviations is shown below. Although the blue distribution is
slightly to the right of the red distribution, the overlap is almost complete. So, the finding that the maximum difference that can be expected (based on a 99% confidence interval) is itself a very small difference would allow the experimenter to conclude that the drug is not effective. The claim would not be that it is totally ineffective, but, at most, its effectiveness is very limited. See also: pages 2-3 of "Confidence intervals & hypothesis testing"
The basic logic of hypothesis testing has been presented somewhat informally in the sections on "Ruling out chance as an explanation" and the "Null hypothesis." In this section the logic will be presented in more detail and more formally. 1. The first step in hypothesis testing is to specify the null hypothesis (H0) and the alternative hypothesis (H1). If the research concerns whether one method of presenting pictorial stimuli leads to better recognition than another, the null hypothesis would most likely be that there is no difference between methods (H0: μ1 - μ2 = 0). The alternative hypothesis would be H1: μ1 ≠ μ2. If the research concerned the correlation between grades and SAT scores, the null hypothesis would most likely be that there is no correlation (H0: ρ= 0). The alternative hypothesis would be H1: ρ ≠ 0. 2. The next step is to select a significance level. Typically the 0.05 or the 0.01 level is used. 3. The third step is to calculate a statistic analogous to the parameter specified by the null hypothesis. If the null hypothesis were defined by the parameter μ1- μ2, then the statistic M1 - M2 would be computed. 4. The fourth step is to calculate the probability value (often called the p value). The p value is the probability of obtaining a statistic as different or more different from the parameter specified in the null hypothesis as the statistic computed from the data. The calculations are made assuming that the null hypothesis is true. (click here for a concrete example) 5. The probability value computed in Step 4 is compared with the significance level chosen in Step 2. If the probability is less than or equal to the significance level, then the null hypothesis is rejected; if the probability is greater than the significance level then the null hypothesis is not rejected. When the null hypothesis is rejected, the outcome is said to be "statistically significant" when the null hypothesis is not rejected then the outcome is said be "not statistically significant." 6. If the outcome is statistically significant, then the null hypothesis is rejected in favor of the alternative hypothesis. If the rejected null hypothesis were that μ1- μ2 = 0, then the alternative hypothesis would be that μ1≠ μ2. If M1 were greater than M2 then the researcher would naturally conclude that μ1 ≥ μ2. (Click here to see why you can conclude more than μ1 ≠ μ2) The final step is to describe the result and the statistical conclusion in an understandable way. Be sure to present the descriptive statistics as well as whether the effect was significant or not. For example, a significant difference between a group that received a drug and a control group might be described as follow: Subjects in the drug group scored significantly higher (M = 23) than did subjects in the control group (M = 17), t(18) = 2.4, p = 0.027.
The statement that "t(18) =2.4" has to do with how the probability value (p) was calculated. A small minority of researchers might object to two aspects of this wording. First, some believe that the significance level rather than the probability level should be reported. The argument for reporting the probability value is presented in another section. Second, since the alternative hypothesis was stated as µ1 ≠ µ2, some might argue that it can only be concluded that the population means differ and not that the population mean for the drug group is higher than the population mean for the control group. This argument is misguided. Intuitively, there are strong reasons for inferring that the direction of the difference in the population is the same as the difference in the sample. There is also a more formal argument. A non significant effect might be described as follows: Although subjects in the drug group scored higher (M = 23) than did subjects in the control group, (M = 20), the difference between means was not significant, t(18) = 1.4, p = 0.179. It would not have been correct to say that there was no difference between the performance of the two groups. There was a difference. It is just that the difference was not large enough to rule out chance as an explanation of the difference. It would also have been incorrect to imply that there is no difference in the population. Be sure not to accept the null hypothesis. At this point you may wish to see a concrete example of using these seven steps in hypothesis testing. If so, jump to the section on "Tests of μ, σ known."
Why the Null Hypothesis is Not Accepted (1 of 5)
A null hypothesis is not accepted just because it is not rejected. Data not sufficient to show convincingly that a difference between means is not zero do not prove that the difference is zero. Such data may even suggest that the null hypothesis is false but not be strong enough to make a convincing case that the null hypothesis is false. For example, if the probability value were 0.15, then one would not be ready to present one's case that the null hypothesis is false to the (properly) skeptical scientific community. More convincing data would be needed to do that. However, there would be no basis to conclude that the null hypothesis is true. It may or may not be true, there just is not strong enough evidence to reject it. Not even in cases where there is no evidence that the null hypothesis is false is it valid to conclude the null hypothesis is true. If the null hypothesis is that µ1 - µ2 is zero then the
hypothesis is that the difference is exactly zero. No experiment can distinguish between the case of no difference between means and an extremely small difference between means. If data are consistent with the null hypothesis, they are also consistent with other similar hypotheses. Thus, if the data do not provide a basis for rejecting the null hypothesis that µ1- µ2 = 0 then they almost certainly will not provide a basis for rejecting the hypothesis that µ1- µ2 = 0.001. The data are consistent with both hypotheses. When the null hypothesis is not rejected then it is legitimate to conclude that the data are consistent with the null hypothesis. It is not legitimate to conclude that the data support the acceptance of the null hypothesis since the data are consistent with other hypotheses as well. In some respects, rejecting the null hypothesis is comparable to a jury finding a defendant guilty. In both cases, the evidence is convincing beyond a reasonable doubt. Failing to reject the null hypothesis is comparable to a finding of not guilty. The defendant is not declared innocent. There is just not enough evidence to be convincing beyond a reasonable doubt. In the judicial system, a decision has to be made and the defendant is set free. In science, no decision has to be made immediately. More experiments are conducted. One experiment might provide data sufficient to reject the null hypothesis, although no experiment can demonstrate that the null hypothesis is true. Where does this leave the researcher who wishes to argue that a variable does not have an effect? If the null hypothesis cannot be accepted, even in principle, then what type of statistical evidence can be used to support the hypothesis that a variable does not have an effect. The answer lies in relaxing the claim a little and arguing not that a variable has no effect whatsoever but that it has, at most, a negligible effect. This can be done by constructing a confidence interval around the parameter value. Consider a researcher interested in the possible effectiveness of a new psychotherapeutic drug. The researcher conducted an experiment comparing a drug-treatment group to a control group and found no significant difference between them. Although the experimenter cannot claim the drug has no effect, he or she can estimate the size of the effect using a confidence interval. If µ1 were the population mean for the drug group and µ2 were the population mean for the control group, then the confidence interval would be on the parameter µ1 - µ2. Assume the experiment measured "well being" on a 50 point scale (with higher scores representing more well being) that has a standard deviation of 10. Further assume the 99% confidence interval computed from the experimental data was: -0.5 ≤ µ1- µ2 ≤ 1 This says that one can be confident that the mean "true" drug treatment effect is somewhere between -0.5 and 1. If it were -0.5 then the drug would, on average, be slightly detrimental; if it were 1 then the drug would, on average, be slightly
beneficial. But, how much benefit is an average improvement of 1? Naturally that is a question that involves characteristics of the measurement scale. But, since 1 is only 0.10 standard deviations, it can be presumed to be a small effect. The overlap between two distributions whose means differ by 0.10 standard deviations is shown below. Although the blue distribution is
slightly to the right of the red distribution, the overlap is almost complete. So, the finding that the maximum difference that can be expected (based on a 99% confidence interval) is itself a very small difference would allow the experimenter to conclude that the drug is not effective. The claim would not be that it is totally ineffective, but, at most, its effectiveness is very limited. See also: pages 2-3 of "Confidence intervals & hypothesis testing"
