Multiple Comparison Test

Introduction

Hypothesis Test

Hypothesis Test Procedure

• Determining H0 and HA
• Determining its statistical value
  • Confidence level
  • Degrees of freedom
  • Number of samples
• Determines the value of the calculated statistic, this value depends on the parametric method used..
• Making decisions, this is determined by comparing the calculated statistical value with the statistical value of the table or its critical value.

Error Type

1. If we reject the hypothesis (H0 is rejected), while the true hypothesis says we are committing a type I error
2. Conversely if we accept the hypothesis (H0 is accepted) while the hypothesis is wrong, it says we committed a type II error

Level of Significant

1. The probability of making a Type I error is known as the significant level or level of significance and is usually expressed by α.
2. Commonly used values are α = 0.05 and α = 0.01.
3. Example: α = 0.05
   • α = 0.05 is often also called the significant rate of 5% which means roughly 5 out of 100 conclusions that we would reject a hypothesis that should be accepted.
   • This means we may be wrong with a 5% chance or in other words: roughly 95% are sure that we have made the correct conclusion

Analysis of Variance (Anova)

Table of Analysis of Variance for Nitrogen Content

1. Test F (Anova) was used to determine whether there were differences between the treatment means
   • H₀:  m₁ = m₂ = m₃ = ^{.  .  .} = m_k
   • H_A:  one or more of the mean treatments differ from others

   •   

2. When H₀ is accepted, we conclude that all treatment means do not differ
3. If H₀ is rejected (H_A is accepted), we conclude that there are one or more different treatment means from others, but we cannot conclude that all treatment means are different.
4. If H₀ is rejected, which of the mean treatments is different from others??
5. Post hoc test => to find out the mean of which treatments are different and how they are in order

Mean comparison

1. Planned comparison
   • It has been planned before the data (experimental results) are obtained
   • Linear Contrasts (Complex Comparisons)
   • Scheffé's Test
   • Comparison with controls:
     • Dunnet 
     • Bonferroni
     • Sidak

2. Unplanned comparison – post hoc test
   • Pair-wise comparisons or Multiple Comparisons:
     • LSD (not recommended)
     • Tukey HSD (recommended)
     • Scheffé
     • Bonferroni
     • Sidak
     • Gabriel
     • Hochberg
   • Multistage test (Multiple Range Test): comparison of all combinations of mean pairs determined after the results of the experiment were obtained (Post-Hoc Comparison)
     • SNK (Student Newman Keul)
     • Duncan
     • Tukey HSD
     • Tukey B
     • Scheffé
     • Gabriel
     • REGWQ (Ryan, Einot, Gabriel and Welsh.  Q = the studentized range statistic) recommended if supporting software is available)

The number of possible combinations of pairs

In general, the many possible combinations of comparisons for the level of treatment are:

$$\frac{k(k-1)}{2}$$

In the example above, k=3, so that the number of possible combinations:

$$\frac{3(3-1)}{2}=\frac{3(2)}{2}=3$$

Problems in Multiple Comparison

1. Any pairwise comparison at a certain significant level (e.g., a = 5%) will result in a type I error a
2. Example: a= 0.05
   • a = 0.05 is often also called the significant rate of 5% which means roughly 5 out of 100 conclusions that we would reject a hypothesis that should be accepted (type I error);
   • This means we may be wrong with a 5% chance or in other words: roughly 95% are sure that we have made the correct conclusion
   • 
3. When we compare 20 treatments at a significant level of 0.05, where all the null hypotheses are actually significant:
   • 0.95²⁰ = 0.3585 and
   • Type I error = 1-0.3585 = 0.6415
4. In general, the odds of making a type I error on the comparison j are:
   • $\alpha_{EW}=1-(1-\alpha_{PC})^j$
   • It is known as Experiment wise (EW), or Familywise type I error rate.
   • Suppose we want to do 3 treatment comparisons at a significant level: a_PC = 0.05.  The probability to make Type I Mistakes are:
     $\begin{matrix}\alpha_{EW}=1-(1-\alpha_{PC})^j\\=1-(1-0.05)^3\\=1-(0.95)^3\\=1-0.8574\\=0.1426\\\end{matrix}$

General Procedure Comparison of Treatments 

Suppose the mean values of t:  ${\bar{x}}_1,{\bar{x}}_2,\cdots,\bar{x}t$

1. F-test showed significant differences between treatment means
2. Perform an analysis (post hoc test) to determine exactly where there is a difference
3. Two means are expressed differently when the difference is greater than the critical value

Tips in Comparison means

1. Select a pairwise comparison when no treatment will be compared to the control.
2. Use comparison with controls when we are going to compare treatment with controls (Dunnet).

Presentation of Treatment Mean Table

In addition to the methods already mentioned above, there are several ways of presenting differences between treatment means, namely in the form of crosstabulation of mean differences or odds, giving the same line to the mean of different treatments, and in the form of graphs. 

In the difference in the difference in the mean value of the treatment:

Description: if the mean difference is >4.482 mg, then the two treatments are significantly different

In the probability form:

Description: if the probability value < α=0.05, then the two treatments are significantly different

In the notation form:

After the order is returned based on the order of treatment:

Description: the number followed by the same letter in one column does not differ markedly at a significant level of 5% according to the LSD test

Overall Summary Post Hoc Test: