Controlling Error in Multiple Comparisons, with Special Attention to the National Assessment of Educational Progress (1994)

Abstract:

Adjustment procedures for multiplicity are investigated, including the traditional Bonferroni technique, a sequential Bonferroni technique developed by Hochberg (1988), and a sequential approach proposed by Benjamini and Hochberg (in press) for controlling the false discovery rate (FDR). The procedures are illustrated and compared, based on examples from several data sets, including the National Assessment of Educational Progress (NAEP) and the NAEP Trial State Assessment. One advantage of the FDR procedure is shown to be its consistency about the statistical significance of comparisons over alternative choices of family size. Simulation studies show that all three procedures maintain a false discovery rate bounded above by alpha (or alpha / 2). For both uncorrelated and pairwise families of comparisons, the FDR technique results in greater power than that for the Hochberg or Bonferroni procedures and the power advantage of the FDR procedure increases with the number of comparisons. We recommend that in reporting results from the Trial State Assessment, NAEP discontinue use of the Bonferroni procedure in favor of the FDR technique.