Analysis of Variance

Introduction to ANOVA

The ANalysis Of VAriance test (ANOVA) is commonly used to determine the statistical significance of observed differences in mean values between three or more independent groups. The presence of a statistically significant difference after calculating an ANOVA suggests that the independent variable(s) in a study influenced the overall variance between at least two of the study groups. Before using an ANOVA, three assumptions must be satisfied in a data set:

1. The dependent data in each study group is normally distributed on a Gaussian curve.^1,2
2. The intrinsic variance within each study group is the same.^1,2
3. The observations measured within study groups are independent of one another.^1,2

If the above assumptions are met or controlled for, the results of any statistically significant findings in a data set analyzed with an ANOVA can be considered valid and worthy of further post hoc testing.^1,2

An ANOVA determines statistical significance by calculating the ratio of the variance present between group means and the variance intrinsically present within the groups, which is defined as the F statistic. When coupled with P value significance thresholds (commonly defined as P ≤ 0.05), a smaller F statistic (equal or close to 1.0) supports the null hypothesis. In comparison, a larger F statistic (greater than 1.0) rejects the null hypothesis.^1,2

Importantly, ANOVA is classified as an omnibus test, meaning it can only confirm the validity of the overall study hypothesis, not the specific statistical relationships of the independent data groups. In other words, ANOVA can determine that a significant difference exists between at least two of the group means but cannot distinguish which specific group means differ significantly from one another. This limitation necessitates the use of post hoc (after the event) tests to further interpret the data generated from an ANOVA into a meaningful conclusion.^2,3

There are multiple iterations of the ANOVA test, each with its own applications to medical research. The choice of ANOVA test for a particular data set depends on the nature of the group means that are to be compared, which includes but is not limited to the distinct number of categorical independent variables analyzed, the presence of interaction effects from multiple independent variables, the presence of confounding variables, and the frequency of data sampling (continuous vs. discontinuous). Four of the most popular ANOVA tests are discussed in more detail in the following sections.^2,3

1. One-Way ANOVA

This is the simplest form of the ANOVA test. The purpose of a one-way ANOVA is to determine the statistical significance of observed differences in mean values between three or more study groups in relation to one independent variable. A One-Way ANOVA cannot analyze a data set for interaction effects because the test assumes only one independent variable is present.¹

2. Two-Way ANOVA

The purpose of a two-way ANOVA is to determine the statistical significance of observed differences in mean values between three or more study groups in relation to two independent variables. Notably, a two-way ANOVA delineates whether the distinct effects that each of the independent variables have on the variance of the group means is significant between at least two groups. It also can determine whether the interaction effect that the combination of the two independent variables has on the variance of the group means is statistically significant.³

3. Analysis of Covariance (ANCOVA)

The purpose of an ANCOVA is to determine the statistical significance of observed differences in mean values between three or more study groups while simultaneously controlling for potential error variance introduced into the data set means by continuous, confounding variables known as covariates. These covariates affect the observed differences in data set means, skewing the intended effects of the independent variables being investigated. By controlling for confounding variance from covariates, ANCOVA increases the statistical power and reduces error variance within a study.^2,3

4. Repeated Measures ANOVA

The purpose of repeated measures ANOVA is to determine the statistical significance of observed differences in mean values between three or more groups of serial observations or measurements taken from the same group of subjects. The repeated measures ANOVA resolves the breach of assumption three for ANOVA tests because it controls for the correlation present between measurements in non-independent subjects.³

Interpretation of ANOVA

The following section will summarize the general interpretation of ANOVA test results and provide some examples of post hoc tests to consider when further analyzing data sets based on promising ANOVA results.

One-Way ANOVA

• Informs a researcher that a significant difference in outcome means from a single independent variable was or was not present between at least two independent groups in a study that consisted of three or more independent groups.¹

Two-Way ANOVA

• Informs a researcher that a significant difference in outcome means from two independent variables was or was not present between at least two independent groups in a study that consisted of three or more independent groups.³
• Interaction effects between the two independent variables that may influence the outcome can add more depth to the final conclusions of a study.²

Analysis of Covariance (ANCOVA)

• Informs a researcher that a significant difference in outcome means from a single independent variable that would otherwise have been skewed by recognized covariates was or was not present between at least two independent groups in a study that consisted of three or more independent groups.^2,3
• Control of covariates allows for error reduction and increased statistical power, which increases researcher confidence in ANOVA results and guides future post hoc testing of study results.^2,3

Repeated Measures ANOVA

• Informs a researcher that a significant difference in outcome means from a single independent variable was or was not present between at least two non-independent measurement groups in a study that consisted of three or more non-independent groups of measurement taken from the same patient cohort over time.³
• Correlation effects from non-independent measurements normally create unacceptable levels of error variance in an ANOVA but are controlled for with a repeated measures ANOVA. The effect of the independent variable on the outcome over time can then be interpreted with greater confidence.³

Post hoc tests

• Post hoc tests are used to distinguish which specific group means differ from one another in a data set of three or more groups that were previously determined to contain a significant difference via an ANOVA test.^2,4
• The correct choice of test is at the researcher’s discretion and depends largely on the specific characteristics of the data set to be analyzed.⁴
• Some common post hoc tests include Tukey’s HSD (Honestly Significant Difference), Scheffe’s procedure, Bonferroni’s procedure, Newman-Keuls procedure, Duncan’s procedure, Dunnett’s test, Hsu’s multiple comparisons with the best (MCB) test.⁴

Application of ANOVA

The following section will demonstrate example study designs that could benefit from using one of the four ANOVA tests previously introduced.

Study 1

A group of researchers are investigating the effect of a new anesthetic drug on blood pressure. The study enrolls three independent groups of patients. The first group will receive a low dose of the new drug, the second group will receive a medium dose, and the third group will receive a high dose. Each patient’s blood pressure is measured continuously after the intravenous administration of the drug. Each group is determined to have equal variance among the patients, the measured data is normally distributed, and all observations are independent of one another. Which ANOVA is most appropriate to apply?

In Study 1, a one-way ANOVA would be most appropriate because the study design uses one independent variable (the drug dosage) and measures its effect on a defined dependent variable (blood pressure). The study also allocated the patients into three independent groups (low dose, medium dose, and high dose).³ A one-way ANOVA would determine if a statistically significant difference in mean blood pressure is present between at least two of these three patient groups.³

Study 2

A group of researchers are investigating the effect of a new anesthetic drug on blood pressure. The researchers are also interested in determining if the patient’s gender influences blood pressure fluctuations caused by the new drug. The study enrolls three independent groups of patients, with equal representation of males and females in each group. The first group will receive a low dose of the new drug, the second group will receive a medium dose, and the third group will receive a high dose. Each patient’s blood pressure is measured continuously after the intravenous administration of the drug. Each group is determined to have equal variance among the subjects, the measured data is normally distributed, and all observations are independent of one another. Which ANOVA is most appropriate to apply?

In study 2, a two-way ANOVA would be most appropriate because the study design incorporates two independent variables (the drug dosage and patient gender) and measures their separate and combined effects on a defined dependent variable (blood pressure). The study also allocated the patients into three independent groups (low dose, medium dose, and high dose).³ A two-way ANOVA would determine if a statistically significant difference in mean blood pressure is present between at least two of these three patient groups due to the variance caused by patient gender, the dose of the anesthetic administered, and/or the interaction effect that patient gender and the drug may have with one another.³

Study 3

A residency program is investigating the effect of a new training method on resident performance during a particular procedure. The residency program chooses to test the new training method on first-year, second-year, and third-year residents within the program. Each resident group is graded on their performance by attending preceptors before applying the training method and then reassessed in the future after undergoing the new training. Each group is determined to have equal variance among the subjects, the measured data is normally distributed, and all observations were independent of one another. Which ANOVA is most appropriate to apply?

In study 3, an ANCOVA would be most appropriate because the study design incorporates one independent variable (the new training method), controls for a covariate through pretesting (potential prior knowledge about the procedure), and measures a defined dependent variable (resident performance). The study also allocated the residents into three independent groups (first-year, second-year, and third-year).^2,3

An ANCOVA would determine if a statistically significant difference in mean resident performance exists between at least two of these three resident groups due to the variance caused by the new training method. The confounding effect of the continuous covariate of prior resident knowledge can be controlled for within the ANCOVA via measurements taken from the performance pre-test.^2,3

Study 4

A group of researchers are investigating the effect of a new analgesic drug on pain relief over time. The researchers measured the self-reported pain scale from 1 to 10 of one cohort of patients with chronic, refractory low back pain at three different time points: before treatment, one month after beginning treatment, and six months after beginning treatment. The patients were determined to have equal variance among them, and the measured data was normally distributed. Still, the observations were not independent of one another due to multiple observations being recorded from the same patients throughout the study. Which ANOVA is most appropriate to apply?

In study 4, a repeated measures ANOVA would be most appropriate because the study design uses one independent variable (the new drug) and measures its effect on a defined dependent variable (reported pain on a scale of 1 to 10). Although the study only included one group of patients, it was designed to investigate three independent time points of data (before treatment, one month after treatment, and six months after treatment).³

A repeated measures ANOVA would determine if a statistically significant difference in mean pain relief over time is present between at least two of the three timepoints analyzed in study 4, because this test controls for error variance that stems from correlation effects present in non-independent data.³