Post-hoc reasoning on two-ways

Outline

-- Comparing two examples
-- Pair-wise comparisons
-- no effects
-- just main effects
2 levels
3 or more levels
-- interactions
2 x 2 designs
more complex designs

1 -- plot the cell means and make predictions (get a feel for your data)

2 -- compute the ANOVA (do the math) if ANOVA says not significant it does not matter that it looks like it is in the graph

3 -- Interpret (follow-up comparisons)

a. if MEs only, then do comparisons on marginal means
b. if Interaction only, find where it is
c. If MEs and Interaction, find interaction and double check your MEs (make sure that they are consistent)

Considering two examples

a 2x2 Example

 . Menu Options . Rest. Type Mild Hot . Dive M1,1 = 3.00 N1,1 = 5 M2,1= 7.00 N2,1 = 5 MB=1= 5.00 Fancy M1,2 = 3.00 N1,2 = 5 M2,2 = 5.00 N2,2 = 5 MB=2= 4.00 . MA=1= 3.00 MA=2 = 6.00 MT = 4.50
 Source SS df MS F p b/t factors . . . . . Menu Options 45.00 1 45.00 90.00 <.001 Rest Type 5.00 1 5.00 10.00 <.01 interaction . . . . . Mx R 5.00 1 5.00 10.00 <.01 error 8.00 16 0.50 . . total 63.00 19 . . .

In this case, the math (the ANOVA) told us that we had two main effects and an interaction. The graph, however, says that the main effect of restaurant type is an artifact of the interaction, since it is not consistent.

a 3x2 Example

 . . Menu Options . . Rest. Style Mild Medium Hot . Dive M1,1 = 4.00 N1,1 = 6 M2,1= 7.00 N2,1 = 6 M3,1= 8.00 N3,1 = 6 MB=1= 6.33 Fancy M1,2 = 3.00 N1,2 = 6 M2,2 = 3.00 N2,2 = 6 M3,2 = 5.00 N3,2 = 6 MB=2= 3.67 . MA=1= 3.50 MA=2 = 5.00 MA=3 = 6.50 MT = 5.00
 Source SS df MS F p b/t factors . . . . . Menu Options 54.00 2 27.00 16.875 <.001 Rest Type 64.00 1 64.00 40.000 <.001 interaction . . . . . Mx R 14.00 2 7.00 4.375 <.025 error 48.00 30 1.60 . . total 180.00 35 . . .

There are some fundamental guidelines in interpretting 2-ways

1. regardless of your graph says, must be statistically significant

2. must determine where main effects lie

3. if you have interactions, you must describe them -- that is, how the effects of one IV depend on the level of the other IV

4. must interpret main effects when interactions occur (are they true MEs, that is do they show up consistently)

pair-wise comparisons

When we are looking at an ANOVA we already know what it doesn't tell us -- direction! So there is some need to do post-hoc comparisons when you are finished. There are three general possible situations: 1. no effects at all; 2. just main effects; and 3. interactions (with or without main effects).

No Effects:

No follow-ups needed. There is no effect to find.

Just main effects :

The simplest case is when all you need to do is deal with main effects. In this case you can act as if you have two separate experiments that are each one-way designs. Consider the two examples if you had not had the interaction. There are two possible situations based on the number of levels for you IVs.

2 levels

Since there are only 2 levels, in this case you know direction, even though you are dealing with an ANOVA. The difference is between the 2 levels and whichever one has the higher Mean is the one with the higher Mean.  The Means you attend to here are the marginal means, not the cell means.  Thus in a 2x2 with just Main Effects -- you know direction and you are done!!! In a 3x2, if there are only main effects, for the variable with 2 levels, there is no need for follow-ups and you are done with that IV (but maybe not the other IV).

3 or more levels

The interpretation is just like a multilevel one-way ANOVA: you must do post-hoc analyses. You know there are differences among the levels, but since there are multiple levels, you don't know where exactly. I, again, recommend Tukey's. In this case, treating it like a one-way design, you work with the marginal means and marginal Ns. In this case:

CD = q  [sqrt (MSE / Na)]

MSE = experiment MSE

Na = marginal N

q is based on number of A levels, dfe, and experiment-wide p

@ .05 level CD = 3.49 [sqrt (1.60/12)] = 1.274

Compare marginal means:

 . . Menu Options . Menu Options Mild 3.50 Medium 5.00 Hot 6.50 Mild - Medium 1.50* - Hot 3.00* 1.50* -

* p < .05 (because difference between the Means exceeded the CD)

Interactions

With interactions all bets are off. You have one primary goal: describe the interaction. You have to describe how the effects of one IV depend on the level of the other IV. You will have a second goal if your ANOVA found a significant main effect: check that the main effect is consistent (if it is not consistent, then it is not a true main effect; it is an artifact of the interaction).

2 x 2 designs

Well, this isn't so bad, because you can usually do things visually by interpretting the graph. Take our first example -- could see that it wasn't the case that rest. mattered consistently. The difference between the dive and the fancy restaurant only showed up for hot food. Thus the effect of rest. type depended on the menu option. This also means that the effect was not consistent and was thus an artifact.

You can also describe the interaction in terms of the other IV: Although menu option mattered in both dive and fancy rest type, the effect was larger in the dive than in the fancy restaurant. The size of the effect was determined by the level of the other IV.

more complex designs

Here there is a bit of work involved in interpretation. Now you have to look for simple effects. These are cell to cell comparisons. They are the effects of one IV at just one level of the other IV.  Let's look at our second example in which the interaction was significant. This was the 3x2 design. We would want to know about the effects of A at each level of B. (Was there an effect of menu option at each level of rest. type?) We would want to know if there was an effect of B at each level of A. (This means, was there an effect of rest type at each level of menu option?) I recommend a simple expansion of what we did with a one way -- using Tukey's again, but this time comparing cell, rather than marginal, means.

CD = q [sqrt (MSE / Nab)]

MSe = experiment MSe
Nab = cell N
q is based number of means being compared which is based on the number of AxB levels, dfe, and level of p
2x2 is 3
2x3 or 3x2 is 5
2x4 or 4x2 is 7
3x3 is 7

@.05 level CD = 4.10 [sqrt (1.60 / 6)] = 2.117

First deal with the simple effects of A at each level of B: the Simple Effect of Menu Options at each level of Rest. Type: Are there differences among the levels of menu options in the Dive level of rest.type? and Are there differences among the levels of menu options in the Fancy level of rest.type?

Rest Type = Dive

 . . Menu Options . Menu Options Mild 4.00 Medium 7.00 Hot 8.00 Mild - Medium 3.00* - Hot 4.00* 1.00 -

* p < .05 (because difference between the Means exceeded the CD)

Rest Type = Fancy

 . . Menu Options . Menu Options Mild 3.00 Medium 3.00 Hot 5.00 Mild - Medium 0.00 - Hot 2.00 2.00 -

* p < .05 (because difference between the Means exceeded the CD)

Now deal with the simple effects of B at each level of A: the Simple Effect of Rest. Type at each level of Menu Options: Are there differences among the levels of rest type for the mild level of menu options? and Are there differences among the levels of rest type for the medium level of menu options? and Are there differences among the levels of rest type for the hotlevel of menu options?

 . Rest Type Rest Type Dive 4.00 Fancy 3.00 Dive - Fancy 1.00 -

* p < .05 (because difference between the Means exceeded the CD)

 . Rest Type Rest Type Dive 7.00 Fancy 3.00 Dive - Fancy 4.00* -

* p < .05 (because difference between the Means exceeded the CD)

 . Rest Type Rest Type Dive 8.00 Fancy 5.00 Dive - Fancy 3.00* -

* p < .05 (because difference between the Means exceeded the CD)

Now you can describe the interaction:

In the dive, mild is less spicy than medium and hot; while in the fancy rest, there are no differences among the levels of menu options: thus the effects of menu options depends on the level of rest.type.

For mild food, there is not a difference between the dive and the fancy rest.; while the food in the dive is rated spicier than the food in the fancy rest. for medium and hot food: thus the effect of rest. depends on the level of menu options.

You can also check that you main effects are consistent:

In this case, neither shows up consistently across every level of the other IV.