Lecture 1:  Experimental Design

Outline
Basic Terms
-- population
-- sample
-- subject
-- independent variable
-- dependent variable

Designs
-- true experimental variables 1:  between-subjects
basic 2 group between-subjects
multiple levels
pretest/posttest designs
-- true experimental variables 2:  within-subjects
basic 2 group within-subjects
multiple levels
-- quasi-experimental variables:  cross-sectional or status variables
basic 2 group within-subjects
multiple levels
-- factorial designs
correlational designs

Control
-- confounds
-- limiting noise
-- control groups
-- placebos

Basic Terms

Population
All members of a group
Group can be variously defined

Sample
A subset of a population
Unbiased sample vs. biased sample
Random sample:
All members of the population have equal chance of being selected
Generalizing from sample to the population

Subject
A single member of the sample
Human subjects are often called participants

Independent Variable (IV)
Treatment or Factor
The manipulated variable in an experiment
IV has various levels that different groups/conditions experience
Theory claims IV is causal

Dependent Variable
Outcome variable
Measured in experiment
Theory claims IV affects DV

Designs
Today we are going to worry about a basic step in designing and controlling your observations:  experimental design.

OBS --> Theory --> Hypothesis --> OBS

I want to start by explaining something about this design process:
Everything is determined by theories.
· Theories tell you what IVs should affect the variable of interest
· You can then make precise hypotheses concerning what levels of the IVs should lead to high and low outcomes on the DV
· You then design an experiment (observation) that causes those levels, controls other possible confounds, and measures the outcome
· Thus you start with theories in a given domain.

One other point:  the experimental design and the scale of measurement determine your statistics:  so once you know your question, everything else follows.
Thus start thinking about what questions are worth asking.  Do not do it the other way around, say I want to do this stat so what design, what question.

True experiments:  between-subjects variables
True experiments are where you manipulate the Independent Variable and you control all other factors that may influence the Dependent, or outcome, Variable.  In this case you can really answer a question; you can know if changes in the IV cause changes in the DV.
If you want to know the differences between X and Y, you set things up such that X happens to one group and Y happens to another.  You then measure outcomes.  If you want to know the difference between X, Y, and Z, you set things up so that X, Y, and Z happen to different groups and you measure outcomes.

Basic 2 group between-subjects true experiment
Here is the basic design

 IV DV Pop    --> Sample    --> A Level 1 measure Compare groups B Level 2 measure

Keep in mind that you start with a question and you then figure out which design will allow you to investigate that question.

Imagine that you wanted to investigate the decay theory of forgetting.  Say, for example, you were interested in how time since learning affects memory.  Then IV would be time; level 1 might be 0 (immediate recall) and level 2 might be 1 minute.  DV would be determined by how you operationally defined memory.  Say number of words from a word list.

Multiple levels between-subjects true experiment

Say if you had different ideas about how forgetting works.  If you thought interference was important and you wanted to know if visual information interferes with verbal information, but you realize that just time itself does allow for some decay in memory.

 IV DV A Level 1 measure Pop   ---> Sample    --> B Level 2 measure Compare groups C Level 3 measure

IV:  Type of Interference
Levels: 1 = verbal inference; 2 = visual interference, 3 = no interference (control)

Pretest/posttest designs
This is another way to control for individual differences:  measure each subject before and each after IV, then the DV is the change.  Say effects of different types instruction but you know some people know a lot before the course and some know less.

 Pretest DV IV Posttest DV Pop   ---> Sample    --> A measure Level 1 measure compare changes B measure Level 2 measure

True experiments, II:  Within-subjects designs
Sometimes it is possible to conduct within-subjects designs:  This, like a pretest-posttest design, eliminates the problem of individual differences.
In this case, each subject acts as his/her own control and experiences all conditions.  So for time study each subject would get one word list at each delay:
You can do this with multiple levels as well, each subject experiences every level of IV and gets measured on the DV after each.

Controls for Individual Differences
Powerful statistically
Uses fewer subjects

Design is transparent to subjects
May lead to practice effects

Quasi-experimental designs:  Cross-sectional or status variables
What if we want to investigate the differences between two different types of people.  Say you are still concerned about learning little words on a list, but what about the difference between people with large and small vocabularies.  You have reason to believe that people with more knowledge of words will be better able to process and store words in a list and thus should recall more of the a word list.
Now in this case, we can’t randomly assign some people to have large vocabularies and some to have small.  Instead you are drawing samples from two separate populations

 IV DV Pop 1    --> Sample  1  --> measure Compare groups Pop 2    --> Sample  2  --> measure

IV:  Vocab size
Levels:  1= large vocab; 2 = small vocab,
Both experience word list and are asked to recall it.
Of course, you can have multiple levels here just like in true experiments.

True Experimental Variables vs. Quasi-Experimental Variables
Difference is in your ability to infer
Can infer that IV is related to DV, but what about the IV?
Why are people that way?
Is there something causing them to have large vocabs and do well on word list tasks?
Cannot determine causality!

What factor (IV) you are interested in determines whether or not you can manipulate it and thus whether you can conduct a true experiment.

Factorial designs
What if you think two IVs affect the DV in question?

Say for forgetting, you want to look at both time and interference.

IV1: Time
1= 1 minute
2= 2 minute
IV2:  Type of Interference
1= verbal
2 = visual

Draw as a box

 IV 2 IV 1 - Time L1 - Verbal L2 - Visual L1 - 1 minute 1m of verbal 1m of visual L2 - 2 minutes 2m of verbal 2m of visual

Note: In a factorial design the IVs can be between-subjects, within-subjects, or cross-sectional.  Can have lots of IVs

Correlational designs
What if you think that vocabulary size determines ability to do these simple word list memory experiments.

You could do a cross-sectional quasi-experiment where you grouped people by vocab size into high and low, and measure memory for a word list.

Or you could take each persons score on vocab measure and number of words recalled and correlate to memory.

(Draw graphs with two groups vs. 20 individuals.)

Other:
There are other styles of collecting observations but they are not the topic of this course really.

Control

Confounds
Any factor that varies with your IV and that could explain observed differences in your DV.

You do not want confounds.

Creep in because of
Nature of IV
Carelessness

Affects causal claims
In true experiment, something caused effect on DV
IV or Confound?

Limiting noise/random variability
Variability within groups
Individual Differences
Minor differences in experimental sessions
You want to limit

Contributes to total variability and limits ability to see effects

Formula for most stats is:

Variability due to the Effect
Total Variability

Control groups
Baseline comparison conditions
Time and memory
Immediate is control
Stroke victims and cognitive functioning
Same age
What is the best comparison condition
Nothing
Existing treatment in medicine

Eliminate other explanations
Misinformation effect
Debriefing group for conformity effects

Placebos
Is it the IV or just getting “special treatment”?

Medical Research
Sugar pills

Education Research
Other effective instruction techniques

Counseling
Wait list control

Blind and Double Blind
Blind – Subject is unaware of IV level
Double Blind – Subject and experimenter are unaware