Lecture 1:  Experimental Design

Outline
Basic Terms
-- population
-- sample
-- subject
-- independent variable
-- dependent variable

Designs
 -- true experimental variables 1:  between-subjects
  basic 2 group between-subjects
  multiple levels
  pretest/posttest designs
 -- true experimental variables 2:  within-subjects
  basic 2 group within-subjects
  multiple levels
-- quasi-experimental variables:  cross-sectional or status variables
  basic 2 group within-subjects
  multiple levels
 -- factorial designs
correlational designs

Control
-- confounds
-- limiting noise
-- control groups
-- placebos
 

Basic Terms

Population
All members of a group
Group can be variously defined

Sample
A subset of a population
Unbiased sample vs. biased sample
Random sample:
All members of the population have equal chance of being selected
Generalizing from sample to the population

Subject
A single member of the sample
Human subjects are often called participants

Independent Variable (IV)
Treatment or Factor
The manipulated variable in an experiment
IV has various levels that different groups/conditions experience
Theory claims IV is causal

Dependent Variable
Outcome variable
Measured in experiment
Theory claims IV affects DV
 
 

Designs
Today we are going to worry about a basic step in designing and controlling your observations:  experimental design.

  OBS --> Theory --> Hypothesis --> OBS

I want to start by explaining something about this design process:
Everything is determined by theories.
· Theories tell you what IVs should affect the variable of interest
· You can then make precise hypotheses concerning what levels of the IVs should lead to high and low outcomes on the DV
· You then design an experiment (observation) that causes those levels, controls other possible confounds, and measures the outcome
· Thus you start with theories in a given domain.

One other point:  the experimental design and the scale of measurement determine your statistics:  so once you know your question, everything else follows.
Thus start thinking about what questions are worth asking.  Do not do it the other way around, say I want to do this stat so what design, what question.
 

True experiments:  between-subjects variables
 True experiments are where you manipulate the Independent Variable and you control all other factors that may influence the Dependent, or outcome, Variable.  In this case you can really answer a question; you can know if changes in the IV cause changes in the DV.
 If you want to know the differences between X and Y, you set things up such that X happens to one group and Y happens to another.  You then measure outcomes.  If you want to know the difference between X, Y, and Z, you set things up so that X, Y, and Z happen to different groups and you measure outcomes.

Basic 2 group between-subjects true experiment
Here is the basic design
 
IV DV
Pop    --> Sample    --> A Level 1 measure Compare groups
B Level 2 measure

Keep in mind that you start with a question and you then figure out which design will allow you to investigate that question.

 Imagine that you wanted to investigate the decay theory of forgetting.  Say, for example, you were interested in how time since learning affects memory.  Then IV would be time; level 1 might be 0 (immediate recall) and level 2 might be 1 minute.  DV would be determined by how you operationally defined memory.  Say number of words from a word list.

Multiple levels between-subjects true experiment

Say if you had different ideas about how forgetting works.  If you thought interference was important and you wanted to know if visual information interferes with verbal information, but you realize that just time itself does allow for some decay in memory.
 
 
IV DV
A Level 1 measure
Pop   ---> Sample    --> B Level 2 measure Compare groups
C Level 3 measure

IV:  Type of Interference
Levels: 1 = verbal inference; 2 = visual interference, 3 = no interference (control)

Pretest/posttest designs
This is another way to control for individual differences:  measure each subject before and each after IV, then the DV is the change.  Say effects of different types instruction but you know some people know a lot before the course and some know less.
 
 
Pretest DV IV Posttest DV
Pop   ---> Sample    --> A measure Level 1 measure compare changes
B measure Level 2 measure

True experiments, II:  Within-subjects designs
Sometimes it is possible to conduct within-subjects designs:  This, like a pretest-posttest design, eliminates the problem of individual differences.
 In this case, each subject acts as his/her own control and experiences all conditions.  So for time study each subject would get one word list at each delay:
 You can do this with multiple levels as well, each subject experiences every level of IV and gets measured on the DV after each.

Advantages
 Controls for Individual Differences
 Powerful statistically
 Uses fewer subjects

Disadvantages
 Design is transparent to subjects
 May lead to practice effects
 

Quasi-experimental designs:  Cross-sectional or status variables
What if we want to investigate the differences between two different types of people.  Say you are still concerned about learning little words on a list, but what about the difference between people with large and small vocabularies.  You have reason to believe that people with more knowledge of words will be better able to process and store words in a list and thus should recall more of the a word list.
 Now in this case, we can’t randomly assign some people to have large vocabularies and some to have small.  Instead you are drawing samples from two separate populations
 
 
IV DV
Pop 1    --> Sample  1  --> measure Compare groups
Pop 2    --> Sample  2  --> measure

IV:  Vocab size
Levels:  1= large vocab; 2 = small vocab,
Both experience word list and are asked to recall it.
Of course, you can have multiple levels here just like in true experiments.

True Experimental Variables vs. Quasi-Experimental Variables
Difference is in your ability to infer
  Can infer that IV is related to DV, but what about the IV?
Why are people that way?
Is there something causing them to have large vocabs and do well on word list tasks?
 Cannot determine causality!

What factor (IV) you are interested in determines whether or not you can manipulate it and thus whether you can conduct a true experiment.

Factorial designs
What if you think two IVs affect the DV in question?

Say for forgetting, you want to look at both time and interference.

IV1: Time
 1= 1 minute
2= 2 minute
IV2:  Type of Interference
1= verbal
2 = visual

Draw as a box
 
 
IV 
2
IV 1 - Time
L1 - Verbal
L2 - Visual
   L1 - 1 minute
1m of verbal
1m of visual
   L2 - 2 minutes
2m of verbal
2m of visual

Note: In a factorial design the IVs can be between-subjects, within-subjects, or cross-sectional.  Can have lots of IVs
 

Correlational designs
What if you think that vocabulary size determines ability to do these simple word list memory experiments.

You could do a cross-sectional quasi-experiment where you grouped people by vocab size into high and low, and measure memory for a word list.

Or you could take each persons score on vocab measure and number of words recalled and correlate to memory.

(Draw graphs with two groups vs. 20 individuals.)

Other:
There are other styles of collecting observations but they are not the topic of this course really.
 

Control

Confounds
Any factor that varies with your IV and that could explain observed differences in your DV.

You do not want confounds.

Creep in because of
 Nature of IV
 Carelessness

Affects causal claims
 In true experiment, something caused effect on DV
 IV or Confound?

Limiting noise/random variability
Variability within groups
 Individual Differences
 Minor differences in experimental sessions
You want to limit

Contributes to total variability and limits ability to see effects

Formula for most stats is:

        Variability due to the Effect
            Total Variability

Control groups
Baseline comparison conditions
 Time and memory
  Immediate is control
 Stroke victims and cognitive functioning
  Same age
  Other head injury
  Young adult
 What is the best comparison condition
  Nothing
  Existing treatment in medicine

Eliminate other explanations
 Misinformation effect
  Debriefing group for conformity effects

Placebos
Is it the IV or just getting “special treatment”?

Medical Research
 Sugar pills

Education Research
 Other effective instruction techniques

Counseling
 Wait list control

Blind and Double Blind
 Blind – Subject is unaware of IV level
 Double Blind – Subject and experimenter are unaware