Home
Subject: Re: Experimental Research In Education: The Most Exciting
Talk at the 2005 Joint Statistical Meetings
To: EdStat E-Mail List
ApStat E-Mail List
Teaching Statistics E-Mail List
sci.stat.edu Usenet Newsgroup
From: Donald B. Macnaughton < donmac@matstat.com >
Date: Sunday February 11, 2007
-----------------------------------------------------------------
Some Basic Ideas of Science
This post replies to comments by Mike Palij. But first, to sup-
port the discussion, it is helpful to propose definitions of some
basic ideas of science, as follows:
EMPIRICAL RESEARCH is any activity in which measurements
(observations) are gathered from some area of experience
and then reasonable conclusions are drawn from the meas-
urements.
Measurements in empirical research are usefully viewed as
reflecting the values of VARIABLES, which reflect the
measured values of properties of entities. (Entities may
be people, other living things, physical objects, or any
other type of object or thing.)
An EMPIRICAL RESEARCH PROJECT (or logical sub-unit of an
empirical research project) is usually usefully viewed as
studying the relationship between a single response vari-
able and one or more predictor variables in the entities
in the population of interest. This study is performed
by analyzing the measured values of the variables ob-
tained from a sample of entities from the population.
An EXPERIMENT is an empirical research project in which
at least one of the predictor variables is “manipulated”
(i.e., caused to take certain values in the entities in
the sample) by the researcher.
An OBSERVATIONAL RESEARCH PROJECT is an empirical re-
search project in which none of the predictor variables
is manipulated by the researcher, and the values of the
predictor variables are simply observed.
I discuss the ideas in more detail in a paper (2002).
The description of an empirical research project is qualified
with the word “usually”. This implies that some empirical re-
search projects (or sub-units) don’t satisfy the description --
that is, they can’t be reasonably viewed as studying the rela-
tionship between a single response variable and one or more pre-
dictor variables. My experience suggests that research projects
that can’t be reasonably viewed as satisfying the description ap-
pear in less than three percent of published empirical research
reports (in science, technology, and business). I discuss some
of these research projects in the paper (2002, Appendix I.2).
For economy of words, this small group of research projects is
mostly ignored in the discussion below.
Novelty of Experiments in Education
Quoting my August 25, 2005 post, Mike Palij wrote (on August 26
in the EdStat e-mail list)
< snip >
> Experimental studies in education are not a new idea.
I agree. I suspect that the idea of performing proper experi-
ments in education can be traced back to the 1920’s or 1930’s
when Fisher, Neyman, Pearson, and other statisticians first ex-
plained and debated the idea of a proper experiment. However,
proper experiments in education haven’t often been performed.
What is a “proper” experiment? Here is a reasonable definition:
An experiment (or randomized trial) is a PROPER EXPERI-
MENT if it has been performed according to widely ac-
cepted principles of scientific practice, experimental
design, and data analysis, as described by Bailar and
Mosteller (1992), Box, Hunter, and Hunter (2005), Fleiss
(1986), Kirk (1995), Winer, Brown, and Michels (1991),
and many others.
I suspect that proper experiments haven’t often been performed in
education for two reasons: (a) proper experiments in education
are difficult, and (b) perhaps simply due to tradition, research-
ers in education have lacked a strong connection to the ideas of
experimental research.
Problematic Nature of Experiments in Education
< snip >
> There are a number of reasons why experiments are problematic
> in educational settings
I fully agree. Some key problems in performing a good education
experiment to compare teaching approaches are
1. What is the RESPONSE VARIABLE (or variables) that we will use
to compare the teaching approaches? Will it be marks, or at-
titudes, or aptitudes, or some other measure of the students
(or the classes of students) under study?
2. What are the PREDICTOR VARIABLES that we will measure in the
research? One predictor variable will reflect the different
teaching approaches under study. What other predictor vari-
ables (e.g., students’ age or gender) should we measure to as-
sist our understanding? Is it reasonable to use students’
race as a predictor variable?
3. How can we design the experiment in a way that will ELIMINATE
teacher effects and other REASONABLE ALTERNATIVE EXPLANATIONS
of any significant differences we find in the values of the
response variable between the teaching approaches?
4. How can we design the experiment so that it is as LIKELY as
possible that we will FIND STRONG EVIDENCE of the relationship
we are looking for between the response variable and the pre-
dictor variable(s) (assuming that the relationship actually
exists)?
5. How can we RECRUIT TEACHERS to participate in the experiment?
6. How can we OBTAIN FUNDING to pay for the experiment?
7. After we have performed the experiment, how can we ANALYZE the
RESULTS and DRAW SCIENTIFICALLY VALID CONCLUSIONS?
8. While properly addressing the preceding seven problems, how
can we MINIMIZE the COSTS of the experiment?
Appendix A expands the eight problems and discusses some solu-
tions.
Necessity of Experiments in Education
< snip >
> As for the necessity of experimental research to provide the
> basis for either (a) science or (b) valid conclusions, the as-
> tronomers say “Hi! We’ve been engaging in science for hundreds
> of years without having performed a single experiment!”.
Mike suggests that if astronomers don’t perform experiments, then
education researchers also don’t need to perform experiments. I
think that this is a thought-provoking point, but an invalid ar-
gument. The argument is invalid because astronomers CAN’T per-
form experiments because they can’t manipulate distant astronomi-
cal events. If they could manipulate these events (at a reason-
able cost), it seems doubtless that they would. That is, they
would perform proper experiments just like scientists in disci-
plines that regularly perform experiments, such as in most
branches of physics, chemistry, engineering, medicine, biology,
and psychology.
The fact that astronomers can’t perform astronomical experiments
makes astronomy an “observational” discipline. Other observa-
tional disciplines that generally can’t perform experiments (due
to the remoteness or untouchability of the phenomena they study)
include anthropology, archaeology, economics, epidemiology, geol-
ogy, paleontology, and some areas of sociology. Such observa-
tional disciplines base their inferences on careful observational
empirical research, often studying relationships between vari-
ables. (In historical disciplines, observational research is
sometimes [due to the paucity of data] reduced to careful consid-
eration of physical or anecdotal information about entities,
properties of entities, or variables, without focusing on the
concept of ‘relationship between variables’.)
Proper observational research often enables us to reliably
PREDICT the values of the response variable (in new situations),
which is an important benefit. However, in education research we
would generally like to learn how to reliably CONTROL the values
of the response variable (in new situations). That is, we would
like to learn how to structure an education program so that it
provides students with the BEST education. Unfortunately, obser-
vational research is almost always equivocal about control --
subject to multiple competing reasonable explanations.
For example, suppose we are presented with the results of an ob-
servational research project in education that suggests that a
certain teaching approach A is better (in the sense of exhibiting
significantly better average values in students of the chosen re-
sponse variable) than another teaching approach B. In this case
(due to the nature of observational research) it is almost always
possible to find a reasonable alternative explanation of the re-
search finding, and this explanation implies that approach A may
NOT be better than approach B. But if we find such an explana-
tion, this implies that the research is equivocal. This means
that the research is of substantially less value because it can’t
reliably help us to decide which teaching approach is better.
(A reasonable alternative explanation in observational education
research is often in terms of “confounding” of the teaching ap-
proaches under study in the research with some other aspect
[i.e., variable] of the research situation. Then it is generally
possible that this other variable can fully account for the dif-
ference between the average values of the response variable under
the different teaching approaches. For example, an observational
research project might confound two teaching approaches with two
different schools -- school 1 uses teaching approach A, and
school 2 uses teaching approach B. In this case if we find a
significant difference in the average values of the response
variable between the two approaches, it is possible that the
teaching approaches have no differential effect on the values of
the response variable. That is, [unless the School variable is
appropriately (and expensively) taken account of in the design of
the research] it is possible that a certain difference between
the SCHOOLS caused the observed significant differences in the
average values of the response variable [in students or classes]
between the teaching approaches.)
In contrast, suppose that a proper EXPERIMENT provides good evi-
dence that teaching approach A is better than teaching approach
B. In this case the finding is unequivocal. This is because
proper experiments are explicitly designed to eliminate confound-
ing and other reasonable alternative explanations. Thus in this
case we can safely (tentatively) conclude that approach A will be
better than approach B in new situations (if the relevant condi-
tions are sufficiently similar to those of the experiment). Thus
proper experiments are preferred to observational research pro-
jects in education research.
(The equivocation in observational research relates to drawing
conclusions about causation. That is, we are interested in
whether teaching approach A CAUSES students to do better than
teaching approach B. Evidence about relationships between vari-
ables obtained in observational research is generally equivocal
about causation. In contrast, evidence about relationships ob-
tained in proper experimental research is unequivocal about cau-
sation.)
Changing Attitudes Toward Experiments in Education
Mike noted that members of the respected American Educational Re-
search Association (AERA) have carefully considered the issue of
experimental research in education. Mike’s point is directly re-
flected in the opening sentence of the official description of
the theme of the 2006 AERA annual meeting:
Current social and political pressures on education re-
search suggest that research must meet the demands of
evidence-based and scientifically based inquiry (Ladson-
Billings and Tate 2006).
The idea of “current” pressures reflects the fact that the pres-
sures on education research are new, having arrived over the last
decade or so. The sentence implies that education researchers
are moving toward “evidence-based and scientifically based” re-
search. This suggests that education researchers should gener-
ally perform proper experiments (because observational research
results are generally equivocal).
(Having acknowledged the importance of proper research, the de-
scription of the theme of the 2006 AERA meeting turns to the
theme itself, which pertains to education research in the public
interest, education research that will “increase the common-
wealth”. The discussion of the theme is available at
http://www.aera.net/annualmeeting/?id=694 )
Opportunities for Experiments in Education
The preceding discussion suggests that (a) the area of experimen-
tal studies in education is only now beginning to open up and (b)
this area will become the mainstream of education research as
granting agencies and journal editors reinforce the point that
proper experiments are preferred to observational research. Be-
cause the area is opening up, it has many opportunities for
thoughtful researchers.
To perform a proper education experiment a researcher must be fa-
miliar with the principles of experimental design, power analy-
sis, and (often) repeated measurements analysis of variance.
Some education researchers are less familiar with these topics.
They may find it helpful to follow the path of many medical re-
searchers who collaborate with a statistician with experience in
the topics. To ensure that the research design is efficient, I
recommend that this collaboration begin early in the design phase
of the research.
Opportunities exist for statisticians to present courses to edu-
cation researchers about the statistical and scientific aspects
of education research. I propose topics for such a course in ap-
pendix B.
I believe that the movement toward experimentally based education
research will yield a body of reliable research results that will
substantially improve education.
Don Macnaughton
Donald B. Macnaughton
donmac@matstat.com
Appendices
Appendix A: Eight Problems in Experiments in Education
Appendix B: Courses About Experiments For Education Researchers
Appendix C: Can Human Performance or Behavior Be Predicted from
a Person’s Race?
Appendix D: Specifying a Repeated Measurements Analysis of
Variance
Appendix A: Eight Problems in Experiments in Education
The body of this post lists eight problems that arise in experi-
ments in education. This appendix briefly expands the problems
and discusses some general solutions.
Problem 1: Choosing the Response Variable
Suppose we are designing an education experiment to compare two
teaching approaches. In choosing the response variable we can
reasonably begin by answering the following question:
What would we like the teaching approaches under study to
do (accomplish)? That is, what is our teaching goal?
This question can be answered at a high level by deciding which
of the following is the main goal:
1. maximize student knowledge and understanding of the subject
area
2. maximize certain student attitudes toward the subject area
3. maximize student aptitudes in the use of the subject area in
practical applications
4. optimize some other property (or combination of properties) of
the students.
The definition of the teaching goal in an education experiment
depends on the particular topic or discipline being taught, on
the type of students being taught, and on the course designer’s
and researcher’s interests. The definition of the goal deserves
careful attention because it lies at the heart of the research.
After we have defined the teaching goal, we can choose the re-
sponse variable by addressing a second important question, which
is
How can we best measure the effectiveness of a teaching
approach to satisfy the teaching goal?
The answer to this question defines the response variable. For
example, if the goal is to maximize students’ knowledge and un-
derstanding, the main response variable in an education experi-
ment will be a measure of each student’s knowledge and under-
standing, typically a weighted average of marks on assignments or
tests. In this case it is important to devise a “fair” measure
of students’ knowledge and understanding by devising “fair” as-
signments or tests -- a challenging but doable task.
Similarly, if the teaching goal is to maximize certain student
attitudes toward the subject area of the course, the response
variable(s) will be one or more measures of students’ attitudes.
Attitudes are important because they play a key role in peoples’
decisions. It is generally unnecessary to devise a test of atti-
tudes toward the subject area of a course because reliable stan-
dardized attitude tests (which can be administered in less than
twenty minutes) are available for many subject areas.
(You can find tests of students’ attitudes toward a subject area
by searching the Web of Science [available in some university li-
braries], other science literature databases, and the world wide
web for articles and books that have the word “attitude” [or “at-
titudes”] and the name of your subject area in their titles or
keywords. You may also be able to find acceptable generic atti-
tude tests. If you are studying a subject area in which an ac-
ceptable test of student attitudes toward the area is not avail-
able, and if you are comfortable with statistical ideas, you can
study other attitude tests and then use the principles of atti-
tude scale development [Aiken 2002, Krosnick, Judd, and
Wittenbrink 2005] and the statistical procedure of exploratory
factor analysis [Thompson 2004] to develop such a test.)
Similarly, if the teaching goal is to maximize students’ apti-
tudes (which is a key goal in courses that teach applied skills),
the response variable will be a measure of students’ aptitudes.
Standardized measures of aptitudes are available in some areas,
and can be found with the methods in the first sentence of the
preceding paragraph.
Problem 2: Choosing the Predictor Variables
Choosing the predictor variables in an education experiment to
compare teaching approaches requires first that we choose the two
(or more) teaching approaches that we wish to compare. These
teaching approaches define the values of the “teaching approach”
predictor variable. This variable is manipulated in the students
in the sense that some students receive one of the teaching ap-
proaches, and other students receive the other. Two reasonable
teaching approaches to compare are (a) the traditional approach
to teaching some topic or discipline and (b) the top contender to
replace the traditional approach. The experiment stages a fair
contest between the two approaches.
As noted above, in addition to choosing the main predictor vari-
able, we must also decide which other variables of the situation
under study we will measure. Generally, the more “relevant” pre-
dictor variables we measure in an experiment, the better the un-
derstanding we obtain of the relationship between variables we
are studying. Thus it may be useful to measure each student’s
age and gender. It may also be relevant to measure students’
previous experience with the material, intelligence, socioeco-
nomic status, and years of experience speaking the language that
the course is taught in.
Appendix C discusses the use of “race” as a predictor variable in
research studying measures of human performance or behavior.
Problem 3: Eliminating Reasonable Alternative Explanations
The need to eliminate reasonable alternative explanations of a
research finding stems from the sensible principle that good re-
search must be unequivocal. Eliminating reasonable alternative
explanations is difficult because many forms of reasonable expla-
nations are possible, and some are hard to recognize. Thus ex-
perienced researchers spend considerable time trying to think of
reasonable alternative explanations of research results, espe-
cially results of their own planned research. If the elimination
of reasonable alternative explanations is properly done (gener-
ally through careful research design), it (mostly) eliminates the
possibility that the associated research conclusion will be ruled
invalid due to a reasonable alternative explanation that someone
thinks of later.
Confounding alternative explanations are eliminated by random as-
signment of experimental entities to treatments. That is, in an
education experiment we randomly assign students (or classes of
students) to teaching approaches. Such random assignment helps
to ensure (in a probabilistic sense) that the different teaching
approaches could not be confounded with schools or with many
other variables.
For logistical reasons, random assignment is sometimes difficult
in education research. In the case of randomly assigning stu-
dents to different teaching approaches, the problem can often be
solved (albeit at some expense) by running both (or all) the
teaching approaches together at the same time on the same day of
the week in nearby (similar) facilities. (If enough resources
exist, this concurrent presentation of the set of treatments can
be repeated on different days of the week or at different loca-
tions.) This means that students can be readily randomly as-
signed to the treatments (teaching approaches) without confound-
ing because none of the treatments has extraneous time or loca-
tion advantages. I recommend that the researcher (a) discourage
students from switching between classes, but permit such
switches, and (b) identity any students who switch, perhaps to be
with a friend. The students who switch classes after the course
has begun can be studied and then (due to unrepresentativeness)
be omitted from the analysis.
Problems 4, 7, and 8: Power, Analysis, and Minimizing Costs
The fourth, seventh, and eighth problems respectively pertain to
maximizing the power of the statistical tests in an experiment,
analyzing the data obtained in the experiment, and minimizing the
costs of the experiment. Detailed technical help with these
problems is available from the field of statistics, which has ef-
ficient general methods for the design and analysis of powerful
but inexpensive experiments. I discuss an introduction to the
methods of statistics in the paper (2002).
One can get practical help with problems 4, 7, and 8 by studying
the medical research techniques of multicenter clinical trials.
These techniques can help because proper experiments in education
must be performed in parallel at several teaching institutions
(or at least in several classes) to eliminate teacher effects and
to provide power and generality. This is similar to how medical
research is performed in parallel in several hospitals in a mul-
ticenter clinical trial.
(Technical Aside: In medical research the patient is often the
entity [unit] of analysis, but in education research the class of
students is often the [implicit] entity of analysis. A class of
students in education research is analogous to a hospital [or
other grouping] of patients in a multicenter clinical trial. A
researcher can often substantially increase the power of statis-
tical tests in education research [with often only a minor in-
crease in costs] by designing the research so that the student
[instead of the class] is the entity of analysis. This can be
done by using a pre-post [i.e., repeated measurements] experimen-
tal design. That is, the value of the response variable is meas-
ured in each student in the experiment both before and after the
students experience their assigned teaching approach. Such a de-
sign is feasible with many but not all response variables.)
Problem 5: Recruiting Teachers
Recruiting teachers or teaching departments to participate in ex-
perimental research in education is facilitated if the researcher
explains how the research will provide important educational
benefits. If this is carefully done, and if the disruption of
the existing program is not too great, appropriate participants
can generally be recruited, just as appropriate medical personnel
at different hospitals are recruited as a first step in a multi-
center clinical trial.
Problem 6: Obtaining Funding
To obtain research funding a researcher submits a carefully writ-
ten grant proposal to an appropriate funding agency. The pro-
posal competes with other proposals for funds from the pool of
funds distributed by the agency. Research proposals are judged
on the following criteria:
- the reasonableness of the hypothesized phenomenon that the re-
search will study
- the clarity of thinking in the rationale and implications of
the research
- the potential of the hypothesized phenomenon to make a worth-
while contribution to the field under study, and
- the conformity of the proposal to the correct style and proto-
col for grant proposals to the agency.
The research projects whose proposals best satisfy the above cri-
teria generally receive funding.
Appendix B: Courses About Experiments
For Education Researchers
I recommend that courses about experimental research for educa-
tion researchers discuss the eight problems discussed in appendix
A in terms of examples of good and bad education experiments. I
recommend that teachers discuss real or realistic experiments (as
opposed to abstract experiments) because realistic experiments
enable students to consider the specific research goal of each
example. Considering researchers’ goals helps students to formu-
late the goals of their own research.
In discussing a “good” education experiment it is important to
convey to students the practical benefits that are provided by
the results because these benefits generally justify the care
taken to perform the experiment.
In view of the usefulness of a pre-post experimental design for
increasing power, I recommend that this type of design and the
proper analysis of the results of this type of experiment be dis-
cussed in detail.
Discussion of data analysis can best omit all mathematical con-
cepts and focus on interpreting the output from the computer
analysis of the data. Most experiments to compare teaching ap-
proaches have a continuous reasonably-well-behaved response vari-
able (e.g., marks or attitude scores), and the main predictor
variable (i.e., “teaching approach”) is discrete. Thus the re-
sults of these experiments are best analyzed with analysis of
variance (which becomes repeated measurements analysis of vari-
ance if a pre-post design is used).
(Repeated measurements analysis of variance is also called “mixed
model” analysis because the right-hand side of model equation of
the relationship between the variables contains a mixture of (a)
“fixed” terms (associated with the predictor variables) and (b)
“random” terms associated with unaccounted-for variation in the
values of the response variable. However, I prefer the term “re-
peated measurements” because it is more intuitive for beginners.)
I recommend that discussion of data analysis be in terms of exam-
ples of good experiments in education research. I recommend that
the discussion cover the following topics:
1. For each example, a discussion of the research hypothesis (or
research question) under study, a discussion of the conduct of
the experiment, and a discussion of the layout of the data ta-
ble that was obtained in the experiment and that is the basis
of the data analysis.
2. What the computer (software) must be told in order to perform
the analysis of variance. We must tell it
- the location of the data table (often in a file on the
computer)
- which variable in the data table is the response variable
in the analysis
- which variable(s) in the table is (are) the predictor
variable(s)
- in repeated measurements experiments which of the predic-
tor variables vary within experimental entities and which
vary between experimental entities (or, equivalently,
which variable uniquely identifies the experimental enti-
ties)
- in more complicated experiments details of the relation-
ship between variables we are studying, such as the hy-
pothesized form of the model equation of the relationship.
3. How to tell the computer what it must be told. This varies
among different software products, and is discussed further in
appendix D.
4. How to interpret each p-value in an analysis of variance table
produced by the computer from the analysis of the data.
5. How to understand tables of means of the response variable for
effects with low p-values, as produced by the computer.
6. How to graphically illustrate means of the response variable
for effects with low p-values for ease of understanding of the
results.
7. How to understand other output from the computer such as (a) a
measure of strength or “effect size” of each relationship be-
tween variables studied in the experiment and (b) the esti-
mates of the values of the parameters of the model equation.
8. How (to avoid embarrassing errors) the researcher must confirm
that the underlying assumptions of a statistical analysis are
adequately satisfied before drawing final conclusions. (This
includes (a) checking of the univariate distribution of each
of the variables for anomalies and (b) appropriate verifica-
tion that the data have certain necessary properties. The
specific properties depend on which statistical procedure is
used to study the relationship.) These assumptions are often
but not always adequately satisfied.
9. How to interpret the analysis in terms of the research hy-
pothesis.
Some statistics courses omit or minimize the first and last top-
ics, which pertain to the research hypothesis. This omission oc-
curs because some courses are more focused either on data analy-
sis or on statistical theory, which are both vast topics. How-
ever, consideration of the implications of the research for the
research hypothesis is clearly important in courses aimed at edu-
cation researchers.
In high-level terms, the results of a data analysis have one of
three implications for the research hypothesis, which are:
1. the results support the research hypothesis (and there is no
reasonable alternative explanation of the results)
2. the results neither support nor refute the research hypothesis
(either because the research found no good evidence of the
sought-after relationship between variables or because a rea-
sonable alternative explanation of the results is available)
3. the results refute (contradict) the research hypothesis.
Because of the positive way that research hypotheses are framed
(e.g., drug D reduces cancer), researchers performing a research
project almost always hope that the first outcome will occur.
That is, they hope that the results of the research project will
support their research hypothesis. In this case, if the hypothe-
sis was carefully thought out, the finding will make a contribu-
tion (perhaps substantial) to the associated field of study. Un-
fortunately, the second outcome sometimes occurs, perhaps because
the research hypothesis is false, or because the research was
poorly designed, or due to the whims of chance. Although the
third outcome is generally possible, it rarely occurs in prac-
tice.
After students understand the basic ideas of drawing conclusions
from data analysis, I recommend that they learn how to use the
computer to generate realistic artificial data. Such data gen-
eration is not difficult if students are given the appropriate
instructions (or program templates) and if they are shown how to
generalize the instructions as necessary.
The ability to generate realistic artificial data has three bene-
fits: First, generating artificial data helps students to under-
stand the postulated model equation of the relationship between
the variables that is under study. This understanding comes be-
cause generating data is most easily understood through writing a
simple computer program that substitutes the values of the pre-
dictor variables into the model equation and then evaluates the
equation to generate the predicted value of the response vari-
able. (This evaluation is done repeatedly through the use of
program loops and with one or more random number generators to
generate the values of the random term[s] in the equation.) From
a theoretical perspective the model equation is the essence of a
relationship between variables, and the students’ experience in
programming the essence helps them to understand it.
Second, the ability to generate artificial data gives students a
source of “tame” data, which they can then analyze with data
analysis procedures. Because the students generate the data,
they know exactly what relationship(s) is (are) present (or not)
in the data. This allows them to see how the data analysis pro-
cedures work to detect and characterize relationships between
variables. This helps students to develop knowledge and trust of
the procedures.
Third, students can be encouraged to use their ability to gener-
ate artificial data when designing their own research projects.
That is, during the design phase of a research project students
can generate sets of realistic artificial data that resemble the
data they expect to obtain in the research. Then they can care-
fully analyze these artificial data with the planned data analy-
sis procedure. This gives students a thorough review of the
planned research design and data analysis procedure before the
design and analysis are put into practice. Such a review is of-
ten helpful in eliminating later serious problems and in increas-
ing efficiency, especially for beginning researchers.
(Statistical software vendors can help users to generate artifi-
cial data by providing easy-to-follow instructions for generating
data for all the common types of relationships between variables.
I recommend that these instructions be placed prominently in the
software documentation so that beginners can easily find this im-
portant resource. A component of the software that can generate
a data table from fill-in-the-blanks specifications might also be
helpful for beginners.)
I believe that the topics in this appendix, when developed in ap-
propriate detail, give prospective researchers a reasonable in-
troduction to how to perform experiments in education research.
Appendix C: Can Human Performance or Behavior
Be Predicted from a Person’s Race?
The Model Equation of a Relationship Between Performance and Pre-
dictors of Performance
Suppose that the variable y is a particular measure of human per-
formance or human behavior. For example, y might reflect stu-
dents’ grade point averages or it might reflect some measure of
athletes’ ability. Under the scientific approach we think that y
“depends” on a number of other variables. For example, we think
that each student’s grade point average probably depends on the
students’ intelligence, on their motivation, on their parents’
style of parenting, on the attitudes of their friends, perhaps on
their diet, and on various other variables.
The relationship between y and the other variables can be written
in a general model equation as
y = f(x1, x2, ..., xn) + ε.
The performance measure, y (e.g., grade point average), is the
response variable in the relationship, and the x1, x2, ..., xn
are the relevant predictor variables (e.g., intelligence, motiva-
tion, etc.). The notation f(...) stands for a mathematical func-
tion that outputs the estimated value of y for a person when the
values of the x’s for the person are substituted into it. [The
detailed mathematical form of f(...) is discovered through analy-
sis of relevant empirical research data.] The symbol n indicates
the number of predictor variables under consideration, which is
typically between one and five.
The Greek letter ε (epsilon) on the right end of the equation is
the “error” term. It takes account of the fact that f(...) gen-
erally can’t perfectly predict the actual measured value of y for
a person from the x’s. The error term is a “random” variable be-
cause it has a different seemingly random value each time the
equation is applied. Usually ε is sensibly modeled as being half
the time greater than zero and half the time symmetrically less
than zero, so it has an average value of zero. Invariably ε is
modeled as being more often closer to zero than farther away.
(Technical Aside: In any particular [standard] instance of a
variable [and at a given time] the variable [just like the prop-
erty behind the variable] has a single value. [For simplicity, I
ignore here (a) the idea that a particular value of a variable
may be “missing” and (b) the more general but infrequent case of
variables that are vectors.] A variable can be classified as be-
ing either a continuous variable or a discrete variable. If a
variable is a continuous variable, its value in a particular in-
stance can theoretically be any value between the minimum and
maximum permissible values. Continuous variables almost always
have numeric values. For example, grade point average is a con-
tinuous variable that for a given student [in some schools] can
have any value between 0.00 and 4.00, such as 3.82. The values
of variables that are obtained from conventional measuring in-
struments [of any type, e.g., ruler, stopwatch] are usually con-
tinuous variables. [The values of any continuous variable are
limited to a certain maximum number of significant digits (often
between two and four) due to limitations in the accuracy of the
measuring instrument that is used to measure the values.] In
contrast, if a variable is a discrete variable, its value in a
particular instance can be one of only a limited number of dif-
ferent values, usually less than thirty, and sometimes as few as
only two. [Discrete variables can be ordinal -- with an implicit
ordering -- or categorical.] For example, the variable “likes to
dance” is reasonably viewed as a [ordinal] discrete variable that
for a given person has one of a range of five [or perhaps seven]
possible values indicating different levels of liking or dislik-
ing to dance. [The limitation to five or seven values generally
occurs in raw variables that reflect human judgments or opinions,
as discussed by Miller, 1956.] For simplicity, the discussion in
this appendix assumes that a continuous response variable is al-
ways used in research projects because using continuous response
variables is [when feasible] the more efficient and more common
approach. If the response variable in a particular empirical re-
search project is discrete, some of the technical ideas behind
model equations of relationships between variables change, but
the main principles in this appendix still apply, as shown by the
theory of generalized linear models [McCullagh and Nelder,
1989].)
We (i.e., society) can use the model equation of a (properly
verified) relationship between variables to help us to predict
and sometimes control the values of the response variable in a
new situation on the basis of measuring or controlling the values
of the predictor variables in the situation. If the variables
are carefully chosen, the ability to predict or control can be of
substantial value. For example, if we can find a (ethical)
method to control (i.e., raise) grade point averages in students
by controlling the values of other variables, we can use the
method to help students to excel.
As research into a relationship between variables advances, more
predictor variables may be discovered that can be (correctly) in-
cluded in the function f(...) for the relationship, which makes
the predictions (or control) made by the function more accurate.
As the predictions become more accurate, the average (absolute)
size of the error term ε in the model equation for the relation-
ship becomes accordingly smaller. In some areas of research
(e.g., in many areas of the hard sciences) the model equations
make almost perfect predictions. Thus the error terms in these
equations are so small that they are often sensibly ignored.
The Concept of ‘Race’
The discussion below uses the concept of ‘race’. Most people
above the age of ten or so have a reasonable intuitive under-
standing of this concept in the sense that they can reliably
(though not perfectly) assign themselves and other people to ra-
cial categories. (The assignments are “reliable” in the sense
that different people generally agree with each other [at a mutu-
ally acceptable level of classification] about the assignments.)
Although most people understand the concept of ‘race’ at an in-
tuitive level, formal definitions of the concept are difficult.
The definitions break into three classes, which are
1. definitions in terms of a person’s biological ancestry (e.g.,
in terms of classification of the person’s genetic DNA)
2. definitions in terms of a person’s self-reported race
3. definitions in terms of a person’s observable attributes such
as skin color, hair color, facial characteristics, and speech
characteristics.
Each of the three classes contains various definitions of race.
Each definition provides (at least in theory) a way of assigning
people to racial categories. The categories are usually discrete
categories (e.g., Asian, Black, Mixed, Native American, White,
Other) rather than reflecting one or more continuous scales.
For example, using the self-report approach a researcher might
ask each person studied in a research project which of the above
six racial categories they belong to. Thus race would be defined
and measured in terms of the six categories. A second researcher
might define race in terms of eight or ten or even more catego-
ries reflecting the many identifiable groups of people in the
world.
The various definitions of race are closely associated, but are
different because assignments to racial categories by one of the
definitions will sometimes disagree with assignments by another.
For example, a person may ancestrally belong in whole or in large
part to one race, but may report or appear as belonging to an-
other.
Definitions of race in the first class (biological ancestry) are
generally preferred to definitions in the other two classes be-
cause the first class seems basic, and the other two classes seem
to be merely less accurate reflections of it. However, defini-
tions in the first (and third) class are often difficult to im-
plement in practice in research. (Some of the difficulties arise
from respected ethical considerations.) Thus if a research pro-
ject is performed on people in which each person’s race is meas-
ured, the researcher will often measure race in terms of self-
reported race.
Is Performance “Causally Dependent” on Race?
Suppose that a research project is carried out to study the rela-
tionship between (a) a measure of human performance (or behavior)
as the response variable and (b) a set of other variables that
are predictor variables. The predictor variables may reflect a
person’s attributes and may reflect manipulations applied to the
person in an experiment. Suppose the researcher includes in the
research a predictor variable that reflects race (or ethnicity).
And suppose that the research project finds good evidence of a
relationship between performance and race -- the average level of
performance of people from one race is significantly higher than
the average level of performance of people from another race.
This raises a key question: Can we conclude from this relation-
ship between variables that human performance depends to some ex-
tent on one’s race? In other words, can we conclude that differ-
ences in race cause differences in performance?
For example, evidence exists that Asians score somewhat higher
(on average) on intelligence tests than Whites, who in turn score
somewhat higher (on average) than Blacks. Does this imply that a
person’s intelligence depends (partly) on their race?
No. The evidence of the relationship doesn’t imply dependence or
a causal relationship because this relationship between variables
is invariably studied with observational research, as opposed to
proper experimental research. Observational research must be
used because it is impossible in a practical sense to manipulate
“race” in a proper experiment. That is, unlike assigning treat-
ments to people (or people to treatments) in an experiment, a re-
searcher can’t assign races to people (or people to races) be-
cause race has already been assigned. Because research projects
studying the relationship between performance and race are in-
variably (in that aspect) observational research projects, the
results of the research are open to reasonable alternative expla-
nations, as discussed in problem 3 in appendix A.
For example, the relationship between intelligence test scores
and race was found through observational research. This rela-
tionship could easily be accounted for by other causal variables
that are confounded with race and that have (unfortunately) been
omitted from the analysis (typically because they are unknown or
are deemed unimportant). For example, due to a history of op-
pression that began with Whites’ slavery of Blacks, many Blacks
throughout the world have had less access to opportunities and
resources, which might account for the differences in average in-
telligence test scores between Blacks and Whites. Thus if vari-
ables that properly reflect the relevant types of oppression (in-
cluding relevant historical effects) are included in the analy-
sis, the Black/White aspect of the relationship between intelli-
gence test scores and race might easily vanish. Similarly,
Asians might score higher (on average) on intelligence tests than
Blacks and Whites due to cultural childhood influences among
Asians that emphasize disciplined logical thinking. Thus if a
variable reflecting childhood encouragement of logical thinking
is included in the analysis, this second aspect of the relation-
ship between intelligence test scores and race might also vanish.
To minimize expensive errors, science demands unequivocal evi-
dence of causation before causation can be inferred. But, as
noted, the results of research projects studying the relationship
between performance and race are generally equivocal because they
are open to reasonable alternative explanations. Therefore, it
is generally scientifically impossible to infer that performance
in humans is causally dependent on a person’s race.
The word “generally” in the preceding paragraph indicates the
possibility of exceptions. That is, it is conceivable that an
ingenious researcher might find a way to perform a proper experi-
ment or find a way to deal with all of the confounding variables
and all of the alternative explanations. This researcher might
still find good evidence of a causal relationship between a cer-
tain response variable reflecting performance and a person’s
race. Then we could conclude that a causal relationship exists
between performance and race. However, the chance of this occur-
ring is low because
1. Eliminating all reasonable alternative explanations would be
very difficult or impossible.
2. Knowing that performance depends to a small extent on race is
not of much obvious theoretical or practical use. Therefore,
there is little scientific incentive to study this type of re-
lationship. (In contrast, knowing that certain other response
variables depend on race is sometimes quite useful, such as in
the prevention and treatment of diseases.)
3. Knowing that performance depends on race may suggest to some
people a basis for racial discrimination. Discrimination is
undesirable because it usually harms everyone involved.
Therefore, there is a social disincentive to study this type
of relationship.
Can Performance Be Predicted in Individuals on the Basis of Their
Race?
Although it may be true that performance doesn’t depend on race,
it is still true that certain relationships exist between per-
formance and race. For example, as noted, a relationship exists
between intelligence test scores and race. Thus perhaps race
could be used to predict (and thereby indirectly control) per-
formance, even though a direct causal relationship between these
variables may not exist. For example, Asians perform better (on
average) on intelligence tests than Blacks or Whites. Therefore,
a company president wishing to maximize the intelligence of the
employees of the company might decide to hire only Asians as em-
ployees. Although such an approach to hiring is unethical, it is
instructive to temporarily ignore the ethics and consider it from
a strictly scientific point of view. Is it scientifically sensi-
ble to hire on the basis of a known relationship between some im-
portant measure of performance and race?
No. It is generally inefficient to predict individual human per-
formance from race because other predictor variables are substan-
tially more accurate. In particular, instead of using race, a
company will hire more effective employees if it bases its hiring
decisions on each job candidate’s education and experience, to-
gether with the candidate’s performance in an interview, and per-
haps the candidate’s performance on empirically valid aptitude
tests. This approach to hiring selects more effective employees
because any differences in performance between the races are (if
present at all) very small when compared to the vast (and identi-
fiable) differences in performance that occur within each racial
group.
(Perhaps an employer could properly use education, experience,
interview performance, and aptitude test scores in a model equa-
tion to predict job performance, but still reasonably include a
predictor variable reflecting race in the equation. Perhaps in-
cluding race would significantly improve the predictions made by
the equation, even though all the other variables are also prop-
erly used in the equation. That is, including the other vari-
ables in the relationship might make the relationship between
performance and race more “visible” in the analysis. This is
theoretically possible if a certain “interactive” type of rela-
tionship between variables occurs. However, the presently by far
more common outcome in social research when more predictor vari-
ables are added is that the individual predictor variables become
weaker rather than stronger due to relationships [confoundings]
among them. Since race is already only at best a very weak pre-
dictor of performance, it too is likely to become weaker or non-
existent in a model equation as the number of predictor variables
is increased.)
Therefore, even if ethical considerations are ignored, it is gen-
erally not scientifically reasonable to predict performance in
individuals on the basis of a known relationship between perform-
ance and race.
The Social Taboo Against Concluding That Performance Depends On
or Is Predictable From Race
Despite the preceding points, there is a tendency among some peo-
ple to think that one race (or religion) is superior to others in
one or more areas of performance. Unfortunately, this point of
view can lead to appalling undeserved human suffering. There-
fore, civilized society uses another important incentive to work
in concert with the somewhat complicated logical arguments in the
preceding paragraphs that performance can’t be reasonably viewed
as depending on (or as predictable from) race. This incentive
operates in the ethical realm and exists in the form of a strong
social taboo against concluding that performance depends on race.
This taboo exists without any need for justification in the sense
that many people accept it on an intuitive level without ques-
tioning it (because it is fair).
The taboo is vividly illustrated by the experience of Glayde
Whitney, a behavior geneticist with a record of distinguished re-
search in the genetics of mouse taste, and who was the 1995
President of the Behavior Genetics Association (BGA). In view of
the BGA’s name, many of its members have considered the idea of
relationships between performance and race. Interestingly, most
behavior geneticists believe that no such relationships exist.
Thus Whitney astounded the association by suggesting in his
Presidential Address that race plays a role in causing murders.
He presented (in a speech at an evening banquet) reliable evi-
dence that the murder rate in the United States was significantly
higher among non-Whites than among Whites. He then said
Like it or not, it is a reasonable scientific hypothesis
that some, perhaps much, of the race difference in murder
rate is caused by genetic differences in contributory
variables such as low intelligence, lack of empathy, ag-
gressive acting out, and impulsive lack of foresight
(1995, p. 336).
The next morning Whitney was shunned at the meeting of the BGA
Executive Committee, and the committee voted (with Whitney ab-
staining) to issue an official statement denouncing his comments.
Also, the editor of the BGA journal declined (contrary to stan-
dard policy) to publish the text of the Presidential Address in
the journal (Whitney, 1995). After the meeting the incoming 1996
BGA president circulated an open letter calling Whitney’s com-
ments “nonscientific, misleading, and cruel,” and urging Whitney
to resign from the association (“Specter at the Feast,” 1995).
Whitney’s hypothesis is that race exerts a causal influence on
murder, and he was correct in saying that this hypothesis is a
“reasonable scientific hypothesis”. However, due to the possi-
bility of reasonable alternative explanations (perhaps in terms
of poverty and alienation), he erred in believing that the murder
statistics properly support the hypothesis.
In view of the error in scientific logic and in view of the taboo
against concluding that performance depends on race, the members
of the Behavior Genetics Association moved quickly to distance
themselves from Whitney’s scientifically unfounded and socially
inappropriate causal conclusion.
(A similar taboo pertains to concluding that performance in indi-
viduals depends on their sex [gender]. We [society] allow meas-
ures of physical performance to depend on sex because sufficient
obvious differences exist between the sexes in determinants of
physical performance [e.g., in average body weight] to justify
such differences. We also generally allow differences in “emo-
tional” performance between the sexes, although the distinction
may be diminishing. However, we have a justified strong social
prohibition against concluding that intellectual performance de-
pends on sex because such differences might be used by some peo-
ple as a basis for sex discrimination.)
Does Performance Not Depend on Race?
The discussion above suggests that we can’t reasonably conclude
that performance depends on race. It is instructive to consider
the negation of this idea. That is, can we conclude that per-
formance doesn’t depend on race?
Many people believe that human performance doesn’t directly de-
pend on race. (I am in this group.) However, the statement that
performance doesn’t depend on race is a statement of a scientific
“null hypothesis” -- a statement that something doesn’t exist.
(Here the null hypothesis says that no causal relationship exists
in humans between a given measure of performance and race.) It
is impossible to scientifically prove that something that is
logically possible doesn’t exist (assuming that the size of the
thing isn’t specified). Thus a null hypothesis can’t be directly
empirically supported. Thus we can’t scientifically prove that
performance doesn’t depend (in some perhaps very small way) on
race.
Despite the preceding point, scientific logic dictates (through
the principle of parsimony) that we assume that a null hypothesis
is true until (if ever) incontrovertible empirical evidence to
the contrary is brought forward. Thus (since no incontrovertible
evidence is presently available) we assume that performance
doesn’t depend on race, even though we can’t prove it is true.
Rejecting the Null Hypothesis
As a rule, scientists are highly interested in properly rejecting
null hypotheses about causal relationships between variables.
This rejection is performed by finding empirical evidence that
implies the existence of the relationship. Scientists are inter-
ested in rejecting null hypotheses because the knowledge gained
in rejecting a (carefully chosen) null hypothesis is generally of
theoretical or practical use.
However, the case of relationships between performance and race
is an important exception. In this case most scientists and
other thoughtful people are not interested in trying to reject
the null hypothesis because, as noted, rejection is not seen as
being particularly scientifically useful, and rejection might be
used by some people as a basis for racial discrimination.
Summing Up
The preceding discussion leads to a certain type of negative
(null) conclusion. A conclusion of this type is often unstated
because experienced scientists take such a conclusion for granted
until (if ever) it is rejected. However, in view of the harmful-
ness of racial discrimination, the conclusion is worth stating:
There is presently no convincing scientific evidence that per-
formance (or behavior) in individuals can be reasonably predicted
from their race or ethnicity.
Appendix D: Specifying a Repeated Measurements
Analysis of Variance
The procedure for requesting a repeated measurements analysis of
variance from a statistical analysis computer program is compli-
cated because one must understand two somewhat complicated lan-
guages:
- the language of statistical ideas related to repeated measure-
ments analysis of variance (i.e., variation, within- and be-
tween-entity variation, main effect, interaction, and p-value)
- the language of the computer program chosen to analyze the
data. (In general, each program uses a different proprietary
language to specify the required information.)
Also, requesting a repeated measurements analysis of variance is
complicated because two layouts are available for organizing the
data table, and most software is only capable of analyzing data
organized according to one of the layouts, and software manuals
sometimes don’t carefully distinguish between the layouts.
One layout for organizing the data is with one row of data per
response-variable value. For example, suppose we perform a re-
peated measurements experiment to compare teaching approach A
with teaching approach B using a measure of knowledge as the re-
sponse variable. And suppose we measure the students’ knowledge
of the subject area before they are exposed to the teaching ap-
proaches and we measure their knowledge again after each student
has had three months of exposure to one or the other of the ap-
proaches. Then our data table might be organized as follows:
--------------------------------------
Teaching
Student Approach Time Knowledge
--------------------------------------
Jack A Before 55
Jack A After 65
Mary B Before 63
Mary B After 75
Jean A Before 68
Jean A After 69
Bill B Before 49
Bill B After 82
etc.
-------------------------------------
The table indicates that Jack had a measured knowledge value of
55 before receiving teaching approach A and a measured knowledge
value of 65 after receiving teaching approach A, and so on for
the other students.
A second layout for organizing the data is with one row of data
per experimental entity, i.e., one row per student in the present
discussion. Under this layout the information in the above table
could be organized as follows:
------------------------------------------
Teaching Knowledge Knowledge
Student Approach Before After
------------------------------------------
Jack A 55 65
Mary B 63 75
Jean A 68 69
Bill B 49 82
etc.
------------------------------------------
In this second layout for organizing the data the response vari-
able (Knowledge) has multiple columns in the data table, with
these columns reflecting the repeated measurements aspect of the
research, with one column for each time the response variable was
measured in the students. Traditionally this second layout has
been used to organize repeated measurements data. (This may be
because this layout is non-redundant and thus more compact than
the first layout.) However, the first layout may be slightly
easier to understand because each variable has only a single col-
umn in the data table and no variables are hidden or implicit.
(In the table immediately above the variable Knowledge has two
columns in the table and the variable Time has no column -- time
is implied by the two Knowledge columns.) I hope that statisti-
cal software developers will debate the advantages of the two
layouts for organizing repeated measurements data and then stan-
dardize on the better layout (or perhaps make both layouts avail-
able).
It is easy for an expert to use statistical software to convert a
data table from one of the two layouts for organizing repeated
measurements data to the other. However, for a less experienced
researcher this conversion can be surprisingly difficult in the
details. Thus I recommend that less experienced researchers de-
termine which data organization their software requires and then
ensure that the data table is organized properly from the start.
References
Aiken, L. R. 2002. Attitudes and related psychosocial constructs:
Theories, assessment, and research. Thousand Oaks, CA: Sage.
Bailar, J. C., III, and Mosteller, F., eds. 1992. Medical uses of
statistics (2nd ed.). Boston: NEJM (New England Journal of
Medicine) Books.
Box, G. E. P., Hunter, J. S., and Hunter, W. G. 2005. Statistics
for experimenters (2nd ed.). New York: John Wiley.
Fleiss, J. L. 1986. The design and analysis of clinical experi-
ments. New York: John Wiley.
Kirk, R. E. 1995. Experimental design: Procedures for behavioral
sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole.
Krosnick, J. A., Judd, C. M., and Wittenbrink, B. 2005. The meas-
urement of attitudes. In D. Albarracin, B. T. Johnson, and M.
P. Zannna (Eds.), The handbook of attitudes, (pp. 21-76). Mah-
wah, NJ: Lawrence Erlbaum.
Ladson-Billings, G., and Tate, W. 2006. 2006. American Education
Research Association annual meeting theme: Education research
in the public interest. Available at
http://www.aera.net/annualmeeting/?id=694
Macnaughton, D. B. 2002. The introductory statistics course: The
entity-property-relationship approach. Available at
http://www.matstat.com/teach
McCullagh, P., and Nelder, J. A. 1989. Generalized linear models
(2nd ed.). London: Chapman and Hall.
Miller, G. A. 1956. The magical number seven, plus or minus two:
Some limits on our capacity for processing information. Psy-
chological Review 63:81-97. Also available at
http://www.well.com/~smalin/miller.html
Specter at the Feast. 1995 (July 7). Science 269:35.
Thompson, B. 2004. Exploratory and confirmatory factor analysis:
Understanding concepts and applications. Washington, DC:
American Psychological Association.
Whitney, G. 1995. Ideology and censorship in behavior genetics.
Mankind Quarterly 35:327-342. Also available at
http://www.lrainc.com/swtaboo/taboos/gw-icbg.html
Winer, B. J., Brown, D. R., and Michels, K. M. 1991. Statistical
principles in experimental design (3rd ed.). New York: McGraw-
Hill.
Home page for the Entity-Property-Relationship Approach to Introductory Statistics