EVIDENCE-BASED PRACTICE IN SCHOOL MENTAL HEALTH
OXFORD WORKSHOP SERIES:
SCHOOL SOCIAL WORK ASSOCIATION OF AMERICA
Series Advisory Board
Judith Shine, President
Susan L. Ellis, Executive Director
Randy A. Fisher
Stephen P. Hydon
Donna J. Secor
Evidence-Based Practice in School Mental Health
James C. Raines
The Domains and Demands of School Social Work Practice
Michael S. Kelly
Solution-Focused Brief Therapy in Schools
Michael S. Kelly, Johnny S. Kim, and Cynthia Franklin
EVIDENCE-BASED PRACTICE IN SCHOOL
MENTAL HEALTH
A Primer for School Social Workers,
Psychologists, and Counselors
James C. Raines
1
2008
OXFORD WORKSHOP SERIES
■■■
3
Oxford University Press, Inc., publishes works that further
Oxford University’s objective of excellence
in research, scholarship, and education.
Oxford New York
Auckland Cape Town Dar es Salaam Hong Kong Karachi

Printed in the United States of America
on acid-free paper
v
Contents
Preface vii
Chapter 1 Introduction 3
Chapter 2 Evidence-Based Practice: De nition
and Process
25
Chapter 3 Creating Answerable Questions 45
Chapter 4 Investigating the Evidence 67
Chapter 5 Appraising the Evidence 103
Chapter 6 Adapting and Applying the Evidence 133
Chapter 7 Evaluating Group Outcomes Using Descriptive
Designs
163
Chapter 8 Evaluating Individual Outcomes Using
Single-Subject Designs
197
Chapter 9 Ethics for Evidence-Based Practice 221
Chapter 10 Systemic Change 255
Glossary 277
Appendix A Internet Resources for Evidence-Based
Practice
285
vi Contents
Appendix B Annotated Bibliography on Empirically Supported
Interventions
293
Appendix C Ten Threats to Internal Validity 295

youth treatment (Durlak, Wells, Cotton, & Johnson, 1995; Kazdin, 2000).
The later studies have overcome earlier methodological weaknesses and
demonstrated signifi cantly stronger results (Weisz, 2004).
This book, however, is not about evidence-based practices, which may
be defi ned as techniques or treatments that have empirical support. This
book is about evidence-based practice or the process of continually infusing
practice with the current best research (Drake, Hovmand, Jonson-Reid, &
Zayas, 2007; Evidence-Based Medicine Working Group, 1992). Why choose
viii Preface
a process approach? Paul (2004) describes the problem when he reviews
multiple books about empirically supported treatments:
Even books published in the last year, are 3 to 4 years behind
the times. By virtue of the lag between writing and publication,
even the most recently released book will have citations that are
a few years old. Journals and conventions remain the best vehi-
cle to stay on top of the learning curve (p. 39).
Staying on top of the curve is what professional practice is all about. Each
new client brings a unique amalgam of strengths and problems, develop-
mental issues, cultural diversity, and value preferences. There are no clini-
cians who can sit back and comfortably assume that they have no more
to learn. The world is changing much too fast to believe that we can stop
growing along with it.
What’s up ahead in this book? In Chapter 1—Introduction—I provide
fi ve compelling reasons for evidence-based practice. They include ethical,
legal, clinical, educational, and economic justifi cations. I also describe the
philosophy of science that undergirds this book. In Chapter 2—Evidence-
Based Practice: Defi nition and Process—I defi ne evidence-based practice
and dispel some myths. I also describe the fi ve basic steps of evidence-based
practice. These include determining answerable questions, investigating the
evidence, appraising the evidence, adapting and applying the evidence, and

ence behind evidence-based practice (EBP). The “Reasons to Practice EBP”
section includes ethical, legal, clinical, educational, and economic reasons.
The subsection on ethical reasons examines the code of ethics for each of
the three major student service provider groups: school social workers,
school psychologists, and school counselors. Despite their differences in
training and perspective, all three groups concur about the necessity for
EBP. The subsection on legal reasons is split into two parts: case law and
federal legislation. Case law looks at three U.S. Supreme Court cases that
have clarifi ed the question, “What is scientifi c evidence?” Federal legislation
looks at the two most important laws governing both general and special
education. The No Child Left Behind Act is the latest reauthorization of
the Elementary and Secondary Education Act (1965) and has demanded
accountability from every school based on student results. The Individuals
with Disabilities Education Improvement Act of 2004 is the latest reautho-
rization of the Education of All Handicapped Children’s Act (1975) and has
Evidence-Based Practice in School Mental Health4
aimed to align the accountability mandates of general education with those
of special education. Both use the same defi nition of scientifi cally based
research provided here. The subsection on clinical reasons is divided into
“Standards of Care,” “Avoiding Harm,” and “Optimal Treatment.” “Standards
of Care” represent the expectations of an ordinary school service provider
based on similar circumstances in similar locales, while “Avoiding Harm”
addresses the iatrogenic or treatment-related damage that can occur. I provide
three true cases of children who have been killed by inept service providers.
“Optimal Treatment” looks at seven ways by which EBP can be used to pro-
vide the best care for students in our schools. The subsection “Educational
Reasons” examines the effect on learning when mental health practitio-
ners eliminate some of the barriers to learning for children with social or
emotional diffi culties. The subsection “Economic Reasons” argues that EBP
can help schools and society save time, money, and resources while still

professional literature when providing service to their clients.
Legal Reasons
Second, there is a legal mandate to practice EBP. This mandate rests on both
case law and federal legislation.
Case Law
Case law requires experts to use scientifi c support for their conclusions.
In Daubert v. Merrell Dow Pharmaceuticals (1993), two children and their
parents alleged that the children’s birth defects had been caused by the
mother’s ingestion of the antinausea drug, bendectin, while she was preg-
nant. The initial District Court found that the drug did not cause human
birth defects. The parents appealed against the verdict and produced eight
experts who claimed on the basis of their unpublished studies that ben-
dectin could have caused birth defects. Both the District Court and the
Court of Appeals determined that the parents’ experts did not meet the
1923 standard (Frye v. United States) according to which experts could use
only techniques “generally accepted” in their chosen fi eld. In 1975, however,
Congress passed the Federal Rules of Evidence (1975), which introduced
the standards of relevance and reliability—the evidence had to be relevant
to the issue at hand and experts had to be qualifi ed on the basis of reliable
foundation of scientifi c, technical, or other specialized knowledge. The two
pertinent federal rules of evidence are as follows.
Rule 401. De nition of “
Relevant
Evidence”
“Relevant evidence” means evidence having any tendency to make the exis-
tence of any fact that is of consequence to the determination of the action
more probable or less probable than it would be without the evidence.
Rule 702. Testimony by Experts
If scientifi c, technical, or other specialized knowledge will assist the
trier of fact to understand the evidence or to determine a fact in issue, a

exhaustive list, but an illustrative one that gave judges broad discretion to
screen all expert testimony.
Psychologists have been the fi rst to understand the implications of
federal case law on the practice (Youngstrom & Busch, 2000). Mental health
practitioners have to become wary of pseudoscience (Lilienfeld, Lynn,
& Lohr, 2003). Assessment techniques based on projective testing (e.g.,
Rorschach ink blot tests) and controversial diagnoses (e.g., dissociative iden-
tity disorder) have become suspect under these standards (Grove & Barden,
1999). Likewise, unsupported experimental treatments for children with
Introduction 7
attention-defi cit/hyperactivity disorder and autism should warrant extreme
caution (Reamer, 2006a; Romanczyk, Arnstein, Soorya, & Gillis, 2003;
Waschbusch & Hill, 2003). Just because desperate parents will try almost
anything for the benefi t of their children does not mean that school service
providers should join them in their quixotic quests.
Legal experts have also noticed an important difference between judges
and scientists: “Judges cannot suspend judgment until research studies have
addressed their sources of doubt” (Rothstein, 2005, p. S4). School-based
clinicians are more like judges than social scientists—they cannot suspend
judgment and wait until research catches up with their needs. Mental health
practitioners must be able to make an immediate judgment about the evi-
dence they have at their disposal. When the current state of the evidence
does not apply to the client in front of them, clinicians must use their best
judgment on the basis of what they know. Greenhalgh (2006) describes
the quandary of coping with ever-changing scientifi c evidence thus: “It is
not so much about what you have read in the past, but about how you go
about applying your knowledge appropriately and consistently in new clin-
ical situations” (p. 9).
Federal Legislation
There are also two important laws passed in the last 5 years that require

5. Ensures that experimental studies are presented in suffi cient
detail and clarity to allow for replication or, at a minimum, offer
the opportunity to build systematically on their fi ndings; and
6. Has been accepted by a peer-reviewed journal or approved by
a panel of independent experts through a comparably rigorous,
objective, and scientifi c review. (§ 300.35)
When one considers the continuum of evidence (see Figure 1.1), one
cannot help but notice that the U.S. Department of Education has clearly
chosen a standard in which randomized controlled trials and quasi-experi-
mental designs are the only type of research that can be considered for their
systematic reviews. Randomized controlled trials are experimental designs
in which participants are randomly assigned to either a no-treatment control
group or a treatment group. Measures are taken at the beginning to establish
a baseline and to ensure that both groups are relatively equal. Treatment
is carefully administered so that threats to internal validity are minimal.
Finally, the same measures are taken at the end to determine if the treat-
ment group has changed signifi cantly more than the control group. Quasi-
experimental designs are similar, but they do not use a control group. This
Clinical
wisdom
Random
controlled
trials
Qualitative
case
studies
Pretest
posttest
research
Posttest

The standard of care is what ordinary, reasonable, and prudent professionals
with similar training would do under similar circumstances (Reamer, 2006b).
Practitioners who do not meet the standard of care for clients make themselves
liable for malpractice. Gambrill (2006a) identifi es four essential elements in
professional liability. First, there has to be a fi duciary relationship—one that
involves a commitment of trust between the helper and client. Second, the
practitioner’s treatment must be below the accepted standard for the profes-
sion. Third, the client must have sustained some kind of injury (emotional,
physical, psychological, or social). Fourth, the practitioner’s substandard
treatment must be the proximate cause of the client’s injury. Where does a
professional fi nd these standards of care? Each of the school-based helping
professional associations publish and update these standards regularly.
The NASW Standards for School Social Work (National Association of Social
Workers, 2002) addresses the importance of research-infused practice in
two standards. First, “School social workers shall use research to inform
Evidence-Based Practice in School Mental Health10
practice and understand social policies related to schools” (Standard 17).
Second, “School social workers shall be able to evaluate their practice and
disseminate the fi ndings to consumers, the local education agency, the com-
munity, and the profession” (Standard 23).
The National Association of School Psychologists (2000b) also has two
standards that explicitly require practitioners to keep abreast of current
research:
School psychologists must (a) utilize current professional lit-
erature on various aspects of education and child development,
(b) translate research into practice through the problem-solving
process, and (c) use research design and statistic skills to con-
duct investigations to develop and facilitate effective services.
(Practice Guideline 1)
School psychologists (in collaboration with others) develop

assistance until the 3rd or 4th grade, losing precious years of early interven-
tion (Shaywitz et al., 1999; U.S. Department of Education, 2002a). In 2004,
Congress decided that this standard method was insuffi cient and passed the
Individuals with Disabilities Education Improvement Act (P.L. 108-446),
which allowed states to evaluate how a child responds to scientifi c research-
based intervention. This response to intervention approach does not have a
long or wide track record for identifying children with learning disabilities
(Bender, Ulmer, Baskette, & Shores, 2007; Kavale, Holdnack, & Mostert, 2006).
It has been recommended since the turn of the century (Fletcher et al., 2001;
Marston, 2001) and has primarily focused on response to reading interventions
(Lyon et al., 2001). Thus, the adoption of response to intervention refl ects what
the philosopher of science Thomas Kuhn called faith in a new paradigm:
The man who embraces a new paradigm at an early stage must
often do so in defi ance of the evidence provided by problem-
solving. He must, that is, have faith that the new paradigm will
succeed with the many large problems that confront it, knowing
only that the older paradigm has failed with a few. A decision of that
kind can only be made on faith.
Kuhn, 1970, p. 158, italics added
So what is the current “standard of care” for evaluation of children who are
suspected to have learning disabilities? The answer must be that we do not
know—the standard is evolving and this kind of uncertainty is what requires
school-based professionals to stay current with the most recent research.
Avoiding Harm
What is the worst that can happen? Consider real life examples in
Boxes 1.1, 1.2, or 1.3. Although these cases represent the extreme, children
and youth have died from a variety of mental health treatments, including
antidepressant medications (Green, 2001; Nelson, 2004; Potter, Padich,
Rudorfer, & Krishnan, 2006) or unsupervised seclusion and restraint pro-
cedures (Busch & Shore, 2000; Masters & Bellonci, 2001). The youth also

after intervention had ended. Third, the results were quite specifi c. Youth
treated with empirically supported interventions showed more progress
in the target problems than they did with their untargeted problems.
Fourth, the results were wide-ranging in two ways. There are empirically
supported treatments for a wide variety of problems—attention-defi cit dis-
orders, anxiety, conduct disorders, depression, and eating disorders, and
so forth. There are also interventions available for a wide range of ages
from preschool to late adolescence. Fifth, there are an increasing number
Introduction 13
BOX 1.1 Death by Therapy: Candace Newmaker
In April 2000, a 10-year-old girl named Candace Newmaker
underwent treatment for reactive attachment disorder at the
request of her adoptive mother. Candace’s story begins with her
removal from her birth parents by child welfare workers in North
Carolina. Her natural mother, Angie Elmore, was a rural teenage
mother with a violent husband who attempted to evade investiga-
tions into child neglect by moving to a different county. Eventually,
social services tracked them down and placed all three of their
children in foster care. Candace was placed in fi ve different foster
homes before being adopted at age six by registered nurse, Jeanne
Newmaker.
According to her adoptive mother, Candace was always
diffi cult—hitting other children and starting a fi re at home. Her
adoptive mother sought help from child psychiatrists who never
bothered to seek input from Candace’s teachers or neighbors.
The doctors diagnosed her with reactive attachment disorder and
gave Ms. Newmaker a referral for attachment therapy. Unable
to fi nd an attachment therapist in North Carolina, she was told
about a prominent leader in the fi eld, Connell Jane Watkins
(a.k.a. C. J. Cooil), who operated out of her home in Evergreen,

cerebral edema.
Prosecutors used a videotape of the incident the two therapists
made as evidence against them. The two therapists were convicted
of reckless child abuse resulting in death and sentenced to 16 years
in Colorado State Prison (Advocates for Children in Therapy, 2006;
Mercer, Sarner, & Rosa, 2003).
Questions for Discussion
What evidence was there that Candace had a “reactive 1.
attachment disorder”?
What forms of therapy exist for such a problem?2.
What were the warning signs that C. J. Watkins was not 3.
properly qualifi ed?
What should have told the therapists to stop the 4.
intervention?
Was the punishment fair?5.
BOX 1.2 Death by Restraint: Angie Arndt
Angie was a foster child taken in by Dan and Donna Pavlik. They
enrolled her in the Marriage & Family Health Services “Mikan”
program, where she made progress for 8 weeks. Then an agency
social worker recommended day treatment in order for her to get
caught up in school. She was admitted to Northwest Guidance &
Counseling’s Day Treatment clinic in Rice Lake, Wisconsin, in the
Spring of 2006. At the time, she was seven years old and weighed
56 lbs. Soon afterward, her parents noticed a negative change in
her behavior and made an appointment with the Director for June.
On May 24, Angie arrived late in the morning. While having
lunch she was reprimanded for blowing bubbles in her milk and
laughing. When she laughed again, she was taken to a “time-out”
room where she was told to sit on a hard chair. She crossed her legs
and rested her head in her lap. Since this was not exactly what she

How should seven-year-old children be disciplined?3.
Who was most at fault—the agency or the staff member?4.
Was the punishment fair?5.
BOX 1.3 Death by Medication: Rebecca Riley
Rebecca was diagnosed with attention-defi cit hyperactivity
disorder and bipolar disorder (ADHD) at 28 months by a psychi-
atrist who based her assessment of the family’s medical history,
parental descriptions of her behavior, and brief offi ce visits. Other
adults, such as preschool teachers, were never consulted. She was
prescribed both clonidine (for the ADHD) and Depakote (for the
bipolar disorder). Both medications are approved by the FDA for
adults only, though doctors have been known to prescribe them for
children as well. School teachers, the school nurse, the child’s ther-
apist, and social workers with the Massachusetts Department of
Social Services all raised concerns about the side effects of the med-
ications to no avail. The psychiatrist and the medical center where
she worked assured the other professionals that the prescriptions
(continued)