what is empirical research in education

• Tools and Resources
• Customer Services
• Original Language Spotlight
• Alternative and Non-formal Education 
• Cognition, Emotion, and Learning
• Curriculum and Pedagogy
• Education and Society
• Education, Change, and Development
• Education, Cultures, and Ethnicities
• Education, Gender, and Sexualities
• Education, Health, and Social Services
• Educational Administration and Leadership
• Educational History
• Educational Politics and Policy
• Educational Purposes and Ideals
• Educational Systems
• Educational Theories and Philosophies
• Globalization, Economics, and Education
• Languages and Literacies
• Professional Learning and Development
• Research and Assessment Methods
• Technology and Education
• Share This Facebook LinkedIn Twitter

Article contents

Evidence-based educational practice.

• Tone Kvernbekk Tone Kvernbekk University of Oslo
• https://doi.org/10.1093/acrefore/9780190264093.013.187
• Published online: 19 December 2017

Evidence-based practice (EBP) is a buzzword in contemporary professional debates, for example, in education, medicine, psychiatry, and social policy. It is known as the “what works” agenda, and its focus is on the use of the best available evidence to bring about desirable results or prevent undesirable ones. We immediately see here that EBP is practical in nature, that evidence is thought to play a central role, and also that EBP is deeply causal: we intervene into an already existing practice in order to produce an output or to improve the output. If our intervention brings the results we want, we say that it “works.”

How should we understand the causal nature of EBP? Causality is a highly contentious issue in education, and many writers want to banish it altogether. But causation denotes a dynamic relation between factors and is indispensable if one wants to be able to plan the attainment of goals and results. A nuanced and reasonable understanding of causality is therefore necessary to EBP, and this we find in the INUS-condition approach.

The nature and function of evidence is much discussed. The evidence in question is supplied by research, as a response to both political and practical demands that educational research should contribute to practice. In general, evidence speaks to the truth value of claims. In the case of EBP, the evidence emanates from randomized controlled trials (RCTs) and presumably speaks to the truth value of claims such as “if we do X, it will lead to result Y.” But what does research evidence really tell us? It is argued here that a positive RCT result will tell you that X worked where the RCT was conducted and that an RCT does not yield general results.

Causality and evidence come together in the practitioner perspective. Here we shift from finding causes to using them to bring about desirable results. This puts contextual matters at center stage: will X work in this particular context? It is argued that much heterogeneous contextual evidence is required to make X relevant for new contexts. If EBP is to be a success, research evidence and contextual evidence must be brought together.

• effectiveness
• INUS conditions
• practitioner perspective

Introduction

Evidence-based practice, hereafter EBP, is generally known as the “what works” agenda. This is an apt phrase, pointing as it does to central practical issues: how to attain goals and produce desirable results, and how we know what works. Obviously, this goes to the heart of much (but not all) of the everyday activity that practitioners engage in. The “what works” agenda is meant to narrow the gap between research and practice and be an area in which research can make itself directly useful to practice. David Hargreaves, one of the instigators of the EBP debate in education, has stated that the point of evidence-based research is to gather evidence about what works in what circumstances (Hargreaves, 1996a , 1996b ). Teachers, Hargreaves said, want to know what works; only secondarily are they interested in understanding the why of classroom events. The kind of research we talk about is meant to be relevant not only for teachers but also for policymakers, school developers, and headmasters. Its purpose is to improve practice, which largely comes down to improving student achievement. Hargreaves’s work was supported by, for example, Robert Slavin, who stated that education research not only can address questions about “what works” but also must do so (Slavin, 2004 ).

All the same, despite the fact that EBP, at least at the outset, seems to speak directly to the needs of practitioners, it has met with much criticism. It is difficult to characterize both EBP and the debate about it, but let me suggest that the debate branches off in different but interrelated directions. We may roughly identify two: what educational research can and should contribute to practice and what EBP entails for the nature of educational practice and the teaching profession. There is ample space here for different definitions, different perspectives, different opinions, as well as for some general unclarity and confusions. To some extent, advocates and critics bring different vocabularies to the debate, and to some extent, they employ the same vocabulary but take very different stances. Overall in the EBP conceptual landscape we find such concepts as relevance, effectiveness, generality, causality, systematic reviews, randomized controlled trials (RCTs), what works, accountability, competences, outcomes, measurement, practical judgment, professional experience, situatedness, democracy, appropriateness, ends, and means as constitutive of ends or as instrumental to the achievement of ends. Out of this tangle we shall carefully extract and examine a selection of themes, assumptions, and problems. These mainly concern the causal nature of EBP, the function of evidence, and EBP from the practitioner point of view.

Definition, History, and Context

The term “evidence-based” originates in medicine—evidence-based medicine—and was coined in 1991 by a group of doctors at McMaster University in Hamilton, Ontario. Originally, it denoted a method for teaching medicine at the bedside. It has long since outgrown the hospital bedside and has become a buzzword in many contemporary professions and professional debates, not only in education, but also leadership, psychiatry, and policymaking. The term EBP can be defined in different ways, broadly or more narrowly. We shall here adopt a parsimonious, minimal definition, which says that EBP involves the use of the best available evidence to bring about desirable outcomes, or conversely, to prevent undesirable outcomes (Kvernbekk, 2016 ). That is to say, we intervene to bring about results, and this practice should be guided by evidence of how well it works. This minimal definition does not specify what kinds of evidence are allowed, what “based” should mean, what practice is, or how we should understand the causality that is inevitably involved in bringing about and preventing results. Minimal definitions are eminently useful because they are broad in their phenomenal range and thus allow differing versions of the phenomenon in question to fall under the concept.

We live in an age which insists that practices and policies of all kinds be based on research. Researchers thus face political demands for better research bases to underpin, inform and guide policy and practice, and practitioners face political demands to make use of research to produce desirable results or improve results already produced. Although the term EBP is fairly recent, the idea that research should be used to guide and improve practice is by no means new. To illustrate, in 1933 , the School Commission of Norwegian Teacher Unions (Lærerorganisasjonenes skolenevnd, 1933 ) declared that progress in schooling can only happen through empirical studies, notably, by different kinds of experiments and trials. Examples of problems the commission thought research should solve are (a) in which grade the teaching of a second language should start and (b) what the best form of differentiation is. The accumulated evidence should form the basis for policy, the unions argued. Thus, the idea that pedagogy should be based on systematic research is not entirely new. What is new is the magnitude and influence of the EBP movement and other, related trends, such as large-scale international comparative studies (e.g., the Progress in International Reading Literacy Study, PIRLS, and the Programme for International Student Assessment, PISA). Schooling is generally considered successful when the predetermined outcomes have been achieved, and education worldwide therefore makes excessive requirements of assessment, measurement, testing, and documentation. EBP generally belongs in this big picture, with its emphasis on knowing what works in order to maximize the probability of attaining the goal. What is also new, and quite unprecedented, is the growth of organizations such as the What Works Clearinghouses, set up all around the world. The WWCs collect, review, synthesize, and report on studies of educational interventions. Their main functions are, first, to provide hierarchies that rank evidence. The hierarchies may differ in their details, but they all rank RCTs, meta-analyses, and systematic reviews on top and professional judgment near the bottom (see, e.g., Oancea & Pring, 2008 ). Second, they provide guides that offer advice about how to choose a method of instruction that is backed by good evidence; and third, they serve as a warehouse, where a practitioner might find methods that are indeed backed by good evidence (Cartwright & Hardie, 2012 ).

Educationists today seem to have a somewhat ambiguous relationship to research and what it can do for practice. Some, such as Robert Slavin ( 2002 ), a highly influential educational researcher and a defender of EBP, think that education is on the brink of a scientific revolution. Slavin has argued that over time, rigorous research will yield the same step-by-step, irreversible progress in education that medicine has enjoyed because all interventions would be subjected to strict standards of evaluation before being recommended for general use. Central to this optimism is the RCT. Other educationists, such as Gert Biesta ( 2007 , 2010 ), also a highly influential figure in the field and a critic of EBP, are wary of according such weight to research and to the advice guides and practical guidelines of the WWCs for fear that this might seriously restrict, or out and out replace, the experience and professional judgment of practitioners. And there matters stand: EBP is a huge domain with many different topics, issues, and problems, where advocates and critics have criss-crossing perspectives, assumptions, and value stances.

The Causal Nature of Evidence-Based Practice

As the slogan “what works” suggests, EBP is practical in nature. By the same token, EBP is also deeply causal. Works is a causal term, as are intervention, effectiveness , bring about , influence , and prevent . In EBP we intervene into an already existing practice in order to change its outcomes in what we judge to be a more desirable direction. To say that something (an intervention) works is roughly to say that doing it yields the outcomes we want. If we get other results or no results at all, we say that it does not work. To put it crudely, we do X, and if it leads to some desirable outcome Y, we judge that X works. It is the ambition of EBP to provide knowledge of how intervention X can be used to bring about or produce Y (or improvements in Y) and to back this up by solid evidence—for example, how implementing a reading-instruction program can improve the reading skills of slow or delayed readers, or how a schoolwide behavioral support program can serve to enhance students’ social skills and prevent future problem behavior. For convenience, I adopt the convention of calling the cause (intervention, input) X and the effect (result, outcome, output) Y. This is on the explicit understanding that both X and Y can be highly complex in their own right, and that the convention, as will become clear, is a simplification.

There can be no doubt that EBP is causal. However, the whole issue of causality is highly contentious in education. Many educationists and philosophers of education have over the years dismissed the idea that education is or can be causal or have causal elements. In EBP, too, this controversy runs deep. By and large, advocates of EBP seem to take for granted that causality in the social and human realm simply exists, but they tend not to provide any analysis of it. RCTs are preferred because they allow causal inferences to be made with a high degree of certainty. As Slavin ( 2002 ) put it, “The experiment is the design of choice for studies that seek to make causal conclusions, and particularly for evaluations of educational innovations” (p. 18). In contrast, critics often make much of the causal of nature of EBP, since for many of them this is reason to reject EBP altogether. Biesta is a case in point. For him and many others, education is a moral and social practice and therefore non causal. According to Biesta ( 2010 ):

The most important argument against the idea that education is a causal process lies in the fact that education is not a process of physical interaction but a process of symbolic or symbolically mediated interaction. (p. 34)

Since education is noncausal and EBP is causal, on this line of reasoning, it follows that EBP must be rejected—it fundamentally mistakes the nature of education.

Such wholesale dismissals rest on certain assumptions about the nature of causality, for example, that it is deterministic, positivist, and physical and that it essentially belongs in the natural sciences. Biesta, for example, clearly assumes that causality requires a physical process. But since the mid-1900s our understanding of causality has witnessed dramatic developments; arguably the most important of which is its reformulation in probabilistic terms, thus making it compatible with indeterminism. A quick survey of the field reveals that causality is a highly varied thing. The concept is used in different ways in different contexts, and not all uses are compatible. There are several competing theories, all with counterexamples. As Nancy Cartwright ( 2007b ) has pointed out, “There is no single interesting characterizing feature of causation; hence no off-the-shelf or one-size-fits-all method for finding out about it, no ‘gold standard’ for judging causal relations” (p. 2).

The approach to causality taken here is twofold. First, there should be room for causality in education; we just have to be very careful how we think about it. Causality is an important ingredient in education because it denotes a dynamic relationship between factors of various kinds. Causes make their effects happen; they make a difference to the effect. Causality implies change and how it can be brought about, and this is something that surely lies at the heart of education. Ordinary educational talk is replete with causal verbs, for example, enhance, improve, reduce, increase, encourage, motivate, influence, affect, intervene, bring about, prevent, enable, contribute. The short version of the causal nature of education, and so EBP, is therefore that EBP is causal because it concerns the bringing about of desirable results (or the preventing of undesirable results). We have a causal connection between an action or an intervention and its effect, between X and Y. The longer version of the causal nature of EBP takes into account the many forms of causality: direct, indirect, necessary, sufficient, probable, deterministic, general, actual, potential, singular, strong, weak, robust, fragile, chains, multiple causes, two-way connections, side-effects, and so on. What is important is that we adopt an understanding of causality that fits the nature of EBP and does not do violence to the matter at hand. That leads me to my second point: the suggestion that in EBP causes are best understood as INUS conditions.

The understanding of causes as INUS conditions was pioneered by the philosopher John Mackie ( 1975 ). He placed his account within what is known as the regularity theory of causality. Regularity theory is largely the legacy of David Hume, and it describes causality as the constant conjunction of two entities (cause and effect, input and output). Like many others, Mackie took (some version of) regularity theory to be the common view of causality. Regularities are generally expressed in terms of necessity and sufficiency. In a causal law, the cause would be held to be both necessary and sufficient for the occurrence of the effect; the cause would produce its effect every time; and the relation would be constant. This is the starting point of Mackie’s brilliant refinement of the regularity view. Suppose, he said, that a fire has broken out in a house, and that the experts conclude that it was caused by an electrical short circuit. How should we understand this claim? The short circuit is not necessary, since many other events could have caused the fire. Nor is it sufficient, since short circuits may happen without causing a fire. But if the short circuit is neither necessary nor sufficient, then what do we mean by saying that it caused the fire? What we mean, Mackie ( 1975 ) suggests, is that the short circuit is an INUS condition: “an insufficient but necessary part of a condition which is itself unnecessary but sufficient for the result” (p. 16), INUS being an acronym formed of the initial letters of the italicized words. The main point is that a short circuit does not cause a fire all by itself; it requires the presence of oxygen and combustible material and the absence of a working sprinkler. On this approach, therefore, a cause is a complex set of conditions, of which some may be positive (present), and some may be negative (absent). In this constellation of factors, the event that is the focus of the definition (the insufficient but necessary factor) is the one that is salient to us. When we speak of an event causing another, we tend to let this factor represent the whole complex constellation.

In EBP, our own intervention X (strategy, method of instruction) is the factor we focus on, the factor that is salient to us, is within our control, and receives our attention. I propose that we understand any intervention we implement as an INUS condition. Then it immediately transpires that X not only does not bring about Y alone, but also that it cannot do so.

Before inquiring further into interventions as INUS -conditions, we should briefly characterize causality in education more broadly. Most causal theories, but not all of them, understand causal connections in terms of probability—that is, causing is making more likely. This means that causes sometimes make their effects happen, and sometimes not. A basic understanding of causality as indeterministic is vitally important in education, for two reasons. First, because the world is diverse, it is to some extent unpredictable, and planning for results is by no means straightforward. Second, because we can here clear up a fundamental misunderstanding about causality in education: causality is not deterministic and the effect is therefore not necessitated by the cause. The most common phrase in causal theory seems to be that causes make a difference for the effect (Schaffer, 2007 ). We must be flexible in our thinking here. One factor can make a difference for another factor in a great variety of ways: prevent it, contribute to it, enhance it as part of a causal chain, hinder it via one path and increase it via another, delay it, or produce undesirable side effects, and so on. This is not just conceptual hair-splitting; it has great practical import. Educational researchers may tell us that X causes Y, but what a practitioner can do with that knowledge differs radically if X is a potential cause, a disabler, a sufficient cause, or the absence of a hindrance.

Interventions as INUS Conditions

Human affairs, including education, are complex, and it stands to reason that a given outcome will have several sources and causes. While one of the factors in a causal constellation is salient to us, the others jointly enable X to have an effect. This enabling role is eminently generalizable and crucial to understanding how interventions bring about their effects. As Mackie’s example suggests, enablers may also be absences—that is vital to note, since absences normally go under our radar.

The term “intervention” deserves brief mention. To some it seems to denote a form of practice that is interested only (or mainly) in producing measurable changes on selected output variables. It is not obvious that there is a clear conception of intervention in EBP, but we should refrain from imposing heavy restrictions on it. I thus propose to employ the broad understanding suggested by Peter Menzies and Huw Price ( 1993 )—namely, interventions as a natural part of human agency. We all have the ability to intervene in the world and influence it; that is, to act as agents. Educational interventions may thus take many forms and encompass actions, strategies, programs and methods of instruction. Most interventions will be composites consisting of many different activities, and some, for instance, schoolwide behavioral programs, are meant to run for a considerable length of time.

When practitioners consider implementing an intervention X, the INUS approach encourages them to also consider what the enabling conditions are and how they might allow X to produce Y (or to contribute to its production). Our general knowledge of house fires and how they start prompts us to look at factors such as oxygen, materials, and fire extinguishers. In other cases, we might not know what the enabling conditions are. Suppose a teacher observes that some of his first graders are reading delayed. What to do? The teacher may decide to implement what we might call “Hatcher’s method” (Hatcher et al., 2006 ). This “method” focuses on letter knowledge, single-word reading, and phoneme awareness and lasts for two consecutive 10-week periods. Hatcher and colleagues’ study showed that about 75% of the children who received it made significant progress. So should our teacher now simply implement the method and expect the results with his own students to be (approximately) the same? As any teacher knows, what worked in one context might not work in another context. What we can infer from the fact that the method, X, worked where the data were collected is that a sufficient set of support factors were present to enable X to work. That is, Hatcher’s method serves as an INUS condition in a larger constellation of factors that together are sufficient for a positive result for a good many of the individuals in the study population. Do we know what the enabling factors are—the factors that correspond to presence of oxygen and inflammable material and absence of sprinkler in Mackie’s example? Not necessarily. General educational knowledge may tell us something, but enablers are also contextual. Examples of possible enablers include student motivation, parental support (important if the method requires homework), adequate materials, a separate room, and sufficient time. Maybe the program requires a teacher’s assistant? The enablers are factors that X requires to bring about or improve Y; if they are missing, X might not be able to do its work.

Understanding X as an INUS condition adds quite a lot of complexity to the simple X–Y picture and may thus alleviate at least some of the EBP critics’ fear that EBP is inherently reductionist and oversimplified. EBP is at heart causal, but that does not entail a deterministic, simplistic or physical understanding. Rather, I have argued, to do justice to EBP in education its causal nature must be understood to be both complex and sophisticated. We should also note here that X can enter into different constellations. The enablers in one context need not be the same as the enablers in another context. In fact, we should expect them to be different, simply because contexts are different.

Evidence and Its Uses

Evidence is an epistemological concept. In its immediate surroundings we find such concepts as justification, support, hypotheses, reasons, grounds, truth, confirmation, disconfirmation, falsification, and others. It is often unclear what people take evidence and its function to be. In epistemology, evidence is that which serves to confirm or disconfirm a hypothesis (claim, belief, theory; Achinstein, 2001 ; Kelly, 2008 ). The basic function of evidence is thus summed up in the word “support”: evidence is something that stands in a relation of support (confirmation, disconfirmation) to a claim or hypothesis, and provides us with good reason to believe that a claim is true (or false). The question of what can count as evidence is the question of what kind of stuff can enter into such evidential relations with a claim. This question is controversial in EBP and usually amounts to criticism of evidence hierarchies. The standard criticisms are that such hierarchies unduly privilege certain forms of knowledge and research design (Oancea & Pring, 2008 ), undervalue the contribution of other research perspectives (Pawson, 2012 ), and undervalue professional experience and judgment (Hammersley, 1997 , 2004 ). It is, however, not of much use to discuss evidence in and of itself—we must look at what we want evidence for . Evidence is that which can perform a support function, including all sorts of data, facts, personal experiences, and even physical traces and objects. In murder mysteries, bloody footprints, knives, and witness observations count as evidence, for or against the hypothesis that the butler did it. In everyday life, a face covered in ice cream is evidence of who ate the dessert before dinner.

There are three important things to keep in mind concerning evidence. First, in principle, many different entities can play the role of evidence and enter into an evidentiary relation with a claim (hypothesis, belief). Second, what counts as evidence in each case has everything to do with the type of claim we are interested in. If we want evidence that something is possible, observation of one single instance is sufficient evidence. If we want evidence for a general claim, we at least need enough data to judge that the hypothesis has good inductive support. If we want to bolster the normative conclusion that means M1 serves end E better than means M2, we have to adduce a range of evidences and reasons, from causal connections to ethical considerations (Hitchcock, 2011 ). If we want to back up our hypothesis that the butler is guilty of stealing Lady Markham’s necklace, we have to take into consideration such diverse pieces of evidence as fingerprints, reconstructed timelines, witness observations and alibis. Third, evidence comes in different degrees of trustworthiness, which is why evidence must be evaluated—bad evidence cannot be used to support a hypothesis and does not speak to its truth value; weak evidence can support a hypothesis and speak to its truth value, but only weakly.

The goal in EBP is to find evidence for a causal claim. Here we meet with a problem, because causal claims come in many different shapes: for example, “X leads to Y,” “doing X sometimes leads to Y and sometimes to G,” “X contributes moderately to Y” and “given Z, X will make a difference to Y.” On the INUS approach the hypothesis is that X, in conjunction with a suitable set of support factors, in all likelihood will lead to Y (or will contribute positively to Y, or make a difference to the bringing about of Y). The reason why RCTs are preferred is precisely that we are dealing with causal claims. Provided that the RCT design satisfies all requirements, it controls for confounders, and makes it possible to distinguish correlations from causal connections and to draw causal inferences with a high degree of confidence. In RCTs we compare two groups, the study group and the control group. Random assignment is supposed to ensure that the groups have the same distribution of causal and other factors, save one—namely, the intervention X (but do note that the value of randomization has recently been problematized, most notably by John Worrall ( 2007 ). The standard result from an RCT is a treatment effect, expressed in terms of an effect size. An effect size is a statistical measure denoting average effect in the treatment group minus average effect in the control group (to simplify). We tend to assume that any difference between the groups requires a causal explanation. Since other factors and confounders are (assumed to be) evenly distributed and thus controlled for, we infer that the treatment, whatever it is, is the cause of the difference. Thus, the evidence-ranking schemes seem to have some justification, despite Cartwright’s insistence that there is no gold standard for drawing causal inferences. We want evidence for causal claims, and RCTs yield highly trustworthy evidence and, hence, give us good reason to believe the causal hypothesis. In most cases the causal hypothesis is of the form “if we do X it will lead to Y.”

Effectiveness

Effectiveness is much sought after in EBP. For example, Philip Davies ( 2004 ) describes the role of the Campbell Collaboration as helping both policymakers and practitioners make good decisions by providing systematic reviews of the effectiveness of social and behavioral interventions in education. The US Department of Education’s Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide ( 2003 ) provides an example of how evidence, evidence hierarchies, effectiveness, and “what works” are tied together. The aim of the guide is to provide practitioners with the tools to distinguish practices that are supported by rigorous evidence from practices that are not. “Rigorous evidence” is here identical to RCT evidence, and the guide devotes an entire chapter to RCTs and why they yield strong evidence for the effectiveness of some intervention. Thus:

The intervention should be demonstrated effective, through well-designed randomized controlled trials, in more than one site of implementation;

These sites should be typical school or community settings, such as public school classrooms taught by regular teachers; and

The trials should demonstrate the intervention’s effectiveness in school settings similar to yours, before you can be confident that it will work in your schools/classrooms (p. 17).

Effectiveness is clearly at the heart of EBP, but what does it really mean? “Effectiveness” is a complex multidimensional concept containing causal, normative, and conceptual dimensions, all of which have different sides to them. Probabilistic causality comes in two main versions, one concerning causal strength and one concerning causal frequency or tendency (Kvernbekk, 2016 ). One common interpretation says that effectiveness concerns the relation between input and output—that is, the degree to which an intervention works. Effect sizes would seem to fall into this category, expressing as they do the magnitude of the effect and thereby the strength of the cause.

But a large effect size is not the only thing we want; we also want the cause to make its effect happen regularly across different contexts. In other words, we are interested in frequency . A cause may not produce its effect every time but often enough to be of interest. If we are to be able to plan for results, X must produce its effect regularly. Reproducibility of desirable results thus depends crucially of the tendency of the cause to produce its effect wherever and whenever it appears. Hence, the term “effectiveness” signals generality. In passing, the same generality hides in the term “works”—if an intervention works, it can be relied on to produce its desired results wherever it is implemented. The issue of scope also belongs to this generality picture: for which groups do we think our causal claim holds? All students of a certain kind, for example, first graders who are responsive to extra word and phoneme training? Some first graders somewhere in the world? All first graders everywhere?

The normative dimension of “what works,” or effectiveness, is equally important, also because it demonstrates so well that effectiveness is a judgment we make. We sometimes gauge effectiveness by the relation between desired output and actual output; that is, if the correlation between the two is judged to be sufficiently high, we conclude that the method of instruction in question is effective. In such cases, the result (actual or desired) is central to our judgment, even if the focus of EBP undeniably lies on the means, and not on the goals. In a similar vein, to conclude that X works, you must judge the output to be satisfactory (enough), and that again depends on which success criteria you adopt (Morrison, 2001 ). Next, we have to consider the temporal dimension: how long must an effect linger for us to judge that X works? Three weeks? Two months? One year? Indefinitely? Finally, there is a conceptual dimension to judgments of effectiveness: the judgment of how well X works also depends on how the target is defined. For example, an assessment of the effectiveness of reading-instruction methods depends on what it means to say that students can read. Vague target articulations give much leeway for judgments of whether the target (Y) is attained, which, in turn, opens the possibility that many different Xs are judged to lead to Y.

Given the different dimensions of the term “effectiveness,” we should not wonder that effectiveness claims often equivocate on whether they mean effectiveness in terms of strength or frequency or perhaps both. The intended scope is often unclear, the target may be imprecise and the success criteria too broad or too narrow or left implicit altogether. However, since reproducibility of results is vitally important in EBP, it stands to reason that generality—external validity—should be of the greatest interest. All the strategies Dean, Hubbell, Pitler, and Stone ( 2012 ) discuss in their book about classroom instruction that works are explicitly general, for example, that providing feedback on homework assignments will benefit students and help enhance their achievements. This is generality in the frequency and (large) scope sense. It is future oriented: we expect interventions to produce much the same results in the future as they did in the past, and this makes planning possible.

The evidence for general causal claims is thought to emanate from RCTs, so let us turn again to RCTs to see whether they supply us with evidence that can support such claims. It would seem that we largely assume that they do. The Department of Education’s guide, as we have seen, presupposes that two RCTs are sufficient to demonstrate general effectiveness. Keith Morrison ( 2001 ) thinks that advocates of EBP simply assume that RCTs ensure generalizability, which is, of course, exactly what one wants in EBP—if results are generalizable, we may assume that the effect travels to other target populations so that results are reproducible and we can plan for their attainment. But does RCT evidence tell us that a cause holds widely? No, Cartwright ( 2007a ) argued, RCTs require strong premises, and strong premises do not hold widely. Because of design restrictions, RCT results hold formally for the study group (the sample) and only for that group, she insists. Methods that are strong on internal validity are correspondingly weak on external validity. RCTs establish efficacy, not effectiveness. We tend to assume without question, Cartwright argues, that efficacy is evidence for effectiveness. But we should not take this for granted—either it presumes that the effect depends exclusively on the intervention and not on who receives it, or it relies on presumed commonalities between the study group and the target group. This is a matter of concern to EBP and its advocates, because if Cartwright is correct, RCT evidence does not tell us what we think it tells us. Multiple RCTs will not solve this problem; the weakness of enumerative induction—inferences from single instances to a general conclusion—is well known. So how then can we ground our expectation that results are reproducible and can be planned for?

The Practitioner Perspective

EBP, as it is mostly discussed, is researcher centered. The typical advice guides, such as that of the What Works Clearinghouse, tend to focus on the finding of causes and the quality of the evidence produced. Claims and interventions should be rigorously tested by stringent methods such as RCTs and ranked accordingly. The narrowness of the kind of evidence thus admitted (or preferred) is pointed out by many critics, but it is of equal importance that the kind of claims RCT evidence is evidence for is also rather narrow. Shifting the focus from research to practice significantly changes the game. And bring in the practitioners we must—EBP is eminently practical in nature, concerning as it does the production of desirable results. Putting practice center stage means shifting from finding causes and assessing the quality of research evidence to using causes to produce change. In research we can control for confounders and keep variables fixed. In practice we can do no such thing; hence the significant change of the game.

The claim a practitioner wants evidence for is not the same claim that a researcher wants evidence for. The researcher wants evidence for a causal hypothesis, which we have seen can be of many different kinds, for example, the contribution of X to Y. The practitioner wants evidence for a different kind of claim—namely, whether X will contribute positively to Y for his students, in his context. This is the practitioner’s problem: the evidence that research provides, rigorous as it may be, does not tell him whether a proposed intervention will work here , for this particular target group. Something more is required.

Fidelity is a demand for faithfulness in implementation: if you are to implement an intervention that is backed by, say, two solid RCTs, you should do it exactly as it was done where the evidence was collected. The minimal definition of EBP adopted here leaves it open whether fidelity should be included or not, but there can be no doubt that both advocates and critics take it that it is—making fidelity one of the most controversial issues in EBP. The advocate argument centers on quality of implementation (e.g., Arnesen, Ogden, & Sørlie, 2006 ). It basically says that if X is implemented differently than is prescribed by researchers or program developers, we can no longer know exactly what it is that works. If unfaithfully implemented, the intervention might not produce the expected results, and the program developers cannot be held responsible for the results that do obtain. Failure to obtain the expected results is to be blamed on unsystematic or unfaithful implementation of a program, the argument goes. Note that the results are described as expected.

The critics, on the other hand, interpret fidelity as an attempt to curb the judgment and practical knowledge of the teachers; perhaps even as an attempt to replace professional judgment with research evidence. Biesta ( 2007 ), for example, argues that in the EBP framework the only thing that remains for practitioners to do is to follow rules for action. These rules are thought to be somehow directly derived from the evidence. Biesta is by no means the only EBP critic to voice this criticism; we find the same view in Bridges, Smeyers, and Smith ( 2008 ):

The evidence-based policy movement seems almost to presuppose an algorithm which will generate policy decisions: If A is what you want to achieve and if research shows R1, R2 and R3 to be the case, and if furthermore research shows that doing P is positively correlated with A, then it follows that P is what you need to do. So provided you have your educational/political goals sorted out, all you need to do is slot in the appropriate research findings—the right information—to extract your policy. (p. 9)

No consideration of the concrete situation is deemed necessary, and professional judgment therefore becomes practically superfluous. Many critics of EBP make the same point: teaching should not be a matter of following rules, but a matter of making judgments. If fidelity implies following highly scripted lessons to the letter, the critics have a good point. If fidelity means being faithful to higher level principles, such as “provide feedback on home assignments,” it becomes more open and it is no longer clear exactly what one is supposed to be faithful to, since feedback can be given in a number of ways. We should also note here that EBP advocates, for example, David Hargreaves ( 1996b ), emphatically insist that evidence should enhance professional judgment, not replace it. Let us also note briefly the usage of the term “evidence,” since it deviates from the epistemological usage of the term. Biesta (and other critics) picture evidence as something from which rules for action can be inferred. But evidence is (quantitative) data that speak to the truth value of a causal hypothesis, not something from which you derive rules for action. Indeed, the word “based” in evidence-based practice is misleading—practice is not based on the RCT evidence; it is based on the hypothesis (supposedly) supported by the evidence. Remember that the role of evidence can be summed up as support . Evidence surely can enhance judgment, although EBP advocates tend to be rather hazy about how this is supposed to happen, especially if they also endorse the principle of fidelity.

Contextual Matters

If we hold that causes are easily exportable and can be relied on to produce their effect across a variety of different contexts, we rely on a number of assumptions about causality and about contexts. For example, we must assume that the causal X–Y relation is somehow basic, that it simply holds in and of itself. This assumption is easy to form; if we have conducted an RCT (or several, and pooled the results in a meta-analysis) and found a relation between an intervention and an effect of a decent magnitude, chances are that we conclude that this relation simply exists. Causal relations that hold in and of themselves naturally also hold widely; they are stable, and the cause can be relied on as sufficient to bring about its effect most of the time, in most contexts, if not all. This is a very powerful set of assumptions indeed—it underpins the belief that desirable results are reproducible and can be planned for, which is exactly what not only EBP wants but what practical pedagogy wants and what everyday life in general runs on.

The second set of assumptions concerns context. The US Department of Education guide ( 2003 ) advises that RCTs should demonstrate the intervention’s effectiveness in school settings similar to yours, before you can be confident that it will work for you. The guide provides no information about what features should be similar or how similar those features should be; still, a common enough assumption is hinted at here: if two contexts are (sufficiently) similar (on the right kind of features) the cause that worked in one will also work in the other. But as all teachers know, students are different, teachers are different, parents are different, headmasters are different, and school cultures are different. The problem faced by EBP is how deep these differences are and what they imply for the exportability of interventions.

On the view taken here, causal relations are not general, not basic, and therefore do not hold in and of themselves. Causal relations are context dependent, and contexts should be expected to be different, just as people are different. This view poses problems for the practitioner, because it means that an intervention that is shown by an RCT to work somewhere (or in many somewheres ) cannot simply be assumed to work here . Using causes in practice to bring about desirable changes is very different from finding them, and context is all-important (Cartwright, 2012 ).

All interventions are inserted into an already existing practice, and all practices are highly complex causal/social systems with many factors, causes, effects, persons, beliefs, values, interactions and relations. This system already produces an output Y; we are just not happy with it and wish to improve it. Suppose that most of our first graders do learn to read, but that some are reading delayed. We wish to change that, so we consider whether to implement Hatcher’s method. We intervene by changing the cause that we hold to be (mainly) responsible for Y—namely, X—or we implement a brand-new X. But when we implement X or change it from x i to x j (shifting from one method of reading instruction to another), we generally thereby also change other factors in the system (context, practice), not just the ones causally downstream from X. We might (inadvertently) have changed both A, B, and C—all of which may have an effect on Y. Some of these contextual changes might reinforce the effect of X; others might counteract it. For example, in selecting the group of reading-delayed children for special treatment, we might find that we change the interactional patterns in the class, and that we change the attitudes of parents toward their children’s education and toward the teacher or the school. With the changes to A, B, and C, we are no longer in system g but in system h . The probability of Y might thereby change; it might increase or it might decrease. Hence, insofar as EBP focuses exclusively on the X–Y relation, natural as this is, it tells only half the story. If we take the context into account, it transpires that if X is going to be an efficacious strategy for changing (bringing about, enhancing, improving, preventing, reducing) Y, then it is not the relation between X and Y that matters the most. What matters instead is that the probability of Y given X-in-conjunction-with-system is higher than the probability of Y given not-X-in-conjunction-with-system. But what do we need to know to make such judgments?

Relevance and Evidence

On the understanding of EBP advanced here, fidelity is misguided. It rests on causal assumptions that are at least problematic; it fails to distinguish between finding causes and using causes; and it fails to pay proper attention to contextual matters.

What, then, should a practitioner look for when trying to make a decision about whether to implement X or not? X has worked somewhere ; that has been established by RCTs. But when is the fact that X has worked somewhere relevant to a judgment that X will also work here ? If the world is diverse, we cannot simply export a causal connection, insert it into a different context, and expect it to work there. The practitioner will need to gather a lot of heterogeneous evidence, put it together, and make an astute all-things-considered judgment about the likelihood that X will bring about the desired results here were it to be implemented. The success of EBP depends not only on rigorous research evidence but also on the steps taken to use an intervention to bring about desirable changes in a context where the intervention is as yet untried.

What are the things to be considered for an all-things-considered decision about implementing X? First, the practitioner already knows that X has worked somewhere ; the RCT evidence tells him or her that. Thus, we do know that X played a positive causal role for many of the individuals in the study group (but not necessarily all of them; effect sizes are aggregate results and thus compatible with negative results for some individuals).

Second, the practitioner must think about how the intervention might work if it were implemented. RCTs run on an input–output logic and do not tell us anything about how the cause is thought to bring about its effect. But a practitioner needs to ask whether X can play a positive causal role in his or her context, and then the question to ask is how , rather than what .

Third, given our understanding of causes as INUS conditions, the practitioner will have to map the contextual factors that are necessary for X to be able to do its work and bring about Y. What are the enabling factors? If they are not present, can they be easily procured? Do they outweigh any disabling factors that may be present? It is important to remember that enablers may be absences of hindrances. Despite their adherence to the principle of fidelity, Arnesen, Ogden, and Sørlie ( 2006 ) acknowledge the importance of context for bringing about Y. For example, they point out that there must be no staff conflicts if the behavioural program is to work. Such conflicts would be a contextual disabler, and their absence is necessary. If you wish to implement Hatcher’s method, you have to look at your students and decide whether you think this will suit them, whether they are motivated, and how they might interact with the method and the materials. As David Olson ( 2004 ) points out, the effect of an intervention depends on how it is “taken” or understood by the learner. But vital contextual factors also include mundane things such as availability of adequate materials, whether the parents will support and help if the method requires homework, whether you have a suitable classroom and sufficient extra time, whether a teacher assistant is available, and so on. Hatcher’s method is the INUS condition, the salient factor, but it requires a contextual support team to be able to do its work.

Fourth, the practitioner needs to have some idea of how the context might change as a result of implementing X. Will it change the interactions among the students? Create jealousy? Take resources meant for other activities? The stability of the system into which an intervention is inserted is generally of vital importance for our chances of success. If the system is shifting and unstable X may never be able to make its effect happen. The practitioner must therefore know what the stabilizing factors are and how to control them (assuming they are within his or her control).

In sum, the INUS approach to causality and the all-important role of contextual factors and the target group members themselves in bringing about results strongly suggest that fidelity is misguided. The intervention is not solely responsible for the result; one has to take both the target group (whatever the scope) and contextual factors into consideration. On the other hand, similarity of contexts loses its significance because an intervention that worked somewhere can be made to be relevant here—there is no reason to assume that one needs exactly the same contextual support factors. The enablers that made X work there need not the same enablers that will make X work here . What is important is that the practitioner carefully considers how X can be made to work in his or her context.

EBP is a complex enterprise. The seemingly simple question of using the best available evidence to bring about desirable results and prevent undesirable ones branches out in different directions to involve problems concerning what educational research can and should contribute to practice, the nature of teaching, what kind of knowledge teachers need, what education should be all about, how we judge what works, the role of context and the exportability of interventions, what we think causality is, and so on. We thus meet both ontological, epistemological, and normative questions.

It is important to distinguish between the evidence and the claim which it is evidence for . Evidence serves to support (confirm, disconfirm) a claim, and strictly speaking practice is based on claims, not on evidence. Research evidence (as well as everyday types of evidence) should always be evaluated for its trustworthiness, its relevance, and its scope.

EBP as it is generally discussed emphasizes research at the expense of practice. The demands of rigor made on research evidence are very high. There is a growing literature on implementation and a growing understanding of the importance of quality of implementation, but insofar as this focuses on fidelity, it is misguided. Fidelity fails to take into account the diversity of the world and the importance of the context into which an intervention is to be inserted. It is argued here that implementation centers on the matter of whether an intervention will work here and that a reasonable answer to that question requires much local, heterogeneous evidence. The local evidence concerning target group and context must be provided by the practitioner. The research evidence tells only part of the story.

If EBP is to be a success, the research story and the local-practice story must be brought together, and this is the practitioner’s job. The researcher does not know what is relevant in the concrete context faced by the practitioner; that is for the practitioner to decide.

EBP thus demands much knowledge, good thinking, and astute judgments by practitioners.

As a recommendation for future research, I would suggest inquiries into how the research story and the contextual story come together; how practitioners understand the causal systems they work within, how they understand effectiveness, and how they adapt or translate generalized guidelines into concrete local practice.

• Achinstein, P. (2001). The book of evidence . Oxford: Oxford University Press.
• Arnesen, A. , Ogden, T. , & Sørlie, M.-A. (2006). Positiv atferd og støttende læringsmiljø i skolen . Oslo: Universitetsforlaget.
• Biesta, G. (2007). Why “what works” won’t work: Evidence-based practice and the democratic deficit in educational research. Educational Theory , 57 , 1–22.
• Biesta, G. (2010). Good education in an age of measurement: Ethics, politics, democracy . Boulder, CO: Paradigm.
• Bridges, D. , Smeyers, P. , & Smith, R. (2008). Educational research and the practical judgment of policy makers. Journal of Philosophy of Education , 42 (Suppl. 1), 5–11.
• Cartwright, N. (2007a). Are RCTs the gold standard? BioSocieties , 2 , 11–20.
• Cartwright, N. (2007b). Hunting causes and using them: Approaches in philosophy and e conomics . Cambridge, U.K.: Cambridge University Press.
• Cartwright, N. (2012). Will this policy work for you? Predicting effectiveness better: How philosophy helps. Philosophy of Science , 79 , 973–989.
• Cartwright, N. , & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better . Oxford: Oxford University Press.
• Davies, P. (2004). Systematic reviews and the Campbell Collaboration. In G. Thomas & R. Pring (Eds.), Evidence-based practice in education (pp. 21–33). Maidenhead, U.K.: Open University Press.
• Dean, C. B. , Hubbell, E. R. , Pitler, H. , & Stone, Bj. (2012). Classroom instruction that works: Research-based strategies for increasing student achievement (2d ed.). Denver, CO: Mid-continent Research for Education and Learning.
• Hammersley, M. (1997). Educational research and teaching: A response to David Hargreaves’ TTA lecture. British Educational Research Journal , 23 , 141–161.
• Hammersley, M. (2004). Some questions about evidence-based practice in education. In G. Thomas & R. Pring (Eds.), Evidence-based practice in education (pp. 133–149). Maidenhead, U.K.: Open University Press.
• Hargreaves, D. (1996a). Educational research and evidence-based educational practice: A response to critics. Research Intelligence , 58 , 12–16.
• Hargreaves, D. (1996b). Teaching as a research-based profession: Possibilities and prospects. Teacher Training Agency Annual Lecture, London. Retrieved from https://eppi.ioe.ac.uk/cms/Portals/0/PDF%20reviews%20and%20summaries/TTA%20Hargreaves%20lecture.pdf .
• Hatcher, P. , Hulme, C. , Miles, J. N. , Caroll, J. M. , Hatcher, J. , Gibbs, S. , . . . Snowling, M. J. (2006). Efficacy of small group reading intervention for readers with reading delay: A randomised controlled trial. Journal of Child Psychology and Psychiatry , 47 (8), 820–827.
• Hitchcock, D. (2011). Instrumental rationality. In P. McBurney , I. Rahwan , & S. Parsons (Eds.), Argumentation in multi-agent systems: Proceedings of the 7th international ArgMAS Workshop (pp. 1–11). New York: Springer.
• Kelly, T. (2008). Evidence. In E. Zalta (Ed.), Stanford encyclopedia of philosophy . Retrieved from http://plato.stanford.edu/entries/evidence/ .
• Kvernbekk, T. (2016). Evidence-based practice in education: Functions of evidence and causal presuppositions . London: Routledge.
• Lærerorganisasjonenes skolenevnd . (1933). Innstilling . Oslo: O. Fredr. Arnesens Bok- og Akcidenstrykkeri.
• Mackie, J. L. (1975). Causes and conditions. In E. Sosa (Ed.), Causation and conditionals (pp. 15–38). Oxford: Oxford University Press.
• Menzies, P. , & Price, H. (1993). Causation as a secondary quality. British Journal for the Philosophy of Science , 44 , 187–203.
• Morrison, K. (2001). Randomised controlled trials for evidence-based education: Some problems in judging “what works.” Evaluation and Research in Education , 15 (2), 69–83.
• Oancea, A. , & Pring, R. (2008). The importance of being thorough: On systematic accumulation of “what works” in education research. Journal of Philosophy of Education , 42 (Suppl. 1), 15–39.
• Olson, D. R. (2004). The triumph of hope over experience in the search for “what works”: A response to Slavin. Educational Researcher , 33 , 24–26.
• Pawson, R. (2012). Evidence-based policy: A realist perspective . Los Angeles: SAGE.
• Schaffer, J. (2007). The metaphysics of causation. In E. Zalta (Ed.), Stanford encyclopedia of philosophy . Retrieved from http://plato.stanford.edu/entries/causation-metaphysics/ .
• Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher , 31 , 15–21.
• Slavin, R. E. (2004). Education research can and must address “what works” questions. Educational Researcher , 33 , 27–28.
• U.S. Department of Education . (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide . Washington, DC: Coalition for Evidence-Based Policy. Retrieved from http://www2.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid/pdf .
• Worrall, J. (2007). Why there’s no cause to randomize. British Journal for the Philosophy of Science , 58 , 451–488.

Bibliography

• Biesta, G. (2010). Why “what works” still won’t work: From evidence-based education to value-based education. Studies in Philosophy and Education , 29 , 491–503.
• Bridges, D. , & Watts, M. (2008). Educational research and policy: Epistemological considerations. Journal of Philosophy of Education , 42 (Suppl. 1), 41–62.
• Cartwright, N. (2013). Knowing what we are talking about: Why evidence doesn’t always travel. Evidence and Policy , 9 , 97–112.
• Cartwright, N. , & Munro, E. (2010): The limitations of randomized controlled trials in predicting effectiveness. Journal of Evaluation in Clinical Practice , 16 , 260–266.
• Cartwright, N. , & Stegenga, J. (2011). A theory of evidence for evidence-based policy. Proceedings of the British Academy , 171 , 289–319.
• Hammersley, M. (Ed.). (2007). Educational research and evidence-based practice . Los Angeles: SAGE.
• Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement . London: Routledge.
• Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning . London: Routledge.
• Kvernbekk, T. (2013). Evidence-based practice: On the functions of evidence in practical reasoning. Studier i Pædagogisk Filosofi , 2 (2), 19–33.
• Pearl, J. (2009). Causality: Models, reasoning, and inference . Cambridge, U.K.: Cambridge University Press.
• Phillips, D. C. (2007). Adding complexity: Philosophical perspectives on the relationship between evidence and policy. In P. Moss (Ed.), Evidence and decision making . Yearbook of the National Society for the Study of Education, 106 (pp. 376–402). Malden, MA: Blackwell.
• Psillos, S. (2009). Regularity theories. In H. Beebee , C. Hitchcock , & P. Menzies (Eds.), The Oxford handbook of causation (pp. 131–157). Oxford: Oxford University Press.
• Reiss, J. (2009). Causation in the social sciences: Evidence, inference, and purpose. Philosophy of the Social Sciences , 39 , 20–40.
• Rosenfield, S. , & Berninger, V. (Eds.). (2009). Implementing evidence-based academic interventions in school settings . Oxford: Oxford University Press.
• Sanderson, I. (2003). Is it “what works” that matters? Evaluation and evidence-based policy- making. Research Papers in Education , 18 , 331–345.
• Sloman, S. (2005). Causal models: How people think about the world and its alternatives . Oxford: Oxford University Press.
• Smeyers, P. , & Depaepe, M. (Eds.). (2006). Educational research: Why “what works” doesn’t work . Dordrecht, The Netherlands: Springer.
• Thomas, G. , & Pring, R. (Eds.). (2004). Evidence-based practice in education . Maidenhead, U.K.: Open University Press.
• Williamson, J. (2009). Probabilistic theories. In H. Beebee , C. Hitchcock , & P. Menzies (Eds.), The Oxford handbook of causation (pp. 185–212). Oxford: Oxford University Press.
• Woodward, J. (2003). Making things happen: A theory of causal explanation . Oxford: Oxford University Press.

Related Articles

• Rethinking Curriculum and Teaching
• Accountabilities in Schools and School Systems

Printed from Oxford Research Encyclopedias, Education. Under the terms of the licence agreement, an individual user may print out a single article for personal use (for details see Privacy Policy and Legal Notice).

date: 25 April 2024

• Cookie Policy
• Privacy Policy
• Legal Notice
• Accessibility
• [66.249.64.20|185.80.151.9]
• 185.80.151.9

Character limit 500 /500

• Research Guides

Empirical Research in the Social Sciences and Education

What is empirical research.

• Finding Empirical Research
• Ethics & Anti-Racism in Research
• Citing and Writing

Education Librarian

Or contact at

[email protected]

925-631-8632

An empirical research article is a primary source where the authors reported on experiments or observations that they conducted. Their research includes their observed and measured data that they derived from an actual experiment rather than theory or belief. 

How do you know if you are reading an empirical article? Ask yourself: "What did the authors actually do?" or "How could this study be re-created?"

Key characteristics to look for:

• Specific research questions  to be answered
• Definition of the  population, behavior, or phenomena  being studied
• Description of the  process or methodology  used to study this population or phenomena, including selection criteria, controls, and testing instruments (example: surveys, questionnaires, etc)
• You can readily describe what the authors actually did 

Layout of Empirical Articles

Scholarly journals sometimes use a specific layout for empirical articles, called the "IMRaD" format, to communicate empirical research findings. There are four main components:

• Introduction : aka "literature review". This section summarizes what is known about the topic at the time of the article's publication. It brings the reader up-to-speed on the research and usually includes a theoretical framework 
• Methodology : aka "research design". This section describes exactly how the study was done. It describes the population, research process, and analytical tools
• Results : aka "findings". This section describes what was learned in the study. It usually contains statistical data or substantial quotes from research participants
• Discussion : aka "conclusion" or "implications". This section explains why the study is important, and also describes the limitations of the study. While research results can influence professional practices and future studies, it's important for the researchers to clarify if specific aspects of the study should limit its use. For example, a study using undergraduate students at a small, western, private college can not be extrapolated to include all undergraduates. 
• Next: Finding Empirical Research >>
• Last Updated: Aug 29, 2023 10:19 AM
• URL: https://stmarys-ca.libguides.com/empirical

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License .

SPED 664: Research in Special Education

• Research Methods Resources
• Finding Empirical Research
• Articles/Databases
• Sources for Education Data & Statistics
• Professional Organizations & Web Resources
• Citing Sources (APA)
• Reference Books
• Films of Interest
• Find Books/Catalog

What is it Empirical Research?

Empirical research  is research based on observed and measured phenomena and derives knowledge from actual experience rather than from theory or belief. An empirical article is a research article that reports the results of a study that uses data derived from actual observation or experimentation.

How can I tell if a study is empirical? 

Key characteristics to look for:

• Specific research questions  to be answered
• Definition of the  population, behavior, or   phenomena  being studied
• Description of the  process  used to study this population or phenomena, including selection criteria, controls, and testing instruments (such as surveys)

Another hint: some scholarly journals use a specific layout, called the "IMRaD" format, to communicate empirical research findings. IMRaD  is an acronym for Introduction – Method – Results – and – Discussion:

• Introduction : sometimes called "literature review" -- what is currently known about the topic -- usually includes a theoretical framework and/or discussion of previous studies--typically states the hypothesis of the research.
• Methodology:  sometimes called "research design" -- a description of how the research was conducted and  how to recreate the study -- usually describes the participants, study design, and analytical tools
• Results : sometimes called "findings" --describes the outcomes --what was learned through the study -- usually appears as statistical data or as substantial quotations from research participants
• Discussion : sometimes called "conclusion" or "implications" -- why the study is important -- usually describes how the research results influence professional practices or future studies

Link to Finding Empirical Research 

Credits: The Empirical Research portion of this guide was adapted with permission from  

Elyssa Cahoy's "Empirical Research in Education and the Behavioral/Social Sciences" Library Guide. Please see  Penn State University Library for additional information.

• << Previous: Research Methods Resources
• Next: Finding Empirical Research >>
• Last Updated: Jul 18, 2022 10:15 AM
• URL: https://libguides.tcnj.edu/sped664

• What is Educational Research? + [Types, Scope & Importance]

Education is an integral aspect of every society and in a bid to expand the frontiers of knowledge, educational research must become a priority. Educational research plays a vital role in the overall development of pedagogy, learning programs, and policy formulation. 

Educational research is a spectrum that bothers on multiple fields of knowledge and this means that it draws from different disciplines. As a result of this, the findings of this research are multi-dimensional and can be restricted by the characteristics of the research participants and the research environment. 

What is Educational Research?

Educational research is a type of systematic investigation that applies empirical methods to solving challenges in education. It adopts rigorous and well-defined scientific processes in order to gather and analyze data for problem-solving and knowledge advancement. 

J. W. Best defines educational research as that activity that is directed towards the development of a science of behavior in educational situations. The ultimate aim of such a science is to provide knowledge that will permit the educator to achieve his goals through the most effective methods.

The primary purpose of educational research is to expand the existing body of knowledge by providing solutions to different problems in pedagogy while improving teaching and learning practices. Educational researchers also seek answers to questions bothering on learner motivation, development, and classroom management. 

Characteristics of Education Research  

While educational research can take numerous forms and approaches, several characteristics define its process and approach. Some of them are listed below:

• It sets out to solve a specific problem.
• Educational research adopts primary and secondary research methods in its data collection process . This means that in educational research, the investigator relies on first-hand sources of information and secondary data to arrive at a suitable conclusion. 
• Educational research relies on empirical evidence . This results from its largely scientific approach.
• Educational research is objective and accurate because it measures verifiable information.
• In educational research, the researcher adopts specific methodologies, detailed procedures, and analysis to arrive at the most objective responses
• Educational research findings are useful in the development of principles and theories that provide better insights into pressing issues.
• This research approach combines structured, semi-structured, and unstructured questions to gather verifiable data from respondents.
• Many educational research findings are documented for peer review before their presentation. 
• Educational research is interdisciplinary in nature because it draws from different fields and studies complex factual relations.

Types of Educational Research 

Educational research can be broadly categorized into 3 which are descriptive research , correlational research , and experimental research . Each of these has distinct and overlapping features. 

Descriptive Educational Research

In this type of educational research, the researcher merely seeks to collect data with regards to the status quo or present situation of things. The core of descriptive research lies in defining the state and characteristics of the research subject being understudied. 

Because of its emphasis on the “what” of the situation, descriptive research can be termed an observational research method . In descriptive educational research, the researcher makes use of quantitative research methods including surveys and questionnaires to gather the required data.

Typically, descriptive educational research is the first step in solving a specific problem. Here are a few examples of descriptive research: 

• A reading program to help you understand student literacy levels.
• A study of students’ classroom performance.
• Research to gather data on students’ interests and preferences. 

From these examples, you would notice that the researcher does not need to create a simulation of the natural environment of the research subjects; rather, he or she observes them as they engage in their routines. Also, the researcher is not concerned with creating a causal relationship between the research variables. 

Correlational Educational Research

This is a type of educational research that seeks insights into the statistical relationship between two research variables. In correlational research, the researcher studies two variables intending to establish a connection between them. 

Correlational research can be positive, negative, or non-existent. Positive correlation occurs when an increase in variable A leads to an increase in variable B, while negative correlation occurs when an increase in variable A results in a decrease in variable B. 

When a change in any of the variables does not trigger a succeeding change in the other, then the correlation is non-existent. Also, in correlational educational research, the research does not need to alter the natural environment of the variables; that is, there is no need for external conditioning. 

Examples of educational correlational research include: 

• Research to discover the relationship between students’ behaviors and classroom performance.
• A study into the relationship between students’ social skills and their learning behaviors. 

Experimental Educational Research

Experimental educational research is a research approach that seeks to establish the causal relationship between two variables in the research environment. It adopts quantitative research methods in order to determine the cause and effect in terms of the research variables being studied. 

Experimental educational research typically involves two groups – the control group and the experimental group. The researcher introduces some changes to the experimental group such as a change in environment or a catalyst, while the control group is left in its natural state. 

The introduction of these catalysts allows the researcher to determine the causative factor(s) in the experiment. At the core of experimental educational research lies the formulation of a hypothesis and so, the overall research design relies on statistical analysis to approve or disprove this hypothesis.

Examples of Experimental Educational Research

• A study to determine the best teaching and learning methods in a school.
• A study to understand how extracurricular activities affect the learning process. 

Based on functionality, educational research can be classified into fundamental research , applied research , and action research. The primary purpose of fundamental research is to provide insights into the research variables; that is, to gain more knowledge. Fundamental research does not solve any specific problems. 

Just as the name suggests, applied research is a research approach that seeks to solve specific problems. Findings from applied research are useful in solving practical challenges in the educational sector such as improving teaching methods, modifying learning curricula, and simplifying pedagogy. 

Action research is tailored to solve immediate problems that are specific to a context such as educational challenges in a local primary school. The goal of action research is to proffer solutions that work in this context and to solve general or universal challenges in the educational sector. 

Importance of Educational Research

• Educational research plays a crucial role in knowledge advancement across different fields of study. 
• It provides answers to practical educational challenges using scientific methods.
• Findings from educational research; especially applied research, are instrumental in policy reformulation. 
• For the researcher and other parties involved in this research approach, educational research improves learning, knowledge, skills, and understanding.
• Educational research improves teaching and learning methods by empowering you with data to help you teach and lead more strategically and effectively.
• Educational research helps students apply their knowledge to practical situations.

Educational Research Methods 

• Surveys/Questionnaires

A survey is a research method that is used to collect data from a predetermined audience about a specific research context. It usually consists of a set of standardized questions that help you to gain insights into the experiences, thoughts, and behaviors of the audience. 

Surveys can be administered physically using paper forms, face-to-face conversations, telephone conversations, or online forms. Online forms are easier to administer because they help you to collect accurate data and to also reach a larger sample size. Creating your online survey on data-gathering platforms like Formplus allows you to.also analyze survey respondent’s data easily. 

In order to gather accurate data via your survey, you must first identify the research context and the research subjects that would make up your data sample size. Next, you need to choose an online survey tool like Formplus to help you create and administer your survey with little or no hassles. 

An interview is a qualitative data collection method that helps you to gather information from respondents by asking questions in a conversation. It is typically a face-to-face conversation with the research subjects in order to gather insights that will prove useful to the specific research context. 

Interviews can be structured, semi-structured , or unstructured . A structured interview is a type of interview that follows a premeditated sequence; that is, it makes use of a set of standardized questions to gather information from the research subjects. 

An unstructured interview is a type of interview that is fluid; that is, it is non-directive. During a structured interview, the researcher does not make use of a set of predetermined questions rather, he or she spontaneously asks questions to gather relevant data from the respondents. 

A semi-structured interview is the mid-point between structured and unstructured interviews. Here, the researcher makes use of a set of standardized questions yet, he or she still makes inquiries outside these premeditated questions as dedicated by the flow of the conversations in the research context. 

Data from Interviews can be collected using audio recorders, digital cameras, surveys, and questionnaires. 

• Observation

Observation is a method of data collection that entails systematically selecting, watching, listening, reading, touching, and recording behaviors and characteristics of living beings, objects, or phenomena. In the classroom, teachers can adopt this method to understand students’ behaviors in different contexts. 

Observation can be qualitative or quantitative in approach . In quantitative observation, the researcher aims at collecting statistical information from respondents and in qualitative information, the researcher aims at collecting qualitative data from respondents. 

Qualitative observation can further be classified into participant or non-participant observation. In participant observation, the researcher becomes a part of the research environment and interacts with the research subjects to gather info about their behaviors. In non-participant observation, the researcher does not actively take part in the research environment; that is, he or she is a passive observer. 

How to Create Surveys and Questionnaires with Formplus

• On your dashboard, choose the “create new form” button to access the form builder. You can also choose from the available survey templates and modify them to suit your need.
• Save your online survey to access the form customization section. Here, you can change the physical appearance of your form by adding preferred background images and inserting your organization’s logo.
• Formplus has a form analytics dashboard that allows you to view insights from your data collection process such as the total number of form views and form submissions. You can also use the reports summary tool to generate custom graphs and charts from your survey data. 

Steps in Educational Research

Like other types of research, educational research involves several steps. Following these steps allows the researcher to gather objective information and arrive at valid findings that are useful to the research context. 

• Define the research problem clearly. 
• Formulate your hypothesis. A hypothesis is the researcher’s reasonable guess based on the available evidence, which he or she seeks to prove in the course of the research.
• Determine the methodology to be adopted. Educational research methods include interviews, surveys, and questionnaires.
• Collect data from the research subjects using one or more educational research methods. You can collect research data using Formplus forms.
• Analyze and interpret your data to arrive at valid findings. In the Formplus analytics dashboard, you can view important data collection insights and you can also create custom visual reports with the reports summary tool. 
• Create your research report. A research report details the entire process of the systematic investigation plus the research findings. 

Conclusion 

Educational research is crucial to the overall advancement of different fields of study and learning, as a whole. Data in educational research can be gathered via surveys and questionnaires, observation methods, or interviews – structured, unstructured, and semi-structured. 

You can create a survey/questionnaire for educational research with Formplu s. As a top-tier data tool, Formplus makes it easy for you to create your educational research survey in the drag-and-drop form builder, and share this with survey respondents using one or more of the form sharing options. 

Connect to Formplus, Get Started Now - It's Free!

• education research
• educational research types
• examples of educational research
• importance of educational research
• purpose of educational research
• busayo.longe

You may also like:

Assessment Tools: Types, Examples & Importance

In this article, you’ll learn about different assessment tools to help you evaluate performance in various contexts

What is Pure or Basic Research? + [Examples & Method]

Simple guide on pure or basic research, its methods, characteristics, advantages, and examples in science, medicine, education and psychology

Goodhart’s Law: Definition, Implications & Examples

In this article, we will discuss Goodhart’s law in different fields, especially in survey research, and how you can avoid it.

User Research: Definition, Methods, Tools and Guide

In this article, you’ll learn to provide value to your target market with user research. As a bonus, we’ve added user research tools and...

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

Advertisement

Trends in Highly Cited Empirical Research in STEM Education: a Literature Review

• Published: 07 December 2022
• Volume 5 , pages 303–321, ( 2022 )

Cite this article

• Yeping Li 1 ,
• Ke Wang 2 ,
• Yu Xiao 1 &
• Suzanne M. Wilson 3  

3728 Accesses

5 Citations

2 Altmetric

Explore all metrics

The development of STEM education scholarship takes time and collective effort. Identifying and examining trends in highly cited empirical STEM education research over time will help inform the field and future research. In this study, we searched the Web of Science core database to identify the top 100 most-cited empirical research journal publications in each of three consecutive years. Our analyses revealed consistencies and important changes over the years in terms of the inclusion of articles themselves, the journals where they were published, disciplinary content coverage, and research topics. The results demonstrate that STEM education research is increasingly recognized as important in education, both through publications and citations, and that the field is moving toward conducting more multi- and interdisciplinary STEM education research.

Avoid common mistakes on your manuscript.

Introduction

The importance of STEM (science, technology, engineering, and mathematics) education has been well-recognized, not only due to the importance of each component discipline but also the connection and integration of different STEM disciplines. Different from the traditional approach of focusing on individual disciplines such as mathematics and physics, STEM education opens the door for new opportunities and approaches for students’ learning and preparation (see Li, 2018 ). However, new opportunities come with new challenges, as STEM itself is not a discipline. The nature and content of this new integrative field of STEM education scholarship cannot be pre-defined, but emerges from the collective contributions of numerous scholars over time. To gain insights into STEM education scholarship development, we sought to identify and examine trends in empirical research publications that have had high impact in the field over the past 3 years.

A recent research review (Li et al., 2022 ) served as the foundation for this study. In that review, we specified high impact empirical research publications as those that gained high citations. Although many databases are available to search for publication citation counts, the Web of Science (WoS) is the world’s leading scientific citation search and analytical information platform (Li et al., 2018 ). Its core database has been commonly used as a reliable indexing database with close attention to high standard research publications with a peer-review process, thus used in many research review studies (e.g., Li et al., 2018 ; Marín-Marín et al., 2021 ). The WoS core database is more selective than many others, such as Scopus. For these reasons, we searched the WoS core database to identify the top 100 most-cited empirical studies in STEM education published in journals from 2000 to 2021. The search was conducted on September 12, 2022, and allowed us to identify and select those research publications that gained high citations for inclusion up to that day. However, as publication citations keep changing on a daily basis and also over time, it remains unclear whether and how the top 100 most-cited research publications may be changing over time. Learning about publication citation changes provides a glimpse into the dynamic evolution in STEM education research field.

Li et al. ( 2022 ) examined multiple aspects of the top 100 most-cited empirical studies published in journals from 2000 to 2021, including journals in which these high impact empirical studies were published, publication years, disciplinary content coverage, and research topics. We planned for the current study to also focus on those aspects. Different from the recent review, however, the current study aimed to identify and examine possible trends from changes in the top 100 most-cited empirical research publications identified in three different years (i.e., August 2020, September 2021, and September 2022). Taken together, this study was designed to use data from three searches conducted over the years to systematically analyze and report:

consistencies and changes in the top 100 most-cited empirical research journal publications over three years

distributions and patterns of the top 100 most-cited empirical research publications in different journals

disciplinary content coverage of the top 100 most-cited empirical research journal publications and possible trends

research topics being focused on by the top 100 most-cited empirical research journal publications and topic trends

Methodological Considerations

Searching and identifying.

To be consistent with Li et al. ( 2022 ), we used the same process to search and identify the top 100 most-cited empirical research publications in different years. The process started with searching the WoS core database under the field of “topic” (covering title, abstract, author keywords, and keywords plus), using the same search terms: “STEM” OR “STEAM” OR “science, technology, engineering, and mathematics.” Because there are many different categories in the WoS database, we conducted publication searches under the same WoS category: Education Educational Research, on August 9, 2020, September 20, 2021, and September 12, 2022, respectively. In each of these 3 years, the time period of publications was set as starting from 2000 to the year right before the search was conducted. For example, in 2022, all publications from 2000 to 2021 were specified in the search.

After obtaining a list of publications from the search, all publications were placed in descending order in terms of citation counts. Each publication was then carefully checked using the same criteria for inclusion or exclusion (see Table 1 ). The process identified the top 100 most-cited empirical research journal publications. The follow-up publication coding and analysis were then carried out in the same way as Li et al. ( 2022 ).

Accounting for Different Search Category Coverage

During the article search process in September 2022, we noticed that the WoS database has four categories listed under “education”: Education Educational Research, Education Scientific Disciplines, Psychology Educational, and Education Special. It occurred to us that empirical research articles in STEM education may also be published in journals that are not classified and listed under the category of “Education Educational Research.”

Thus, we conducted another search of the WoS database in 2022 under all of the four categories using the same search terms for journal publications from 2000 to 2021. The search returned 9275 publications under “Education Educational Research,” 2161 under “Education Scientific Disciplines,” 247 under “Psychology Educational,” and 15 under “Education Special.” The combined list of all publications was then placed in descending order in terms of citation counts, and each publication was screened using the same inclusion or exclusion criteria (Table 1 ).

As these two searches with the inclusion of different WoS categories were conducted in the same year, possible connections and differences in the inclusion of publications and journals across these two searches will not reveal possible trends over time. However, comparing the results may illuminate the diversity of journal outlets that researchers are using.

Trends in Highly Cited STEM Education Research Publications

Consistencies and changes from 2020 to 2022.

Across the 3 years, 76 publications were identified as included each year. Eleven publications were changed in the 2nd search in 2021 in comparison to the list from the 1st search in 2020; twenty-three publications were changed in 2022. Across the lists between searches in 2021 and 2022, there were 84 same publications and 16 different publications. The results suggest that the majority (76) of high impact empirical research journal publications were stable, over the 3-year period, in terms of gaining high citations. At the same time, quite a substantial number of publications (24) were dropped.

Figure  1 shows the distributions of top 100 most-cited publications in 2022, 2021, and 2020, respectively. Several overall consistencies are noteworthy. Specifically, across these three distributions, the majority of publications were published between 2010 and 2015, suggesting that publications would typically need about 5–10 years to gain extensive exposure to obtain enough citations for inclusion. At the same time, the distribution of those most-cited publications identified in 2022 shows that some more recent publications (2016–2019) also emerged with high citations. Multiple factors might account for this, including possible changes in journal inclusion in the WoS core database or the appearance of high quality research that was disseminated in high profile ways. Further examination of possible contributing factors is beyond the scope of this study.

Distributions of the top 100 most-cited empirical research journal publications in STEM education over the years in three different searches

Table 2 provides the top 10 list of most-cited publications identified from each search. Nine articles (with citation counts bolded) appear each year. In particular, the top 4 are the same across the 3 years. All top 10 articles from the 2021 search made it to the top 10 list again in 2022 search. There is one article difference of inclusion in the top 10 list between 2021 and 2020 searches, and two article differences of inclusion in the top 10 between 2022 and 2020 searches.

Table 2 also shows that all of these most-cited articles were published between 2008 and 2013. The top 10 most-cited publications in 2020 had an average of 180 citations (range, 134–271 per article). In 2021, the articles had an average of 226 citations (range, 180–352). For the top 11 (with two articles in a tie at the 10th place) in 2022, the average was 263 citations (range, 211–421). The nine articles that appeared each year had an average of 185 citations with a range of 140 to 271 in 2020, an average of 231 citations with a range of 192 to 352 in 2021, and an average of 272 citations with a range of 211 to 421 in 2022. The results again provide a clear indication of increased citations for publications over time, albeit with different increasing rates for different publications.

We also observed that the top 10 list of most-cited empirical research publications were published in different journals, but not in a journal specifically on STEM education. In fact, there was no well-established journal in STEM education before 2019 (Li, 2019 ). It is not surprising that the top 10 list of most-cited empirical research articles in STEM education were published in other well-established journals in education or science education at that time. The results suggest the potential value of examining what journals published highly cited empirical research in STEM education and related patterns.

Distributions and Patterns of Highly Cited Publications in Different Journals

We identified and sorted all journals in which publications appeared. Forty-eight journals published these articles across the 3 years (see Table 3 ), 43 journals published the top 100 identified in 2022, 41 journals in 2021, and 40 journals in 2020. Thirty-five of these journals appeared in all three searches.

Thirty-seven journals were covered by SSCI (Social Sciences Citation Index) and 11 were covered by ESCI (Emerging Sources Citation Index). These are clearly well-established and quality journals in the professional community. Moreover, the majority of these journals have a long publishing history, with 35 journals being established 30 or more years ago, and 13 journals having less than a 30-year history. Eleven journals have been established since 2000, eight SSCI journals and three ESCI journals. This suggests that the most highly cited empirical research has been published in well-established and reputable journals with a long publishing history. It is not surprising as STEM education itself has too a short history to establish top journals (Li et al., 2020 ).

To take a closer look at the possible impact of different journals, we examined the top 10 journals and their publications from the list in Table 3 . These 10 journals contributed 57 articles (57%) for the top 100 collection from 2022 search, 56 articles (56%) for the top 100 collection in 2021, and 53 articles (53%) for the top 100 collection in 2020. The results provide clear indications that these 10 journals carried a heavy weight in publishing high impact research articles from the three searches. All of these 10 journals were SSCI journals. They all had 30 or more years of history, except one journal, International Journal of STEM Education (IJSTEM), that started to publish in 2014 (Li, 2014 ). In fact, IJSTEM is the only journal, out of the 48 journals in the list, with a clear focus on STEM education research. The result provides a confirmation about the initial stage of STEM education research journal development at this time (Li, 2018 ; Li et al., 2020 ), and the leading journal status of IJSTEM in promoting and publishing STEM education research (Li, 2021 ).

Among the 48 journals listed in Table 3 , we classified them into two general categories: general education research journals (without a discipline of STEM specified in a journal’s title) and those with STEM discipline specified in a journal’s title, an approach similar to what we used in a previous research review (Li et al., 2020 ). Thirty journals spoke to general educational education research readers; 18 to readers in specific discipline(s) of STEM. The result suggests that researchers published their high impact empirical research in STEM education in a wide range of journals, with more in general educational research journals as these journals tend to be well-established with a long history, and spoke both to scholars with interests in STEM and to broader communities.

Figure  2 shows that the distributions of the top 100 articles over these two types of journals were very stable across these three searches, with about 50% of the articles published in educational research journals and 50% in journals with STEM discipline(s) specified.

Distributions of top 100 most-cited empirical research publications in general and STEM-specific journals. Note: 0 = journals without STEM discipline specified, 1 = journals with STEM discipline specified

Going beyond the cumulative counts of these publications in two types of journals from each search, Fig.  3 shows the distributions of top 100 publications in each type of journal over the years for each search, where dotted line segments refer to the distributions in journals without STEM discipline specified and solid line segments refer to the distributions in journals with STEM discipline specified.

Trends of top 100 most-cited empirical STEM education research publications in general vs. STEM-specific journals, 2020–2022

Overall, there are some general consistencies in trends between the group of dotted line segments and the group of solid line segments, an observation consistent with what we learned from Fig.  2 about overall distributions of publications. However, we also noticed that the dotted line segments stay above the solid line segments from 2009 to 2012, and solid line segments tend to stay higher since 2014 especially from the 2022 search. The results suggest a general trend that those highly cited articles tended to be published in general educational research journals before 2013, but started to have more in journals with content discipline of STEM specified after 2013 in recent search. There are many possible reasons for this trend (e.g., Li et al., 2020 ; Li & Xiao, 2022 ; Moore et al., 2017 ; Wilsdon et al., 2015 ). Researchers might have developed disciplinary consciousness in publications, especially when STEM education publications started to pick up since that time. There is also evidence that there has been a rise in higher education in the pressure to publish, both due to the use of productivity metrics in universities and funders’ use of publication records in evaluating research grants. It may be that, on the global front, institutions of higher education are broadening the list of recommended outlets for publications to STEM-specific journals. This result may also relate to disciplinary content coverage specified in these top 100 publications in different searches, a topic that we will examine further in next section.

Disciplinary Content Coverage

The top 100 most-cited empirical research publications in STEM education were identified through topic search of specific terms (“STEM,” “STEAM,” or “science, technology, engineering, and mathematics”), as it was done in other research reviews (Li et al., 2020 , 2022 ). However, the author’s self-inclusion of such identifier(s) did not mean that all four STEM disciplines were focused on in their studies. Thus, we took a further look at each publication to examine if it focused on a single discipline of STEM or multiple disciplines of STEM.

Figure  4 presents summarized results. The vast majority (75% or more) of these top 100 highly cited articles focused on multidisciplinary STEM education. There is a small but notable change in disciplinary foci between those publications identified from 2022 search and those publications from 2021 and 2020 searches: 25 publications on a single discipline of STEM and 75 on multidisciplinary STEM education from 2022 search, while about 15 publications on a single discipline of STEM and 85 on multidisciplinary STEM education from 2021 and 2020 searches. It would be interesting to see how the disciplinary foci emerged from those highly cited empirical research publications evolve.

Disciplinary content coverage in top 100 most-cited articles, 2020–2022. Note: 1 = single discipline of STEM education, 2 = multidisciplinary STEM education

To take a further look at possible changes across different searches, Fig.  5 presents the distributions of top 100 publications with different disciplinary content coverages from 2020 to 2022. We used solid line segments for the distributions of publications on a single discipline of STEM and dotted line segments for the distributions of publications on multidisciplinary STEM education.

Trends in disciplinary content coverage, 2020–2022

Several general consistencies in trends between the group of dotted line segments and the group of solid line segments, an observation consistent with what we learned from Fig.  4 . At the same time, we also noticed that no publications on a single discipline of STEM before 2011 made to the list in 2021 and 2022 searches, and more publications on multidisciplinary STEM education after 2015 made to the list in these two recent searches than the 2020 search. The results suggest a possible trend of shifting research interest and development toward multi- and interdisciplinary STEM education in the field through recent publications and citations.

Research Topics

To examine research topics, we used the same list of topics from previous reviews (Li & Xiao, 2022 ; Li et al., 2019 ). The following list contains seven topic categories (TC) that was used to classify and examine all publications identified and selected from the three searches in this study.

TC1: Teaching, teacher, and teacher education in STEM (including both pre-service and in-service teacher education) in K-12 education

TC2: Teacher and teaching in STEM (including faculty development, etc.) at post-secondary level

TC3: STEM learner, learning, and learning environment in K-12 education

TC4: STEM learner, learning, and learning environments (excluding pre-service teacher education) at post-secondary level

TC5: Policy, curriculum, evaluation, and assessment in STEM (including literature review about a field in general)

TC6: Culture, social, and gender issues in STEM education

TC7: History, epistemology, and perspectives about STEM and STEM education

Consistent with the coding practice used in the previous reviews, we assigned each publication to only one topic. When there were cases that more than one topic could have been used, a decision was made after discussion.

Figure  6 shows that more publications on four TCs (i.e., TC3, TC4, TC5, and TC6) were more highly cited than articles focused on the other three TCs (i.e., TC1, TC2, and TC7). Moreover, TC4 (STEM learner, learning, and learning environments at post-secondary level) and TC6 (culture, social, and gender issues in STEM education) were the two TCs in 2021 and 2020 searches that had the most publications. In 2022, culture, social, and gender issues in STEM education was the focus of many more publications than the other TCs. The results suggest that publications in TC4 and TC6 were more likely to gain high citations than publications in other TCs, and the trend seems to go further for publications in TC6 but not for TC4 in 2022 search. At the same time, it is a bit surprising to observe that teaching, teacher, and teacher education in STEM in K-12 education (TC1) and teachers and teaching in STEM at post-secondary level (TC2) were not popular topic areas among those highly cited research publications. It would be interesting to see if possible changes may take place in the future.

Trends in research topic distributions, 2020–2022

In comparison to what we can learn from previous research reviews (Li et al., 2020 ; Li & Xiao, 2022 ), the results from this study seemingly present a different picture in terms of the “hot” topics. However, it should be pointed out that this study was restricted in identifying and selecting high impact empirical research publications in STEM education from the WoS database, different from previous reviews in terms of both journal coverages and the scope of publication inclusion. In fact, there were many highly cited research reviews and conceptual papers in STEM education but excluded from review in this study.

Consistencies and Changes in the Top 100 Most-cited Empirical STEM Research Journal Publications Identified from Two Searches Using Different WoS Categories

Now we come to the two searches conducted in 2022 using different WoS categories, as mentioned above. One search was conducted using one category (“Education Educational Research”). The results of the top 100 most-cited publications were presented above together with the results from 2021 and 2020 searches. The second search was conducted using all four categories listed under “education,” including “Education Educational Research,” “Education Scientific Disciplines,” “Psychology Educational,” and “Education Special.” Some journals may be classified and listed in more than one category, for example, IJSTEM that is listed under both “Education Educational Research” and “Education Scientific Disciplines.” Nevertheless, it is clear that the 2nd search covered more journals and publications with all four categories.

Table 4 shows the list of all journals (50) that published the top 100 most-cited empirical research articles from these two searches, with 43 journals published the top 100 identified in the 1st search and 45 journals in the second search. The vast majority of the journals were the same (38). At the same time, we observed a few important differences. First, there were a few journals (see journal titles in bold in Table 4 , all SSCI journals) likely not listed under “Education Educational Research” but published highly cited empirical research articles in STEM education, especially CBE-Life Science Education (CBE-LSE) and the Journal of Educational Psychology . The inclusion of these new journals from the search actually resulted in significant changes to the allocation of top 100 most-cited empirical research publications in other journals. Moreover, if taking a close look at these nine articles published in CBE-LSE (a journal specified with STEM discipline), we found that these articles were all published more recently from 2014 to 2019, with three in 2014, two in 2015, and one in 2016, 2017, 2018, and 2019, respectively. The result is consistent with a trend we noticed above from Fig.  3 .

Second, some journals (see journal titles in italics in Table 4 ) from the first search being pushed out in the second search, as publications in these journals did not have high enough citations for inclusion as part of the top 100 list. Moreover, these five journals were all on general education research, except one ( Physical Review Physics Education Research ).

Taken together, the results suggest that differences can occur with the use of different categories in the WoS searches. The use of all four categories under “education” would make the search more inclusive, which is what we conducted in the WoS core database search in the recent research review (Li et al., 2022 ).

Concluding Remarks

This study, for the first time, examined trends in highly cited empirical research publications in STEM education, through reviewing the top 100 most-cited journal articles identified from the WoS core database in each of three consecutive years. The systematic analysis of these publications reveals the on-going accumulation and development of STEM education scholarship. Although empirical research in STEM education has been consistently published in many well-established journals especially in educational research, our analysis shows that a growing number of those highly cited research articles were published in journals specified more or less with STEM discipline(s). In particular, among the top 10 journals that published more than 50% of those top 100 most-cited articles in each search, the International Journal of STEM Education was able to make to the top 10 journal list even though it is the only one established after 2000.

Across these three searches, the vast majority of those highly cited research publications focused on multidisciplinary STEM education. At the same time, the lists of publications identified from recent searches in 2021 and 2022 contain no publication on a single discipline of STEM before 2011 but an increasing number of publications on multidisciplinary STEM education in recent years. A trend shows an increased interest and development in recent publications on multi- and interdisciplinary STEM education.

With a restriction on empirical research in STEM education from the WoS database, this study differed from previous research reviews that covered all types of articles on STEM education in different journals (e.g., Li & Xiao, 2022 ; Li et al., 2020 ). Nevertheless, our analysis in this study shows that culture, social, and gender issues in STEM education (TC6) and STEM learner, learning, and learning environments at post-secondary level (TC4) were popular topic areas among those highly cited research publications. In contrast, teaching, teacher, and teacher education in STEM in K-12 education (TC1) and teacher and teaching in STEM at post-secondary level (TC2) were not popular. As research in these TCs has been growing with increased publications (Li & Xiao, 2022 ), we would not be surprised if more publications in these areas will appear in the top 100 research publication list in the future.

It may also be the case that teaching, teachers, and teacher preparation are the focus of already-published research that is still too recent to have accumulated the citations necessary to become highly cited. If one major goal of literature reviews is to help scholars identify promising topics of inquiry, this “lag” time problem suggests that citations—while a helpful proxy—need to be supplemented with other indicators, including those that may not be even dependent on the arduous (and often lengthy) process of getting one’s work published in a journal. A recent review of publications in the International Journal of STEM Education shows the value of using different indicators for publication performance measurements, such as altmetrics (Li, 2022 ). Finding other ways to locate the field’s promising topics will benefit researchers and journals such as ours which can play an important role in providing a platform for sharing and promoting integrated research in STEM education.

Data Availability

The data and materials used and analyzed for the report were obtained through searching the Web of Science database, and related journal information are available directly from these journals’ websites.

Li, Y. (2014). International journal of STEM education – A platform to promote STEM education and research worldwide. International Journal of STEM Education, 1 , 1. https://doi.org/10.1186/2196-7822-1-1

Article   Google Scholar  

Li, Y. (2018). Journal for STEM education research – Promoting the development of interdisciplinary research in STEM education. Journal for STEM Education Research, 1 , 1–6. https://doi.org/10.1007/s41979-018-0009-z

Li, Y. (2019). Five years of development in pursuing excellence in quality and global impact to become the first journal in STEM education covered in SSCI. International Journal of STEM Education, 6 , 42. https://doi.org/10.1186/s40594-019-0198-8

Li, Y. (2021). Seven years of development as building a foundation for the journal’s leadership in promoting STEM education internationally. International Journal of STEM Education, 7 , 58. https://doi.org/10.1186/s40594-021-00316-w

Li, Y. (2022). Eight years of development in welcoming and engaging diverse scholars to share and promote STEM education research worldwide. International Journal of STEM Education, 9 , 69. https://doi.org/10.1186/s40594-022-00385-5

Li, Y., & Xiao, Y. (2022). Authorship and topic trends in STEM education research. International Journal of STEM Education, 9 , 62. https://doi.org/10.1186/s40594-022-00378-4

Li, K., Rollins, J., & Yan, E. (2018). Web of Science use in published research and review papers 1997–2017: A selective, dynamic, cross-domain, content-based analysis. Scientometrics, 115 (1), 1–20.

Li, Y., Froyd, J. E., & Wang, K. (2019). Learning about research and readership development in STEM education: A systematic analysis of the journal’s publications from 2014 to 2018. International Journal of STEM Education, 6 , 19. https://doi.org/10.1186/s40594-019-0176-1

Li, Y., Wang, K., Xiao, Y., & Froyd, J. E. (2020). Research and trends in STEM education: A systematic review of journal publications. International Journal of STEM Education, 7 , 11. https://doi.org/10.1186/s40594-020-00207-6

Li, Y., Xiao, Y., Wang, K., Zhang, N., Pang, Y., Wang, R., Qi, C., Yuan, Z., Xu, J., Nite, S. B., & Star, J. R. (2022). A systematic review of high impact empirical studies in STEM education. International Journal of STEM Education, 9: 73.

Marín-Marín, J.-A., Moreno-Guerrero, A.-J., Dúo-Terrón, P., & López-Belmonte, J. (2021). STEAM in education: A bibliometric analysis of performance and co-words in Web of Science. International Journal of STEM Education, 8 , 41. https://doi.org/10.1186/s40594-021-00296-x

Moore, S., Neylon, C., Paul Eve, M., O’Donnell, D. P., & Pattinson, D. (2017). “Excellence R Us”: University research and the fetishisation of excellence. Palgrave Commun., 3 , 16105.

Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J., & Johnson, B. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management . London: SAGE Publications Ltd. https://doi.org/10.13140/RG.2.1.4929.1363

Download references

Acknowledgements

The authors wish to thank Jeff Davis and the staff at Springer for their support in publishing this article.

Author information

Authors and affiliations.

Texas A&M University, College Station, TX, 77843-4232, USA

Yeping Li & Yu Xiao

Nicholls State University, Thibodaux, LA, USA

University of Connecticut, Storrs, CT, USA

Suzanne M. Wilson

You can also search for this author in PubMed   Google Scholar

Contributions

YL conceived the study, conducted data analyses, and drafted the manuscript. KW and YX contributed with article search, identification, selection, and coding. SMW reviewed drafts and contributed to manuscript revisions. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yeping Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Li, Y., Wang, K., Xiao, Y. et al. Trends in Highly Cited Empirical Research in STEM Education: a Literature Review. Journal for STEM Educ Res 5 , 303–321 (2022). https://doi.org/10.1007/s41979-022-00081-7

Download citation

Published : 07 December 2022

Issue Date : December 2022

DOI : https://doi.org/10.1007/s41979-022-00081-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Empirical research
• Literature review
• STEM education research
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

What is research? Educators’ conceptions and alignment with United States federal policies

This study draws on two communities theory to address two major research questions related conceptions of research in educational practice and policy. First, how do educators conceptualize research? Second, to what extent do educators’ conceptions of research align with recent U.S. federal educational policies? We conducted 90 semi-structured interviews with educators in the United States, asking them what comes to mind when they think of research. We used open, axial, and selective coding to characterize educators’ conceptions of research. We also compared educators’ conceptions of research to two U.S. federal educational policies that define scientifically based research and evidence-based interventions. Findings indicate that educators and policies defined research in similar ways, but each included some unique characteristics. Implications from the study include the need for increased communication between federal policy-makers and educators and improved reporting by researchers to better attend to the needs of educators and policymakers.

INTRODUCTION

Some education researchers are producing research informed by practice and some practitioners are using practices informed by research. However, there remains a research-practice gap (e.g., Farley-Ripple, May, Karpyn, Tilley, & McDonough, 2018 ; Neal, Neal, Kornbluh, Mills, & Lawlor, 2015 ). This gap is marked by a communication deficit between researchers and practitioners, where there is limited uptake of practitioners’ needs and expertise in research settings, and limited uptake of educational research in practice settings ( Neal et al., 2015 ). Several factors contribute to this gap, including the limited accessibility of research, limited relevance to student needs, and lack of resources ( Bartels, 2003 ; Barwick, Barac, Akrong, Johnson, & Chaban, 2014 ; Coburn & Talbert, 2006 ; Long et al., 2016 ; Farley-Ripple, 2012 ; Farley-Ripple et al., 2018 ; Malin, Brown, & Trubceac, 2018 ; Neal et al., 2015 ; Neal, Neal, Lawlor, Mills, & McAlindon, 2018 ; Neal, Mills, McAlindon, Neal, & Lawlor, in press ).

In the United States, federal policies including the No Child Left Behind Act of 2001 (NCLB) and the Every Student Succeeds Act (ESSA) encourage educators to use research or evidence to aid in improving student outcomes and support equal educational opportunities for all students ( ESSA, 2015 ; NCLB, 2002 ). However, in addition to the research-practice gap noted above, there may also be a practice-policy gap wherein practitioners do not engage in the research use encouraged or required by policy. One potential explanation for such a gap is that the policy and practitioner communities are speaking different languages, that what practitioners mean by “research” is different from what policies mean by “research” (e.g., Hill, 2001 ; Spillane, Reiser, & Riemer, 2002 ). Accordingly, work is needed to understand how educators conceptualize research and how these conceptions align with policies requiring its use (see Davidson, Penuel, & Farrell, 2018 ; Finnigan, Daly, & Che, 2013 ; Penuel, Farrell, Allen, Toyama, & Coburn, 2018 ; Joram, 2007 ). Understanding how educators conceptualize research provides insight about how educators are locating, evaluating, and applying research to advance educational efforts, and may identify opportunities for additional training. Moreover, exploring how educators’ conceptions of research align with federal educational policies may identify potential mismatches between the aspects of research that are important to educators and in policy.

This study explores how educators conceptualize research, and the extent to which their conceptions of research align with recent U.S. federal educational policies. Drawing on Two-Communities theory ( Caplan, 1979 ; Farley-Ripple et al., 2018 ), we argue that educators and policymakers represent separate groups with their own conceptions of research, which would lead to a practice-policy gap . Through semi-structured interviews with 90 K-12 public school educators throughout the U.S. State of Michigan, we show that educators’ conceptions of research are quite broad, and only partially align with federal policy definitions. We conclude with a discussion of the findings on educators’ conceptions of research, and their implications for federal education policymakers and researchers.

LITERATURE REVIEW

Two-communities theory and the practice-policy gap.

Originating in the knowledge utilization literature, two-communities theory provides a rationale for exploring educators’ conceptions of research, and whether these conceptions align with U.S. federal education policies ( Caplan, 1979 ; Farley-Ripple et al., 2018 ). Caplan (1979) aimed to describe why government policymakers often fail to use social science research. He argued that the users and producers of research often work in disconnected social communities where their limited boundary crossing (e.g. Penuel, Allen, Coburn, and Farrell, 2015 ) leads to limited overlap in definitions and expectations (see also Green, Ottoson, Garcia, & Hiatt, 2009 ; Lomas, 2007 ; Neal et al., 2015 ; Nutley, Walter, and Davies, 2003 ). Although recent work suggests there may be more communication between the users and producers of research than two-communities theory originally implied (e.g., Newman, Cherney, & Head, 2016 ), two-communities theory continues to provide a useful explanation for mismatches between these groups. For example, Farley-Ripple et al. (2018) used two-communities theory to show that educators tend to emphasize the demographic fit of research to their own context, while researchers tend to emphasize internal validity and research design.

While two-communities theory has been applied to explain the research-practice gap (e.g., Caplan, 1979 ; Farley-Ripple et al., 2018 ; Newman et al., 2016 ), it is also helpful for understanding a practice-policy gap between the users of educational policy (e.g., educators) and those who enact it (e.g., federal policymakers). Like educators and researchers, educators and policymakers work in different communities with different definitions and expectations ( Locock & Boaz, 2004 ). Supporting this idea, a number of studies of educators’ sense-making suggest that educators interpret the language and messages in educational policy based on their own experiences and values, and in ways that are different from policymakers’ original objectives (e.g., Hill, 2001 ; Spillane et. al., 2002 ). Therefore, it is possible that educators and federal policymakers have distinct conceptions of research.

Educators can use research in instrumental, conceptual, and symbolic ways ( Weiss, 1979 ; Weiss & Bucuvalas, 1980 ). First, instrumental use of research (i.e., direct use) occurs when educators use research to solve specific problems or make specific decisions. Second, conceptual use of research (i.e., enlightenment or indirect use) occurs when educators use research broadly to inform their thinking on a topic. Third, symbolic use of research (i.e., political or tactical use) occurs when educators use research to retroactively defend already existing decisions or actions. Although district and building educators report using research in all three ways (e.g., Cain, 2015 ; Coburn, Toure, & Yamashita, 2009 ; Penuel et al., 2017 ; Weiss, Murphy-Graham, & Birkeland, 2005 ), federal policies tend to emphasize instrumental use of research (e.g., Penuel et al., 2017 ). Thus, it is possible that federal policymakers take a narrower stance than educators in their conceptualization of research, focusing policies on qualities of research that are useful for solving specific problems or driving decision-making (e.g., study design, reliability and validity of the findings, outcomes). To determine what factors might lead to a practice-policy gap, it is critical to understand both how educators conceptualize research, and the extent to which their conceptions align with those outlined in federal policy.

Educators’ Conceptions of Research

Educators’ conceptions of research often include data such as standardized tests scores and student performance data, while they less often think about peer-reviewed empirical studies. For instance, Finnigan et al. (2013) found that high school educators in low performing schools relied more heavily on data use and equated research with standardized test scores. They used research to describe inquiring about other schools’ practices, but rarely to describe research studies, which they perceived as not fitting their school’s context. Consistent with Joram’s (2007) findings, although published research was considered nearly as credible as student data, fewer than one-third of these educators consulted scholarly or practitioner journals.

In contrast, both Honig and Coburn (2008) and Farley-Ripple (2012) found that district administrators consulted a range of evidence including student data (e.g., standardized test scores), but also site observations, social science research, and evaluation. Administrators with well-developed conceptions of research based the quality of research on scientific and theoretical rigor, whereas those with less developed conceptions focused exclusively on a single factor like the demographic fit between a research study and the school context or the researcher’s reputation ( Coburn and Talbert, 2006 ). More recent studies have focused on identifying educators’ conceptions of useful research, finding that educators prefer practical guide books to empirical studies ( Penuel et al., 2018 ) and that research is most useful when educators viewed it as compatible with their existing practice and could be readily observed it in use elsewhere ( Neal et al., 2018 ).

Based upon theory and the literature we expect educators’ conceptions of research to be broad, reflecting a range of types of evidence (e.g., local data, other schools’ programs and practices, empirical studies) and sources (e.g., books, practitioner journals, peer-reviewed journals). We might also expect educators to consider aspects of reliability, validity, fit (e.g., sample overlap with student body demographics), and credibility (e.g., the reputation of the researcher) in their conceptions of research. Aiming to capture this breadth, Reis-Jorge (2007) found that teachers conceptions of research were either functional and focused on what it does , or structural and focused on how it was produced .

Conceptions of Research in Federal Educational Policies

Federal, state, and local educational policies govern educators’ work. In this paper, we focus on federal educational policies because these apply broadly across public education, and restrict our focus on U.S. policies because only these are likely to influence the U.S.-based educators in our sample (for national education policy outside the US, see Blackmore, 2002 ; Canada: Campbell, Pollock Briscoe; Carr-Harris & Tuters, 2017 ). We specifically focus on two policies, NCLB and ESSA, which are both reauthorizations of the Education and Secondary Education Act (ESEA) originally passed in 1965 to increase equity in public schools through the distribution of federal funds ( Kantor, 1991 ). Since its original passage, reauthorizations of the law have shifted its focus areas (as we describe below), but the charge to distribute funds remains the same ( Farley-Ripple, May, Karpyn, Tilley, & McDonough, 2018 ).

For many public schools, funds associated with federal policies like NCLB or ESSA (e.g., Title 1 – Improving the Academic Achievement of the Disadvantaged) significantly contribute to their operating budget. Access to these funds requires that educators engage in instrumental use of research to make decisions about programs and practices, but differ in the details: NCLB emphasizes the use of scientifically-based research, while ESSA emphasizes the use of evidence-based interventions ( Farley-Ripple et al., 2018 ). The NCLB defined scientifically-based research as:

…the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs; and (B) includes research that—(i) employs systematic, empirical methods that draw on observation or experiment; (ii) involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn; (iii) relies on measurements or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same or different investigators; (iv) is evaluated using experimental or quasi experimental designs in which individuals, entities, programs, or activities are assigned to different conditions and with appropriate controls to evaluate the effects of the condition of interest, with a preference for random-assignment experiments, or other designs to the extent that those designs contain within-condition or across-condition controls; (v) ensures that experimental studies are presented in sufficient detail and clarity to allow for replication or, at a minimum, offer the opportunity to build systematically on their findings; and (vi) has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review (115 STAT. 1964).

Following this definition, a randomized control trial to test the efficacy of an instructional practice is an example of research, while a published collection of program testimonials about the same instructional practice is not.

The ESSA shifted the focus from scientifically-based research to evidence-based interventions, which was purported to increase the implementation of effective interventions and to improve outcomes. ESSA defined an evidence-based intervention as: “… an activity, strategy, or intervention that – (i) demonstrates a statistically significant effect on improving student outcomes or other relevant outcomes” (p. 388). Following this definition, a character education program that shows positive statistically significant effects on behavior is an example of an evidence-based intervention, while a school-developed student peer-to-peer mentoring program solely evaluated via program attendance is not. ESSA further outlines criteria used to evaluate evidence-based interventions:

(I) strong evidence from at least one well-designed and well-implemented experimental study; (II) moderate evidence from at least one well-designed and well-implemented quasi-experimental study; or (III) promising evidence from at least one well-designed and well-implemented correlational study with statistical controls for selection bias; or (ii) (I) demonstrates a rationale based on high-quality research findings or positive evaluation that such activity, strategy, or intervention is likely to improve student outcomes or other relevant outcomes; and (II) includes ongoing efforts to examine the effects of such activity, strategy, or intervention (p. 388)

These criteria focus on evaluating components of scientifically-based research studies (e.g., design, sample size, and effects), and favor those studies with experimental or quasi-experimental designs that allow for causal inference.

To date, there is limited research examining the alignment of educators’ conceptions of research to those outlined in federal educational policies. One exception is Davidson et al. (2018) , who found that when research was pre-defined, educators in a nationally representative sample named multiple sources of ‘useful’ research: books (57%), research/policy reports (16%), journal articles (13%), and other sources (14%). However, of the unique sources with original analyses identified by educators, less than 20% of sources met criteria for strong, moderate, or promising evidence as outlined in ESSA. This finding supports two-communities theory’s proposition that educators and policymakers represent different communities, each with differing understandings about what research is. However, more research is needed to determine exactly where this mismatch (or lack of alignment) occurs.

Sample and Data Collection

Data for this study were collected from public school building-, district-, and intermediate school district (ISD)-level educators throughout Michigan, within the context of a larger study. Both a snowball referral strategy and a social network-based strategy were used to recruit participants. The snowball recruitment strategy began by interviewing the superintendent of each of two intermediate school districts, with an invitation to suggest other study participants at the ISD, district, and building levels. The suggested participants were invited to participate in an interview, and were also invited to suggest participants. This yielded 34 interview participants from referrals originating in one county, and 40 from referrals originating in another county. Separately, the network-based strategy began by using a relational chain design to identify educators who serve as knowledge brokers. A random sample of Michigan principals and superintendents were asked who they talk to for information about school programs. Those named as sources of information were contacted and asked the same question, and this process was repeated up to 11 times, thereby identifying chains of knowledge brokers. From all educator knowledge brokers located using this approach, to capture perspectives from elsewhere in the state, we purposively sampled 24 from 17 counties to participate in interviews. Together, these strategies yielded a sample of 98 educator interviews.

Here we focus on an analytic sample of 90 interviews with educators across 21 counties, 19 districts, and 16 school buildings. 1 The sample was primarily white (N = 87.78%), non-Hispanic (N = 97.8%), and female (N = 67.78%). Interviewees had worked in their current district for an average of 11.39 years, and in their current job for an average of 7.21 years, holding a range of academic qualifications (BA = 42.22%, MA = 23.33%, Ph.D., Ed.D., or Professional = 34.44%; see Table 1 ).

Participant Demographics

Interviews were conducted between Fall 2015 and Spring 2016, and were conducted in-person (N = 79) and by phone (N = 11). All interviews were recorded and transcribed with the consent of the interviewee, who each received a $30 Amazon.com gift card. The interview focused on how school districts acquired information about school programs, interventions, policies, and practices, and the role that different people and organizations played in the process. However, the current study focuses narrowly on educators’ responses to one question that elicited many thoughts: When you hear the word “research” in the context of school-based programs, what kinds of things do you think of?

Data Analysis

To address the first research question (i.e., How do educators conceptualize research?), open, axial, then selective coding were used to analyze the data. First, open coding involves examining, comparing, conceptualizing, and categorizing data ( Creswell, 2014 ). In this step, two coders independently reviewed each interview to gain additional familiarity with the data. 2 Quotes or phrases that captured participants’ conceptualization(s) of research were identified and initial codes were generated. Each interview could receive more than one code. The two coders met to discuss, revise, and reach consensus on initial codes. Next, axial coding involves grouping or theming data based upon patterns in the initial codes. In this step, the two coders met to group initial codes, focusing on codes that were observed across at least 10% (n = 9) interviews, then they met to discuss, revise, and reach consensus on broad themes. Initial codes were grouped to create 2 broad themes, Research Process and Research Products, which closely mirror Reis-Jorge’s (2007) distinction between structural and functional aspects of research (see Table 2 ). Finally, selective coding involves identifying and describing each theme and their interrelationships ( Creswell, 2014 ). In this step, the two coders met to identify the definition for each theme, supported by example quotes from the data.

Educators’ Conceptions of Research Themes

To address the second research question (i.e., To what extent do educators’ conceptions of research align with recent U.S. federal educational policies?), the two coders coded the NCLB and ESSA definitions of research, aligning them with the codes identified in their analysis of educator interviews when possible (see Table 3 ). For example, the NCLB scientifically-based research definition clause “relies on measurements or observational methods that provide reliable and valid data” was coded as “Data,” which had previously been identified as a code in the educator interviews. Then the coders reviewed the policy definitions to identify key concepts that did not fit any of the codes identified in the educator interviews, and developed an additional set of codes. For example, NCLB discusses the need to “test the stated hypotheses,” but because no educators had mentioned hypotheses in their interviews, an applicable code did not already exist. In such cases, a new code was created. This two-stage policy coding process ensured that alignments between educators’ conceptions of research and policy definitions could be identified, but that elements of policy definitions not present in educators’ conceptions could also be identified. The final set of codes was grouped into two larger themes: research process and research products.

Alignment of Educators’ Conceptions of Research with NCLB and ESSA Policy Definitions

The final column of Table 3 reports, for each code, the number of interviews in which the educators’ conception of research aligned with the policy definition. We did not expect any instances of verbatim alignment, that is, that educators would use the exact policy language in their interview responses. Therefore, an educator’s conception is counted as aligned with policy here when the educator mentioned one or more elements of the relevant policy excerpt. For example, although ESSA provides very detailed guidance about appropriate research designs, educators were counted as having policy-aligned conceptions regarding design if they discussed one or more of the design features (e.g. random assignment, experimental design, control group) identified by the policy.

Educators had broad conceptions of research that were grouped around two major themes: research process (e.g., design, methods, etc.) and research products (e.g., data, outcomes, etc.). Process refers to how research is conducted , including who conducts the research, how research is designed, and how data is collected. Products refer to what results from the research process , including data and outcomes, as well as educators’ evaluations of these products (e.g., fit and credibility). We find that educators often talked about multiple aspects of process and products, and there was variation in the ways in which they discussed them. These same themes emerged from the NCLB and ESSA policies, but their scope was more focused. For instance, within the research process, while educators discussed a range of different research designs (e.g. meta-analysis, evaluations, experimental and quasi-experimental designs), policies focused more narrowly on experimental or quasi-experimental designs.

How do educators conceptualize research?

In total, 45 (50%) of interviews discussed one or more categories of the research process, however each category appeared in only 10 (11.11%) to 17 (18.89%) interviews.

Investigators

Investigators refers to the person or organization who conducts a systematic investigation. Investigators were mentioned in 10 (11.11%) interviews. Educators discussed various types of investigators, including universities or university employees who produced empirical research (n = 5), and teachers and principals who were involved in school or district-based research that could directly inform school-level practices (n = 4). For example, one educator shared:

You have your purely academic research that you might find in a peer reviewed journal ….at the same time we also like to look at action research and build capacity within teachers and principals to do that kind of research within their own districts and communities…

Educators also mentioned researchers and practitioners as co-investigators in research-practice partnerships (n = 4) They noted the desire for university researchers to understand and incorporate problems of practice in their work, and the desire for researchers to be involved in practice settings to help address concerns of practitioners. For instance, one educator shared:

I want the researcher at that table while we sit there and go oh my gosh now what, right. and I want the researcher there because I want the researcher to understand a genuine problem of practice. Does that make sense?

Design refers to the logical structure of systematic inquiry guided by a research question and was mentioned in 17 (18.89%) interviews. While design appeared in a number of interviews, there was great variation in the specific types of designs that were discussed. Several educators (n = 5) identified meta-analysis as a significant research design for them because it includes a process of synthesizing and identifying important information from existing research:

We try to ascribe to the scientifically-based research and definition, where, you know, the studies are done basically with a code that, you know are required and that it includes more than one piece of research. So, a meta-analysis. So, you know, when you think of the [author] work, which I think has really taken us to another level of how we look at the, you know, research for teachers.

A few educators described evaluations as a type of design (n = 3). Other examples of design included experiments or quasi-experiments (n = 2) or mentions of specific features typically associated with designs like controlled studies (n = 6) or pre-post tests (n = 4):

So that might be you know something that’s on the promising practices list or the Department of Ed’s best practices list… something that has, maybe its not you know a double blind random control study but something that has some evidence base behind it that shows some kind of results.

Another educator shared:

“so, again, I think of someone who’s had a control group, someone who’s had- you know, and then measure the control group and measure the strategy and seeing if there’s a difference in outcomes.”

Methods refers to the process and tools for collecting information to investigate a research problem or question and was mentioned in 13 (14.44%) interviews. Educators focused on several different aspects of research methods including sample size (n = 3), data collection (n = 4) and data analysis (n = 3). When discussing sample size, educators tended to equate larger samples with higher quality evidence:

I’m not interested in very small studies unless we can combine those studies to you know an aggregate or something like that. So I need to be able to count on the information I’m going to get from the research that will help me understand that whatever it is we’re studying is going to be effective for students.

They also focused on conventional modes of data collection (e.g. surveys) that closely resemble the methods used to collect performance data from students (e.g. standardized tests):

I think of surveys…that would be a parent survey, that would be a student survey, it’d be a staff survey, and community survey, all the stakeholders involved.

Implementation

Implementation refers to the process of implementing an intervention or practice to achieve a desired outcome and was mentioned in 15 (16.67%) interviews. Educators often discussed fidelity, occasionally only in passing (n = 3), but more often to emphasize importance of contextual factors in implementation (n = 5):

I’ve always worked in grants where you had to say obviously do everything with fidelity, which I totally agree with, but sometimes when you’re choosing an evidence-based strategy, there is little room for flexibility and sometimes what is a part of that strategy may not work with the target population you’re dealing with. So not every strategy a hundred percent fits every population.

Educators also frequently discussed wanting to see implementation in action (n = 6), for example, as evidence that implementing a program was possible and effective:

For me research is again seeing the program in action at a school, being able to go to school districts websites to find out if the program worked or if it didn’t. in other words, hearing the success stories.

In total, 74 (82.22%) interviews mentioned one or more categories of research products, however each category appeared in only 17 (18.89%) to 47 (52.22%) interviews

Data refers to the information that comes from systematic investigation and was mentioned in 22 (24.44%) interviews. Participants were consistent in the way they discussed data, frequently describing it as an intermediate product that facilitated development of other types of products, like outcomes or evidence-based practices. In most cases participants mentioned data in general terms, for example noting that:

They [Michigan Department of Education multi-tiered system of support] have a wealth of data that is based upon research-based practices that shows marked increases in student achievement.

However, when participants were more specific about the type of data, they were focused on quantitative data (n = 3):

I think of data. I think of stats. I think of looking at the data, looking at the impact of what is done from a climate perspective.

Outcomes refer to the documented results of a research study or an implementation effort and were mentioned in 47 (52.22%) interviews, more often than any other aspect of research. Educators’ mentions of outcomes included student achievement, student behavior, and school climate, and at times were the first thing that came to mind in their conceptions of research: “[I think of] student outcomes. That’s it.” Indeed, some educators saw demonstrated outcomes as the part of research makes it useful for decision-making around the adoption of programs or practices:

so it’s always helpful to know that yes they’ve tested it here and there and students reported this and that and therefore there was a x percentage of gain or loss or whatever you’re- so to see facts and figures is helpful in deciding.

Thus, for educators, whether research or the use of an evidence-based intervention yields desirable outcomes is a key feature of what research is.

In many cases (n = 15), educators’ discussions of research outcomes focused specifically on evidence-based practices, which they sometimes also called best practices: “I’m thinking of like best practices, things that have been known to work, not necessarily something you purchase.” Educators described challenges in identifying best practices because of the need to interpret research activities as part of the process:

But I also know teachers aren’t researchers, so we have a hard time understanding what is a best practice because you can find research studies with really small sample sizes, research studies with really small effect sizes and be misled, which is why the [author] work was- with the [book] was really good cause it was research on research. So he looked at what does all the research say on class size, not just one…

This quote also suggests that certain ways of presenting findings may be important for helping educators to identifying best practices. Things like meta-analyses or research summaries that evaluate available evidence can reduce burden on teachers to interpret research.

Fit refers to the degree to which research is compatible with school or district context and was discussed in 22 (22.44%) interviews. Educators mentioned research being compatible with context (n = 10), needs (n = 6), resources (n = 3) and demographics (n = 7). In some cases, educators focused on the need for specificity, describing how research may not be appropriate for all contexts, and about challenges locating research that was conducted on populations demographically similar to their own:

I look at what the research was done on, like what kind of demographics and would it be transferrable to our population. So that’s something to always keep in mind.

However, in other cases, educators seemed to recognize the value of research that is generalizable, and discussed considering whether the research is transferable to multiple populations:

you know again you have to look at the populations and everything that the research was done on to make sure that it is you know transferable to any population or what population it, you know the research was done on.

Credibility

Credibility refers to research conducted, promoted or disseminated by a reputable or trusted source and was mentioned in 17 (18.89%) interviews. Educators most often thought about research from universities (n = 4), and occasionally referred to research conducted by specific individuals (n = 3). Participants considered the source when consuming research as a critical factor in evaluating its quality, in some cases placing it above more conventional markers of quality such as peer review:

So even if it has not been published formally in academia, I think that it carries more weight when you can say this person works at [university 1 or university 2] and they’ve found this to be a valid and reliable program, that goes farther. However, I can also go down the hall and talk to five different teachers about practices that are evidence-based that they see working and it may not be a formalized product or program and to me there’s a lot of value in that and that goes back to teachers trusting each other…

Group differences

Although we focused on conceptions of research held by educators who play a role in selecting and deciding to use school-based programs, this can still represent quite a diverse range of individuals working in different roles and in different local contexts. These differences may systematically impact how educators conceptualize research. To explore this possibility, we compared the conceptions held by educators in different groups in three ways. First, we compared educators working in each of the two counties where the majority of our sample was drawn. Second, we compared educators working at different levels of the public education system: county-level intermediate school districts, local school districts, and individual school buildings. Third, we compared educators in executive roles (i.e. principals and superintendents) to those in non-executive roles. We did not observe significant differences between any of these groups in the frequency with which educators mentioned either the process or product aspects of research. This finding of no group differences lends support for viewing educators as a single community within the lens of two-communities theory.

To what extent do educators’ conceptions of research align with NCLB and ESSA?

Many of the components of educators’ conceptions of research also appear in policy definitions of research, however we observed only partial alignment between how educators discuss these components and how they appear in policy. Typically, educators’ conceptions were broader and more inclusive, while policy definitions were narrow and precise. Additionally, educators’ conceptions of research included some components that do not appear in policy, and policy definitions refer to concepts that educators did not mention. Thus, providing some support for a practice-policy gap and two-communities theory, we find that educators’ conceptions of research align only partially with federal policy definitions.

Like educators, NCLB and ESSA also discuss both design and methods as key features of research. Policy definitions of research design were quite detailed and specific, focusing on a narrow set of experimental or quasi-experimental designs, while not referencing and thus implicitly excluding other types of designs (e.g. meta-analysis, descriptive, etc.). As a result, although educators mentioned design-related features of research in 17 interviews, only 9 of these interviews referred to one or more aspects of design described in policy. Policy definitions of research methods were more general, focusing not on specific methods, but broadly on data collection, multiple measurements across studies, and data analysis. Thus, all 13 educators who discussed methods-related features of research in their interviews referred to one or more aspects of method described in policy. Therefore, educators’ conceptions of research are aligned with policy to the extent that they both consider design and method, but the alignment on these aspects is partial.

These policies’ consideration of the research process was restricted to the actual research activities, however educators’ conceptions of the process of research also included parts of the process that both precede (the identity of the investigator) and follow (implementation) the research activities themselves. For policies defining research, these parts of the process fall outside the scope of what research is; research depends on what is done, but not who is doing it, or whether the research is subsequently used. This highlights a misalignment between educators and policy, with educators viewing research as a more encompassing process.

Similarly, NCLB defines research as a hypothesis-driven process, that is, a set of activities designed to test hypotheses and thereby draw general conclusions. However, none of the educators in this sample mentioned the role of hypotheses or hypothesis testing when describing their understanding of research. This highlights a second misalignment between educators and policy, with educators conceptualizing research more broadly as including a range of empirical activities (i.e. data gathering), but policy more narrowly defining it as a specifically intended to test a priori hypotheses.

Like educators, NCLB and ESSA discuss data, outcomes, and credibility as key features of research. Policy definitions of data focused specifically on reliable and valid data, while educators discuss data more broadly. As a result, although educators mentioned data in 22 interviews, none of those interviews mentioned reliable or valid data. Thus, educators’ conceptions of data did not align with policy definitions. Policy definitions of outcomes were more general, focusing on outcomes relevant to educational activities and programs, and those that improve student outcomes or other relevant outcomes. Thus, all 47 educators who discussed outcomes in their interviews aligned with policy because they referred to those relevant to educational activities and programs or those that improve student outcomes. Lastly, policy definitions of credibility focused on external review via a peer-review journal or independent experts, while not referencing other ways to establish credibility such as through professional organizations or colleagues. As a result, although educators mentioned credibility of research in 17 interviews, only 6 of these interviews referred to credibility as described in policy. Thus, educators’ conceptions of credibility partially aligned with federal policy definitions.

Policy consideration of research products was restricted to data, outcomes, and credibility. However, educators’ conceptions of research products also included their judgment of those products (i.e., fit). Thus, while educators’ conceptions of research products included not only products of the research itself, but also secondary products, these fell outside the scope of research as defined by policy, and highlights a misalignment between educators and policy.

Although U.S. federal policies encourage educators to use research (ESSA, 2005; NCLB, 2002), few studies have focused on how educators conceptualize research and whether their conceptions align with these policies (see Davidson et al., 2018 for an exception). Indeed, despite a growing literature on gaps among the research, practice, and policy communities in many fields beyond education, it remains rare to explicitly study how each of these communities conceptualize and define research. Two-communities theory ( Caplan, 1979 ; Locock & Boaz, 2004 ; Farley-Ripple et al., 2018 ) suggests that because educators and federal policymakers work in different social communities and emphasize different types of research use ( Weiss, 1979 ; Weiss & Bucuvalas, 1980 ), there may be a practice-policy gap in the ways that these two groups conceptualize research. This study extends prior research and builds upon two-communities theory by examining educators’ conceptions of research among Michigan educators, and assessing how these conceptions align with the NCLB definition of scientifically-based research and ESSA definition of evidence-based interventions. Exploring educators’ conceptions of research and their alignment with federal policy can highlight opportunities for promoting educators’ use of research, and relatedly bridging the practice-policy gap .

Educators’ Conceptions of Research.

In this study, we found that educators’ conceptions of research included two main themes – research process and research products – which did not vary across educators at different levels, in different roles, or in different local contexts. When educators conceptualized research, they were less likely to mention research process than products. Conceptions of the research process included the full range of process activities, starting with the investigators, and including design, methods, and implementation. Conceptions of research products were similarly inclusive, ranging from immediate products (e.g. data and outcomes) to evaluations of those products (e.g. fit and credibility). These findings are consistent with prior literature, which highlighted the broad scope of educators’ conceptions of research ( Coburn & Talbert, 2006 ; Farley-Ripple, 2012 ; Finnigan et al., 2013 ; Honig & Coburn, 2008 ; Joram, 2007 ). Specifically, related to process, these studies have found that educators focus on broad aspects of design (e.g., empirical investigation), and methods (e.g., psychometric properties and use of multiple measures). However, unlike the past literature, we also found that educators’ conceptions of research include more long-range processes (e.g., implementation). Related to products, these studies have found that educators focus on a range of data and outcomes as well as perceived fit between a research study and school context, and credibility of research or researchers. In our study, educators’ conceptions of the products of research were largely consistent with the past literature.

Alignment with NCLB and ESSA.

At the broadest level, educators’ conceptions of research aligned with the NCLB definition of scientifically-based research and the ESSA definition of evidence-based interventions in that they focused on aspects of both process and products. However, a more detailed look revealed that NCLB and ESSA offer a much narrower conception of the research process and research products than educators, suggesting some evidence of a practice-policy gap . For instance, NCLB and ESSA focused on experimental, quasi-experimental, and correlational designs, whereas educators also considered meta-analytic and evaluation designs. Similarly, NCLB and ESSA focused on reliable and valid data, whereas educators often discuss data in more general terms. Compared to these policies, educators also had a much broader view of what makes research credible. Whereas these policies primarily view credibility as deriving from publication in a peer-reviewed outlet, educators reported that the status of the author (e.g. at a well-known university) or the source of the information (e.g. a trusted colleague) can also serve as markers of credibility. This misalignment in conceptions of credibility may derive from another misalignment: the slowness of research, and the immediacy of educators’ needs. Research can take a long time to make its way through the peer review and publication processes, while educators simply cannot wait and must turn to other indicators of credibility to facilitate their quicker access to research.

Unlike aspects of research such as design and credibility, because NCLB and ESSA have a relatively broad conception of methods and outcomes, their definitions of these aspects of research aligned more closely with educators’ conceptions. Educators’ particularly frequent focus on outcomes, which aligned with policy, is perhaps not surprising because the central aim of these policies is to improve student outcomes. Indeed, the names of the policies themselves – No Child Left Behind and Every Student Succeeds – explicitly reference outcomes in the form of reducing achievement gaps and promoting universal achievement, respectively. Moreover, the ultimate metric against which educators’ adherence to these policies is measured is evidence of improved student outcomes. That is, although these policies advocate research use, this is a proximal goal and there has generally not been a concerted effort to measure or reward educators for using research. Instead, measurement and rewards focus on the more distal goal of improving outcomes. Thus, we observe high degrees of alignment between educators on policy that improving outcomes – “moving the needle” – is critical, but somewhat less alignment on what it means to use research to achieve such a goal.

Implications and conclusions.

These findings have implications for federal policymakers, educators, and researchers. Our finding that educators’ conceptions of research are often misaligned or only partially aligned with definitions of research in education policy suggests the need for more bidirectional communication between policymakers and educators. Federal policymakers can be clearer about what kinds of research “count,” and take steps to help educators identify research that meets this definition and thus fulfills policy requirements. For example, definitions of research that appear in education policies might be accompanied by short practitioner-friendly checklists, like the ESSA summary of recommended study criteria, that can be used to assess when and to what extent a piece of evidence satisfies policy requirements. Additionally, federal policymakers and educators can work together when education policy is being developed to ensure that they are speaking the same language and that the resulting policy documents will be understandable in both the policy and practitioner communities.

Similarly, researchers (especially education researchers) can conduct and disseminate their research in a way that matches the conceptions of both educators and policymakers. Researchers are already attuned to the report features of their work that are essential components of education policy definitions of research like methods (e.g., data collection and analysis). However, researchers should also routinely report things like validity, reliability, and details regarding experimental designs that attend to federal policy definitions of research. Researchers should also attend to the features of their work that matter to educators, but are not mentioned in federal education policy. Specifically, researchers should include information about how to implement research findings and should report the demographic and contextual details that allow educators to assess the findings’ fit and generalizability. Recognizing that policy and practice audiences approach research from different, but overlapping, perspectives and considering these differences when conducting and reporting research can maximize the potential for bridging the research-practice gap. To improve dissemination efforts, it might be helpful to develop checklists that researchers can follow to ensure that their reporting includes components that are consistent with both federal policy and educators’ conceptions of research.

Limitations and Future Directions.

This study represents an initial exploration into educators’ conceptions of research and their alignment with federal education policy, and thus it must be interpreted with some limitations in mind. The sample was restricted to educators in two predominantly white Michigan counties; future studies would benefit from exploring educators’ conceptions of research in other locations and in demographically diverse settings. This study examined the extent to which educators’ conceptions of research aligned with definitions provided by two recent U.S. federal educational policies. However, future research might extend the current study by examining how educators’ conceptions of research align with additional resources provided by policymakers. For example, ESSA includes supplementary non-regulatory guidance for using evidence that highlights aspects like sample size, fit with demographics and setting, and What Works Clearinghouse standards. Future research could also explore how educators’ conceptions align with other educational policies like state or local mandates, and how this localized alignment relates to the research-practice gap. Additionally, this study is focused on United States federal education policy; future research should examine educators’ conceptions of research and their alignment with education policy in other national contexts. Lastly, future research should extend this study by exploring the extent to which differences between educators’ and policymakers’ understandings of research influence how and when educators use research. For example, future studies could determine whether educators who have conceptions of research that are more aligned with federal policy are more likely to engage in instrumental forms of research use that align with expectations in federal policy (e.g., using What Works Clearinghouse to identify programs and practices).

The purpose of this study was to understand how public-school educators conceptualize research, and how these conceptions aligned with recent U.S. federal educational policies, NCLB and ESSA. Through the analysis of interview data, findings suggested that educators’ conceptions of research were broad and multifaceted. Educators most often discussed research outcomes , but to varying degrees discussed many aspects of the research process and the products of that process. These conceptions only partially aligned with definitions of research in federal policy (i.e. NCLB and ESSA), which tended to be much narrower. These findings suggest that educators and policymakers have some overlap in their conceptions of research, but also approach research from different perspectives.

Key messages:

• We examine how 90 U.S. educators define research and how these definitions align with federal policy
• Educators’ definitions of research reflected two major themes: process and products.
• Educators’ definitions of research were broad, while policy definitions were narrow and precise.
• Findings have implications for both federal policymakers and educational researchers.

Acknowledgments:

The authors would like to Camren Wilson for his helpful feedback on the study results. We would also like to thank all participating educators for contributing their time and perspectives to this study.

Funding Details:

This study was funded by Officer’s Research Award (#182241) and a Use of Research Evidence Award (#183010) from the William T. Grant Foundation. Additional support for this research also came from an R21 research grant from the National Institute of Mental Health (#1R21MH100238-01A1).

Conflicts of Interest: The authors declare that there is no conflict of interest.

1 An unexpected data loss event resulted in the loss of a random 9 interviews.

2 These coders are the first and second authors, who also served as interviewers during the data collection stage, and therefore already had familiarity with the data.

• Bartels N, 2003, How teachers and researchers read academic articles . Teaching and Teacher Education , 19 , 737–753. [ Google Scholar ]
• Barwick MA, Barac R, Akrong LM, Johnson S, & Chaban P, 2014, Bringing evidence to the classroom: Exploring educator notions of evidence and preferences for practice change . International Education Research , 2 , 4 , 1–15. [ Google Scholar ]
• Blackmore J, 2002, Is it only ‘What works’ that ‘Counts’ in New Knowledge Economies? Evidence-based Practice, Educational Research and Teacher Education in Australia . Social Policy and Society , 1 , 257–266. [ Google Scholar ]
• Cain T, 2015, Teachers’ engagement with research texts: beyond instrumental, conceptual, or strategic use . Journal of Education and Teaching , 41 , 5 , 478–492. doi: 10.1080/02607476.2015.1105536 [ CrossRef ] [ Google Scholar ]
• Campbell C, Pollock K, Briscoe P, Carr-Harris S & Tuters S, 2017, Developing a knowledge network for applied education research to mobilise evidence in and for educational practice . Educational Research , 59 , 209–227. [ Google Scholar ]
• Caplan N, 1979, The two communities theory and knowledge utilization . The American Behavioral Scientist , 22 , 459–471. [ Google Scholar ]
• Coburn CE, & Talbert JE, 2006, Conceptions of evidence use in school districts: Mapping the terrain . American Journal of Education , 112 , 4 , 469–495. [ Google Scholar ]
• Coburn CE, Toure J, & Yamashita M, 2009, Evidence, interpretation, and persuasion: Instructional decision making at the district central office . Teachers College Record , 111 , 4 , 1115–1161. [ Google Scholar ]
• Creswell JW, 2014, Research Design: Qualitative, Quantitative and Mixed Methods Approaches . Thousand Oaks, CA: SAGE Publications. [ Google Scholar ]
• Davidson KL, Penuel WR, & Farrell CC, 2018, What Counts as Research Evidence? How Educational Leaders’ Reports of the Research they Use Compare to ESSA Guidelines . Paper presented at the Society for Research on Educational Effectiveness, Washington, D.C. [ Google Scholar ]
• Every Student Succeeds Act (ESSA) of 2015, 20 U.S.C.A. § 6301 et seq . (U.S. Government Publishing Office, 2015). [ Google Scholar ]
• Farley-Ripple EN, 2012, Research use in school district central office decision making: A case study . Educational Management Administration & Leadership , 40 , 6 , 786–806. [ Google Scholar ]
• Farley-Ripple EN, May H, Karpyn A, Tilley K, & McDonough K, 2018, Rethinking connections between research and practice in education: A conceptual framework . Educational Researcher 47 , 235–245 [ Google Scholar ]
• Finnigan KS, Daly AJ, & Che J, 2013, Systemwide reform in districts under pressure: The role of social networks in defining, acquiring, using, and diffusing research evidence . Journal of Educational Administration , 51 , 4 , 476–497. [ Google Scholar ]
• Hill HC, 2001, Policy is not enough: Language and interpretation of state standards . American Educational Research Journal , 38 , 289–318. [ Google Scholar ]
• Honig MI, & Coburn C, 2008, Evidence-based decision making in school district central offices: Toward a policy and research agenda . Educational Policy , 22 , 4 , 578–608. [ Google Scholar ]
• Green LW, Ottoson JM, Garcia C, & Hiatt RA, 2009, Diffusion theory and knowledge dissemination, utilization, and integration in public health . Annual Review of Public Health , 30 , 151–174. [ PubMed ] [ Google Scholar ]
• Joram E, 2007, Clashing epistemologies: Aspiring teachers’, practicing teachers’, and professors’ beliefs about knowledge and research in education . Teaching and teacher education , 23 , 2 , 123–135. [ Google Scholar ]
• Kantor H, 1991, Education, social reform, and the state: ESEA and federal education policy in the 1960s . American Journal of Education , 100 , 1 , 47–83. [ Google Scholar ]
• Locock L, Boaz A, 2004, Research, policy, and practice – worlds apart? Social Policy and Society , 3 , 375–384. [ Google Scholar ]
• Lomas J, 2007, The in-between world of knowledge brokering . BMJ: British Medical Journal , 334 , 7585 , 129–132. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Long ACJ, Sanetti LMH, Collier-Meek MA, Gallucci J, Altschaefl M, & Kratochwill TR, 2016, An exploratory investigation of teachers’ intervention planning and perceived implementation barriers . Journal of School Psychology , 55 , 1–26. [ PubMed ] [ Google Scholar ]
• Malin JR, Brown C, Trubceac AS, 2018, Going for broke: A multiple-case study of brokerage in education . AERA Open , 4 , 1–4. [ Google Scholar ]
• Neal JW, Mills KJ, McAlindon K, Neal ZP, & Lawlor JA, in press, Multiple audiences for encouraging research use: Uncovering a typology of educators . Educational Administration Quarterly . [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Neal JW, Neal ZP, Kornbluh M, Mills KJ, & Lawlor JA, 2015, Brokering the research-practice gap: A typology . American Journal of Community Psychology , 56 , 3/4 , 422–435. doi: 10.1007/s10464-015-9745-8. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Neal JW, Neal ZP, Lawlor JA, Mills KJ, & McAlindon K, 2018, What Makes Research Useful for Public School Educators? Administration and Policy in Mental Health and Mental Health Services Research , 45 , 3 , 432–446. doi: 10.1007/s10488-017-0834-x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Newman J, Cherney A, & Head BW, 2016, Do policymakers use academic research? Rexamining the “two communities” theory of research utilization . Public Administration Review , 76 , 1 , 24–32. [ Google Scholar ]
• No Child Left Behind (NCLB) Act of 2001, 20 U.S.C.A. § 6319 et seq . (2002).
• Nutley S, Walter I, & Davies HTO, 2003, From knowing to doing: A framework for understanding the evidence-into-practice agenda . Evaluation , 9 , 125–148. doi: 10.1177/1356389003009002002 [ CrossRef ] [ Google Scholar ]
• Penuel WR, Allen A-R, Coburn CE, & Farrell C, 2015, Conceptualizing research–practice partnerships as joint work at boundaries , Journal of Education for Students at Risk , 20 , 182–197. [ Google Scholar ]
• Penuel WR, Briggs DC, Davidson KL, Herlihy C, Sherer D, Hill HC, Farrell CC, & Allen A, 2017, How school district leaders access, perceive, and use research , AERA Open , 3 , 1–17. [ Google Scholar ]
• Penuel WR, Farrell CC, Allen A-R, Toyama Y, Coburn CE, 2018, What research district leaders find useful . Educational Policy , 32 , 540–568. [ Google Scholar ]
• Spillane JP, Reiser BJ, & Reimer, 2002, Policy implementation and cognition: Reframing and refocusing implementation research . Review of Educational Research , 72 , 387–431. [ Google Scholar ]
• Reis-Jorge J, 2007, Teachers’ conceptions of teacher-research and self-perceptions as enquiring practitioners—A longitudinal case study . Teaching and Teacher Education , 23 , 402–417. [ Google Scholar ]
• Weiss CH, 1979, The many meanings of research utilization . Public Administration Review , 39 , 426–431. [ Google Scholar ]
• Weiss CH, & Bucuvalas MJ, 1980, Social science and decision-making . New York: Columbia University Press. [ Google Scholar ]
• Weiss CH, Murphy-Graham E, & Birkeland S, 2005, An alternate route to policy influence: How evaluations affect D.A.R.E . American Journal of Evaluation , 26 , 12–30. [ Google Scholar ]

• Subject List
• Take a Tour
• For Authors
• Subscriber Services
• Publications
• African American Studies
• African Studies
• American Literature
• Anthropology
• Architecture Planning and Preservation
• Art History
• Atlantic History
• Biblical Studies
• British and Irish Literature
• Childhood Studies
• Chinese Studies
• Cinema and Media Studies
• Communication
• Criminology
• Environmental Science
• Evolutionary Biology
• International Law
• International Relations
• Islamic Studies
• Jewish Studies
• Latin American Studies
• Latino Studies
• Linguistics
• Literary and Critical Theory
• Medieval Studies
• Military History
• Political Science
• Public Health
• Renaissance and Reformation
• Social Work
• Urban Studies
• Victorian Literature
• Browse All Subjects

How to Subscribe

• Free Trials

In This Article Expand or collapse the "in this article" section Empirical Perspectives in Education Leadership

Introduction, general overviews.

• Foundational Education Leadership Texts

Related Articles Expand or collapse the "related articles" section about

About related articles close popup.

Lorem Ipsum Sit Dolor Amet

Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Aliquam ligula odio, euismod ut aliquam et, vestibulum nec risus. Nulla viverra, arcu et iaculis consequat, justo diam ornare tellus, semper ultrices tellus nunc eu tellus.

• Coaching and Feedback for Leadership Development
• Distributed Leadership
• Education Finance

Other Subject Areas

Forthcoming articles expand or collapse the "forthcoming articles" section.

• English as an International Language for Academic Publishing
• Girls' Education in the Developing World
• History of Education in Europe
• Find more forthcoming articles...
• Export Citations
• Share This Facebook LinkedIn Twitter

Empirical Perspectives in Education Leadership by James W. Guthrie , Patrick J. Schuermann LAST REVIEWED: 15 December 2011 LAST MODIFIED: 15 December 2011 DOI: 10.1093/obo/9780199756810-0022

Leaders ensure for an organization that it does the right things, and they pay equal attention to ensuring that, within an organization, things are done right. “Leadership” refers to the traits and behaviors of those who direct collectives of human beings, be they formal organizations such as religions, nations, tribes, or clans, or informal groups such as families. A leader is one who pursues personal or group purposes by organizing and motivating the actions of others. Leaders have as potential tools dynamic interactions and conditions such as a vision of an organization’s future, the accumulation and allocation of resources and rewards, the recruitment and training of followers, and the manipulation of incentives. The “inherent quality” of a formal or informal organization will always be further improved by extraordinary leadership—and will not, in the long run, survive unusually poor leadership. In this article is a summary set of references regarding generalized leadership. Subsequent sections provide citations to specific topics within educational leadership. However, there is a caveat here. There are literally thousands of popular books and articles focused on leadership and leadership-related matters such as management, motivation, personal appearance, and how to use one’s time. This bibliography omits references to most of this body of writing because the tendency is for these popular publications to be highly normative and lacking an empirical grounding. The following listings emphasize research and objective reports. There is general expert consensus regarding the scientific or technical part of leadership. Leaders routinely perform a consistent set of functions, no matter which organization they lead: They contribute to shaping an organization’s purposes, create a vision or a roadmap of means for achieving these purposes, select and motivate followers to follow this roadmap, obtain and allocate resources consistent with the plan for achieving organizational purposes, continually evaluate the success of the organization and its members’ performance in pursuing an agreed-upon path, guide midcourse corrections when information suggests such are in order, and represent the organization to its external audiences.

Early leadership studies concentrated on individual traits of successful leaders. These usually produced a laundry list regarding identifiable characteristics, such as physical size, strength, energy, intelligence, education, early training, childhood and family background, ethnicity, and psychological disposition. A historical summary of “trait theory” is important for an understanding of the operating frameworks in the field today ( Bass 1990 ). Bass 2008 and Bennis 2009 provide foundational overviews, including among them the historical evolution, methodologies, and strategies within the field. Core challenges to the ideology and practice of leadership further the reader’s understanding of the field ( Kotter 1999 ). Personal and organizational leadership development that furthers continual community learning can be found in Senge 2006 . These texts provide the reader with a robust overview of the historical framework, practice, and challenges in the broad field of educational leadership.

Bass, Bernard M. 1990. Traits of leadership: A follow-up. In Bass & Stogdill’s handbook of leadership: Theory, research, & managerial applications . 3d ed. By Bernard M. Bass, 81–88. New York: Free Press.

This article provides a summary of the “Trait Theory” of leadership and is important for historical purposes.

Bass, Bernard M. 2008. Bass handbook of leadership: Theory, research and managerial application . 4th ed. New York: Free Press.

This is a comprehensive publication on leadership, providing an overview of the field of study and chapters that explain to readers the historical evolution of the field and crucial writings en route to contemporary research and understanding.

Bennis, Warren G. 2009. On becoming a leader . 4th ed. New York: Basic Books.

This book has served for fifteen years as a classic publication regarding leadership issues, describing qualities that define leadership, people who exemplify it, and strategies that can be pursued to become an effective leader. This recent edition features an introduction on the challenges and opportunities facing 21st-century leaders.

Kotter, John P. 1999. John P. Kotter on what leaders really do . Boston: Harvard Business School Press.

This book describes core issues that reside at the heart of leadership and enables a reader to rethink one’s own relationship to the work of leaders.

Senge, Peter M. 2006. The fifth discipline: The art and practice of the learning organization . Rev. ed. New York: Doubleday.

This book advocates construction, by a leader, of an organization where people expand their capacity to create results they desire, where new patterns of thinking are nurtured, where aspiration is set free, and where people are continually learning how to learn together.

back to top

Users without a subscription are not able to see the full content on this page. Please subscribe or login .

Oxford Bibliographies Online is available by subscription and perpetual access to institutions. For more information or to contact an Oxford Sales Representative click here .

• About Education »
• Meet the Editorial Board »
• Academic Achievement
• Academic Audit for Universities
• Academic Freedom and Tenure in the United States
• Action Research in Education
• Adjuncts in Higher Education in the United States
• Administrator Preparation
• Adolescence
• Advanced Placement and International Baccalaureate Courses
• Advocacy and Activism in Early Childhood
• African American Racial Identity and Learning
• Alaska Native Education
• Alternative Certification Programs for Educators
• Alternative Schools
• American Indian Education
• Animals in Environmental Education
• Art Education
• Artificial Intelligence and Learning
• Assessing School Leader Effectiveness
• Assessment, Behavioral
• Assessment, Educational
• Assessment in Early Childhood Education
• Assistive Technology
• Augmented Reality in Education
• Beginning-Teacher Induction
• Bilingual Education and Bilingualism
• Black Undergraduate Women: Critical Race and Gender Perspe...
• Blended Learning
• Case Study in Education Research
• Changing Professional and Academic Identities
• Character Education
• Children’s and Young Adult Literature
• Children's Beliefs about Intelligence
• Children's Rights in Early Childhood Education
• Citizenship Education
• Civic and Social Engagement of Higher Education
• Classroom Learning Environments: Assessing and Investigati...
• Classroom Management
• Coherent Instructional Systems at the School and School Sy...
• College Admissions in the United States
• College Athletics in the United States
• Community Relations
• Comparative Education
• Computer-Assisted Language Learning
• Computer-Based Testing
• Conceptualizing, Measuring, and Evaluating Improvement Net...
• Continuous Improvement and "High Leverage" Educational Pro...
• Counseling in Schools
• Critical Approaches to Gender in Higher Education
• Critical Perspectives on Educational Innovation and Improv...
• Critical Race Theory
• Crossborder and Transnational Higher Education
• Cross-National Research on Continuous Improvement
• Cross-Sector Research on Continuous Learning and Improveme...
• Cultural Diversity in Early Childhood Education
• Culturally Responsive Leadership
• Culturally Responsive Pedagogies
• Culturally Responsive Teacher Education in the United Stat...
• Curriculum Design
• Data Collection in Educational Research
• Data-driven Decision Making in the United States
• Deaf Education
• Desegregation and Integration
• Design Thinking and the Learning Sciences: Theoretical, Pr...
• Development, Moral
• Dialogic Pedagogy
• Digital Age Teacher, The
• Digital Citizenship
• Digital Divides
• Disabilities
• Distance Learning
• Doctoral Education and Training
• Early Childhood Education and Care (ECEC) in Denmark
• Early Childhood Education and Development in Mexico
• Early Childhood Education in Aotearoa New Zealand
• Early Childhood Education in Australia
• Early Childhood Education in China
• Early Childhood Education in Europe
• Early Childhood Education in Sub-Saharan Africa
• Early Childhood Education in Sweden
• Early Childhood Education Pedagogy
• Early Childhood Education Policy
• Early Childhood Education, The Arts in
• Early Childhood Mathematics
• Early Childhood Science
• Early Childhood Teacher Education
• Early Childhood Teachers in Aotearoa New Zealand
• Early Years Professionalism and Professionalization Polici...
• Economics of Education
• Education For Children with Autism
• Education for Sustainable Development
• Education Leadership, Empirical Perspectives in
• Education of Native Hawaiian Students
• Education Reform and School Change
• Educational Statistics for Longitudinal Research
• Educator Partnerships with Parents and Families with a Foc...
• Emotional and Affective Issues in Environmental and Sustai...
• Emotional and Behavioral Disorders
• Environmental and Science Education: Overlaps and Issues
• Environmental Education
• Environmental Education in Brazil
• Epistemic Beliefs
• Equity and Improvement: Engaging Communities in Educationa...
• Equity, Ethnicity, Diversity, and Excellence in Education
• Ethical Research with Young Children
• Ethics and Education
• Ethics of Teaching
• Ethnic Studies
• Evidence-Based Communication Assessment and Intervention
• Family and Community Partnerships in Education
• Family Day Care
• Federal Government Programs and Issues
• Feminization of Labor in Academia
• Finance, Education
• Financial Aid
• Formative Assessment
• Future-Focused Education
• Gender and Achievement
• Gender and Alternative Education
• Gender, Power and Politics in the Academy
• Gender-Based Violence on University Campuses
• Gifted Education
• Global Mindedness and Global Citizenship Education
• Global University Rankings
• Governance, Education
• Grounded Theory
• Growth of Effective Mental Health Services in Schools in t...
• Higher Education and Globalization
• Higher Education and the Developing World
• Higher Education Faculty Characteristics and Trends in the...
• Higher Education Finance
• Higher Education Governance
• Higher Education Graduate Outcomes and Destinations
• Higher Education in Africa
• Higher Education in China
• Higher Education in Latin America
• Higher Education in the United States, Historical Evolutio...
• Higher Education, International Issues in
• Higher Education Management
• Higher Education Policy
• Higher Education Research
• Higher Education Student Assessment
• High-stakes Testing
• History of Early Childhood Education in the United States
• History of Education in the United States
• History of Technology Integration in Education
• Homeschooling
• Inclusion in Early Childhood: Difference, Disability, and ...
• Inclusive Education
• Indigenous Education in a Global Context
• Indigenous Learning Environments
• Indigenous Students in Higher Education in the United Stat...
• Infant and Toddler Pedagogy
• Inservice Teacher Education
• Integrating Art across the Curriculum
• Intelligence
• Intensive Interventions for Children and Adolescents with ...
• International Perspectives on Academic Freedom
• Intersectionality and Education
• Knowledge Development in Early Childhood
• Leadership Development, Coaching and Feedback for
• Leadership in Early Childhood Education
• Leadership Training with an Emphasis on the United States
• Learning Analytics in Higher Education
• Learning Difficulties
• Learning, Lifelong
• Learning, Multimedia
• Learning Strategies
• Legal Matters and Education Law
• LGBT Youth in Schools
• Linguistic Diversity
• Linguistically Inclusive Pedagogy
• Literacy Development and Language Acquisition
• Literature Reviews
• Mathematics Identity
• Mathematics Instruction and Interventions for Students wit...
• Mathematics Teacher Education
• Measurement for Improvement in Education
• Measurement in Education in the United States
• Meta-Analysis and Research Synthesis in Education
• Methodological Approaches for Impact Evaluation in Educati...
• Methodologies for Conducting Education Research
• Mindfulness, Learning, and Education
• Mixed Methods Research
• Motherscholars
• Multiliteracies in Early Childhood Education
• Multiple Documents Literacy: Theory, Research, and Applica...
• Multivariate Research Methodology
• Museums, Education, and Curriculum
• Music Education
• Narrative Research in Education
• Native American Studies
• Nonformal and Informal Environmental Education
• Note-Taking
• Numeracy Education
• One-to-One Technology in the K-12 Classroom
• Online Education
• Open Education
• Organizing for Continuous Improvement in Education
• Organizing Schools for the Inclusion of Students with Disa...
• Outdoor Play and Learning
• Outdoor Play and Learning in Early Childhood Education
• Pedagogical Leadership
• Pedagogy of Teacher Education, A
• Performance Objectives and Measurement
• Performance-based Research Assessment in Higher Education
• Performance-based Research Funding
• Phenomenology in Educational Research
• Philosophy of Education
• Physical Education
• Podcasts in Education
• Policy Context of United States Educational Innovation and...
• Politics of Education
• Portable Technology Use in Special Education Programs and ...
• Post-humanism and Environmental Education
• Pre-Service Teacher Education
• Problem Solving
• Productivity and Higher Education
• Professional Development
• Professional Learning Communities
• Program Evaluation
• Programs and Services for Students with Emotional or Behav...
• Psychology Learning and Teaching
• Psychometric Issues in the Assessment of English Language ...
• Qualitative Data Analysis Techniques
• Qualitative, Quantitative, and Mixed Methods Research Samp...
• Qualitative Research Design
• Quantitative Research Designs in Educational Research
• Queering the English Language Arts (ELA) Writing Classroom
• Race and Affirmative Action in Higher Education
• Reading Education
• Refugee and New Immigrant Learners
• Relational and Developmental Trauma and Schools
• Relational Pedagogies in Early Childhood Education
• Reliability in Educational Assessments
• Religion in Elementary and Secondary Education in the Unit...
• Researcher Development and Skills Training within the Cont...
• Research-Practice Partnerships in Education within the Uni...
• Response to Intervention
• Restorative Practices
• Risky Play in Early Childhood Education
• Scale and Sustainability of Education Innovation and Impro...
• Scaling Up Research-based Educational Practices
• School Accreditation
• School Choice
• School Culture
• School District Budgeting and Financial Management in the ...
• School Improvement through Inclusive Education
• School Reform
• Schools, Private and Independent
• School-Wide Positive Behavior Support
• Science Education
• Secondary to Postsecondary Transition Issues
• Self-Regulated Learning
• Self-Study of Teacher Education Practices
• Service-Learning
• Severe Disabilities
• Single Salary Schedule
• Single-sex Education
• Single-Subject Research Design
• Social Context of Education
• Social Justice
• Social Network Analysis
• Social Pedagogy
• Social Science and Education Research
• Social Studies Education
• Sociology of Education
• Standards-Based Education
• Statistical Assumptions
• Student Access, Equity, and Diversity in Higher Education
• Student Assignment Policy
• Student Engagement in Tertiary Education
• Student Learning, Development, Engagement, and Motivation ...
• Student Participation
• Student Voice in Teacher Development
• Sustainability Education in Early Childhood Education
• Sustainability in Early Childhood Education
• Sustainability in Higher Education
• Teacher Beliefs and Epistemologies
• Teacher Collaboration in School Improvement
• Teacher Evaluation and Teacher Effectiveness
• Teacher Preparation
• Teacher Training and Development
• Teacher Unions and Associations
• Teacher-Student Relationships
• Teaching Critical Thinking
• Technologies, Teaching, and Learning in Higher Education
• Technology Education in Early Childhood
• Technology, Educational
• Technology-based Assessment
• The Bologna Process
• The Regulation of Standards in Higher Education
• Theories of Educational Leadership
• Three Conceptions of Literacy: Media, Narrative, and Gamin...
• Tracking and Detracking
• Traditions of Quality Improvement in Education
• Transformative Learning
• Transitions in Early Childhood Education
• Tribally Controlled Colleges and Universities in the Unite...
• Understanding the Psycho-Social Dimensions of Schools and ...
• University Faculty Roles and Responsibilities in the Unite...
• Using Ethnography in Educational Research
• Value of Higher Education for Students and Other Stakehold...
• Virtual Learning Environments
• Vocational and Technical Education
• Wellness and Well-Being in Education
• Women's and Gender Studies
• Young Children and Spirituality
• Young Children's Learning Dispositions
• Young Children's Working Theories
• Privacy Policy
• Cookie Policy
• Legal Notice
• Accessibility

Powered by:

• [66.249.64.20|185.80.151.9]
• 185.80.151.9

What is Empirical Evidence?

Published Jan 5, 2018

Empirical evidence is information that researchers generate to help uncover answers to questions that can have significant implications for our society.

Take seatbelts. Prior to their invention, people were killed or maimed in what today we would think of as minor traffic accidents. So smart engineers put their heads together to try to do something about it.

Let’s try tying people down! Let’s change what the steering wheel is made of! Let’s put an exploding bag of air in the steering wheel! (Imagine how crazy that sounded in a pitch meeting.) These all seem like reasonable ideas (well except that exploding airbag one), so how do we know which one we should do?

The answer is to generate and weigh empirical evidence.

Theory vs. Empirical Evidence

One might have a theory about how something will play out, but what one observes or experiences can be different from what a theory might predict. People want to know the effectiveness of all sorts of things, which means they have to test them.

Social scientists produce empirical evidence in a variety of ways to test theories and measure the ability of A to produce an expected result: B.

Usually, researchers collect data through direct or indirect observation, and they analyze these data to answer empirical questions (questions that can be answered through observation).

Let’s look at our car safety example. Engineers and scientists equipped cars with various safety devices in various configurations, then smashed them into walls, poles and other cars and recorded what happened. Over time, they were able to figure out what types of safety devices worked and which ones didn’t. As it turns out, that whole airbag thing wasn’t so crazy after all.

They didn’t get everything right immediately. For instance, early seatbelts weren’t retractable. Some airbags shot pieces of metal into passengers . But, in fits and in starts, auto safety got better, and even though people are driving more and more miles, fewer and fewer are dying on the road .

How Gathering Empirical Evidence in Social Science is Different

Testing the effects of, say, a public policy on a group of people puts us in the territory of social science.

For instance, education research is not the same as automotive research because children (people) aren’t cars (objects). Education, though, can be made better by attempting new things, gathering data on those efforts, rigorously analyzing that data and then weighing all available empirical evidence to see if those new things accomplish what we hope they do.

Unfortunately, the “rigorously analyzing” bit is often missing from education research. In the labs of automobile engineers, great care is taken to only change one bit of design (a variable) at a time so that each test isolates the individual factor that is making a car more or less safe. OK, for this test, let’s just change the material of the steering wheel and keep everything else the same, so we’ll know if it is the wheel that is hurting people.

Comparing Apples with Apples

In social science and especially in education, trying to isolate variables is challenging, but possible, if researchers can make “apples-to-apples” comparisons.

The best way to get an apples-to-apples comparison is to perform something called a randomized control trial (RCT). You might have heard about these in relation to the testing of medicine. Drug testing uses RCTs all the time.

In an educational RCT, students are divided into two groups by a randomized lottery and half of the students receive whatever the educational “treatment” is (a new reading program, a change in approach to discipline, a school voucher, etc.) while the other does not. Researchers compare the results of those two groups and estimate the “treatment” effect. This approach gives us confidence that the observed effect is caused by the intervention and no other factors.

RCTs are not always possible. Sometimes researchers can get close by using random events that separate kids into two groups, such as school district boundaries that are created by rivers or creeks that split a community more or less by chance or birthday cutoffs for preschool that place a child born on August 31st in one grade but one born September 1st in another even though there is basically no difference between them. Depending on the exact nature of the event, these can be known as “regression discontinuity” or “instrumental variable” analyses, and they can be useful tools to estimate the effects of a program.

Researchers can also follow individual children that receive a treatment if they have data from before and after to see how that child’s educational trajectory changes over time. These are known as “fixed effects” analyses.

All three of these—randomized control trials, regression discontinuity analyses and fixed effects analyses —have their drawbacks.

Very few outside events are truly random. If, as regression discontinuity analysis often does, researchers only look at children just above or just below the cutoff, or, as fixed effects analysis often does, researchers look at only those children who switch from one school to another, those children might not be representative of the population. How would an intervention affect kids who are not close to a cutoff or border? Or kids who do not switch schools?

In the SlideShare below, we present empirical evidence based on rigorous research on private school choice programs as an example of how we, as academics and researchers ourselves, identify and characterize the high-quality empirical evidence in a given area of study.

[Slideshare no longer available]

A Couple Considerations

It’s a lot to wade through, so before you do, we’d like to offer two notes.

First, it is always important to understand the tradeoffs between internal and external validity.

Internal validity refers to how well a study is conducted—it gives us confidence that the effects we observe can be attributed to the intervention or program, not other factors.

For example, when the federal government wanted to know if Washington, D.C.’s school voucher program increased students’ reading and math test scores, researchers took the 2,308 students who applied for the program and randomly assigned 1,387 to get vouchers and 921 not to . They then followed the two groups over time, and when they analyzed the results, they could reasonably conclude that any differences were due to the offer of a voucher, because that is the only thing that was different between the two groups and they were different only because of random chance. This study had high internal validity.

External validity refers to the extent that we can generalize the findings from a study to other settings.

Let’s think about that same study. The D.C. program was unique. The amount of money that students receive, the regulations that participating schools had to agree to, the size of the program, its politically precarious situation and numerous other factors were different in that program than in others, not to mention the fact that Washington, D.C. is not representative of the United States as a whole demographically, politically or in really any way we can possibly imagine. As a result, we have to be cautious when we try to generalize the findings. The study has lower external validity.

To combat issues around lower external validity, researchers can collect and analyze empirical evidence on program design to understand its impact. We can also look at multiple studies to see how similar interventions affect students in different settings.

Second, the respect and use of research does not endorse technocracy. Research and expertise is incredibly useful. When you get on an airplane or head into surgery, you want the person who is doing the work to be an expert. Empirical evidence can help us know more about the world and be better at what we do. But we should also exercise restraint and humility by recognizing the limits of social science.

Public policy involves weighing tradeoffs that social science cannot do for us. Social science can tell us that a program increases reading scores but also increases anxiety and depression in children. Should that program be allowed to continue? Ultimately, that comes down to human judgment and values. That should never be forgotten.

With that, we hope you found this article helpful. Please feel free to reach out with any questions by emailing [email protected] or posting your question in the comments section below.

Michael Q. McShane

Director of national research, edchoice.

Dr. Michael McShane is Director of National Research at EdChoice. He is the author, editor, co-author, or co-editor eleven books on education policy, including his most recent Hybrid Homeschooling: A Guide to the Future of Education (Rowman and Littlefield, 2021) He is currently an opinion contributor to Forbes, and his analyses and commentary have been published widely in the media, including in USA Today, The Washington Post, and the Wall Street Journal. He has also been featured in education-specific outlets such as Teachers College Record, Education Week, Phi Delta Kappan, and Education Next. In addition to authoring numerous white papers, McShane has had academic work published in Education Finance and Policy, The Handbook of Education Politics and Policy, and the Journal of School Choice. A former high school teacher, he earned a Ph.D. in education policy from the University of Arkansas, an M.Ed. from the University of Notre Dame, and a B.A. in English from St. Louis University.

Will Flanders

Education research director, wisconsin institute for law & liberty.

Will Flanders is the education research director at the Wisconsin Institute for Law & Liberty. He holds a Ph.D. in Political Science with a specialization in American Politics and Public Policy, an M.S. in Political Science and an M.S. in Applied American Politics and Policy from Florida State University.

What New Teen Polling Tells Us About the State of K-12 Education in 2024

Colyn Ritter

Updated April 17, 2024

New Polling Shows Parental Support for School Choice Policies Remains Strong in 2024

Updated April 10, 2024

RECEIVE EDUCATIONAL CHOICE UPDATES STRAIGHT TO YOUR INBOX

• First Name *
• Last Name *
• © 2024 EdChoice
• Privacy Policy
• Terms of Use

EDS 624: Contemporary Issues and Research in Special Education in the Classroom: What is Empirical Research?

• Logging in from Home
• Encyclopedia & Special Education EBooks (General)

What is Empirical Research?

• Journal Finder
• Interlibrary Loan Services
• Ask a Librarian
• Troubleshooting Access while Off-Campus This link opens in a new window
• APA Style This link opens in a new window
• Writing Center

What is Peer Review?

Types of Scholarly Articles

Empirical Research

Empirical research articles report on research based on actual observations or experiments. This type of research may make use of either a quantitative or qualitative research design. Quantitative research is research which generates numerical data and looks to create relationships between two or more variables. Qualitative research looks to critically and objectively analyze behaviors, beliefs, feelings, or values. 

Key characteristics of empirical research articles include: specific research questions, definition of a problem, behavior, or phenomenon, a complete description of the process used to study the population or phenomenon, including selection criteria, controls, as well as a description of the instruments used such as interviews, tests or surveys. 

Empirical articles normally include the following elements:

1. An Introduction : The Introduction provides a brief summary of the research.

2. Methodology : The methodology describes how the research was conducted -- including who the participants are, how they were selected, the steps taken to design the study, what the participants did, and the types of measurements used. 

3. Results : The results section describes the outcomes of the measures of the study.

4. Discussion : The discussion section contains interpretation and implications of the study.

5. Conclusion

6. References

Note: Empirical   research articles may combine these elements so that one section of the paper may handle two or three of these areas at once.

Empirical research articles are usually lengthy (>7 pages) and can be difficult to read -- particularly if you are not a practitioner in the field of study in which they are written.  Whenever you are in doubt always speak to your professor or ask a librarian for advice. 

• << Previous: Encyclopedia & Special Education EBooks (General)
• Next: Databases >>
• Last Updated: Feb 11, 2024 6:16 PM
• URL: https://liu.cwp.libguides.com/c.php?g=1124196

Higher Education

• Books, E-Books, and New Titles
• Articles and Databases
• Web Resources
• Open Education Resources (OER) This link opens in a new window
• Citing Sources
• Search Tutorials
• Empirical Studies
• Thesis & Dissertation Submission
• An empirical study is research derived from actual observation or experimentation.
• The written articles resulting from empirical studies undergo a rigorous review by experts in the field of study prior to being published in journals.
• After passing this review the articles are published in a scholarly, peer-reviewed, or academic journal.
• Empirical study articles will generally contain the following features:

               Abstract - This is a summary of the article.                Introduction - This is often identified as the hypothesis of the study and describes the researcher's intent.                           Method - A description of how the research was conducted.                Results - A description of the findings obtained as a result of the research. Most often answers the hypothesis.                Conclusion - A description of how/if the findings were successful and the impact made as a result.                References - A detailed listing of all resources cited in the article that support the written work.              

Keyword Terms

Methodology

Observation

Sample Size

Statistical Analysis

• << Previous: Search Tutorials
• Next: Thesis & Dissertation Submission >>
• Last Updated: Apr 24, 2024 4:53 PM
• URL: https://campbellsville.libguides.com/c.php?g=1384663

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 17 April 2024

How popularising higher education affects economic growth and poverty alleviation: empirical evidence from 38 countries

• Jian Li   ORCID: orcid.org/0000-0002-3228-8163 1   na1 ,
• Eryong Xue   ORCID: orcid.org/0000-0002-7079-5027 2   na1 ,
• Yukai Wei   ORCID: orcid.org/0000-0002-5202-7307 2 &
• Yunshu He   ORCID: orcid.org/0000-0003-4814-9835 2  

Humanities and Social Sciences Communications volume  11 , Article number:  520 ( 2024 ) Cite this article

609 Accesses

3 Altmetric

Metrics details

The popularisation of higher education supports UNESCO’s aim of developing inclusive and equitable quality education to achieve the fourth Sustainable Development Goal. However, the effect of popularising higher education on economic growth and poverty alleviation remains unexplored. Therefore, this study investigated the effects of higher education and adult education within populations (popularisation of higher education) on economic growth (gross domestic product; GDP) and the poverty line using panel data from 38 countries. OLS and quantile regression were performed using data for the period 1995–2021 extracted from the OECD and World Bank databases. The results showed that the population segments with higher education had a significantly positive impact on GDP growth. Moreover, an increased proportion of the population with higher education, of working age, was found to be a contributing factor to GDP growth. Popularising higher education also played a positive role during the initial stage of social and economic development. This study also highlighted that popularising higher education play a key role to influence a country’s educational development and scientific and technological innovation drives the deepening of a country’s economy. It suggested that both national and local governments worldwide should pay much attention to the popularisation degree of higher education to greatly improve the innovative ability of talents and scientific and technological innovation in higher education for both the economic growth and poverty alleviation.

Similar content being viewed by others

The effect of intergenerational mobility on family education investment: evidence from China

Dynamics of returns to vocational education in China: 2010–2017

Measuring and forecasting progress in education: what about early childhood?

Introduction.

The popularisation of higher education critically contributes to UNESCO’s efforts to realise the fourth Sustainable Development Goal of inclusive and equitable quality education (Ainscow, 2020 ; Bamberger and Kim, 2022 ).Popularisation of higher education expands the scale of higher education and its high growth rate introduces considerable challenges to the management structure of higher education, triggering a series of theoretical and practical concerns relating to the nature and function of higher education (Balestra and Ruiz, 2015 ; Brand, 2021 ). Given that education and social and economic development are mutually reinforcing, the expansion of higher education leads to an ascending spiral of development for individuals and/or economies. By contrast, a lack of education or early withdrawal from education leads to a downward spiral for them (Camilla, 2023 ). This relationship between education and development undergirds the model of poverty alleviation based on the return on education (Decancq, 2017 ). The previous studies emphasise the importance of the return on education as a multidimensional anti-poverty mechanism and thus a key factor in poverty alleviation (Fang et al., 2022 ; Chelli et al., 2022 ; Garritzmann, 2016 ). For example, return on education is the key factor enabling a transition from poverty alleviation through education to poverty alleviation through education (Gillian et al., 2021 ; Gong and Hong, 2021 ). Poverty alleviation is realised through an interlinking of these two processes and the promotion of the latter (Granata, 2022 ; Habibi and Zabardast, 2020 ). The educational resources can meet the needs of the poor mainly through the return on education at the levels of survival and life quality. In addition, the previous studies highlighted that, with a continuous expansion in the scale of higher education, its economic effect gradually appears to become marginal (Hoeller et al., 2014 ). The density of colleges and universities worldwide has increased considerably in recent years, but it is still inadequate to meet increasing demands resulting from the ongoing popularisation of higher education (Jericho, 2016 ). The increase in the number of colleges and universities has a positive effect in promoting economic development but with marginal benefits. (Julian, 2018 ).

Through reviewed the current relevant studies, it is found that there have limited studies that have simultaneously explored the effects of popularising higher education on economic growth and poverty alleviation. The previous research revealed that most studies have focused on the relations between popularisation of higher education and economic growth. However, a few empirical investigations have examined the effect of population segments with higher education and adult education (popularisation of higher education) on economic growth (GDP) and poverty reduction. Considering the scope and limitations of previous studies, it aimed to address the above research gap by investigating the effect of a population segment with high levels of higher education and adult education (popularisation of higher education) on economic growth (GDP) and the poverty line at a wide scale using panel data from 38 countries. The main research questions addressed in this study are as follows.

Q1: What is the effect of a population segment with higher education on GDP growth?

Q2: What is the effect of adult education on GDP growth?

Q3: What impact does a population segment with higher education have on reducing the proportion of those experiencing poverty?

Q4: What is the relation between an increased level of adult education and the proportion of the population experiencing poverty?

All these questions are relevant to an exploration of the effect of the population segment with higher education and adult education (popularisation of higher education) on economic growth (GDP) and the poverty line. This study is divided into several sections: the first section concentrates on examining the effect of popularising higher education on economic growth and the poverty line, the relationship between popularisation of higher education and poverty alleviation, and the relationship between popularisation of higher education and poverty alleviation. In the second section of method, to address this research gap, this study performed OLS and quantile regressions using data extracted from the OECD and World Bank databases for the period 1995–2021. An OLS regression model and a panel quantile model were used to analyse the effect of a population segment with higher education and adult education (popularisation of higher education) on economic growth (GDP) and the poverty line within 38 OECD countries. The impact of the proportion of people aged 24–64 years and 25–34 years who had completed higher education in relation to their peers on GDP and the proportion of people living in poverty in 38 OECD countries have been measured and analysed. The results and discussion have been provided at the last.

Literature review

The effect of popularising higher education on economic growth.

The population segment with higher education is regarded as an important contributor to economic growth, generating scientific knowledge and providing labour, which in turn increases human capital and productivity (Jungblut, 2017 ; Kalamova, 2020 ; Liu, 2017 ). As the scale of higher education expands, the emergence of knowledge power as a large-scale global phenomenon reflects the important role of an expanded educated labour force in the advancement of science and technology and the economy. For example, the relationship between higher education and economic development in European Union countries between 1997 and 2016 was analysed. Their findings revealed a statistically significant correlation between expanding higher education and economic growth in the selected countries. The one-way cause-and-effect relationship between education and economic development in these countries suggests that an increase in the proportion of the population enroled in higher education boosts economic performance. In addition, using a survey sample of 35 households, a retrospective study in Brazil, examined the role of educational expansion in reducing income inequality and poverty. Its findings suggest that it would take decades to reduce inequality and poverty in this country and that this outcome could only be achieved through a major expansion of the higher education sector. The growth needed to achieve this outcome would be considerable (Lamichhane et al., 2021 ). This reduction in inequality and poverty could only be achieved if optimistic assumptions about growth, matching job skills and the return on education do not fall short. In brief, education is not a panacea for reducing poverty and inequality. How three major stages of education contributed to the growth in labour productivity in 125 countries during the period 1999–2014 was also explored. They found that human capital is consistent with the educational returns of an average number of years of formal education at the levels of primary, secondary, and higher education. Their analysis showed that higher education had the greatest impact on labour productivity in the economies under study (Ledger et al., 2019 ). In addition, popularising higher education plays an important role in promoting economic growth, as the scale of higher education can guarantee the scale of human resources development by improving the quality of human resources and cultivating and distributing innovative scientific and technological talents. The scale of higher education guarantees the spread of science and technology and the popularisation of scientific and technological achievements (Mathias, 2023 ; Megyesiova and Lieskovska, 2018 ). The expanded scale of higher education worldwide has a spatial spillover effect on economic growth, which is strengthened through international cooperation in the fields of science and technology.

Popularising higher education also plays a direct role in cultivating and transporting scientific and technological talents to promote international scientific and technological cooperation (Mitic, 2018 ; Özdoğan Özbal, 2021 ; OECD, 2022 ; Pinheiro and Pillay, 2016 ). The scale of postgraduate education inhibited the total number of scientific and technological innovation achievements, indicating that there may be a trade-off between ‘quantity expansion’ and ‘quality upgrading’ of scientific and technological innovation achievements. Nevertheless, the positive effect on the number of high-tech innovation outcomes is significant, indicating that the supporting effect of graduate education on scientific and technological innovation is mainly concentrated in the high-tech fields (Pinheiro and Pillay, 2016 ; Rowe, 2019 ; Sahnoun and Abdennadher, 2022 ). The ‘talent increment’ of regional expansion and the ‘resource stock’ of graduate education have a combined promoting effect on high-tech innovation. There are differences in the effect of graduate education supporting high-tech innovation among provinces with different characteristics relating to the development of graduate education. The incremental expansion of high-quality talent is essential for enhancing the efficiency of material capital and stabilising the advantage of resource stocks. Using education statistics from OECD countries, Russia, and several other countries that participate in OECD education research, comparative and correlational analysis methods were applied to analyse how the scale of growth in young people’s participation in higher education is reflected in changes in their employment and economic activity. The results of their analysis showed that the growth in economic activity involving young graduates with a master’s degree exceeded that of college graduates after the 2009 financial crisis, and graduates fared better in the 2020 crisis, which was triggered by the COVID-19 pandemic.

The effect of popularisation of higher education on poverty alleviation

Popularisation of higher education is regarded as an essential factor contributing to poverty alleviation (Samo, 2022 ; Adams, 2013 ; Zapp, 2022 ). The higher education’s role in promoting economic growth can only be fully realised through the cultivation of talents suitable for the actual development situation of the country. Countries with food shortages, for example in Africa, also need to procure and train the right agricultural talent. Key drivers of sustainable agricultural production include access to improved technologies, sustainable growth of human, biological and natural resource capital, improvements in institutional performance and a favourable economic policy environment. Higher education graduates with the necessary ‘soft skills and business skills constitute an important pillar. Chakeredza ( 2008 ), who explored the effect of popularising higher education on poverty alleviation, suggested that the number of hungry people in Africa will continue to increase. Higher education in agriculture must be transformed, and efforts must focus on retaining faculty and on reviewing and redesigning institutional management systems, curriculum content and education delivery.

There are many reasons for poverty, with a lack of education being an important one. Insufficient quality education leads to educational poverty. Using PISA data, Agasisti et al. ( 2021 ) investigated the extent of educational poverty in European countries, considering its incidence, breadth, depth and severity. For this study, they adopted an additive multidimensional poverty measure proposed by Alkirew and Foster. Their findings indicated that between 2006 and 2015, the depth and severity of poverty decreased in most of the countries under study. Moreover, the incidence of educational poverty in many European countries was related mainly to student characteristics and school factors. The expansion of higher education has a positive effect on economic development and poverty reduction by improving work skills within the labour force. Increased enrolment in higher education encourages individuals born in families with low education levels to avail of higher education opportunities. Evidently, the expanded scale of higher education in the process of promoting economic growth has enhanced the equity effect of intergenerational social mobility. The expansion of higher education improves total factor productivity, thus promoting economic transformation and advancement globally (Samo, 2022 ; Adams, 2013 ; Zapp, 2022 ). Furthermore, the previous studies have shown that the structure of higher education talent training has a significant impact on economic development. Therefore, government departments need to make constant efforts to improve relevant systems and promote the optimisation and upgrading of the structure of higher education talent training to meet the needs of future economic development.

Theoretical underpinnings

The relationship between education and economic growth is a classic issue in the study of educational economics. For example, in Solow’s view, the growth of per capita output comes from per capita capital stock and technological progress, but capital investment has the problem of diminishing marginal returns, and the long-term sustainable development of the economy depends on technological progress (Solow, 1957 ). The emphasis on technological progress is a very important point in Solow’s growth theory. It was Schultz who systematically analyzed the contribution of education to economic growth. Influenced by the progress of economic growth theory and national accounting methods, Schulz proposed human capital theory in the process of explaining Solow residuals (Schultz, 1961 ). believes that once human capital is included in economic growth, it will solve the paradoxes and puzzles faced in economic growth research. Starting with the difference in income of different types of workers in the labour market, he found that education and health factors are the main reasons for the income difference, and further clarified that the reason for the income difference is the difference in labor productivity (Schultz, 1961 ). Schultz ( 1961 ) believes that human resources include the quantity and quality of labor, and he mainly focuses on the skills and knowledge of people who can improve labor productivity. As for how to measure human capital investment, Schulz believes that the cost of human capital can be measured in the same way as physical capital. Lucas ( 1988 ) focuses on the mechanism of human capital accumulation and why human capital does not show diminishing marginal returns like physical capital. Lucas divides the effect of human capital into internal effect and external effect. Romer ( 1990 ) internalised technological progress, revealed the relationship between human capital and technological progress, and proposed that the stock of human capital determines the economic growth rate, and it is human capital rather than population that determines economic growth. Romer starts with three hypotheses: first, technological progress is central to long-term economic growth; Second, technological progress is formed by people’s response to market incentives, and market incentives determine technological progress. Third, technology is a special kind of product, and once the cost of the initial input is produced, the technology can be reproduced indefinitely at no cost or very low cost.

In other words, higher education is more about improving students’ ability and productivity, thereby increasing students’ income, and promoting economic growth. Higher education mainly affects economic growth through two aspects: one is the same as Schulz’s improvement of individual ability, and the internal effect of human capital, which directly affects the production process (Schultz, 1961 ). Second, Lucas emphasised the external effect of human capital, and the comprehensive effect of human capital on the whole society, which has the characteristics of increasing marginal benefit (Lucas, 1988 ). It emphasises that the human capital invested in technological innovation and the existing knowledge and technology stock of the whole society jointly determine technological innovation.

Research hypotheses and analytical model

In this study, an OLS regression model and a panel quantile model were used to analyse the effect of a population segment with higher education and adult education (popularisation of higher education) on economic growth (GDP) and the poverty line within 38 OECD countries. The study’s hypotheses were as follows:

Hypothesis 1: The effect of a population segment with higher education has a positive impact on GDP growth.

Hypothesis 2: Some level of adult education has a positive impact on GDP growth.

Hypothesis 3: A population segment with higher education has a positive impact by reducing the proportion of the population experiencing poverty.

Hypothesis 4: An increase in the level of adult education has a positive impact by reducing the proportion of the population experiencing poverty.

The widely used Mankiw-Romer-Weil model was applied in this study. The overall level of development of higher education and the popularisation of higher education were considered core elements that independently promote economic development and alleviate poverty. The following model was constructed by incorporating the variable of quality higher education into the Solow model:

where Y it refers to the output of i country in t year. The independent variables Qit and P it respectively represent the scale of development and the degree of popularisation of higher education in i country in t year. The following specific model was constructed:

The independent variables were the proportion of people aged 25–64 years with higher education (A) and the proportion of people aged 25–34 years with higher education within the same age group (B). The first variable reflects the population segment that has completed higher education and can work in the corresponding age group. The second reflects the degree of popularisation of higher education. The proportion of those who have completed higher education in relation to their peers is in the normal state, which can reflect the enrolment rate for the previous process of higher education, thus indicating the degree of popularisation of higher education.

The dependent variables were GDP and the poverty line (D). GDP is a measure the overall level of a country’s economic and social development. The poverty line refers to the proportion of people living on less than US$1.25 a day as a percentage of the country’s total population or the proportion of people living in poverty. Thus, it reflects the level of equity in social development. The figure of US$2.15 is used in the World Bank’s index and is based on the purchasing power parity in 2017 (see Table 1 ).

Data sources and selection of variables

This study measured the impact of the proportion of people aged 24–64 years and 25–34 years who had completed higher education in relation to their peers on GDP and the proportion of people living in poverty in 28 OECD countries. Specifically, this study assessed the impact of the overall level of development of higher education and the degree of its popularisation (the breadth of development of higher education) on GDP (the height of development of economic and social development) and the poverty line (the breadth of development of economic and social development). Data were sourced from the OECD database and the World Bank website covering the period 1995–2021. This study selected 38 OECD countries for this study: the United States, UK, France, Germany, Italy, Canada, Ireland, the Netherlands, Belgium, Luxembourg, Austria, Switzerland, Norway, Iceland, Denmark, Sweden, Spain, Portugal, Greece, Turkey, Japan, Finland, Australia, New Zealand, Mexico, the Czech Republic, Hungary, Poland, South Korea, Slovakia, Chile, Slovenia, Estonia, Israel, Latvia, Lithuania Colombia and Costa Rica. Figure 1 shows the distribution of the 38 OECD countries. Of these countries, 20 were founding members of the OECD when it was established in 1961, while the remaining 18 subsequently became members. After 1994, OECD membership expanded rapidly. Five new members were added within three years. OECD then entered a period of accelerated development, and its operations and advancement reached an optimal stage. Therefore, this study selected data from the OECD database and the World Bank website covering the period 1995–2021 to explore the relationship between higher education and economic and social development in OECD member countries.

It expresses the geographical relations of the Atlantic region and simplifies the latitude and longitude lines and country symbols, highlighting the geographical distribution by highlighting OECD countries in color and other countries in apricot color.

The impact of the population segment with higher education on GDP growth

This study explored the impact of the population segment with higher education on GDP, taking the proportion of people aged 25–34 years who had completed higher education (B) and the proportion of people aged 25–64 years who had completed higher education (A) as the independent variables for the OLS regression. The square value of model R was 0.097, indicating that the two independent variables could explain 9.73% of the change in GDP. The model passed an F test ( F  = 46.137, p  = 0.000 < 0.05), indicating that at least one of the two independent variables impacted the GDP regression coefficient (C). The following formula was used:

The final analysis revealed that the regression coefficient value of A was 1.553 and the significance level was 0.01 ( t  = 7.141, p  = 0.000 < 0.01). Therefore, A had a significantly positive influence on C. Accordingly, the proportion of the population aged 25–64 years who had completed higher education, that is, the overall level of development of higher education was found to have a positive impact on GDP. The influence coefficient value was 1.533, indicating that an increase in the proportion of the population with completed higher education led to an increase in GDP.

The regression coefficient value of B was −0.813 at a 0.01 level of significance ( t  = −4.300, p  = 0.000 < 0.01), indicating that B had a significantly negative influence on C. The proportion of the population aged 25–34 years who had completed higher education, that is, the degree of popularisation of higher education had a negative effect on GDP, and the influence coefficient value was −0.813.

The negative impact on economic and social development caused by an increase in the popularity of higher education and the proportion of young people’s higher education experience may be attributed to the excess capacity of higher education. The development of higher education should be adapted to the national context. An excess of higher education and a lack of investment lead to a rise in the social cost of education and a decline in social outputs, which hinder social and economic development. At the same time, young people aged between 25 and 34 years occupy the position of’ export’ in the education process. With the increasing popularity of higher education, the supply of talents in the labour market generated through the recruitment of former higher education exceeds the demand for graduates with higher education within recruiting organisations. Consequently, issues such as wasted educational resources and knowledge, unemployment, excessive education, excess talents, an imbalance in the structure of higher education, excessive expansion and decreasing compatibility undermine economic operations and hinder GDP growth.

In this study, the variance decomposition and Pearson coefficient based on covariance calculation were analyzed. The variable of the number of 25–34-year-old who have completed higher education as a percentage of their peers explains 50.74% of the change in GDP. The variable of the proportion of 25–64-year-old who have completed higher education explains 49.26% of the change in GDP. The variable of 25- to 34-year-olds who completed higher education as a percentage of their peers explained 45.88% of the change in poverty line. The variable of the proportion of people aged 25–64 who have completed higher education explains 54.12% of the change in GDP (See Table 2 ).

The proportion of people aged 25–34 who have completed higher education in their peers and the proportion of people aged 25–64 who have completed higher education in their peers, GDP and poverty line showed significant correlation coefficients. The correlation between the proportion of people who have completed higher education at the age of 25–34 and the proportion of people who have completed higher education at the age of 25–64 is 0.931, and shows a significance of 0.01, which indicates that there is a significant positive correlation between the proportion of people who have completed higher education at the age of 25–34 and the proportion of people who have completed higher education at the age of 25–64. The correlation between the proportion of the number of people who have completed higher education at the age of 25–34 and the GDP is 0.209, and the significance is 0.01, which indicates that there is a significant positive correlation between the number of people who have completed higher education at the age of 25–34 and the GDP. The correlation between the number of people who have completed higher education and the poverty line at the age of 25–34 is −0.365, with a significance of 0.01, indicating a significant negative correlation between the number of people who have completed higher education and the poverty line at the age of 25–34 (See Table 2 ).

White test and BP test were used in this study. The test null hypothesis is that the model has no heteroscedasticity. The table above shows that both tests reject the null hypothesis ( p  < 0.05), indicating that the model does have heteroscedasticity. When there is a heteroscedasticity problem, Robust and robust standard false regression is used (See Table 3 ).

The impact of a population segment with higher education on the poverty line

This study also explored the impact of a population segment with higher education on the poverty line. Specifically, this study performed an OLS regression in which the proportion of people aged 25–34 years who had completed higher education (B) and the proportion of those aged 25–64 years who had completed higher education (A) were the independent variables. As Table 2 shows, the R squared value was 0.134. This means that variables A and B could explain 13.37% of the change in the poverty line (D). The model passed the F test ( F  = 48.771, p  = 0.000 < 0.05), which means that at least one variable (A or B) had an impact on the poverty line. The formula for the change in the poverty line was expressed as follows:

The final analysis revealed that the regression coefficient value of the proportion of people aged 25–64 years who had completed higher education (A) was 0.005 but with no significance ( t  = 0.428, p  = 0.669 > 0.05), indicating that the population segment with higher education did not have an impact on the poverty line.

The regression coefficient value of the proportion of people aged 25–34 years who had completed higher education (B) was −0.048 at a significance level of 0.01 ( t  = −4.305, p  = 0.000 < 0.01), which means that in relation to their peers, the proportion of people aged 25–34 years who had completed higher education had a significantly negative impact on the proportion of poor people. A higher proportion of people aged 25–34-years who had completed higher education corresponded to a higher penetration rate of higher education and a lower proportion of those living in poverty. This phenomenon can be attributed to OECD’s support for the development of higher education in various countries. When the development of higher education reaches a certain level, the reduction of the proportion of the population segment experiencing poverty will no longer be affected by a simple expansion of the scale of extended higher education and the superposition of the total number of highly educated human resources. It will be influenced more by the reasonable distribution of educational resources and educational equity within higher education and its popularisation, that is, the increase in the proportion of the school-aged population aged 25–34 years based on the increase of the previous enrolment rate (see Table 4 ).

The effect of adult education on GDP growth

For quantile regression analysis, a total of nine models (with decimal points ranging from 0.10 to 0.90 and at intervals of 0.10) were estimated in this analysis, which aimed to explore the impact of the independent variables A and B on the dependent variable, GDP (C). When the quantile value was between 0.1 and 0.3, the proportion of the population aged 25–64 years who had completed higher education (A) had no significant positive impact on GDP growth, indicating that the development of higher education did not significantly affect economic and social development in poorer OECD countries. When the quantile value was between 0.4 and 0.6, the level of development of higher education had a significantly negative impact on economic and social development. Thus, for a country that had developed over a period, the advancement of higher education required multiple inputs, such as capital, material, and human resources.

During the early stage of the development of higher education, such inputs may, however, have a negative and weakening impact on social and economic development. The added cost of education and the lag between the output of educational achievements and the input of talents puts increased pressure on economic and social development during a certain period. When the quantile value was 0.7 or higher, the improvement of the overall level of higher education had a significantly positive impact on GDP growth, indicating the realisation of the talent training outcomes of higher education. Teaching and research outcomes were thus transformed into socially productive resources and power, with talents with higher education contributing to economic and social development.

When the quantile value was 0.1, the proportion of people aged 25–34 years who had completed higher education in relation to their peers (variable B), indicating the popularisation of higher education, had no significant impact on GDP growth. Thus, in extremely backward countries, the popularisation of higher education had little effect on economic and social development. When the quantile value ranged between 0.2 and 0.6, the popularisation of higher education had a significantly positive effect on GDP growth, indicating its contribution to economic growth.

When the quantile value was 0.7, the influence of variable B on variable C was no longer significant, indicating that social development would soon face the problem of overcapacity in higher education. When it exceeded 0.7, the ratio of eligible people aged 25–34 years who had completed higher education in relation to their peers had a significantly negative impact on GDP growth, revealing that with the development of the economy, society and education, higher education had become overexpanded. Thus, the cost of investing in education exceeded the social benefits, leading to overcapacity whereby the supply of higher education talents exceeded the demand. This situation led to wasted educational resources and excessive competition of talents, hindering economic growth (See Table 5 ).

The increased level of adult education and the proportion of the population experiencing poverty

Using the same model, this study explored the influence of the independent variables, A and B, on the poverty line (dependent variable D). The proportion of the population aged 25–64 years who had completed higher education (independent variable A) had no significant influence on the proportion of the population living in poverty, indicating that popularisation of education and economic and social development have been achieved to a certain extent in OECD countries, and improvements targeting the population experiencing poverty could no longer be achieved simply by increasing the volume and quantity of higher education. When the quantile value was 0.1, the proportion of people aged 25–34 years who had completed higher education in relation to their peers (independent variable B) had no significant effect on the proportion of the population experiencing poverty (dependent variable D). Therefore, the strategy of increasing higher education enrolment and the ratio of the eligible population through the fair allocation of educational resources, and thus the popularisation of education, would not be effective for a small population segment experiencing poverty. In other words, the population segment experiencing poverty in highly developed countries is less receptive to the popularisation of higher education. When the quantile value was 0.2, the independent variable, B, had a significantly positive impact on the dependent variable D, that is, an increase in the popularity of higher education led to an increase in the population segment experiencing poverty. This phenomenon can be interpreted as reflecting the inherent disadvantages of the welfare state in the field of education. A rise in the number of eligible young people aged 25–34 years who have completed higher education reflects the development trend of higher education towards fairness and popularisation following the redistribution of higher education resources.

The fair distribution of higher education resources leads to a lack of competition in the areas of teaching and career development. To a certain extent, reducing students’ willingness and enthusiasm to work may lead to poverty caused by the failure to achieve teaching results. When the quantile value was between 0.3 and 0.4, the independent variable, B, had no significant influence on the dependent variable D. In relatively poor countries, the popularisation of higher education contributes little to reducing the degree of poverty, so it may be necessary to explore ways of alleviating poverty from the perspective of improving the overall level and expanding the scale of basic higher education. When the quantile value was 0.5 or above, the independent variable B had a significantly negative impact on the dependent variable D, indicating that for countries with a relatively high proportion of their population experiencing poverty, the following strategy would be more effective.

Considering the quantile data, this study deemed that the degree of sensitivity of countries at different stages of economic development to the level of development and popularisation of higher education could be more intuitively evaluated using a radar map (see Fig. 2 ). Countries with sub-points 0.1–0.9 were defined along a spectrum as extremely backward, backward, moderately backward, slightly backward, moderate, preliminarily developed, moderately developed, developed, and highly developed. From the perspective of economic development, increasing the proportion of young people who complete higher education and popularising higher education had an obvious positive effect in backward and medium-developed countries, whereas the effect in highly developed countries was not obvious. Overall, the sensitivity of OECD countries to the high level of education penetration was found to be higher than the level of development of higher education. From the perspective of equitable economic development, the overall level of development of higher education had no significant impact on the poverty link in OECD countries, whereas OECD countries with differing economic development backgrounds and at varying stages of development evidenced relatively significant and stable sensitivity to the proportion of young and middle-aged people who completed higher education and the popularisation of higher education.

The dashed line represents the proportion of people aged 25–34 years who have completed higher education. The solid line represents the proportion of people aged 25–64 years who have completed higher education, the impact of the overall level of higher education.

Our findings indicated that population segments with higher education had a significantly positive impact on GDP growth in 38 OECD countries. An increase in the proportion of the population segment of working age who completed higher education was found to contribute to GDP growth. Moreover, an improvement in the popularity of higher education played a positive role during the initial stage of economic and social development.

At the same time, oversupply and overcapacity may result from a continuous improvement of higher education. A very large number of young people who have completed higher education can lead to excessive competition and wasted academic qualifications (Mathias, 2023 ; Megyesiova and Lieskovska, 2018 ). In turn, higher education knowledge unemployment, overinvestment, a structural imbalance, disorderly expansion and wasted resources can occur, which have detrimental impacts on economic operations.

Some studies have shown that strengthening the quality of higher education helps to improve cognitive abilities within the labour force, thereby enhancing the growth of the knowledge economy (Ainscow, 2020 ; Bamberger and Kim, 2022 ). Other studies have reported regional heterogeneity relating to the marginal effect of improving the quality of higher education on economic growth. Some scholars have analysed the influence of the quality of higher education on economic development from the perspective of human capital investment theory. Their findings indicate that the quality of higher education determines the composition and growth trend of social human capital. Because of differences in the degrees of development of different economies, the quality of higher education has a phased influence on economic growth (Balestra and Ruiz, 2015 ; Brand, 2021 ). Case studies of African developing countries by international scholars have revealed that quality factors are key to realising the economic development function of higher education. From the perspectives of both efficient financial investments by states in education poverty alleviation and the effects of economic, time and emotional investments of poor families and individuals in education poverty alleviation, it is necessary to take the return on education into consideration. Moreover, it is important to respond to reasonable concerns regarding the return on education for poor people and to strengthen their cognitive capacities to rationalise as well as their expectations regarding returns on education (Li et al., 2023 ). In this way, the intention to participate and behaviour of anti-poverty education will be generated, and the strategic upgrading of poverty alleviation combined with the promotion of aspirations and cognitive capacities will be emphasised.

Implications

Our use of panel data from 38 countries to deepen understanding of the effect of popularising higher education on economic growth and poverty reduction also has practical implications. The economic, social, and higher education undertakings in OECD countries evidence a certain level of development. The population segment with higher education has no significant impact on reducing the proportion of the population segment experiencing poverty. Simply increasing the proportion of people who complete higher education and expanding the scale of higher education will not effectively reduce poverty (Li and Xue, 2021 ). Providing more educational opportunities to poor people through the slanting of educational resources can help to reduce the proportion of poor people (Ainscow, 2020 ; Bamberger and Kim, 2022 ). For example, popularising higher education plays a key role to influence a country’s development level and scientific and technological innovation drives the deepening of a country’s economy (Bamberger and Kim, 2022 ). Technological progress is the core of economic growth, scientific and technological innovation brings technological change and development in all aspects, human capital promotes economic growth, and higher education trains talents and improves the capital attribute of human (Camilla, 2023 ). For endogenous economic growth theory, the economy does not rely on external forces to achieve sustained growth, and endogenous technological progress determines sustained economic growth. Popularising higher education worldwide brings the accumulation of human capital, improves the quality of workers, and scientific and technological innovation makes technological progress and high-quality economic development, practically. Human capital accumulation is also the process of continuous input of labour force, which covers the accumulation of human capital by labour force factors in formal education, training, and other learning processes. From the perspective of human capital, popularising higher education is the most direct and efficient way to promote the accumulation of human capital and improve the quality of labour force (Balestra and Ruiz, 2015 ; Brand, 2021 ). The popularisation degree of higher education is one of the important indicators to measure the development level of a country’s economic, and it is also the common trend of the development of higher education in all countries after World War II. In this transitional era, how to continue the achievements of higher education in the popular era and solve the existing problems as soon as possible is the heavy responsibility of our times. Therefore, at the initial stage of popularisation of higher education, it is necessary to re-examine the process of higher education popularisation globally and explore the internal logics between the popularisation of higher education and Sustainable Development Goal of inclusive and equitable quality education (Ainscow, 2020 ; Bamberger and Kim, 2022 ).

For policy suggestions, this study suggests that both national and local governments worldwide should pay much attention to the popularisation degree of higher education to greatly improve the innovative ability of talents and scientific and technological innovation in higher education. For example, they could promote scientific and technological innovation in an organised manner to serve national and regional economic and social development. Faced with the current situation in which global higher education has entered a stage of popularisation and new challenges and problems in serving regional economic and social development, national governments should continue to optimise the distribution and structure of higher education resources to support different regions, focusing on the major strategy of enhancing national competitiveness, serving economic and social development, and promoting common prosperity.

Contributions

This study novelty contributes on examining how popularising higher education affects economic growth and poverty alleviation, conceptually, methodologically, and practically. For instance, this study focuses on epitomising the conceptual and analytical model to explore the effects of higher education and adult education within populations (popularisation of higher education) on economic growth (gross domestic product; GDP) and the poverty line. In addition, this study novelty combines both Mankiw-Romer-Weil model Solow model to investigate the effects of higher education and adult education within populations on economic growth and the poverty through OLS regression model and quantile model. For the practical aspect, this study practically uncovers the implicit significance of the popularisation of higher education for advocating UNESCO’s aim of developing inclusive and equitable quality education to achieve the fourth Sustainable Development Goal.

Limitations

This study had some limitations. Data could have been collected from a larger sample of OECD countries to explore the effect of population segments with higher education and adult education (popularisation of higher education) on economic growth (GDP) and the poverty line. In addition, a qualitative component could be included in future studies to uncover the cultural and historical contexts of the effect of popularising higher education on economic growth and poverty reduction at the local level. Future studies should also investigate the causal relationship between the popularisation of higher education and economic growth. Additional empirical data and advanced research methods can be used to enable a shift from correlation to causality.

In conclusion, this study examined the effect of the population segment with higher education and adult education (popularisation of higher education) on economic growth (GDP) and the poverty line using panel data from 38 countries. The population segment with higher education was found to have a significant positive impact on promoting GDP growth. An increase in the proportion of the working-age population segment that had completed higher education was evidently conducive to GDP growth. Popularisation of higher education was also found to play a positive role in the initial stage of economic and social development.

Data availability

The data of OECD country GDP is retrieved from https://data.worldbank.org/indicator/NY.GDP.MKTP.CD?locations=1W , The data of OECD country poverty line is retrieved from https://data.worldbank.org/indicator/SI.POV.DDAY?locations=1W&start=1984&view=chart , The data of OECD country Population with tertiary education 25–34-year-old is retrieved from https://data.oecd.org/eduatt/population-with-tertiary-education.htm#indicator-chart , The data of OECD country Percentage of 25–64-year old’s who have completed higher education (%) is retrieved from https://data.oecd.org/eduatt/adult-education-level.htm#indicator-chart , The datasets generated during and/or analysed during the current study are available in Harvard Dataverse https://doi.org/10.7910/DVN/TP43QS .

Adams, R (2013). Education: poverty and inequality blamed as UK sinks in adult literacy league: OECD survey flags up big problems for England: Post-16 education blamed for failure to address issue. The Guardian . (2013, October 9). London, England

Agasisti T, Longobardi S, Prete V, Russo F (2021) The relevance of educational poverty in europe: determinants and remedies. J Policy Model 43(3):692–709. https://doi.org/10.1016/j.jpolmod.2020.03.015

Article   Google Scholar  

Ainscow M (2020) Promoting inclusion and equity in education: lessons from international experiences. Nord J Stud Educ Policy 6(1):7–16

Balestra C, Ruiz N (2015) Scale-invariant measurement of inequality and welfare in ordinal achievements: an application to subjective well-being and education in OECD Countries. Soc Indic Res 123(2):479–500. https://doi.org/10.1007/s11205-014-0751-2

Bamberger, A, & Kim, MJ (2022) The OECD’s influence on national higher education policies: internationalisation in Israel and South Korea. Comp Educ 18. https://doi.org/10.1080/03050068.2022.2147635

Brand M (2021) The OECD poverty rate: lessons from the russian case. Glob Soc Policy 21(1):144–147

Camilla S. (2023) The design of upper secondary education across OECD countries: Managing choice, coherence and specialisation. OECD Education Working Papers. https://doi.org/10.1787/158101f0-en

Chakeredza S, Temu A, Saka J, Munthali D, Muir-Leresche K, Akinnifesi F, Ajayi O, Sileshi G (2008) Tailoring Tertiary Agricultural Education for Sustainable Development in Sub-Saharan Africa: Opportunities and Challenges. J Sci Res Essay 3(8):326–332

Google Scholar  

Chelli F, Ciommi M, Mariani F, Polinesi G, Recchioni MC, Lamonica GR, Salvati L (2022) A story of strengths and weaknesses in tertiary education: evaluating “Mobility” and “Opportunities” in OECD countries with composite indicators. Sustainability 14(24):16463. https://doi.org/10.3390/su142416463

Decancq, K (2017) Measuring multidimensional inequality in the OECD member countries with a distribution-sensitive better life index. Soc Indic Res 131(3):1057–1086. https://doi.org/10.1007/s11205-016-1289-2

Fang W, Farooq U, Liu Z, Lan J, Iram R (2022) Measuring energy efficiency financing: a way forward for reducing energy poverty through financial inclusion in OECD. Environ Sci Pollut Res 29(47):71923. https://doi.org/10.1007/s11356-022-20139-8

Garritzmann, JL (2016) The political economy of higher education finance: the politics of tuition fees and subsidies in OECD Countries, 1945–2015. Garritzmann. Germany, Berlin: Springer International Publishing

Gillian G, Lisa T, & Thomas W (2021) How are higher education systems in OECD countries resourced?: Evidence from an OECD Policy Survey.OCDE. https://doi.org/10.1787/0ac1fbad-en

Gong HJ, Hong JE (2021) Does postsecondary education attainment matter in community service engagement? Evidence from Across 18 OECD Countries. Educ Sci 11(3):1–17. https://doi.org/10.3390/educsci11030096

Article   MathSciNet   Google Scholar  

Granata, M (2022) The OECD and technical education in post-war Mediterranean Europe. Labor History. https://doi.org/10.1080/0023656X.2022.2057459

Habibi, F & Zabardast, MA (2020) Digitalization, education and economic growth: a comparative analysis of Middle East and OECD countries. Technol Soc 63. https://doi.org/10.1016/j.techsoc.2020.101370

Hoeller P, Joumard I, Koske I (2014) Reducing income inequality while boosting economic growth: can it be done? Evidence from OECD Countries. Singap Econ Rev 59(1):1–22

Jericho, G (2016) Australia didn’t have a ‘great recession’? Tell that to young people; an OECD report has found a rise in youth not in employment, education or training (Neets) since 2008. While historically the rate isn’t high, benefits are so low that youth are now more likely to be living in poverty. Guardian. London, England. https://link.gale.com/apps/doc/A463552813/AONE?u=unimelb&sid=bookmark-AONE&xid=701996a9

Julian LG (2018) The political economy of higher education finance. the politics of tuition fees and subsidies in OECD Countries, 1945–2015. Czech Sociol Rev 54(3):479–482

Jungblut J (2017) The political economy of higher education finance. The politics of tuition fees and subsidies in OECD countries, 1945–2015. Eur J High Educ 7(1):96–99. https://doi.org/10.1080/21568235.2017.1265789

Kalamova, M (2020) Resourcing higher education: challenges, choices and consequences. Higher Education Policy Team. Paris: OECD Publishing

Lamichhane, S, Egilmez, G, Gedik, R, Bhutta, MKS, & Erenay, B (2021) Benchmarking OECD countries’ sustainable development performance: a goal-specific principal component analysis approach. J Clea Prod 287. https://doi.org/10.1016/j.jclepro.2020.125040

Ledger, S, Thier, M, Bailey, L, & Pitts, C (2019) OECD’s approach to measuring global competency: powerful voices shaping education. Teachers College Record, 121(8):1–40

Li, J & Xue. E (2021) Returnee faculty responses to internationalizing “Academic Ecology” for Creating World-class Universities in China’ Elite Universities. Higher Education. 81(5), 1063–1078

Li, J, & Xue. E, Liu, C, Li, X (2023) Integrated macro and micro analyses of student burden reduction policies in China: call for a collaborative “family–school–society” model. Humanities & Social Sciences Communications. 10, 184 (2023). https://doi.org/10.1057/s41599-023-01695-x

Liu, L (2017) Exploring the relationship between education economy and individual civic attitudes: a cross-national analysis in England, Spain, Sweden, Poland and Chinese Taipei

Lucas R (1988) On the mechanics of economic development. J. Monetary Econ 22(1):342

Mathias B (2023). The assessment of students’ creative and critical thinking skills in higher education across OECD countries: a review of policies and related practices. OECD Education Working Papers. https://doi.org/10.1787/35dbd439-en

Megyesiova, S, & Lieskovska, V (2018) Analysis of the sustainable development indicators in the OECD Countries. Sustainability 10(12). https://doi.org/10.3390/su10124554

Mitic RR (2018) The political economy of higher education finance: the politics of tuition fees and subsidies in OECD Countries, 1945-2015. Rev High Educ 41(3):484–487

OECD (2022) Education at a Glance 2022: OECD Indicators, OECD Publishing, Paris, https://doi.org/10.1787/3197152b-en

Özdoğan Özbal E (2021) Dynamic effects of higher education expenditures on human capital and economic growth: an evaluation of OECD countries. Policy Rev High Educ 5(2):174–196. https://doi.org/10.1080/23322969.2021.1893125

Pinheiro R, Pillay P (2016) Higher education and economic development in the OECD: policy lessons for other countries and regions. J High Educ Policy Manag 38(2):150

Romer M (1990) Endogenous technological change. J Political Econ 98(5):S71–S102

Rowe E (2019) Capitalism without capital: the intangible economy of education reform. Discourse 40(2):271

Sahnoun M, Abdennadher C (2022) Returns to investment in education in the OECD Countries: does governance quality matter? J Knowldege Econ 13(3):1819–1842. https://doi.org/10.1007/s13132-021-00783-0

Samo V. (2022). A snapshot of equity and inclusion in OECD education systems: findings from the Strength through Diversity Policy Survey. OECD Eduction Working Papers. https://doi.org/10.1787/801dd29b-en

Schultz (1961) Investment in human capital. Am Econ Rev 51(1):1–17

Solow RM (1957) Technical change and the aggregate production function. Rev Econ Stat 39(3):312–320

Zapp M (2022) Revisiting the global knowledge economy: the worldwide expansion of research and development personnel, 1980–2015. Minerva 60(2):181–208. https://doi.org/10.1007/s11024-021-09455-4

Article   PubMed   PubMed Central   Google Scholar  

Download references

Acknowledgements

This study is funded by 2021 National Social Science Foundation of Higher Education Ideological and Political Course research (Key project) Ideological and Political Education System Construction System Mechanism Research in New Era (No.: 21VSZ004).

Author information

These authors contributed equally: Jian Li, Eryong Xue.

Authors and Affiliations

Institute of International and Comparative Education, Beijing Normal University, Beijing, 100875, China

China Institute of Education Policy, Faculty of Education, Beijing Normal University, Beijing, 100875, China

Eryong Xue, Yukai Wei & Yunshu He

You can also search for this author in PubMed   Google Scholar

Substantial contributions to the conception or design of the work: Jian Li and Eryong Xue; the acquisition, analysis, or interpretation of data for the work: Jian Li and Eryong Xue; drafting the work or revising it critically for important intellectual content: Eryong Xue, Jian Li, Yukai Wei, and Yunshu He; final approval of the version to be published: Eryong Xue and Jian Li.

Corresponding authors

Correspondence to Jian Li or Eryong Xue .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants. It was not required as this article used published data.

Informed consent

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, J., Xue, E., Wei, Y. et al. How popularising higher education affects economic growth and poverty alleviation: empirical evidence from 38 countries. Humanit Soc Sci Commun 11 , 520 (2024). https://doi.org/10.1057/s41599-024-03013-5

Download citation

Received : 29 June 2023

Accepted : 02 April 2024

Published : 17 April 2024

DOI : https://doi.org/10.1057/s41599-024-03013-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

How the American middle class has changed in the past five decades

The middle class, once the economic stratum of a clear majority of American adults, has steadily contracted in the past five decades. The share of adults who live in middle-class households fell from 61% in 1971 to 50% in 2021, according to a new Pew Research Center analysis of government data.

From 2020: Are you in the American middle class? Find out with our income calculator

The shrinking of the middle class has been accompanied by an increase in the share of adults in the upper-income tier – from 14% in 1971 to 21% in 2021 – as well as an increase in the share who are in the lower-income tier, from 25% to 29%. These changes have occurred gradually, as the share of adults in the middle class decreased in each decade from 1971 to 2011, but then held steady through 2021.

The analysis below presents seven facts about how the economic status of the U.S. middle class and that of America’s major demographic groups have changed since 1971. A related analysis examines the impact of the coronavirus pandemic on the financial well-being of households in the lower-, middle- and upper-income tiers, with comparisons to the Great Recession era. (In the source data for both analyses, demographic figures refer to the 1971-2021 period, while income figures refer to the 1970-2020 period. Thus, the shares of adults in an income tier are based on their household incomes in the previous year.)

This report analyzes data from the Annual Social and Economic Supplements (ASEC) of the Current Population Survey (CPS) to study how the economic status of the American middle class has changed since 1971. It also examines the movement of demographic groups in and out of the American middle class and across lower- and upper-income tiers from 1971 to 2021.

The CPS is the U.S. government’s official source for monthly estimates of unemployment ; the ASEC, conducted in March each year, is the official source for its estimates of income and poverty . The COVID-19 outbreak has affected data collection efforts by the U.S. government in its surveys, limiting in-person data collection and affecting the response rate. It is possible that some measures of economic outcomes and how they vary across demographic groups are affected by these changes in data collection. This report makes use of updated weights released by the Census Bureau to correct for nonresponse in 2019, 2020 and 2021.

In this analysis, “middle-income” adults in 2021 are those with an annual household income that was two-thirds to double the national median income in 2020, after incomes have been adjusted for household size, or about $52,000 to $156,000 annually in 2020 dollars for a household of three. “Lower-income” adults have household incomes less than $52,000 and “upper-income” adults have household incomes greater than $156,000.

The income it takes to be middle income varies by household size, with smaller households requiring less to support the same lifestyle as larger households. The boundaries of the income tiers also vary across years with changes in the national median income. Read the methodology for more details.

The terms “middle income” and “middle class” are used interchangeably in this analysis for the sake of exposition. But being middle class can refer to more than just income, be it the level of education, the type of profession, economic security, home ownership, or one’s social and political values. Class also could simply be a matter of self-identification.

Household incomes have risen considerably since 1970, but those of middle-class households have not climbed nearly as much as those of upper-income households. The median income of middle-class households in 2020 was 50% greater than in 1970 ($90,131 vs. $59,934), as measured in 2020 dollars. These gains were realized slowly, but for the most part steadily, with the exception of the period from 2000 to 2010, the so-called “ lost decade ,” when incomes fell across the board.

The median income for lower-income households grew more slowly than that of middle-class households, increasing from $20,604 in 1970 to $29,963 in 2020, or 45%.

The rise in income from 1970 to 2020 was steepest for upper-income households. Their median income increased 69% during that timespan, from $130,008 to $219,572.

As a result of these changes, the gap in the incomes of upper-income and other households also increased. In 2020, the median income of upper-income households was 7.3 times that of lower-income households, up from 6.3 in 1970. The median income of upper-income households was 2.4 times that of middle-income households in 2020, up from 2.2 in 1970.

The share of aggregate U.S. household income held by the middle class has fallen steadily since 1970. The widening of the income gap and the shrinking of the middle class has led to a steady decrease in the share of U.S. aggregate income held by middle-class households. In 1970, adults in middle-income households accounted for 62% of aggregate income, a share that fell to 42% in 2020.

Meanwhile, the share of aggregate income accounted for by upper-income households has increased steadily, from 29% in 1970 to 50% in 2020. Part of this increase reflects the rising share of adults who are in the upper-income tier.

The share of U.S. aggregate income held by lower-income households edged down from 10% to 8% over these five decades, even though the proportion of adults living in lower-income households increased over this period.

Older Americans and Black adults made the greatest progress up the income ladder from 1971 to 2021. Among adults overall, the share who were in the upper-income tier increased from 14% in 1971 to 21% in 2021, or by 7 percentage points. Meanwhile, the share in the lower-income tier increased from 25% to 29%, or by 4 points. On balance, this represented a net gain of 3 percentage points in income status for all adults.

Those ages 65 and older made the most notable progress up the income ladder from 1971 to 2021. They increased their share in the upper-income tier while reducing their share in the lower-income tier, resulting in a net gain of 25 points. Progress among adults 65 and older was likely driven by an increase in labor force participation , rising educational levels and by the role of Social Security payments in reducing poverty.

Black adults, as well as married men and women, were also among the biggest gainers from 1971 to 2021, with net increases ranging from 12 to 14 percentage points.

On the other hand, not having at least a bachelor’s degree resulted in a notable degree of economic regression over this period. Adults with a high school diploma or less education, as well as those with some college experience but no degree, saw sizable increases in their shares in the lower-income tier in the past five decades. Although no single group of adults by education category moved up the income ladder from 1971 to 2021, adults overall realized gains by boosting their education levels . The share of adults 25 and older who had completed at least four years of college stood at 38% in 2021, compared with only 11% in 1971.

Progress up the income ladder for a demographic group does not necessarily signal its economic status in comparison with other groups at a given point in time. For example, in 2021, adults ages 65 and older and Black adults were still more likely than many other groups to be lower income, and less likely to be middle or upper income.

Married adults and those in multi-earner households made more progress up the income ladder from 1971 to 2021 than their immediate counterparts. Generally, partnered adults have better outcomes on a range of economic outcomes than the unpartnered. One reason is that marriage is increasingly linked to educational attainment , which bears fruit in terms of higher incomes.

Married men and women were distributed across the income tiers identically to each other in both 1971 and 2021. Both groups nearly doubled their shares in the upper-income tier in the past five decades, from 14% in 1971 to 27% in 2021. And neither group experienced an increase in the share in the lower-income tier.

Unmarried men and women were much more likely than their married counterparts to be in the lower-income tier in 2021. And unmarried men, in particular, experienced a sizable increase in their share in the lower-income tier from 1971 t0 2021 and a similarly large decrease in their share in the middle-income tier. Nonetheless, unmarried men are less likely than unmarried women to be lower income and more likely to be middle income.

Adults in households with more than one earner fare much better economically than adults in households with only one earner. In 2021, some 20% of adults in multi-earner households were in the lower-income tier, compared with 53% of adults in single-earner households. Also, adults in multi-earner households were more than twice as likely as adults in single-earner households to be in the upper-income tier in 2021. In the long haul, adults in single-earner households are among the groups who slid down the income ladder the most from 1971 to 2021.

Despite progress, Black and Hispanic adults trail behind other groups in their economic status. Although Black adults made some of the biggest strides up the income tiers from 1971 to 2021, they, along with Hispanic adults, are more likely to be in the lower-income tier than are White or Asian adults. About 40% of both Black and Hispanic adults were lower income in 2021, compared with 24% of White adults and 22% of Asian adults.

Black adults are the only major racial and ethnic group that did not experience a decrease in its middle-class share, which stood at 47% in 2021, about the same as in 1971. White adults are the only group in which more than half (52%) lived in middle-class households in 2021, albeit after declining from 63% in 1971. At the top end, only about one-in-ten Black and Hispanic adults were upper income in 2021, compared with one-in-four or more White and Asian adults.

The relative economic status of men and women has changed little from 1971 to 2021. Both experienced similar percentage point increases in the shares in the lower- and upper-income tiers, and both saw double-digit decreases in the shares who are middle class. Women remained more likely than men to live in lower-income households in 2021 (31% vs. 26%).

Adults 65 and older continue to lag economically, despite decades of progress. The share of adults ages 65 and older in the lower-income tier fell from 54% in 1971 to 37% in 2021. Their share in the middle class rose from 39% to 47% and their share in the upper-income tier increased from 7% to 16%. However, adults 65 and older are the only age group in which more than one-in-three adults are in lower-income households, and they are much less likely than adults ages 30 to 44 – as well as those ages 45 to 64 – to be in the upper-income tier.

All other age groups experienced an increase in the shares who are lower income from 1971 to 2021, as well as a decrease in the shares who are middle income. But they also saw increases in the shares who are upper income. Among adults ages 30 to 44, for instance, the share in upper-income households almost doubled, from 12% in 1971 to 21% in 2021.

There is a sizable and growing income gap between adults with a bachelor’s degree and those with lower levels of education. In 2021, about four-in-ten adults with at least a bachelor’s degree (39%) were in the upper-income tier, compared with 16% or less among those without a bachelor’s degree. The share of adults in the upper-income tier with at least a bachelor’s degree edged up from 1971 to 2021, while the share without a bachelor’s degree either edged down or held constant.

About half or a little more of adults with either some college education or a high school diploma only were in the middle class in 2021. But these two groups, along with those with less than a high school education, experienced notable drops in their middle class shares from 1971 to 2021 – and notable increases in the shares in the lower-income tier. In 2021, about four-in-ten adults with only a high school diploma or its equivalent (39%) were in the lower-income tier, about double the share in 1971.

Note: Here is the methodology for this analysis.

• Economic Inequality
• Income & Wages
• Middle Class

Rakesh Kochhar is a senior researcher at Pew Research Center

Stella Sechopoulos is a former research assistant focusing on social and demographic trends research at Pew Research Center

1 in 10: Redefining the Asian American Dream (Short Film)

The hardships and dreams of asian americans living in poverty, a booming u.s. stock market doesn’t benefit all racial and ethnic groups equally, black americans’ views on success in the u.s., wealth surged in the pandemic, but debt endures for poorer black and hispanic families, most popular.

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Age & Generations
• Coronavirus (COVID-19)
• Economy & Work
• Family & Relationships
• Gender & LGBTQ
• Immigration & Migration
• International Affairs
• Internet & Technology
• Methodological Research
• News Habits & Media
• Non-U.S. Governments
• Other Topics
• Politics & Policy
• Race & Ethnicity
• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

Copyright 2024 Pew Research Center

Terms & Conditions

Privacy Policy

Cookie Settings

Reprints, Permissions & Use Policy