research study on instruction

SYSTEMATIC REVIEW article

Differentiated instruction in secondary education: a systematic review of research evidence.

• Department of Teacher Education, University of Groningen, Groningen, Netherlands

Differentiated instruction is a pedagogical-didactical approach that provides teachers with a starting point for meeting students' diverse learning needs. Although differentiated instruction has gained a lot of attention in practice and research, not much is known about the status of the empirical evidence and its benefits for enhancing student achievement in secondary education. The current review sets out to provide an overview of the theoretical conceptualizations of differentiated instruction as well as prior findings on its effectiveness. Then, by means of a systematic review of the literature from 2006 to 2016, empirical evidence on the effects of within-class differentiated instruction for secondary school students' academic achievement is evaluated and summarized. After a rigorous search and selection process, only 14 papers about 12 unique empirical studies on the topic were selected for review. A narrative description of the selected papers shows that differentiated instruction has been operationalized in many different ways. The selection includes studies on generic teacher trainings for differentiated instruction, ability grouping and tiering, individualization, mastery learning, heterogeneous grouping, and remediation in flipped classroom lessons. The majority of the studies show small to moderate positive effects of differentiated instruction on student achievement. Summarized effect sizes across studies range from d = +0.741 to +0.509 (omitting an outlier). These empirical findings give some indication of the possible benefits of differentiated instruction. However, they also point out that there are still severe knowledge gaps. More research is needed before drawing convincing conclusions regarding the effectiveness and value of different approaches to differentiated instruction for secondary school classes.

Introduction

Differentiation is a hot-topic in education nowadays. Policy-makers and researchers urge teachers to embrace diversity and to adapt their instruction to the diverse learning needs of students in their classrooms ( Schleicher, 2016 ; Unesco, 2017 ). Differentiation is a philosophy of teaching rooted in deep respect for students, acknowledgment of their differences, and the drive to help all students thrive. Such ideas imply that teachers proactively modify curricula, teaching methods, resources, learning activities, or requirements for student products to better meet students' learning needs ( Tomlinson et al., 2003 ). When teachers deliberately plan such adaptations to facilitate students' learning and execute these adaptations during their lessons we call it differentiated instruction. A number of developments in education have boosted the need for differentiated instruction. First, contemporary classes are becoming relatively heterogeneous because of policies focused on detracking, the inclusion of students from culturally and linguistically diverse backgrounds, and inclusive education in which special education students (SEN) attend classes along with non-SEN students ( Rock et al., 2008 ; Tomlinson, 2015 ). Since early stratification of students may have unintended effects on the educational opportunities of students with varying background characteristics, addressing students' learning needs by teaching adaptively within heterogeneous classrooms has been proposed as the best choice for a fair educational system ( Oakes, 2008 ; Schütz et al., 2008 ; Schofield, 2010 ; OECD, 2012 , 2018 ). In addition, even within relatively homogeneous classrooms, there are considerable differences between students that need attention ( Wilkinson and Penney, 2014 ). Second, the idea that learners have different learning needs and that a one-size-fits-all approach does not suffice, is gaining momentum ( Subban, 2006 ). Policy makers stress that all students should be supported to develop their knowledge and skills at their own level ( Rock et al., 2008 ; Schleicher, 2016 ) and there is the wish to improve equity or equality among students ( Unesco, 2017 ; Kyriakides et al., 2018 ). When the aim is to decrease the gap between low and high achieving students, teachers could invest most in supporting low achieving students. This is called convergent differentiation ( Bosker, 2005 ). Alternatively, teachers may apply divergent differentiation in which they strive for equality by dividing their efforts equally across all students, allowing for variation between students in the learning goals they reach, time they use, and outcomes they produce ( Bosker, 2005 ).

Although the concept of differentiated instruction is quite well-known, teachers find it difficult to grasp how differentiated instruction should be implemented in their classrooms ( Van Casteren et al., 2017 ). A recent study found that teachers across different countries infrequently adapt their instruction to student characteristics ( Schleicher, 2016 ). Struggling students may work on too difficult tasks or, conversely, high ability students may practice skills they have already mastered ( Tomlinson et al., 2003 ). Clearly, more information about effective practices is needed. A recent review and meta-analysis of differentiated instruction practices in primary education shows that differentiated instruction has some potential for improving student outcomes, when implemented well ( Deunk et al., 2018 ). However, these results may not generalize directly to secondary education, since the situation in which teachers teach multiple classes in secondary education is rather different in nature compared to primary education ( Van Casteren et al., 2017 ). For secondary education, evidence for the benefits of differentiated instruction is scarce ( Coubergs et al., 2013 ). The bulk of studies in secondary education focus on differentiation of students between classes by means of streaming or tracking ( Slavin, 1990a ; Schofield, 2010 ). Alternatively, the current study seeks to scrutinize which empirical evidence there is on the effectiveness of within-class differentiated instruction in secondary education, how studies operationalize the approach, and in which contexts the studies were performed.

Theory and Operationalizations

Operationalizing differentiated instruction in the classroom.

Theories of differentiation are bound by several guiding principles. They include a focus on essential ideas and skills in each content area, responsiveness to individual differences, integration of assessment and instruction, and ongoing adjustment of content, process, and products to meet students' learning needs ( Rock et al., 2008 ). Differentiation typically includes pro-active and deliberate adaptations of the content, process, product, learning environment or learning time, based on the assessment of students' readiness or another relevant student characteristic such as learning preference or interest ( Roy et al., 2013 ; Tomlinson, 2014 ). In Table 1 , we have schematized the theoretical construct of differentiated instruction in the lesson within the broader definition of within-class differentiation.

Table 1 . Theoretical model of within-class differentiation.

Differentiated instruction in the classroom entails two aspects. First is the pedagogy and didactics of differentiated instruction : which teaching practices and techniques do teachers use and what do they differentiate ( McQuarrie et al., 2008 ; Valiande and Koutselini, 2009 )? Teachers may offer students' adapted content , offer various options in the learning process , use different assessment products , or adapt the learning environment to students' learning needs ( Tomlinson, 2014 ). Teachers may also offer certain students more learning time or conversely, encourage high achievers to speed up their learning process ( Coubergs et al., 2013 ). Regarding the process , they may use pre-teaching or extended instruction to cater to the needs of students ( Smets and Struyven, 2018 ), or they could adapt instructions throughout the lesson. Second, the organizational aspect of differentiated instruction entails the structure in which it is embedded. There are different approaches a teacher may choose (see Table 1 ). In macro-adaptive approaches, teachers use some form of homogeneous clustering to organize their differentiated instruction ( Corno, 2008 ), including fixed or flexible grouping of students based on a common characteristic such as readiness or interest. Alternatively, teachers could use heterogeneous grouping to organize their differentiated instruction. Differentiation of the learning process may occur because students divide tasks within the group based on their learning preferences or abilities. Alternatively, a teacher may suggest a division of tasks or support based on assessment of learning needs ( Coubergs et al., 2013 ). When adaptations are taken to the level at which individual students work at their own rate on their level, this is called individualization ( Education Endowment Foundation, n.d. ). The learning goals are the same, but learning trajectories are tailored to individuals' needs. Some authors include individualized approaches into the theoretical construct of differentiated instruction ( Smit et al., 2011 ; Coubergs et al., 2013 ; Tomlinson, 2014 ), whereas others separate it from differentiated instruction ( Bray and McClaskey, 2013 ; Roy et al., 2013 ).

Lastly, there are teaching models or strategies in which differentiated instruction has a central place. One well-known example is group-based mastery learning . In this approach, subject matter is divided into small blocks or units. For each unit, the teacher gives uniform instructions to the whole group of students. Then, a formative assessment informs the teacher which students reach the desired level of mastery of the unit (usually set at 80–90% correct). Students below this criterion receive corrective instruction in small groups, or alternatively, forms of tutoring, peer tutoring or independent practice are also possible to differentiate the learning process ( Slavin, 1987 ). Differentiated instruction may also be embedded in other instructional approaches like peer tutoring, problem-based learning, flipped classroom models etc. ( Mastropieri et al., 2006 ; Coubergs et al., 2013 ; Altemueller and Lindquist, 2017 ).

Immediate, unplanned adaptations to student needs, so-called “micro-adaptations” ( Corno, 2008 ), are not included in the theoretical model in Table 1 , since differentiated instruction is—by nature—planned and deliberate ( Coubergs et al., 2013 ; Tomlinson, 2014 ; Keuning et al., 2017 ). Furthermore, we did not include the concept of “personalization” in our model since in personalized approaches students follow their own learning trajectories, pursue their own learning goals, and co-construct the learning trajectory, which makes it notably different from typical operationalizations of differentiated instruction ( Bray and McClaskey, 2013 ; Cavanagh, 2014 ).

Differentiation as a Sum of Its Parts

As noted above, differentiated instruction during the lesson is in fact only one piece of the mosaic ( Tomlinson, 1999 ). There are a lot of other steps that are crucial for successful implementation of differentiated instruction ( Keuning et al., 2017 ; Van Geel et al., 2019 ). Table 1 shows other behaviors that are related to what teachers do in the classroom. First, continuous monitoring and (formative) assessment and differentiated instruction are inseparable ( Hall, 1992 ; Valiande and Koutselini, 2009 ; Roy et al., 2013 ; Tomlinson, 2014 ; Denessen and Douglas, 2015 ; Prast et al., 2015 ). Some teachers may be inclined to use rather one-dimensional, fixed categorizations of students based on their learning needs at some point in time ( Smets and Struyven, 2018 ). Nevertheless, high quality differentiated instruction is based on the frequent assessment of learning needs and flexible adaptations to meet those needs. Prior to the lesson including differentiated instruction, teachers should have clear goals for their students, use some form of pre-assessment , and plan their adaptive instruction ( Prast et al., 2015 ; Keuning et al., 2017 ; Van Geel et al., 2019 ). Then, teachers proceed to the actual differentiated instruction during the lesson . After the lesson, teachers should evaluate students' progress toward their goals.

Besides these steps, more general high-quality teaching behaviors are preconditions to create a good context for differentiated instruction ( Wang et al., 1990 ; Tomlinson, 2014 ). For instance, creating a safe and stimulating learning environment in which students feel welcomed and respected is essential ( Tomlinson, 2014 ). In addition, good classroom management may help teachers to implement differentiated instruction in an orderly manner ( Maulana et al., 2015 ; Prast et al., 2015 ). In empirical studies, differentiated instruction has been found to be a separate domain of teaching, while at the same time being strongly interrelated with other high quality teaching behaviors ( Van de Grift et al., 2014 ; Maulana et al., 2015 ; Van der Lans et al., 2017 , 2018 ). In turn, high quality teaching behaviors like questioning, explaining the lesson content, or giving examples can be applied in a differentiated way, stressing that high quality teaching is both a contextual factor as a direct source of input for teachers' differentiated instruction.

Prior Review Studies on Differentiated Instruction

Although studies on within-class differentiated instruction in secondary education are scarce, a number of reviews and meta-analyses have shed some light on the effects on student achievement. Subban (2006) discusses a number of studies showing that adapting content or processes can make learning more engaging for students than one-size-fits-all teaching, and some studies showed positive effects of differentiated instruction on student achievement. The narrative review by Tomlinson et al. (2003) revealed studies showing that students achieve better results in mixed-ability classrooms in which the teacher differentiates instruction than in homogeneous classes were a more single-size approach is used. In a recent narrative research synthesis on adaptive teaching, one study on differentiated instruction was included. The authors found positive results of different types of adaptive teaching on students' academic and non-academic outcomes in primary education ( Parsons et al., 2018 ). In a large-scale meta-analysis by Scheerens (2016) , adaptive teaching was operationalized with some relevant indicators such as using variable teaching methods, orientation toward individual learning processes, and considering students' prerequisites. In this meta-analysis, a very small effect of adaptive teaching on student achievement was found.

A number of reviews report on specific operationalizations of within-class differentiated instruction. One of the most frequently reviewed forms is ability grouping . In within-class ability grouping, teachers cluster students into different homogeneous groups based on their abilities or readiness. In her narrative review, Tieso (2003) summarizes that ability grouping has a potential influence on student achievement when grouping is flexible, and teachers adapt their instruction to the needs of different groups. Steenbergen-Hu et al. (2016) performed a meta-synthesis including five other meta-analyses of the effects of ability grouping in K-12 education. In their study, within-class grouping was found to have at least a small positive impact on students' academic achievement (Hedges g = + 0.25). In the study of Kulik (1992) , who also combined results from different meta-analyses, a comparable effect size of Glass's Δ = + 0.25 in favor of within-class ability grouping was found. In the meta-analysis of Lou et al. (1996) on grouping in secondary education, within-class grouping was found to have a small positive effect (Cohen's d = + 0.12) on student outcomes. Substantive achievement gains were found in studies in which teachers adapted their teaching to needs of the different ability groups (Cohen's d = + 0.25), but not in studies in which teachers provided the same instruction for the different groups (Cohen's d = + 0.02). In his large meta-analysis of effects of instructional approaches on student outcomes, Hattie (2009) reported a small positive effect of within-class ability grouping on students' academic achievement (Cohen's d = +0.16). Conversely, Slavin (1990a) did not find significant effects of (between and within-class) ability grouping on achievement in secondary education. In a meta-synthesis of multiple meta-analyses on ability grouping—including between-class ability grouping—no overall positive effects of the approach were found ( Sipe and Curlette, 1996 ). Some studies have found that ability grouping effects may differ for subgroups of students. For instance, Lou et al. (1996) found that low-ability students learned significantly more in heterogeneous (mixed-ability) groups, average-ability students benefitted most in homogeneous ability groups, and for high-ability students group composition made no significant difference. In primary education, Deunk et al. (2018) found a negative effect of within-class homogeneous grouping for low achieving pupils. Conversely, Steenbergen-Hu et al. (2016) concluded that high-, average-, and low-ability students all benefited equally from ability grouping. Thus, the findings on differential effects of ability grouping remain inconclusive.

Another possible approach to differentiated instruction is tiering. Tiering refers to using the same curriculum material for all learners, but adjusting the depth of content, the learning activity process, and/or the type of product developed by the student to students' readiness, interest or learning style ( Pierce and Adams, 2005 ; Richards and Omdal, 2007 ). Teachers design a number of variations or tiers to a learning task, process or product, to which students are assigned based on assessed abilities. To our knowledge, there are no specific reviews of the literature or meta-analyses summarizing the effects of tiering on student achievement, but the approach is often combined with homogeneous (ability) grouping.

Alternatively, turning to heterogeneous grouping as an organizational structure for differentiated instruction, there is evidence that students of varying backgrounds working together may learn from each other's knowledge, from observing each other, and from commenting on each other's errors ( Nokes-Malach et al., 2015 ). However, based on their narrative review about differentiated instruction in secondary schools, Coubergs et al. (2013) concluded that there is little known about the effectiveness of differentiated instruction in heterogeneous settings They found that guiding heterogeneous groups is challenging for teachers, and that it is difficult to address the learning needs of all students in these mixed groups.

Reviews of effectiveness of individualized instruction indicate small effects on student outcomes. Hattie (2009) reports a small effect of individualization on student achievement (Cohen's d = +0.23). In addition, in another review a wide range of effects across meta-analyses was found of individualization on academic achievement of students (from −0.07 to +0.40; Education Endowment Foundation, n.d. ). Currently, mostly ICT-applications are used to individualize instruction. Review studies show that such adaptive ICT applications may considerably improve student achievement ( Ma et al., 2014 ; Van der Kleij et al., 2015 ; Kulik and Fletcher, 2016 ; Shute and Rahimi, 2017 ).

Guskey and Pigott (1988) performed a meta-analysis on the effects of group-based mastery learning on students' academic outcomes from grade one up to college. They reported positive effects on students' academic achievement as a result of the application of group-based mastery learning for, among others, high school students (Hedges g = +0.48). Later on, Kulik et al. (1990) and Hattie (2009) also reported relatively large positive effects of group-based mastery learning on student achievement (ES = +0.59 and Cohen's d = +0.58, respectively). Low ability students were generally found to profit most from the convergent approach ( Guskey and Pigott, 1988 ; Kulik et al., 1990 ). Mastery learning was among the most effective educational approaches in a meta-synthesis of multiple meta-analyses ( Sipe and Curlette, 1996 ). However, mastery learning may be particularly valuable to train specific skills but may yield fewer positive results for more general skills as measured by standardized tests ( Slavin, 1987 , 1990b ). Mastery learning has also been incorporated into broader interventions in secondary education such as the IMPROVE method ( Mevarech and Kramarski, 1997 ).

Overall, from previous review studies we can draw the conclusion that there is some evidence that differentiated instruction has potential power to affect students' academic achievement positively with small to medium effects. However, the evidence is limited and heterogeneous in nature. The effectiveness of some approaches to differentiated instruction, such as ability grouping, has been reviewed extensively, while other approaches have received less attention. Furthermore, most studies were executed some time ago and were executed in the context of primary education, while only few studies focus specifically on secondary education.

Contextual and Personal Factors Influencing Differentiated Instruction

When analyzing the effectiveness of differentiated instruction, it is important to acknowledge that classroom processes do not occur in a vacuum. Both internal and external sources determine whether teachers will succeed in developing complex teaching skills ( Clarke and Hollingsworth, 2002 ). In the case of differentiated instruction, teacher-level variables like education, professional development and personal characteristics like knowledge, attitudes, beliefs, values and self-efficacy may influence their behavior ( Tomlinson, 1995 ; Tomlinson et al., 2003 ; Kiley, 2011 ; De Jager, 2013 ; Parsons et al., 2013 ; Dixon et al., 2014 ; De Neve and Devos, 2016 ; Suprayogi et al., 2017 ; Stollman, 2018 ). Teachers need thorough content knowledge and a broad range of pedagogical and didactic skills to plan and execute differentiated instruction ( Van Casteren et al., 2017 ). At the classroom level, diversity of the student population ( De Neve and Devos, 2016 ) and class-size ( Blatchford et al., 2011 ; Suprayogi et al., 2017 ; Stollman, 2018 ) influence interactions between teachers and their students. Moreover, school characteristics matter. For instance, a school principal's support can influence implementation of differentiated instruction ( Hertberg-Davis and Brighton, 2006 ). Additionally, structural organizational conditions, such as time and resources available for professional development, and cultural organizational conditions such as the learning environment, support from the school board, and a professional culture of collaboration may influence teaching ( Imants and Van Veen, 2010 ; Stollman, 2018 ). Teachers have reported that preparation time is a crucial factor determining the implementation of differentiated instruction ( De Jager, 2013 ; Van Casteren et al., 2017 ). Moreover, collaboration is key; a high pedagogical team culture influences both the learning climate and the implementation of differentiated instruction ( Smit and Humpert, 2012 ; Stollman, 2018 ). Lastly, country level requirements and (assessment) policies that stress differentiated instruction may influence implementation ( Mills et al., 2014 ).

Research Questions

Researchers and teachers lack a systematic overview of the current empirical evidence for different approaches to within-class differentiated instruction in secondary education. Therefore, we aim to (1) give an overview of the empirical literature on effects of differentiated instruction on student achievement in secondary education, and (2) consider the degree to which contextual and personal factors inhibit or enhance the effects of within-class differentiated instruction.

Our study is guided by the following research questions:

RQ1. What is the research base regarding the effects of within-class differentiated instruction on students' academic achievement in secondary education?

RQ2. How are the selected approaches to differentiated instruction operationalized?

RQ3. What are the overall effects of differentiated instruction on students' academic achievement?

RQ4. Which contextual and personal factors inhibit or enhance the effects of differentiated instruction on student achievement?

Based on previous research, we hypothesize to find literature on multiple possible approaches to differentiated instruction in the classroom. Probably, there will be more evidence for some operationalizations (like ability grouping) than for others. Overall, we hypothesize that differentiated instruction will have a small to medium positive effect on students' academic achievement. Several contextual and personal factors may affect the implementation. In this review, we will include information about relevant contextual and personal variables—when provided—into the interpretation of the literature.

Study Design

In order to provide a systematic overview of the literature on within-class differentiated instruction, a best evidence synthesis ( Slavin, 1986 , 1995 ; Best Evidence Encyclopedia, n.d.) was applied. This was done by a-priori defining consistent, transparent standards to identify relevant studies about within-class differentiated instruction. Each selected study is discussed in some detail and results are evaluated. In case enough papers are found that are comparable, findings can be pooled across studies. The best-evidence strategy is particularly suitable for topics—such as differentiated instruction—for which the body of literature is expected to be rather small and diverse. In such cases, it is important to learn as much as possible from each study, not just to average quantitative outcomes and study characteristics (compare Slavin and Cheung, 2005 ). In a recent review study on differentiated instruction in primary schools, the best evidence synthesis approach was used as well ( Deunk et al., 2018 ). In this study, the authors mentioned the benefits of selecting studies using strict pre-defined criteria (to avoid a garbage in-garbage-out effect). Moreover, combining a meta-analysis with relatively extended descriptions of the included studies in order to make the information more fine-grained was found to improve the interpretability of the results.

Working Definition of Differentiated Instruction

To select relevant studies for our review, we used the following working definition of differentiated instruction: Differentiated teaching in the classroom consisting of planned adaptations in process, learning time, content, product or learning environment for groups of students or individual students. Adaptations can be based on achievement/readiness or another relevant student characteristic (such as prior knowledge, learning preferences, and interest) with the goal of meeting students' learning needs.

Adaptations that are merely organizational, such as placing students in homogeneous groups without adapting the teaching to relevant inter-learner differences, were excluded. Interventions using approaches like peer tutoring, project-based learning and other types of collaborative leaning were eligible, but only when planned differentiated instruction was applied based on relevant student characteristics (e.g., by assigning specific roles based on students' abilities). Beyond the scope of this review were studies on differentiated instruction outside the classroom such as between-class differentiation (streaming or tracking), tutoring outside the classroom, or stratification of students between schools.

Search Strategy

The studies for our best evidence synthesis were identified in a number of steps. First, we performed a systematic search in the online databases ERIC, PsycINFO, and Web of Science (SSCI). Following the guidelines of Petticrew and Roberts (2006) , a set of keywords referring to the intervention (differentiation combined with keywords referring to instruction), the population (secondary education) and the outcomes of interest (academic outcomes) were used. We limited the findings to studies published between 2006 and 2016 that were published in academic journals. Although this first search yielded relevant studies, it failed to identify a number of important studies on differentiated instruction practices known from the literature. This was because search terms like “differentiation” and “adaptive” were not used in all relevant studies. Some authors used more specific terms such as ability grouping, tiered lessons, flexible grouping and mastery learning. Therefore, an additional search was performed in ERIC and PsycINFO with more specific keywords associated with differentiated instruction. We added keywords referring to various homogeneous or heterogeneous clustering approaches, to mastery learning approaches, or to convergent or divergent approaches (see Appendix A for the full search string) 1 .

Additional to this protocol-driven approach, we used more informal approaches to trace relevant studies. We cross-referenced the selected papers and recent review studies on related topics, used personal knowledge about relevant papers, and consulted experts in the field. We only used newly identified papers in case they were from journals indexed in the online databases Ebscohost, Web of Science, or Scopus to avoid selecting predatory journal outputs.

Selection of Papers

The identified papers were screened in pre-designed Excel sheets in two stages. First, two independent coders applied a set of inclusion criteria (criteria 1–8) to all papers based on title, abstract, and keywords. The papers that met the following conditions were reviewed in full text: (1) one or both of the coders judged the paper to be included for full text review based on the inclusion criteria using the title, abstract, and keywords, or (2) the study fulfilled some of the inclusion criteria but not all criteria could be discerned clearly from the title, abstract or keywords. Second, in a full text review, two coders applied the inclusion criteria again after reading the full paper. If a study met the basic criteria 1–8, additional methodological criteria (9–13) were checked in order to make the final selection. To assure the quality of the coding process, full-text coding of both coders was compared. Differences between coders about whether the study met certain inclusion criteria were resolved by discussion and consensus. The dual coding process by two reviewers was used since this substantially increases the chance that eligible studies are rightfully included ( Edwards et al., 2002 ). Only studies that met all 13 inclusion criteria were included in the review.

Inclusion Criteria

The following inclusion criteria were used to select the relevant papers. These criteria were based on a prior review study on differentiated instruction in primary education ( Deunk et al., 2018 ) and the best evidence studies by Slavin and colleagues ( Slavin and Cheung, 2005 ; Slavin et al., 2008 , 2009 ; Slavin, 2013 ; Cheung et al., 2017 ).

1. Within-class differentiated instruction: The study is about the effect of within-class differentiated instruction, as defined in our study (see section Working Definition of Differentiated Instruction).

2. Practicality : The differentiated instruction approach is practical for teachers ( Janssen et al., 2015 ). Teachers must be able to apply this intervention themselves in a regular classroom. In addition, the intervention is time- and cost-effective, meaning that it should not take excessive training or coaching nor use of external teachers in the classroom to implement the approach. Interventions in which ICT applications are used to support the teachers' instruction and can be controlled by the teacher (e.g., in blended learning environments in which teachers make use of on-line tools or PowerPoint) could be included. However, studies on the effects of fully computerized adaptive programs (e.g., with adaptive feedback or intelligent tutors) or differentiation approaches for which an external teacher (or tutor) is needed (such as pullout interventions) were excluded.

3. Study type: Students in a differentiated instruction intervention condition are compared to those in a control condition in which students are taught using standard practice (“business as usual”), or to an alternative intervention (compare Slavin et al., 2008 , 2009 ; Slavin, 2013 ; Cheung et al., 2017 ; Deunk et al., 2018 ). The design could be truly randomized or quasi-experimental or matched (the control condition could be a group of other students in a between-group design, or students could be their own control group in a within-groups design) 2 . Additionally, large-scale survey designs in which within-class differentiated instruction is retrospectively linked to academic outcomes were eligible for inclusion (compare Deunk et al., 2018 ). Surveys have increasingly included been used in reviews of effectiveness, although one must keep in mind that no finding from a survey is definitive ( Petticrew and Roberts, 2006 ).

4. Quantitative empirical study : The study contains quantitative empirical data of at least 15 students per experimental group (compare Slavin et al., 2008 , 2009 ; Slavin, 2013 ; Cheung et al., 2017 ; Deunk et al., 2018 ). Other studies such as qualitative studies, case studies with fewer than 15 students, or theoretical or descriptive studies were excluded.

5. Secondary education: The study was executed in secondary education. For example, in middle schools, high schools, vocational schools, sixth-form schools or comparable levels of education for students from an age of about 11 or 12 years onwards. In some contexts, secondary schools could include grades as low as five, but they usually start with sixth or seventh grades (compare Slavin, 1990a ).

6. Mainstream education : The study was performed in a mainstream school setting (in a regular school, during school hours). Studies that were performed in non-school settings (e.g., in a laboratory or the workplace) or in an alternate school setting (e.g., an on-line course, a summer school, a special needs school) were excluded.

7. Academic achievement : Academic achievement of students is reported as a quantitative dependent variable, such as mathematics skills, language comprehension, or knowledge of history.

8. Language : The paper is written in English or Dutch (all authors master these languages), but the actual studies could be performed in any country.

Additional inclusion criteria used in the full-text review:

9. Differentiated instruction purpose: The study is about differentiated instruction with the aim of addressing cognitive differences (e.g., readiness, achievement level, intelligence) or differences in motivation / interest or learning profiles ( Tomlinson et al., 2003 ). Studies in which adaptions were made based on other factors such as culture (“culturally responsive teaching”) or physical or mental disabilities are beyond the scope of this review.

10. Implementation : The intervention is (at least partly) implemented. If this was not specifically reported, implementation was assumed.

11. Outcome measurement: The dependent variables/outcome measures include quantitative measures of achievement. Experimenter-made measures were accepted if they were comprehensive and fair to the both groups; no treatment-inherent measures were included ( Slavin and Madden, 2011 ).

12. Effect sizes : The paper provides enough information to calculate or extract effect sizes about the effectiveness of the differentiated instruction approach.

13. Comparability : Pretest information is provided (unless random assignments of at least 30 units was used and there were no indications of initial inequality). Studies with pretest differences of more than 50% of a standard deviation were excluded because—even with analyses of covariance—large pretest differences cannot be adequately adjusted for ( Slavin et al., 2009 ; Slavin, 2013 ; Cheung et al., 2017 ; compare Deunk et al., 2018 ).

Data Extraction

After the final selection of papers based on the criteria above, relevant information was extracted from the papers and coded by two independent reviewers in a pre-designed Excel sheet (see Appendix B ). Discrepancies between the extractions of both reviewers were discussed until consensus was reached. Missing information regarding the methodology or results was requested from the authors by e-mail (although only few responses were received). The content coding was used (additional to the full texts) to inform the literature synthesis and to extract data for the calculation of effect sizes.

Data Analysis

We transformed all outcomes on student achievement from the selected papers to Cohen's d , which is the standardized mean difference between groups ( Petticrew and Roberts, 2006 ; Borenstein et al., 2009 ). To do so, the program Comprehensive Meta-Analysis (CMA) version 2 was used ( Borenstein et al., 2009 ). Effect sizes were calculated using a random effects model since we have no reason to assume that the studies are “identical” in the sense that the true effect size is exactly the same in all studies ( Borenstein et al., 2010 ). Methods of calculating effects using different types of data are described in Borenstein et al. (2009) and Lyons (2003) . When outcomes were reported in multiple formats in the paper, we chose the means and standard deviations to come to transparent and comparable outcomes. The effects were standardized using post-score standard deviations for measures where this was needed. For some outcome formats, CMA requires the user to insert a pre-post correlation. Since none of the selected papers provided this number, we assumed a correlation of 0.80 in the analyses since it is reasonable to assume such a pre- post correlation in studies in secondary education ( Swanson and Lussier, 2001 ; Cole et al., 2011 ). This correlation does not affect the Cohen's d statistic but has impact on its variance component. For the papers in which multiple outcome measures were reported, we used the means of the different measures. In case only subgroup means (of subgroups within classes of schools) were reported, we combined the outcomes of the subgroups with study as the unit of analysis to calculate a combined effect ( Borenstein et al., 2009 ). For one study in which the intervention was executed in separate schools differing in implementation and findings, we have included the schools in the analyses separately (using schools in which the intervention took place as the unit of analysis).

Search Results

Our search led to 1,365 hits from the online databases ERIC, PsycINFO and Web of Science and 34 cross-referenced papers. Excluding duplicates, 1,029 papers were reviewed. See Appendix C for a flow-chart of the selection process. In total, 14 papers met the eligibility criteria for inclusion. Papers reporting on the same project and outcomes were taken together as one study. The papers by Altintas and Özdemir (2015a , b) report on the same project. The same applies to two other papers as well ( Vogt and Rogalla, 2009 ; Bruhwiler and Blatchford, 2011 ). Thus, in the end, 12 unique studies were included in our review and meta-analysis leading to 15 effects in total (since for one study the four different schools in which the intervention was executed were taken as the unit of analysis).

Study Characteristics

In Table 2 , the characteristics and individual effects of the studies included in our review are summarized. The selection of studies includes eight quasi-experimental studies in which classes were randomly allocated to a control or experimental condition ( Mastropieri et al., 2006 ; Richards and Omdal, 2007 ; Huber et al., 2009 ; Vogt and Rogalla, 2009 ; Little et al., 2014 ; Altintas and Özdemir, 2015a , b ; Bal, 2016 ; Bhagat et al., 2016 ), three studies in which schools were randomly allocated to conditions ( Wambugu and Changeiywo, 2008 ; Mitee and Obaitan, 2015 ; Bikić et al., 2016 ), and one survey-study ( Smit and Humpert, 2012 ). These studies covered a wide range of academic subjects, including science, mathematics and reading. In terms of the number of participating students, six studies were small-scale studies ( N < 250) and six were large-scale studies ( N > 250). However, note that all experiments had nested designs. Only the studies of Little et al. (2014) and Vogt and Rogalla (2009) have at least 15 cases in each experimental condition at the level of randomization. Four studies were performed in the United States of America, five in Europe, one in Taiwan, and two in Africa. All studies were performed in secondary education, but the Vogt and Rogalla study represents a combined sample of primary- and secondary education students.

Table 2 . Summary of contents of the selected papers and the effects of the individual studies on student achievement.

Literature Synthesis

To further reflect on the findings from the selected studies in respect to our research questions, we will give a more detailed description of the study designs, implementations and findings here.

Studies on Generic Approaches to Differentiated Instruction

Although adaptive teaching does not necessarily include differentiated instruction, we found two quasi-experimental studies on adaptive teaching that (to some extent) matched our definition of differentiated instruction. In the large-scale study by Vogt and Rogalla (2009) , teachers were trained in adaptive teaching competency to improve their teaching and, in turn, to maximize students' learning. In the project “Adaptive Teaching Competency,” that was also included in the paper of Bruhwiler and Blatchford (2011) , adaptive teaching was characterized as including: sufficient subject knowledge, taking the diverse pre-conditions and learning processes of students into account, using various effective teaching methods for the whole group, differentiating for students' varying learning needs, supporting students in the regulation of learning processes, and using effective classroom management. In the project, teachers learned to focus on both adaptive planning prior to the lesson, as well as making adaptations during the lesson. Teachers of 27 primary school classes and 23 secondary school classes with 623 students were recruited to learn more about adaptive teaching. They participated in a 2-day workshop, received several coaching sessions in the classroom and used the adaptive teaching framework in their classes for eight science lessons. After the intervention, it was measured—among others—whether teachers differentiated to meet students' diverse skills and interests. After the intervention, teachers' competency in planning adaptive lessons significantly increased but their “Adaptive Implementation” did not change much. Unfortunately, in the coaching sessions, teachers often did not discuss about issues of adapting to the diversity of students' skills and their pre-existing knowledge. The results of students in the experimental classes were compared to those of 299 control students. The authors reported that the secondary students in the experimental group outperformed their counterparts in control classrooms on a science achievement test after the intervention. However, since we only had access to the means of the combined sample in primary and secondary education we used the combined sample results. Our calculation based on these means shows a small non-significant intervention effect of d = +0.133 (see Table 2 ). The authors argue that more coaching may be needed to foster the implementation of adaptive teaching in the classroom, although it would decrease the cost-effectiveness of the approach.

In the study by Huber et al. (2009) , teachers learned about adaptive teaching in a workshop, and were asked to incorporate it into their lessons. The intervention was the Prevention through Alternative Learning Styles (PALS) program aimed at prevention of alcohol-, tobacco-, and other drug (AOTD) abuse. Prevention of alcohol-, tobacco-, and other drugs is rather commonplace in secondary schools. For instance, in the US, students typically get into prevention programs more than once in their school career ( Kumar et al., 2013 ) and European schools are also encouraged to take action in promoting students' health ( World Health Organiasation, 2011 ). Teachers attended a 1-day workshop about adaptive teaching by means of: modifying time, increasing or decreasing the number of items to be learned or completed, increasing the level of support, changing the input or the way the material is presented, changing the output, adapting the amount of active participation, changing to alternate goals and expectations, adapting the level of difficulty for each individual, and providing different instruction and materials. In addition, teachers learned about alternative learning styles and disabilities. PALS materials were developed by the research team to match students' specific needs and related abilities. In a quasi-experimental study, four grade 6–8 teachers taught the 10 PALS intervention lessons to their classes and PALS team members taught another 24 classes. School officials suggested a convenient comparison group receiving the traditional prevention program. In reference to the control group, the PALS program had a large significant effect of d = +1.374 on students' knowledge of the effects of ATOD (see Table 2 ). These results were replicated in a second, within-group repeated measures design. Although the findings seem promising, more information is needed about how the approach was implemented; in the paper, it is unclear how teachers applied the information from the training in their instruction. Moreover, replication of the findings in a study in which teachers teach all project lessons may also help clarify whether the effects of the intervention were affected by the fact that project staff taught most lessons in the experimental condition.

We only selected two studies using a generic approach to differentiated instruction and the effects of the studies described above differ considerably regarding their intervention, school subject, and findings. This makes it hard to estimate the overall effectiveness of generic approaches. The study of Huber seems promising, but unfortunately, the study of Vogt and Rogalla did not lead to positive achievement effects for students across the primary and secondary school group. More studies are needed to gain insight in how teachers could effectively and efficiently be supported or coached to master the multifaceted approach of differentiated instruction.

Studies on Differentiated Instruction Using Homogeneous Clustering

A number of selected studies use a macro-adaptive approach to differentiated instruction ( Richards and Omdal, 2007 ; Altintas and Özdemir, 2015a , b ; Bal, 2016 ; Bikić et al., 2016 ). Of these studies, the study of Richards and Omdal (2007) has the most robust design. In this study, first year students were randomized over 14 classes and then classes were randomly assigned to conditions. Within the experimental condition, the science content for ability groups was adapted to students' learning needs by means of tiering. To study the effectiveness of the approach, 194 students were randomly assigned to classes in which the teachers used tiered content, while 194 other students were in the control group that worked with the midrange curriculum for 4 weeks. Each teacher was assigned at least one treatment and one control class. After a pretest, students in the experimental condition were assigned to three ability groups: a low background knowledge group (around the lowest scoring 10 percent of all students), a midrange group (about 80 percent), and a high background group (the highest scoring 10 percent). One of the researchers produced the instructional materials for the study. To develop the differentiated materials, first core instructional materials were developed that were aimed at the midrange group. Next, the content was differentiated for the low and high background students. Adaptations were made to the depth of content, the degree of teacher dependence and structuring, the number of steps, the skills, time on task, the product, and the available resources. Students were asked to work together within their tiers. There was an overall small significant effect of the intervention of d = +0.284 in favor of the tiering condition (see Table 2 ). Closer analyses of subgroup results (see Table 2 ) show that this is particularly due to a large effect for the low background learners of d = +1.057. For high-range learners, differences between the control condition and the experimental condition are near to zero ( d = +0.077), although this may be partly due to a ceiling effect on the test. The authors conclude that curriculum differentiation through tiered assignments can be an effective way to address the needs of low achieving students. They recommend, however, that it should be accompanied by professional support and that teachers who design the tiers should have substantial subject matter knowledge and experience with learners with different needs.

In the study by Bikić et al. (2016) , the effectiveness of differentiated instruction of geometry content within a problem-based learning approach is studied. In the quasi-experiment, the authors compare an approach in which students solved mathematics problems on three levels differing in complexity using problem-based learning to a control condition. The study design is not described in detail, but since the authors state “students of the experimental group and control group were not the students from the same school” it seems that schools were allocated to an experimental or control condition to study the effectiveness of the approach. Within the experimental condition, 88 secondary school students were assigned to three groups (low- average-, or high-achievers) based on an initial test, and then worked on adapted levels of geometry problems for 16 lessons before completing a final test. An example of the differentiated materials in the paper shows that the three ability groups all received a different task (which was a variation of the same task differing in complexity). Unfortunately, it is not described how the students exactly processed the content. In the control condition, 77 other students were taught in the usual, traditional manner. Students in the ability grouping condition outperformed the control students with a moderate positive effect of d = +0.539 (see Table 2 ). Subgroup analyses indicate that the approach was most effective for average ability students; students in the high achieving group did not outperform high achieving students in the control group. Do note however that the high achieving groups were small (12 exp. vs. 14 contr. students), hence, these results should be interpreted with caution. More research would be needed to clarify to which extent the differentiated content improved the effectiveness of the problem-based learning approach.

A different grouping approach is one based on preferred learning styles. In the study of Bal (2016) , grade 6 students completed an algebra pre-test as well as filling out a learning style inventory (kinesthetic, visual, affective learning styles). Algebra-learning materials an activities are adapted for two tiers; for low performing students and high performing students, also adapted for different learning styles of students in the experimental group. Despite the fact that there are reasons not to use learning styles as a distinction between students (see e.g., Kirschner et al., 2018 ), the authors did find large positive effects of the tiering approach after 4 weeks of teaching ( d = + 1.085, see Table 2 ). Do note however that ANCOVA results were used to calculate the effects which may lead to some positive bias in this estimate. Based on information from student-interviews presented in the paper, it seems that students experienced success in learning and enjoyed the materials and activities developed for the experimental condition. It is unclear however, how the materials and activities were made more appropriate for students' readiness (and learning style) and how they differed from the approach in the control condition that used traditional teaching. In that sense, it is difficult to judge what caused these positive findings. In another study on mathematics by Altintas and Özdemir (2015a , b) , teachers assessed students' preferred learning modalities by taking a multiple intelligences inventory. The data obtained from the inventory were used to determine the students' project topics, to select the teachers' teaching strategies, and to determine the relevant factors for motivating students. The effectiveness of the approach, which was originally designed for gifted students, was evaluated in a sample of 5 to 7th grade students in Turkey. After pretesting, one class of students was allocated to the experimental condition and one class of the same grade formed the control group. The authors report a very large effect of the intervention after six practices lasting 7 weeks each when compared to classes working with the Purdue model for both grade 6 and grade 7 students ( d = +4.504 across subgroups, see Table 2 ). However, it is difficult to discern what exactly caused this finding. Little information was provided about how exactly the teachers planned and executed the lessons and how students' activities and objectives were matched to their dominant intelligences, nor was there much information about possible confounding factors. In addition, since the researcher who developed the multiple intelligences theory admits that the theory is no longer up to date ( Gardner, 2016 ), one could question whether learning preferences could be better determined based on another distinction.

In summary, from the studies we found on the effectiveness approaches to differentiated instruction using homogeneous clustering, we could infer that overall small to medium sized effects (and in some cases also large effects) of the approach on student achievement can be achieved in beta subjects. The study of Altintas and Özdemir shows a very large effect of this approach and the study of Bal also shows large effects. However, before we can corroborate these findings, more information would be needed. When we look at the operationalizations of differentiated instruction in the two larger studies, we see that teachers used variations of learning tasks that were designed to better match the learning needs of different ability groups. Differential effects for student outcomes are somewhat variable; the results are most profound for the low achieving group in the study by Richards and Omdal (2007) , and for the low and average achieving group in the study of Bikić et al. (2016) . In both studies, effectiveness for the high achieving group seemed negligible.

Studies on Mastery Learning

In two included studies, mastery learning was used to boost student achievement in physics and mathematics. The quasi-experimental studies reporting on mastery learning approaches in secondary education used randomization of schools to conditions and were both performed in African schools ( Wambugu and Changeiywo, 2008 ; Mitee and Obaitan, 2015 ). In the papers, the authors describe similar characteristics of mastery learning in their theoretical framework, such as specifying learning goals, breaking down the curriculum into small units, formative assessment, using corrective instruction for students who did not reach mastery, and retesting. This process continues until virtually all the students master the taught material ( Mitee and Obaitan, 2015 ), which emphasizes its aim of convergent differentiation. Mittee and Obaitan report a large effect of the mastery learning approach of d = +1.461 based on an experiment in which about 400 students from four schools were allocated to a mastery learning or a control condition (see Table 2 ). Wambugu and Changeiywo randomly divided four classes from four schools over the mastery learning or the experimental condition. Comparing the results on the physics achievement test of the two experimental classes a two control classes, they found a large effect of mastery learning ( d = +1.322 based on the findings of an ANOVA, see Table 2 ). However, do note that pretests were only available for two out of four classes (one control and one experimental).

Unfortunately, the information on the mastery learning approach in the lessons is rather limited in both papers. Therefore, it is difficult to judge how such large achievement gains can be reached by implementing mastery learning in secondary education. Nevertheless, we can extract a number of recommendations: First, both studies use corrective instruction for helping students gain mastery. Secondly, in both studies the authors refer to some type of collaborative learning in the corrective instruction phase. Lastly, Wambugu and Changeiywo note that the time needed to develop the learning objectives, formative tests, and corrective activities is considerable so teachers may want to work together in teacher teams to achieve these goals. More high-quality research is needed to replicate these findings and to gain insight in how teachers can apply this approach in practice.

Studies on Individualized Differentiated Instruction

The large-scale quasi-experimental study on differentiated reading instruction in middle schools by Little et al. (2014) used individualized adaptations to address students' learning needs. They used a program called the Schoolwide Enrichment Model-Reading Framework (SEM-R) to support students' reading adaptively. The SEM-R approach consists of three phases: (1) short read-alouds by the teacher (“Book Hooks”) and brief discussions about books, (2) students read independently in self-selected, challenging books while the teacher organizes individualized 5- to 7-min conferences with each student once every 1 to 2 weeks, (3) interest-based and more project-oriented activities. Professional development of teachers included workshops as well as classroom support from project staff. The focus of the intervention was on phases 1 and 2. Teachers were expected to implement SEM-R on a daily basis for about 40 to 45 min per day or 3 h per week. In a cluster-randomized design executed in four middle schools with 2,150 students, the effectiveness of the approach was compared to that of traditional teaching. The effects of the approach varied considerably across the different schools. The authors reported that, for the reading fluency outcome, SEM-R students significantly outperformed their control counterparts in two out of four schools. The standardized mean differences ranged from about −0.1 to +0.3 between the schools (see Table 2 ). The authors conclude that the intervention was at least as effective as traditional instruction. However, the wide range of implementations and effects on student outcomes between classes and schools illustrates the difficulty of implementing intensive forms of individualization in practice.

In the survey study of Smit and Humpert (2012) , the authors assessed which teaching practices teachers used to differentiate their teaching. In this sub-study of the project “Schools in Alpine Regions,” teachers from 8 primary schools and 14 secondary schools in the rural Alpine region of Switzerland participated. Teachers responded to a teacher questionnaire about differentiated instruction. They mainly reported to make adaptations at the individual level by, for instance, providing students with individual tasks (tiered assignments), adapting the number of tasks, or providing more time to work on tasks. Teachers often used “learning plans” as well as tasks in which students could take individual learning trajectories varying the content or learning rate. Flexible grouping was less common and alternative assessments were very rare. Peer tutoring occurred frequently, and tiered assignments were very common. On average, 38% of teachers' weekly lessons were differentiated. The authors conclude that teachers in their sample, on average, did not execute very elaborate differentiated instruction. Moreover, no significant relation between differentiated instruction and student achievement was found for neither a standardized language test ( d = −0.092) nor a standardized mathematics test ( d = −0.085, see Table 2 ). Following the survey study, an intervention study was executed with 10 of the schools that were included in the survey-study. In this study (that was not included in our selection since it was not published in an academic journal), teachers participated in workshops and team meetings and logged their learning experiences in portfolios. Teachers barely progressed in their differentiated instruction during the 2.5-year project ( Smit et al., 2011 ). Nevertheless, a high pedagogical team culture in schools was found to have a positive influence teachers' differentiated instruction ( Smit et al., 2011 ; Smit and Humpert, 2012) , and as such may be one of the keys to achieve improvement.

Overall, it seems that it is rather difficult to boost the achievement of the whole class by means of individualized approaches. However, as Little et al. (2014) suggest, individualization may be used as an approach to increase students' engagement with the learning content. A drawback of the approach may be that the requirements for organizing and monitoring learning activities by the teacher in individualized approaches could leave less time for high quality pedagogical interaction. Possibly, future research on individualization supported by digital technology may open up more possibilities for this approach to have high impact on student achievement ( Education Endowment Foundation, n.d. ).

Studies on Differentiated Instruction Using Heterogeneous Clustering

One of the included studies used differentiated instruction within mixed-ability learning settings. In the study by Mastropieri et al. (2006) , grade eight students worked on science assignments in groups of two or three. Peer-mediated differentiated instruction and tiering was used to adapt the content to students' learning needs within the groups. The authors developed three tiers of each assignment varying in complexity. Within the peer groups, students could work on activities on their own appropriate level and continue to the next level once proficiency was obtained. All lower ability level students—including students with learning disabilities—were required to begin with the lowest tier. In the experiment, 13 classes with a total of 216 students were assigned to the peer-mediated differentiated content condition or a teacher-led control condition. The researchers divided the classes in such a way that each teacher taught at least one experimental and one control classroom. After about 12 weeks, a small positive effect was found in favor of the peer-mediated condition with tiered content on both the unit test and the high stakes end of year test (respectively d = + 0.466 and d = + 0.306, see Table 2 ). The overall effect of d = +0.386 is comparable to that of the tiering intervention of Richards and Omdal (2007) discussed earlier. The effect is slightly higher, but this may also partly be affected by the use of adjusted means. In any case, more research is needed to disentangle the effects of the peer-learning and the differentiated content.

Studies on Differentiated Instruction in Flipped Classrooms

In flipped classroom instruction, content dissemination (lecture) is moved outside of the classroom, typically by letting students watch instructional videos before the lesson. This opens up more time for active learning inside the classroom ( Leo and Puzio, 2016 ). This format implies differentiation of learning time and pace before the lesson since students may rewind, pause or watch the video's multiple times according to their learning needs. However, whether the activities during the lesson encompass our operationalization of differentiated instruction (see Table 1 ) varies. From a recent meta-analysis on flipping the classroom ( Akçayir and Akçayir, 2018 ), we found one study in secondary education in which remediation in the classroom was mentioned as being part of the intervention. Bhagat et al. (2016) report on a quasi-experiment in which 41 high school students were assigned to a classroom using flipping-the-classroom and 41 students were in the control condition. The experimental group underwent “flipped” lessons on trigonometry for 6 weeks, while the control group followed similar lessons using the conventional learning method. Students in the flipped condition watched videos of 15–20 min before the lesson. During the lesson, students discussed problems collaboratively and, in the meantime, students who needed remediation were provided with extra instruction. After the intervention, students from the flipped classrooms outperformed their counterparts on a mathematics test and were more motivated. The authors report a large effect of the intervention on students' mathematics achievement based on analysis of covariance. However, the combined effect across the subgroup mean differences is modest d = 0.376, see Table 2 ). On average, experimental students of all abilities performed better, except for high achievers who did not significantly outperform the control group. These differential effects should be interpreted with caution because of the limited number of students in the subgroups. The pro of this study is that it gives some insights in the benefits of differentiated instruction embedded in an innovative approach to teaching. Yet, the authors did not specify clearly what the remediation and collaborative learning in the classroom consisted of and cannot disentangle effects of different elements of the intervention. More research would be needed to clarify the role and effectiveness of differentiated instruction in flipped settings.

Contextual and Personal Variables

As we discussed in the theoretical framework, many variables may influence teachers' implementation of differentiated instruction. We hoped to find evidence for this assumption in our selection of papers. However, in general, little information was provided about contextual and personal factors such as school, class, or teacher characteristics.

In our sample of studies, differentiated instruction was mostly applied to teaching mathematics and science. Additionally, there were also papers on literacy and social sciences. No clear differences in effectiveness could be observed between the subjects. Students varied in background characteristics across the studies. In the study by Little et al. (2014) , for instance, about 48 to 77 percent of students were from low SES. In the study by Mastropieri et al. (2006) , many ethnicities were represented. In the studies by Huber et al. (2009) , students were mostly European-American. Student ages varied from about 11 to 17 years old (see Table 2 ). Teacher characteristics were rarely reported. In the study by Mastropieri et al. (2006) , relatively inexperienced teachers participated with a mean of about 3 years in their current position, and in the studies by Vogt and Rogalla (2009) and Smit and Humpert (2012) , years of teaching experience varied considerably, with an average of about 15 to 17 years.

The only variable that is rather consistent across the studies is that teachers in the included studies relied considerably on external sources of information or support to help them implement differentiated instruction within their classrooms. In most of the selected studies, the research team developed materials for students, and teachers were instructed or coached in implementing the interventions (see Table 2 ). Although we aimed to select practical interventions, little information is provided about whether teachers were able to successfully execute the differentiated instruction practices independently in the long run.

Overall Effects of Differentiated Instruction

Ideally, combining our narrative reflection on the included papers with a meta-analysis of the findings would give us an answer as to how effective within-class differentiated instruction in secondary education may be. However, unfortunately, the number of papers that remained after applying our selection criteria is limited and the studies are heterogeneous in nature so meta-analyses of results should be interpreted with caution. To inform the readers however, we did add a forest plot with an overview of the average effect size of each individual study to the appendix (see Appendix D ). In Table 2 the effects and intermediate calculations for individual studies are described. A summary effect across all studies is also reported ( d = +0.741; 95% CI = 0.397–1.1085; Q = 507.701; df = 14; p < 0.01). The p -value of the Q statistic was significant which may indicate heterogeneity of the papers meaning that the true effects of the interventions may vary. Noticeably, the largest studies in our sample show small positive effects of differentiated instruction. In contrast, the relatively small studies reported on large effects, and the other studies mostly show moderate effects of the approach. A cumulative analysis (see Appendix D ) illustrates that the small study by Altintas and Özdemir (2015a , b) considerably shifts the point estimate of the effect size in the positive direction. Excluding this outlier, the summary effect of differentiated instruction is d = +0.509 (95% CI = 0.215–0.803; see Appendix D ). A funnel plot was made to check for publication bias (see Appendix E ). Using Duval and Tweedie's Trim and Fill method ( Duval and Tweedie, 2000 ), no adjusted values were estimated. This indicates that there is no evidence of publication bias. These analyses give some information about the range of effects that can be achieved with differentiated instruction interventions ranging. However, unquestionably, more information is needed before drawing a more definitive conclusion about the overall and relative effects of different approaches to differentiated instruction in secondary schools.

Suggestions for Reporting on Differentiated Instruction Interventions

One of the issues we encountered when performing this review, was that interventions and research methodologies were often described rather briefly. In addition, relevant context information was frequently missing. This is problematic, not only from a scientific point of view, but also to judge the transferability of the findings to practice. Therefore, we encourage researchers to diligently report on the methods and analytical techniques they used and to be specific about the outcomes that led to their conclusions (see e.g., Hancock and Mueller, 2010 ). Except for this general suggestion, we would like to provide a number of specific recommendations for reporting on differentiated instruction interventions (see Appendix F ).

Conclusion and discussion

The most important conclusion from our systematic review of the literature is that there are too few high-quality studies on the effectiveness of differentiated instruction in secondary education. Only 12 studies from 14 papers were selected after applying strict selection criteria to a large amount of literature on the topic. As expected, we found papers on various operationalizations of differentiated instruction like homogeneous grouping, differentiated instruction in peer-learning, and individualization. However, even within the most well-known approaches like ability grouping, the empirical evidence was limited. High quality teacher-led differentiated instruction studies in secondary education are scarce, although the literature on ICT-applications for differentiated instruction seems to be on the rise. This paucity has not changed much after our search, although there are some recent interesting endeavors for teacher professionalization in differentiated instruction ( Brink and Bartz, 2017 ; Schipper et al., 2017 , 2018 ; Valiandes and Neophytou, 2018 ) and there have been some recent small-scale studies including aspects of differentiated instruction ( Sezer, 2017 ; Adeniji et al., 2018 ). This paucity is remarkable given the large interest for the topic of differentiated instruction in both the literature as well as in policy and practice. Apparently, the premises of differentiated instruction seems substantial enough for schools and policy makers to move towards implementation before a solid research base has been established. On the one hand, this seems defendable; differentiated instruction matches the ambitions of educationists to be more student-oriented and to improve equity among students. In addition, there is prior research showing benefits of approaches like ability grouping and mastery learning for K-12 students' achievement ( Guskey and Pigott, 1988 ; Kulik et al., 1990 ; Kulik, 1992 ; Lou et al., 1996 ; Hattie, 2009 ; Steenbergen-Hu et al., 2016 ). Furthermore, the ideas behind differentiated instruction are in line with approaches which have repeatedly been linked to better learning such as having students work on an appropriate level of moderate challenge according to their “zone of proximal development” and matching learning tasks to students' abilities and interests to create “flow” ( Tomlinson et al., 2003 ). On the other hand, more research on different operationalizations of differentiated instruction is needed to help teachers and policy makers to determine which approaches are helpful for students of different characteristics and to gain insight in how these could be implemented successfully. From prior research in primary education, we know that it is likely that not all approaches have comparable effects, and that effects for low- average- and high ability students may vary ( Deunk et al., 2018 ). Our current review shows that there is much work to be done in order to further clarify which approaches work and why within the context of secondary education.

Having said that, the studies that we did find do give us some directions about the expectations we may have about the effectiveness of differentiated instruction in secondary education. Most well-designed studies in our sample reported small to medium-sized positive effects of differentiated instruction on student achievement. This finding is comparable to the moderate effects found in most differentiated instruction reviews (e.g., Kulik, 1992 ; Lou et al., 1996 ; Steenbergen-Hu et al., 2016 ) and other studies on educational interventions ( Sipe and Curlette, 1996 ). The overall effect in our study is a bit higher than in prior reviews, possibly due to the inclusion of various approaches to differentiated instruction, including mastery learning and more holistic approaches. Although we cannot give a conclusive answer about the effectiveness of differentiated instruction in secondary education, most of the included studies do illustrate the possibility of improving student achievement by means of differentiated instruction.

Moreover, the selected papers give insight in the many different ways that differentiated instruction can be operationalized and studied in secondary education. For instance, a number of studies used generic training of teachers in principles of differentiated instruction. Based on the findings, we would suggest that more research is needed to study how teachers can adequately be guided to implement such holistic approaches into their daily teaching (compare practicality theory by Janssen et al., 2015 ). Alternatively, in four of the selected studies homogeneous clustering by means of tiering and ability grouping was used as a structure for differentiated instruction. For the subgroups, learning content was adapted to better fit the needs of the students ( Richards and Omdal, 2007 ; Altintas and Özdemir, 2015a , b ; Bal, 2016 ; Bikić et al., 2016 ). Medium to large positive effects were reported of such an approach, indicating this may be one of the ways teachers may address differentiated instruction. This finding is comparable to findings on ability grouping in the meta-analyses by Steenbergen-Hu et al. (2016) and Lou et al. (1996) . The effects were somewhat larger compared to those in the studies in primary education discussed by Deunk et al. (2018) and Slavin (1990a) . One possible explanation might be that some of the studies mentioned in those previous reviews may have included grouping without any instructional adaptations, which was excluded from the current review. Also, in our selected papers on homogeneous clustering, researcher-developed outcome measures were used. Researcher-developed measures have previously been associated with larger effects than standardized measures ( Slavin, 1987 ; Lou et al., 1996 ). Turning to another approach, two studies were reviewed on the effectiveness of mastery learning. The authors reported large effects of mastery learning on student achievement. However, since the research methods were not thoroughly described in the papers, we cannot say much about the quality of the intervention nor the implementation. Two other studies focused on individualization. Overall, small and non-significant effects of this approach were found. It could be that teachers grapple with the organizational requirements of individualized instruction ( Education Endowment Foundation, n.d. ). Additionally, a study was found that successfully embedded differentiated instruction in a peer-learning setting by means tiered content matching students' learning needs ( Mastropieri et al., 2006 ). Lastly, one of the studies embedded remediation and collaboration in a flipped-classroom format illustrating how differentiated instruction can be applied within different approaches to teaching ( Bhagat et al., 2016 ).

Unfortunately, in only three studies, authors reported on differential effects for subgroups of students within classes. This makes it difficult to judge which differentiated instruction approach is most suitable for whom. In the studies ( Richards and Omdal, 2007 ; Bhagat et al., 2016 ; Bikić et al., 2016 ) that did report effects for subgroups, the interventions were shown to be most beneficial for low achieving (and in case of Bikić also the average achieving) subgroups of students, even though the learning content was adapted to better match the needs of other students too. However, it remains unclear whether this was caused by the differentiated instruction, by the fact that the teachers directed more attention toward low performing students, or by the fact that the outcome measures did not match the adapted content. In addition, the subgroups were relatively small, limiting the power of the findings. Therefore, more empirical evidence is needed about the implementation and relative effects of differentiated instruction to further inform the “differentiation-dilemma” of how to best divide time over students with different needs ( Denessen, 2017 ).

Regarding the contextual and personal variables across studies, students' age, the school subjects and teaching experience of teachers varied. The fact that positive results have been replicated in several settings with different populations, gives a first indication that the approach may be transferable across different contexts ( Petticrew and Roberts, 2006 ). One consistent finding across the studies is that teachers relied on external support to implement within-class differentiated instruction during the interventions. This is to be expected, since prior reviews found that implementing differentiated instruction is quite complex for teachers and that they may need considerable guidance to get it right ( Tomlinson et al., 2003 ; Subban, 2006 ; Van Casteren et al., 2017 ). Previous studies show that teachers receiving more professional development in differentiated instruction perceive higher efficacy and adapt their teaching to students more often ( Dixon et al., 2014 ; Suprayogi et al., 2017 ).

The contribution of the current review to existing knowledge of the effects of differentiated instruction on students' achievement in secondary education is as follows: First, it provides an overview of theoretical concepts and operationalizations of differentiated instruction in the classroom. Next, it shows that a systematic review of the literature leads to a limited body of evidence regarding the effectiveness of within-class differentiated instruction in secondary education. This overview of the state of the art within this theme may inform further research initiatives. Additionally, the study addresses some contextual and personal factors that may affect teachers' differentiated instruction.

Limitations

The most salient drawback of the review is the limited number of studies that were included. On the one hand, it is unfortunate that the limited number of selected papers makes it difficult to come to definitive conclusions about the effectiveness of within-class differentiated instruction. On the other hand, the importance of using systematic reviews to identify research gaps to inform further development of the field should not be underestimated ( Petticrew and Roberts, 2006 ). Defining consistent criteria for the selection of the best evidence available—as we have done in this study—may limit the number of selected studies but does help to ensure that the studies that are selected are highly informative ( Slavin, 1995 ). The limited number of studies we found is just about comparable to the number of within-class approaches that were selected in a recent review of between-class and within-class differentiated instruction in primary education ( Deunk et al., 2018 ). We only included studies in which student achievement was reported as an outcome measure. In future research, adding other types of outcomes and other types of study designs could add to the breadth of the research base.

Another limitation has to do with the quality of the selected papers and consequently with our approach to the analyses. First, the fact that we did not locate any truly randomized designs necessitates caution in interpreting the findings. Potential biases are likely to be greater for non-randomized studies compared to randomized trials ( Higgins and Green, 2011 ). Second, the number of participants at the level of randomization (often the classroom level) was mostly low. Furthermore, it was sometimes difficult to determine the quality of the studies due to a lack of information in the papers. We tried to gain insight in the differentiated instruction interventions, but often essential information was omitted. Also, the conversion to Cohen's d could not always be done using an identical approach across the different studies. Must studies reported pre- and/or post-scores on achievement tests that we could use to calculate the effects in a rather straightforward manner, but in a few cases we had to estimate effects based on other types of information (for instance adjusted means or analyses of variance) which may complicate comparability across studies. Another drawback is that authors sometimes provided the outcomes of subgroups (for instance classes or ability groups within classes), sometimes only outcomes of the experimental conditions, or sometimes both. In the case of differentiated teaching, researchers should clearly explain their aims regarding which students they want to support (convergent or divergent). And if the aims differ per subgroup, they should ideally report these separate effects too. To inform future research on the topic, we have suggested some reporting guidelines that may help to clarify the content of future approaches to differentiated instruction and how they were studied in the Appendix.

A final limitation, inherent to a topic that is so multifaceted, is that the choices we have made in how we defined within-class differentiated instruction have influenced our selection of the literature and, thus, should be considered when interpreting the findings. The existing literature is marked by different ways of defining and operationalizing differentiated instruction ( Suprayogi et al., 2017 ; Deunk et al., 2018 ). As such, our review may differ from the operationalizations of other authors. In addition, other ways to adapt teaching to students' learning needs are also certainly interesting to consider by teachers who want to better align teaching to students' needs. For example, the use of scaffolding techniques in which instruction is broken up in chunks, and instruction in each chunk is provided contingent to students' level of understanding is a promising instructional technique ( Van de Pol et al., 2010 , 2015 ). In addition, formative assessment is a helpful starting point for differentiated instruction or other types of adaptive teaching ( Kingston and Nash, 2011 ). Furthermore, as discussed in the theoretical framework, differentiated instruction is a broad construct that adds up as a sum of its parts including lesson planning, differentiated instruction, evaluation and general high-quality teaching behaviors. We could not include all these factors into the working definition used to select and synthesize the studies. Therefore, readers should keep in mind that in order to understand differentiated instruction comprehensively and apply it in practice, there is more to it than just executing a differentiated lesson. A thoughtful approach using different steps starting from planning to evaluation including high quality teaching behaviors is key.

Recommendations for Research and Practice

We would like to urge researchers to further study the impact and implementation of differentiated instruction. First, reviews and meta-analyses combining quantitative and qualitative information on the effects of different approaches to differentiated instruction for different outcomes may add further to the current knowledge base ( Dixon-Woods et al., 2005 ). When more quantitative studies are located, this enables more statistical possibilities that can be used to gain insight in differential effects and predictive characteristics of different student outcomes ( Lou et al., 1996 ; Moeyaert et al., 2016 ; Deunk et al., 2018 ). And qualitative studies may help us understand how teachers differentiate and how their subjective experiences in the classroom influence their differentiated instruction ( Civitillo et al., 2016 ). In addition, authors may want to add studies on affective student outcomes as well. For example, students may have better attitudes and motivation in differentiated classes in which teaching better matches their learning needs ( Kulik and Kulik, 1982 ; Lou et al., 1996 ; Maulana et al., 2017 ; Van Casteren et al., 2017 ).

Second, future studies on the development and evaluation of differentiated instruction interventions could add to the knowledge base about how to reach differentiated instruction's potential in practice. In order to support teachers, specific coaching on the job by experienced peers or external coaches or other types of professionalization may help to develop awareness and implementation of differentiated instruction ( Latz et al., 2009 ; Smit and Humpert, 2012 ; Parsons et al., 2018 ; Valiandes and Neophytou, 2018 ). Teachers should learn to reflect upon the decisions they make when adapting their teaching ( Parsons et al., 2018 ). Moreover, teachers need team support and sufficient time to develop their differentiated instruction ( Stollman, 2018 ). Research shows that teachers themselves are quite enthusiastic about bottom-up professionalization approaches like peer-coaching or professional learning communities ( Van Casteren et al., 2017 ). Whatever approach one chooses, there are some characteristics which may facilitate the effectiveness of professionalization including: a focus on both content and pedagogical knowledge, sufficient duration of the intervention, initial training and follow-up sessions, a facilitation of collaboration and communication with colleagues and experts, constant on-site support and help during the implementation- and the development of personal skills for reflection and self-evaluation of teachers ( Valiandes and Neophytou, 2018 ). In addition, teacher educators should be mindful of teacher differences themselves too by providing differentiated professionalization ( Stollman, 2018 ). In this review, we did not include studies on the effectiveness of adaptive ICT applications on students' progress. However, ICT can play a significant role in the creation of student-centered learning environments when used as more than a simple add-on to regular teaching ( Smeets and Mooij, 2001 ; Deunk et al., 2018 ). Some recent studies on adaptive or personalized ICT programs, digital pen technologies, and blended learning show that such interventions can support differentiated instruction and have positive effects on student achievement ( Walkington, 2013 ; Chen et al., 2016 ; Van Halem et al., 2017 ; Ghysels and Haelermans, 2018 ), although more research is needed to assess for whom and for which type of outcomes these approaches are beneficial ( Van Klaveren et al., 2017 ). In the studies in this review, fixed outcome measures were used to assess students' learning. Possibly, adaptive testing will provide more room for assessing differentiated growth trajectories in future studies ( Martin and Lazendic, 2018 ).

Lastly, when aiming to gain further insight in the effectiveness of differentiated instruction, authors may want to reflect on how differentiated instruction is operationalized and measured. In prior research, teacher questionnaires were often used to assess teachers' differentiated instruction practices ( Roy et al., 2013 ; Prast et al., 2015 ). In addition, classroom observations of differentiated instruction or adaptive teaching behavior have been used ( Cassady et al., 2004 ; Van Tassel-Baska et al., 2006 ; Van de Grift, 2007 ). Alternatively, in our selection of papers, we found some interesting ways to determine how teachers differentiate. For example, using vignette or video tests ( Vogt and Rogalla, 2009 ; Bruhwiler and Blatchford, 2011 ) or by means of teacher logs or observations ( Little et al., 2014 ). Enriching measures of teacher behavior with information about the match of the behavior with students' needs may be another step forward ( Van Geel et al., 2019 ). We would like to recommend authors to further develop, evaluate and apply measures for differentiated instruction that can be used to gain insight in how differentiated instruction is linked to various student outcomes.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Author Contributions

AS-J set up the methods of the paper, analyzed the theoretical backgrounds and is responsible for the concept of the article, and together with co-authors, extracted data, performed the analyses, and wrote the paper. AM coordinated the selection of studies, worked on data selection and extraction, and contributed to writing the paper. MH-L and RM designed the overarching project, acquired funding for the execution, and contributed to the conceptualization of differentiated instruction and the review process.

This work was supported by the Dutch scientific funding agency (NRO) under Grant number 405-15-732.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We want to thank Bernie Helms for his contribution to the practical work needed to execute this study. Additionally, we greatly value the consultations regarding the analyses with our colleagues Dr. Hester de Boer and Prof. Dr. Roel Bosker from GION Educational Sciences.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.02366/full#supplementary-material

1. ^ We did not include search terms specifically referring to heterogeneous approaches in the search string. Although heterogeneous grouping may include differentiation, adaptiveness is often not the focus of these studies.

2. ^ Quasi-experimental studies in which experimental and control groups are well matched, and covariates that correlate strongly with pretests are used to adjust outcomes, can be a valuable source of information usable for meta-analyses ( Slavin et al., 2008 ; Slavin and Smith, 2009 ), although the results of (especially small-scale) quasi-experimental studies should be evaluated with caution ( Cheung and Slavin, 2016 ).

3. ^ References included in the systematic review are marked with an asterisk.

Adeniji, S. M., Ameen, S. K., Dambatta, B. U., and Orilonise, R. (2018). Effect of mastery learning approach on senior school students' academic performance and retention in circle geometry. Int. J. Instruct. 11, 951–962. doi: 10.12973/iji.2018.11460a

CrossRef Full Text | Google Scholar

Akçayir, G., and Akçayir, M. (2018). The flipped classroom: a review of its advantages and challenges. Comput. Educ. 126, 334–345. doi: 10.1016/j.compedu.2018.07.021

Altemueller, L., and Lindquist, C. (2017). Flipped classroom instruction for inclusive learning. Br. J. Spec. Educ. 44, 341–358. doi: 10.1111/1467-8578.12177

* Altintas, E., and Özdemir, A. S. (2015a). The effect of the developed differentiation approach on the achievements of the students. Eurasian J. Educ. Res. 61, 199–216. doi: 10.14689/ejer.2015.61.11

* Altintas, E., and Özdemir, A. S. (2015b). Evaluating a newly developed differentiation approach in terms of student achievement and teachers' opinions. Educ. Sci. Theor. Pract. 15, 1103–1118. doi: 10.12738/estp.2015.4.2540

* and Bal, A. P. (2016). The effect of the differentiated teaching approach in the algebraic learning field on students' academic achievements. Eurasian J. Educ. Res. 63, 185–204. doi: 10.14689/ejer.2016.63.11

Best Evidence Encyclopedia (n.d.). Review Methods. Criteria for Inclusion in the Best Evidence Encyclopedia . Available online at: http://www.bestevidence.org/methods/criteria.htm

Google Scholar

* Bhagat, K. K., Chang, C., and Chang, C. (2016). The impact of the flipped classroom on mathematics concept learning in high school. J. Educ. Technol. Soc. 19, 134–142. Available online at: https://psycnet.apa.org/record/2016-35586-003

* Bikić, N., Maričić, S. M., and Pikula, M. (2016). The effects of differentiation of content in problem-solving in learning geometry in secondary school. EURASIA J. Math. Sci. Technol. Educ. 12, 2783–2795. doi: 10.12973/eurasia.2016.02304a

Blatchford, P., Bassett, P., and Brown, P. (2011). Examining the effect of class size on classroom engagement and teacher-pupil interaction: differences in relation to pupil prior attainment and primary vs. secondary schools. Learn. Instruct. 21, 715–730. doi: 10.1016/j.learninstruc.2011.04.001

Borenstein, M., Hedges, L. V., Higgins, J. P. T., and Rothstein, H. R. (2009). Introduction to Meta-Analysis . Chichester: John Wiley and Sons. doi: 10.1002/9780470743386

PubMed Abstract | CrossRef Full Text | Google Scholar

Borenstein, M., Hedges, L. V., Higgins, J. P. T., and Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Res. Synth. Methods 1, 97–111. doi: 10.1002/jrsm.12

Bosker, R. J. (2005). De Grenzen van Gedifferentiëerd Onderwijs. Groningen: Rijksuniversiteit Groningen . Available online at: http://www.rug.nl/research/portal/files/14812458/bosker.pdf

Bray, B., and McClaskey, K. (2013). Personalization vs. Differentiation vs. Individualization. (No. version 3) . Available online at: http://www.personalizelearning.com/2012/04/explaining-chart.html

Brink, M., and Bartz, D. E. (2017). Effective use of formative assessment by high school teachers. Pract. Assess. Res. Eval. 22, 1–10. Available online at: https://pareonline.net/getvn.asp?v=22&n=8

* Bruhwiler, C., and Blatchford, P. (2011). Effects of class size and adaptive teaching competency on classroom processes and academic outcome. Learn. Instruct. 21, 95–108. doi: 10.1016/j.learninstruc.2009.11.004

Cassady, J. C., Neumeister, K. L. S., Adams, C. M., Cross, T. L., Dixon, F. A., and Pierce, R. L. (2004). The differentiated classroom observation scale. Roeper Rev. 26, 139–146. doi: 10.1080/02783190409554259

Cavanagh, S. (2014). What is personalised learning? Educators seek clarity. Education Week . Available online at: https://www.edweek.org/ew/articles/2014/10/22/09pl-overview.h34.html

Chen, C., Tan, C., and Lo, B. (2016). Facilitating English-language learners' oral reading fluency with digital pen technology. Interact. Learn. Environ. 24, 96–118. doi: 10.1080/10494820.2013.817442

Cheung, A. C. K., and Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educ. Res. 45, 283–292. doi: 10.3102/0013189X16656615

Cheung, A. C. K., Slavin, R. E., Kim, E., and Lake, C. (2017). Effective secondary science programs: a best-evidence synthesis. J. Res. Sci. Teach. 54, 58–81. doi: 10.1002/tea.21338

Civitillo, S., Denessen, E., and Molenaar, I. (2016). How to see the classroom through the eyes of a teacher: consistency between perceptions on diversity and differentiation practices. J. Res. Spec. Educ. Needs 16, 587–591. doi: 10.1111/1471-3802.12190

Clarke, D., and Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teach. Teach. Educ. 18, 947–967. doi: 10.1016/S0742-051X(02)00053-7

Cole, R., Haimson, J., Perez-Johnson, I., and May, H. (2011). Variability in Pretest-Posttest Correlation Coefficients by Student Achievement Level. (NCEE Reference Report 2011-4033). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Corno, L. (2008). On teaching adaptively. Educ. Psychol. 43, 161–173. doi: 10.1080/00461520802178466

Coubergs, C., Struyven, K., Engels, N., Cools, W., and De Martelaer, K. (2013). Binnenklas-Differentiatie. Leerkansen Voor Alle Leerlingen . Leuven: Uitgeverij Acco.

De Jager, T. (2013). Guidelines to assist the implementation of differentiated learning activities in south African secondary schools. Int. J. Inclus. Educ. 17, 80–94. doi: 10.1080/13603116.2011.580465

De Neve, D., and Devos, G. (2016). The role of environmental factors in beginning teachers' professional learning related to differentiated instruction. Sch. Effect. Sch. Improv. 27, 557–579. doi: 10.1080/09243453.2015.1122637

Denessen, E. J. P. G. (2017). Verantwoord Omgaan met Verschillen: Social-Culturele Achtergronden en Differentiatie in Het Onderwijs. [Soundly Dealing with Differences: Socialcultural Background and Differentiation in Education]. (Inaugural lecture). Leiden: Leiden University . Available online at: https://openaccess.leidenuniv.~nl/handle/1887/51574

Denessen, E. J. P. G., and Douglas, A. S. (2015). “Teacher expectations and within-classroom differentiation,” in Routledge International Handbook of Social Psychology of the Classroom , eds C. M. Rubie-Davies, J. M. Stephens, and P. Watson (London: Routledge; Taylor and Francis Group, 296–303.

Deunk, M. I., Smale-Jacobse, A. E., de Boer, H., Doolaard, S., and Bosker, R. J. (2018). Effective differentiation practices: a systematic review and meta-analysis of studies on the cognitive effects of differentiation practices in primary education. Educ. Res. Rev. 24, 31–54. doi: 10.1016/j.edurev.2018.02.002

Dixon, F. A., Yssel, N., McConnell, J. M., and Hardin, T. (2014). Differentiated instruction, professional development, and teacher efficacy. J. Educ. Gifted 37, 111–127. doi: 10.1177/0162353214529042

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., and Sutton, A. (2005). Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Serv. Res. Policy 10, 45–53. doi: 10.1177/135581960501000110

Duval, S., and Tweedie, R. (2000). Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56, 455–463. doi: 10.1111/j.0006-341X.2000.00455.x

Education Endowment Foundation (n.d.). Teaching Learning Toolkit An Accessible Summary of the International Evidence on Teaching 5-16 year-Olds . Available online at: https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit/

Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., and Wentz, R. (2002). Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat. Med. 21, 1635–1640. doi: 10.1002/sim.1190

Gardner, H. (2016). “Multiple intelligences: prelude, theory, and aftermath,” in Scientists Making a Difference , eds R. J. Sternberg, S. T. Fiske, and D. J. Foss (New York, NY: Cambridge University Press).doi: 10.1017/CBO9781316422250

Ghysels, J., and Haelermans, C. (2018). New evidence on the effect of computerized individualized practice and instruction on language skills. J. Comput. Assist. Learn. 34, 440–449. doi: 10.1111/jcal.12248

Guskey, T. R., and Pigott, T. D. (1988). Research on group-based mastery learning programs: a meta-analysis. J. Educ. Res. 81, 197–216. doi: 10.1080/00220671.1988.10885824

Hall, E. F. (1992). Assessment for differentiation. Br. J. Spec. Educ. 19, 20–23. doi: 10.1111/j.1467-8578.1992.tb00397.x

Hancock, G. R., and Mueller, R. O. (2010). The Reviewer's Guide to Quantitative Methods in the Social Sciences . New York, NY: Routledge.

Hattie, J. (2009). Visible Learning. A Synthesis of Over 800 Meta-Analyses Relating to Achievement. Oxon: Routledge.

Hertberg-Davis, H., and Brighton, C. M. (2006). Support and sabotage: principals' influence on middle school teachers' responses to differentiation. J. Secondary Gifted Educ. 17, 90–102. doi: 10.4219/jsge-2006-685

Higgins, J.P.T., and Green, S., (eds). (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration . Available online at: www.handbook.cochrane.org .

* Huber, M. J., Workman, J., Ford, J. A., Moore, D., and Mayer, T. (2009). Evaluating the prevention through alternative learning styles program. J. Drug Educ. 39, 239–259. doi: 10.2190/DE.39.3.b

Imants, J., and Van Veen, K. (2010). “Teacher learning as workplace learning,” in International Encyclopedia of Education, 3rd Edn . eds P. Peterson, E. Baker, and B. McGaw (Oxford: Elsevier), 569–574. doi: 10.1016/B978-0-08-044894-7.00657-6

Janssen, F., Westbroek, H., and Doyle, W. (2015). Practicality studies: how to move from what works in principle to what works in practice. J. Learn. Sci. 24, 176–186. doi: 10.1080/10508406.2014.954751

Keuning, T., Van Geel, M., Frèrejean, J., Van Merriënboer, J., Dolmans, D., and Visscher, A. (2017). Differentiëren bij rekenen: Een cognitieve taakanalyse van het denken en handelen van basisschoolleerkrachten [Differentiating in mathematics: a cognitive task analysis of primary school teachers' reflections and practices]. Pedagog. Stud. 94, 160–181. Available online at: http://pedagogischestudien.nl/download?type=document&identifier=640319

Kiley, D. (2011). Differentiated Instruction in the Secondary Classroom: Analysis of the Level of Implementation and Factors that Influence Practice (Partial FULFILLMENT of the Requirements for the Degree of Doctor of Education) . Kalamazoo: Western Michigan University.

Kingston, N., and Nash, B. (2011). Formative assessment: a meta-analysis and a call for research. Educ. Meas. Issues Pract. 30, 28–37. doi: 10.1111/j.1745-3992.2011.00220.x

Kirschner, P. A., Claessens, L., and Raaijmakers, S. (2018). Op de Schouders van Reuzen. Inspirerende Inzichten uit de Cognitieve Psychologie voor Leerkrachten. [On the Shoulders of Giants. Inspiring Insights from Cognitive Psychology for Teachers] . Meppel: Drukkerij Ten Brink Uitgevers.

Kulik, C. C., and Kulik, J. A. (1982). Effects of ability grouping on secondary school students: a meta-analysis of evaluation findings. Am. Educ. Res. J. 19, 415–428. doi: 10.3102/00028312019003415

Kulik, C. C., Kulik, J. A., and Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs: a meta-analysis. Rev. Educ. Res. 60, 265–299. doi: 10.3102/00346543060002265

Kulik, J. A. (1992). An Analysis of the Research on Ability Grouping: Historical and Contemporary Perspectives. Research-Based Decision Making Series. National Research Center on the Gifted and Talented . Available online at: http://search.ebscohost.com.proxy-ub.rug.nl/login.aspx?direct=trueanddb=ericandAN=ED350777andsite=ehost-liveandscope=site

Kulik, J. A., and Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Rev. Educ. Res. 86, 42–78. doi: 10.3102/0034654315581420

Kumar, R., O'malley, P. M., Johnston, L. D., and Laetz, V. B. (2013). Alcohol, tobacco, and other drug use prevention programs in U.S. schools: a descriptive summary. Prev. Sci. 14, 581–592. doi: 10.1007/s11121-012-0340-z

Kyriakides, L., Creemers, B., and Charalambous, E. (2018). Equity and Quality Dimensions in Educational Effectiveness. Dordrecht: Springer International Publishing. doi: 10.1007/978-3-319-72066-1

Latz, A. O., Speirs Neumeister, K. L., Adams, C. M., and Pierce, R. L. (2009). Peer coaching to improve classroom differentiation: perspectives from project CLUE. Roeper Rev. 31, 27–39. doi: 10.1080/02783190802527356

Leo, J., and Puzio, K. (2016). Flipped instruction in a high school science classroom. J. Sci. Educ. Technol. 25, 775–781. doi: 10.1007/s10956-016-9634-4

* Little, C. A., McCoach, D. B., and Reis, S. M. (2014). Effects of differentiated reading instruction on student achievement in middle school. J. Adv. Acad. 25, 384–402. doi: 10.1177/1932202X14549250

Lou, Y., Abrami, P. C., Spence, J. C., Poulsen, C., Chambers, B., and d'Apollonia, S. (1996). Within-class grouping: a meta-analysis. Rev. Educ. Res. 66, 423–458. doi: 10.3102/00346543066004423

Lyons, L. C. (2003). Meta-Analysis: Methods of Accumulating Results Across Research Domains . Available online at: http://www.lyonsmorris.com/lyons/metaAnalysis/index.cfm

Ma, W., Adesope, O. O., Nesbit, J. C., and Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. J. Educ. Psychol. 106, 901–918. doi: 10.1037/a0037123

Martin, A. J., and Lazendic, G. (2018). Computer-adaptive testing: implications for students' achievement, motivation, engagement, and subjective test experience. J. Educ. Psychol. 110, 27–45. doi: 10.1037/edu0000205

* Mastropieri, M. A., Scruggs, T. E., Norland, J. J., Berkeley, S., McDuffie, K., Tornquist, E. H., et al. (2006). Differentiated curriculum enhancement in inclusive middle school science: effects on classroom and high-stakes tests. J. Spec. Educ. 40, 130–137. doi: 10.1177/00224669060400030101

Maulana, R., Helms-Lorenz, M., and Van de Grift, W. J. C. M. (2015). Development and evaluation of a questionnaire measuring pre-service teachers' teaching behaviour: a rasch modelling approach. Sch. Effect. Sch. Improv. 26, 169–194. doi: 10.1080/09243453.2014.939198

Maulana, R., Helms-Lorenz, M., and Van de Grift, W. J. C. M. (2017). Validating a model of effective teaching behaviour of pre-service teachers. Teach. Teach. Theor. Pract. 23, 471–493. doi: 10.1080/13540602.2016.1211102

McQuarrie, L., McRae, P., and Stack-Cutler, H. (2008). Differentiated Instruction Provincial Research Review. Edmonton, AB: Alberta Initiative for School Improvement.

Mevarech, Z. R., and Kramarski, B. (1997). IMPROVE: a multidimensional method for teaching mathematics in heterogeneous classrooms. Am. Educ. Res. J. 34, 365–394. doi: 10.3102/00028312034002365

Mills, M., Monk, S., Keddie, A., Renshaw, P., Christie, P., Geelan, D., et al. (2014). Differentiated learning: from policy to classroom. Oxford Rev. Educ. 40, 331–348. doi: 10.1080/03054985.2014.911725

* Mitee, T. L., and Obaitan, G. N. (2015). Effect of mastery learning on senior secondary school students' cognitive learning outcome in quantitative chemistry. J. Educ. Pract. 6, 34–38. Available online at: https://files.eric.ed.gov/fulltext/EJ1083639.pdf

Moeyaert, M., Ugille, M., Beretvas, N., Ferron, J., Bunuan, R., and Van den Noortgate, W. (2016). Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. Int. J. Soc. Res. Methodol. 20, 559–572. doi: 10.1080/13645579.2016.1252189

Nokes-Malach, T., Richey, J., and Gadgil, S. (2015). When is it better to learn together? Insights from research on collaborative learning. Educ. Psychol. Rev. 27, 645–656. doi: 10.1007/s10648-015-9312-8

Oakes, J. (2008). Keeping track: structuring equality and inequality in an era of accountability. Teach. College Rec. 110, 700–712. Available online at: https://www.tcrecord.org/Content.asp?ContentId=14610

OECD (2012). Equity and Quality in Education. Supporting Disadvantaged Students and Schools. Paris: OECD Publishing. doi: 10.1787/9789264130852-en

OECD (2018). The Resilience of Students with an Immigrant Background. Factors that Shape Well-being. Paris: OECD Publishing. doi: 10.1787/9789264292093-en

Parsons, S. A., Dodman, S. L., and Cohen Burrowbridge, S. (2013). Broadening the view of differentiated instruction differentiation shouldn't end with planning but should continue as teachers adapt their instruction during lessons. Kappan 95, 38–42. doi: 10.1177/003172171309500107

Parsons, S. A., Vaughn, M., Scales, R. Q., Gallagher, M. A., Parsons, A. W., Davis, S. G., et al. (2018). Teachers' instructional adaptations: a research synthesis. Rev. Educ. Res. 88, 205–242. doi: 10.3102/0034654317743198

Petticrew, M., and Roberts, H. (2006). Systematic Reviews in the Social Sciences. A Practical Guide. Malden, MA: USA Blackwell publishing. doi: 10.1002/9780470754887

Pierce, R., and Adams, C. (2005). Using tiered lessons in mathematics. Math. Teach. Middle Sch. 11, 144–149.

Prast, E. J., Van de Weijer-Bergsma, E., Kroesbergen, E. H., Van Luit, and Johannes, E. H. (2015). Readiness-based differentiation in primary school mathematics: expert recommendations and teacher self-assessment. Frontline Learn. Res. 3, 90–116. doi: 10.14786/flr.v3i2.163

* Richards, M. R. E., and Omdal, S. N. (2007). Effects of tiered instruction on academic achievement in a secondary science course. J. Adv. Acad. 18, 424–453. doi: 10.4219/jaa-2007-499

Rock, M. L., Gregg, M., Ellis, E., and Gable, R. A. (2008). REACH: a framework for differentiating classroom instruction. Prev. Sch. Fail. 52, 31–47. doi: 10.3200/PSFL.52.2.31-47

Roy, A., Guay, F., and Valois, P. (2013). Teaching to address diverse learning needs: development and validation of a differentiated instruction scale. Int. J. Inclus. Educ. 17, 1186–1204. doi: 10.1080/13603116.2012.743604

Scheerens, J. (2016). “Meta-analyses of school and instructional effectiveness,” in Educational Effectiveness and Ineffectiveness , ed J. Scheerens (Dordrecht: Springer Science + Business Media), 175–223. doi: 10.1007/978-94-017-7459-8_8

Schipper, T., Goei, S. L., de Vries, S., and van Veen, K. (2017). Professional growth in adaptive teaching competence as a result of lesson study. Teach. Teach. Educ. 68, 289–303. doi: 10.1016/j.tate.2017.09.015

Schipper, T., Goei, S. L., de Vries, S., and van Veen, K. (2018). Developing teachers' self-efficacy and adaptive teaching behaviour through lesson study. International Journal of Educational Research , 88, 109–120. doi: 10.1016/j.ijer.2018.01.011

Schleicher, A. (2016). Teaching Excellence Through Professional Learning and Policy Reform: Lessons from Around the World . Paris: International Summit on the Teaching Profession; OECD Publishing. doi: 10.1787/9789264252059-en

Schofield, J. W. (2010). International evidence on ability grouping with curriculum differentiation and the achievement gap in secondary schools. Teach. College Rec. 112, 1492–1528. Available online at: https://www.tcrecord.org/Content.asp?ContentId=15684

Schütz, G., Ursprung, H., and Wößmann, L. (2008). Education Policy and Equality of Opportunity, Vol. 61 (Kyklos: Wiley Blackwell), 279–308.

Sezer, B. (2017). The effectiveness of a technology-enhanced flipped science classroom. J. Educ. Comput. Res. 55, 471–494. doi: 10.1177/0735633116671325

Shute, V. J., and Rahimi, S. (2017). Review of computer-based assessment for learning in elementary and secondary education. J. Comput. Assist. Learn. 33, 1–19. doi: 10.1111/jcal.12172

Sipe, T. A., and Curlette, W. L. (1996). A meta-synthesis of factors related to educational achievement: a methodological approach to summarizing and synthesizing meta-analyses. Int. J. Educ. Res. 25, 83–698. doi: 10.1016/S0883-0355(96)80001-2

Slavin, R., and Smith, D. (2009). The relationship between sample sizes and effect sizes in systematic reviews in education. Educ. Eval. Policy Anal. 31, 500–506. doi: 10.3102/0162373709352369

Slavin, R. E. (1986). Best-evidence synthesis: an alternative to meta-analytic and traditional reviews. Educ. Res. 15, 5–11. doi: 10.3102/0013189X015009005

Slavin, R. E. (1987). Mastery learning reconsidered. Rev. Educ. Res. 57, 175–214. doi: 10.3102/00346543057002175

Slavin, R. E. (1990a). Achievement effects of ability grouping in secondary schools: a best-evidence synthesis. Rev. Educ. Res. 60, 471–499. doi: 10.3102/00346543060003471

Slavin, R. E. (1990b). Mastery learning re-reconsidered. Rev. Educ. Res. 60, 300–302. doi: 10.3102/00346543060002300

Slavin, R. E. (1995). Best evidence synthesis. An intelligent alternative to meta-analysis. J. Clin. Epidemiol. 48, 9–18. doi: 10.1016/0895-4356(94)00097-A

Slavin, R. E. (2013). Effective programmes in reading and mathematics: lessons from the Best Evidence Encyclopaedia. Sch. Effect. Sch. Improv. 24, 383–391. doi: 10.1080/09243453.2013.797913

Slavin, R. E., and Cheung, A. (2005). A synthesis of research on language of reading instruction for English language learners. Rev. Educ. Res. 75, 247–284. doi: 10.3102/00346543075002247

Slavin, R. E., Cheung, A., Groff, C., and Lake, C. (2008). Effective reading programs for middle and high schools: a best-evidence synthesis. Read. Res. Q. 43, 290–322. doi: 10.1598/RRQ.43.3.4

Slavin, R. E., Lake, C., and Groff, C. (2009). Effective programs in middle and high school mathematics: a best-evidence synthesis. Rev. Educ. Res. 79, 839–911. doi: 10.3102/0034654308330968

Slavin, R. E., and Madden, N. A. (2011). Measures inherent to treatments in program effectiveness reviews. J. Res. Educ. Effect. 4, 370–380. doi: 10.1080/19345747.2011.558986

Smeets, E., and Mooij, T. (2001). Pupil-centred learning, ICT, and teacher behaviour: observations in educational practice. Br. J. Educ. Technol. 32, 403. doi: 10.1111/1467-8535.00210

Smets, W., and Struyven, K. (2018). Realist review of literature on catering for different instructional needs with preteaching and extended instruction. Educ. Sci . 8, 113. doi: 10.3390/educsci8030113

* Smit, R., and Humpert, W. (2012). Differentiated instruction in small schools. Teach. Teach. Educ. 28, 1152–1162. doi: 10.1016/j.tate.2012.07.003

Smit, R., Humpert, W., Obertüfer-Gahler, R., Engeli, E., and Breuer-Brodmüller, M. (2011). “Differenzierung als Chance für kleine schulen - empirische befunde im längsschnitt,” in Schule im Alpinen Raum , eds R. Müller, A. Keller, U. Kerle, A. Raggl, and E. Steiner (Innsbruck: Studienverlag), 435–488.

Steenbergen-Hu, S., Makel, M. C., and Olszewski-Kubilius, P. (2016). What one hundred years of research says about the effects of ability grouping and acceleration on K−12 students' academic achievement: findings of two second-order meta-analyses. Rev. Educ. Res. 86, 849–899. doi: 10.3102/0034654316675417

Stollman, S. H. M. (2018). Differentiated Instruction in Practice: A Teacher Perspective . Leiden: ICLON, Leiden University Graduate School of Teaching.

Subban, P. (2006). Differentiated instruction: a research basis. Int. Educ. J. 7, 935–947. Available online at: http://ehlt.flinders.edu.au/education/iej/articles/v7n7/Subban/BEGIN.HTM

Suprayogi, M. N., Valcke, M., and Godwin, R. (2017). Teachers and their implementation of differentiated instruction in the classroom. Teach. Teach. Educ. 67, 291–301. doi: 10.1016/j.tate.2017.06.020

Swanson, H. L., and Lussier, C. M. (2001). A selective synthesis of the experimental literature on dynamic assessment. Rev. Educ. Res. 71, 321–363. doi: 10.3102/00346543071002321

Tieso, C. L. (2003). Ability grouping is not just tracking anymore. Roeper Rev. 26, 29–36. doi: 10.1080/02783190309554236

Tomlinson, C. (2015). Teaching for excellence in academically diverse classrooms. Society 52, 203–209. doi: 10.1007/s12115-015-9888-0

Tomlinson, C. A. (1995). Deciding to differentiate instruction in middle school: one school's journey. Gifted Child Q. 39, 77–87. doi: 10.1177/001698629503900204

Tomlinson, C. A. (1999). Mapping a route toward differentiated instruction. Pers. Learn. 57, 12–16.

Tomlinson, C. A. (2014). The Differentiated Classroom. Responding to the Needs of All Learrners, 2nd Edn . Alexandria, VA: ASCD.

Tomlinson, C. A., Brighton, C., Hertberg, H., Callahan, C. M., Moon, T. R., Brimijoin, K., et al. (2003). Differentiating instruction in response to student readiness, interest, and learning profile in academically diverse classrooms: a review of literature. J. Educ. Gifted 27, 119–145. doi: 10.1177/016235320302700203

Unesco (2017). A Guide for Ensuring Inclusion and Equity in Education. Paris: United Nations Educational, Scientific and Cultural Organization . Available online at: https://unesdoc.unesco.org/ark:/48223/pf0000248254

Valiande, S., and Koutselini, M. I. (2009). (2009). “Application and evaluation of differentiation instruction in mixed ability classrooms,” Paper presented at the 4th Hellenic Observatory PhD Symposium (London: LSE, 25–26.

Valiandes, S., and Neophytou, L. (2018). Teachers' professional development for differentiated instruction in mixed-ability classrooms: investigating the impact of a development program on teachers' professional learning and on students' achievement. Teach. Dev. 22, 123–138. doi: 10.1080/13664530.2017.1338196

Van Casteren, W., Bendig-Jacobs, J., Wartenbergh-Cras, F., Van Essen, M., and Kurver, B. (2017). Differentiëren en Differentiatievaardigheden in Het Voortgezet Onderwijs. Nijmegen: ResearchNed.

Van de Grift, W. J. C. M. (2007). Quality of teaching in four European countries: A review of the literature and application of an assessment instrument. Educ. Res. 49, 127–152. doi: 10.1080/00131880701369651

Van de Grift, W. J. C. M., Helms-Lorenz, M., and Maulana, R. (2014). Teaching skills of student teachers: calibration of an evaluation instrument and its value in predicting student academic engagement. Stud. Educ. Eval. 43, 150–159. doi: 10.1016/j.stueduc.2014.09.003

Van de Pol, J., Volman, M., and Beishuizen, J. (2010). Scaffolding in Teacher–Student interaction: a decade of research. Educ. Psychol. Rev. 22, 271–296. doi: 10.1007/s10648-010-9127-6

Van de Pol, J., Volman, M., Oort, F., and Beishuizen, J. (2015). The effects of scaffolding in the classroom: support contingency and student independent working time in relation to student achievement, task effort and appreciation of support. Instruct. Sci. 43, 615–641. doi: 10.1007/s11251-015-9351-z

Van der Kleij, F., Feskens, R. C. W., and Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning environment on students' learning outcomes. Rev. Educ. Res. 85, 475–511. doi: 10.3102/0034654314564881

Van der Lans, R. M., Van de Grift, W. J. C. M., and van Veen, K. (2017). Individual differences in teacher development: an exploration of the applicability of a stage model to assess individual teachers. Learn. Individ. Diff. 58, 46–55. doi: 10.1016/j.lindif.2017.07.007

Van der Lans, R. M., Van de Grift, W. J. C. M., and van Veen, K. (2018). Developing an instrument for teacher feedback: using the rasch model to explore teachers' development of effective teaching strategies and behaviors. J. Exp. Educ. 86, 247–264. doi: 10.1080/00220973.2016.1268086

Van Geel, M., Keuning, T., Frèrejean, J., Dolmans, D., Van Merriënboer, J., and Visscher, A. J. (2019). Capturing the complexity of differentiated instruction. Sch. Effect. Sch. Improv. 30, 51–67. doi: 10.1080/09243453.2018.1539013

Van Halem, N., Van Klaveren, C. P. B. J., and Cornelisz, I. (2017). Oefent een leerling meer door niveaudifferentiatie? Het effect van data-gestuurde differentiatie op leerinspanning en de rol van eerder behaalde cijfers. [Does a learner practice more because of readiness-based differentiation? The effect of data-driven differentiation on learning effort and the role of prior grades]. Pedagog. Stud. 94, 182–195. Available online at: http://pedagogischestudien.nl/download?type=document&identifier=640298

Van Klaveren, C., Vonk, S., and Cornelisz, I. (2017). The effect of adaptive versus static practicing on student learning - evidence from a randomized field experiment. Econ. Educ. Rev. 58, 175–187. doi: 10.1016/j.econedurev.2017.04.003

Van Tassel-Baska, J., Quek, C., and Feng, A. X. (2006). The development and use of a structured teacher observation scale to assess differentiated best practice. Roeper Rev. 29, 84–92. doi: 10.1080/02783190709554391

* Vogt, F., and Rogalla, M. (2009). Developing adaptive teaching competency through coaching. Teach. Teach. Educ. 25, 1051–1060. doi: 10.1016/j.tate.2009.04.002

Walkington, C. A. (2013). Using adaptive learning technologies to personalize instruction to student interests: the impact of relevant contexts on performance and learning outcomes. J. Educ. Psychol. 105, 932–945. doi: 10.1037/a0031882

* Wambugu, P. W., and Changeiywo, J. M. (2008). Effects of mastery learning approach on secondary school students' physics achievement. EURASIA J. Math. Sci. Technol. Educ. 4, 293–302. doi: 10.12973/ejmste/75352

Wang, M. C., Haertel, G. D., and Walberg, H. J. (1990). What influences learning? A content analysis of review literature. J. Educ. Res. 84, 30–43. doi: 10.1080/00220671.1990.10885988

Wilkinson, S. D., and Penney, D. (2014). The effects of setting on classroom teaching and student learning in mainstream mathematics, English and science lessons: a critical review of the literature in England. Educ. Rev. 66, 411–427. doi: 10.1080/00131911.2013.787971

World Health Organiasation (2011). European Action Plan to Reduce the Harmful Use of Alcohol 2012–2020. Copenhagen: World Health Organization Regional Office for Europe . Available online at: https://www.stap.nl/en/home/european-alcohol-policy.html

Keywords: review, differentiation, differentiated instruction, adaptive teaching, ability grouping, secondary education, student performance, effectiveness

Citation: Smale-Jacobse AE, Meijer A, Helms-Lorenz M and Maulana R (2019) Differentiated Instruction in Secondary Education: A Systematic Review of Research Evidence. Front. Psychol. 10:2366. doi: 10.3389/fpsyg.2019.02366

Received: 14 May 2019; Accepted: 04 October 2019; Published: 22 November 2019.

Reviewed by:

Copyright © 2019 Smale-Jacobse, Meijer, Helms-Lorenz and Maulana. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Annemieke E. Smale-Jacobse, a.e.smale-jacobse@rug.nl

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Original article
• Open access
• Published: 09 April 2020

Why does peer instruction benefit student learning?

• Jonathan G. Tullis 1 &
• Robert L. Goldstone 2  

Cognitive Research: Principles and Implications volume  5 , Article number:  15 ( 2020 ) Cite this article

83k Accesses

42 Citations

52 Altmetric

Metrics details

In peer instruction, instructors pose a challenging question to students, students answer the question individually, students work with a partner in the class to discuss their answers, and finally students answer the question again. A large body of evidence shows that peer instruction benefits student learning. To determine the mechanism for these benefits, we collected semester-long data from six classes, involving a total of 208 undergraduate students being asked a total of 86 different questions related to their course content. For each question, students chose their answer individually, reported their confidence, discussed their answers with their partner, and then indicated their possibly revised answer and confidence again. Overall, students were more accurate and confident after discussion than before. Initially correct students were more likely to keep their answers than initially incorrect students, and this tendency was partially but not completely attributable to differences in confidence. We discuss the benefits of peer instruction in terms of differences in the coherence of explanations, social learning, and the contextual factors that influence confidence and accuracy.

Significance

Peer instruction is widely used in physics instruction across many universities. Here, we examine how peer instruction, or discussing one’s answer with a peer, affects students’ decisions about a class assignment. Across six different university classes, students answered a question, discussed their answer with a peer, and finally answered the question again. Students’ accuracy consistently improved through discussion with a peer. Our peer instruction data show that students were hesitant to switch away from their initial answer and that students did consider both their own confidence and their partner’s confidence when making their final decision, in accord with basic research about confidence in decision making. More broadly, the data reveal that peer discussion helped students select the correct answer by prompting them to create new knowledge. The benefit to student accuracy that arises when students discuss their answers with a partner is a “process gain”, in which working in a group yields better performance than can be predicted from individuals’ performance alone.

Peer instruction is specific evidence-based instructional strategy that is well-known and widely used, particularly in physics (Henderson & Dancy, 2009 ). In fact, peer instruction has been advocated as a part of best methods in science classrooms (Beatty, Gerace, Leonard, & Dufresne, 2006 ; Caldwell, 2007 ; Crouch & Mazur, 2001 ; Newbury & Heiner, 2012 ; Wieman et al., 2009 ) and over a quarter of university physics professors report using peer instruction (Henderson & Dancy, 2009 ). In peer instruction, instructors pose a challenging question to students, students answer the question individually, students discuss their answers with a peer in the class, and finally students answer the question again. There are variations of peer instruction in which instructors show the class’s distribution of answers before discussion (Nielsen, Hansen-Nygård, & Stav, 2012 ; Perez et al., 2010 ), in which students’ answers are graded for participation or for correctness (James, 2006 ), and in which instructors’ norms affect whether peer instruction offers opportunities for answer-seeking or for sense-making (Turpen & Finkelstein, 2007 ).

Despite wide variations in its implementation, peer instruction consistently benefits student learning. Switching classroom structure from didactic lectures to one centered around peer instruction improves learners’ conceptual understanding (Duncan, 2005 ; Mazur, 1997 ), reduces student attrition in difficult courses (Lasry, Mazur, & Watkins, 2008 ), decreases failure rates (Porter, Bailey-Lee, & Simon, 2013 ), improves student attendance (Deslauriers, Schelew, & Wieman, 2011 ), and bolsters student engagement (Lucas, 2009 ) and attitudes to their course (Beekes, 2006 ). Benefits of peer instruction have been found across many fields, including physics (Mazur, 1997 ; Pollock, Chasteen, Dubson, & Perkins, 2010 ), biology (Knight, Wise, & Southard, 2013 ; Smith, Wood, Krauter, & Knight, 2011 ), chemistry (Brooks & Koretsky, 2011 ), physiology (Cortright, Collins, & DiCarlo, 2005 ; Rao & DiCarlo, 2000 ), calculus (Lucas, 2009 ; Miller, Santana-Vega, & Terrell, 2007 ), computer science (Porter et al., 2013 ), entomology (Jones, Antonenko, & Greenwood, 2012 ), and even philosophy (Butchart, Handfield, & Restall, 2009 ). Additionally, benefits of peer instruction have been found at prestigious private universities, two-year community colleges (Lasry et al., 2008 ), and even high schools (Cummings & Roberts, 2008 ). Peer instruction benefits not just the specific questions posed during discussion, but also improves accuracy on later similar problems (e.g., Smith et al., 2009 ).

One of the consistent empirical hallmarks of peer instruction is that students’ answers are more frequently correct following discussion than preceding it. For example, in introductory computer science courses, post-discussion performance was higher on 70 out of 71 questions throughout the semester (Simon, Kohanfars, Lee, Tamayo, & Cutts, 2010 ). Further, gains in performance from discussion are found on many different types of questions, including recall, application, and synthesis questions (Rao & DiCarlo, 2000 ). Performance improvements are found because students are more likely to switch from an incorrect answer to the correct answer than from the correct answer to an incorrect answer. In physics, 59% of incorrect answers switched to correct following discussion, but only 13% of correct answers switched to incorrect (Crouch & Mazur, 2001 ). Other research on peer instruction shows the same patterns: 41% of incorrect answers are switched to correct ones, while only 18% of correct answers are switched to incorrect (Morgan & Wakefield, 2012 ). On qualitative problem-solving questions in physiology, 57% of incorrect answers switched to correct after discussion, and only 7% of correct answers to incorrect (Giuliodori, Lujan, & DiCarlo, 2006 ).

There are two explanations for improvements in pre-discussion to post-discussion accuracy. First, switches from incorrect to correct answers may be driven by selecting the answer from the peer who is more confident. When students discuss answers that disagree, they may choose whichever answer belongs to the more confident peer. Evidence about decision-making and advice-taking substantiates this account. First, confidence is correlated with correctness across many settings and procedures (Finley, Tullis, & Benjamin, 2010 ). Students who are more confident in their answers are typically more likely to be correct. Second, research examining decision-making and advice-taking indicates that (1) the less confident you are, the more you value others’ opinions (Granovskiy, Gold, Sumpter, & Goldstone, 2015 ; Harvey & Fischer, 1997 ; Yaniv, 2004a , 2004b ; Yaniv & Choshen-Hillel, 2012 ) and (2) the more confident the advisor is, the more strongly they influence your decision (Kuhn & Sniezek, 1996 ; Price & Stone, 2004 ; Sah, Moore, & MacCoun, 2013 ; Sniezek & Buckley, 1995 ; Van Swol & Sniezek, 2005 ; Yaniv, 2004b ). Consequently, if students simply choose their final answer based upon whoever is more confident, accuracy should increase from pre-discussion to post-discussion. This explanation suggests that switches in answers should be driven entirely by a combination of one’s own initial confidence and one’s partner’s confidence. In accord with this confidence view, Koriat ( 2015 ) shows that an individual’s confidence typically reflects the group’s most typically given answer. When the answer most often given by group members is incorrect, peer interactions amplify the selection of and confidence in incorrect answers. Correct answers have no special draw. Rather, peer instruction merely amplifies the dominant view through differences in the individual’s confidence.

In a second explanation, working with others may prompt students to verbalize explanations and verbalizations may generate new knowledge. More specifically, as students discuss the questions, they need to create a common representation of the problem and answer. Generating a common representation may compel students to identify gaps in their existing knowledge and construct new knowledge (Schwartz, 1995 ). Further, peer discussion may promote students’ metacognitive processes of detecting and correcting errors in their mental models. Students create more new knowledge and better diagnostic tests of answers together than alone. Ultimately, then, the new knowledge and improved metacognition may make the correct answer appear more compelling or coherent than incorrect options. Peer discussion would draw attention to coherent or compelling answers, more so than students’ initial confidence alone and the coherence of the correct answer would prompt students to switch away from incorrect answers. Similarly, Trouche, Sander, and Mercier ( 2014 ) argue that interactions in a group prompt argumentation and discussion of reasoning. Good arguments and reasoning should be more compelling to change individuals’ answers than confidence alone. Indeed, in a reasoning task known to benefit from careful deliberation, good arguments and the correctness of the answers change partners’ minds more than confidence in one’s answer (Trouche et al., 2014 ). This explanation predicts several distinct patterns of data. First, as seen in prior research, more students should switch from incorrect answers to correct than vice versa. Second, the intrinsic coherence of the correct answer should attract students, so the likelihood of switching answers would be predicted by the correctness of an answer above and beyond differences in initial confidence. Third, initial confidence in an answer should not be as tightly related to initial accuracy as final confidence is to final accuracy because peer discussion should provide a strong test of the coherence of students’ answers. Fourth, because the coherence of an answer is revealed through peer discussion, student confidence should increase more from pre-discussion to post-discussion when they agree on the correct answers compared to agreeing on incorrect answers.

Here, we examined the predictions of these two explanations of peer instruction across six different classes. We specifically examined whether changes in answers are driven exclusively through the confidence of the peers during discussion or whether the coherence of an answer is better constructed and revealed through peer instruction than on one’s own. We are interested in analyzing cognitive processes at work in a specific, but common, implementation of classroom-based peer instruction; we do not intend to make general claims about all kinds of peer instruction or to evaluate the long-term effectiveness of peer instruction. This research is the first to analyze how confidence in one’s answer relates to answer-switching during peer instruction and tests the impact of peer instruction in new domains (i.e., psychology and educational psychology classes).

Participants

Students in six different classes participated as part of their normal class procedures. More details about these classes are presented in Table  1 . The authors served as instructors for these classes. Across the six classes, 208 students contributed a total of 1657 full responses to 86 different questions.

The instructors of the courses developed multiple-choice questions related to the ongoing course content. Questions were aimed at testing students’ conceptual understanding, rather than factual knowledge. Consequently, questions often tested whether students could apply ideas to new settings or contexts. An example of a cognitive psychology question used is: Which is a fixed action pattern (not a reflex)?

Knee jerks up when patella is hit

Male bowerbirds building elaborate nests [correct]

Eye blinks when air is blown on it

Can play well learned song on guitar even when in conversation

The procedures for peer instruction across the six different classes followed similar patterns. Students were presented with a multiple-choice question. First, students read the question on their own, chose their answer, and reported their confidence in their answer on a scale of 1 “Not at all confident” to 10 “Highly confident”. Students then paired up with a neighbor in their class and discussed the question with their peer. After discussion, students answered the question and reported the confidence for a second time. The course instructor indicated the correct answer and discussed the reasoning for the answer after all final answers had been submitted. Instruction was paced based upon how quickly students read and answered questions. Most student responses counted towards their participation grade, regardless of the correctness of their answer (the last question in each of the cognitive psychology classes was graded for correctness).

There were small differences in procedures between classes. Students in the cognitive psychology classes input their responses using classroom clickers, but those in other classes wrote their responses on paper. Further, students in the cognitive psychology classes explicitly reported their partner’s answer and confidence, while students in other classes only reported the name of their partner (the partners’ data were aligned during data recording). The cognitive psychology students then were required to mention their own answer and their confidence to their partner during peer instruction; students in other classes were not required to tell their answer or their confidence to their peer. Finally, the questions appeared at any point during the class period for the cognitive psychology classes, while the questions typically happened at the beginning of each class for the other classes.

Analytic strategy

Data are available on the OpenScienceFramework: https://mfr.osf.io/render?url=https://osf.io/5qc46/?action=download%26mode=render .

For most of our analyses we used linear mixed-effects models (Baayen, Davidson, & Bates, 2008 ; Murayama, Sakaki, Yan, & Smith, 2014 ). The unit of analysis in a mixed-effect model is the outcome of a single trial (e.g., whether or not a particular question was answered correctly by a particular participant). We modeled these individual trial-level outcomes as a function of multiple fixed effects - those of theoretical interest - and multiple random effects - effects for which the observed levels are sampled out of a larger population (e.g., questions, students, and classes sampled out of a population of potential questions, students, and classes).

Linear mixed-effects models solve four statistical problems involved with the data of peer instruction. First, there is large variability in students’ performance and the difficulty of questions across students and classes. Mixed-effect models simultaneously account for random variation both across participants and across items (Baayen et al., 2008 ; Murayama et al., 2014 ). Second, students may miss individual classes and therefore may not provide data across every item. Similarly, classes varied in how many peer instruction questions were posed throughout the semester and the number of students enrolled. Mixed-effects models weight each response equally when drawing conclusions (rather than weighting each student or question equally) and can easily accommodate missing data. Third, we were interested in how several different characteristics influenced students’ performance. Mixed effects models can include multiple predictors simultaneously, which allows us to test the effect of one predictor while controlling for others. Finally, mixed effects models can predict the log odds (or logit) of a correct answer, which is needed when examining binary outcomes (i.e., correct or incorrect; Jaeger, 2008 ).

We fit all models in R using the lmer() function of the lme4 package (Bates, Maechler, Bolker, & Walker, 2015 ). For each mixed-effect model, we included random intercepts that capture baseline differences in difficulty of questions, in classes, and in students, in addition to multiple fixed effects of theoretical interest. In mixed-effect models with hundreds of observations, the t distribution effectively converges to the normal, so we compared the t statistic to the normal distribution for analyses involving continuous outcomes (i.e., confidence; Baayen, 2008 ). P values can be directly obtained from Wald z statistics for models with binary outcomes (i.e., correctness).

Does accuracy change through discussion?

First, we examined how correctness changed across peer discussion. A logit model predicting correctness from time point (pre-discussion to post-discussion) revealed that the odds of correctness increased by 1.57 times (95% confidence interval (conf) 1.31–1.87) from pre-discussion to post-discussion, as shown in Table  2 . In fact, 88% of students showed an increase or no change in accuracy from pre-discussion to post-discussion. Pre-discussion to post-discussion performance for each class is shown in Table  3 . We further examined how accuracy changed from pre-discussion to post-discussion for each question and the results are plotted in Fig.  1 . The data show a consistent improvement in accuracy from pre-discussion to post-discussion across all levels of initial difficulty.

The relationship between pre-discussion accuracy (x axis) and post-discussion accuracy (y axis). Each point represents a single question. The solid diagonal line represents equal pre-discussion and post-discussion accuracy; points above the line indicate improvements in accuracy and points below represent decrements in accuracy. The dashed line indicates the line of best fit for the observed data

We examined how performance increased from pre-discussion to post-discussion by tracing the correctness of answers through the discussion. Figure  2 tracks the percent (and number of items) correct from pre-discussion to post-discussion. The top row shows whether students were initially correct or incorrect in their answer; the middle row shows whether students agreed or disagreed with their partner; the last row show whether students were correct or incorrect after discussion. Additionally, Fig. 2 shows the confidence associated with each pathway. The bottow line of each entry shows the students’ average confidence; in the middle white row, the confidence reported is the average of the peer’s confidence.

The pathways of answers from pre-discussion (top row) to post-discussion (bottom row). Percentages indicate the portion of items from the category immediately above in that category, the numbers in brackets indicate the raw numbers of items, and the numbers at the bottom of each entry indicate the confidence associated with those items. In the middle, white row, confidence values show the peer’s confidence. Turquoise indicates incorrect answers and yellow indicates correct answers

Broadly, only 5% of correct answers were switched to incorrect, while 28% of incorrect answers were switched to correct following discussion. Even for the items in which students were initially correct but disagreed with their partner, only 21% of answers were changed to incorrect answers after discussion. However, out of the items where students were initially incorrect and disagreed with their partner, 42% were changed to the correct answer.

Does confidence predict switching?

Differences in the amount of switching to correct or incorrect answers could be driven solely by differences in confidence, as described in our first theory mentioned earlier. For this theory to hold, answers with greater confidence must have a greater likelihood of being correct. To examine whether initial confidence is associated with initial correctness, we calculated the gamma correlation between correctness and confidence in the answer before discussion, as shown in the first column of Table  4 . The average gamma correlation between initial confidence and initial correctness (mean (M) = 0.40) was greater than zero, t (160) = 8.59, p  < 0.001, d  = 0.68, indicating that greater confidence was associated with being correct.

Changing from an incorrect to a correct answer, then, may be driven entirely by selecting the answer from the peer with the greater confidence during discussion, even though most of the students in our sample were not required to explicitly disclose their confidence to their partner during discussion. We examined how frequently students choose the more confident answer when peers disagree. When peers disagreed, students’ final answers aligned with the more confident peer only 58% of the time. Similarly, we tested what the performance would be if peers always picked the answer of the more confident peer. If peers always chose the more confident answer during discussion, the final accuracy would be 69%, which is significantly lower than actual final accuracy (M = 72%, t (207) = 2.59, p  = 0.01, d  = 0.18). While initial confidence is related to accuracy, these results show that confidence is not the only predictor of switching answers.

Does correctness predict switching beyond confidence?

Discussion may reveal information about the correctness of answers by generating new knowledge and testing the coherence of each possible answer. To test whether the correctness of an answer added predictive power beyond the confidence of the peers involved in discussion, we analyzed situations in which students disagreed with their partner. Out of the instances when partners initially disagreed, we predicted the likelihood of keeping one’s answer based upon one’s own confidence, the partner’s confidence, and whether one’s answer was initially correct. The results of a model predicting whether students keep their answers is shown in Table  5 . For each increase in a point of one’s own confidence, the odds of keeping one’s answer increases 1.25 times (95% conf 1.13–1.38). For each decrease in a point of the partner’s confidence, the odds of keeping one’s answer increased 1.19 times (1.08–1.32). The beta weight for one’s confidence did not differ from the beta weight of the partner’s confidence, χ 2  = 0.49, p  = 0.48. Finally, if one’s own answer was correct, the odds of keeping one’s answer increased 4.48 times (2.92–6.89). In other words, the more confident students were, the more likely they were to keep their answer; the more confident their peer was, the more likely they were to change their answer; and finally, if a student was correct, they were more likely to keep their answer.

To illustrate this relationship, we plotted the probability of keeping one’s own answer as a function of the difference between one’s own and their partner’s confidence for initially correct and incorrect answers. As shown in Fig.  3 , at every confidence level, being correct led to equal or more frequently keeping one’s answer than being incorrect.

The probability of keeping one’s answer in situations where one’s partner initially disagreed as a function of the difference between partners’ levels of confidence. Error bars indicate the standard error of the proportion and are not shown when the data are based upon a single data point

As another measure of whether discussion allows learners to test the coherence of the correct answer, we analyzed how discussion impacted confidence when partners’ answers agreed. We predicted confidence in answers by the interaction of time point (i.e., pre-discussion versus post-discussion) and being initially correct for situations in which peers initially agreed on their answer. The results, displayed in Table  6 , show that confidence increased from pre-discussion to post-discussion by 1.08 points and that confidence was greater for initially correct answers (than incorrect answers) by 0.78 points. As the interaction between time point and initial correctness shows, confidence increased more from pre-discussion to post-discussion when students were initially correct (as compared to initially incorrect). To illustrate this relationship, we plotted pre-confidence against post-confidence for initially correct and initially incorrect answers when peers agreed (Fig.  4 ). Each plotted point represents a student; the diagonal blue line indicates no change between pre-confidence and post-confidence. The graph reflects that confidence increases more from pre-discussion to post-discussion for correct answers than for incorrect answers, even when we only consider cases where peers agreed.

The relationship between pre-discussion and post-discussion confidence as a function of the accuracy of an answer when partners agreed. Each dot represents a student

If students engage in more comprehensive answer testing during discussion than before, the relationship between confidence in their answer and the accuracy of their answer should be stronger following discussion than it is before. We examined whether confidence accurately reflected correctness before and after discussion. To do so, we calculated the gamma correlation between confidence and accuracy, as is typically reported in the literature on metacognitive monitoring (e.g., Son & Metcalfe, 2000 ; Tullis & Fraundorf, 2017 ). Across all students, the resolution of metacognitive monitoring increases from pre-discussion to post-discussion ( t (139) = 2.98, p  = 0.003, d  = 0.24; for a breakdown of gamma calculations for each class, see Table 4 ). Confidence was more accurately aligned with accuracy following discussion than preceding it. The resolution between student confidence and correctness increases through discussion, suggesting that discussion offers better coherence testing than answering alone.

To examine why peer instruction benefits student learning, we analyzed student answers and confidence before and after discussion across six psychology classes. Discussing a question with a partner improved accuracy across classes and grade levels with small to medium-sized effects. Questions of all difficulty levels benefited from peer discussion; even questions where less than half of students originally answered correctly saw improvements from discussion. Benefits across the spectrum of question difficulty align with prior research showing improvements when even very few students initially know the correct answer (Smith et al., 2009 ). More students switched from incorrect answers to correct answers than vice versa, leading to an improvement in accuracy following discussion. Answer switching was driven by a student’s own confidence in their answer and their partner’s confidence. Greater confidence in one’s answer indicated a greater likelihood of keeping the answer; a partner’s greater confidence increased the likelihood of changing to their answer.

Switching answers depended on more than just confidence: even when accounting for students’ confidence levels, the correctness of the answer impacted switching behavior. Across several measures, our data showed that the correctness of an answer carried weight beyond confidence. For example, the correctness of the answer predicted whether students switched their initial answer during peer disagreements, even after taking the confidence of both partners into account. Further, students’ confidence increased more when partners agreed on the correct answer compared to when they agreed on an incorrect answer. Finally, although confidence increased from pre-discussion to post-discussion when students changed their answers from incorrect to the correct ones, confidence decreased when students changed their answer away from the correct one. A plausible interpretation of this difference is that when students switch from a correct answer to an incorrect one, their decrease in confidence reflects the poor coherence of their final incorrect selection.

Whether peer instruction resulted in optimal switching behaviors is debatable. While accuracy improved through discussion, final accuracy was worse than if students had optimally switched their answers during discussion. If students had chosen the correct answer whenever one of the partners initially chose it, the final accuracy would have been significantly higher (M = 0.80 (SD = 0.19)) than in our data (M = 0.72 (SD = 0.24), t (207) = 6.49, p  < 0.001, d  = 0.45). While this might be interpreted as “process loss” (Steiner, 1972 ; Weldon & Bellinger, 1997 ), that would assume that there is sufficient information contained within the dyad to ascertain the correct answer. One individual selecting the correct answer is inadequate for this claim because they may not have a compelling justification for their answer. When we account for differences in initial confidence, students’ final accuracy was better than expected. Students’ final accuracy was better than that predicted from a model in which students always choose the answer of the more confident peer. This over-performance, often called “process gain”, can sometimes emerge when individuals collaborate to create or generate new knowledge (Laughlin, Bonner, & Miner, 2002 ; Michaelsen, Watson, & Black, 1989 ; Sniezek & Henry, 1989 ; Tindale & Sheffey, 2002 ). Final accuracy reveals that students did not simply choose the answer of the more confident student during discussion; instead, students more thoroughly probed the coherence of answers and mental models during discussion than they could do alone.

Students’ final accuracy emerges from the interaction between the pairs of students, rather than solely from individuals’ sequestered knowledge prior to discussion (e.g. Wegner, Giuliano, & Hertel, 1985 ). Schwartz ( 1995 ) details four specific cognitive products that can emerge through working in dyads. Specifically, dyads force verbalization of ideas through discussion, and this verbalization facilitates generating new knowledge. Students may not create a coherent explanation of their answer until they engage in discussion with a peer. When students create a verbal explanation of their answer to discuss with a peer, they can identify knowledge gaps and construct new knowledge to fill those gaps. Prior research examining the content of peer interactions during argumentation in upper-level biology classes has shown that these kinds of co-construction happen frequently; over three quarters of statements during discussion involve an exchange of claims and reasoning to support those claims (Knight et al., 2013 ). Second, dyads have more information processing resources than individuals, so they can solve more complex problems. Third, dyads may foster greater motivation than individuals. Finally, dyads may stimulate the creation of new, abstract representations of knowledge, above and beyond what one would expect from the level of abstraction created by individuals. Students need to communicate with their partner; to create common ground and facilitate discourse, dyads negotiate common representations to coordinate different perspectives. The common representations bridge multiple perspectives, so they lose idiosyncratic surface features of individuals’ representation. Working in pairs generates new knowledge and tests of answers that could not be predicted from individuals’ performance alone.

More broadly, teachers often put students in groups so that they can learn from each other by giving and receiving help, recognizing contradictions between their own and others’ perspectives, and constructing new understandings from divergent ideas (Bearison, Magzamen, & Filardo, 1986 ; Bossert, 1988-1989 ; Brown & Palincsar, 1989 ; Webb & Palincsar, 1996 ). Giving explanations to a peer may encourage explainers to clarify or reorganize information, recognize and rectify gaps in understandings, and build more elaborate interpretations of knowledge than they would have alone (Bargh & Schul, 1980 ; Benware & Deci, 1984 ; King, 1992 ; Yackel, Cobb, & Wood, 1991 ). Prompting students to explain why and how problems are solved facilitates conceptual learning more than reading the problem solutions twice without self-explanations (Chi, de Leeuw, Chiu, & LaVancher, 1994 ; Rittle-Johnson, 2006 ; Wong, Lawson, & Keeves, 2002 ). Self-explanations can prompt students to retrieve, integrate, and modify their knowledge with new knowledge; self-explanations can also help students identify gaps in their knowledge (Bielaczyc, Pirolli, & Brown, 1995 ; Chi & Bassock, 1989 ; Chi, Bassock, Lewis, Reimann, & Glaser, 1989 ; Renkl, Stark, Gruber, & Mandl, 1998 ; VanLehn, Jones, & Chi, 1992 ; Wong et al., 2002 ), detect and correct errors, and facilitate deeper understanding of conceptual knowledge (Aleven & Koedinger, 2002 ; Atkinson, Renkl, & Merrill, 2003 ; Chi & VanLehn, 2010 ; Graesser, McNamara, & VanLehn, 2005 ). Peer instruction, while leveraging these benefits of self-explanation, also goes beyond them by involving what might be called “other-explanation” processes - processes recruited not just when explaining a situation to oneself but to others. Mercier and Sperber ( 2019 ) argue that much of human reason is the result of generating explanations that will be convincing to other members of one’s community, thereby compelling others to act in the way that one wants.

Conversely, students receiving explanations can fill in gaps in their own understanding, correct misconceptions, and construct new, lasting knowledge. Fellow students may be particularly effective explainers because they can better take the perspective of their peer than the teacher (Priniski & Horne, 2019 ; Ryskin, Benjamin, Tullis, & Brown-Schmidt, 2015 ; Tullis, 2018 ). Peers may be better able than expert teachers to explain concepts in familiar terms and direct peers’ attention to the relevant features of questions that they do not understand (Brown & Palincsar, 1989 ; Noddings, 1985 ; Vedder, 1985 ; Vygotsky, 1981 ).

Peer instruction may benefit from the generation of explanations, but social influences may compound those benefits. Social interactions may help students monitor and regulate their cognition better than self-explanations alone (e.g., Jarvela et al., 2015 ; Kirschner, Kreijns, Phielix, & Fransen, 2015 ; Kreijns, Kirschner, & Vermeulen, 2013 ; Phielix, Prins, & Kirschner, 2010 ; Phielix, Prins, Kirschner, Erkens, & Jaspers, 2011 ). Peers may be able to judge the quality of the explanation better than the explainer. In fact, recent research suggests that peer instruction facilitates learning even more than self-explanations (Versteeg, van Blankenstein, Putter, & Steendijk, 2019 ).

Not only does peer instruction generate new knowledge, but it may also improve students’ metacognition. Our data show that peer discussion prompted more thorough testing of the coherence of the answers. Specifically, students’ confidences were better aligned with accuracy following discussion than before. Improvements in metacognitive resolution indicate that discussion provides more thorough testing of answers and ideas than does answering questions on one’s own. Discussion facilitates the metacognitive processes of detecting errors and assessing the coherence of an answer.

Agreement among peers has important consequences for final behavior. For example, when peers agreed, students very rarely changed their answer (less than 3% of the time). Further, large increases in confidence occurred when students agreed (as compared to when they disagreed). Alternatively, disagreements likely engaged different discussion processes and prompted students to combine different answers. Whether students weighed their initial answer more than their partner’s initial answer remains debatable. When students disagreed with their partner, they were more likely to stick with their own answer than switch; they kept their own answer 66% of the time. Even when their partner was more confident, students only switched to their partner’s answer 50% of the time. The low rate of switching during disagreements suggests that students weighed their own answer more heavily than their partner’s answer. In fact, across prior research, deciders typically weigh their own thoughts more than the thoughts of an advisor (Harvey, Harries, & Fischer, 2000 ; Yaniv & Kleinberger, 2000 ).

Interestingly, peers agreed more frequently than expected by chance. When students were initially correct (64% of the time), 78% of peers agreed. When students were initially incorrect (36% of the time), peers agreed 43% of the time. Pairs of students, then, agree more than expected by a random distribution of answers throughout the classroom. These data suggest that students group themselves into pairs based upon likelihood of sharing the same answer. Further, these data suggest that student understanding is not randomly distributed throughout the physical space of the classroom. Across all classes, students were instructed to work with a neighbor to discuss their answer. Given that neighbors agreed more than predicted by chance, students seem to tend to sit near and pair with peers that share their same levels of understanding. Our results from peer instruction reveal that students physically locate themselves near students of similar abilities. Peer instruction could potentially benefit from randomly pairing students together (i.e. not with a physically close neighbor) to generate the most disagreements and generative activity during discussion.

Learning through peer instruction may involve deep processing as peers actively challenge each other, and this deep processing may effectively support long-term retention. Future research can examine the persistence of gains in accuracy from peer instruction. For example, whether errors that are corrected during peer instruction stay corrected on later retests of the material remains an open question. High and low-confidence errors that are corrected during peer instruction may result in different long-term retention of the correct answer; more specifically, the hypercorrection effect suggests that errors committed with high confidence are more likely to be corrected on subsequent tests than errors with low confidence (e.g., Butler, Fazio, & Marsh, 2011 ; Butterfield & Metcalfe, 2001 ; Metcalfe, 2017 ). Whether hypercorrection holds for corrections from classmates during peer instruction (rather than from an absolute authority) could be examined in the future.

The influence of partner interaction on accuracy may depend upon the domain and kind of question posed to learners. For simple factual or perceptual questions, partner interaction may not consistently benefit learning. More specifically, partner interaction may amplify and bolster wrong answers when factual or perceptual questions lead most students to answer incorrectly (Koriat, 2015 ). However, for more “intellective tasks,” interactions and arguments between partners can produce gains in knowledge (Trouche et al., 2014 ). For example, groups typically outperform individuals for reasoning tasks (Laughlin, 2011 ; Moshman & Geil, 1998 ), math problems (Laughlin & Ellis, 1986 ), and logic problems (Doise & Mugny, 1984; Perret-Clermont, 1980 ). Peer instruction questions that allow for student argumentation and reasoning, therefore, may have the best benefits in student learning.

The underlying benefits of peer instruction extend beyond the improvements in accuracy seen from pre-discussion to post-discussion. Peer instruction prompts students to retrieve information from long-term memory, and these practice tests improve long-term retention of information (Roediger III & Karpicke, 2006 ; Tullis, Fiechter, & Benjamin, 2018 ). Further, feedback provided by instructors following peer instruction may guide students to improve their performance and correct misconceptions, which should benefit student learning (Bangert-Drowns, Kulik, & Kulik, 1991 ; Thurlings, Vermeulen, Bastiaens, & Stijnen, 2013 ). Learners who engage in peer discussion can use their new knowledge to solve new, but similar problems on their own (Smith et al., 2009 ). Generating new knowledge and revealing gaps in knowledge through peer instruction, then, effectively supports students’ ability to solve novel problems. Peer instruction can be an effective tool to generate new knowledge through discussion between peers and improve student understanding and metacognition.

Availability of data and materials

As described below, data and materials are available on the OpenScienceFramework: https://mfr.osf.io/render?url=https://osf.io/5qc46/?action=download%26mode=render .

Aleven, V., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer based cognitive tutor. Cognitive Science , 26 , 147–179.

Article   Google Scholar  

Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology , 95 , 774–783.

Baayen, R. H. (2008). Analyzing linguistic data: A practical introduction to statistics . Cambridge: Cambridge University Press.

Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language , 59 , 390–412.

Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C.-L. C. (1991). Effects of frequent classroom testing. Journal of Educational Research , 85 , 89–99.

Bargh, J. A., & Schul, Y. (1980). On the cognitive benefit of teaching. Journal of Educational Psychology , 72 , 593–604.

Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software , 67 , 1–48.

Bearison, D. J., Magzamen, S., & Filardo, E. K. (1986). Sociocognitive conflict and cognitive growth in young children. Merrill-Palmer Quarterly , 32 (1), 51–72.

Google Scholar  

Beatty, I. D., Gerace, W. J., Leonard, W. J., & Dufresne, R. J. (2006). Designing effective questions for classroom response system teaching. American Journal of Physics , 74 (1), 31e39.

Beekes, W. (2006). The “millionaire” method for encouraging participation. Active Learning in Higher Education , 7 , 25–36.

Benware, C. A., & Deci, E. L. (1984). Quality of learning with an active versus passive motivational set. American Educational Research Journal , 21 , 755–765.

Bielaczyc, K., Pirolli, P., & Brown, A. L. (1995). Training in self-explanation and self regulation strategies: Investigating the effects of knowledge acquisition activities on problem solving. Cognition and Instruction , 13 , 221–251.

Bossert, S. T. (1988-1989). Cooperative activities in the classroom. Review of Research in Education , 15 , 225–252.

Brooks, B. J., & Koretsky, M. D. (2011). The influence of group discussion on students’ responses and confidence during peer instruction. Journal of Chemistry Education , 88 , 1477–1484.

Brown, A. L., & Palincsar, A. S. (1989). Guided, cooperative learning and individual knowledge acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction: essays in honor of Robert Glaser , (pp. 393–451). Hillsdale: Erlbaum.

Butchart, S., Handfield, T., & Restall, G. (2009). Using peer instruction to teach philosophy, logic and critical thinking. Teaching Philosophy , 32 , 1–40.

Butler, A. C., Fazio, L. K., & Marsh, E. J. (2011). The hypercorrection effect persists over a week, but high-confidence errors return. Psychonomic Bulletin & Review , 18 (6), 1238–1244.

Butterfield, B., & Metcalfe, J. (2001). Errors committed with high confidence are hypercorrected. Journal of Experimental Psychology: Learning, Memory, and Cognition , 27 (6), 1491.

PubMed   Google Scholar  

Caldwell, J. E. (2007). Clickers in the large classroom: current research and best-practice tips. CBE-Life Sciences Education , 6 (1), 9–20.

Article   PubMed   PubMed Central   Google Scholar  

Chi, M., & VanLehn, K. A. (2010). Meta-cognitive strategy instruction in intelligent tutoring systems: How, when and why. Journal of Educational Technology and Society , 13 , 25–39.

Chi, M. T. H., & Bassock, M. (1989). Learning from examples via self-explanations. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser , (pp. 251–282). Hillsdale: Erlbaum.

Chi, M. T. H., Bassock, M., Lewis, M., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science , 13 , 145–182.

Chi, M. T. H., de Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science , 18 , 439–477.

Cortright, R. N., Collins, H. L., & DiCarlo, S. E. (2005). Peer instruction enhanced meaningful learning: Ability to solve novel problems. Advances in Physiology Education , 29 , 107–111.

Article   PubMed   Google Scholar  

Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American Journal of Physics , 69 , 970–977.

Cummings, K., & Roberts, S. (2008). A study of peer instruction methods with school physics students. In C. Henderson, M. Sabella, & L. Hsu (Eds.), Physics education research conference , (pp. 103–106). College Park: American Institute of Physics.

Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science , 332 , 862–864.

Duncan, D. (2005). Clickers in the classroom: How to enhance science teaching using classroom response systems . San Francisco: Pearson/Addison-Wesley.

Finley, J. R., Tullis, J. G., & Benjamin, A. S. (2010). Metacognitive control of learning and remembering. In M. S. Khine, & I. M. Saleh (Eds.), New science of learning: Cognition, computers and collaborators in education . New York: Springer Science & Business Media, LLC.

Giuliodori, M. J., Lujan, H. L., & DiCarlo, S. E. (2006). Peer instruction enhanced student performance on qualitative problem solving questions. Advances in Physiology Education , 30 , 168–173.

Graesser, A. C., McNamara, D., & VanLehn, K. (2005). Scaffolding deep comprehension strategies through AutoTutor and iSTART. Educational Psychologist , 40 , 225–234.

Granovskiy, B., Gold, J. M., Sumpter, D., & Goldstone, R. L. (2015). Integration of social information by human groups. Topics in Cognitive Science , 7 , 469–493.

Harvey, N., & Fischer, I. (1997). Taking advice: Accepting help, improving judgment, and sharing responsibility. Organizational Behavior and Human Decision Processes , 70 , 117–133.

Harvey, N., Harries, C., & Fischer, I. (2000). Using advice and assessing its quality. Organizational Behavior and Human Decision Processes , 81 , 252–273.

Henderson, C., & Dancy, M. H. (2009). The impact of physics education research on the teaching of introductory quantitative physics in the United States. Physical Review Special Topics: Physics Education Research , 5 (2), 020107.

Jaeger, T. F. (2008). Categorical data analysis: away from ANOVAs (transformation or not) and towards logit mixed models. Journal of Memory and Language , 59 , 434–446.

James, M. C. (2006). The effect of grading incentive on student discourse in peer instruction. American Journal of Physics , 74 (8), 689–691.

Jarvela, S., Kirschner, P., Panadero, E., Malmberg, J., Phielix, C., Jaspers, J., … Jarvenoja, H. (2015). Enhancing socially shared regulation in collaborative learning groups: Designing for CSCL regulation tools. Educational Technology Research and Development , 63 (1), 125e142.

Jones, M. E., Antonenko, P. D., & Greenwood, C. M. (2012). The impact of collaborative and individualized student response system strategies on learner motivation, metacognition, and knowledge transfer. Journal of Computer Assisted Learning , 28 (5), 477–487.

King, A. (1992). Facilitating elaborative learning through guided student-generated questioning. Educational Psychologist , 27 , 111–126.

Kirschner, P. A., Kreijns, K., Phielix, C., & Fransen, J. (2015). Awareness of cognitive and social behavior in a CSCL environment. Journal of Computer Assisted Learning , 31 (1), 59–77.

Knight, J. K., Wise, S. B., & Southard, K. M. (2013). Understanding clicker discussions: student reasoning and the impact of instructional cues. CBE-Life Sciences Education , 12 , 645–654.

Koriat, A. (2015). When two heads are better than one and when they can be worse: The amplification hypothesis. Journal of Experimental Psychology: General , 144 , 934–950. https://doi.org/10.1037/xge0000092 .

Kreijns, K., Kirschner, P. A., & Vermeulen, M. (2013). Social aspects of CSCL environments: A research framework. Educational Psychologist , 48 (4), 229e242.

Kuhn, L. M., & Sniezek, J. A. (1996). Confidence and uncertainty in judgmental forecasting: Differential effects of scenario presentation. Journal of Behavioral Decision Making , 9 , 231–247.

Lasry, N., Mazur, E., & Watkins, J. (2008). Peer instruction: From Harvard to the two-year college. American Journal of Physics , 76 (11), 1066–1069.

Laughlin, P. R. (2011). Group problem solving. Princeton: Princeton University Press.

Book   Google Scholar  

Laughlin, P. R., Bonner, B. L., & Miner, A. G. (2002). Groups perform better than individuals on letters-to-numbers problems. Organisational Behaviour and Human Decision Processes , 88 , 605–620.

Laughlin, P. R., & Ellis, A. L. (1986). Demonstrability and social combination processes on mathematical intellective tasks. Journal of Experimental Social Psychology, 22, 177–189.

Lucas, A. (2009). Using peer instruction and i-clickers to enhance student participation in calculus. Primus , 19 (3), 219–231.

Mazur, E. (1997). Peer instruction: A user’s manual . Upper Saddle River: Prentice Hall.

Mercier, H., & Sperber, D. (2019). The enigma of reason . Cambridge: Harvard University Press.

Metcalfe, J. (2017). Learning from errors. Annual Review of Psychology , 68 , 465–489.

Michaelsen, L. K., Watson, W. E., & Black, R. H. (1989). Realistic test of individual versus group decision making. Journal of Applied Psychology , 64 , 834–839.

Miller, R. L., Santana-Vega, E., & Terrell, M. S. (2007). Can good questions and peer discussion improve calculus instruction? Primus , 16 (3), 193–203.

Morgan, J. T., & Wakefield, C. (2012). Who benefits from peer conversation? Examining correlations of clicker question correctness and course performance. Journal of College Science Teaching , 41 (5), 51–56.

Moshman, D., & Geil, M. (1998). Collaborative reasoning: Evidence for collective rationality. Thinking and Reasoning, 4, 231–248.

Murayama, K., Sakaki, M., Yan, V. X., & Smith, G. M. (2014). Type I error inflation in the traditional by-participant analysis to metamemory accuracy: A generalized mixed-effects model perspective. Journal of Experimental Psychology: Learning, Memory, and Cognition , 40 , 1287–1306.

Newbury, P., & Heiner, C. (2012). Ready, set, react! getting the most out of peer instruction using clickers. Retrieved October 28, 2015, from http://www.cwsei.ubc.ca/Files/ReadySetReact_3fold.pdf .

Nielsen, K. L., Hansen-Nygård, G., & Stav, J. B. (2012). Investigating peer instruction: how the initial voting session affects students’ experiences of group discussion. ISRN Education , 2012 , article 290157.

Noddings, N. (1985). Small groups as a setting for research on mathematical problem solving. In E. A. Silver (Ed.), Teaching and learning mathematical problem solving , (pp. 345–360). Hillsdale: Erlbaum.

Perret-Clermont, A. N. (1980). Social Interaction and Cognitive Development in Children. London: Academic Press.

Perez, K. E., Strauss, E. A., Downey, N., Galbraith, A., Jeanne, R., Cooper, S., & Madison, W. (2010). Does displaying the class results affect student discussion during peer instruction? CBE Life Sciences Education , 9 , 133–140.

Phielix, C., Prins, F. J., & Kirschner, P. A. (2010). Awareness of group performance in a CSCL-environment: Effects of peer feedback and reflection. Computers in Human Behavior , 26 (2), 151–161.

Phielix, C., Prins, F. J., Kirschner, P. A., Erkens, G., & Jaspers, J. (2011). Group awareness of social and cognitive performance in a CSCL environment: Effects of a peer feedback and reflection tool. Computers in Human Behavior , 27 (3), 1087–1102.

Pollock, S. J., Chasteen, S. V., Dubson, M., & Perkins, K. K. (2010). The use of concept tests and peer instruction in upper-division physics. In M. Sabella, C. Singh, & S. Rebello (Eds.), AIP conference proceedings , (vol. 1289, p. 261). New York: AIP Press.

Porter, L., Bailey-Lee, C., & Simon, B. (2013). Halving fail rates using peer instruction: A study of four computer science courses. In SIGCSE ‘13: Proceedings of the 44th ACM technical symposium on computer science education , (pp. 177–182). New York: ACM Press.

Price, P. C., & Stone, E. R. (2004). Intuitive evaluation of likelihood judgment producers. Journal of Behavioral Decision Making , 17 , 39–57.

Priniski, J. H., & Horne, Z. (2019). Crowdsourcing effective educational interventions. In A. K. Goel, C. Seifert, & C. Freska (Eds.), Proceedings of the 41st annual conference of the cognitive science society . Austin: Cognitive Science Society.

Rao, S. P., & DiCarlo, S. E. (2000). Peer instruction improves performance on quizzes. Advances in Physiological Education , 24 , 51–55.

Renkl, A., Stark, R., Gruber, H., & Mandl, H. (1998). Learning from worked-out examples: The effects of example variability and elicited self-explanations. Contemporary Educational Psychology , 23 , 90–108.

Rittle-Johnson, B. (2006). Promoting transfer: Effects of self-explanation and direct instruction. Child Development , 77 , 1–15.

Roediger III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science , 17 , 249–255.

Ryskin, R., Benjamin, A. S., Tullis, J. G., & Brown-Schmidt, S. (2015). Perspective-taking in comprehension, production, and memory: An individual differences approach. Journal of Experimental Psychology: General , 144 , 898–915.

Sah, S., Moore, D. A., & MacCoun, R. J. (2013). Cheap talk and credibility: The consequences of confidence and accuracy on advisor credibility and persuasiveness. Organizational Behavior and Human Decision Processes , 121 , 246–255.

Schwartz, D. L. (1995). The emergence of abstract representations in dyad problem solving. The Journal of the Learning Sciences , 4 , 321–354.

Simon, B., Kohanfars, M., Lee, J., Tamayo, K., & Cutts, Q. (2010). Experience report: peer instruction in introductory computing. In Proceedings of the 41st SIGCSE technical symposium on computer science education .

Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild, N., & Su, T. T. (2009). Why peer discussion improves student performance on in-class concept questions. Science , 323 , 122–124.

Smith, M. K., Wood, W. B., Krauter, K., & Knight, J. K. (2011). Combining peer discussion with instructor explanation increases student learning from in-class concept questions. CBE-Life Sciences Education , 10 , 55–63.

Sniezek, J. A., & Buckley, T. (1995). Cueing and cognitive conflict in judge–Advisor decision making. Organizational Behavior and Human Decision Processes , 62 , 159–174.

Sniezek, J. A., & Henry, R. A. (1989). Accuracy and confidence in group judgment. Organizational Behavior and Human Decision Processes , 43 , 1–28.

Son, L. K., & Metcalfe, J. (2000). Metacognitive and control strategies in study-time allocation. Journal of Experimental Psychology: Learning, Memory, and Cognition , 26 , 204–221.

Steiner, I. D. (1972). Group processes and productivity . New York: Academic Press.

Thurlings, M., Vermeulen, M., Bastiaens, T., & Stijnen, S. (2013). Understanding feedback: A learning theory perspective. Educational Research Review , 9 , 1–15.

Tindale, R. S., & Sheffey, S. (2002). Shared information, cognitive load, and group memory. Group Processes & Intergroup Relations , 5 (1), 5–18.

Trouche, E., Sander, E., & Mercier, H. (2014). Arguments, more than confidence, explain the good performance of reasoning groups. Journal of Experimental Psychology: General , 143 , 1958–1971.

Tullis, J. G. (2018). Predicting others’ knowledge: Knowledge estimation as cue-utilization. Memory & Cognition , 46 , 1360–1375.

Tullis, J. G., Fiechter, J. L., & Benjamin, A. S. (2018). The efficacy of learners’ testing choices. Journal of Experimental Psychology: Learning, Memory, and Cognition , 44 , 540–552.

Tullis, J. G., & Fraundorf, S. H. (2017). Predicting others’ memory performance: The accuracy and bases of social metacognition. Journal of Memory and Language , 95 , 124–137.

Turpen, C., & Finkelstein, N. (2007). Understanding how physics faculty use peer instruction. In L. Hsu, C. Henderson, & L. McCullough (Eds.), Physics education research conference , (pp. 204–209). College Park: American Institute of Physics.

Van Swol, L. M., & Sniezek, J. A. (2005). Factors affecting the acceptance of expert advice. British Journal of Social Psychology , 44 , 443–461.

VanLehn, K., Jones, R. M., & Chi, M. T. H. (1992). A model of the self-explanation effect. Journal of the Learning Sciences , 2 (1), 1–59.

Vedder, P. (1985). Cooperative learning: A study on processes and effects of cooperation between primary school children . Westerhaven: Rijkuniversiteit Groningen.

Versteeg, M., van Blankenstein, F. M., Putter, H., & Steendijk, P. (2019). Peer instruction improves comprehension and transfer of physiological concepts: A randomized comparison with self-explanation. Advances in Health Sciences Education , 24 , 151–165.

Vygotsky, L. S. (1981). The genesis of higher mental functioning. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology , (pp. 144–188). Armonk: Sharpe.

Webb, N. M., & Palincsar, A. S. (1996). Group processes in the classroom. In D. C. Berliner, & R. C. Calfee (Eds.), Handbook of educational psychology , (pp. 841–873). New York: Macmillan Library Reference USA: London: Prentice Hall International.

Wegner, D. M., Giuliano, T., & Hertel, P. (1985). Cognitive interdependence in close relationships. In W. J. Ickes (Ed.), Compatible and incompatible relationships , (pp. 253–276). New York: Springer-Verlag.

Chapter   Google Scholar  

Weldon, M. S., & Bellinger, K. D. (1997). Collective memory: Collaborative and individual processes in remembering. Journal of Experimental Psychology: Learning, Memory, and Cognition , 23 , 1160–1175.

Wieman, C., Perkins, K., Gilbert, S., Benay, F., Kennedy, S., Semsar, K., et al. (2009). Clicker resource guide: An instructor’s guide to the effective use of personalresponse systems (clickers) in teaching . Vancouver: University of British Columbia Available from http://www.cwsei.ubc.ca/resources/files/Clicker_guide_CWSEI_CU-SEI.pdf .

Wong, R. M. F., Lawson, M. J., & Keeves, J. (2002). The effects of self-explanation training on students’ problem solving in high school mathematics. Learning and Instruction , 12 , 23.

Yackel, E., Cobb, P., & Wood, T. (1991). Small-group interactions as a source of learning opportunities in second-grade mathematics. Journal for Research in Mathematics Education , 22 , 390–408.

Yaniv, I. (2004a). The benefit of additional opinions. Current Directions in Psychological Science , 13 , 75–78.

Yaniv, I. (2004b). Receiving other people’s advice: Influence and benefit. Organizational Behavior and Human Decision Processes , 93 , 1–13.

Yaniv, I., & Choshen-Hillel, S. (2012). Exploiting the wisdom of others to make better decisions: Suspending judgment reduces egocentrism and increases accuracy. Journal of Behavioral Decision Making , 25 , 427–434.

Yaniv, I., & Kleinberger, E. (2000). Advice taking in decision making: Egocentric discounting and reputation formation. Organizational Behavior and Human Decision Processes , 83 , 260–281.

Download references

Acknowledgements

Not applicable.

No funding supported this manuscript.

Author information

Authors and affiliations.

Department of Educational Psychology, University of Arizona, 1430 E. Second St., Tucson, AZ, 85721, USA

Jonathan G. Tullis

Department of Psychology, Indiana University, Bloomington, IN, USA

Robert L. Goldstone

You can also search for this author in PubMed   Google Scholar

Contributions

JGT collected some data, analyzed the data, and wrote the first draft of the paper. RLG collected some data, contributed significantly to the framing of the paper, and edited the paper. The authors read and approved the final manuscript.

Authors’ information

JGT: Assistant Professor in Educational Psychology at University of Arizona. RLG: Chancellor’s Professor in Psychology at Indiana University.

Corresponding author

Correspondence to Jonathan G. Tullis .

Ethics declarations

Ethics approval and consent to participate.

The ethics approval was waived by the Indiana University Institutional Review Board (IRB) and the University of Arizona IRB, given that these data are collected as part of normal educational settings and processes.

Consent for publication

No individual data are presented in the manuscript.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Tullis, J.G., Goldstone, R.L. Why does peer instruction benefit student learning?. Cogn. Research 5 , 15 (2020). https://doi.org/10.1186/s41235-020-00218-5

Download citation

Received : 08 October 2019

Accepted : 25 February 2020

Published : 09 April 2020

DOI : https://doi.org/10.1186/s41235-020-00218-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Group decisions
• Peer instruction
• Metacognition
• Decision making

• Research article
• Open access
• Published: 15 February 2018

Blended learning: the new normal and emerging technologies

• Charles Dziuban 1 ,
• Charles R. Graham 2 ,
• Patsy D. Moskal   ORCID: orcid.org/0000-0001-6376-839X 1 ,
• Anders Norberg 3 &
• Nicole Sicilia 1  

International Journal of Educational Technology in Higher Education volume  15 , Article number:  3 ( 2018 ) Cite this article

551k Accesses

355 Citations

118 Altmetric

Metrics details

This study addressed several outcomes, implications, and possible future directions for blended learning (BL) in higher education in a world where information communication technologies (ICTs) increasingly communicate with each other. In considering effectiveness, the authors contend that BL coalesces around access, success, and students’ perception of their learning environments. Success and withdrawal rates for face-to-face and online courses are compared to those for BL as they interact with minority status. Investigation of student perception about course excellence revealed the existence of robust if-then decision rules for determining how students evaluate their educational experiences. Those rules were independent of course modality, perceived content relevance, and expected grade. The authors conclude that although blended learning preceded modern instructional technologies, its evolution will be inextricably bound to contemporary information communication technologies that are approximating some aspects of human thought processes.

Introduction

Blended learning and research issues.

Blended learning (BL), or the integration of face-to-face and online instruction (Graham 2013 ), is widely adopted across higher education with some scholars referring to it as the “new traditional model” (Ross and Gage 2006 , p. 167) or the “new normal” in course delivery (Norberg et al. 2011 , p. 207). However, tracking the accurate extent of its growth has been challenging because of definitional ambiguity (Oliver and Trigwell 2005 ), combined with institutions’ inability to track an innovative practice, that in many instances has emerged organically. One early nationwide study sponsored by the Sloan Consortium (now the Online Learning Consortium) found that 65.2% of participating institutions of higher education (IHEs) offered blended (also termed hybrid ) courses (Allen and Seaman 2003 ). A 2008 study, commissioned by the U.S. Department of Education to explore distance education in the U.S., defined BL as “a combination of online and in-class instruction with reduced in-class seat time for students ” (Lewis and Parsad 2008 , p. 1, emphasis added). Using this definition, the study found that 35% of higher education institutions offered blended courses, and that 12% of the 12.2 million documented distance education enrollments were in blended courses.

The 2017 New Media Consortium Horizon Report found that blended learning designs were one of the short term forces driving technology adoption in higher education in the next 1–2 years (Adams Becker et al. 2017 ). Also, blended learning is one of the key issues in teaching and learning in the EDUCAUSE Learning Initiative’s 2017 annual survey of higher education (EDUCAUSE 2017 ). As institutions begin to examine BL instruction, there is a growing research interest in exploring the implications for both faculty and students. This modality is creating a community of practice built on a singular and pervasive research question, “How is blended learning impacting the teaching and learning environment?” That question continues to gain traction as investigators study the complexities of how BL interacts with cognitive, affective, and behavioral components of student behavior, and examine its transformation potential for the academy. Those issues are so compelling that several volumes have been dedicated to assembling the research on how blended learning can be better understood (Dziuban et al. 2016 ; Picciano et al. 2014 ; Picciano and Dziuban 2007 ; Bonk and Graham 2007 ; Kitchenham 2011 ; Jean-François 2013 ; Garrison and Vaughan 2013 ) and at least one organization, the Online Learning Consortium, sponsored an annual conference solely dedicated to blended learning at all levels of education and training (2004–2015). These initiatives address blended learning in a wide variety of situations. For instance, the contexts range over K-12 education, industrial and military training, conceptual frameworks, transformational potential, authentic assessment, and new research models. Further, many of these resources address students’ access, success, withdrawal, and perception of the degree to which blended learning provides an effective learning environment.

Currently the United States faces a widening educational gap between our underserved student population and those communities with greater financial and technological resources (Williams 2016 ). Equal access to education is a critical need, one that is particularly important for those in our underserved communities. Can blended learning help increase access thereby alleviating some of the issues faced by our lower income students while resulting in improved educational equality? Although most indicators suggest “yes” (Dziuban et al. 2004 ), it seems that, at the moment, the answer is still “to be determined.” Quality education presents a challenge, evidenced by many definitions of what constitutes its fundamental components (Pirsig 1974 ; Arum et al. 2016 ). Although progress has been made by initiatives, such as, Quality Matters ( 2016 ), the OLC OSCQR Course Design Review Scorecard developed by Open SUNY (Open SUNY n.d. ), the Quality Scorecard for Blended Learning Programs (Online Learning Consortium n.d. ), and SERVQUAL (Alhabeeb 2015 ), the issue is by no means resolved. Generally, we still make quality education a perceptual phenomenon where we ascribe that attribute to a course, educational program, or idea, but struggle with precisely why we reached that decision. Searle ( 2015 ), summarizes the problem concisely arguing that quality does not exist independently, but is entirely observer dependent. Pirsig ( 1974 ) in his iconic volume on the nature of quality frames the context this way,

“There is such thing as Quality, but that as soon as you try to define it, something goes haywire. You can’t do it” (p. 91).

Therefore, attempting to formulate a semantic definition of quality education with syntax-based metrics results in what O’Neil (O'Neil 2017 ) terms surrogate models that are rough approximations and oversimplified. Further, the derived metrics tend to morph into goals or benchmarks, losing their original measurement properties (Goodhart 1975 ).

Information communication technologies in society and education

Blended learning forces us to consider the characteristics of digital technology, in general, and information communication technologies (ICTs), more specifically. Floridi ( 2014 ) suggests an answer proffered by Alan Turing: that digital ICTs can process information on their own, in some sense just as humans and other biological life. ICTs can also communicate information to each other, without human intervention, but as linked processes designed by humans. We have evolved to the point where humans are not always “in the loop” of technology, but should be “on the loop” (Floridi 2014 , p. 30), designing and adapting the process. We perceive our world more and more in informational terms, and not primarily as physical entities (Floridi 2008 ). Increasingly, the educational world is dominated by information and our economies rest primarily on that asset. So our world is also blended, and it is blended so much that we hardly see the individual components of the blend any longer. Floridi ( 2014 ) argues that the world has become an “infosphere” (like biosphere) where we live as “inforgs.” What is real for us is shifting from the physical and unchangeable to those things with which we can interact.

Floridi also helps us to identify the next blend in education, involving ICTs, or specialized artificial intelligence (Floridi 2014 , 25; Norberg 2017 , 65). Learning analytics, adaptive learning, calibrated peer review, and automated essay scoring (Balfour 2013 ) are advanced processes that, provided they are good interfaces, can work well with the teacher— allowing him or her to concentrate on human attributes such as being caring, creative, and engaging in problem-solving. This can, of course, as with all technical advancements, be used to save resources and augment the role of the teacher. For instance, if artificial intelligence can be used to work along with teachers, allowing them more time for personal feedback and mentoring with students, then, we will have made a transformational breakthrough. The Edinburg University manifesto for teaching online says bravely, “Automation need not impoverish education – we welcome our robot colleagues” (Bayne et al. 2016 ). If used wisely, they will teach us more about ourselves, and about what is truly human in education. This emerging blend will also affect curricular and policy questions, such as the what? and what for? The new normal for education will be in perpetual flux. Floridi’s ( 2014 ) philosophy offers us tools to understand and be in control and not just sit by and watch what happens. In many respects, he has addressed the new normal for blended learning.

Literature of blended learning

A number of investigators have assembled a comprehensive agenda of transformative and innovative research issues for blended learning that have the potential to enhance effectiveness (Garrison and Kanuka 2004 ; Picciano 2009 ). Generally, research has found that BL results in improvement in student success and satisfaction, (Dziuban and Moskal 2011 ; Dziuban et al. 2011 ; Means et al. 2013 ) as well as an improvement in students’ sense of community (Rovai and Jordan 2004 ) when compared with face-to-face courses. Those who have been most successful at blended learning initiatives stress the importance of institutional support for course redesign and planning (Moskal et al. 2013 ; Dringus and Seagull 2015 ; Picciano 2009 ; Tynan et al. 2015 ). The evolving research questions found in the literature are long and demanding, with varied definitions of what constitutes “blended learning,” facilitating the need for continued and in-depth research on instructional models and support needed to maximize achievement and success (Dringus and Seagull 2015 ; Bloemer and Swan 2015 ).

Educational access

The lack of access to educational technologies and innovations (sometimes termed the digital divide) continues to be a challenge with novel educational technologies (Fairlie 2004 ; Jones et al. 2009 ). One of the promises of online technologies is that they can increase access to nontraditional and underserved students by bringing a host of educational resources and experiences to those who may have limited access to on-campus-only higher education. A 2010 U.S. report shows that students with low socioeconomic status are less likely to obtain higher levels of postsecondary education (Aud et al. 2010 ). However, the increasing availability of distance education has provided educational opportunities to millions (Lewis and Parsad 2008 ; Allen et al. 2016 ). Additionally, an emphasis on open educational resources (OER) in recent years has resulted in significant cost reductions without diminishing student performance outcomes (Robinson et al. 2014 ; Fischer et al. 2015 ; Hilton et al. 2016 ).

Unfortunately, the benefits of access may not be experienced evenly across demographic groups. A 2015 study found that Hispanic and Black STEM majors were significantly less likely to take online courses even when controlling for academic preparation, socioeconomic status (SES), citizenship, and English as a second language (ESL) status (Wladis et al. 2015 ). Also, questions have been raised about whether the additional access afforded by online technologies has actually resulted in improved outcomes for underserved populations. A distance education report in California found that all ethnic minorities (except Asian/Pacific Islanders) completed distance education courses at a lower rate than the ethnic majority (California Community Colleges Chancellor’s Office 2013 ). Shea and Bidjerano ( 2014 , 2016 ) found that African American community college students who took distance education courses completed degrees at significantly lower rates than those who did not take distance education courses. On the other hand, a study of success factors in K-12 online learning found that for ethnic minorities, only 1 out of 15 courses had significant gaps in student test scores (Liu and Cavanaugh 2011 ). More research needs to be conducted, examining access and success rates for different populations, when it comes to learning in different modalities, including fully online and blended learning environments.

Framing a treatment effect

Over the last decade, there have been at least five meta-analyses that have addressed the impact of blended learning environments and its relationship to learning effectiveness (Zhao et al. 2005 ; Sitzmann et al. 2006 ; Bernard et al. 2009 ; Means et al. 2010 , 2013 ; Bernard et al. 2014 ). Each of these studies has found small to moderate positive effect sizes in favor of blended learning when compared to fully online or traditional face-to-face environments. However, there are several considerations inherent in these studies that impact our understanding the generalizability of outcomes.

Dziuban and colleagues (Dziuban et al. 2015 ) analyzed the meta-analyses conducted by Means and her colleagues (Means et al. 2013 ; Means et al. 2010 ), concluding that their methods were impressive as evidenced by exhaustive study inclusion criteria and the use of scale-free effect size indices. The conclusion, in both papers, was that there was a modest difference in multiple outcome measures for courses featuring online modalities—in particular, blended courses. However, with blended learning especially, there are some concerns with these kinds of studies. First, the effect sizes are based on the linear hypothesis testing model with the underlying assumption that the treatment and the error terms are uncorrelated, indicating that there is nothing else going on in the blending that might confound the results. Although the blended learning articles (Means et al. 2010 ) were carefully vetted, the assumption of independence is tenuous at best so that these meta-analysis studies must be interpreted with extreme caution.

There is an additional concern with blended learning as well. Blends are not equivalent because of the manner on which they are configured. For instance, a careful reading of the sources used in the Means, et al. papers will identify, at minimum, the following blending techniques: laboratory assessments, online instruction, e-mail, class web sites, computer laboratories, mapping and scaffolding tools, computer clusters, interactive presentations and e-mail, handwriting capture, evidence-based practice, electronic portfolios, learning management systems, and virtual apparatuses. These are not equivalent ways in which to configure courses, and such nonequivalence constitutes the confounding we describe. We argue here that, in actuality, blended learning is a general construct in the form of a boundary object (Star and Griesemer 1989 ) rather than a treatment effect in the statistical sense. That is, an idea or concept that can support a community of practice, but is weakly defined fostering disagreement in the general group. Conversely, it is stronger in individual constituencies. For instance, content disciplines (i.e. education, rhetoric, optics, mathematics, and philosophy) formulate a more precise definition because of commonly embraced teaching and learning principles. Quite simply, the situation is more complicated than that, as Leonard Smith ( 2007 ) says after Tolstoy,

“All linear models resemble each other, each non nonlinear system is unique in its own way” (p. 33).

This by no means invalidates these studies, but effect size associated with blended learning should be interpreted with caution where the impact is evaluated within a particular learning context.

Study objectives

This study addressed student access by examining success and withdrawal rates in the blended learning courses by comparing them to face-to-face and online modalities over an extended time period at the University of Central Florida. Further, the investigators sought to assess the differences in those success and withdrawal rates with the minority status of students. Secondly, the investigators examined the student end-of-course ratings of blended learning and other modalities by attempting to develop robust if-then decision rules about what characteristics of classes and instructors lead students to assign an “excellent” value to their educational experience. Because of the high stakes nature of these student ratings toward faculty promotion, awards, and tenure, they act as a surrogate measure for instructional quality. Next, the investigators determined the conditional probabilities for students conforming to the identified rule cross-referenced by expected grade, the degree to which they desired to take the course, and course modality.

Student grades by course modality were recoded into a binary variable with C or higher assigned a value of 1, and remaining values a 0. This was a declassification process that sacrificed some specificity but compensated for confirmation bias associated with disparate departmental policies regarding grade assignment. At the measurement level this was an “on track to graduation index” for students. Withdrawal was similarly coded by the presence or absence of its occurrence. In each case, the percentage of students succeeding or withdrawing from blended, online or face-to-face courses was calculated by minority and non-minority status for the fall 2014 through fall 2015 semesters.

Next, a classification and regression tree (CART) analysis (Brieman et al. 1984 ) was performed on the student end-of-course evaluation protocol ( Appendix 1 ). The dependent measure was a binary variable indicating whether or not a student assigned an overall rating of excellent to his or her course experience. The independent measures in the study were: the remaining eight rating items on the protocol, college membership, and course level (lower undergraduate, upper undergraduate, and graduate). Decision trees are efficient procedures for achieving effective solutions in studies such as this because with missing values imputation may be avoided with procedures such as floating methods and the surrogate formation (Brieman et al. 1984 , Olshen et al. 1995 ). For example, a logistic regression method cannot efficiently handle all variables under consideration. There are 10 independent variables involved here; one variable has three levels, another has nine, and eight have five levels each. This means the logistic regression model must incorporate more than 50 dummy variables and an excessively large number of two-way interactions. However, the decision-tree method can perform this analysis very efficiently, permitting the investigator to consider higher order interactions. Even more importantly, decision trees represent appropriate methods in this situation because many of the variables are ordinally scaled. Although numerical values can be assigned to each category, those values are not unique. However, decision trees incorporate the ordinal component of the variables to obtain a solution. The rules derived from decision trees have an if-then structure that is readily understandable. The accuracy of these rules can be assessed with percentages of correct classification or odds-ratios that are easily understood. The procedure produces tree-like rule structures that predict outcomes.

The model-building procedure for predicting overall instructor rating

For this study, the investigators used the CART method (Brieman et al. 1984 ) executed with SPSS 23 (IBM Corp 2015 ). Because of its strong variance-sharing tendencies with the other variables, the dependent measure for the analysis was the rating on the item Overall Rating of the Instructor , with the previously mentioned indicator variables (college, course level, and the remaining 8 questions) on the instrument. Tree methods are recursive, and bisect data into subgroups called nodes or leaves. CART analysis bases itself on: data splitting, pruning, and homogeneous assessment.

Splitting the data into two (binary) subsets comprises the first stage of the process. CART continues to split the data until the frequencies in each subset are either very small or all observations in a subset belong to one category (e.g., all observations in a subset have the same rating). Usually the growing stage results in too many terminate nodes for the model to be useful. CART solves this problem using pruning methods that reduce the dimensionality of the system.

The final stage of the analysis involves assessing homogeneousness in growing and pruning the tree. One way to accomplish this is to compute the misclassification rates. For example, a rule that produces a .95 probability that an instructor will receive an excellent rating has an associated error of 5.0%.

Implications for using decision trees

Although decision-tree techniques are effective for analyzing datasets such as this, the reader should be aware of certain limitations. For example, since trees use ranks to analyze both ordinal and interval variables, information can be lost. However, the most serious weakness of decision tree analysis is that the results can be unstable because small initial variations can lead to substantially different solutions.

For this study model, these problems were addressed with the k-fold cross-validation process. Initially the dataset was partitioned randomly into 10 subsets with an approximately equal number of records in each subset. Each cohort is used as a test partition, and the remaining subsets are combined to complete the function. This produces 10 models that are all trained on different subsets of the original dataset and where each has been used as the test partition one time only.

Although computationally dense, CART was selected as the analysis model for a number of reasons— primarily because it provides easily interpretable rules that readers will be able evaluate in their particular contexts. Unlike many other multivariate procedures that are even more sensitive to initial estimates and require a good deal of statistical sophistication for interpretation, CART has an intuitive resonance with researcher consumers. The overriding objective of our choice of analysis methods was to facilitate readers’ concentration on our outcomes rather than having to rely on our interpretation of the results.

Institution-level evaluation: Success and withdrawal

The University of Central Florida (UCF) began a longitudinal impact study of their online and blended courses at the start of the distributed learning initiative in 1996. The collection of similar data across multiple semesters and academic years has allowed UCF to monitor trends, assess any issues that may arise, and provide continual support for both faculty and students across varying demographics. Table  1 illustrates the overall success rates in blended, online and face-to-face courses, while also reporting their variability across minority and non-minority demographics.

While success (A, B, or C grade) is not a direct reflection of learning outcomes, this overview does provide an institutional level indication of progress and possible issues of concern. BL has a slight advantage when looking at overall success and withdrawal rates. This varies by discipline and course, but generally UCF’s blended modality has evolved to be the best of both worlds, providing an opportunity for optimizing face-to-face instruction through the effective use of online components. These gains hold true across minority status. Reducing on-ground time also addresses issues that impact both students and faculty such as parking and time to reach class. In addition, UCF requires faculty to go through faculty development tailored to teaching in either blended or online modalities. This 8-week faculty development course is designed to model blended learning, encouraging faculty to redesign their course and not merely consider blended learning as a means to move face-to-face instructional modules online (Cobb et al. 2012 ; Lowe 2013 ).

Withdrawal (Table  2 ) from classes impedes students’ success and retention and can result in delayed time to degree, incurred excess credit hour fees, or lost scholarships and financial aid. Although grades are only a surrogate measure for learning, they are a strong predictor of college completion. Therefore, the impact of any new innovation on students’ grades should be a component of any evaluation. Once again, the blended modality is competitive and in some cases results in lower overall withdrawal rates than either fully online or face-to-face courses.

The students’ perceptions of their learning environments

Other potentially high-stakes indicators can be measured to determine the impact of an innovation such as blended learning on the academy. For instance, student satisfaction and attitudes can be measured through data collection protocols, including common student ratings, or student perception of instruction instruments. Given that those ratings often impact faculty evaluation, any negative reflection can derail the successful implementation and scaling of an innovation by disenfranchised instructors. In fact, early online and blended courses created a request by the UCF faculty senate to investigate their impact on faculty ratings as compared to face-to-face sections. The UCF Student Perception of Instruction form is released automatically online through the campus web portal near the end of each semester. Students receive a splash page with a link to each course’s form. Faculty receive a scripted email that they can send to students indicating the time period that the ratings form will be available. The forms close at the beginning of finals week. Faculty receive a summary of their results following the semester end.

The instrument used for this study was developed over a ten year period by the faculty senate of the University of Central Florida, recognizing the evolution of multiple course modalities including blended learning. The process involved input from several constituencies on campus (students, faculty, administrators, instructional designers, and others), in attempt to provide useful formative and summative instructional information to the university community. The final instrument was approved by resolution of the senate and, currently, is used across the university. Students’ rating of their classes and instructors comes with considerable controversy and disagreement with researchers aligning themselves on both sides of the issue. Recently, there have been a number of studies criticizing the process (Uttl et al. 2016 ; Boring et al. 2016 ; & Stark and Freishtat 2014 ). In spite of this discussion, a viable alternative has yet to emerge in higher education. So in the foreseeable future, the process is likely to continue. Therefore, with an implied faculty senate mandate this study was initiated by this team of researchers.

Prior to any analysis of the item responses collected in this campus-wide student sample, the psychometric quality (domain sampling) of the information yielded by the instrument was assessed. Initially, the reliability (internal consistency) was derived using coefficient alpha (Cronbach 1951 ). In addition, Guttman ( 1953 ) developed a theorem about item properties that leads to evidence about the quality of one’s data, demonstrating that as the domain sampling properties of items improve, the inverse of the correlation matrix among items will approach a diagonal. Subsequently, Kaiser and Rice ( 1974 ) developed the measure of sampling adequacy (MSA) that is a function of the Guttman Theorem. The index has an upper bound of one with Kaiser offering some decision rules for interpreting the value of MSA. If the value of the index is in the .80 to .99 range, the investigator has evidence of an excellent domain sample. Values in the .70s signal an acceptable result, and those in the .60s indicate data that are unacceptable. Customarily, the MSA has been used for data assessment prior to the application of any dimensionality assessments. Computation of the MSA value gave the investigators a benchmark for the construct validity of the items in this study. This procedure has been recommended by Dziuban and Shirkey ( 1974 ) prior to any latent dimension analysis and was used with the data obtained for this study. The MSA for the current instrument was .98 suggesting excellent domain sampling properties with an associated alpha reliability coefficient of .97 suggesting superior internal consistency. The psychometric properties of the instrument were excellent with both measures.

The online student ratings form presents an electronic data set each semester. These can be merged across time to create a larger data set of completed ratings for every course across each semester. In addition, captured data includes course identification variables including prefix, number, section and semester, department, college, faculty, and class size. The overall rating of effectiveness is used most heavily by departments and faculty in comparing across courses and modalities (Table  3 ).

The finally derived tree (decision rules) included only three variables—survey items that asked students to rate the instructor’s effectiveness at:

Helping students achieve course objectives,

Creating an environment that helps students learn, and

Communicating ideas and information.

None of the demographic variables associated with the courses contributed to the final model. The final rule specifies that if a student assigns an excellent rating to those three items, irrespective of their status on any other condition, the probability is .99 that an instructor will receive an overall rating of excellent. The converse is true as well. A poor rating on all three of those items will lead to a 99% chance of an instructor receiving an overall rating of poor.

Tables  4 , 5 and 6 present a demonstration of the robustness of the CART rule for variables on which it was not developed: expected course grade, desire to take the course and modality.

In each case, irrespective of the marginal probabilities, those students conforming to the rule have a virtually 100% chance of seeing the course as excellent. For instance, 27% of all students expecting to fail assigned an excellent rating to their courses, but when they conformed to the rule the percentage rose to 97%. The same finding is true when students were asked about their desire to take the course with those who strongly disagreed assigning excellent ratings to their courses 26% of the time. However, for those conforming to the rule, that category rose to 92%. When course modality is considered in the marginal sense, blended learning is rated as the preferred choice. However, from Table  6 we can observe that the rule equates student assessment of their learning experiences. If they conform to the rule, they will see excellence.

This study addressed increasingly important issues of student success, withdrawal and perception of the learning environment across multiple course modalities. Arguably these components form the crux of how we will make more effective decisions about how blended learning configures itself in the new normal. The results reported here indicate that blending maintains or increases access for most student cohorts and produces improved success rates for minority and non-minority students alike. In addition, when students express their beliefs about the effectiveness of their learning environments, blended learning enjoys the number one rank. However, upon more thorough analysis of key elements students view as important in their learning, external and demographic variables have minimal impact on those decisions. For example college (i.e. discipline) membership, course level or modality, expected grade or desire to take a particular course have little to do with their course ratings. The characteristics they view as important relate to clear establishment and progress toward course objectives, creating an effective learning environment and the instructors’ effective communication. If in their view those three elements of a course are satisfied they are virtually guaranteed to evaluate their educational experience as excellent irrespective of most other considerations. While end of course rating protocols are summative the three components have clear formative characteristics in that each one is directly related to effective pedagogy and is responsive to faculty development through units such as the faculty center for teaching and learning. We view these results as encouraging because they offer potential for improving the teaching and learning process in an educational environment that increases the pressure to become more responsive to contemporary student lifestyles.

Clearly, in this study we are dealing with complex adaptive systems that feature the emergent property. That is, their primary agents and their interactions comprise an environment that is more than the linear combination of their individual elements. Blending learning, by interacting with almost every aspect of higher education, provides opportunities and challenges that we are not able to fully anticipate.

This pedagogy alters many assumptions about the most effective way to support the educational environment. For instance, blending, like its counterpart active learning, is a personal and individual phenomenon experienced by students. Therefore, it should not be surprising that much of what we have called blended learning is, in reality, blended teaching that reflects pedagogical arrangements. Actually, the best we can do for assessing impact is to use surrogate measures such as success, grades, results of assessment protocols, and student testimony about their learning experiences. Whether or not such devices are valid indicators remains to be determined. We may be well served, however, by changing our mode of inquiry to blended teaching.

Additionally, as Norberg ( 2017 ) points out, blended learning is not new. The modality dates back, at least, to the medieval period when the technology of textbooks was introduced into the classroom where, traditionally, the professor read to the students from the only existing manuscript. Certainly, like modern technologies, books were disruptive because they altered the teaching and learning paradigm. Blended learning might be considered what Johnson describes as a slow hunch (2010). That is, an idea that evolved over a long period of time, achieving what Kaufmann ( 2000 ) describes as the adjacent possible – a realistic next step occurring in many iterations.

The search for a definition for blended learning has been productive, challenging, and, at times, daunting. The definitional continuum is constrained by Oliver and Trigwell ( 2005 ) castigation of the concept for its imprecise vagueness to Sharpe et al.’s ( 2006 ) notion that its definitional latitude enhances contextual relevance. Both extremes alter boundaries such as time, place, presence, learning hierarchies, and space. The disagreement leads us to conclude that Lakoff’s ( 2012 ) idealized cognitive models i.e. arbitrarily derived concepts (of which blended learning might be one) are necessary if we are to function effectively. However, the strong possibility exists that blended learning, like quality, is observer dependent and may not exist outside of our perceptions of the concept. This, of course, circles back to the problem of assuming that blending is a treatment effect for point hypothesis testing and meta-analysis.

Ultimately, in this article, we have tried to consider theoretical concepts and empirical findings about blended learning and their relationship to the new normal as it evolves. Unfortunately, like unresolved chaotic solutions, we cannot be sure that there is an attractor or that it will be the new normal. That being said, it seems clear that blended learning is the harbinger of substantial change in higher education and will become equally impactful in K-12 schooling and industrial training. Blended learning, because of its flexibility, allows us to maximize many positive education functions. If Floridi ( 2014 ) is correct and we are about to live in an environment where we are on the communication loop rather than in it, our educational future is about to change. However, if our results are correct and not over fit to the University of Central Florida and our theoretical speculations have some validity, the future of blended learning should encourage us about the coming changes.

Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C., & Ananthanarayanan, V. (2017). NMC horizon report: 2017 higher Education Edition . Austin: The New Media Consortium.

Google Scholar  

Alhabeeb, A. M. (2015). The quality assessment of the services offered to the students of the College of Education at King Saud University using (SERVQUAL) method. Journal of Education and Practice , 6 (30), 82–93.

Allen, I. E., & Seaman, J. (2003). Sizing the opportunity: The quality and extent of online education in the United States, 2002 and 2003. Retrieved from http://files.eric.ed.gov/fulltext/ED530060.pdf

Allen, I. E., Seaman, J., Poulin, R., & Straut, T. T. (2016). Online report card: Tracking online education in the United States, 1–4. Retrieved from http://onlinelearningsurvey.com/reports/onlinereportcard.pdf

Arum, R., Roksa, J., & Cook, A. (2016). Improving quality in American higher education: Learning outcomes and assessments for the 21st century . San Francisco: Jossey-Bass.

Aud, S., Hussar, W., Planty, M., Snyder, T., Bianco, K., Fox, M. A., & Drake, L. (2010). The condition of education - 2010. Education, 4–29. https://doi.org/10.1037/e492172006-019

Balfour, S. P. (2013). Assessing writing in MOOCs: Automated essay scoring and calibrated peer review. Research and Practice in Assessment , 2013 (8), 40–48.

Bayne, S., Evans, P., Ewins, R.,Knox, J., Lamb, J., McLeod, H., O’Shea, C., Ross, J., Sheail, P. & Sinclair, C, (2016) Manifesto for teaching online. Digital Education at Edinburg University. Retrieved from https://onlineteachingmanifesto.wordpress.com/the-text/

Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research , 79 (3), 1243–1289. https://doi.org/10.3102/0034654309333844 .

Article   Google Scholar  

Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014). A meta-analysis of blended learning and technology use in higher education: From the general to the applied. Journal of Computing in Higher Education , 26 (1), 87–122.

Bloemer, W., & Swan, K. (2015). Investigating informal blending at the University of Illinois Springfield. In A. G. Picciano, C. D. Dziuban, & C. R. Graham (Eds.), Blended learning: Research perspectives , (vol. 2, pp. 52–69). New York: Routledge.

Bonk, C. J., & Graham, C. R. (2007). The handbook of blended learning: Global perspectives, local designs . San Francisco: Pfeiffer.

Boring, A., Ottoboni, K., & Stark, P.B. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. EGERA.

Brieman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees . New York: Chapman & Hall.

California Community Colleges Chancellor’s Office. (2013). Distance education report.

Cobb, C., deNoyelles, A., & Lowe, D. (2012). Influence of reduced seat time on satisfaction and perception of course development goals: A case study in faculty development. The Journal of Asynchronous Learning , 16 (2), 85–98.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika , 16 (3), 297–334 Retrieved from http://psych.colorado.edu/~carey/courses/psyc5112/readings/alpha_cronbach.pdf .

Article   MATH   Google Scholar  

Dringus, L. P., and A. B. Seagull. 2015. A five-year study of sustaining blended learning initiatives to enhance academic engagement in computer and information sciences campus courses. In Blended learning: Research perspectives. Vol. 2. Edited by A. G. Picciano, C. D. Dziuban, and C. R. Graham, 122-140. New York: Routledge.

Dziuban, C. D., & Shirkey, E. C. (1974). When is a correlation matrix appropriate for factor analysis? Some decision rules. Psychological Bulletin , 81(6), 358. https://doi.org/10.1037/h0036316 .

Dziuban, C., Hartman, J., Cavanagh, T., & Moskal, P. (2011). Blended courses as drivers of institutional transformation. In A. Kitchenham (Ed.), Blended learning across disciplines: Models for implementation , (pp. 17–37). Hershey: IGI Global.

Chapter   Google Scholar  

Dziuban, C., & Moskal, P. (2011). A course is a course is a course: Factor invariance in student evaluation of online, blended and face-to-face learning environments. The Internet and Higher Education , 14 (4), 236–241.

Dziuban, C., Moskal, P., Hermsdorfer, A., DeCantis, G., Norberg, A., & Bradford, G., (2015) A deconstruction of blended learning. Presented at the 11 th annual Sloan-C blended learning conference and workshop

Dziuban, C., Picciano, A. G., Graham, C. R., & Moskal, P. D. (2016). Conducting research in online and blended learning environments: New pedagogical frontiers . New York: Routledge, Taylor & Francis Group.

Dziuban, C. D., Hartman, J. L., & Moskal, P. D. (2004). Blended learning. EDUCAUSE Research Bulletin , 7 , 1–12.

EDUCAUSE. (2017) 2017 key issues in teaching & learning. Retrieved from https://www.EDUCAUSE.edu/eli/initiatives/key-issues-in-teaching-and-learning

Fairlie, R. (2004). Race and the digital divide. The B.E. Journal of Economic Analysis & Policy , 3 (1). https://doi.org/10.2202/1538-0645.1263 .

Fischer, L., Hilton, J., Robinson, T. J., & Wiley, D. (2015). A Multi-institutional Study of the Impact of Open Textbook Adoption on the Learning Outcomes of Post-secondary Students . Journal of Computing in Higher Education. https://doi.org/10.1007/s12528-015-9101-x .

Floridi, L. (2008). A defence of informational structural realism. Synthese , 161 (2), 219–253.

Article   MathSciNet   Google Scholar  

Floridi, L. (2014). The 4th revolution: How the infosphere is reshaping human reality . Oxford: Oxford University Press.

Garrison, D. R., & Vaughan, N. D. (2013). Blended learning in higher education , (1st ed., ). San Francisco: Jossey-Bass Print.

Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education , 7 , 95–105.

Goodhart, C.A.E. (1975). “Problems of monetary management: The U.K. experience.” Papers in Monetary Economics. Reserve Bank of Australia. I.

Graham, C. R. (2013). Emerging practice and research in blended learning. In M. G. Moore (Ed.), Handbook of distance education , (3rd ed., pp. 333–350). New York: Routledge.

Guttman, L. (1953). Image theory for the structure of quantitative variates. Psychometrika , 18 , 277–296.

Article   MathSciNet   MATH   Google Scholar  

Hilton, J., Fischer, L., Wiley, D., & Williams, L. (2016). Maintaining momentum toward graduation: OER and the course throughput rate. International Review of Research in Open and Distance Learning , 17 (6) https://doi.org/10.19173/irrodl.v17i6.2686 .

IBM Corp. Released (2015). IBM SPSS statistics for windows, version 23.0 . Armonk: IBM Corp.

Jean-François, E. (2013). Transcultural blended learning and teaching in postsecondary education . Hershey: Information Science Reference.

Book   Google Scholar  

Jones, S., Johnson-Yale, C., Millermaier, S., & Pérez, F. S. (2009). U.S. college students’ internet use: Race, gender and digital divides. Journal of Computer-Mediated Communication , 14 (2), 244–264 https://doi.org/10.1111/j.1083-6101.2009.01439.x .

Kaiser, H. F., & Rice, J. (1974). Little Jiffy, Mark IV. Journal of Educational and Psychological Measurement , 34(1), 111–117.

Kaufmann, S. (2000). Investigations . New York: Oxford University Press.

Kitchenham, A. (2011). Blended learning across disciplines: Models for implementation . Hershey: Information Science Reference.

Lakoff, G. (2012). Women, fire, and dangerous things: What categories reveal about the mind . Chicago: The University of Chicago Press.

Lewis, L., & Parsad, B. (2008). Distance education at degree-granting postsecondary institutions : 2006–07 (NCES 2009–044) . Washington: Retrieved from http://nces.ed.gov/pubs2009/2009044.pdf .

Liu, F., & Cavanaugh, C. (2011). High enrollment course success factors in virtual school: Factors influencing student academic achievement. International Journal on E-Learning , 10 (4), 393–418.

Lowe, D. (2013). Roadmap of a blended learning model for online faculty development. Invited feature article in Distance Education Report , 17 (6), 1–7.

Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record , 115 (3), 1–47.

Means, B., Toyama, Y., Murphy, R., Kaia, M., & Jones, K. (2010). Evaluation of evidence-based practices in online learning . Washington: US Department of Education.

Moskal, P., Dziuban, C., & Hartman, J. (2013). Blended learning: A dangerous idea? The Internet and Higher Education , 18 , 15–23.

Norberg, A. (2017). From blended learning to learning onlife: ICTs, time and access in higher education (Doctoral dissertation, Umeå University).

Norberg, A., Dziuban, C. D., & Moskal, P. D. (2011). A time-based blended learning model. On the Horizon , 19 (3), 207–216. https://doi.org/10.1108/10748121111163913 .

Oliver, M., & Trigwell, K. (2005). Can ‘blended learning’ be redeemed? e-Learning , 2 (1), 17–25.

Olshen, Stone , Steinberg , and Colla (1995). CART classification and regression trees. Tree-structured nonparametric data analysis. Statistical algorithms. Salford systems interface and documentation. Salford Systems .

O'Neil, C. (2017). Weapons of math destruction: How big data increases inequality and threatens democracy . Broadway Books.

Online Learning Consortium. The OLC quality scorecard for blended learning programs. Retrieved from https://onlinelearningconsortium.org/consult/olc-quality-scorecard-blended-learning-programs/

Open SUNY. The OSCQR course design review scorecard. Retrieved from https://onlinelearningconsortium.org/consult/oscqr-course-design-review/

Picciano, A. G. (2009). Blending with purpose: The multimodal model. Journal of Asynchronous Learning Networks , 13 (1), 7–18.

Picciano, A. G., Dziuban, C., & Graham, C. R. (2014). Blended learning: Research perspectives , (vol. 2). New York: Routledge.

Picciano, A. G., & Dziuban, C. D. (2007). Blended learning: Research perspectives . Needham: The Sloan Consortium.

Pirsig, R. M. (1974). Zen and the art of motorcycle maintenance: An inquiry into values . New York: Morrow.

Quality Matters. (2016). About Quality Matters. Retrieved from https://www.qualitymatters.org/research

Robinson, T. J., Fischer, L., Wiley, D. A., & Hilton, J. (2014). The Impact of Open Textbooks on Secondary Science Learning Outcomes . Educational Researcher. https://doi.org/10.3102/0013189X14550275 .

Ross, B., & Gage, K. (2006). Global perspectives on blended learning: Insight from WebCT and our customers in higher education. In C. J. Bonk, & C. R. Graham (Eds.), Handbook of blended learning: Global perspectives, local designs , (pp. 155–168). San Francisco: Pfeiffer.

Rovai, A. P., & Jordan, H. M. (2004). Blended learning and sense of community: A comparative analysis with traditional and fully online graduate courses. International Review of Research in Open and Distance Learning , 5 (2), 1–13.

Searle, J. R. (2015). Seeing things as they are: A theory of perception . Chicago: Oxford University Press.

Sharpe, R., Benfield, G., Roberts, G., & Francis, R. (2006). The undergraduate experience of blended learning: A review of UK literature and research. The Higher Education Academy, (October 2006).

Shea, P., & Bidjerano, T. (2014). Does online learning impede degree completion? A national study of community college students. Computers and Education , 75 , 103–111 https://doi.org/10.1016/j.compedu.2014.02.009 .

Shea, P., & Bidjerano, T. (2016). A National Study of differences between distance and non-distance community college students in time to first associate degree attainment, transfer, and dropout. Online Learning , 20 (3), 14–15.

Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology , 59 (3), 623–664.

Smith, L. A. (2007). Chaos: a very short introduction . Oxford: Oxford University Press.

Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, translations and boundary objects: Amatuers and professionals in Berkely’s Museum of Vertebrate Zoology, 1907-39. Social Studies of Science , 19 (3), 387–420.

Stark, P. & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen. Retrieved from https://www.stat.berkeley.edu/~stark/Preprints/evaluations14.pdf .

Tynan, B., Ryan, Y., & Lamont-Mills, A. (2015). Examining workload models in online and blended teaching. British Journal of Educational Technology , 46 (1), 5–15.

Uttl, B., White, C. A., & Gonzalez, D. W. (2016). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation , 54 , 22–42.

Williams, J. (2016). College and the new class divide. Inside Higher Ed July 11, 2016.

Wladis, C., Hachey, A. C., & Conway, K. (2015). Which STEM majors enroll in online courses, and why should we care? The impact of ethnicity, gender, and non-traditional student characteristics. Computers and Education , 87 , 285–308 https://doi.org/10.1016/j.compedu.2015.06.010 .

Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, H. S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record , 107 (8), 1836–1884. https://doi.org/10.1111/j.1467-9620.2005.00544.x .

Download references

Acknowledgements

The authors acknowledge the contributions of several investigators and course developers from the Center for Distributed Learning at the University of Central Florida, the McKay School of Education at Brigham Young University, and Scholars at Umea University, Sweden. These professionals contributed theoretical and practical ideas to this research project and carefully reviewed earlier versions of this manuscript. The Authors gratefully acknowledge their support and assistance.

Author information

Authors and affiliations.

University of Central Florida, Orlando, Florida, USA

Charles Dziuban, Patsy D. Moskal & Nicole Sicilia

Brigham Young University, Provo, Utah, USA

Charles R. Graham

Campus Skellefteå, Skellefteå, Sweden

Anders Norberg

You can also search for this author in PubMed   Google Scholar

Contributions

The Authors of this article are listed in alphabetical order indicating equal contribution to this article. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Patsy D. Moskal .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Student Perception of Instruction

Instructions: Please answer each question based on your current class experience. You can provide additional information where indicated.

All responses are anonymous. Responses to these questions are important to help improve the course and how it is taught. Results may be used in personnel decisions. The results will be shared with the instructor after the semester is over.

Please rate the instructor’s effectiveness in the following areas:

Organizing the course:

Excellent b) Very Good c) Good d) Fair e) Poor

Explaining course requirements, grading criteria, and expectations:

Communicating ideas and/or information:

Showing respect and concern for students:

Stimulating interest in the course:

Creating an environment that helps students learn:

Giving useful feedback on course performance:

Helping students achieve course objectives:

Overall, the effectiveness of the instructor in this course was:

What did you like best about the course and/or how the instructor taught it?

What suggestions do you have for improving the course and/or how the instructor taught it?

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Dziuban, C., Graham, C.R., Moskal, P.D. et al. Blended learning: the new normal and emerging technologies. Int J Educ Technol High Educ 15 , 3 (2018). https://doi.org/10.1186/s41239-017-0087-5

Download citation

Received : 09 October 2017

Accepted : 20 December 2017

Published : 15 February 2018

DOI : https://doi.org/10.1186/s41239-017-0087-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Blended learning
• Higher education
• Student success
• Student perception of instruction

‘Explicit Instruction’ Provides Dramatic Benefits in Learning to Read

Summary: When it comes to learning to read, new research suggests that explicit instruction—a phonics teaching method in which the relationship between sound and spelling is taught directly and systematically—is more effective than self-discovery through reading.

The ability to read is foundational to education, but prolonged school closures and distance learning due to the pandemic have imposed unique challenges on the teaching of many fundamental skills. When in-person classes resume, many students will likely need a period of catch-up learning, especially those who lag behind in basic reading skills.

New research published in the journal Psychological Science shows that people who were taught to read by receiving explicit instructions on the relationship between sounds and spelling experienced a dramatic improvement compared to learners who discovered this relationship naturally through the reading process. These results contribute to an ongoing debate about how best to teach children to read.

A team of researchers from Royal Holloway, University of London, tested both techniques on a group of 48 adults who, over an intensive two-week period, were taught to read a new language that was printed in unfamiliar symbols.

“The provision of evidence-based instructional methods has never been more important. Our research highlights the significance of explicit instruction in ensuring that all pupils have the opportunity to develop strong reading skills.” — K. Rastle, Royal Holloway

One half of the participants learned spelling-to-sound and spelling-to-meaning regularities solely through experience with reading the novel words during training. The other half received a brief session of explicit instruction on these regularities before training commenced. At the end of the two-week period, both groups were given reading tests to gauge how well they had learned the new language.

“Our results were really striking. By the end of the two weeks, virtually all learners who had received explicit instruction were able to read words printed in the unfamiliar symbols,” said Kathleen Rastle, a researcher at Royal Holloway and lead author on the paper.

In contrast, despite up to 18 hours of experience with the new language, less than 25% of the “discovery learners” reached the same standard, and some showed very poor learning.

“Reading is the foundation for children’s learning throughout their schooling; for this reason, the learning loss that we are seeing is very concerning and has the potential for lifelong consequences,” said Rastle. “The provision of evidence-based instructional methods has never been more important. Our research highlights the significance of explicit instruction in ensuring that all pupils have the opportunity to develop strong reading skills.”

Psychological Science , the flagship journal of APS, is the leading peer-reviewed journal publishing empirical research spanning the entire spectrum of the science of psychology. For a copy of this article, contact [email protected] .

Reference: Rastle, K., Lally, C., Davis, M., & Taylor, J. S. H. (2021). The Dramatic Impact of Explicit Instruction on Learning to Read in a New Writing System. Psychological Science . Advance online publication. https://doi.org/10.1177/0956797620968790

APS regularly opens certain online articles for discussion on our website. Effective February 2021, you must be a logged-in APS member to post comments. By posting a comment, you agree to our Community Guidelines and the display of your profile information, including your name and affiliation. Any opinions, findings, conclusions, or recommendations present in article comments are those of the writers and do not necessarily reflect the views of APS or the article’s author. For more information, please see our Community Guidelines .

Please login with your APS account to comment.

The Value of Values in Poverty Reduction

Anti-poverty programs in the developing world often sputter because they clash with local culture and values. But researchers are finding success with programs that align with the tenets of the communities they’re trying to help.

Inside Grants: Institute of Education Sciences Grant Funding

The Institute of Education Sciences and Educational Testing Services collaboration research grant is a $1,399,520 effort toward identifying evidence of collaborative problem solving skills.

APS Expands Career Resources, Professional Development in 2023

A new series of educational workshops and trainings in 2023 is designed to help psychological scientists further their careers inside and outside of academia, while fostering a more transparent and valid science on the global stage. 

Privacy Overview

What Research Says About . . . / Differentiated Learning

What we know, what you can do, educators take note.

Researchers at the National Center on Accessing the General Curriculum define differentiated instruction asa process to approach teaching and learning for students of differing abilities in the same class. The intent is to maximize each student's growth and individual success by meeting each student where he or she is . . . rather than expecting students to modify themselves for the curriculum. (Hall, 2002)

• Focus on the essential ideas and skills of the content area, eliminating ancillary tasks and activities.
• Respond to individual student differences (such as learning style, prior knowledge, interests, and level of engagement).
• Group students flexibly by shared interest, topic, or ability.
• Integrate ongoing and meaningful assessments with instruction.
• Continually assess; reflect; and adjust content, process, and product to meet student needs.

Allan, S. D., &amp; Tomlinson, C. A. (2000). Leadership for differentiating schools and classrooms . Alexandria, VA: ASCD.

Anderson, K. M., (2007). Differentiating instruction to include all students. Preventing School Failure, 51 (3), 49–54.

Baumgartner, T., Lipowski, M. B., &amp; Rush, C. (2003). Increasing reading achievement of primary and middle school students through differentiated instruction (Master's research). Available from Education Resources Information Center (ERIC No. ED479203).

Ellis, E. S., &amp; Worthington, L. A. (1994). Research synthesis on effective teaching principles and the design of quality tools for educators (Technical Report No. 5). Eugene: University of Oregon, National Center to Improve the Tools of Educators.

Hall, T. (2002). Differentiated instruction [Online]. Wakefield, MA: CAST. Available: www.cast.org/publications/ncac/ncac_diffinstruc.html

Lawrence-Brown, D. (2004). Differentiated instruction: Inclusive strategies for standards-based learning that benefit the whole class. American Secondary Education 32 (3), 34.

McQuarrie, L., McRae, P., &amp; Stack-Cutler, H. (2008). Differentiated instruction provincial research review . Edmonton: Alberta Initiative for School Improvement.

Rock, M., Gregg, M., Ellis, E., &amp; Gable, R. A. (2008). REACH: A framework for differentiating classroom instruction. Preventing School Failure, 52 (2), 31–47.

Tieso, C. (2005). The effects of grouping practices and curricular adjustments on achievement. Journal for the Education of the Gifted, 29 (1), 60–89.

Tomlinson, C. A. (1999). Leadership for differentiated classrooms. The School Administrator, 56 (9), 6–11.

Tomlinson, C. A. (2000). Differentiation of instruction in the elementary grades. ERIC Digest . Available: www.ericdigests.org/2001-2/elementary.html

Tomlinson, C., &amp; Kalbfleisch, M. L. (1998). Teach me, teach my brain: A call for differentiated classrooms. Educational Leadership, 56 (3), 52–55.

Tomlinson, C. A., &amp; Strickland, C. A. (2005). Differentiation in practice: A resource guide for differentiating curriculum, grades 9–12 . Alexandria, VA: ASCD.

Vaughn, S., Bos, C., &amp; Schumm, J. (2000). Teaching exceptional, diverse, and at-risk students in the general education classroom (2nd ed.). Boston: Allyn and Bacon.

Vygotsky, L. S., (1978). Mind in society: The development of higher psychological processes . Cambridge, MA: Harvard University Press.

ASCD is a community dedicated to educators' professional growth and well-being.

Let us help you put your vision into action., from our issue.

To process a transaction with a Purchase Order please send to [email protected]

Exploring the impact of the adaptive gamified assessment on learners in blended learning

• Open access
• Published: 06 May 2024

Cite this article

You have full access to this open access article

• Zhihui Zhang   ORCID: orcid.org/0000-0002-0277-4937 1 &
• Xiaomeng Huang   ORCID: orcid.org/0009-0008-6864-5821 2  

Blended learning combines online and traditional classroom instruction, aiming to optimize educational outcomes. Despite its potential, student engagement with online components remains a significant challenge. Gamification has emerged as a popular solution to bolster engagement, though its effectiveness is contested, with research yielding mixed results. This study addresses this gap by examining the impact of adaptive gamified assessments on young learners' motivation and language proficiency within a blended learning framework. Under Self-Determination Theory (SDT) and Language Assessment Principles, the study evaluates how adaptive gamified tests affect learner engagement and outcomes. A 20-week comparative experiment involved 45 elementary school participants in a blended learning environment. The experimental group ( n  = 23) took the adaptive gamified test, while the control group ( n  = 22) engaged with non-gamified e-tests. Statistical analysis using a paired t-test in SPSS revealed that the implementation of adaptive gamified testing in the blended learning setting significantly decreased learner dissatisfaction (t (44) = 10.13, p  < .001, SD = 0.87). Moreover, this approach markedly improved learners' accuracy rates (t (44) = -25.75, p  < .001, SD = 2.09), indicating enhanced language proficiency and motivation, as also reflected in the attitude scores (t(44) = -14.47, p  < .001, SD = 4.73). The adaptive gamified assessment primarily enhanced intrinsic motivation related to competence, with 69% of students in the experimental group reporting increased abilities. The findings suggest that adaptive gamified testing is an effective instructional method for fostering improved motivation and learning outcomes.

Similar content being viewed by others

Online learning readiness and attitudes towards gaming in gamified online learning – a mixed methods case study.

EFL learners’ motivation in a gamified formative assessment: The case of Quizizz

Game-Play: Effects of Online Gamified and Game-Based Learning on Dispositions, Abilities and Behaviours of Primary Learners

Avoid common mistakes on your manuscript.

1 Introduction

In the rapidly evolving landscape of educational technology, blended learning (BL) has become a prominent approach, seamlessly integrating face-to-face and online learning experiences (Hill & Smith, 2023 ; Rasheed et al., 2020 ). While previous research has emphasized the widespread adoption and benefits of BL, including improved academic achievement (Boelens et al., 2017 ; Hill & Smith, 2023 ), challenges faced by students, teachers, and educational institutions in its implementation are also recognized (Rasheed et al., 2020 ). Designing effective blended learning (BL) presents several key challenges, including diminished learner attention and a decline in motivation, which result in decreased engagement and participation in courses (Khaldi et al., 2023 ). Furthermore, students may face difficulties with preparatory tasks and quizzes prior to in-person classes, often due to inadequate motivation (Albiladi & Alshareef, 2019 ; Boelens et al., 2017 ).

Gamification, defined as the educational use of game mechanics and design principles extending beyond traditional games (Schell, 2008 ), has garnered attention. Studies highlight its positive impact on learning motivation, emphasizing the mediating role of psychological needs under the self-determination theory (Deci & Ryan, 2016 ). This positions gamification as a potential solution for addressing challenges in blended learning. Recent systematic research underscores the significance of leveraging gamification in online environments to enhance student engagement (Bizami et al., 2023 ; Jayawardena et al., 2021 ).

The increasing popularity of gamified tests has positively influenced academia, supporting blended learning models and formal education settings (Bolat & Taş, 2023 ; Saleem et al., 2022 ). Recent findings suggest that gamified assessments contribute to higher process satisfaction among students compared to traditional assessments (Cortés-Pérez et al., 2023 ; Sanchez et al., 2020 ). The advent of machine learning algorithms has given rise to adaptive gamified assessments, offering a novel approach to personalized testing and feedback, thereby enhancing learning autonomy (Llorens-Largo et al., n.d. ). Therefore, this study focuses on investigating the impact of gamified assessment on blended learning.

While existing research has explored the impact of gamification in online environments (Can & Dursun, 2019 ; Ramírez-Donoso et al., 2023 ), a noticeable gap remains in understanding the specific effects of gamified tests in online settings, particularly within the context of K-12 education. Research on adaptive gamified assessments is limited, emphasizing the need for further exploration (Bennani et al., 2022 ). Consequently, this study primarily focuses on investigating adaptive gamified assessments, with research objectives centered around motivation and knowledge levels in early education. The research objectives are outlined as follows:

Does adaptive gamified assessment enhance learners' motivation on blended learning? What is the effect of the adaptive gamified assessment on learners' motivation?

Does adaptive gamified assessment improve learners’ academic performance on blended learning?

To address the challenges present in blended learning, this research contributes to the field by providing insights into the effects of machine learning-based gamified assessments on motivation and performance, offering valuable recommendations for the improvement of blended learning. The findings could also facilitate the design and adoption of blended learning, particularly in the context of K-12 education.

The subsequent sections will delve into a comprehensive literature review, conceptual framework, outline the chosen methodology, present results, and discussions, and conclude with implications and avenues for future research.

2 Literature review

2.1 blended learning challenges and benefits.

Blended learning has emerged as a popular educational model, distinct from traditional instructional methods. It represents a convergence of in-person and online learning, leveraging the strengths of each to enhance the educational experience (Poon, 2013 ). The hybrid approach combines classroom effectiveness and socialization with the technological benefits of online modules, offering a compelling alternative to conventional teaching models. It has identified significant improvements in academic performance attributable to blended learning's efficiency, flexibility, and capacity (Hill & Smith, 2023 ). The approach also facilitates increased interaction between staff and students, promotes active engagement, and provides opportunities for continuous improvement (Can & Dursun, 2019 ).

Despite these advantages, blended learning is not without its challenges, particularly for students, teachers, and educational institutions during implementation. Boelens et al. ( 2017 ) highlight that students often face self-regulation challenges, including poor time management and procrastination. The degree of autonomy in blended courses requires heightened self-discipline, especially online, to mitigate learner isolation and the asynchronous nature of digital interactions (Hill & Smith, 2023 ). Isolation can be a critical issue, as students engaged in pre-class activities such as reading and assignments often do so in solitude, which can lead to a decrease in motivation and an increase in feelings of alienation (Chuang et al., 2018 ; Yang & Ogata, 2023 ).

Teachers, on the other hand, encounter obstacles in technological literacy and competency. Personalizing learning content, providing feedback, and assessing each student can demand considerable time and effort (Cuesta Medina, 2018 ; Bower et al., 2015 ). These challenges can adversely affect teachers' perceptions and attitudes towards technology (Albiladi & Alshareef, 2019 ). Furthermore, from a systems perspective, implementing Learning Management Systems (LMSs) that accommodate diverse learning styles is a significant hurdle. It necessitates a custom approach to effectively support differentiated learning trajectories (Albiladi & Alshareef, 2019 ; Boelens et al., 2017 ; Brown, 2016 ). Current research efforts are thus focused on enhancing the effectiveness of blended learning and its facilitation of independent learning practices.

2.2 Gamification in education

Gamification in education signifies the integration of game design elements into teaching activities that are not inherently game-related. This approach is distinct from game-based learning, where the primary focus is on engaging learners in gameplay to acquire knowledge. Gamification introduces game dynamics into non-gaming environments to enrich the learning experience (Alsawaier, 2018 ).

With the progression of technology, gamification has become increasingly prevalent within educational frameworks, aiming to amplify student engagement, motivation, and interactivity (Oliveira et al., 2023 ). Empirical evidence supports that gamification can effectively address issues such as the lack of motivation and frustration in educational contexts (Alt, 2023 ; Buckley & Doyle, 2016 ). Components like levels and leaderboards have been successful as external motivators, promoting a competitive spirit among learners (Mekler et al., 2017 ). Furthermore, research indicates that gamification can have enduring effects on student participation, fostering beneficial learning behaviors (Alsawaier, 2018 ).

Despite these positive aspects, some scholarly inquiries have presented a more nuanced view, suggesting that gamification does not unilaterally enhance academic outcomes. These varying results invite deeper investigation into the conditions under which gamification can truly enhance the educational experience (Oliveira et al., 2023 ). In light of such findings, recent gamified designs have increasingly emphasized personalization, taking into account the unique characteristics, needs, and preferences of each student. Studies have explored the tailoring of gamification frameworks to align with diverse student profiles (Dehghanzadeh et al., 2023 ; Ghaban & Hendley, 2019 ), learning styles (Hassan et al., 2021 ), pedagogical approaches, and knowledge structures (Oliveira et al., 2023 ). However, the literature still presents contradictory findings, and there is a relative dearth of research focusing on learning outcomes (Oliveira et al., 2023 ).

2.3 Adaptive assessment in education

Adaptive learning harnesses technological advancements to create a supportive educational environment where instructional content is tailored to individual learning processes (Muñoz et al., 2022 ). This pedagogical approach is grounded in the principle of differentiated instruction, allowing for the personalization of educational resources to meet diverse learning requirements (Reiser & Tabak, 2014 ).

Adaptive assessments, stemming from the philosophy of adaptive learning, dynamically adjust the difficulty of questions based on a learner's previous answers, terminating the assessment once enough data is collected to form a judgment (Barney & Fisher Jr, 2016 ). In the digital age, with the proliferation of e-learning, there has been a significant shift towards adaptive computer-based assessments (Muñoz et al., 2022 ), utilizing AI-based modeling techniques (Coşkun, 2023 ), and emotion-based adaptation in e-learning environments (Boughida et al., 2024 ). These assessments are characterized by their ability to modify testing parameters in response to student performance, employing machine learning algorithms to ascertain a student’s proficiency level.

Prior studies on adaptive methods have revealed several advantages, such as delivering personalized feedback promptly, forecasting academic achievement, and facilitating interactive learning support. These advantages extend to potentially enhancing learner engagement and outcomes (Muñoz et al., 2022 ). However, adapting instruction to cater to varied skill levels remains a challenge, as does addressing the issues of demotivation and anxiety among students undergoing assessment (Akhtar et al., 2023 ). Consequently, current research is concentrated on boosting student motivation and engagement in adaptive assessments.

In the field of gamified education, adaptive gamification aims to merge adaptive learning principles with game elements to enrich the learning experience. This approach has been explored through the use of data mining techniques on student logs to foster motivation within adaptive gamified web-based environments (Hassan et al., 2021 ). Despite these innovative efforts, empirical research on gamified adaptive assessment is limited, as the field is still developing.

2.4 Integration of blended learning and gamified assessment

The combination of blended learning with gamified assessment has been recognized for its potential to increase student engagement, a critical factor often lacking in online learning compared to traditional classroom settings (Dumford & Miller, 2018 ; Hill & Smith, 2023 ). Studies investigating the role of gamification within online learning environments have found that it can enhance students’ achievement by fostering greater interaction with content (Taşkın & Kılıç Çakmak, 2023 ). Moreover, gamified activities that demand active participation can promote active engagement (Özhan & Kocadere, 2020 ).

Investigations into the efficacy of Gamified Assessment in online environments suggest that students may reap the benefits of its motivational potential. For instance, research has adapted motivational formative assessment tools from massively multiplayer online role-playing games (MMORPGs) for use in cMOOCs, demonstrating positive outcomes (Danka, 2020 ). Another study compared the effects of traditional online assessment environments to those employing gamified elements, such as point systems, observing the impact on student task completion and quality in mathematics assessments (Attali & Arieli-Attali, 2015 ). Collectively, these studies indicate that gamified tests can indirectly benefit learning by enhancing the instructional content.

While many studies affirm the efficacy of gamified tests as a valuable, cost-effective tool for educators in blended learning environments (Sanchez et al., 2020 ), there is a noted gap in research addressing individual differences within gamified testing. Particularly, empirical research on adaptive gamified assessment is scarce, with more focus on the computational aspects of system development than on the impacts on motivation and academic achievement. Furthermore, while studies suggest that gamified tests may enhance the 'testing effect'—the phenomenon where testing an individual's memory improves long-term retention—most of this research is centered in higher education (Pitoyo & Asib, 2020 ).

The use of gamification spans various educational levels, from primary and secondary schooling to university and lifelong learning programs. However, research focusing on the implementation of gamification in primary and secondary education tends to prioritize the perspective of educators and the application within instructional activities (Yang & Ogata, 2023 ), rather than the online assessment itself. Therefore, this study aims to advance the empirical understanding of the application of gamification in assessments and its potential to improve learning outcomes, particularly in early education.

3 Theoretical framework

3.1 self-determination theory (sdt).

Self-Determination Theory (SDT) is a well-known theory of motivation that offers an in-depth understanding of human needs, motivation, and well-being within social and cultural environments (Chiu, 2021 ). Gamification, which applies gaming elements in non-game settings, frequently employs SDT to address educational challenges in both gamified and online learning platforms (Chapman et al., 2023 ). SDT distinguishes itself by its focus on autonomous versus controlled forms of motivation and the impact of intrinsic and extrinsic motivators, as characterized by Ryan and Deci ( 2000 ). Unlike intrinsic motivation, which is driven by internal desires, extrinsic motivation relies on external incentives such as rewards, points, or deadlines to elicit behaviors—commonly seen in the reward structures of gamified learning environments. In these adaptive gamified assessments, the provision of points and rewards for task completion serves to regulate extrinsic motivation, offering various rewards and titles each time a student completes an exercise task.

SDT is a comprehensive theory that explores the intricacies of human motivation. A subset of SDT, Cognitive Evaluation Theory, postulates that three innate psychological needs—autonomy, competence, and relatedness—propel individuals to act (Deci & Ryan, 2012 ). Autonomy is experienced when individuals feel they have control over their learning journey, making choices that align with their self-identity, such as selecting specific content areas or types of questions in an adaptive gamified assessment. Competence emerges when individuals encounter optimal challenges that match their skills, where adaptive gamified assessments can adjust in difficulty and provide feedback, thereby promoting skill acquisition and mastery. Relatedness is the sense of connection with others, fostered by supportive and engaging learning environments. In gamified contexts, this can be achieved through competitive elements and parental involvement in the learning process, enhancing the learning atmosphere.

The fulfillment of these psychological needs, particularly those of autonomy and competence, is central to fostering intrinsic motivation according to SDT. Figure  1 examines the adaptive gamified assessment process and how it aligns with SDT.

The structure of the adaptive gamified assessment

3.2 Principles of language assessment

The adaptive gamified assessment in this study utilizes Quizizz, an online educational technology platform that offers formative gamified tests to help students develop academic skills in various subjects, including English language (Göksün & Gürsoy, 2019 ). Drawing on the five principles of language assessment as outlined by Brown and Abeywickrama ( 2004 ), this study analyzes the adaptive gamified assessment. These principles—authenticity, practicality, reliability, validity, and washback—are foundational in foreign language teaching and assessment.

Practicality refers to the flexibility of the test to operate without constraints of time, resources, and technical requirements. Quizizz’s adaptive assessments are seamlessly integrated into blended learning environments, designed for time efficiency, and require minimal resources, making them suitable for a broad range of educational contexts. The platform's user-friendly design ensures that assessments are easily administered and completed by students, necessitating only an internet connection and a digital device (Göksün & Gürsoy, 2019 ).

Reliability is the extent to which an assessment consistently yields stable results over time and across different learner groups, providing dependable measures of language proficiency. Quizizz's algorithms adapt task difficulty based on learner responses, offering consistent outcomes and measuring student performance reliably over time (Munawir & Hasbi, 2021 ).

Validity concerns the assessment's ability to accurately measure language abilities in alignment with intended learning outcomes and real-world language application. Quizizz's assessments measure language skills that correlate directly with curriculum-defined learning outcomes, ensuring that results are valid representations of a student's language capabilities. The gamified context also mirrors competitive real-life scenarios, enhancing the authenticity of language use (Priyanti et al., 2019 ).

Authenticity indicates that assessments should mirror real-life language usage, providing tasks that are meaningful and indicative of actual communication situations. Quizizz's assessments incorporate tasks resembling real-world communicative scenarios, such as reading passages, interactive dialogues, and written responses that reflect authentic language use (Brown & Abeywickrama, 2004 ).

Washback refers to the influence of assessments on teaching and learning practices, which should be constructive and foster language learning. Quizizz's immediate feedback from adaptive assessments can positively affect teaching and learning. Instructors can utilize the results to pinpoint student strengths and areas for improvement, customizing their teaching strategies accordingly. Students benefit from being challenged at the appropriate level, bolstering motivation and facilitating the acquisition of new language skills in a gradual, supportive manner (Munawir & Hasbi, 2021 ).

Previous research has demonstrated that Quizizz has a significant impact on academic performance across various educational institutions (Munawir & Hasbi, 2021 ). As an exemplar of gamified adaptive assessment, Quizizz is designed to be practical and reliable while offering valid and authentic assessments of language proficiency. Moreover, it strives for a positive washback effect on the learning process, promoting effective language learning strategies and accommodating personalized education.

4 Methodology

4.1 research design.

This study employed a controlled experimental design within a quantitative research framework. The methodology involved several stages, as illustrated in Fig.  2 . Firstly, participants were selected based on their responses to a pre-questionnaire and a pre-assessment, ensuring comparable baseline levels in English proficiency and computer literacy among all participants. Subsequently, participants were randomly assigned to either the control or the experimental group to ensure variability and minimize bias. Over a period of 20 weeks, a blended language learning intervention was administered to both groups. This intervention involved accessing identical online learning resources before and after traditional classroom sessions, with equal amounts of offline instruction time. Daily assessments were conducted throughout the intervention period. The experimental group completed gamified adaptive tests via Quizizz, while the control group undertook non-gamified adaptive tests on a computer. Upon completion of the intervention, surveys were conducted to assess the motivation levels of both groups and compare their English language proficiency. Data were collected from both pre- and post-assessments, as well as responses from the questionnaires and structured interviews.

Flowchart of the experimental process for assessing the impact of gamified learning on student outcomes

4.2 Participants

Forty-five English learners from primary schools in China, aged 8 to 10 years (M = 9.40, SD = 0.62), participated in this study. The sample comprised 25 girls (55.56%) and 20 boys (44.44%). Insights into students' previous experiences, motivations for formative assessments, and attitudes toward language learning were gathered through a pre-questionnaire. Informed consent was obtained from all participants and their guardians, and confidentiality and anonymity were maintained throughout the study. Participants see in Fig.  3 were randomly divided into a control group (n = 22; 12 girls and 10 boys) and an experimental group (n = 23; 13 girls and 10 boys). The experimental group received instructions on completing and utilizing the adaptive gamified assessment, Quizizz, while the control group completed non-gamified adaptive tests on a computer. Both groups adopted the same blended learning model and were informed of identical deadlines for weekly formative assessments, requiring an accuracy rate of over 90%. Immediate feedback was provided on the accuracy rates, and participants were informed they could attempt the assessment again if the target was not met.

Comparison of number and gender ratio in two groups

4.3 Instruments

The study utilized Quizizz's Adaptive Question Bank mode, offering a range of question difficulties and allowing students to progress at their own pace. The questionnaire was adapted from the Student Evaluation of Educational Quality (SEEQ), which has demonstrated a high level of reliability, with Cronbach's alpha ranging from 0.88 to 0.97. Additionally, according to Pecheux and Derbaix ( 1999 ), the questionnaire was designed to be as concise as possible for young learners and was administered in their native language, Chinese.

The content of the questionnaire includes a 5-point Likert scale used to measure students' attitudes toward adaptive gamified tests. The response options are as follows: strongly agree = 9, agree = 7, neutral = 5, disagree = 3, and strongly disagree = 1. The statements cover various aspects of gamified testing, including Engagement and Enjoyment, exemplified by 'You enjoy learning new words through game tests. Game tests make learning grammar and vocabulary more fun for you.' Anxiety and Confidence, as indicated by 'Game tests help you feel less worried about making mistakes in learning.' Understanding and Retention, highlighted by 'Playing educational games helps you understand your lessons better.' And preference over traditional testing methods, as shown by 'You prefer taking quizzes when they are like games compared to regular tests.' This total score will provide a cumulative measure of their attitude toward gamified language tests. In addition, there are questions asking participants to express their overall satisfaction with the blended learning experience as a percentage. This metric is instrumental in assessing the role of gamified testing within the blended learning framework. Furthermore, there are specific aspects of gamification: binary yes/no questions that delve into specific components and potential effects of gamified tests, such as the impact of leaderboards and rewards on motivation, and willingness to spend extra time on gamified tests.

Moreover, to explore the impact of adaptive gamified assessment on motivation, structured interviews were conducted with the experimental group. The questions, adapted from Chiu ( 2021 ), primarily focused on aspects of motivation such as amotivation, external regulation, intrinsic motivation, and the psychological needs related to relatedness, autonomy, and competency, as seen in Table  1 . Responses were quantified on the same Likert scale, with options ranging from 'strongly agree' to 'strongly disagree.'

5 Results and discussion

5.1 comparison language learning attitude scores and satisfaction of participants.

To analyze the impact of adaptive gamified assessments on learners, the trajectory of language learning attitude scores and satisfaction percentage for two groups over the course of the experiment was explored, with results depicted in Fig. 4 and Fig. 5.

In Fig.  4 , the total score of language learning attitude for the control group's online assessment and the experimental group's adaptive gamified assessment demonstrates an increasing trend as the experiment progressed. After 4 weeks, the language learning attitude scores of the control and experimental groups were 10 and 47, respectively. By week 16, the experimental group's score increased to 70, and after 20 weeks, the control group's score was 50, while the experimental group's score reached 75. A paired-samples t-test conducted via SPSS indicated that the attitude scores were significantly higher in the experimental group than in the control group (t(44) = -14.47, p  < 0.001, SD = 4.73), as detailed in Table  2 . This significant difference in attitude scores demonstrates the effectiveness of the adaptive gamified assessment in enhancing the language learning attitude of students over the duration of the experiment.

Change of language learning attitude scores of two groups

Figure  5 reveals that as the experiment progressed, the students' dissatisfaction rates with gamification online learning decreased significantly in both groups. Initially, after 4 weeks, the average dissatisfaction rate for the control and experimental groups was 11% and 6%, respectively. As the experiment continued, the dissatisfaction rates declined, dropping to about 5% in the experimental group and 8% in the control group after 20 weeks. Paired t-test results further show a significant decrease in dissatisfaction (t(44) = 10.13, p  < 0.001, SD = 0.87). This suggests a marked downward trend in students' dissatisfaction with gamified online learning over the duration of the study, in accordance with their attitudes towards adaptive gamified assessment.

Variation curve of dissatisfaction rate of gamification in two groups

Our research found that students maintain a positive attitude towards the blended learning model of online assessment, which aligns with previous research (Abduh, 2021 ; Albiladi & Alshareef, 2019 ), indicating that e-assessment can benefit online learning and teaching. However, a deeper comparison between non-gamified and gamified adaptive testing groups in terms of satisfaction and students' subjective perceptions reveals differences. The experimental group, which incorporated gamified adaptive testing, demonstrated a more positive attitude, corroborating the positive role of gamification in education as outlined by Bolat and Taş ( 2023 ). Gamified assessment promotes student motivation in a manner consistent with previous research (Bennani et al., 2022 ), and our study has similarly shown that gamified assessment positively influences learners' behaviors and attitudes (Özhan & Kocadere, 2020 ).

This result appears to contradict the findings of Kwon and Özpolat ( 2021 ), which suggest that gamification of assessments had a significantly adverse effect on students' perceptions of satisfaction and their experience of courses in higher education. Our findings, however, indicate that adaptive gamified assessments enhance motivation and engagement, thus contributing positively to the learning process for young learners. Furthermore, the motivational levels in the experimental group remained stable, whereas motivation in the control group decreased over time. This suggests that adaptive gamified assessments may help to sustain or enhance learner motivation within a blended learning environment.

5.2 Effect of adaptive gamified assessment on learners' motivation

To further examine the effect of adaptive gamified assessments, the standard error of dissatisfaction for both groups was evaluated, while also including a statistical analysis of the distribution of motivation within the experimental group. The outcomes of these analyses are depicted in Fig.  6 .

Changing Curves of Satisfaction of Standard Errors of Two Groups

In Fig.  6 , a notable decrease in standard error scores for both the control and experimental groups is observed as the experiment progresses. Initially, after 4 weeks, the standard error scores stood at 8 for the control group and 5 for the experimental group. At the end of the 20-week study period, these scores had diminished to 5.4 and 2.8, respectively.

This study's findings are consistent with previous research on the benefits of personalization in gamification. Rodrigues et al. ( 2021 ) reported that personalized gamification mitigated negative perceptions of common assessment activities while simultaneously motivating and supporting learners. This reinforces the pivotal role of adaptive assessment in tailoring learning experiences compared to traditional e-assessment methods. Furthermore, structured interviews conducted with the experimental group revealed the distribution of students' motivation in Table  3 . For younger learners, external motivation induced by gamified testing was found to be predominant, with 73% of the children acknowledging its influence. Notably, the tests' impact on students' intrinsic motivation was also significant, especially regarding the sense of competency; 69% of students reported feeling an enhancement in their abilities. This finding presents a nuanced difference from Dahlstrøm's ( 2012 ) proposition that gamified products and services could both facilitate and undermine intrinsic motivation through supporting or neglecting the basic psychological needs for autonomy and competence. It suggests an alternate conclusion: the gamified adaptive assessment enhances intrinsic motivation and participation. Of course, the effectiveness of such interventions is significantly dependent on individual and contextual factors, thus highlighting the adaptive gamified approach's role in effectively moderating these effects.

5.3 Impact of adaptive gamified assessment on academic performance

To evaluate the impact of adaptive gamified assessment on learners’ academic performance, the errors and system score data from the model tests of different groups were organized. Figure  7 depicts the error variation of the system model test, while Fig.  8 analyzes the change curve of the system’s average score data.

Variation curves of test errors of different models in two groups

Change curve of average learning scores of learners in Two Groups

Figure  7 demonstrates that systematic errors in model testing for both groups exhibited a decreasing trend over the course of the experiment. Initially, after 4 weeks, the model test errors were 22% for the control group and 23% for the experimental group. Following 16 weeks, both groups reached a minimum test error value of 3%. However, after 20 weeks, a rebound and increasing trend in model test errors were observed in both groups. Consequently, setting the experiment duration to 16 weeks appears to effectively improve the accuracy of the gamified assessment. A paired-samples t-test in Table  4 indicates a significant reduction in standard error (t(44) = -25.75, p  < 0.001, SD = 2.09), reinforcing the effectiveness of the adaptive gamified strategy optimization in reducing learning standard errors and, consequently, improving learners' efficiency and knowledge acquisition.

As shown in Fig.  8 , the average learning scores of students in both groups increased as the experiment progressed. After four weeks, the average learning score was 25 for the control group and 48 for the experimental group. After 16 weeks, these scores increased to 36 and 66, respectively. By week 20, the average score for the experimental group slightly decreased to 63. This indicates that learners' average scores in different experimental groups peaked after 16 weeks. A comprehensive evaluation, which included a comparison of average learning scores and standard deviation (SD) changes, was used to assess the impact of the gamified assessment. The results are detailed in Table  5 .

These comparisons reveal that adaptive gamified assessments can enhance students' online learning experiences. This supports the findings of Attali and Arieli-Attali ( 2015 ), who demonstrated that under a points-gamification condition, participants, particularly eighth graders, showed higher likeability ratings. Additionally, the effect of gamified assessment on students' final scores was mediated by intrinsic motivation levels. This contrasts with previous studies on gamification in education, such as Alsawaier ( 2018 ), which indicated that students in gamified courses exhibited less motivation and lower final exam scores than those in non-gamified classes. Furthermore, the element of peer competition introduced by gamification was more meaningful to students with better results, aligning with the findings of Göksün and Gürsoy ( 2019 ). Adaptive gamified tests, serving as a formative assessment platform, have been found to positively influence young learners' learning outcomes. Moreover, gamified testing could reduce language anxiety, consistent with the study by Hong et al. ( 2022 ). Compared to traditional gamified assessments, adaptive assessments are better equipped to address issues of repetition and learner capability fit, and they align more closely with the principles of scaffolding in education, thereby enhancing students' academic performance.

6 Conclusion

This research explores the influence of adaptive gamified assessment within a blended learning context on young learners' motivation and academic performance. Grounded in Self-Determination Theory (SDT), this investigation categorizes student motivation and analyzes their engagement and learning capabilities in relation to non-gamified and gamified adaptive tests. The findings suggest that the gamified adaptive test can significantly help learners improve their motivation and foster enhanced language proficiency performance in a blended learning environment.

The study verifies the enhancing effect of gamified evaluation on the internalization of students' motivation (Özhan & Kocadere, 2020 ) and confirms the regulatory role of gamified elements in blended learning, aiding in increasing student participation and satisfaction (Jayawardena et al., 2021 ). Furthermore, the positive role of gamification in language learning and as a tool for reinforcing assessment is corroborated (Priyanti et al., 2019 ). This study extends our understanding of the motivational impacts of gamification in younger education settings, suggesting that while previous research indicated a lesser effect on intrinsic motivation for young learners (Mekler et al., 2017 ), the adaptive mode of gamified assessment could enhance students' sense of competency and, thereby, their intrinsic motivation. Additionally, this research integrates the relationship between motivation and academic level, suggesting that the transition from external motivations provided by rewards in adaptive gamified assessments to enjoying personalized feedback and growth can enhance satisfaction in blended learning, facilitating the internalization of motivation towards participation and language proficiency.

In terms of managerial and policy implications, the introduction of gamification into blended learning environments is advisable, not only as a teaching method but also as an assessment tool. Gamified assessment, with its interactive nature, can be used to alleviate negative impacts of language learning, such as anxiety and lack of confidence, especially for young learners who may benefit from guided external motivational factors. Educators should implement a variety of formative assessments using technology in evaluation activities, especially to promote active learning.

However, the short duration of the experiment and the limited sample size are insufficient to substantiate long-term positive effects of gamification. Future research should delve into a more nuanced examination of students' intrinsic motivation, with longitudinal tracking to observe the internalization of motivation. The inclusion of delayed tests can help study the long-term effects of gamification. Further research could also compare adaptive gamified experiments with gamified experiments to enhance understanding of how gamification influences the internalization of students' intrinsic motivation.

Data availability

All data and questionnaires seen in the attachment.

Data will be available on request due to privacy/ethical restrictions.

Code availability

Not Applicable.

Abduh, M. Y. M. (2021). Full-time online assessment during COVID-19 lockdown: EFL teachers’ perceptions. Asian EFL Journal, 28 (1.1), 26–46.

Google Scholar  

Akhtar, H., Silfiasari, Vekety, B., & Kovacs, K. (2023). The effect of computerized adaptive testing on motivation and anxiety: a systematic review and meta-analysis. Assessment, 30 (5), 1379–1390.

Article   Google Scholar  

Albiladi, W. S., & Alshareef, K. K. (2019). Blended learning in English teaching and learning: A review of the current literature. Journal of Language Teaching and Research, 10 (2), 232–238.

Alsawaier, R. S. (2018). The effect of gamification on motivation and engagement. The International Journal of Information and Learning Technology, 35 (1), 56–79.

Alt, D. (2023). Assessing the benefits of gamification in mathematics for student gameful experience and gaming motivation. Computers & Education, 200 , 104806.

Attali, Y., & Arieli-Attali, M. (2015). Gamification in assessment: Do points affect test performance? Computers & Education, 83 , 57–63.

Barney, M., & Fisher, W. P., Jr. (2016). Adaptive measurement and assessment. Annual Review of Organizational Psychology and Organizational Behavior, 3 , 469–490.

Bennani, S., Maalel, A., & Ben Ghezala, H. (2022). Adaptive gamification in E-learning: A literature review and future challenges. Computer Applications in Engineering Education, 30 (2), 628–642.

Bizami, N. A., Tasir, Z., & Kew, S. N. (2023). Innovative pedagogical principles and technological tools capabilities for immersive blended learning: A systematic literature review. Education and Information Technologies, 28 (2), 1373–1425.

Boelens, R., De Wever, B., & Voet, M. (2017). Four key challenges to the design of blended learning: A systematic literature review. Educational Research Review, 22 , 1–18.

Bolat, Y. I., & Taş, N. (2023). A meta-analysis on the effect of gamified-assessment tools’ on academic achievement in formal educational settings. Education and Information Technologies, 28 (5), 5011–5039.

Boughida, A., Kouahla, M. N., & Lafifi, Y. (2024). emoLearnAdapt: A new approach for an emotion-based adaptation in e-learning environments.  Education and Information Technologies , 1–55. https://doi.org/10.1007/s10639-023-12429-6

Bower, M., Dalgarno, B., Kennedy, G., Lee, M. J. W., & Kenney, J. (2015). Design and implementation factors in blended synchronous learning environments: Outcomes from a cross-case analysis. Computers and Education/Computers & Education, 86 , 1–17.

Brown, M. G. (2016). Blended instructional practice: A review of the empirical literature on instructors’ adoption and use of online tools in face-to-face teaching. The Internet and Higher Education, 31 , 1–10.

Brown, H. D., & Abeywickrama, P. (2004). Language assessment. Principles and Classroom Practices . Pearson Education.

Buckley, P., & Doyle, E. (2016). Gamification and student motivation. Interactive Learning Environments, 24 (6), 1162–1175.

Can, M. E. S. E., & Dursun, O. O. (2019). Effectiveness of gamification elements in blended learning environments. Turkish Online Journal of Distance Education, 20 (3), 119–142.

Chapman, J. R., Kohler, T. B., Rich, P. J., & Trego, A. (2023). Maybe we’ve got it wrong. An experimental evaluation of self-determination and Flow Theory in gamification. Journal of Research on Technology in Education , 1-20. https://doi.org/10.1080/15391523.2023.2242981

Chiu, T. K. (2021). Digital support for student engagement in blended learning based on self-determination theory. Computers in Human Behavior, 124 , 106909.

Chuang, H. H., Weng, C. Y., & Chen, C. H. (2018). Which students benefit most from a flipped classroom approach to language learning? British Journal of Educational Technology, 49 (1), 56–68.

Cortés-Pérez, I., Zagalaz-Anula, N., López-Ruiz, M. D. C., Díaz-Fernández, Á., Obrero-Gaitán, E., & Osuna-Pérez, M. C. (2023). Study based on gamification of tests through Kahoot!™ and reward game cards as an innovative tool in physiotherapy students: A preliminary study. Healthcare, 11 (4), 578. https://doi.org/10.3390/healthcare11040578

Coşkun, L. (2023). An advanced modeling approach to examine factors affecting preschool children’s phonological and print awareness.  Education and Information Technologies , 1–28. https://doi.org/10.1007/s10639-023-12216-3

Cuesta Medina, L. (2018). Blended learning: Deficits and prospects in higher education. Australasian Journal of Educational Technology, 34 (1).  https://doi.org/10.14742/ajet.3100

Dahlstrøm, C. (2012). Impacts of gamification on intrinsic motivation.  Education and Humanities Research , 1–11.

Danka, I. (2020). Motivation by gamification: Adapting motivational tools of massively multiplayer online role playing games (MMORPGs) for peer-to-peer assessment in connectivist massive open online courses (cMOOCs). International Review of Education, 66 (1), 75–92.

Deci, E. L., & Ryan, R. M. (2016). Optimizing students’ motivation in the era of testing and pressure: A self-determination theory perspective. In  Building autonomous learners  (pp. 9–29).  https://doi.org/10.1007/978-981-287-630-0_2

Deci, E. L., & Ryan, R. M. (2012). Self-determination theory. Handbook of Theories of Social Psychology, 1 (20), 416–436.

Dehghanzadeh, H., Farrokhnia, M., Dehghanzadeh, H., Taghipour, K., & Noroozi, O. (2023). Using gamification to support learning in K‐12 education: A systematic literature review.  British Journal of Educational Technology, 55 (1), 34–70. https://doi.org/10.1111/bjet.133

Dumford, A. D., & Miller, A. L. (2018). Online learning in higher education: Exploring advantages and disadvantages for engagement. Journal of Computing in Higher Education, 30 , 452–465.

Ghaban, W., & Hendley, R. (2019). How different personalities benefit from gamification. Interacting with Computers, 31 (2), 138–153.

Göksün, D. O., & Gürsoy, G. (2019). Comparing success and engagement in gamified learning experiences via Kahoot and Quizizz. Computers & Education, 135 , 15–29.

Hassan, M. A., Habiba, U., Majeed, F., & Shoaib, M. (2021). Adaptive gamification in e-learning based on students’ learning styles. Interactive Learning Environments, 29 (4), 545–565.

Hill, J., & Smith, K. (2023). Visions of blended learning: Identifying the challenges and opportunities in shaping institutional approaches to blended learning in higher education. Technology, Pedagogy and Education, 32 (3), 289–303.

Hong, J. C., Hwang, M. Y., Liu, Y. H., & Tai, K. H. (2022). Effects of gamifying questions on English grammar learning mediated by epistemic curiosity and language anxiety. Computer Assisted Language Learning, 35 (7), 1458–1482.

Jayawardena, N. S., Ross, M., Quach, S., Behl, A., & Gupta, M. (2021). Effective online engagement strategies through gamification: A systematic literature review and a future research agenda. Journal of Global Information Management (JGIM), 30 (5), 1–25.

Khaldi, A., Bouzidi, R., & Nader, F. (2023). Gamification of e-learning in higher education: a systematic literature review. Smart Learning Environments, 10 (1). https://doi.org/10.1186/s40561-023-00227-z

Kwon, H. Y., & Özpolat, K. (2021). The dark side of narrow gamification: Negative impact of assessment gamification on student perceptions and content knowledge. INFORMS Transactions on Education, 21 (2), 67–81.

Llorens-Largo, F., Gallego-Durán, F. J., Villagrá-Arnedo, C. J., Compañ-Rosique, P., Satorre-Cuerda, R., & Molina-Carmona, R. (2016). Gamification of the learning process: Lessons learned. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 11 (4), 227–234. https://doi.org/10.1109/rita.2016.2619138

Mekler, E. D., Brühlmann, F., Tuch, A. N., & Opwis, K. (2017). Towards understanding the effects of individual gamification elements on intrinsic motivation and performance. Computers in Human Behavior, 71 , 525–534.

Munawir, A., & Hasbi, N. P. (2021). The effect of using quizizz to efl students’engagement and learning outcome. English Review: Journal of English Education, 10 (1), 297–308.

Muñoz, J. L. R., Ojeda, F. M., Jurado, D. L. A., Peña, P. F. P., Carranza, C. P. M., Berríos, H. Q., … & Vasquez-Pauca, M. J. (2022). Systematic review of adaptive learning technology for learning in higher education.  Eurasian Journal of Educational Research ,  98 (98), 221–233.

Oliveira, W., Hamari, J., Shi, L., Toda, A. M., Rodrigues, L., Palomino, P. T., & Isotani, S. (2023). Tailored gamification in education: A literature review and future agenda. Education and Information Technologies, 28 (1), 373–406.

Özhan, ŞÇ., & Kocadere, S. A. (2020). The effects of flow, emotional engagement, and motivation on success in a gamified online learning environment. Journal of Educational Computing Research, 57 (8), 2006–2031.

Pecheux, C., & Derbaix, C. (1999). Children and attitude toward the brand: A new measurement scale. Journal of Advertising Research, 39 (4), 19–19.

Pitoyo, M. D., & Asib, A. (2020). Gamification-Based Assessment: The Washback Effect of Quizizz on Students’ Learning in Higher Education. International Journal of Language Education, 4 (1), 1–10.

Poon, J. (2013). Blended learning: An institutional approach for enhancing students’ learning experiences. Journal of Online Learning and Teaching, 9 (2), 271.

Priyanti, N. W. I., Santosa, M. H., & Dewi, K. S. (2019). Effect of quizizz towards the eleventh-grade english students’reading comprehension in mobile learning context. Language and Education Journal Undiksha, 2 (2), 71–80.

Ramírez-Donoso, L., Pérez-Sanagustín, M., Neyem, A., Alario-Hoyos, C., Hilliger, I., & Rojos, F. (2023). Fostering the use of online learning resources: Results of using a mobile collaboration tool based on gamification in a blended course. Interactive Learning Environments, 31 (3), 1564–1578.

Rasheed, R. A., Kamsin, A., & Abdullah, N. A. (2020). Challenges in the online component of blended learning: A systematic review. Computers & Education, 144 , 103701.

Reiser, B. J., & Tabak, I. (2014). Scaffolding. In  The Cambridge Handbook of the Learning Sciences  (pp. 44–62). https://doi.org/10.1017/cbo9781139519526.005

Rodrigues, L., Palomino, P. T., Toda, A. M., Klock, A. C., Oliveira, W., Avila-Santos, A. P., … & Isotani, S. (2021). Personalization improves gamification: Evidence from a mixed-methods study.  Proceedings of the ACM on Human-Computer Interaction ,  5 (CHI PLAY), 1–25.

Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivation: Classic definitions and new directions. Contemporary Educational Psychology, 25 , 54–67.

Saleem, A. N., Noori, N. M., & Ozdamli, F. (2022). Gamification applications in E-learning: A literature review. Technology, Knowledge and Learning, 27 (1), 139–159.

Sanchez, D. R., Langer, M., & Kaur, R. (2020). Gamification in the classroom: Examining the impact of gamified quizzes on student learning. Computers & Education, 144 , 103666.

Schell, J. (2008). The Art of Game Design: A Book of Lenses. Elsevier Inc

Taşkın, N., & Kılıç Çakmak, E. (2023). Effects of gamification on behavioral and cognitive engagement of students in the online learning environment. International Journal of Human-Computer Interaction, 39 (17), 3334–3345.

Yang, C. C., & Ogata, H. (2023). Personalized learning analytics intervention approach for enhancing student learning achievement and behavioral engagement in blended learning. Education and Information Technologies, 28 (3), 2509–2528.

Download references

Open access funding provided by SCELC, Statewide California Electronic Library Consortium

Author information

Authors and affiliations.

Rossier School of Education, University of Southern California, 3551 Trousdale Pkwy, Los Angeles, CA, 90089, USA

Zhihui Zhang

Alibaba Cloud, Alibaba Hangzhou Headquarters, 699 Wangshang Rd, Binjiang District, Hangzhou, 310052, Zhejiang, China

Xiaomeng Huang

You can also search for this author in PubMed   Google Scholar

Contributions

Zhihui Zhang, 60%, conceived and designed the experiments, performed the experiments, prepared the figures and/or tables, drafted the work.

Xiaomeng Huang, 40%, analyzed the data, performed the experiments, wrote the code, designed the software or performed the computation work, drafted the work, revised it.

Corresponding author

Correspondence to Zhihui Zhang .

Ethics declarations

Ethics approval.

Consent to participate—(include appropriate consent statements).

Conflicts of interest/Competing interests

There is no conflicts of interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Zhang, Z., Huang, X. Exploring the impact of the adaptive gamified assessment on learners in blended learning. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12708-w

Download citation

Received : 13 September 2023

Accepted : 10 April 2024

Published : 06 May 2024

DOI : https://doi.org/10.1007/s10639-024-12708-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Blended learning
• Gamification
• Adaptive assessment
• Language proficiency
• Learner motivation
• Find a journal
• Publish with us
• Track your research

share this!

May 6, 2024

This article has been reviewed according to Science X's editorial process and policies . Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

peer-reviewed publication

Do good lessons promote students' attention and behavior?

by Anke Wilde, Leibniz-Institut für Bildungsforschung und Bildungsinformation

Students are better able to regulate themselves in lessons that they consider to be particularly well implemented. This is the conclusion drawn from a study by the DIPF | Leibniz Institute for Research and Information in Education, published in the journal Learning and Instruction .

The link between teaching quality and self-regulation tends to be particularly true for pupils who have problems controlling their behavior and following lessons, for example due to ADHD symptoms.

Good teaching is characterized by the teacher leading the class through the lesson without disruption, encouraging the students to think, taking an interest in them and supporting them individually. The better the teacher is at this, the better the students will be able to regulate their behavior, for example by paying attention, cooperating and adhering to the class rules.

As a result, they learn better. This link, which has already been established in research, has now been examined in more detail in this daily diary study and evaluated with the help of multilevel analyses.

It became clear that the quality of teaching has an impact not only on self-regulation overall, but also in each individual lesson, as Dr. Friederike Blume, lead author of the now published study, summarizes the results.

"When teachers are particularly good at classroom management and providing student support in a lesson, students are better able to regulate their behavior. When these two characteristics of good teaching are not working well in a lesson, students also reported that they were less able to concentrate and engage."

Cognitive activation, the third characteristic of good teaching, was hardly relevant for self-regulation. Therefore, the personal relationship between teacher and student is particularly important, emphasizes Dr. Blume.

This is especially true for students who have difficulties with self-regulation, such as those with attention deficit hyperactivity disorder (ADHD).

"Many teachers find it difficult to establish a positive relationship with children with ADHD symptoms," says the educational researcher. "However, our study showed that in lessons where children with self-regulation difficulties felt particularly supported by their teacher, they were more likely to report being able to concentrate better and follow class rules.

"It is therefore worth taking a positive approach to these children in the classroom and showing a genuine interest in them, as this can reduce the pressure on teachers in the long term and bring more calm to the classroom."

The DIPF researcher also recommends that teachers ask their students for feedback on their teaching from time to time. Although this is still a taboo for many, it can provide valuable information on how to better tailor their teaching to the needs of individual students.

A total of 64 pupils in years 5 and 6 took part in the study. They did not necessarily belong to the same school or class, but were recruited through an email appeal to music schools, sports and leisure centers, for example.

At the start of the study, the children completed a questionnaire about general information such as their grade level and type of school, as well as how they rated their self-regulation skills. Over the next three school weeks, the children answered daily questions about the last lesson of each day.

The questions related to the quality of teaching (e.g., support from the teacher, disruptions in lessons, stimulation of reflection), as well as their ability to regulate themselves in that lesson (e.g., attention, impulse control, motor activity).

The links between the individual lessons and the corresponding daily entries were evaluated using multilevel analysis. Among other things, the results were analyzed on an intrapersonal level, which allows conclusions to be drawn at the level of the individual child. In addition, interpersonal associations were examined, which allows conclusions to be drawn about all participants together.

Limitations of the study

Studies with such an elaborate design, involving daily diaries, always aim to collect data in as short time as possible. As a result, teaching quality was only measured here on the basis of only few statements, which certainly do not cover all the characteristics of good teaching.

Future studies should therefore take a closer look at classroom interaction processes to explore which features of teaching are particularly beneficial, especially for children with stronger ADHD symptoms.

Furthermore, future studies must show whether the results found here apply to all subjects or only to certain subjects, and the role of different teaching methods.

Journal information: Learning and Instruction

Provided by Leibniz-Institut für Bildungsforschung und Bildungsinformation

Explore further

Feedback to editors

Astronomers explore globular cluster NGC 2419

26 minutes ago

Research team develops AI to perform chemical synthesis

56 minutes ago

US restorationist solves 60-million-year-old dinosaur fossil 'puzzles'

2 hours ago

Researchers reveal how molecular roadblocks slow the breakdown of cellulose for biofuels

12 hours ago

In South Africa, tiny primates could struggle to adapt to climate change

Study reveals flaw in long-accepted approximation used in water simulations

13 hours ago

Researchers develop nanotechnology for creating wafer-scale nanoparticle monolayers in seconds

Study underscores new strategies to fight drug-resistant bacteria

14 hours ago

Researchers establish commercially viable process for manufacturing with promising new class of metals

Chimps shown to learn and improve tool-using skills even as adults

15 hours ago

Relevant PhysicsForums posts

Physics instructor minimum education to teach community college, studying "useful" vs. "useless" stuff in school.

Apr 30, 2024

Why are Physicists so informal with mathematics?

Apr 29, 2024

Plagiarism & ChatGPT: Is Cheating with AI the New Normal?

Apr 28, 2024

Digital oscilloscope for high school use

Apr 25, 2024

Motivating high school Physics students with Popcorn Physics

Apr 3, 2024

More from STEM Educators and Teaching

Related Stories

Virtual pupils make for more confident teachers

Jun 22, 2021

Small class sizes not better for pupils' grades or resilience, says study

Mar 8, 2024

Explicit socioemotional learning can have a key role in physical education lessons, study says

Mar 13, 2024

Study finds clear instruction, parental support predict students' sense of school belonging

Mar 27, 2024

How do we help students from disadvantaged backgrounds feel confident about school?

Apr 1, 2024

Positive teacher-student relationships lead to better teaching

Mar 8, 2022

Recommended for you

Investigation reveals varied impact of preschool programs on long-term school success

May 2, 2024

Training of brain processes makes reading more efficient

Apr 18, 2024

Researchers find lower grades given to students with surnames that come later in alphabetical order

Apr 17, 2024

Earth, the sun and a bike wheel: Why your high-school textbook was wrong about the shape of Earth's orbit

Apr 8, 2024

Touchibo, a robot that fosters inclusion in education through touch

Apr 5, 2024

More than money, family and community bonds prep teens for college success: Study

Let us know if there is a problem with our content.

Use this form if you have come across a typo, inaccuracy or would like to send an edit request for the content on this page. For general inquiries, please use our contact form . For general feedback, use the public comments section below (please adhere to guidelines ).

Please select the most appropriate category to facilitate processing of your request

Thank you for taking time to provide your feedback to the editors.

Your feedback is important to us. However, we do not guarantee individual replies due to the high volume of messages.

E-mail the story

Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient's address will be used for any other purpose. The information you enter will appear in your e-mail message and is not retained by Phys.org in any form.

Newsletter sign up

Get weekly and/or daily updates delivered to your inbox. You can unsubscribe at any time and we'll never share your details to third parties.

More information Privacy policy

Donate and enjoy an ad-free experience

We keep our content available to everyone. Consider supporting Science X's mission by getting a premium account.

E-mail newsletter

Research and Economic Development

Citi program training for irb requirements.

• About Research Compliance
• Frequently Asked Questions (FAQ)
• Institutional Registration and Accreditation Numbers
• Staff Duties/Contacts
• Kuali Research

Committees:

• Institutional Review Board (IRB)
• Institutional Animal Care and Use Committee (IACUC)
• Conflict of interest (COI)
• Stem Cell Research Oversight (SCRO)
• Dual Use Research of Concern (DURC)

Related Links:

• Research Misconduct (RM) Allegations
• Responsible Conduct of Research (RCR) Requirements
• Research Participant/Subject Resources
• HIPAA & Human Subjects Research

This page provides instructions for using CITI Program courses to satisfy Human Subjects Research (HSR) training requirements for IRB applications. The training for Human Subjects Research courses varies by discipline. You only need to complete one of these based on your field of study:

• Biomedical Research Investigators
• Social & Behavioral Research Investigators
• Research with Data or Laboratory Specimens Only

Step 1: Log in

(click on an image to see it larger)

Step 2: Select Curriculum

If there are no courses associated with your account yet, you will be taken to the Select Curriculum page. If you decide to wait to add courses, click the My Courses link when you return to the account and then click the  View Courses button next to University of California at Riverside.

Also on this page, you will find links that will allow you to affiliate your account with UCR. If you logged in using UCR's Single Sign On application, then your account is probably already affiliated with UCR and you don't need to do this. But if you do not see UCR on this page then your account is not yet affiliated with UCR. Adding a UCR affiliation will allow any future training completion records to be automatically added to Kuali Research protocols where you are listed as a laboratory member.

For details on adding affiliations and transferring completion records for courses completed prior to affiliating your account with UCR , see the Knowledge Base article  at https://support.citiprogram.org/s/article/how-to-add-change-your-affiliated-institution-or-transfer-completions?name=how-to-add-change-your-affiliated-institution-or-transfer-completions .

To add new courses to your account, click the Add a Course link in the gray box near the bottom of the My Courses page.

This brings up the Select Curriculum page, which displays a series of checkboxes to identify the course category(ies) that you'd like to add.

The next section asks you to choose specific courses from the selected category. Choose the courses that most closely match your area of research. The regulations require you to complete at least one course. You are welcome to select as many as you like.

At the end of the selection, a message indicates that your enrollment was successful and the courses now appear on your My Courses page. You can repeat the Add Courses process whenever you want to add other courses to your account.  You can start the training right away or complete it later by exiting and logging back in when you are ready.

Step 3: Verify Completion Records

CITI automatically records Human Subjects training data and in most cases your completion can be confirmed by UCR staff. However, you may choose to generate a certificate of completion for your own records. This is especially important if you completed the training through a different institution, since UCR staff can only access records for people who affiliated with UCR prior to completing the training.

• Open the CITI Program Log In webpage
• Click Log In Through My Organization
• click the My Courses link at the top of the page to view your enrollments.

Transferring Course Credits

If you completed a course or some of its modules previously at another institution, you can transfer the credits to your UCR-affiliated account:

Instructions to transfer credits to your UCR-affiliated account

For questions regarding HSR training requirements, contact [email protected] . For technical questions, contact CITI at [email protected] or 1-888-529-5929.

• Skip to Content
• Skip to Main Navigation
• Skip to Search

Indiana University Indianapolis Indiana University Indianapolis IU Indianapolis

• Undergraduate Majors
• Apply to the Accelerated Program
• Master's Degrees
• Doctoral Degrees & Minors
• Minors & Certificates
• General Education
• Artificial Intelligence
• Bioinformatics
• Computer Science
• Data Science
• Health Informatics
• Health Information Management
• Library & Information Science
• Informatics
• Media Arts and Science
• Study Abroad in Greece
• Study Abroad in Finland
• Micro-Credentials
• Freshman Applicants
• Returning Students
• Master's Degree
• Doctoral Program
• Graduate Certificates
• Change or Declare your Major
• Admitted Students
• Student Ambassadors
• Virtual Tour
• Undergraduate Webinars & Information Sessions
• Graduate Student Information Sessions
• Summer Camp
• Earn College Credit
• Biomedical Informatics Challenge
• Computer Science Challenge
• Incoming Undergraduate Scholarships
• Undergraduate Scholarships
• Graduate Scholarships
• Accelerated Program Cost & Aid
• Travel Funding
• Tuition Reduction
• Peer Advisors
• Forms & Policies
• Become a Student Leader
• Student Organizations
• Honors Program
• Laptop Requirements
• Equipment Checkout
• Luddy Knowledge Base
• Student Facility Access
• Biomedical Informatics B.S.
• Health Information Management B.S.
• Informatics B.S.
• Media Arts and Science B.S.
• Bioinformatics M.S.
• Health Informatics M.S.
• Applied Data Science M.S.
• Human-Computer Interaction M.S.
• Master of Library and Information Science
• Media Arts and Science M.S.
• Find a Job or Internship
• F-1 Students & Internships
• Library & Information Science Internships
• Internship Checklist
• Forage: Virtual Job Simulations
• Forage: Earn Credit
• Network with LinkedIn
• Big Interview
• Elevator Pitch
• Cover Letter
• Informational Interview
• Interviewing
• Technical Interviewing
• The Offer Process
• The Negotiation Process
• Freelance Work
• Grant Proposal Writing
• Schedule an Appointment
• Request a Career Services Presentation
• Featured Employer Days
• Resume Reviews
• Portfolio Reviews
• Presentations and Workshops
• Employer Career Fair Registration
• Research Centers & Labs
• Undergraduate Research
• Research Events
• Luddy Strategic Plan
• Meet Fred Luddy
• Faculty Openings
• Faculty Directory
• Staff Directory
• Media Requests
• Contact Admissions
• Request Undergraduate Information
• Request Graduate Information
• Get involved
• Advisory Boards
• Advisory Board
• Department Blog
• Strategic Plan
• Multimedia Stories
• Luddy Leads Blog
• Student Showcases
• LIS Industry Speaker Series

Luddy School of Informatics, Computing, and Engineering

• Alumni & Giving
• Departments
• News & Blog
• Degrees & Courses

Luddy School of Informatics, Computing, and Engineering resources and social media channels

• Schedule a Visit

Additional links and resources

• Degrees & Majors
• Scholarships

Happening at Luddy

• Pre-college Programs

Information For

• Current Students
• Faculty & Staff Intranet

Luddy Indianapolis