introduction about reading comprehension research

BRIEF RESEARCH REPORT article

The use of new technologies for improving reading comprehension.

• 1 Department of General Psychology, University of Padova, Padua, Italy
• 2 Azienda Sociosanitaria Ligure 5 Spezzino, La Spezia, Italy

Since the introduction of writing systems, reading comprehension has always been a foundation for achievement in several areas within the educational system, as well as a prerequisite for successful participation in most areas of adult life. The increased availability of technologies and web-based resources can be a really valid support, both in the educational and clinical field, to devise training activities that can also be carried out remotely. There are studies in current literature that has examined the efficacy of internet-based programs for reading comprehension for children with reading comprehension difficulties but almost none considered distance rehabilitation programs. The present paper reports data concerning a distance program Cloze , developed in Italy, for improving language and reading comprehension. Twenty-eight children from 3rd to 6th grade with comprehension difficulties were involved. These children completed the distance program for 15–20 min for at least three times a week for about 4 months. The program was presented separately to each child, with a degree of difficulty adapted to his/her characteristics. Text reading comprehension (assessed distinguishing between narrative and informative texts) increased after intervention. These findings have clinical and educational implications as they suggest that it is possible to promote reading comprehension with a distance individualized program, avoiding the need for the child displacements, necessary for reaching a rehabilitation center.

Introduction

Reading comprehension is a fundamental cognitive ability for children, that supports school achievement and successively participation in most areas of adult life ( Hulme and Snowling, 2011 ). Therefore, children with learning disabilities (LD) and special educational needs who show difficulties in text comprehension, sometimes also in association with other problems, may have an increased risk of life and school failure ( Woolley, 2011 ). Reading comprehension is, indeed, a complex cognitive ability which involves not only linguistic (e.g., vocabulary, grammatical knowledge), but also cognitive (such as working memory, De Beni and Palladino, 2000 ), and metacognitive skills (both for the aspects of knowledge and control, Channa et al., 2015 ), and, more specifically, higher order comprehension skills such as the generation of inferences ( Oakhill et al., 2003 ).

Recently, due to the diffusion of technology in many fields of daily life, text comprehension at school, at home during homework, and at work is based on an increasing number of digital reading devices (computers and laptops, e-books, and tablet devices) that can become a fundamental support to improve traditional reading comprehension and learning skills (e.g., inference generation).

Some authors contrasted in children with typical development the effects of the technological interface on reading comprehension vs printed texts ( Kerr and Symons, 2006 ; Rideout et al., 2010 ; Mangen et al., 2013 ; Singer and Alexander, 2017 ; Delgado et al., 2018 ). Results were consistent and showed a worse comprehension performance in screen texts compared to printed texts for children ( Mangen et al., 2013 ; Delgado et al., 2018 ) and adolescents who nonetheless showed a preference for digital texts compared to printed texts ( Singer and Alexander, 2017 ). Regarding children with learning problems, only few studies considered the differences between printed texts and digital devices ( Chen, 2009 ; Gonzalez, 2014 ; Krieger, 2017 ) finding no significant differences, suggesting that the use of compensative digital tools for children with a learning difficulty could be a valid alternative with respect to the traditional written texts in facilitating their academic and work performance. This conclusion is also supported by the results of a meta-analysis ( Moran et al., 2008 ), regarding the use of digital tools and learning environments for enhancing literacy acquisition in middle school students, which demonstrates that technology can improve reading comprehension.

Different procedures and abilities are targeted in the international literature concerning computerized training programs for reading comprehension. In particular, various studies include activities promoting cognitive (e.g., vocabulary, inference making) and metacognitive (e.g., the use of strategies, comprehension monitoring, and identification of relevant parts in a text) components of reading comprehension. Table 1 reports the list of papers proposing computerized training programs with a summary of the findings encountered. Participants involved cover different ages and school grades, the majority belonging to middle school and high school. The general outcome of the studies is positive due to a significant improvement in comprehension skills after the training program with long-lasting effects also during follow-up; indeed, the majority of participants involved in training programs outperformed their peers assigned to comparison groups and maintained their improvements. Specifically, several studies ( O’Reilly et al., 2004 ; Magliano et al., 2005 ; McNamara et al., 2006 ) used the iSTART program with adolescents and young adults. This program promotes self-explanation, prior knowledge and reading strategies to enhance understanding of descriptive scientific texts. Results demonstrated that students who followed the iSTART program received more benefits than their peers, improving self-explanation and summarization. Additionally, strategic knowledge was a relevant factor for the outcome in comprehension tasks including multiple choice questions: students who already possessed good strategic knowledge improved their accuracy when answering to bridging inference questions, whereas students with low strategic knowledge became more accurate with text-based questions. Another program, ITSS, was used with younger students ( Meyer et al., 2011 ; Wijekumar et al., 2012 , 2013 , 2017 ), with the objective to support activities based on identifying main parts and key words in a text and classifying information in a hierarchical order. Positive outcomes were found also with such program since students who followed the ITSS program significantly improved text comprehension compared to their peers in the control group.

Table 1. Synthesis of the main results of the computerized training programs on comprehension present in the literature.

Although most of the literature deals with typical development, also cases of students with learning difficulties were considered. For example, Potocki et al. (2013) (see also Potocki et al., 2015 ) examined the effects of two different computerized programs with specific aims: one focusing on comprehension features, such as inference making and the analysis of text structure, the other considering decoding skills. Both training programs brought some benefits to reading comprehension, however larger effects were found with the program focused on comprehension with long-lasting effects in listening and reading comprehension (see also Kleinsz et al., 2017 ). Studies by Johnson-Glenberg (2005) and Kim et al. (2006) , using respectively the programs 3D Readers and CACSR, were able to promote reading comprehension abilities in middle school students through metacognitive activities. Thanks to these programs students also became more aware of reading strategies and implemented them more successfully during text comprehension. In particular, a study by Niedo et al. (2014) , obtained positive results on silent reading in a small group of children struggling with reading using the “cloze” procedure. This procedure proposes exercises in which parts of a text, typically words, are missing and participants are required to complete the text guessing what is missing.

Thus, computerized programs generally seem to improve reading comprehension skills. However, it should be noticed that, in most cases, students were trained at school, without the personalized support of a clinician taking into consideration the cognitive and psychological needs of the child. In particular, to our knowledge, no program examined the effects of an internet-based distance reading comprehension program which allows the child to be trained at home in a personalized way. A useful aspect of an internet-based distance training is that the psychologist can monitor with the application ( app ) the child’s results and activities and write him/her some motivational messages, reducing the attritions present in programs carried out at home with the only supervision of parents. Literature concerning distance trainings is still rare, however, some evidence suggests that these programs may represent a good integration to other types of intervention, usually carried out at school, in a rehabilitation center or at home (e.g., Mich et al., 2013 ).

Therefore, despite still preliminary, we think that it is relevant to present data about a distance program developed in Italy named Cloze ( Cornoldi and Bertolo, 2013 ), devised for rehabilitation purposes but with potential implication also for educational contexts. Cloze has been developed to promote inferential abilities both at a sentence- and discourse-level using the “cloze” procedure. Several findings in the literature demonstrate that abilities, such as anticipating text parts and inference making, bring improvements in text comprehension (e.g., Yuill and Oakhill, 1988 ) and it has been shown that one way to promote inferential competences is to improve the ability to predict parts of the text that are missing or that follow, considering the available information: the “cloze” technique appears to be one of the most successful ways for this purpose (e.g., Greene, 2001 ).

In the current study the effectiveness of this training program has been tested on a clinical population who exhibited, for various reasons, difficulties in reading comprehension. Participants were 28 children (16 male and 12 female) attending a private practice for learning difficulties in the city of La Spezia, in the north-west of Italy, from 3rd to 6th school grade (5 of 3rd, 9 of 4th, 11 of 5th and 3 of 6th grade), with a mean age of children of M = 9.79 years (SD = 1.03). Seventeen children had a current or past speech disorder: of these children 10 also had a LD (Learning Disabilities) and one was bilingual (speech problems were not due to bilingualism). The other 11 children had a LD or important learning difficulties, and one of them had also ADHD (Attention Deficit/Hyperactivity Disorder). For the goals of the study, all these children were considered together as they all presented a severe reading comprehension difficulty as reported by parents and teachers and confirmed by the initial assessment.

All children had received a comprehensive psychological assessment (see Table 2 ), adapted to their particular needs and ages. In particular all children had an IQ >80 assessed with the Wechsler Intelligence Scale for Children-IV (WISC-IV; Wechsler, 2003 ) and did not have anxiety disorders, mood affective disorders or other developmental disorders, with the exception of the cases with language disorder and the case with ADHD. Children were not receiving any additional treatment, including medication. Written consent was obtained from the children’s parents in the context of the private practice.

Table 2. Main characteristics of the sample in terms of reading and cognitive abilities.

Materials and Methods

Pre-/post-test assessment and procedure of the training.

Each child started a training program through the distance rehabilitation platform Ridinet, using the Cloze app, after the assessment of learning and cognitive abilities, including comprehension assessment with two texts, one narrative and one informative ( Cornoldi and Carretti, 2016 ; Cornoldi et al., 2017 ). Connection to the Ridinet web site was required in order to access to the app, three or four times a week for more or less 15/20 min. The period of use was of 3 months for 6 children and 4 months for 22 children. After this period children’s comprehension was assessed again. Additionally, some questions were asked to parents and children about the app’s utility and pleasantness. In particular, children were asked: “Do you think the program helped you improve your text comprehension skills?,” “Did you like doing this program instead of the same exercises on paper?”; and parents were asked: “Was it difficult to start the Cloze activities on days when it had to be done?,” “Compared to the beginning of the treatment, how do you currently judge the ability of your child to understand the texts?”. For all questions, except the last one, the answer had to be given on a 5-point scale with 1 = not at all, 2 = a little, 3 = enough, 4 = very, 5 = very much. For the last question the answer changed on a 4-point scale with 1 = got worse, 2 = unchanged, 3 = slightly improved, and 4 = greatly improved.

Comprehension Tasks

Reading comprehension was assessed with two texts, the first narrative and the other informative, taken from Italian batteries for the assessment of reading ( Cornoldi and Carretti, 2016 ; Cornoldi et al., 2017 ). The texts range between 226 and 455 words in length, and their length increases with school grade (in order to have texts and questions matching the degrees of expertise at different grades the batteries include a different pair of texts for each grade). Students read the text in silence at their own pace, then answer a variable number of multiple-choice questions (depending on school grade), choosing one of four possible answers. There is no time limit, and students can reread the text whenever they wish. The final score is calculated as the total number of correct answers for each text. Alpha coefficients, as reported by the manuals, range between 0.61 and 0.83. For the purposes of the study we decided to use the same two comprehension texts, at pre-test and post-test, as the procedure offered the opportunity of directly examining and showing to parents changes in comprehension and previous evidence had shown the absence of relevant retest effects with this material in a retest carried out after 3 months ( Viola and Carretti, 2019 ).

Distance Rehabilitation Program: Cloze

Cloze ( Cornoldi and Bertolo, 2013 ) is an app for the promotion of text comprehension with the specific aim to recover processes of lexical and semantic inference. At each work session the child works with texts that lack words and must complete the empty spaces by choosing the correct alternative from those automatically proposed by the app, so that the text becomes congruent. The program is adaptive, as text complexity and proportion of missing words vary according to the previous level of response, and is designed for children who have weaknesses in written text comprehension, mainly due to poor skills in lexical and semantic inferential processes. The app also allows to enhance a set of language skills (phonology, syntax, semantics) which contribute to ensuring the fluidity of text and production processing. The recommended age range for the use of this program is between 7 and 14 years. In this study the semantic mode (only content words may be missing and no syntactic cues can be used for deciding between the alternatives) was proposed to 21 children and the syntactic mode (where all words may be missing) to 7 children. The mode type selected for each child depends from the performance at pre-test and diagnosis. A clinician, co-author of the present study (LB), monitored the child’s results and activities with the app and sent him/her from time to time some motivational messages. The motivational messages were typically sent once a week for congratulating with children for the work done and check with him/her possible problems emerged. Training lasted from 3 to 4 months and involved between 3 and 4 sessions of 15–20 min per week. The variation in duration depended on the decision of each individual family. In fact, children were required to use the software for about 4 months or in any case for a minimum period of 3 months (choice made by six families).

Effects on Reading Comprehension of Cloze Training

All analyses were carried out with SPSS 25 ( IBM Corp, 2017 ). A preliminary analysis found that all the examined variables met the assumptions of normality (K-S between 0.106 and 0.143, p > 0.05). Then, we compared the reading comprehension performance of children before and after the computerized training with Cloze . For this analysis, a repeated measure Analysis of Variance (ANOVA) was conducted on comprehension scores to examine the differences in the whole group of children between the scores obtained before and after the training. A significant difference was found for both comprehension texts [ F (1,27) = 22.37, p < 0.001, η 2 p = 0.453 and F (1,27) = 38.90, p < 0.001, η 2 p = 0.599, respectively]. Possible differences between the two training modalities (semantic vs syntactic) and between different training periods (3 months vs 4 months) were then analyzed; no significant differences emerged between groups in both cases [ F (1,27) < 1].

Secondly, to analyze the role of individual differences at pre-test, the standardized training gain score (STG; Jaeggi et al., 2011 ) – computed by subtracting post-test score minus pre-test score, divided by the SD of the pre-test – was calculated for the two texts comprehension. Pearson correlations were computed between the STG and the variable collected at pre-test (reading speed and errors, WISC IV – Full scale IQ, Verbal Comprehension, Perceptual Reasoning, Working Memory and Processing Speed indexes). The only significant correlation was between STG of the narrative text and Verbal Comprehension Index of the WISC-IV Scale ( r = 0.38, p = 0.048). Finally, individual improvements from pre- to post-test were also confirmed considering changes in performance in terms of standard deviation in relations to norms (provided by the manual). Table 3 shows the number of children for each comprehension text who improved their performance moving from a performance at least 2 standard deviations or between 1 and 2 negative standard deviations under the mean to a performance above one negative standard deviation.

Table 3. Changes in performance in relations to norms (provided by the manual) after the training program Cloze.

Perceived Utility, Pleasantness, Parents and Child’s Improvements of Cloze

Results concerning the answers of parents and children about utility, pleasantness and self-perceived efficacy of the app, were also analyzed. At the first question, addressing children’s perceived improvement in comprehension skills, more than half of the sample chose the alternatives “very” or “very much” (15 “very” and 5 “very much”), only 1 child answered “a little” and the others chose “enough.” At the second question, about the pleasure of doing this kind of activity instead of pen and paper activities, all children answered “very” or “very much.” Concerning parents’ questions, at the first question about the difficulty to start the Cloze activity, only one parent answered “enough,” a quarter of the sample chose “a little” (seven families) and all the other 20 families chose the alternative “not at all.” At the last question about the perceived training efficacy on their child’s performance, the large majority of the families chose “slightly improved” or “greatly improved” and only three parents thought their children’s ability had remained unchanged. However, no correlations between parents and child’s perceived improvements and STG in reading comprehension were found.

The present study examined the effects of the use of Cloze , a distance rehabilitation program focused on inference skills, for improving reading comprehension, on the basis of the hypothesis that, being inference making related to reading comprehension at different ages (e.g., Oakhill and Cain, 2012 ), positive effects of the training activities on reading comprehension should be found.

Concerning the efficacy of computer-assisted training programs, literature highlights that many training programs are devised for an educational context. Results are generally encouraging with positive effects on reading comprehension, measured with materials different from those practiced during the training. However, few studies analyzed the efficacy in children with specific reading comprehension problems, and no studies considered the possibility of carrying out a training at home under the distance supervision of an expert. The latter characteristics are those that make the Cloze peculiar compared to the existent literature. Cloze is indeed based on a rehabilitation online platform which allows the child to complete personalized training activities several times a week, without moving from his/her home, and concurrently enabling the clinician to monitor the child’s progress or manage activities’ characteristics. The advantage of this procedure is twofold: on one hand it increases the potential number of training sessions per week, on the other hand it permits to save the necessary time to reach the center for rehabilitation and to reduce the costs of the intervention.

The preliminary data on Cloze were generally positive: children, working on either two slightly different versions of the same program, showed a generalized improvement in reading comprehension tasks and, together with their families, expressed appreciation for the pleasantness and the efficacy of the program. Encouraging results emerged also from the analysis of individual improvements referring to normative scores, as reported in Table 3 : most of the children’s performance migrated from a highly negative level to an average level.

It is noticeable that the efficacy of the training was assessed with materials different from those practiced during the training sessions, since reading comprehension tasks required to read a paper text and complete a series of multiple-choice questions. In future studies it would be interesting to analyze the effects of the program on skills known to be related to text comprehension, such as vocabulary or comprehension monitoring, for example. There is good reason to believe that since these variables are highly predictive of comprehension skills (and given that training in these skills sometimes improve comprehension; e.g., Beck et al., 1982 ; see also Hulme and Snowling, 2011 ), training that specifically targets comprehension might, in turn, lead to improvements in vocabulary or comprehension monitoring skills. Further studies are needed to explore this hypothesis.

A second relevant finding of the present study is the presence of a positive correlation between the gain obtained in one of the reading comprehension text (the narrative one) and the Verbal Comprehension Intelligence Quotient (VCIQ) index of the WISC-IV battery, showing that children who started with more resources in verbal intelligence achieved greater improvements in text comprehension at least with one type of text through the Cloze . The activities probably required to develop some kind of strategies, and for this reason students with larger verbal intellectual resources, who were presumably more able to develop new strategies, were more advantaged. Indeed, this amplification effect is usually found when training activities require the development of strategies ( von Bastian and Oberauer, 2014 ). Such result has clinical and educational implications, inviting professionals and teachers to consider children’s starting resources and, if necessary, to combine activities conducted through distance rehabilitation programs with personal intervention sessions that could teach strategies and promote a metacognitive approach to reading comprehension. However, some limitations of the present study must be acknowledged. Firstly, study did not include a control group, therefore findings should be taken with caution, although normative data and previous results obtained with the same test offer support to the robustness of our results and the use of normative data offers a control measure of how reading comprehension skills are acquired in typically developing children without specific training, therefore functioning as a sort of passive control group. Secondly, the treated group, although characterized by a common reading comprehension difficulty, was partly heterogeneous, as children attended different grades and could have different diagnoses. Unfortunately, the limited number of subjects, with the consequence that it was not possible to form groups defined both by the grade and the diagnosis, did not permit to make analyses taking into account the grade and the diagnosis as between-subjects factors. Future studies should examine a more homogeneous population or consider a larger sample of children, giving more information about the efficacy of training in different children population. Additionally, the fact that the treatment was concluded with the post-training assessment did not offer the opportunity to further examine the procedure and maintenance effects with a follow-up. Despite the limitations, this study offers evidence concerning the efficacy of new methods, based on computer-assisted training programs that could be beneficial in training high-level skills such as comprehension and inference generation. Such tools can be extremely worthwhile for struggling readers who may need to receive further attention in mastering higher level reading comprehension.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent to participate in this study was provided by the participants’ legal guardian/next of kin. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author Contributions

AC, CC and BC contributed to the design and implementation of the research. LB provided the data. BC organized the database. AC performed the statistical analysis. ED did the literature research and wrote the section about the review of the literature. AC and BC wrote the other sections. CC contributed to the manuscript revision, read and approved the submitted version.

The present work was carried out within the scope of the research program Dipartimenti di Eccellenza (art.1, commi 314-337 legge 232/2016), which was supported by a grant from MIUR to the Department of General Psychology, University of Padua and partially supported by a grant (PRIN 2015, 2015AR52F9_003) to Cesare Cornoldi funded by the Italian Ministry of Research and Education (MIUR).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Beck, I. L., Perfetti, C. A., and McKeown, M. G. (1982). Effects of long-term vocabulary instruction on lexical access and reading comprehension. J. Educ. Psychol. 74, 506–521. doi: 10.1037/0022-0663.74.4.506

CrossRef Full Text | Google Scholar

Channa, M. A., Nordin, Z. S., Siming, I. A., Chandio, A. A., and Koondher, M. A. (2015). Developing reading comprehension through metacognitive strategies: a review of previous studies. Eng. Lang. Teach. 8, 181–186. doi: 10.5539/elt.v8n8p181

Chen, H. (2009). Online reading comprehension strategies among fifth- and sixth-grade general and special education students. Educ. Res. Perspect. 37, 79–109.

Google Scholar

Cornoldi, C., and Bertolo, L. (2013). Cloze Ridinet. Bologna: Anastasis.

Cornoldi, C., and Carretti, B. (2016). Prove MT-3-Clinica. Firenze: Giunti Edu.

Cornoldi, C., Carretti, B., and Colpo, C. (2017). Prove MT-Kit Scuola. Dalla valutazione degli Apprendimenti di Lettura E Comprensione Al Potenziamento. [MT-Kit for the Assessment In The School. From Reading Assessment To Its Enhancement]. Firenze: Giunti Edu.

Cullen, J. M., Alber-Morgan, S. R., Schnell, S. T., and Wheaton, J. E. (2014). Improving reading skills of students with disabilities using headsprout comprehension. Remed. Spec. Educ. 35, 356–365. doi: 10.1177/0741932514534075

De Beni, R., and Palladino, P. (2000). Intrusion errors in working memory tasks: are they related to reading comprehension ability? Learn. Individ. Differ. 12, 131–143. doi: 10.1016/s1041-6080(01)00033-4

Delgado, P., Vargas, C., Ackerman, R., and Salmerón, L. (2018). Don’t throw away your printed books: a meta-analysis on the effects of reading media on reading comprehension. Educ. Res. Rev. 25, 23–38. doi: 10.1016/j.edurev.2018.09.003

Gonzalez, M. (2014). The effect of embedded text-to-speech and vocabulary eBook scaffolds on the comprehension of students with reading disabilities. Intern. J. Spec. Educ. 29, 111–125.

Greene, B. (2001). Testing reading comprehension of theoretical discourse with cloze. J. Res. Read. 24, 82–98. doi: 10.1111/1467-9817.00134

Hulme, C., and Snowling, M. J. (2011). Children’s reading comprehension difficulties: nature, causes, and treatments. Curr. Direct. Psychol. Sci. 20, 139–142. doi: 10.1177/0963721411408673

IBM Corp (2017). IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp.

Jaeggi, S. M., Buschkuehl, M., Jonides, J., and Shah, P. (2011). Short-and long-term benefits of cognitive training. Proc. Natl. Acad. Sci. U.S.A. 108, 10081–10086. doi: 10.1073/pnas.1103228108

Johnson-Glenberg, M. C. (2005). Web-based training of metacognitive strategies for text comprehension: focus on poor comprehenders. Read. Writ. 18, 755–786. doi: 10.1007/s11145-005-0956-5

Kerr, M. A., and Symons, S. E. (2006). Computerized presentation of text: effects on children’s reading of informational material. Read. Writ. 19, 1–19. doi: 10.1007/s11145-003-8128-y

Kim, A. H., Vaughn, S., Klingner, J. K., Woodruff, A. L., Klein Reutebuch, C., and Kouzekanani, K. (2006). Improving the reading comprehension of middle school students with disabilities through computer-assisted collaborative strategic reading. Remed. Spec. Educ. 27, 235–249. doi: 10.1177/07419325060270040401

Kleinsz, N., Potocki, A., Ecalle, J., and Magnan, A. (2017). Profiles of French poor readers: underlying difficulties and effects of computerized training programs. Learn. Individ. Differ. 57, 45–57. doi: 10.1016/j.lindif.2017.05.009

Krieger, R. (2017). The Effect of Electronic Text Reading on Reading Comprehension Scores of Students with Disabilities. Master thesis, Governors State University, Park, IL.

Leong, C. K. (1992). Enhancing reading comprehension with text-to-speech (DECtalk) computer system. Read. Writ. 4, 205–217. doi: 10.1007/bf01027492

Magliano, J. P., Todaro, S., Millis, K., Wiemer-Hastings, K., Kim, H. J., and McNamara, D. S. (2005). Changes in reading strategies as a function of reading training: a comparison of live and computerized training. J. Educ. Comput. Res. 32, 185–208. doi: 10.2190/1ln8-7bqe-8tn0-m91l

Mangen, A., Walgermo, B. R., and Brønnick, K. (2013). Reading linear texts on paper versus computer screen: effects on reading comprehension. Intern. J. Educ. Res. 58, 61–68. doi: 10.1016/j.ijer.2012.12.002

McNamara, D. S., O’Reilly, T. P., Best, R. M., and Ozuru, Y. (2006). Improving adolescent students’ reading comprehension with iSTART. Intern. J. Educ. Res. 34, 147–171. doi: 10.2190/1ru5-hdtj-a5c8-jvwe

Meyer, B. J., Wijekumar, K. K., and Lin, Y. C. (2011). Individualizing a web-based structure strategy intervention for fifth graders’ comprehension of nonfiction. J. Educ. Psychol. 103, 140–168. doi: 10.1037/a0021606

Mich, O., Pianta, E., and Mana, N. (2013). Interactive stories and exercises with dynamic feedback for improving reading comprehension skills in deaf children. Comput. Educ. 65, 34–44. doi: 10.1016/j.compedu.2013.01.016

Moran, J., Ferdig, R. E., Pearson, P. D., Wardrop, J., and Blomeyer, R. L. Jr. (2008). Technology and reading performance in the middle-school grades: a meta-analysis with recommendations for policy and practice. J. Liter. Res. 40, 6–58. doi: 10.1080/10862960802070483

Niedo, J., Lee, Y. L., Breznitz, Z., and Berninger, V. W. (2014). Computerized silent reading rate and strategy instruction for fourth graders at risk in silent reading rate. Learn. Disabil. Q. 37, 100–110. doi: 10.1177/0731948713507263

Oakhill, J. V., and Cain, K. (2012). The precursors of reading ability in young readers: evidence from a four-year longitudinal study. Sci. Stud. Read. 162, 91–121. doi: 10.1080/10888438.2010.529219

Oakhill, J. V., Cain, K., and Bryant, P. E. (2003). The dissociation of word reading and text comprehension: evidence from component skills. Lang. Cogn. Process. 18, 443–468. doi: 10.1080/01690960344000008

O’Reilly, T. P., Sinclair, G. P., and McNamara, D. S. (2004). “iSTART: A web-based reading strategy intervention that improves students’ science comprehension,” in Proceedings of the IADIS International Conference Cognition and Exploratory Learning in Digital Age: CELDA 2004 , eds D. G. Kinshuk and P. Isaias (Lisbon: IADIS), 173–180.

Ortlieb, E., Sargent, S., and Moreland, M. (2014). Evaluating the efficacy of using a digital reading environment to improve reading comprehension within a reading clinic. Read. Psychol. 35, 397–421. doi: 10.1080/02702711.2012.683236

Potocki, A., Ecalle, J., and Magnan, A. (2013). Effects of computer-assisted comprehension training in less skilled comprehenders in second grade: a one-year follow-up study. Comput. Educ. 63, 131–140. doi: 10.1016/j.compedu.2012.12.011

Potocki, A., Magnan, A., and Ecalle, J. (2015). Computerized trainings in four groups of struggling readers: specific effects on word reading and comprehension. Res. Dev. Disabil. 45, 83–92. doi: 10.1016/j.ridd.2015.07.016

Rideout, V. J., Foehr, U. G., and Roberts, D. F. (2010). Generation M 2: Media in the Lives of 8-to 18-Year-Olds. San Francisco, CA: Henry J. Kaiser Family Foundation.

Singer, L. M., and Alexander, P. A. (2017). Reading on paper and digitally: what the past decades of empirical research reveal. Rev. Educ. Res. 87, 1007–1041. doi: 10.3102/0034654317722961

Viola, F., and Carretti, B. (2019). Cambiamentinelleabilità di letturanelcorso di unostesso anno scolastico [Changes in readingskillsduring the sameschoolyear]. Dislessia 16, 147–159. doi: 10.14605/DIS1621902

von Bastian, C. C., and Oberauer, K. (2014). Effects and mechanisms of working memory training: a review. Psychol. Res. 78, 803–820. doi: 10.1007/s00426-013-0524-526

Wechsler, D. (2003). Wechsler Intelligence Scale For Children–Fourth Edition (WISC-IV). San Antonio, TX: The Psychological Corporation.

Wijekumar, K. K., Meyer, B. J., and Lei, P. (2012). Large-scale randomized controlled trial with 4th graders using intelligent tutoring of the structure strategy to improve nonfiction reading comprehension. Educ. Technol. Res. Dev. 60, 987–1013. doi: 10.1007/s11423-012-9263-4

Wijekumar, K. K., Meyer, B. J., and Lei, P. (2013). High-fidelity implementation of web-based intelligent tutoring system improves fourth and fifth graders content area reading comprehension. Comput. Educ. 68, 366–379. doi: 10.1016/j.compedu.2013.05.021

Wijekumar, K. K., Meyer, B. J., and Lei, P. (2017). Web-based text structure strategy instruction improves seventh graders’ content area reading comprehension. J. Educ. Psychol. 109, 741–760. doi: 10.1037/edu0000168

Woolley, G. (2011). “Reading comprehension,” in Reading Comprehension: Assisting Children With Learning Difficulties , ed Springer Science+Business Media (Dordrecht, NL: Springer), doi: 10.1007/978-94-007-1174-7_2

Yuill, N., and Oakhill, J. (1988). Effects of inference awareness training on poor reading comprehension. Appl. Cogn. Psychol. 2, 33–45. doi: 10.1002/acp.2350020105

Keywords : reading comprehension, training, distance rehabilitation program, digital device, Cloze app

Citation: Capodieci A, Cornoldi C, Doerr E, Bertolo L and Carretti B (2020) The Use of New Technologies for Improving Reading Comprehension. Front. Psychol. 11:751. doi: 10.3389/fpsyg.2020.00751

Received: 20 November 2019; Accepted: 27 March 2020; Published: 23 April 2020.

Reviewed by:

Copyright © 2020 Capodieci, Cornoldi, Doerr, Bertolo and Carretti. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Agnese Capodieci, [email protected] ; Laura Bertolo, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Preventing Reading Difficulties in Young Children (1998)

Chapter: part i: introduction to reading, part i introduction to reading.

Reading is a complex developmental challenge that we know to be intertwined with many other developmental accomplishments: attention, memory, language, and motivation, for example. Reading is not only a cognitive psycholinguistic activity but also a social activity.

Being a good reader in English means that a child has gained a functional knowledge of the principles of the English alphabetic writing system. Young children gain functional knowledge of the parts, products, and uses of the writing system from their ability to attend to and analyze the external sound structure of spoken words. Understanding the basic alphabetic principle requires an awareness that spoken language can be analyzed into strings of separable words, and words, in turn, into sequences of syllables and phonemes within syllables.

Beyond knowledge about how the English writing system works, though, there is a point in a child's growth when we expect "real reading" to start. Children are expected, without help, to read some unfamiliar texts, relying on the print and drawing meaning from it. There are many reasons why children have difficulty learning to read. These issues and problems led to the initiation of this study.

Even though quite accurate estimates can be made on the basis of known risk factors, it is still difficult to predict precisely which young children will have difficulty learning to read. We therefore propose that prevention efforts must reach all children. To wait to initiate treatment until the child has been diagnosed with a specific disability is too late. However, we can begin treatment of conditions associated with reading problems, for example, hearing impairments.

Ensuring success in reading requires different levels of effort for different segments of the population. The prevention and intervention efforts described in this report can be thought of in terms of three levels (Caplan and Grunebaum, 1967, cited in Simeonsson, 1994; Pianta, 1990; and Needlman, 1997).  Primary prevention is concerned with reducing the number of new cases (incidence) of an identified condition or problem in the population, such as ensuring that all children attend schools in which instruction is coherent and competent.

Secondary prevention is concerned with reducing the number of existing cases (prevalence) of an identified condition or problem in the population. Secondary prevention likewise involves the promotion of compensatory skills and behaviors. Children who are growing up in poverty, for example, may need excellent, enriched preschool environments or schools that address their particular learning needs with highly effective and focused instruction. The extra effort is focused on children at higher risk of developing reading difficulties but before any serious, long-term deficit has emerged.

Tertiary prevention is concerned with reducing the complications associated with identified problem, or conditions. Programs, strategies, and interventions at this level have an explicit remedial or rehabilitative focus. If children demonstrate inadequate progress under secondary prevention conditions, they may need instruction that is specially designed and supplemental—special education, tutoring from a reading specialist—to their current instruction.

While most children learn to read fairly well, there remain many young Americans whose futures are imperiled because they do not read well enough to meet the demands of our competitive, technology-driven society. This book explores the problem within the context of social, historical, cultural, and biological factors.

Recommendations address the identification of groups of children at risk, effective instruction for the preschool and early grades, effective approaches to dialects and bilingualism, the importance of these findings for the professional development of teachers, and gaps that remain in our understanding of how children learn to read. Implications for parents, teachers, schools, communities, the media, and government at all levels are discussed.

The book examines the epidemiology of reading problems and introduces the concepts used by experts in the field. In a clear and readable narrative, word identification, comprehension, and other processes in normal reading development are discussed.

Against the background of normal progress, Preventing Reading Difficulties in Young Children examines factors that put children at risk of poor reading. It explores in detail how literacy can be fostered from birth through kindergarten and the primary grades, including evaluation of philosophies, systems, and materials commonly used to teach reading.

READ FREE ONLINE

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

Switch between the Original Pages , where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol

Levels of Reading Comprehension in Higher Education: Systematic Review and Meta-Analysis

Cristina de-la-peña.

1 Departamento de Métodos de Investigación y Diagnóstico en Educación, Universidad Internacional de la Rioja, Logroño, Spain

María Jesús Luque-Rojas

2 Department of Theory and History of Education and Research Methods and Diagnosis in Education, University of Malaga, Málaga, Spain

Associated Data

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Higher education aims for university students to produce knowledge from the critical reflection of scientific texts. Therefore, it is necessary to develop a deep mental representation of written information. The objective of this research was to determine through a systematic review and meta-analysis the proportion of university students who have an optimal performance at each level of reading comprehension. Systematic review of empirical studies has been limited from 2010 to March 2021 using the Web of Science, Scopus, Medline, and PsycINFO databases. Two reviewers performed data extraction independently. A random-effects model of proportions was used for the meta-analysis and heterogeneity was assessed with I 2 . To analyze the influence of moderating variables, meta-regression was used and two ways were used to study publication bias. Seven articles were identified with a total sample of the seven of 1,044. The proportion of students at the literal level was 56% (95% CI = 39–72%, I 2 = 96.3%), inferential level 33% (95% CI = 19–46%, I 2 = 95.2%), critical level 22% (95% CI = 9–35%, I 2 = 99.04%), and organizational level 22% (95% CI = 6–37%, I 2 = 99.67%). Comparing reading comprehension levels, there is a significant higher proportion of university students who have an optimal level of literal compared to the rest of the reading comprehension levels. The results have to be interpreted with caution but are a guide for future research.

Introduction

Reading comprehension allows the integration of knowledge that facilitates training processes and successful coping with academic and personal situations. In higher education, this reading comprehension has to provide students with autonomy to self-direct their academic-professional learning and provide critical thinking in favor of community service ( UNESCO, 2009 ). However, research in recent years ( Bharuthram, 2012 ; Afflerbach et al., 2015 ) indicates that a part of university students are not prepared to successfully deal with academic texts or they have reading difficulties ( Smagorinsky, 2001 ; Cox et al., 2014 ), which may limit academic training focused on written texts. This work aims to review the level of reading comprehension provided by studies carried out in different countries, considering the heterogeneity of existing educational models.

The level of reading comprehension refers to the type of mental representation that is made of the written text. The reader builds a mental model in which he can integrate explicit and implicit data from the text, experiences, and previous knowledge ( Kucer, 2016 ; van den Broek et al., 2016 ). Within the framework of the construction-integration model ( Kintsch and van Dijk, 1978 ; Kintsch, 1998 ), the most accepted model of reading comprehension, processing levels are differentiated, specifically: A superficial level that identifies or memorizes data forming the basis of the text and a deep level in which the text situation model is elaborated integrating previous experiences and knowledge. At these levels of processing, the cognitive strategies used, are different according to the domain-learning model ( Alexander, 2004 ) from basic coding to a transformation of the text. In the scientific literature, there are investigations ( Yussof et al., 2013 ; Ulum, 2016 ) that also identify levels of reading comprehension ranging from a literal level of identification of ideas to an inferential and critical level that require the elaboration of inferences and the data transformation.

Studies focused on higher education ( Barletta et al., 2005 ; Yáñez Botello, 2013 ) show that university students are at a literal or basic level of understanding, they often have difficulties in making inferences and recognizing the macrostructure of the written text, so they would not develop a model of a situation of the text. These scientific results are in the same direction as the research on reading comprehension in the mother tongue in the university population. Bharuthram (2012) indicates that university students do not access or develop effective strategies for reading comprehension, such as the capacity for abstraction and synthesis-analysis. Later, Livingston et al. (2015) find that first-year education students present limited reading strategies and difficulties in understanding written texts. Ntereke and Ramoroka (2017) found that only 12.4% of students perform well in a reading comprehension task, 34.3% presenting a low level of execution in the task.

Factors related to the level of understanding of written information are the mode of presentation of the text (printed vs. digital), the type of metacognitive strategies used (planning, making inferences, inhibition, monitoring, etc.), the type of text and difficulties (novel vs. a science passage), the mode of writing (text vs. multimodal), the type of reading comprehension task, and the diversity of the student. For example, several studies ( Tuncer and Bahadir, 2014 ; Trakhman et al., 2019 ; Kazazoglu, 2020 ) indicate that reading is more efficient with better performance in reading comprehension tests in printed texts compared to the same text in digital and according to Spencer (2006) college students prefer to read in print vs. digital texts. In reading the written text, metacognitive strategies are involved ( Amril et al., 2019 ) but studies ( Channa et al., 2018 ) seem to indicate that students do not use them for reading comprehension, specifically; Korotaeva (2012) finds that only 7% of students use them. Concerning the type of text and difficulties, for Wolfe and Woodwyk (2010) , expository texts benefit more from the construction of a situational model of the text than narrative texts, although Feng (2011) finds that expository texts are more difficult to read than narrative texts. Regarding the modality of the text, Mayer (2009) and Guo et al. (2020) indicate that multimodal texts that incorporate images into the text positively improve reading comprehension. In a study of Kobayashi (2002) using open questions, close, and multiple-choice shows that the type and format of the reading comprehension assessment test significantly influence student performance and that more structured tests help to better differentiate the good ones and the poor ones in reading comprehension. Finally, about student diversity, studies link reading comprehension with the interest and intrinsic motivation of university students ( Cartwright et al., 2019 ; Dewi et al., 2020 ), with gender ( Saracaloglu and Karasakaloglu, 2011 ), finding that women present a better level of reading comprehension than men and with knowledge related to reading ( Perfetti et al., 1987 ). In this research, it was controlled that all were printed and unimodal texts, that is, only text. This is essential because the cognitive processes involved in reading comprehension can vary with these factors ( Butcher and Kintsch, 2003 ; Xu et al., 2020 ).

The Present Study

Regardless of the educational context, in any university discipline, preparing essays or developing arguments are formative tasks that require a deep level of reading comprehension (inferences and transformation of information) that allows the elaboration of a situation model, and not having this level can lead to limited formative learning. Therefore, the objective of this research was to know the state of reading comprehension levels in higher education; specifically, the proportion of university students who perform optimally at each level of reading comprehension. It is important to note that there is not much information about the different levels in university students and that it is the only meta-analytic review that explores different levels of reading comprehension in this educational stage. This is a relevant issue because the university system requires that students produce knowledge from the critical reflection of scientific texts, preparing them for innovation, employability, and coexistence in society.

Materials and Methods

Eligibility criteria: inclusion and exclusion.

Empirical studies written in Spanish or English are selected that analyze the reading comprehension level in university students.

The exclusion criteria are as follows: (a) book chapters or review books or publications; (b) articles in other languages; (c) studies of lower educational levels; (d) articles that do not identify the age of the sample; (e) second language studies; (f) students with learning difficulties or other disorders; (g) publications that do not indicate the level of reading comprehension; (h) studies that relate reading competence with other variables but do not report reading comprehension levels; (i) pre-post program application work; (j) studies with experimental and control groups; (k) articles comparing pre-university stages or adults; (l) publications that use multi-texts; (m) studies that use some type of technology (computer, hypertext, web, psychophysiological, online questionnaire, etc.); and (n) studies unrelated to the subject of interest.

Only those publications that meet the following criteria are included as: (a) be empirical research (article, thesis, final degree/master’s degree, or conference proceedings book); (b) university stage; (c) include data or some measure on the level of reading comprehension that allows calculating the effect size; (d) written in English or Spanish; (e) reading comprehension in the first language or mother tongue; and (f) the temporary period from January 2010 to March 2021.

Search Strategies

A three-step procedure is used to select the studies included in the meta-analysis. In the first step, a review of research and empirical articles in English and Spanish from January 2010 to March 2021. The search is carried out in online databases of languages in Spanish and English, such as Web of Science (WoS), Scopus, Medline, and PsycINFO, to review empirical productions that analyze the level of reading comprehension in university students. In the second step, the following terms (titles, abstracts, keywords, and full text) are used to select the articles: Reading comprehension and higher education, university students, in Spanish and English, combined with the Boolean operators AND and OR. In the last step, secondary sources, such as the Google search engine, Theseus, and references in publications, are explored.

The search reports 4,294 publications (articles, theses, and conference proceedings books) in the databases and eight records of secondary references, specifically, 1989 from WoS, 2001 from Scopus, 42 from Medline, and 262 of PsycINFO. Of the total (4,294), 1,568 are eliminated due to duplications, leaving 2,734 valid records. Next, titles and abstracts are reviewed and 2,659 are excluded because they do not meet the inclusion criteria. The sample of 75 publications is reduced to 40 articles, excluding 35 because the full text cannot be accessed (the authors were contacted but did not respond), the full text did not show specific statistical data, they used online questionnaires or computerized presentations of the text. Finally, seven articles in Spanish were selected for use in the meta-analysis of the reading comprehension level of university students. Data additional to those included in the articles were not requested from the selected authors.

The PRISMA-P guidelines ( Moher et al., 2015 ) are followed to perform the meta-analysis and the flow chart for the selection of publications relevant to the subject is exposed (Figure 1) .

Flow diagram for the selection of articles.

Encoding Procedure

This research complies with what is established in the manual of systematic reviews ( Higgins and Green, 2008 ) in which clear objectives, specific search terms, and eligibility criteria for previously defined works are established. Two independent coders, reaching a 100% agreement, carry out the study search process. Subsequently, the research is codified, for this, a coding protocol is used as a guide to help resolve the ambiguities between the coders; the proposals are reflected and discussed and discrepancies are resolved, reaching a degree of agreement between the two coders of 97%.

For all studies, the reference, country, research objective, sample size, age and gender, reading comprehension test, other tests, and reading comprehension results were coded in percentages. All this information was later systematized in Table 1 .

Results of the empirical studies included in the meta-analysis.

In relation to the type of reading comprehension level, it was coded based on the levels of the scientific literature as follows: 1 = literal; 2 = inferential; 3 = critical; and 4 = organizational.

Regarding the possible moderating variables, it was coded if the investigations used a standardized reading comprehension measure (value = 1) or non-standardized (value = 0). This research considers the standardized measures of reading comprehension as the non-standardized measures created by the researchers themselves in their studies or questionnaires by other authors. By the type of evaluation test, we encode between multiple-choice (value = 0) or multiple-choices plus open question (value = 1). By type of text, we encode between argumentative (value = 1) or unknown (value = 0). By the type of career, we encode social sciences (value = 1) or other careers (health sciences; value = 0). Moreover, by the type of publication, we encode between article (value = 1) or doctoral thesis (value = 0).

Effect Size and Statistical Analysis

This descriptive study with a sample k = 7 and a population of 1,044 university students used a continuous variable and the proportions were used as the effect size to analyze the proportion of students who had an optimal performance at each level of reading comprehension. As for the percentages of each level of reading comprehension of the sample, they were transformed into absolute frequencies. A random-effects model ( Borenstein et al., 2009 ) was used as the effect size. These random-effects models have a greater capacity to generalize the conclusions and allow estimating the effects of different sources of variation (moderating variables). The DerSimonian and Laird method ( Egger et al., 2001 ) was used, calculating raw proportion and for each proportion its standard error, value of p and 95% confidence interval (CI).

To examine sampling variability, Cochran’s Q test (to test the null hypothesis of homogeneity between studies) and I 2 (proportion of variability) were used. According to Higgins et al. (2003) , if I 2 reaches 25%, it is considered low, if it reaches 50% and if it exceeds 75% it is considered high. A meta-regression analysis was used to investigate the effect of the moderator variables (type of measure, type of evaluation test, type of text, type of career, and type of publication) in each level of reading comprehension of the sample studies. For each moderating variable, all the necessary statistics were calculated (estimate, standard error, CI, Q , and I 2 ).

To compare the effect sizes of each level (literal, inferential, critical, and organizational) of reading comprehension, the chi-square test for the proportion recommended by Campbell (2007) was used.

Finally, to analyze publication bias, this study uses two ways: Rosenthal’s fail-safe number and regression test. Rosenthal’s fail-safe number shows the number of missing studies with null effects that would make the previous correlations insignificant ( Borenstein et al., 2009 ). When the values are large there is no bias. In the regression test, when the regression is not significant, there is no bias.

The software used to classify and encode data and produce descriptive statistics was with Microsoft Excel and the Jamovi version 1.6 free software was used to perform the meta-analysis.

The results of the meta-analysis are presented in three parts: the general descriptive analysis of the included studies; the meta-analytic analysis with the effect size, heterogeneity, moderating variables, and comparison of effect sizes; and the study of publication bias.

Overview of Included Studies

The search carried out of the scientific literature related to the subject published from 2010 to March 2021 generated a small number of publications, because it was limited to the higher education stage and required clear statistical data on reading comprehension.

Table 1 presents all the publications reviewed in this meta-analysis with a total of students evaluated in the reviewed works that amounts to 1,044, with the smallest sample size of 30 ( Del Pino-Yépez et al., 2019 ) and the largest with 570 ( Guevara Benítez et al., 2014 ). Regarding gender, 72% women and 28% men were included. Most of the sample comes from university degrees in social sciences, such as psychology and education (71.42%) followed by health sciences (14.28%) engineering and a publication (14.28%) that does not indicate origin. These publications selected according to the inclusion criteria for the meta-analysis come from more countries with a variety of educational systems, but all from South America. Specifically, the countries that have more studies are Mexico (28.57%) and Colombia, Chile, Bolivia, Peru, and Ecuador with 14.28% each, respectively. The years in which they were published are 2.57% in 2018 and 2016 and 14.28% in 2019, 2014, and 2013.

A total of 57% of the studies analyze four levels of reading comprehension (literal, inferential, critical, and organizational) and 43% investigate three levels of reading comprehension (literal, inferential, and critical). Based on the moderating variables, 57% of the studies use standardized reading comprehension measures and 43% non-standardized measures. According to the evaluation test used, 29% use multiple-choice questions and 71% combine multiple-choice questions plus open questions. 43% use an argumentative text and 57% other types of texts (not indicated in studies). By type of career, 71% are students of social sciences and 29% of other different careers, such as engineering or health sciences. In addition, 71% are articles and 29% with research works (thesis and degree works).

Table 2 shows the reading comprehension assessment instruments used by the authors of the empirical research integrated into the meta-analysis.

Reading comprehension assessment tests used in higher education.

Meta-Analytic Analysis of the Level of Reading Comprehension

The literal level presents a mean proportion effect size of 56% (95% CI = 39–72%; Figure 2 ). The variability between the different samples of the literal level of reading comprehension was significant ( Q = 162.066, p < 0.001; I 2 = 96.3%). No moderating variable used in this research had a significant contribution to heterogeneity: type of measurement ( p = 0.520), type of test ( p = 0.114), type of text ( p = 0.520), type of career ( p = 0.235), and type of publication ( p = 0.585). The high variability is explained by other factors not considered in this work, such as the characteristics of the students (cognitive abilities) or other issues.

Forest plot of literal level.

The inferential level presents a mean proportion effect size of 33% (95% CI = 19–46%; Figure 3 ). The variability between the different samples of the inferential level of reading comprehension was significant ( Q = 125.123, p < 0.001; I 2 = 95.2%). The type of measure ( p = 0.011) and the type of text ( p = 0.011) had a significant contribution to heterogeneity. The rest of the variables had no significance: type of test ( p = 0.214), type of career ( p = 0.449), and type of publication ( p = 0.218). According to the type of measure, the proportion of students who have an optimal level in inferential administering a standardized test is 28.7% less than when a non-standardized test is administered. The type of measure reduces variability by 2.57% and explains the differences between the results of the studies at the inferential level. According to the type of text, the proportion of students who have an optimal level in inferential using an argumentative text is 28.7% less than when using another type of text. The type of text reduces the variability by 2.57% and explains the differences between the results of the studies at the inferential level.

Forest plot of inferential level.

The critical level has a mean effect size of the proportion of 22% (95% CI = 9–35%; Figure 4 ). The variability between the different samples of the critical level of reading comprehension was significant ( Q = 627.044, p < 0.001; I 2 = 99.04%). No moderating variable used in this research had a significant contribution to heterogeneity: type of measurement ( p = 0.575), type of test ( p = 0.691), type of text ( p = 0.575), type of career ( p = 0.699), and type of publication ( p = 0.293). The high variability is explained by other factors not considered in this work, such as the characteristics of the students (cognitive abilities).

Forest plot of critical level.

The organizational level presents a mean effect size of the proportion of 22% (95% CI = 6–37%; Figure 5 ). The variability between the different samples of the organizational level of reading comprehension was significant ( Q = 1799.366, p < 0.001; I 2 = 99.67%). The type of test made a significant contribution to heterogeneity ( p = 0.289). The other moderating variables were not significant in this research: type of measurement ( p = 0.289), type of text ( p = 0.289), type of career ( p = 0.361), and type of publication ( p = 0.371). Depending on the type of test, the proportion of students who have an optimal level in organizational with multiple-choices tests plus open questions is 37% higher than while using only multiple-choice tests. The type of text reduces the variability by 0.27% and explains the differences between the results of the studies at the organizational level.

Forest plot of organizational level.

Table 3 shows the difference between the estimated effect sizes and the significance. There is a larger proportion of students having an optimal level of reading comprehension at the literal level compared to the inferential, critical, and organizational level; an optimal level of reading comprehension at the inferential level vs. the critical and organizational level.

Results of effect size comparison.

Analysis of Publication Bias

This research uses two ways to verify the existence of bias independently of the sample size. Table 4 shows the results and there is no publication bias at any level of reading comprehension.

Publication bias results.

This research used a systematic literature search and meta-analysis to provide estimates of the number of cases of university students who have an optimal level in the different levels of reading comprehension. All the information available on the subject at the international level was analyzed using international databases in English and Spanish, but the potentially relevant publications were limited. Only seven Spanish language studies were identified internationally. In these seven studies, the optimal performance at each level of reading comprehension varied, finding heterogeneity associated with the very high estimates, which indicates that the summary estimates have to be interpreted with caution and in the context of the sample and the variables used in this meta-analysis.

In this research, the effects of the type of measure, type of test, type of text, type of career, and type of publication have been analyzed. Due to the limited information in the publications, it was not possible to assess the effect of any more moderating variables.

We found that some factors significantly influence heterogeneity according to the level of reading comprehension considered. The type of measure influenced the optimal performance of students in the inferential level of reading comprehension; specifically, the proportion of students who have an optimal level in inferential worsens if the test is standardized. Several studies ( Pike, 1996 ; Koretz, 2002 ) identify differences between standardized and non-standardized measures in reading comprehension and a favor of non-standardized measures developed by the researchers ( Pyle et al., 2017 ). The ability to generate inferences of each individual may difficult to standardize because each person differently identifies the relationship between the parts of the text and integrates it with their previous knowledge ( Oakhill, 1982 ; Cain et al., 2004 ). This mental representation of the meaning of the text is necessary to create a model of the situation and a deep understanding ( McNamara and Magliano, 2009 ; van den Broek and Espin, 2012 ).

The type of test was significant for the organizational level of reading comprehension. The proportion of students who have an optimal level in organizational improves if the reading comprehension assessment test is multiple-choice plus open questions. The organizational level requires the reordering of written information through analysis and synthesis processes ( Guevara Benítez et al., 2014 ); therefore, it constitutes a production task that is better reflected in open questions than in reproduction questions as multiple choice ( Dinsmore and Alexander, 2015 ). McNamara and Kintsch (1996) identify that open tasks require an effort to make inferences related to previous knowledge and multidisciplinary knowledge. Important is to indicate that different evaluation test formats can measure different aspects of reading comprehension ( Zheng et al., 2007 ).

The type of text significantly influenced the inferential level of reading comprehension. The proportion of students who have an optimal level in inferential decreases with an argumentative text. The expectations created before an argumentative text made it difficult to generate inferences and, therefore, the construction of the meaning of the text. This result is in the opposite direction to the study by Diakidoy et al. (2011) who find that the refutation text, such as the argumentative one, facilitates the elaboration of inferences compared to other types of texts. It is possible that the argumentative text, given its dialogical nature of arguments and counterarguments, with a subject unknown by the students, has determined the decrease of inferences based on their scarce previous knowledge of the subject, needing help to elaborate the structure of the text read ( Reznitskaya et al., 2007 ). It should be pointed out that in meta-analysis studies, 43% use argumentative texts. Knowing the type of the text is relevant for generating inferences, for instance, according to Baretta et al. (2009) the different types of text are processed differently in the brain generating more or fewer inferences; specifically, using the N400 component, they find that expository texts generate more inferences from the text read.

For the type of career and the type of publication, no significance was found at any level of reading comprehension in this sample. This seems to indicate that university students have the same level of performance in tasks of literal, critical inferential, and organizational understanding regardless of whether they are studying social sciences, health sciences, or engineering. Nor does the type of publication affect the state of the different levels of reading comprehension in higher education.

The remaining high heterogeneity at all levels of reading comprehension was not captured in this review, indicating that there are other factors, such as student characteristics, gender, or other issues, that are moderating and explaining the variability at the literal, inferential, critical, and organizational reading comprehension in university students.

To the comparison between the different levels of reading comprehension, the literal level has a significantly higher proportion of students with an optimal level than the inferential, critical, and organizational levels. The inferential level has a significantly higher proportion of students with an optimal level than the critical and organizational levels. This corresponds with data from other investigations ( Márquez et al., 2016 ; Del Pino-Yépez et al., 2019 ) that indicate that the literal level is where university students execute with more successes, being more difficult and with less success at the inferential, organizational, and critical levels. This indicates that university students of this sample do not generate a coherent situation model that provides them with a global mental representation of the read text according to the model of Kintsch (1998) , but rather they make a literal analysis of the explicit content of the read text. This level of understanding can lead to less desirable results in educational terms ( Dinsmore and Alexander, 2015 ).

The educational implications of this meta-analysis in this sample are aimed at making universities aware of the state of reading comprehension levels possessed by university students and designing strategies (courses and workshops) to optimize it by improving the training and employability of students. Some proposals can be directed to the use of reflection tasks, integration of information, graphic organizers, evaluation, interpretation, nor the use of paraphrasing ( Rahmani, 2011 ). Some studies ( Hong-Nam and Leavell, 2011 ; Parr and Woloshyn, 2013 ) demonstrate the effectiveness of instructional courses in improving performance in reading comprehension and metacognitive strategies. In addition, it is necessary to design reading comprehension assessment tests in higher education that are balanced, validated, and reliable, allowing to have data for the different levels of reading comprehension.

Limitations and Conclusion

This meta-analysis can be used as a starting point to report on reading comprehension levels in higher education, but the results should be interpreted with caution and in the context of the study sample and variables. Publications without sufficient data and inaccessible articles, with a sample of seven studies, may have limited the international perspective. The interest in studying reading comprehension in the mother tongue, using only unimodal texts, without the influence of technology and with English and Spanish has also limited the review. The limited amount of data in the studies has limited meta-regression.

This review is a guide to direct future research, broadening the study focus on the level of reading comprehension using digital technology, experimental designs, second languages, and investigations that relate reading comprehension with other factors (gender, cognitive abilities, etc.) that can explain the heterogeneity in the different levels of reading comprehension. The possibility of developing a comprehensive reading comprehension assessment test in higher education could also be explored.

This review contributes to the scientific literature in several ways. In the first place, this meta-analytic review is the only one that analyzes the proportion of university students who have an optimal performance in the different levels of reading comprehension. This review is made with international publications and this topic is mostly investigated in Latin America. Second, optimal performance can be improved at all levels of reading comprehension, fundamentally inferential, critical, and organizational. The literal level is significantly the level of reading comprehension with the highest proportion of optimal performance in university students. Third, the students in this sample have optimal performance at the inferential level when they are non-argumentative texts and non-standardized measures, and, in the analyzed works, there is optimal performance at the organizational level when multiple-choice questions plus open questions are used.

The current research is linked to the research project “Study of reading comprehension in higher education” of Asociación Educar para el Desarrollo Humano from Argentina.

Data Availability Statement

Author contributions.

Cd-l-P had the idea for the article and analyzed the data. ML-R searched the data. Cd-l-P and ML-R selected the data and contributed to the valuable comments and manuscript writing. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor declared a shared affiliation though no other collaboration with one of the authors ML-R at the time of the review.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Funding. This paper was funded by the Universidad Internacional de la Rioja and Universidad de Málaga.

• Afflerbach P., Cho B.-Y., Kim J.-Y. (2015). Conceptualizing and assessing higher-order thinking in reading . Theory Pract. 54 , 203–212. 10.1080/00405841.2015.1044367 [ CrossRef ] [ Google Scholar ]
• Alexander P. A. (2004). “ A model of domain learning: reinterpreting expertise as a multidimensional, multistage process ,” in Motivation, Emotion, and Cognition: Integrative Perspectives on Intellectual Functioning and Development. eds. Dai D. Y., Sternberg R. J. (Mahwah, NJ: Erlbaum; ), 273–298. [ Google Scholar ]
• Amril A., Hasanuddin W. S., Atmazaki (2019). The contributions of reading strategies and reading frequencies toward students’ reading comprehension skill in higher education . Int. J. Eng. Adv. Technol. (IJEAT) 8 , 593–595. 10.35940/ijeat.F1105.0986S319 [ CrossRef ] [ Google Scholar ]
• Baretta L., Braga Tomitch L. M., MacNair N., Kwan Lim V., Waldie K. E. (2009). Inference making while reading narrative and expository texts: an ERP study . Psychol. Neurosci. 2 , 137–145. 10.3922/j.psns.2009.2.005 [ CrossRef ] [ Google Scholar ]
• Barletta M., Bovea V., Delgado P., Del Villar L., Lozano A., May O., et al.. (2005). Comprensión y Competencias Lectoras en Estudiantes Universitarios. Barranquilla: Uninorte. [ Google Scholar ]
• Bharuthram S. (2012). Making a case for the teaching of reading across the curriculum in higher education . S. Afr. J. Educ. 32 , 205–214. 10.15700/saje.v32n2a557 [ CrossRef ] [ Google Scholar ]
• Borenstein M., Hedges L. V., Higgins J. P. T., Rothstein H. R. (2009). Introduction to Meta-Analysis. United Kingdom: John Wiley and Sons, Ltd, 45–49. [ Google Scholar ]
• Butcher K. R., Kintsch W. (2003). “ Text comprehension and discourse processing ,” in Handbook of Psychology: Experimental Psychology. 2nd Edn . Vol . 4 . eds. Healy A. F., Proctor R. W., Weiner I. B. (New Jersey: John Wiley and Sons, Inc.), 575–595. [ Google Scholar ]
• Cain K., Oakhill J., Bryant P. (2004). Children’s reading comprehension ability: concurrent prediction by working memory, verbal ability, and component skills . J. Educ. Psychol. 96 , 31–42. 10.1037/0022-0663.96.1.31 [ CrossRef ] [ Google Scholar ]
• Campbell I. (2007). Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations . Stat. Med. 26 , 3661–3675. 10.1002/sim.2832, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cartwright K. B., Lee S. A., Barber A. T., DeWyngaert L. U., Lane A. B., Singleton T. (2019). Contributions of executive function and cognitive intrinsic motivation to university students’ reading comprehension . Read. Res. Q. 55 , 345–369. 10.1002/rrq.273 [ CrossRef ] [ Google Scholar ]
• Channa M. A., Abassi A. M., John S., Sahito J. K. M. (2018). Reading comprehension and metacognitive strategies in first-year engineering university students in Pakistan . Int. J. Engl. Ling. 8 , 78–87. 10.5539/ijel.v8n6p78 [ CrossRef ] [ Google Scholar ]
• Cox S. R., Friesner D. L., Khayum M. (2014). Do Reading skills courses help underprepared readers achieve academic success in college? J. College Reading and Learn. 33 , 170–196. 10.1080/10790195.2003.10850147 [ CrossRef ] [ Google Scholar ]
• Del Pino-Yépez G. M., Saltos-Rodríguez L. J., Moreira-Aguayo P. Y. (2019). Estrategias didácticas para el afianzamiento de la comprensión lectora en estudiantes universitarios . Revista científica Dominio de las Ciencias 5 , 171–187. 10.23857/dc.v5i1.1038 [ CrossRef ] [ Google Scholar ]
• Dewi R. S., Fahrurrozi, Hasanah U., Wahyudi A. (2020). Reading interest and Reading comprehension: a correlational study in Syarif Hidayatullah State Islamic University, Jakarta . Talent Dev. Excell. 12 , 241–250. [ Google Scholar ]
• Diakidoy I. N., Mouskounti T., Ioannides C. (2011). Comprehension and learning from refutation and expository texts . Read. Res. Q. 46 , 22–38. 10.1598/RRQ.46.1.2 [ CrossRef ] [ Google Scholar ]
• Dinsmore D. J., Alexander P. A. (2015). A multidimensional investigation of deep-level and surface-level processing . J. Exp. Educ. 84 , 213–244. 10.1080/00220973.2014.979126 [ CrossRef ] [ Google Scholar ]
• Egger M., Smith D., Altmand D. G. (2001). Systematic Reviews in Health Care: Meta-Analysis in Context. London: BMJ Publishing Group. [ Google Scholar ]
• Feng L. (2011). A short analysis of the text variables affecting reading and testing reading . Stud. Lit. Lang. 2 , 44–49. [ Google Scholar ]
• Figueroa Romero R. L., Castañeda Sánchez W., Tamay Carranza I. A. (2016). Nivel de comprensión lectora en los estudiantes del primer ciclo de la Universidad San Pedro, filial Caraz, 2016. (Trabajo de investigación, Universidad San Pedro). Repositorio Institucional USP. Available at: http://repositorio.usanpedro.edu.pe/bitstream/handle/USANPEDRO/305/PI1640418.pdf?sequence=1andisAllowed=y (Accessed February 15, 2021).
• Guevara Benítez Y., Guerra García J., Delgado Sánchez U., Flores Rubí C. (2014). Evaluación de distintos niveles de comprensión lectora en estudiantes mexicanos de Psicología . Acta Colombiana de Psicología 17 , 113–121. 10.14718/ACP.2014.17.2.12 [ CrossRef ] [ Google Scholar ]
• Guo D., Zhang S., Wright K. L., McTigue E. M. (2020). Do you get the picture? A meta-analysis of the effect of graphics on reading comprehension . AERA Open 6 , 1–20. 10.1177/2332858420901696 [ CrossRef ] [ Google Scholar ]
• Higgins J. P., Green S. (2008). Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration. [ Google Scholar ]
• Higgins J. P., Thompson S. G., Deeks J. J., Altman D. G. (2003). Measuring inconsistency in meta-analyses . BMJ 327 , 327–557. 10.1136/bmj.327.7414.557, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hong-Nam K., Leavell A. G. (2011). Reading strategy instruction, metacognitive awareness, and self-perception of striving college developmental readers . J. College Literacy Learn. 37 , 3–17. [ Google Scholar ]
• Kazazoglu S. (2020). Is printed text the best choice? A mixed-method case study on reading comprehension . J. Lang. Linguistic Stud. 16 , 458–473. 10.17263/jlls.712879 [ CrossRef ] [ Google Scholar ]
• Kintsch W. (1998). Comprehension: A Paradigm for Cognition. New York: Cambridge University Press. [ Google Scholar ]
• Kintsch W., van Dijk T. A. (1978). Toward a model of text comprehension and production . Psychol. Rev. 85 , 363–394. 10.1037/0033-295X.85.5.363 [ CrossRef ] [ Google Scholar ]
• Kobayashi M. (2002). Method effects on reading comprehension test performance: test organization and response format . Lang. Test. 19 , 193–220. 10.1191/0265532202lt227oa [ CrossRef ] [ Google Scholar ]
• Koretz D. (2002). Limitations in the use of achievement tests as measures of educators’ productivity . J. Hum. Resour. 37 , 752–777. 10.2307/3069616 [ CrossRef ] [ Google Scholar ]
• Korotaeva I. V. (2012). Metacognitive strategies in reading comprehension of education majors . Procedural-Social and Behav. Sci. 69 , 1895–1900. 10.1016/j.sbspro.2012.12.143 [ CrossRef ] [ Google Scholar ]
• Kucer S. B. (2016). Accuracy, miscues, and the comprehension of complex literary and scientific texts . Read. Psychol. 37 , 1076–1095. 10.1080/02702711.2016.1159632 [ CrossRef ] [ Google Scholar ]
• Livingston C., Klopper B., Cox S., Uys C. (2015). The impact of an academic reading program in the bachelor of education (intermediate and senior phase) degree . Read. Writ. 6 , 1–11. 10.4102/rw.v6i1.66 [ CrossRef ] [ Google Scholar ]
• Márquez H., Díaz C., Muñoz R., Fuentes R. (2016). Evaluación de los niveles de comprensión lectora en estudiantes universitarios pertenecientes a las carreras de Kinesiología y Nutrición y Dietética de la Universidad Andrés Bello, Concepción . Revista de Educación en Ciencias de la Salud 13 , 154–160. [ Google Scholar ]
• Mayer R. E. (ed.) (2009). “ Modality principle ,” in Multimedia Learning. United States: Cambridge University Press, 200–2020. [ Google Scholar ]
• McNamara D. S., Kintsch W. (1996). Learning from texts: effects of prior knowledge and text coherence . Discourse Process 22 , 247–288. 10.1080/01638539609544975 [ CrossRef ] [ Google Scholar ]
• McNamara D. S., Magliano J. (2009). “ Toward a comprehensive model of comprehension ,” in The psychology of learning and motivation. ed. Ross E. B. (New York: Elsevier; ), 297–384. [ Google Scholar ]
• Moher D., Shamseer L., Clarke M., Ghersi D., Liberati A., Petticrew M., et al.. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement . Syst. Rev. 4 :1. 10.1186/2046-4053-4-1, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ntereke B. B., Ramoroka B. T. (2017). Reading competency of first-year undergraduate students at the University of Botswana: a case study . Read. Writ. 8 :a123. 10.4102/rw.v8i1.123 [ CrossRef ] [ Google Scholar ]
• Oakhill J. (1982). Constructive processes in skilled and less skilled comprehenders’ memory for sentences . Br. J. Psychol. 73 , 13–20. 10.1111/j.2044-8295.1982.tb01785.x, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Parr C., Woloshyn V. (2013). Reading comprehension strategy instruction in a first-year course: an instructor’s self-study . Can. J. Scholarship Teach. Learn. 4 :3. 10.5206/cjsotl-rcacea.2013.2.3 [ CrossRef ] [ Google Scholar ]
• Perfetti C. A., Beck I., Bell L. C., Hughes C. (1987). Phonemic knowledge and learning to read are reciprocal: a longitudinal study of first-grade children . Merrill-Palmer Q. 33 , 283–319. [ Google Scholar ]
• Pike G. R. (1996). Limitations of using students’ self-reports of academic development as proxies for traditional achievement measures . Res. High. Educ. 37 , 89–114. 10.1007/BF01680043 [ CrossRef ] [ Google Scholar ]
• Pyle N., Vasquez A. C., Lignugaris B., Gillam S. L., Reutzel D. R., Olszewski A., et al.. (2017). Effects of expository text structure interventions on comprehension: a meta-analysis . Read. Res. Q. 52 , 469–501. 10.1002/rrq.179 [ CrossRef ] [ Google Scholar ]
• Rahmani M. (2011). Effects of note-taking training on reading comprehension and recall . Reading Matrix: An Int. Online J. 11 , 116–126. [ Google Scholar ]
• Reznitskaya A., Anderson R., Kuo L. J. (2007). Teaching and learning argumentation . Elem. Sch. J. 107 , 449–472. 10.1086/518623 [ CrossRef ] [ Google Scholar ]
• Sáez Sánchez B. K. (2018). La comprensión lectora en jóvenes universitarios de una escuela formadora de docentes . Revista Electrónica Científica de Investigación Educativa 4 , 609–618. [ Google Scholar ]
• Sanabria Mantilla T. R. (2018). Relación entre comprensión lectora y rendimiento académico en estudiantes de primer año de Psicología de la Universidad Pontificia Bolivariana. (Trabajo de grado, Universidad Pontificia Bolivariana). Repositorio Institucional UPB. Available at: https://repository.upb.edu.co/bitstream/handle/20.500.11912/5443/digital_36863.pdf?sequence=1andisAllowed=y (Accessed February 15, 2021).
• Saracaloglu A. S., Karasakaloglu N. (2011). An investigation of prospective teachers’ reading comprehension levels and study and learning strategies related to some variables . Egit. ve Bilim 36 , 98–115. [ Google Scholar ]
• Smagorinsky P. (2001). If meaning is constructed, what is it made from? Toward a cultural theory of reading . Rev. Educ. Res. 71 , 133–169. 10.3102/00346543071001133 [ CrossRef ] [ Google Scholar ]
• Spencer C. (2006). Research on learners’ preferences for reading from a printed text or a computer screen . J. Dist. Educ. 21 , 33–50. [ Google Scholar ]
• Trakhman L. M. S., Alexander P., Berkowitz L. E. (2019). Effects of processing time on comprehension and calibration in print and digital mediums . J. Exp. Educ. 87 , 101–115. 10.1080/00220973.2017.1411877 [ CrossRef ] [ Google Scholar ]
• Tuncer M., Bhadir F. (2014). Effect of screen reading and reading from printed out material on student success and permanency in introduction to computer lesson . Turk. Online J. Educ. Technol. 13 , 41–49. [ Google Scholar ]
• Ulum O. G. (2016). A descriptive content analysis of the extent of Bloom’s taxonomy in the reading comprehension questions of the course book Q: skills for success 4 reading and writing . Qual. Rep. 21 , 1674–1683. [ Google Scholar ]
• UNESCO (2009). “Conferencia mundial sobre la Educación Superior – 2009.” La nueva dinámica de la educación superior y la investigación para el cambio social y el desarrollo; July 5-8, 2009; Paris.
• van den Broek P., Espin C. A. (2012). Connecting cognitive theory and assessment: measuring individual differences in reading comprehension . Sch. Psychol. Rev. 43 , 315–325. 10.1080/02796015.2012.12087512 [ CrossRef ] [ Google Scholar ]
• van den Broek P., Mouw J. M., Kraal A. (2016). “ Individual differences in reading comprehension ,” in Handbook of Individual Differences in Reading: Reader, Text, and Context. ed. Afflerbach E. P. (New York: Routledge; ), 138–150. [ Google Scholar ]
• Wolfe M. B. W., Woodwyk J. M. (2010). Processing and memory of information presented in narrative or expository texts . Br. J. Educ. Psychol. 80 , 341–362. 10.1348/000709910X485700, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Xu Y., Wong R., He S., Veldre A., Andrews S. (2020). Is it smart to read on your phone? The impact of reading format and culture on the continued influence of misinformation . Mem. Cogn. 48 , 1112–1127. 10.3758/s13421-020-01046-0, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Yáñez Botello C. R. (2013). Caracterización de los procesos cognoscitivos y competencias involucrados en los niveles de comprensión lectora en Estudiantes Universitarios . Cuadernos Hispanoamericanos de Psicología 13 , 75–90. 10.18270/chps.v13i2.1350 [ CrossRef ] [ Google Scholar ]
• Yussof Y. M., Jamian A. R., Hamzah Z. A. Z., Roslan A. (2013). Students’ reading comprehension performance with emotional . Int. J. Edu. Literacy Stud. 1 , 82–88. 10.7575/aiac.ijels.v.1n.1p.82 [ CrossRef ] [ Google Scholar ]
• Zheng Y., Cheng L., Klinger D. A. (2007). Do test format in reading comprehension affect second-language students’ test performance differently? TESL Can. J. 25 , 65–78. 10.18806/tesl.v25i1.108 [ CrossRef ] [ Google Scholar ]

• Introduction to Reading

Why should we evaluate academic reading strategies?

Reading is fundamental to writing and research at University, but often gets overlooked – lecturers assume that students know how to read, and students assume there’s only one way to read – but neither of these things is necessarily true! There are ways to read that can improve information processing, can help with building an argument, and importantly for many students, can save lots of time!! — Academic Literacy Workshops, University of Cape Town [1]

The passage above makes an important point: most of us assume we know how to read for school. However, methods that may have been fine in the past (skimming, quick reviews, relying upon class lectures or notes) won’t hold up well as we move further into higher education.

This module defines a specific category of reading–academic reading–and discusses a range of skill sets and strategies that are specific to this type of reading.

It’s helpful to remember that academic reading is an act of performance . Rather than sitting back and passively receiving information we read in college, we will be asked to directly act upon that information in some way. We will be quizzed or tested. We will be asked to debate, analyze, or critique what we read. We will need to read closely, remember the text accurately, and compare it to other texts for style and content.

The following video addresses how academic reading is a key component of inter-related skills that demonstrate mastery of critical thinking.

As this video points out, as a reader in college you will be asked to embrace a “healthy skepticism” for every idea you come in contact with. This will take energy and work–it’s much easier to accept what others tell us on face value than to critically assess each idea that comes our way. However, education in the fullest sense means developing the tools for this critical response, building it into an automatic reflex that makes us thoughtful, engaged citizens of the world around us.

Learning Outcomes

• Evaluate various types of reading material
• Evaluate general reading strategies
• Evaluate reading strategies for specialized texts
• Evaluate vocabulary usage
• Evaluate thesis ideas of texts
• Evaluate supporting claims of texts
• Evaluate use of logic and structure in texts
• Evaluate summary skills for reading comprehension
• Hurst, Ellen, Ed. Academic Literacy Workshops: A Handbook for Students and Instructors . U of Capetown. 2011. ↵
• Why It Matters: Reading. Provided by : Lumen Learning. License : CC BY: Attribution
• Critical Thinking Skills by David Sotir. Authored by : University of Technology Sydney. Located at : https://youtu.be/9PsLktb7HTA . License : All Rights Reserved . License Terms : Standard YouTube License
• Table of Contents

Instructor Resources (available upon sign-in)

• Overview of Instructor Resources
• Quiz Survey

Reading: Types of Reading Material

• Outcome: Types of Reading Material
• Characteristics of Texts, Part 1
• Characteristics of Texts, Part 2
• Characteristics of Texts, Part 3
• Characteristics of Texts, Conclusion
• Self Check: Types of Writing

Reading: Reading Strategies

• Outcome: Reading Strategies
• The Rhetorical Situation
• Academic Reading Strategies
• Self Check: Reading Strategies

Reading: Specialized Reading Strategies

• Outcome: Specialized Reading Strategies
• Online Reading Comprehension
• How to Read Effectively in Math
• How to Read Effectively in the Social Sciences
• How to Read Effectively in the Sciences
• 5 Step Approach for Reading Charts and Graphs
• Self Check: Specialized Reading Strategies

Reading: Vocabulary

• Outcome: Vocabulary
• Strategies to Improve Your Vocabulary
• Using Context Clues
• The Relationship Between Reading and Vocabulary
• Self Check: Vocabulary

Reading: Thesis

• Outcome: Thesis
• Locating and Evaluating Thesis Statements
• The Organizational Statement
• Self Check: Thesis

Reading: Supporting Claims

• Outcome: Supporting Claims
• Types of Support
• Supporting Claims
• Self Check: Supporting Claims

Reading: Logic and Structure

• Outcome: Logic and Structure
• Rhetorical Modes
• Inductive and Deductive Reasoning
• Diagramming and Evaluating Arguments
• Logical Fallacies
• Evaluating Appeals to Ethos, Logos, and Pathos
• Self Check: Logic and Structure

Reading: Summary Skills

• Outcome: Summary Skills
• How to Annotate
• Paraphrasing
• Quote Bombs
• Summary Writing
• Self Check: Summary Skills
• Conclusion to Reading

Writing Process: Topic Selection

• Introduction to Writing Process
• Outcome: Topic Selection
• Starting a Paper
• Choosing and Developing Topics
• Back to the Future of Topics
• Developing Your Topic
• Self Check: Topic Selection

Writing Process: Prewriting

• Outcome: Prewriting
• Prewriting Strategies for Diverse Learners
• Rhetorical Context
• Working Thesis Statements
• Self Check: Prewriting

Writing Process: Finding Evidence

• Outcome: Finding Evidence
• Using Personal Examples
• Performing Background Research
• Listening to Sources, Talking to Sources
• Self Check: Finding Evidence

Writing Process: Organizing

• Outcome: Organizing
• Moving Beyond the Five-Paragraph Theme
• Introduction to Argument
• The Three-Story Thesis
• Organically Structured Arguments
• Logic and Structure
• The Perfect Paragraph
• Introductions and Conclusions
• Self Check: Organizing

Writing Process: Drafting

• Outcome: Drafting
• From Outlining to Drafting
• Flash Drafts
• Self Check: Drafting

Writing Process: Revising

• Outcome: Revising
• Seeking Input from Others
• Responding to Input from Others
• The Art of Re-Seeing
• Higher Order Concerns
• Self Check: Revising

Writing Process: Proofreading

• Outcome: Proofreading
• Lower Order Concerns
• Proofreading Advice
• "Correctness" in Writing
• The Importance of Spelling
• Punctuation Concerns
• Self Check: Proofreading
• Conclusion to Writing Process

Research Process: Finding Sources

• Introduction to Research Process
• Outcome: Finding Sources
• The Research Process
• Finding Sources
• What are Scholarly Articles?
• Finding Scholarly Articles and Using Databases
• Database Searching
• Advanced Search Strategies
• Preliminary Research Strategies
• Reading and Using Scholarly Sources
• Self Check: Finding Sources

Research Process: Source Analysis

• Outcome: Source Analysis
• Evaluating Sources
• CRAAP Analysis
• Evaluating Websites
• Synthesizing Sources
• Self Check: Source Analysis

Research Process: Writing Ethically

• Outcome: Writing Ethically
• Academic Integrity
• Defining Plagiarism
• Avoiding Plagiarism
• Using Sources in Your Writing
• Self Check: Writing Ethically

Research Process: MLA Documentation

• Introduction to MLA Documentation
• Outcome: MLA Documentation
• MLA Document Formatting
• MLA Works Cited
• Creating MLA Citations
• MLA In-Text Citations
• Self Check: MLA Documentation
• Conclusion to Research Process

Grammar: Nouns and Pronouns

• Introduction to Grammar
• Outcome: Nouns and Pronouns
• Pronoun Cases and Types
• Pronoun Antecedents
• Try It: Nouns and Pronouns
• Self Check: Nouns and Pronouns

Grammar: Verbs

• Outcome: Verbs
• Verb Tenses and Agreement
• Non-Finite Verbs
• Complex Verb Tenses
• Try It: Verbs
• Self Check: Verbs

Grammar: Other Parts of Speech

• Outcome: Other Parts of Speech
• Comparing Adjectives and Adverbs
• Adjectives and Adverbs
• Conjunctions
• Prepositions
• Try It: Other Parts of Speech
• Self Check: Other Parts of Speech

Grammar: Punctuation

• Outcome: Punctuation
• End Punctuation
• Hyphens and Dashes
• Apostrophes and Quotation Marks
• Brackets, Parentheses, and Ellipses
• Semicolons and Colons
• Try It: Punctuation
• Self Check: Punctuation

Grammar: Sentence Structure

• Outcome: Sentence Structure
• Parts of a Sentence
• Common Sentence Structures
• Run-on Sentences
• Sentence Fragments
• Parallel Structure
• Try It: Sentence Structure
• Self Check: Sentence Structure

Grammar: Voice

• Outcome: Voice
• Active and Passive Voice
• Using the Passive Voice
• Conclusion to Grammar
• Try It: Voice
• Self Check: Voice

Success Skills

• Introduction to Success Skills
• Habits for Success
• Critical Thinking
• Time Management
• Writing in College
• Computer-Based Writing
• Conclusion to Success Skills

Promoting pre-service teachers’ knowledge integration from multiple text sources across domains with instructional prompts

• Development Article
• Open access
• Published: 10 April 2024

Cite this article

You have full access to this open access article

• Inka Sara Hähnlein   ORCID: orcid.org/0000-0003-2758-634X 1 &
• Pablo Pirnay-Dummer 1  

Multiple document comprehension and knowledge integration across domains are particularly important for pre-service teachers, as integrated professional knowledge forms the basis for teaching expertise and competence. This study examines the effects of instructional prompts and relevance prompts embedded in pre-service teachers’ learning processes on the quality their knowledge integration in multiple document comprehension across domains. 109 pre-service teachers participated in an experimental study. They read four texts on “competencies” from different knowledge domains and wrote a text on a given scenario. Experimental group 1 was aided with instructional and relevance prompts, while experimental group 2 received only relevance prompts. The control group received no prompting. Perceived relevance of knowledge integration was assessed in a pre-post-test. Pre-service teachers’ separative and integrative learning, epistemological beliefs, metacognition, study-specific self-concept, and post-experimental motivation were assessed as control variables. Participants’ texts were analyzed concerning knowledge integration by raters and with computer linguistic measures. A key finding is that combined complex prompting enhances pre-service teachers perceived relevance of knowledge integration. This study found effects of prompting types on the pre-service teachers’ semantic knowledge structures. Implications for transfer are discussed.

Avoid common mistakes on your manuscript.

Introduction

Multiple document comprehension and knowledge integration across domains are especially important for pre-service teachers, as they are necessary for forming integrated professional knowledge. This in turn is the basis for the professional competence of future teachers (Lehmann, 2020a , 2020b ). Since pre-service teachers find it difficult to integrate knowledge, and teacher education is hardly designed to help in the integration of knowledge (Hudson & Zgaga, 2017 ), pre-service teachers are in need of support. Instructional strategies such as cognitive prompts have been found to be effective aids (Lehmann, Pirnay-Dummer, et al., 2019a , 2019b ; Lehmann, Rott, et al., 2019a , 2019b ). However, more empirical research is needed on how to promote cognitive knowledge integration across domains.

The present study aims to promote and facilitate pre-service teachers’ knowledge integration from multiple text sources across domains with two kinds of cognitive instructional strategies. In our experimental study, pre-actional relevance prompts and pre-actional instructional prompts combined with relevance prompts are embedded at multiple time points in pre-service teachers’ self-regulated learning processes. The learning process includes a reading phase and a writing task designed to integrate and transfer what has been read. We hypothesize that the types of prompts will influence the three criteria of pre-service teachers’ text quality, knowledge structure, and perceived relevance of knowledge integration in various ways.

Theoretical background

Knowledge integration across disciplines as a precondition for teaching expertise.

For pre-service teachers, pedagogical knowledge (PK), pedagogical content knowledge (PCK), and content knowledge (CK) form the core of their professional competence as future teachers (Shulman, 1986 ; Voss et al., 2011 ). Technological, organizational and counseling knowledge as well as self-regulation skills, motivational orientations, beliefs, and values are also part of teachers’ professional competence (Model of Professional Competence of Teachers, Baumert & Kunter, 2006 ; Technological Pedagogical Content Knowledge, TPACK, Mishra & Koehler, 2006 ; Koehler & Mishra, 2008 ; Mishra, 2019 ; Krauskopf et al., 2020 ).

These aspects constitute the professional knowledge of teachers in training and are the basis for successfully teaching a specific subject (i.e., analyzing, planning, designing, developing, and evaluating instruction and instructional interactions; Seel et al., 2017 ). Teacher education includes study components from educational sciences, didactics, pedagogy, and the teaching subjects. Text is still by far the main source of information in academia (Pirnay-Dummer, 2020 ). For pre-service teachers, these texts come from different disciplines. Pre-service teachers are thus confronted with texts that take different perspectives on the same topics, and integrating these perspectives is not a simple straightforward task. Texts provide different domain-specific rationales for specific instructional decisions that interact with each other in a complex way (Lehmann, 2020a ; Pirnay-Dummer, 2020 ).

Knowledge integration and multiple-document comprehension

According to Lehmann ( 2020a ), knowledge integration is defined in two ways, as first- and second-order knowledge integration. First-order knowledge integration, as a form of constructive learning, is the active linking, merging, distinguishing, organizing, and structuring of knowledge structures into a coherent model (Lehmann, 2020a , 2020b ; Linn, 2000 ; Schneider, 2012 ). Second-order knowledge integration, as a form of knowledge application, is the simultaneous transfer of knowledge from different domains with the goal of reaching a suitable problem solution (Graichen et al., 2019 ; Janssen & Lazonder, 2016 ; Lehmann, 2020a , 2020b ).

Integrating knowledge from text across disciplines, a competency pre-service teachers are expected to possess is a complex process that relies on single-text comprehension (Construction-Integration Model; Kintsch, 1988 , 1998 ; Trevors et al., 2016 ; van Dijk & Kintsch, 1983 ) but furthermore requires multiple document comprehension (Document Model Framework; Braasch et al., 2012 ; Britt & Rouet, 2012 ; Britt et al., 2018 ; Perfetti et al., 1999 ; Rouet, 2006 ).

The construction of a coherent integrated model of multiple texts requires readers to form an integrated mental model of the content of the texts (integrated mental model), including contradictions and their possible or impossible resolutions, as well as representations of the text sources and how these sources are related to each other as an intertext model (Bråten & Braasch, 2018 ; Britt & Rouet, 2012 ; Perfetti et al., 1999 ; Rouet, 2006 ).

Relevance of knowledge integration for teaching

Empirical evidence shows the importance of knowledge integration for pre-service teachers: The level of knowledge integration influences their degree of expertise and professional competence as teachers later on (Baumert & Kunter, 2006 ; Bromme, 2014 ; Graichen et al., 2019 ; Janssen & Lazonder, 2016 ; König, 2010 ; Lehmann, 2020a ). Teachers’ knowledge integration is also positively related to the learning performances of students (Hill et al., 2005 ).

For pre-service teachers and in-service teachers, successful knowledge integration means being able to apply integrated knowledge to solve complex (ill-defined) problems (Jonassen, 2012 ), such as instructional planning (Brunner et al., 2006 ; Dörner, 1976 ; Lehmann, 2020a ; Norton et al., 2009 ), and to make informed instructional decisions, taking into account multiple perspectives and their complex interaction (Lehmann, 2020a ).

Although successful knowledge integration is particularly important for prospective teachers, teacher education is hardly designed to link subject areas. Subject knowledge, didactics, and educational science are taught separately (Ball, 2000 ; Blömeke, 2009 ; Darling-Hammond, 2006 ; Hudson & Zgaga, 2017 ). As Zeeb et al. ( 2020 , p. 202) point out, this way of knowledge acquisition increases the risk that students will develop structural deficits in their knowledge, in the sense of compartmentalization of knowledge (e.g., Whitehead, 1929 ), which in turn explains the fragmented, inert knowledge (Renkl et al., 1996 ). Accordingly, the professional knowledge present in pre-service teachers and in-service teachers tends to be fragmented and poorly integrated. There is hardly any systematic linking across subject areas and faculty boundaries. The principle of separating subject knowledge, subject didactics, educational science, and pedagogy prevails (Ball, 2000 ; Darling-Hammond, 2006 ; Graichen et al., 2019 ; Harr et al., 2014 ; Janssen & Lazonder, 2016 ).

Since integrating knowledge is an important but complex task, and teacher education is hardly designed to initiate and train integration of knowledge (Hudson & Zgaga, 2017 ), pre-service teachers are in need of support.

Supporting pre-service teachers’ knowledge integration and multiple document comprehension

There are different approaches to the promotion of knowledge integration:

The MD-TRACE model (“Multiple-Document Task-Based Relevance Assessment and Content Extraction”) describes how readers represent their goals by forming a task model in multiple-document comprehension (Rouet, 2006 ; Rouet & Britt, 2011 ). The reading context is vital for multiple-document processing. According to the RESOLV theory, readers initially construct a model of the reading context that influences the interpretation of the task at hand, for instance the interpretation of writing tasks (Rouet & Britt, 2011 ; Rouet et al., 2017 ). There is ample evidence to suggest that specific writing tasks can elicit changes in learning activities (Wiley & Voss, 1996 , 1999 ; Bråten & Strømsø 2009 , 2012 ; Lehmann, Rott, et al., 2019a , 2019b ). Specific writing tasks can promote integrated understanding by stimulating elaborative processes of knowledge-transforming rather than simple knowledge-telling (Scardamalia & Bereiter, 1987 , 1991 ). Writing a summary of multiple documents or answering overarching questions may also promote knowledge integration (Britt & Sommer, 2004 ; Wiley & Voss, 1999 ). Tasks asking pre-service teachers to combine information from multiple sources can foster the integrated application of knowledge (Graichen, et al., 2019 ; Harr et al., 2015 ; Lehmann, Rott, et al., 2019a , 2019b ; Wäschle et al., 2015 ).

A growing body of empirical evidence shows that prompts are effective instructional strategies for supporting knowledge integration across domains (Lehmann, Pirnay-Dummer, et al., 2019a , 2019b ; Lehmann, Rott, et al., 2019a , 2019b ). Implemented as statements, focus questions, incomplete sentences, or relevance instructions, among other formats, they promote the cyclical process of self-regulated learning (SRL) (Schiefele & Pekrun, 1996 ; Zimmerman, 2002 ). Prompts can be embedded in the pre-actional, actional, or post-actional phase of SRL. They elicit the use of cognitive, metacognitive, and motivational learning strategies that promote learning at the respective level (Bannert, 2009 ; Ifenthaler, 2012 ; Lehmann et al., 2014 ; Lehmann, Rott, et al., 2019a , 2019b ; Reigeluth & Stein, 1983 ; Zimmerman, 2002 ).

Implemented pre-actionally, that is, provided prior to learning, prompts can assist learners in constructing an appropriate task model (Ifenthaler & Lehmann, 2012 ; Lehmann et al., 2014 ). Empirical evidence demonstrates that pre-actional cognitive prompts promote an integrated deep understanding across core domains of professional knowledge of pre-service teachers (Lehmann, Rott, et al., 2019a , 2019b ; Wäschle et al., 2015 ).

Relevance instruction enhances integrated knowledge in pre-service teachers by encouraging the use of integrative strategies (e.g. Zeeb et al., 2020 ). Relevance prompts can be implemented either specifically or independently. They can emphasize the relevance of the specific learning content (Cerdán & Vidal-Abarca, 2008 ) or learning task (Gil et al., 2010 ), or they can be implemented independently and refer to the relevance of knowledge integration in general (Zeeb, Biwer, et al., 2019 ; Zeeb et al., 2020 ). Zeeb et al. ( 2020 ) argue in favor of the independent implementation, since knowledge integration is important for teacher education across domains and not just within specific subjects or topics. Repeated relevance instructions have been shown to be superior to one-time instructions (Zeeb et al., 2020 ).

The above described theoretical foundation shows the importance of multiple document comprehension and knowledge integration across domains for pre-service teachers and that prompts might be used to help them.

In our experiment, we investigated how to promote and facilitate pre-service teachers’ knowledge integration from multiple text sources across domains. The design examined the effects of two different kinds of prompts embedded in the students’ learning processes at three different points in an experimental design.

Research question

Do instructional and relevance prompts embedded in the learning process promote pre-service teachers’ knowledge integration from multiple texts across domains?

The evidence described above indicates that providing students with pre-actional instructional prompts should lead to an increased quality of knowledge integration. The same is to be expected for repeated pre-actional relevance prompts . Our experiment combined the two types of prompts and embedded them at different stages in the learning process, in experimental contrast to providing relevance prompts only. A control group was provided with no support, as this is still often the standard in pre-service teacher training.

We hypothesized that the combined prompts would lead to higher rated text quality (dependent variable 1, DV1) than relevance prompts alone. We also hypothesized that support from relevance prompts alone would be better than no support at all.

Moreover, we hypothesized that there would be systematic differences between the three groups both in terms of knowledge structure and knowledge semantics (dependent variable 2, DV2) when compared with the knowledge model of the source material.

In addition, we hypothesized that repeated pre-actional relevance prompts would lead to an increase in perceived relevance (dependent variable 3, DV3).

We assessed pre-service teacher students’ perceived integration and separation learning in teacher education, epistemological beliefs, metacognition, study-specific self-efficacy, and post-experimental motivation as control variables, because of their relevance for self-regulated knowledge integration (Barzilai & Strømsø, 2018 ; Lehmann, 2022 ).

Participants

The experiment was conducted with N = 109 of 119 pre-service teacher students (see Analysis and Results for explanation of dropout) who attended one of four courses on research methods and statistics in the 2021/22 fall term at a German university (81 females, 27 males, 1 n/a; age: M = 22.46, SD = 3.23). Of the 108 student teachers who specified their school type, most are studying elementary school teaching (45), while 36 are studying high school teaching, 17 secondary school teaching, 8 special education, and only 1 elementary and special education. The most common major subject is German (43), followed my mathematics (23), history (9), biology (6), English (5), sports (5) and others. The most chosen second subject are mathematics (29), German (24), biology (8), English (5). The participants have been studying their major subject for an average of about six semesters ( M = 5.9, SD = 2.3).

Procedure and design

This experimental intervention study has a cross-sectional control group design. Figure 1 shows the experimental procedure and design. First, the pre-service teacher students were informed about the study at the beginning of the term (calendar week 40) in the introductory sessions of the courses on research methods. Participation was voluntary with no consequences for not participating. All students enrolled in the course chose to participate in this study giving their informed consent. The participants were anonymized by means of randomly assigned codes known only to themselves. Then, the participants were randomly assigned to three experimental conditions: experimental group 1 (EG1), experimental group 2 (EG2), and control group (CG). All assessments and the intervention took place at course time in the course room to make it as easy as possible for the students to participate. Students who dropped out of the course automatically terminated their participation in this study.

Experimental procedure with three experimental conditions in an experimental intervention study with cross-sectional design

In weeks two and three of the term (calender week 41/42), we collected the students’ demographics, and as control variables assessed their epistemological beliefs (Students’ Epistemological Beliefs; StEB; Hähnlein, 2018 ), metacognition in the learning process (MILP; Hähnlein & Pirnay-Dummer, 2019 ), study-specific self-efficacy (WIRKSTUD; Schwarzer & Jerusalem, 2003 ), as well as their self-reported separative and integrative learning in teacher education (SILTE; Lehmann et al.; 2020 ). All data collection was implemented online on Limesurvey.

To avoid test fatigue in the participants, the actual intervention was not conducted until several weeks after the control variables were collected. For all participants, the time interval between pre-survey and experiment was the same. The experiment was conducted six to seven weeks (calender week 46/47) into the term and took approximately 90 to 105 minutes for CG and EG2, but about 120 minutes for EG1 to account for the prolonged instruction.

The experiment started with a 3-minute pre-test regarding students’ perceived relevance of knowledge integration (Figure 1 ). After that, to initiate a learning process, all participants received a learning task with an instructional part, a reading and a writing task (Figure 1 ). Participants were instructed to work through the learning process and remaining test procedure on their own time and were allowed to leave the experiment when they were finished, but not before 90 minutes had passed (normal course duration).

For all participants, the reading task consisted of four texts about “the concept of competence in teaching” from different subject domains of teacher education (8 pages in total. We recorded the time participants spent studying the text material including reading time (M = 38, SD = 17, Min = 14, Max = 78). Students’ perceived handling of the text material was evaluated right after the reading phase (Text Material Questionnaire I; Deci & Ryan, n.d.; German adaption). This took just 2 minutes. The writing task was on a fictional scenario that required integrating the knowledge from all four source texts to derive implications for the application of the knowledge.

The participants of the control group (CG: no prompts) received just organizational information in the instructional phase of the learning process and no aid regarding knowledge integration for the reading and writing task (Figure 1 ).

The participants of experimental group 2 (EG2: relevance prompts) received relevance prompts in verbal and written form embedded in their learning processes in the instructional phase as well as the reading and writing phase. Experimental group 1 (EG1: relevance prompts and instructional prompts) received both relevance prompts and instructional prompts (Figure 1 ). Both types of prompts were embedded in the students’ learning process at three time points: the instructional phase, the reading phase, and the writing phase.

Following the learning process, we again assessed students’ perceived relevance of knowledge integration (3 minutes) as well as their post-experimental intrinsic motivation (Intrinsic Motivation Inventory, IMI; Deci & Ryan, n.d.; German version, 12 minutes). All survey instruments used are introduced below in the section Survey Instruments .

Instructional prompts in experimental group 1

In the instructional phase, the participants of experimental group 1 received a 10-minute PowerPoint-based introduction to knowledge integration to stimulate their pre-flexion prior to reading. The introduction used the example of lesson planning to outline how knowledge integration works, how it works, and why it is important for future teachers. This served as a pre-actional cognitive prompt.

To support the reading phase, focus questions were developed in our research team specifically to the text material at hand. The participants received the following focus questions related to knowledge integration to apply to the texts to be read (translated from German):

What are similarities and differences in the understanding of competence between the texts?

What level does the knowledge from the different texts refer to?

Does the knowledge refer to what competencies are?

Does the knowledge relate to abstract or specific objectives? (What is to be achieved?)

Does the knowledge relate to application? Does it relate to the process of how to accomplish something?

Does the knowledge relate to why? (Why does something work this way and not another way?)

How do the competency perspectives interact with each other? Do the knowledge contents and their levels complement each other or do contradictions arise? (Bridges between texts?)

Can different things be derived from the different perspectives for application?

What can be derived for the application from the integrated impression of all 4 texts?

Furthermore, the participants were asked to model the interrelation between the texts, for instance by drawing a mind map. The focus questions served as pre-actional cognitive prompts for the reading task, while the modeling served as an actional cognitive prompt for the reading phase and a pre-actional cognitive prompt for the writing phase.

Before writing another pre-actional prompt was given: The participants in this group were explicitly asked to use their elaborations from the reading phase while writing and to explicitly connect the knowledge instead of just summarizing it.

Relevance prompts in experimental group 1 and 2

During instruction, the relevance prompt was provided to the participants via an oral explanation of the importance of knowledge integration for their future teaching proficiency and the reasons for it, supported by an anthology on knowledge integration research held up during the presentation of a PowerPoint slide. This served as pre-actional prompt.

In both the reading and the writing phase, the participants were again reminded in writing of the relevance of knowledge integration.

For both experimental groups, the task sheet remained with the participants so that they could access the prompts even while performing the task.

The control group received neither relevance prompts nor instructional prompts but just the reading and writing task.

Reading material

The reading material was four selected texts on the concept of competence in teaching. The texts come from different disciplines and are all academic in source and nature (educational science/humanities, educational psychology, didactics, and policy-making). The texts were selected and discussed beforehand by an interdisciplinary team both for their relevance within each field and for their potential for not being too easy to integrate. However, since the four texts are from different disciplines, each of them takes a different professional perspective on the topic.

Text 1 takes the perspective of educational science or pedagogy. It is taken from a textbook on the introduction to educational science and two pages long (Textbook: Thompson, 2020 , pp. 131–133). This text is about how the concept of competence is defined from the perspective of competence research.

Text 2 takes the point of view of educational psychology. It is two and a half pages long and about what distinguishes the concept of competence from established categories such as ability, skill, or intelligence. (Article: Wilhelm & Nicolaus, 2013 , pp. 23–26)

Using the example of learning to read, text 3 deals with the distinction between different levels of competence (Textbook: Philipp, 2012, pp. 11–15). It is a text from the didactics and one and a half pages long.

Text 4 is a curricular description from the Standing Conference of the Ministers of Education and Cultural Affairs (Kultusministerkonferenz, 2009 , pp. 1–5). It takes an educational policy perspective on the subject matter and is two and a half pages long. This text is about the competency level model for the educational standards in the competency area speaking and listening for secondary school.

Writing task

The writing task was part of a fictional scenario requiring participants to integrate knowledge from the four source texts and to draw integrated conclusions for its application to teaching in order to help a friend in need. The scenario with task (translated from German) read as follows:

During your school internship, a future colleague has guided you through many a challenging situation thanks to her professional experience and appreciative nature. This teacher, Monique Gerber, recently turned to you and somewhat bashfully told you that she herself is currently facing a rather challenging situation. She has a school evaluation coming up next week. In itself, this is not a problem for Monique. However, she has learned in advance that dealing with competence and its scientific foundation in teaching is a central theme of the evaluation. She says that the academic discussion has been going on for far too long and that she would like you to give her an informative summary of the topic of competence: What should she look for when teaching? How should she justify things? How does she relate what she does well in class to existing scientific knowledge? A little flattered, and knowing of Monique’s distress, you set out to help her. Task: Write a text yourself on the basis of the four short texts on the topic of competence. Your text for Monique should explain step by step the current scientific understanding of competence and show her how it can be used to help her plan and design lessons. (The text should be written in complete sentences. It must be at least 400 words.)

This complex task requires both knowledge-telling and knowledge-transforming (Scardamalia & Bereiter, 1987 , 1991 ). An integrated mental model of the text sources (Bråten & Strømsø, 2009 ; Wiley & Voss, 1999 ) is needed to derive implications for lesson planning.

Survey instruments

The StEB Inventory (Hähnlein, 2018 ) is designed to assess pre-service teachers’ epistemological beliefs. The instrument development is based on the theoretical conceptualization of the epistemological belief system by Schommer ( 1994 ), the core dimensions by Hofer and Pintrich ( 1997 ) and Conley et al. ( 2004 ), as well as the Integrative Model for Personal Epistemology ( IM, Bendixen & Rule, 2004 ; Rule & Bendixen, 2010 ) to explain the mechanism of change, and the Theory of Integrated Domains in Epistemology (TIDE, Muis et al., 2006 ) to explain the context dependency and discipline specificity of epistemological beliefs. The StEB questionnaire consists of four subscales: beliefs about the simplicity of knowledge , the absoluteness of knowledge , the multimodality of knowledge , and the development of knowledge . The questionnaire consists of 26 items. Agreement with the statements is indicated on a 5-point Likert scale (from does not apply at all to completely applies ).

The MiLP Inventory (Hähnlein & Pirnay-Dummer, 2019 ) assesses students’ metacognitive activities in the form of learning judgments. The instrument development is based on the theoretical model of Nelson and Narens ( 1990 , 1994 ). It distinguishes between metacognitive monitoring and control as well as the three phases of learning: knowledge acquisition, retention, and retrieval. The questionnaire consists of 33 items and six subscales. Four subscales concern the metacognitive activities in the knowledge acquisition phase. Two each concern metacognitive monitoring and metacognitive regulation. One subscale concerns metacognitive observation in the retention phase and one that of knowledge retrieval. The response scale has a 5-level Likert format (from does not apply to does apply ). The six subscales are as follows:

Anco: Assesses a learners’ ability to regulate his/her learning in the phase of knowledge acquisition by means of adequate learning strategies. (10 items)

Abmo: Assesses a learners’ ability to monitor his/her retrieval of knowledge in a way that her/she is able to successfully remember the learning content. (8 items)

Anmo: Assesses a learners’ ability to monitor his/her knowledge acquisition by means of assessing the difficulty of the learning content. (5 items)

Akco: Assesses a learners’ ability to regulate his/her knowledge acquisition in a way that he/she is able to successfully differentiate between important and unimportant learning content. (3 items)

Bemo: Assesses a learners’ ability to monitor his/her retention of knowledge in a way that he/she is able to remember the learning content. (4 items)

Akmo: Assesses a learners’ ability to monitor his/her knowledge acquisition in a way that he/she is able to figure out if the knowledge acquisition was successful. (3 items)

The SILTE Short Scales (Lehmann et al., 2020 ) are used to measure the self-reported knowledge integration of pre-service teacher students in teacher education across domains. With its two dimensions, it measures integrative learning with 7 items and separative learning with 5 items. The two scales have a five-point response format (ranging from does not apply at all to fully applies ). According to Lehmann et al. ( 2020 , p. 156), the theoretical foundation of the SILTE questionnaire is the model of knowledge building (e.g. Chan et al., 1997 ; Scardamalia & Bereiter, 1994 , 1999 ), which can be assigned to the constructivist approaches to strategic learning. In addition, the questionnaire is based on the concepts of cognitive fragmentation and knowledge integration in teacher education and learning to teach (e.g. Ball, 2000 ; Darling-Hammond, 2006 ; Lehmann, 2020b ).

Study-specific self-efficacy is assessed using the WIRKSTUD scale (Schwarzer & Jerusalem, 2003 ). It is one-dimensional and has 7 items with a four-point rating scale ( does not apply at all , hardly applies , applies , applies completely ). The conception of the scale is based on Bandura’s ( 1978 ) social-cognitive learning theory and the concept of positive situation-action expectations contained therein.

The Intrinsic Motivation Inventory (IMI) is a multidimensional measurement that comes in different versions. It is intended to assess “participants’ subjective experience related to a target activity in laboratory experiments” (Deci & Ryan, n.d., p. 1). The Post-Experimental Intrinsic Motivation Inventory (Deci & Ryan, n.d.) originally consists of 45 items and seven scales that can be selected according to the requirements of the experimental setting. In our study, the following six scales were used: Interest/enjoyment (7 items), perceived competence (6 items), effort/importance (5 items), pressure/tension (5 items), perceived choice (7 items), and value/usefulness (7 items). The scale relatedness (8 items) was not used in this study. A five-point response format (ranging from does not apply at all to fully applies ) were used for all IMI measures.

The Text Material Questionnaire consists of three of the subscales of the IMI questionnaire (Deci & Ryan, n.d.) adapted to text material. It assesses students’ interest and pleasure (5 items), felt pressure (2 items), and perceived competence (2 items) in dealing with the text.

The value/usefulness (7 items) subscale of the IMI (Deci & Ryan, n.d.) which is adaptable to different content, was used to measure pre-service teachers’ perceived relevance of knowledge integration in the pretest and posttest.

Table 1 shows the internal consistencies of the survey instruments used. Reliabilities are reported for the current study as well as for the previous development and validation studies. All in all, the reliabilities can be considered acceptable. For individual scales, however, there are very low internal consistencies with values below α = .70.

Text rating measure

To score the quality of the participants’ texts, we used a rating scheme with three criteria: degree of transfer, validity of conclusions, and degree of integration. Each criterion was rated on a scale of 0 to 3 points. The criteria were initially developed by an expert group of five persons regarding content validity. For this study, a two-person group re-evaluated the criteria, but only minor changes were made and only with respect to the specific content. The text criteria were not revealed to the participants. The rating criteria (translated from German) were as follows:

Degree of transfer (transfers are only given if they can also be derived from the texts read).

0: There is no transfer to the action level.

1: A transfer to the action level is only made by naming goals. For this purpose, less specified should statements are used (for example, as in “the lessons should be designed in a friendly way” or “the lessons should have a good relationship level”).

2: Ideas are formulated sporadically (or in an unconnected list form) on how aspects from the transfer can be implemented.

3: There are concrete, interrelated derivations from the texts with regard to a realistic lesson design.

Validity of the conclusions

0: The conclusions cannot be derived with certainty from the sources (e.g., purely intuitive assumptions).

1: Unconnected (e.g., purely abductive) assumptions are present, but at most as a list-like series of unrelated individual statements.

2: The conclusions are largely clear from the sources.

3: The conclusions emerge unambiguously and deductively from the sources and are logically related to application.

Degree of integration

0: Assumptions are treated separately per text.

1: The assumptions are treated separately, but, e.g., any contradictions and compatibilities discovered are contrasted, mentioned, and/or discussed.

2: The different models are treated together, with reference to each other. They are not worked through sequentially.

3: The different models in the texts are processed in an integrated and coherent way, explicitly integrating the areas of knowledge into each other.

Computer linguistic methods

Computer linguistic methods were used in this research project for computational modeling of the semantic knowledge structures contained in both source texts and student texts (Pirnay-Dummer, 2006 , 2010 , 2014 , 2015a , 2015b ; Pirnay-Dummer et al., 2010 ).

Mental model-based (Seel, 1991 , 2003 ) knowledge elicitation techniques have relied on recreating propositional networks from human knowledge (Ifenthaler, 2010 ; Jonassen, 2000 , 2006 ; Jonassen & Cho, 2008 ; Pirnay-Dummer, 2015a , 2015b ).

The computational linguistic heuristic technology T-MITOCAR (Text-Model Inspection Trace of Concepts and Relations; Pirnay-Dummer, 2006 , 2007 ; Pirnay-Dummer et al., 2010 ) was developed as a means of automatically analyzing, modeling, visualizing, and comparing the semantic knowledge structure of texts. The approach behind T-MITOCAR is closely based on the psychology of knowledge, knowing, and epistemology (Spector, 2010; Strasser, 2010). Its associative core functions are founded strictly on mental model theory (Gentner & Stevens, 1983 ; Johnson-Laird, 1983 ; Pirnay-Dummer et al., 2012 ; Pirnay-Dummer & Seel, 2018 ; Seel, 2012 ; Seel et al., 2013 ) and on how, when, and why parts of knowledge are reproduced in the semantics of natural language (Evans & Green, 2006 ; Helbig, 2006 ; Partee, 2004 ; Taylor, 2007 ). The algorithms work through the propositional relations of a text to identify central relations between concepts and build a network (a graph), while always heuristically reconstructing the parts as closely as possible to how human knowledge is constructed, conveyed, and reconstructed through text.

T-MITOCAR can also automatically compare different knowledge structures from text quantitatively using measures based on graph theory (Tittmann, 2010 ), four structural measures (Table 2 ), and three semantic measures (Table 3 ). The measures lead to similarity measures s between zero and one. Tables 2 and 3 only provide an overview for their interpretations, which is necessary to understand them as criteria within this study.

In this research project, we used T-MITOCAR technology to compare the student texts with a reference model of the source texts on the basis of the seven similarity measures. Although we classified the measures into structural and semantic indices, as Tables 2 and 3 show, the similarity indices measure different features and can therefore not be treated like a subsuming scale (e.g., on a test): The measures are not items for the same but for different properties of knowledge graphs. The structural measures indicate different properties of structure, whereas structure itself is not a property. The same holds true for the semantic measures.

Analysis and results

The statistical software R (R-Core-Team, 2022 ) was used for analysis. Of the 119 students enrolled in the four courses at the beginning, five dropped out during term and terminated their participation in this study. Two people were absent on the day of the experiment for health reasons. Three other participants participated in the experiment but did not submit their self-written texts for unknown reasons. Since this was interpreted as a withdrawal of consent, these three datasets were not evaluated. The data of 109 participants were available for further analysis.

First, we checked for differences in the control variables between the experimental conditions using MANOVA and ANOVA. Results from MANOVA showed no significant differences in students’ separative and integrative learning in teacher education between the groups (Wilks’ λ = 0.97, F [4,220] = 0.93, p = .45). Also, no significant differences between the three groups were found in the students’ metacognitive abilities (MANOVA, Wilks’ λ = 0.88, F [12,222] = 1.27, p = .24), epistemological beliefs (MANOVA, Wilks’ λ = 0.95, F [8,226] = 0.81, p = .60), and study-specific self-efficacy (ANOVA, F [2,110] = 2.29, p = .11). The participants of the three groups did not perceive the reading material differently (MANOVA, Wilks’ λ = 0.95, F [6,212] = 0.99, p = .43). For post-experimental intrinsic motivation, no significant differences between the three groups were found (MANOVA, Wilks’ λ = 0.79, F [12,176] = 1.79, p = .053).

Using ANOVA, we analyzed the group differences in the time (minutes) participants spent studying the text material, to check if the instructional prompts (focus questions) were used. Time spent studying the text material differed significantly between the three groups (ANOVA, F [2,89] = 238.2, p < .001; , η p 2 =.84). Participants with the combined prompts spent an average of 63 minutes ( SD = 7.40) studying the text material, while participants with only relevance prompts (EG2: M = 29, SD = 7.25, p < .001) or no prompts (CG: M = 27, SD = 5.67, p < .001) spent significantly less time.

As a further check on prompt usage, participants of EG1 were asked to hand in their models showing the interrelations between the texts. Three subjects who had not created a model, i.e. had not completed this learning phase, were excluded from further analysis.

Quality of knowledge integration (DV1)

We analyzed the quality of the pre-service teachers’ texts (DV1). Two trained raters assessed the student texts ( N = 109, EG1: n1 = 37, EG2: n2 = 37, CG: n3 = 35) for quality on a scale of 0 to 3 points on the basis of the following criteria: degree of transfer, validity of conclusions, and degree of integration.

Table 4 shows the mean ratings for the criteria per group for both raters. The average points achieved for integration, transfer and conclusions are lower for rater 1. Reviewer one seems to evaluate the student texts more strictly than reviewer two.

Interrater reliability (Kendalls-τ, Type b) was high for transfer ( r τ = .52, z = 6.08, p < .001) but low for conclusion ( r τ = .21, z = 2.23, p = .02) and integration ( r τ = .13, z = 1.60, p = .11), as well as for the overall rating ( r τ = .24, z = 3.30, p < .001). The ratings were treated separately in the following analysis due to low interrater reliability. All raters used the whole range of the criteria (0–3) for each item. MANOVA with three rating scores per rater as dependent measures and the experimental conditions as independent measures indicated no significant differences between type of prompting (Wilks’ λ = 0.85, F [12,202] = 1.43, p = .16).

Computer linguistic analyses (DV2)

Comparison of participants’ texts with the reference model of source material.

Using T-MITOCAR technology, we combined the four source texts to generate a reference model of their semantic knowledge structures across the respective domains (see Figure 3 ). Figure 2 shows only a section of the resulting reference model for illustrative purposes because the entire model would be too large.

Section of the reference model of the source material

The source reference model consists of concepts that are bound by links. The links are associations strengths as determined by T-MITOCAR. At the links (Figure 2 ) are measures of association as weights between 0 and 1. One stands for the strongest association within the text and zero would stand for no association. Only the strongest ones are included in such a graph. Within the parenthesis is a linear transformation of the same value, so that the weakest links that still made it to the graph show a zero and the strongest show a one (Pirnay-Dummer, 2015b ). The meaning of a concept is constituted by the context structure in which it is located in the network. The meaning of such a T-MITOCAR generated semantic knowledge structures (in this case, the reference model of the source texts) lies the way concepts are linked with each other (e.g. cyclic, hierarchical, sequential) and the connections they make to specific concepts but not others.

To explore the effect of prompting on the pre-service teachers’ knowledge structures, we compared the student texts ( N = 109, EG1: n 1 = 37, EG2: n 2 = 37, CG: n 3 = 35) with the reference model of the source texts using T-MITOCAR similarity measures. The participants in CG and EG2 tended to write longer texts than those in EG1. However, differences in the average word count between the groups were not significant (ANOVA, F [2,106] = 1.67, p = .19; M EG1 = 407.11, SD EG1 = 122.59; M EG2 = 429.73, SD EG2 = 86.21 ; M CG = 448.8, SD CG = 74.56; F (2,106)= 1.75).

The logic of the computer-linguistic analysis in this study is shown in Figure 3 .

Computer-linguistic analysis of the student texts

Means and standard deviations for the computer-linguistic comparison measures per group between participants’ texts and reference text model are shown in Table 5 .

MANOVA (Type III), with the four structural (SUR, GRA, STRU, GAMMA) and three semantic similarity scores (CONC, PROP, BSM) as dependent measures and the experimental conditions as independent measures, indicated significant differences between type of prompting (Wilks’ λ = 0.79, F [14,200] = 1.83, p = .04).

We conducted follow-up univariate ANOVA (Type III) for each of the dependent measures. The results indicated significant differences between the experimental conditions for structural matching (STRU: F [2,106] = 7.44, p < .001, η p 2 = .12; Figure 3 ), propositional matching (PROP: F [2,106] = 4.78, p = .01, η p 2 = .08; Figure 4 ), and balanced semantic matching (BSM: F [2,106] = 3.49, p = .03, η p 2 = .06). The differences in surface (SUR: F [2,106] = 1.69, p = .19), graphical (GRA: F [2,106] = 2.10, p = .13), concept matching (CONC: F [2,106] = 1.80, p = .17) and gamma were not significant (GAMMA: F [2,106] = 1.28, p = .28).

Propositional similarity between participants’ texts and reference source model per group

Tukey HSD post-hoc for structural matching revealed that the control group (STRU: M = 0.43, SD = 0.10) achieved significantly higher similarities with the reference model than experimental group 1 (STRU: M = 0.31, SD = 0.15, p < .001) and experimental group 2 (STRU: M = 0.34, SD = 0.14, p = .02; see Figure 4 ). The control group (PROP: M = 0.08, SD = 0.03) achieved significantly higher propositional similarity to the reference model than experimental group 1 (PROP: M = 0.06, SD = 0.03, p = .009; see Figure 5 ). The contrasts for balanced semantic matching were not significant (BSM: p > .05).

Structural similarity between participants’ texts and reference source model per group

Perceived relevance (DV3)

We used mixed ANOVA (Type III, with Greenhouse-Geisser correction for violation of sphericity) to analyze the differences in the students’ perceived relevance of knowledge integration over time (R-Package: afex; Function: aov_4). There were no significant main effects of the within-subject factor (time, F [1,105] = 0.77, p = .38) or the between-subject factor (group, F [2,105] = 0.20, p = .82) on the differences between the groups over time. However, as Figure 6 illustrates, a significant interaction effect of time (within) and group (between) on perceived relevance was found ( F [2,105] = 5.68, p =.005, η 2 G = .027).

Differences in perceived relevance of knowledge integration between groups over time

Tukey-HSD-adjusted mixed post-hoc analysis revealed a significant ( p = .02) increase in perceived relevance of knowledge integration for only EG1, the group receiving combined prompts ( M Diff = − 0.31, 95% − CI[− .58, − .033]).

The main objective of this study was to examine the effects of instructional prompts and relevance prompts embedded in pre-service teachers’ learning processes on the quality of their knowledge integration in multiple document comprehension across domains.

We found that students receiving no prompts (CG) achieved a closer structural match to the overall reference model than the students aided by prompts (EG1, EG2). Also, their propositions were more similar to the reference model than those of the students with combined prompts (EG1). When interpreting the results, it is important to separate the knowledge level from the integration level of a text. The way the reference model was created to which the student texts were compared explains why higher similarity does not indicate higher knowledge integration, but rather more knowledge-telling (Scardamalia & Bereiter, 1987 , 1991 ). The reference model was created by combining all four source texts into one document. From this, the overall model of the textual knowledge structure was then created using T-MITOCAR. Thus, the resulting model of the source texts is not an integrated knowledge model but rather a combined one. Students who have solved the writing task by preparing a short summary for each of the reference texts in turn achieved a higher structural and propositional similarity than students who actually made the effort to produce an integrated text of their own from the four reference texts. Thus, our results provide for structural and propositional matching provide a substantial empirical indication that students who do not receive prompts about knowledge integration are more prone to knowledge-telling than students who receive prompts.

However, no conclusions can be drawn from this as to whether students supported by the prompts actually integrated their knowledge better. We found no direct evidence that pre-service teachers were supported in their knowledge integration by pre-actional cognitive prompts in the form of task-supplemental focus questions in combination with repeated content-independent relevance instruction. This is contrary to previous studies who succeeded in prompting knowledge integration (e.g. Lehmann, Rott et al., 2019a , 2019b ; Zeeb, Biwer, et al., 2019 ; Zeeb et al., 2020 ).

Overall, very few of the pre-service teachers in our study succeeded in writing integrated texts that contained transfer. This again is in line with previous empirical findings that student teachers struggle with knowledge integration (Graichen et al., 2019 ; Harr et al., 2014 ; Harr et al., 2015 ; Janssen & Lazonder, 2016 ). Pre-service teachers have little or no formal experience in knowledge integration, because their study domains are taught separately (Ball, 2000 ; Darling-Hammond, 2006 ; Hudson & Zgaga, 2017 ). The complex task we used in our experiment required students not only to integrate knowledge from four texts across domains but also to draw integrated conclusions for transfer. Low integration despite of repeated, combined prompting can be interpreted as an indicator that transfer-oriented knowledge integration should not be treated as a function of multi-document integration, even when it is as easy to control as it is within studies for knowledge integration. Instead of a text-inherent process of a task, which is limited to a particular domain and particularly trained expectations towards kinds of integration, transfer extending to the true expected task (what students believe to be their knowledge application later in “real life”) seems to modify the kind of integration as well as its outcome. Students in this study seem to be inspired to actually leave their academic knowledge and rely more on their word knowledge. When the students of this study are induced by the task and its way of introducing integration and thus transfer to leave the specific domain, they no longer perceive the same specific content to be as accessible or relevant. This is only just post-hoc at this point and not at all finite evidence, but it poses a recognizable danger to academic transfer and may even help to explain a lack of dissemination. In future analyses, we will also try to map commonplace knowledge to the solutions—this could be difficult to do, however, because we would first need to carefully create a comparable knowledge base as an additional outside criterion.

Low integration and transfer might suggest a floor effect that may have made it additionally difficult for raters, as there was little transfer and integrated knowledge to be found. This suggests that knowledge integration training and practice in the use of knowledge integration support is needed in handling complex tasks especially when the integration is directed at academic transfer.

We did not find significant group differences for text quality (transfer, conclusion, integration criteria). However, the low interrater reliability is a limitation of this study. It is suggested that new raters to be trained in more detail to obtain more reliable assessments of the texts. At the same time, we found group differences for time participants spent reading and working with the given text material. This shows that the focus questions given as instructional prompts to EG1 were indeed used and stimulated a significantly longer engagement with the texts compared to the relevance prompts alone or no prompts.

Contradicting our third hypothesis (see Objectives) and previous findings (Zeeb, Biwer, et al., 2019 ; Zeeb et al., 2020 ), relevance prompts alone were not enough to enhance the relevance perception, even though they were repeated and emphasized knowledge integration independently from source material content or the task (DV3). Only in combination with the complex instructional prompts did the perceived relevance increase over time. However, long-term effects were not part of this study, which limits the validity of this point.

The results of this study are surprising in some important ways: Knowledge integration seems to be even more complex than already known, particularly when it uses interdisciplinary domains aimed at transfer. Everyday knowledge may get in the way of academically sound transfer in a much deeper sense than previously assumed. This should be considered as a prerequisite of transfer. Just leaving it to the practical imagination clearly does not suffice. Scholars and practitioners alike need to know about this gap before they can effectively train for integrated transfer.

Data availability

The data that support the findings of this study are not openly available but are available from the corresponding author upon request.

Ball, D. L. (2000). Bridging practices: Intertwining content and pedagogy in teaching and learning to teach. Journal of Teacher Education, 51 (3), 241–247. https://doi.org/10.1177/0022487100051003013

Article   Google Scholar  

Bandura, A. (1978). Self-efficacy: Toward a unifying theory of behavioral change. Advances in Behaviour Research and Therapy, 1 (4), 139–161. https://doi.org/10.1016/0146-6402(78)90002-4

Bannert, M. (2009). Promoting self-regulated learning through prompts. Zeitschrift für Pädagogische Psychologie, 23 (2), 139–145. https://doi.org/10.1024/1010-0652.23.2.139

Barzilai, S., & Strømsø, H. I. (2018). Individual differences in multiple document comprehension. In J. L. G. Braasch, I. Bråten, & M. T. McCrudden (Eds.), Handbook of multiple source use (pp. 99–115). Routledge. https://doi.org/10.4324/9781315627496-6

Chapter   Google Scholar  

Baumert, J., & Kunter, M. (2006). Stichwort: Professionelle Kompetenz von Lehrkräften. Zeitschrift für Erziehungswissenschaft, 9 (4), 469–520. https://doi.org/10.1007/s11618-006-0165-2

Bendixen, L. D., & Rule, D. C. (2004). An integrative approach to personal epistemology: A guiding model. Educational Psychologist, 39 (1), 69–80. https://doi.org/10.1207/s15326985ep3901_7

Blömeke, S. (2009). Lehrerausbildung. In S. Blömeke, T. Bohl, L. Haag, G. Lang-Wojtasik, & W. Sacher (Eds.), Handbuch schule. Theorie—organisation—entwicklung (pp. 483–490). Klinkhardt.

Braasch, J. L. G., Rouet, J.-F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40 , 450–465. https://doi.org/10.3758/s13421-011-0160-6

Bråten, I., & Braasch, J. L. G. (2018). The role of conflict in multiple source use. In J. L. G. Braasch, I. Bråten, & M. T. McCrudden (Eds.), Handbook of multiple source use (pp. 184–201). Routledge. https://doi.org/10.4324/9781315627496-11

Bråten, I., & Strømsø, H. I. (2009). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes, 47 (1), 1–31. https://doi.org/10.1080/01638530902959646

Bråten, I., & Strømsø, H. I. (2012). Knowledge acquisition: Constructing meaning from multiple information sources. In N. M. Seel (Ed.), Encyclopedia of the sciences of learning (pp. 1677–1680). Springer. https://doi.org/10.1007/978-1-4419-1428-6_807

Britt, M. A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. Kirby & M. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). Cambridge University Press. https://doi.org/10.1017/CBO9781139048224.017

Britt, M. A., Rouet, J.-F., & Durik, A. M. (2018). Representations and processes in multiple source use. In J. L. G. Braasch, I. Bråten, & M. T. McCrudden (Eds.), Handbook of multiple source use (pp. 17–32). Routledge. https://doi.org/10.4324/9781315627496-2

Britt, M. A., & Sommer, J. (2004). Facilitating textual integration with marco-structure focusing tasks. Reading Psychology, 25 (4), 313–339. https://doi.org/10.1080/02702710490522658

Bromme, R. (2014). Der Lehrer als Experte: Zur Psychologie des professionellen Wissens (Vol. 7). Waxmann Verlag.

Google Scholar  

Brunner, M., Kunter, M., Krauss, S., Baumert, J., Blum, W., Dubberke, T., & Neubrand, M. (2006). Welche Zusammenhänge bestehen zwischen dem fachspezifischen Professionswissen von Mathematiklehrkräften und ihrer Ausbildung sowie beruflichen Fortbildung? Zeitschrift für Erziehungswissenschaft, 9 (4), 521–544. https://doi.org/10.1007/s11618-006-0166-1

Cerdán, R., & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100 (1), 209–222. https://doi.org/10.1037/0022-0663.100.1.209

Chan, C., Burtis, J., & Bereiter, C. (1997). Knowledge building as a mediator of conflict in conceptual change. Cognition and Instruction, 15 (1), 1–40. https://doi.org/10.1207/s1532690xci1501_1

Conley, A. M., Pintrich, P. R., Vekiri, I., & Harrison, D. (2004). Changes in epistemological beliefs in elementary science students. Epistemological Development and Its Impact on Cognition in Academic Domains, 29 (2), 186–204. https://doi.org/10.1016/j.cedpsych.2004.01.004

Darling-Hammond, L. (2006). Constructing 21st-century teacher education. Journal of Teacher Education, 57 (3), 300–314. https://doi.org/10.1177/0022487105285962

Deci, E. L., & Ryan, R. M. (n.d.). Intrinsic motivation inventory (IMI). http://selfdeterminationtheory.org/intrinsic-motivation-inventory/

Dörner, D. (1976). Problemlösen als Informationsverarbeitung (1st ed.). Kohlhammer.

Evans, V., & Green, M. (2006). Cognitive linguistics. An Introduction. Edinburgh: Edinburgh University Press. https://doi.org/10.4324/9781315864327

Gentner, D., & Stevens, A. L. (1983). Mental models . Erlbaum.

Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35 (3), 157–173. https://doi.org/10.1016/j.cedpsych.2009.11.002

Graichen, M., Wegner, E., & Nückles, M. (2019). Wie können Lehramtsstudierende beim Lernen durch Schreiben von Lernprotokollen unterstützt werden, dass die Kohärenz und Anwendbarkeit des erworbenen Professionswissens verbessert wird? Unterrichtswissenschaft, 47 (1), 7–28. https://doi.org/10.1007/s42010-019-00042-x

Hähnlein, I. (2018). Erfassung epistemologischer Überzeugungen von Lehramtsstudierenden. Entwicklung und Validierung des StEB Inventar. (Dissertation). Universität Passau. https://opus4.kobv.de/opus4-uni-passau/frontdoor/index/index/docId/588 .

Hähnlein, I., & Pirnay-Dummer, P. (2019). Assessing Metacognition in the Learning Process. Construction of the Metacognition in the Learning Process Inventory MILP . Paper presented at the European Association for Research on Learning and Instruction (EARLI) Conference, Aachen, Germany.

Harr, N., Eichler, A., & Renkl, A. (2014). Integrating pedagogical content knowledge and pedagogical/psychological knowledge in mathematics. Frontiers in Psychology, 5 , 1–10. https://doi.org/10.3389/fpsyg.2014.00924

Harr, N., Eichler, A., & Renkl, A. (2015). Integrated learning: Ways of fostering the applicability of teachers’ pedagogical and psychological knowledge. Frontiers in Psychology, 6 , 1–16. https://doi.org/10.3389/fpsyg.2015.00738

Helbig, H. (2006). Knowledge representation and the semantics of natural language. Springer . https://doi.org/10.1007/3-540-29966-1

Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on students’ achievement. American educational research journal, 42 (2), 371–406. https://doi.org/10.3102/00028312042002371

Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67 (1), 88–140. https://doi.org/10.3102/00346543067001088

Hudson, B., & Zgaga, P. (2017). History, context and overview: Implications for teacher education policy, practice and future research. In H. Brian (Ed.), Overcoming Fragmentation in Teacher Education Policy and Practice (pp. 1–10). Cambridge University Press.

Ifenthaler, D. (2010). Relational, structural, and semantic analysis of graphical representations and concept maps. Educational Technology Research and Development, 58 (1), 81–97. https://doi.org/10.1007/s11423-008-9087-4

Ifenthaler, D. (2012). Determining the effectiveness of prompts for self-regulated learning in problem-solving scenarios. Journal of Educational Technology & Society, 15 (1), 38–52.

Ifenthaler, D., & Lehmann, T. (2012). Preactional self-regulation as a tool for successful problem solving and learning. Technology, Instruction, Cognition and Learning, 9 (1–2), 97–110.

Janssen, N., & Lazonder, A. W. (2016). Supporting pre-service teachers in designing technology-infused lesson plans. Journal of Computer Assisted Learning, 32 (5), 456–467. https://doi.org/10.1111/jcal.12146

Johnson-Laird, P. N. (1983). Mental models: Toward a cognitive science of language, inference, and consicousness . Cambridge Univ Press. https://doi.org/10.2307/414498

Book   Google Scholar  

Jonassen, D. H. (2000). Computers as mindtools for schools: Engaging critical thinking . Cham: Prentice hall.

Jonassen, D. H. (2006). On the role of concepts in learning and instructional design. Educational Technology Research and Development, 54 (2), 177–196. https://doi.org/10.1007/s11423-006-8253-9

Jonassen, D. H. (2012). Problem typology. In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning. Springer. https://doi.org/10.1007/978-1-4419-1428-6_209

Jonassen, D. H., & Cho, Y. H. (2008). Externalizing mental models with mindtools. In D. Ifenthaler, P. Pirnay-Dummer, & J. M. Spector (Eds.), Understanding models for learning and instruction Essays in honor of Norbert M Seel (pp. 145–159). Springer. https://doi.org/10.1007/978-0-387-76898-4_7

Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction integration model. Psychological review, 95 (2), 163–182. https://doi.org/10.1037/0033-295X.95.2.163

Kintsch, W. (1998). Comprehension: A paradigm for cognition . Cambridge University Press.

Koehler, M. J., & Mishra, P. (2008). Introducing TPCK. In AACTE Committee on Innovation and Technology (Eds.), Handbook of technological pedagogical content knowledge (TPCK) for educators (pp. 2–29). Routledge.

König, J. (2010). Lehrerprofessionalität. Konzepte und Ergebnisse der internationalen und deutschen Forschung am Beispiel fachübergreifender, pädagogischer Kompetenzen. In J. König & B. Hofmann (Eds.), Professionalität von Lehrkräften. Was sollen Lehrkräfte im Lese- und Schreibunterricht wissen und können (pp. 40–106). Dgls.

Krauskopf, K., Zahn, C., & Hesse, F. W. (2020). Concetualizing (pre-service) teachers’ professional knowledge for complex domains. In T. Lehmann (Ed.), International perspectives on knowledge integration: Theory, research, and good practice in pre-service teacher and higher education (pp. 31–57). Brill Sense. https://doi.org/10.1163/9789004429499_003

Kulturministerkonferenz. (2009). Kompetenzstufenmodell zu den Bildungsstandards im Kompetenzbereich: Sprechen und Zuhören—hier Zuhören—für den Mittleren Schulabschluss . Beschluss der Kultusministerkonferenz.

Lehmann, T. (2020a). What is knowledge integration of multiple domains and how does it relate to teachers’ professional competence? In T. Lehmann (Ed.), International perspectives on knowledge integration: Theory, research, and good practice in pre-service teacher and higher education (pp. 9–29). Brill Sense. https://doi.org/10.1163/9789004429499_002

Lehmann, T. (2020b). International perspectives on knowledge integration: Theory, research, and good practice in pre-service teacher and higher education . Brill Academic Publishers. https://doi.org/10.1163/9789004429499

Lehmann, T. (2022). Student teachers’ knowledge integration across conceptual borders: The role of study approaches, learning strategies, beliefs, and motivation. European Journal of Psychology of Education . https://doi.org/10.1007/s10212-021-00577-7

Lehmann, T., Hähnlein, I., & Ifenthaler, D. (2014). Cognitive, metacognitive and motivational perspectives on preflection in self-regulated online learning. Computers in Human Behavior, 32 , 313–323. https://doi.org/10.1016/j.chb.2013.07.051

Lehmann, T., Klieme, K., & Schmidt-Borcherding, F. (2020). Separative and integrative learing in teacher education: Validity and reliability of the “SILTE” short scales. In T. Lehmann (Ed.), International perspectives on knowledge integration: Theory, research, and good practice in pre-service teacher and higher education (pp. 155–177). Brill Academic Publishers.

Lehmann, T., Pirnay-Dummer, P., & Schmidt-Borcherding, F. (2019a). Fostering integrated mental models of different professional knowledge domains: Instructional approaches and model-based analyses. Educational Technology Research and Development, 68 (3), 905–927. https://doi.org/10.1007/s11423-019-09704-0

Lehmann, T., Rott, B., & Schmidt-Borcherding, F. (2019b). Promoting pre-service teachers’ integration of professional knowledge: Effects of writing tasks and prompts on learning from multiple documents. Instructional Science, 47 (1), 99–126. https://doi.org/10.1007/s11251-018-9472-2

Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science Education, 22 (8), 781–796. https://doi.org/10.1080/095006900412275

Mishra, P. (2019). Considering contextual knowledge: The TPACK diagram gets an upgrade. Journal of Digital Learning in Teacher Education, 35 (2), 76–78. https://doi.org/10.1080/21532974.2019.1588611

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108 (6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x

Muis, K. R., Bendixen, L. D., & Haerle, F. C. (2006). Domain-generality and domain-specificity in personal epistemology research: Philosophical and empirical reflections in the development of a theoretical framework. Educational Psychology Review, 18 (1), 3–54.

Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and some new findings. The Psychology of Learning and Motivation, 26 , 125–173. https://doi.org/10.1016/S0079-7421(08)60053-5

Nelson, T. O., & Narens, L. (1994). Why investigate metacognition. In J. Metcalfe & A. Shimamura (Eds.), Metacognition knowing about knowing (pp. 1–25). Cambridge University Press.

Norton, P., van Rooij, S. W., Jerome, M. K., Clark, K., Behrmann, M., & Bannan-Ritland, B. (2009). Linking theory and practice through design: An instructional technology program. In M. Orey, V. J. McClendon, & R. M. Branch (Eds.), Educational media and technology yearbook (pp. 47–59). Springer. https://doi.org/10.1007/978-0-387-09675-9_4

Partee, B. H. (2004). Compositionality in formal semantics: Selected papers of Barbara Partee. Blackwell Pub . https://doi.org/10.1002/9780470751305

Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Lawrence Erlbaum Associates.

Pirnay-Dummer, P. (2006). Expertise und Modellbildung-MITOCAR. (Dissertation). Albert-Ludwigs-Universität Freiburg (Breisgau).

Pirnay-Dummer, P. (2007). Model inspection trace of concepts and relations: A heuristic approach to language-oriented model assessment. Paper presented at the annual meeting of AERA, Chicago IL.

Pirnay-Dummer, P. (2010). Complete structure comparison. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 235–258). Springer. https://doi.org/10.1007/978-1-4419-5662-0_13

Pirnay-Dummer, P. (2014). Gainfully guided misconception: How automatically generated knowledge maps can help companies within and across their projects. In D. Ifenthaler & R. Hanewald (Eds.), Digital knowledge maps in higher education: Technology-enhanced support for teachers and learners (pp. 253–274). Springer. https://doi.org/10.1007/978-1-4614-3178-7_14

Pirnay-Dummer, P. (2015a). Knowledge elicitation. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology (pp. 438–442). SAGE. https://doi.org/10.4135/9781483346397.n183

Pirnay-Dummer, P. (2015b). Linguistic analysis tools. In C. A. MacArthur, M. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 427–442). Guilford Publications.

Pirnay-Dummer, P. (2020). Knowledge and structure to teach: A model-based computer-linguistic approach to track, visualize, compare and cluster knowledge and knowledge integration in pre-service teachers. In T. Lehmann (Ed.), International perspectives on knowledge integration: Theory, research, and good practice in pre-service teacher and higher education (pp. 133–154). Brill Sense. https://doi.org/10.1163/9789004429499_007

Pirnay-Dummer, P., Ifenthaler, D., & Seel, N. M. (2012). Designing model-based learning environments to support mental models for learning. In D. H. Jonassen & S. M. Land (Eds.), Theoretical foundations of learning environments (pp. 55–90). Routledge. https://doi.org/10.4324/9780203813799

Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2010). Highly integrated model assessment technology and tools. Educational Technology Research and Development, 58 (1), 3–18. https://doi.org/10.1007/s11423-009-9119-8

Pirnay-Dummer, P., & Seel, N. M. (2018). The sciences of learning. In L. Lin & J. M. Spector (Eds.), The Sciences of learning and instructional design: Constructive articulation between communities (pp. 8–35). Routledge. https://doi.org/10.4324/9781315684444-2

R-Core-Team (2022). R: A language and environment for statistical computing (Version 4.2.2): R Foundation for Statistical Computing. https://www.R-project.org/ .

Reigeluth, C. M., & Stein, F. S. (1983). The elaboration theory of instruction. In C. M. Reigeluth (Ed.), Instructional design theories and models: An overview of their current status (pp. 335–382). Erlbaum. https://doi.org/10.4324/9780203824283

Renkl, A., Mandl, H., & Gruber, H. (1996). Inert knowledge: Analyses and remedies. Educational Psychologist, 31 (2), 115–121. https://doi.org/10.1207/s15326985ep3102_3

Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning. Lawrence Erlbaum Associates . https://doi.org/10.4324/9780203820094

Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). IAP Information Age Publishing.

Rouet, J.-F., Britt, M. A., & Durik, A. M. (2017). RESOLV: Readers’ representation of reading contexts and tasks. Educational Psychologist, 52 (3), 200–215. https://doi.org/10.1080/00461520.2017.1329015

Rule, D. C., & Bendixen, L. D. (2010). The integrative model of personal epistemology development: Theoretical underpinnings and implications for education. In L. D. Bendixen & F. C. Feucht (Eds.), Personal epistemology in the classroom (pp. 94–123). Cambridge University Press. https://doi.org/10.1017/CBO9780511691904.004

Scardamalia, M., & Bereiter, C. (1987). Knowledge telling and knowledge transforming in written composition. In S. Rosenberg (Ed.), Advances in applied psycholinguistics (Vol. 2, pp. 142–175). Cham: Cambridge University Press. https://doi.org/10.1017/S0142716400008596

Scardamalia, M., & Bereiter, C. (1991). Literate expertise. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits. Cham: Cambridge University Press.

Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3 (3), 265–283. https://doi.org/10.1207/s15327809jls0303_3

Scardamalia, M., & Bereiter, C. (1999). Schools as knowledge-building organizations. In D. P. Keating & C. Hertzman (Eds.), Developmental health and the wealth of nations: Social, biological, and educational dynamics (pp. 274–289). Guilford Press.

Schiefele, U., & Pekrun, R. (1996). Psychologische Modelle des selbstgesteuerten und fremdgesteuerten Lernens. In F. E. Weinert (Ed.), Enzyklopädie der Psychologie—Psychologie des Lernens und der Instruktion Pädagogische Psychologie (Vol. 2, pp. 249–287). Hogrefe.

Schneider, M. (2012). Knowledge integration. In N. M. Seel (Ed.), Encyclopedia of the sciences of learning (pp. 1684–1686). Springer. https://doi.org/10.1007/978-1-4419-1428-6_807

Schommer, M. (1994). An emerging conceptualization of epistemological beliefs and their role in learning. In R. Garner & P. A. Alexander (Eds.), Beliefs about text and instruction with text (pp. 25–40). Erlbaum.

Schwarzer, R., & Jerusalem, M. (2003). SWE: Skala zur Allgemeinen Selbstwirksamkeitserwartung [Verfahrensdokumentation, Autorenbeschreibung und Fragebogen]. In L.-I. f. P. (ZPID) (Ed.), Open Test Archive . https://doi.org/10.23668/psycharchives.4515

Seel, N. M. (1991). Weltwissen und mentale Modelle . Hogrefe Verlag f.

Seel, N. M. (2003). Model-centered learning and instruction. Technology, Instruction, Cognition and Learning, 1 (1), 59–85.

Seel, N. M. (2012). Learning and thinking. In N. M. Seel (Ed.), Encyclopedia of the sciences of learning (Vol. 4, pp. 1797–1799). Springer.

Seel, N. M., Ifenthaler, D., & Pirnay-Dummer, P. (2013). Mental models and their role in learning by insight and creative problem solving. In J. M. Spector, B. B. Lockee, S. E. Smaldino, & M. Herring (Eds.), Learning, problem solving, and mind tools: Essays in honor of David H. Jonassen (pp. 10–34). Springer.

Seel, N. M., Lehmann, T., Blumschein, P., & Podolskiy, O. A. (2017). Instructional design for learning: Theoretical foundations. Sense . https://doi.org/10.1007/978-94-6300-941-6

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15 (2), 4–14. https://doi.org/10.3102/0013189X015002004

Taylor, J. R. (2007). Cognitive linguistics and autonomous linguistics. In D. Geeraerts & H. Cuyckens (Eds.), Cognitive Linguistics (pp. 566–588). Oxford University Press.

Thompson, C. (2020). Allgemeine Erziehungswissenschaft . Eine Einführung (Vol. 3). http://www.klinkhardt.de/ewr/97831726165.html .

Tittmann, P. (2010). Graphs and networks. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 177–188). Springer. https://doi.org/10.1007/978-1-4419-5662-0_10

Trevors, G. J., Muis, K. R., Pekrun, R., Sinatra, G. M., & Winne, P. H. (2016). Identity and epistemic emotions during knowledge revision: A potential account for the backfire effect. Discourse Processes, 53 (5–6), 339–370. https://doi.org/10.1080/0163853X.2015.1136507

van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. Academic Press . https://doi.org/10.2307/415483

Voss, T., Kunter, M., & Baumert, J. (2011). Assessing teacher candidates’ general pedagogical/psychological knowledge: Test construction and validation. Journal of Educational Psychology, 103 (4), 952–969. https://doi.org/10.1037/a0025125

Wäschle, K., Lehmann, T., Brauch, N., & Nückles, M. (2015). Prompted journal writing supports preservice history teachers in drawing on multiple knowledge domains for designing learning tasks. Peabody Journal of Education, 90 (4), 546–559. https://doi.org/10.1080/0161956X.2015.1068084

Whitehead, A. N. (1929). The aims of education . Macmillan.

Wiley, J., & Voss, J. F. (1996). The effects of ‘playing historian’ on learning in history. Applied Cognitive Psychology, 10 (7), 63–72.

Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory from text. Journal of Educational Psychology, 91 (2), 301–311. https://doi.org/10.1037/0022-0663.91.2.301

Wilhelm, O., & Nickolaus, R. (2013). Was grenzt das Kompetenzkonzept von etablierten Kategorien wie Fähigkeit, Fertigkeit oder Intelligenz ab? Zeitschrift für Erziehungswissenschaft, 16 , 23–26. https://doi.org/10.1007/s11618-013-0380-6

Zeeb, H., Biwer, F., Brunner, G., Leuders, T., & Renkl, A. (2019). Make it relevant! How prior instructions foster the integration of teacher knowledge. Instructional Science, 47 (6), 711–739. https://doi.org/10.1007/s11251-019-09497-y

Zeeb, H., Spitzmesser, E., Röddiger, A., Leuders, T., & Renkl, A. (2020). Using relevance instructions to support the integration of teacher knowledge. In T. Lehmann (Ed.), International perspectives on knowledge integration: Theory, research, and good practice in pre-service teacher and higher education (pp. 201–229). Brill, Sense. https://doi.org/10.1163/9789004429499_010

Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into practice, 41 (2), 64–70. https://doi.org/10.1207/s15430421tip4102_2

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Faculty of Philosophy III, Institut of Pedagogy, Educational Psychology, Martin-Luther-University Halle-Wittenberg, Halle, Germany

Inka Sara Hähnlein & Pablo Pirnay-Dummer

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Inka Sara Hähnlein .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Hähnlein, I.S., Pirnay-Dummer, P. Promoting pre-service teachers’ knowledge integration from multiple text sources across domains with instructional prompts. Education Tech Research Dev (2024). https://doi.org/10.1007/s11423-024-10363-z

Download citation

Accepted : 19 February 2024

Published : 10 April 2024

DOI : https://doi.org/10.1007/s11423-024-10363-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Knowledge integration
• Multiple-document comprehension
• Pre-service teacher education
• Find a journal
• Publish with us
• Track your research