research methodology based on literature review

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Feb 29, 2024 12:00 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jan 17, 2024 10:05 AM
• URL: https://libguides.northwestern.edu/literaturereviews

Chapter 9. Reviewing the Literature

What is a “literature review”.

No researcher ever comes up with a research question that is wholly novel. Someone, somewhere, has asked the same thing. Academic research is part of a larger community of researchers, and it is your responsibility, as a member of this community, to acknowledge others who have asked similar questions and to put your particular research into this greater context. It is not simply a convention or custom to begin your study with a review of previous literature (the “ lit review ”) but an important responsibility you owe the scholarly community.

Too often, new researchers pursue a topic to study and then write something like, “No one has ever studied this before” or “This area is underresearched.” It may be that no one has studied this particular group or setting, but it is highly unlikely no one has studied the foundational phenomenon of interest. And that comment about an area being underresearched? Be careful. The statement may simply signal to others that you haven’t done your homework. Rubin ( 2021 ) refers to this as “free soloing,” and it is not appreciated in academic work:

The truth of the matter is, academics don’t really like when people free solo. It’s really bad form to omit talking about the other people who are doing or have done research in your area. Partly, I mean we need to cite their work, but I also mean we need to respond to it—agree or disagree, clarify for extend. It’s also really bad form to talk about your research in a way that does not make it understandable to other academics.…You have to explain to your readers what your story is really about in terms they care about . This means using certain terminology, referencing debates in the literature, and citing relevant works—that is, in connecting your work to something else. ( 51–52 )

A literature review is a comprehensive summary of previous research on a topic. It includes both articles and books—and in some cases reports—relevant to a particular area of research. Ideally, one’s research question follows from the reading of what has already been produced. For example, you are interested in studying sports injuries related to female gymnasts. You read everything you can find on sports injuries related to female gymnasts, and you begin to get a sense of what questions remain open. You find that there is a lot of research on how coaches manage sports injuries and much about cultures of silence around treating injuries, but you don’t know what the gymnasts themselves are thinking about these issues. You look specifically for studies about this and find several, which then pushes you to narrow the question further. Your literature review then provides the road map of how you came to your very specific question, and it puts your study in the context of studies of sports injuries. What you eventually find can “speak to” all the related questions as well as your particular one.

In practice, the process is often a bit messier. Many researchers, and not simply those starting out, begin with a particular question and have a clear idea of who they want to study and where they want to conduct their study but don’t really know much about other studies at all. Although backward, we need to recognize this is pretty common. Telling students to “find literature” after the fact can seem like a purposeless task or just another hurdle for completing a thesis or dissertation. It is not! Even if you were not motivated by the literature in the first place, acknowledging similar studies and connecting your own research to those studies are important parts of building knowledge. Acknowledgment of past research is a responsibility you owe the discipline to which you belong.

Literature reviews can also signal theoretical approaches and particular concepts that you will incorporate into your own study. For example, let us say you are doing a study of how people find their first jobs after college, and you want to use the concept of social capital . There are competing definitions of social capital out there (e.g., Bourdieu vs. Burt vs. Putnam). Bourdieu’s notion is of one form of capital, or durable asset, of a “network of more or less institutionalized relationships of mutual acquaintance or recognition” ( 1984:248 ). Burt emphasizes the “brokerage opportunities” in a social network as social capital ( 1997:355 ). Putnam’s social capital is all about “facilitating coordination and cooperation for mutual benefit” ( 2001:67 ). Your literature review can adjudicate among these three approaches, or it can simply refer to the one that is animating your own research. If you include Bourdieu in your literature review, readers will know “what kind” of social capital you are talking about as well as what kind of social scientist you yourself are. They will likely understand that you are interested more in how some people are advantaged by their social capital relative to others rather than being interested in the mechanics of how social networks operate.

The literature review thus does two important things for you: firstly, it allows you to acknowledge previous research in your area of interest, thereby situating you within a discipline or body of scholars, and, secondly, it demonstrates that you know what you are talking about. If you present the findings of your research study without including a literature review, it can be like singing into the wind. It sounds nice, but no one really hears it, or if they do catch snippets, they don’t know where it is coming from.

Examples of Literature Reviews

To help you get a grasp of what a good literature review looks like and how it can advance your study, let’s take a look at a few examples.

Reader-Friendly Example: The Power of Peers

The first is by Janice McCabe ( 2016 ) and is from an article on peer networks in the journal Contexts . Contexts presents articles in a relatively reader-friendly format, with the goal of reaching a large audience for interesting sociological research. Read this example carefully and note how easily McCabe is able to convey the relevance of her own work by situating it in the context of previous studies:

Scholars who study education have long acknowledged the importance of peers for students’ well-being and academic achievement. For example, in 1961, James Coleman argued that peer culture within high schools shapes students’ social and academic aspirations and successes. More recently, Judith Rich Harris has drawn on research in a range of areas—from sociological studies of preschool children to primatologists’ studies of chimpanzees and criminologists’ studies of neighborhoods—to argue that peers matter much more than parents in how children “turn out.” Researchers have explored students’ social lives in rich detail, as in Murray Milner’s book about high school students, Freaks, Geeks, and Cool Kids , and Elizabeth Armstrong and Laura Hamilton’s look at college students, Paying for the Party . These works consistently show that peers play a very important role in most students’ lives. They tend, however, to prioritize social over academic influence and to use a fuzzy conception of peers rather than focusing directly on friends—the relationships that should matter most for student success. Social scientists have also studied the power of peers through network analysis, which is based on uncovering the web of connections between people. Network analysis involves visually mapping networks and mathematically comparing their structures (such as the density of ties) and the positions of individuals within them (such as how central a given person is within the network). As Nicholas Christakis and James Fowler point out in their book Connected , network structure influences a range of outcomes, including health, happiness, wealth, weight, and emotions. Given that sociologists have long considered network explanations for social phenomena, it’s surprising that we know little about how college students’ friends impact their experiences. In line with this network tradition, I focus on the structure of friendship networks, constructing network maps so that the differences we see across participants are due to the underlying structure, including each participant’s centrality in their friendship group and the density of ties among their friends. ( 23 )

What did you notice? In her very second sentence, McCabe uses “for example” to introduce a study by Coleman, thereby indicating that she is not going to tell you every single study in this area but is going to tell you that (1) there is a lot of research in this area, (2) it has been going on since at least 1961, and (3) it is still relevant (i.e., recent studies are still being done now). She ends her first paragraph by summarizing the body of literature in this area (after giving you a few examples) and then telling you what may have been (so far) left out of this research. In the second paragraph, she shifts to a separate interesting focus that is related to the first but is also quite distinct. Lit reviews very often include two (or three) distinct strands of literature, the combination of which nicely backgrounds this particular study . In the case of our female gymnast study (above), those two strands might be (1) cultures of silence around sports injuries and (2) the importance of coaches. McCabe concludes her short and sweet literature review with one sentence explaining how she is drawing from both strands of the literature she has succinctly presented for her particular study. This example should show you that literature reviews can be readable, helpful, and powerful additions to your final presentation.

Authoritative Academic Journal Example: Working Class Students’ College Expectations

The second example is more typical of academic journal writing. It is an article published in the British Journal of Sociology of Education by Wolfgang Lehmann ( 2009 ):

Although this increase in post-secondary enrolment and the push for university is evident across gender, race, ethnicity, and social class categories, access to university in Canada continues to be significantly constrained for those from lower socio-economic backgrounds (Finnie, Lascelles, and Sweetman 2005). Rising tuition fees coupled with an overestimation of the cost and an underestimation of the benefits of higher education has put university out of reach for many young people from low-income families (Usher 2005). Financial constraints aside, empirical studies in Canada have shown that the most important predictor of university access is parental educational attainment. Having at least one parent with a university degree significantly increases the likelihood of a young person to attend academic-track courses in high school, have high educational and career aspirations, and ultimately attend university (Andres et al. 1999, 2000; Lehmann 2007a). Drawing on Bourdieu’s various writing on habitus and class-based dispositions (see, for example, Bourdieu 1977, 1990), Hodkinson and Sparkes (1997) explain career decisions as neither determined nor completely rational. Instead, they are based on personal experiences (e.g., through employment or other exposure to occupations) and advice from others. Furthermore, they argue that we have to understand these decisions as pragmatic, rather than rational. They are pragmatic in that they are based on incomplete and filtered information, because of the social context in which the information is obtained and processed. New experiences and information can, however, also be allowed into one’s world, where they gradually or radically transform habitus, which in turn creates the possibility for the formation of new and different dispositions. Encountering a supportive teacher in elementary or secondary school, having ambitious friends, or chance encounters can spark such transformations. Transformations can be confirming or contradictory, they can be evolutionary or dislocating. Working-class students who enter university most certainly encounter such potentially transformative situations. Granfield (1991) has shown how initially dislocating feelings of inadequacy and inferiority of working-class students at an elite US law school were eventually replaced by an evolutionary transformation, in which the students came to dress, speak and act more like their middle-class and upper-class peers. In contrast, Lehmann (2007b) showed how persistent habitus dislocation led working-class university students to drop out of university. Foskett and Hemsley-Brown (1999) argue that young people’s perceptions of careers are a complex mix of their own experiences, images conveyed through adults, and derived images conveyed by the media. Media images of careers, perhaps, are even more important for working-class youth with high ambitions as they offer (generally distorted) windows into a world of professional employment to which they have few other sources of access. It has also been argued that working-class youth who do continue to university still face unique, class-specific challenges, evident in higher levels of uncertainty (Baxter and Britton 2001; Lehmann 2004, 2007a; Quinn 2004), their higher education choices (Ball et al. 2002; Brooks 2003; Reay et al. 2001) and fears of inadequacy because of their cultural outsider status (Aries and Seider 2005; Granfield 1991). Although the number of working-class university students in Canada has slowly increased, that of middle-class students at university has risen far more steeply (Knighton and Mizra 2002). These different enrolment trajectories have actually widened the participation gap, which in tum explains our continued concerns with the potential outsider status Indeed, in a study comparing first-generation working-class and traditional students who left university without graduating, Lehmann (2007b) found that first-generation working-class students were more likely to leave university very early in some cases within the first two months of enrollment. They were also more likely to leave university despite solid academic performance. Not “fitting in,” not “feeling university,” and not being able to “relate to these people” were key reasons for eventually withdrawing from university. From the preceding review of the literature, a number of key research questions arise: How do working-class university students frame their decision to attend university? How do they defy the considerable odds documented in the literature to attend university? What are the sources of information and various images that create dispositions to study at university? What role does their social-class background- or habitus play in their transition dispositions and how does this translate into expectations for university? ( 139 )

What did you notice here? How is this different from (and similar to) the first example? Note that rather than provide you with one or two illustrative examples of similar types of research, Lehmann provides abundant source citations throughout. He includes theory and concepts too. Like McCabe, Lehmann is weaving through multiple literature strands: the class gap in higher education participation in Canada, class-based dispositions, and obstacles facing working-class college students. Note how he concludes the literature review by placing his research questions in context.

Find other articles of interest and read their literature reviews carefully. I’ve included two more for you at the end of this chapter . As you learned how to diagram a sentence in elementary school (hopefully!), try diagramming the literature reviews. What are the “different strands” of research being discussed? How does the author connect these strands to their own research questions? Where is theory in the lit review, and how is it incorporated (e.g., Is it a separate strand of its own or is it inextricably linked with previous research in this area)?

One model of how to structure your literature review can be found in table 9.1. More tips, hints, and practices will be discussed later in the chapter.

Table 9.1. Model of Literature Review, Adopted from Calarco (2020:166)

Embracing Theory

A good research study will, in some form or another, use theory. Depending on your particular study (and possibly the preferences of the members of your committee), theory may be built into your literature review. Or it may form its own section in your research proposal/design (e.g., “literature review” followed by “theoretical framework”). In my own experience, I see a lot of graduate students grappling with the requirement to “include theory” in their research proposals. Things get a little squiggly here because there are different ways of incorporating theory into a study (Are you testing a theory? Are you generating a theory?), and based on these differences, your literature review proper may include works that describe, explain, and otherwise set forth theories, concepts, or frameworks you are interested in, or it may not do this at all. Sometimes a literature review sets forth what we know about a particular group or culture totally independent of what kinds of theoretical framework or particular concepts you want to explore. Indeed, the big point of your study might be to bring together a body of work with a theory that has never been applied to it previously. All this is to say that there is no one correct way to approach the use of theory and the writing about theory in your research proposal.

Students are often scared of embracing theory because they do not exactly understand what it is. Sometimes, it seems like an arbitrary requirement. You’re interested in a topic; maybe you’ve even done some research in the area and you have findings you want to report. And then a committee member reads over what you have and asks, “So what?” This question is a good clue that you are missing theory, the part that connects what you have done to what other researchers have done and are doing. You might stumble upon this rather accidentally and not know you are embracing theory, as in a case where you seek to replicate a prior study under new circumstances and end up finding that a particular correlation between behaviors only happens when mediated by something else. There’s theory in there, if you can pull it out and articulate it. Or it might be that you are motivated to do more research on racial microaggressions because you want to document their frequency in a particular setting, taking for granted the kind of critical race theoretical framework that has done the hard work of defining and conceptualizing “microaggressions” in the first place. In that case, your literature review could be a review of Critical Race Theory, specifically related to this one important concept. That’s the way to bring your study into a broader conversation while also acknowledging (and honoring) the hard work that has preceded you.

Rubin ( 2021 ) classifies ways of incorporating theory into case study research into four categories, each of which might be discussed somewhat differently in a literature review or theoretical framework section. The first, the least theoretical, is where you set out to study a “configurative idiographic case” ( 70 ) This is where you set out to describe a particular case, leaving yourself pretty much open to whatever you find. You are not expecting anything based on previous literature. This is actually pretty weak as far as research design goes, but it is probably the default for novice researchers. Your committee members should probably help you situate this in previous literature in some way or another. If they cannot, and it really does appear you are looking at something fairly new that no one else has bothered to research before, and you really are completely open to discovery, you might try using a Grounded Theory approach, which is a methodological approach that foregrounds the generation of theory. In that case, your “theory” section can be a discussion of “Grounded Theory” methodology (confusing, yes, but if you take some time to ponder, you will see how this works). You will still need a literature review, though. Ideally one that describes other studies that have ever looked at anything remotely like what you are looking at—parallel cases that have been researched.

The second approach is the “disciplined configurative case,” in which theory is applied to explain a particular case or topic. You are not trying to test the theory but rather assuming the theory is correct, as in the case of exploring microaggressions in a particular setting. In this case, you really do need to have a separate theory section in addition to the literature review, one in which you clearly define the theoretical framework, including any of its important concepts. You can use this section to discuss how other researchers have used the concepts and note any discrepancies in definitions or operationalization of those concepts. This way you will be sure to design your study so that it speaks to and with other researchers. If everyone who is writing about microaggressions has a different definition of them, it is hard for others to compare findings or make any judgments about their prevalence (or any number of other important characteristics). Your literature review section may then stand alone and describe previous research in the particular area or setting, irrespective of the kinds of theory underlying those studies.

The third approach is “heuristic,” one in which you seek to identify new variables, hypotheses, mechanisms, or paths not yet explained by a theory or theoretical framework. In a way, you are generating new theory, but it is probably more accurate to say that you are extending or deepening preexisting theory. In this case, having a single literature review that is focused on the theory and the ways the theory has been applied and understood (with all its various mechanisms and pathways) is probably your best option. The focus of the literature reviewed is less on the case and more on the theory you are seeking to extend.

The final approach is “theory testing,” which is much rarer in qualitative studies than in quantitative, where this is the default approach. Theory-testing cases are those where a particular case is used to see if an existing theory is accurate or accurate under particular circumstances. As with the heuristic approach, your literature review will probably draw heavily on previous uses of the theory, but you may end up having a special section specifically about cases very close to your own . In other words, the more your study approaches theory testing, the more likely there is to be a set of similar studies to draw on or even one important key study that you are setting your own study up in parallel to in order to find out if the theory generated there operates here.

If we wanted to get very technical, it might be useful to distinguish theoretical frameworks properly from conceptual frameworks. The latter are a bit looser and, given the nature of qualitative research, often fit exploratory studies. Theoretical frameworks rely on specific theories and are essential for theory-testing studies. Conceptual frameworks can pull in specific concepts or ideas that may or may not be linked to particular theories. Think about it this way: A theory is a story of how the world works. Concepts don’t presume to explain the whole world but instead are ways to approach phenomena to help make sense of them. Microaggressions are concepts that are linked to Critical Race Theory. One could contextualize one’s study within Critical Race Theory and then draw various concepts, such as that of microaggressions from the overall theoretical framework. Or one could bracket out the master theory or framework and employ the concept of microaggression more opportunistically as a phenomenon of interest. If you are unsure of what theory you are using, you might want to frame a more practical conceptual framework in your review of the literature.

Helpful Tips

How to maintain good notes for what your read.

Over the years, I have developed various ways of organizing notes on what I read. At first, I used a single sheet of full-size paper with a preprinted list of questions and points clearly addressed on the front side, leaving the second side for more reflective comments and free-form musings about what I read, why it mattered, and how it might be useful for my research. Later, I developed a system in which I use a single 4″ × 6″ note card for each book I read. I try only to use the front side (and write very small), leaving the back for comments that are about not just this reading but things to do or examine or consider based on the reading. These notes often mean nothing to anyone else picking up the card, but they make sense to me. I encourage you to find an organizing system that works for you. Then when you set out to compose a literature review, instead of staring at five to ten books or a dozen articles, you will have ten neatly printed pages or notecards or files that have distilled what is important to know about your reading.

It is also a good idea to store this data digitally, perhaps through a reference manager. I use RefWorks, but I also recommend EndNote or any other system that allows you to search institutional databases. Your campus library will probably provide access to one of these or another system. Most systems will allow you to export references from another manager if and when you decide to move to another system. Reference managers allow you to sort through all your literature by descriptor, author, year, and so on. Even so, I personally like to have the ability to manually sort through my index cards, recategorizing things I have read as I go. I use RefWorks to keep a record of what I have read, with proper citations, so I can create bibliographies more easily, and I do add in a few “notes” there, but the bulk of my notes are kept in longhand.

What kinds of information should you include from your reading? Here are some bulleted suggestions from Calarco ( 2020:113–114 ), with my own emendations:

• Citation . If you are using a reference manager, you can import the citation and then, when you are ready to create a bibliography, you can use a provided menu of citation styles, which saves a lot of time. If you’ve originally formatted in Chicago Style but the journal you are writing for wants APA style, you can change your entire bibliography in less than a minute. When using a notecard for a book, I include author, title, date as well as the library call number (since most of what I read I pull from the library). This is something RefWorks is not able to do, and it helps when I categorize.

I begin each notecard with an “intro” section, where I record the aims, goals, and general point of the book/article as explained in the introductory sections (which might be the preface, the acknowledgments, or the first two chapters). I then draw a bold line underneath this part of the notecard. Everything after that should be chapter specific. Included in this intro section are things such as the following, recommended by Calarco ( 2020 ):

• Key background . “Two to three short bullet points identifying the theory/prior research on which the authors are building and defining key terms.”
• Data/methods . “One or two short bullet points with information about the source of the data and the method of analysis, with a note if this is a novel or particularly effective example of that method.” I use [M] to signal methodology on my notecard, which might read, “[M] Int[erview]s (n-35), B[lack]/W[hite] voters” (I need shorthand to fit on my notecard!).
• Research question . “Stated as briefly as possible.” I always provide page numbers so I can go back and see exactly how this was stated (sometimes, in qualitative research, there are multiple research questions, and they cannot be stated simply).
• Argument/contributions . “Two to three short bullet points briefly describing the authors’ answer to the central research question and its implication for research, theory, and practice.” I use [ARG] for argument to signify the argument, and I make sure this is prominently visible on my notecard. I also provide page numbers here.

For me, all of this fits in the “intro” section, which, if this is a theoretically rich, methodologically sound book, might take up a third or even half of the front page of my notecard. Beneath the bold underline, I report specific findings or particulars of the book as they emerge chapter by chapter. Calarco’s ( 2020 ) next step is the following:

• Key findings . “Three to four short bullet points identifying key patterns in the data that support the authors’ argument.”

All that remains is writing down thoughts that occur upon finishing the article/book. I use the back of the notecard for these kinds of notes. Often, they reach out to other things I have read (e.g., “Robinson reminds me of Crusoe here in that both are looking at the effects of social isolation, but I think Robinson makes a stronger argument”). Calarco ( 2020 ) concludes similarly with the following:

• Unanswered questions . “Two to three short bullet points that identify key limitations of the research and/or questions the research did not answer that could be answered in future research.”

As I mentioned, when I first began taking notes like this, I preprinted pages with prompts for “research question,” “argument,” and so on. This was a great way to remind myself to look for these things in particular. You can do the same, adding whatever preprinted sections make sense to you, given what you are studying and the important aspects of your discipline. The other nice thing about the preprinted forms is that it keeps your writing to a minimum—you cannot write more than the allotted space, even if you might want to, preventing your notes from spiraling out of control. This can be helpful when we are new to a subject and everything seems worth recording!

After years of discipline, I have finally settled on my notecard approach. I have thousands of notecards, organized in several index card filing boxes stacked in my office. On the top right of each card is a note of the month/day I finished reading the item. I can remind myself what I read in the summer of 2010 if the need or desire ever arose to do so…those invaluable notecards are like a memento of what my brain has been up to!

Where to Start Looking for Literature

Your university library should provide access to one of several searchable databases for academic books and articles. My own preference is JSTOR, a service of ITHAKA, a not-for-profit organization that works to advance and preserve knowledge and to improve teaching and learning through the use of digital technologies. JSTOR allows you to search by several keywords and to narrow your search by type of material (articles or books). For many disciplines, the “literature” of the literature review is expected to be peer-reviewed “articles,” but some disciplines will also value books and book chapters. JSTOR is particularly useful for article searching. You can submit several keywords and see what is returned, and you can also narrow your search by a particular journal or discipline. If your discipline has one or two key journals (e.g., the American Journal of Sociology and the American Sociological Review are key for sociology), you might want to go directly to those journals’ websites and search for your topic area. There is an art to when to cast your net widely and when to refine your search, and you may have to tack back and forth to ensure that you are getting all that is relevant but not getting bogged down in all studies that might have some marginal relevance.

Some articles will carry more weight than others, and you can use applications like Google Scholar to see which articles have made and are continuing to make larger impacts on your discipline. Find these articles and read them carefully; use their literature review and the sources cited in those articles to make sure you are capturing what is relevant. This is actually a really good way of finding relevant books—only the most impactful will make it into the citations of journals. Over time, you will notice that a handful of articles (or books) are cited so often that when you see, say, Armstrong and Hamilton ( 2015 ), you know exactly what book this is without looking at the full cite. This is when you know you are in the conversation.

You might also approach a professor whose work is broadly in the area of your interest and ask them to recommend one or two “important” foundational articles or books. You can then use the references cited in those recommendations to build up your literature. Just be careful: some older professors’ knowledge of the literature (and I reluctantly add myself here) may be a bit outdated! It is best that the article or book whose references and sources you use to build your body of literature be relatively current.

Keep a List of Your Keywords

When using searchable databases, it is a good idea to keep a list of all the keywords you use as you go along so that (1) you do not needlessly duplicate your efforts and (2) you can more easily adjust your search as you get a better sense of what you are looking for. I suggest you keep a separate file or even a small notebook for this and you date your search efforts.

Here’s an example:

Table 9.2. Keep a List of Your Keywords

Think Laterally

How to find the various strands of literature to combine? Don’t get stuck on finding the exact same research topic you think you are interested in. In the female gymnast example, I recommended that my student consider looking for studies of ballerinas, who also suffer sports injuries and around whom there is a similar culture of silence. It turned out that there was in fact research about my student’s particular questions, just not about the subjects she was interested in. You might do something similar. Don’t get stuck looking for too direct literature but think about the broader phenomenon of interest or analogous cases.

Read Outside the Canon

Some scholars’ work gets cited by everyone all the time. To some extent, this is a very good thing, as it helps establish the discipline. For example, there are a lot of “Bourdieu scholars” out there (myself included) who draw ideas, concepts, and quoted passages from Bourdieu. This makes us recognizable to one another and is a way of sharing a common language (e.g., where “cultural capital” has a particular meaning to those versed in Bourdieusian theory). There are empirical studies that get cited over and over again because they are excellent studies but also because there is an “echo chamber effect” going on, where knowing to cite this study marks you as part of the club, in the know, and so on. But here’s the problem with this: there are hundreds if not thousands of excellent studies out there that fail to get appreciated because they are crowded out by the canon. Sometimes this happens because they are published in “lower-ranked” journals and are never read by a lot of scholars who don’t have time to read anything other than the “big three” in their field. Other times this happens because the author falls outside of the dominant social networks in the field and thus is unmentored and fails to get noticed by those who publish a lot in those highly ranked and visible spaces. Scholars who fall outside the dominant social networks and who publish outside of the top-ranked journals are in no way less insightful than their peers, and their studies may be just as rigorous and relevant to your work, so it is important for you to take some time to read outside the canon. Due to how a person’s race, gender, and class operate in the academy, there is also a matter of social justice and ethical responsibility involved here: “When you focus on the most-cited research, you’re more likely to miss relevant research by women and especially women of color, whose research tends to be under-cited in most fields. You’re also more likely to miss new research, research by junior scholars, and research in other disciplines that could inform your work. Essentially, it is important to read and cite responsibly, which means checking that you’re not just reading and citing the same white men and the same old studies that everyone has cited before you” ( Calarco 2020:112 ).

Consider Multiple Uses for Literature

Throughout this chapter, I’ve referred to the literature of interest in a rather abstract way, as what is relevant to your study. But there are many different ways previous research can be relevant to your study. The most basic use of the literature is the “findings”—for example, “So-and-so found that Canadian working-class students were concerned about ‘fitting in’ to the culture of college, and I am going to look at a similar question here in the US.” But the literature may be of interest not for its findings but theoretically—for example, employing concepts that you want to employ in your own study. Bourdieu’s definition of social capital may have emerged in a study of French professors, but it can still be relevant in a study of, say, how parents make choices about what preschools to send their kids to (also a good example of lateral thinking!).

If you are engaged in some novel methodological form of data collection or analysis, you might look for previous literature that has attempted that. I would not recommend this for undergraduate research projects, but for graduate students who are considering “breaking the mold,” find out if anyone has been there before you. Even if their study has absolutely nothing else in common with yours, it is important to acknowledge that previous work.

Describing Gaps in the Literature

First, be careful! Although it is common to explain how your research adds to, builds upon, and fills in gaps in the previous research (see all four literature review examples in this chapter for this), there is a fine line between describing the gaps and misrepresenting previous literature by failing to conduct a thorough review of the literature. A little humility can make a big difference in your presentation. Instead of “This is the first study that has looked at how firefighters juggle childcare during forest fire season,” say, “I use the previous literature on how working parents juggling childcare and the previous ethnographic studies of firefighters to explore how firefighters juggle childcare during forest fire season.” You can even add, “To my knowledge, no one has conducted an ethnographic study in this specific area, although what we have learned from X about childcare and from Y about firefighters would lead us to expect Z here.” Read more literature review sections to see how others have described the “gaps” they are filling.

Use Concept Mapping

Concept mapping is a helpful tool for getting your thoughts in order and is particularly helpful when thinking about the “literature” foundational to your particular study. Concept maps are also known as mind maps, which is a delightful way to think about them. Your brain is probably abuzz with competing ideas in the early stages of your research design. Write/draw them on paper, and then try to categorize and move the pieces around into “clusters” that make sense to you. Going back to the gymnasts example, my student might have begun by jotting down random words of interest: gymnasts * sports * coaches * female gymnasts * stress * injury * don’t complain * women in sports * bad coaching * anxiety/stress * careers in sports * pain. She could then have begun clustering these into relational categories (bad coaching, don’t complain culture) and simple “event” categories (injury, stress). This might have led her to think about reviewing literature in these two separate aspects and then literature that put them together. There is no correct way to draw a concept map, as they are wonderfully specific to your mind. There are many examples you can find online.

Ask Yourself, “How Is This Sociology (or Political Science or Public Policy, Etc.)?”

Rubin ( 2021:82 ) offers this suggestion instead of asking yourself the “So what?” question to get you thinking about what bridges there are between your study and the body of research in your particular discipline. This is particularly helpful for thinking about theory. Rubin further suggests that if you are really stumped, ask yourself, “What is the really big question that all [fill in your discipline here] care about?” For sociology, it might be “inequality,” which would then help you think about theories of inequality that might be helpful in framing your study on whatever it is you are studying—OnlyFans? Childcare during COVID? Aging in America? I can think of some interesting ways to frame questions about inequality for any of those topics. You can further narrow it by focusing on particular aspects of inequality (Gender oppression? Racial exclusion? Heteronormativity?). If your discipline is public policy, the big questions there might be, How does policy get enacted, and what makes a policy effective? You can then take whatever your particular policy interest is—tax reform, student debt relief, cap-and-trade regulations—and apply those big questions. Doing so would give you a handle on what is otherwise an intolerably vague subject (e.g., What about student debt relief?).

Sometimes finding you are in new territory means you’ve hit the jackpot, and sometimes it means you’ve traveled out of bounds for your discipline. The jackpot scenario is wonderful. You are doing truly innovative research that is combining multiple literatures or is addressing a new or under-examined phenomenon of interest, and your research has the potential to be groundbreaking. Congrats! But that’s really hard to do, and it might be more likely that you’ve traveled out of bounds, by which I mean, you are no longer in your discipline . It might be that no one has written about this thing—at least within your field— because no one in your field actually cares about this topic . ( Rubin 2021:83 ; emphases added)

Don’t Treat This as a Chore

Don’t treat the literature review as a chore that has to be completed, but see it for what it really is—you are building connections to other researchers out there. You want to represent your discipline or area of study fairly and adequately. Demonstrate humility and your knowledge of previous research. Be part of the conversation.

Supplement: Two More Literature Review Examples

Elites by harvey ( 2011 ).

In the last two decades, there has been a small but growing literature on elites. In part, this has been a result of the resurgence of ethnographic research such as interviews, focus groups, case studies, and participant observation but also because scholars have become increasingly interested in understanding the perspectives and behaviors of leaders in business, politics, and society as a whole. Yet until recently, our understanding of some of the methodological challenges of researching elites has lagged behind our rush to interview them.

There is no clear-cut definition of the term elite, and given its broad understanding across the social sciences, scholars have tended to adopt different approaches. Zuckerman (1972) uses the term ultraelites to describe individuals who hold a significant amount of power within a group that is already considered elite. She argues, for example, that US senators constitute part of the country’s political elite but that among them are the ultraelites: a “subset of particularly powerful or prestigious influentials” (160). She suggests that there is a hierarchy of status within elite groups. McDowell (1998) analyses a broader group of “professional elites” who are employees working at different levels for merchant and investment banks in London. She classifies this group as elite because they are “highly skilled, professionally competent, and class-specific” (2135). Parry (1998:2148) uses the term hybrid elites in the context of the international trade of genetic material because she argues that critical knowledge exists not in traditional institutions “but rather as increasingly informal, hybridised, spatially fragmented, and hence largely ‘invisible,’ networks of elite actors.” Given the undertheorization of the term elite, Smith (2006) recognizes why scholars have shaped their definitions to match their respondents . However, she is rightly critical of the underlying assumption that those who hold professional positions necessarily exert as much influence as initially perceived. Indeed, job titles can entirely misrepresent the role of workers and therefore are by no means an indicator of elite status (Harvey 2010).

Many scholars have used the term elite in a relational sense, defining them either in terms of their social position compared to the researcher or compared to the average person in society (Stephens 2007). The problem with this definition is there is no guarantee that an elite subject will necessarily translate this power and authority in an interview setting. Indeed, Smith (2006) found that on the few occasions she experienced respondents wanting to exert their authority over her, it was not from elites but from relatively less senior workers. Furthermore, although business and political elites often receive extensive media training, they are often scrutinized by television and radio journalists and therefore can also feel threatened in an interview, particularly in contexts that are less straightforward to prepare for such as academic interviews. On several occasions, for instance, I have been asked by elite respondents or their personal assistants what they need to prepare for before the interview, which suggests that they consider the interview as some form of challenge or justification for what they do.

In many cases, it is not necessarily the figureheads or leaders of organizations and institutions who have the greatest claim to elite status but those who hold important social networks, social capital, and strategic positions within social structures because they are better able to exert influence (Burt 1992; Parry 1998; Smith 2005; Woods 1998). An elite status can also change, with people both gaining and losing theirs over time. In addition, it is geographically specific, with people holding elite status in some but not all locations. In short, it is clear that the term elite can mean many things in different contexts, which explains the range of definitions. The purpose here is not to critique these other definitions but rather to highlight the variety of perspectives.

When referring to my research, I define elites as those who occupy senior-management- and board-level positions within organizations. This is a similar scope of definition to Zuckerman’s (1972) but focuses on a level immediately below her ultraelite subjects. My definition is narrower than McDowell’s (1998) because it is clear in the context of my research that these people have significant decision-making influence within and outside of the firm and therefore present a unique challenge to interview. I deliberately use the term elite more broadly when drawing on examples from the theoretical literature in order to compare my experiences with those who have researched similar groups.

”Changing Dispositions among the Upwardly Mobile” by Curl, Lareau, and Wu ( 2018 )

There is growing interest in the role of cultural practices in undergirding the social stratification system. For example, Lamont et al. (2014) critically assess the preoccupation with economic dimensions of social stratification and call for more developed cultural models of the transmission of inequality. The importance of cultural factors in the maintenance of social inequality has also received empirical attention from some younger scholars, including Calarco (2011, 2014) and Streib (2015). Yet questions remain regarding the degree to which economic position is tied to cultural sensibilities and the ways in which these cultural sensibilities are imprinted on the self or are subject to change. Although habitus is a core concept in Bourdieu’s theory of social reproduction, there is limited empirical attention to the precise areas of the habitus that can be subject to change during upward mobility as well as the ramifications of these changes for family life.

In Bourdieu’s (1984) highly influential work on the importance of class-based cultural dispositions, habitus is defined as a “durable system of dispositions” created in childhood. The habitus provides a “matrix of perceptions” that seems natural while also structuring future actions and pathways. In many of his writings, Bourdieu emphasized the durability of cultural tastes and dispositions and did not consider empirically whether these dispositions might be changed or altered throughout one’s life (Swartz 1997). His theoretical work does permit the possibility of upward mobility and transformation, however, through the ability of the habitus to “improvise” or “change” due to “new experiences” (Friedman 2016:131). Researchers have differed in opinion on the durability of the habitus and its ability to change (King 2000). Based on marital conflict in cross-class marriages, for instance, Streib (2015) argues that cultural dispositions of individuals raised in working-class families are deeply embedded and largely unchanging. In a somewhat different vein, Horvat and Davis (2011:152) argue that young adults enrolled in an alternative educational program undergo important shifts in their self-perception, such as “self-esteem” and their “ability to accomplish something of value.” Others argue there is variability in the degree to which habitus changes dependent on life experience and personality (Christodoulou and Spyridakis 2016). Recently, additional studies have investigated the habitus as it intersects with lifestyle through the lens of meaning making (Ambrasat et al. 2016). There is, therefore, ample discussion of class-based cultural practices in self-perception (Horvat and Davis 2011), lifestyle (Ambrasat et al. 2016), and other forms of taste (Andrews 2012; Bourdieu 1984), yet researchers have not sufficiently delineated which aspects of the habitus might change through upward mobility or which specific dimensions of life prompt moments of class-based conflict.

Bourdieu (1999:511; 2004) acknowledged simmering tensions between the durable aspects of habitus and those aspects that have been transformed—that is, a “fractured” or “cleft” habitus. Others have explored these tensions as a “divided” or “fragmented” habitus (Baxter and Britton 2001; Lee and Kramer 2013). Each of these conceptions of the habitus implies that changes in cultural dispositions are possible but come with costs. Exploration of the specific aspects of one’s habitus that can change and generate conflict contributes to this literature.

Scholars have also studied the costs associated with academic success for working-class undergraduates (Hurst 2010; Lee and Kramer 2013; London 1989; Reay 2017; Rondini 2016; Stuber 2011), but we know little about the lasting effects on adults. For instance, Lee and Kramer (2013) point to cross-class tensions as family and friends criticize upwardly mobile individuals for their newly acquired cultural dispositions. Documenting the tension many working-class students experience with their friends and families of origin, they find that the source of their pain or struggle is “shaped not only by their interactions with non-mobile family and friends but also within their own minds, by their own assessments of their social positions, and by how those positions are interpreted by others” (Lee and Kramer 2013:29). Hurst (2010) also explores the experiences of undergraduates who have been academically successful and the costs associated with that success. She finds that decisions about “class allegiance and identity” are required aspects of what it means to “becom[e] educated” (4) and that working-class students deal with these cultural changes differently. Jack (2014, 2016) also argues that there is diversity among lower-income students, which yields varied college experiences. Naming two groups, the “doubly disadvantaged” and the “privileged poor,” he argues that previous experience with “elite environments” (2014:456) prior to college informs students’ ability to take on dominant cultural practices, particularly around engagement, such as help seeking or meeting with professors (2016). These studies shed light on the role college might play as a “lever for mobility” (2016:15) and discuss the pain and difficulty associated with upward mobility among undergraduates, but the studies do not illuminate how these tensions unfold in adulthood. Neither have they sufficiently addressed potential enduring tensions with extended family members as well as the specific nature of the difficulties.

Some scholars point to the positive outcomes upwardly mobile youth (Lehmann 2009) and adults (Stuber 2005) experience when they maintain a different habitus than their newly acquired class position, although, as Jack (2014, 2016) shows, those experiences may vary depending on one’s experience with elite environments in their youth. Researchers have not sufficiently explored the specific aspects of the habitus that upwardly mobile adults change or the conflicts that emerge with family and childhood friends as they reach adulthood and experience colliding social worlds. We contribute to this scholarship with clear examples of self-reported changes to one’s cultural dispositions in three specific areas: “horizons,” food and health, and communication. We link these changes to enduring tension with family members, friends, and colleagues and explore varied responses to this tension based on race.

Further Readings

Bloomberg, Linda Dale, and Marie F. Volpe. 2012. Completing Your Qualitative Dissertation: A Road Map from Beginning to End . 2nd ed. Thousand Oaks, CA: SAGE. In keeping with its general approach to qualitative research, includes a “road map” for conducting a literature review.

Hart, Chris. 1998. Doing a Literature Review: Releasing the Social Science Research Imagination . London: SAGE. A how-to book dedicated entirely to conducting a literature review from a British perspective. Useful for both undergraduate and graduate students.

Machi, Lawrence A., and Brenda T. McEvoy. 2022. The Literature Review: Six Steps to Success . 4th ed. Newbury Park, CA: Corwin. A well-organized guidebook complete with reflection sections to prompt successful thinking about your literature review.

Ridley, Diana. 2008. The Literature Review: A Step-by-Step Guide for Students . London: SAGE. A highly recommended companion to conducting a literature review for doctoral-level students.

The process of systematically searching through pre-existing studies (“literature”) on the subject of research; also, the section of a presentation in which the pre-existing literature is discussed.

Follow-up questions used in a semi-structured interview  to elicit further elaboration.  Suggested prompts can be included in the interview guide  to be used/deployed depending on how the initial question was answered or if the topic of the prompt does not emerge spontaneously.

A tool for identifying relationships among ideas by visually representing them on paper.  Most concept maps depict ideas as boxes or circles (also called nodes), which are structured hierarchically and connected with lines or arrows (also called arcs). These lines are labeled with linking words and phrases to help explain the connections between concepts.  Also known as mind mapping.

The people who are the subjects of an interview-based qualitative study. In general, they are also known as the participants, and for purposes of IRBs they are often referred to as the human subjects of the research.

Introduction to Qualitative Research Methods Copyright © 2023 by Allison Hurst is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License , except where otherwise noted.

Types of Literature Review

There are many types of literature review. The choice of a specific type depends on your research approach and design. The following types of literature review are the most popular in business studies:

Narrative literature review , also referred to as traditional literature review, critiques literature and summarizes the body of a literature. Narrative review also draws conclusions about the topic and identifies gaps or inconsistencies in a body of knowledge. You need to have a sufficiently focused research question to conduct a narrative literature review

Systematic literature review requires more rigorous and well-defined approach compared to most other types of literature review. Systematic literature review is comprehensive and details the timeframe within which the literature was selected. Systematic literature review can be divided into two categories: meta-analysis and meta-synthesis.

When you conduct meta-analysis you take findings from several studies on the same subject and analyze these using standardized statistical procedures. In meta-analysis patterns and relationships are detected and conclusions are drawn. Meta-analysis is associated with deductive research approach.

Meta-synthesis, on the other hand, is based on non-statistical techniques. This technique integrates, evaluates and interprets findings of multiple qualitative research studies. Meta-synthesis literature review is conducted usually when following inductive research approach.

Scoping literature review , as implied by its name is used to identify the scope or coverage of a body of literature on a given topic. It has been noted that “scoping reviews are useful for examining emerging evidence when it is still unclear what other, more specific questions can be posed and valuably addressed by a more precise systematic review.” [1] The main difference between systematic and scoping types of literature review is that, systematic literature review is conducted to find answer to more specific research questions, whereas scoping literature review is conducted to explore more general research question.

Argumentative literature review , as the name implies, examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. It should be noted that a potential for bias is a major shortcoming associated with argumentative literature review.

Integrative literature review reviews , critiques, and synthesizes secondary data about research topic in an integrated way such that new frameworks and perspectives on the topic are generated. If your research does not involve primary data collection and data analysis, then using integrative literature review will be your only option.

Theoretical literature review focuses on a pool of theory that has accumulated in regard to an issue, concept, theory, phenomena. Theoretical literature reviews play an instrumental role in establishing what theories already exist, the relationships between them, to what degree existing theories have been investigated, and to develop new hypotheses to be tested.

At the earlier parts of the literature review chapter, you need to specify the type of your literature review your chose and justify your choice. Your choice of a specific type of literature review should be based upon your research area, research problem and research methods.  Also, you can briefly discuss other most popular types of literature review mentioned above, to illustrate your awareness of them.

[1] Munn, A. et. al. (2018) “Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach” BMC Medical Research Methodology

  John Dudovskiy

Get science-backed answers as you write with Paperpal's Research feature

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 
• How to write a good literature review 
• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

• Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 
• Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 
• Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 
• Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 
• Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 
• Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

How to write a good literature review

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. 

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• Life Sciences Papers: 9 Tips for Authors Writing in Biological Sciences
• What is an Argumentative Essay? How to Write It (With Examples)

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, what is academic writing: tips for students, why traditional editorial process needs an upgrade, paperpal’s new ai research finder empowers authors to..., what is hedging in academic writing  , how to use ai to enhance your college..., ai + human expertise – a paradigm shift..., how to use paperpal to generate emails &..., ai in education: it’s time to change the..., is it ethical to use ai-generated abstracts without..., do plagiarism checkers detect ai content.

Rapid literature review: definition and methodology

Affiliations.

• 1 Assignity, Cracow, Poland.
• 2 Public Health Department, Aix-Marseille University, Marseille, France.
• 3 Studio Slowa, Wroclaw, Poland.
• 4 Clever-Access, Paris, France.
• PMID: 37533549
• PMCID: PMC10392303
• DOI: 10.1080/20016689.2023.2241234

Introduction: A rapid literature review (RLR) is an alternative to systematic literature review (SLR) that can speed up the analysis of newly published data. The objective was to identify and summarize available information regarding different approaches to defining RLR and the methodology applied to the conduct of such reviews. Methods: The Medline and EMBASE databases, as well as the grey literature, were searched using the set of keywords and their combination related to the targeted and rapid review, as well as design, approach, and methodology. Of the 3,898 records retrieved, 12 articles were included. Results: Specific definition of RLRs has only been developed in 2021. In terms of methodology, the RLR should be completed within shorter timeframes using simplified procedures in comparison to SLRs, while maintaining a similar level of transparency and minimizing bias. Inherent components of the RLR process should be a clear research question, search protocol, simplified process of study selection, data extraction, and quality assurance. Conclusions: There is a lack of consensus on the formal definition of the RLR and the best approaches to perform it. The evidence-based supporting methods are evolving, and more work is needed to define the most robust approaches.

Keywords: Delphi consensus; Rapid review; methodology; systematic literature review.

© 2023 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

Publication types

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 20 April 2024

Viral decisions: unmasking the impact of COVID-19 info and behavioral quirks on investment choices

• Wasim ul Rehman   ORCID: orcid.org/0000-0002-9927-2780 1 ,
• Omur Saltik 2 ,
• Faryal Jalil 3 &
• Suleyman Degirmen 4  

Humanities and Social Sciences Communications volume  11 , Article number:  524 ( 2024 ) Cite this article

Metrics details

This study aims to investigate the impact of behavioral biases on investment decisions and the moderating role of COVID-19 pandemic information sharing. Furthermore, it highlights the significance of considering cognitive biases and sociodemographic factors in analyzing investor behavior and in designing agent-based models for market simulation. The findings reveal that these behavioral factors significantly positively affect investment decisions, aligning with prior research. The agent-based model’s outcomes indicate that younger, less experienced agents are more prone to herding behavior and perform worse in the simulation compared to their older, higher-income counterparts. In conclusion, the results offer valuable insights into the influence of behavioral biases and the moderating role of COVID-19 pandemic information sharing on investment decisions. Investors can leverage these insights to devise effective strategies that foster rational decision-making during crises, such as the COVID-19 pandemic.

Similar content being viewed by others

The influence of upward social comparison on retail trading behaviour

Sandra Andraszewicz, Dániel Kaszás, … Christoph Hölscher

The unequal effects of the health–economy trade-off during the COVID-19 pandemic

Marco Pangallo, Alberto Aleta, … J. Doyne Farmer

Structural balance emerges and explains performance in risky decision-making

Omid Askarisichani, Jacqueline Ng Lane, … Brian Uzzi

Introduction

Coronavirus (COVID-19) is recognized as a significant health crisis that has adversely affected the well-being of global economies (Baker et al. 2020 ; Smales 2021 ; Debata et al. 2021 ). First identified in December 2019 as a highly fatal and contagious disease, it was declared a public health emergency by the World Health Organization (WHO) (WHO 2020 ; Baker et al. 2020 ; Altig et al. 2020 ; Smales 2021 ; Li et al. 2020 ). The outbreak swiftly spread across 31 provinces, municipalities, and autonomous regions in China, eventually evolving into a severe global pandemic that significantly impacted the global economy, particularly equity markets and social development (WHO 2020 ; Kazmi et al. 2020 ; Li et al. 2020 ). Since the early 2020 emergence of COVID-19 symptoms, the pandemic has caused considerable market decline and volatility in stock returns, significantly impacting the prosperity of world economies (Rahman et al. 2022 ; Soltani et al. 2021 ; Rubesam and Júnior 2022 ; Debata et al. 2021 ; Baker et al. 2020 ; Altig et al. 2020 ). This situation has garnered the attention of many policymakers and economists since its classification as a public health emergency.

Pakistan’s National Command and Operation Centre reported its first two confirmed COVID-19 cases on February 26, 2020. Following this, the Pakistan Stock Exchange experienced a significant downturn, losing 2266 points and erasing Rs. 436 billion in market equity. Foreign investment saw a notable decline, with stocks worth $22.5 million contracting sharply. By the end of February 2020, stock investments totaling $56.40 million had been liquidated. This dramatic drop in equity markets is attributed to the global outbreak of the COVID-19 pandemic (Khan et al. 2020 ). Additionally, for the first time in 75 years, Pakistan’s economy underwent its most substantial contraction in economic growth, recording a GDP growth rate of −0.4% in the first nine months. All three sectors of the economy—agriculture, services, and industry—fell short of their growth targets, culminating in a loss of one-third of their revenue. Exports declined by more than 50% due to the pandemic. Economists have raised concerns about a potential recession as the country grapples with virus containment efforts (Shafi et al. 2020 ; Naqvi 2020 ). Consequently, the rapid spread of COVID-19 has heightened volatility in financial markets, inflicted substantial losses on investors, and caused widespread turmoil in financial and liquidity markets globally (Zhang et al. 2020 ; Goodell 2020 ; Al-Awadhi et al. 2020 ; Ritika et al. 2023 ). This uncertainty has been exacerbated by an increasing number of positive COVID-19 cases.

Since the magnitude of the COVID-19 outbreak became evident, capital markets worldwide have been experiencing significant declines and volatility in stock returns, affected by all new virus variants despite their effective treatments (Hong et al. 2021 ; Rubesam and Júnior 2022 ; Zhang et al. 2020 ). Previous studies have characterized COVID-19 as a particularly devastating and deadly pandemic, severely impacting socio-economic infrastructures globally (Fernandes 2020 ). The pandemic has disrupted trade and investment activities, leading to imbalances in equity market returns (Xu 2021 ; Shehzad et al. 2020 ; Zaremba et al. 2020 ; Baig et al. 2021 ). In response to the COVID-19 outbreak, various governments, including Pakistan’s, have implemented unprecedented and diverse measures. These include restricting the mobility of the general public and commercial operations, and implementing smart or partial lockdowns, all aimed at mitigating the pandemic’s impact on global economic growth (Rubesam and Júnior 2022 ; Zaremba et al. 2020 ).

Investment decisions become notably complex and challenging when influenced by behavioral biases (Pompian 2012 ). In this context, numerous studies have sought to reconcile various behavioral finance theories with the notion of investors as rational decision-makers. One prominent theory is the Efficient Market Hypothesis, which asserts that capital markets are efficient when decisions are informed by symmetrical information among participants (Fama 1991 ). Yet, in reality, individual investors often struggle to make rational investment choices (Kim and Nofsinger 2008 ), as their decisions are significantly swayed by behavioral biases, leading to market inefficiencies. These biases, including investor sentiment, overconfidence, over/underreaction, and herding behavior, are recognized as widespread in human decision-making (Metawa et al. 2018 ). Prior research has identified various behavioral and psychological biases—such as loss aversion, anchoring, heuristic biases, and the disposition effect—that cause investors to stray from rational investment decisions. Moreover, investors’ responses to COVID-19-related news, like infection rates, vaccine developments, lockdowns, or economic forecasts, often reflect behavioral biases such as investor sentiment, overconfidence, over/underreaction, or herding behavior towards short-term events, thereby affecting market volatility (Soltani and Boujelbene 2023 ; Dash and Maitra 2022 ). These biases may have a wide applicability across different markets, regardless of specific cultural or regulatory differences. Consequently, we posit that these four behavioral biases, in the context of COVID-19, are key factors in reducing vulnerability in investment decisions (Dermawan and Trisnawati 2023 ), especially for individual investors who are more susceptible than in a typical investment environment (Botzen et al. 2021 ; Talwar et al. 2021 ). Therefore, understanding these behavioral biases—such as investor sentiment, overconfidence, over/underreaction, or herding behavior—during the COVID-19 pandemic is crucial, as no previous epidemic has demonstrated such profound impacts of behavioral biases on investment decisions (Baker et al. 2020 ; Sattar et al. 2020 ).

Numerous studies have explored the impact of behavioral biases, including investor sentiment, overconfidence, over/under-reaction, and herding behavior, on investment decisions (Metawa et al. 2018 ; Menike et al. 2015 ; Nofsinger and Varma 2014 ; Qadri and Shabbir 2014 ; Asaad 2012 ; Kengatharan and Kengatharan 2014 ). Recent literature has also shed light on the effects of the COVID-19 pandemic on financial and precious commodity markets (Gao et al. 2023 ; Zhang et al. 2020 ; Corbet et al. 2020 ; Baker et al. 2020 ; Mumtaz and Ahmad 2020 ; Ahmed et al. 2022 ; Hamidon and Kehelwalatenna 2020 ). However, academic research specifically addressing the moderating role of COVID-19 pandemic information sharing on behavioral biases remains limited. It has been observed that global pandemics, such as the Ebola Virus Disease (EVD) and Severe Acute Respiratory Syndrome (SARS), significantly influence stock market dynamics, sparking widespread fear among investors and leading to market uncertainty (Del Giudice and Paltrinieri 2017 ; He et al. 2020 ). This study contributes to the field by examining how behavioral biases, such as investor sentiment, overconfidence, over/under-reaction, and herding behavior, are influenced by the unique circumstances of the COVID-19 crisis. Furthermore, this research provides novel insights into real-time investor behavior and policymaking, thus advancing the academic debate on the role of COVID-19 pandemic information sharing within behavioral finance.

The primary goal of this study is to explore the impact of the COVID-19 crisis on behavioral biases and their effect on investment decisions. Additionally, it aims to assess how various socio-demographic factors influence investment decision-making. These factors include age, occupation, gender, educational qualifications, type of investor, investment objectives, reasons for investing, preferred investment duration, and considerations prior to investing, such as the safety of the principal, risk level, expected returns, maturity period, and sources of investment advice. We hypothesize that these factors significantly influence investment decisions, and our analysis endeavors to investigate the relationship between these factors and investment behavior. By thoroughly examining these variables, the study aims to shed light on the role socio-demographic factors play in investment behavior and enhance the understanding of the investment decision-making process. Additionally, the study seeks to conduct a cluster analysis to identify hierarchical relationships and causality, alongside an agent-based learning model that illustrates the susceptibility of low-income and younger age groups to herding behavior. The article provides the codes and outcomes of the model.

The study will commence with an introduction that outlines the scope and significance of the research. Following this, a literature review will be provided, along with the development of hypotheses concerning the behavioral biases affecting investment decisions and the role of socio-demographic factors in shaping investment behavior. The methodology section will detail the research approach, data collection process, variables considered for analysis, and the statistical methods applied. Subsequently, the results section will present findings from the regression and moderating analyses, cluster analysis, and the agent-based learning model. This will include a detailed explanation of the model codes and their interpretations. The discussion section will interpret the study’s results, highlighting their relevance to policymakers, financial advisors, and individual investors. The article will conclude by summarizing the main discoveries and offering suggestions for further inquiry in this domain.

Literature review and development of hypotheses

Invsetor sentiments and investment decisions.

Pandemic-driven sentiments play a crucial role in determining market returns, making it imperative to understand pandemic-related sentiments to predict future investor returns. Consequently, we posit that the sharing of COVID-19 pandemic information is a critical factor influencing investor sentiments towards investment decisions (Li et al. 2021 ; Anusakumar et al. 2017 ; Zhu and Niu 2016 ; Jiang et al. 2021 ). Generally, investors’ sentiments refer to their beliefs, anticipations, and outlooks regarding future cash flows, which are significantly influenced by external factors (Baker and Wurgler 2006 ). Ding et al. ( 2021 ) define investor sentiment as the collective attitude of investors towards a particular market or security, reflected in trading activities and price movements of securities. A trend of rising prices signals bullish sentiments, while decreasing prices indicate bearish investor sentiment. These sentiments, including emotions and beliefs about investment risks, notably affect investors’ behavior and yield (Baker and Wurgler 2006 ; Anusakumar et al. 2017 ; Jansen and Nahuis 2003 ). Sentiment reacts to stock price news (Mian and Sankaraguruswamy 2012 ), with stock prices responding more positively to favorable earnings news during periods of high sentiment than in low sentiment periods, and vice versa. This sentiment-driven reaction to share price movements is observed across all types of stocks (Mian and Sankaraguruswamy 2012 ). Furthermore, research indicates that market responses to earnings announcements are asymmetrical, especially in the context of pessimistic investor sentiments (Jiang et al. 2019 ). Such reactions were notably pronounced during COVID-19 pandemic news, where sentiments such as fear, greed, or optimism significantly influenced market dynamics (Jiang et al. 2021 ). Thus, information related to the COVID-19 pandemic emerges as a valuable resource for forecasting future returns and market volatility, ultimately affecting investment decision-making (Debata et al. 2021 ).

Overconfidence and investment decision

Standard finance theories suggest that investors aim for rational decision-making (Statman et al. 2006 ). However, their judgments are often swayed by personal sentiments or cognitive errors, leading to overconfidence (Apergis and Apergis 2021 ). Overconfidence in investing can be described as an inflated belief in one’s financial insight and decision-making capabilities (Pikulina et al. 2017 ; Lichtenstein and Fischhoff 1977 ), or a tendency to overvalue one’s skills and knowledge (Dittrich et al. 2005 ). This results in investors perceiving themselves as more knowledgeable than they are (Moore and Healy 2008 ; Pikulina et al. 2017 ).

Overconfidence has been categorized into overestimation, where investors believe their abilities and chances of success are higher than actual, and over-placement, where individuals see themselves as superior to others (Moore and Healy 2008 ). Such overconfidence affects investment choices, leading to potentially inappropriate high-risk investments (Pikulina et al. 2017 ). Overconfident investors often attribute success to personal abilities and failures to external factors (Barber and Odean 2000 ; Tariq and Ullah 2013 ). Overconfidence also leads to suboptimal decision-making, especially under uncertainty (Dittrich et al. 2005 ).

Behavioral finance research shows that individual investors tend to overestimate their chances of success and underestimate risks (Wei et al. 2011 ; Dittrich et al. 2005 ). Excessive overconfidence prompts over-investment, whereas insufficient confidence causes under-investment; moderate confidence, however, leads to more prudent investing (Pikulina et al. 2017 ). The lack of market information often triggers this scenario (Wang 2001 ). Amidst recent market anomalies, COVID-19 information has significantly impacted investors’ overconfidence in their investment decisions. Studies have shown that overconfident investors underestimate their personal risk of COVID-19 compared to the general risk perception (Bottemanne et al. 2020 ; Heimer et al. 2020 ; Boruchowicz and Lopez Boo 2022 ; Druica et al. 2020 ; Raude et al. 2020 ). Overconfidence may lead to adverse selection and undervaluing others’ actions, underestimating the likelihood of loss due to inadequate COVID-19 information (Hossain and Siddiqua 2022 ). Consequently, this study hypothesizes that certain exogenous factors, integral to COVID-19 information sharing, may moderate investment decisions in the context of investor overconfidence.

Over/under reaction and investment decision

The Efficient Market Hypothesis (EMH) suggests that investors’ attempts to act rationally are based on the availability of market information (Fama 1998 ; Fama et al. 1969 ; De Bondt 2000 ). However, psychological biases in investors systematically respond to unwelcome news, leading to overreaction and underreaction, thus challenging the notion of market efficiency (Maher and Parikh 2011 ; De Bondt and Thaler 1985 ). Overreaction and underreaction biases refer to exaggerated responses to recent market news, resulting in the overbuying or overselling of securities in financial markets (Durand et al. 2021 ; Spyrou et al. 2007 ). Barberis et al. ( 1998 ) identified both underreaction and overreaction as pervasive anomalies that drive investors toward irrational investment decisions. Similarly, Hirshleifer ( 2001 ) noted that noisy trading contributes to overreaction, which in turn leads to excessive market volatility.

The impact of the COVID-19 outbreak extends far beyond the loss of millions of lives, disrupting financial markets from every angle (Zhang et al. 2020 ; Iqbal and Bilal 2021 ; Tauni et al. 2020 ; Borgards et al. 2021 ). Market reactions have been significantly shaped by COVID-19 pandemic information sharing, affecting investors’ decisions (Kannadas 2021 ). Recent studies have found that investors’ biases in evaluating the precision and predictive accuracy of COVID-19 information can lead to overreactions and underreactions (Borgards et al. 2021 ; Xu et al. 2022 ; Kannadas 2021 ). Furthermore, research documents the growing influence of COVID-19 information sharing on market reactions worldwide, including in the US, Asian, European, and Australian markets (Xu et al. 2022 ; Nguyen et al. 2020 ; Nguyen and Hoang Dinh 2021 ; Naidu and Ranjeeni 2021 ; Heyden and Heyden 2021 ), indicating that market reactions, characterized by non-linear behavior, are driven by investors’ beliefs.

Previous literature has scarcely explored the role of investors’ overreaction and underreaction in decision-making. Recently, emerging research has begun to enrich the literature by examining the moderating role of COVID-19 pandemic information sharing.

Herding behavior and investment decision

According to the assumptions of Efficient Market Hypothesis (EMH), optimal decision-making is facilitated by the availability of market information and stability of stock returns (Fama 1970 ; Raza et al. 2023 ). However, these conditions are seldom met in reality, as decisions are influenced by human behavior shaped by socio-economic norms (Summers 1986 ; Shiller 1989 ). Behavioral finance research suggests that herding behavior plays a significant role in the decline of asset and stock prices, implying that identifying herding can aid investors in making more rational decisions (Bharti and Kumar 2022 ; Jiang et al. 2022 ; Jiang and Verardo 2018 ; Ali 2022 ). Bikhchandani and Sharma ( 2000 ) define herding as investors’ tendency to mimic others’ trading behaviors, often ignoring their own information. It is essentially a group dynamic where decisions are irrationally based on others’ information, overlooking personal insights, experiences, or beliefs (Bikhchandani and Sharma 2000 ; Huang and Wang 2017 ). Echoing this, Hirshleifer and Hong Teoh ( 2003 ) argue that herding is characterized by investment decisions being influenced by the actions of others.

The sharp market declines prompted by events such as the COVID-19 pandemic raise questions about its influence on investors’ herding behaviors (Rubesam and Júnior 2022 ; Mandaci and Cagli 2022 ; Espinosa-Méndez and Arias 2021 ). Christie and Huang ( 1995 ) observed that investor herding becomes more evident during market uncertainties. Hwang and Salmon ( 2004 ) noted that investors are less likely to exhibit herding during crises compared to stable market periods when confidence in future market prospects is higher. The COVID-19 pandemic, as a major market disruptor, necessitates that investors pay close attention to market fundamentals before making investment decisions. Recent studies suggest that an overload of COVID-19 information could lead to irrational decision-making, potentially challenging the EMH by influencing herding behavior (Jiang et al. 2022 ; Mandaci and Cagli 2022 ). This highlights the importance for investors to be aware of market information asymmetry changes, such as those triggered by the COVID-19 outbreak, which could negatively impact their investment portfolios by altering their herding tendencies. This effect may be more pronounced among individual investors than institutional ones (Metawa et al. 2018 ). A yet unexplored area is the extent to which COVID-19 pandemic information sharing amplifies the herding behavior among investors during investment decision-making processes (Mandaci and Cagli 2022 ).

COVID-19 pandemic information sharing moderating the relationship between behavioral biases and investment decisions

Recent research indicates that the COVID-19 pandemic has notably influenced behavioral biases among investors, affecting their decision-making processes (Betthäuser et al. 2023 ; Vasileiou 2020 ). Since the pandemic’s onset, investors have shown increased sensitivity to pandemic-related news or developments, leading to intensified behavioral biases. This heightened sensitivity poses challenges to investors’ abilities to respond effectively. Specifically, information related to economic uncertainty, infection rates, and vaccination progress has shifted investor sentiment regarding risk perception (Gao et al. 2023 ). Additionally, pandemic news has altered the risk perception of overconfident investors, who previously may have underestimated the risks associated with COVID-19 (Bouteska et al. 2023 ). The increased uncertainty and market volatility triggered by COVID-19 news have also prompted investors to adapt their reactions based on new information, potentially fostering more rational decision-making (Jiang et al. 2022 ). The rapid spread of COVID-19-related news has been shown to diminish mimicry in investment decisions (Nguyen et al. 2023 ). This indicates that viral news about the pandemic makes investors more discerning regarding risk perceptions and investment strategies, moving away from mere herd behavior. Based on this discussion, the study proposes that COVID-19 pandemic information sharing acts as a moderating factor in the relationship between behavioral biases and investment decisions.

Sociodemographic factors and investment decision

The influence of demographic factors like gender, age, income, and marital status on investor behavior is well-documented in financial literature. However, examining these relationships within specific geographical contexts—such as countries, regions, states, and provinces—reveals that cultural values, beliefs, and experiences may blur the distinctions between human and cognitive biases in terms of their nuanced impacts. Evidence shows that certain demographic groups, particularly young male investors with lower portfolio values from regions less developed in terms of education and income, are more prone to overconfidence and familiarity bias in their trading activities. Conversely, investors with higher education levels and female investors are inclined to trade less frequently, resulting in better investment returns (Barber and Odean 2000 ; Gervais and Odean 2001 ; Glaser and Weber 2007 ).

This study’s findings further suggest that with increased stock market experience, investors tend to discount emotional factors, leading to more rational investment choices. Nonetheless, experience alone does not appear to markedly influence the decision-making process among investors (Al-Hilu et al. 2017 ; Metawa et al. 2019 ).

In summary, demographic variables such as age, gender, and education significantly impact investment decisions, especially when considered alongside behavioral aspects like investor sentiment, overconfidence, and herd behavior. Gaining insight into these dynamics is crucial for investors, financial advisors, and policymakers to devise effective investment strategies and enhance financial literacy.

Research methodology

Data and sampling.

The research methodology outlines the strategy for achieving the study’s objectives. This research adopted a quantitative approach, utilizing a survey method (questionnaire) to examine the behavioral biases of individual investors in Pakistan during the COVID-19 pandemic. The target population comprised individual investors from Punjab province, specifically those interested in capital investments. Data were collected through convenient sampling techniques. A total of 750 questionnaires were distributed via an online survey (Google Form) to investors in four major cities of Punjab province: Karachi, Lahore, Islamabad, and Faisalabad. Initially, 257 respondents completed the survey following follow-up reminder emails. Out of these, 223 responses were deemed usable, yielding a valid response rate of 29.73% for further analysis (Saunders et al. 2012 ).

To mitigate potential biases during the data collection process, we conducted analyses for non-response and common method biases. Non-response bias, which arises when there is a significant difference between early and late respondents in a survey, was addressed by comparing the mean scores of early and late respondents using the independent samples t -test (Armstrong and Overton 1977 ). Results (see Table 1 ) indicated no statistically significant ( p  > 0.05) difference between early and late responses, suggesting that response bias was not a significant issue in the dataset.

Furthermore, to assess the potential threat of common method variance, we applied Harman’s single-factor test, a widely used method to evaluate common method biases in datasets (Podsakoff et al. 2003 ). This technique is aimed at identifying systematic biases that could compromise the validity of the scale. Through exploratory factor analysis (EFA) conducted without rotation, it was determined that no single factor accounted for a variance greater than the threshold (i.e., 50%). Consequently, common method variance was not considered a problem in the dataset, ensuring the reliability of the findings.

Figure 1 illustrates the framework of the model established for regression and moderating analyses that reveal the interactions between behavioral biases, investment decisions and COVID-19 pandemic information sharing.

Covid-19 pandemic informing sharing.

Measures for behavioral biases

A close-ended questionnaire based on five-point Likert measurement scales was prepared scaling (1= “strongly disagree” to 5= “strongly agree”) to operationalize the behavioral biases of investors. The first predictor is investor sentiments. It refers to investors’ beliefs and perspectives related to future cash flows or discourses of specific assets. It is a crucial behavioral factor that often drives the market movements, especially during pandemic. We used the modified 5-items scale from the study of (Metawa et al. 2018 ; Baker and Wurgler 2006 ). Second important behavioral factor is overconfidence, which measured the tendency of decision-makers to unwittingly give excessive weight to the judgment of knowledge and correctness of information possessed and ignore the public information (Lichtenstein and Fischhoff 1977 ; Metawa et al. 2018 ). This construct was measured by using the 3-items scale developed by Dittrich et al. ( 2005 ). In line with the studies of (see for example (De Bondt and Thaler 1985 ; Metawa et al. 2018 ), we opted the 4-items scale to measure the over/under reactions. It illustrates that investors systematically overreact to unexpected news, and this leads to the violation of market efficiency. They conclude that investors attach great importance to past performance, ignoring trends back to the average of that performance (Boubaker et al. 2014 ). Last, herding behavior effect means theoretical set-up suggesting that investment managers are imitating the strategy of others despite having exclusive information. Such managers prefer to make decisions according to the connected group to avoid the risk of reputational damage (Scharfstein and Stein 1990 ). In sense, a modified scale was anchored to examine the herd behavior of investors from the studies of Bikhchandani and Sharma ( 2000 ) and Metawa et al. ( 2018 ).

Measures for COVID-19 pandemic information sharing

To assess the moderating effect of COVID-19 pandemic information sharing, it was examined in terms of uncertainty, fear, and perceived risk associated with the virus (Kiruba and Vasantha 2021 ). Previous studies indicate that COVID-19 news and developments have markedly affected the behavioral biases of investors (Jiang et al. 2022 ; Nguyen et al. 2023 ). To this end, an initial scale was developed to measure the moderating effect of COVID-19 pandemic information sharing. The primary reason for creating a new scale was that existing scales lacked clarity and were not specifically designed to assess how anchoring behavioral biases affect investment decisions. Subsequently, a self-developed scale was refined with input from a panel of experts, including two academicians specializing in neuro or behavioral finance and two investors with expertise in the capital market, to ensure the scale’s face and content validity regarding COVID-19 pandemic information sharing. They reviewed the scale in terms of format, content, and wording. Based on their comprehensive review, minor modifications were made, particularly aligning the scale with pandemic news and developments to accurately measure the impact of the COVID-19 health crisis on investors’ behavioral biases. Ultimately, a four-item scale, employing a five-point Likert scale (1= “strongly disagree” to 5= “strongly agree”), focusing on COVID-19 related aspects (e.g., infection rates, lockdowns, vaccine development, and government stimulus packages) was utilized to operationalize the construct of COVID-19 pandemic information sharing (Bin-Nashwan and Muneeza 2023 ; Li and Cao 2021 ).

I believe that increasing information about rate of COVID-19 infections influenced my investment decisions.

I believe that increasing information about COVID-19 lockdowns influenced my investment decisions.

I believe that increasing information about COVID-19 vaccinations development, influenced my investment decisions, and

I believe that increasing information about government stimulus packages influenced my investment decisions.

Measures for investment decisions

To measure investment decision, the modified five points Likert scale ranging from (1= “strongly disagree” to 5= “strongly agree”) has been opted from the study of Metawa et al. ( 2018 ).

Hypotheses of study

The hypotheses of the study regarding regression analysis and moderating analyses are as follows in Table 2 :

The hypotheses outlined above were tested using regression analyses and moderating analyses. To reveal the clustering tendencies of investors exhibiting similar behaviors, cognitive biases, and sociodemographic variables, the feature importance values were investigated using K-means clustering analyses. Furthermore, findings and recommendations were provided to policymakers using agent-based models to develop policy suggestions within the scope of these hypotheses, offering insights for academic purposes.

Demographic profile of respondents

Table 3 provides a brief demographic profile of respondents.

Based on the percentages presented in Table 3 , the study primarily focuses on a specific demographic profile. Most participants were 20–30 years old (61.0%) with a higher educational background, particularly a master’s degree (67.3%). They were mostly salaried individuals (56.5%), male (61.0%), and identified as seasonal investors (63.7%). The investment objective of this group was mostly focused on growth and income (37.2%), while wealth creation (41.3%) was their primary purpose for investing. They preferred to invest equally in medium-term (43.5%) and long-term (28.3%) periods and considered high returns (38.6%) as the primary factor before investing. They received investment advice primarily from family and friends (44.8%) and social media (29.6%). Overall, the study indicates that the sample consisted of younger, male, salaried individuals with higher education levels who rely on personal networks and social media for investment advice. Their investment objectives are focused on wealth creation through growth and income, with an equal preference for medium and long-term investments.

Analysis and results

Descriptive summary.

Table 4 outlines the measures used to evaluate the constructs of the study, detailing the number of items for each construct, mean values, standard deviations, zero-order bivariate correlations among the variables, and Cronbach’s Alpha values. The evaluation encompasses a total of 29 items spread across six constructs: investor sentiments (5 items), overconfidence (3 items), over/under reaction (4 items), herding theory (3 items), investment decision (10 items), and COVID-19 information impact (4 items). The mean scores for these items fall between 3.535 and 3.779, with standard deviations ranging from 0.877 to 0.965.

Parallel coordinates (see Figs. 2 – 5 ) visualization is employed as a method to depict high-dimensional data on a two-dimensional plane, proving particularly beneficial for datasets with a large number of features or attributes. This technique involves the use of vertical axes to represent each feature, connected by horizontal lines that represent individual data points. This visualization method facilitates the identification of patterns, detection of clusters or outliers, and discovery of correlations among the features. Therefore, parallel coordinates visualization is instrumental in analyzing complex datasets, aiding in the informed decision-making process based on the insights obtained.

Strongly disagree (CIS1) choice parallel coordinates.

Disagree (CIS2) choice parallel coordinates.

Agree (CIS3) choice parallel coordinates.

Strongly agree (CIS4) choice parallel coordinates.

The analysis of responses to the COVID-19 information sharing questions reveals a significant correlation with the second and fourth-level responses concerning cognitive biases, including investor sentiment, overconfidence, over/under reaction, and herding behavior. This observation leads to two key insights. Firstly, participants demonstrate an ability to perceive, respond to, and comprehend the nuances of their investment decisions as related to investor sentiment, overconfidence, over/under reaction, and herding behavior. Consequently, they show a propensity to make clear decisions, indicating agreement or disagreement in their responses. Secondly, it is noted that individuals who acknowledge being significantly influenced by COVID-19 news tend to adopt more balanced investment strategies concerning these cognitive biases. Additionally, younger individuals, particularly those self-employed or not professionally investing, who show a preference for long-term value investments, are more inclined to exhibit these tendencies.

The value of the Pearson correlation coefficient (r) was calculated to investigate the nature, strength and relationship between variables. The results of correlation analysis reveal that all the constructs positively correlated.

To investigate the interconnections among variables in the dataset, correlations were computed and illustrated through a network graph. The correlation matrix’s values served as the basis for edge weights in the graph, with more robust correlations depicted by thicker lines (see Fig. 6a ). Each variable received a unique color, and connections showcasing higher correlations utilized a distinct color scheme to enhance visual clarity. This method offers a graphical depiction of the intricate relationships among various variables, facilitating the discovery of patterns and insights that might remain obscured within a conventional correlation matrix.

a Correlation diagraphs and matrix. b Correlation diagraphs and matrix.

The correlation analysis revealed a pronounced relationship between cognitive biases (such as investor sentiments, overconfidence, herd behavior, and investment decisions), COVID-19 information sharing, and socio-demographic factors (including age group, occupation, gender, educational qualifications, type of investor, investment objectives, investment purposes, preferred investment duration, factors considered prior to investing, and sources of investment advice). A correlation matrix graph was constructed to further elucidate these correlations, assigning different colors to each variable for visual differentiation (see Fig. 6b ). The thickness of the lines in the graph correlates with the strength of the relationships, indicating variables with high correlation more prominently.

These findings underscore the interconnected nature of the study variables, demonstrating that cognitive biases and socio-demographic factors exert a considerable impact on investment decisions. This analytical approach highlights the complexity of investor behavior and underscores the multifaceted influences on investment choices, providing valuable insights for understanding how various factors interact within the investment decision-making process.

Reliability test

For reliability test, the Cronbach alpha values were examined to check the internal consistency of the measure. The internal consistency of an instrument tends to indicate whether a metric or an indicator measure what it is intended to measure (Creswell 2009 ). The Cronbach’s alpha greater than 0.7 indicates that all the items or the questions regarding the respective variable are good, highly correlated and reliable. The calculated Cronbach coefficient value for Investor sentiments (alpha = 0.888), over confidence (alpha = 0.827), over/under reaction (alpha = 0.858), herding behavior theory (alpha = 0.741), Investment decision (alpha = 0.933) and COVID-19 (alpha = 0.782) indicates that all of the constructs are reliable.

Validity test

Validity refers to the extent to which an instrument accurately measures or performs what it is designed to measure (Kothari 2004 ). To ensure the validity of the questionnaire and its constructs, the researcher engaged in a comprehensive literature review, sought the advice of consultants, and incorporated feedback from other professionals in the field. Additionally, the concepts of convergent validity and discriminant validity were evaluated to further assess the instrument’s validity.

Convergent validity assesses the extent to which items that are theoretically related to a single construct are, in fact, related in practice (Wang et al. 2017 ). To determine convergent validity, factor loading, Average Variance Extracted (AVE), and Composite Reliability (CR) were calculated. According to Hair et al. ( 1998 ), factor loading values should exceed 0.60, composite reliability should be 0.70 or higher, and AVE should surpass 0.50 to confirm adequate convergent validity.

Table 5 demonstrates that all constructs utilized in this study surpass these threshold values, indicating strong convergent validity. This suggests that the items within each construct are consistently measuring the same underlying structure, reinforcing the validity of the questionnaire’s design and the constructs it aims to measure.

Discriminant validity measures the degree that the concepts are distinct from each other (Bagozzi et al. 1991 ) and it is evident that if alpha value of a construct is greater than the average correlation of the construct with other variables in model, the existence of discriminant validity exist (Ghiselli et al. 1981 ).

Hypotheses testing

To examine the conditional moderating effect of COVID-19 on the influence of behavioral factors (investor sentiments, overconfidence, over/under reaction, and herding behavior) on investment decision-making, moderation analysis was conducted using the Process Macro (Model 1) for SPSS, as developed by Hayes, with bootstrapping samples at 95% confidence intervals. According to Hayes ( 2018 ), the analysis first explores the direct impact of the behavioral factors on investment decisions. Subsequently, it assesses the indirect influence exerted by the moderating variable (COVID-19). This two-step approach allows for a comprehensive understanding of how COVID-19 modifies the relationship between investors’ behavioral biases and their decision-making processes, shedding light on the extent to which the pandemic acts as a moderating factor in these dynamics.

For this study the mathematical model to test moderating role of COVID-19 pandemic information sharing can be explained as:

Y = Investment decisions (Dependent variable)

β 0  = Intercept

X 1  = Investment sentiments (Independent variable)

X 2  = Overconfidence (Independent variable)

X 3  = Over/under reaction (Independent variable)

X 4  = Herding behavior (Independent variable)

β 1 X 1  = Intercept of investors sentiments

β 2 X 2  = Intercept of overconfidence

β 3 X 3  = Intercept of over/under reaction

β 4 X 4  = Intercept of herding behavior

(X 1 * COVID-19) = Investors’ sentiments and moderation effect of COVID-19 information

(X 2 * COVID-19) = Overconfidence and moderation effect of COVID-19 information

(X 3 * COVID-19) = Over/under reaction and moderation effect of COVID-19 information

(X 4 * COVID-19) = Herding behavior and moderation effect of COVID-19 information

μ = Residual term.

Direct effect

In Table 6 , the direct effect of the independent variables on the dependent variable demonstrates that the behavioral factors (investor sentiments, overconfidence, over/under reaction, and herding behavior) significantly influence investment decision (ID) with beta values of 0.961, 0.867, 0.884, and 0.698, respectively. The confidence interval (CI) values presented in Table 6 confirm these relationships are statistically significant. The positive and significant outcomes underline that behavioral factors critically impact investors’ decision-making attitudes. Consequently, Hypotheses 1, 2, 3, and 4 (H1, H2, H3, and H4) are accepted, affirming the substantial role of investor sentiments, overconfidence, over/under reaction, and herding behavior in shaping investment decisions.

Indirect moderating effect

In the context of the COVID-19 pandemic and its associated risks, the impact of behavioral factors (investor sentiments, overconfidence, over/under reaction, and herding behavior) on investment decisions tends to diminish. The findings presented in Table 6 and illustrated in Fig. 7 indicate that COVID-19 information sharing significantly and negatively moderates the relationship between these factors and investment decisions, leading to the acceptance of Hypotheses 5, 6, 7, and 8 (H5, H6, H7, and H8). The negative beta values underscore that the presence of COVID-19 adversely influences investors’ behavior, steering them away from rational investment decisions. This demonstrates that the pandemic context acts as a moderating factor, altering how behavioral biases impact investment choices, ultimately guiding investors towards more cautious or altered decision-making processes.

Moderating effect of Covid-19 pandemic information sharing.

K-means clustering analysis

K-means clustering analysis is utilized to uncover natural groupings within datasets by analyzing similarities between observations. This technique is especially beneficial for managing large and complex datasets as it reveals patterns and relationships among variables that may not be immediately evident. In this study, K-means clustering helps identify natural groupings based on socio-demographic factors, cognitive biases regarding investment decisions, and COVID-19 pandemic information sharing, thereby offering insights into the data’s underlying structure and identifying potential patterns or relationships among key variables.

The cluster analysis aims to ascertain the feature importance value of groups with similar investor behaviors, which is crucial for determining agents’ investment functions in subsequent agent-based modeling. Selecting the appropriate number of clusters in the K-means algorithm is essential, yet challenging, as different numbers of clusters can yield varying results (Li and Wu 2012 ).

Two prevalent methods for determining the optimal number of clusters are:

Elbow Method: This approach involves running the K-means algorithm with varying cluster numbers and calculating the total sum of squared errors (SSE) for each. SSE represents the squared distances of each data point from its cluster’s centroid. Plotting the SSE values against the number of clusters reveals a point known as the “elbow,” where the rate of SSE decrease markedly slows, indicating the optimal cluster number (Syakur et al. 2018 ).

Silhouette Analysis: Not mentioned directly in the narrative, but it’s another method that measures how similar an object is to its own cluster compared to other clusters. The silhouette score ranges from −1 to 1, where a high value indicates the object is well matched to its own cluster and poorly matched to neighboring clusters.

The sklearn library provides tools for implementing the elbow method and silhouette analysis. For example, the code snippet described applies the elbow method by varying the number of clusters from 1 to 10 and calculating SSE for each scenario. The optimal number of clusters is identified by selecting a value near the elbow point on the resulting plot.

After clustering, the analysis progresses by using the fit () method from sklearn’s K-Means class to cluster the data, determine each cluster’s center coordinates, and assign each data point to a cluster. Feature importance values can be calculated using the Extra Trees Classifier class from sklearn, and these values can be visualized through a line graph.

Finally, to illustrate the clusters’ membership to the CIS1, CIS2, CIS3, and CIS4 inputs as a color scale bar, the seaborn library is used (see Fig. 8 (top) and Fig. 8 (bottom)). This involves calculating the average membership values for each cluster and visualizing these averages, providing a clear depiction of how each cluster associates with the different inputs, enriching the analysis of investor behaviors and their responses to COVID-19 information sharing.

Elbow method sum of squared error class determination (top) and clustering analysis results (bottom).

After employing a network diagram constructed from a correlation matrix to elucidate the interrelationships among variables, and utilizing the Elbow method to ascertain the optimal number of clusters, the K-means clustering algorithm was applied (see Fig. 9 ). This approach successfully identified three distinct clusters, highlighting the variables that exerted a significant influence on these clusters. Notably, the COVID-19 pandemic information sharing variable, along with its corresponding CIS1, CIS2, CIS3, and CIS4 values, emerged as significant factors. The analysis indicated that overconfidence and overreaction were the predominant factors in crucial clustering, alongside cognitive biases and investment strategies that lead to similar behaviors among investors and varying levels of impact from COVID-19.

Cluster analysis feature importance value results.

Furthermore, sociodemographic factors such as age, occupation, and investor type were also identified as influential determinants. Leveraging these insights, policymakers and researchers can develop an agent-based model that incorporates herd behavior, along with age and income levels categorized by occupation, to effectively simulate market dynamics. This approach facilitates a comprehensive understanding of how different factors, particularly those related to the COVID-19 pandemic, influence investor behavior and market movements, thereby enabling the formulation of more informed strategies and policies.

An ingenious agent-based simulation for herding behavior

In this study, the findings of behavioral economics and finance research may contain results that are easy to interpret for policymakers but may involve certain difficulties in practical implementation. Specifically, for policymakers, an agent-based model has been created (see Appendix 1 for pseudo codes. In case, requested python codes are available). In a model consisting of 223 agents who trade on a single stock, prototypes of investors have been created based on the analysis presented here, and characteristics such as age group and income status, which are relatively easy to access or predict regarding their socio-demographic profiles, have been taken into account in the herd behavior function, considering the decision to follow the group or make independent decisions. Younger and lower-income agents were allowed to exhibit a greater tendency to follow the group, while 50 successful transactions were monitored to determine in which trend of stock price increase or decrease the balance of the most successful agent was increased or decreased (Gervais and Odean 2001 ).

In addressing the influence of age and income status on herding behavior, it is imperative to underscore the nuanced interplay between various socio-economic and psychological factors within our agent-based model framework. The model’s robustness stems from its capacity to simulate a range of investor behaviors by integrating key determinants such as investor sentiment, overconfidence, reaction to market events, and socio-demographic characteristics. Herein we expound on the contributory elements:

Investor Sentiment (IS1–IS5)

The model encapsulates the variability of investor sentiment, which oscillates with age and income, influencing individuals’ financial perspectives and risk propensities. Younger investors’ sentiment may tilt towards optimism driven by a more extensive investment horizon, while lower-income investors’ sentiment could lean towards caution, primarily driven by the pressing requirement for financial dsecurity (Baker and Wurgler 2007 ).

Overconfidence (OF1–OF5)

The tendency towards overconfidence is dynamically modeled, particularly among younger investors who may overrate their market acumen and predictive capabilities. This overconfidence may also manifest among lower-income investors as a psychological compensatory mechanism for resource inadequacy (Malmendier and Tate 2005 ).

Over/Under Reaction (OUR1–OUR5)

The model accounts for the influence of age and income on the velocity and extent of response to market stimuli. Inexperienced or financially restricted investors may be prone to overreactions due to a lack of market exposure or intensified economic strain (Daniel et al. 1998 ).

Herding Behavior (HB1–HB4)

Within the simulated environment, herding is more pronounced among younger investors, possibly due to peer influence, and among lower-income investors who may seek safety in conformity (Bikhchandani et al. 1992 ).

Investment Decision (ID1–ID10)

The model intricately reflects the complexities of investment decisions influenced by age-specific factors such as projected earnings and lifecycle influences. Investors with limited income may exhibit a predilection for security, swaying their investment choices (Yao and Curl 2011 ).

COVID-19 Information Sharing (CIS1–CIS4)

The pandemic era’s nuances are integrated into the model, acknowledging that younger investors could be more susceptible to digitally disseminated information, which, in turn, impacts their investment decisions. The credibility and source of information are also calibrated based on income levels (Shiller 2020 ).

Socio-demographic factors

Age: The model simulates younger investors’ reliance on the conduct of others, utilizing it as a heuristic substitute for experience (Dobni and Racine 2016 ).

Occupation: It captures how occupational background can broaden or restrict access to information and influence herding tendencies (Hong et al. 2000 ).

Gender: Gender disparities are incorporated, reflecting on investment styles where men may be more disposed to herding due to overconfidence (Barber and Odean 2001 ).

Qualification (Qualif.): The model acknowledges that higher education and financial literacy levels can curtail herding by fostering self-reliant decision-making (Lusardi and Mitchell 2007 ).

Investor Type (InvTyp): It differentiates between retail and institutional investors, noting that limited resources might push retail investors towards herding (Nofsinger and Sias 1999 ).

Investment Objective (InvObj): The model recognizes that short-term objectives might amplify herding as investors chase swift gains (Odean 1998 ).

Purpose: It contemplates the conservative herding behavior that is aligned with goals like retirement savings (Yao and Curl 2011 ).

Investment Horizon (Horizon): A lengthier investment horizon is modeled to potentially dampen herding tendencies (Kaustia and Knüpfer 2008 ).

Factors Considered Before Investing (factors): The model simulates a range of investment considerations, including risk tolerance and expected returns, which influence herding propensities (Shefrin and Statman 2000 ).

Source of Investment Advice (source): The influence of advice sources, such as analysts or financial media, on herding is also captured within the model (Tetlock 2007 ).

In conclusion, the agent-based model we present is meticulously designed to reflect the intricate fabric of financial market behavior. It is particularly attuned to the multi-layered aspects that drive herding, informed by empirical evidence and theoretical underpinnings that rigorously define the interrelations between investor demographics and market behavior. The aforementioned socio-economic and psychological facets provide a comprehensive backdrop against which the validity and consistency of the model are substantiated.

The following code has been prepared using Python programming language with the Mesa, Pandas, SciPy, NumPy, Random and Matplotlib libraries. This code simulates a herd behavior of stock traders in a simple market (Hunt and Thomas 2010 ; McKinney 2010 ; Harris et al. 2020 ; Virtanen et al. 2020 ; Van Rossum 2020 ; Hunter 2007 ). The simulation runs for 50-time steps, with the stock price and balance of each agent printed at each step. The decision-making process of agents in the simulation is stochastic, with agents randomly choosing to buy, sell, or follow the market trend based on their characteristics and decision-making strategy.

The Stock Trader class in the model symbolizes individual agents, each characterized by a unique ID, balance, and a stock price. These agents are equipped with a method to compute the current stock price. The step() function within each agent embodies their decision-making process, which is influenced by their current balance and the prevailing stock price. Agents have the option to buy, sell, or align with the market trend, reflecting various investment strategies.

The Herding Model class encapsulates the entire simulation framework. It generates a population of Stock Trader agents and progresses the simulation over a designated number of time steps. Within this class, the agent_decision() method orchestrates each agent’s decision-making, factoring in individual characteristics and strategies. The step() method, in turn, adjusts the stock price based on the aggregate current stock prices of all agents before executing the step() method for each agent, thereby simulating the dynamic nature of the stock market.

Socio-demographic factors, specifically age and income status, are integrated into the agent-based model simulations, drawing upon insights from Parallel Coordinates and Cluster Analysis as well as relevant literature. The simulation posits that agents of younger age and lower income are predisposed to mimicking the market trend, whereas other agents exhibit a propensity for independent decision-making. Given the stochastic nature of the decision-making process, the behavior of agents varies across different runs of the simulation, introducing an element of unpredictability.

At each time step, the simulation outputs the stock price and balance of each agent, offering a snapshot of the market dynamics at that moment. Figure 10 provides a flow diagram elucidating the operational framework of the model’s code, presenting a visual representation of how the simulation unfolds over time.

Flowchart of agent-based model.

This model architecture allows for the exploration of how socio-demographic characteristics influence investment behaviors within a simulated market environment, offering valuable insights into the mechanisms driving market trends and individual investor decisions.

Within our agent-based model (ABM), “performance” embodies multiple dimensions reflective of the agents’ investment outcomes, influenced by socio-demographic factors and behavioral biases. The provided pseudo-code conceptualizes the implementation of these facets in the model.

Metrics used to quantify agent performance

Balance trajectory.

This primary indicator tracks the evolution of each agent’s financial balance over time, reflecting the impact of their buy, sell, or market trend-following decisions (Arthur 1991 ).

Decision strategy efficacy

Evaluates the effectiveness of an agent’s decision-making strategy (‘buy’, ‘sell’, or ‘follow’), influenced by socio-demographic variables such as age and income, as delineated in the agent_decision method (Tesfatsion and Judd 2006 ).

Market trend alignment

Assesses the correlation between an agent’s balance trajectory and overall market trends, indicating successful performance if an agent’s balance increases with market prices (Shiller 2003 ).

Risk management

Infers risk management skill from the volatility of balance changes, with less volatility indicating stable and potentially successful investment strategies (Markowitz 1952 ).

Wealth accumulation

Agents are ranked by their final balance at the simulation’s end to identify the most financially successful outcomes (De Long et al. 1990 ).

Adaptive behavior

The model evaluates agents’ adaptability to market price changes, revealing their capacity to capitalize on market movements (Gode and Sunder 1993 ).

Herding influence

Considers how herding behavior impacts financial outcomes, especially for younger and lower-income agents as programmed in the Herding Model class (Bikhchandani et al. 1992 ).

These performance metrics are quantified through agents’ balance and stock price histories, updated at each simulation step. These histories offer a time series analysis of financial trajectories, enabling pattern identification such as herding tendencies or the effects of overconfidence.

The model’s realism is enhanced by parameters like young_follow_factor and low_income_follow_factor, adjusting the propensity for herding among different socio-demographic groups. This inclusion allows the model to reflect real-world dynamics where age and income significantly impact investment performance.

In conclusion, our ABM presents a detailed framework for examining investment performance’s complex nature. It integrates behavioral economics and socio-demographic data, providing insights into investor behavior under simulated market conditions.

Characteristics of agents in the agent-based model

Demographics (age and income): Consistent with the focus of our study on socio-demographic factors, each agent is characterized by age and income parameters, which influence their investment behavior, particularly their propensity towards herding. Age and income are randomly assigned within realistic bounds reflecting the demographic distribution of typical investor populations.

Cognitive biases: Agents are imbued with behavioral attributes such as overconfidence, herding instinct, and over/under-reaction tendencies to market news, reflecting the psychological dimensions of real-world investors.

Investment strategy: Each agent follows a distinct investment strategy categorized broadly as ‘buy’, ‘sell’, or ‘follow’ (herding). The strategy is influenced by the agent’s demographic characteristics and cognitive biases.

Adaptability: Agents are capable of learning and adapting to market changes over time, simulating the dynamic and evolving nature of real-world investor behavior.

Social influence: Agents are influenced by other agents’ behaviors, especially under conditions conducive to herding, modeling the social dynamics of investment communities.

Wealth and portfolio: Agents have a variable representing their wealth, which fluctuates based on investment decisions and market performance. Their portfolio composition and changes therein are also tracked, offering insights into their risk-taking and diversification behaviors.

Significance of agent-based modeling

Agent-based modeling is a powerful tool that allows researchers to simulate and analyze complex systems composed of interacting agents. Its significance and utility in various fields, including economics and finance are profound:

Complexity and emergence: ABM can capture the emergent phenomena that arise from the interactions of many individual agents, providing insights into complex market dynamics that are not apparent at the individual level (Epstein and Axtell 1996 ).

Customizability and scalability: ABMs can be tailored to include various levels of detail and complexity, allowing for the simulation of systems ranging from small groups to entire markets (Tesfatsion and Judd 2006 ).

Experimental flexibility: ABMs facilitate virtual experiments that would be impractical or impossible in the real world, enabling researchers to explore hypothetical scenarios and policy implications (Gilbert and Troitzsch 2005 ).

Realism in behavioral representation: By incorporating cognitive biases and decision-making rules, ABMs can realistically represent human behavior, providing deeper behavioral insights than models assuming perfect rationality (Hommes 2006 ).

Policy analysis and forecasting: In economics and finance, ABMs are particularly useful for policy analysis, risk assessment, and forecasting, as they can incorporate a wide range of real-world factors and individual behaviors (LeBaron and Tesfatsion 2008 ).

By integrating these agent characteristics into our ABM and considering the broader implications of agent-based modeling, our study aims to provide nuanced insights into herding behavior among investors. We believe that our approach not only aligns with best practices in the field but also significantly contributes to the understanding of complex investment behaviors and market dynamics. We trust that this expanded description addresses the reviewer’s comment and underscores the robustness and relevance of our agent-based simulation approach.

Figure 11a, b panels display the balance changes of agents with respect to stock prices, age, and income status. By coding the balance increases and decreases as +1 and −1, respectively, and employing a line graph that matches the changes in stock prices, it has become possible to provide information about the agents’ performance. In panels a and b, it is observed that agents created after the age of 37.5 have been included in the higher income group on average, and during transitions of stock prices below 12.75 units, between 17 and 20 units, and between 26 and 27.50 units, the agents’ responses to price state changes are accompanied by noticeable transitions (increases and decreases) in their portfolio states, depending on age and income status.

a Agents’ performance. b Agents’ responses.

In Fig. 12 , in the agent-based model’s 50 repeated simulations, at the 45th simulation, the stock price is 20.03 units, and the balance of agent number 74 reaches 911 units. The price-income-balance change graph for the agent throughout the 50 transactions is presented below.

Balance change according to stock price for agent 74.

Upon examining the descriptive statistics of the income for agent number 74, who diverges from the herding tendency profile of the model and is in the higher income group aged 40 and above, the highest balance value is 911 units, the lowest balance level is 732 units, the average is 799 units, and the standard deviation is 41 units. When the overall balance of the agents is investigated, it is observed that the average balance of the agents is around 84 units. Considering the existence of an agent with the lowest balance of −670 units, it can be concluded that agent number 74 has demonstrated a significantly superior performance.

Discussion and conclusion

The influence of behavioral biases on investors’ decision-making has yielded mixed findings in literature. Wan ( 2018 ) observed a positive impact of behavioral biases, considered forward-looking factors, on investment decisions. Conversely, Zulfiqar et al. ( 2018 ) noted a markedly negative impact of overconfidence on investment decisions. Similarly, Aziz and Khan ( 2016 ) explored the role of heuristic factors (representative, anchoring, overconfidence, and availability bases) and found them significantly influencing investment decision and performance. However, they reported that prospect factors (loss aversion, regret aversion, and mental accounting biases) had an insignificant impact on these outcomes.

These varied results may stem from a complex interplay of factors such as cultural differences, pandemic-related information, economic conditions, regulatory environments, historical context, and investors’ financial literacy levels, contributing to differences in how behavioral biases influence investment decisions across regions (Metawa et al. 2018 ).

This study contributes to the field of behavioral finance by revealing the moderating role of COVID-19 pandemic information sharing on the relationship between behavioral quirks and investment choices, specifically in the context of Pakistan. Key contributions include:

Investors’ sentiments

This study shows that COVID-19 pandemic information sharing significantly moderates the relationship between investors’ sentiments and their investment decisions, validating that pandemic-related information, such as infection rates and economic downturns, heavily influences investors’ sentiments and alters their risk perceptions (Anastasiou et al. 2022 ; Hsu and Tang 2022 ; Bin-Nashwan and Muneeza 2023 ; Gao et al. 2023 ; Sohail et al. 2020 ).

Overconfidence

It reveals how COVID-19 information reshapes overconfident investors’ risk perceptions, urging them to reassess their investment portfolios in light of the pandemic’s uncertainties and economic implications (Bouteska et al. 2023 ; Li and Cao 2021 ).

Over/under reaction

The study uncovers that the pandemic information moderates the relationship between over-under reaction and investment decisions, suggesting that investors adjust their reactions based on evolving pandemic information, leading to more informed and rational investment choices (Jiang et al. 2022 ).

Herd behavior

It finds that COVID-19 pandemic information significantly reduces herd behavior among investors, encouraging them to make rational decisions rather than blindly following the majority (Nguyen et al. 2023 ).

In conclusion, this study illustrates that the COVID-19 pandemic has significantly moderated the relationship between behavioral biases and investment decisions. Furthermore, clustering analyses and agent-based outcomes suggest that younger, less experienced agents prone to herding behavior exhibit a higher propensity for such behavior and demonstrate lower performance in agent-based models. These findings pave the way for further research into additional cognitive biases and socio-demographic variables’ effects on investment decisions.

Implications

This study contributes to the field of behavioral finance that COVID-19 pandemic information sharing significantly moderates the relationship between behavioral biases (e.g., investors’ sentiments, overconfidence, over/under reaction, and herd behavior) and investment decisions. Therefore, policy implications stem from findings are substantial, and thus addressing behavioral biases during COVID-19 pandemic to mitigate the market inefficiencies and promote better decision-making. First, this study suggests that investing in comprehensive financial education plans will enhance the financial literacy of investors and enable them to better recognize the behavioral biases during times of uncertainty and crises. Second, findings imply that accurate and transparent information sharing about COVID-19 pandemic can better mitigate the behavioral biases, especially government interventions (e.g., National Command and Coordination Centre) ensuring reliable information can lead the investors to make more rational and informed investment decisions during the time of uncertainty and crises. Last, findings provide insights to policy makers that pandemic news and developments significantly influenced behavioral biases of investment decisions (Khurshid et al. 2021 ). For example, news about number of causalities, infection rates, vaccine progress, government stimulus packages, or stock market downturns had immediate effects on behavioral biases especially when an investor is overconfidence, over/under reaction, and herd behavior. In this sense, enhancing information transparency about COVID-19 news in media can reduce the influence of sensationalized news on investor decisions.

Limitations and call for future research

This study significantly enhances the understanding of behavioral factors’ impact on investors’ decision-making processes, presenting important findings within the context of the COVID-19 pandemic. While these contributions are notable, the research is subject to certain limitations that pave the way for future exploration and deeper investigation into this complex field.

Firstly, the study underscores the necessity for further research to validate its results through larger sample sizes and a more diverse array of respondents. Adopting a longitudinal design could prove particularly insightful, enabling an analysis of behavioral biases across different stages of the pandemic and providing a dynamic perspective on how investor behaviors evolve over time.

In addition, there’s a highlighted opportunity for future studies to delve into the behaviors influencing institutional investor decisions within Pakistan. The complex decision-making processes and investment portfolios of institutional investors, coupled with challenges like data availability and the heterogeneity among institutions, present a fertile ground for investigation. Such research could unravel how various factors, including market conditions and macroeconomic assessments, impact institutional investment strategies.

The study also points out the need to broaden the investigation to include other potential behavioral factors beyond those focused on in the current research, such as loss aversion, personality traits, anchoring, and recency biases. Expanding the scope of behavioral factors examined could significantly enrich the behavioral finance field by offering a more comprehensive view of the influences on investment decisions.

Moreover, while the insights gained from a Pakistani context during the COVID-19 pandemic are invaluable, extending the research to include global (e.g., China, Japan, USA) and other emerging markets (e.g., BRICS) would enhance understanding of the universality or specificity of behavioral biases in investment decisions across various economic, cultural, and regulatory environments.

Lastly, the study’s reliance on quantitative data points to the potential benefits of incorporating qualitative data into future research. Undertaking case studies within specific securities brokerages or investment banks could provide an in-depth investigation of investor behavior, generating new insights that could inspire further research.

To support the development of more sophisticated agent-based models and to foster collaborative research efforts, the study makes its source code available to other researchers. This openness to collaboration promises to stimulate innovative approaches to understanding and modeling investor behavior across diverse contexts, contributing to the advancement of the behavioral finance field.

Author information

Authors and affiliations.

Department of Business Administration, University of the Punjab, Gujranwala Campus, Gujranwala, Pakistan

Wasim ul Rehman

Manager of Economics Research Department, Marbas Securities Co., Istanbul, Turkey

Omur Saltik

Institute of Quality and Technology Management, University of the Punjab, Lahore, Pakistan

Faryal Jalil

Department of Economics, Mersin University, Mersin, Turkey

Suleyman Degirmen

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed equally to this research work.

Corresponding author

Correspondence to Wasim ul Rehman .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

The data was collated through an online survey approach (questionnaire) during the last variant of COVID-19 where anonymity of the respondents is meticulously preserved. The respondents were not asked to provide their names, identification, address, or any other identifying elements. The authors minutely observed the ethical guidelines of the Declaration of Helsinki. In addition, we hereby certify that this study was conducted under the ethical approval guidelines of Office of Research Innovation and Commercialization, University of the Punjab granted under the office order No. D/ 409/ORIC dated 31-12-2021.

Informed consent

The consent of participants was obtained through consent form during the last variant of COVID-19. The consent form contains the title of study, intent of study, procedure to participate, confidentiality, voluntary participation of respondents, questions/query and consent of the respondents. The respondents were requested to provide their willingness to participate in survey on consent form via email before filling the online-surveyed (questionnaire). Further, participants were also assured that their anonymity would be maintained and that no personal information or identifying element would be disclosed. The consent form is in the supplementary files.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Consent form, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Rehman, W.u., Saltik, O., Jalil, F. et al. Viral decisions: unmasking the impact of COVID-19 info and behavioral quirks on investment choices. Humanit Soc Sci Commun 11 , 524 (2024). https://doi.org/10.1057/s41599-024-03011-7

Download citation

Received : 17 June 2023

Accepted : 28 March 2024

Published : 20 April 2024

DOI : https://doi.org/10.1057/s41599-024-03011-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Open access
• Published: 23 April 2024

Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review

• Javiera Fuentes-Cimma 1 , 2 ,
• Dominique Sluijsmans 3 ,
• Arnoldo Riquelme 4 ,
• Ignacio Villagran   ORCID: orcid.org/0000-0003-3130-8326 1 ,
• Lorena Isbej   ORCID: orcid.org/0000-0002-4272-8484 2 , 5 ,
• María Teresa Olivares-Labbe 6 &
• Sylvia Heeneman 7  

BMC Medical Education volume  24 , Article number:  440 ( 2024 ) Cite this article

Metrics details

Feedback processes are crucial for learning, guiding improvement, and enhancing performance. In workplace-based learning settings, diverse teaching and assessment activities are advocated to be designed and implemented, generating feedback that students use, with proper guidance, to close the gap between current and desired performance levels. Since productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured feedback activities within undergraduate workplace-based learning settings. However, these settings are characterized by their unpredictable nature, which can either promote learning or present challenges in offering structured learning opportunities for students. This scoping review maps literature on how feedback processes are organised in undergraduate clinical workplace-based learning settings, providing insight into the design and use of feedback.

A scoping review was conducted. Studies were identified from seven databases and ten relevant journals in medical education. The screening process was performed independently in duplicate with the support of the StArt program. Data were organized in a data chart and analyzed using thematic analysis. The feedback loop with a sociocultural perspective was used as a theoretical framework.

The search yielded 4,877 papers, and 61 were included in the review. Two themes were identified in the qualitative analysis: (1) The organization of the feedback processes in workplace-based learning settings, and (2) Sociocultural factors influencing the organization of feedback processes. The literature describes multiple teaching and assessment activities that generate feedback information. Most papers described experiences and perceptions of diverse teaching and assessment feedback activities. Few studies described how feedback processes improve performance. Sociocultural factors such as establishing a feedback culture, enabling stable and trustworthy relationships, and enhancing student feedback agency are crucial for productive feedback processes.

Conclusions

This review identified concrete ideas regarding how feedback could be organized within the clinical workplace to promote feedback processes. The feedback encounter should be organized to allow follow-up of the feedback, i.e., working on required learning and performance goals at the next occasion. The educational programs should design feedback processes by appropriately planning subsequent tasks and activities. More insight is needed in designing a full-loop feedback process, in which specific attention is needed in effective feedforward practices.

Peer Review reports

The design of effective feedback processes in higher education has been important for educators and researchers and has prompted numerous publications discussing potential mechanisms, theoretical frameworks, and best practice examples over the past few decades. Initially, research on feedback primarily focused more on teachers and feedback delivery, and students were depicted as passive feedback recipients [ 1 , 2 , 3 ]. The feedback conversation has recently evolved to a more dynamic emphasis on interaction, sense-making, outcomes in actions, and engagement with learners [ 2 ]. This shift aligns with utilizing the feedback process as a form of social interaction or dialogue to enhance performance [ 4 ]. Henderson et al. (2019) defined feedback processes as "where the learner makes sense of performance-relevant information to promote their learning." (p. 17). When a student grasps the information concerning their performance in connection to the desired learning outcome and subsequently takes suitable action, a feedback loop is closed so the process can be regarded as successful [ 5 , 6 ].

Hattie and Timperley (2007) proposed a comprehensive perspective on feedback, the so-called feedback loop, to answer three key questions: “Where am I going? “How am I going?” and “Where to next?” [ 7 ]. Each question represents a key dimension of the feedback loop. The first is the feed-up, which consists of setting learning goals and sharing clear objectives of learners' performance expectations. While the concept of the feed-up might not be consistently included in the literature, it is considered to be related to principles of effective feedback and goal setting within educational contexts [ 7 , 8 ]. Goal setting allows students to focus on tasks and learning, and teachers to have clear intended learning outcomes to enable the design of aligned activities and tasks in which feedback processes can be embedded [ 9 ]. Teachers can improve the feed-up dimension by proposing clear, challenging, but achievable goals [ 7 ]. The second dimension of the feedback loop focuses on feedback and aims to answer the second question by obtaining information about students' current performance. Different teaching and assessment activities can be used to obtain feedback information, and it can be provided by a teacher or tutor, a peer, oneself, a patient, or another coworker. The last dimension of the feedback loop is the feedforward, which is specifically associated with using feedback to improve performance or change behaviors [ 10 ]. Feedforward is crucial in closing the loop because it refers to those specific actions students must take to reduce the gap between current and desired performance [ 7 ].

From a sociocultural perspective, feedback processes involve a social practice consisting of intricate relationships within a learning context [ 11 ]. The main feature of this approach is that students learn from feedback only when the feedback encounter includes generating, making sense of, and acting upon the information given [ 11 ]. In the context of workplace-based learning (WBL), actionable feedback plays a crucial role in enabling learners to leverage specific feedback to enhance their performance, skills, and conceptual understandings. The WBL environment provides students with a valuable opportunity to gain hands-on experience in authentic clinical settings, in which students work more independently on real-world tasks, allowing them to develop and exhibit their competencies [ 3 ]. However, WBL settings are characterized by their unpredictable nature, which can either promote self-directed learning or present challenges in offering structured learning opportunities for students [ 12 ]. Consequently, designing purposive feedback opportunities within WBL settings is a significant challenge for clinical teachers and faculty.

In undergraduate clinical education, feedback opportunities are often constrained due to the emphasis on clinical work and the absence of dedicated time for teaching [ 13 ]. Students are expected to perform autonomously under supervision, ideally achieved by giving them space to practice progressively and providing continuous instances of constructive feedback [ 14 ]. However, the hierarchy often present in clinical settings places undergraduate students in a dependent position, below residents and specialists [ 15 ]. Undergraduate or junior students may have different approaches to receiving and using feedback. If their priority is meeting the minimum standards given pass-fail consequences and acting merely as feedback recipients, other incentives may be needed to engage with the feedback processes because they will need more learning support [ 16 , 17 ]. Adequate supervision and feedback have been recognized as vital educational support in encouraging students to adopt a constructive learning approach [ 18 ]. Given that productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured teaching and learning feedback activities within undergraduate WBL settings.

Despite the extensive research on feedback, a significant proportion of published studies involve residents or postgraduate students [ 19 , 20 ]. Recent reviews focusing on feedback interventions within medical education have clearly distinguished between undergraduate medical students and residents or fellows [ 21 ]. To gain a comprehensive understanding of initiatives related to actionable feedback in the WBL environment for undergraduate health professions, a scoping review of the existing literature could provide insight into how feedback processes are designed in that context. Accordingly, the present scoping review aims to answer the following research question: How are the feedback processes designed in the undergraduate health professions' workplace-based learning environments?

A scoping review was conducted using the five-step methodological framework proposed by Arksey and O'Malley (2005) [ 22 ], intertwined with the PRISMA checklist extension for scoping reviews to provide reporting guidance for this specific type of knowledge synthesis [ 23 ]. Scoping reviews allow us to study the literature without restricting the methodological quality of the studies found, systematically and comprehensively map the literature, and identify gaps [ 24 ]. Furthermore, a scoping review was used because this topic is not suitable for a systematic review due to the varied approaches described and the large difference in the methodologies used [ 21 ].

Search strategy

With the collaboration of a medical librarian, the authors used the research question to guide the search strategy. An initial meeting was held to define keywords and search resources. The proposed search strategy was reviewed by the research team, and then the study selection was conducted in two steps:

An online database search included Medline/PubMed, Web of Science, CINAHL, Cochrane Library, Embase, ERIC, and PsycINFO.

A directed search of ten relevant journals in the health sciences education field (Academic Medicine, Medical Education, Advances in Health Sciences Education, Medical Teacher, Teaching and Learning in Medicine, Journal of Surgical Education, BMC Medical Education, Medical Education Online, Perspectives on Medical Education and The Clinical Teacher) was performed.

The research team conducted a pilot or initial search before the full search to identify if the topic was susceptible to a scoping review. The full search was conducted in November 2022. One team member (MO) identified the papers in the databases. JF searched in the selected journals. Authors included studies written in English due to feasibility issues, with no time span limitation. After eliminating duplicates, two research team members (JF and IV) independently reviewed all the titles and abstracts using the exclusion and inclusion criteria described in Table  2 and with the support of the screening application StArT [ 25 ]. A third team member (AR) reviewed the titles and abstracts when the first two disagreed. The reviewer team met again at a midpoint and final stage to discuss the challenges related to study selection. Articles included for full-text review were exported to Mendeley. JF independently screened all full-text papers, and AR verified 10% for inclusion. The authors did not analyze study quality or risk of bias during study selection, which is consistent with conducting a scoping review.

The analysis of the results incorporated a descriptive summary and a thematic analysis, which was carried out to clarify and give consistency to the results' reporting [ 22 , 24 , 26 ]. Quantitative data were analyzed to report the characteristics of the studies, populations, settings, methods, and outcomes. Qualitative data were labeled, coded, and categorized into themes by three team members (JF, SH, and DS). The feedback loop framework with a sociocultural perspective was used as the theoretical framework to analyze the results.

The keywords used for the search strategies were as follows:

Clinical clerkship; feedback; formative feedback; health professions; undergraduate medical education; workplace.

Definitions of the keywords used for the present review are available in Appendix 1 .

As an example, we included the search strategy that we used in the Medline/PubMed database when conducting the full search:

("Formative Feedback"[Mesh] OR feedback) AND ("Workplace"[Mesh] OR workplace OR "Clinical Clerkship"[Mesh] OR clerkship) AND (("Education, Medical, Undergraduate"[Mesh] OR undergraduate health profession*) OR (learner* medical education)).

Inclusion and exclusion criteria

The following inclusion and exclusion criteria were used (Table  1 ):

Data extraction

The research group developed a data-charting form to organize the information obtained from the studies. The process was iterative, as the data chart was continuously reviewed and improved as necessary. In addition, following Levac et al.'s recommendation (2010), the three members involved in the charting process (JF, LI, and IV) independently reviewed the first five selected studies to determine whether the data extraction was consistent with the objectives of this scoping review and to ensure consistency. Then, the team met using web-conferencing software (Zoom; CA, USA) to review the results and adjust any details in the chart. The same three members extracted data independently from all the selected studies, considering two members reviewing each paper [ 26 ]. A third team member was consulted if any conflict occurred when extracting data. The data chart identified demographic patterns and facilitated the data synthesis. To organize data, we used a shared Excel spreadsheet, considering the following headings: title, author(s), year of publication, journal/source, country/origin, aim of the study, research question (if any), population/sample size, participants, discipline, setting, methodology, study design, data collection, data analysis, intervention, outcomes, outcomes measure, key findings, and relation of findings to research question.

Additionally, all the included papers were uploaded to AtlasTi v19 to facilitate the qualitative analysis. Three team members (JF, SH, and DS) independently coded the first six papers to create a list of codes to ensure consistency and rigor. The group met several times to discuss and refine the list of codes. Then, one member of the team (JF) used the code list to code all the rest of the papers. Once all papers were coded, the team organized codes into descriptive themes aligned with the research question.

Preliminary results were shared with a number of stakeholders (six clinical teachers, ten students, six medical educators) to elicit their opinions as an opportunity to build on the evidence and offer a greater level of meaning, content expertise, and perspective to the preliminary findings [ 26 ]. No quality appraisal of the studies is considered for this scoping review, which aligns with the frameworks for guiding scoping reviews [ 27 ].

The datasets analyzed during the current study are available from the corresponding author upon request.

A database search resulted in 3,597 papers, and the directed search of the most relevant journals in the health sciences education field yielded 2,096 titles. An example of the results of one database is available in Appendix 2 . Of the titles obtained, 816 duplicates were eliminated, and the team reviewed the titles and abstracts of 4,877 papers. Of these, 120 were selected for full-text review. Finally, 61 papers were included in this scoping review (Fig.  1 ), as listed in Table  2 .

PRISMA flow diagram for included studies, incorporating records identified through the database and direct searching

The selected studies were published between 1986 and 2022, and seventy-five percent (46) were published during the last decade. Of all the articles included in this review, 13% (8) were literature reviews: one integrative review [ 28 ] and four scoping reviews [ 29 , 30 , 31 , 32 ]. Finally, fifty-three (87%) original or empirical papers were included (i.e., studies that answered a research question or achieved a research purpose through qualitative or quantitative methodologies) [ 15 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 ].

Table 2 summarizes the papers included in the present scoping review, and Table  3 describes the characteristics of the included studies.

The thematic analysis resulted in two themes: (1) the organization of feedback processes in WBL settings, and (2) sociocultural factors influencing the organization of feedback processes. Table 4 gives a summary of the themes and subthemes.

Organization of feedback processes in WBL settings.

Setting learning goals (i.e., feed-up dimension).

Feedback that focuses on students' learning needs and is based on known performance standards enhances student response and setting learning goals [ 30 ]. Discussing goals and agreements before starting clinical practice enhances students' feedback-seeking behavior [ 39 ] and responsiveness to feedback [ 83 ]. Farrell et al. (2017) found that teacher-learner co-constructed learning goals enhance feedback interactions and help establish educational alliances, improving the learning experience [ 50 ]. However, Kiger (2020) found that sharing individualized learning plans with teachers aligned feedback with learning goals but did not improve students' perceived use of feedback [ 64 ]

Two papers of this set pointed out the importance of goal-oriented feedback, a dynamic process that depends on discussion of goal setting between teachers and students [ 50 ] and influences how individuals experience, approach, and respond to upcoming learning activities [ 34 ]. Goal-oriented feedback should be embedded in the learning experience of the clinical workplace, as it can enhance students' engagement in safe feedback dialogues [ 50 ]. Ideally, each feedback encounter in the WBL context should conclude, in addition to setting a plan of action to achieve the desired goal, with a reflection on the next goal [ 50 ].

Feedback strategies within the WBL environment. (i.e., feedback dimension)

In undergraduate WBL environments, there are several tasks and feedback opportunities organized in the undergraduate clinical workplace that can enable feedback processes:

Questions from clinical teachers to students are a feedback strategy [ 74 ]. There are different types of questions that the teacher can use, either to clarify concepts, to reach the correct answer, or to facilitate self-correction [ 74 ]. Usually, questions can be used in conjunction with other communication strategies, such as pauses, which enable self-correction by the student [ 74 ]. Students can also ask questions to obtain feedback on their performance [ 54 ]. However, question-and-answer as a feedback strategy usually provides information on either correct or incorrect answers and fewer suggestions for improvement, rendering it less constructive as a feedback strategy [ 82 ].

Direct observation of performance by default is needed to be able to provide information to be used as input in the feedback process [ 33 , 46 , 49 , 86 ]. In the process of observation, teachers can include clarification of objectives (i.e., feed-up dimension) and suggestions for an action plan (i.e., feedforward) [ 50 ]. Accordingly, Schopper et al. (2016) showed that students valued being observed while interviewing patients, as they received feedback that helped them become more efficient and effective as interviewers and communicators [ 33 ]. Moreover, it is widely described that direct observation improves feedback credibility [ 33 , 40 , 84 ]. Ideally, observation should be deliberate [ 33 , 83 ], informal or spontaneous [ 33 ], conducted by a (clinical) expert [ 46 , 86 ], provided immediately after the observation, and clinical teacher if possible, should schedule or be alert on follow-up observations to promote closing the gap between current and desired performance [ 46 ].

Workplace-based assessments (WBAs), by definition, entail direct observation of performance during authentic task demonstration [ 39 , 46 , 56 , 87 ]. WBAs can significantly impact behavioral change in medical students [ 55 ]. Organizing and designing formative WBAs and embedding these in a feedback dialogue is essential for effective learning [ 31 ].

Summative organization of WBAs is a well described barrier for feedback uptake in the clinical workplace [ 35 , 46 ]. If feedback is perceived as summative, or organized as a pass-fail decision, students may be less inclined to use the feedback for future learning [ 52 ]. According to Schopper et al. (2016), using a scale within a WBA makes students shift their focus during the clinical interaction and see it as an assessment with consequences [ 33 ]. Harrison et al. (2016) pointed out that an environment that only contains assessments with a summative purpose will not lead to a culture of learning and improving performance [ 56 ]. The recommendation is to separate the formative and summative WBAs, as feedback in summative instances is often not recognized as a learning opportunity or an instance to seek feedback [ 54 ]. In terms of the design, an organizational format is needed to clarify to students how formative assessments can promote learning from feedback [ 56 ]. Harrison et al. (2016) identified that enabling students to have more control over their assessments, designing authentic assessments, and facilitating long-term mentoring could improve receptivity to formative assessment feedback [ 56 ].

Multiple WBA instruments and systems are reported in the literature. Sox et al. (2014) used a detailed evaluation form to help students improve their clinical case presentation skills. They found that feedback on oral presentations provided by supervisors using a detailed evaluation form improved clerkship students’ oral presentation skills [ 78 ]. Daelmans et al. (2006) suggested that a formal in-training assessment programme composed by 19 assessments that provided structured feedback, could promote observation and verbal feedback opportunities through frequent assessments [ 43 ]. However, in this setting, limited student-staff interactions still hindered feedback follow-up [ 43 ]. Designing frequent WBA improves feedback credibility [ 28 ]. Long et al. (2021) emphasized that students' responsiveness to assessment feedback hinges on its perceived credibility, underlining the importance of credibility for students to effectively engage and improve their performance [ 31 ].

The mini-CEX is one of the most widely described WBA instruments in the literature. Students perceive that the mini-CEX allows them to be observed and encourages the development of interviewing skills [ 33 ]. The mini-CEX can provide feedback that improves students' clinical skills [ 58 , 60 ], as it incorporates a structure for discussing the student's strengths and weaknesses and the design of a written action plan [ 39 , 80 ]. When mini-CEXs are incorporated as part of a system of WBA, such as programmatic assessment, students feel confident in seeking feedback after observation, and being systematic allows for follow-up [ 39 ]. Students suggested separating grading from observation and using the mini-CEX in more informal situations [ 33 ].

Clinical encounter cards allow students to receive weekly feedback and make them request more feedback as the clerkship progresses [ 65 ]. Moreover, encounter cards stimulate that feedback is given by supervisors, and students are more satisfied with the feedback process [ 72 ]. With encounter card feedback, students are responsible for asking a supervisor for feedback before a clinical encounter, and supervisors give students written and verbal comments about their performance after the encounter [ 42 , 72 ]. Encounter cards enhance the use of feedback and add approximately one minute to the length of the clinical encounter, so they are well accepted by students and supervisors [ 72 ]. Bennett (2006) identified that Instant Feedback Cards (IFC) facilitated mid-rotation feedback [ 38 ]. Feedback encounter card comments must be discussed between students and supervisors; otherwise, students may perceive it as impersonal, static, formulaic, and incomplete [ 59 ].

Self-assessments can change students' feedback orientation, transforming them into coproducers of learning [ 68 ]. Self-assessments promote the feedback process [ 68 ]. Some articles emphasize the importance of organizing self-assessments before receiving feedback from supervisors, for example, discussing their appraisal with the supervisor [ 46 , 52 ]. In designing a feedback encounter, starting with a self-assessment as feed-up, discussing with the supervisor, and identifying areas for improvement is recommended, as part of the feedback dialogue [ 68 ].

Peer feedback as an organized activity allows students to develop strategies to observe and give feedback to other peers [ 61 ]. Students can act as the feedback provider or receiver, fostering understanding of critical comments and promoting evaluative judgment for their clinical practice [ 61 ]. Within clerkships, enabling the sharing of feedback information among peers allows for a better understanding and acceptance of feedback [ 52 ]. However, students can find it challenging to take on the peer assessor/feedback provider role, as they prefer to avoid social conflicts [ 28 , 61 ]. Moreover, it has been described that they do not trust the judgment of their peers because they are not experts, although they know the procedures, tasks, and steps well and empathize with their peer status in the learning process [ 61 ].

Bedside-teaching encounters (BTEs) provide timely feedback and are an opportunity for verbal feedback during performance [ 74 ]. Rizan et al. (2014) explored timely feedback delivered within BTEs and determined that it promotes interaction that constructively enhances learner development through various corrective strategies (e.g., question and answers, pauses, etc.). However, if the feedback given during the BTEs was general, unspecific, or open-ended, it could go unnoticed [ 74 ]. Torre et al. (2005) investigated which integrated feedback activities and clinical tasks occurred on clerkship rotations and assessed students' perceived quality in each teaching encounter [ 81 ]. The feedback activities reported were feedback on written clinical history, physical examination, differential diagnosis, oral case presentation, a daily progress note, and bedside feedback. Students considered all these feedback activities high-quality learning opportunities, but they were more likely to receive feedback when teaching was at the bedside than at other teaching locations [ 81 ].

Case presentations are an opportunity for feedback within WBL contexts [ 67 , 73 ]. However, both students and supervisors struggled to identify them as feedback moments, and they often dismissed questions and clarifications around case presentations as feedback [ 73 ]. Joshi (2017) identified case presentations as a way for students to ask for informal or spontaneous supervisor feedback [ 63 ].

Organization of follow-up feedback and action plans (i.e., feedforward dimension).

Feedback that generates use and response from students is characterized by two-way communication and embedded in a dialogue [ 30 ]. Feedback must be future-focused [ 29 ], and a feedback encounter should be followed by planning the next observation [ 46 , 87 ]. Follow-up feedback could be organized as a future self-assessment, reflective practice by the student, and/or a discussion with the supervisor or coach [ 68 ]. The literature describes that a lack of student interaction with teachers makes follow-up difficult [ 43 ]. According to Haffling et al. (2011), follow-up feedback sessions improve students' satisfaction with feedback compared to students who do not have follow-up sessions. In addition, these same authors reported that a second follow-up session allows verification of improved performances or confirmation that the skill was acquired [ 55 ].

Although feedback encounter forms are a recognized way of obtaining information about performance (i.e., feedback dimension), the literature does not provide many clear examples of how they may impact the feedforward phase. For example, Joshi et al. (2016) consider a feedback form with four fields (i.e., what did you do well, advise the student on what could be done to improve performance, indicate the level of proficiency, and personal details of the tutor). In this case, the supervisor highlighted what the student could improve but not how, which is the missing phase of the co-constructed action plan [ 63 ]. Whichever WBA instrument is used in clerkships to provide feedback, it should include a "next steps" box [ 44 ], and it is recommended to organize a long-term use of the WBA instrument so that those involved get used to it and improve interaction and feedback uptake [ 55 ]. RIME-based feedback (Reporting, Interpreting, Managing, Educating) is considered an interesting example, as it is perceived as helpful to students in knowing what they need to improve in their performance [ 44 ]. Hochberg (2017) implemented formative mid-clerkship assessments to enhance face-to-face feedback conversations and co-create an improvement plan [ 59 ]. Apps for structuring and storing feedback improve the amount of verbal and written feedback. In the study of Joshi et al. (2016), a reasonable proportion of students (64%) perceived that these app tools help them improve their performance during rotations [ 63 ].

Several studies indicate that an action plan as part of the follow-up feedback is essential for performance improvement and learning [ 46 , 55 , 60 ]. An action plan corresponds to an agreed-upon strategy for improving, confirming, or correcting performance. Bing-You et al. (2017) determined that only 12% of the articles included in their scoping review incorporated an action plan for learners [ 32 ]. Holmboe et al. (2004) reported that only 11% of the feedback sessions following a mini-CEX included an action plan [ 60 ]. Suhoyo et al. (2017) also reported that only 55% of mini-CEX encounters contained an action plan [ 80 ]. Other authors reported that action plans are not commonly offered during feedback encounters [ 77 ]. Sokol-Hessner et al. (2010) implemented feedback card comments with a space to provide written feedback and a specific action plan. In their results, 96% contained positive comments, and only 5% contained constructive comments [ 77 ]. In summary, although the recommendation is to include a “next step” box in the feedback instruments, evidence shows these items are not often used for constructive comments or action plans.

Sociocultural factors influencing the organization of feedback processes.

Multiple sociocultural factors influence interaction in feedback encounters, promoting or hampering the productivity of the feedback processes.

Clinical learning culture

Context impacts feedback processes [ 30 , 82 ], and there are barriers to incorporating actionable feedback in the clinical learning context. The clinical learning culture is partly determined by the clinical context, which can be unpredictable [ 29 , 46 , 68 ], as the available patients determine learning opportunities. Supervisors are occupied by a high workload, which results in limited time or priority for teaching [ 35 , 46 , 48 , 55 , 68 , 83 ], hindering students’ feedback-seeking behavior [ 54 ], and creating a challenge for the balance between patient care and student mentoring [ 35 ].

Clinical workplace culture does not always purposefully prioritize instances for feedback processes [ 83 , 84 ]. This often leads to limited direct observation [ 55 , 68 ] and the provision of poorly informed feedback. It is also evident that this affects trust between clinical teachers and students [ 52 ]. Supervisors consider feedback a low priority in clinical contexts [ 35 ] due to low compensation and lack of protected time [ 83 ]. In particular, lack of time appears to be the most significant and well-known barrier to frequent observation and workplace feedback [ 35 , 43 , 48 , 62 , 67 , 83 ].

The clinical environment is hierarchical [ 68 , 80 ] and can make students not consider themselves part of the team and feel like a burden to their supervisor [ 68 ]. This hierarchical learning environment can lead to unidirectional feedback, limit dialogue during feedback processes, and hinder the seeking, uptake, and use of feedback [ 67 , 68 ]. In a learning culture where feedback is not supported, learners are less likely to want to seek it and feel motivated and engaged in their learning [ 83 ]. Furthermore, it has been identified that clinical supervisors lack the motivation to teach [ 48 ] and the intention to observe or reobserve performance [ 86 ].

In summary, the clinical context and WBL culture do not fully use the potential of a feedback process aimed at closing learning gaps. However, concrete actions shown in the literature can be taken to improve the effectiveness of feedback by organizing the learning context. For example, McGinness et al. (2022) identified that students felt more receptive to feedback when working in a safe, nonjudgmental environment [ 67 ]. Moreover, supervisors and trainees identified the learning culture as key to establishing an open feedback dialogue [ 73 ]. Students who perceive culture as supportive and formative can feel more comfortable performing tasks and more willing to receive feedback [ 73 ].

Relationships

There is a consensus in the literature that trusting and long-term relationships improve the chances of actionable feedback. However, relationships between supervisors and students in the clinical workplace are often brief and not organized as more longitudinally [ 68 , 83 ], leaving little time to establish a trustful relationship [ 68 ]. Supervisors change continuously, resulting in short interactions that limit the creation of lasting relationships over time [ 50 , 68 , 83 ]. In some contexts, it is common for a student to have several supervisors who have their own standards in the observation of performance [ 46 , 56 , 68 , 83 ]. A lack of stable relationships results in students having little engagement in feedback [ 68 ]. Furthermore, in case of summative assessment programmes, the dual role of supervisors (i.e., assessing and giving feedback) makes feedback interactions perceived as summative and can complicate the relationship [ 83 ].

Repeatedly, the articles considered in this review describe that long-term and stable relationships enable the development of trust and respect [ 35 , 62 ] and foster feedback-seeking behavior [ 35 , 67 ] and feedback-giver behavior [ 39 ]. Moreover, constructive and positive relationships enhance students´ use of and response to feedback [ 30 ]. For example, Longitudinal Integrated Clerkships (LICs) promote stable relationships, thus enhancing the impact of feedback [ 83 ]. In a long-term trusting relationship, feedback can be straightforward and credible [ 87 ], there are more opportunities for student observation, and the likelihood of follow-up and actionable feedback improves [ 83 ]. Johnson et al. (2020) pointed out that within a clinical teacher-student relationship, the focus must be on establishing psychological safety; thus, the feedback conversations might be transformed [ 62 ].

Stable relationships enhance feedback dialogues, which offer an opportunity to co-construct learning and propose and negotiate aspects of the design of learning strategies [ 62 ].

Students as active agents in the feedback processes

The feedback response learners generate depends on the type of feedback information they receive, how credible the source of feedback information is, the relationship between the receiver and the giver, and the relevance of the information delivered [ 49 ]. Garino (2020) noted that students who are most successful in using feedback are those who do not take criticism personally, who understand what they need to improve and know they can do so, who value and feel meaning in criticism, are not surprised to receive it, and who are motivated to seek new feedback and use effective learning strategies [ 52 ]. Successful users of feedback ask others for help, are intentional about their learning, know what resources to use and when to use them, listen to and understand a message, value advice, and use effective learning strategies. They regulate their emotions, find meaning in the message, and are willing to change [ 52 ].

Student self-efficacy influences the understanding and use of feedback in the clinical workplace. McGinness et al. (2022) described various positive examples of self-efficacy regarding feedback processes: planning feedback meetings with teachers, fostering good relationships with the clinical team, demonstrating interest in assigned tasks, persisting in seeking feedback despite the patient workload, and taking advantage of opportunities for feedback, e.g., case presentations [ 67 ].

When students are encouraged to seek feedback aligned with their own learning objectives, they promote feedback information specific to what they want to learn and improve and enhance the use of feedback [ 53 ]. McGinness et al. (2022) identified that the perceived relevance of feedback information influenced the use of feedback because students were more likely to ask for feedback if they perceived that the information was useful to them. For example, if students feel part of the clinical team and participate in patient care, they are more likely to seek feedback [ 17 ].

Learning-oriented students aim to seek feedback to achieve clinical competence at the expected level [ 75 ]; they focus on improving their knowledge and skills and on professional development [ 17 ]. Performance-oriented students aim not to fail and to avoid negative feedback [ 17 , 75 ].

For effective feedback processes, including feed-up, feedback, and feedforward, the student must be feedback-oriented, i.e., active, seeking, listening to, interpreting, and acting on feedback [ 68 ]. The literature shows that feedback-oriented students are coproducers of learning [ 68 ] and are more involved in the feedback process [ 51 ]. Additionally, students who are metacognitively aware of their learning process are more likely to use feedback to reduce gaps in learning and performance [ 52 ]. For this, students must recognize feedback when it occurs and understand it when they receive it. Thus, it is important to organize training and promote feedback literacy so that students understand what feedback is, act on it, and improve the quality of feedback and their learning plans [ 68 ].

Table 5 summarizes those feedback tasks, activities, and key features of organizational aspects that enable each phase of the feedback loop based on the literature review.

The present scoping review identified 61 papers that mapped the literature on feedback processes in the WBL environments of undergraduate health professions. This review explored how feedback processes are organized in these learning contexts using the feedback loop framework. Given the specific characteristics of feedback processes in undergraduate clinical learning, three main findings were identified on how feedback processes are being conducted in the clinical environment and how these processes could be organized to support feedback processes.

First, the literature lacks a balance between the three dimensions of the feedback loop. In this regard, most of the articles in this review focused on reporting experiences or strategies for delivering feedback information (i.e., feedback dimension). Credible and objective feedback information is based on direct observation [ 46 ] and occurs within an interaction or a dialogue [ 62 , 88 ]. However, only having credible and objective information does not ensure that it will be considered, understood, used, and put into practice by the student [ 89 ].

Feedback-supporting actions aligned with goals and priorities facilitate effective feedback processes [ 89 ] because goal-oriented feedback focuses on students' learning needs [ 7 ]. In contrast, this review showed that only a minority of the studies highlighted the importance of aligning learning objectives and feedback (i.e., the feed-up dimension). To overcome this, supervisors and students must establish goals and agreements before starting clinical practice, as it allows students to measure themselves on a defined basis [ 90 , 91 ] and enhances students' feedback-seeking behavior [ 39 , 92 ] and responsiveness to feedback [ 83 ]. In addition, learning goals should be shared, and co-constructed, through a dialogue [ 50 , 88 , 90 , 92 ]. In fact, relationship-based feedback models emphasize setting shared goals and plans as part of the feedback process [ 68 ].

Many of the studies acknowledge the importance of establishing an action plan and promoting the use of feedback (i.e., feedforward). However, there is yet limited insight on how to best implement strategies that support the use of action plans, improve performance and close learning gaps. In this regard, it is described that delivering feedback without perceiving changes, results in no effect or impact on learning [ 88 ]. To determine if a feedback loop is closed, observing a change in the student's response is necessary. In other words, feedback does not work without repeating the same task [ 68 ], so teachers need to observe subsequent tasks to notice changes [ 88 ]. While feedforward is fundamental to long-term performance, it is shown that more research is needed to determine effective actions to be implemented in the WBL environment to close feedback loops.

Second, there is a need for more knowledge about designing feedback activities in the WBL environment that will generate constructive feedback for learning. WBA is the most frequently reported feedback activity in clinical workplace contexts [ 39 , 46 , 56 , 87 ]. Despite the efforts of some authors to use WBAs as a formative assessment and feedback opportunity, in several studies, a summative component of the WBA was presented as a barrier to actionable feedback [ 33 , 56 ]. Students suggest separating grading from observation and using, for example, the mini-CEX in informal situations [ 33 ]. Several authors also recommend disconnecting the summative components of WBAs to avoid generating emotions that can limit the uptake and use of feedback [ 28 , 93 ]. Other literature recommends purposefully designing a system of assessment using low-stakes data points for feedback and learning. Accordingly, programmatic assessment is a framework that combines both the learning and the decision-making function of assessment [ 94 , 95 ]. Programmatic assessment is a practical approach for implementing low-stakes as a continuum, giving opportunities to close the gap between current and desired performance and having the student as an active agent [ 96 ]. This approach enables the incorporation of low-stakes data points that target student learning [ 93 ] and provide performance-relevant information (i.e., meaningful feedback) based on direct observations during authentic professional activities [ 46 ]. Using low-stakes data points, learners make sense of information about their performance and use it to enhance the quality of their work or performance [ 96 , 97 , 98 ]. Implementing multiple instances of feedback is more effective than providing it once because it promotes closing feedback loops by giving the student opportunities to understand the feedback, make changes, and see if those changes were effective [ 89 ].

Third, the support provided by the teacher is fundamental and should be built into a reliable and long-term relationship, where the teacher must take the role of coach rather than assessor, and students should develop feedback agency and be active in seeking and using feedback to improve performance. Although it is recognized that institutional efforts over the past decades have focused on training teachers to deliver feedback, clinical supervisors' lack of teaching skills is still identified as a barrier to workplace feedback [ 99 ]. In particular, research indicates that clinical teachers lack the skills to transform the information obtained from an observation into constructive feedback [ 100 ]. Students are more likely to use feedback if they consider it credible and constructive [ 93 ] and based on stable relationships [ 93 , 99 , 101 ]. In trusting relationships, feedback can be straightforward and credible, and the likelihood of follow-up and actionable feedback improves [ 83 , 88 ]. Coaching strategies can be enhanced by teachers building an educational alliance that allows for trustworthy relationships or having supervisors with an exclusive coaching role [ 14 , 93 , 102 ].

Last, from a sociocultural perspective, individuals are the main actors in the learning process. Therefore, feedback impacts learning only if students engage and interact with it [ 11 ]. Thus, feedback design and student agency appear to be the main features of effective feedback processes. Accordingly, the present review identified that feedback design is a key feature for effective learning in complex environments such as WBL. Feedback in the workplace must ideally be organized and implemented to align learning outcomes, learning activities, and assessments, allowing learners to learn, practice, and close feedback loops [ 88 ]. To guide students toward performances that reflect long-term learning, an intensive formative learning phase is needed, in which multiple feedback processes are included that shape students´ further learning [ 103 ]. This design would promote student uptake of feedback for subsequent performance [ 1 ].

Strengths and limitations

The strengths of this study are (1) the use of an established framework, the Arksey and O'Malley's framework [ 22 ]. We included the step of socializing the results with stakeholders, which allowed the team to better understand the results from another perspective and offer a realistic look. (2) Using the feedback loop as a theoretical framework strengthened the results and gave a more thorough explanation of the literature regarding feedback processes in the WBL context. (3) our team was diverse and included researchers from different disciplines as well as a librarian.

The present scoping review has several limitations. Although we adhered to the recommended protocols and methodologies, some relevant papers may have been omitted. The research team decided to select original studies and reviews of the literature for the present scoping review. This caused some articles, such as guidelines, perspectives, and narrative papers, to be excluded from the current study.

One of the inclusion criteria was a focus on undergraduate students. However, some papers that incorporated undergraduate and postgraduate participants were included, as these supported the results of this review. Most articles involved medical students. Although the authors did not limit the search to medicine, maybe some articles involving students from other health disciplines needed to be included, considering the search in other databases or journals.

The results give insight in how feedback could be organized within the clinical workplace to promote feedback processes. On a small scale, i.e., in the feedback encounter between a supervisor and a learner, feedback should be organized to allow for follow-up feedback, thus working on required learning and performance goals. On a larger level, i.e., in the clerkship programme or a placement rotation, feedback should be organized through appropriate planning of subsequent tasks and activities.

More insight is needed in designing a closed loop feedback process, in which specific attention is needed in effective feedforward practices. The feedback that stimulates further action and learning requires a safe and trustful work and learning environment. Understanding the relationship between an individual and his or her environment is a challenge for determining the impact of feedback and must be further investigated within clinical WBL environments. Aligning the dimensions of feed-up, feedback and feedforward includes careful attention to teachers’ and students’ feedback literacy to assure that students can act on feedback in a constructive way. In this line, how to develop students' feedback agency within these learning environments needs further research.

Boud D, Molloy E. Rethinking models of feedback for learning: The challenge of design. Assess Eval High Educ. 2013;38:698–712.

Article   Google Scholar  

Henderson M, Ajjawi R, Boud D, Molloy E. Identifying feedback that has impact. In: The Impact of Feedback in Higher Education. Springer International Publishing: Cham; 2019. p. 15–34.

Chapter   Google Scholar  

Winstone N, Carless D. Designing effective feedback processes in higher education: A learning-focused approach. 1st ed. New York: Routledge; 2020.

Google Scholar  

Ajjawi R, Boud D. Assessment & Evaluation in Higher Education Researching feedback dialogue: an interactional analysis approach. 2015. https://doi.org/10.1080/02602938.2015.1102863 .

Carless D. Feedback loops and the longer-term: towards feedback spirals. Assess Eval High Educ. 2019;44:705–14.

Sadler DR. Formative assessment and the design of instructional systems. Instr Sci. 1989;18:119–44.

Hattie J, Timperley H. The Power of Feedback The Meaning of Feedback. Rev Educ Res. 2007;77:81–112.

Zarrinabadi N, Rezazadeh M. Why only feedback? Including feed up and feed forward improves nonlinguistic aspects of L2 writing. Language Teaching Research. 2023;27(3):575–92.

Fisher D, Frey N. Feed up, back, forward. Educ Leadersh. 2009;67:20–5.

Reimann A, Sadler I, Sambell K. What’s in a word? Practices associated with ‘feedforward’ in higher education. Assessment evaluation in higher education. 2019;44:1279–90.

Esterhazy R. Re-conceptualizing Feedback Through a Sociocultural Lens. In: Henderson M, Ajjawi R, Boud D, Molloy E, editors. The Impact of Feedback in Higher Education. Cham: Palgrave Macmillan; 2019. https://doi.org/10.1007/978-3-030-25112-3_5 .

Bransen D, Govaerts MJB, Sluijsmans DMA, Driessen EW. Beyond the self: The role of co-regulation in medical students’ self-regulated learning. Med Educ. 2020;54:234–41.

Ramani S, Könings KD, Ginsburg S, Van Der Vleuten CP. Feedback Redefined: Principles and Practice. J Gen Intern Med. 2019;34:744–53.

Atkinson A, Watling CJ, Brand PL. Feedback and coaching. Eur J Pediatr. 2022;181(2):441–6.

Suhoyo Y, Schonrock-Adema J, Emilia O, Kuks JBM, Cohen-Schotanus JA. Clinical workplace learning: perceived learning value of individual and group feedback in a collectivistic culture. BMC Med Educ. 2018;18:79.

Bowen L, Marshall M, Murdoch-Eaton D. Medical Student Perceptions of Feedback and Feedback Behaviors Within the Context of the “Educational Alliance.” Acad Med. 2017;92:1303–12.

Bok HGJ, Teunissen PW, Spruijt A, Fokkema JPI, van Beukelen P, Jaarsma DADC, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47:282–91.

Al-Kadri HM, Al-Kadi MT, Van Der Vleuten CPM. Workplace-based assessment and students’ approaches to learning: A qualitative inquiry. Med Teach. 2013;35(SUPPL):1.

Dennis AA, Foy MJ, Monrouxe LV, Rees CE. Exploring trainer and trainee emotional talk in narratives about workplace-based feedback processes. Adv Health Sci Educ. 2018;23:75–93.

Watling C, LaDonna KA, Lingard L, Voyer S, Hatala R. ‘Sometimes the work just needs to be done’: socio-cultural influences on direct observation in medical training. Med Educ. 2016;50:1054–64.

Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. Feedback for Learners in Medical Education: What is Known? A Scoping Review Academic Medicine. 2017;92:1346–54.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169:467–73.

Colquhoun HL, Levac D, O’brien KK, Straus S, Tricco AC, Perrier L, et al. Scoping reviews: time for clarity in definition methods and reporting. J Clin Epidemiol. 2014;67:1291–4.

StArt - State of Art through Systematic Review. 2013.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: Advancing the methodology. Implementation Science. 2010;5:1–9.

Peters MDJ, BPharm CMGHK, Parker PMD, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc. 2015;13:141–6.

Bing-You R, Varaklis K, Hayes V, Trowbridge R, Kemp H, McKelvy D, et al. The Feedback Tango: An Integrative Review and Analysis of the Content of the Teacher-Learner Feedback Exchange. Acad Med. 2018;93:657–63.

Ossenberg C, Henderson A, Mitchell M. What attributes guide best practice for effective feedback? A scoping review. Adv Health Sci Educ. 2019;24:383–401.

Spooner M, Duane C, Uygur J, Smyth E, Marron B, Murphy PJ, et al. Self -regulatory learning theory as a lens on how undergraduate and postgraduate learners respond to feedback: A BEME scoping review: BEME Guide No. 66. Med Teach. 2022;44:3–18.

Long S, Rodriguez C, St-Onge C, Tellier PP, Torabi N, Young M. Factors affecting perceived credibility of assessment in medical education: A scoping review. Adv Health Sci Educ. 2022;27:229–62.

Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. Feedback for Learners in Medical Education: What is Known? A Scoping Review: Lippincott Williams and Wilkins; 2017.

Schopper H, Rosenbaum M, Axelson R. “I wish someone watched me interview:” medical student insight into observation and feedback as a method for teaching communication skills during the clinical years. BMC Med Educ. 2016;16:286.

Crommelinck M, Anseel F. Understanding and encouraging feedback-seeking behaviour: a literature review. Med Educ. 2013;47:232–41.

Adamson E, Kinga L, Foy L, McLeodb M, Traynor J, Watson W, et al. Feedback in clinical practice: Enhancing the students’ experience through action research. Nurse Educ Pract. 2018;31:48–53.

Al-Mously N, Nabil NM, Al-Babtain SA, et al. Undergraduate medical students’ perceptions on the quality of feedback received during clinical rotations. Med Teach. 2014;36(Supplement 1):S17-23.

Bates J, Konkin J, Suddards C, Dobson S, Pratt D. Student perceptions of assessment and feedback in longitudinal integrated clerkships. Med Educ. 2013;47:362–74.

Bennett AJ, Goldenhar LM, Stanford K. Utilization of a Formative Evaluation Card in a Psychiatry Clerkship. Acad Psychiatry. 2006;30:319–24.

Bok HG, Jaarsma DA, Spruijt A, Van Beukelen P, Van Der Vleuten CP, Teunissen PW, et al. Feedback-giving behaviour in performance evaluations during clinical clerkships. Med Teach. 2016;38:88–95.

Bok HG, Teunissen PW, Spruijt A, Fokkema JP, van Beukelen P, Jaarsma DA, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47:282–91.

Calleja P, Harvey T, Fox A, Carmichael M, et al. Feedback and clinical practice improvement: A tool to assist workplace supervisors and students. Nurse Educ Pract. 2016;17:167–73.

Carey EG, Wu C, Hur ES, Hasday SJ, Rosculet NP, Kemp MT, et al. Evaluation of Feedback Systems for the Third-Year Surgical Clerkship. J Surg Educ. 2017;74:787–93.

Daelmans HE, Overmeer RM, Van der Hem-Stokroos HH, Scherpbier AJ, Stehouwer CD, van der Vleuten CP. In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Medical education. 2006;40(1):51–8.

DeWitt D, Carline J, Paauw D, Pangaro L. Pilot study of a ’RIME’-based tool for giving feedback in a multi-specialty longitudinal clerkship. Med Educ. 2008;42:1205–9.

Dolan BM, O’Brien CL, Green MM. Including Entrustment Language in an Assessment Form May Improve Constructive Feedback for Student Clinical Skills. Med Sci Educ. 2017;27:461–4.

Duijn CC, Welink LS, Mandoki M, Ten Cate OT, Kremer WD, Bok HG. Am I ready for it? Students’ perceptions of meaningful feedback on entrustable professional activities. Perspectives on medical education. 2017;6:256–64.

Elnicki DM, Zalenski D. Integrating medical students’ goals, self-assessment and preceptor feedback in an ambulatory clerkship. Teach Learn Med. 2013;25:285–91.

Embo MP, Driessen EW, Valcke M, Van der Vleuten CP. Assessment and feedback to facilitate self-directed learning in clinical practice of Midwifery students. Medical teacher. 2010;32(7):e263-9.

Eva KW, Armson H, Holmboe E, Lockyer J, Loney E, Mann K, et al. Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ. 2012;17:15–26.

Farrell L, Bourgeois-Law G, Ajjawi R, Regehr G. An autoethnographic exploration of the use of goal oriented feedback to enhance brief clinical teaching encounters. Adv Health Sci Educ. 2017;22:91–104.

Fernando N, Cleland J, McKenzie H, Cassar K. Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Med Educ. 2008;42:89–95.

Garino A. Ready, willing and able: a model to explain successful use of feedback. Adv Health Sci Educ. 2020;25:337–61.

Garner MS, Gusberg RJ, Kim AW. The positive effect of immediate feedback on medical student education during the surgical clerkship. J Surg Educ. 2014;71:391–7.

Bing-You R, Hayes V, Palka T, Ford M, Trowbridge R. The Art (and Artifice) of Seeking Feedback: Clerkship Students’ Approaches to Asking for Feedback. Acad Med. 2018;93:1218–26.

Haffling AC, Beckman A, Edgren G. Structured feedback to undergraduate medical students: 3 years’ experience of an assessment tool. Medical teacher. 2011;33(7):e349-57.

Harrison CJ, Könings KD, Dannefer EF, Schuwirth LWTT, Wass V, van der Vleuten CPMM. Factors influencing students’ receptivity to formative feedback emerging from different assessment cultures. Perspect Med Educ. 2016;5:276–84.

Harrison CJ, Könings KD, Schuwirth LW, Wass V, Van der Vleuten CP, Konings KD, et al. Changing the culture of assessment: the dominance of the summative assessment paradigm. BMC medical education. 2017;17:1–4.

Harvey P, Radomski N, O’Connor D. Written feedback and continuity of learning in a geographically distributed medical education program. Medical teacher. 2013;35(12):1009–13.

Hochberg M, Berman R, Ogilvie J, Yingling S, Lee S, Pusic M, et al. Midclerkship feedback in the surgical clerkship: the “Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills” application utilizing learner self-assessment. Am J Surg. 2017;213:212–6.

Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the mini clinical evaluation exercise. Journal of general internal medicine. 2004;19(5):558–61.

Tai JHM, Canny BJ, Haines TP, Molloy EK. The role of peer-assisted learning in building evaluative judgement: opportunities in clinical medical education. Adv Health Sci Educ. 2016;21:659–76.

Johnson CE, Keating JL, Molloy EK. Psychological safety in feedback: What does it look like and how can educators work with learners to foster it? Med Educ. 2020;54:559–70.

Joshi A, Generalla J, Thompson B, Haidet P. Facilitating the Feedback Process on a Clinical Clerkship Using a Smartphone Application. Acad Psychiatry. 2017;41:651–5.

Kiger ME, Riley C, Stolfi A, Morrison S, Burke A, Lockspeiser T. Use of Individualized Learning Plans to Facilitate Feedback Among Medical Students. Teach Learn Med. 2020;32:399–409.

Kogan J, Shea J. Implementing feedback cards in core clerkships. Med Educ. 2008;42:1071–9.

Lefroy J, Walters B, Molyneux A, Smithson S. Can learning from workplace feedback be enhanced by reflective writing? A realist evaluation in UK undergraduate medical education. Educ Prim Care. 2021;32:326–35.

McGinness HT, Caldwell PHY, Gunasekera H, Scott KM. ‘Every Human Interaction Requires a Bit of Give and Take’: Medical Students’ Approaches to Pursuing Feedback in the Clinical Setting. Teach Learn Med. 2022. https://doi.org/10.1080/10401334.2022.2084401 .

Noble C, Billett S, Armit L, Collier L, Hilder J, Sly C, et al. ``It’s yours to take{’’}: generating learner feedback literacy in the workplace. Adv Health Sci Educ. 2020;25:55–74.

Ogburn T, Espey E. The R-I-M-E method for evaluation of medical students on an obstetrics and gynecology clerkship. Am J Obstet Gynecol. 2003;189:666–9.

Po O, Reznik M, Greenberg L. Improving a medical student feedback with a clinical encounter card. Ambul Pediatr. 2007;7:449–52.

Parkes J, Abercrombie S, McCarty T, Parkes J, Abercrombie S, McCarty T. Feedback sandwiches affect perceptions but not performance. Adv Health Sci Educ. 2013;18:397–407.

Paukert JL, Richards ML, Olney C. An encounter card system for increasing feedback to students. Am J Surg. 2002;183:300–4.

Rassos J, Melvin LJ, Panisko D, Kulasegaram K, Kuper A. Unearthing Faculty and Trainee Perspectives of Feedback in Internal Medicine: the Oral Case Presentation as a Model. J Gen Intern Med. 2019;34:2107–13.

Rizan C, Elsey C, Lemon T, Grant A, Monrouxe L. Feedback in action within bedside teaching encounters: a video ethnographic study. Med Educ. 2014;48:902–20.

Robertson AC, Fowler LC. Medical student perceptions of learner-initiated feedback using a mobile web application. Journal of medical education and curricular development. 2017;4:2382120517746384.

Scheidt PC, Lazoritz S, Ebbeling WL, Figelman AR, Moessner HF, Singer JE. Evaluation of system providing feedback to students on videotaped patient encounters. J Med Educ. 1986;61(7):585–90.

Sokol-Hessner L, Shea J, Kogan J. The open-ended comment space for action plans on core clerkship students’ encounter cards: what gets written? Acad Med. 2010;85:S110–4.

Sox CM, Dell M, Phillipi CA, Cabral HJ, Vargas G, Lewin LO. Feedback on oral presentations during pediatric clerkships: a randomized controlled trial. Pediatrics. 2014;134:965–71.

Spickard A, Gigante J, Stein G, Denny JC. Automatic capture of student notes to augment mentor feedback and student performance on patient write-ups. J Gen Intern Med. 2008;23:979–84.

Suhoyo Y, Van Hell EA, Kerdijk W, Emilia O, Schönrock-Adema J, Kuks JB, et al. nfluence of feedback characteristics on perceived learning value of feedback in clerkships: does culture matter? BMC medical education. 2017;17:1–7.

Torre DM, Simpson D, Sebastian JL, Elnicki DM. Learning/feedback activities and high-quality teaching: perceptions of third-year medical students during an inpatient rotation. Acad Med. 2005;80:950–4.

Urquhart LM, Ker JS, Rees CE. Exploring the influence of context on feedback at medical school: a video-ethnography study. Adv Health Sci Educ. 2018;23:159–86.

Watling C, Driessen E, van der Vleuten C, Lingard L. Learning culture and feedback: an international study of medical athletes and musicians. Med Educ. 2014;48:713–23.

Watling C, Driessen E, van der Vleuten C, Vanstone M, Lingard L. Beyond individualism: Professional culture and its influence on feedback. Med Educ. 2013;47:585–94.

Soemantri D, Dodds A, Mccoll G. Examining the nature of feedback within the Mini Clinical Evaluation Exercise (Mini-CEX): an analysis of 1427 Mini-CEX assessment forms. GMS J Med Educ. 2018;35:Doc47.

Van De Ridder JMMM, Stokking KM, McGaghie WC, ten Cate OTJ, van der Ridder JM, Stokking KM, et al. What is feedback in clinical education? Med Educ. 2008;42:189–97.

van de Ridder JMMM, McGaghie WC, Stokking KM, ten Cate OTJJ. Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review. Med Educ. 2015;49:658–73.

Boud D. Feedback: ensuring that it leads to enhanced learning. Clin Teach. 2015. https://doi.org/10.1111/tct.12345 .

Brehaut J, Colquhoun H, Eva K, Carrol K, Sales A, Michie S, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164:435–41.

Ende J. Feedback in clinical medical education. J Am Med Assoc. 1983;250:777–81.

Cantillon P, Sargeant J. Giving feedback in clinical settings. Br Med J. 2008;337(7681):1292–4.

Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29:855–71.

Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Med Educ. 2019;53:76–85.

van der Vleuten CPM, Schuwirth LWT, Driessen EW, Dijkstra J, Tigelaar D, Baartman LKJ, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34:205–14.

Schuwirth LWT, der Vleuten CPM. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33:478–85.

Schut S, Driessen E, van Tartwijk J, van der Vleuten C, Heeneman S. Stakes in the eye of the beholder: an international study of learners’ perceptions within programmatic assessment. Med Educ. 2018;52:654–63.

Henderson M, Boud D, Molloy E, Dawson P, Phillips M, Ryan T, Mahoney MP. Feedback for learning. Closing the assessment loop. Framework for effective learning. Canberra, Australia: Australian Government, Department for Education and Training; 2018.

Heeneman S, Pool AO, Schuwirth LWT, van der Vleuten CPM, Driessen EW, Oudkerk A, et al. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 2015;49:487–98.

Lefroy J, Watling C, Teunissen P, Brand P, Watling C. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ. 2015;4:284–99.

Ramani S, Krackov SK. Twelve tips for giving feedback effectively in the clinical environment. Med Teach. 2012;34:787–91.

Telio S, Ajjawi R, Regehr G. The, “Educational Alliance” as a Framework for Reconceptualizing Feedback in Medical Education. Acad Med. 2015;90:609–14.

Lockyer J, Armson H, Könings KD, Lee-Krueger RC, des Ordons AR, Ramani S, et al. In-the-Moment Feedback and Coaching: Improving R2C2 for a New Context. J Grad Med Educ. 2020;12:27–35.

Black P, Wiliam D. Developing the theory of formative assessment. Educ Assess Eval Account. 2009;21:5–31.

Download references

Author information

Authors and affiliations.

Department of Health Sciences, Faculty of Medicine, Pontificia Universidad Católica de Chile, Avenida Vicuña Mackenna 4860, Macul, Santiago, Chile

Javiera Fuentes-Cimma & Ignacio Villagran

School of Health Professions Education, Maastricht University, Maastricht, Netherlands

Javiera Fuentes-Cimma & Lorena Isbej

Rotterdam University of Applied Sciences, Rotterdam, Netherlands

Dominique Sluijsmans

Centre for Medical and Health Profession Education, Department of Gastroenterology, Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile

Arnoldo Riquelme

School of Dentistry, Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile

Lorena Isbej

Sistema de Bibliotecas UC (SIBUC), Pontificia Universidad Católica de Chile, Santiago, Chile

María Teresa Olivares-Labbe

Department of Pathology, Faculty of Health, Medicine and Health Sciences, Maastricht University, Maastricht, Netherlands

Sylvia Heeneman

You can also search for this author in PubMed   Google Scholar

Contributions

J.F-C, D.S, and S.H. made substantial contributions to the conception and design of the work. M.O-L contributed to the identification of studies. J.F-C, I.V, A.R, and L.I. made substantial contributions to the screening, reliability, and data analysis. J.F-C. wrote th e main manuscript text. All authors reviewed the manuscript.

Corresponding author

Correspondence to Javiera Fuentes-Cimma .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1, supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Fuentes-Cimma, J., Sluijsmans, D., Riquelme, A. et al. Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review. BMC Med Educ 24 , 440 (2024). https://doi.org/10.1186/s12909-024-05439-6

Download citation

Received : 25 September 2023

Accepted : 17 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1186/s12909-024-05439-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical clerkship
• Feedback processes
• Feedforward
• Formative feedback
• Health professions
• Undergraduate medical education
• Undergraduate healthcare education
• Workplace learning

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Hybrid intelligence failure analysis for industry 4.0: a literature review and future prospective

• Open access
• Published: 22 April 2024

Cite this article

You have full access to this open access article

• Mahdi Mokhtarzadeh   ORCID: orcid.org/0000-0002-0348-6718 1 , 2 ,
• Jorge Rodríguez-Echeverría 1 , 2 , 3 ,
• Ivana Semanjski 1 , 2 &
• Sidharta Gautama 1 , 2  

Industry 4.0 and advanced technology, such as sensors and human–machine cooperation, provide new possibilities for infusing intelligence into failure analysis. Failure analysis is the process of identifying (potential) failures and determining their causes and effects to enhance reliability and manufacturing quality. Proactive methodologies, such as failure mode and effects analysis (FMEA), and reactive methodologies, such as root cause analysis (RCA) and fault tree analysis (FTA), are used to analyze failures before and after their occurrence. This paper focused on failure analysis methodologies intelligentization literature applied to FMEA, RCA, and FTA to provide insights into expert-driven, data-driven, and hybrid intelligence failure analysis advancements. Types of data to establish an intelligence failure analysis, tools to find a failure’s causes and effects, e.g., Bayesian networks, and managerial insights are discussed. This literature review, along with the analyses within it, assists failure and quality analysts in developing effective hybrid intelligence failure analysis methodologies that leverage the strengths of both proactive and reactive methods.

Avoid common mistakes on your manuscript.

Introduction

Failure analysis entails activities to identify, categorize, and prioritize (potential) failures and determine causes and effects of each failure and failure propagation and interdependencies (Rausand & Øien, 1996 ). Failure analysis significance in manufacturing has grown since Industry 3.0 to mitigate defects and/or failures in production processes, thereby maximizing reliability and quality and minimizing production interruptions, associated risks, and costs (Wu et al., 2021 ; Ebeling, 2019 ).

Failure analysis methodologies have been supported by mathematical, statistical, and graph theories and tools, including MCDM theory, fuzzy theory, six-sigma, SPC, DOE, simulation, Pareto charts, and analysis of mean and variance (Oliveira et al., 2021 ; Huang et al., 2020 ; Tari & Sabater, 2004 ). Industry 4.0 is driven by (real-time) data from sensors, the Internet of Things (IoT), such as Internet-enabled machines and tools, and artificial intelligence (AI). Advances in artificial intelligence theory and technology have brought new tools to strengthen failure analysis methodologies (Oztemel & Gursev, 2020 ). Examples of tools include Bayesian networks (BNs), case-based reasoning (CBR), neural networks, classifications, clusterings algorithms, principal component analysis (PCA), deep learning, decision trees, and ontology-driven methods (Zheng et al., 2021 ). These Industry 4.0 advancments enable more efficient data collection and analysis, enhancing predictive capabilities, increasing efficiency and automation, and improving collaboration and knowledge sharing.

Failure analysis methodologies can be categorized into expert-driven, data-driven, and hybrid ones. Expert-driven failure analysis methods rely on experts’ knowledge and experience (Yucesan et al., 2021 ; Huang et al., 2020 ). This approach is useful when the data is limited or when there is a high degree of uncertainty. Expert-driven methods are also useful when the failure structure is complex and difficult to understand. However, this approach is limited by the availability and expertise of the experts, and is prone to bias and subjective interpretations (Liu et al., 2013 ).

Data-driven failure analysis methods, on the other hand, rely on statistical analysis and machine learning algorithms to identify patterns in the data and predict the causes of the failure (Zhang et al., 2023 ; Mazzoleni et al., 2017 ). This approach is useful when there is a large amount of data available and when the failure structure is well-defined. However, data-driven methods is limited by the quality and completeness of the data (Oliveira et al., 2021 ).

Until recently, most tools have focused on replacing humans with artificial intelligence (Yang et al., 2020 ; Filz et al., 2021b ), which causes them to remove human intellect and capabilities from intelligence systems. Hybrid intelligence creates hybrid human–machine intelligence systems, in which humans and machines collaborate synergistically, proactively, and purposefully to augment human intellect and capabilities rather than replace them with machine intellect and capabilities to achieve shared goals (Akata et al., 2020 ).

Collaboration between humans and machines can enhance the failure analysis process, allowing for analyses that were previously unattainable by either humans or machines alone. Thus, hybrid failure analysis provides a more comprehensive analysis of the failure by incorporating strengths of both expert-driven and data-driven approaches to identify the most likely causes and effects of failures (Dellermann et al., 2019 ; van der Aalst, 2021 ).

Benefits from a smart failure analysis may include reduced costs and production stoppages, improved use of human resources, improved use of knowledge, improved failure, root causes, and effects identification, and real-time failure analysis. Yet, only a few studies specifically addressed hybrid failure analysis (Chhetri et al., 2023 ). A case example of hybrid expert data-driven failure analysis involves using data from similar product assemblies to construct a Bayesian network for proccess failure mode and effects analysis (pFMEA), while also incorporating expert knowledge as constraints based on the specific product being analyzed (Chhetri et al., 2023 ).

Over the past few years, several literature reviews, as reported in Section Literature review , have been accomplished under different outlooks in relation to different failure analysis methodologies including failure mode and effects analysis (FMEA), root cause analysis (RCA), and fault tree analysis (FTA). Currently, most existing literature does not systematically summarize the research status of these failure analysis methodologies from the perspective of Industry 4.0 and (hybrid) intelligence failure analysis with the benefits from new technologies. Therefore, this study aims to review, categorize, and analyze the literature of these three general failure analysis methodologies in production systems. The objective is to provide researchers with a comprehensive overview of these methodologies, with a specific focus on hybrid intelligence, and its benefits for quality issues in production. We address two questions "How can failure analysis methodologies benefit from hybrid intelligence?" and "Which tools are suitable for a good fusion of human and machine intelligence?" Consequently, the main contributions of this study to the failure analysis literature are as follows:

Analysis of 86 papers out of 7113 papers from FMEA, RCA, and FTA with respect to methods and data types that might be useful for a hybrid intelligence failure analysis.

Identification of data and methods to construct and detect multiple failures within different research related to FMEA, RCA, and FTA methodologies.

Identification of the most effective methods for analyzing failures, identifying their sources and effects, and assessing related risks.

Proposal of a categorization of research based on the levels of automation/intelligence, along with the identification of limitations in current research in this regard.

Provision of hybrid intelligent failure analysis future research, along with other future directions such as future research on failure propagation and correlation.

The plan of this paper is as follows. Section Literature review briefly introduces related literature reviews on FMEA, RCA, and FTA. A brief description of other failure analysis methodologies is also provided. Section Research methodology presents our review methodology, including the review scope and protocols, defining both our primary and secondary questions, and the criteria for selecting journals and papers to be reviewed. A bibliography summary of the selected papers is provided. Literature has been categorized in Section Literature categorization based on the four general steps of a failure analysis methodology, involving failure structure detection, failure event probability detection, failure risk analysis, and outputs. Managerial insights, limitations, and future research are discussed in Section Managerial insights, limitations, and future research . This assists researchers with applications and complexity, levels of intelligence, how knowledge is introduced into the failure analysis. A more in-depth discussion of hybrid intelligence, failure propagation and correlation, hybrid methodologies, and other areas of future research is also included. Conclusions are presented in Section Conclusion .

Literature review

General and industry/field-specific failure analysis methodologies have been developed over the last few decades. In this section, we provide useful review papers regarding FMEA, RCA, and FTA, which are the focus of our paper. Additionally, some other general and industry/field-specific failure analysis methodologies are briefly discussed.

FMEA is a most commonly used bottom-up proactive qualitative methodologies for potential quality failure analysis (Huang et al., 2020 ; Stamatis, 2003 ). Among its extensions, process FMEA (pFMEA) proactively identifies potential quality failures in production processes such as assembly lines (Johnson & Khan, 2003 ). Typically, (p)FMEA uses expert knowledge to determine potential failures, effects, and causes, and to prioritize the failures based on the risk priority number (RPN). RPN is a product of severity, occurrence, and detection rates for each failure (Wu et al., 2021 ). Some of the FMEA shortcomings include time-consuming, subjectivity, inability to determine multiple failures, and failure propagation and interdependency (Liu et al., 2013 ).

RCA is a bottom-up reactive quantitative methodology that determines the causal mechanism behind a failure to prevent the recurrence of the failure in manufacturing processes (Oliveira et al., 2023 ). To locate, identify, and/or explain the reasons behind the occurrence of root causes, RCA utilizes statistical analysis tools, such as regression, statistical process control (SPC), design of experiments (DOE), PCA, and cause-effect diagram (Williams, 2001 ). Limited ability to predict future failures and difficulty in identifying complex or systemic issues are among RCA limitations (Yuniarto, 2012 ).

FTA is a top-down reactive graphical method to model failure propagation through a system, i.e., how component failures lead to system failures (Kumar & Kaushik, 2020 ). FTA uses qualitative data to model the structure of a system and quantitative data, including probabilities and graph methods such as minimal cut/path sets, binary decision diagrams, simulation, and BNs, to model failures propagation. Requiring extensive data, limited ability to identify contributing factors, and time-consuming are among the FTA limitations (Ruijters & Stoelinga, 2015 ).

In recent years, several literature reviews have been conducted on failure analysis methodologies, exploring various perspectives and approaches. Liu et al. ( 2013 ) reviewed FMEA risk evaluation tools including rule-based systems, mathematical programming, and multi-criteria decision-making (MCDM). They concluded that artificial intelligence and MCDM tools, particularly fuzzy rule base systems, grey theory, and cost-based models, are the most cited tools to prioritize risks in FMEA. Liu et al. ( 2019a ) and Dabous et al. ( 2021 ) reviewed MCDM tools application for FMEA. Papers with different areas, automotive, electronics, machinery and equipment, and steel manufacturing were considered. The most used MCDM tools, namely technique for order of preference by similarity to ideal solution (TOPSIS), analytic hierarchy process (AHP), decision-making trial and evaluation laboratory (DEMATEL), and grey theory, were identified.

Spreafico et al. ( 2017 ) provided a FMEA/Failure mode, effects, and criticality analysis (FMECA) critical review by classifying FMEA/FMECA limitations and issues and reviewing suggested improvements and solutions for the limitations. FMEA issues were classified into four groups of applicabilities, cause and effect analysis, risk analysis, and problem-solving. Main problems (and solutions) are being time-consuming (integration with design tools, using more structured templates, and automation), lack of secondary effects modeling (integration with other tools such as FTA, BN, and Petri net), being too subjective (using statistical evaluation and cost-based approaches), and lack in evaluating the implementation of a solution (using the improved presentation of the results and integration with other tools such as maintenance management tools), respectively. Huang et al. ( 2020 ) provided a bibliographic analysis of FMEA and its applications in manufacturing, marine, healthcare, aerospace, and electronics. Wu et al. ( 2021 ) sorted out potential failure mode identification approaches such as analyzing entry point for system failure mode identification, failure mode recognition tools, and failure mode specification description. Then a review of FMEA risk assessment tools had been provided.

Oliveira et al. ( 2023 ) reviewed automatic RCA literature in manufacturing. Different data types, location-time, physical, and log-action, that are usually used were identified. Industries with the most use of RCA are ranked, semiconductor, chemical, automotive, and others. Then different tools used to automate RCA, including decision trees, regression models, classification methods, clustering methods, neural networks, BNs, PCA, statistical tests, and control charts, were discussed. Ruijters and Stoelinga ( 2015 ) provided FTA qualitative and quantitative analysis methods. Also, different types of FTA, standard FTA, dynamic FTA, and other extensions, were discussed. Zhu and Zhang ( 2022 ) also reviewed dynamic FTA. Cai et al. ( 2017 ) reviewed the application of BN in fault diagnosis. First, an overview of BN types (static, dynamic, and object-oriented), structure modeling, parameters modeling, and interference has been provided. Then applicability of BN for fault identification in process, energy, structural, manufacturing, and network systems has been discussed. BN verification and validation methods are provided. Future prospects including integration of big data with BN, real-time fault diagnosis BN inference algorithms, and hybrid fault diagnosis methods are finally resulted. More relevant BN reviews include BN application in reliability (Insua et al., 2020 ) and safety and risk assessments (Kabir & Papadopoulos, 2019 ).

The integration of FMEA, RCA, and FTA holds immense potential for quality and production managers to minimize failures and enhance system efficiency. By capitalizing on the unique strengths of each approach, the integration of these failure analysis methodologies enables a more comprehensive and effective examination of failures. However, existing studies and literature reviews have predominantly focused on individual methodologies, leading to a lack of integration and limited familiarity with three approaches among engineers and industry experts. To address this gap and promote the integration of them, this study aims to review the progress of intelligence failure analysis within FMEA, RCA, and FTA.

Other general failure analysis methodologies include, but are not limited to, the following methodologies. Event Tree Analysis, similar to FTA, is a graphical representation that models the progression of events following an initiating event, helping to analyze the potential consequences (Ruijters & Stoelinga, 2015 ). Bow-Tie Analysis, usually used in risk management, visualizes the relationship between different potential causes of a hazard and their possible consequences (Khakzad et al., 2012 ). Human Reliability Analysis focuses on assessing the probability of human error and its potential impact on systems and processes (French et al., 2011 ). The Fishbone Diagram visually represents potential causes of a problem to identify root causes by categorizing them into specific factors like people, process, equipment, materials, etc.

There are also industry-specific methodologies, including but not limited to the following ones. Electrostatic Discharge (ESD) Failure Analysis focuses on identifying failures caused by electrostatic discharge, a common concern in the electronics industry. Hazard and Operability Study is widely used in the chemical industry to examine deviations from the design intent and identify potential hazards and operability issues. Incident Response and Post-Incident Analysis, in the IT industry, is used for analyzing and responding to security incidents, with a focus on preventing future occurrences. Hazard Analysis and Critical Control Points is a systematic preventive approach to food safety that identifies, evaluates, and controls hazards throughout the production process. Maximum credible accident analysis assesses and mitigates the most severe accidents that could occur in high-risk industries. For more information on industry-specific methodologies, an interested reader may consult the paper on that industry, as they are wide and out of the scope of this paper for deep discussion.

Our review focuses on the historical progress of (hybrid) intelligence failure analysis to identify and classify methodologies and tools used within them. In Industry 4.0, (hybrid) intelligence failure analysis can contribute to improve quality management and automate quality through an improved human cyber-physical experience. Different from the abovementioned reviews, the purpose of our study is to provide a rich comprehensive understanding of the recent developments in these methodologies from industry 4.0 and hybrid intelligence, the benefits of making them intelligent, i.e., (augmented) automatic and/or data-driven, and their limitations.

Research methodology

A systematic literature review analyses a particular knowledge domain’s body of literature to provide insights into research and practice and identify research gaps (Thomé et al., 2016 ). This section discusses our review scope and protocols, defining both our primary and secondary questions, and the criteria for selecting journals and papers to be reviewed. A bibliography analysis of the selected papers is also presented, including distributions by year, affiliation, and journals.

Review scope and protocol

We follow Thomé et al. ( 2016 ) 8-step literature review methodology to assure a rigorous literature review of intelligence, automated/data-driven, failure analysis methodology for Industry 4.0.

In Step 1, our (hybrid) intelligence failure analysis problem is planned and formulated by identifying the needs, scope, and questions for this research. Our initial need for this literature review comes from a relevant industrial project entitled "assembly quality management using system intelligence" which aims to reduce the quality failures in assembly lines. The trend towards automated and data-driven methodologies in recent years signifies the need for this systematic literature review. Thus, three general failure analysis methodologies, FMEA, RCA, and FTA, are reviewed with respect to tools to make them intelligent and to derive benefits from hybrid intelligence.

Our primary questions are as follows. (i) What are the failure analysis general methodologies and what tools have been used to make them intelligent? (ii) How these methodologies may benefit from hybrid intelligence? (iii) What are the strengths and weaknesses of these methodologies and tools? Our secondary questions are as follows. (i) How intelligent are these tools? (ii) What types of data do they use? Which tools allow a good fusion of human and machine intelligence? (iii) How well do they identify the root causes of failures? (iv) What are the possible future prospectives?

Distribution of papers by year and affliation

Step 2 concerns searching the literature by selecting relevant journals, databases, keywords, and criteria to include or exclude papers. We select the SCOPUS database to scan the relevant paper from 1990 to the first half of 2022. SCOPUS contains all high-quality English publications and covers other databases such as ScienceDirect and IEEE Xplore. A two-level keyword structure is used. The first level retrieves all papers that have either failure mode and effect analysis, FMEA, failure mode and effects and criticality analysis, FMECA, fault tree analysis, FTA, event tree analysis, ETA, root cause analysis, RCA, failure identification, failure analysis, or fault diagnosis in the title, abstract, and/or keywords. The second level limits the retrieved paper by the first level keywords to papers that have either Bayesian network, BN, automated, automatic, automation, smart, intelligence or data-driven in the title, abstract, and/or keywords.

To ensure the scientific rigor of our literature review process, we have removed papers that met at least one of the following criteria: Publications with concise and/or ambiguous information that would make it impossible to re-implement the tools and methodologies described in the paper later on. Publications in low-level journals, i.e., journals in the third quarter (Q3) or lower in the Scimago Journal & Country Rank. Papers with subject areas that are irrelevant to our research topic, such as physics and astronomy.

Steps 3 and 4 involve gathering data and evaluating data quality. We download papers and check their sources according to exclusion criteria. Step 5 concerns data analysis. Step 6 focuses on interpreting the data. The final selected papers are analyzed and interpreted in Section Managerial insights, limitations, and future research . Step 7 involves preparing the results and report. Step 8 requires the review to be updated continuously.

Discussion and statistical analysis

Here is a bibliometric analysis of our literature review. About 15,977 papers were found in our first search. By excluding criteria, we shortened the search to 7113. Then, we checked the titles of 7113 papers including 4359 conference and 2754 journal papers. We downloaded 1,203 papers to read their abstracts and skim their bodies. Then, 1114 low-quality/irrelevant papers were excluded. The remaining 86 high-quality papers were examined for this study.

Distributions of papers by year and affiliation are shown in Fig. 1 . 28 countries have contributed in total. Most affiliations are in advanced countries including China, Germany, and the UK. Surprisingly, we found no publications from Japan and only five from the USA. Only one papers had been published between 1990 and 1999 because of limited data and technology, e.g., sensors and industrial cameras. A slow growth observed between 2000 and 2014 coincides with the technology advancement and Industry 4.0 emergence. The advanced technology and researchers focus on Industry 4.0 have led to significant growth every year since 2015. Worth to note that 2022 information is incomplete because this research has been conducted in the middle of 2022. We expect more publications, at least equal to 2021, for 2022.

Papers distribution by journal is in Fig. 2 . 58 journals and conferences have contributed. Journals with a focus on production and quality, e.g., International Journal of Production Research , have published most papers. Technology-focused journals, e.g., IEEE Access , also have contributed.

Distribution of papers by journal

Literature categorization

Selected papers are now categorized based on the four general steps of a failure analysis methodology, involving failure structure detection, failure event probabilities detection, failure risk analysis, and outputs. Then, a statistical analysis of these categorizations is provided.

These four steps of a failure analysis methodology are illustrated in Fig. 3 . The first two steps deal with input data. In step 1, the failure structure is identified, encompassing all (possible) failures, the failure propagation structure, failure interdependency, and causes and effects. Step 2 involves detecting event probabilities in a failure structure. For example, classical FMEA scores each failure with severity, occurrence, and detection rates.

Four general steps of a failure analysis methodology

To analyze failures in a (production) system, data should be collected to identify the failure structure and detect failures. Reactive methodologies, such as RCA, are data-driven and typically gather available data in a system, while proactive methodologies, such as FMEA, are expert-driven and gather data through expert knowledge. However, a (hybrid) intelligence failure analysis methodology should take advantage of both advanced technologies, such as sensors and Internet-enabled machines and tools, and experts to automatically gather required data, combining proactive and reactive approaches, and providing highly reliable analyses and solutions.

In step 3, all input data are processed to determine the associated risk value with each failure, and the most probable causes (usually based on an observed or potential effect). Typically, a main tool, such as Bayesian networks, neural rule-based systems, statistical analysis, or expert analysis, is used to determine root causes, classify failures, and/or rank failures.

Step 4 outputs results that may include failures and sources, reasons behind the sources, and mitigation actions. The output of this tool is post-processed to provide possible solutions and information that is explainable and easy to use for both humans and machines.

Steps 1: failure structure

Failure structure identification is the first step in a failure analysis methodology. (Potential) failures, causes, effects, and/or failure interdependency are identified. We categorize the literature to develop a (hybrid) intelligence failure methodology to identify failure structure, causes, effects, interdependencies, and relationships between failures, failures and causes, and failures and effects.

Traditionally, experts have defined failure structures by analyzing causes, effects, and the interdependency of failures. However, recent studies have explored alternative approaches to identifying failure structures, leveraging available data sources such as problem-solving databases, design forms, and process descriptions. Problem-solving databases include quality issue records, maintenance records, failure analysis records, and CBR databases. These records could be stored in structured databases and sheets, or unstructured texts. Design forms may include design FMEA forms, reliability characteristics, and product quality characteristics. Process descriptions may include operations, stations, and key operational characteristics. Moreover, simulation can be used to generate failures, causes, and effects (Snooke & Price, 2012 ). Design forms and process descriptions are generated by experts, usually for other purposes, and are re-used for failure analysis. Problem-solving databases could be generated by experts, such as previous FMEAs, or by an automated failure analysis methodology, such as automated RCA. Table 1 classifies studies based on the data sources used to identify the failure structure.

Data processing methods

To define failure structure from operational expert-driven data, no specific tool has been used. In the industry, failure structures are typically defined by an expert (or group of experts). When expert-driven or data-driven historical data and/or design forms and process descriptions are available, ontology-driven algorithms, including heuristics (Sayed & Lohse, 2014 ; Zhou et al., 2015 ; Steenwinckel et al., 2018 ; Xu & Dang, 2023 ) and SysML modeling language (Hecht & Baum, 2019 ), process/system decomposition (the operation, the station, and the key characteristics levels) (Zuo et al., 2016 ; Khorshidi et al., 2015 ; Zhou et al., 2015 ), rule-based algorithms that use CBR (Yang et al., 2018 ; Liu & Ke, 2007 ; Xu & Dang, 2023 ; Oliveira et al., 2022 , 2021 ), and FTA/BN modeling from FMEA/expert data (Yang et al., 2022 ; Steenwinckel et al., 2018 ; Palluat et al., 2006 ) and from Perti net (Yang & Liu, 1998 ) have been suggested. Rivera Torres et al. ( 2018 ) divided a system into components and related failures to each of the components to make a tree of components and failures.

Component-failure matrix is generated using unstructured and quality problem texts mining from historical documents such as bills of material and failure analysis. Apriori algorithms were used to find synonyms in the set of failure modes (Xu et al., 2020 ). The 8D method is used to describe a failure. Ontology was used to store and retrieve data in a knowledge base CBR system.

Yang et al. ( 2022 ), Leu and Chang ( 2013 ) and Waghen and Ouali ( 2021 ) have suggested building a BN structure from the FTA model. Wang et al. ( 2018 ) has proposed to use the fault feature diagram, the fault-labeled transition system based on the Kripke structure to describe the system behavior. The MASON (manufacturing semantic ontology) has been used to construct the structure of the failure class by Psarommatis and Kiritsis ( 2022 ). Teoh and Case ( 2005 ) has developed a functional diagram to construct a failure structure between components of a system and to identify causes and effect propagation. Yang et al. ( 2018 ) used an FMEA style CBR to collect failures to search for similarity. They then used CBR to build a BN using a heuristic algorithm.

Step 2: failure detection

Failure detection data are gathered to determine the strength of relationships among failures, causes, and effects.

Failure detection can be based on operational or historical expert-driven data, as well as data-driven historical and/or real-time data obtained from sensors. Such data can come from a variety of sources, including design and control parameters (such as machine age or workpiece geometry), state variables (such as power demand), performance criteria (such as process time or acoustic emission), and internal/external influencing factors (such as environmental conditions) (Filz et al., 2021b ; Dey & Stori, 2005 ). These data are usually used to determine occurrence probability of failures. To determine the severity and detection probabilities of failures, conditional severity utility data/tables may be used (Lee, 2001 ). Simulation can also be used to determine occurrence, severity, and detection (Price & Taylor, 2002 ). Table 2 summarizes types of data that are usually used to detect failures in the literature.

Processing data refers to the transformation of raw data into meaningful information. A data processing tool is needed that provides accurate and complete information about the system and relationships between data and potential failures.

First, data from different sources should be pre-processed. In a data pre-processing step, data is cleaned, edited, reduced, or wrangled to ensure or enhance performance, such as replacing a missing value with the mean value of the entire column (Filz et al., 2021b ; Schuh et al., 2021 ; Zhang et al., 2023 ; Musumeci et al., 2020 ; Jiao et al., 2020 ; Yang et al., 2015 ; Chien et al., 2017 ).

Data then may need to be processed according to the tools used in Step 3. Common data processing methods between all tools include data normalization using the min-max method (Filz et al., 2021b ; Musumeci et al., 2020 ) and other methods (Yang et al., 2018 ; Schuh et al., 2021 ; Jiao et al., 2020 ; Sariyer et al., 2021 ; Chien et al., 2017 ).

Feature selection/extraction algorithms have been used to select the most important features of data (Filz et al., 2021b ; Xu & Dang, 2020 ; Mazzoleni et al., 2017 ; Duan et al., 2020 ; Schuh et al., 2021 ; Zhang et al., 2023 ; Musumeci et al., 2020 ; Yang et al., 2015 ; Sariyer et al., 2021 ).

For BN-based failure analysis, maximum entropy theory is proposed to calculate failure probabilities from expert-based data (Rastayesh et al., 2019 ). Fuzzy methods have also been used to convert linguistic terms to occurrence probabilities (Yucesan et al., 2021 ; Wan et al., 2019 ; Nie et al., 2019 ; Nepal & Yadav, 2015 ; Ma & Wu, 2020 ; Li et al., 2013 ; Duan et al., 2020 ). Euclidean distance-based similarity measure (Chang et al., 2015 ) and fuzzy rule base RPN model (Tay et al., 2015 ), heuristic algorithms (Brahim et al., 2019 ; Dey & Stori, 2005 ; Yang et al., 2022 ), and a fuzzy probability function (Khorshidi et al., 2015 ) have been suggested to build failure probabilities.

Failure analysis data may be incomplete, inaccurate, imprecise, and limited. Therefore, several studies have used tools to deal with uncertainty in data. The most commonly used methods are fuzzy FMEA (Yang et al., 2022 ; Nepal & Yadav, 2015 ; Ma & Wu, 2020 ), fuzzy BN (Yucesan et al., 2021 ; Wan et al., 2019 ; Nie et al., 2019 ), fuzzy MCDM (Yucesan et al., 2021 ; Nie et al., 2019 ; Nepal & Yadav, 2015 ), fuzzy neural network (Tay et al., 2015 ; Palluat et al., 2006 ), and fuzzy evidential reasoning and Petri nets (Shi et al., 2020 ).

Step 3: analysis

A failure analysis tool is essential for conducting any failure analysis. Table 3 categorizes various data-driven tools, such as BNs, Clustering/Classification, Rule-based Reasoning, and other tools used in the literature and the aspects they support.

BNs model probabilistic relationships among failure causes, modes, and effects using directed acyclic graphs and conditional probabilities. Pieces of evidence, i.e., known variables, are propagated through the graph to evaluate unobserved variables (Cai et al., 2017 ). For example, Rastayesh et al. ( 2019 ) applied BNs for FMEA and perform risk analysis of a Proton Exchange Membrane Fuel Cell. Various elements and levels of the system were identified along with possible routes of failure, including failure causes, modes, and effects. A BN was constructed to perform the failure analysis. Some other examples of the BNs application include an assembly system (Sayed & Lohse, 2014 ), kitchen equipment manufacturing (Yucesan et al., 2021 ), and Auxiliary Power Unit (APU) fault isolation (Yang et al., 2015 ).

Classification assigns predefined labels to input data based on learned patterns, Clustering organizes data into groups based on similarities. Neural networks are commonly used for failure classification and have been employed in most studies. Hence, we separated these studies from those that used other clustering/classification tools. Neural networks consist of layers of interconnected nodes, with an input layer receiving data, one or more hidden layers for processing, and an output layer providing the final classification (Jiang et al., 2024 ). For example, Ma and Wu ( 2020 ) applied neural networks to assess the quality of 311 apartments in Shanghai, China, for FMEA. The input includes various APIs collected for the apartments, and the output was the risk rate of each apartment. In another study, Ma et al. ( 2021 ) applied neural networks for RCA to predict the root causes of multiple quality problems in an automobile factory. Some other examples of the neural networks application include industrial valve manufacturing (Pang et al., 2021 ), complex cyber–physical systems (Liu et al., 2021 ), and an electronic module designed for use in a medical device (Psarommatis & Kiritsis, 2022 ).

Other clustering/classification tools include evolving tree (Chang et al., 2015 ), reinforced concrete columns (Mangalathu et al., 2020 ), K-means, random forest algorithms (Xu & Dang, 2020 ; Chien et al., 2017 ; Oliveira et al., 2022 , 2021 ), contrasting clusters (Zhang et al., 2023 ), K-nearest neighbors (Ma et al., 2021 ), self-organizing maps (Gómez-Andrades et al., 2015 ), and Naive Bayes (Schuh et al., 2021 ; Yang et al., 2015 ).

Rule-based reasoning represents knowledge in the form of "if-then" rules. Rule-based reasoning involves a knowledge base containing the rules and a reasoning engine that applies these rules to incoming data or situations. For instance, Jacobo et al. ( 2007 ) utilized rule-based reasoning for analyzing failures in mechanical components. This approach serves as a knowledgeable assistant, offering guidance to less experienced users with foundational knowledge in materials science and related engineering fields throughout the failure analysis process. Also, the application of the rule-based reasoning for wind turbines FMEA is studied by (Zhou et al., 2015 ).

Other tools include gradient-boosted trees, logistic regression (Filz et al., 2021b ), CBR (Tönnes, 2018 ; Camarillo et al., 2018 ; Jacobo et al., 2007 ), analyzing sensitivities of the machining operation by the stream of variations and errors probability distribution determination (Zuo et al., 2016 ), causal reasoning (Teoh & Case, 2005 ), probabilistic Boolean networks with interventions (Rivera Torres et al., 2018 ), principal component analysis (PCA) (Duan et al., 2020 ; Zhang et al., 2023 ; Jiao et al., 2020 ; Sun et al., 2021 ), factor ranking algorithms (Oliveira et al., 2022 , 2021 ), heuristics and/or new frameworks (Camarillo et al., 2018 ; Yang et al., 2009 , 2020 ; Snooke & Price, 2012 ; Xu & Dang, 2023 ; Rokach & Hutter, 2012 ; Wang et al., 2018 ; Hecht & Baum, 2019 ; Yang & Liu, 1998 ; Liu & Ke, 2007 ), and mathematical optimization methods (Khorshidi et al., 2015 ).

These tools may be integrated by other tools including sequential state switching and artificial anomaly association in a neural network (Liu et al., 2021 ), MCDM/optimization (Yucesan et al., 2021 ; Jomthanachai et al., 2021 ; Ma et al., 2021 ; Sun et al., 2021 ), game theory (Mangalathu et al., 2020 ), fuzzy evidential reasoning and Petri nets (Shi et al., 2020 ), and maximum spanning tree, conditional Granger causality, and multivariate time series (Chen et al., 2018 ).

Step 4: output

A data analysis process can benefit not only humans but also machines and tools in a hybrid intelligence failure analysis methodology. Therefore, the output information should be carefully designed. Table 4 ranks the output data, and the list of studies for each output is available in Online Appendix EC.1. Most studies have focused on automatically identifying the root causes of failures, which is the primary objective of a failure analysis methodology. In addition, researchers have also focused on failure occurrence rating, ranking, and classification. While automatically finding the root causes of failures is important, a hybrid intelligence failure analysis process needs to interpret the related data and information and automatically provide mitigation actions for both operators and machines. However, only a few studies have proposed tools to automatically find possible mitigation actions, usually based on CBR databases and only readable for humans. Therefore, future studies may focus on finding possible automated mitigation actions for failures and developing a quality inspection strategy.

Data post-processing

A data post-processing step transforms data from the main tool into readable, actionable, and useful information for both humans and machines. Adapting solutions from similar failures in a database (i.e., CBR) to propose a solution for a detected failure has been proposed by Tönnes ( 2018 ), Camarillo et al. ( 2018 ), Hecht and Baum ( 2019 ), Jacobo et al. ( 2007 ), Liu and Ke ( 2007 ) and Ma et al. ( 2021 ). Simulation to analyze different scenarios (Psarommatis & Kiritsis, 2022 ; Jomthanachai et al., 2021 ; Chien et al., 2017 ; Oliveira et al., 2022 ), mathematical optimization model (Khorshidi et al., 2015 ; Ma et al., 2021 ) and self-organizing map (SOM) neural network (Chang et al., 2017 ) to automatically select the best corrective action have also been proposed. Also, fuzzy rule-based systems to obtain RPN (Nepal & Yadav, 2015 ) and visualisation (Xu & Dang, 2020 ; Yang et al., 2009 ) are discussed.

The statistical analysis of the paper reveals that most FMEA-based studies rely solely on expert-based information to construct failure structures, while RCA-based papers tend to use a hybrid of problem-solving and system-related data. This is depicted in Fig. 4 , which shows the distribution of papers by data used over time. FMEA is used to identify potential failures when there is not enough data available to construct a failure structure based on system-based data. The trend shows some effort to use data, instead of expert knowledge, to construct failure structures, using data from similar products/processes. RCA and FTA are a reactive methodology that analyzes more information than FMEA. Advances in data mining techniques, along with increased data availability, have led to a growing trend of using data to construct failure structures. For a comprehensive and reliable intelligence failure analysis, a combination of all kinds of data is necessary. It is worth noting that Waghen and Ouali ( 2021 ) proposed a heuristic method to augment failure structure identification that uses expert and historical data. They suggested engaging expert knowledge when historical data are insufficient to identify a failure structure and/or the reliability of a failure structure is low. Other studies have solely focused on failure identification through expert knowledge or historical data, without considering the potential benefits of combining different types of data.

Input data statistical analysis

While most FMEA-based papers use only expert-based data to determine failure probability, there is a significant growth in the utilization of problem-solving data and a hybrid of problem-solving and system-related data, i.e., production line data, over time. RCA and FTA usually tend to use more problem-solving and system-related data. Moreover, this figure and Fig. 5 show that the literature on RCA has been growing in recent years, while the trend for FMEA has remained the same over time. We found that Filz et al. ( 2021b ), Mazzoleni et al. ( 2017 ), Ma and Wu ( 2020 ) and Yang et al. ( 2015 ) improved FMEA to use a combination of expert-based, problem-solving, and system-related data to determine potential failures and their causes. They analyzed these data using deep learning, classification, and neural networks, respectively. Duan et al. ( 2020 ), Ma et al. ( 2021 ) tried to use the benefits of both expert-based data and problem-solving and system-related data in the RCA context. They analyzed the root cause of failures using neural networks.

The distribution of papers by the tools used is shown in Fig. 5 . BNs have been mainly used within the context of FMEA methodologies with a growing trend during the recent years, while RCA researchers have used them less frequently. BNs have the potential to model failure propagation, multi-failure scenarios, and solution analysis to propose potential solutions. However, all of the studies reviewed in this paper only used BNs to identify the root causes of failures. BNs offer a clear graphical representation of failures, their causes, and their effects, which facilitates the interpretation of results by humans. They also provide an easy way for humans to intervene and analyze the sensitivity of results and correct processed data if it appears unrealistic. BNs are well-developed tool and have the ability to work with expert-based, historical, and system-based data, even when data is fuzzy or limited. Developing methodologies that leverage the advantages of BNs seems promising for FMEA, RCA, and FTA.

Tools distribution statistical analysis

RCA and FTA are reliant on various tools over time with no trend of using a specific tool, such as PCA and regression, due to their need for a large amount of data. However, these methods have limitations in incorporating both human and machine intelligence and mostly rely on machine intelligence. Although neural networks and classification algorithms have gained attention in both FMEA and RCA during the last few years, they are black boxes and difficult for humans to modify. Also, classification algorithms typically do not address failure propagation or multi-failure modes. BNs offer a promising alternative, as they can model failure propagation, multiple-failures, and provide a clear graphical representation of failures, causes, and effects. Furthermore, BNs can incorporate both expert-based and historical data, making them well-suited for FMEA, RCA, and FTA. Therefore, developing methodologies that fully leverage the benefits of BNs in these domains would be valuable.

Managerial insights, limitations, and future research

In this section, we discuss managerial insights, limitations, and future research related to different aspects of a Hybrid Intelligence failure analysis methodology. The aim is to assist researchers in focusing on relevant recommendations. Section Section Applications and complexity delves into the applications and complexity of each study, and provides examples for each tool. Section Levels of automation/intelligence presents the levels of intelligence for a failure analysis methodology. Section Introducing knowledge into tools discusses how knowledge is introduced into the failure analysis tools for an effective failure analysis. A more in-depth discussion of hybrid intelligence is in Section Hybrid intelligence . The last three sections provide insights into failure propagation and correlation, hybrid methodologies, and other areas of future research.

Applications and complexity

Intelligent FMEA, RCA, and FTA have been applied to various applications, including production quality management, computer systems, reliability and safety, chemical systems, and others. Table 5 presents the distribution of reviewed papers by application. The list of studies per application is available in Online Appendix EC.2. Production quality management has been the most common application of intelligent failure analysis methodologies due to the significant costs associated with quality assurance. Smart failure analysis methodologies have also been impacted by the increased use of sensors and IoT to collect precise data from machines, tools, operators, and stations, as well as powerful computers to analyze the data. Computer systems failure analysis and system reliability and safety rank second, while chemical systems rank third, as these systems often require specific methodologies, such as hazard and operability analysis.

We checked every paper dataset to find information about the complexity of their case-study and reasons behind their good results to help readers select a validated study on a large set of data. An enriched dataset of problem-solving data are used by Xu et al. ( 2020 ), Du et al. ( 2012 ), Oliveira et al. ( 2021 ), Gómez-Andrades et al. ( 2015 ), Leu and Chang ( 2013 ), Price and Taylor ( 2002 ), Sariyer et al. ( 2021 ), Gomez-Andrades et al. ( 2016 ) and Xu and Dang ( 2023 ). An enriched dataset of historical problem-solving and sensors data is used by

Filz et al. ( 2021b ), Sun et al. ( 2021 ), Mazzoleni et al. ( 2017 ), Hireche et al. ( 2018 ), Yanget al. ( 2015 ) Demirbaga et al. ( 2021 ), Waghen and Ouali ( 2021 ), Zhang et al. ( 2023 ), Oliveira et al. ( 2022 ), Sun et al. ( 2021 ). Data from the system and processes are used by Teoh and Case ( 2005 ), Ma et al. ( 2021 ), Schuh et al. ( 2021 ), Waghen and Ouali ( 2021 ). Other studies demonstrated their methodology on a small problem.

Levels of automation/intelligence

Failure analysis intelligence can be divided into five levels based on the data used. Level 1 involves analyzing failures using expert-based data with the use of intelligence tools. This level can be further improved by incorporating fuzzy-based tools, such as fuzzy BNs, fuzzy neural networks, and fuzzy rule-based systems. If the amount of historical data can be increased over time, we suggest using BNs in a heuristic-based algorithm, as they have the capability to work with all possible data, resulting in fewer modifications in the failure analysis methodology over time. Good examples for Level 1 include Yucesan et al. ( 2021 ) and Brahim et al. ( 2019 ).

Level 2 involves analyzing failures using experts to identify failure structures and problem-solving and system-related data to determine failure probabilities. This level can be used by a professional team who can correctly and completely identify failure structure. It can also be used by those who work with variable structures where updating the structure requires a lot of data modification. Identifying failure structures and analyzing failures are both automated at level 3. This level is the most applicable when a good amount of data is available. BNs, classification algorithms, and neural networks are among the best tools to analyze failure within RCA, FMEA, and FTA methodologies. Studies such as Filz et al. ( 2021b ) Zuo et al. ( 2016 ), Dey and Stori ( 2005 ), Mangalathu et al. ( 2020 ), Yang et al. ( 2015 ) and Ma et al. ( 2021 ) are good examples for Levels 2 and 3.

In level 4, mitigation actions are also determined automatically. This level represents a whole automation of failure analysis. BNs are among the few tools that can encompass all steps of failure analysis. As such, we suggest using them. CBR databases can be used by BNs plus system-based data to provide possible corrective actions. Tönnes ( 2018 ), Zuo et al. ( 2016 ) and Hecht and Baum ( 2019 ) are among good studies for Level 4. Chang et al. ( 2017 ) has focused to automate and visualize corrective actions using a self-organizing map (SOM) neural network in an FMEA methodology. Future research should concentrate on the development of an automated FMEA that dynamically updates the current RPN (Risk Priority Number). This can aid in predicting failures in parts or components of a system using a "Live RPN." The predictive capability of such a tool can be utilized to optimize the overall system. It enables the transformation of a manufacturing system into a self-controlling system, allowing adjustments based on current parameters (Filz et al., 2021b ).

Level 5 is a hybrid intelligence approach to failure analysis that encompasses all other levels and can be implemented within FMEA, RCA, and FTA methodologies when a limited amount of historical and system-based data is available until a comprehensive CBR database is built. BNs provide a good graphical representation and can work with all possible data types. The advantages of BNs are significant enough to be suggested for hybrid intelligence failure analysis. However, we did not find any comprehensive study for this level. A combination of studies that proposed methods to use integrated expert-based, problem-solving, and system-based data, such as Waghen and Ouali ( 2021 ); Filz et al. ( 2021b ), is suggested. Nonetheless, this level remains open and needs to be the focus of future research by scholars. To facilitate the implementation of hybrid intelligence failure analysis, a user-friendly interface is crucial for operators to interact with. Several studies have proposed user-interface applications for this purpose, including (Chan & McNaught, 2008 ; Camarillo et al., 2018 ; Li et al., 2013 ; Jacobo et al., 2007 ; Yang et al., 2009 , 2020 ; Demirbaga et al., 2021 ; Snooke & Price, 2012 ; Palluat et al., 2006 ).

Introducing knowledge into tools

In this section, we analyze which types of knowledge, expert-driven, data-driven, or a hybrid of both, are usually used with which tools and what the implications are for providing insights on suitable tools for hybrid intelligence failure analyses.

Figure 6 shows the distribution of literature based on the input data, tools, and outputs (four general steps of a failure analysis methodology in Fig. 3 ). The first column of nodes shows various combinations of types of knowledge, expert-driven, data-driven, or a hybrid of both, that are usually used in the literature to identify the structure of failure and to detect the probability of failures. The second column of nodes shows various tools that are used to analyze the failure. The third column of nodes shows outputs of a failure analysis. The number of studies with each particular focus is shown by the thickness of an arrow. Details are in Appendix EC.1.

Literature distribution based on inputs, tools, and outputs

The following studies have tried to introduce knowledge and data from expert and data based sources to a failure analysis methodology. Filz et al. ( 2021b ) utilized expert knowledge to identify the structure of failure, the components involved, and the necessary sensors to be used. They then employed sensors to capture data and leveraged problem-solving data from the recorded expert archive to identify failures in a deep learning model. Similarly, Musumeci et al. ( 2020 ) used supervised algorithms to classify failures. Mazzoleni et al. ( 2017 ) they used data from sensors to select the most effective features related to a failure, and subsequently employed sensor data and failure expert data-sets within a gradient boosting tree algorithm to identify the possibility of the failure. Duan et al. ( 2020 ) used data from different sources in a similar way for a neural network to identify the root cause of a failure. Ma and Wu ( 2020 ) utilized expert knowledge to identify failures in construction projects. Subsequently, expert datasets were employed in conjunction with project performance indices to predict the possibility of a failure and determine the root cause of the failure using a neural network tool.

Hireche et al. ( 2018 ), Yang et al. ( 2015 ) gathered data from sensors to determine the conditions of each failure/component node. Then, a BN was used to identify the risks and causes. A multi-level tree is developed by Waghen and Ouali ( 2021 ). Each level contains a solution, pattern, and condition level. Solutions are retrieved from a historical failure database as a combination of certain patterns. The pattern in each problem has been identified and related to the solution using a supervised machine-learning tool. Each level is linked to the next level until the root cause of a failure is correctly identified.

Other usefull tips for introducing knowledge from different sources to a failure analysis methodology can be found in the following studies. Zuo et al. ( 2016 ) divided a multi-operation machining process operation, station, and key characteristics levels. Stream of variations (SoV) was used to evaluate the sensitivities of the machining operations level by level. Results were used to find the sources affecting the quality. Distribution techniques for each quality precision using multi-objective optimization were chosen. Dey and Stori ( 2005 ) used a message-passing method (Pearl, 1988 ) to update a BN using data from sensors to estimate the condition of the system and update the CPTs, when each sensor output is considered as a node in the BN. Chan and McNaught ( 2008 ) also used sensor data to change the probabilities in a BN. A user interface is also developed to make inferences and present the results to operators.

Rokach and Hutter ( 2012 ) used the sequence of machines and a commonality graph of steps and failure causes data to cluster failures to find commonalities between them. A GO methodology is used by Liu et al. ( 2019b ) to model the system and a heuristic is used to construct BN structure and probabilities from the GO methodology model. Teoh and Case ( 2005 ) developed an objective-oriented framework that considers conceptual design information. A hierarchy of components, an assembly tree, and a functional diagram are built to capture data from processes and feed it to FMEA. Bhardwaj et al. ( 2022 ) used historical data from a similar system to estimate failure detection probabilities. Hecht and Baum ( 2019 ) used SysML to describe components and failures.

Zhou et al. ( 2015 ) used a tree of a system. Two classes of knowledge, shallow knowledge and deep knowledge, were gathered to generate rules for failure analysis. The former indicates the experiential knowledge of domain experts, and the latter is the knowledge about the structure and basic principle of the diagnosis system. Liu and Ke ( 2007 ) used CBR to find similar problems and solutions, text mining to find key concepts of the failure in the historical failure record texts, and rule mining to find hidden patterns among system features and failures. Filz et al. ( 2021a ) gathered process parameters after each station using a quality check station. Then a self-organizing Map was used to find failure propagation and cause and effect. Ma et al. ( 2021 ) used data from the system to determine features of problems, products, and operators. Data from problem-solving databases was used to find new failures and classified them using the features and historical data.

Psarommatis and Kiritsis ( 2022 ) developed a methodology that uses data-driven and knowledge-based approaches, an ontology base on the MASON ontology to describe the production domain and enrich the available data. Wang et al. ( 2018 ) developed a data acquisition system including a monitor, sensor, and filter modules. A fault diagram models failure propagation. They extended the Kripke structure by proposing the feature-labeled transition system, which is used to distinguish the behavior of the transition relationship by adding a signature to the transition relationship.

This section highlights that in the realm of failure analysis, a majority of research papers have utilized a hybrid approach, combining expert and data knowledge for tasks such as failure detection, classification, and feature selection. However, to achieve real-time failure analysis, a more effective integration of these two sources is crucial. This integration should enable operators and engineers to provide timely input to the system and observe immediate results. Furthermore, only a limited number of studies have specifically focused on the identification of failure structures using either data or a hybrid of expert and data knowledge.

The use of BNs has emerged as a highly promising approach for achieving real-time input and structure identification in the field of failure analysis. By leveraging both expert knowledge and data sources, BNs have the capability to effectively incorporate expert knowledge as constraints within structure identification algorithms. Unlike traditional classification algorithms that are primarily designed for continuous data, BNs are versatile in handling both discrete and continuous data types. Moreover, BNs possess several strengths that make them particularly suitable for failure analysis. They excel at performing real-time inferences, engaging in counterfactual reasoning, and effectively managing confounding factors. Given these advantages, it is essential to allocate more attention to the application of BNs in hybrid intelligence failure analysis. This involves further exploration of their capabilities and conducting comparative analyses with other tools to assess their effectiveness in various scenarios. By focusing on BNs and conducting comprehensive evaluations, researchers can enhance the understanding and adoption of these powerful tools for improved failure analysis in real-time settings.

Hybrid intelligence

A collaborative failure analysis methodology is needed, in which artificial intelligence tools, machines, and humans can communicate. While hybrid intelligence has gained attention in various fields, literature on the subject for failure analysis is still limited. For example, Piller et al. ( 2022 ) discussed methods to enhance productivity in manufacturing using hybrid intelligence. They explored considerations such as task allocation between humans and machines and the degree of machine intelligence integrated into manufacturing processes. Petrescu and Krishen ( 2023 ) and references within have delved into the benefits and future directions of hybrid intelligence for marketing analytics. Mirbabaie et al. ( 2021 ) has reviewed challenges associated with hybrid intelligence, focusing particularly on conversational agents in hospital settings. Ye et al. ( 2022 ) developed a parallel cognition model. This model draws on both a psychological model and user behavioral data to adaptively learn an individual’s cognitive knowledge. Lee et al. ( 2020 ) combined a data-driven prediction model with a rule-based system to benefit from the combination of human and machine intelligence for personalized rehabilitation assessment.

An artificial intelligence tool should not only provide its final results but also provide its reasoning. A human can analyze the artificial intelligence tool reasoning through a user-interface application and correct possible mistakes instantly and effortlessly. To enable this capability, the use of a white-box artificial tool, such as Bayesian networks, is essential. Explainable AI aids in comprehending and trusting the decision-making process of the hybrid intelligence system by providing the reasoning behind it (Confalonieri et al., 2021 ). Moreover, a machine should be able to interpret and implement an artificial intelligence tool and/or human solutions. Artificial intelligence tools, machines, and humans can learn from mistakes (Correia et al., 2023 ).

To fully exploit the complementarity in human–machine collaborations and effectively utilize the strengths of both, it is important to recognize and understand their roles, limitations, and capabilities in the context of failure analysis. Future research should focus on developing a clear plan for their teamwork and joint actions, including determining the optimal sensor types and locations, quality inspection stations, and human/machine analysis processes. In other words, How to design a decision support system that integrates both human knowledge and machine intelligence with respect to quality management? should be answered. Additionally, tools should be developed to propose possible mitigation actions based on the unique characteristics of the system, environment, humans, and machines. To achieve this, system-related data along with CBR data can be analyzed to find potential mitigation actions.

A general framework for human–machine fusion could involve the following steps: identifying applicable human knowledge and machine data for the problem, determining machine intelligence tools that facilitate the integration of human–machine elements like BNs, identifying the suitable points in the decision-making process to combine human knowledge and machine intelligence effectively, designing the user interface, and incorporating online learning using input from human knowledge (Jarrahi et al., 2022 ). However, human–machine fusion is not an easy task due to the complexity of human–machine interaction, the need for effective and online methods to work with both human and machine data, and the challenge of online learning from human knowledge. For instance, while ChatGPT interacts well with humans, it currently does not update its knowledge using human knowledge input for future cases (Dellermann et al., 2019 ; Correia et al., 2023 ).

Failure propagation and correlation

Most FMEA papers concentrated on analyzing failures in individual products, processes, or machines. It is essential to acknowledge that production processes and machines are interconnected, leading to the correlation and propagation of failures among them. Consequently, it becomes crucial to address the challenge of analyzing failures in multiple machines. To effectively tackle this issue, a holistic approach is necessary. Rather than focusing solely on individual machines, take a broader perspective by considering the entire production system to identify the interdependencies and interactions among different machines, multiple processes, and within the system.

For an intelligence failure analysis, it is necessary to exploit detailed system-related data to carefully and comprehensively identify the relations between different parts of a system, product, and/or process. Some papers have suggested methods to identify failure propagation and correlation (Wang et al., 2021 ; Zhu et al., 2021 ; Chen et al., 2017 ). They usually proposed methods to analyze correlations only between failures or risk criteria using MCDM or statistical methods. However, an intelligence failure analysis should go beyond this and identify failure propagation and correlation among parts of a system.

In the literature, Chen and Jiao ( 2017 ) proposed finite state machine (FSM) theory to model the interactive behaviors between the components, constructing the transition process of fault propagation through the extraction of the state, input, output, and state function of the component. Zuo et al. ( 2016 ) used SoV to model propagation of variations from station to station and operation to operation. A propagation from one station (operation) to the next station (operation) was modeled using a regression like formula. Ament and Goch ( 2001 ) used quality check data after each station to train a neural network for failure progagation and estimate the relationships betweenfailure in stations using a regression model to find patterns in quality check data. Ma et al. ( 2021 ) used patterns in data to classify failures and identify causes.

To conduct an intelligence failure analysis, it is important to identify every part involved, their roles, characteristics, and states. The analysis should include the identification of failure propagation and effects on functions, parts, and other failures. One approach to analyzing failures is through simulation, which can help assess the changes in the characteristics of every part of a system, including humans, machines, and the environment. To analyze the complexity of failure propagation and mutual interactions among different parts of a system, data-driven tools and heuristic algorithms need to be developed. These tools should be capable of managing a large bill of materials and analyzing the failure structure beyond the traditional statistical and MCDM methods. Rule mining can be a useful tool for detecting failure correlation and propagation, especially in situations where there is limited data available, and human interpretation is crucial.

Hybrid methodologies

FMEA, RCA, and FTA methodologies are all complementary and can improve each other’s performance. Furthermore, the availability of data, advanced tools to process data, and the ability to gather online data may lead to a unified FMEA, RCA, and FTA methodology. The reason for this is that while FMEA tries to find potential failures, RCA and FTA try to find root causes of failures, they use similar data and tools to analyze data.

In the literature, FTA has been used as an internal part of FMEA by Steenwinckel et al. ( 2018 ), Palluat et al. ( 2006 )and RCA by Chen et al. ( 2018 ). Using automated mappings from FMEA data to a domain-specific ontology and rules derived from a constructed FTA, Steenwinckel et al. ( 2018 ) annotated and reasoned on sensor observations. Palluat et al. ( 2006 ) used FTA to illustrate the failure structure of a system within an FMEA methodology and developed a neuro-fuzzy network to analyze failures. Chen et al. ( 2018 ) used FTA and graph theory tools, such as the maximum spanning tree, to find the root cause of failures in an RCA methodology. However, studies on the integration of these methodologies regarding the availability of data, tools, and applications should be done to use their advantages within a unified methodology that detects potential failures, finds root causes and effects, and improves the system.

Other future research

Several promising future research directions can be pursued. Cost-based and economic quantification approaches can be integrated into intelligent methodologies to enable more informed decision-making related to failures, their effects, and corrective actions. Additionally, incorporating customer satisfaction criteria, such as using the Kano model, can be useful in situations where there are several costly failures in a system, and budget constraints make it necessary to select the most effective corrective action. This approach has been successfully applied in previous studies (Madzík & Kormanec, 2020 ), and can help optimize decision-making in complex failure scenarios.

Data management is a critical aspect of intelligence methodologies, given the large volume and diverse types of data that need to be processed. Therefore, it is important to design reliable databases that can store and retrieve all necessary data. Ontology can be a valuable tool to help integrate and connect different types of data (Rajpathak & De, 2016 ; Ebrahimipour et al., 2010 ). However, it is also essential to consider issues such as data obsolescence and updates, especially when corrective actions are taken and root causes are removed. Failure to address these issues can lead to incorrect analysis and decision-making.

Traditionally, only single failures were considered in analysis because analyzing a combination of multiple failures was impossible. However, in a system, two or more failures may occur simultaneously or sequentially. It is also possible that a failure occurs as a consequence of another failure. These circumstances are complicated because each failure can have several root causes, and another failure is only one of its causes. Therefore, a clear and powerful tool, such as Bayesian Networks (BNs), should be used to analyze failures and accurately identify possible causes.

The traditional failure analysis methodologies had limitations such as repeatability, subjectivity, and time consumption, which have been addressed by intelligence failure analysis. However, there is a need for more focus on explainability, objective evaluation criteria, and results reliability as some intelligent tools, such as neural networks, act as black boxes. Therefore, suitable tools, such as BNs, should be well-developed and adapted for (hybrid) intelligence failure analysis. Details such as the time and location of the detected failure, possible factors of the causes, such as location, time, conditions, and description of the cause, and reasons behind the causes, such as human fatigue, should be considered within a methodology. These can help to go beyond the CBR and propose intelligence solutions based on the reasons behind a cause. While RCA has implemented these data to a limited extent, FMEA lacks such implementation.

This paper has collected information on both proactive and reactive failure analysis methodologies from 86 papers that focus on FMEA, RCA, or FTA. The goal is to identify areas for improvement, trends, and open problems regarding intelligent failure analysis. This information can help researchers learn the benefits of both methodologies, use their tools, and integrate them to strengthen failure analysis. Each paper has been read and analyzed to extract data and tools used within the paper and their benefits. It was observed that the literature on the three methodologies, FMEA, RCA, and FTA, is diverse. In Industry 4.0, the availability of data, and advances in technology are helping these methodologies benefit from the same tools, such as BNs and neural networks, and make them more integrated.

The literature was classified based on the data needed for a (hybrid) intelligence failure analysis methodology and the tools used for failure analysis to be data-driven and automated. In addition, trends to make these methodologies smart and possible future research in this regard were discussed.

Two main classes of failure structure and failure detection data are usually needed for a failure analysis methodology, each of which can be classified as expert-driven and data-driven. However, a combination of all types of data can lead to more reliable failure analysis. Most papers focused on operational and historical expert-driven and/or data-driven problem-solving data. Among the tools used within FMEA, RCA, and FTA methodologies, BNs have the capability to make a methodology smart and interact with both humans and machines to benefit from hybrid intelligence. BNs not only can analyze failures to identify root causes but also can analyze possible solutions to provide necessary action to prevent failures. A BN’s are also capable of real-time inference, counterfactual reasoning, and managing confounding factors. BNs handle both discrete and continuous data types, unlike traditional classification algorithms. Besides BNs, classification by neural networks, other classification tools, rule-based algorithms, and other tools have been proposed in the literature.

Finally, managerial insights and future research are provided. Most studies have focused on the determination of root causes. It is necessary to automatically find possible mitigation and corrective actions. This step of a failure analysis methodology needs more interaction with humans. Thus, the benefits of hybrid intelligence can be more evident here. It is imperative for humans and machines to work together to properly identify and resolve failures. System-related data should be analyzed to find possible corrective actions. This data is usually available for both proactive and reactive methodologies. Our study showed an effectively tool to integrate knowledge from experts and sensors in needed, enabling operators and engineers to provide timely input and observe immediate results. There is a need to identify failure structures using a hybrid approach that combines expert and data knowledge. Real-time input and structure identification with Bayesian networks can be achieved through the use of Bayesian networks. Further exploration of BNs and comparative analyses with other tools is necessary to enhance understanding and adoption of the best tools for a hybrid intelligence failure analysis in real-time scenarios to prevent failures.

Data availability

There is no data related to this paper.

Agrawal, V., Panigrahi, B. K., & Subbarao, P. (2016). Intelligent decision support system for detection and root cause analysis of faults in coal mills. IEEE Transactions on Fuzzy Systems, 25 (4), 934–944.

Article   Google Scholar  

Akata, Z., Balliet, D., De Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020). A research agenda for hybrid intelligence: Augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. Computer, 53 (08), 18–28.

Al-Mamory, S. O., & Zhang, H. (2009). Intrusion detection alarms reduction using root cause analysis and clustering. Computer Communications, 32 (2), 419–430.

Ament, C., & Goch, G. (2001). A process oriented approach to automated quality control. CIRP Annals, 50 (1), 251–254.

Bhardwaj, U., Teixeira, A., & Soares, C. G. (2022). Bayesian framework for reliability prediction of subsea processing systems accounting for influencing factors uncertainty. Reliability Engineering & System Safety, 218 , 108143.

Brahim, I. B., Addouche, S. A., El Mhamedi, A., & Boujelbene, Y. (2019). Build a Bayesian network from FMECA in the production of automotive parts: Diagnosis and prediction. IFAC-PapersOnLine, 52 (13), 2572–2577.

Cai, B., Huang, L., & Xie, M. (2017). Bayesian networks in fault diagnosis. IEEE Transactions on Industrial Informatics, 13 (5), 2227–2240.

Camarillo, A., Ríos, J., & Althoff, K. D. (2018). Knowledge-based multi-agent system for manufacturing problem solving process in production plants. Journal of Manufacturing Systems, 47 , 115–127.

Chan, A., & McNaught, K. R. (2008). Using Bayesian networks to improve fault diagnosis during manufacturing tests of mobile telephone infrastructure. Journal of the Operational Research Society, 59 (4), 423–430.

Chang, W. L., Pang, L. M., & Tay, K. M. (2017). Application of self-organizing map to failure modes and effects analysis methodology. Neurocomputing, 249 , 314–320.

Chang, W. L., Tay, K. M., & Lim, C. P. (2015). Clustering and visualization of failure modes using an evolving tree. Expert Systems with Applications, 42 (20), 7235–7244.

Chen, H. S., Yan, Z., Zhang, X., Liu, Y., & Yao, Y. (2018). Root cause diagnosis of process faults using conditional Granger causality analysis and maximum spanning tree. IFAC-PapersOnLine, 51 (18), 381–386.

Chen, L., Jiao, J., Wei, Q., & Zhao, T. (2017). An improved formal failure analysis approach for safety-critical system based on mbsa. Engineering Failure Analysis, 82 , 713–725.

Chen, X., & Jiao, J. (2017). A fault propagation modeling method based on a finite state machine. Annual Reliability and Maintainability Symposium (RAMS), 2017 , 1–7.

Google Scholar  

Chhetri, T. R., Aghaei, S., Fensel, A., Göhner, U., Gül-Ficici, S., & Martinez-Gil, J. (2023). Optimising manufacturing process with Bayesian structure learning and knowledge graphs. Computer Aided Systems Theory - EUROCAST, 2022 , 594–602.

Chien, C. F., Liu, C. W., & Chuang, S. C. (2017). Analysing semiconductor manufacturing big data for root cause detection of excursion for yield enhancement. International Journal of Production Research, 55 (17), 5095–5107.

Clancy, R., O’Sullivan, D., & Bruton, K. (2023). Data-driven quality improvement approach to reducing waste in manufacturing. The TQM Journal, 35 (1), 51–72.

Confalonieri, R., Coba, L., Wagner, B., & Besold, T. R. (2021). A historical perspective of explainable artificial intelligence. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11 (1), e1391.

Correia, A., Grover, A., Schneider, D., Pimentel, A. P., Chaves, R., De Almeida, M. A., & Fonseca, B. (2023). Designing for hybrid intelligence: A taxonomy and survey of crowd-machine interaction. Applied Sciences, 13 (4), 2198.

Dabous, S. A., Ibrahim, F., Feroz, S., & Alsyouf, I. (2021). Integration of failure mode, effects, and criticality analysis with multi-criteria decision-making in engineering applications: Part I- manufacturing industry. Engineering Failure Analysis, 122 , 105264.

Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019). Hybrid intelligence. Business & Information Systems Engineering, 61 , 637–643.

Demirbaga, U., Wen, Z., Noor, A., Mitra, K., Alwasel, K., Garg, S., Zomaya, A. Y., & Ranjan, R. (2021). Autodiagn: An automated real-time diagnosis framework for big data systems. IEEE Transactions on Computers, 71 (5), 1035–1048.

Dey, S., & Stori, J. (2005). A Bayesian network approach to root cause diagnosis of process variations. International Journal of Machine Tools and Manufacture, 45 (1), 75–91.

Du, S., Lv, J., & Xi, L. (2012). A robust approach for root causes identification in machining processes using hybrid learning algorithm and engineering knowledge. Journal of Intelligent Manufacturing, 23 (5), 1833–1847.

Duan, P., He, Z., He, Y., Liu, F., Zhang, A., & Zhou, D. (2020). Root cause analysis approach based on reverse cascading decomposition in QFD and fuzzy weight ARM for quality accidents. Computers & Industrial Engineering, 147 , 106643.

Ebeling, C. E. (2019). An introduction to reliability and maintainability engineering . Waveland Press.

Ebrahimipour, V., Rezaie, K., & Shokravi, S. (2010). An ontology approach to support FMEA studies. Expert Systems with Applications, 37 (1), 671–677.

Filz, M. A., Gellrich, S., Lang, F., Zietsch, J., Abraham, T., & Herrmann, C. (2021). Data-driven analysis of product property propagation to support process-integrated quality management in manufacturing systems. Procedia CIRP, 104 , 900–905.

Filz, M. A., Langner, J. E. B., Herrmann, C., & Thiede, S. (2021). Data-driven failure mode and effect analysis (FMEA) to enhance maintenance planning. Computers in Industry, 129 , 103451.

French, S., Bedford, T., Pollard, S. J., & Soane, E. (2011). Human reliability analysis: A critique and review for managers. Safety Science, 49 (6), 753–763.

Gomez-Andrades, A., Barco, R., Serrano, I., Delgado, P., Caro-Oliver, P., & Munoz, P. (2016). Automatic root cause analysis based on traces for LTE self-organizing networks. IEEE Wireless Communications, 23 (3), 20–28.

Gómez-Andrades, A., Munoz, P., Serrano, I., & Barco, R. (2015). Automatic root cause analysis for lte networks based on unsupervised techniques. IEEE Transactions on Vehicular Technology, 65 (4), 2369–2386.

Hecht, M., & Baum, D. (2019). Failure propagation modeling in FMEAs for reliability, safety, and cybersecurity using SysML. Procedia Computer Science, 153 , 370–377.

Hireche, C., Dezan, C., Mocanu, S., Heller, D., & Diguet, J. P. (2018). Context/resource-aware mission planning based on BNs and concurrent MDPs for autonomous UAVs. Sensors, 18 (12), 4266.

Huang, J., You, J. X., Liu, H. C., & Song, M. S. (2020). Failure mode and effect analysis improvement: A systematic literature review and future research agenda. Reliability Engineering & System Safety, 199 , 106885.

Insua, D. R., Ruggeri, F., Soyer, R., & Wilson, S. (2020). Advances in Bayesian decision making in reliability. European Journal of Operational Research, 282 (1), 1–18.

Jacobo, V., Ortiz, A., Cerrud, Y., & Schouwenaars, R. (2007). Hybrid expert system for the failure analysis of mechanical elements. Engineering Failure Analysis, 14 (8), 1435–1443.

Jarrahi, M. H., Lutz, C., & Newlands, G. (2022). Artificial intelligence, human intelligence and hybrid intelligence based on mutual augmentation. Big Data & Society, 9 (2), 20539517221142824.

Jiang, S., Qin, S., Pulsipher, J. L., & Zavala, V. M. (2024). Convolutional neural networks: Basic concepts and applications in manufacturing. Artificial Intelligence in Manufacturing, 8 , 63–102.

Jiao, J., Zhen, W., Zhu, W., & Wang, G. (2020). Quality-related root cause diagnosis based on orthogonal kernel principal component regression and transfer entropy. IEEE Transactions on Industrial Informatics, 17 (9), 6347–6356.

Johnson, K., & Khan, M. K. (2003). A study into the use of the process failure mode and effects analysis (PFMEA) in the automotive industry in the UK. Journal of Materials Processing Technology, 139 (1–3), 348–356.

Jomthanachai, S., Wong, W. P., & Lim, C. P. (2021). An application of data envelopment analysis and machine learning approach to risk management. IEEE Access, 9 , 85978–85994.

Kabir, S., & Papadopoulos, Y. (2019). Applications of Bayesian networks and Petri nets in safety, reliability, and risk assessments: A review. Safety Science, 115 , 154–175.

Khakzad, N., Khan, F., & Amyotte, P. (2012). Dynamic risk analysis using bow-tie approach. Reliability Engineering & System Safety, 104 , 36–44.

Khorshidi, H. A., Gunawan, I., & Ibrahim, M. Y. (2015). Data-driven system reliability and failure behavior modeling using FMECA. IEEE Transactions on Industrial Informatics, 12 (3), 1253–1260.

Kumar, M., & Kaushik, M. (2020). System failure probability evaluation using fault tree analysis and expert opinions in intuitionistic fuzzy environment. Journal of Loss Prevention in the Process Industries, 67 , 104236.

Lee BH (2001) Using Bayes belief networks in industrial FMEA modeling and analysis. Annual Reliability and Maintainability Symposium. 2001 Proceedings. International Symposium on Product Quality and Integrity (Cat. No.01CH37179) , pp. 7–15.

Lee MH, Siewiorek DP, Smailagic A, Bernardino A, Bermúdez i Badia S (2020) Interactive hybrid approach to combine machine and human intelligence for personalized rehabilitation assessment. Proceedings of the ACM Conference on Health, Inference, and Learning , pp. 160–169.

Leu, S. S., & Chang, C. M. (2013). Bayesian-network-based safety risk assessment for steel construction projects. Accident Analysis & Prevention, 54 , 122–133.

Li, B., Han, T., & Kang, F. (2013). Fault diagnosis expert system of semiconductor manufacturing equipment using a Bayesian network. International Journal of Computer Integrated Manufacturing, 26 (12), 1161–1171.

Liu, C., Lore, K. G., Jiang, Z., & Sarkar, S. (2021). Root-cause analysis for time-series anomalies via spatiotemporal graphical modeling in distributed complex systems. Knowledge-Based Systems, 211 , 106527.

Liu, D. R., & Ke, C. K. (2007). Knowledge support for problem-solving in a production process: A hybrid of knowledge discovery and case-based reasoning. Expert Systems with Applications, 33 (1), 147–161.

Liu, H. C., Chen, X. Q., Duan, C. Y., & Wang, Y. M. (2019). Failure mode and effect analysis using multi-criteria decision making methods: A systematic literature review. Computers & Industrial Engineering, 135 , 881–897.

Liu, H. C., Liu, L., & Liu, N. (2013). Risk evaluation approaches in failure mode and effects analysis: A literature review. Expert Systems with Applications, 40 (2), 828–838.

Liu, L., Fan, D., Wang, Z., Yang, D., Cui, J., Ma, X., & Ren, Y. (2019). Enhanced GO methodology to support failure mode, effects and criticality analysis. Journal of Intelligent Manufacturing, 30 (3), 1451–1468.

Ma, G., & Wu, M. (2020). A big data and FMEA-based construction quality risk evaluation model considering project schedule for shanghai apartment projects. International Journal of Quality & Reliability Management, 37 (1), 18–33.

Ma, Q., Li, H., & Thorstenson, A. (2021). A big data-driven root cause analysis system: Application of machine learning in quality problem solving. Computers & Industrial Engineering, 160 , 107580.

Madzík, P., & Kormanec, P. (2020). Developing the integrated approach of Kano model and failure mode and effect analysis. Total Quality Management & Business Excellence, 31 (15–16), 1788–1810.

Mangalathu, S., Hwang, S. H., & Jeon, J. S. (2020). Failure mode and effects analysis of RC members based on machine-learning-based shapley additive explanations (shap) approach. Engineering Structures, 219 , 110927.

Mazzoleni, M., Maccarana, Y., & Previdi, F. (2017). A comparison of data-driven fault detection methods with application to aerospace electro-mechanical actuators. IFAC-PapersOnLine, 50 (1), 12797–12802.

Mirbabaie, M., Stieglitz, S., & Frick, N. R. (2021). Hybrid intelligence in hospitals: Towards a research agenda for collaboration. Electronic Markets, 31 , 365–387.

Musumeci, F., Magni, L., Ayoub, O., Rubino, R., Capacchione, M., Rigamonti, G., Milano, M., Passera, C., & Tornatore, M. (2020). Supervised and semi-supervised learning for failure identification in microwave networks. IEEE Transactions on Network and Service Management, 18 (2), 1934–1945.

Nepal, B., & Yadav, O. P. (2015). Bayesian belief network-based framework for sourcing risk analysis during supplier selection. International Journal of Production Research, 53 (20), 6114–6135.

Nie, W., Liu, W., Wu, Z., Chen, B., & Wu, L. (2019). Failure mode and effects analysis by integrating Bayesian fuzzy assessment number and extended gray relational analysis-technique for order preference by similarity to ideal solution method. Quality and Reliability Engineering International, 35 (6), 1676–1697.

Oliveira, E. E., Miguéis, V. L., & Borges, J. L. (2021). Understanding overlap in automatic root cause analysis in manufacturing using causal inference. IEEE Access, 10 , 191–201.

Oliveira, E. E., Miguéis, V. L., & Borges, J. L. (2022). On the influence of overlap in automatic root cause analysis in manufacturing. International Journal of Production Research, 60 (21), 6491–6507.

Oliveira, E. E., Miguéis, V. L., & Borges, J. L. (2023). Automatic root cause analysis in manufacturing: An overview & conceptualization. Journal of Intelligent Manufacturing, 34 , 2061–2078.

Oztemel, E., & Gursev, S. (2020). Literature review of industry 4.0 and related technologies. Journal of Intelligent Manufacturing, 31 (1), 127–182.

Palluat, N., Racoceanu, D., & Zerhouni, N. (2006). A neuro-fuzzy monitoring system: Application to flexible production systems. Computers in Industry, 57 (6), 528–538.

Pang, J., Zhang, N., Xiao, Q., Qi, F., & Xue, X. (2021). A new intelligent and data-driven product quality control system of industrial valve manufacturing process in CPS. Computer Communications, 175 , 25–34.

Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference . Morgan kaufmann.

Petrescu, M., & Krishen, A. S. (2023). Hybrid intelligence: Human-ai collaboration in marketing analytics. Journal of Marketing Analytics, 11 (3), 263–274.

Piller, F. T., Nitsch, V., & van der Aalst, W. (2022). Hybrid intelligence in next generation manufacturing: An outlook on new forms of collaboration between human and algorithmic decision-makers in the factory of the future (pp. 139–158). Forecasting Next Generation Manufacturing: Digital Shadows, Human-Machine Collaboration, and Data-driven Business Models.

Price, C. J., & Taylor, N. S. (2002). Automated multiple failure FMEA. Reliability Engineering & System Safety, 76 (1), 1–10.

Psarommatis, F., & Kiritsis, D. (2022). A hybrid decision support system for automating decision making in the event of defects in the era of zero defect manufacturing. Journal of Industrial Information Integration, 26 , 100263.

Rajpathak, D., & De, S. (2016). A data-and ontology-driven text mining-based construction of reliability model to analyze and predict component failures. Knowledge and Information Systems, 46 (1), 87–113.

Rastayesh, S., Bahrebar, S., Blaabjerg, F., Zhou, D., Wang, H., & Dalsgaard Sørensen, J. (2019). A system engineering approach using FMEA and Bayesian network for risk analysis-a case study. Sustainability, 12 (1), 77.

Rausand, M., & Øien, K. (1996). The basic concepts of failure analysis. Reliability Engineering & System Safety, 53 (1), 73–83.

Rivera Torres, P. J., Serrano Mercado, E. I., Llanes Santiago, O., & Anido Rifón, L. (2018). Modeling preventive maintenance of manufacturing processes with probabilistic Boolean networks with interventions. Journal of Intelligent Manufacturing, 29 (8), 1941–1952.

Rokach, L., & Hutter, D. (2012). Automatic discovery of the root causes for quality drift in high dimensionality manufacturing processes. Journal of Intelligent Manufacturing, 23 (5), 1915–1930.

Ruijters, E., & Stoelinga, M. (2015). Fault tree analysis: A survey of the state-of-the-art in modeling, analysis and tools. Computer Science Review, 15 , 29–62.

Sariyer, G., Mangla, S. K., Kazancoglu, Y., Ocal Tasar, C., & Luthra, S. (2021). Data analytics for quality management in Industry 4.0 from a MSME perspective. Annals of Operations Research, 23 , 1–19.

Sayed, M. S., & Lohse, N. (2014). Ontology-driven generation of Bayesian diagnostic models for assembly systems. The International Journal of Advanced Manufacturing Technology, 74 (5), 1033–1052.

Schuh, G., Gützlaff, A., Thomas, K., & Welsing, M. (2021). Machine learning based defect detection in a low automated assembly environment. Procedia CIRP, 104 , 265–270.

Shi, H., Wang, L., Li, X. Y., & Liu, H. C. (2020). A novel method for failure mode and effects analysis using fuzzy evidential reasoning and fuzzy Petri nets. Journal of Ambient Intelligence and Humanized Computing, 11 (6), 2381–2395.

Snooke, N., & Price, C. (2012). Automated FMEA based diagnostic symptom generation. Advanced Engineering Informatics, 26 (4), 870–888.

Spreafico, C., Russo, D., & Rizzi, C. (2017). A state-of-the-art review of FMEA/FMECA including patents. Computer Science Review, 25 , 19–28.

Stamatis, D. H. (2003). Failure mode and effect analysis: FMEA from theory to execution . ASQ Quality Press.

Steenwinckel B, Heyvaert P, De Paepe D, Janssens O, Vanden Hautte S, Dimou A, De Turck F, Van Hoecke S, Ongenae F (2018) Towards adaptive anomaly detection and root cause analysis by automated extraction of knowledge from risk analyses. 9th International Semantic Sensor Networks Workshop, Co-Located with 17th International Semantic Web Conference (ISWC 2018) , Vol. 2213, pp. 17–31.

Sun, Y., Qin, W., Zhuang, Z., & Xu, H. (2021). An adaptive fault detection and root-cause analysis scheme for complex industrial processes using moving window KPCA and information geometric causal inference. Journal of Intelligent Manufacturing, 32 (7), 2007–2021.

Tari, J. J., & Sabater, V. (2004). Quality tools and techniques: Are they necessary for quality management? International Journal of Production Economics, 92 (3), 267–280.

Tay, K. M., Jong, C. H., & Lim, C. P. (2015). A clustering-based failure mode and effect analysis model and its application to the edible bird nest industry. Neural Computing and Applications, 26 (3), 551–560.

Teoh, P. C., & Case, K. (2004). Failure modes and effects analysis through knowledge modelling. Journal of Materials Processing Technology, 153 , 253–260.

Teoh, P. C., & Case, K. (2005). An evaluation of failure modes and effects analysis generation method for conceptual design. International Journal of Computer Integrated Manufacturing, 18 (4), 279–293.

Thomé, A. M. T., Scavarda, L. F., & Scavarda, A. J. (2016). Conducting systematic literature review in operations management. Production Planning & Control, 27 (5), 408–420.

Tönnes, W. (2018). Applying data of historical defects to increase efficiency of rework in assembly. Procedia CIRP, 72 , 255–260.

van der Aalst, W. M. (2021). Hybrid intelligence: To automate or not to automate, that is the question. International Journal of Information Systems and Project Management, 9 (2), 5–20.

Waghen, K., & Ouali, M. S. (2021). Multi-level interpretable logic tree analysis: A data-driven approach for hierarchical causality analysis. Expert Systems with Applications, 178 , 115035.

Wan, C., Yan, X., Zhang, D., Qu, Z., & Yang, Z. (2019). An advanced fuzzy Bayesian-based FMEA approach for assessing maritime supply chain risks. Transportation Research Part E, 125 , 222–240.

Wang, L., Li, S., Wei, O., Huang, M., & Hu, J. (2018). An automated fault tree generation approach with fault configuration based on model checking. IEEE Access, 6 , 46900–46914.

Wang, Q., Jia, G., Jia, Y., & Song, W. (2021). A new approach for risk assessment of failure modes considering risk interaction and propagation effects. Reliability Engineering & System Safety, 216 , 108044.

Williams, P. M. (2001). Techniques for root cause analysis. Baylor University Medical Center Proceedings, 14 (2), 154–157.

Wu, Z., Liu, W., & Nie, W. (2021). Literature review and prospect of the development and application of FMEA in manufacturing industry. The International Journal of Advanced Manufacturing Technology, 112 (5), 1409–1436.

Xu, Z., & Dang, Y. (2020). Automated digital cause-and-effect diagrams to assist causal analysis in problem-solving: A data-driven approach. International Journal of Production Research, 58 (17), 5359–5379.

Xu, Z., & Dang, Y. (2023). Data-driven causal knowledge graph construction for root cause analysis in quality problem solving. International Journal of Production Research, 61 (10), 3227–3245.

Xu, Z., Dang, Y., Munro, P., & Wang, Y. (2020). A data-driven approach for constructing the component-failure mode matrix for FMEA. Journal of Intelligent Manufacturing, 31 (1), 249–265.

Yang, C., Zou, Y., Lai, P., & Jiang, N. (2015). Data mining-based methods for fault isolation with validated fmea model ranking. Applied Intelligence, 43 (4), 913–923.

Yang, S., Bian, C., Li, X., Tan, L., & Tang, D. (2018). Optimized fault diagnosis based on FMEA-style CBR and BN for embedded software system. The International Journal of Advanced Manufacturing Technology, 94 (9), 3441–3453.

Yang, S., Liu, H., Zhang, Y., Arndt, T., Hofmann, C., Häfner, B., & Lanza, G. (2020). A data-driven approach for quality analytics of screwing processes in a global learning factory. Procedia Manufacturing, 45 , 454–459.

Yang, S., & Liu, T. (1998). A Petri net approach to early failure detection and isolation for preventive maintenance. Quality and Reliability Engineering International, 14 (5), 319-330.

Yang, Y. J., Xiong, Y. L., Zhang, X. Y., Wang, G. H., & Zou, B. (2022). Reliability analysis of continuous emission monitoring system with common cause failure based on fuzzy FMECA and Bayesian networks. Annals of Operations Research, 311 , 451–467.

Yang, Z. X., Zheng, Y. Y., & Xue, J. X. (2009). Development of automatic fault tree synthesis system using decision matrix. International Journal of Production Economics, 121 (1), 49–56.

Ye, P., Wang, X., Zheng, W., Wei, Q., & Wang, F. Y. (2022). Parallel cognition: Hybrid intelligence for human-machine interaction and management. Frontiers of Information Technology & Electronic Engineering, 23 (12), 1765–1779.

Yucesan, M., Gul, M., & Celik, E. (2021). A holistic FMEA approach by fuzzy-based Bayesian network and best-worst method. Complex & Intelligent Systems, 7 (3), 1547–1564.

Yuniarto, H. (2012). The shortcomings of existing root cause analysis tools. Proceedings of the World Congress on Engineering, 3 , 186–191.

Zhang, S., Xie, X., & Qu, H. (2023). A data-driven workflow for evaporation performance degradation analysis: A full-scale case study in the herbal medicine manufacturing industry. Journal of Intelligent Manufacturing, 34 , 651–668.

Zheng, T., Ardolino, M., Bacchetti, A., & Perona, M. (2021). The applications of industry 4.0 technologies in manufacturing context: a systematic literature review. International Journal of Production Research, 59 (6), 1922–1954.

Zhou, A., Yu, D., & Zhang, W. (2015). A research on intelligent fault diagnosis of wind turbines based on ontology and FMECA. Advanced Engineering Informatics, 29 (1), 115–125.

Zhu, C., & Zhang, T. (2022). A review on the realization methods of dynamic fault tree. Quality and Reliability Engineering International, 38 (6), 3233–3251.

Zhu, J. H., Chen, Z. S., Shuai, B., Pedrycz, W., Chin, K. S., & Martínez, L. (2021). Failure mode and effect analysis: A three-way decision approach. Engineering Applications of Artificial Intelligence, 106 , 104505.

Zuo, X., Li, B., & Yang, J. (2016). Error sensitivity analysis and precision distribution for multi-operation machining processes based on error propagation model. The International Journal of Advanced Manufacturing Technology, 86 (1), 269–280.

Download references

This research is funded by Flanders Make under the project AQUME_SBO, project number 2022-0151. Flanders Make is the Flemish strategic research center for the manufacturing industry in Belgium.

Author information

Authors and affiliations.

Department of Industrial Systems Engineering and Product Design, Ghent University, 9052, Ghent, Belgium

Mahdi Mokhtarzadeh, Jorge Rodríguez-Echeverría, Ivana Semanjski & Sidharta Gautama

FlandersMake@UGent–corelab ISyE, Lommel, Belgium

Escuela Superior Politécnica del Litoral, ESPOL, Facultad de Ingeniería en Electricidad y Computación, ESPOL Polytechnic University, Campus Gustavo Galindo, Km 30.5 Vía Perimetral, P.O. Box 09-01-5863, 090112, Guayaquil, Ecuador

Jorge Rodríguez-Echeverría

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mahdi Mokhtarzadeh .

Ethics declarations

Competing interest.

The authors report there are no competing interests to declare.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 137 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Mokhtarzadeh, M., Rodríguez-Echeverría, J., Semanjski, I. et al. Hybrid intelligence failure analysis for industry 4.0: a literature review and future prospective. J Intell Manuf (2024). https://doi.org/10.1007/s10845-024-02376-5

Download citation

Received : 27 June 2023

Accepted : 14 March 2024

Published : 22 April 2024

DOI : https://doi.org/10.1007/s10845-024-02376-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Automated failure analysis
• Data-driven failure analysis
• Human–machine cooperation
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Grad Med Educ
• v.8(3); 2016 Jul

The Literature Review: A Foundation for High-Quality Medical Education Research

a  These are subscription resources. Researchers should check with their librarian to determine their access rights.

Despite a surge in published scholarship in medical education 1 and rapid growth in journals that publish educational research, manuscript acceptance rates continue to fall. 2 Failure to conduct a thorough, accurate, and up-to-date literature review identifying an important problem and placing the study in context is consistently identified as one of the top reasons for rejection. 3 , 4 The purpose of this editorial is to provide a road map and practical recommendations for planning a literature review. By understanding the goals of a literature review and following a few basic processes, authors can enhance both the quality of their educational research and the likelihood of publication in the Journal of Graduate Medical Education ( JGME ) and in other journals.

The Literature Review Defined

In medical education, no organization has articulated a formal definition of a literature review for a research paper; thus, a literature review can take a number of forms. Depending on the type of article, target journal, and specific topic, these forms will vary in methodology, rigor, and depth. Several organizations have published guidelines for conducting an intensive literature search intended for formal systematic reviews, both broadly (eg, PRISMA) 5 and within medical education, 6 and there are excellent commentaries to guide authors of systematic reviews. 7 , 8

• A literature review forms the basis for high-quality medical education research and helps maximize relevance, originality, generalizability, and impact.
• A literature review provides context, informs methodology, maximizes innovation, avoids duplicative research, and ensures that professional standards are met.
• Literature reviews take time, are iterative, and should continue throughout the research process.
• Researchers should maximize the use of human resources (librarians, colleagues), search tools (databases/search engines), and existing literature (related articles).
• Keeping organized is critical.

Such work is outside the scope of this article, which focuses on literature reviews to inform reports of original medical education research. We define such a literature review as a synthetic review and summary of what is known and unknown regarding the topic of a scholarly body of work, including the current work's place within the existing knowledge . While this type of literature review may not require the intensive search processes mandated by systematic reviews, it merits a thoughtful and rigorous approach.

Purpose and Importance of the Literature Review

An understanding of the current literature is critical for all phases of a research study. Lingard 9 recently invoked the “journal-as-conversation” metaphor as a way of understanding how one's research fits into the larger medical education conversation. As she described it: “Imagine yourself joining a conversation at a social event. After you hang about eavesdropping to get the drift of what's being said (the conversational equivalent of the literature review), you join the conversation with a contribution that signals your shared interest in the topic, your knowledge of what's already been said, and your intention.” 9

The literature review helps any researcher “join the conversation” by providing context, informing methodology, identifying innovation, minimizing duplicative research, and ensuring that professional standards are met. Understanding the current literature also promotes scholarship, as proposed by Boyer, 10 by contributing to 5 of the 6 standards by which scholarly work should be evaluated. 11 Specifically, the review helps the researcher (1) articulate clear goals, (2) show evidence of adequate preparation, (3) select appropriate methods, (4) communicate relevant results, and (5) engage in reflective critique.

Failure to conduct a high-quality literature review is associated with several problems identified in the medical education literature, including studies that are repetitive, not grounded in theory, methodologically weak, and fail to expand knowledge beyond a single setting. 12 Indeed, medical education scholars complain that many studies repeat work already published and contribute little new knowledge—a likely cause of which is failure to conduct a proper literature review. 3 , 4

Likewise, studies that lack theoretical grounding or a conceptual framework make study design and interpretation difficult. 13 When theory is used in medical education studies, it is often invoked at a superficial level. As Norman 14 noted, when theory is used appropriately, it helps articulate variables that might be linked together and why, and it allows the researcher to make hypotheses and define a study's context and scope. Ultimately, a proper literature review is a first critical step toward identifying relevant conceptual frameworks.

Another problem is that many medical education studies are methodologically weak. 12 Good research requires trained investigators who can articulate relevant research questions, operationally define variables of interest, and choose the best method for specific research questions. Conducting a proper literature review helps both novice and experienced researchers select rigorous research methodologies.

Finally, many studies in medical education are “one-offs,” that is, single studies undertaken because the opportunity presented itself locally. Such studies frequently are not oriented toward progressive knowledge building and generalization to other settings. A firm grasp of the literature can encourage a programmatic approach to research.

Approaching the Literature Review

Considering these issues, journals have a responsibility to demand from authors a thoughtful synthesis of their study's position within the field, and it is the authors' responsibility to provide such a synthesis, based on a literature review. The aforementioned purposes of the literature review mandate that the review occurs throughout all phases of a study, from conception and design, to implementation and analysis, to manuscript preparation and submission.

Planning the literature review requires understanding of journal requirements, which vary greatly by journal ( table 1 ). Authors are advised to take note of common problems with reporting results of the literature review. Table 2 lists the most common problems that we have encountered as authors, reviewers, and editors.

Sample of Journals' Author Instructions for Literature Reviews Conducted as Part of Original Research Article a

Common Problem Areas for Reporting Literature Reviews in the Context of Scholarly Articles

Locating and Organizing the Literature

Three resources may facilitate identifying relevant literature: human resources, search tools, and related literature. As the process requires time, it is important to begin searching for literature early in the process (ie, the study design phase). Identifying and understanding relevant studies will increase the likelihood of designing a relevant, adaptable, generalizable, and novel study that is based on educational or learning theory and can maximize impact.

Human Resources

A medical librarian can help translate research interests into an effective search strategy, familiarize researchers with available information resources, provide information on organizing information, and introduce strategies for keeping current with emerging research. Often, librarians are also aware of research across their institutions and may be able to connect researchers with similar interests. Reaching out to colleagues for suggestions may help researchers quickly locate resources that would not otherwise be on their radar.

During this process, researchers will likely identify other researchers writing on aspects of their topic. Researchers should consider searching for the publications of these relevant researchers (see table 3 for search strategies). Additionally, institutional websites may include curriculum vitae of such relevant faculty with access to their entire publication record, including difficult to locate publications, such as book chapters, dissertations, and technical reports.

Strategies for Finding Related Researcher Publications in Databases and Search Engines

Search Tools and Related Literature

Researchers will locate the majority of needed information using databases and search engines. Excellent resources are available to guide researchers in the mechanics of literature searches. 15 , 16

Because medical education research draws on a variety of disciplines, researchers should include search tools with coverage beyond medicine (eg, psychology, nursing, education, and anthropology) and that cover several publication types, such as reports, standards, conference abstracts, and book chapters (see the box for several information resources). Many search tools include options for viewing citations of selected articles. Examining cited references provides additional articles for review and a sense of the influence of the selected article on its field.

Box Information Resources

• Web of Science a
• Education Resource Information Center (ERIC)
• Cumulative Index of Nursing & Allied Health (CINAHL) a
• Google Scholar

Once relevant articles are located, it is useful to mine those articles for additional citations. One strategy is to examine references of key articles, especially review articles, for relevant citations.

Getting Organized

As the aforementioned resources will likely provide a tremendous amount of information, organization is crucial. Researchers should determine which details are most important to their study (eg, participants, setting, methods, and outcomes) and generate a strategy for keeping those details organized and accessible. Increasingly, researchers utilize digital tools, such as Evernote, to capture such information, which enables accessibility across digital workspaces and search capabilities. Use of citation managers can also be helpful as they store citations and, in some cases, can generate bibliographies ( table 4 ).

Citation Managers

Knowing When to Say When

Researchers often ask how to know when they have located enough citations. Unfortunately, there is no magic or ideal number of citations to collect. One strategy for checking coverage of the literature is to inspect references of relevant articles. As researchers review references they will start noticing a repetition of the same articles with few new articles appearing. This can indicate that the researcher has covered the literature base on a particular topic.

Putting It All Together

In preparing to write a research paper, it is important to consider which citations to include and how they will inform the introduction and discussion sections. The “Instructions to Authors” for the targeted journal will often provide guidance on structuring the literature review (or introduction) and the number of total citations permitted for each article category. Reviewing articles of similar type published in the targeted journal can also provide guidance regarding structure and average lengths of the introduction and discussion sections.

When selecting references for the introduction consider those that illustrate core background theoretical and methodological concepts, as well as recent relevant studies. The introduction should be brief and present references not as a laundry list or narrative of available literature, but rather as a synthesized summary to provide context for the current study and to identify the gap in the literature that the study intends to fill. For the discussion, citations should be thoughtfully selected to compare and contrast the present study's findings with the current literature and to indicate how the present study moves the field forward.

To facilitate writing a literature review, journals are increasingly providing helpful features to guide authors. For example, the resources available through JGME include several articles on writing. 17 The journal Perspectives on Medical Education recently launched “The Writer's Craft,” which is intended to help medical educators improve their writing. Additionally, many institutions have writing centers that provide web-based materials on writing a literature review, and some even have writing coaches.

The literature review is a vital part of medical education research and should occur throughout the research process to help researchers design a strong study and effectively communicate study results and importance. To achieve these goals, researchers are advised to plan and execute the literature review carefully. The guidance in this editorial provides considerations and recommendations that may improve the quality of literature reviews.

• Study Guides
• Homework Questions

Literature Review