literature review as a process

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 22 April 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Research Process :: Step by Step

• Introduction
• Select Topic
• Identify Keywords
• Background Information
• Develop Research Questions
• Refine Topic
• Search Strategy
• Popular Databases
• Evaluate Sources
• Types of Periodicals
• Reading Scholarly Articles
• Primary & Secondary Sources
• Organize / Take Notes
• Writing & Grammar Resources
• Annotated Bibliography
• Literature Review
• Citation Styles
• Paraphrasing
• Privacy / Confidentiality
• Research Process
• Selecting Your Topic
• Identifying Keywords
• Gathering Background Info
• Evaluating Sources

Organize the literature review into sections that present themes or identify trends, including relevant theory. You are not trying to list all the material published, but to synthesize and evaluate it according to the guiding concept of your thesis or research question.  

What is a literature review?

A literature review is an account of what has been published on a topic by accredited scholars and researchers. Occasionally you will be asked to write one as a separate assignment, but more often it is part of the introduction to an essay, research report, or thesis. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries

A literature review must do these things:

• be organized around and related directly to the thesis or research question you are developing
• synthesize results into a summary of what is and is not known
• identify areas of controversy in the literature
• formulate questions that need further research

Ask yourself questions like these:

• What is the specific thesis, problem, or research question that my literature review helps to define?
• What type of literature review am I conducting? Am I looking at issues of theory? methodology? policy? quantitative research (e.g. on the effectiveness of a new procedure)? qualitative research (e.g., studies of loneliness among migrant workers)?
• What is the scope of my literature review? What types of publications am I using (e.g., journals, books, government documents, popular media)? What discipline am I working in (e.g., nursing psychology, sociology, medicine)?
• How good was my information seeking? Has my search been wide enough to ensure I've found all the relevant material? Has it been narrow enough to exclude irrelevant material? Is the number of sources I've used appropriate for the length of my paper?
• Have I critically analyzed the literature I use? Do I follow through a set of concepts and questions, comparing items to each other in the ways they deal with them? Instead of just listing and summarizing items, do I assess them, discussing strengths and weaknesses?
• Have I cited and discussed studies contrary to my perspective?
• Will the reader find my literature review relevant, appropriate, and useful?

Ask yourself questions like these about each book or article you include:

• Has the author formulated a problem/issue?
• Is it clearly defined? Is its significance (scope, severity, relevance) clearly established?
• Could the problem have been approached more effectively from another perspective?
• What is the author's research orientation (e.g., interpretive, critical science, combination)?
• What is the author's theoretical framework (e.g., psychological, developmental, feminist)?
• What is the relationship between the theoretical and research perspectives?
• Has the author evaluated the literature relevant to the problem/issue? Does the author include literature taking positions she or he does not agree with?
• In a research study, how good are the basic components of the study design (e.g., population, intervention, outcome)? How accurate and valid are the measurements? Is the analysis of the data accurate and relevant to the research question? Are the conclusions validly based upon the data and analysis?
• In material written for a popular readership, does the author use appeals to emotion, one-sided examples, or rhetorically-charged language and tone? Is there an objective basis to the reasoning, or is the author merely "proving" what he or she already believes?
• How does the author structure the argument? Can you "deconstruct" the flow of the argument to see whether or where it breaks down logically (e.g., in establishing cause-effect relationships)?
• In what ways does this book or article contribute to our understanding of the problem under study, and in what ways is it useful for practice? What are the strengths and limitations?
• How does this book or article relate to the specific thesis or question I am developing?

Text written by Dena Taylor, Health Sciences Writing Centre, University of Toronto

http://www.writing.utoronto.ca/advice/specific-types-of-writing/literature-review

• << Previous: Annotated Bibliography
• Next: Step 5: Cite Sources >>
• Last Updated: Apr 19, 2024 12:43 PM
• URL: https://libguides.uta.edu/researchprocess

University of Texas Arlington Libraries 702 Planetarium Place · Arlington, TX 76019 · 817-272-3000

• Internet Privacy
• Accessibility
• Problems with a guide? Contact Us.

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

How To Write An A-Grade Literature Review

3 straightforward steps (with examples) + free template.

By: Derek Jansen (MBA) | Expert Reviewed By: Dr. Eunice Rautenbach | October 2019

Quality research is about building onto the existing work of others , “standing on the shoulders of giants”, as Newton put it. The literature review chapter of your dissertation, thesis or research project is where you synthesise this prior work and lay the theoretical foundation for your own research.

Long story short, this chapter is a pretty big deal, which is why you want to make sure you get it right . In this post, I’ll show you exactly how to write a literature review in three straightforward steps, so you can conquer this vital chapter (the smart way).

Overview: The Literature Review Process

• Understanding the “ why “
• Finding the relevant literature
• Cataloguing and synthesising the information
• Outlining & writing up your literature review
• Example of a literature review

But first, the “why”…

Before we unpack how to write the literature review chapter, we’ve got to look at the why . To put it bluntly, if you don’t understand the function and purpose of the literature review process, there’s no way you can pull it off well. So, what exactly is the purpose of the literature review?

Well, there are (at least) four core functions:

• For you to gain an understanding (and demonstrate this understanding) of where the research is at currently, what the key arguments and disagreements are.
• For you to identify the gap(s) in the literature and then use this as justification for your own research topic.
• To help you build a conceptual framework for empirical testing (if applicable to your research topic).
• To inform your methodological choices and help you source tried and tested questionnaires (for interviews ) and measurement instruments (for surveys ).

Most students understand the first point but don’t give any thought to the rest. To get the most from the literature review process, you must keep all four points front of mind as you review the literature (more on this shortly), or you’ll land up with a wonky foundation.

Okay – with the why out the way, let’s move on to the how . As mentioned above, writing your literature review is a process, which I’ll break down into three steps:

• Finding the most suitable literature
• Understanding , distilling and organising the literature
• Planning and writing up your literature review chapter

Importantly, you must complete steps one and two before you start writing up your chapter. I know it’s very tempting, but don’t try to kill two birds with one stone and write as you read. You’ll invariably end up wasting huge amounts of time re-writing and re-shaping, or you’ll just land up with a disjointed, hard-to-digest mess . Instead, you need to read first and distil the information, then plan and execute the writing.

Step 1: Find the relevant literature

Naturally, the first step in the literature review journey is to hunt down the existing research that’s relevant to your topic. While you probably already have a decent base of this from your research proposal , you need to expand on this substantially in the dissertation or thesis itself.

Essentially, you need to be looking for any existing literature that potentially helps you answer your research question (or develop it, if that’s not yet pinned down). There are numerous ways to find relevant literature, but I’ll cover my top four tactics here. I’d suggest combining all four methods to ensure that nothing slips past you:

Method 1 – Google Scholar Scrubbing

Google’s academic search engine, Google Scholar , is a great starting point as it provides a good high-level view of the relevant journal articles for whatever keyword you throw at it. Most valuably, it tells you how many times each article has been cited, which gives you an idea of how credible (or at least, popular) it is. Some articles will be free to access, while others will require an account, which brings us to the next method.

Method 2 – University Database Scrounging

Generally, universities provide students with access to an online library, which provides access to many (but not all) of the major journals.

So, if you find an article using Google Scholar that requires paid access (which is quite likely), search for that article in your university’s database – if it’s listed there, you’ll have access. Note that, generally, the search engine capabilities of these databases are poor, so make sure you search for the exact article name, or you might not find it.

Method 3 – Journal Article Snowballing

At the end of every academic journal article, you’ll find a list of references. As with any academic writing, these references are the building blocks of the article, so if the article is relevant to your topic, there’s a good chance a portion of the referenced works will be too. Do a quick scan of the titles and see what seems relevant, then search for the relevant ones in your university’s database.

Method 4 – Dissertation Scavenging

Similar to Method 3 above, you can leverage other students’ dissertations. All you have to do is skim through literature review chapters of existing dissertations related to your topic and you’ll find a gold mine of potential literature. Usually, your university will provide you with access to previous students’ dissertations, but you can also find a much larger selection in the following databases:

• Open Access Theses & Dissertations
• Stanford SearchWorks

Keep in mind that dissertations and theses are not as academically sound as published, peer-reviewed journal articles (because they’re written by students, not professionals), so be sure to check the credibility of any sources you find using this method. You can do this by assessing the citation count of any given article in Google Scholar. If you need help with assessing the credibility of any article, or with finding relevant research in general, you can chat with one of our Research Specialists .

Alright – with a good base of literature firmly under your belt, it’s time to move onto the next step.

Need a helping hand?

Step 2: Log, catalogue and synthesise

Once you’ve built a little treasure trove of articles, it’s time to get reading and start digesting the information – what does it all mean?

While I present steps one and two (hunting and digesting) as sequential, in reality, it’s more of a back-and-forth tango – you’ll read a little , then have an idea, spot a new citation, or a new potential variable, and then go back to searching for articles. This is perfectly natural – through the reading process, your thoughts will develop , new avenues might crop up, and directional adjustments might arise. This is, after all, one of the main purposes of the literature review process (i.e. to familiarise yourself with the current state of research in your field).

As you’re working through your treasure chest, it’s essential that you simultaneously start organising the information. There are three aspects to this:

• Logging reference information
• Building an organised catalogue
• Distilling and synthesising the information

I’ll discuss each of these below:

2.1 – Log the reference information

As you read each article, you should add it to your reference management software. I usually recommend Mendeley for this purpose (see the Mendeley 101 video below), but you can use whichever software you’re comfortable with. Most importantly, make sure you load EVERY article you read into your reference manager, even if it doesn’t seem very relevant at the time.

2.2 – Build an organised catalogue

In the beginning, you might feel confident that you can remember who said what, where, and what their main arguments were. Trust me, you won’t. If you do a thorough review of the relevant literature (as you must!), you’re going to read many, many articles, and it’s simply impossible to remember who said what, when, and in what context . Also, without the bird’s eye view that a catalogue provides, you’ll miss connections between various articles, and have no view of how the research developed over time. Simply put, it’s essential to build your own catalogue of the literature.

I would suggest using Excel to build your catalogue, as it allows you to run filters, colour code and sort – all very useful when your list grows large (which it will). How you lay your spreadsheet out is up to you, but I’d suggest you have the following columns (at minimum):

• Author, date, title – Start with three columns containing this core information. This will make it easy for you to search for titles with certain words, order research by date, or group by author.
• Categories or keywords – You can either create multiple columns, one for each category/theme and then tick the relevant categories, or you can have one column with keywords.
• Key arguments/points – Use this column to succinctly convey the essence of the article, the key arguments and implications thereof for your research.
• Context – Note the socioeconomic context in which the research was undertaken. For example, US-based, respondents aged 25-35, lower- income, etc. This will be useful for making an argument about gaps in the research.
• Methodology – Note which methodology was used and why. Also, note any issues you feel arise due to the methodology. Again, you can use this to make an argument about gaps in the research.
• Quotations – Note down any quoteworthy lines you feel might be useful later.
• Notes – Make notes about anything not already covered. For example, linkages to or disagreements with other theories, questions raised but unanswered, shortcomings or limitations, and so forth.

If you’d like, you can try out our free catalog template here (see screenshot below).

2.3 – Digest and synthesise

Most importantly, as you work through the literature and build your catalogue, you need to synthesise all the information in your own mind – how does it all fit together? Look for links between the various articles and try to develop a bigger picture view of the state of the research. Some important questions to ask yourself are:

• What answers does the existing research provide to my own research questions ?
• Which points do the researchers agree (and disagree) on?
• How has the research developed over time?
• Where do the gaps in the current research lie?

To help you develop a big-picture view and synthesise all the information, you might find mind mapping software such as Freemind useful. Alternatively, if you’re a fan of physical note-taking, investing in a large whiteboard might work for you.

Step 3: Outline and write it up!

Once you’re satisfied that you have digested and distilled all the relevant literature in your mind, it’s time to put pen to paper (or rather, fingers to keyboard). There are two steps here – outlining and writing:

3.1 – Draw up your outline

Having spent so much time reading, it might be tempting to just start writing up without a clear structure in mind. However, it’s critically important to decide on your structure and develop a detailed outline before you write anything. Your literature review chapter needs to present a clear, logical and an easy to follow narrative – and that requires some planning. Don’t try to wing it!

Naturally, you won’t always follow the plan to the letter, but without a detailed outline, you’re more than likely going to end up with a disjointed pile of waffle , and then you’re going to spend a far greater amount of time re-writing, hacking and patching. The adage, “measure twice, cut once” is very suitable here.

In terms of structure, the first decision you’ll have to make is whether you’ll lay out your review thematically (into themes) or chronologically (by date/period). The right choice depends on your topic, research objectives and research questions, which we discuss in this article .

Once that’s decided, you need to draw up an outline of your entire chapter in bullet point format. Try to get as detailed as possible, so that you know exactly what you’ll cover where, how each section will connect to the next, and how your entire argument will develop throughout the chapter. Also, at this stage, it’s a good idea to allocate rough word count limits for each section, so that you can identify word count problems before you’ve spent weeks or months writing!

PS – check out our free literature review chapter template…

3.2 – Get writing

With a detailed outline at your side, it’s time to start writing up (finally!). At this stage, it’s common to feel a bit of writer’s block and find yourself procrastinating under the pressure of finally having to put something on paper. To help with this, remember that the objective of the first draft is not perfection – it’s simply to get your thoughts out of your head and onto paper, after which you can refine them. The structure might change a little, the word count allocations might shift and shuffle, and you might add or remove a section – that’s all okay. Don’t worry about all this on your first draft – just get your thoughts down on paper.

Once you’ve got a full first draft (however rough it may be), step away from it for a day or two (longer if you can) and then come back at it with fresh eyes. Pay particular attention to the flow and narrative – does it fall fit together and flow from one section to another smoothly? Now’s the time to try to improve the linkage from each section to the next, tighten up the writing to be more concise, trim down word count and sand it down into a more digestible read.

Once you’ve done that, give your writing to a friend or colleague who is not a subject matter expert and ask them if they understand the overall discussion. The best way to assess this is to ask them to explain the chapter back to you. This technique will give you a strong indication of which points were clearly communicated and which weren’t. If you’re working with Grad Coach, this is a good time to have your Research Specialist review your chapter.

Finally, tighten it up and send it off to your supervisor for comment. Some might argue that you should be sending your work to your supervisor sooner than this (indeed your university might formally require this), but in my experience, supervisors are extremely short on time (and often patience), so, the more refined your chapter is, the less time they’ll waste on addressing basic issues (which you know about already) and the more time they’ll spend on valuable feedback that will increase your mark-earning potential.

Literature Review Example

In the video below, we unpack an actual literature review so that you can see how all the core components come together in reality.

Let’s Recap

In this post, we’ve covered how to research and write up a high-quality literature review chapter. Let’s do a quick recap of the key takeaways:

• It is essential to understand the WHY of the literature review before you read or write anything. Make sure you understand the 4 core functions of the process.
• The first step is to hunt down the relevant literature . You can do this using Google Scholar, your university database, the snowballing technique and by reviewing other dissertations and theses.
• Next, you need to log all the articles in your reference manager , build your own catalogue of literature and synthesise all the research.
• Following that, you need to develop a detailed outline of your entire chapter – the more detail the better. Don’t start writing without a clear outline (on paper, not in your head!)
• Write up your first draft in rough form – don’t aim for perfection. Remember, done beats perfect.
• Refine your second draft and get a layman’s perspective on it . Then tighten it up and submit it to your supervisor.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

38 Comments

Thank you very much. This page is an eye opener and easy to comprehend.

This is awesome!

I wish I come across GradCoach earlier enough.

But all the same I’ll make use of this opportunity to the fullest.

Thank you for this good job.

Keep it up!

You’re welcome, Yinka. Thank you for the kind words. All the best writing your literature review.

Thank you for a very useful literature review session. Although I am doing most of the steps…it being my first masters an Mphil is a self study and one not sure you are on the right track. I have an amazing supervisor but one also knows they are super busy. So not wanting to bother on the minutae. Thank you.

You’re most welcome, Renee. Good luck with your literature review 🙂

This has been really helpful. Will make full use of it. 🙂

Thank you Gradcoach.

Really agreed. Admirable effort

thank you for this beautiful well explained recap.

Thank you so much for your guide of video and other instructions for the dissertation writing.

It is instrumental. It encouraged me to write a dissertation now.

Thank you the video was great – from someone that knows nothing thankyou

an amazing and very constructive way of presetting a topic, very useful, thanks for the effort,

It is timely

It is very good video of guidance for writing a research proposal and a dissertation. Since I have been watching and reading instructions, I have started my research proposal to write. I appreciate to Mr Jansen hugely.

I learn a lot from your videos. Very comprehensive and detailed.

Thank you for sharing your knowledge. As a research student, you learn better with your learning tips in research

I was really stuck in reading and gathering information but after watching these things are cleared thanks, it is so helpful.

Really helpful, Thank you for the effort in showing such information

This is super helpful thank you very much.

Thank you for this whole literature writing review.You have simplified the process.

I’m so glad I found GradCoach. Excellent information, Clear explanation, and Easy to follow, Many thanks Derek!

You’re welcome, Maithe. Good luck writing your literature review 🙂

Thank you Coach, you have greatly enriched and improved my knowledge

Great piece, so enriching and it is going to help me a great lot in my project and thesis, thanks so much

This is THE BEST site for ANYONE doing a masters or doctorate! Thank you for the sound advice and templates. You rock!

Thanks, Stephanie 🙂

This is mind blowing, the detailed explanation and simplicity is perfect.

I am doing two papers on my final year thesis, and I must stay I feel very confident to face both headlong after reading this article.

thank you so much.

if anyone is to get a paper done on time and in the best way possible, GRADCOACH is certainly the go to area!

This is very good video which is well explained with detailed explanation

Thank you excellent piece of work and great mentoring

Thanks, it was useful

Thank you very much. the video and the information were very helpful.

Good morning scholar. I’m delighted coming to know you even before the commencement of my dissertation which hopefully is expected in not more than six months from now. I would love to engage my study under your guidance from the beginning to the end. I love to know how to do good job

Thank you so much Derek for such useful information on writing up a good literature review. I am at a stage where I need to start writing my one. My proposal was accepted late last year but I honestly did not know where to start

Like the name of your YouTube implies you are GRAD (great,resource person, about dissertation). In short you are smart enough in coaching research work.

This is a very well thought out webpage. Very informative and a great read.

Very timely.

I appreciate.

Very comprehensive and eye opener for me as beginner in postgraduate study. Well explained and easy to understand. Appreciate and good reference in guiding me in my research journey. Thank you

Thank you. I requested to download the free literature review template, however, your website wouldn’t allow me to complete the request or complete a download. May I request that you email me the free template? Thank you.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• University of Oregon Libraries
• Research Guides

How to Write a Literature Review

Who is my subject librarian.

• Literature Reviews: A Recap
• Reading Journal Articles
• Does it Describe a Literature Review?
• 1. Identify the Question
• 2. Review Discipline Styles
• Searching Article Databases
• Finding Full-Text of an Article
• Citation Chaining
• When to Stop Searching
• 4. Manage Your References
• 5. Critically Analyze and Evaluate
• 6. Synthesize
• 7. Write a Literature Review

This guide has a Creative Commons Attribution-NonCommercial-ShareAlike (CC BY-NC-SA) License.

This license lets others remix, tweak, and build upon your work non-commercially, as long as they credit you and license their new creations under the identical terms.

• Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License Deed

This guide is organized around the 7 steps in the graphic to the left to help you understand the process of writing a literature review.

This guide will help you to

• Define a literature review
• Recognize that different fields of study have their own way to perform and write literature reviews
• Prepare to search the literature
• Read critically -- analyze and synthesize
• Prepare to write a literature review

At the end of the tutorial, you will find a quiz that you can submit through Canvas for course credit. 

Graphic from Literature Review (2009) by Machi  and McEvoy.

Text description of "Literature Review Process" for web accessibility

If you have questions related to a field or discipline, consider reaching out to a Subject Librarian by email, phone, or by scheduling an appointment for a free consultation:

• Subject Librarians
• Next: What's a Literature Review? >>
• Last Updated: Jan 10, 2024 4:46 PM
• URL: https://researchguides.uoregon.edu/litreview

Contact Us Library Accessibility UO Libraries Privacy Notices and Procedures

1501 Kincaid Street Eugene, OR 97403 P: 541-346-3053 F: 541-346-3485

• Visit us on Facebook
• Visit us on Twitter
• Visit us on Youtube
• Visit us on Instagram
• Report a Concern
• Nondiscrimination and Title IX
• Accessibility
• Privacy Policy
• Find People

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Apr 22, 2024 9:12 AM
• URL: https://libguides.usc.edu/writingguide

News alert: UC Berkeley has announced its next university librarian

Secondary menu

• Log in to your Library account
• Hours and Maps
• Connect from Off Campus
• UC Berkeley Home

Search form

Conducting a literature review: why do a literature review, why do a literature review.

• How To Find "The Literature"
• Found it -- Now What?

Besides the obvious reason for students -- because it is assigned! -- a literature review helps you explore the research that has come before you, to see how your research question has (or has not) already been addressed.

You identify:

• core research in the field
• experts in the subject area
• methodology you may want to use (or avoid)
• gaps in knowledge -- or where your research would fit in

It Also Helps You:

• Publish and share your findings
• Justify requests for grants and other funding
• Identify best practices to inform practice
• Set wider context for a program evaluation
• Compile information to support community organizing

Great brief overview, from NCSU

Want To Know More?

• Next: How To Find "The Literature" >>
• Last Updated: Dec 8, 2023 10:11 AM
• URL: https://guides.lib.berkeley.edu/litreview

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

Literature Reviews

• Getting Started

Selecting a topic

Remember the goal of your literature review, planning your research, popular vs. scholarly, finding scholarly journal articles, finding or requesting full-text articles, why not just use google scholar, finding books, open access resources, manage your own downloads and citations, using your own method, managing your citations, organizing your notes: synthesis matrix.

• Literature Review as a Product: Organizing your Writing
• Finding Examples of Literature Reviews
• General Resources

Need help? Ask a librarian or chat

Ask A Librarian  for help finding sources, narrowing or expanding your topic, and more!​

• Research Desk: 314-246-6950
• Toll-free: 800-985-4279
• Visit us:  Library hours
• Search our FAQs or email us 

One of the most important steps in research is selecting a good research topic.  This page on the Capstone Research Guide will guide you through selecting a topic, writing a thesis statement and/or research questions, selecting keywords, and recommended databases for preliminary research.

The goal of the literature review is to provide an analysis of the literature and research already published on your subject. You want to read as much as possible on your topic to gain a foundation of the information which already exists and analyze that information to understand how it all relates. This gives you the opportunity to identify possible gaps in the research or justify your own research in relation to what already exists. 

This page will walk through the steps in: 

• Planning your research 
• Collecting your research 
• Organizing your notes 

After you have chosen your topic, you will want to make a plan for all of the places you should look for information. This will allow you to organize your search and keep track of information as you find it. In a document, make a list of possible places you might find information on your topic. This list might include: 

• Library databases - See section "Finding scholarly journal articles" below
• Research facilities - Are there organizations or institutions which publish and share research in your field of study?
• Open Access Journals - See section "Open Access Journals" below
• Webster University library catalog - See section "Finding books" below
• The catalogs of other libraries - See section "Finding books" below

Keep in mind that a literature review necessitates the use of scholarly research. These are peer-reviewed articles written by graduate or post-graduate students, educators, researchers, or professionals in the field. These types of articles wil include standard citations for the works they reference in their research. 

What is a scholarly, peer-reviewed journal article?

Scholarly articles are sometimes "peer-reviewed" or "refereed" because they are evaluated by other scholars or experts in the field before being accepted for publication.  A scholarly article is commonly an experimental or research study, or an in-depth theoretical or literature review. It is usually many more pages than a magazine article.

The clearest and most reliable indicator of a scholarly article is the presence of references or citations. Look for a list of works cited, a reference list, and/or numbered footnotes or endnotes. Citations are not merely a check against plagiarism. They set the article in the context of a scholarly discussion and provide useful suggestions for further research. 

Many of our databases allow you to limit your search to just scholarly articles. This is a useful feature, but it is not 100% accurate in terms of what it includes and what it excludes. You should still check to see if the article has references or citations.

The table below compares some of the differences between magazines (e.g. Psychology Today) and journals (e.g Journal of Abnormal Psychology).

How to find scholarly, peer-reviewed articles

• FAQ: How do I find peer-reviewed or scholarly articles?
• FAQ: How can I tell if an article is peer-reviewed?

To get a sense of the available research, you may want to start with a multidisciplinary article database such as  Academic Search Premier  or Business Source Complete (for management and business).  Then, you may want to do a more thorough search in additional specialized sources--see the link below.

• Academic Search Premier (EBSCO) A scholarly multidisciplinary database of periodical articles, most with full-text, through Ebscohost.
• Business Source Complete (EBSCO) A great starting point for articles on business and management topics including peer-reviewed journals and classic publications such Harvard Business Review. Also contains SWOT, industry, and market research reports.
• Additional specialized sources for finding journal articles Select the areas that correspond to your topic or academic program to find specialized journal article databases.

From a library database

When the PDF or HTML full-text is not available in one of our databases, use the "Full Text Finder" button. Full Text Finder will allow you to link to the article in another database. If no full text is available you may request an electronic copy of the article through Interlibrary Loan .

From another source (e.g. online, Google Scholar)

Many articles that you find online may require payment ( aka  paywall) to download the article. In many cases the library can get the article for you for free to keep you from having to pay out of pocket. For more information, please visit our: 

• Request Articles and Books page For information on how to request items depending on your campus or location
• Ask a Librarian Reach out to us in person or via phone, email or 24/7 chat. We are here to help

While  Google Scholar can be a useful source for finding journal articles, there are advantages found in using Webster University Libraries' databases, including:

• Features that let you customize your search
• Access to more full text materials
• Integration with other library services (e.g., chat, delivery services, etc.).

For more information on using Google Scholar, view the FAQ:  How can I connect Google Scholar to the Library?

Do not forget books when you are surveying the literature. They often provide historical information and overviews of current research in a topic area.

• Search for books in the Library Catalog
• Find an electronic book on a topic
• MOBIUS Catalog Here you can borrow from a consortium system of college & university libraries in Missouri and other states. (Some public libraries have also joined MOBIUS.)

Here you can search a large catalog of books and other materials owned by U.S libraries and some worldwide locations. This source shows local libraries where a given book is housed and also indicates if there is an ebook available.  

When something is published as an open access resource, it is published online and can be accessed for free with few or no copyright restrictions. Open access resources allows you to search, download, and cite researchers who have chosen to publish open access without paying for each article.

Searching through open access resources might be a great option for your research once you have exhausted the databases. Please note that sometimes, an open resource repository can be difficult to search through as often there are fewer ways to limit the search. 

• About Open Access guide This research guide provides information about Open Access (OA) and Open Educational Resources (OER) and includes a list of resources by discipline.

As you download articles and begin to identify helpful resources, you will need to develop a method of keeping track of this research. No matter the method you choose to use, make sure that: 

• Any article you've downloaded is saved in a labeled folder which is easy to access
• Any citation you have created is saved and accessible
• EBSCO tools If using an EBSCO database, this document provides information on how to create folders and save links to articles within the database.

You can certainly create a system for organizing your downloads, citations, and other electronic notes. Use the file storage system on your computer, or cloud computing software like  Google Drive  or  Dropbox . Create folders specifically for your project and save everything you think you might use. The benefit of managing your research this way is that these options are often free and allow you to have access to materials beyond your time as a Webster student.

There are a number of software programs available that help students store references and notes, create bibliographies, etc. While not needed for every assignment, they are useful for when you are gathering a large number of articles and other resources for projects such as capstone papers, theses, and dissertations. Some of the main citation management software applications are listed here.

Because the purpose of the literature review is to analyze the research on the topic and find relationships between resources, simply reading the resources and keeping notes may not be enough to help you see the connections between the resources. 

One option is to create a document with a chart used solely to compare the ideas and methods of various scholars and researchers. This is called a synthesis matrix . Before starting a matrix, you may want to identify a couple subtopics or themes to track within the articles you read. Other themes will reveal themselves as you read the literature and can be added to the chart. 

Below, you will find some examples of synthesis matrixes. Use inspiration from any of them to design  your own matrix which works best for your style and ideas. No matter the design of your matrix, some of the items you may want to compare across resources are: 

Publishing Date : Show how an idea has developed over time due to continuous research. 

Research Methods : Discuss findings based upon research type. You might ask yourself: How does the method used to collect the data impact the findings of the study? 

• Themes : Which topics are covered in the article and what does the author believe about that topic? 
• Literature Review: Synthesizing Multiple Sources (Indiana University Bloomington)
• Synthesis Matrix (NC State University)
• Writing a Literature Review and Using a Synthesis Matrix PDF Document created by North Carolina State University and posted by Wayne State University University.
• << Previous: Getting Started
• Next: Literature Review as a Product: Organizing your Writing >>
• Last Updated: Apr 20, 2024 2:21 PM
• URL: https://library.webster.edu/litreview

How to write a Literature Review: Literature review process

• Literature review process
• Purpose of a literature review
• Evaluating sources
• Managing sources
• Request a literature search
• Selecting the approach to use
• Quantitative vs qualitative method
• Summary of different research methodologies
• Research design vs research methodology
• Diagram: importance of research
• Attributes of a good research scholar

Step 1: Select a topic

• Select a topic you can manage in the time frame you have to complete your project.
• Establish your research questions and organize your literature into logical categories around the subject/ topic areas of your questions.  Your research questions must be specific enou gh to guide you to the relevant literature.
• Make sure you understand the concept of ‘broader’ and ‘narrower’ terms.  The narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good survey of the literature.

Step 2: Identify the most relevant sources on your topic

Use a variety of resources - locate books , journals , and documents that contain useful information and ideas on your topic. Internet sites , theses & dissertations , conference papers , ePrints and government or industry reports can also be included. Do not rely solely on electronic full-text material which is more easily available. Reference sources such as dictionaries can assist in defining terminology, and encyclopaedias may provide useful introductions to your topic by experts in the field and will list key references.

Step 3 : Search and refine

• Unisa has a number of databases that provide full text access to articles, that allow you to refine your search to ‘peer reviewed’ journals.  These are scholarly journals which go through a rigorous process of quality assessment by several researchers or subject specialists in the academic community before they are accepted for publication. 
• Use the And, Or, Not operators, Wildcards and Logical Brackets when searching in the databases.  For instance, you can use And to narrow your search while the operator OR expands your search.  Not, on the other hand, helps to exclude irrelevant information from your search results.  Please click here for more information on searching.

Literature review process - an overview

Step 3: search and refine.

• Unisa has a number of  databases  that provide full text access to articles, that allow you to refine your search to ‘peer reviewed’ journals.  These are scholarly journals which go through a rigorous process of quality assessment by several researchers or subject specialists in the academic community before they are accepted for publication. 
• Use the  And, Or, Not  operators,  Wildcards  and  Logical Brackets  when searching in the databases.  For instance, you can use  And  to narrow your search while  the  operator  OR  expands your search.   Not,  on the other hand,   helps to exclude   irrelevant information from your search results.  Please click  here  for more information on searching.

How do I write a literature review

See the chapter below for a helpful overview of the literature review process, especially the sections on how to analyse the literature you have gathered and how to write up your literature review:

Literature Reviews and Bibliographic Searches. 2006. In V. Desai, & R. Potter (Eds.),  Doing Development Research.  (pp. 209-222). London, England: SAGE Publications, Ltd. Available at:  http://0-dx.doi.org.oasis.unisa.ac.za/10.4135/9781849208925.n22     (A student will be prompted at some stage for his/ her student number and myUnisa password. A staff member will be prompted at some stage for his/ her Unisa Network username and login password).

This book is available in the  Sage Research Methods Online  database.

Step 4: Read and analyse

Group the sources into the  themes  and  sub-themes  of your topic.  As you read widely but selectively in your topic area, consider what themes or issues connect your sources together.

• Do they present one or different solutions?
• Is there an aspect of the field that is missing?
• How well do they present the material and do they portray it according to an appropriate theory?
• Do they reveal a trend in the field?
• A raging debate?
• Pick one of these themes to focus the organization of your review.

Step 5: Write the literature review

You can organize the review in many ways; for example, you can center the review  historically  (how the topic has been dealt with over time); or center it on the  theoretical positions  surrounding your topic (those for a position vs. those against, for example); or you can focus on how each of your sources contributes to your understanding of your project.

Your literature review should include:

• an  introduction  which explains how your review is organized.
• a  body  which contains the  headings  and  subheadings  that provide a map to show the various perspectives of your argument. In other words the body contains the evaluation of the materials you want to include on your topic.
• a  summary .

Some of the information on this page is indebted to the sources below:

Caldwell College Library

Monmouth University Library

University of Cape Town Libraries

• << Previous: Home
• Next: Purpose of a literature review >>
• Last Updated: Dec 7, 2023 1:32 PM
• URL: https://libguides.unisa.ac.za/literature_review

Literature Reviews

What this handout is about.

This handout will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.

Introduction

OK. You’ve got to write a literature review. You dust off a novel and a book of poetry, settle down in your chair, and get ready to issue a “thumbs up” or “thumbs down” as you leaf through the pages. “Literature review” done. Right?

Wrong! The “literature” of a literature review refers to any collection of materials on a topic, not necessarily the great literary texts of the world. “Literature” could be anything from a set of government pamphlets on British colonial methods in Africa to scholarly articles on the treatment of a torn ACL. And a review does not necessarily mean that your reader wants you to give your personal opinion on whether or not you liked these sources.

What is a literature review, then?

A literature review discusses published information in a particular subject area, and sometimes information in a particular subject area within a certain time period.

A literature review can be just a simple summary of the sources, but it usually has an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information. It might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. And depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

But how is a literature review different from an academic research paper?

The main focus of an academic research paper is to develop a new argument, and a research paper is likely to contain a literature review as one of its parts. In a research paper, you use the literature as a foundation and as support for a new insight that you contribute. The focus of a literature review, however, is to summarize and synthesize the arguments and ideas of others without adding new contributions.

Why do we write literature reviews?

Literature reviews provide you with a handy guide to a particular topic. If you have limited time to conduct research, literature reviews can give you an overview or act as a stepping stone. For professionals, they are useful reports that keep them up to date with what is current in the field. For scholars, the depth and breadth of the literature review emphasizes the credibility of the writer in his or her field. Literature reviews also provide a solid background for a research paper’s investigation. Comprehensive knowledge of the literature of the field is essential to most research papers.

Who writes these things, anyway?

Literature reviews are written occasionally in the humanities, but mostly in the sciences and social sciences; in experiment and lab reports, they constitute a section of the paper. Sometimes a literature review is written as a paper in itself.

Let’s get to it! What should I do before writing the literature review?

If your assignment is not very specific, seek clarification from your instructor:

• Roughly how many sources should you include?
• What types of sources (books, journal articles, websites)?
• Should you summarize, synthesize, or critique your sources by discussing a common theme or issue?
• Should you evaluate your sources?
• Should you provide subheadings and other background information, such as definitions and/or a history?

Find models

Look for other literature reviews in your area of interest or in the discipline and read them to get a sense of the types of themes you might want to look for in your own research or ways to organize your final review. You can simply put the word “review” in your search engine along with your other topic terms to find articles of this type on the Internet or in an electronic database. The bibliography or reference section of sources you’ve already read are also excellent entry points into your own research.

Narrow your topic

There are hundreds or even thousands of articles and books on most areas of study. The narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good survey of the material. Your instructor will probably not expect you to read everything that’s out there on the topic, but you’ll make your job easier if you first limit your scope.

Keep in mind that UNC Libraries have research guides and to databases relevant to many fields of study. You can reach out to the subject librarian for a consultation: https://library.unc.edu/support/consultations/ .

And don’t forget to tap into your professor’s (or other professors’) knowledge in the field. Ask your professor questions such as: “If you had to read only one book from the 90’s on topic X, what would it be?” Questions such as this help you to find and determine quickly the most seminal pieces in the field.

Consider whether your sources are current

Some disciplines require that you use information that is as current as possible. In the sciences, for instance, treatments for medical problems are constantly changing according to the latest studies. Information even two years old could be obsolete. However, if you are writing a review in the humanities, history, or social sciences, a survey of the history of the literature may be what is needed, because what is important is how perspectives have changed through the years or within a certain time period. Try sorting through some other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to consider what is currently of interest to scholars in this field and what is not.

Strategies for writing the literature review

Find a focus.

A literature review, like a term paper, is usually organized around ideas, not the sources themselves as an annotated bibliography would be organized. This means that you will not just simply list your sources and go into detail about each one of them, one at a time. No. As you read widely but selectively in your topic area, consider instead what themes or issues connect your sources together. Do they present one or different solutions? Is there an aspect of the field that is missing? How well do they present the material and do they portray it according to an appropriate theory? Do they reveal a trend in the field? A raging debate? Pick one of these themes to focus the organization of your review.

Convey it to your reader

A literature review may not have a traditional thesis statement (one that makes an argument), but you do need to tell readers what to expect. Try writing a simple statement that lets the reader know what is your main organizing principle. Here are a couple of examples:

The current trend in treatment for congestive heart failure combines surgery and medicine. More and more cultural studies scholars are accepting popular media as a subject worthy of academic consideration.

Consider organization

You’ve got a focus, and you’ve stated it clearly and directly. Now what is the most effective way of presenting the information? What are the most important topics, subtopics, etc., that your review needs to include? And in what order should you present them? Develop an organization for your review at both a global and local level:

First, cover the basic categories

Just like most academic papers, literature reviews also must contain at least three basic elements: an introduction or background information section; the body of the review containing the discussion of sources; and, finally, a conclusion and/or recommendations section to end the paper. The following provides a brief description of the content of each:

• Introduction: Gives a quick idea of the topic of the literature review, such as the central theme or organizational pattern.
• Body: Contains your discussion of sources and is organized either chronologically, thematically, or methodologically (see below for more information on each).
• Conclusions/Recommendations: Discuss what you have drawn from reviewing literature so far. Where might the discussion proceed?

Organizing the body

Once you have the basic categories in place, then you must consider how you will present the sources themselves within the body of your paper. Create an organizational method to focus this section even further.

To help you come up with an overall organizational framework for your review, consider the following scenario:

You’ve decided to focus your literature review on materials dealing with sperm whales. This is because you’ve just finished reading Moby Dick, and you wonder if that whale’s portrayal is really real. You start with some articles about the physiology of sperm whales in biology journals written in the 1980’s. But these articles refer to some British biological studies performed on whales in the early 18th century. So you check those out. Then you look up a book written in 1968 with information on how sperm whales have been portrayed in other forms of art, such as in Alaskan poetry, in French painting, or on whale bone, as the whale hunters in the late 19th century used to do. This makes you wonder about American whaling methods during the time portrayed in Moby Dick, so you find some academic articles published in the last five years on how accurately Herman Melville portrayed the whaling scene in his novel.

Now consider some typical ways of organizing the sources into a review:

• Chronological: If your review follows the chronological method, you could write about the materials above according to when they were published. For instance, first you would talk about the British biological studies of the 18th century, then about Moby Dick, published in 1851, then the book on sperm whales in other art (1968), and finally the biology articles (1980s) and the recent articles on American whaling of the 19th century. But there is relatively no continuity among subjects here. And notice that even though the sources on sperm whales in other art and on American whaling are written recently, they are about other subjects/objects that were created much earlier. Thus, the review loses its chronological focus.
• By publication: Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on biological studies of sperm whales if the progression revealed a change in dissection practices of the researchers who wrote and/or conducted the studies.
• By trend: A better way to organize the above sources chronologically is to examine the sources under another trend, such as the history of whaling. Then your review would have subsections according to eras within this period. For instance, the review might examine whaling from pre-1600-1699, 1700-1799, and 1800-1899. Under this method, you would combine the recent studies on American whaling in the 19th century with Moby Dick itself in the 1800-1899 category, even though the authors wrote a century apart.
• Thematic: Thematic reviews of literature are organized around a topic or issue, rather than the progression of time. However, progression of time may still be an important factor in a thematic review. For instance, the sperm whale review could focus on the development of the harpoon for whale hunting. While the study focuses on one topic, harpoon technology, it will still be organized chronologically. The only difference here between a “chronological” and a “thematic” approach is what is emphasized the most: the development of the harpoon or the harpoon technology.But more authentic thematic reviews tend to break away from chronological order. For instance, a thematic review of material on sperm whales might examine how they are portrayed as “evil” in cultural documents. The subsections might include how they are personified, how their proportions are exaggerated, and their behaviors misunderstood. A review organized in this manner would shift between time periods within each section according to the point made.
• Methodological: A methodological approach differs from the two above in that the focusing factor usually does not have to do with the content of the material. Instead, it focuses on the “methods” of the researcher or writer. For the sperm whale project, one methodological approach would be to look at cultural differences between the portrayal of whales in American, British, and French art work. Or the review might focus on the economic impact of whaling on a community. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed. Once you’ve decided on the organizational method for the body of the review, the sections you need to include in the paper should be easy to figure out. They should arise out of your organizational strategy. In other words, a chronological review would have subsections for each vital time period. A thematic review would have subtopics based upon factors that relate to the theme or issue.

Sometimes, though, you might need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. Put in only what is necessary. Here are a few other sections you might want to consider:

• Current Situation: Information necessary to understand the topic or focus of the literature review.
• History: The chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Methods and/or Standards: The criteria you used to select the sources in your literature review or the way in which you present your information. For instance, you might explain that your review includes only peer-reviewed articles and journals.

Questions for Further Research: What questions about the field has the review sparked? How will you further your research as a result of the review?

Begin composing

Once you’ve settled on a general pattern of organization, you’re ready to write each section. There are a few guidelines you should follow during the writing stage as well. Here is a sample paragraph from a literature review about sexism and language to illuminate the following discussion:

However, other studies have shown that even gender-neutral antecedents are more likely to produce masculine images than feminine ones (Gastil, 1990). Hamilton (1988) asked students to complete sentences that required them to fill in pronouns that agreed with gender-neutral antecedents such as “writer,” “pedestrian,” and “persons.” The students were asked to describe any image they had when writing the sentence. Hamilton found that people imagined 3.3 men to each woman in the masculine “generic” condition and 1.5 men per woman in the unbiased condition. Thus, while ambient sexism accounted for some of the masculine bias, sexist language amplified the effect. (Source: Erika Falk and Jordan Mills, “Why Sexist Language Affects Persuasion: The Role of Homophily, Intended Audience, and Offense,” Women and Language19:2).

Use evidence

In the example above, the writers refer to several other sources when making their point. A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid.

Be selective

Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the review’s focus, whether it is thematic, methodological, or chronological.

Use quotes sparingly

Falk and Mills do not use any direct quotes. That is because the survey nature of the literature review does not allow for in-depth discussion or detailed quotes from the text. Some short quotes here and there are okay, though, if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Notice that Falk and Mills do quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. But if you find yourself wanting to put in more quotes, check with your instructor.

Summarize and synthesize

Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. The authors here recapitulate important features of Hamilton’s study, but then synthesize it by rephrasing the study’s significance and relating it to their own work.

Keep your own voice

While the literature review presents others’ ideas, your voice (the writer’s) should remain front and center. Notice that Falk and Mills weave references to other sources into their own text, but they still maintain their own voice by starting and ending the paragraph with their own ideas and their own words. The sources support what Falk and Mills are saying.

Use caution when paraphrasing

When paraphrasing a source that is not your own, be sure to represent the author’s information or opinions accurately and in your own words. In the preceding example, Falk and Mills either directly refer in the text to the author of their source, such as Hamilton, or they provide ample notation in the text when the ideas they are mentioning are not their own, for example, Gastil’s. For more information, please see our handout on plagiarism .

Revise, revise, revise

Draft in hand? Now you’re ready to revise. Spending a lot of time revising is a wise idea, because your main objective is to present the material, not the argument. So check over your review again to make sure it follows the assignment and/or your outline. Then, just as you would for most other academic forms of writing, rewrite or rework the language of your review so that you’ve presented your information in the most concise manner possible. Be sure to use terminology familiar to your audience; get rid of unnecessary jargon or slang. Finally, double check that you’ve documented your sources and formatted the review appropriately for your discipline. For tips on the revising and editing process, see our handout on revising drafts .

Works consulted

We consulted these works while writing this handout. This is not a comprehensive list of resources on the handout’s topic, and we encourage you to do your own research to find additional publications. Please do not use this list as a model for the format of your own reference list, as it may not match the citation style you are using. For guidance on formatting citations, please see the UNC Libraries citation tutorial . We revise these tips periodically and welcome feedback.

Anson, Chris M., and Robert A. Schwegler. 2010. The Longman Handbook for Writers and Readers , 6th ed. New York: Longman.

Jones, Robert, Patrick Bizzaro, and Cynthia Selfe. 1997. The Harcourt Brace Guide to Writing in the Disciplines . New York: Harcourt Brace.

Lamb, Sandra E. 1998. How to Write It: A Complete Guide to Everything You’ll Ever Write . Berkeley: Ten Speed Press.

Rosen, Leonard J., and Laurence Behrens. 2003. The Allyn & Bacon Handbook , 5th ed. New York: Longman.

Troyka, Lynn Quittman, and Doug Hesse. 2016. Simon and Schuster Handbook for Writers , 11th ed. London: Pearson.

You may reproduce it for non-commercial use if you use the entire handout and attribute the source: The Writing Center, University of North Carolina at Chapel Hill

Make a Gift

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jan 17, 2024 10:05 AM
• URL: https://libguides.northwestern.edu/literaturereviews

• Library databases
• Library website

Library Guide to Capstone Literature Reviews: Role of the Literature Review

The role of the literature review.

Your literature review gives readers an understanding of the scholarly research on your topic.

In your literature review you will:

• demonstrate that you are a well-informed scholar with expertise and knowledge in the field by giving an overview of the current state of the literature
• find a gap in the literature, or address a business or professional issue, depending on your doctoral study program; the literature review will illustrate how your research contributes to the scholarly conversation
• provide a synthesis of the issues, trends, and concepts surrounding your research

Be aware that the literature review is an iterative process. As you read and write initial drafts, you will find new threads and complementary themes, at which point you will return to search, find out about these new themes, and incorporate them into your review.

The purpose of this guide is to help you through the literature review process. Take some time to look over the resources in order to become familiar with them. The tabs on the left side of this page have additional information.

Short video: Research for the Literature Review

Short Video: Research for the Literature Review

(4 min 10 sec) Recorded August 2019 Transcript 

Literature review as a dinner party

To think about the role of the literature review, consider this analogy:  pretend that you throw a dinner party for the other researchers working in your topic area. First, you’d need to develop a guest list.

• The guests of honor would be early researchers or theorists; their work likely inspired subsequent studies, ideas, or controversies that the current researchers pursue.
• Then, think about the important current researchers to invite. Which guests might agree with each other?  Which others might provide useful counterpoints?
• You likely won’t be able to include everyone on the guest list, so you may need to choose carefully so that you don’t leave important figures out. 
• Alternatively, if there aren’t many researchers working in your topic area, then your guest list will need to include people working in other, related areas, who can still contribute to the conversation.

After the party, you describe the evening to a friend. You’ll summarize the evening’s conversation. Perhaps one guest made a comment that sparked a conversation, and then you describe who responded and how the topic evolved. There are other conversations to share, too. This is how you synthesize the themes and developments that you find in your research. Thinking about your literature research this way will help you to present your dinner party (and your literature review) in a lively and engaging way.

Short video: Empirical research

Video: How to locate and identify empirical research for your literature review

(6 min 16 sec) Recorded May 2020 Transcript 

Here are some useful resources from the Writing Center, the Office of Research and Doctoral Services, and other departments within the Office of Academic Support. Take some time to look at what is available to help you with your capstone/dissertation.

• Familiarize yourself with Walden support
• Doctoral Capstone Resources website
• Capstone writing resources
• Office of Student Research Administration
• Office of Research and Doctoral Services
• Visit the Writing Center

You can watch recorded webinars on the literature review in our Library Webinar Archives .

• Next Page: Scope
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Grad Med Educ
• v.8(3); 2016 Jul

The Literature Review: A Foundation for High-Quality Medical Education Research

a  These are subscription resources. Researchers should check with their librarian to determine their access rights.

Despite a surge in published scholarship in medical education 1 and rapid growth in journals that publish educational research, manuscript acceptance rates continue to fall. 2 Failure to conduct a thorough, accurate, and up-to-date literature review identifying an important problem and placing the study in context is consistently identified as one of the top reasons for rejection. 3 , 4 The purpose of this editorial is to provide a road map and practical recommendations for planning a literature review. By understanding the goals of a literature review and following a few basic processes, authors can enhance both the quality of their educational research and the likelihood of publication in the Journal of Graduate Medical Education ( JGME ) and in other journals.

The Literature Review Defined

In medical education, no organization has articulated a formal definition of a literature review for a research paper; thus, a literature review can take a number of forms. Depending on the type of article, target journal, and specific topic, these forms will vary in methodology, rigor, and depth. Several organizations have published guidelines for conducting an intensive literature search intended for formal systematic reviews, both broadly (eg, PRISMA) 5 and within medical education, 6 and there are excellent commentaries to guide authors of systematic reviews. 7 , 8

• A literature review forms the basis for high-quality medical education research and helps maximize relevance, originality, generalizability, and impact.
• A literature review provides context, informs methodology, maximizes innovation, avoids duplicative research, and ensures that professional standards are met.
• Literature reviews take time, are iterative, and should continue throughout the research process.
• Researchers should maximize the use of human resources (librarians, colleagues), search tools (databases/search engines), and existing literature (related articles).
• Keeping organized is critical.

Such work is outside the scope of this article, which focuses on literature reviews to inform reports of original medical education research. We define such a literature review as a synthetic review and summary of what is known and unknown regarding the topic of a scholarly body of work, including the current work's place within the existing knowledge . While this type of literature review may not require the intensive search processes mandated by systematic reviews, it merits a thoughtful and rigorous approach.

Purpose and Importance of the Literature Review

An understanding of the current literature is critical for all phases of a research study. Lingard 9 recently invoked the “journal-as-conversation” metaphor as a way of understanding how one's research fits into the larger medical education conversation. As she described it: “Imagine yourself joining a conversation at a social event. After you hang about eavesdropping to get the drift of what's being said (the conversational equivalent of the literature review), you join the conversation with a contribution that signals your shared interest in the topic, your knowledge of what's already been said, and your intention.” 9

The literature review helps any researcher “join the conversation” by providing context, informing methodology, identifying innovation, minimizing duplicative research, and ensuring that professional standards are met. Understanding the current literature also promotes scholarship, as proposed by Boyer, 10 by contributing to 5 of the 6 standards by which scholarly work should be evaluated. 11 Specifically, the review helps the researcher (1) articulate clear goals, (2) show evidence of adequate preparation, (3) select appropriate methods, (4) communicate relevant results, and (5) engage in reflective critique.

Failure to conduct a high-quality literature review is associated with several problems identified in the medical education literature, including studies that are repetitive, not grounded in theory, methodologically weak, and fail to expand knowledge beyond a single setting. 12 Indeed, medical education scholars complain that many studies repeat work already published and contribute little new knowledge—a likely cause of which is failure to conduct a proper literature review. 3 , 4

Likewise, studies that lack theoretical grounding or a conceptual framework make study design and interpretation difficult. 13 When theory is used in medical education studies, it is often invoked at a superficial level. As Norman 14 noted, when theory is used appropriately, it helps articulate variables that might be linked together and why, and it allows the researcher to make hypotheses and define a study's context and scope. Ultimately, a proper literature review is a first critical step toward identifying relevant conceptual frameworks.

Another problem is that many medical education studies are methodologically weak. 12 Good research requires trained investigators who can articulate relevant research questions, operationally define variables of interest, and choose the best method for specific research questions. Conducting a proper literature review helps both novice and experienced researchers select rigorous research methodologies.

Finally, many studies in medical education are “one-offs,” that is, single studies undertaken because the opportunity presented itself locally. Such studies frequently are not oriented toward progressive knowledge building and generalization to other settings. A firm grasp of the literature can encourage a programmatic approach to research.

Approaching the Literature Review

Considering these issues, journals have a responsibility to demand from authors a thoughtful synthesis of their study's position within the field, and it is the authors' responsibility to provide such a synthesis, based on a literature review. The aforementioned purposes of the literature review mandate that the review occurs throughout all phases of a study, from conception and design, to implementation and analysis, to manuscript preparation and submission.

Planning the literature review requires understanding of journal requirements, which vary greatly by journal ( table 1 ). Authors are advised to take note of common problems with reporting results of the literature review. Table 2 lists the most common problems that we have encountered as authors, reviewers, and editors.

Sample of Journals' Author Instructions for Literature Reviews Conducted as Part of Original Research Article a

Common Problem Areas for Reporting Literature Reviews in the Context of Scholarly Articles

Locating and Organizing the Literature

Three resources may facilitate identifying relevant literature: human resources, search tools, and related literature. As the process requires time, it is important to begin searching for literature early in the process (ie, the study design phase). Identifying and understanding relevant studies will increase the likelihood of designing a relevant, adaptable, generalizable, and novel study that is based on educational or learning theory and can maximize impact.

Human Resources

A medical librarian can help translate research interests into an effective search strategy, familiarize researchers with available information resources, provide information on organizing information, and introduce strategies for keeping current with emerging research. Often, librarians are also aware of research across their institutions and may be able to connect researchers with similar interests. Reaching out to colleagues for suggestions may help researchers quickly locate resources that would not otherwise be on their radar.

During this process, researchers will likely identify other researchers writing on aspects of their topic. Researchers should consider searching for the publications of these relevant researchers (see table 3 for search strategies). Additionally, institutional websites may include curriculum vitae of such relevant faculty with access to their entire publication record, including difficult to locate publications, such as book chapters, dissertations, and technical reports.

Strategies for Finding Related Researcher Publications in Databases and Search Engines

Search Tools and Related Literature

Researchers will locate the majority of needed information using databases and search engines. Excellent resources are available to guide researchers in the mechanics of literature searches. 15 , 16

Because medical education research draws on a variety of disciplines, researchers should include search tools with coverage beyond medicine (eg, psychology, nursing, education, and anthropology) and that cover several publication types, such as reports, standards, conference abstracts, and book chapters (see the box for several information resources). Many search tools include options for viewing citations of selected articles. Examining cited references provides additional articles for review and a sense of the influence of the selected article on its field.

Box Information Resources

• Web of Science a
• Education Resource Information Center (ERIC)
• Cumulative Index of Nursing & Allied Health (CINAHL) a
• Google Scholar

Once relevant articles are located, it is useful to mine those articles for additional citations. One strategy is to examine references of key articles, especially review articles, for relevant citations.

Getting Organized

As the aforementioned resources will likely provide a tremendous amount of information, organization is crucial. Researchers should determine which details are most important to their study (eg, participants, setting, methods, and outcomes) and generate a strategy for keeping those details organized and accessible. Increasingly, researchers utilize digital tools, such as Evernote, to capture such information, which enables accessibility across digital workspaces and search capabilities. Use of citation managers can also be helpful as they store citations and, in some cases, can generate bibliographies ( table 4 ).

Citation Managers

Knowing When to Say When

Researchers often ask how to know when they have located enough citations. Unfortunately, there is no magic or ideal number of citations to collect. One strategy for checking coverage of the literature is to inspect references of relevant articles. As researchers review references they will start noticing a repetition of the same articles with few new articles appearing. This can indicate that the researcher has covered the literature base on a particular topic.

Putting It All Together

In preparing to write a research paper, it is important to consider which citations to include and how they will inform the introduction and discussion sections. The “Instructions to Authors” for the targeted journal will often provide guidance on structuring the literature review (or introduction) and the number of total citations permitted for each article category. Reviewing articles of similar type published in the targeted journal can also provide guidance regarding structure and average lengths of the introduction and discussion sections.

When selecting references for the introduction consider those that illustrate core background theoretical and methodological concepts, as well as recent relevant studies. The introduction should be brief and present references not as a laundry list or narrative of available literature, but rather as a synthesized summary to provide context for the current study and to identify the gap in the literature that the study intends to fill. For the discussion, citations should be thoughtfully selected to compare and contrast the present study's findings with the current literature and to indicate how the present study moves the field forward.

To facilitate writing a literature review, journals are increasingly providing helpful features to guide authors. For example, the resources available through JGME include several articles on writing. 17 The journal Perspectives on Medical Education recently launched “The Writer's Craft,” which is intended to help medical educators improve their writing. Additionally, many institutions have writing centers that provide web-based materials on writing a literature review, and some even have writing coaches.

The literature review is a vital part of medical education research and should occur throughout the research process to help researchers design a strong study and effectively communicate study results and importance. To achieve these goals, researchers are advised to plan and execute the literature review carefully. The guidance in this editorial provides considerations and recommendations that may improve the quality of literature reviews.

Hybrid intelligence failure analysis for industry 4.0: a literature review and future prospective

• Open access
• Published: 22 April 2024

Cite this article

You have full access to this open access article

• Mahdi Mokhtarzadeh   ORCID: orcid.org/0000-0002-0348-6718 1 , 2 ,
• Jorge Rodríguez-Echeverría 1 , 2 , 3 ,
• Ivana Semanjski 1 , 2 &
• Sidharta Gautama 1 , 2  

90 Accesses

Explore all metrics

Industry 4.0 and advanced technology, such as sensors and human–machine cooperation, provide new possibilities for infusing intelligence into failure analysis. Failure analysis is the process of identifying (potential) failures and determining their causes and effects to enhance reliability and manufacturing quality. Proactive methodologies, such as failure mode and effects analysis (FMEA), and reactive methodologies, such as root cause analysis (RCA) and fault tree analysis (FTA), are used to analyze failures before and after their occurrence. This paper focused on failure analysis methodologies intelligentization literature applied to FMEA, RCA, and FTA to provide insights into expert-driven, data-driven, and hybrid intelligence failure analysis advancements. Types of data to establish an intelligence failure analysis, tools to find a failure’s causes and effects, e.g., Bayesian networks, and managerial insights are discussed. This literature review, along with the analyses within it, assists failure and quality analysts in developing effective hybrid intelligence failure analysis methodologies that leverage the strengths of both proactive and reactive methods.

Similar content being viewed by others

A comprehensive literature review of the applications of AI techniques through the lifecycle of industrial equipment

Artificial intelligence, cyber-threats and Industry 4.0: challenges and opportunities

A failure mode and effect analysis (FMEA)-based approach for risk assessment of scientific processes in non-regulated research laboratories

Avoid common mistakes on your manuscript.

Introduction

Failure analysis entails activities to identify, categorize, and prioritize (potential) failures and determine causes and effects of each failure and failure propagation and interdependencies (Rausand & Øien, 1996 ). Failure analysis significance in manufacturing has grown since Industry 3.0 to mitigate defects and/or failures in production processes, thereby maximizing reliability and quality and minimizing production interruptions, associated risks, and costs (Wu et al., 2021 ; Ebeling, 2019 ).

Failure analysis methodologies have been supported by mathematical, statistical, and graph theories and tools, including MCDM theory, fuzzy theory, six-sigma, SPC, DOE, simulation, Pareto charts, and analysis of mean and variance (Oliveira et al., 2021 ; Huang et al., 2020 ; Tari & Sabater, 2004 ). Industry 4.0 is driven by (real-time) data from sensors, the Internet of Things (IoT), such as Internet-enabled machines and tools, and artificial intelligence (AI). Advances in artificial intelligence theory and technology have brought new tools to strengthen failure analysis methodologies (Oztemel & Gursev, 2020 ). Examples of tools include Bayesian networks (BNs), case-based reasoning (CBR), neural networks, classifications, clusterings algorithms, principal component analysis (PCA), deep learning, decision trees, and ontology-driven methods (Zheng et al., 2021 ). These Industry 4.0 advancments enable more efficient data collection and analysis, enhancing predictive capabilities, increasing efficiency and automation, and improving collaboration and knowledge sharing.

Failure analysis methodologies can be categorized into expert-driven, data-driven, and hybrid ones. Expert-driven failure analysis methods rely on experts’ knowledge and experience (Yucesan et al., 2021 ; Huang et al., 2020 ). This approach is useful when the data is limited or when there is a high degree of uncertainty. Expert-driven methods are also useful when the failure structure is complex and difficult to understand. However, this approach is limited by the availability and expertise of the experts, and is prone to bias and subjective interpretations (Liu et al., 2013 ).

Data-driven failure analysis methods, on the other hand, rely on statistical analysis and machine learning algorithms to identify patterns in the data and predict the causes of the failure (Zhang et al., 2023 ; Mazzoleni et al., 2017 ). This approach is useful when there is a large amount of data available and when the failure structure is well-defined. However, data-driven methods is limited by the quality and completeness of the data (Oliveira et al., 2021 ).

Until recently, most tools have focused on replacing humans with artificial intelligence (Yang et al., 2020 ; Filz et al., 2021b ), which causes them to remove human intellect and capabilities from intelligence systems. Hybrid intelligence creates hybrid human–machine intelligence systems, in which humans and machines collaborate synergistically, proactively, and purposefully to augment human intellect and capabilities rather than replace them with machine intellect and capabilities to achieve shared goals (Akata et al., 2020 ).

Collaboration between humans and machines can enhance the failure analysis process, allowing for analyses that were previously unattainable by either humans or machines alone. Thus, hybrid failure analysis provides a more comprehensive analysis of the failure by incorporating strengths of both expert-driven and data-driven approaches to identify the most likely causes and effects of failures (Dellermann et al., 2019 ; van der Aalst, 2021 ).

Benefits from a smart failure analysis may include reduced costs and production stoppages, improved use of human resources, improved use of knowledge, improved failure, root causes, and effects identification, and real-time failure analysis. Yet, only a few studies specifically addressed hybrid failure analysis (Chhetri et al., 2023 ). A case example of hybrid expert data-driven failure analysis involves using data from similar product assemblies to construct a Bayesian network for proccess failure mode and effects analysis (pFMEA), while also incorporating expert knowledge as constraints based on the specific product being analyzed (Chhetri et al., 2023 ).

Over the past few years, several literature reviews, as reported in Section Literature review , have been accomplished under different outlooks in relation to different failure analysis methodologies including failure mode and effects analysis (FMEA), root cause analysis (RCA), and fault tree analysis (FTA). Currently, most existing literature does not systematically summarize the research status of these failure analysis methodologies from the perspective of Industry 4.0 and (hybrid) intelligence failure analysis with the benefits from new technologies. Therefore, this study aims to review, categorize, and analyze the literature of these three general failure analysis methodologies in production systems. The objective is to provide researchers with a comprehensive overview of these methodologies, with a specific focus on hybrid intelligence, and its benefits for quality issues in production. We address two questions "How can failure analysis methodologies benefit from hybrid intelligence?" and "Which tools are suitable for a good fusion of human and machine intelligence?" Consequently, the main contributions of this study to the failure analysis literature are as follows:

Analysis of 86 papers out of 7113 papers from FMEA, RCA, and FTA with respect to methods and data types that might be useful for a hybrid intelligence failure analysis.

Identification of data and methods to construct and detect multiple failures within different research related to FMEA, RCA, and FTA methodologies.

Identification of the most effective methods for analyzing failures, identifying their sources and effects, and assessing related risks.

Proposal of a categorization of research based on the levels of automation/intelligence, along with the identification of limitations in current research in this regard.

Provision of hybrid intelligent failure analysis future research, along with other future directions such as future research on failure propagation and correlation.

The plan of this paper is as follows. Section Literature review briefly introduces related literature reviews on FMEA, RCA, and FTA. A brief description of other failure analysis methodologies is also provided. Section Research methodology presents our review methodology, including the review scope and protocols, defining both our primary and secondary questions, and the criteria for selecting journals and papers to be reviewed. A bibliography summary of the selected papers is provided. Literature has been categorized in Section Literature categorization based on the four general steps of a failure analysis methodology, involving failure structure detection, failure event probability detection, failure risk analysis, and outputs. Managerial insights, limitations, and future research are discussed in Section Managerial insights, limitations, and future research . This assists researchers with applications and complexity, levels of intelligence, how knowledge is introduced into the failure analysis. A more in-depth discussion of hybrid intelligence, failure propagation and correlation, hybrid methodologies, and other areas of future research is also included. Conclusions are presented in Section Conclusion .

Literature review

General and industry/field-specific failure analysis methodologies have been developed over the last few decades. In this section, we provide useful review papers regarding FMEA, RCA, and FTA, which are the focus of our paper. Additionally, some other general and industry/field-specific failure analysis methodologies are briefly discussed.

FMEA is a most commonly used bottom-up proactive qualitative methodologies for potential quality failure analysis (Huang et al., 2020 ; Stamatis, 2003 ). Among its extensions, process FMEA (pFMEA) proactively identifies potential quality failures in production processes such as assembly lines (Johnson & Khan, 2003 ). Typically, (p)FMEA uses expert knowledge to determine potential failures, effects, and causes, and to prioritize the failures based on the risk priority number (RPN). RPN is a product of severity, occurrence, and detection rates for each failure (Wu et al., 2021 ). Some of the FMEA shortcomings include time-consuming, subjectivity, inability to determine multiple failures, and failure propagation and interdependency (Liu et al., 2013 ).

RCA is a bottom-up reactive quantitative methodology that determines the causal mechanism behind a failure to prevent the recurrence of the failure in manufacturing processes (Oliveira et al., 2023 ). To locate, identify, and/or explain the reasons behind the occurrence of root causes, RCA utilizes statistical analysis tools, such as regression, statistical process control (SPC), design of experiments (DOE), PCA, and cause-effect diagram (Williams, 2001 ). Limited ability to predict future failures and difficulty in identifying complex or systemic issues are among RCA limitations (Yuniarto, 2012 ).

FTA is a top-down reactive graphical method to model failure propagation through a system, i.e., how component failures lead to system failures (Kumar & Kaushik, 2020 ). FTA uses qualitative data to model the structure of a system and quantitative data, including probabilities and graph methods such as minimal cut/path sets, binary decision diagrams, simulation, and BNs, to model failures propagation. Requiring extensive data, limited ability to identify contributing factors, and time-consuming are among the FTA limitations (Ruijters & Stoelinga, 2015 ).

In recent years, several literature reviews have been conducted on failure analysis methodologies, exploring various perspectives and approaches. Liu et al. ( 2013 ) reviewed FMEA risk evaluation tools including rule-based systems, mathematical programming, and multi-criteria decision-making (MCDM). They concluded that artificial intelligence and MCDM tools, particularly fuzzy rule base systems, grey theory, and cost-based models, are the most cited tools to prioritize risks in FMEA. Liu et al. ( 2019a ) and Dabous et al. ( 2021 ) reviewed MCDM tools application for FMEA. Papers with different areas, automotive, electronics, machinery and equipment, and steel manufacturing were considered. The most used MCDM tools, namely technique for order of preference by similarity to ideal solution (TOPSIS), analytic hierarchy process (AHP), decision-making trial and evaluation laboratory (DEMATEL), and grey theory, were identified.

Spreafico et al. ( 2017 ) provided a FMEA/Failure mode, effects, and criticality analysis (FMECA) critical review by classifying FMEA/FMECA limitations and issues and reviewing suggested improvements and solutions for the limitations. FMEA issues were classified into four groups of applicabilities, cause and effect analysis, risk analysis, and problem-solving. Main problems (and solutions) are being time-consuming (integration with design tools, using more structured templates, and automation), lack of secondary effects modeling (integration with other tools such as FTA, BN, and Petri net), being too subjective (using statistical evaluation and cost-based approaches), and lack in evaluating the implementation of a solution (using the improved presentation of the results and integration with other tools such as maintenance management tools), respectively. Huang et al. ( 2020 ) provided a bibliographic analysis of FMEA and its applications in manufacturing, marine, healthcare, aerospace, and electronics. Wu et al. ( 2021 ) sorted out potential failure mode identification approaches such as analyzing entry point for system failure mode identification, failure mode recognition tools, and failure mode specification description. Then a review of FMEA risk assessment tools had been provided.

Oliveira et al. ( 2023 ) reviewed automatic RCA literature in manufacturing. Different data types, location-time, physical, and log-action, that are usually used were identified. Industries with the most use of RCA are ranked, semiconductor, chemical, automotive, and others. Then different tools used to automate RCA, including decision trees, regression models, classification methods, clustering methods, neural networks, BNs, PCA, statistical tests, and control charts, were discussed. Ruijters and Stoelinga ( 2015 ) provided FTA qualitative and quantitative analysis methods. Also, different types of FTA, standard FTA, dynamic FTA, and other extensions, were discussed. Zhu and Zhang ( 2022 ) also reviewed dynamic FTA. Cai et al. ( 2017 ) reviewed the application of BN in fault diagnosis. First, an overview of BN types (static, dynamic, and object-oriented), structure modeling, parameters modeling, and interference has been provided. Then applicability of BN for fault identification in process, energy, structural, manufacturing, and network systems has been discussed. BN verification and validation methods are provided. Future prospects including integration of big data with BN, real-time fault diagnosis BN inference algorithms, and hybrid fault diagnosis methods are finally resulted. More relevant BN reviews include BN application in reliability (Insua et al., 2020 ) and safety and risk assessments (Kabir & Papadopoulos, 2019 ).

The integration of FMEA, RCA, and FTA holds immense potential for quality and production managers to minimize failures and enhance system efficiency. By capitalizing on the unique strengths of each approach, the integration of these failure analysis methodologies enables a more comprehensive and effective examination of failures. However, existing studies and literature reviews have predominantly focused on individual methodologies, leading to a lack of integration and limited familiarity with three approaches among engineers and industry experts. To address this gap and promote the integration of them, this study aims to review the progress of intelligence failure analysis within FMEA, RCA, and FTA.

Other general failure analysis methodologies include, but are not limited to, the following methodologies. Event Tree Analysis, similar to FTA, is a graphical representation that models the progression of events following an initiating event, helping to analyze the potential consequences (Ruijters & Stoelinga, 2015 ). Bow-Tie Analysis, usually used in risk management, visualizes the relationship between different potential causes of a hazard and their possible consequences (Khakzad et al., 2012 ). Human Reliability Analysis focuses on assessing the probability of human error and its potential impact on systems and processes (French et al., 2011 ). The Fishbone Diagram visually represents potential causes of a problem to identify root causes by categorizing them into specific factors like people, process, equipment, materials, etc.

There are also industry-specific methodologies, including but not limited to the following ones. Electrostatic Discharge (ESD) Failure Analysis focuses on identifying failures caused by electrostatic discharge, a common concern in the electronics industry. Hazard and Operability Study is widely used in the chemical industry to examine deviations from the design intent and identify potential hazards and operability issues. Incident Response and Post-Incident Analysis, in the IT industry, is used for analyzing and responding to security incidents, with a focus on preventing future occurrences. Hazard Analysis and Critical Control Points is a systematic preventive approach to food safety that identifies, evaluates, and controls hazards throughout the production process. Maximum credible accident analysis assesses and mitigates the most severe accidents that could occur in high-risk industries. For more information on industry-specific methodologies, an interested reader may consult the paper on that industry, as they are wide and out of the scope of this paper for deep discussion.

Our review focuses on the historical progress of (hybrid) intelligence failure analysis to identify and classify methodologies and tools used within them. In Industry 4.0, (hybrid) intelligence failure analysis can contribute to improve quality management and automate quality through an improved human cyber-physical experience. Different from the abovementioned reviews, the purpose of our study is to provide a rich comprehensive understanding of the recent developments in these methodologies from industry 4.0 and hybrid intelligence, the benefits of making them intelligent, i.e., (augmented) automatic and/or data-driven, and their limitations.

Research methodology

A systematic literature review analyses a particular knowledge domain’s body of literature to provide insights into research and practice and identify research gaps (Thomé et al., 2016 ). This section discusses our review scope and protocols, defining both our primary and secondary questions, and the criteria for selecting journals and papers to be reviewed. A bibliography analysis of the selected papers is also presented, including distributions by year, affiliation, and journals.

Review scope and protocol

We follow Thomé et al. ( 2016 ) 8-step literature review methodology to assure a rigorous literature review of intelligence, automated/data-driven, failure analysis methodology for Industry 4.0.

In Step 1, our (hybrid) intelligence failure analysis problem is planned and formulated by identifying the needs, scope, and questions for this research. Our initial need for this literature review comes from a relevant industrial project entitled "assembly quality management using system intelligence" which aims to reduce the quality failures in assembly lines. The trend towards automated and data-driven methodologies in recent years signifies the need for this systematic literature review. Thus, three general failure analysis methodologies, FMEA, RCA, and FTA, are reviewed with respect to tools to make them intelligent and to derive benefits from hybrid intelligence.

Our primary questions are as follows. (i) What are the failure analysis general methodologies and what tools have been used to make them intelligent? (ii) How these methodologies may benefit from hybrid intelligence? (iii) What are the strengths and weaknesses of these methodologies and tools? Our secondary questions are as follows. (i) How intelligent are these tools? (ii) What types of data do they use? Which tools allow a good fusion of human and machine intelligence? (iii) How well do they identify the root causes of failures? (iv) What are the possible future prospectives?

Distribution of papers by year and affliation

Step 2 concerns searching the literature by selecting relevant journals, databases, keywords, and criteria to include or exclude papers. We select the SCOPUS database to scan the relevant paper from 1990 to the first half of 2022. SCOPUS contains all high-quality English publications and covers other databases such as ScienceDirect and IEEE Xplore. A two-level keyword structure is used. The first level retrieves all papers that have either failure mode and effect analysis, FMEA, failure mode and effects and criticality analysis, FMECA, fault tree analysis, FTA, event tree analysis, ETA, root cause analysis, RCA, failure identification, failure analysis, or fault diagnosis in the title, abstract, and/or keywords. The second level limits the retrieved paper by the first level keywords to papers that have either Bayesian network, BN, automated, automatic, automation, smart, intelligence or data-driven in the title, abstract, and/or keywords.

To ensure the scientific rigor of our literature review process, we have removed papers that met at least one of the following criteria: Publications with concise and/or ambiguous information that would make it impossible to re-implement the tools and methodologies described in the paper later on. Publications in low-level journals, i.e., journals in the third quarter (Q3) or lower in the Scimago Journal & Country Rank. Papers with subject areas that are irrelevant to our research topic, such as physics and astronomy.

Steps 3 and 4 involve gathering data and evaluating data quality. We download papers and check their sources according to exclusion criteria. Step 5 concerns data analysis. Step 6 focuses on interpreting the data. The final selected papers are analyzed and interpreted in Section Managerial insights, limitations, and future research . Step 7 involves preparing the results and report. Step 8 requires the review to be updated continuously.

Discussion and statistical analysis

Here is a bibliometric analysis of our literature review. About 15,977 papers were found in our first search. By excluding criteria, we shortened the search to 7113. Then, we checked the titles of 7113 papers including 4359 conference and 2754 journal papers. We downloaded 1,203 papers to read their abstracts and skim their bodies. Then, 1114 low-quality/irrelevant papers were excluded. The remaining 86 high-quality papers were examined for this study.

Distributions of papers by year and affiliation are shown in Fig. 1 . 28 countries have contributed in total. Most affiliations are in advanced countries including China, Germany, and the UK. Surprisingly, we found no publications from Japan and only five from the USA. Only one papers had been published between 1990 and 1999 because of limited data and technology, e.g., sensors and industrial cameras. A slow growth observed between 2000 and 2014 coincides with the technology advancement and Industry 4.0 emergence. The advanced technology and researchers focus on Industry 4.0 have led to significant growth every year since 2015. Worth to note that 2022 information is incomplete because this research has been conducted in the middle of 2022. We expect more publications, at least equal to 2021, for 2022.

Papers distribution by journal is in Fig. 2 . 58 journals and conferences have contributed. Journals with a focus on production and quality, e.g., International Journal of Production Research , have published most papers. Technology-focused journals, e.g., IEEE Access , also have contributed.

Distribution of papers by journal

Literature categorization

Selected papers are now categorized based on the four general steps of a failure analysis methodology, involving failure structure detection, failure event probabilities detection, failure risk analysis, and outputs. Then, a statistical analysis of these categorizations is provided.

These four steps of a failure analysis methodology are illustrated in Fig. 3 . The first two steps deal with input data. In step 1, the failure structure is identified, encompassing all (possible) failures, the failure propagation structure, failure interdependency, and causes and effects. Step 2 involves detecting event probabilities in a failure structure. For example, classical FMEA scores each failure with severity, occurrence, and detection rates.

Four general steps of a failure analysis methodology

To analyze failures in a (production) system, data should be collected to identify the failure structure and detect failures. Reactive methodologies, such as RCA, are data-driven and typically gather available data in a system, while proactive methodologies, such as FMEA, are expert-driven and gather data through expert knowledge. However, a (hybrid) intelligence failure analysis methodology should take advantage of both advanced technologies, such as sensors and Internet-enabled machines and tools, and experts to automatically gather required data, combining proactive and reactive approaches, and providing highly reliable analyses and solutions.

In step 3, all input data are processed to determine the associated risk value with each failure, and the most probable causes (usually based on an observed or potential effect). Typically, a main tool, such as Bayesian networks, neural rule-based systems, statistical analysis, or expert analysis, is used to determine root causes, classify failures, and/or rank failures.

Step 4 outputs results that may include failures and sources, reasons behind the sources, and mitigation actions. The output of this tool is post-processed to provide possible solutions and information that is explainable and easy to use for both humans and machines.

Steps 1: failure structure

Failure structure identification is the first step in a failure analysis methodology. (Potential) failures, causes, effects, and/or failure interdependency are identified. We categorize the literature to develop a (hybrid) intelligence failure methodology to identify failure structure, causes, effects, interdependencies, and relationships between failures, failures and causes, and failures and effects.

Traditionally, experts have defined failure structures by analyzing causes, effects, and the interdependency of failures. However, recent studies have explored alternative approaches to identifying failure structures, leveraging available data sources such as problem-solving databases, design forms, and process descriptions. Problem-solving databases include quality issue records, maintenance records, failure analysis records, and CBR databases. These records could be stored in structured databases and sheets, or unstructured texts. Design forms may include design FMEA forms, reliability characteristics, and product quality characteristics. Process descriptions may include operations, stations, and key operational characteristics. Moreover, simulation can be used to generate failures, causes, and effects (Snooke & Price, 2012 ). Design forms and process descriptions are generated by experts, usually for other purposes, and are re-used for failure analysis. Problem-solving databases could be generated by experts, such as previous FMEAs, or by an automated failure analysis methodology, such as automated RCA. Table 1 classifies studies based on the data sources used to identify the failure structure.

Data processing methods

To define failure structure from operational expert-driven data, no specific tool has been used. In the industry, failure structures are typically defined by an expert (or group of experts). When expert-driven or data-driven historical data and/or design forms and process descriptions are available, ontology-driven algorithms, including heuristics (Sayed & Lohse, 2014 ; Zhou et al., 2015 ; Steenwinckel et al., 2018 ; Xu & Dang, 2023 ) and SysML modeling language (Hecht & Baum, 2019 ), process/system decomposition (the operation, the station, and the key characteristics levels) (Zuo et al., 2016 ; Khorshidi et al., 2015 ; Zhou et al., 2015 ), rule-based algorithms that use CBR (Yang et al., 2018 ; Liu & Ke, 2007 ; Xu & Dang, 2023 ; Oliveira et al., 2022 , 2021 ), and FTA/BN modeling from FMEA/expert data (Yang et al., 2022 ; Steenwinckel et al., 2018 ; Palluat et al., 2006 ) and from Perti net (Yang & Liu, 1998 ) have been suggested. Rivera Torres et al. ( 2018 ) divided a system into components and related failures to each of the components to make a tree of components and failures.

Component-failure matrix is generated using unstructured and quality problem texts mining from historical documents such as bills of material and failure analysis. Apriori algorithms were used to find synonyms in the set of failure modes (Xu et al., 2020 ). The 8D method is used to describe a failure. Ontology was used to store and retrieve data in a knowledge base CBR system.

Yang et al. ( 2022 ), Leu and Chang ( 2013 ) and Waghen and Ouali ( 2021 ) have suggested building a BN structure from the FTA model. Wang et al. ( 2018 ) has proposed to use the fault feature diagram, the fault-labeled transition system based on the Kripke structure to describe the system behavior. The MASON (manufacturing semantic ontology) has been used to construct the structure of the failure class by Psarommatis and Kiritsis ( 2022 ). Teoh and Case ( 2005 ) has developed a functional diagram to construct a failure structure between components of a system and to identify causes and effect propagation. Yang et al. ( 2018 ) used an FMEA style CBR to collect failures to search for similarity. They then used CBR to build a BN using a heuristic algorithm.

Step 2: failure detection

Failure detection data are gathered to determine the strength of relationships among failures, causes, and effects.

Failure detection can be based on operational or historical expert-driven data, as well as data-driven historical and/or real-time data obtained from sensors. Such data can come from a variety of sources, including design and control parameters (such as machine age or workpiece geometry), state variables (such as power demand), performance criteria (such as process time or acoustic emission), and internal/external influencing factors (such as environmental conditions) (Filz et al., 2021b ; Dey & Stori, 2005 ). These data are usually used to determine occurrence probability of failures. To determine the severity and detection probabilities of failures, conditional severity utility data/tables may be used (Lee, 2001 ). Simulation can also be used to determine occurrence, severity, and detection (Price & Taylor, 2002 ). Table 2 summarizes types of data that are usually used to detect failures in the literature.

Processing data refers to the transformation of raw data into meaningful information. A data processing tool is needed that provides accurate and complete information about the system and relationships between data and potential failures.

First, data from different sources should be pre-processed. In a data pre-processing step, data is cleaned, edited, reduced, or wrangled to ensure or enhance performance, such as replacing a missing value with the mean value of the entire column (Filz et al., 2021b ; Schuh et al., 2021 ; Zhang et al., 2023 ; Musumeci et al., 2020 ; Jiao et al., 2020 ; Yang et al., 2015 ; Chien et al., 2017 ).

Data then may need to be processed according to the tools used in Step 3. Common data processing methods between all tools include data normalization using the min-max method (Filz et al., 2021b ; Musumeci et al., 2020 ) and other methods (Yang et al., 2018 ; Schuh et al., 2021 ; Jiao et al., 2020 ; Sariyer et al., 2021 ; Chien et al., 2017 ).

Feature selection/extraction algorithms have been used to select the most important features of data (Filz et al., 2021b ; Xu & Dang, 2020 ; Mazzoleni et al., 2017 ; Duan et al., 2020 ; Schuh et al., 2021 ; Zhang et al., 2023 ; Musumeci et al., 2020 ; Yang et al., 2015 ; Sariyer et al., 2021 ).

For BN-based failure analysis, maximum entropy theory is proposed to calculate failure probabilities from expert-based data (Rastayesh et al., 2019 ). Fuzzy methods have also been used to convert linguistic terms to occurrence probabilities (Yucesan et al., 2021 ; Wan et al., 2019 ; Nie et al., 2019 ; Nepal & Yadav, 2015 ; Ma & Wu, 2020 ; Li et al., 2013 ; Duan et al., 2020 ). Euclidean distance-based similarity measure (Chang et al., 2015 ) and fuzzy rule base RPN model (Tay et al., 2015 ), heuristic algorithms (Brahim et al., 2019 ; Dey & Stori, 2005 ; Yang et al., 2022 ), and a fuzzy probability function (Khorshidi et al., 2015 ) have been suggested to build failure probabilities.

Failure analysis data may be incomplete, inaccurate, imprecise, and limited. Therefore, several studies have used tools to deal with uncertainty in data. The most commonly used methods are fuzzy FMEA (Yang et al., 2022 ; Nepal & Yadav, 2015 ; Ma & Wu, 2020 ), fuzzy BN (Yucesan et al., 2021 ; Wan et al., 2019 ; Nie et al., 2019 ), fuzzy MCDM (Yucesan et al., 2021 ; Nie et al., 2019 ; Nepal & Yadav, 2015 ), fuzzy neural network (Tay et al., 2015 ; Palluat et al., 2006 ), and fuzzy evidential reasoning and Petri nets (Shi et al., 2020 ).

Step 3: analysis

A failure analysis tool is essential for conducting any failure analysis. Table 3 categorizes various data-driven tools, such as BNs, Clustering/Classification, Rule-based Reasoning, and other tools used in the literature and the aspects they support.

BNs model probabilistic relationships among failure causes, modes, and effects using directed acyclic graphs and conditional probabilities. Pieces of evidence, i.e., known variables, are propagated through the graph to evaluate unobserved variables (Cai et al., 2017 ). For example, Rastayesh et al. ( 2019 ) applied BNs for FMEA and perform risk analysis of a Proton Exchange Membrane Fuel Cell. Various elements and levels of the system were identified along with possible routes of failure, including failure causes, modes, and effects. A BN was constructed to perform the failure analysis. Some other examples of the BNs application include an assembly system (Sayed & Lohse, 2014 ), kitchen equipment manufacturing (Yucesan et al., 2021 ), and Auxiliary Power Unit (APU) fault isolation (Yang et al., 2015 ).

Classification assigns predefined labels to input data based on learned patterns, Clustering organizes data into groups based on similarities. Neural networks are commonly used for failure classification and have been employed in most studies. Hence, we separated these studies from those that used other clustering/classification tools. Neural networks consist of layers of interconnected nodes, with an input layer receiving data, one or more hidden layers for processing, and an output layer providing the final classification (Jiang et al., 2024 ). For example, Ma and Wu ( 2020 ) applied neural networks to assess the quality of 311 apartments in Shanghai, China, for FMEA. The input includes various APIs collected for the apartments, and the output was the risk rate of each apartment. In another study, Ma et al. ( 2021 ) applied neural networks for RCA to predict the root causes of multiple quality problems in an automobile factory. Some other examples of the neural networks application include industrial valve manufacturing (Pang et al., 2021 ), complex cyber–physical systems (Liu et al., 2021 ), and an electronic module designed for use in a medical device (Psarommatis & Kiritsis, 2022 ).

Other clustering/classification tools include evolving tree (Chang et al., 2015 ), reinforced concrete columns (Mangalathu et al., 2020 ), K-means, random forest algorithms (Xu & Dang, 2020 ; Chien et al., 2017 ; Oliveira et al., 2022 , 2021 ), contrasting clusters (Zhang et al., 2023 ), K-nearest neighbors (Ma et al., 2021 ), self-organizing maps (Gómez-Andrades et al., 2015 ), and Naive Bayes (Schuh et al., 2021 ; Yang et al., 2015 ).

Rule-based reasoning represents knowledge in the form of "if-then" rules. Rule-based reasoning involves a knowledge base containing the rules and a reasoning engine that applies these rules to incoming data or situations. For instance, Jacobo et al. ( 2007 ) utilized rule-based reasoning for analyzing failures in mechanical components. This approach serves as a knowledgeable assistant, offering guidance to less experienced users with foundational knowledge in materials science and related engineering fields throughout the failure analysis process. Also, the application of the rule-based reasoning for wind turbines FMEA is studied by (Zhou et al., 2015 ).

Other tools include gradient-boosted trees, logistic regression (Filz et al., 2021b ), CBR (Tönnes, 2018 ; Camarillo et al., 2018 ; Jacobo et al., 2007 ), analyzing sensitivities of the machining operation by the stream of variations and errors probability distribution determination (Zuo et al., 2016 ), causal reasoning (Teoh & Case, 2005 ), probabilistic Boolean networks with interventions (Rivera Torres et al., 2018 ), principal component analysis (PCA) (Duan et al., 2020 ; Zhang et al., 2023 ; Jiao et al., 2020 ; Sun et al., 2021 ), factor ranking algorithms (Oliveira et al., 2022 , 2021 ), heuristics and/or new frameworks (Camarillo et al., 2018 ; Yang et al., 2009 , 2020 ; Snooke & Price, 2012 ; Xu & Dang, 2023 ; Rokach & Hutter, 2012 ; Wang et al., 2018 ; Hecht & Baum, 2019 ; Yang & Liu, 1998 ; Liu & Ke, 2007 ), and mathematical optimization methods (Khorshidi et al., 2015 ).

These tools may be integrated by other tools including sequential state switching and artificial anomaly association in a neural network (Liu et al., 2021 ), MCDM/optimization (Yucesan et al., 2021 ; Jomthanachai et al., 2021 ; Ma et al., 2021 ; Sun et al., 2021 ), game theory (Mangalathu et al., 2020 ), fuzzy evidential reasoning and Petri nets (Shi et al., 2020 ), and maximum spanning tree, conditional Granger causality, and multivariate time series (Chen et al., 2018 ).

Step 4: output

A data analysis process can benefit not only humans but also machines and tools in a hybrid intelligence failure analysis methodology. Therefore, the output information should be carefully designed. Table 4 ranks the output data, and the list of studies for each output is available in Online Appendix EC.1. Most studies have focused on automatically identifying the root causes of failures, which is the primary objective of a failure analysis methodology. In addition, researchers have also focused on failure occurrence rating, ranking, and classification. While automatically finding the root causes of failures is important, a hybrid intelligence failure analysis process needs to interpret the related data and information and automatically provide mitigation actions for both operators and machines. However, only a few studies have proposed tools to automatically find possible mitigation actions, usually based on CBR databases and only readable for humans. Therefore, future studies may focus on finding possible automated mitigation actions for failures and developing a quality inspection strategy.

Data post-processing

A data post-processing step transforms data from the main tool into readable, actionable, and useful information for both humans and machines. Adapting solutions from similar failures in a database (i.e., CBR) to propose a solution for a detected failure has been proposed by Tönnes ( 2018 ), Camarillo et al. ( 2018 ), Hecht and Baum ( 2019 ), Jacobo et al. ( 2007 ), Liu and Ke ( 2007 ) and Ma et al. ( 2021 ). Simulation to analyze different scenarios (Psarommatis & Kiritsis, 2022 ; Jomthanachai et al., 2021 ; Chien et al., 2017 ; Oliveira et al., 2022 ), mathematical optimization model (Khorshidi et al., 2015 ; Ma et al., 2021 ) and self-organizing map (SOM) neural network (Chang et al., 2017 ) to automatically select the best corrective action have also been proposed. Also, fuzzy rule-based systems to obtain RPN (Nepal & Yadav, 2015 ) and visualisation (Xu & Dang, 2020 ; Yang et al., 2009 ) are discussed.

The statistical analysis of the paper reveals that most FMEA-based studies rely solely on expert-based information to construct failure structures, while RCA-based papers tend to use a hybrid of problem-solving and system-related data. This is depicted in Fig. 4 , which shows the distribution of papers by data used over time. FMEA is used to identify potential failures when there is not enough data available to construct a failure structure based on system-based data. The trend shows some effort to use data, instead of expert knowledge, to construct failure structures, using data from similar products/processes. RCA and FTA are a reactive methodology that analyzes more information than FMEA. Advances in data mining techniques, along with increased data availability, have led to a growing trend of using data to construct failure structures. For a comprehensive and reliable intelligence failure analysis, a combination of all kinds of data is necessary. It is worth noting that Waghen and Ouali ( 2021 ) proposed a heuristic method to augment failure structure identification that uses expert and historical data. They suggested engaging expert knowledge when historical data are insufficient to identify a failure structure and/or the reliability of a failure structure is low. Other studies have solely focused on failure identification through expert knowledge or historical data, without considering the potential benefits of combining different types of data.

Input data statistical analysis

While most FMEA-based papers use only expert-based data to determine failure probability, there is a significant growth in the utilization of problem-solving data and a hybrid of problem-solving and system-related data, i.e., production line data, over time. RCA and FTA usually tend to use more problem-solving and system-related data. Moreover, this figure and Fig. 5 show that the literature on RCA has been growing in recent years, while the trend for FMEA has remained the same over time. We found that Filz et al. ( 2021b ), Mazzoleni et al. ( 2017 ), Ma and Wu ( 2020 ) and Yang et al. ( 2015 ) improved FMEA to use a combination of expert-based, problem-solving, and system-related data to determine potential failures and their causes. They analyzed these data using deep learning, classification, and neural networks, respectively. Duan et al. ( 2020 ), Ma et al. ( 2021 ) tried to use the benefits of both expert-based data and problem-solving and system-related data in the RCA context. They analyzed the root cause of failures using neural networks.

The distribution of papers by the tools used is shown in Fig. 5 . BNs have been mainly used within the context of FMEA methodologies with a growing trend during the recent years, while RCA researchers have used them less frequently. BNs have the potential to model failure propagation, multi-failure scenarios, and solution analysis to propose potential solutions. However, all of the studies reviewed in this paper only used BNs to identify the root causes of failures. BNs offer a clear graphical representation of failures, their causes, and their effects, which facilitates the interpretation of results by humans. They also provide an easy way for humans to intervene and analyze the sensitivity of results and correct processed data if it appears unrealistic. BNs are well-developed tool and have the ability to work with expert-based, historical, and system-based data, even when data is fuzzy or limited. Developing methodologies that leverage the advantages of BNs seems promising for FMEA, RCA, and FTA.

Tools distribution statistical analysis

RCA and FTA are reliant on various tools over time with no trend of using a specific tool, such as PCA and regression, due to their need for a large amount of data. However, these methods have limitations in incorporating both human and machine intelligence and mostly rely on machine intelligence. Although neural networks and classification algorithms have gained attention in both FMEA and RCA during the last few years, they are black boxes and difficult for humans to modify. Also, classification algorithms typically do not address failure propagation or multi-failure modes. BNs offer a promising alternative, as they can model failure propagation, multiple-failures, and provide a clear graphical representation of failures, causes, and effects. Furthermore, BNs can incorporate both expert-based and historical data, making them well-suited for FMEA, RCA, and FTA. Therefore, developing methodologies that fully leverage the benefits of BNs in these domains would be valuable.

Managerial insights, limitations, and future research

In this section, we discuss managerial insights, limitations, and future research related to different aspects of a Hybrid Intelligence failure analysis methodology. The aim is to assist researchers in focusing on relevant recommendations. Section Section Applications and complexity delves into the applications and complexity of each study, and provides examples for each tool. Section Levels of automation/intelligence presents the levels of intelligence for a failure analysis methodology. Section Introducing knowledge into tools discusses how knowledge is introduced into the failure analysis tools for an effective failure analysis. A more in-depth discussion of hybrid intelligence is in Section Hybrid intelligence . The last three sections provide insights into failure propagation and correlation, hybrid methodologies, and other areas of future research.

Applications and complexity

Intelligent FMEA, RCA, and FTA have been applied to various applications, including production quality management, computer systems, reliability and safety, chemical systems, and others. Table 5 presents the distribution of reviewed papers by application. The list of studies per application is available in Online Appendix EC.2. Production quality management has been the most common application of intelligent failure analysis methodologies due to the significant costs associated with quality assurance. Smart failure analysis methodologies have also been impacted by the increased use of sensors and IoT to collect precise data from machines, tools, operators, and stations, as well as powerful computers to analyze the data. Computer systems failure analysis and system reliability and safety rank second, while chemical systems rank third, as these systems often require specific methodologies, such as hazard and operability analysis.

We checked every paper dataset to find information about the complexity of their case-study and reasons behind their good results to help readers select a validated study on a large set of data. An enriched dataset of problem-solving data are used by Xu et al. ( 2020 ), Du et al. ( 2012 ), Oliveira et al. ( 2021 ), Gómez-Andrades et al. ( 2015 ), Leu and Chang ( 2013 ), Price and Taylor ( 2002 ), Sariyer et al. ( 2021 ), Gomez-Andrades et al. ( 2016 ) and Xu and Dang ( 2023 ). An enriched dataset of historical problem-solving and sensors data is used by

Filz et al. ( 2021b ), Sun et al. ( 2021 ), Mazzoleni et al. ( 2017 ), Hireche et al. ( 2018 ), Yanget al. ( 2015 ) Demirbaga et al. ( 2021 ), Waghen and Ouali ( 2021 ), Zhang et al. ( 2023 ), Oliveira et al. ( 2022 ), Sun et al. ( 2021 ). Data from the system and processes are used by Teoh and Case ( 2005 ), Ma et al. ( 2021 ), Schuh et al. ( 2021 ), Waghen and Ouali ( 2021 ). Other studies demonstrated their methodology on a small problem.

Levels of automation/intelligence

Failure analysis intelligence can be divided into five levels based on the data used. Level 1 involves analyzing failures using expert-based data with the use of intelligence tools. This level can be further improved by incorporating fuzzy-based tools, such as fuzzy BNs, fuzzy neural networks, and fuzzy rule-based systems. If the amount of historical data can be increased over time, we suggest using BNs in a heuristic-based algorithm, as they have the capability to work with all possible data, resulting in fewer modifications in the failure analysis methodology over time. Good examples for Level 1 include Yucesan et al. ( 2021 ) and Brahim et al. ( 2019 ).

Level 2 involves analyzing failures using experts to identify failure structures and problem-solving and system-related data to determine failure probabilities. This level can be used by a professional team who can correctly and completely identify failure structure. It can also be used by those who work with variable structures where updating the structure requires a lot of data modification. Identifying failure structures and analyzing failures are both automated at level 3. This level is the most applicable when a good amount of data is available. BNs, classification algorithms, and neural networks are among the best tools to analyze failure within RCA, FMEA, and FTA methodologies. Studies such as Filz et al. ( 2021b ) Zuo et al. ( 2016 ), Dey and Stori ( 2005 ), Mangalathu et al. ( 2020 ), Yang et al. ( 2015 ) and Ma et al. ( 2021 ) are good examples for Levels 2 and 3.

In level 4, mitigation actions are also determined automatically. This level represents a whole automation of failure analysis. BNs are among the few tools that can encompass all steps of failure analysis. As such, we suggest using them. CBR databases can be used by BNs plus system-based data to provide possible corrective actions. Tönnes ( 2018 ), Zuo et al. ( 2016 ) and Hecht and Baum ( 2019 ) are among good studies for Level 4. Chang et al. ( 2017 ) has focused to automate and visualize corrective actions using a self-organizing map (SOM) neural network in an FMEA methodology. Future research should concentrate on the development of an automated FMEA that dynamically updates the current RPN (Risk Priority Number). This can aid in predicting failures in parts or components of a system using a "Live RPN." The predictive capability of such a tool can be utilized to optimize the overall system. It enables the transformation of a manufacturing system into a self-controlling system, allowing adjustments based on current parameters (Filz et al., 2021b ).

Level 5 is a hybrid intelligence approach to failure analysis that encompasses all other levels and can be implemented within FMEA, RCA, and FTA methodologies when a limited amount of historical and system-based data is available until a comprehensive CBR database is built. BNs provide a good graphical representation and can work with all possible data types. The advantages of BNs are significant enough to be suggested for hybrid intelligence failure analysis. However, we did not find any comprehensive study for this level. A combination of studies that proposed methods to use integrated expert-based, problem-solving, and system-based data, such as Waghen and Ouali ( 2021 ); Filz et al. ( 2021b ), is suggested. Nonetheless, this level remains open and needs to be the focus of future research by scholars. To facilitate the implementation of hybrid intelligence failure analysis, a user-friendly interface is crucial for operators to interact with. Several studies have proposed user-interface applications for this purpose, including (Chan & McNaught, 2008 ; Camarillo et al., 2018 ; Li et al., 2013 ; Jacobo et al., 2007 ; Yang et al., 2009 , 2020 ; Demirbaga et al., 2021 ; Snooke & Price, 2012 ; Palluat et al., 2006 ).

Introducing knowledge into tools

In this section, we analyze which types of knowledge, expert-driven, data-driven, or a hybrid of both, are usually used with which tools and what the implications are for providing insights on suitable tools for hybrid intelligence failure analyses.

Figure 6 shows the distribution of literature based on the input data, tools, and outputs (four general steps of a failure analysis methodology in Fig. 3 ). The first column of nodes shows various combinations of types of knowledge, expert-driven, data-driven, or a hybrid of both, that are usually used in the literature to identify the structure of failure and to detect the probability of failures. The second column of nodes shows various tools that are used to analyze the failure. The third column of nodes shows outputs of a failure analysis. The number of studies with each particular focus is shown by the thickness of an arrow. Details are in Appendix EC.1.

Literature distribution based on inputs, tools, and outputs

The following studies have tried to introduce knowledge and data from expert and data based sources to a failure analysis methodology. Filz et al. ( 2021b ) utilized expert knowledge to identify the structure of failure, the components involved, and the necessary sensors to be used. They then employed sensors to capture data and leveraged problem-solving data from the recorded expert archive to identify failures in a deep learning model. Similarly, Musumeci et al. ( 2020 ) used supervised algorithms to classify failures. Mazzoleni et al. ( 2017 ) they used data from sensors to select the most effective features related to a failure, and subsequently employed sensor data and failure expert data-sets within a gradient boosting tree algorithm to identify the possibility of the failure. Duan et al. ( 2020 ) used data from different sources in a similar way for a neural network to identify the root cause of a failure. Ma and Wu ( 2020 ) utilized expert knowledge to identify failures in construction projects. Subsequently, expert datasets were employed in conjunction with project performance indices to predict the possibility of a failure and determine the root cause of the failure using a neural network tool.

Hireche et al. ( 2018 ), Yang et al. ( 2015 ) gathered data from sensors to determine the conditions of each failure/component node. Then, a BN was used to identify the risks and causes. A multi-level tree is developed by Waghen and Ouali ( 2021 ). Each level contains a solution, pattern, and condition level. Solutions are retrieved from a historical failure database as a combination of certain patterns. The pattern in each problem has been identified and related to the solution using a supervised machine-learning tool. Each level is linked to the next level until the root cause of a failure is correctly identified.

Other usefull tips for introducing knowledge from different sources to a failure analysis methodology can be found in the following studies. Zuo et al. ( 2016 ) divided a multi-operation machining process operation, station, and key characteristics levels. Stream of variations (SoV) was used to evaluate the sensitivities of the machining operations level by level. Results were used to find the sources affecting the quality. Distribution techniques for each quality precision using multi-objective optimization were chosen. Dey and Stori ( 2005 ) used a message-passing method (Pearl, 1988 ) to update a BN using data from sensors to estimate the condition of the system and update the CPTs, when each sensor output is considered as a node in the BN. Chan and McNaught ( 2008 ) also used sensor data to change the probabilities in a BN. A user interface is also developed to make inferences and present the results to operators.

Rokach and Hutter ( 2012 ) used the sequence of machines and a commonality graph of steps and failure causes data to cluster failures to find commonalities between them. A GO methodology is used by Liu et al. ( 2019b ) to model the system and a heuristic is used to construct BN structure and probabilities from the GO methodology model. Teoh and Case ( 2005 ) developed an objective-oriented framework that considers conceptual design information. A hierarchy of components, an assembly tree, and a functional diagram are built to capture data from processes and feed it to FMEA. Bhardwaj et al. ( 2022 ) used historical data from a similar system to estimate failure detection probabilities. Hecht and Baum ( 2019 ) used SysML to describe components and failures.

Zhou et al. ( 2015 ) used a tree of a system. Two classes of knowledge, shallow knowledge and deep knowledge, were gathered to generate rules for failure analysis. The former indicates the experiential knowledge of domain experts, and the latter is the knowledge about the structure and basic principle of the diagnosis system. Liu and Ke ( 2007 ) used CBR to find similar problems and solutions, text mining to find key concepts of the failure in the historical failure record texts, and rule mining to find hidden patterns among system features and failures. Filz et al. ( 2021a ) gathered process parameters after each station using a quality check station. Then a self-organizing Map was used to find failure propagation and cause and effect. Ma et al. ( 2021 ) used data from the system to determine features of problems, products, and operators. Data from problem-solving databases was used to find new failures and classified them using the features and historical data.

Psarommatis and Kiritsis ( 2022 ) developed a methodology that uses data-driven and knowledge-based approaches, an ontology base on the MASON ontology to describe the production domain and enrich the available data. Wang et al. ( 2018 ) developed a data acquisition system including a monitor, sensor, and filter modules. A fault diagram models failure propagation. They extended the Kripke structure by proposing the feature-labeled transition system, which is used to distinguish the behavior of the transition relationship by adding a signature to the transition relationship.

This section highlights that in the realm of failure analysis, a majority of research papers have utilized a hybrid approach, combining expert and data knowledge for tasks such as failure detection, classification, and feature selection. However, to achieve real-time failure analysis, a more effective integration of these two sources is crucial. This integration should enable operators and engineers to provide timely input to the system and observe immediate results. Furthermore, only a limited number of studies have specifically focused on the identification of failure structures using either data or a hybrid of expert and data knowledge.

The use of BNs has emerged as a highly promising approach for achieving real-time input and structure identification in the field of failure analysis. By leveraging both expert knowledge and data sources, BNs have the capability to effectively incorporate expert knowledge as constraints within structure identification algorithms. Unlike traditional classification algorithms that are primarily designed for continuous data, BNs are versatile in handling both discrete and continuous data types. Moreover, BNs possess several strengths that make them particularly suitable for failure analysis. They excel at performing real-time inferences, engaging in counterfactual reasoning, and effectively managing confounding factors. Given these advantages, it is essential to allocate more attention to the application of BNs in hybrid intelligence failure analysis. This involves further exploration of their capabilities and conducting comparative analyses with other tools to assess their effectiveness in various scenarios. By focusing on BNs and conducting comprehensive evaluations, researchers can enhance the understanding and adoption of these powerful tools for improved failure analysis in real-time settings.

Hybrid intelligence

A collaborative failure analysis methodology is needed, in which artificial intelligence tools, machines, and humans can communicate. While hybrid intelligence has gained attention in various fields, literature on the subject for failure analysis is still limited. For example, Piller et al. ( 2022 ) discussed methods to enhance productivity in manufacturing using hybrid intelligence. They explored considerations such as task allocation between humans and machines and the degree of machine intelligence integrated into manufacturing processes. Petrescu and Krishen ( 2023 ) and references within have delved into the benefits and future directions of hybrid intelligence for marketing analytics. Mirbabaie et al. ( 2021 ) has reviewed challenges associated with hybrid intelligence, focusing particularly on conversational agents in hospital settings. Ye et al. ( 2022 ) developed a parallel cognition model. This model draws on both a psychological model and user behavioral data to adaptively learn an individual’s cognitive knowledge. Lee et al. ( 2020 ) combined a data-driven prediction model with a rule-based system to benefit from the combination of human and machine intelligence for personalized rehabilitation assessment.

An artificial intelligence tool should not only provide its final results but also provide its reasoning. A human can analyze the artificial intelligence tool reasoning through a user-interface application and correct possible mistakes instantly and effortlessly. To enable this capability, the use of a white-box artificial tool, such as Bayesian networks, is essential. Explainable AI aids in comprehending and trusting the decision-making process of the hybrid intelligence system by providing the reasoning behind it (Confalonieri et al., 2021 ). Moreover, a machine should be able to interpret and implement an artificial intelligence tool and/or human solutions. Artificial intelligence tools, machines, and humans can learn from mistakes (Correia et al., 2023 ).

To fully exploit the complementarity in human–machine collaborations and effectively utilize the strengths of both, it is important to recognize and understand their roles, limitations, and capabilities in the context of failure analysis. Future research should focus on developing a clear plan for their teamwork and joint actions, including determining the optimal sensor types and locations, quality inspection stations, and human/machine analysis processes. In other words, How to design a decision support system that integrates both human knowledge and machine intelligence with respect to quality management? should be answered. Additionally, tools should be developed to propose possible mitigation actions based on the unique characteristics of the system, environment, humans, and machines. To achieve this, system-related data along with CBR data can be analyzed to find potential mitigation actions.

A general framework for human–machine fusion could involve the following steps: identifying applicable human knowledge and machine data for the problem, determining machine intelligence tools that facilitate the integration of human–machine elements like BNs, identifying the suitable points in the decision-making process to combine human knowledge and machine intelligence effectively, designing the user interface, and incorporating online learning using input from human knowledge (Jarrahi et al., 2022 ). However, human–machine fusion is not an easy task due to the complexity of human–machine interaction, the need for effective and online methods to work with both human and machine data, and the challenge of online learning from human knowledge. For instance, while ChatGPT interacts well with humans, it currently does not update its knowledge using human knowledge input for future cases (Dellermann et al., 2019 ; Correia et al., 2023 ).

Failure propagation and correlation

Most FMEA papers concentrated on analyzing failures in individual products, processes, or machines. It is essential to acknowledge that production processes and machines are interconnected, leading to the correlation and propagation of failures among them. Consequently, it becomes crucial to address the challenge of analyzing failures in multiple machines. To effectively tackle this issue, a holistic approach is necessary. Rather than focusing solely on individual machines, take a broader perspective by considering the entire production system to identify the interdependencies and interactions among different machines, multiple processes, and within the system.

For an intelligence failure analysis, it is necessary to exploit detailed system-related data to carefully and comprehensively identify the relations between different parts of a system, product, and/or process. Some papers have suggested methods to identify failure propagation and correlation (Wang et al., 2021 ; Zhu et al., 2021 ; Chen et al., 2017 ). They usually proposed methods to analyze correlations only between failures or risk criteria using MCDM or statistical methods. However, an intelligence failure analysis should go beyond this and identify failure propagation and correlation among parts of a system.

In the literature, Chen and Jiao ( 2017 ) proposed finite state machine (FSM) theory to model the interactive behaviors between the components, constructing the transition process of fault propagation through the extraction of the state, input, output, and state function of the component. Zuo et al. ( 2016 ) used SoV to model propagation of variations from station to station and operation to operation. A propagation from one station (operation) to the next station (operation) was modeled using a regression like formula. Ament and Goch ( 2001 ) used quality check data after each station to train a neural network for failure progagation and estimate the relationships betweenfailure in stations using a regression model to find patterns in quality check data. Ma et al. ( 2021 ) used patterns in data to classify failures and identify causes.

To conduct an intelligence failure analysis, it is important to identify every part involved, their roles, characteristics, and states. The analysis should include the identification of failure propagation and effects on functions, parts, and other failures. One approach to analyzing failures is through simulation, which can help assess the changes in the characteristics of every part of a system, including humans, machines, and the environment. To analyze the complexity of failure propagation and mutual interactions among different parts of a system, data-driven tools and heuristic algorithms need to be developed. These tools should be capable of managing a large bill of materials and analyzing the failure structure beyond the traditional statistical and MCDM methods. Rule mining can be a useful tool for detecting failure correlation and propagation, especially in situations where there is limited data available, and human interpretation is crucial.

Hybrid methodologies

FMEA, RCA, and FTA methodologies are all complementary and can improve each other’s performance. Furthermore, the availability of data, advanced tools to process data, and the ability to gather online data may lead to a unified FMEA, RCA, and FTA methodology. The reason for this is that while FMEA tries to find potential failures, RCA and FTA try to find root causes of failures, they use similar data and tools to analyze data.

In the literature, FTA has been used as an internal part of FMEA by Steenwinckel et al. ( 2018 ), Palluat et al. ( 2006 )and RCA by Chen et al. ( 2018 ). Using automated mappings from FMEA data to a domain-specific ontology and rules derived from a constructed FTA, Steenwinckel et al. ( 2018 ) annotated and reasoned on sensor observations. Palluat et al. ( 2006 ) used FTA to illustrate the failure structure of a system within an FMEA methodology and developed a neuro-fuzzy network to analyze failures. Chen et al. ( 2018 ) used FTA and graph theory tools, such as the maximum spanning tree, to find the root cause of failures in an RCA methodology. However, studies on the integration of these methodologies regarding the availability of data, tools, and applications should be done to use their advantages within a unified methodology that detects potential failures, finds root causes and effects, and improves the system.

Other future research

Several promising future research directions can be pursued. Cost-based and economic quantification approaches can be integrated into intelligent methodologies to enable more informed decision-making related to failures, their effects, and corrective actions. Additionally, incorporating customer satisfaction criteria, such as using the Kano model, can be useful in situations where there are several costly failures in a system, and budget constraints make it necessary to select the most effective corrective action. This approach has been successfully applied in previous studies (Madzík & Kormanec, 2020 ), and can help optimize decision-making in complex failure scenarios.

Data management is a critical aspect of intelligence methodologies, given the large volume and diverse types of data that need to be processed. Therefore, it is important to design reliable databases that can store and retrieve all necessary data. Ontology can be a valuable tool to help integrate and connect different types of data (Rajpathak & De, 2016 ; Ebrahimipour et al., 2010 ). However, it is also essential to consider issues such as data obsolescence and updates, especially when corrective actions are taken and root causes are removed. Failure to address these issues can lead to incorrect analysis and decision-making.

Traditionally, only single failures were considered in analysis because analyzing a combination of multiple failures was impossible. However, in a system, two or more failures may occur simultaneously or sequentially. It is also possible that a failure occurs as a consequence of another failure. These circumstances are complicated because each failure can have several root causes, and another failure is only one of its causes. Therefore, a clear and powerful tool, such as Bayesian Networks (BNs), should be used to analyze failures and accurately identify possible causes.

The traditional failure analysis methodologies had limitations such as repeatability, subjectivity, and time consumption, which have been addressed by intelligence failure analysis. However, there is a need for more focus on explainability, objective evaluation criteria, and results reliability as some intelligent tools, such as neural networks, act as black boxes. Therefore, suitable tools, such as BNs, should be well-developed and adapted for (hybrid) intelligence failure analysis. Details such as the time and location of the detected failure, possible factors of the causes, such as location, time, conditions, and description of the cause, and reasons behind the causes, such as human fatigue, should be considered within a methodology. These can help to go beyond the CBR and propose intelligence solutions based on the reasons behind a cause. While RCA has implemented these data to a limited extent, FMEA lacks such implementation.

This paper has collected information on both proactive and reactive failure analysis methodologies from 86 papers that focus on FMEA, RCA, or FTA. The goal is to identify areas for improvement, trends, and open problems regarding intelligent failure analysis. This information can help researchers learn the benefits of both methodologies, use their tools, and integrate them to strengthen failure analysis. Each paper has been read and analyzed to extract data and tools used within the paper and their benefits. It was observed that the literature on the three methodologies, FMEA, RCA, and FTA, is diverse. In Industry 4.0, the availability of data, and advances in technology are helping these methodologies benefit from the same tools, such as BNs and neural networks, and make them more integrated.

The literature was classified based on the data needed for a (hybrid) intelligence failure analysis methodology and the tools used for failure analysis to be data-driven and automated. In addition, trends to make these methodologies smart and possible future research in this regard were discussed.

Two main classes of failure structure and failure detection data are usually needed for a failure analysis methodology, each of which can be classified as expert-driven and data-driven. However, a combination of all types of data can lead to more reliable failure analysis. Most papers focused on operational and historical expert-driven and/or data-driven problem-solving data. Among the tools used within FMEA, RCA, and FTA methodologies, BNs have the capability to make a methodology smart and interact with both humans and machines to benefit from hybrid intelligence. BNs not only can analyze failures to identify root causes but also can analyze possible solutions to provide necessary action to prevent failures. A BN’s are also capable of real-time inference, counterfactual reasoning, and managing confounding factors. BNs handle both discrete and continuous data types, unlike traditional classification algorithms. Besides BNs, classification by neural networks, other classification tools, rule-based algorithms, and other tools have been proposed in the literature.

Finally, managerial insights and future research are provided. Most studies have focused on the determination of root causes. It is necessary to automatically find possible mitigation and corrective actions. This step of a failure analysis methodology needs more interaction with humans. Thus, the benefits of hybrid intelligence can be more evident here. It is imperative for humans and machines to work together to properly identify and resolve failures. System-related data should be analyzed to find possible corrective actions. This data is usually available for both proactive and reactive methodologies. Our study showed an effectively tool to integrate knowledge from experts and sensors in needed, enabling operators and engineers to provide timely input and observe immediate results. There is a need to identify failure structures using a hybrid approach that combines expert and data knowledge. Real-time input and structure identification with Bayesian networks can be achieved through the use of Bayesian networks. Further exploration of BNs and comparative analyses with other tools is necessary to enhance understanding and adoption of the best tools for a hybrid intelligence failure analysis in real-time scenarios to prevent failures.

Data availability

There is no data related to this paper.

Agrawal, V., Panigrahi, B. K., & Subbarao, P. (2016). Intelligent decision support system for detection and root cause analysis of faults in coal mills. IEEE Transactions on Fuzzy Systems, 25 (4), 934–944.

Article   Google Scholar  

Akata, Z., Balliet, D., De Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020). A research agenda for hybrid intelligence: Augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. Computer, 53 (08), 18–28.

Al-Mamory, S. O., & Zhang, H. (2009). Intrusion detection alarms reduction using root cause analysis and clustering. Computer Communications, 32 (2), 419–430.

Ament, C., & Goch, G. (2001). A process oriented approach to automated quality control. CIRP Annals, 50 (1), 251–254.

Bhardwaj, U., Teixeira, A., & Soares, C. G. (2022). Bayesian framework for reliability prediction of subsea processing systems accounting for influencing factors uncertainty. Reliability Engineering & System Safety, 218 , 108143.

Brahim, I. B., Addouche, S. A., El Mhamedi, A., & Boujelbene, Y. (2019). Build a Bayesian network from FMECA in the production of automotive parts: Diagnosis and prediction. IFAC-PapersOnLine, 52 (13), 2572–2577.

Cai, B., Huang, L., & Xie, M. (2017). Bayesian networks in fault diagnosis. IEEE Transactions on Industrial Informatics, 13 (5), 2227–2240.

Camarillo, A., Ríos, J., & Althoff, K. D. (2018). Knowledge-based multi-agent system for manufacturing problem solving process in production plants. Journal of Manufacturing Systems, 47 , 115–127.

Chan, A., & McNaught, K. R. (2008). Using Bayesian networks to improve fault diagnosis during manufacturing tests of mobile telephone infrastructure. Journal of the Operational Research Society, 59 (4), 423–430.

Chang, W. L., Pang, L. M., & Tay, K. M. (2017). Application of self-organizing map to failure modes and effects analysis methodology. Neurocomputing, 249 , 314–320.

Chang, W. L., Tay, K. M., & Lim, C. P. (2015). Clustering and visualization of failure modes using an evolving tree. Expert Systems with Applications, 42 (20), 7235–7244.

Chen, H. S., Yan, Z., Zhang, X., Liu, Y., & Yao, Y. (2018). Root cause diagnosis of process faults using conditional Granger causality analysis and maximum spanning tree. IFAC-PapersOnLine, 51 (18), 381–386.

Chen, L., Jiao, J., Wei, Q., & Zhao, T. (2017). An improved formal failure analysis approach for safety-critical system based on mbsa. Engineering Failure Analysis, 82 , 713–725.

Chen, X., & Jiao, J. (2017). A fault propagation modeling method based on a finite state machine. Annual Reliability and Maintainability Symposium (RAMS), 2017 , 1–7.

Google Scholar  

Chhetri, T. R., Aghaei, S., Fensel, A., Göhner, U., Gül-Ficici, S., & Martinez-Gil, J. (2023). Optimising manufacturing process with Bayesian structure learning and knowledge graphs. Computer Aided Systems Theory - EUROCAST, 2022 , 594–602.

Chien, C. F., Liu, C. W., & Chuang, S. C. (2017). Analysing semiconductor manufacturing big data for root cause detection of excursion for yield enhancement. International Journal of Production Research, 55 (17), 5095–5107.

Clancy, R., O’Sullivan, D., & Bruton, K. (2023). Data-driven quality improvement approach to reducing waste in manufacturing. The TQM Journal, 35 (1), 51–72.

Confalonieri, R., Coba, L., Wagner, B., & Besold, T. R. (2021). A historical perspective of explainable artificial intelligence. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11 (1), e1391.

Correia, A., Grover, A., Schneider, D., Pimentel, A. P., Chaves, R., De Almeida, M. A., & Fonseca, B. (2023). Designing for hybrid intelligence: A taxonomy and survey of crowd-machine interaction. Applied Sciences, 13 (4), 2198.

Dabous, S. A., Ibrahim, F., Feroz, S., & Alsyouf, I. (2021). Integration of failure mode, effects, and criticality analysis with multi-criteria decision-making in engineering applications: Part I- manufacturing industry. Engineering Failure Analysis, 122 , 105264.

Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019). Hybrid intelligence. Business & Information Systems Engineering, 61 , 637–643.

Demirbaga, U., Wen, Z., Noor, A., Mitra, K., Alwasel, K., Garg, S., Zomaya, A. Y., & Ranjan, R. (2021). Autodiagn: An automated real-time diagnosis framework for big data systems. IEEE Transactions on Computers, 71 (5), 1035–1048.

Dey, S., & Stori, J. (2005). A Bayesian network approach to root cause diagnosis of process variations. International Journal of Machine Tools and Manufacture, 45 (1), 75–91.

Du, S., Lv, J., & Xi, L. (2012). A robust approach for root causes identification in machining processes using hybrid learning algorithm and engineering knowledge. Journal of Intelligent Manufacturing, 23 (5), 1833–1847.

Duan, P., He, Z., He, Y., Liu, F., Zhang, A., & Zhou, D. (2020). Root cause analysis approach based on reverse cascading decomposition in QFD and fuzzy weight ARM for quality accidents. Computers & Industrial Engineering, 147 , 106643.

Ebeling, C. E. (2019). An introduction to reliability and maintainability engineering . Waveland Press.

Ebrahimipour, V., Rezaie, K., & Shokravi, S. (2010). An ontology approach to support FMEA studies. Expert Systems with Applications, 37 (1), 671–677.

Filz, M. A., Gellrich, S., Lang, F., Zietsch, J., Abraham, T., & Herrmann, C. (2021). Data-driven analysis of product property propagation to support process-integrated quality management in manufacturing systems. Procedia CIRP, 104 , 900–905.

Filz, M. A., Langner, J. E. B., Herrmann, C., & Thiede, S. (2021). Data-driven failure mode and effect analysis (FMEA) to enhance maintenance planning. Computers in Industry, 129 , 103451.

French, S., Bedford, T., Pollard, S. J., & Soane, E. (2011). Human reliability analysis: A critique and review for managers. Safety Science, 49 (6), 753–763.

Gomez-Andrades, A., Barco, R., Serrano, I., Delgado, P., Caro-Oliver, P., & Munoz, P. (2016). Automatic root cause analysis based on traces for LTE self-organizing networks. IEEE Wireless Communications, 23 (3), 20–28.

Gómez-Andrades, A., Munoz, P., Serrano, I., & Barco, R. (2015). Automatic root cause analysis for lte networks based on unsupervised techniques. IEEE Transactions on Vehicular Technology, 65 (4), 2369–2386.

Hecht, M., & Baum, D. (2019). Failure propagation modeling in FMEAs for reliability, safety, and cybersecurity using SysML. Procedia Computer Science, 153 , 370–377.

Hireche, C., Dezan, C., Mocanu, S., Heller, D., & Diguet, J. P. (2018). Context/resource-aware mission planning based on BNs and concurrent MDPs for autonomous UAVs. Sensors, 18 (12), 4266.

Huang, J., You, J. X., Liu, H. C., & Song, M. S. (2020). Failure mode and effect analysis improvement: A systematic literature review and future research agenda. Reliability Engineering & System Safety, 199 , 106885.

Insua, D. R., Ruggeri, F., Soyer, R., & Wilson, S. (2020). Advances in Bayesian decision making in reliability. European Journal of Operational Research, 282 (1), 1–18.

Jacobo, V., Ortiz, A., Cerrud, Y., & Schouwenaars, R. (2007). Hybrid expert system for the failure analysis of mechanical elements. Engineering Failure Analysis, 14 (8), 1435–1443.

Jarrahi, M. H., Lutz, C., & Newlands, G. (2022). Artificial intelligence, human intelligence and hybrid intelligence based on mutual augmentation. Big Data & Society, 9 (2), 20539517221142824.

Jiang, S., Qin, S., Pulsipher, J. L., & Zavala, V. M. (2024). Convolutional neural networks: Basic concepts and applications in manufacturing. Artificial Intelligence in Manufacturing, 8 , 63–102.

Jiao, J., Zhen, W., Zhu, W., & Wang, G. (2020). Quality-related root cause diagnosis based on orthogonal kernel principal component regression and transfer entropy. IEEE Transactions on Industrial Informatics, 17 (9), 6347–6356.

Johnson, K., & Khan, M. K. (2003). A study into the use of the process failure mode and effects analysis (PFMEA) in the automotive industry in the UK. Journal of Materials Processing Technology, 139 (1–3), 348–356.

Jomthanachai, S., Wong, W. P., & Lim, C. P. (2021). An application of data envelopment analysis and machine learning approach to risk management. IEEE Access, 9 , 85978–85994.

Kabir, S., & Papadopoulos, Y. (2019). Applications of Bayesian networks and Petri nets in safety, reliability, and risk assessments: A review. Safety Science, 115 , 154–175.

Khakzad, N., Khan, F., & Amyotte, P. (2012). Dynamic risk analysis using bow-tie approach. Reliability Engineering & System Safety, 104 , 36–44.

Khorshidi, H. A., Gunawan, I., & Ibrahim, M. Y. (2015). Data-driven system reliability and failure behavior modeling using FMECA. IEEE Transactions on Industrial Informatics, 12 (3), 1253–1260.

Kumar, M., & Kaushik, M. (2020). System failure probability evaluation using fault tree analysis and expert opinions in intuitionistic fuzzy environment. Journal of Loss Prevention in the Process Industries, 67 , 104236.

Lee BH (2001) Using Bayes belief networks in industrial FMEA modeling and analysis. Annual Reliability and Maintainability Symposium. 2001 Proceedings. International Symposium on Product Quality and Integrity (Cat. No.01CH37179) , pp. 7–15.

Lee MH, Siewiorek DP, Smailagic A, Bernardino A, Bermúdez i Badia S (2020) Interactive hybrid approach to combine machine and human intelligence for personalized rehabilitation assessment. Proceedings of the ACM Conference on Health, Inference, and Learning , pp. 160–169.

Leu, S. S., & Chang, C. M. (2013). Bayesian-network-based safety risk assessment for steel construction projects. Accident Analysis & Prevention, 54 , 122–133.

Li, B., Han, T., & Kang, F. (2013). Fault diagnosis expert system of semiconductor manufacturing equipment using a Bayesian network. International Journal of Computer Integrated Manufacturing, 26 (12), 1161–1171.

Liu, C., Lore, K. G., Jiang, Z., & Sarkar, S. (2021). Root-cause analysis for time-series anomalies via spatiotemporal graphical modeling in distributed complex systems. Knowledge-Based Systems, 211 , 106527.

Liu, D. R., & Ke, C. K. (2007). Knowledge support for problem-solving in a production process: A hybrid of knowledge discovery and case-based reasoning. Expert Systems with Applications, 33 (1), 147–161.

Liu, H. C., Chen, X. Q., Duan, C. Y., & Wang, Y. M. (2019). Failure mode and effect analysis using multi-criteria decision making methods: A systematic literature review. Computers & Industrial Engineering, 135 , 881–897.

Liu, H. C., Liu, L., & Liu, N. (2013). Risk evaluation approaches in failure mode and effects analysis: A literature review. Expert Systems with Applications, 40 (2), 828–838.

Liu, L., Fan, D., Wang, Z., Yang, D., Cui, J., Ma, X., & Ren, Y. (2019). Enhanced GO methodology to support failure mode, effects and criticality analysis. Journal of Intelligent Manufacturing, 30 (3), 1451–1468.

Ma, G., & Wu, M. (2020). A big data and FMEA-based construction quality risk evaluation model considering project schedule for shanghai apartment projects. International Journal of Quality & Reliability Management, 37 (1), 18–33.

Ma, Q., Li, H., & Thorstenson, A. (2021). A big data-driven root cause analysis system: Application of machine learning in quality problem solving. Computers & Industrial Engineering, 160 , 107580.

Madzík, P., & Kormanec, P. (2020). Developing the integrated approach of Kano model and failure mode and effect analysis. Total Quality Management & Business Excellence, 31 (15–16), 1788–1810.

Mangalathu, S., Hwang, S. H., & Jeon, J. S. (2020). Failure mode and effects analysis of RC members based on machine-learning-based shapley additive explanations (shap) approach. Engineering Structures, 219 , 110927.

Mazzoleni, M., Maccarana, Y., & Previdi, F. (2017). A comparison of data-driven fault detection methods with application to aerospace electro-mechanical actuators. IFAC-PapersOnLine, 50 (1), 12797–12802.

Mirbabaie, M., Stieglitz, S., & Frick, N. R. (2021). Hybrid intelligence in hospitals: Towards a research agenda for collaboration. Electronic Markets, 31 , 365–387.

Musumeci, F., Magni, L., Ayoub, O., Rubino, R., Capacchione, M., Rigamonti, G., Milano, M., Passera, C., & Tornatore, M. (2020). Supervised and semi-supervised learning for failure identification in microwave networks. IEEE Transactions on Network and Service Management, 18 (2), 1934–1945.

Nepal, B., & Yadav, O. P. (2015). Bayesian belief network-based framework for sourcing risk analysis during supplier selection. International Journal of Production Research, 53 (20), 6114–6135.

Nie, W., Liu, W., Wu, Z., Chen, B., & Wu, L. (2019). Failure mode and effects analysis by integrating Bayesian fuzzy assessment number and extended gray relational analysis-technique for order preference by similarity to ideal solution method. Quality and Reliability Engineering International, 35 (6), 1676–1697.

Oliveira, E. E., Miguéis, V. L., & Borges, J. L. (2021). Understanding overlap in automatic root cause analysis in manufacturing using causal inference. IEEE Access, 10 , 191–201.

Oliveira, E. E., Miguéis, V. L., & Borges, J. L. (2022). On the influence of overlap in automatic root cause analysis in manufacturing. International Journal of Production Research, 60 (21), 6491–6507.

Oliveira, E. E., Miguéis, V. L., & Borges, J. L. (2023). Automatic root cause analysis in manufacturing: An overview & conceptualization. Journal of Intelligent Manufacturing, 34 , 2061–2078.

Oztemel, E., & Gursev, S. (2020). Literature review of industry 4.0 and related technologies. Journal of Intelligent Manufacturing, 31 (1), 127–182.

Palluat, N., Racoceanu, D., & Zerhouni, N. (2006). A neuro-fuzzy monitoring system: Application to flexible production systems. Computers in Industry, 57 (6), 528–538.

Pang, J., Zhang, N., Xiao, Q., Qi, F., & Xue, X. (2021). A new intelligent and data-driven product quality control system of industrial valve manufacturing process in CPS. Computer Communications, 175 , 25–34.

Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference . Morgan kaufmann.

Petrescu, M., & Krishen, A. S. (2023). Hybrid intelligence: Human-ai collaboration in marketing analytics. Journal of Marketing Analytics, 11 (3), 263–274.

Piller, F. T., Nitsch, V., & van der Aalst, W. (2022). Hybrid intelligence in next generation manufacturing: An outlook on new forms of collaboration between human and algorithmic decision-makers in the factory of the future (pp. 139–158). Forecasting Next Generation Manufacturing: Digital Shadows, Human-Machine Collaboration, and Data-driven Business Models.

Price, C. J., & Taylor, N. S. (2002). Automated multiple failure FMEA. Reliability Engineering & System Safety, 76 (1), 1–10.

Psarommatis, F., & Kiritsis, D. (2022). A hybrid decision support system for automating decision making in the event of defects in the era of zero defect manufacturing. Journal of Industrial Information Integration, 26 , 100263.

Rajpathak, D., & De, S. (2016). A data-and ontology-driven text mining-based construction of reliability model to analyze and predict component failures. Knowledge and Information Systems, 46 (1), 87–113.

Rastayesh, S., Bahrebar, S., Blaabjerg, F., Zhou, D., Wang, H., & Dalsgaard Sørensen, J. (2019). A system engineering approach using FMEA and Bayesian network for risk analysis-a case study. Sustainability, 12 (1), 77.

Rausand, M., & Øien, K. (1996). The basic concepts of failure analysis. Reliability Engineering & System Safety, 53 (1), 73–83.

Rivera Torres, P. J., Serrano Mercado, E. I., Llanes Santiago, O., & Anido Rifón, L. (2018). Modeling preventive maintenance of manufacturing processes with probabilistic Boolean networks with interventions. Journal of Intelligent Manufacturing, 29 (8), 1941–1952.

Rokach, L., & Hutter, D. (2012). Automatic discovery of the root causes for quality drift in high dimensionality manufacturing processes. Journal of Intelligent Manufacturing, 23 (5), 1915–1930.

Ruijters, E., & Stoelinga, M. (2015). Fault tree analysis: A survey of the state-of-the-art in modeling, analysis and tools. Computer Science Review, 15 , 29–62.

Sariyer, G., Mangla, S. K., Kazancoglu, Y., Ocal Tasar, C., & Luthra, S. (2021). Data analytics for quality management in Industry 4.0 from a MSME perspective. Annals of Operations Research, 23 , 1–19.

Sayed, M. S., & Lohse, N. (2014). Ontology-driven generation of Bayesian diagnostic models for assembly systems. The International Journal of Advanced Manufacturing Technology, 74 (5), 1033–1052.

Schuh, G., Gützlaff, A., Thomas, K., & Welsing, M. (2021). Machine learning based defect detection in a low automated assembly environment. Procedia CIRP, 104 , 265–270.

Shi, H., Wang, L., Li, X. Y., & Liu, H. C. (2020). A novel method for failure mode and effects analysis using fuzzy evidential reasoning and fuzzy Petri nets. Journal of Ambient Intelligence and Humanized Computing, 11 (6), 2381–2395.

Snooke, N., & Price, C. (2012). Automated FMEA based diagnostic symptom generation. Advanced Engineering Informatics, 26 (4), 870–888.

Spreafico, C., Russo, D., & Rizzi, C. (2017). A state-of-the-art review of FMEA/FMECA including patents. Computer Science Review, 25 , 19–28.

Stamatis, D. H. (2003). Failure mode and effect analysis: FMEA from theory to execution . ASQ Quality Press.

Steenwinckel B, Heyvaert P, De Paepe D, Janssens O, Vanden Hautte S, Dimou A, De Turck F, Van Hoecke S, Ongenae F (2018) Towards adaptive anomaly detection and root cause analysis by automated extraction of knowledge from risk analyses. 9th International Semantic Sensor Networks Workshop, Co-Located with 17th International Semantic Web Conference (ISWC 2018) , Vol. 2213, pp. 17–31.

Sun, Y., Qin, W., Zhuang, Z., & Xu, H. (2021). An adaptive fault detection and root-cause analysis scheme for complex industrial processes using moving window KPCA and information geometric causal inference. Journal of Intelligent Manufacturing, 32 (7), 2007–2021.

Tari, J. J., & Sabater, V. (2004). Quality tools and techniques: Are they necessary for quality management? International Journal of Production Economics, 92 (3), 267–280.

Tay, K. M., Jong, C. H., & Lim, C. P. (2015). A clustering-based failure mode and effect analysis model and its application to the edible bird nest industry. Neural Computing and Applications, 26 (3), 551–560.

Teoh, P. C., & Case, K. (2004). Failure modes and effects analysis through knowledge modelling. Journal of Materials Processing Technology, 153 , 253–260.

Teoh, P. C., & Case, K. (2005). An evaluation of failure modes and effects analysis generation method for conceptual design. International Journal of Computer Integrated Manufacturing, 18 (4), 279–293.

Thomé, A. M. T., Scavarda, L. F., & Scavarda, A. J. (2016). Conducting systematic literature review in operations management. Production Planning & Control, 27 (5), 408–420.

Tönnes, W. (2018). Applying data of historical defects to increase efficiency of rework in assembly. Procedia CIRP, 72 , 255–260.

van der Aalst, W. M. (2021). Hybrid intelligence: To automate or not to automate, that is the question. International Journal of Information Systems and Project Management, 9 (2), 5–20.

Waghen, K., & Ouali, M. S. (2021). Multi-level interpretable logic tree analysis: A data-driven approach for hierarchical causality analysis. Expert Systems with Applications, 178 , 115035.

Wan, C., Yan, X., Zhang, D., Qu, Z., & Yang, Z. (2019). An advanced fuzzy Bayesian-based FMEA approach for assessing maritime supply chain risks. Transportation Research Part E, 125 , 222–240.

Wang, L., Li, S., Wei, O., Huang, M., & Hu, J. (2018). An automated fault tree generation approach with fault configuration based on model checking. IEEE Access, 6 , 46900–46914.

Wang, Q., Jia, G., Jia, Y., & Song, W. (2021). A new approach for risk assessment of failure modes considering risk interaction and propagation effects. Reliability Engineering & System Safety, 216 , 108044.

Williams, P. M. (2001). Techniques for root cause analysis. Baylor University Medical Center Proceedings, 14 (2), 154–157.

Wu, Z., Liu, W., & Nie, W. (2021). Literature review and prospect of the development and application of FMEA in manufacturing industry. The International Journal of Advanced Manufacturing Technology, 112 (5), 1409–1436.

Xu, Z., & Dang, Y. (2020). Automated digital cause-and-effect diagrams to assist causal analysis in problem-solving: A data-driven approach. International Journal of Production Research, 58 (17), 5359–5379.

Xu, Z., & Dang, Y. (2023). Data-driven causal knowledge graph construction for root cause analysis in quality problem solving. International Journal of Production Research, 61 (10), 3227–3245.

Xu, Z., Dang, Y., Munro, P., & Wang, Y. (2020). A data-driven approach for constructing the component-failure mode matrix for FMEA. Journal of Intelligent Manufacturing, 31 (1), 249–265.

Yang, C., Zou, Y., Lai, P., & Jiang, N. (2015). Data mining-based methods for fault isolation with validated fmea model ranking. Applied Intelligence, 43 (4), 913–923.

Yang, S., Bian, C., Li, X., Tan, L., & Tang, D. (2018). Optimized fault diagnosis based on FMEA-style CBR and BN for embedded software system. The International Journal of Advanced Manufacturing Technology, 94 (9), 3441–3453.

Yang, S., Liu, H., Zhang, Y., Arndt, T., Hofmann, C., Häfner, B., & Lanza, G. (2020). A data-driven approach for quality analytics of screwing processes in a global learning factory. Procedia Manufacturing, 45 , 454–459.

Yang, S., & Liu, T. (1998). A Petri net approach to early failure detection and isolation for preventive maintenance. Quality and Reliability Engineering International, 14 (5), 319-330.

Yang, Y. J., Xiong, Y. L., Zhang, X. Y., Wang, G. H., & Zou, B. (2022). Reliability analysis of continuous emission monitoring system with common cause failure based on fuzzy FMECA and Bayesian networks. Annals of Operations Research, 311 , 451–467.

Yang, Z. X., Zheng, Y. Y., & Xue, J. X. (2009). Development of automatic fault tree synthesis system using decision matrix. International Journal of Production Economics, 121 (1), 49–56.

Ye, P., Wang, X., Zheng, W., Wei, Q., & Wang, F. Y. (2022). Parallel cognition: Hybrid intelligence for human-machine interaction and management. Frontiers of Information Technology & Electronic Engineering, 23 (12), 1765–1779.

Yucesan, M., Gul, M., & Celik, E. (2021). A holistic FMEA approach by fuzzy-based Bayesian network and best-worst method. Complex & Intelligent Systems, 7 (3), 1547–1564.

Yuniarto, H. (2012). The shortcomings of existing root cause analysis tools. Proceedings of the World Congress on Engineering, 3 , 186–191.

Zhang, S., Xie, X., & Qu, H. (2023). A data-driven workflow for evaporation performance degradation analysis: A full-scale case study in the herbal medicine manufacturing industry. Journal of Intelligent Manufacturing, 34 , 651–668.

Zheng, T., Ardolino, M., Bacchetti, A., & Perona, M. (2021). The applications of industry 4.0 technologies in manufacturing context: a systematic literature review. International Journal of Production Research, 59 (6), 1922–1954.

Zhou, A., Yu, D., & Zhang, W. (2015). A research on intelligent fault diagnosis of wind turbines based on ontology and FMECA. Advanced Engineering Informatics, 29 (1), 115–125.

Zhu, C., & Zhang, T. (2022). A review on the realization methods of dynamic fault tree. Quality and Reliability Engineering International, 38 (6), 3233–3251.

Zhu, J. H., Chen, Z. S., Shuai, B., Pedrycz, W., Chin, K. S., & Martínez, L. (2021). Failure mode and effect analysis: A three-way decision approach. Engineering Applications of Artificial Intelligence, 106 , 104505.

Zuo, X., Li, B., & Yang, J. (2016). Error sensitivity analysis and precision distribution for multi-operation machining processes based on error propagation model. The International Journal of Advanced Manufacturing Technology, 86 (1), 269–280.

Download references

This research is funded by Flanders Make under the project AQUME_SBO, project number 2022-0151. Flanders Make is the Flemish strategic research center for the manufacturing industry in Belgium.

Author information

Authors and affiliations.

Department of Industrial Systems Engineering and Product Design, Ghent University, 9052, Ghent, Belgium

Mahdi Mokhtarzadeh, Jorge Rodríguez-Echeverría, Ivana Semanjski & Sidharta Gautama

FlandersMake@UGent–corelab ISyE, Lommel, Belgium

Escuela Superior Politécnica del Litoral, ESPOL, Facultad de Ingeniería en Electricidad y Computación, ESPOL Polytechnic University, Campus Gustavo Galindo, Km 30.5 Vía Perimetral, P.O. Box 09-01-5863, 090112, Guayaquil, Ecuador

Jorge Rodríguez-Echeverría

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mahdi Mokhtarzadeh .

Ethics declarations

Competing interest.

The authors report there are no competing interests to declare.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 137 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Mokhtarzadeh, M., Rodríguez-Echeverría, J., Semanjski, I. et al. Hybrid intelligence failure analysis for industry 4.0: a literature review and future prospective. J Intell Manuf (2024). https://doi.org/10.1007/s10845-024-02376-5

Download citation

Received : 27 June 2023

Accepted : 14 March 2024

Published : 22 April 2024

DOI : https://doi.org/10.1007/s10845-024-02376-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Automated failure analysis
• Data-driven failure analysis
• Human–machine cooperation
• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 23 April 2024

Research on flipped classrooms in foreign language teaching in Chinese higher education

• Wen Kong 1 ,
• Di Li 2 &
• Quanjiang Guo   ORCID: orcid.org/0000-0002-7846-1363 3  

Humanities and Social Sciences Communications volume  11 , Article number:  525 ( 2024 ) Cite this article

Metrics details

• Language and linguistics

This review examines 233 articles published in Chinese academic journals between 2011 and 2021, documenting the state of research concerning flipped classrooms (FCs) in foreign language teaching within the context of higher education in China. Employing the methodological approach of a scoping review, the investigation is underpinned by the five-stage framework articulated by Arksey and O’Malley. The results reveal a notable surge in FC-related studies between 2013 and 2017, with a subsequent decline in scholarly attention. The majority of the reviewed studies on FCs focused on English instruction at the college level, with a conspicuous dearth of inquiry into the application of FCs in the teaching of other foreign languages. All studies were categorized as either empirical or non-empirical, and the most frequently used instruments for data collection were surveys and interviews; case studies were underrepresented in the literature. Early studies focused on the introduction of the new model, while more recent investigations focused on the impact of its implementation. The findings of the in-depth content analysis unearthed a prevailing trend of high learner satisfaction with the FC model, along with favorable direct and indirect educational outcomes. Noteworthy factors influencing the efficacy of FCs included learners’ foreign language proficiency and their self-regulation or self-discipline abilities. The paper concludes with a discussion of the challenges in FC implementation and a call for future research on this promising pedagogy.

Similar content being viewed by others

The implementation of a flipped classroom approach at a UK dental school

Effectiveness of the flipped classroom model on students’ self-reported motivation and learning during the COVID-19 pandemic

Content validity of the Constructivist Learning in Higher Education Settings (CLHES) scale in the context of the flipped classroom in higher education

Introduction.

The flipped classroom (FC), also known as the “inverted classroom”, is a pedagogical approach that first emerged in the 1980s and came into more widespread use in the 2000s (Baker, 2000 ; Bergmann and Sams, 2012 ; Khan, 2012 ). It has gained prominence as advances in technology afford increasing opportunities for ubiquitous access to a variety of online resources. The FC model removes in-class lectures, freeing up classroom time for more in-depth exploration of topics through discussion with peers or problem-solving activities facilitated by instructors. The removed content is often delivered to learners through pre-class materials like video recordings. As a result, in the FC, learning activities that are active and social occur inside the classroom while most information transmission occurs outside the classroom. Today, the FC has been implemented in many different disciplines and in schools and universities around the world (Akcayir and Akcayir, 2018 ).

Proponents of the FC assert its pedagogical merits on several fronts. First, it alleviates the constraints associated with requiring all learning to happen at the same time and place, furnishing learners with an individualized education that enables flexible online study at their own pace as long as an internet connection is available (Hung, 2014 ). Second, it allocates class time to the cultivation of learners’ higher-order cognitive skills, emphasizing application, analysis, and evaluation, as opposed to lower-order skills such as knowledge and comprehension (Brinks-Lockwood, 2014 ; Lee and Wallace, 2018 ). Third, in contrast to traditional lecturing, the FC is a student-centered approach emphasizing engagement and active learning (Steen-Utheim and Foldnes, 2018 ), fostering students’ autonomy by endowing them with heightened responsibility for their learning (Brinks-Lockwood, 2014 ; O’Flaherty and Philips, 2015 ).

Vygotsky’s social constructivism ( 1978 ) has frequently been adopted as a theoretical foundation for designing learning experiences in technologically rich environments (Marzouki et al., 2017 ), and this framework highlights the particular benefits of technology-enhanced FC pedagogy (Jarvis et al., 2014 ). As mentioned above, in an FC model, learners can watch pre-recorded videos in their own time before class to remember basic information and understand concepts as they prepare for classroom activities, while the higher-order skills of analyzing, applying, evaluating, and creating can be collaborative and interactive, taking place in class with the guidance of a teacher, and thus facilitating progression within the learners’ proximal developmental zone.

Since its introduction in foreign language teaching (FLT) in China in 2011, the FC has attracted increasing research attention and has been welcomed by foreign language teachers (Yan and Zhou, 2021 ). Over the past decade, the Ministry of Education of the People’s Republic of China has exerted increasing pressure on higher education institutions to transition from traditional teacher-centered lecture-style approaches to innovative methods integrating technology and the internet, with the goals of enhancing learning, sustaining student engagement, and improving student satisfaction (Ministry of Education of People’s Republic China, 2021 ). The FC model, combined with traditional face-to-face teaching and personalized online learning, has emerged as a popular strategy in China to meet ministry requirements while delivering cost-effective and learner-centered curricula in response to the increasing student enrollment in higher education.

Despite the wide adoption of FCs in FLT in China, literature reviews about their implementation and effects have been notably scarce in the last decade. A search of the China National Knowledge Infrastructure (CNKI), the largest national research and information publishing company housing China’s most extensive academic database, revealed only three reviews—by Deng ( 2016 ), Qu ( 2019 ), and Su et al. ( 2019 )—published prior to the end of 2021. These reviews primarily focused on FCs in the context of English as a foreign language (EFL) education, overlooking most of the over 100 foreign languages taught in Chinese higher education. As a result, these reviews fell short of delivering a comprehensive analysis of research pertaining to FCs, and the reliability and generalizability of their findings in non-EFL contexts are questionable. Moreover, Deng ( 2016 ) and Su et al.’s (2019) reviews included all published papers without establishing clear inclusion and exclusion criteria. For example, they did not exclude articles that made a passing or token reference to the FC model, short papers of only one or two pages in length, book reviews, or editorials. Qu’s study ( 2019 ), on the other hand, was constrained in scope to articles within the Chinese Social Sciences Citation Index (CSSCI), a sub-database developed by Nanjing University of China Academy of Social Sciences Research Evaluation Center and the Hong Kong University of Science and Technology, and thus omitted relevant contributions from other academic journals. The CNKI incorporates both the CSSCI and the Core Journals of China (CJC), an equally significant sub-database overseen by the Peking University Library and experts from relevant institutions. Given the exclusion of the latter, a reevaluation of the scope and potential limitations of Qu’s study is warranted.

Thus, there persists an imperative for a comprehensive synthesis of the extant studies on FCs in FLT within Chinese higher education over the past decade. The restricted visibility of studies conducted in China, owing to their publication in Chinese and confinement to Chinese academic journals, makes it difficult for international researchers and practitioners to access and comprehend this body of literature. Such understanding among the global academic community is necessary for exploring both the strengths and limitations of FCs in diverse cultural and linguistic contexts.

Research method

The current study adopts a scoping review approach based on the methodological framework developed by Arksey and O’Malley ( 2005 ) to provide both quantitative and qualitative data for researchers and practitioners.

A scoping review is a relatively new approach to synthesizing research data which has been gaining popularity in many disciplines (Davis et al., 2009 ; Daudt et al., 2013 ). It is often undertaken as an independent project when a research area is complex, and no review of that area has previously been made available. A scoping review serves to highlight the relevant literature to researchers with the aim of rapidly mapping the key concepts characterizing a research area and the main sources and types of evidence available (Arksey and O’Malley, 2005 ; Mays et al., 2005 ; Levac et al., 2010 ). According to Arksey and O’Malley ( 2005 ), this kind of review addresses four goals: to examine the extent, range, and nature of research activity; to determine the value of undertaking a full systematic review; to summarize and disseminate research findings; and to identify research gaps in the existing literature. The scoping review is increasingly being employed in the field of foreign language education to provide a comprehensive view of FLT studies, identify implications for theory and pedagogy, or inform subsequent in-depth reviews and empirical studies (Chan et al., 2022 ; Hillman et al., 2020 ; Tullock and Ortega, 2017 ).

The difference between a scoping review and a narrative or traditional literature review lies in the transparency of the review process. A narrative review usually depends on the author’s own knowledge or experience to describe the studies reviewed and uses an implicit process to provide evidence (Garg et al., 2008 ). The reader cannot determine how much literature has been consulted or whether certain studies have been ignored due to contradictory findings. A scoping review, in contrast, uses an explicit, rigorous, and systematic approach to retrieve relevant articles to ensure the transparency and replicability of the data extraction process. For example, the methodological framework adopted by Arksey and O’Malley ( 2005 ) for conducting a scoping study comprises five stages: identifying the research questions; identifying relevant studies; selecting studies for inclusion; charting the data; and collating, summarizing, and reporting the results. By presenting the process and results in an accessible and summarized format, reviewers are in a position to illustrate the field of interest in terms of the volume, nature, and characteristics of the primary research, enabling researchers, practitioners, and policymakers to make effective use of the findings.

Figure 1 presents the process of the scoping review in the current study based on the five-stage methodological framework developed by Arksey and O’Malley ( 2005 ).

Process of the scoping review.

Process of the scoping review

Identifying research questions.

This scoping review is driven by four research questions:

RQ1. What is the current state of FC research in FLT within the context of higher education in China?

RQ2. What research methods and instruments have been employed in the included FC studies?

RQ3. What research foci and trends are displayed in the included FC studies?

RQ4. What are the major findings of the included FC studies?

RQ1 aims to provide an overview of studies on FCs in FLT in Chinese higher education by providing details about the basic information about existing publications, such as the number of publications per year and the distribution of publications by foreign language context. RQ2 leads to a classification of the research methods and instruments used to collect data in FC research. RQ3 explores the topics and trends in FC research over the past decade with the help of the literature visualization and analysis tool CiteSpace5.8R3. RQ4 reveals the effects of the FC model on direct and indirect educational outcomes, learners’ satisfaction with FCs, and the factors influencing the impact of FCs, as documented in the reviewed sources.

Searching for relevant studies

To be as comprehensive as possible in identifying primary evidence and to ensure the quality of the published articles, we searched both CSSCI and CJC in the CNKI database. The key search terms were developed and categorized based on two dimensions according to the purpose of the review. One dimension related to teaching or learning in FCs, while the other dimension related to the types of foreign languages. The key search terms and search methods are listed in Table 1 .

As the FC approach was introduced into FLT in China in 2011, the search included articles published between 2011 and 2021. Further inclusion and exclusion criteria were developed to focus on the scope of the review; these are outlined in Table 2 .

Study selection

Figure 2 shows a process diagram of the study selection process, which consisted of four phases: searching the databases; identifying the total number of articles in each database; screening titles, abstracts, and full texts; and selecting eligible articles for inclusion.

Flowchart diagram for article selection.

The final database search was conducted on January 16, 2022, and resulted in the identification of a total of 333 articles. Subsequently, all potentially relevant articles went through a three-step screening process. The first step excluded 9 duplicates. The second step excluded irrelevant articles by screening titles and abstracts; 37 articles were removed at this stage as they were book reviews, conference proceedings, reports, editorials, or other non-refereed publications. The third step filtered articles by screening full texts; 54 articles were excluded because they made only passing reference to the FC or were not related to higher education. This meticulous selection yielded a corpus of 233 articles suitable for in-depth analysis, each of which was scrutinized by the authors to confirm its suitability for inclusion. During the selection process, the 233 articles were also systematically categorized into two groups: 131 non-empirical and 102 empirical studies. The non-empirical studies were further divided into two subcategories. The first type was literature reviews; the second type was those drawing on personal observations, reflections on current events, or the authority or experience of the author (Dan, 2021 ). The empirical studies used a variety of systematic methods of collecting materials and analyzing data, including quantitative methods (e.g., survey, correlational research, experimental research) and/or qualitative methods (e.g., interview, case study, record keeping, observation, ethnographic research) (Dan, 2021 ).

Data charting and collation

The fourth stage of Arksey and O’Malley’s scoping review framework is the charting of the selected articles. Summaries of each study were developed. for all studies, these summaries included the author, year of publication, citations per year, foreign language taught, and a brief description of the outcomes. For empirical sources, details related to the research design, study population, and sample size were also provided. Tables 3 and 4 list the top ten most-cited non-empirical and empirical sources. In Table 4 , which references experimental and control groups in results summaries, the experimental group (EG) was the group that took courses in the FC model, while the control group (CG) took courses in a traditional classroom.

Results and analysis

In accordance with the fifth stage of Arksey and O’Malley’s framework for a scoping review, the findings from the 233 included studies are summarized and discussed in the following three sections. Section 4.1 summarizes basic information regarding the included studies; section 4.2 presents a holistic analysis of the research foci and trends over time using keyword clustering analysis and keyword burst analysis; and section 4.3 offers an in-depth content analysis focusing on the categorization of the included studies and discussion of the major findings.

Basic information on the included studies

Distribution by year of publication.

As Fig. 3 shows, the first studies on FCs in the field of FLT in China emerged in 2013. The number of such studies began to steadily increase and reached a peak in 2016 and 2017. Although there was some decrease after that, the FC model has continued to attract research attention, in line with global trends. According to Akçayir and Akçayir’s (2018) review of the literature on FCs published in Social Sciences Citation Index (SSCI) journals as of 31 December 2016, the first article about the FC was published in 2000, but the second was not published until more than a decade later, in 2012; 2013 was also the year that FC studies became popular among scholars. A possible explanation for this increase in interest is the growing availability of internet technologies and the popularity of online learning platforms, such as MOOCs and SPOCs (Small Private Online Courses), along with the view of the FC as a promising model that can open doors to new approaches in higher education in the new century.

Number of articles published by year.

Distribution by foreign language

Figure 4 shows the distribution of foreign languages discussed in the FC literature. The FC model was mainly implemented in EFL teaching (93%), which reflects the dominance of English in FLT in Chinese higher education. Only five articles discussed the use of FC models in Japanese teaching, while one article was related to French teaching. Ten non-empirical studies (4%) reported the feasibility of FC models in FLT without mentioning a specific foreign language.

Distribution by foreign language type.

Research methods of the included studies

Figure 5 shows a breakdown of the methodologies adopted by the studies included in our review. Among the 131 non-empirical studies, three were literature reviews, while the remaining 128 (55%) were descriptive studies based on the introduction of the FC model, including descriptions of its strengths and associated challenges and discussions of its design and implementation in FLT.

Methodological paradigms.

Of the 102 empirical studies, 60 (26%) used quantitative methods for data collection, eight (3%) used qualitative methods, and 34 (15%) used mixed methods. It is interesting to note that although quantitative methods are more common in FC studies, seven of the top ten most-cited empirical studies (as listed above in Table 4 ) used mixed methods. A potential reason may be that research findings collected with triangulation from various data sources or methods are seen as more reliable and valid and, hence, more accepted by scholars.

A breakdown of the data collection approaches used in the 102 reviewed empirical studies is displayed in Table 5 . It is important to note that most studies used more than one instrument, and therefore, it is possible for percentages to add up to more than 100%. The survey, as a convenient, cost-effective, and reliable research method, was the tool most frequently used to gain a comprehensive picture of the attitudes and characteristics of a large group of learners. Surveys were used in 79 of the 102 studies—73 times with learners and six times with teachers—to explore students’ learning experiences, attitudes, and emotions, as well as teachers’ opinions. Some studies used paper-based surveys, while others used online ones. Interviews with learners were used in 33 studies to provide in-depth information; one study used interviews with teachers. Surveys and interviews were combined in 24 studies to obtain both quantitative and qualitative data. Other research approaches included comparing the test scores between experimental and control groups (used in 25 studies) or using the results of course assessments (17 studies) to investigate the effects of the FC on academic performance. Learners’ self-reports (9 studies) were also used to capture the effects of the FC on learners’ experience and cognitive changes that could not be obtained in other ways, while one study used a case study for a similar purpose. Teachers’ class observations and reflections were used in eight studies to evaluate students’ engagement, interaction, activities, and learning performance.

Holistic analysis of the research foci and the changing trends of the included studies

A holistic analysis of the research foci in studies of FCs in China was conducted using CiteSpace5.8.R3, a software developed by Chaomei Chen ( http://cluster.cis.drexel.edu/~cchen/citespace/ , accessed on 20 February 2022), to conduct a visual analysis of the literature. This software can help conduct co-citation analysis, keyword co-occurrence analysis, keyword clustering analysis, keyword burst analysis, and social network analysis (Chen, 2016 ). In this study, keyword clustering analysis and keyword burst analysis were chosen to capture important themes and reveal changing trends in FC research.

Keyword clustering analysis primarily serves to identify core topics in a corpus. Figure 6 presents a graph of the top ten keyword clusters identified in the included studies. In this graph, the lower the ID number of a given cluster, the more keywords are in that cluster. As shown in the top left corner of Fig. 6 , the value of modularity q is 0.8122, which is greater than the critical value of 0.3, indicating that the clustering effect is good; the mean silhouette value is 0.9412, which is >0.5, indicating that the clustering results are significant and can accurately represent hot spots and topics in FC research (Hu and Song, 2021 ). The top ten keyword clusters include #0翻转课堂 (flipped classroom), #1大学英语 (college English), #2 MOOC, #3教学模式 (teaching model), #4元认知 (metacognition), #5微课 (micro lecture), #6微课设计 (micro lecture design), #7英语教学 (English teaching), #8 SPOC, and #9 POA (production-oriented approach).

The graph of the top ten keyword clusters.

Keyword burst analysis is used to showcase the changes in keyword frequencies over a given period of time. By analyzing the rise and decline of keywords, and in particular, the years in which some keywords suddenly become significantly more prevalent (“burst”), we can identify emerging trends in the evolution of FC research. Figure 7 displays the 11 keywords with the strongest citation bursts. We can roughly divide the evolution of FC research documented in Fig. 7 into two periods. The first period (2014 to 2017) focused on the introduction of the new model and the analysis of its feasibility in FLT. The keywords that underwent bursts in this period included “MOOC”, “自主学习” (independent learning), “模式” (model), “学习模式” (learning model), “教师话语” (teacher discourse), “茶文化” (tea culture), and “可行性” (feasibility). The reason for the appearance of the keyword “tea culture” lies in the fact that three articles discussing the use of FCs in teaching tea culture in an EFL environment were published in the same journal, entitled Tea in Fujian , during this period. The second period (2018–2021) focused on the investigation of the effect of FCs and the design of micro lectures. Keywords undergoing bursts during this period included “互联网+” (internet plus), “课堂环境” (classroom environment), “教学效果” (teaching effect), and “微课设计” (micro lecture design). The latter two topics (“teaching effect” and “micro lecture design”) may continue to be prevalent in the coming years.

Top 11 keywords with the strongest citation bursts.

In-depth content analysis of the included studies

Along with the findings from the keyword clustering analysis and keyword burst analysis, an open coding system was created to categorize the research topics and contents of the 233 articles for in-depth analysis. Non-empirical and empirical studies were classified further into detailed sub-categories based on research foci and findings. It is important to note that some studies reported more than one research focus. For such studies, more than one sub-category or more than one code was applied; therefore, it is possible for percentages to add up to more than 100%. The findings for each category are discussed in detail in the following sections.

Non-empirical studies

The 131 non-empirical studies can be roughly divided into two categories, as shown in Table 6 . The first category, literature reviews, has no sub-categories. The second, descriptive studies, includes discussions of how to use FCs in FLT; descriptions of the process of implementing the FC in FLT; and comparisons between FCs and traditional classes or comparisons of FCs in Chinese and American educational contexts.

The sub-categories of “introduction and discussion” and “introduction and description” in Table 6 comprise 91.6% of the non-empirical studies included in our review. The difference between them lies in that the former is based on the introduction of the FC literature, while the latter is based both on the introduction of the FC literature and exploration of researchers’ teaching experience; the latter might have become qualitative studies if researchers had gone further in providing systematic methods of collecting information or an analysis of the impact of FCs.

Empirical studies

The 102 empirical studies were divided into four categories based on the domain of their reported findings: the effect of FCs on learners; learners’ satisfaction with FCs; factors influencing FCs; or other research foci. Each group was further classified into more detailed sub-categories.

Effect of FCs on learners

Studies on the effect of FCs on learners were divided into two types, as presented in Table 7 : those concerned with the direct effect of FCs on learning performance and those exploring the indirect effect on learners’ perceptions. Eight codes were applied to categorize the direct effect of FCs on learning performance, which was usually evaluated through test scores; 14 codes were used to categorize the indirect effect of FCs on learners’ perceptions, which were usually investigated through surveys or questionnaires. We do not provide percentages for each code in Tables 7 – 9 because, given that the total number of empirical studies is 102, the percentages are almost identical to the frequencies.

The results shown in Table 7 reveal that 84 studies of direct educational outcomes reported that FCs had a positive effect on basic language skills, content knowledge, and foreign language proficiency. Of these, 64 were concerned with the positive effect of FCs on foreign language proficiency, speaking skills, or listening skills. This result might be explained by the features of FCs. The main difference between FCs and traditional classrooms is that the teaching of content in FCs has been removed from the classes themselves and is often delivered to the students through video recordings, which can be viewed repeatedly outside of the class. In-class time can thus be used for discussion, presentations, or the extension of the knowledge provided in the videos. It is evident that students have more opportunities to practice listening and speaking in FCs, and foreign language proficiency is naturally expected. Only three studies reported that FCs had no effect or a negative effect on the development of foreign language proficiency, speaking, listening, and writing skills. Yan and Zhou ( 2021 ) found that after the FC model had been in place for one semester, college students’ reading abilities improved significantly, while there was no significant improvement in their listening and writing abilities. Yin ( 2016 ) reported that after FC had been implemented for one semester, there was no significant difference in college students’ speaking scores.

A total of 96 studies reported positive effects on indirect educational outcomes, including: boosting learners’ motivation, interest, or confidence; enhancing engagement, interaction, cooperation, creativity, independent learning ability, or critical thinking ability; fostering information literacy, learning strategies, learning efficiency, or self-efficacy; or relieving stress or anxiety. The most frequently documented indirect effect of FCs is improvement in students’ independent learning ability. Only one study found that the FC did not significantly increase student interest in the course (Wang, 2015 ). Similarly, only one study found that students’ anxiety in the FC was significantly higher than that in a traditional class (Gao and Li, 2016 ).

Learners’ satisfaction with FCs

Table 8 presents the results regarding learners’ satisfaction with FCs. Nine codes were used to categorize the different aspects of learners’ satisfaction investigated in the 102 empirical studies. Some researchers represented learner satisfaction using the percentage of students choosing each answer on a five-point Likert scale from 1 (not at all satisfied) to 5 (very satisfied), while others used average scores based on Likert scale values. For the purposes of our synthesis of findings, if the percentage is above 60% or the average score is above 3, the finding is categorized as satisfied; otherwise, it is categorized as not satisfied.

The results in Table 8 show that among the nine aspects investigated, teaching approach and learning outcomes were most frequently asked about in the research, and learners were generally satisfied with both. Only one study (Li and Cao, 2015 ) reported significant dissatisfaction; in this case, 76.19% of students were not satisfied with the videos used in college English teaching due to their poor quality.

Factors influencing the effect of FCs

Eleven factors were found to influence the effect of FCs; these are categorized in Table 9 .

The results shown in Table 9 indicate that learners’ foreign language proficiency and self-regulation or self-discipline abilities are two important factors influencing the effect of FCs. Learners with high foreign language proficiency benefited more from FCs than those with low foreign language proficiency (Lv and Wang, 2016 ; Li and Cao, 2015 ; Wang and Zhang, 2014 ; Qu and Miu, 2016 ; Wang and Zhang, 2013 ; Cheng, 2016 ; Jia et al., 2016 ; Liu, 2016 ), and learners with good self-regulation and self-discipline abilities benefited more than those with limited abilities (Wang and Zhang, 2014 ; Lu, 2014 ; Lv and Wang, 2016 ; Dai and Chen 2016 ; Jia et al. 2016 ; Ling, 2018 ). It is interesting to note that two studies explored the relationship between gender and FCs (Wang and Zhang, 2014 ; Zhang and He, 2020 ), and both reported that girls benefited more from FCs because they were generally more self-disciplined than boys.

Studies with other research foci

There were six studies with other research foci, three of which investigated teachers’ attitudes toward FCs (Liao and Zou, 2019 ; Zhang and Xu, 2018 ; Zhang et al., 2015 ). The results of the surveys in these three studies showed that teachers generally held positive attitudes towards FCs and felt that the learning outcomes were better than those of traditional classes. However, some problems were also revealed in these studies. First, 56% of teachers expressed the desire to receive training before using FCs due to a lack of theoretical and practical expertise regarding this new model. Second, 87% of teachers thought that the FC increased their workload, as they were spending a significant amount of time learning to use new technology and preparing online videos or materials, yet no policy was implemented in the schools to encourage them to undertake this work. Third, 72% of teachers felt that the FC increased the academic burden students faced in their spare time (Zhang and Xu, 2018 ; Zhang et al., 2015 ). The final three studies include Cheng’s ( 2016 ) investigation of the mediative functions of college EFL teachers in the FC, Wang and Ma’s ( 2017 ) construction of a model for assessing the teaching quality of classes using the FC model, and Luo’s ( 2018 ) evaluation of the learning environment of an FC-model college English MOOC.

Discussion and conclusions

This investigation employed literature visualization to systematically analyze 233 research papers sourced from CSSCI and CJC in the CNKI database, thereby conducting a scoping review delineating the landscape of FC research within the domain of FLT in the context of higher education in China.

Our findings in relation to RQ1 highlight a substantial surge in the number of articles relating to FCs in FLT between 2013 and 2017, followed by a discernible, albeit moderate, decrease. Despite this trend, FC studies continue to be of significant interest to foreign language educators and researchers. This may be attributed to Chinese government policies encouraging higher education reform, increased internet access among educators and learners, and the burgeoning popularity of online courses such as MOOCs and SPOCs. However, the majority of the reviewed FC studies were conducted in college English classes, with only 6 studies on classes teaching foreign languages other than English. It seems that foreign language education in China (and in much of the world) has become synonymous with the teaching and learning of English, with other languages occupying a marginal position, struggling to find space in educational programs. In a multilingual world in which each language offers different possibilities for understanding others, their cultures, their epistemologies, and their experiences, this monolingual approach to FLT is dangerous (Liddicoat, 2022 ). The promotion of linguistic diversity in foreign language education policies and research is thus imperative. Another gap that needs to be addressed is the paucity of studies on the implementation of FCs in adult education. The FC model is expected to be potentially effective for teaching adult learners because it is similar in some respects to online distance learning.

In answer to RQ2, we found that the commonly used research methods and instruments in studies of the FC model include surveys, interviews, comparisons of academic measures between EGs and CGs, and course assessments. The case study is the least used method, likely due to limitations such as time demand, researcher bias, and the fact that it provides little basis for the generalization of results to the wider population. However, more case studies are needed in future research on FCs because they can provide detailed and insightful qualitative information that cannot be gathered in other ways.

Our findings regarding RQ3 show that research foci within the FC domain have evolved over time from initial exploration and feasibility discussions to a subsequent focus on the design of FCs incorporating micro-lectures based on MOOC or SPOC structures, and then to the present focus on the examination of FCs’ impacts on learners. The results of the keyword burst analysis indicate that these thematic areas are likely to persist as prominent subjects of research interest for the foreseeable future.

In response to RQ4, our in-depth content analysis found that FCs, on the whole, yield positive outcomes, although isolated studies identify limited negative impacts. FCs are most frequently associated with enhancements in student learning performance, fostering independent learning, promoting engagement and cooperation, and mitigating stress or anxiety. The results of this study suggest that well-designed FCs present a significant opportunity for foreign language educators to revolutionize instructional approaches. Furthermore, well-structured FCs can facilitate the development of learners’ potential while concurrently enabling the seamless integration of digital technology into FLT.

Most learners are satisfied with FCs, particularly with the innovative pedagogical approach of reversing traditional classes. FCs are perceived as beneficial for improving learning outcomes, creating an environment conducive to peer interaction, and gaining immediate teacher feedback and support. In addition, students’ interest in classes is enhanced by the rich and diverse online learning materials uploaded by teachers, which can be accessed conveniently at any time in any place. Furthermore, the dynamic and formative online assessment approach is also welcomed by students because it provides immediate feedback and the ability to discuss any problems they have with teachers or peers online or offline.

However, it is worth noting that most of the reviewed studies on FCs focused on one course, usually over only one semester. Students’ increase in motivation or improvements in learning outcomes might, therefore, be a result of the Hawthorne effect. Compared with the traditional didactic lecture format, the novelty of FCs, when used for the first time, might generate excitement among students, thus increasing their attention and enhancing learning outcomes, but such benefits will diminish over time. Therefore, there is a need to examine whether this model is suitable for large-scale implementation and whether its effects might be sustained over longer periods of implementation.

Learners’ foreign language proficiency and self-regulation or self-discipline abilities are the two key factors influencing the effect of FCs. These two factors are closely related; self-regulation or self-discipline is a prerequisite for successful foreign language learning in FC contexts and plays a crucial role in students’ success in the pre-class sessions for which they are personally responsible. In addition, factors such as learners’ attitudes, expectations of and adaptability to the FC model, the learning tasks and learning environment, the teaching organization and assessment methods, and the learner’s gender also have some impact on the effect of FCs. However, due to the limited number of studies, there is not sufficient evidence to warrant the generalization of any of these effects.

This scoping review highlights some potential challenges that need to be addressed for the effective implementation of FCs.

First, despite the benefits of the FC model, FCs are not equally advantageous to all students due to the self-regulated nature of the model. Many learners have reported difficulties in completing their individual online tasks outside the classroom (Yoon et al., 2021 ). The non-traditional configuration of FCs poses a formidable challenge, particularly for students less inclined to engage in pre-class online activities characterized by a lack of interactivity and for those who are less self-disciplined. Consequentially, students may attend class without having assimilated the pre-assigned material, thereby diminishing the efficacy of this instructional approach. To address this issue, additional support or prompts for students should be provided to remind them of the need to self-regulate their learning. For example, Park and Jo ( 2015 ) employed a learning analytics dashboard displaying visual representations of students’ learning patterns derived from login traces, such as login frequency and interval regularity, within the course’s learning management system. These visual indicators allowed students to monitor their learning engagement and performance in comparison to those of their peers.

Second, a persistent problem with FCs is the inability of students to interact with their peers or receive prompt feedback from instructors after completing independent online learning activities. While some researchers identified a need for teachers to provide immediate online feedback or opportunities for peer discussion, our review of the literature shows that scant attention has been given to this issue. Researchers note that under-stimulation, low perceived control over tasks, and delayed or insufficient feedback in online learning contribute significantly to learner boredom or absenteeism (Yazdanmehr et al., 2021 ; Tao and Gao, 2022 ). Online pedagogical innovations are needed to solve these new problems. For instance, the establishment of online groups employing chat software like QQ or WeChat could facilitate instantaneous feedback or peer interaction through text-based communication, thereby enhancing learners’ satisfaction with FC courses.

Third, despite recognizing the value of FCs in enhancing the learning experience for students, teachers often lack the requisite training to implement FCs effectively. Insights derived from interviews with teachers, as noted in several of the reviewed studies, reveal a pronounced desire for increased opportunities to learn about the underlying theories of FCs and acquire the skills necessary for the translation of FC concepts into pedagogical practice. Specifically, teachers express a need for guidance in creating engaging instructional videos, determining optimal video length to sustain learner interest, and ascertaining the ideal duration for online quizzes to foster optimal learner performance. Further research is required on strategies and technologies that can help teachers produce high-quality videos despite limited time and technical skills. Support from professional communities, institutions, and technology specialists is thus essential for the provision of effective hybrid offline and online instruction.

Fourth, additional research is required to determine whether workloads for students and teachers are increased by the use of FCs. If this is the case, as found in some of the reviewed studies, then the compelling benefits of FCs would be offset by the extra time needed, making it difficult to draw the conclusion that FCs are more efficient than traditional classes. The majority of language teachers, due to limited skills in technology, online environment management, and online interaction, feel too physically and emotionally overworked to expend more time and energy on enhancing teaching effectiveness. With few teachers having excess spare time, the thought of designing and creating new content might discourage even the most enthusiastic teachers.

Finally, robust empirical evidence is needed to evaluate whether FCs can facilitate students’ higher-order thinking through the use of creative technologies and assessment approaches. Constructs such as creativity and critical thinking are not always easily reduced to measurable items on survey instruments or scores on examinations (Haladyna et al., 2002 ).

In conclusion, the insights garnered from this study have the potential to enrich the global discourse on the benefits and limitations of FCs in diverse cultural and linguistic contexts. Our review included literature accessible through CSSCI and CJC in the CNKI database, and while this provides a thorough selection of the Chinese literature on the subject, our search approach may have excluded valuable FC-related papers published in other languages and countries. Consequently, different search criteria might yield different selection and data results. Future researchers are encouraged to undertake more comprehensive literature reviews encompassing broader databases to fill the gaps in our work and to augment the depth and breadth of knowledge in this domain.

Data availability

The raw data for this paper were collected from articles in Chinese Social Sciences Citation Index (CSSCI) journals and A Guide to the Core Journals of China of Peking University (PKU journals) in the database of China National Knowledge Infrastructure (CNKI) ( https://www.cnki.net/ ). The raw data used to support the findings of this study are available from the corresponding author upon request.

Akcayir G, Akcayir M (2018) The flipped classroom: a review of its advantages and challenges. Comput Educ 126:334–345. https://doi.org/10.1016/j.compedu.2018.07.021

Article   Google Scholar  

Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Methodol 8:19–32. https://doi.org/10.1080/1364557032000119616

Baker JW (2000) The ‘Classroom Flip’: using web course management tools to become the guide by the side. In: Chambers JA ed Selected papers from the 11th international conference on college teaching and learning. Florida Community College at Jacksonville, Jacksonville, FL

Google Scholar  

Bergmann J, Sams A (2012) Flip your classroom: reach every student in every class every day. International Society for Technology in Education, Eugene, OR

Brinks-Lockwood R (2014) Flip it! Strategies for the ESL classroom. University of Michigan Press, Ann Arbor, MI

Book   Google Scholar  

Cao P (2020) Construction of blended learning and evaluation of its effect based on a flipped class approach in an ESP course. Foreign Lang World 6:87–94

Chan L, Woore R, Molway L, Mutton T (2022) Learning and teaching Chinese as a foreign language: a scoping review. Rev Educ 10:1–35. https://doi.org/10.1002/rev3.3370

Chen C (2016) CiteSpace: a practical guide for mapping scientific literature. Nova Science Publishers, Hauppauge, NY, USA

Cheng Y (2014) Facing the challenge and flipping ourselves—opportunities and challenges faced by foreign language teachers under the new paradigm of education. Technol Enhanc Foreign Lang 157:44–47

Cheng X (2016) An empirical study of implementations of mediative functions of college EFL teachers in flipped classrooms. Technol Enhanc Foreign Lang 168:48–53

Cui Y, Wang Y (2014) Flipped class model and its application to college English teaching. China Educ Technol 334:116–121

Dan V (2021) Empirical and nonempirical methods. https://www.ls1.ifkw.uni-muenchen.de/personen/wiss_ma/dan_viorela/empirical_and_non_empirical.pdf . Accessed 23 Aug 2021

Daudt HM, van Mossel C, Scott SJ (2013) Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Med Res Methodol 13:48, http://www.biomedcentral.com/1471-2288/13/48

Article   PubMed   PubMed Central   Google Scholar  

Dai C, Chen J (2016) An analysis of contributing factors of college English flipped classroom with MOOC philosophy. Technol Enhanc Foreign Lang 172:35–41

Davis K, Drey N, Gould D (2009) What are scoping studies? A review of the nursing literature. Int J Nurs Stud 46:1386–1400. https://doi.org/10.1016/j.ijnurstu.2009.02.010

Article   PubMed   Google Scholar  

Deng D (2016) A review of the research on the application of the flipped classroom model in college English teaching. Foreign Lang World 175:89–96

Dou J, Wen S (2015) The teaching reform exploration of college English flipped classroom based on APP. Heilongjiang Res High Educ 5:162–167

Gao Z, Li J (2016) English classroom anxiety of Chinese learners: flipped vs. traditional. Technol Enhanc Foreign Lang 167:37–42

Garg AX, Hackam D, Tonelli M (2008) Systematic review and meta-analysis: When one study is just not enough. Clin J Am Soc Nephrol 3:253–260. https://doi.org/10.2215/CJN.01430307

Haladyna T, Downing S, Rodriguez M (2002) A review of multiple-choice item writing guidelines for classroom assessment. Appl Meas Educ 15:309–334. https://doi.org/10.1207/S15324818AME15035

Hillman S, Selvi AF, Yazan B (2020) A scoping review of world Englishes in the Middle East and North Africa. World Englishes 40:1–17. https://doi.org/10.1111/weng.12505

Hu J, Wu Z (2014) An empirical study on the MOOC-based college English flipped classroom instructional model. Technol Enhanc Foreign Lang 160:40–45

Hu Z, Song J (2021) Research hotspots and trends of online and offline hybrid teaching in China—bibliometric analysis of journal literature based on CNKI (2015–2020). In: 2021 2nd international conference on education, knowledge and information management. IEEE. https://doi.org/10.1109/ICEKIM52309.2021.00032

Hung HT (2014) Flipping the classroom for English language learners to foster active learning. Comput Assist Lang Learn 28:81–96. https://doi.org/10.1080/09588221.2014.967701

Jarvis W, Halvorson W, Sadeque S, Johnston S (2014) A large class engagement(LCE) model based on service-dominant logic (SDL) and flipped classrooms. Educ Res Perspect 41:1–24. http://www.erpjournal.net/

Jia L, Zhang G, Shi C (2016) Flipping medical English reading class based on Wechat public platform and Wechat community. Technol Enhanc Foreign Lang 168:65–69

Jiang Q, Tao Y (2018) The application of “flipped classroom” in teaching MTI translation theory and its effect analysis. Foreign Lang Educ 5:70–74

ADS   Google Scholar  

Jiang Y, Hu J (2018) A study on the large-scale instruction mechanism of SPOC-based college English flipped classroom. Technol Enhanc Foreign Lang 182:9–15

Khan S (2012) The one world schoolhouse: education reimagined. Hodder and Stoughton, London

Lee G, Wallace A (2018) Flipped learning in the English as a foreign language classroom: outcomes and perceptions. TESOL Q 52:62–84

Levac D, Colquhoun H, O’Brien K (2010) Scoping studies: advancing the methodology. Implement. Sci. 5(69):1–9

Li X, Cao H (2015) Research on flipped classroom based on micro-lecture—a case study in college English teaching through videos. Mod Educ Technol 9:70–76. https://doi.org/10.3969/j.issn.1009-8097.2015.09.011

Li X, Wang B (2017) An empirical study on college English flipped classroom model based on context awareness. Technol Enhanc Foreign Lang 178:71–77

Li J, Wu Z (2015) Practice and reflections on flipped college English class. Foreign Lang China 6:4–9. https://doi.org/10.13564/j.cnki.issn.1672-9382.2015.06.002

Liao G, Zou X (2019) Optimizing the teaching design to develop learners’ autonomous study. Educ Res Mon 10:105–111

Liddicoat AJ (2022) Language planning for diversity in foreign language education. Curr Issues Lang Plan 23(5):457–465. https://doi.org/10.1080/14664208.2022.2088968

Ling R (2018) Research on Japanese pronunciation teaching with the flipped classroom model. Jpn Learn Res 4:68–76. https://doi.org/10.13508/j.cnki.jsr.2018.04.010

Article   ADS   Google Scholar  

Liu Y (2016) Research on the construction of flipped classroom model for vocabulary instruction and its effectiveness. Technol Enhanc Foreign Lang 167:43–49

Liu Z, Wu Q (2015) Cultivation of college students’ learning autonomy from the perspective of “Flipped Classroom”. Mod Educ Technol 11:67–72. https://doi.org/10.3969/j.issn.1009-8097.2015.11.010

Lu H (2014) Feasibility analysis on the application of micro-class based “Flipped Classroom” mode in college English teaching. Technol Enhanc Foreign Lang 158:33–36

Luo S (2018) Evaluating the learning environment of MOOC-based college English flipped classrooms. Technol Enhanc Foreign Lang 182:16–22

Lv T, Wang N (2016) A study on the effect of the flipped classroom based on SPOC+teaching resource platform. China Educ Technol 352:85–90

Mays N, Pope C, Popay J (2005) Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy 1:6–20. https://doi.org/10.1258/1355819054308576

Marzouki OF, Idrissi MK, Bennani S (2017) Effects of social constructivist mobile learning environments on knowledge acquisition: a meta-analysis. Int J Interact Mob Technol 11(1):18–39. https://doi.org/10.3991/ijim.v11i1.5982

Ministry of Education of People’s Republic China (2021) 10-year development plan for education information (2011–2020). http://www.moe.gov.cn/srcsite/A16/s3342/201203/t20120313_133322.html . Accessed 30 Dec 2021

O’ Flaherty J, Philips C (2015) The use of flipped classroom in higher education: a scoping review. Internet High Educ 25:85–95. https://doi.org/10.1016/j.iheduc.2015.02.002

Park Y, Jo I-H (2015) Development of the learning analytics dashboard to support students’ learning performance. J Univers Comput Sci 21(1):110–133. https://doi.org/10.3217/jucs-021-01-0110

Qiao H (2017) Study of blended English learning model based on community of inquiry. Technol Enhanc Foreign Lang 176:43–48

Qu S (2019) A content analysis of researches on EFL flipped classrooms in China’s universities. Technol Enhanc Foreign Lang 187:62–68

Qu Q, Miu R (2016) A research on learning strategies of flipped classroom teaching model. China Educ Technol 350:114–119

Shen Y, Sheng Y (2015) Construction of the flipped college English classroom based on Community of Inquiry. Foreign Lang. World 4:81–89

Steen-Utheim AT, Foldnes NA (2018) Qualitative investigation of student engagement in a flipped classroom. Teach High Educ 23:30–324. https://doi.org/10.1080/13562517.2017.1379481

Su X, Liu S, Ma W (2019) The use of flipped classrooms in Chinese college English teaching: a review. Foreign Lang Lit 35:142–148. 1674-6414(2019) 01-0142-07

Tao J, Gao X (2022) Teaching and learning languages online: challenges and responses. System 107:1–9. https://doi.org/10.1016/j.system.2022.102819

Tullock B, Ortega L (2017) Fluency and multilingualism in study abroad: Lessons from a scoping review. System 71:7–21. https://doi.org/10.1016/j.system.2017.09.019

Vygotsky LS (1978) Mind in society: the development of higher psychological processes. Harvard University Press, Cambridge

Wan M(2016) An empirical study on the application of flipped classroom in college English teaching High Educ Explor 5:69–72. www.cnki.net

Wang H (2015) An action research on interpreting teaching based on the flipped classroom. Chin Transl J 1:59–62

Wang X (2014) College foreign language teaching based on MOOC. Heilongjiang Res High Educ 8:157–159

Wang Y (2016) Research on English learning anxiety of non-English majors under flipped class model. J Guangxi Norm Univ: Philos Soc Sci Ed 4:134–139. https://doi.org/10.16088/j.issn.1001-6597.2016.04.020

Wang N, Chen J, Zhang D (2016) SPOC-based flipped classroom of college English: Construction of an efficient learning model. Technol Enhanc Foreign Lang 169:52–57

Wang G, Ma S (2017) A study on college English classroom “Flip Degree” assessment index. Technol Enhanc Foreign Lang 177:23–28

Wang H, Zhang L (2014) The application of flipped classroom in English teaching. Teach Manag 7:141–144. www.cnki.net

Wang S, Zhang L (2014) A study of college EFL learners’ acceptance towards flipped classroom. Mod Educ Technol 3:71–78. https://doi.org/10.3969/j.issn.1009-8097.2014.03.010

Article   CAS   Google Scholar  

Wang X, Zhang C (2013) The application research of flipped classroom in university teaching. Mod Educ Technol 8:11–16. https://doi.org/10.3969/j.issn.1009-8097.2013.08.002

Wang L, Zhao M, Yang W (2018) An empirical study of the instructional model of college oral English flipped classroom based on principles of CDIO engineering education. Technol Enhanc Foreign Lang 180:72–77

Wang Z, Wu M (2017) The analysis of the impact factors on students’ learning behaviors under flipped classroom model. Technol Enhanc Foreign Lang 177:29–34

Wu L (2015) A study on the flipped classroom model in teaching listening and speaking for academic English for general purposes. E-Educ Res 11:81–87. https://doi.org/10.13811/j.cnki.eer.2015.11.013

Xie P (2020) Applying MOOC-based blended learning to the courses of English education. Foreign Lang Educ China 2:43–49

Xu H (2017) An explorative study of college students’ critical thinking under flipped classroom. Technol Enhanc Foreign Lang 173:29–33

Xu T, Li X (2014) Exploration to project-based flipped classroom—new concept, method and technology in English teaching: a case study based on new college English. Foreign Lang China 15:81–87. https://doi.org/10.13564/j.cnki.issn.1672-9382.2014.05.011

Yan J, Zhang W, Yu Y (2016) School-based flipped classroom teaching reform on video-aural-oral course of college English. Mod Educ Technol 2:94–99. https://doi.org/10.3969/j.issn.1009-8097.2016.02.014

Yan Z, Zhou P(2021) Effect of flipped teaching on college students’ EFL acquisition. Foreign Language Learn Theory Pract 2:86–96

Yazdanmehr E, Shirvan ME, Saghafi K (2021) A process tracing study of the dynamic patterns of boredom in an online L3 course of German during COVID-19 pandemic. Foreign Lang Ann 54(3):714–739. https://doi.org/10.1111/flan.12548

Yin H (2016) An empirical study and critical reflection on the flipped classroom model. J Res Educ Ethn Minor 1:25–30

Yoon M, Hill J, Kim D (2021) Designing supports for promoting self-regulated learning in the flipped classroom. J Comput High Educ 33:398–418. https://doi.org/10.1007/s12528-021-09269-z

Yu P (2014) The application of flipped classroom in English teaching. Teach Manag 9:62–64. www.cnki.net

Yu Z, Chen W (2016) The Influence of clicker-aided college English flipped classroom on meta-cognition, cognitive loads and learning achievements. Technol Enhanc Foreign Lang 170:32–37

Zhai X, Lin L (2014) Factors analysis of Chinese learners’ satisfaction in western flipped classroom model (FCM) teaching—an empirical study based on college English. China Educ Technol 327:104–109

Zhang D (2021) Construction and application of a blended golden course framework of college English. Technol Enhanc Foreign Lang 197:71–77

Zhang M, Deng L(2021) Construction and application of “emergent flipped learning” teaching model in the integration of Chinese culture into college English teaching Foreign Language Learn Theory Pract 1:61–70

Zhang M, He X (2020) A study on the adaptability of blended college English learning. Technol Enhanc Foreign Lang. 4:89–94

Zhang M, Sun X (2015) MOOC-based flipped classroom teaching model for follow-up college English courses. Mod Educ Technol 8:81–87. https://doi.org/10.3969/j.issn.1009-8097.2015.08.01

Zhang J, Li K, Du X (2015) College English classroom: based on the survey of the current situation. Mod Educ Technol 7:68–74. https://doi.org/10.3969/j.issn.1009-8097.2015.07.011

Zhang Y, Xu Z (2018) Research on the acceptance of college English teachers’ flipped classroom. Theory Pract Educ 36:53–55

ADS   CAS   Google Scholar  

Zhou P (2015) Flipped classroom based on modern educational technology and its theoretical foundations. Technol Enhanc Foreign Lang 162:72–76

Zhu L, Xu Y, Han J (2021) An approach for the integration of foreign language teaching and information technology. Foreign Lang World 2:46–62

Download references

Acknowledgements

This research was funded by The 14th Five-year Plan for Education Science of Jiangsu Province (Grant number: D/2021/01/79), Changzhou University (Grant number: GJY2021013), and Department of Education of Zhejiang Province, China (Project of Ideological and Political Construction of Courses 2021-337).

Author information

Authors and affiliations.

Department of English, Changzhou University, 213100, Changzhou, Jiangsu Province, China

School of English Education, Guangdong University of Foreign Studies, 510006, Guangzhou, Guangdong Province, China

College of Science & Technology, Ningbo University, 315200, Ningbo, Zhejiang Province, China

Quanjiang Guo

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization, WK and Q-JG; methodology, WK; software, DL; validation, WK, DL, and Q-JG; formal analysis, WK and Q-JG; investigation, WK and Q-JG; resources, DL; data curation, DL; writing—original draft preparation, WK and Q-JG; writing—review and editing, WK and Q-JG; visualization, WK and Q-JG; supervision, WK; project administration, WK; funding acquisition, WK and Q-JG. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Quanjiang Guo .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

Informed consent was not required as the study did not involve human participants.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kong, W., Li, D. & Guo, Q. Research on flipped classrooms in foreign language teaching in Chinese higher education. Humanit Soc Sci Commun 11 , 525 (2024). https://doi.org/10.1057/s41599-024-03019-z

Download citation

Received : 17 October 2023

Accepted : 02 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1057/s41599-024-03019-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Open access
• Published: 19 April 2024

Person-centered care assessment tool with a focus on quality healthcare: a systematic review of psychometric properties

• Lluna Maria Bru-Luna 1 ,
• Manuel Martí-Vilar 2 ,
• César Merino-Soto 3 ,
• José Livia-Segovia 4 ,
• Juan Garduño-Espinosa 5 &
• Filiberto Toledano-Toledano 5 , 6 , 7  

BMC Psychology volume  12 , Article number:  217 ( 2024 ) Cite this article

268 Accesses

Metrics details

The person-centered care (PCC) approach plays a fundamental role in ensuring quality healthcare. The Person-Centered Care Assessment Tool (P-CAT) is one of the shortest and simplest tools currently available for measuring PCC. The objective of this study was to conduct a systematic review of the evidence in validation studies of the P-CAT, taking the “Standards” as a frame of reference.

First, a systematic literature review was conducted following the PRISMA method. Second, a systematic descriptive literature review of validity tests was conducted following the “Standards” framework. The search strategy and information sources were obtained from the Cochrane, Web of Science (WoS), Scopus and PubMed databases. With regard to the eligibility criteria and selection process, a protocol was registered in PROSPERO (CRD42022335866), and articles had to meet criteria for inclusion in the systematic review.

A total of seven articles were included. Empirical evidence indicates that these validations offer a high number of sources related to test content, internal structure for dimensionality and internal consistency. A moderate number of sources pertain to internal structure in terms of test-retest reliability and the relationship with other variables. There is little evidence of response processes, internal structure in measurement invariance terms, and test consequences.

The various validations of the P-CAT are not framed in a structured, valid, theory-based procedural framework like the “Standards” are. This can affect clinical practice because people’s health may depend on it. The findings of this study show that validation studies continue to focus on the types of validity traditionally studied and overlook interpretation of the scores in terms of their intended use.

Peer Review reports

Person-centered care (PCC)

Quality care for people with chronic diseases, functional limitations, or both has become one of the main objectives of medical and care services. The person-centered care (PCC) approach is an essential element not only in achieving this goal but also in providing high-quality health maintenance and medical care [ 1 , 2 , 3 ]. In addition to guaranteeing human rights, PCC provides numerous benefits to both the recipient and the provider [ 4 , 5 ]. Additionally, PCC includes a set of necessary competencies for healthcare professionals to address ongoing challenges in this area [ 6 ]. PCC includes the following elements [ 7 ]: an individualized, goal-oriented care plan based on individuals’ preferences; an ongoing review of the plan and the individual’s goals; support from an interprofessional team; active coordination among all medical and care providers and support services; ongoing information exchange, education and training for providers; and quality improvement through feedback from the individual and caregivers.

There is currently a growing body of literature on the application of PCC. A good example of this is McCormack’s widely known mid-range theory [ 8 ], an internationally recognized theoretical framework for PCC and how it is operationalized in practice. This framework forms a guide for care practitioners and researchers in hospital settings. This framework is elaborated in PCC and conceived of as “an approach to practice that is established through the formation and fostering of therapeutic relationships between all care providers, service users, and others significant to them, underpinned by values of respect for persons, [the] individual right to self-determination, mutual respect, and understanding” [ 9 ].

Thus, as established by PCC, it is important to emphasize that reference to the person who is the focus of care refers not only to the recipient but also to everyone involved in a care interaction [ 10 , 11 ]. PCC ensures that professionals are trained in relevant skills and methodology since, as discussed above, carers are among the agents who have the greatest impact on the quality of life of the person in need of care [ 12 , 13 , 14 ]. Furthermore, due to the high burden of caregiving, it is essential to account for caregivers’ well-being. In this regard, studies on professional caregivers are beginning to suggest that the provision of PCC can produce multiple benefits for both the care recipient and the caregiver [ 15 ].

Despite a considerable body of literature and the frequent inclusion of the term in health policy and research [ 16 ], PCC involves several complications. There is no standard consensus on the definition of this concept [ 17 ], which includes problematic areas such as efficacy assessment [ 18 , 19 ]. In addition, the difficulty of measuring the subjectivity involved in identifying the dimensions of the CPC and the infrequent use of standardized measures are acute issues [ 20 ]. These limitations and purposes motivated the creation of the Person-Centered Care Assessment Tool (P-CAT; [ 21 ]), which emerged from the need for a brief, economical, easily applied, versatile and comprehensive assessment instrument to provide valid and reliable measures of PCC for research purposes [ 21 ].

Person-centered care assessment tool (P-CAT)

There are several instruments that can measure PCC from different perspectives (i.e., the caregiver or the care recipient) and in different contexts (e.g., hospitals and nursing homes). However, from a practical point of view, the P-CAT is one of the shortest and simplest tools and contains all the essential elements of PCC described in the literature. It was developed in Australia to measure the approach of long-term residential settings to older people with dementia, although it is increasingly used in other healthcare settings, such as oncology units [ 22 ] and psychiatric hospitals [ 23 ].

Due to the brevity and simplicity of its application, the versatility of its use in different medical and care contexts, and its potential emic characteristics (i.e., constructs that can be cross-culturally applicable with reasonable and similar structure and interpretation; [ 24 ]), the P-CAT is one of the most widely used tests by professionals to measure PCC [ 25 , 26 ]. It has expanded to several countries with cultural and linguistic differences. Since its creation, it has been adapted in countries separated by wide cultural and linguistic differences, such as Norway [ 27 ], Sweden [ 28 ], China [ 29 ], South Korea [ 30 ], Spain [ 25 ], and Italy [ 31 ].

The P-CAT comprises 13 items rated on a 5-point ordinal scale (from “strongly disagree” to “strongly agree”), with high scores indicating a high degree of person-centeredness. The scale consists of three dimensions: person-centered care (7 items), organizational support (4 items) and environmental accessibility (2 items). In the original study ( n  = 220; [ 21 ]), the internal consistency of the instrument yielded satisfactory values for the total scale ( α  = 0.84) and good test-retest reliability ( r  =.66) at one-week intervals. A reliability generalization study conducted in 2021 [ 32 ] that estimated the internal consistency of the P-CAT and analyzed possible factors that could affect the it revealed that the mean α value for the 25 meta-analysis samples (some of which were part of the validations included in this study) was 0.81, and the only variable that had a statistically significant relationship with the reliability coefficient was the mean age of the sample. With respect to internal structure validity, three factors (56% of the total variance) were obtained, and content validity was assessed by experts, literature reviews and stakeholders [ 33 ].

Although not explicitly stated, the apparent commonality between validation studies of different versions of the P-CAT may be influenced by an influential decades-old validity framework that differentiates three categories: content validity, construct validity, and criterion validity [ 34 , 35 ]. However, a reformulation of the validity of the P-CAT within a modern framework, which would provide a different definition of validity, has not been performed.

Scale validity

Traditionally, validation is a process focused on the psychometric properties of a measurement instrument [ 36 ]. In the early 20th century, with the frequent use of standardized measurement tests in education and psychology, two definitions emerged: the first defined validity as the degree to which a test measures what it intends to measure, while the second described the validity of an instrument in terms of the correlation it presents with a variable [ 35 ].

However, in the past century, validity theory has evolved, leading to the understanding that validity should be based on specific interpretations for an intended purpose. It should not be limited to empirically obtained psychometric properties but should also be supported by the theory underlying the construct measured. Thus, to speak of classical or modern validity theory suggests an evolution in the classical or modern understanding of the concept of validity. Therefore, a classical approach (called classical test theory, CTT) is specifically differentiated from a modern approach. In general, recent concepts associated with a modern view of validity are based on (a) a unitary conception of validity and (b) validity judgments based on inferences and interpretations of the scores of a measure [ 37 , 38 ]. This conceptual advance in the concept of validity led to the creation of a guiding framework to for obtaining evidence to support the use and interpretation of the scores obtained by a measure [ 39 ].

This purpose is addressed by the Standards for Educational and Psychological Testing (“Standards”), a guide created by the American Educational Research Association (AERA), the American Psychological Association (APA) and the National Council on Measurement in Education (NCME) in 2014 with the aim of providing guidelines to assess the validity of the interpretations of scores of an instrument based on their intended use. Two conceptual aspects stand out in this modern view of validity: first, validity is a unitary concept centered on the construct; second, validity is defined as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” [ 37 ]. Thus, the “Standards” propose several sources that serve as a reference for assessing different aspects of validity. The five sources of valid evidence are as follows [ 37 ]: test content, response processes, internal structure, relations to other variables and consequences of testing. According to AERA et al. [ 37 ], test content validity refers to the relationship of the administration process, subject matter, wording and format of test items to the construct they are intended to measure. It is measured predominantly with qualitative methods but without excluding quantitative approaches. The validity of the responses is based on analysis of the cognitive processes and interpretation of the items by respondents and is measured with qualitative methods. Internal structure validity is based on the interrelationship between the items and the construct and is measured by quantitative methods. Validity in terms of the relationship with other variables is based on comparison between the variable that the instrument intends to measure and other theoretically relevant external variables and is measured by quantitative methods. Finally, validity based on the results of the test analyses consequences, both intended and unintended, that may be due to a source of invalidity. It is measured mainly by qualitative methods.

Thus, although validity plays a fundamental role in providing a strong scientific basis for interpretations of test scores, validation studies in the health field have traditionally focused on content validity, criterion validity and construct validity and have overlooked the interpretation and use of scores [ 34 ].

“Standards” are considered a suitable validity theory-based procedural framework for reviewing the validity of questionnaires due to its ability to analyze sources of validity from both qualitative and quantitative approaches and its evidence-based method [ 35 ]. Nevertheless, due to a lack of knowledge or the lack of a systematic description protocol, very few instruments to date have been reviewed within the framework of the “Standards” [ 39 ].

Current study

Although the P-CAT is one of the most widely used instruments by professionals and has seven validations [ 25 , 27 , 28 , 29 , 30 , 31 , 40 ], no analysis has been conducted of its validity within the framework of the “Standards”. That is, empirical evidence of the validity of the P-CAT has not been obtained in a way that helps to develop a judgment based on a synthesis of the available information.

A review of this type is critical given that some methodological issues seem to have not been resolved in the P-CAT. For example, although the multidimensionality of the P-CAT was identified in the study that introduced it, Bru-Luna et al. [ 32 ] recently stated that in adaptations of the P-CAT [ 25 , 27 , 28 , 29 , 30 , 40 ], the total score is used for interpretation and multidimensionality is disregarded. Thus, the multidimensionality of the original study was apparently not replicated. Bru-Luna et al. [ 32 ] also indicated that the internal structure validity of the P-CAT is usually underreported due to a lack of sufficiently rigorous approaches to establish with certainty how its scores are calculated.

The validity of the P-CAT, specifically its internal structure, appears to be unresolved. Nevertheless, substantive research and professional practice point to this measure as relevant to assessing PCC. This perception is contestable and judgment-based and may not be sufficient to assess the validity of the P-CAT from a cumulative and synthetic angle based on preceding validation studies. An adequate assessment of validity requires a model to conceptualize validity followed by a review of previous studies of the validity of the P-CAT using this model.

Therefore, the main purpose of this study was to conduct a systematic review of the evidence provided by P-CAT validation studies while taking the “Standards” as a framework.

The present study comprises two distinct but interconnected procedures. First, a systematic literature review was conducted following the PRISMA method ( [ 41 ]; Additional file 1; Additional file 2) with the aim of collecting all validations of the P-CAT that have been developed. Second, a systematic description of the validity evidence for each of the P-CAT validations found in the systematic review was developed following the “Standards” framework [ 37 ]. The work of Hawkins et al. [ 39 ], the first study to review validity sources according to the guidelines proposed by the “Standards”, was also used as a reference. Both provided conceptual and pragmatic guidance for organizing and classifying validity evidence for the P-CAT.

The procedure conducted in the systematic review is described below, followed by the procedure for examining the validity studies.

Systematic review

Search strategy and information sources.

Initially, the Cochrane database was searched with the aim of identifying systematic reviews of the P-CAT. When no such reviews were found, subsequent preliminary searches were performed in the Web of Science (WoS), Scopus and PubMed databases. These databases play a fundamental role in recent scientific literature since they are the main sources of published articles that undergo high-quality content and editorial review processes [ 42 ]. The search formula was as follows. The original P-CAT article [ 21 ] was located, after which all articles that cited it through 2021 were identified and analyzed. This approach ensured the inclusion of all validations. No articles were excluded on the basis of language to avoid language bias [ 43 ]. Moreover, to reduce the effects of publication bias, a complementary search in Google Scholar was also performed to allow the inclusion of “gray” literature [ 44 ]. Finally, a manual search was performed through a review of the references of the included articles to identify other articles that met the search criteria but were not present in any of the aforementioned databases.

This process was conducted by one of the authors and corroborated by another using the Covidence tool [ 45 ]. A third author was consulted in case of doubt.

Eligibility criteria and selection process

The protocol was registered in PROSPERO, and the search was conducted according to these criteria. The identification code is CRD42022335866.

The articles had to meet the following criteria for inclusion in the systematic review: (a) a methodological approach to P-CAT validations, (b) an experimental or quasiexperimental studies, (c) studies with any type of sample, and (d) studies in any language. We discarded studies that met at least one of the following exclusion criteria: (a) systematic reviews or bibliometric reviews of the instrument or meta-analyses or (b) studies published after 2021.

Data collection process

After the articles were selected, the most relevant information was extracted from each article. Fundamental data were recorded in an Excel spreadsheet for each of the sections: introduction, methodology, results and discussion. Information was also recorded about the limitations mentioned in each article as well as the practical implications and suggestions for future research.

Given the aim of the study, information was collected about the sources of validity of each study, including test content (judges’ evaluation, literature review and translation), response processes, internal structure (factor analysis, design, estimator, factor extraction method, factors and items, interfactor R, internal replication, effect of the method, and factor loadings), and relationships with other variables (convergent, divergent, concurrent and predictive validity) and consequences of measurement.

Description of the validity study

To assess the validity of the studies, an Excel table was used. Information was recorded for the seven articles included in the systematic review. The data were extracted directly from the texts of the articles and included information about the authors, the year of publication, the country where each P-CAT validation was produced and each of the five standards proposed in the “Standards” [ 37 ].

The validity source related to internal structure was divided into three sections to record information about dimensionality (e.g., factor analysis, design, estimator, factor extraction method, factors and items, interfactor R, internal replication, effect of the method, and factor loadings), reliability expression (i.e., internal consistency and test-retest) and the study of factorial invariance according to the groups into which it was divided (e.g., sex, age, profession) and the level of study (i.e., metric, intercepts). This approach allowed much more information to be obtained than relying solely on source validity based on internal structure. This division was performed by the same researcher who performed the previous processes.

Study selection and study characteristics

The systematic review process was developed according to the PRISMA methodology [ 41 ].

The WoS, Scopus, PubMed and Google Scholar databases were searched on February 12, 2022 and yielded a total of 485 articles. Of these, 111 were found in WoS, 114 in Scopus, 43 in PubMed and 217 in Google Scholar. In the first phase, the title and abstracts of all the articles were read. In this first screening, 457 articles were eliminated because they did not include studies with a methodological approach to P-CAT validation and one article was excluded because it was the original P-CAT article. This resulted in a total of 27 articles, 19 of which were duplicated in different databases and, in the case of Google Scholar, within the same database. This process yielded a total of eight articles that were evaluated for eligibility by a complete reading of the text. In this step, one of the articles was excluded due to a lack of access to the full text of the study [ 31 ] (although the original manuscript was found, it was impossible to access the complete content; in addition, the authors of the manuscript were contacted, but no reply was received). Finally, a manual search was performed by reviewing the references of the seven studies, but none were considered suitable for inclusion. Thus, the review was conducted with a total of seven articles.

Of the seven studies, six were original validations in other languages. These included Norwegian [ 27 ], Swedish [ 28 ], Chinese (which has two validations [ 29 , 40 ]), Spanish [ 25 ], and Korean [ 30 ]. The study by Selan et al. [ 46 ] included a modification of the Swedish version of the P-CAT and explored the psychometric properties of both versions (i.e., the original Swedish version and the modified version).

The item selection and screening process are illustrated in detail in Fig.  1 .

PRISMA 2020 flow diagram for new systematic reviews including database searches

Validity analysis

To provide a clear overview of the validity analyses, Table  1 descriptively shows the percentages of items that provide information about the five standards proposed by the “Standards” guide [ 37 ].

The table shows a high number of validity sources related to test content and internal structure in relation to dimensionality and internal consistency, followed by a moderate number of sources for test-retest and relationship with other variables. A rate of 0% is observed for validity sources related to response processes, invariance and test consequences. Below, different sections related to each of the standards are shown, and the information is presented in more detail.

Evidence based on test content

The first standard, which focused on test content, was met for all items (100%). Translation, which refers to the equivalence of content between the original language and the target language, was met in the six articles that conducted validation in another language and/or culture. These studies reported that the validations were translated by bilingual experts and/or experts in the area of care. In addition, three studies [ 25 , 29 , 40 ] reported that the translation process followed International Test Commission guidelines, such as those of Beaton et al. [ 47 ], Guillemin [ 48 ], Hambleton et al. [ 49 ], and Muñiz et al. [ 50 ]. Evaluation by judges, who referred to the relevance, clarity and importance of the content, was divided into two categories: expert evaluation (a panel of expert judges for each of the areas to consider in the evaluation instrument) and experiential evaluation (potential participants testing the test). The first type of evaluation occurred in three of the articles [ 28 , 29 , 46 ], while the other occurred in two [ 25 , 40 ]. Only one of the items [ 29 ] reported that the scale contained items that reflected the dimension described in the literature. The validity evidence related to the test content presented in each article can be found in Table  2 .

Evidence based on response processes

The second standard, related to the validity of the response process, was obtained according to the “Standards” from the analysis of individual responses: “questioning test takers about their performance strategies or response to particular items (…), maintaining records that monitor the development of a response to a writing task (…), documentation of other aspects of performance, like eye movement or response times…” [ 37 ] (p. 15). According to the analysis of the validity of the response processes, none of the articles complied with this evidence.

Evidence based on internal structure

The third standard, validity related to internal structure, was divided into three sections. First, the dimensionality of each study was examined in terms of factor analysis, design, estimator, factor extraction method, factors and items, interfactor R, internal replication, effect of the method, and factor loadings. Le et al. [ 40 ] conducted an exploratory-confirmatory design while Sjögren et al. [ 28 ] conducted a confirmatory-exploratory design to assess construct validity using confirmatory factor analysis (CFA) and investigated it further using exploratory factor analysis (EFA). The remaining articles employed only a single form of factor analysis: three employed EFA, and two employed CFA. Regarding the next point, only three of the articles reported the factor extraction method used, including Kaiser’s eigenvalue, criterion, scree plot test, parallel analysis and Velicer’s MAP test. Instrument validations yielded a total of two factors in five of the seven articles, while one yielded a single dimension [ 25 ] and the other yielded three dimensions [ 29 ], as in the original instrument. The interfactor R was reported only in the study by Zhong and Lou [ 29 ], whereas in the study by Martínez et al. [ 25 ], it could be easily obtained since it consisted of only one dimension. Internal replication was also calculated in the Spanish validation by randomly splitting the sample into two to test the correlations between factors. The effectiveness of the method was not reported in any of the articles. This information is presented in Table  3 in addition to a summary of the factor loadings.

The second section examined reliability. All the studies presented measures of internal consistency conducted in their entirety with Cronbach’s α coefficient for both the total scale and the subscales. The ω coefficient of McDonald was not used in any case. Four of the seven articles performed a test-retest test. Martínez et al. [ 25 ] conducted a test-retest after a period of seven days, while Le et al. [ 40 ] and Rokstad et al. [ 27 ] performed it between one and two weeks later and Sjögren et al. [ 28 ] allowed approximately two weeks to pass after the initial test.

The third section analyzes the calculation of invariance, which was not reported in any of the studies.

Evidence based on relationships with other variables

In the fourth standard, based on validity according to the relationship with other variables, the articles that reported it used only convergent validity (i.e., it was hypothesized that the variables related to the construct measured by the test—in this case, person-centeredness—were positively or negatively related to another construct). Discriminant validity hypothesizes that the variables related to the PCC construct are not correlated in any way with any other variable studied. No article (0%) measured discriminant evidence, while four (57%) measured convergent evidence [ 25 , 29 , 30 , 46 ]. Convergent validity was obtained through comparisons with instruments such as the Person-Centered Climate Questionnaire–Staff Version (PCQ-S), the Staff-Based Measures of Individualized Care for Institutionalized Persons with Dementia (IC), the Caregiver Psychological Elder Abuse Behavior Scale (CPEAB), the Organizational Climate (CLIOR) and the Maslach Burnout Inventory (MBI). In the case of Selan et al. [ 46 ], convergent validity was assessed on two items considered by the authors as “crude measures of person-centered care (i.e., external constructs) giving an indication of the instruments’ ability to measure PCC” (p. 4). Concurrent validity, which measures the degree to which the results of one test are or are not similar to those of another test conducted at more or less the same time with the same participants, and predictive validity, which allows predictions to be established regarding behavior based on comparison between the values of the instrument and the criterion, were not reported in any of the studies.

Evidence based on the consequences of testing

The fifth and final standard was related to the consequences of the test. It analyzed the consequences, both intended and unintended, of applying the test to a given sample. None of the articles presented explicit or implicit evidence of this.

The last two sources of validity can be seen in Table  4 .

Table  5 shows the results of the set of validity tests for each study according to the described standards.

The main purpose of this article is to analyze the evidence of validity in different validation studies of the P-CAT. To gather all existing validations, a systematic review of all literature citing this instrument was conducted.

The publication of validation studies of the P-CAT has been constant over the years. Since the publication of the original instrument in 2010, seven validations have been published in other languages (taking into account the Italian version by Brugnolli et al. [ 31 ], which could not be included in this study) as well as a modification of one of these versions. The very unequal distribution of validations between languages and countries is striking. A recent systematic review [ 51 ] revealed that in Europe, the countries where the PCC approach is most widely used are the United Kingdom, Sweden, the Netherlands, Northern Ireland, and Norway. It has also been shown that the neighboring countries seem to exert an influence on each other due to proximity [ 52 ] such that they tend to organize healthcare in a similar way, as is the case for Scandinavian countries. This favors the expansion of PCC and explains the numerous validations we found in this geographical area.

Although this approach is conceived as an essential element of healthcare for most governments [ 53 ], PCC varies according to the different definitions and interpretations attributed to it, which can cause confusion in its application (e.g., between Norway and the United Kingdom [ 54 ]). Moreover, facilitators of or barriers to implementation depend on the context and level of development of each country, and financial support remains one of the main factors in this regard [ 53 ]. This fact explains why PCC is not globally widespread among all territories. In countries where access to healthcare for all remains out of reach for economic reasons, the application of this approach takes a back seat, as does the validation of its assessment tools. In contrast, in a large part of Europe or in countries such as China or South Korea that have experienced decades of rapid economic development, patients are willing to be involved in their medical treatment and enjoy more satisfying and efficient medical experiences and environments [ 55 ], which facilitates the expansion of validations of instruments such as the P-CAT.

Regarding validity testing, the guidelines proposed by the “Standards” [ 37 ] were followed. According to the analysis of the different validations of the P-CAT instrument, none of the studies used a structured validity theory-based procedural framework for conducting validation. The most frequently reported validity tests were on the content of the test and two of the sections into which the internal structure was divided (i.e., dimensionality and internal consistency).

In the present article, the most cited source of validity in the studies was the content of the test because most of the articles were validations of the P-CAT in other languages, and the authors reported that the translation procedure was conducted by experts in all cases. In addition, several of the studies employed International Test Commission guidelines, such as those by Beaton et al. [ 47 ], Guillemin [ 48 ], Hambleton et al. [ 49 ], and Muñiz et al. [ 50 ]. Several studies also assessed the relevance, clarity and importance of the content.

The third source of validity, internal structure, was the next most often reported, although it appeared unevenly among the three sections into which this evidence was divided. Dimensionality and internal consistency were reported in all studies, followed by test-retest consistency. In relation to the first section, factor analysis, a total of five EFAs and four CFAs were presented in the validations. Traditionally, EFA has been used in research to assess dimensionality and identify key psychological constructs, although this approach involves a number of inconveniences, such as difficulty testing measurement invariance and incorporating latent factors into subsequent analyses [ 56 ] or the major problem of factor loading matrix rotation [ 57 ]. Studies eventually began to employ CFA, a technique that overcame some of these obstacles [ 56 ] but had other drawbacks; for example, the strict requirement of zero cross-loadings often does not fit the data well, and misspecification of zero loadings tends to produce distorted factors [ 57 ]. Recently, exploratory structural equation modeling (ESEM) has been proposed. This technique is widely recommended both conceptually and empirically to assess the internal structure of psychological tools [ 58 ] since it overcomes the limitations of EFA and CFA in estimating their parameters [ 56 , 57 ].

The next section, reliability, reports the total number of items according to Cronbach’s α reliability coefficient. Reliability is defined as a combination of systematic and random influences that determine the observed scores on a psychological test. Reporting the reliability measure ensures that item-based scores are consistent, that the tool’s responses are replicable and that they are not modified solely by random noise [ 59 , 60 ]. Currently, the most commonly employed reliability coefficient in studies with a multi-item measurement scale (MIMS) is Cronbach’s α [ 60 , 61 ].

Cronbach’s α [ 62 ] is based on numerous strict assumptions (e.g., the test must be unidimensional, factor loadings must be equal for all items and item errors should not covary) to estimate internal consistency. These assumptions are difficult to meet, and their violation may produce small reliability estimates [ 60 ]. One of the alternative measures to α that is increasingly recommended by the scientific literature is McDonald’s ω [ 63 ], a composite reliability measure. This coefficient is recommended for congeneric scales in which tau equivalence is not assumed. It has several advantages. For example, estimates of ω are usually robust when the estimated model contains more factors than the true model, even with small samples, or when skewness in univariate item distributions produces lower biases than those found when using α [ 59 ].

The test-retest method was the next most commonly reported internal structure section in these studies. This type of reliability considers the consistency of the scores of a test between two measurements separated by a period [ 64 ]. It is striking that test-retest consistency does not have a prevalence similar to that of internal consistency since, unlike internal consistency, test-retest consistency can be assessed for practically all types of patient-reported outcomes. It is even considered by some measurement experts to report reliability with greater relevance than internal consistency since it plays a fundamental role in the calculation of parameters for health measures [ 64 ]. However, the literature provides little guidance regarding the assessment of this type of reliability.

The internal structure section that was least frequently reported in the studies in this review was invariance. A lack of invariance refers to a difference between scores on a test that is not explained by group differences in the structure it is intended to measure [ 65 ]. The invariance of the measure should be emphasized as a prerequisite in comparisons between groups since “if scale invariance is not examined, item bias may not be fully recognized and this may lead to a distorted interpretation of the bias in a particular psychological measure” [ 65 ].

Evidence related to other variables was the next most reported source of validity in the studies included in this review. Specifically, the four studies that reported this evidence did so according to convergent validity and cited several instruments. None of the studies included evidence of discriminant validity, although this may be because there are currently several obstacles related to the measurement of this type of validity [ 66 ]. On the one hand, different definitions are used in the applied literature, which makes its evaluation difficult; on the other hand, the literature on discriminant validity focuses on techniques that require the use of multiple measurement methods, which often seem to have been introduced without sufficient evidence or are applied randomly.

Validity related to response processes was not reported by any of the studies. There are several methods to analyze this validity. These methods can be divided into two groups: “those that directly access the psychological processes or cognitive operations (think aloud, focus group, and interviews), compared to those which provide indirect indicators which in turn require additional inference (eye tracking and response times)” [ 38 ]. However, this validity evidence has traditionally been reported less frequently than others in most studies, perhaps because there are fewer clear and accepted practices on how to design or report these studies [ 67 ].

Finally, the consequences of testing were not reported in any of the studies. There is debate regarding this source of validity, with two main opposing streams of thought. On the one hand [ 68 , 69 ]) suggests that consequences that appear after the application of a test should not derive from any source of test invalidity and that “adverse consequences only undermine the validity of an assessment if they can be attributed to a problem of fit between the test and the construct” (p. 6). In contrast, Cronbach [ 69 , 70 ] notes that adverse social consequences that may result from the application of a test may call into question the validity of the test. However, the potential risks that may arise from the application of a test should be minimized in any case, especially in regard to health assessments. To this end, it is essential that this aspect be assessed by instrument developers and that the experiences of respondents be protected through the development of comprehensive and informed practices [ 39 ].

This work is not without limitations. First, not all published validation studies of the P-CAT, such as the Italian version by Brugnolli et al. [ 31 ], were available. These studies could have provided relevant information. Second, many sources of validity could not be analyzed because the studies provided scant or no data, such as response processes [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ], relationships with other variables [ 27 , 28 , 40 ], consequences of testing [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ], or invariance [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ] in the case of internal structure and interfactor R [ 27 , 28 , 30 , 40 , 46 ], internal replication [ 27 , 28 , 29 , 30 , 40 , 46 ] or the effect of the method [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ] in the case of dimensionality. In the future, it is hoped that authors will become aware of the importance of validity, as shown in this article and many others, and provide data on unreported sources so that comprehensive validity studies can be performed.

The present work also has several strengths. The search was extensive, and many studies were obtained using three different databases, including WoS, one of the most widely used and authoritative databases in the world. This database includes a large number and variety of articles and is not fully automated due to its human team [ 71 , 72 , 73 ]. In addition, to prevent publication bias, gray literature search engines such as Google Scholar were used to avoid the exclusion of unpublished research [ 44 ]. Finally, linguistic bias was prevented by not limiting the search to articles published in only one or two languages, thus avoiding the overrepresentation of studies in one language and underrepresentation in others [ 43 ].

Conclusions

Validity is understood as the degree to which tests and theory support the interpretations of instrument scores for their intended use [ 37 ]. From this perspective, the various validations of the P-CAT are not presented in a structured, valid, theory-based procedural framework like the “Standards” are. After integration and analysis of the results, it was observed that these validation reports offer a high number of sources of validity related to test content, internal structure in dimensionality and internal consistency, a moderate number of sources for internal structure in terms of test-retest reliability and the relationship with other variables, and a very low number of sources for response processes, internal structure in terms of invariance, and test consequences.

Validity plays a fundamental role in ensuring a sound scientific basis for test interpretations because it provides evidence of the extent to which the data provided by the test are valid for the intended purpose. This can affect clinical practice as people’s health may depend on it. In this sense, the “Standards” are considered a suitable and valid theory-based procedural framework for studying this modern conception of questionnaire validity, which should be taken into account in future research in this area.

Although the P-CAT is one of the most widely used instruments for assessing PCC, as shown in this study, PCC has rarely been studied. The developers of measurement tests applied to the health care setting, on which the health and quality of life of many people may depend, should use this validity framework to reflect the clear purpose of the measurement. This approach is important because the equity of decision making by healthcare professionals in daily clinical practice may depend on the source of validity. Through a more extensive study of validity that includes the interpretation of scores in terms of their intended use, the applicability of the P-CAT, an instrument that was initially developed for long-term care homes for elderly people, could be expanded to other care settings. However, the findings of this study show that validation studies continue to focus on traditionally studied types of validity and overlook the interpretation of scores in terms of their intended use.

Data availability

All data relevant to the study were included in the article or uploaded as additional files. Additional template data extraction forms are available from the corresponding author upon reasonable request.

Abbreviations

American Educational Research Association

American Psychological Association

Confirmatory factor analysis

Organizational Climate

Caregiver Psychological Elder Abuse Behavior Scale

Exploratory factor analysis

Exploratory structural equation modeling

Staff-based Measures of Individualized Care for Institutionalized Persons with Dementia

Maslach Burnout Inventory

Multi-item measurement scale

Maximum likelihood

National Council on Measurement in Education

Person-Centered Care Assessment Tool

• Person-centered care

Person-Centered Climate Questionnaire–Staff Version

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

International Register of Systematic Review Protocols

Standards for Educational and Psychological Testing

weighted least square mean and variance adjusted

Web of Science

Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy; 2001.

Google Scholar  

International Alliance of Patients’ Organizations. What is patient-centred healthcare? A review of definitions and principles. 2nd ed. London, UK: International Alliance of Patients’ Organizations; 2007.

World Health Organization. WHO global strategy on people-centred and integrated health services: interim report. Geneva, Switzerland: World Health Organization; 2015.

Britten N, Ekman I, Naldemirci Ö, Javinger M, Hedman H, Wolf A. Learning from Gothenburg model of person centred healthcare. BMJ. 2020;370:m2738.

Article   PubMed   Google Scholar  

Van Diepen C, Fors A, Ekman I, Hensing G. Association between person-centred care and healthcare providers’ job satisfaction and work-related health: a scoping review. BMJ Open. 2020;10:e042658.

Article   PubMed   PubMed Central   Google Scholar  

Ekman N, Taft C, Moons P, Mäkitalo Å, Boström E, Fors A. A state-of-the-art review of direct observation tools for assessing competency in person-centred care. Int J Nurs Stud. 2020;109:103634.

American Geriatrics Society Expert Panel on Person-Centered Care. Person-centered care: a definition and essential elements. J Am Geriatr Soc. 2016;64:15–8.

Article   Google Scholar  

McCormack B, McCance TV. Development of a framework for person-centred nursing. J Adv Nurs. 2006;56:472–9.

McCormack B, McCance T. Person-centred practice in nursing and health care: theory and practice. Chichester, England: Wiley; 2016.

Nolan MR, Davies S, Brown J, Keady J, Nolan J. Beyond person-centred care: a new vision for gerontological nursing. J Clin Nurs. 2004;13:45–53.

McCormack B, McCance T. Person-centred nursing: theory, models and methods. Oxford, UK: Wiley-Blackwell; 2010.

Book   Google Scholar  

Abraha I, Rimland JM, Trotta FM, Dell’Aquila G, Cruz-Jentoft A, Petrovic M, et al. Systematic review of systematic reviews of non-pharmacological interventions to treat behavioural disturbances in older patients with dementia. The SENATOR-OnTop series. BMJ Open. 2017;7:e012759.

Anderson K, Blair A. Why we need to care about the care: a longitudinal study linking the quality of residential dementia care to residents’ quality of life. Arch Gerontol Geriatr. 2020;91:104226.

Bauer M, Fetherstonhaugh D, Haesler E, Beattie E, Hill KD, Poulos CJ. The impact of nurse and care staff education on the functional ability and quality of life of people living with dementia in aged care: a systematic review. Nurse Educ Today. 2018;67:27–45.

Smythe A, Jenkins C, Galant-Miecznikowska M, Dyer J, Downs M, Bentham P, et al. A qualitative study exploring nursing home nurses’ experiences of training in person centred dementia care on burnout. Nurse Educ Pract. 2020;44:102745.

McCormack B, Borg M, Cardiff S, Dewing J, Jacobs G, Janes N, et al. Person-centredness– the ‘state’ of the art. Int Pract Dev J. 2015;5:1–15.

Wilberforce M, Challis D, Davies L, Kelly MP, Roberts C, Loynes N. Person-centredness in the care of older adults: a systematic review of questionnaire-based scales and their measurement properties. BMC Geriatr. 2016;16:63.

Rathert C, Wyrwich MD, Boren SA. Patient-centered care and outcomes: a systematic review of the literature. Med Care Res Rev. 2013;70:351–79.

Sharma T, Bamford M, Dodman D. Person-centred care: an overview of reviews. Contemp Nurse. 2016;51:107–20.

Ahmed S, Djurkovic A, Manalili K, Sahota B, Santana MJ. A qualitative study on measuring patient-centered care: perspectives from clinician-scientists and quality improvement experts. Health Sci Rep. 2019;2:e140.

Edvardsson D, Fetherstonhaugh D, Nay R, Gibson S. Development and initial testing of the person-centered Care Assessment Tool (P-CAT). Int Psychogeriatr. 2010;22:101–8.

Tamagawa R, Groff S, Anderson J, Champ S, Deiure A, Looyis J, et al. Effects of a provincial-wide implementation of screening for distress on healthcare professionals’ confidence and understanding of person-centered care in oncology. J Natl Compr Canc Netw. 2016;14:1259–66.

Degl’ Innocenti A, Wijk H, Kullgren A, Alexiou E. The influence of evidence-based design on staff perceptions of a supportive environment for person-centered care in forensic psychiatry. J Forensic Nurs. 2020;16:E23–30.

Hulin CL. A psychometric theory of evaluations of item and scale translations: fidelity across languages. J Cross Cult Psychol. 1987;18:115–42.

Martínez T, Suárez-Álvarez J, Yanguas J, Muñiz J. Spanish validation of the person-centered Care Assessment Tool (P-CAT). Aging Ment Health. 2016;20:550–8.

Martínez T, Martínez-Loredo V, Cuesta M, Muñiz J. Assessment of person-centered care in gerontology services: a new tool for healthcare professionals. Int J Clin Health Psychol. 2020;20:62–70.

Rokstad AM, Engedal K, Edvardsson D, Selbaek G. Psychometric evaluation of the Norwegian version of the person-centred Care Assessment Tool. Int J Nurs Pract. 2012;18:99–105.

Sjögren K, Lindkvist M, Sandman PO, Zingmark K, Edvardsson D. Psychometric evaluation of the Swedish version of the person-centered Care Assessment Tool (P-CAT). Int Psychogeriatr. 2012;24:406–15.

Zhong XB, Lou VW. Person-centered care in Chinese residential care facilities: a preliminary measure. Aging Ment Health. 2013;17:952–8.

Tak YR, Woo HY, You SY, Kim JH. Validity and reliability of the person-centered Care Assessment Tool in long-term care facilities in Korea. J Korean Acad Nurs. 2015;45:412–9.

Brugnolli A, Debiasi M, Zenere A, Zanolin ME, Baggia M. The person-centered Care Assessment Tool in nursing homes: psychometric evaluation of the Italian version. J Nurs Meas. 2020;28:555–63.

Bru-Luna LM, Martí-Vilar M, Merino-Soto C, Livia J. Reliability generalization study of the person-centered Care Assessment Tool. Front Psychol. 2021;12:712582.

Edvardsson D, Innes A. Measuring person-centered care: a critical comparative review of published tools. Gerontologist. 2010;50:834–46.

Hawkins M, Elsworth GR, Nolte S, Osborne RH. Validity arguments for patient-reported outcomes: justifying the intended interpretation and use of data. J Patient Rep Outcomes. 2021;5:64.

Sireci SG. On the validity of useless tests. Assess Educ Princ Policy Pract. 2016;23:226–35.

Hawkins M, Elsworth GR, Osborne RH. Questionnaire validation practice: a protocol for a systematic descriptive literature review of health literacy assessments. BMJ Open. 2019;9:e030753.

American Educational Research Association, American Psychological Association. National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.

Padilla JL, Benítez I. Validity evidence based on response processes. Psicothema. 2014;26:136–44.

PubMed   Google Scholar  

Hawkins M, Elsworth GR, Hoban E, Osborne RH. Questionnaire validation practice within a theoretical framework: a systematic descriptive literature review of health literacy assessments. BMJ Open. 2020;10:e035974.

Le C, Ma K, Tang P, Edvardsson D, Behm L, Zhang J, et al. Psychometric evaluation of the Chinese version of the person-centred Care Assessment Tool. BMJ Open. 2020;10:e031580.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88:105906.

Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008;22:338–42.

Grégoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a tower of Babel bias? J Clin Epidemiol. 1995;48:159–63.

Arias MM. Aspectos metodológicos Del metaanálisis (1). Pediatr Aten Primaria. 2018;20:297–302.

Covidence. Covidence systematic review software. Veritas Health Innovation, Australia. 2014. https://www.covidence.org/ . Accessed 28 Feb 2022.

Selan D, Jakobsson U, Condelius A. The Swedish P-CAT: modification and exploration of psychometric properties of two different versions. Scand J Caring Sci. 2017;31:527–35.

Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25:3186–91.

Guillemin F. Cross-cultural adaptation and validation of health status measures. Scand J Rheumatol. 1995;24:61–3.

Hambleton R, Merenda P, Spielberger C. Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Lawrence Erlbaum Associates; 2005.

Muñiz J, Elosua P, Hambleton RK. International test commission guidelines for test translation and adaptation: second edition. Psicothema. 2013;25:151–7.

Rosengren K, Brannefors P, Carlstrom E. Adoption of the concept of person-centred care into discourse in Europe: a systematic literature review. J Health Organ Manag. 2021;35:265–80.

Alharbi T, Olsson LE, Ekman I, Carlström E. The impact of organizational culture on the outcome of hospital care: after the implementation of person-centred care. Scand J Public Health. 2014;42:104–10.

Bensbih S, Souadka A, Diez AG, Bouksour O. Patient centered care: focus on low and middle income countries and proposition of new conceptual model. J Med Surg Res. 2020;7:755–63.

Stranz A, Sörensdotter R. Interpretations of person-centered dementia care: same rhetoric, different practices? A comparative study of nursing homes in England and Sweden. J Aging Stud. 2016;38:70–80.

Zhou LM, Xu RH, Xu YH, Chang JH, Wang D. Inpatients’ perception of patient-centered care in Guangdong province, China: a cross-sectional study. Inquiry. 2021. https://doi.org/10.1177/00469580211059482 .

Marsh HW, Morin AJ, Parker PD, Kaur G. Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis. Annu Rev Clin Psychol. 2014;10:85–110.

Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Model Multidiscip J. 2009;16:397–438.

Cabedo-Peris J, Martí-Vilar M, Merino-Soto C, Ortiz-Morán M. Basic empathy scale: a systematic review and reliability generalization meta-analysis. Healthc (Basel). 2022;10:29–62.

Flora DB. Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Adv Methods Pract Psychol Sci. 2020;3:484–501.

McNeish D. Thanks coefficient alpha, we’ll take it from here. Psychol Methods. 2018;23:412–33.

Hayes AF, Coutts JJ. Use omega rather than Cronbach’s alpha for estimating reliability. But… Commun Methods Meas. 2020;14:1–24.

Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.

McDonald R. Test theory: a unified approach. Mahwah, NJ: Erlbaum; 1999.

Polit DF. Getting serious about test-retest reliability: a critique of retest research and some recommendations. Qual Life Res. 2014;23:1713–20.

Ceylan D, Çizel B, Karakaş H. Testing destination image scale invariance for intergroup comparison. Tour Anal. 2020;25:239–51.

Rönkkö M, Cho E. An updated guideline for assessing discriminant validity. Organ Res Methods. 2022;25:6–14.

Hubley A, Zumbo B. Response processes in the context of validity: setting the stage. In: Zumbo B, Hubley A, editors. Understanding and investigating response processes in validation research. Cham, Switzerland: Springer; 2017. pp. 1–12.

Messick S. Validity of performance assessments. In: Philips G, editor. Technical issues in large-scale performance assessment. Washington, DC: Department of Education, National Center for Education Statistics; 1996. pp. 1–18.

Moss PA. The role of consequences in validity theory. Educ Meas Issues Pract. 1998;17:6–12.

Cronbach L. Five perspectives on validity argument. In: Wainer H, editor. Test validity. Hillsdale, MI: Erlbaum; 1988. pp. 3–17.

Birkle C, Pendlebury DA, Schnell J, Adams J. Web of Science as a data source for research on scientific and scholarly activity. Quant Sci Stud. 2020;1:363–76.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6:245.

Web of Science Group. Editorial selection process. Clarivate. 2024. https://clarivate.com/webofsciencegroup/solutions/%20editorial-selection-process/ . Accessed 12 Sept 2022.

Download references

Acknowledgements

The authors thank the casual helpers for their aid in information processing and searching.

This work is one of the results of research project HIM/2015/017/SSA.1207, “Effects of mindfulness training on psychological distress and quality of life of the family caregiver”. Main researcher: Filiberto Toledano-Toledano Ph.D. The present research was funded by federal funds for health research and was approved by the Commissions of Research, Ethics and Biosafety (Comisiones de Investigación, Ética y Bioseguridad), Hospital Infantil de México Federico Gómez, National Institute of Health. The source of federal funds did not control the study design, data collection, analysis, or interpretation, or decisions regarding publication.

Author information

Authors and affiliations.

Departamento de Educación, Facultad de Ciencias Sociales, Universidad Europea de Valencia, 46010, Valencia, Spain

Lluna Maria Bru-Luna

Departamento de Psicología Básica, Universitat de València, Blasco Ibáñez Avenue, 21, 46010, Valencia, Spain

Manuel Martí-Vilar

Departamento de Psicología, Instituto de Investigación de Psicología, Universidad de San Martín de Porres, Tomás Marsano Avenue 242, Lima 34, Perú

César Merino-Soto

Instituto Central de Gestión de la Investigación, Universidad Nacional Federico Villarreal, Carlos Gonzalez Avenue 285, 15088, San Miguel, Perú

José Livia-Segovia

Unidad de Investigación en Medicina Basada en Evidencias, Hospital Infantil de México Federico Gómez Instituto Nacional de Salud, Dr. Márquez 162, 06720, Doctores, Cuauhtémoc, Mexico

Juan Garduño-Espinosa & Filiberto Toledano-Toledano

Unidad de Investigación Multidisciplinaria en Salud, Instituto Nacional de Rehabilitación Luis Guillermo Ibarra Ibarra, México-Xochimilco 289, Arenal de Guadalupe, 14389, Tlalpan, Mexico City, Mexico

Filiberto Toledano-Toledano

Dirección de Investigación y Diseminación del Conocimiento, Instituto Nacional de Ciencias e Innovación para la Formación de Comunidad Científica, INDEHUS, Periférico Sur 4860, Arenal de Guadalupe, 14389, Tlalpan, Mexico City, Mexico

You can also search for this author in PubMed   Google Scholar

Contributions

L.M.B.L. conceptualized the study, collected the data, performed the formal anal- ysis, wrote the original draft, and reviewed and edited the subsequent drafts. M.M.V. collected the data and reviewed and edited the subsequent drafts. C.M.S. collected the data, performed the formal analysis, wrote the original draft, and reviewed and edited the subsequent drafts. J.L.S. collected the data, wrote the original draft, and reviewed and edited the subsequent drafts. J.G.E. collected the data and reviewed and edited the subsequent drafts. F.T.T. conceptualized the study and reviewed and edited the subsequent drafts. L.M.B.L. conceptualized the study and reviewed and edited the subsequent drafts. M.M.V. conceptualized the study and reviewed and edited the subsequent drafts. C.M.S. reviewed and edited the subsequent drafts. J.G.E. reviewed and edited the subsequent drafts. F.T.T. conceptualized the study; provided resources, software, and supervision; wrote the original draft; and reviewed and edited the subsequent drafts.

Corresponding author

Correspondence to Filiberto Toledano-Toledano .

Ethics declarations

Ethics approval and consent to participate.

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Commissions of Research, Ethics and Biosafety (Comisiones de Investigación, Ética y Bioseguridad), Hospital Infantil de México Federico Gómez, National Institute of Health. HIM/2015/017/SSA.1207, “Effects of mindfulness training on psychological distress and quality of life of the family caregiver”.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Bru-Luna, L.M., Martí-Vilar, M., Merino-Soto, C. et al. Person-centered care assessment tool with a focus on quality healthcare: a systematic review of psychometric properties. BMC Psychol 12 , 217 (2024). https://doi.org/10.1186/s40359-024-01716-7

Download citation

Received : 17 May 2023

Accepted : 07 April 2024

Published : 19 April 2024

DOI : https://doi.org/10.1186/s40359-024-01716-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Person-centered care assessment tool

BMC Psychology

ISSN: 2050-7283

• General enquiries: [email protected]

• Open access
• Published: 23 April 2024

Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review

• Javiera Fuentes-Cimma 1 , 2 ,
• Dominique Sluijsmans 3 ,
• Arnoldo Riquelme 4 ,
• Ignacio Villagran   ORCID: orcid.org/0000-0003-3130-8326 1 ,
• Lorena Isbej   ORCID: orcid.org/0000-0002-4272-8484 2 , 5 ,
• María Teresa Olivares-Labbe 6 &
• Sylvia Heeneman 7  

BMC Medical Education volume  24 , Article number:  440 ( 2024 ) Cite this article

Metrics details

Feedback processes are crucial for learning, guiding improvement, and enhancing performance. In workplace-based learning settings, diverse teaching and assessment activities are advocated to be designed and implemented, generating feedback that students use, with proper guidance, to close the gap between current and desired performance levels. Since productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured feedback activities within undergraduate workplace-based learning settings. However, these settings are characterized by their unpredictable nature, which can either promote learning or present challenges in offering structured learning opportunities for students. This scoping review maps literature on how feedback processes are organised in undergraduate clinical workplace-based learning settings, providing insight into the design and use of feedback.

A scoping review was conducted. Studies were identified from seven databases and ten relevant journals in medical education. The screening process was performed independently in duplicate with the support of the StArt program. Data were organized in a data chart and analyzed using thematic analysis. The feedback loop with a sociocultural perspective was used as a theoretical framework.

The search yielded 4,877 papers, and 61 were included in the review. Two themes were identified in the qualitative analysis: (1) The organization of the feedback processes in workplace-based learning settings, and (2) Sociocultural factors influencing the organization of feedback processes. The literature describes multiple teaching and assessment activities that generate feedback information. Most papers described experiences and perceptions of diverse teaching and assessment feedback activities. Few studies described how feedback processes improve performance. Sociocultural factors such as establishing a feedback culture, enabling stable and trustworthy relationships, and enhancing student feedback agency are crucial for productive feedback processes.

Conclusions

This review identified concrete ideas regarding how feedback could be organized within the clinical workplace to promote feedback processes. The feedback encounter should be organized to allow follow-up of the feedback, i.e., working on required learning and performance goals at the next occasion. The educational programs should design feedback processes by appropriately planning subsequent tasks and activities. More insight is needed in designing a full-loop feedback process, in which specific attention is needed in effective feedforward practices.

Peer Review reports

The design of effective feedback processes in higher education has been important for educators and researchers and has prompted numerous publications discussing potential mechanisms, theoretical frameworks, and best practice examples over the past few decades. Initially, research on feedback primarily focused more on teachers and feedback delivery, and students were depicted as passive feedback recipients [ 1 , 2 , 3 ]. The feedback conversation has recently evolved to a more dynamic emphasis on interaction, sense-making, outcomes in actions, and engagement with learners [ 2 ]. This shift aligns with utilizing the feedback process as a form of social interaction or dialogue to enhance performance [ 4 ]. Henderson et al. (2019) defined feedback processes as "where the learner makes sense of performance-relevant information to promote their learning." (p. 17). When a student grasps the information concerning their performance in connection to the desired learning outcome and subsequently takes suitable action, a feedback loop is closed so the process can be regarded as successful [ 5 , 6 ].

Hattie and Timperley (2007) proposed a comprehensive perspective on feedback, the so-called feedback loop, to answer three key questions: “Where am I going? “How am I going?” and “Where to next?” [ 7 ]. Each question represents a key dimension of the feedback loop. The first is the feed-up, which consists of setting learning goals and sharing clear objectives of learners' performance expectations. While the concept of the feed-up might not be consistently included in the literature, it is considered to be related to principles of effective feedback and goal setting within educational contexts [ 7 , 8 ]. Goal setting allows students to focus on tasks and learning, and teachers to have clear intended learning outcomes to enable the design of aligned activities and tasks in which feedback processes can be embedded [ 9 ]. Teachers can improve the feed-up dimension by proposing clear, challenging, but achievable goals [ 7 ]. The second dimension of the feedback loop focuses on feedback and aims to answer the second question by obtaining information about students' current performance. Different teaching and assessment activities can be used to obtain feedback information, and it can be provided by a teacher or tutor, a peer, oneself, a patient, or another coworker. The last dimension of the feedback loop is the feedforward, which is specifically associated with using feedback to improve performance or change behaviors [ 10 ]. Feedforward is crucial in closing the loop because it refers to those specific actions students must take to reduce the gap between current and desired performance [ 7 ].

From a sociocultural perspective, feedback processes involve a social practice consisting of intricate relationships within a learning context [ 11 ]. The main feature of this approach is that students learn from feedback only when the feedback encounter includes generating, making sense of, and acting upon the information given [ 11 ]. In the context of workplace-based learning (WBL), actionable feedback plays a crucial role in enabling learners to leverage specific feedback to enhance their performance, skills, and conceptual understandings. The WBL environment provides students with a valuable opportunity to gain hands-on experience in authentic clinical settings, in which students work more independently on real-world tasks, allowing them to develop and exhibit their competencies [ 3 ]. However, WBL settings are characterized by their unpredictable nature, which can either promote self-directed learning or present challenges in offering structured learning opportunities for students [ 12 ]. Consequently, designing purposive feedback opportunities within WBL settings is a significant challenge for clinical teachers and faculty.

In undergraduate clinical education, feedback opportunities are often constrained due to the emphasis on clinical work and the absence of dedicated time for teaching [ 13 ]. Students are expected to perform autonomously under supervision, ideally achieved by giving them space to practice progressively and providing continuous instances of constructive feedback [ 14 ]. However, the hierarchy often present in clinical settings places undergraduate students in a dependent position, below residents and specialists [ 15 ]. Undergraduate or junior students may have different approaches to receiving and using feedback. If their priority is meeting the minimum standards given pass-fail consequences and acting merely as feedback recipients, other incentives may be needed to engage with the feedback processes because they will need more learning support [ 16 , 17 ]. Adequate supervision and feedback have been recognized as vital educational support in encouraging students to adopt a constructive learning approach [ 18 ]. Given that productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured teaching and learning feedback activities within undergraduate WBL settings.

Despite the extensive research on feedback, a significant proportion of published studies involve residents or postgraduate students [ 19 , 20 ]. Recent reviews focusing on feedback interventions within medical education have clearly distinguished between undergraduate medical students and residents or fellows [ 21 ]. To gain a comprehensive understanding of initiatives related to actionable feedback in the WBL environment for undergraduate health professions, a scoping review of the existing literature could provide insight into how feedback processes are designed in that context. Accordingly, the present scoping review aims to answer the following research question: How are the feedback processes designed in the undergraduate health professions' workplace-based learning environments?

A scoping review was conducted using the five-step methodological framework proposed by Arksey and O'Malley (2005) [ 22 ], intertwined with the PRISMA checklist extension for scoping reviews to provide reporting guidance for this specific type of knowledge synthesis [ 23 ]. Scoping reviews allow us to study the literature without restricting the methodological quality of the studies found, systematically and comprehensively map the literature, and identify gaps [ 24 ]. Furthermore, a scoping review was used because this topic is not suitable for a systematic review due to the varied approaches described and the large difference in the methodologies used [ 21 ].

Search strategy

With the collaboration of a medical librarian, the authors used the research question to guide the search strategy. An initial meeting was held to define keywords and search resources. The proposed search strategy was reviewed by the research team, and then the study selection was conducted in two steps:

An online database search included Medline/PubMed, Web of Science, CINAHL, Cochrane Library, Embase, ERIC, and PsycINFO.

A directed search of ten relevant journals in the health sciences education field (Academic Medicine, Medical Education, Advances in Health Sciences Education, Medical Teacher, Teaching and Learning in Medicine, Journal of Surgical Education, BMC Medical Education, Medical Education Online, Perspectives on Medical Education and The Clinical Teacher) was performed.

The research team conducted a pilot or initial search before the full search to identify if the topic was susceptible to a scoping review. The full search was conducted in November 2022. One team member (MO) identified the papers in the databases. JF searched in the selected journals. Authors included studies written in English due to feasibility issues, with no time span limitation. After eliminating duplicates, two research team members (JF and IV) independently reviewed all the titles and abstracts using the exclusion and inclusion criteria described in Table  2 and with the support of the screening application StArT [ 25 ]. A third team member (AR) reviewed the titles and abstracts when the first two disagreed. The reviewer team met again at a midpoint and final stage to discuss the challenges related to study selection. Articles included for full-text review were exported to Mendeley. JF independently screened all full-text papers, and AR verified 10% for inclusion. The authors did not analyze study quality or risk of bias during study selection, which is consistent with conducting a scoping review.

The analysis of the results incorporated a descriptive summary and a thematic analysis, which was carried out to clarify and give consistency to the results' reporting [ 22 , 24 , 26 ]. Quantitative data were analyzed to report the characteristics of the studies, populations, settings, methods, and outcomes. Qualitative data were labeled, coded, and categorized into themes by three team members (JF, SH, and DS). The feedback loop framework with a sociocultural perspective was used as the theoretical framework to analyze the results.

The keywords used for the search strategies were as follows:

Clinical clerkship; feedback; formative feedback; health professions; undergraduate medical education; workplace.

Definitions of the keywords used for the present review are available in Appendix 1 .

As an example, we included the search strategy that we used in the Medline/PubMed database when conducting the full search:

("Formative Feedback"[Mesh] OR feedback) AND ("Workplace"[Mesh] OR workplace OR "Clinical Clerkship"[Mesh] OR clerkship) AND (("Education, Medical, Undergraduate"[Mesh] OR undergraduate health profession*) OR (learner* medical education)).

Inclusion and exclusion criteria

The following inclusion and exclusion criteria were used (Table  1 ):

Data extraction

The research group developed a data-charting form to organize the information obtained from the studies. The process was iterative, as the data chart was continuously reviewed and improved as necessary. In addition, following Levac et al.'s recommendation (2010), the three members involved in the charting process (JF, LI, and IV) independently reviewed the first five selected studies to determine whether the data extraction was consistent with the objectives of this scoping review and to ensure consistency. Then, the team met using web-conferencing software (Zoom; CA, USA) to review the results and adjust any details in the chart. The same three members extracted data independently from all the selected studies, considering two members reviewing each paper [ 26 ]. A third team member was consulted if any conflict occurred when extracting data. The data chart identified demographic patterns and facilitated the data synthesis. To organize data, we used a shared Excel spreadsheet, considering the following headings: title, author(s), year of publication, journal/source, country/origin, aim of the study, research question (if any), population/sample size, participants, discipline, setting, methodology, study design, data collection, data analysis, intervention, outcomes, outcomes measure, key findings, and relation of findings to research question.

Additionally, all the included papers were uploaded to AtlasTi v19 to facilitate the qualitative analysis. Three team members (JF, SH, and DS) independently coded the first six papers to create a list of codes to ensure consistency and rigor. The group met several times to discuss and refine the list of codes. Then, one member of the team (JF) used the code list to code all the rest of the papers. Once all papers were coded, the team organized codes into descriptive themes aligned with the research question.

Preliminary results were shared with a number of stakeholders (six clinical teachers, ten students, six medical educators) to elicit their opinions as an opportunity to build on the evidence and offer a greater level of meaning, content expertise, and perspective to the preliminary findings [ 26 ]. No quality appraisal of the studies is considered for this scoping review, which aligns with the frameworks for guiding scoping reviews [ 27 ].

The datasets analyzed during the current study are available from the corresponding author upon request.

A database search resulted in 3,597 papers, and the directed search of the most relevant journals in the health sciences education field yielded 2,096 titles. An example of the results of one database is available in Appendix 2 . Of the titles obtained, 816 duplicates were eliminated, and the team reviewed the titles and abstracts of 4,877 papers. Of these, 120 were selected for full-text review. Finally, 61 papers were included in this scoping review (Fig.  1 ), as listed in Table  2 .

PRISMA flow diagram for included studies, incorporating records identified through the database and direct searching

The selected studies were published between 1986 and 2022, and seventy-five percent (46) were published during the last decade. Of all the articles included in this review, 13% (8) were literature reviews: one integrative review [ 28 ] and four scoping reviews [ 29 , 30 , 31 , 32 ]. Finally, fifty-three (87%) original or empirical papers were included (i.e., studies that answered a research question or achieved a research purpose through qualitative or quantitative methodologies) [ 15 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 ].

Table 2 summarizes the papers included in the present scoping review, and Table  3 describes the characteristics of the included studies.

The thematic analysis resulted in two themes: (1) the organization of feedback processes in WBL settings, and (2) sociocultural factors influencing the organization of feedback processes. Table 4 gives a summary of the themes and subthemes.

Organization of feedback processes in WBL settings.

Setting learning goals (i.e., feed-up dimension).

Feedback that focuses on students' learning needs and is based on known performance standards enhances student response and setting learning goals [ 30 ]. Discussing goals and agreements before starting clinical practice enhances students' feedback-seeking behavior [ 39 ] and responsiveness to feedback [ 83 ]. Farrell et al. (2017) found that teacher-learner co-constructed learning goals enhance feedback interactions and help establish educational alliances, improving the learning experience [ 50 ]. However, Kiger (2020) found that sharing individualized learning plans with teachers aligned feedback with learning goals but did not improve students' perceived use of feedback [ 64 ]

Two papers of this set pointed out the importance of goal-oriented feedback, a dynamic process that depends on discussion of goal setting between teachers and students [ 50 ] and influences how individuals experience, approach, and respond to upcoming learning activities [ 34 ]. Goal-oriented feedback should be embedded in the learning experience of the clinical workplace, as it can enhance students' engagement in safe feedback dialogues [ 50 ]. Ideally, each feedback encounter in the WBL context should conclude, in addition to setting a plan of action to achieve the desired goal, with a reflection on the next goal [ 50 ].

Feedback strategies within the WBL environment. (i.e., feedback dimension)

In undergraduate WBL environments, there are several tasks and feedback opportunities organized in the undergraduate clinical workplace that can enable feedback processes:

Questions from clinical teachers to students are a feedback strategy [ 74 ]. There are different types of questions that the teacher can use, either to clarify concepts, to reach the correct answer, or to facilitate self-correction [ 74 ]. Usually, questions can be used in conjunction with other communication strategies, such as pauses, which enable self-correction by the student [ 74 ]. Students can also ask questions to obtain feedback on their performance [ 54 ]. However, question-and-answer as a feedback strategy usually provides information on either correct or incorrect answers and fewer suggestions for improvement, rendering it less constructive as a feedback strategy [ 82 ].

Direct observation of performance by default is needed to be able to provide information to be used as input in the feedback process [ 33 , 46 , 49 , 86 ]. In the process of observation, teachers can include clarification of objectives (i.e., feed-up dimension) and suggestions for an action plan (i.e., feedforward) [ 50 ]. Accordingly, Schopper et al. (2016) showed that students valued being observed while interviewing patients, as they received feedback that helped them become more efficient and effective as interviewers and communicators [ 33 ]. Moreover, it is widely described that direct observation improves feedback credibility [ 33 , 40 , 84 ]. Ideally, observation should be deliberate [ 33 , 83 ], informal or spontaneous [ 33 ], conducted by a (clinical) expert [ 46 , 86 ], provided immediately after the observation, and clinical teacher if possible, should schedule or be alert on follow-up observations to promote closing the gap between current and desired performance [ 46 ].

Workplace-based assessments (WBAs), by definition, entail direct observation of performance during authentic task demonstration [ 39 , 46 , 56 , 87 ]. WBAs can significantly impact behavioral change in medical students [ 55 ]. Organizing and designing formative WBAs and embedding these in a feedback dialogue is essential for effective learning [ 31 ].

Summative organization of WBAs is a well described barrier for feedback uptake in the clinical workplace [ 35 , 46 ]. If feedback is perceived as summative, or organized as a pass-fail decision, students may be less inclined to use the feedback for future learning [ 52 ]. According to Schopper et al. (2016), using a scale within a WBA makes students shift their focus during the clinical interaction and see it as an assessment with consequences [ 33 ]. Harrison et al. (2016) pointed out that an environment that only contains assessments with a summative purpose will not lead to a culture of learning and improving performance [ 56 ]. The recommendation is to separate the formative and summative WBAs, as feedback in summative instances is often not recognized as a learning opportunity or an instance to seek feedback [ 54 ]. In terms of the design, an organizational format is needed to clarify to students how formative assessments can promote learning from feedback [ 56 ]. Harrison et al. (2016) identified that enabling students to have more control over their assessments, designing authentic assessments, and facilitating long-term mentoring could improve receptivity to formative assessment feedback [ 56 ].

Multiple WBA instruments and systems are reported in the literature. Sox et al. (2014) used a detailed evaluation form to help students improve their clinical case presentation skills. They found that feedback on oral presentations provided by supervisors using a detailed evaluation form improved clerkship students’ oral presentation skills [ 78 ]. Daelmans et al. (2006) suggested that a formal in-training assessment programme composed by 19 assessments that provided structured feedback, could promote observation and verbal feedback opportunities through frequent assessments [ 43 ]. However, in this setting, limited student-staff interactions still hindered feedback follow-up [ 43 ]. Designing frequent WBA improves feedback credibility [ 28 ]. Long et al. (2021) emphasized that students' responsiveness to assessment feedback hinges on its perceived credibility, underlining the importance of credibility for students to effectively engage and improve their performance [ 31 ].

The mini-CEX is one of the most widely described WBA instruments in the literature. Students perceive that the mini-CEX allows them to be observed and encourages the development of interviewing skills [ 33 ]. The mini-CEX can provide feedback that improves students' clinical skills [ 58 , 60 ], as it incorporates a structure for discussing the student's strengths and weaknesses and the design of a written action plan [ 39 , 80 ]. When mini-CEXs are incorporated as part of a system of WBA, such as programmatic assessment, students feel confident in seeking feedback after observation, and being systematic allows for follow-up [ 39 ]. Students suggested separating grading from observation and using the mini-CEX in more informal situations [ 33 ].

Clinical encounter cards allow students to receive weekly feedback and make them request more feedback as the clerkship progresses [ 65 ]. Moreover, encounter cards stimulate that feedback is given by supervisors, and students are more satisfied with the feedback process [ 72 ]. With encounter card feedback, students are responsible for asking a supervisor for feedback before a clinical encounter, and supervisors give students written and verbal comments about their performance after the encounter [ 42 , 72 ]. Encounter cards enhance the use of feedback and add approximately one minute to the length of the clinical encounter, so they are well accepted by students and supervisors [ 72 ]. Bennett (2006) identified that Instant Feedback Cards (IFC) facilitated mid-rotation feedback [ 38 ]. Feedback encounter card comments must be discussed between students and supervisors; otherwise, students may perceive it as impersonal, static, formulaic, and incomplete [ 59 ].

Self-assessments can change students' feedback orientation, transforming them into coproducers of learning [ 68 ]. Self-assessments promote the feedback process [ 68 ]. Some articles emphasize the importance of organizing self-assessments before receiving feedback from supervisors, for example, discussing their appraisal with the supervisor [ 46 , 52 ]. In designing a feedback encounter, starting with a self-assessment as feed-up, discussing with the supervisor, and identifying areas for improvement is recommended, as part of the feedback dialogue [ 68 ].

Peer feedback as an organized activity allows students to develop strategies to observe and give feedback to other peers [ 61 ]. Students can act as the feedback provider or receiver, fostering understanding of critical comments and promoting evaluative judgment for their clinical practice [ 61 ]. Within clerkships, enabling the sharing of feedback information among peers allows for a better understanding and acceptance of feedback [ 52 ]. However, students can find it challenging to take on the peer assessor/feedback provider role, as they prefer to avoid social conflicts [ 28 , 61 ]. Moreover, it has been described that they do not trust the judgment of their peers because they are not experts, although they know the procedures, tasks, and steps well and empathize with their peer status in the learning process [ 61 ].

Bedside-teaching encounters (BTEs) provide timely feedback and are an opportunity for verbal feedback during performance [ 74 ]. Rizan et al. (2014) explored timely feedback delivered within BTEs and determined that it promotes interaction that constructively enhances learner development through various corrective strategies (e.g., question and answers, pauses, etc.). However, if the feedback given during the BTEs was general, unspecific, or open-ended, it could go unnoticed [ 74 ]. Torre et al. (2005) investigated which integrated feedback activities and clinical tasks occurred on clerkship rotations and assessed students' perceived quality in each teaching encounter [ 81 ]. The feedback activities reported were feedback on written clinical history, physical examination, differential diagnosis, oral case presentation, a daily progress note, and bedside feedback. Students considered all these feedback activities high-quality learning opportunities, but they were more likely to receive feedback when teaching was at the bedside than at other teaching locations [ 81 ].

Case presentations are an opportunity for feedback within WBL contexts [ 67 , 73 ]. However, both students and supervisors struggled to identify them as feedback moments, and they often dismissed questions and clarifications around case presentations as feedback [ 73 ]. Joshi (2017) identified case presentations as a way for students to ask for informal or spontaneous supervisor feedback [ 63 ].

Organization of follow-up feedback and action plans (i.e., feedforward dimension).

Feedback that generates use and response from students is characterized by two-way communication and embedded in a dialogue [ 30 ]. Feedback must be future-focused [ 29 ], and a feedback encounter should be followed by planning the next observation [ 46 , 87 ]. Follow-up feedback could be organized as a future self-assessment, reflective practice by the student, and/or a discussion with the supervisor or coach [ 68 ]. The literature describes that a lack of student interaction with teachers makes follow-up difficult [ 43 ]. According to Haffling et al. (2011), follow-up feedback sessions improve students' satisfaction with feedback compared to students who do not have follow-up sessions. In addition, these same authors reported that a second follow-up session allows verification of improved performances or confirmation that the skill was acquired [ 55 ].

Although feedback encounter forms are a recognized way of obtaining information about performance (i.e., feedback dimension), the literature does not provide many clear examples of how they may impact the feedforward phase. For example, Joshi et al. (2016) consider a feedback form with four fields (i.e., what did you do well, advise the student on what could be done to improve performance, indicate the level of proficiency, and personal details of the tutor). In this case, the supervisor highlighted what the student could improve but not how, which is the missing phase of the co-constructed action plan [ 63 ]. Whichever WBA instrument is used in clerkships to provide feedback, it should include a "next steps" box [ 44 ], and it is recommended to organize a long-term use of the WBA instrument so that those involved get used to it and improve interaction and feedback uptake [ 55 ]. RIME-based feedback (Reporting, Interpreting, Managing, Educating) is considered an interesting example, as it is perceived as helpful to students in knowing what they need to improve in their performance [ 44 ]. Hochberg (2017) implemented formative mid-clerkship assessments to enhance face-to-face feedback conversations and co-create an improvement plan [ 59 ]. Apps for structuring and storing feedback improve the amount of verbal and written feedback. In the study of Joshi et al. (2016), a reasonable proportion of students (64%) perceived that these app tools help them improve their performance during rotations [ 63 ].

Several studies indicate that an action plan as part of the follow-up feedback is essential for performance improvement and learning [ 46 , 55 , 60 ]. An action plan corresponds to an agreed-upon strategy for improving, confirming, or correcting performance. Bing-You et al. (2017) determined that only 12% of the articles included in their scoping review incorporated an action plan for learners [ 32 ]. Holmboe et al. (2004) reported that only 11% of the feedback sessions following a mini-CEX included an action plan [ 60 ]. Suhoyo et al. (2017) also reported that only 55% of mini-CEX encounters contained an action plan [ 80 ]. Other authors reported that action plans are not commonly offered during feedback encounters [ 77 ]. Sokol-Hessner et al. (2010) implemented feedback card comments with a space to provide written feedback and a specific action plan. In their results, 96% contained positive comments, and only 5% contained constructive comments [ 77 ]. In summary, although the recommendation is to include a “next step” box in the feedback instruments, evidence shows these items are not often used for constructive comments or action plans.

Sociocultural factors influencing the organization of feedback processes.

Multiple sociocultural factors influence interaction in feedback encounters, promoting or hampering the productivity of the feedback processes.

Clinical learning culture

Context impacts feedback processes [ 30 , 82 ], and there are barriers to incorporating actionable feedback in the clinical learning context. The clinical learning culture is partly determined by the clinical context, which can be unpredictable [ 29 , 46 , 68 ], as the available patients determine learning opportunities. Supervisors are occupied by a high workload, which results in limited time or priority for teaching [ 35 , 46 , 48 , 55 , 68 , 83 ], hindering students’ feedback-seeking behavior [ 54 ], and creating a challenge for the balance between patient care and student mentoring [ 35 ].

Clinical workplace culture does not always purposefully prioritize instances for feedback processes [ 83 , 84 ]. This often leads to limited direct observation [ 55 , 68 ] and the provision of poorly informed feedback. It is also evident that this affects trust between clinical teachers and students [ 52 ]. Supervisors consider feedback a low priority in clinical contexts [ 35 ] due to low compensation and lack of protected time [ 83 ]. In particular, lack of time appears to be the most significant and well-known barrier to frequent observation and workplace feedback [ 35 , 43 , 48 , 62 , 67 , 83 ].

The clinical environment is hierarchical [ 68 , 80 ] and can make students not consider themselves part of the team and feel like a burden to their supervisor [ 68 ]. This hierarchical learning environment can lead to unidirectional feedback, limit dialogue during feedback processes, and hinder the seeking, uptake, and use of feedback [ 67 , 68 ]. In a learning culture where feedback is not supported, learners are less likely to want to seek it and feel motivated and engaged in their learning [ 83 ]. Furthermore, it has been identified that clinical supervisors lack the motivation to teach [ 48 ] and the intention to observe or reobserve performance [ 86 ].

In summary, the clinical context and WBL culture do not fully use the potential of a feedback process aimed at closing learning gaps. However, concrete actions shown in the literature can be taken to improve the effectiveness of feedback by organizing the learning context. For example, McGinness et al. (2022) identified that students felt more receptive to feedback when working in a safe, nonjudgmental environment [ 67 ]. Moreover, supervisors and trainees identified the learning culture as key to establishing an open feedback dialogue [ 73 ]. Students who perceive culture as supportive and formative can feel more comfortable performing tasks and more willing to receive feedback [ 73 ].

Relationships

There is a consensus in the literature that trusting and long-term relationships improve the chances of actionable feedback. However, relationships between supervisors and students in the clinical workplace are often brief and not organized as more longitudinally [ 68 , 83 ], leaving little time to establish a trustful relationship [ 68 ]. Supervisors change continuously, resulting in short interactions that limit the creation of lasting relationships over time [ 50 , 68 , 83 ]. In some contexts, it is common for a student to have several supervisors who have their own standards in the observation of performance [ 46 , 56 , 68 , 83 ]. A lack of stable relationships results in students having little engagement in feedback [ 68 ]. Furthermore, in case of summative assessment programmes, the dual role of supervisors (i.e., assessing and giving feedback) makes feedback interactions perceived as summative and can complicate the relationship [ 83 ].

Repeatedly, the articles considered in this review describe that long-term and stable relationships enable the development of trust and respect [ 35 , 62 ] and foster feedback-seeking behavior [ 35 , 67 ] and feedback-giver behavior [ 39 ]. Moreover, constructive and positive relationships enhance students´ use of and response to feedback [ 30 ]. For example, Longitudinal Integrated Clerkships (LICs) promote stable relationships, thus enhancing the impact of feedback [ 83 ]. In a long-term trusting relationship, feedback can be straightforward and credible [ 87 ], there are more opportunities for student observation, and the likelihood of follow-up and actionable feedback improves [ 83 ]. Johnson et al. (2020) pointed out that within a clinical teacher-student relationship, the focus must be on establishing psychological safety; thus, the feedback conversations might be transformed [ 62 ].

Stable relationships enhance feedback dialogues, which offer an opportunity to co-construct learning and propose and negotiate aspects of the design of learning strategies [ 62 ].

Students as active agents in the feedback processes

The feedback response learners generate depends on the type of feedback information they receive, how credible the source of feedback information is, the relationship between the receiver and the giver, and the relevance of the information delivered [ 49 ]. Garino (2020) noted that students who are most successful in using feedback are those who do not take criticism personally, who understand what they need to improve and know they can do so, who value and feel meaning in criticism, are not surprised to receive it, and who are motivated to seek new feedback and use effective learning strategies [ 52 ]. Successful users of feedback ask others for help, are intentional about their learning, know what resources to use and when to use them, listen to and understand a message, value advice, and use effective learning strategies. They regulate their emotions, find meaning in the message, and are willing to change [ 52 ].

Student self-efficacy influences the understanding and use of feedback in the clinical workplace. McGinness et al. (2022) described various positive examples of self-efficacy regarding feedback processes: planning feedback meetings with teachers, fostering good relationships with the clinical team, demonstrating interest in assigned tasks, persisting in seeking feedback despite the patient workload, and taking advantage of opportunities for feedback, e.g., case presentations [ 67 ].

When students are encouraged to seek feedback aligned with their own learning objectives, they promote feedback information specific to what they want to learn and improve and enhance the use of feedback [ 53 ]. McGinness et al. (2022) identified that the perceived relevance of feedback information influenced the use of feedback because students were more likely to ask for feedback if they perceived that the information was useful to them. For example, if students feel part of the clinical team and participate in patient care, they are more likely to seek feedback [ 17 ].

Learning-oriented students aim to seek feedback to achieve clinical competence at the expected level [ 75 ]; they focus on improving their knowledge and skills and on professional development [ 17 ]. Performance-oriented students aim not to fail and to avoid negative feedback [ 17 , 75 ].

For effective feedback processes, including feed-up, feedback, and feedforward, the student must be feedback-oriented, i.e., active, seeking, listening to, interpreting, and acting on feedback [ 68 ]. The literature shows that feedback-oriented students are coproducers of learning [ 68 ] and are more involved in the feedback process [ 51 ]. Additionally, students who are metacognitively aware of their learning process are more likely to use feedback to reduce gaps in learning and performance [ 52 ]. For this, students must recognize feedback when it occurs and understand it when they receive it. Thus, it is important to organize training and promote feedback literacy so that students understand what feedback is, act on it, and improve the quality of feedback and their learning plans [ 68 ].

Table 5 summarizes those feedback tasks, activities, and key features of organizational aspects that enable each phase of the feedback loop based on the literature review.

The present scoping review identified 61 papers that mapped the literature on feedback processes in the WBL environments of undergraduate health professions. This review explored how feedback processes are organized in these learning contexts using the feedback loop framework. Given the specific characteristics of feedback processes in undergraduate clinical learning, three main findings were identified on how feedback processes are being conducted in the clinical environment and how these processes could be organized to support feedback processes.

First, the literature lacks a balance between the three dimensions of the feedback loop. In this regard, most of the articles in this review focused on reporting experiences or strategies for delivering feedback information (i.e., feedback dimension). Credible and objective feedback information is based on direct observation [ 46 ] and occurs within an interaction or a dialogue [ 62 , 88 ]. However, only having credible and objective information does not ensure that it will be considered, understood, used, and put into practice by the student [ 89 ].

Feedback-supporting actions aligned with goals and priorities facilitate effective feedback processes [ 89 ] because goal-oriented feedback focuses on students' learning needs [ 7 ]. In contrast, this review showed that only a minority of the studies highlighted the importance of aligning learning objectives and feedback (i.e., the feed-up dimension). To overcome this, supervisors and students must establish goals and agreements before starting clinical practice, as it allows students to measure themselves on a defined basis [ 90 , 91 ] and enhances students' feedback-seeking behavior [ 39 , 92 ] and responsiveness to feedback [ 83 ]. In addition, learning goals should be shared, and co-constructed, through a dialogue [ 50 , 88 , 90 , 92 ]. In fact, relationship-based feedback models emphasize setting shared goals and plans as part of the feedback process [ 68 ].

Many of the studies acknowledge the importance of establishing an action plan and promoting the use of feedback (i.e., feedforward). However, there is yet limited insight on how to best implement strategies that support the use of action plans, improve performance and close learning gaps. In this regard, it is described that delivering feedback without perceiving changes, results in no effect or impact on learning [ 88 ]. To determine if a feedback loop is closed, observing a change in the student's response is necessary. In other words, feedback does not work without repeating the same task [ 68 ], so teachers need to observe subsequent tasks to notice changes [ 88 ]. While feedforward is fundamental to long-term performance, it is shown that more research is needed to determine effective actions to be implemented in the WBL environment to close feedback loops.

Second, there is a need for more knowledge about designing feedback activities in the WBL environment that will generate constructive feedback for learning. WBA is the most frequently reported feedback activity in clinical workplace contexts [ 39 , 46 , 56 , 87 ]. Despite the efforts of some authors to use WBAs as a formative assessment and feedback opportunity, in several studies, a summative component of the WBA was presented as a barrier to actionable feedback [ 33 , 56 ]. Students suggest separating grading from observation and using, for example, the mini-CEX in informal situations [ 33 ]. Several authors also recommend disconnecting the summative components of WBAs to avoid generating emotions that can limit the uptake and use of feedback [ 28 , 93 ]. Other literature recommends purposefully designing a system of assessment using low-stakes data points for feedback and learning. Accordingly, programmatic assessment is a framework that combines both the learning and the decision-making function of assessment [ 94 , 95 ]. Programmatic assessment is a practical approach for implementing low-stakes as a continuum, giving opportunities to close the gap between current and desired performance and having the student as an active agent [ 96 ]. This approach enables the incorporation of low-stakes data points that target student learning [ 93 ] and provide performance-relevant information (i.e., meaningful feedback) based on direct observations during authentic professional activities [ 46 ]. Using low-stakes data points, learners make sense of information about their performance and use it to enhance the quality of their work or performance [ 96 , 97 , 98 ]. Implementing multiple instances of feedback is more effective than providing it once because it promotes closing feedback loops by giving the student opportunities to understand the feedback, make changes, and see if those changes were effective [ 89 ].

Third, the support provided by the teacher is fundamental and should be built into a reliable and long-term relationship, where the teacher must take the role of coach rather than assessor, and students should develop feedback agency and be active in seeking and using feedback to improve performance. Although it is recognized that institutional efforts over the past decades have focused on training teachers to deliver feedback, clinical supervisors' lack of teaching skills is still identified as a barrier to workplace feedback [ 99 ]. In particular, research indicates that clinical teachers lack the skills to transform the information obtained from an observation into constructive feedback [ 100 ]. Students are more likely to use feedback if they consider it credible and constructive [ 93 ] and based on stable relationships [ 93 , 99 , 101 ]. In trusting relationships, feedback can be straightforward and credible, and the likelihood of follow-up and actionable feedback improves [ 83 , 88 ]. Coaching strategies can be enhanced by teachers building an educational alliance that allows for trustworthy relationships or having supervisors with an exclusive coaching role [ 14 , 93 , 102 ].

Last, from a sociocultural perspective, individuals are the main actors in the learning process. Therefore, feedback impacts learning only if students engage and interact with it [ 11 ]. Thus, feedback design and student agency appear to be the main features of effective feedback processes. Accordingly, the present review identified that feedback design is a key feature for effective learning in complex environments such as WBL. Feedback in the workplace must ideally be organized and implemented to align learning outcomes, learning activities, and assessments, allowing learners to learn, practice, and close feedback loops [ 88 ]. To guide students toward performances that reflect long-term learning, an intensive formative learning phase is needed, in which multiple feedback processes are included that shape students´ further learning [ 103 ]. This design would promote student uptake of feedback for subsequent performance [ 1 ].

Strengths and limitations

The strengths of this study are (1) the use of an established framework, the Arksey and O'Malley's framework [ 22 ]. We included the step of socializing the results with stakeholders, which allowed the team to better understand the results from another perspective and offer a realistic look. (2) Using the feedback loop as a theoretical framework strengthened the results and gave a more thorough explanation of the literature regarding feedback processes in the WBL context. (3) our team was diverse and included researchers from different disciplines as well as a librarian.

The present scoping review has several limitations. Although we adhered to the recommended protocols and methodologies, some relevant papers may have been omitted. The research team decided to select original studies and reviews of the literature for the present scoping review. This caused some articles, such as guidelines, perspectives, and narrative papers, to be excluded from the current study.

One of the inclusion criteria was a focus on undergraduate students. However, some papers that incorporated undergraduate and postgraduate participants were included, as these supported the results of this review. Most articles involved medical students. Although the authors did not limit the search to medicine, maybe some articles involving students from other health disciplines needed to be included, considering the search in other databases or journals.

The results give insight in how feedback could be organized within the clinical workplace to promote feedback processes. On a small scale, i.e., in the feedback encounter between a supervisor and a learner, feedback should be organized to allow for follow-up feedback, thus working on required learning and performance goals. On a larger level, i.e., in the clerkship programme or a placement rotation, feedback should be organized through appropriate planning of subsequent tasks and activities.

More insight is needed in designing a closed loop feedback process, in which specific attention is needed in effective feedforward practices. The feedback that stimulates further action and learning requires a safe and trustful work and learning environment. Understanding the relationship between an individual and his or her environment is a challenge for determining the impact of feedback and must be further investigated within clinical WBL environments. Aligning the dimensions of feed-up, feedback and feedforward includes careful attention to teachers’ and students’ feedback literacy to assure that students can act on feedback in a constructive way. In this line, how to develop students' feedback agency within these learning environments needs further research.

Boud D, Molloy E. Rethinking models of feedback for learning: The challenge of design. Assess Eval High Educ. 2013;38:698–712.

Article   Google Scholar  

Henderson M, Ajjawi R, Boud D, Molloy E. Identifying feedback that has impact. In: The Impact of Feedback in Higher Education. Springer International Publishing: Cham; 2019. p. 15–34.

Chapter   Google Scholar  

Winstone N, Carless D. Designing effective feedback processes in higher education: A learning-focused approach. 1st ed. New York: Routledge; 2020.

Google Scholar  

Ajjawi R, Boud D. Assessment & Evaluation in Higher Education Researching feedback dialogue: an interactional analysis approach. 2015. https://doi.org/10.1080/02602938.2015.1102863 .

Carless D. Feedback loops and the longer-term: towards feedback spirals. Assess Eval High Educ. 2019;44:705–14.

Sadler DR. Formative assessment and the design of instructional systems. Instr Sci. 1989;18:119–44.

Hattie J, Timperley H. The Power of Feedback The Meaning of Feedback. Rev Educ Res. 2007;77:81–112.

Zarrinabadi N, Rezazadeh M. Why only feedback? Including feed up and feed forward improves nonlinguistic aspects of L2 writing. Language Teaching Research. 2023;27(3):575–92.

Fisher D, Frey N. Feed up, back, forward. Educ Leadersh. 2009;67:20–5.

Reimann A, Sadler I, Sambell K. What’s in a word? Practices associated with ‘feedforward’ in higher education. Assessment evaluation in higher education. 2019;44:1279–90.

Esterhazy R. Re-conceptualizing Feedback Through a Sociocultural Lens. In: Henderson M, Ajjawi R, Boud D, Molloy E, editors. The Impact of Feedback in Higher Education. Cham: Palgrave Macmillan; 2019. https://doi.org/10.1007/978-3-030-25112-3_5 .

Bransen D, Govaerts MJB, Sluijsmans DMA, Driessen EW. Beyond the self: The role of co-regulation in medical students’ self-regulated learning. Med Educ. 2020;54:234–41.

Ramani S, Könings KD, Ginsburg S, Van Der Vleuten CP. Feedback Redefined: Principles and Practice. J Gen Intern Med. 2019;34:744–53.

Atkinson A, Watling CJ, Brand PL. Feedback and coaching. Eur J Pediatr. 2022;181(2):441–6.

Suhoyo Y, Schonrock-Adema J, Emilia O, Kuks JBM, Cohen-Schotanus JA. Clinical workplace learning: perceived learning value of individual and group feedback in a collectivistic culture. BMC Med Educ. 2018;18:79.

Bowen L, Marshall M, Murdoch-Eaton D. Medical Student Perceptions of Feedback and Feedback Behaviors Within the Context of the “Educational Alliance.” Acad Med. 2017;92:1303–12.

Bok HGJ, Teunissen PW, Spruijt A, Fokkema JPI, van Beukelen P, Jaarsma DADC, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47:282–91.

Al-Kadri HM, Al-Kadi MT, Van Der Vleuten CPM. Workplace-based assessment and students’ approaches to learning: A qualitative inquiry. Med Teach. 2013;35(SUPPL):1.

Dennis AA, Foy MJ, Monrouxe LV, Rees CE. Exploring trainer and trainee emotional talk in narratives about workplace-based feedback processes. Adv Health Sci Educ. 2018;23:75–93.

Watling C, LaDonna KA, Lingard L, Voyer S, Hatala R. ‘Sometimes the work just needs to be done’: socio-cultural influences on direct observation in medical training. Med Educ. 2016;50:1054–64.

Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. Feedback for Learners in Medical Education: What is Known? A Scoping Review Academic Medicine. 2017;92:1346–54.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169:467–73.

Colquhoun HL, Levac D, O’brien KK, Straus S, Tricco AC, Perrier L, et al. Scoping reviews: time for clarity in definition methods and reporting. J Clin Epidemiol. 2014;67:1291–4.

StArt - State of Art through Systematic Review. 2013.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: Advancing the methodology. Implementation Science. 2010;5:1–9.

Peters MDJ, BPharm CMGHK, Parker PMD, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc. 2015;13:141–6.

Bing-You R, Varaklis K, Hayes V, Trowbridge R, Kemp H, McKelvy D, et al. The Feedback Tango: An Integrative Review and Analysis of the Content of the Teacher-Learner Feedback Exchange. Acad Med. 2018;93:657–63.

Ossenberg C, Henderson A, Mitchell M. What attributes guide best practice for effective feedback? A scoping review. Adv Health Sci Educ. 2019;24:383–401.

Spooner M, Duane C, Uygur J, Smyth E, Marron B, Murphy PJ, et al. Self -regulatory learning theory as a lens on how undergraduate and postgraduate learners respond to feedback: A BEME scoping review: BEME Guide No. 66. Med Teach. 2022;44:3–18.

Long S, Rodriguez C, St-Onge C, Tellier PP, Torabi N, Young M. Factors affecting perceived credibility of assessment in medical education: A scoping review. Adv Health Sci Educ. 2022;27:229–62.

Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. Feedback for Learners in Medical Education: What is Known? A Scoping Review: Lippincott Williams and Wilkins; 2017.

Schopper H, Rosenbaum M, Axelson R. “I wish someone watched me interview:” medical student insight into observation and feedback as a method for teaching communication skills during the clinical years. BMC Med Educ. 2016;16:286.

Crommelinck M, Anseel F. Understanding and encouraging feedback-seeking behaviour: a literature review. Med Educ. 2013;47:232–41.

Adamson E, Kinga L, Foy L, McLeodb M, Traynor J, Watson W, et al. Feedback in clinical practice: Enhancing the students’ experience through action research. Nurse Educ Pract. 2018;31:48–53.

Al-Mously N, Nabil NM, Al-Babtain SA, et al. Undergraduate medical students’ perceptions on the quality of feedback received during clinical rotations. Med Teach. 2014;36(Supplement 1):S17-23.

Bates J, Konkin J, Suddards C, Dobson S, Pratt D. Student perceptions of assessment and feedback in longitudinal integrated clerkships. Med Educ. 2013;47:362–74.

Bennett AJ, Goldenhar LM, Stanford K. Utilization of a Formative Evaluation Card in a Psychiatry Clerkship. Acad Psychiatry. 2006;30:319–24.

Bok HG, Jaarsma DA, Spruijt A, Van Beukelen P, Van Der Vleuten CP, Teunissen PW, et al. Feedback-giving behaviour in performance evaluations during clinical clerkships. Med Teach. 2016;38:88–95.

Bok HG, Teunissen PW, Spruijt A, Fokkema JP, van Beukelen P, Jaarsma DA, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47:282–91.

Calleja P, Harvey T, Fox A, Carmichael M, et al. Feedback and clinical practice improvement: A tool to assist workplace supervisors and students. Nurse Educ Pract. 2016;17:167–73.

Carey EG, Wu C, Hur ES, Hasday SJ, Rosculet NP, Kemp MT, et al. Evaluation of Feedback Systems for the Third-Year Surgical Clerkship. J Surg Educ. 2017;74:787–93.

Daelmans HE, Overmeer RM, Van der Hem-Stokroos HH, Scherpbier AJ, Stehouwer CD, van der Vleuten CP. In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Medical education. 2006;40(1):51–8.

DeWitt D, Carline J, Paauw D, Pangaro L. Pilot study of a ’RIME’-based tool for giving feedback in a multi-specialty longitudinal clerkship. Med Educ. 2008;42:1205–9.

Dolan BM, O’Brien CL, Green MM. Including Entrustment Language in an Assessment Form May Improve Constructive Feedback for Student Clinical Skills. Med Sci Educ. 2017;27:461–4.

Duijn CC, Welink LS, Mandoki M, Ten Cate OT, Kremer WD, Bok HG. Am I ready for it? Students’ perceptions of meaningful feedback on entrustable professional activities. Perspectives on medical education. 2017;6:256–64.

Elnicki DM, Zalenski D. Integrating medical students’ goals, self-assessment and preceptor feedback in an ambulatory clerkship. Teach Learn Med. 2013;25:285–91.

Embo MP, Driessen EW, Valcke M, Van der Vleuten CP. Assessment and feedback to facilitate self-directed learning in clinical practice of Midwifery students. Medical teacher. 2010;32(7):e263-9.

Eva KW, Armson H, Holmboe E, Lockyer J, Loney E, Mann K, et al. Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ. 2012;17:15–26.

Farrell L, Bourgeois-Law G, Ajjawi R, Regehr G. An autoethnographic exploration of the use of goal oriented feedback to enhance brief clinical teaching encounters. Adv Health Sci Educ. 2017;22:91–104.

Fernando N, Cleland J, McKenzie H, Cassar K. Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Med Educ. 2008;42:89–95.

Garino A. Ready, willing and able: a model to explain successful use of feedback. Adv Health Sci Educ. 2020;25:337–61.

Garner MS, Gusberg RJ, Kim AW. The positive effect of immediate feedback on medical student education during the surgical clerkship. J Surg Educ. 2014;71:391–7.

Bing-You R, Hayes V, Palka T, Ford M, Trowbridge R. The Art (and Artifice) of Seeking Feedback: Clerkship Students’ Approaches to Asking for Feedback. Acad Med. 2018;93:1218–26.

Haffling AC, Beckman A, Edgren G. Structured feedback to undergraduate medical students: 3 years’ experience of an assessment tool. Medical teacher. 2011;33(7):e349-57.

Harrison CJ, Könings KD, Dannefer EF, Schuwirth LWTT, Wass V, van der Vleuten CPMM. Factors influencing students’ receptivity to formative feedback emerging from different assessment cultures. Perspect Med Educ. 2016;5:276–84.

Harrison CJ, Könings KD, Schuwirth LW, Wass V, Van der Vleuten CP, Konings KD, et al. Changing the culture of assessment: the dominance of the summative assessment paradigm. BMC medical education. 2017;17:1–4.

Harvey P, Radomski N, O’Connor D. Written feedback and continuity of learning in a geographically distributed medical education program. Medical teacher. 2013;35(12):1009–13.

Hochberg M, Berman R, Ogilvie J, Yingling S, Lee S, Pusic M, et al. Midclerkship feedback in the surgical clerkship: the “Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills” application utilizing learner self-assessment. Am J Surg. 2017;213:212–6.

Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the mini clinical evaluation exercise. Journal of general internal medicine. 2004;19(5):558–61.

Tai JHM, Canny BJ, Haines TP, Molloy EK. The role of peer-assisted learning in building evaluative judgement: opportunities in clinical medical education. Adv Health Sci Educ. 2016;21:659–76.

Johnson CE, Keating JL, Molloy EK. Psychological safety in feedback: What does it look like and how can educators work with learners to foster it? Med Educ. 2020;54:559–70.

Joshi A, Generalla J, Thompson B, Haidet P. Facilitating the Feedback Process on a Clinical Clerkship Using a Smartphone Application. Acad Psychiatry. 2017;41:651–5.

Kiger ME, Riley C, Stolfi A, Morrison S, Burke A, Lockspeiser T. Use of Individualized Learning Plans to Facilitate Feedback Among Medical Students. Teach Learn Med. 2020;32:399–409.

Kogan J, Shea J. Implementing feedback cards in core clerkships. Med Educ. 2008;42:1071–9.

Lefroy J, Walters B, Molyneux A, Smithson S. Can learning from workplace feedback be enhanced by reflective writing? A realist evaluation in UK undergraduate medical education. Educ Prim Care. 2021;32:326–35.

McGinness HT, Caldwell PHY, Gunasekera H, Scott KM. ‘Every Human Interaction Requires a Bit of Give and Take’: Medical Students’ Approaches to Pursuing Feedback in the Clinical Setting. Teach Learn Med. 2022. https://doi.org/10.1080/10401334.2022.2084401 .

Noble C, Billett S, Armit L, Collier L, Hilder J, Sly C, et al. ``It’s yours to take{’’}: generating learner feedback literacy in the workplace. Adv Health Sci Educ. 2020;25:55–74.

Ogburn T, Espey E. The R-I-M-E method for evaluation of medical students on an obstetrics and gynecology clerkship. Am J Obstet Gynecol. 2003;189:666–9.

Po O, Reznik M, Greenberg L. Improving a medical student feedback with a clinical encounter card. Ambul Pediatr. 2007;7:449–52.

Parkes J, Abercrombie S, McCarty T, Parkes J, Abercrombie S, McCarty T. Feedback sandwiches affect perceptions but not performance. Adv Health Sci Educ. 2013;18:397–407.

Paukert JL, Richards ML, Olney C. An encounter card system for increasing feedback to students. Am J Surg. 2002;183:300–4.

Rassos J, Melvin LJ, Panisko D, Kulasegaram K, Kuper A. Unearthing Faculty and Trainee Perspectives of Feedback in Internal Medicine: the Oral Case Presentation as a Model. J Gen Intern Med. 2019;34:2107–13.

Rizan C, Elsey C, Lemon T, Grant A, Monrouxe L. Feedback in action within bedside teaching encounters: a video ethnographic study. Med Educ. 2014;48:902–20.

Robertson AC, Fowler LC. Medical student perceptions of learner-initiated feedback using a mobile web application. Journal of medical education and curricular development. 2017;4:2382120517746384.

Scheidt PC, Lazoritz S, Ebbeling WL, Figelman AR, Moessner HF, Singer JE. Evaluation of system providing feedback to students on videotaped patient encounters. J Med Educ. 1986;61(7):585–90.

Sokol-Hessner L, Shea J, Kogan J. The open-ended comment space for action plans on core clerkship students’ encounter cards: what gets written? Acad Med. 2010;85:S110–4.

Sox CM, Dell M, Phillipi CA, Cabral HJ, Vargas G, Lewin LO. Feedback on oral presentations during pediatric clerkships: a randomized controlled trial. Pediatrics. 2014;134:965–71.

Spickard A, Gigante J, Stein G, Denny JC. Automatic capture of student notes to augment mentor feedback and student performance on patient write-ups. J Gen Intern Med. 2008;23:979–84.

Suhoyo Y, Van Hell EA, Kerdijk W, Emilia O, Schönrock-Adema J, Kuks JB, et al. nfluence of feedback characteristics on perceived learning value of feedback in clerkships: does culture matter? BMC medical education. 2017;17:1–7.

Torre DM, Simpson D, Sebastian JL, Elnicki DM. Learning/feedback activities and high-quality teaching: perceptions of third-year medical students during an inpatient rotation. Acad Med. 2005;80:950–4.

Urquhart LM, Ker JS, Rees CE. Exploring the influence of context on feedback at medical school: a video-ethnography study. Adv Health Sci Educ. 2018;23:159–86.

Watling C, Driessen E, van der Vleuten C, Lingard L. Learning culture and feedback: an international study of medical athletes and musicians. Med Educ. 2014;48:713–23.

Watling C, Driessen E, van der Vleuten C, Vanstone M, Lingard L. Beyond individualism: Professional culture and its influence on feedback. Med Educ. 2013;47:585–94.

Soemantri D, Dodds A, Mccoll G. Examining the nature of feedback within the Mini Clinical Evaluation Exercise (Mini-CEX): an analysis of 1427 Mini-CEX assessment forms. GMS J Med Educ. 2018;35:Doc47.

Van De Ridder JMMM, Stokking KM, McGaghie WC, ten Cate OTJ, van der Ridder JM, Stokking KM, et al. What is feedback in clinical education? Med Educ. 2008;42:189–97.

van de Ridder JMMM, McGaghie WC, Stokking KM, ten Cate OTJJ. Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review. Med Educ. 2015;49:658–73.

Boud D. Feedback: ensuring that it leads to enhanced learning. Clin Teach. 2015. https://doi.org/10.1111/tct.12345 .

Brehaut J, Colquhoun H, Eva K, Carrol K, Sales A, Michie S, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164:435–41.

Ende J. Feedback in clinical medical education. J Am Med Assoc. 1983;250:777–81.

Cantillon P, Sargeant J. Giving feedback in clinical settings. Br Med J. 2008;337(7681):1292–4.

Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29:855–71.

Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Med Educ. 2019;53:76–85.

van der Vleuten CPM, Schuwirth LWT, Driessen EW, Dijkstra J, Tigelaar D, Baartman LKJ, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34:205–14.

Schuwirth LWT, der Vleuten CPM. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33:478–85.

Schut S, Driessen E, van Tartwijk J, van der Vleuten C, Heeneman S. Stakes in the eye of the beholder: an international study of learners’ perceptions within programmatic assessment. Med Educ. 2018;52:654–63.

Henderson M, Boud D, Molloy E, Dawson P, Phillips M, Ryan T, Mahoney MP. Feedback for learning. Closing the assessment loop. Framework for effective learning. Canberra, Australia: Australian Government, Department for Education and Training; 2018.

Heeneman S, Pool AO, Schuwirth LWT, van der Vleuten CPM, Driessen EW, Oudkerk A, et al. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 2015;49:487–98.

Lefroy J, Watling C, Teunissen P, Brand P, Watling C. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ. 2015;4:284–99.

Ramani S, Krackov SK. Twelve tips for giving feedback effectively in the clinical environment. Med Teach. 2012;34:787–91.

Telio S, Ajjawi R, Regehr G. The, “Educational Alliance” as a Framework for Reconceptualizing Feedback in Medical Education. Acad Med. 2015;90:609–14.

Lockyer J, Armson H, Könings KD, Lee-Krueger RC, des Ordons AR, Ramani S, et al. In-the-Moment Feedback and Coaching: Improving R2C2 for a New Context. J Grad Med Educ. 2020;12:27–35.

Black P, Wiliam D. Developing the theory of formative assessment. Educ Assess Eval Account. 2009;21:5–31.

Download references

Author information

Authors and affiliations.

Department of Health Sciences, Faculty of Medicine, Pontificia Universidad Católica de Chile, Avenida Vicuña Mackenna 4860, Macul, Santiago, Chile

Javiera Fuentes-Cimma & Ignacio Villagran

School of Health Professions Education, Maastricht University, Maastricht, Netherlands

Javiera Fuentes-Cimma & Lorena Isbej

Rotterdam University of Applied Sciences, Rotterdam, Netherlands

Dominique Sluijsmans

Centre for Medical and Health Profession Education, Department of Gastroenterology, Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile

Arnoldo Riquelme

School of Dentistry, Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile

Lorena Isbej

Sistema de Bibliotecas UC (SIBUC), Pontificia Universidad Católica de Chile, Santiago, Chile

María Teresa Olivares-Labbe

Department of Pathology, Faculty of Health, Medicine and Health Sciences, Maastricht University, Maastricht, Netherlands

Sylvia Heeneman

You can also search for this author in PubMed   Google Scholar

Contributions

J.F-C, D.S, and S.H. made substantial contributions to the conception and design of the work. M.O-L contributed to the identification of studies. J.F-C, I.V, A.R, and L.I. made substantial contributions to the screening, reliability, and data analysis. J.F-C. wrote th e main manuscript text. All authors reviewed the manuscript.

Corresponding author

Correspondence to Javiera Fuentes-Cimma .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1, supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Fuentes-Cimma, J., Sluijsmans, D., Riquelme, A. et al. Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review. BMC Med Educ 24 , 440 (2024). https://doi.org/10.1186/s12909-024-05439-6

Download citation

Received : 25 September 2023

Accepted : 17 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1186/s12909-024-05439-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical clerkship
• Feedback processes
• Feedforward
• Formative feedback
• Health professions
• Undergraduate medical education
• Undergraduate healthcare education
• Workplace learning

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 08 November 2023

Policies to prevent zoonotic spillover: a systematic scoping review of evaluative evidence

• Chloe Clifford Astbury 1 , 2 , 3 ,
• Kirsten M. Lee 1 , 2 ,
• Ryan Mcleod 1 ,
• Raphael Aguiar 2 ,
• Asma Atique 1 ,
• Marilen Balolong 4 ,
• Janielle Clarke 1 ,
• Anastassia Demeshko 1 ,
• Ronald Labonté 5 ,
• Arne Ruckert 5 ,
• Priyanka Sibal 6 ,
• Kathleen Chelsea Togño 4 ,
• A. M. Viens 1 , 3 ,
• Mary Wiktorowicz 1 , 2 ,
• Marc K. Yambayamba 7 ,
• Amy Yau 8 &
• Tarra L. Penney 1 , 2 , 3  

Globalization and Health volume  19 , Article number:  82 ( 2023 ) Cite this article

2931 Accesses

24 Altmetric

Metrics details

Emerging infectious diseases of zoonotic origin present a critical threat to global population health. As accelerating globalisation makes epidemics and pandemics more difficult to contain, there is a need for effective preventive interventions that reduce the risk of zoonotic spillover events. Public policies can play a key role in preventing spillover events. The aim of this review is to identify and describe evaluations of public policies that target the determinants of zoonotic spillover. Our approach is informed by a One Health perspective, acknowledging the inter-connectedness of human, animal and environmental health.

In this systematic scoping review, we searched Medline, SCOPUS, Web of Science and Global Health in May 2021 using search terms combining animal health and the animal-human interface, public policy, prevention and zoonoses. We screened titles and abstracts, extracted data and reported our process in line with PRISMA-ScR guidelines. We also searched relevant organisations’ websites for evaluations published in the grey literature. All evaluations of public policies aiming to prevent zoonotic spillover events were eligible for inclusion. We summarised key data from each study, mapping policies along the spillover pathway.

Our review found 95 publications evaluating 111 policies. We identified 27 unique policy options including habitat protection; trade regulations; border control and quarantine procedures; farm and market biosecurity measures; public information campaigns; and vaccination programmes, as well as multi-component programmes. These were implemented by many sectors, highlighting the cross-sectoral nature of zoonotic spillover prevention. Reports emphasised the importance of surveillance data in both guiding prevention efforts and enabling policy evaluation, as well as the importance of industry and private sector actors in implementing many of these policies. Thoughtful engagement with stakeholders ranging from subsistence hunters and farmers to industrial animal agriculture operations is key for policy success in this area.

This review outlines the state of the evaluative evidence around policies to prevent zoonotic spillover in order to guide policy decision-making and focus research efforts. Since we found that most of the existing policy evaluations target ‘downstream’ determinants, additional research could focus on evaluating policies targeting ‘upstream’ determinants of zoonotic spillover, such as land use change, and policies impacting infection intensity and pathogen shedding in animal populations, such as those targeting animal welfare.

The increasing incidence of zoonotic emerging infectious diseases (EIDs) has been attributed to behavioural practices and ecological and socioeconomic change, and is predicted to continue in the coming years [ 1 ]. Higher levels of anthropogenic activity, including agricultural intensification, urbanisation and other forms of land use change, have led to increased interactions between wildlife, humans and livestock, increasing the risk of cross-species transmission [ 2 , 3 , 4 ]. Meanwhile, accelerating rates of globalisation and urbanisation, leading to increased global movement of people and goods and more dense human settlements, have made outbreaks of disease in human populations more difficult to contain [ 5 ]. In response, a call has been issued by leading organisations and experts, including the United Nations Environment Programme, the International Livestock Research Institute and the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services, to complement reactive policy responses with policies that prevent zoonotic EIDs [ 1 , 6 , 7 , 8 , 9 , 10 ]. This approach, sometimes called deep prevention, would need to target upstream drivers to reduce the risk of outbreaks occuring [ 11 ].

Zoonotic spillover, defined as the transmission of a pathogen from an animal to a human, depends on the alignment of ecological, epidemiological and behavioural factors [ 12 ]. Zoonotic pathogens must be transmitted across a spillover pathway (Fig.  1 ) in order to induce infections in humans [ 12 , 13 ]. This involves meeting a series of conditions including appropriate density and distribution of reservoir hosts, pathogen prevalence, infection intensity and human exposure [ 12 ]. Across this pathway, a number of drivers of zoonotic spillover have been identified, including changes in wildlife and livestock populations [ 14 ]; deforestation, urbanisation and other forms of land use change [ 15 , 16 ]; bushmeat consumption [ 17 , 18 , 19 ]; and a variety of human practices including hunting, farming, animal husbandry, mining, keeping of exotic pets and trade [ 8 , 9 , 20 , 21 , 22 ]. These large-scale changes have repeatedly given rise to spillover events [ 2 , 15 , 23 ], sometimes involving pathogens with epidemic or pandemic potential [ 24 ].

Spillover pathway adapted from Plowright et al. [ 12 , 13 ]

The responsibility for addressing zoonotic disease frequently spans multiple sectors of governance due to its relevance for both animals and humans. A One Health perspective, which recognises the health of humans, animals and the environment as being closely linked and inter-dependent [ 25 ], can be useful in understanding the spillover pathway and drivers of spillover events, as well as informing policy and governance approaches to address this cross-sectoral problem. At the international level, the World Health Organization, the Food and Agriculture Organization, the World Organisation for Animal Health and the United Nations Environment Programme have endorsed a One Health approach to policymaking to respond to zoonotic infectious diseases, emphasising collaboration between agencies [ 26 ].

Operationalising a One Health approach to policy

While One Health is a promising approach to preventing zoonotic EIDs, operationalising this concept remains a challenge. Evaluative evidence exists around the effectiveness of interventions to prevent spillover events [ 13 , 27 , 28 , 29 ], however these have often been implemented as short- to medium-term programmes or academic investigations [ 8 ]. In some cases, zoonoses have re-emerged after successful programmes have ended [ 29 ]. As a result, experts have argued for the incorporation of successful interventions into policy frameworks, providing interventions with the sustainability required for long-term disease control [ 8 , 10 ].

Operationalising a One Health approach to policy involves understanding the policy options, identifying the stakeholders involved and developing insights into how to successfully implement and evaluate these policies. Although the longevity and scope of government actions may make policy an effective vehicle for prevention of emerging diseases, implementing policy is a complex process involving numerous actors with competing views and interests [ 30 ]. This context presents challenges for policy development and implementation. Where relevant policies are designed and implemented in isolation, opportunities for co-benefits may be missed and interventions may produce unintended consequences [ 31 ]. Finally, while evaluative evidence is key to informing future policy decisions, the complex systems in which policies are often implemented make evaluation challenging [ 32 ].

Aims and scope

To provide insights around how to use policy to successfully prevent zoonotic spillover events, it is necessary to synthesise the available evaluative evidence. A One Health perspective allows this evidence synthesis to incorporate a wide range of policy instruments and actors and to identify approaches to successfully implementing and evaluating policies in this complex, multi-sectoral context.

Approaches to managing epidemic and pandemic infectious pathogens when they have entered human populations have been systematically catalogued in the medical literature [ 33 , 34 , 35 , 36 , 37 , 38 , 39 ]. These measures include hand washing, face masks, school closures, contact tracing, vaccination and case isolation. Further upstream, systematic reviews of interventions targeting the spillover pathway have predominantly focused on programmes rather than policies, and have been restricted by various characteristics such as geographic region [ 28 ] or pathogen type [ 29 ], or focused on programmes with an explicit endorsement of a One Health approach [ 27 ]. In consequence, a comprehensive understanding of what policies to prevent zoonotic spillover have been evaluated, what actors are involved, and how to successfully implement and evaluate them, is lacking. To address these research gaps, our objective was to synthesise the existing evaluative evidence around policies that target the determinants of zoonotic spillover.

Our approach to identifying and analysing this literature was informed by a One Health perspective, acknowledging the inter-connectedness of human, animal and environmental health.

We conducted a systematic scoping review of evaluations of policies aimed at preventing zoonotic spillover events, based on a previously published protocol [ 40 ]. Results are reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews [ 41 ]. The scoping review was conducted in line with guidelines published by Arksey and O’Malley and refined by Levac and colleagues [ 42 , 43 , 44 ], which emphasise an iterative approach suited to an exploratory research question.

The One Health perspective guided the development of the review methodology. This included the search strategy and inclusion criteria, which allow for the inclusion of policies focused on human, animal or environmental health (or any combination of these areas) and with leadership from one or more of these sectors, and the research questions, which seek to outline the policies and the range of sectors involved in implementation. While our focus on the spillover pathway meant we only included policies that had been evaluated in terms of their impacts on animal and human population distributions, health and interactions, we explicitly searched for environment-focused policies (e.g., protection of wetlands and other wildlife habitats) that might have been evaluated from this perspective. We also aimed to interrogate the One Health approach to governance, by assessing to what extent cross-sectoral collaboration – a key tenet of One Health practice [ 25 ] – emerged as a reason for policy success.

Stage 1: identifying the research question

Informed by our research objective, our research questions were:

What policies aimed at preventing zoonotic spillover (i.e., policies that target the determinants of zoonotic spillover included in the spillover pathway [ 12 ]: population distribution, health and interactions) have been evaluated?

What are the types of policies?

Which policy actors (single department, multi-sectoral, whole of government) are involved?

What are the reasons for policy success and failure, and the unintended consequences of implementing these policies?

How has evaluation of these policies been approached in the literature?

What are the methods or study designs used?

What are the outcomes?

What are the opportunities and challenges for evaluation?

Stage 2: identifying relevant studies

We systematically searched four electronic databases (Medline, Scopus, Web of Science, Global Health) in May 2021. The search strategy was organized by the main concepts in our research question: the spillover pathway; public policy; prevention; and zoonotic pathogens. The search strategy was developed iteratively, informed by existing systematic reviews focused on related concepts [ 28 , 45 , 46 , 47 , 48 , 49 ] and known indicator papers meeting inclusion criteria. We also searched the websites of 18 organisations involved in the prevention of zoonotic spillover to identify relevant grey literature. The choice of organisations was informed by an actor mapping exercise in which we identified key international organisations working on the prevention of emerging zoonoses using network sampling [ 50 ]. We searched the websites of a subset of these organisations, focusing on inter-governmental organisations and organisations whose main focus was zoonotic disease. See Supplementary File 1 for details of academic database and grey literature search strategies.

Stage 3: study selection

Studies were included if they met the following criteria:

Primary empirical study with an English-language abstract from any country or region (reviews were excluded);

Study reporting empirical findings from an evaluation of any sort; and.

Study focused on a policy implemented by government that targets the determinants of zoonotic spillover.

Academic records identified through the searches were collated and double screened using the online platform Covidence [ 51 ]. Two researchers (CCA and KML) initially screened titles and abstracts. Title and abstract screening of an initial set of 100 papers was undertaken by both researchers independently. Results were compared to ensure consistency in decisions around study eligibility, and discrepancies were resolved through consensus. This process was repeated until an acceptable level of agreement (> 90%) was reached. The remaining papers were then screened by one of the two reviewers. Full-text screening was undertaken by two independent researchers and discrepancies were resolved by consensus. Studies with full-texts in any language were eligible for inclusion if they include an English-language abstract. Full-text studies published in French, Spanish or Chinese were single-screened by a member of the research team fluent in that language (CCA or AY). Studies published in other languages were translated as necessary.

Grey literature was screened by one researcher (CCA) to determine whether it met the inclusion criteria. Publications were initially screened by looking at titles, tables of contents and executive summaries. Where these indicated that the publication might be eligible, documents were read in full to determine if inclusion criteria were met.

In line with published guidelines, the approach to study selection was refined iteratively when reviewing articles for inclusion [ 42 , 43 , 44 ].

Stage 4: charting the data

Data charting was conducted using a form designed to identify the information required to answer the research question and sub-research questions (see Supplementary File 2). Data charting focused on characteristics of the study, the policy, and the evaluation. For each policy, this included identifying which determinant of zoonotic spillover situated along the spillover pathway was being targeted. For the purpose of this study, we used a model of the spillover pathway adapted from Plowright et al.’s work [ 12 , 13 ], in which we differentiated between wildlife and domesticated animals (Fig.  1 ). This differentiation is important in the policy context, as the wildlife-domesticated animal interface is an important site for intervention, as well as the human-animal interface.

The data charting form was piloted with ten records to ensure that it was consistent with the research question, and revised iteratively [ 42 , 43 , 44 ]. Data charting was conducted by one researcher (CCA, RM, JC, AD or PS) and checked by a second researcher (CCA or KML). Discrepancies were resolved by consensus.

Stage 5: collating, summarising and reporting the results

Our protocol stated that we would use the Quality Assessment Tool for Quantitative Studies developed by the Effective Public Health Practice Project [ 52 ] to assess study quality [ 40 ]. However, on reviewing the included studies we selected two tools that were more appropriate to their characteristics: (1) ROBINS-I [ 53 ] for quantitative outcome evaluations and (2) a tool developed by the authors of a previous review [ 54 ] – based on Dixon-Woods et al.’s approach to assessing study credibility and contribution [ 55 ] – for all other study types. Two researchers (CCA and KML) assessed study quality independently for an initial set of 10 studies, before comparing assessments and reaching agreement where discrepancies occurred. This process was repeated until an adequate level of agreement was reached (> 90%). The remaining studies were assessed by a single researcher (CCA or KML). Records were not excluded based on quality assessment. Instead, assessments were primarily used to help synthesize the literature on how policies were evaluated. Quality assessment was not performed on grey literature due to the wide variability in the format and comprehensiveness of included publications.

We analysed the charted data, presenting a numerical summary of the included studies in table form, allowing us to describe the range of policy interventions that have been evaluated, aspects of policy implementation and approaches to evaluation. Based on the charted data, we inductively grouped evaluated policies with similar characteristics into policy types and assigned a policy instrument to each policy type: communication/marketing, guidelines, fiscal, regulation, legislation, environmental/social planning or service provision. We mapped policy types onto the spillover pathway shown in Fig.  1 to outline the policies that have been used to target each of these determinants. Thematic analysis was conducted using the approach described by Braun and Clarke where the focus is guided by the researcher’s analytic interests [ 56 ], with five overarching themes chosen as an a priori coding framework: (1) reasons for policy success; (2) reasons for policy failure; (3) unintended consequences of policy implementation; (4) opportunities for policy evaluation; and (5) challenges for policy evaluation. We selected these themes based on our research questions and previous familiarisation with the included articles during the process of article selection, data extraction and quality assessment. Sub-themes were subsequently identified through close reading and coding of the included articles. Thematic analysis was conducted by one researcher (RM) using the qualitative data analysis software Dedoose [ 57 ] and reviewed by the lead author (CCA).

Study characteristics

After removing duplicates, our searches identified a total of 5064 academic records. After screening titles and abstracts, we considered 330 records for full-text review. We also identified 11 relevant publications through our grey literature search. Grey literature reports were published by five organisations: four organisations focused on health and disease, including an intergovernmental organisation (the World Organisation for Animal Health) and three non-governmental organisations (the One Health Commission, the Global Alliance for Rabies Control and EcoHealth Alliance); and one non-governmental organisation focused on wildlife trade (TRAFFIC). In total, we included 95 publications in this review (PRISMA diagram in Fig.  2 ) [ 58 ].

We excluded studies which assessed the unintended consequences of policies to prevent zoonotic spillover without evaluating their effectiveness. This included studies that looked exclusively at the mental health impacts of mandatory livestock culls on farm workers [ 59 ]; studies which focused on potentially relevant factors, such as the wildlife trade, but with no consideration of outcomes situated on the spillover pathway [ 60 ]; and studies which assessed the detection power of surveillance systems without assessing the impact of associated policy interventions [ 61 , 62 , 63 ].

Policy characteristics

The characteristics of the policies evaluated in the included studies are presented in Supplementary File 3 and summarised in Table  1 . Some studies evaluated more than one policy, particularly modelling studies which compared the impacts of several policy options and process evaluations focused on a range of activities undertaken by a single government. Therefore, the number of evaluated policies (n = 111) is greater than the number of included studies (n = 95).

Most policies were evaluated for their impact on human exposure (21%), pathogen prevalence in domesticated animals (18%), barriers within domesticated animals (15%), and pathogen survival and spread in domesticated animals (9%). There were also a number of multi-component policies studies across multiple stages of the spillover pathway (18%). Fewer studies focused on wildlife health and populations, and none of the included studies evaluated policies for their impact on infection intensity and pathogen release in either domesticated animals or wildlife.

Where the government department responsible for implementing a policy was identified in the paper, most policies were implemented by a single department (35%), although there were a number of multi-sectoral efforts (24%). The range of government sectors responsible for implementing policies to prevent zoonotic spillover included human health, animal health, food safety, agriculture, conservation, national parks, forestry, fisheries, environmental protection, border control and foreign affairs. Policies were predominantly intended to be implemented by private sector actors, including individuals and organisations working in trade, retail, hunting and animal agriculture. However, some policies were also implemented by public sector actors working in public health, veterinary public health and environmental conservation.

Most policies were situated in high-income (49%) and upper middle-income (28%) countries, with studies from East Asia and the Pacific (43%) and Europe and Central Asia (19%) dominating. Publications focused on policies targeting various zoonotic diseases, with the most common being avian influenza (50%), rabies (19%), brucellosis (11%) and Hendra virus (4%).

Most policies were evaluated using process (38%) or outcome (31%) evaluation. The most frequently used policy instrument was legislation (59%), particularly for managing pathogen spread in domesticated animals through measures such as mandatory vaccination, culls or disinfection protocols. Meanwhile, communication and marketing or service provision was more typically used to reduce risk in wildlife and human populations, for example by providing guidance around recommended hygiene protocol, by distributing oral vaccination in wildlife habitat or by offering vaccination to human populations.

PRISMA 2020 diagram [ 58 ]

What policies aimed at preventing zoonotic spillover have been evaluated?

Policy types targeted different determinants across the pathway to zoonotic spillover and used various approaches with different evidence of success (Table  2 ). We identified policy options including culling – both general and targeted – of wild and domesticated animals; habitat protection (limiting activities such as agriculture and animal husbandry in wildlife habitats); supplemental feeding to control wildlife movements; vaccination of both wildlife, domesticated animals and human populations with occupational exposure to animals; policies to improve biosecurity in sites where animals are kept, slaughtered and sold, including mandates and information campaigns; live animal market closures; and bans on hunting and selling wildlife. Where outcomes or impacts were evaluated, most policies saw some level of success (i.e., outcome measures were found to vary in a direction that indicated policy success), though relative effectiveness was not assessed due to variation in study design and outcome measure. Policies with consistent evidence of effectiveness – where outcome measures varied in a direction that indicated policy success in all studies included in the review – included culling and sterilisation of wildlife populations, habitat protection, vaccination in wildlife and domesticated animal populations and mandated disinfection protocols. Policies with equivocal evidence of success (i.e., outcome measures varied in different directions or studies had different findings, some indicating success and some indicating failure) included supplemental feeding of wildlife, pre-emptive livestock culls, live animal market closures and bans on wildlife hunting, trade and consumption. For many policies, there were no impact or outcome evaluations identified in this review.

What are the reasons for policy success?

The evidence from the identified impact and outcome evaluations suggests that most of the policies succeeded to some extent. A range of factors contributed to policy success. First, studies emphasized the importance of effective collaboration and coordination between various agencies, disciplines, and levels of government in the execution of policy directives [ 114 , 115 ], in line with a One Health approach to policy and governance. Policy success was attributed, in part, to strong working relationships that encouraged effective communication between various government agencies, and facilitated timely and appropriate policy responses [ 115 ]. Synergy between agencies responsible for surveillance and the execution of control strategies was also reported to be beneficial. For example, prompt communication and effective collaboration between laboratories testing samples and agencies implementing culls in the field was seen as important in the control of highly pathogenic avian influenza in Nigeria [ 116 ]. Similarly, authors also identified the importance of private-public relations and private sector contributions to implementing policies to prevent zoonotic spillover [ 112 ]. This included stronger government engagement with private veterinarians as a factor for success in reducing the spillover of Hendra virus in Queensland [ 109 ], and with farmers, poultry companies and national farming and poultry processing associations in Ghana as part of a successful campaign to reduce risk from highly pathogenic avian influenza [ 112 ]. Studies suggest that the inclusion of private sector stakeholders in the policy process has the potential to improve compliance through transparent dialogue around disease ecology, risk and risk mitigation [ 90 , 91 , 103 , 117 ]; and highlight the utility of participatory approaches in prompting behaviour changes [ 91 ].

Second, authors emphasised the significance of economic incentives, suggesting that policy impact is dependent on private actors’ appraisal of costs and benefits. Studies illustrated how incentives, including compensation, subsidies, rebates, and fines, have had varying degrees of success [ 91 , 97 , 112 , 115 ]. Compensation levels [ 104 , 114 ] and enforcement practices [ 92 ] were identified as salient factors for compliance and adherence. For example, fear of sanctions for bushmeat hunting while a ban was in place in some parts of West Africa were identified as a stronger incentive to avoid bushmeat hunting than the fear of contracting Ebola virus [ 97 ]. Culls were seen as particularly challenging in this regard: while the long-term benefits for farmers may outweigh the financial loss [ 104 ], authorities need to be conscientious of the substantial economic impacts when considering policies that mandate culling or safe disposal [ 95 ]. The direct losses related to compliance (time, labour and expenses) and indirect losses due to price fluctuations and decreases in trade volume, as well as losses to associated industries, are substantial [ 88 , 96 , 113 , 118 ].

Third, trust in government and public support for implemented policy were specified as critical factors influencing the effectiveness of disease control strategies, and research suggests that strategic engagement to facilitate compliance is a necessary step in the policy process [ 97 ]. Participatory approaches that attempt to identify and understand factors influencing compliance have been consistently used to overcome resistance to policy, as insights from engagement and consultation can lead to solutions that facilitate behaviour change at the population level [ 91 , 103 ]. For example, a World Health Organization initiative to reduce avian influenza transmission in poultry markets in Indonesia worked alongside market vendors to achieve its aims, carrying out repeated consultations with the vendors and implementing market infrastructure (such as energy and running water in the market) in collaboration with local authorities to support vendor behaviour change [ 91 ].

Fourth, studies also demonstrated the importance of public communication. The quality of information, as well as the volume, complexity and delivery of public health messages, were key factors [ 75 , 114 ]. Authors contend that communication strategies must understand the target audience and how they interpret and engage with messages [ 97 ], for example by building on relationships where there is exiting trust, such as between veterinarians advising animal vaccination and animal owners [ 117 ]. Homogenously delivered communication strategies were ineffectual: they limited opportunities for open discourse; discounted contradictory lived experiences and expressions of uncertainty; and ultimately contributed to scepticism surrounding implemented policies [ 97 , 117 ].

Finally, studies underscored the importance of surveillance infrastructure to inform intervention strategies. Surveillance programs with the ability to collect and operationalize relevant data were essential to the development of appropriate interventions that are responsive to each unique context [ 115 , 119 ]. Implementing effective surveillance programmes requires the appropriate evaluation tools [ 120 ] and trained personnel [ 81 ].

What are the reasons for policy failure?

Studies showed that perceptions of acceptability and appropriateness were crucial to the effectiveness of implemented policies [ 101 , 104 ]. Several factors were identified that negatively affected acceptability and appropriateness, including: additional expenses for private sector actors without sufficient support [ 75 , 100 , 104 , 112 , 114 ], particularly were culls were demanded but reimbursement for farmers was slow and inadequate, as in a brucellosis eradication campaign in Macedonia [ 81 ]; lack of affordable alternatives [ 97 ]; impracticality of implemented strategies [ 75 , 101 ]; lack of cultural understanding in designing policy interventions [ 97 , 100 ], for example the distribution of footwear to pig farmers in a Polynesian context where footwear was not traditionally worn [ 100 ]; lack of understanding of viral ecology [ 100 ]; as well as public scepticism and distrust [ 97 , 114 ].

Additionally, policy ineffectiveness was associated with poor planning and execution of intervention strategies, including lack of clear direction [ 114 ]; incomplete or inconsistent implementation of control measures (17); limited scope of intervention [ 114 ]; and poor enforcement [ 92 ]. A lack of adequate resources to implement strategies also contributed to policy failure [ 81 ]. Adequate financial resources were necessary to hire and train staff to run surveillance and control operations [ 81 ]. Financial resources were also necessary to fund compensation mechanisms that facilitate compliance. Willingness to adopt policy-prescribed disposal practices was found to be associated with compensation levels (incentives) as a proportion of production price, dependency on income from activities driving zoonotic risk, and contact with prevention staff [ 92 ].

What are the unintended consequences of implementing policies to prevent zoonotic spillover?

A small number of the included studies collected data on the unintended consequences of policies to prevent zoonotic spillover (n = 18). In some instances, unintended consequences were due to disease ecology or human behaviour as a result of policy failure. For example, a study assessing the impacts of the closure of a live poultry market found that, following the closure, vendors travelled to neighbouring markets to sell their animals [ 94 ]. As a result, while cases of avian influenza decreased in the area surrounding the closed market, cases increased in these neighbouring markets, leading to the wider geographic spread of the disease. In another study, elk were provided with supplementary feeding grounds to discourage them from coming into contact with the livestock who shared their range [ 65 ]. While this intervention had the intended consequence of reducing the transmission of brucellosis between elk and livestock, the spread of brucellosis between the elk using the supplementary feeding grounds – who were gathering in larger, tighter groups for longer periods, resulting in higher within-herd transmission – and other elk populations in the area increased. This resulted in an increasing prevalence of brucellosis among the elk, potentially increasing the risk of spillover to livestock. These examples illustrate the complexity of the social and ecological systems in which these policies are implemented, further suggesting the need for a One Health approach to policies to prevent zoonotic spillover.

A key unintended consequence can be attributed to the loss of profits and livelihoods sometimes associated with policies to prevent zoonotic spillover, as described above. The losses incurred by complying with regulations made farmers, hunters and other private sector actors reluctant to report potential infections, contributing to increased unauthorized or illegal activity, and unrestrained spread of disease [ 90 , 92 , 94 , 98 , 112 , 114 ]. Studies investigated the creative ways policy enforcement was circumvented, including hiding hunting equipment on the outskirts of towns or developing informal trade markets and networks [ 97 , 98 ]. Unintended consequences identified in the included evaluations emphasize an opportunity for policymakers to improve sector compliance through public education, levying the influence of consumer attitudes on industry standards [ 104 , 113 ].

A range of study designs were used to evaluate policies. Outcome evaluations (n = 33) used time series or repeat cross-sectional data to conduct evaluations of natural experiments, though most studies did not include a control group for comparison. Outcome evaluations also used case-control and modelling approaches to assess policy impact on an outcome of interest. Process evaluations (n = 30) used cross-sectional and qualitative approaches, as well as study designs combining multiple sources of data, to understand aspects of policy implementation such as the extent to which the policy was being implemented as designed, and the responses and attitudes of stakeholders involved in policy implementation. Economic evaluations (n = 11) included cost-benefit analyses, risk-benefit analyses and modelling studies. Formative evaluations (n = 17) used modelling approaches to estimate what the impacts of a proposed policy option would be in a specific context.

Outcome variables interpreted as indicators of policy success were also numerous and represented determinants along the spillover pathway. As expected, many studies assessed impact on disease transmission, including disease prevalence and incidence, disease eradication, case numbers, and basic reproduction number in human and animal populations, as well as evidence of disease in environmental samples, such as in live animal markets or at carcass disposal sites. Studies also assessed impacts on intermediate factors indicative of successful implementation of specific policies, such as the availability of wild species in markets where a trade ban had been implemented, or knowledge and practices of stakeholders in response to an educational or information campaign.

While most studies found a reduced risk of zoonotic spillover following policy implementation, comparing the magnitude of these impacts was challenging due to the variety of study designs and outcome measures used in the included studies. However, we identified several studies which used modelling to directly compare the impacts of policy options. These studies evaluated various policy scenarios: different combinations within multi-component policy interventions [ 121 ]; culling versus vaccinating wildlife [ 122 ] and livestock [ 84 , 85 ] populations; targeting strategies to humans exclusively versus targeting humans and livestock [ 108 ]; and altering the parameters for culling and vaccination strategies, for example by modelling different ranges for culling and vaccination near infected farms [ 85 ]. These studies often highlighted trade-offs between the effectiveness of policy measures and their cost. For example, estimates of the number of infected flocks were lower when incorporating a ring cull (cull of animals on farms surrounding an outbreak) into a multi-component control strategy for highly pathogenic avian influenza [ 121 ]. However, livestock vaccination was estimated to be a highly effective strategy, with one study findings livestock vaccination to be as or more effective than a pre-emptive cull for outbreak control purposes (depending on the extent of vaccination coverage), while minimising the number of animals culled [ 85 ]. One study jointly modelled costs and benefits of strategies, and found that livestock vaccination had a higher cost-benefit ratio than a wildlife cull [ 122 ]. A final study highlighted the potential of holistic approaches, with drug administration in humans and livestock having a lower cost per disability-adjusted life year averted than intervention in humans alone [ 108 ].

Study authors noted a number of challenges encountered while evaluating policies to prevent zoonotic spillover. One study noted the difficulty of determining the impact of policies aiming to reduce spillover events between wildlife, livestock and humans, as the number of spillover events is often relatively small [ 65 ]. This highlights the importance of considering upstream determinants and risk factors as outcome measures in attempting to evaluate these policies, particularly where spillover events may happen infrequently or not at all during the period of observation. Studying changes in risk factors for spillover can provide insight on the effectiveness of different policies in tackling spillover risk.

Lack of suitable data was a frequently cited barrier to policy evaluation. As policies to prevent zoonotic spillover are often reactive, being implemented in response to an outbreak in animal populations, accessing data from before a policy was implemented was challenging. Studies highlighted the value of routinely collected data, which was often the only data available and was frequently used for policy evaluation [ 65 , 66 , 94 , 115 , 119 , 123 ]. However, in many contexts routine data on animal health is not collected [ 80 ]. Routine testing data from livestock can sometimes be used for evaluation where it exists, but it does not always provide sufficient detail for examining the potential for a policy to prevent zoonotic spillover. For example, some tests do not differentiate between current and past infection, making it difficult to identify where and when spillover occurred [ 65 ], and animal health data may not be granular enough for policy evaluation, particularly in terms of evaluating local policies [ 94 ]. Studies also highlighted instances where the private sector may own data sets reporting disease prevalence and transmission, but may be reluctant to share the data for evaluation purposes [ 121 ]. In such instances, open communication and good relationships with the private sector may be facilitators to evaluation.

Beyond the lack of baseline data, studies highlighted the difficulty in collecting information about policy compliance. As failing to comply often puts farmers and hunters at risk of fines or imprisonment, they were reluctant to disclose information about non-compliance or participation in illegal trade and sale of animals [ 86 , 92 , 97 , 112 ]. This made it difficult to determine policy effectiveness.

Quality assessment

Of the 44 quantitative evaluations, 37 were evaluated as being at moderate or higher risk of bias (see Supplementary File 4), given the possibility of bias in the assessment of intervention impact due to the presence of confounding effects. A small number of studies were determined to be at serious (n = 6) or critical (n = 1) risk of bias, for two main reasons: only having data from after the intervention was implemented; or using a case-control study model without measuring and adjusting for important potential confounders, such as the prevalence of a targeted disease prior to policy implementation. These limitations may reflect the nature of zoonotic spillover events and policy responses, which can happen quickly and leave little time for baseline data collection. Many of the included studies relied on surveillance data, but where such data sets are not available, post-test and case-control study designs may be the only options.

The quality of studies assessed with the tool developed based on Dixon-Woods’ approach [ 55 ] was high overall (n = 41, see Supplementary file 5). Most studies were rated as high in terms of clearly and comprehensively presenting their results (n = 37), analysis (n = 34), research design (n = 33), aims (n = 32) and research process (n = 28). Most studies also had a high relevance to the research question (n = 31), indicating that the research was embedded in policy, being commissioned, co-designed or conducted in partnership with government stakeholders.

We identified a range of policies targeting different parts of the spillover pathway implemented by various policy and governance sectors, including some multi-sectoral initiatives. Policies tended to rely heavily on private sector actors (including actors ranging from small-scale farmers and hunters to larger commercial operations) for implementation, suggesting that open communication and collaboration with these actors was essential for successful policy implementation. Policy success was undermined by lack of collaboration between government agencies; lack of communication between surveillance and control operations; poor understanding of the context in which policies were implemented; and inadequate financial compensation for private sector actors who lost profits and incurred additional costs by complying with policies. Where policies were ineffective, this tended to be due to unintended consequences relating to complex dynamics within the social and ecological systems where policies were implemented. Lack of appropriate data was a key obstacle to policy evaluation, and studies emphasised the importance of robust surveillance infrastructure in evaluating policies that tended to be implemented reactively, in response to an outbreak of zoonotic disease in animal or human populations.

Implications for policy and practice

The key role that the private sector and industry actors play in implementing policies to prevent zoonotic spillover is an important consideration for policymakers. Our findings suggest that many of these policies must be complied with by farmers – from subsistence and smallholder farmers to large corporations – as well as by other actors, such as hunters. Lack of awareness as well as financial costs of compliance among these groups present key barriers to policy success in this area. This set of stakeholders is complex as some may make very marginal profits, if any, and may struggle to afford the additional costs of implementing preventive policies. However, powerful actors and profitable industries are also involved, including large-scale farms and primary resource extraction enterprises [ 22 ]. Acknowledging the differences across these stakeholder groups, and in particular assessing their capacity to bear some of the costs related to prevention, emerges as crucial in successful policy implementation.

Finally, our findings highlight the importance of disease surveillance in efforts to reduce the risk of spillover events. As well as acting as an early warning system, surveillance provides a source of data to evaluate the impact of preventive policies. We found the availability of surveillance data to be a key enabling factor in evaluating policies. In addition, close collaboration between agencies responsible for disease surveillance and control efforts was key to policy success. National surveillance efforts, as well as cross-country collaboration to support global efforts, such as the United States Agency for International Development’s PREDICT program supporting surveillance in areas at high risk for zoonotic disease outbreaks [ 124 ], must be sustained and expanded. In complex areas such as the prevention of zoonotic spillover, approaches to surveillance which encompass risk factors and transmission pathways [ 125 ], as well as One Health surveillance systems which harmonise and integrate data collection and analysis from across human, animal and environmental sectors [ 126 ], are promising approaches to developing surveillance systems that support risk. This context also involves a need to strengthen surveillance capacity in remote and rural locations, as communities living in these contexts may have exposure to numerous pathogens of wildlife origin. This will require strengthening clinical and diagnostic capacity in these settings, as well as engaging with stakeholders such as community human and animal health workers and wildlife or national park rangers [ 127 ].

Comparison with existing literature

This review sought to map the range of policies implemented to reduce the risk of zoonotic spillover, and the various approaches taken to evaluation, and identify factors behind the success and failure of policy implementation and evaluation. Due to this broad scope, comparing relative effectiveness of policy interventions was challenging. Existing systematic reviews with a more specific focus could apply meta-analysis to determine which interventions were most effective. For example, a review of market-level biosecurity measures aiming to reduce the transmission of avian influenza found that reducing market size, separating poultry species, cleaning and disinfecting premises, closing markets and banning overnight storage were highly effective interventions [ 45 ]. However, our findings suggest that studies focused on the control of avian influenza dominate the literature in this space (55 out of 111 evaluated policies), and many of these are focused on market-level measures. Systematic reviews focused on other approaches to reduce spillover risk, such as on-farm biosecurity [ 47 ]; biosecurity for backyard poultry rearing [ 46 ]; and community-based interventions [ 28 ] comment on the paucity of high-quality evidence around the impacts of such approaches. By taking a broad perspective, we hope our findings will provide policy options for consideration in a number of contexts, and guide researchers in focusing their efforts on areas where evidence is lacking.

Strengths and weaknesses of the study

To our knowledge, this is the first attempt to systematically identify and document evaluations of policies aiming to prevent the spillover of zoonotic pathogens into human populations. However, because of the complex drivers of spillover events, some potentially relevant policy evaluations may be excluded where their outcome measures are too far removed from zoonotic spillover. While relevant, such evaluations will be difficult to systematically identify as they make no reference to zoonotic disease.

In addition, this review focused on policy evaluations that have been reported in the peer-reviewed literature and the grey literature published by international agencies and organisations working on these topics. Policies that have been implemented but not evaluated, or evaluated but not published in these literatures, will therefore be excluded from this review. As a result, potentially effective and important policies in the prevention of zoonotic spillover events may not have been identified. However, we hope that the findings from this review will highlight these gaps in the evaluative evidence. We also hope that this review, by extracting practical dimensions, such as study design, outcome measures and the challenges encountered in the evaluation process, will support policymakers and researchers in carrying out further policy evaluations in this space.

Unanswered questions and future research

Our findings highlight several important gaps in the evidence. First, while observational evidence emphasises the importance of upstream determinants such as environmental and ecosystem health in the increasing rate of zoonotic spillover [ 1 , 15 ], we only identified a single evaluation of a policy attempting to target one of these upstream determinants: an evaluation carried out in China to assess the impact of the Ramstar wetland protection program on avian influenza in migratory waterfowl [ 66 ]. This study found that proximity to protected wetlands reduced outbreak risk. Authors hypothesised that this effect was due to the separation of wild waterfowl and poultry populations and the diversion of wild waterfowl away from human-dominated landscapes and toward protected natural habitats. Our findings support existing calls for more quantitative and mechanistic studies of the impact of interventions supporting environmental and ecosystem health on zoonotic spillover risk [ 128 ], as well as calls for greater integration of the environment into One Health research, policy and practice [ 31 ]. Further evaluations of environment and habitat protection policies would strengthen our understanding of this area. In addition, the impact of policies to reduce deforestation or expand forest coverage, such as China’s Grain-to-Green program [ 129 ], on the spillover pathway could be evaluated. Such evaluations might consider potential unintended consequences, as these policies could promote healthier wildlife populations with better disease resistance, but may also facilitate wildlife population growth and higher rates of wildlife-human encounters [ 130 ].

There is also a lack of evaluation of policies targeting infection intensity and pathogen release in either wildlife or domesticated animals. These could include approaches such as improving animal health and welfare to make these populations more resistant to disease [ 13 ]. While arguments have been made for strengthening legal structures supporting animal welfare in order to reduce the risk of zoonotic pathogen transmission [ 131 ], there is a need to evaluate policies that take this approach.

Our review found publications evaluating a wide range of policy interventions spanning the spillover pathway, including habitat protection; trade regulations; border control and quarantine procedures; farm and market biosecurity measures; public information campaigns; and vaccination programmes for wildlife and domesticated animals, as well as human populations with occupational exposure to animals. A wide range of governance sectors implemented these policies, highlighting the prevention of zoonotic spillover as a cross-sectoral issue, though most policies were implemented by a single sector. Our findings highlight the importance of industry and private actors in implementing policies to prevent zoonotic spillover, and the need for thoughtful and effective engagement with this wide range of actors, from subsistence hunters and farmers through to industrial animal agriculture operations to address their concerns through a range of incentives. We also identified the centrality of surveillance data in evaluating policies that are often implemented reactively, and effective collaboration between surveillance and control operations as a central factor in successful policy implementation.

Data Availability

All data generated or analysed during this study are included in this published article and its supplementary information files. Analysis code for descriptive characteristics of included policies is available on GitHub.

Abbreviations

Emerging infectious disease

Morse SS, Mazet JA, Woolhouse M, Parrish CR, Carroll D, Karesh WB, Zambrana-Torrelio C, Lipkin WI, Daszak P. Prediction and prevention of the next pandemic zoonosis. The Lancet. 2012;380:1956–65.

Article   Google Scholar  

Pulliam JRC, Epstein JH, Dushoff J, Rahman SA, Bunning M, Jamaluddin AA, Hyatt AD, Field HE, Dobson AP, Daszak P. Agricultural intensification, priming for persistence and the emergence of Nipah virus: a lethal bat-borne zoonosis. J Royal Soc Interface. 2012;9:89–101.

IPCC. In: Pörtner H-O, Roberts DC, Tignor M, Poloczanska ES, Mintenbeck K, Alegría A, Craig M, Langsdorf S, Löschke S, Möller V, Okem A, Rama B, editors. Climate Change 2022: impacts, adaptation and vulnerability, contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. In press ed. Cambridge University Press; 2022.

Brenner N, Ghosh S. Between the colossal and the catastrophic: planetary urbanization and the political ecologies of emergent Infectious Disease. Environ Plan A. 2022;54:867–910.

Gallo-Cajiao E, Lieberman S, Dolšak N, et al. Global governance for pandemic prevention and the wildlife trade. Lancet Planet Health. 2023;7:e336–45.

Article   PubMed   PubMed Central   Google Scholar  

Marco MD, Baker ML, Daszak P, et al. Opinion: sustainable development must account for pandemic risk. PNAS. 2020;117:3888–92.

Heymann DL, Dixon M. Infections at the Animal/Human interface: shifting the paradigm from emergency response to Prevention at source. In: Mackenzie JS, Jeggo M, Daszak P, Richt JA, editors. One health: the human-animal-environment interfaces in Emerging Infectious Diseases: Food Safety and Security, and International and National plans for implementation of one health activities. Berlin, Heidelberg: Springer; 2013. pp. 207–15.

Google Scholar  

United Nations Environment Programme, International Livestock Research Institute. (2020) Preventing the next pandemic: Zoonotic diseases and how to break the chain of transmission. 82.

Intergovernmental Science-Policy Platform On Biodiversity And Ecosystem Services (IPBES). (2020) Workshop Report on Biodiversity and Pandemics of the Intergovernmental Platform on Biodiversity and Ecosystem Services (IPBES). https://doi.org/10.5281/ZENODO.4147317 .

One Health theory of change. https://www.who.int/publications/m/item/one-health-theory-of-change . Accessed 30 Jan 2023.

Vinuales J, Moon S, Moli GL, Burci G-L. A global pandemic treaty should aim for deep prevention. The Lancet. 2021;397:1791–2.

Article   CAS   Google Scholar  

Plowright RK, Parrish CR, McCallum H, Hudson PJ, Ko AI, Graham AL, Lloyd-Smith JO. Pathways to zoonotic spillover. Nat Rev Microbiol. 2017;15:502–10.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Sokolow SH, Nova N, Pepin KM, et al. Ecological interventions to prevent and manage zoonotic pathogen spillover. Philosophical Trans Royal Soc B: Biol Sci. 2019;374:20180342.

Johnson CK, Hitchens PL, Pandit PS, Rushmore J, Evans TS, Young CCW, Doyle MM. Global shifts in mammalian population trends reveal key predictors of virus spillover risk. Proc Royal Soc B: Biol Sci. 2020;287:20192736.

Allen T, Murray KA, Zambrana-Torrelio C, Morse SS, Rondinini C, Di Marco M, Breit N, Olival KJ, Daszak P. Global hotspots and correlates of emerging zoonotic Diseases. Nat Commun. 2017;8:1124.

Gandy M. THE ZOONOTIC CITY: Urban Political Ecology and the pandemic imaginary. Int J Urban Reg Res. 2022;46:202–19.

Article   PubMed   Google Scholar  

Hardi R, Babocsay G, Tappe D, Sulyok M, Bodó I, Rózsa L. Armillifer-infected snakes sold at Congolese Bushmeat Markets Represent an emerging zoonotic threat. EcoHealth. 2017;14:743–9.

Steve A-M, Ahidjo A, Placide M-K, et al. High prevalences and a wide genetic diversity of Simian Retroviruses in non-human Primate Bushmeat in Rural areas of the Democratic Republic of Congo. EcoHealth. 2017;14:100–14.

Weiss S, Nowak K, Fahr J, Wibbelt G, Mombouli J-V, Parra H-J, Wolfe ND, Schneider BS, Leendertz FH. Henipavirus-related sequences in Fruit Bat Bushmeat, Republic of Congo. Emerg Infect Dis. 2012;18:1536–7.

Aguirre AA, Catherina R, Frye H, Shelley L. Illicit Wildlife Trade, Wet Markets, and COVID-19: preventing future pandemics. World Med Health Policy. 2020;12:256–65.

Nadimpalli ML, Pickering AJ. A call for global monitoring of WASH in wet markets. Lancet Planet Health. 2020;4:e439–40.

Viliani F, Edelstein M, Buckley E, Llamas A, Dar O. Mining and emerging infectious Diseases: results of the Infectious Disease Risk Assessment and Management (IDRAM) initiative pilot. The Extractive Industries and Society. 2017;4:251–9.

Wegner GI, Murray KA, Springmann M, Muller A, Sokolow SH, Saylors K, Morens DM. Averting wildlife-borne Infectious Disease epidemics requires a focus on socio-ecological drivers and a redesign of the global food system. eClinicalMedicine. 2022. https://doi.org/10.1016/j.eclinm.2022.101386 .

Daszak P. Anatomy of a pandemic. The Lancet. 2012;380:1883–4.

Joint Tripartite (FAO, OIE, WHO) and UNEP Statement. Tripartite and UNEP support OHHLEP’s definition of one health. ” OIE - World Organisation for Animal Health; 2021.

(2022) One Health Joint Plan of Action, 2022–2026. https://doi.org/10.4060/cc2289en .

Baum SE, Machalaba C, Daszak P, Salerno RH, Karesh WB. Evaluating one health: are we demonstrating effectiveness? One Health. 2017;3:5–10.

Halton K, Sarna M, Barnett A, Leonardo L, Graves N. A systematic review of community-based interventions for emerging zoonotic infectious Diseases in Southeast Asia. JBI Database System Rev Implement Rep. 2013;11:1–235.

Article   PubMed Central   Google Scholar  

Meyer A, Holt HR, Selby R, Guitian J. Past and Ongoing Tsetse and Animal Trypanosomiasis Control Operations in five African countries: a systematic review. PLoS Negl Trop Dis. 2016;10:e0005247.

Howlett M, Cashore B. Conceptualizing Public Policy. In: Engeli I, Allison CR, editors. Comparative Policy studies: conceptual and methodological challenges. London: Palgrave Macmillan UK; 2014. pp. 17–33.

Chapter   Google Scholar  

Barrett MA, Bouley TA. Need for enhanced environmental representation in the implementation of one health. EcoHealth. 2015;12:212–9.

Barbrook-Johnson P, Proctor A, Giorgi S, Phillipson J. How do policy evaluators understand complexity? Evaluation. 2020;26:315–32.

Saunders-Hastings P, Crispo JAG, Sikora L, Krewski D. Effectiveness of personal protective measures in reducing pandemic Influenza transmission: a systematic review and meta-analysis. Epidemics. 2017;20:1–20.

Bin Nafisah S, Alamery AH, Al Nafesa A, Aleid B, Brazanji NA. School closure during novel Influenza: a systematic review. J Infect Public Health. 2018;11:657–61.

Viner RM, Russell SJ, Croker H, Packer J, Ward J, Stansfield C, Mytton O, Bonell C, Booy R. School closure and management practices during coronavirus outbreaks including COVID-19: a rapid systematic review. The Lancet Child & Adolescent Health. 2020;4:397–404.

Juneau C-E, Pueyo T, Bell M, Gee G, Collazzo P, Potvin L. (2020) Evidence-Based, cost-effective interventions to suppress the COVID-19 pandemic: a systematic review. medRxiv 2020.04.20.20054726.

MacIntyre CR, Chughtai AA. Facemasks for the prevention of Infection in healthcare and community settings. BMJ. 2015;350:h694.

Smith SMS, Sonego S, Wallen GR, Waterer G, Cheng AC, Thompson P. Use of non-pharmaceutical interventions to reduce the transmission of Influenza in adults: a systematic review. Respirology. 2015;20:896–903.

Jefferson T, Del Mar CB, Dooley L et al. (2011) Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev CD006207.

Astbury CC, Lee KM, Aguiar R, et al. Policies to prevent zoonotic spillover: protocol for a systematic scoping review of evaluative evidence. BMJ Open. 2022;12:e058437.

Tricco AC, Lillie E, Zarin W, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169:467–73.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.

Colquhoun HL, Levac D, O’Brien KK, Straus S, Tricco AC, Perrier L, Kastner M, Moher D. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67:1291–4.

Zhou X, Wang Y, Liu H, Guo F, Doi SA, Smith C, Clements ACA, Edwards J, Huang B, Soares Magalhães RJ. Effectiveness of Market-Level Biosecurity at reducing exposure of Poultry and humans to Avian Influenza: a systematic review and Meta-analysis. J Infect Dis. 2018;218:1861–75.

Conan A, Goutard FL, Sorn S, Vong S. Biosecurity measures for backyard poultry in developing countries: a systematic review. BMC Vet Res. 2012;8:240.

Youssef DM, Wieland B, Knight GM, Lines J, Naylor NR. The effectiveness of biosecurity interventions in reducing the transmission of bacteria from livestock to humans at the farm level: a systematic literature review. Zoonoses Public Health. 2021;68:549–62.

Shi N, Huang J, Zhang X, Bao C, Yue N, Wang Q, Cui T, Zheng M, Huo X, Jin H. Interventions in live poultry markets for the Control of Avian Influenza: a systematic review and Meta-analysis. J Infect Dis. 2020;221:553–60.

Cupertino MC, Resende MB, Mayer NA, Carvalho LM, Siqueira-Batista R. Emerging and re-emerging human infectious Diseases: a systematic review of the role of wild animals with a focus on public health impact. Asian Pac J Trop Med. 2020;13:99.

Clifford Astbury C, Demeshko A, McLeod R, Wiktorowicz M, Gallo Caijao E, Cullerton K, Lee KM, Viens AM, Penney TL. (2023) Governance of the wildlife trade and prevention of emerging zoonoses: a mixed methods network analysis of global organisations. [In preparation].

Covidence - Better. systematic review management. https://www.covidence.org/home . Accessed 17 Jul 2020.

Effective Public Health Practice Project. (2009) Quality Assessment Tool for Quantitative Studies. 4.

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Clifford Astbury C, McGill E, Egan M, Penney TL. Systems thinking and complexity science methods and the policy process in non-communicable Disease prevention: a systematic scoping review protocol. BMJ Open. 2021;11:e049878.

Dixon-Woods M, Cavers D, Agarwal S, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6:35.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3:77–101.

(2021) Dedoose Version 8.3.47, web application for managing, analyzing, and presenting qualitative and mixed method research data.

Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Park H, Chun MS, Joo Y. Traumatic stress of frontline workers in culling livestock animals in South Korea. Animals. 2020;10:1–11.

Programme UNE. Effectiveness of policy interventions relating to the illegal and unsustainable. Wildlife Trade - Policy Brief; 2019.

Cito F, Narcisi V, Danzetta ML, Iannetti S, Sabatino DD, Bruno R, Carvelli A, Atzeni M, Sauro F, Calistri P. Analysis of Surveillance systems in Place in European Mediterranean Countries for West Nile Virus (WNV) and Rift Valley Fever (RVF). Transbound Emerg Dis. 2013;60:40–4.

Schwind JS, Goldstein T, Thomas K, Mazet JA, Smith WA, PREDICT Consortium. Capacity building efforts and perceptions for wildlife surveillance to detect zoonotic pathogens: comparing stakeholder perspectives. BMC Public Health. 2014;14:684.

Reisen WK, Kramer VL, Barker CM. CALIFORNIA STATE MOSQUITO-BORNE VIRUS SURVEILLANCE AND RESPONSE PLAN: A RETROSPECTIVE EVALUATION USING CONDITIONAL SIMULATIONS *. Am J Trop Med Hyg. 2003;68:508–18.

Smith GC, Cheeseman CL. A mathematical model for the control of Diseases in wildlife populations: culling, vaccination and fertility control. Ecol Model. 2002;150:45–53.

Brennan A, Cross PC, Portacci K, Scurlock BM, Edwards WH. Shifting brucellosis risk in livestock coincides with spreading seroprevalence in elk. PLoS ONE. 2017;12:e0178780.

Wu T, Perrings C, Shang C, Collins JP, Daszak P, Kinzig A, Minteer BA. Protection of wetlands as a strategy for reducing the spread of avian Influenza from migratory waterfowl. Ambio. 2020;49:939–49.

Basinski AJ, Nuismer SL, Remien CH. A little goes a long way: weak vaccine transmission facilitates oral vaccination campaigns against zoonotic pathogens. PLoS Negl Trop Dis. 2019;13:e0007251.

Selhorst T. (1999) An evaluation of the efficiency of rabies control strategies in fox (Vulpes 6ulpes) populations using a computer simulation program. Ecol Model 12.

Shwiff SA, Sterner RT, Hale R, Jay MT, Sun B, Slate D. Benefit cost scenarios of potential oral rabies vaccination for Skunks in California. J Wildl Dis. 2009;45:227–33.

García-Díaz P, Ross JV, Woolnough AP, Cassey P. Managing the risk of wildlife Disease introduction: pathway‐level biosecurity for preventing the introduction of alien ranaviruses. J Appl Ecol. 2017;54:234–41.

Hassim A, Dekker EH, Byaruhanga C, Reardon T, van Heerden H. A retrospective study of anthrax on the Ghaap Plateau, Northern Cape province of South Africa, with special reference to the 2007–2008 outbreaks. Onderstepoort J Vet Res. 2017;84:a1414.

Knight-Jones TJD, Gibbens J, Wooldridge M, Staerk KDC. Assessment of Farm-Level Biosecurity Measures after an outbreak of Avian Influenza in the United Kingdom. Transbound Emerg Dis. 2011;58:69–75.

Article   CAS   PubMed   Google Scholar  

Karabozhilova I, Wieland B, Alonso S, Salonen L, Häsler B. Backyard chicken keeping in the Greater London Urban Area: welfare status, biosecurity and Disease control issues. Br Poult Sci. 2012;53:421–30.

Manyweathers J, Field H, Jordan D, Longnecker N, Agho K, Smith C, Taylor M. Risk mitigation of emerging zoonoses: Hendra Virus and Non-vaccinating Horse Owners. Transbound Emerg Dis. 2017;64:1898–911.

Kung N, McLaughlin A, Taylor M, Moloney B, Wright T, Field H. Hendra virus and horse owners - risk perception and management. PLoS ONE. 2013. https://doi.org/10.1371/journal.pone.0080897 .

Rasouli J, Holakoui K, Forouzanfar MH, Salari S, Bahoner, Rashidian A. Cost effectiveness of livestock vaccination for brucellosis in West-Azerbayjan province. Urmia Med J. 2009;20:Pe13–En77.

El Masry I, Rijks J, Peyre M, Taylor N, Lubroth J, Jobre Y. Modelling Influenza A H5N1 vaccination strategy scenarios in the household poultry sector in Egypt. Trop Anim Health Prod. 2014;46:57–63.

Mroz C, Gwida M, El-Ashker M, Ziegler U, Homeier-Bachmann T, Eiden M, Groschup MH. Rift valley Fever virus Infections in Egyptian cattle and their prevention. Transbound Emerg Dis. 2017;64:2049–58.

Pinsent A, Pepin KM, Zhu H, Guan Y, White MT, Riley S. (2017) The persistence of multiple strains of avian influenza in live bird markets. Proceedings of the Royal Society B: Biological Sciences 284:20170715.

Abbas B, Yousif MA, Nur HM. Animal health constraints to livestock exports from the Horn of Africa: -EN- -FR- restrictions sanitaires imposées aux exportations de bétail à partir de la corne de l’Afrique -ES- limitaciones zoosanitarias a las exportaciones de ganado desde El Cuerno De África. Rev Sci Tech OIE. 2014;33:711–21.

Naletoski I, Kirandziski T, Mitrov D, Krstevski K, Dzadzovski I, Acevski S. Gaps in brucellosis eradication campaign in Sheep and goats in Republic of Macedonia: lessons learned. Croat Med J. 2010;51:351–6.

Weaver JT, Malladi S, Bonney PJ, Patyk KA, Bergeron JG, Middleton JL, Alexander CY, Goldsmith TJ, Halvorson DA. A Simulation-based evaluation of Premovement active surveillance Protocol options for the Managed Movement of Turkeys to Slaughter during an outbreak of highly pathogenic avian Influenza in the United States. Avian Dis. 2016;60:132–45.

Andronico A, Courcoul A, Bronner A, Scoizec A, Lebouquin-Leneveu S, Guinat C, Paul MC, Durand B, Cauchemez S. Highly pathogenic avian Influenza H5N8 in south-west France 2016–2017: a modeling study of control strategies. Epidemics. 2019;28:100340.

Backer JA, van Roermund HJW, Fischer EAJ, van Asseldonk MAPM, Bergevoet RHM. Controlling highly pathogenic avian Influenza outbreaks: an epidemiological and economic model analysis. Prev Vet Med. 2015;121:142–50.

Backer JA, Hagenaars TJ, van Roermund HJW, de Jong MCM. Modelling the effectiveness and risks of vaccination strategies to control classical swine Fever epidemics. J R Soc Interface. 2009;6:849–61.

Fournie G, Guitian FJ, Mangtani P, Ghani AC. Impact of the implementation of rest days in live bird markets on the dynamics of H5N1 highly pathogenic avian Influenza. J R Soc Interface. 2011;8:1079–89.

Kung NY, Guan Y, Perkins NR, Bissett L, Ellis T, Sims L, Morris RS, Shortridge KF, Peiris JSM. The impact of a monthly Rest Day on Avian Influenza Virus isolation rates in Retail Live Poultry markets in Hong Kong. Avian Dis. 2003;47:1037–41.

Horigan V, Gale P, Adkin A, Brown I, Clark J, Kelly L. A qualitative risk assessment of cleansing and disinfection requirements after an avian Influenza outbreak in commercial poultry. Br Poult Sci. 2019;60:691–9.

Yuan J, Lau EHY, Li K, et al. Effect of live Poultry Market Closure on Avian Influenza A(H7N9) virus activity in Guangzhou, China, 2014. Emerg Infect Dis. 2015;21:1784–93.

Fournie G, Guitian J, Desvaux S, Cuong VC, Dung DH, Pfeiffer DU, Mangtani P, Ghani AC. Interventions for avian Influenza A (H5N1) risk management in live bird market networks. Proc Natl Acad Sci USA. 2013;110:9177–82.

Samaan G, Hendrawati F, Taylor T, Pitona T, Marmansari D, Rahman R, Lokuge K, Kelly PM. Application of a healthy food markets guide to two Indonesian markets to reduce transmission of avian Flu. Bull World Health Organ. 2012;90:295–300.

Huang Z, Wang J, Zuo A. Chinese farmers’ willingness to accept compensation to practice safe disposal of HPAI infected chicken. Prev Vet Med. 2017;139:67–75.

Graiver DA, Topliff CL, Kelling CL, Bartelt-Hunt SL. Survival of the avian Influenza virus (H6N2) after land disposal. Environ Sci Technol. 2009;43:4063–7.

Li Y, Wang Y, Shen C, Huang J, Kang J, Huang B, Guo F, Edwards J. Closure of live bird markets leads to the spread of H7N9 Influenza in China. PLoS ONE. 2018. https://doi.org/10.1371/journal.pone.0208884 .

Ma J, Yang N, Gu H, Bai L, Sun J, Gu S, Gu J. Effect of closure of live poultry markets in China on prevention and control of human Infection with H7N9 avian Influenza: a case study of four cities in Jiangsu Province. J Public Health Policy. 2019;40:436–47.

Chen Y, Cheng J, Xu Z, Hu W, Lu J. Live poultry market closure and avian Influenza A (H7N9) Infection in cities of China, 2013–2017: an ecological study. BMC Infect Dis. 2020. https://doi.org/10.1186/s12879-020-05091-7 .

Bonwitt J, Dawson M, Kandeh M, Ansumana R, Sahr F, Brown H, Kelly AH. Unintended consequences of the `bushmeat ban’ in West Africa during the 2013–2016 Ebola virus Disease epidemic. Soc Sci Med. 2018;200:166–73.

Brooks-Moizer F, Roberton SI, Edmunds K, Bell D. Avian Influenza H5N1 and the wild Bird Trade in Hanoi, Vietnam. Ecol Soc. 2009;14:28.

Cardador L, Tella JL, Anadon JD, Abellan P, Carrete M. The European trade ban on wild birds reduced invasion risks. Conserv Lett. 2019;12:e12631.

Guerrier G, Foster H, Metge O, Chouvin C, Tui M. Cultural contexts of swine-related Infections in Polynesia. Clin Microbiol Infect. 2013;19:595–9.

Massey PD, Polkinghorne BG, Durrheim DN, Lower T, Speare R. Blood, guts and knife cuts: reducing the risk of swine brucellosis in feral pig hunters in north-west New South Wales, Australia. Rural Remote Health. 2011;11:1793.

CAS   PubMed   Google Scholar  

Lauterbach SE, Nelson SW, Martin AM, Spurck MM, Mathys DA, Mollenkopf DF, Nolting JM, Wittum TE, Bowman AS. (2020) Adoption of recommended hand hygiene practices to limit zoonotic Disease transmission at agricultural fairs. Preventive Veterinary Medicine. https://doi.org/10.1016/j.prevetmed.2020.105116 .

Stewart RJ, Rossow J, Conover JT, et al. Do animal exhibitors support and follow recommendations to prevent transmission of variant Influenza at agricultural fairs? A survey of animal exhibitor households after a variant Influenza virus outbreak in Michigan. Zoonoses Public Health. 2018;65:195–201.

Lin X, Zhang D, Wang X, Huang Y, Du Z, Zou Y, Lu J, Hao Y. Attitudes of consumers and live-poultry workers to central slaughtering in controlling H7N9: a cross-sectional study. BMC Public Health. 2017;17:517.

Huot C, De Serres G, Duval B, Maranda-Aubut R, Ouakki M, Skowronski DM. The cost of preventing rabies at any cost: post-exposure prophylaxis for occult bat contact. Vaccine. 2008;26:4446–50.

De Serres G, Skowronski DM, Mimault P, Ouakki M, Maranda-Aubut R, Duval B. Bats in the bedroom, bats in the Belfry: reanalysis of the rationale for rabies postexposure Prophylaxis. Clin Infect Dis. 2009;48:1493–9.

Vivancos R, Showell D, Keeble B, Goh S, Kroese M, Lipp A, Battersby J. Vaccination of Poultry workers: delivery and uptake of Seasonal Influenza immunization. Zoonoses Public Health. 2011;58:126–30.

Okello AL, Thomas LF. Human taeniasis: current insights into prevention and management strategies in endemic countries. RISK MANAG HEALTHC POLICY. 2017;10:107–16.

Mendez D, Buttner P, Speare R. Hendra virus in Queensland, Australia, during the winter of 2011: veterinarians on the path to better management strategies. Prev Vet Med. 2014;117:40–51.

Häsler B, Howe KS, Hauser R, Stärk KDC. A qualitative approach to measure the effectiveness of active avian Influenza virus surveillance with respect to its cost: a case study from Switzerland. Prev Vet Med. 2012;105:209–22.

Brinkley C, Kingsley JS, Mench J. A Method for Guarding Animal Welfare and Public Health: tracking the rise of Backyard Poultry ordinances. J Community Health. 2018;43:639–46.

Turkson PK, Okike I. Assessment of practices, capacities and incentives of poultry chain actors in implementation of highly pathogenic avian Influenza mitigation measures in Ghana. Vet Med Sci. 2016;2:23–35.

Akunzule AN, Koney EBM, Tiongco M. Economic impact assessment of highly pathogenic avian Influenza on the poultry industry in Ghana. Worlds Poult Sci J. 2009;65:517–27.

Hunter C, Birden HH, Toribio J-A, Booy R, Abdurrahman M, Ambarawati AIGAA, Adiputra N. (2014) Community preparedness for highly pathogenic avian Influenza on Bali and Lombok, Indonesia. Rural Remote Health 14.

Tustin J, Laberge K, Michel P, et al. A National Epidemic of Campylobacteriosis in Iceland, lessons learned. Zoonoses Public Health. 2011;58:440–7.

Oladokun AT, Meseko CA, Ighodalo E, John B. Ekong PS Effect of intervention on the control of highly pathogenic avian Influenza in Nigeria. 8.

Manyweathers J, Field H, Longnecker N, Agho K, Smith C, Taylor M. Why won’t they just vaccinate? Horse owner risk perception and uptake of the Hendra virus vaccine. BMC Vet Res. 2017;13:103.

Zhu G, Kang M, Wei X, Tang T, Liu T, Xiao J, Song T, Ma W. Different intervention strategies toward live poultry markets against avian Influenza A (H7N9) virus: model-based assessment. Environ Res. 2020. https://doi.org/10.1016/j.envres.2020.110465 .

Chowell G, Simonsen L, Towers S, Miller MA, Viboud C. Transmission potential of Influenza A/H7N9, February to May 2013, China. BMC Med. 2013;11:214.

Bodenham RF, Mtui-Malamsha N, Gatei W, et al. Multisectoral cost analysis of a human and livestock anthrax outbreak in Songwe Region, Tanzania (December 2018–January 2019), using a novel Outbreak Costing Tool. One Health. 2021;13:100259.

Lewis N, Dorjee S, Dube C, VanLeeuwen J, Sanchez J. Assessment of Effectiveness of Control Strategies against Simulated Outbreaks of Highly Pathogenic Avian Influenza in Ontario, Canada. Transbound Emerg Dis. 2017;64:938–50.

Anderson A, Shwiff S, Gebhardt K, Ramírez AJ, Shwiff S, Kohler D, Lecuona L. Economic evaluation of Vampire Bat ( Desmodus rotundus) rabies Prevention in Mexico. Transbound Emerg Dis. 2014;61:140–6.

Walker PGT, Cauchemez S, Metras R, Dung DH, Pfeiffer D, Ghani AC. A bayesian Approach to quantifying the effects of Mass Poultry Vaccination upon the spatial and temporal dynamics of H5N1 in Northern Vietnam. PLoS Comput Biol. 2010;6:e1000683.

PREDICT Project. In: PREDICT Project. https://p2.predict.global. Accessed 9 Sep 2022.

Loh EH, Zambrana-Torrelio C, Olival KJ, Bogich TL, Johnson CK, Mazet JAK, Karesh W, Daszak P. Targeting transmission pathways for emerging zoonotic Disease Surveillance and Control. Vector-Borne and Zoonotic Diseases. 2015;15:432–7.

Bordier M, Uea-Anuwong T, Binot A, Hendrikx P, Goutard FL. Characteristics of one health surveillance systems: a systematic literature review. Prev Vet Med. 2020;181:104560.

Worsley-Tonks KEL, Bender JB, Deem SL, et al. Strengthening global health security by improving Disease surveillance in remote rural areas of low-income and middle-income countries. The Lancet Global Health. 2022;10:e579–84.

Reaser JK, Witt A, Tabor GM, Hudson PJ, Plowright RK. Ecological countermeasures for preventing zoonotic Disease outbreaks: when ecological restoration is a human health imperative. Restor Ecol. 2021;29:e13357.

Chen HL, Lewison RL, An L, Tsai YH, Stow D, Shi L, Yang S. Assessing the effects of payments for ecosystem services programs on forest structure and species biodiversity. Biodivers Conserv. 2020;29:2123–40.

Chen Y, Marino J, Tao Q, Sullivan CD, Shi K, Macdonald DW. Predicting hotspots of human-elephant conflict to inform mitigation strategies in Xishuangbanna, Southwest China. PLoS ONE. 2016. https://doi.org/10.1371/journal.pone.0162035 .

Whitfort A. COVID-19 and Wildlife Farming in China: legislating to Protect Wild Animal Health and Welfare in the wake of a global pandemic. J Environ Law. 2021;33:57–84.

Download references

Acknowledgements

Not applicable.

CCA, JC and TLP acknowledge internal research support from York University. MW and CCA acknowledge internal research support from the Dahdaleh Institute for Global Health Research. KML acknowledges funding from the Canadian Institutes of Health Research through a Health System Impact Fellowship. AY is funded by the BBSRC through the Mandala project (grant number BB/V004832/1). AMV acknowledges support from York University through a York Research Chair in Population Health Ethics & Law. This review was undertaken as part of a project funded by the Canadian Institutes of Health Research, Grant Reference Number VR5-172686. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and affiliations.

School of Global Health, York University, Toronto, ON, Canada

Chloe Clifford Astbury, Kirsten M. Lee, Ryan Mcleod, Asma Atique, Janielle Clarke, Anastassia Demeshko, A. M. Viens, Mary Wiktorowicz & Tarra L. Penney

Dahdaleh Institute for Global Health Research, York University, Toronto, ON, Canada

Chloe Clifford Astbury, Kirsten M. Lee, Raphael Aguiar, Mary Wiktorowicz & Tarra L. Penney

Global Strategy Lab, York University, Toronto, ON, Canada

Chloe Clifford Astbury, A. M. Viens & Tarra L. Penney

Applied Microbiology for Health and Environment Research Group, College of Arts and Sciences, University of the Philippines Manila, Manila, Philippines

Marilen Balolong & Kathleen Chelsea Togño

School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada

Ronald Labonté & Arne Ruckert

School of Health Policy and Management, York University, Toronto, ON, Canada

Priyanka Sibal

School of Public Health, University of Kinshasa, Kinshasa, Democratic Republic of Congo

Marc K. Yambayamba

Department of Public Health, Environments and Society, London School of Hygiene & Tropical Medicine, London, UK

You can also search for this author in PubMed   Google Scholar

Contributions

Conception and design: CCA, KLM and TLP. Acquisition of data: CCA, KLM and AY. Analysis and interpretation of data: CCA, KML, RM, JC, AD and PS. Drafting of the manuscript: CCA and RM. Critical revision of the manuscript for important intellectual content: KML, RA, AA, MB, JC, AD, RL, AR, PS, KCT, AMV, MW, MKY, AY and TLP. Obtaining funding: TLP and MW.

Corresponding author

Correspondence to Tarra L. Penney .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests. RL is a co-editor-in-chief of Globalization and Health.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Clifford Astbury, C., Lee, K.M., Mcleod, R. et al. Policies to prevent zoonotic spillover: a systematic scoping review of evaluative evidence. Global Health 19 , 82 (2023). https://doi.org/10.1186/s12992-023-00986-x

Download citation

Received : 05 May 2023

Accepted : 01 November 2023

Published : 08 November 2023

DOI : https://doi.org/10.1186/s12992-023-00986-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Zoonotic spillover
• Public policy
• Emerging zoonoses
• Deep prevention

Globalization and Health

ISSN: 1744-8603

• Submission enquiries: [email protected]
• General enquiries: [email protected]