literature review of 5 articles

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 15, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Apr 16, 2024 10:20 AM
• URL: https://libguides.usc.edu/writingguide

Get science-backed answers as you write with Paperpal's Research feature

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 
• How to write a good literature review 
• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

• Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 
• Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 
• Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 
• Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 
• Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 
• Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

How to write a good literature review

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. 

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• Life Sciences Papers: 9 Tips for Authors Writing in Biological Sciences
• What is an Argumentative Essay? How to Write It (With Examples)

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, why traditional editorial process needs an upgrade, paperpal’s new ai research finder empowers authors to..., what is hedging in academic writing  , how to use ai to enhance your college..., ai + human expertise – a paradigm shift..., how to use paperpal to generate emails &..., ai in education: it’s time to change the..., is it ethical to use ai-generated abstracts without..., do plagiarism checkers detect ai content, word choice problems: how to use the right....

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

Prevent plagiarism, run a free check.

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 15 April 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

How To Write An A-Grade Literature Review

3 straightforward steps (with examples) + free template.

By: Derek Jansen (MBA) | Expert Reviewed By: Dr. Eunice Rautenbach | October 2019

Quality research is about building onto the existing work of others , “standing on the shoulders of giants”, as Newton put it. The literature review chapter of your dissertation, thesis or research project is where you synthesise this prior work and lay the theoretical foundation for your own research.

Long story short, this chapter is a pretty big deal, which is why you want to make sure you get it right . In this post, I’ll show you exactly how to write a literature review in three straightforward steps, so you can conquer this vital chapter (the smart way).

Overview: The Literature Review Process

• Understanding the “ why “
• Finding the relevant literature
• Cataloguing and synthesising the information
• Outlining & writing up your literature review
• Example of a literature review

But first, the “why”…

Before we unpack how to write the literature review chapter, we’ve got to look at the why . To put it bluntly, if you don’t understand the function and purpose of the literature review process, there’s no way you can pull it off well. So, what exactly is the purpose of the literature review?

Well, there are (at least) four core functions:

• For you to gain an understanding (and demonstrate this understanding) of where the research is at currently, what the key arguments and disagreements are.
• For you to identify the gap(s) in the literature and then use this as justification for your own research topic.
• To help you build a conceptual framework for empirical testing (if applicable to your research topic).
• To inform your methodological choices and help you source tried and tested questionnaires (for interviews ) and measurement instruments (for surveys ).

Most students understand the first point but don’t give any thought to the rest. To get the most from the literature review process, you must keep all four points front of mind as you review the literature (more on this shortly), or you’ll land up with a wonky foundation.

Okay – with the why out the way, let’s move on to the how . As mentioned above, writing your literature review is a process, which I’ll break down into three steps:

• Finding the most suitable literature
• Understanding , distilling and organising the literature
• Planning and writing up your literature review chapter

Importantly, you must complete steps one and two before you start writing up your chapter. I know it’s very tempting, but don’t try to kill two birds with one stone and write as you read. You’ll invariably end up wasting huge amounts of time re-writing and re-shaping, or you’ll just land up with a disjointed, hard-to-digest mess . Instead, you need to read first and distil the information, then plan and execute the writing.

Step 1: Find the relevant literature

Naturally, the first step in the literature review journey is to hunt down the existing research that’s relevant to your topic. While you probably already have a decent base of this from your research proposal , you need to expand on this substantially in the dissertation or thesis itself.

Essentially, you need to be looking for any existing literature that potentially helps you answer your research question (or develop it, if that’s not yet pinned down). There are numerous ways to find relevant literature, but I’ll cover my top four tactics here. I’d suggest combining all four methods to ensure that nothing slips past you:

Method 1 – Google Scholar Scrubbing

Google’s academic search engine, Google Scholar , is a great starting point as it provides a good high-level view of the relevant journal articles for whatever keyword you throw at it. Most valuably, it tells you how many times each article has been cited, which gives you an idea of how credible (or at least, popular) it is. Some articles will be free to access, while others will require an account, which brings us to the next method.

Method 2 – University Database Scrounging

Generally, universities provide students with access to an online library, which provides access to many (but not all) of the major journals.

So, if you find an article using Google Scholar that requires paid access (which is quite likely), search for that article in your university’s database – if it’s listed there, you’ll have access. Note that, generally, the search engine capabilities of these databases are poor, so make sure you search for the exact article name, or you might not find it.

Method 3 – Journal Article Snowballing

At the end of every academic journal article, you’ll find a list of references. As with any academic writing, these references are the building blocks of the article, so if the article is relevant to your topic, there’s a good chance a portion of the referenced works will be too. Do a quick scan of the titles and see what seems relevant, then search for the relevant ones in your university’s database.

Method 4 – Dissertation Scavenging

Similar to Method 3 above, you can leverage other students’ dissertations. All you have to do is skim through literature review chapters of existing dissertations related to your topic and you’ll find a gold mine of potential literature. Usually, your university will provide you with access to previous students’ dissertations, but you can also find a much larger selection in the following databases:

• Open Access Theses & Dissertations
• Stanford SearchWorks

Keep in mind that dissertations and theses are not as academically sound as published, peer-reviewed journal articles (because they’re written by students, not professionals), so be sure to check the credibility of any sources you find using this method. You can do this by assessing the citation count of any given article in Google Scholar. If you need help with assessing the credibility of any article, or with finding relevant research in general, you can chat with one of our Research Specialists .

Alright – with a good base of literature firmly under your belt, it’s time to move onto the next step.

Need a helping hand?

Step 2: Log, catalogue and synthesise

Once you’ve built a little treasure trove of articles, it’s time to get reading and start digesting the information – what does it all mean?

While I present steps one and two (hunting and digesting) as sequential, in reality, it’s more of a back-and-forth tango – you’ll read a little , then have an idea, spot a new citation, or a new potential variable, and then go back to searching for articles. This is perfectly natural – through the reading process, your thoughts will develop , new avenues might crop up, and directional adjustments might arise. This is, after all, one of the main purposes of the literature review process (i.e. to familiarise yourself with the current state of research in your field).

As you’re working through your treasure chest, it’s essential that you simultaneously start organising the information. There are three aspects to this:

• Logging reference information
• Building an organised catalogue
• Distilling and synthesising the information

I’ll discuss each of these below:

2.1 – Log the reference information

As you read each article, you should add it to your reference management software. I usually recommend Mendeley for this purpose (see the Mendeley 101 video below), but you can use whichever software you’re comfortable with. Most importantly, make sure you load EVERY article you read into your reference manager, even if it doesn’t seem very relevant at the time.

2.2 – Build an organised catalogue

In the beginning, you might feel confident that you can remember who said what, where, and what their main arguments were. Trust me, you won’t. If you do a thorough review of the relevant literature (as you must!), you’re going to read many, many articles, and it’s simply impossible to remember who said what, when, and in what context . Also, without the bird’s eye view that a catalogue provides, you’ll miss connections between various articles, and have no view of how the research developed over time. Simply put, it’s essential to build your own catalogue of the literature.

I would suggest using Excel to build your catalogue, as it allows you to run filters, colour code and sort – all very useful when your list grows large (which it will). How you lay your spreadsheet out is up to you, but I’d suggest you have the following columns (at minimum):

• Author, date, title – Start with three columns containing this core information. This will make it easy for you to search for titles with certain words, order research by date, or group by author.
• Categories or keywords – You can either create multiple columns, one for each category/theme and then tick the relevant categories, or you can have one column with keywords.
• Key arguments/points – Use this column to succinctly convey the essence of the article, the key arguments and implications thereof for your research.
• Context – Note the socioeconomic context in which the research was undertaken. For example, US-based, respondents aged 25-35, lower- income, etc. This will be useful for making an argument about gaps in the research.
• Methodology – Note which methodology was used and why. Also, note any issues you feel arise due to the methodology. Again, you can use this to make an argument about gaps in the research.
• Quotations – Note down any quoteworthy lines you feel might be useful later.
• Notes – Make notes about anything not already covered. For example, linkages to or disagreements with other theories, questions raised but unanswered, shortcomings or limitations, and so forth.

If you’d like, you can try out our free catalog template here (see screenshot below).

2.3 – Digest and synthesise

Most importantly, as you work through the literature and build your catalogue, you need to synthesise all the information in your own mind – how does it all fit together? Look for links between the various articles and try to develop a bigger picture view of the state of the research. Some important questions to ask yourself are:

• What answers does the existing research provide to my own research questions ?
• Which points do the researchers agree (and disagree) on?
• How has the research developed over time?
• Where do the gaps in the current research lie?

To help you develop a big-picture view and synthesise all the information, you might find mind mapping software such as Freemind useful. Alternatively, if you’re a fan of physical note-taking, investing in a large whiteboard might work for you.

Step 3: Outline and write it up!

Once you’re satisfied that you have digested and distilled all the relevant literature in your mind, it’s time to put pen to paper (or rather, fingers to keyboard). There are two steps here – outlining and writing:

3.1 – Draw up your outline

Having spent so much time reading, it might be tempting to just start writing up without a clear structure in mind. However, it’s critically important to decide on your structure and develop a detailed outline before you write anything. Your literature review chapter needs to present a clear, logical and an easy to follow narrative – and that requires some planning. Don’t try to wing it!

Naturally, you won’t always follow the plan to the letter, but without a detailed outline, you’re more than likely going to end up with a disjointed pile of waffle , and then you’re going to spend a far greater amount of time re-writing, hacking and patching. The adage, “measure twice, cut once” is very suitable here.

In terms of structure, the first decision you’ll have to make is whether you’ll lay out your review thematically (into themes) or chronologically (by date/period). The right choice depends on your topic, research objectives and research questions, which we discuss in this article .

Once that’s decided, you need to draw up an outline of your entire chapter in bullet point format. Try to get as detailed as possible, so that you know exactly what you’ll cover where, how each section will connect to the next, and how your entire argument will develop throughout the chapter. Also, at this stage, it’s a good idea to allocate rough word count limits for each section, so that you can identify word count problems before you’ve spent weeks or months writing!

PS – check out our free literature review chapter template…

3.2 – Get writing

With a detailed outline at your side, it’s time to start writing up (finally!). At this stage, it’s common to feel a bit of writer’s block and find yourself procrastinating under the pressure of finally having to put something on paper. To help with this, remember that the objective of the first draft is not perfection – it’s simply to get your thoughts out of your head and onto paper, after which you can refine them. The structure might change a little, the word count allocations might shift and shuffle, and you might add or remove a section – that’s all okay. Don’t worry about all this on your first draft – just get your thoughts down on paper.

Once you’ve got a full first draft (however rough it may be), step away from it for a day or two (longer if you can) and then come back at it with fresh eyes. Pay particular attention to the flow and narrative – does it fall fit together and flow from one section to another smoothly? Now’s the time to try to improve the linkage from each section to the next, tighten up the writing to be more concise, trim down word count and sand it down into a more digestible read.

Once you’ve done that, give your writing to a friend or colleague who is not a subject matter expert and ask them if they understand the overall discussion. The best way to assess this is to ask them to explain the chapter back to you. This technique will give you a strong indication of which points were clearly communicated and which weren’t. If you’re working with Grad Coach, this is a good time to have your Research Specialist review your chapter.

Finally, tighten it up and send it off to your supervisor for comment. Some might argue that you should be sending your work to your supervisor sooner than this (indeed your university might formally require this), but in my experience, supervisors are extremely short on time (and often patience), so, the more refined your chapter is, the less time they’ll waste on addressing basic issues (which you know about already) and the more time they’ll spend on valuable feedback that will increase your mark-earning potential.

Literature Review Example

In the video below, we unpack an actual literature review so that you can see how all the core components come together in reality.

Let’s Recap

In this post, we’ve covered how to research and write up a high-quality literature review chapter. Let’s do a quick recap of the key takeaways:

• It is essential to understand the WHY of the literature review before you read or write anything. Make sure you understand the 4 core functions of the process.
• The first step is to hunt down the relevant literature . You can do this using Google Scholar, your university database, the snowballing technique and by reviewing other dissertations and theses.
• Next, you need to log all the articles in your reference manager , build your own catalogue of literature and synthesise all the research.
• Following that, you need to develop a detailed outline of your entire chapter – the more detail the better. Don’t start writing without a clear outline (on paper, not in your head!)
• Write up your first draft in rough form – don’t aim for perfection. Remember, done beats perfect.
• Refine your second draft and get a layman’s perspective on it . Then tighten it up and submit it to your supervisor.

Psst… there’s more!

This post is an extract from our bestselling Udemy Course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

38 Comments

Thank you very much. This page is an eye opener and easy to comprehend.

This is awesome!

I wish I come across GradCoach earlier enough.

But all the same I’ll make use of this opportunity to the fullest.

Thank you for this good job.

Keep it up!

You’re welcome, Yinka. Thank you for the kind words. All the best writing your literature review.

Thank you for a very useful literature review session. Although I am doing most of the steps…it being my first masters an Mphil is a self study and one not sure you are on the right track. I have an amazing supervisor but one also knows they are super busy. So not wanting to bother on the minutae. Thank you.

You’re most welcome, Renee. Good luck with your literature review 🙂

This has been really helpful. Will make full use of it. 🙂

Thank you Gradcoach.

Really agreed. Admirable effort

thank you for this beautiful well explained recap.

Thank you so much for your guide of video and other instructions for the dissertation writing.

It is instrumental. It encouraged me to write a dissertation now.

Thank you the video was great – from someone that knows nothing thankyou

an amazing and very constructive way of presetting a topic, very useful, thanks for the effort,

It is timely

It is very good video of guidance for writing a research proposal and a dissertation. Since I have been watching and reading instructions, I have started my research proposal to write. I appreciate to Mr Jansen hugely.

I learn a lot from your videos. Very comprehensive and detailed.

Thank you for sharing your knowledge. As a research student, you learn better with your learning tips in research

I was really stuck in reading and gathering information but after watching these things are cleared thanks, it is so helpful.

Really helpful, Thank you for the effort in showing such information

This is super helpful thank you very much.

Thank you for this whole literature writing review.You have simplified the process.

I’m so glad I found GradCoach. Excellent information, Clear explanation, and Easy to follow, Many thanks Derek!

You’re welcome, Maithe. Good luck writing your literature review 🙂

Thank you Coach, you have greatly enriched and improved my knowledge

Great piece, so enriching and it is going to help me a great lot in my project and thesis, thanks so much

This is THE BEST site for ANYONE doing a masters or doctorate! Thank you for the sound advice and templates. You rock!

Thanks, Stephanie 🙂

This is mind blowing, the detailed explanation and simplicity is perfect.

I am doing two papers on my final year thesis, and I must stay I feel very confident to face both headlong after reading this article.

thank you so much.

if anyone is to get a paper done on time and in the best way possible, GRADCOACH is certainly the go to area!

This is very good video which is well explained with detailed explanation

Thank you excellent piece of work and great mentoring

Thanks, it was useful

Thank you very much. the video and the information were very helpful.

Good morning scholar. I’m delighted coming to know you even before the commencement of my dissertation which hopefully is expected in not more than six months from now. I would love to engage my study under your guidance from the beginning to the end. I love to know how to do good job

Thank you so much Derek for such useful information on writing up a good literature review. I am at a stage where I need to start writing my one. My proposal was accepted late last year but I honestly did not know where to start

Like the name of your YouTube implies you are GRAD (great,resource person, about dissertation). In short you are smart enough in coaching research work.

This is a very well thought out webpage. Very informative and a great read.

Very timely.

I appreciate.

Very comprehensive and eye opener for me as beginner in postgraduate study. Well explained and easy to understand. Appreciate and good reference in guiding me in my research journey. Thank you

Thank you. I requested to download the free literature review template, however, your website wouldn’t allow me to complete the request or complete a download. May I request that you email me the free template? Thank you.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

Deadly diseases and inflatable suits: how I found my niche in virology research

Spotlight 17 APR 24

How young people benefit from Swiss apprenticeships

I dive for fish in the longest freshwater lake in the world

Researchers want a ‘nutrition label’ for academic-paper facts

Nature Index 17 APR 24

How we landed job interviews for professorships straight out of our PhD programmes

Career Column 08 APR 24

Structure peer review to make it more robust

World View 16 APR 24

US COVID-origins hearing puts scientific journals in the hot seat

News 16 APR 24

FACULTY POSITION IN PATHOLOGY RESEARCH

Dallas, Texas (US)

The University of Texas Southwestern Medical Center (UT Southwestern Medical Center)

Postdoc Fellow / Senior Scientist

The Yakoub and Sulzer labs at Harvard Medical School-Brigham and Women’s Hospital and Columbia University

Boston, Massachusetts (US)

Harvard Medical School and Brigham and Women's Hospital

Postdoc in Computational Genomics – Machine Learning for Multi-Omics Profiling of Cancer Evolution

Computational Postdoc - Artificial Intelligence in Oncology and Regulatory Genomics and Cancer Evolution at the DKFZ - limited to 2 years

Heidelberg, Baden-Württemberg (DE)

German Cancer Research Center in the Helmholtz Association (DKFZ)

Computational Postdoc

The German Cancer Research Center is the largest biomedical research institution in Germany.

PhD / PostDoc Medical bioinformatics (m/f/d)

The Institute of Medical Bioinformatics and Systems Medicine / University of Freiburg is looking for a PhD/PostDoc Medical bioinformatics (m/w/d)

Freiburg im Breisgau, Baden-Württemberg (DE)

University of Freiburg

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

How to Write a Literature Review

What is a literature review.

• What Is the Literature
• Writing the Review

A literature review is much more than an annotated bibliography or a list of separate reviews of articles and books. It is a critical, analytical summary and synthesis of the current knowledge of a topic. Thus it should compare and relate different theories, findings, etc, rather than just summarize them individually. In addition, it should have a particular focus or theme to organize the review. It does not have to be an exhaustive account of everything published on the topic, but it should discuss all the significant academic literature and other relevant sources important for that focus.

This is meant to be a general guide to writing a literature review: ways to structure one, what to include, how it supplements other research. For more specific help on writing a review, and especially for help on finding the literature to review, sign up for a Personal Research Session .

The specific organization of a literature review depends on the type and purpose of the review, as well as on the specific field or topic being reviewed. But in general, it is a relatively brief but thorough exploration of past and current work on a topic. Rather than a chronological listing of previous work, though, literature reviews are usually organized thematically, such as different theoretical approaches, methodologies, or specific issues or concepts involved in the topic. A thematic organization makes it much easier to examine contrasting perspectives, theoretical approaches, methodologies, findings, etc, and to analyze the strengths and weaknesses of, and point out any gaps in, previous research. And this is the heart of what a literature review is about. A literature review may offer new interpretations, theoretical approaches, or other ideas; if it is part of a research proposal or report it should demonstrate the relationship of the proposed or reported research to others' work; but whatever else it does, it must provide a critical overview of the current state of research efforts. 

Literature reviews are common and very important in the sciences and social sciences. They are less common and have a less important role in the humanities, but they do have a place, especially stand-alone reviews.

Types of Literature Reviews

There are different types of literature reviews, and different purposes for writing a review, but the most common are:

• Stand-alone literature review articles . These provide an overview and analysis of the current state of research on a topic or question. The goal is to evaluate and compare previous research on a topic to provide an analysis of what is currently known, and also to reveal controversies, weaknesses, and gaps in current work, thus pointing to directions for future research. You can find examples published in any number of academic journals, but there is a series of Annual Reviews of *Subject* which are specifically devoted to literature review articles. Writing a stand-alone review is often an effective way to get a good handle on a topic and to develop ideas for your own research program. For example, contrasting theoretical approaches or conflicting interpretations of findings can be the basis of your research project: can you find evidence supporting one interpretation against another, or can you propose an alternative interpretation that overcomes their limitations?
• Part of a research proposal . This could be a proposal for a PhD dissertation, a senior thesis, or a class project. It could also be a submission for a grant. The literature review, by pointing out the current issues and questions concerning a topic, is a crucial part of demonstrating how your proposed research will contribute to the field, and thus of convincing your thesis committee to allow you to pursue the topic of your interest or a funding agency to pay for your research efforts.
• Part of a research report . When you finish your research and write your thesis or paper to present your findings, it should include a literature review to provide the context to which your work is a contribution. Your report, in addition to detailing the methods, results, etc. of your research, should show how your work relates to others' work.

A literature review for a research report is often a revision of the review for a research proposal, which can be a revision of a stand-alone review. Each revision should be a fairly extensive revision. With the increased knowledge of and experience in the topic as you proceed, your understanding of the topic will increase. Thus, you will be in a better position to analyze and critique the literature. In addition, your focus will change as you proceed in your research. Some areas of the literature you initially reviewed will be marginal or irrelevant for your eventual research, and you will need to explore other areas more thoroughly. 

Examples of Literature Reviews

See the series of Annual Reviews of *Subject* which are specifically devoted to literature review articles to find many examples of stand-alone literature reviews in the biomedical, physical, and social sciences. 

Research report articles vary in how they are organized, but a common general structure is to have sections such as:

• Abstract - Brief summary of the contents of the article
• Introduction - A explanation of the purpose of the study, a statement of the research question(s) the study intends to address
• Literature review - A critical assessment of the work done so far on this topic, to show how the current study relates to what has already been done
• Methods - How the study was carried out (e.g. instruments or equipment, procedures, methods to gather and analyze data)
• Results - What was found in the course of the study
• Discussion - What do the results mean
• Conclusion - State the conclusions and implications of the results, and discuss how it relates to the work reviewed in the literature review; also, point to directions for further work in the area

Here are some articles that illustrate variations on this theme. There is no need to read the entire articles (unless the contents interest you); just quickly browse through to see the sections, and see how each section is introduced and what is contained in them.

The Determinants of Undergraduate Grade Point Average: The Relative Importance of Family Background, High School Resources, and Peer Group Effects , in The Journal of Human Resources , v. 34 no. 2 (Spring 1999), p. 268-293.

This article has a standard breakdown of sections:

• Introduction
• Literature Review
• Some discussion sections

First Encounters of the Bureaucratic Kind: Early Freshman Experiences with a Campus Bureaucracy , in The Journal of Higher Education , v. 67 no. 6 (Nov-Dec 1996), p. 660-691.

This one does not have a section specifically labeled as a "literature review" or "review of the literature," but the first few sections cite a long list of other sources discussing previous research in the area before the authors present their own study they are reporting.

• Next: What Is the Literature >>
• Last Updated: Jan 11, 2024 9:48 AM
• URL: https://libguides.wesleyan.edu/litreview

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jan 17, 2024 10:05 AM
• URL: https://libguides.northwestern.edu/literaturereviews

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

• UWF Libraries

Literature Review: Conducting & Writing

• Sample Literature Reviews
• Steps for Conducting a Lit Review
• Finding "The Literature"
• Organizing/Writing
• APA Style This link opens in a new window
• Chicago: Notes Bibliography This link opens in a new window
• MLA Style This link opens in a new window

Sample Lit Reviews from Communication Arts

Have an exemplary literature review.

• Literature Review Sample 1
• Literature Review Sample 2
• Literature Review Sample 3

Have you written a stellar literature review you care to share for teaching purposes?

Are you an instructor who has received an exemplary literature review and have permission from the student to post?

Please contact Britt McGowan at [email protected] for inclusion in this guide. All disciplines welcome and encouraged.

• << Previous: MLA Style
• Next: Get Help! >>
• Last Updated: Mar 22, 2024 9:37 AM
• URL: https://libguides.uwf.edu/litreview

Advertisement

A Literature Review of Pandemics and Development: the Long-Term Perspective

• Original Paper
• Published: 27 January 2022
• Volume 6 , pages 183–212, ( 2022 )

Cite this article

• Beniamino Callegari   ORCID: orcid.org/0000-0001-5513-7299 1 , 2 &
• Christophe Feder   ORCID: orcid.org/0000-0002-1239-513X 3 , 4  

6441 Accesses

13 Citations

7 Altmetric

Explore all metrics

Pandemics have been a long-standing object of study by economists, albeit with declining interest, that is until COVID-19 arrived. We review current knowledge on the pandemics’ effects on long-term economic development, spanning economic and historical debates. We show that all economic inputs are potentially affected. Pandemics reduce the workforce and human capital, have mixed effects on investment and savings, but potentially positive consequences for innovation and knowledge development, depending on accompanying institutional change. In the absence of an innovative response supporting income redistribution, pandemics tend to increase income inequalities, worsening poverty traps and highlighting the distributional issues built into insurance-based health insurance systems. We find that the effects of pandemics are asymmetric over time, in space, and among sectors and households. Therefore, we suggest that the research focus on the theoretical plausibility and empirical significance of specific mechanisms should be complemented by meta-analytic efforts aimed at reconstructing the resulting complexity. Finally, we suggest that policymakers prioritize the development of organizational learning and innovative capabilities, focusing on the ability to adapt to emergencies rather than developing rigid protocols or mimicking solutions developed and implemented in different contexts.

Similar content being viewed by others

Douglass North’s Theory of Institutions: Lessons for Law and Development

Julio Faundez

Why Do Some Countries Develop and Others Not?

Assessing the relationship between poverty and economic growth: does sustainable development goal can be achieved.

Yu Zhu, Shahid Bashir & Mohamed Marie

Avoid common mistakes on your manuscript.

Introduction

As the COVID-19 emergency appears to slowly and unevenly recede in the wake of medical breakthroughs and the development of more effective prevention and treatment protocols, the question of the long-term impact of the pandemic grows more urgent. There is little doubt that this global health crisis found economists mostly unprepared, as the analysis of the pandemic’s effects has hardly found its way into the discipline’s most central publication avenues (Noy and Managi 2020 ). However, this does not mean that the economic analysis of pandemics is starting from scratch, as economists and economic historians have never ceased to expand our knowledge on the subject.

The connection between pandemics and economic theory has historically been so relevant that it has directly contributed to labeling economics as the ‘dismal science’. Cipolla ( 1974 ) illustrates how reflections on the plague and its consequences led many scholars to develop Malthusian ideas on the complex long-term relationships between population growth, economic growth, and diseases, well in advance of the Essay on the Principle of Population (Malthus 1798 ). However, the Industrial Revolution and the concomitant development of medical knowledge led to a decreased incidence of catastrophic plagues in the West, and a corresponding decline in the interest in pandemics on the part of economists (Easterlin 1995 ). The demographic boom of the West and the visible lack of corresponding pestilence and famine further discredited Malthusian perspectives, leading to a disconnection between the demographic and economic disciplines. Furthermore, from 1900 to 2019, pandemics were either eclipsed by more disruptive events or had a relatively limited economic impact (Garrett 2008 ; Lee and McKibbin 2004 ; Noy and Managi 2020 ). Finally, the marginalist revolution greatly focused economists’ attention on purely economic elements, eliminating from the discipline those elements perceived as spurious, like the study of pandemics’ effects (Schumpeter 1954 ), relegating it to a debate of mainly historical interest.

The expansion of economic analysis beyond its traditional boundaries that has occurred in the last two decades has gradually re-included the consequences of pandemics within economic theory, although most contributions remain on the periphery of academic debate and are relatively hidden (Arora 2001 ; Dunn 2006 ; Weil 2014 ). As Noy and Managi ( 2020 ) observed, the inherently multidisciplinary nature of pandemics, combined with its poor fit with what are called “hard” methods, have both conspired to make the contribution made by economists to the analysis of pandemics modest. The efforts of economists have been greatly augmented by the continuous work done by economic historians to understand the impact of past pandemics on the long-term development of various socioeconomic systems. Yet, while the total contribution to the economic analysis of the long-term impact of pandemics is significant, it is scattered across different journals, disciplines, academic approaches, and debates, making a review work necessary in order for all these contributions to become accessible.

This paper reviews the long-term economic effects of pandemics, defined as health shocks arising from infectious diseases with global diffusion. Within the definition of long-term effects, we include both those mechanisms that are immediately present and persist for a significant amount of time and those effects that arise in the long term. Due to the focus of our analysis, transient short-term effects are not part of our study. To the best of our knowledge, few literature reviews have studied the connection between pandemics and economic development. Bleakley ( 2010 ) critically reviews how diseases, rather than pandemics specifically, affect human capital formation and income growth at the micro and macro levels. Costa ( 2015 ) describes how health improvements affect economic growth, with a specific focus on the US, concluding that improved health is not sufficient to foster growth. Finally, Boucekkine et al. ( 2008 ) formally analyze how and which growth models are better able to mathematically describe the epidemics’ effects. Moreover, some scholars have also reviewed the long-term economic effects of particular health shocks, like the preindustrial epidemics (Alfani 2021 ), Spanish flu (Beach et al. 2021 ), HIV (Gaffeo 2003 ; Zinyemba et al. 2020 ), and modern pandemics (Bloom et al. 2021 ). We differ from these works because we analyze the long-term impact of pandemics in general on economic development. A similar approach has been adopted by Gries and Naudé ( 2021 ) and Callegari and Feder ( 2021a ), but with an entrepreneurship and not a macroeconomic focus.

Our broad approach has led us to review a large number of studies in order to identify recurrent results across very different pandemic events. Pandemics could affect aggregate demand, aggregate supply, and productivity growth (Basco et al. 2021 ; Dieppe 2021 ; Guerrieri et al. 2020 ; Jinjarak et al. 2021 ; Rassy and Smith 2013 ; World Bank 2020 ). Recalling the Solovian framework, we divide the long-term pandemic economic effects into three categories: labor and human capital; investments and physical capital; and knowledge and innovation. We find that all productive inputs are affected in the long term by the pandemic. More specifically, labor and human capital are negatively affected directly by health shocks. However, the intensity of this effect is heterogeneous among countries, labor markets, and industries. Investments and physical capital are affected by pandemics through complex, interacting, and often contrasting mechanisms, leaving long-term effects ambiguous and usually marginal and non-linear. However, the asymmetric impact of pandemics on the capital market and household income leads to the poverty trap and highlights the weakness of the health insurance system in coping with these shocks. Finally, pandemics could positively affect innovations in public and private institutions and bring about relevant technological changes in industries. The scope and direction of these socioeconomic changes appear to mediate the long-term effects of pandemics, determining both their direction and scope. However, relevant and radical institutional changes are necessary if the impact of pandemics on development is to be positive. We therefore suggest that scholars should develop meta-analysis to understand the complex tapestry of long-term pandemic mechanisms. Many policy implications follow directly: an efficient public intervention must be characterized in the long term by flexibility, pro-market orientation, and design customization.

The paper is structured as follows. Section 2 explains the selection methodology used in the review. Sections 3 , 4 , and 5 describe, respectively, the long-term effects of pandemics on: labor and human capital; investment and physical capital; and knowledge and innovation. Section 6 critically discusses the survey and summarizes the main lessons drawn from the literature for researchers and policymakers. Section 7 concludes.

Methodology

This literature review aims to illustrate, compare, and discuss the mechanisms through which pandemics affect long-term economic development. To achieve this goal, we adopted the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology (Moher et al. 2009 ). First, we defined a list of keywords that express the main aspects of the “pandemic” and “economic development” concepts. Second, we identified which data sets to search: JSTOR, IDEAS/RePEc, Google Scholar, and EconLit. We excluded working papers and unpublished articles from our search, to ensure that the mechanisms presented are accepted by the scientific community. Moreover, we restricted our focus to the fields of economics and economic history, to ensure the economic relevance of the mechanisms described. Finally, we excluded papers focused on the COVID-19 pandemic, as it is too early for a comprehensive evaluation of its long-term effects. Applying these criteria, we obtained a first sample of more than 4800 potential articles. Important contributions were not missed due to excessively strict methodological adherence, we also parsed the references lists of the most influential contributions within our initial corpus, identifying in this way 178 additional relevant manuscripts to potentially include in our review.

From this corpus of potential articles, we operated a further selection by analyzing their abstracts and, in uncertain cases, by searching the main body of the paper concerned for evidence of relevant discourse, thereby identifying 805 potential contributions. We then proceeded to evaluate the selected articles for inclusion according to their relevance to our research topic and their relative originality, evaluated in terms of the mechanisms analyzed. We then proceeded to summarize the resulting papers according to their research questions and aims, their theoretical references, their methodology, and their results, focusing on the featured economic mechanisms, in order to identify the structure of our corpus in terms of the main debates, the empirical object of study, the methods applied, and the theoretical foundations. In this way, after eliminating redundant contributions, we selected 88 articles, each describing specific mechanisms through which pandemics may affect the economic system in the long term. Finally, we identified a criterion to organize the resulting mechanisms, inspired by the well-known Solovian model of long-term growth, dividing them into the following three broad categories: labor, capital, and innovation.

We then identified a corpus of high-quality contributions, each offering a specific contribution to the academic debate in terms of one or more relevant mechanisms, supported by either theoretical or empirical arguments. Figure  1 summarizes the main steps of the selection process by using a PRISMA diagram.

The PRISMA process

Labor and Human Capital

The most intuitive and direct effect of pandemics is the adverse shock to the population and the labor market. Delfino and Simmons ( 2005 ) propose a Lotka-Volterra model showing that a negative demographic effect could become persistent if the pandemic is not eradicated. The magnitude of this effect is, however, mediated by contextual factors. Alfani ( 2013 ) shows that, in southern Europe, the plagues of the XVII century had higher mortality and territorial pervasiveness compared with those affecting northern Europe in the same period and the southern Europe plagues of the previous century. Furthermore, the rate of mortality and territorial pervasiveness was heterogeneous among Italian regions and cities. Using a long-term perspective, Rodríguez-Caballero and Vera-Valdés ( 2020 ) find that pandemics reduced the unemployment rate persistently from 1854 to 2016 in Italy, Spain, the Netherlands, the UK, and the US. They also observe that, in the UK, pandemics reduced the GDP per capita over 1270–2019, and that this effect was increasingly persistent in the last 300 years. Fiaschi and Fioroni ( 2019 ) have built a model which shows how pandemics’ impact on growth trajectories is mediated by the production structure and the mortality reduction brought by technological progress. Bloom and Sachs ( 1998 ) observe that the mortality and morbidity of pandemics are highest in tropical regions. They explain that differences in climate and nature, together with anthropomorphic factors, affect the spread of the virus over the population. However, this direct effect on labor and population could decline in the long term.

The long-term effects of pandemics on the labor supply also depend on their impact on fertility. By analyzing 15 relevant infectious diseases from 75 countries between 1940 and 2000, Acemoglu and Johnson ( 2007 ) find that pandemics reduce demographic equilibria in the long term through their impact on fertility rates. Birth rates are influenced directly, as the pandemic reduces the number of fertile women, and indirectly, as future life expectancy influences decisions to have children in the long term. They empirically confirm that the higher mortality of those affected by infectious diseases sharply reduces births and slightly reduces the share of the young in the population because of their lower life expectancy. Lorentzen et al. ( 2008 ) show that a pandemic affects not only fertility, i.e. the number of births, but also the net fertility, i.e. the fertility of the surviving population. Parents care about the number of surviving newborns: higher infant mortality increases births. Moreover, parents invest time and money in their children, who become irreplaceable when they grow up. Therefore, higher adult mortality increases fertility, even more than infant mortality. Finally, given the family budget constraint, parents must choose between the quantity and quality of their children. Consequently, the uncertainty of the pandemic reduces the investment rent in human capital, leading parents to rationally prefer quantity to quality. The authors find empirical support for these hypotheses, observing that the probability of contracting malaria negatively affects adult and infant life expectancy, and that both expectations improve the fertility rate.

Fertility mechanisms interact with human capital accumulation. Lagerlöf ( 2003 ) describes an overlapping-generations model where adults confront the children’s quality-quantity trade-off. Infant survival is negatively affected by both the chance of random pandemics and population density, which both increase the risk of contagion, but is positively affected by human capital (higher medical knowledge), which is cumulative in time and positively affected by population density (knowledge spillovers). They find that, when pandemics are frequent, where the decision to have children is concerned, parents prefer quantity to quality; human capital does not increase; and population density remains low. When sufficient human capital has accumulated, however, the growth path of the economy is no longer affected by new pandemic waves. Consequently, only if, by chance, pandemics do not strike for a sufficiently long time, will parents then increase their investments in future generations, thus reaching the human capital threshold necessary to achieve robust growth trajectories. Gori et al. ( 2020 ) integrate all previously described mechanisms in a three-stage overlapping-generations growth model, including adolescent, adult, and elderly agents. In this model, only the elderly are sexually inactive and are, therefore, not exposed to HIV infection. The probability of dying from the pandemic is negatively associated with human capital endowment and positively associated with the number of virus-spreaders. The pandemic increases both infant and adult mortality. Adult mortality reduces both labor supply and life expectancy. If life expectancy is reduced below a certain threshold level, parents prefer to have more children; otherwise, they prefer to invest in human capital. Parameterizing the model for the Sub-Saharan African countries, Gori et al. ( 2020 ) find that HIV reduces labor supply and human capital but increases fertility. Cervellati and Sunde ( 2015 ) model an economy where parents confront the children’s quality-quantity trade-off, given the child mortality and the inborn ability of the offspring. Also in this model, higher human capital leads to an improvement in both medical care and adult life expectancy: intensive economic growth follows an initial quasi-stagnation. Cervellati and Sunde ( 2015 ) observe, like Lorentzen et al. ( 2008 ), that adult mortality and human capital affect the economic dynamics more than fertility and child mortality. Cervellati and Sunde ( 2011 ) combine Lorentzen et al. ( 2008 )‘s life expectancy effects on mortality and fertility with the Acemoglu and Johnson ( 2007 )‘s model and find non-monotonic patterns of demographic growth. Before the demographic transition, more newborns could compensate for higher mortality, leaving the overall demographic effect ambiguous; after the demographic transition, parents prefer quality over quantity in regard to children, making pandemic demographic effects definitively negative in the long term.

A pandemic’s negative demographic impact reduces the number of available workers. However, its long-term impact is mixed. Gori et al. ( 2020 ) and Dauda ( 2019 ) provide a comprehensive literature review of the complex link between HIV and growth. They conclude that, while strong evidence exists for a negative link at the micro level, the empirical support for the macro effects is weaker. Keogh-Brown et al. ( 2010 ) find four ways in which the pandemic can affect the work supply. Death and infection of workers result in a temporary reduction of the workforce, partially persistent in the long term. However, they observe that these effects could be mitigated by migration (see also Alfani 2013 ), labor market inefficiencies (see also Bloom and Mahal 1997 ), and inventories. Using a structural econometric model of the UK to estimate the economic effects of a modern pandemic, they conclude that it would reduce production and increase firms’ costs, leading to the emergence of inflation in the long term. Voigtländer and Voth ( 2013 ) describe a model where pandemics reduce population but increase labor in the manufacturing sectors. Since the land supply remains constant, labor productivity increases, and therefore survivors’ wages are higher than they would be without the pandemic in the long term. If the welfare increase is sufficiently high, the demand for manufactured goods increases trade and population density. Moreover, manufactured goods are easily taxable, thus enabling the financing of more wars. All these mechanisms increase the transmission of disease, leading to long-term demographic stagnation. Using data on the Black Death, the calibrated model correctly approximates the growth of both the European urbanization and per capita GDP from 1000 to 1700.

Historical research provides further support for the hypothesis. Herlihy ( 1997 ) confirms that wages and the demand for manufactured goods increased after the Plague; however, he observes higher lethality for adults than for both the young and the elderly. The Black Death first reduced the number of available workers and the length of their productive life. Additionally, the Plague took away both the skill and experience of previous workers and the parent’s investment in the education of their children. Moreover, high turnover increased labor demand, further reducing the productivity of new workers in the long term. Low labor supply increased wages, as land rents decreased. Finally, consumption grew quantitatively, shifting qualitatively towards higher-quality goods, leading to the emergence of a positive long-term impact on the real wages and welfare of the survivors. Pamuk ( 2007 ) supports all Herlihy ( 1997 )‘s results. Moreover, he finds that the great difference in economic growth between North and South Europe, which is observed only some centuries later, originates from the Black Death of the fourteenth century. Indeed, if at first the Plague increased wages across Europe, afterwards, when the population began to grow again, the real wages remained persistently higher in North Europe. The higher flexibility of institutions and guilds allowed a better economic and social response to the Black Death in the North, for example by obtaining lease contracts more advantageous for farmers, or making it easier for women to enter the labor market, and then structurally and radically changing the fertility rate and demographic trends in those countries. Alfani and Percoco ( 2019 ) produce empirical evidence that the plagues that infested Italy in the XVII century also led to long-term reductions in real wages. Indeed, although the population remained below pre-plague levels for more than two centuries, the reduction of skills (as well as of capital and technologies) was particularly large for various reasons. First, these plagues were particularly severe compared with the outbreaks in other European countries. Second, these plagues hit all population strata equally, including the poor, nobles, and bourgeois alike. Moreover, the demographic impact was not compensated by migration flows. Finally, the destruction of human capital reduced the competitiveness of the Italian economy.

Economists disagree on the intensity of the long-term effect of pandemics on the accumulation of human capital. Bleakley ( 2010 ) shows that the effect of the pandemic on schooling is uncertain due to the simultaneous decrease in both benefits (following lessons is more difficult) and opportunity costs (labor productivity is lower). Moreover, he observes that the pandemic could already have negative effects on the intellectual development of the child during gestation. Almond ( 2006 ) supports this argument using 1960–80 decennial microdata to analyze the long-term effects on those US children who were conceived during the Spanish flu. He observes that, if the mother was infected during pregnancy, then her offspring had lower educational attainment and a higher probability of being physically disabled. Both factors reduce their future wages and then increase their participation in illegal activities and, more generally, harm their socioeconomic status 40, 50, and 60 years after the pandemic. Parman ( 2015 ) resizes the effect, affirming that in the US the Spanish flu did not affect human capital in aggregate because parents redirected their investments towards older siblings. Meyers and Thomasson ( 2021 ) show that in 1916 the negative effect of polio on human capital differed between the US states and also depended on the age of students and the family income. However, the effect is usually nonlinear on age and more damaging to the richest because of the specific characteristics of polio.

The relationship between pandemics and human capital accumulation has been studied not only in the US. Odugbesan and Rjoub ( 2019 ) show that, for 26 sub-Saharan African countries from 1990 to 2016, the link between a pandemic and human capital is negative and bidirectional due to persistent short-term effects. Using two Tanzanian databases, Wobst and Arndt ( 2004 ) show that the HIV pandemic has decreased human capital (and then wages and income per capita) in at least four ways. First, the pandemic has directly and persistently reduced labor supply and skills availability. Second, the number of teachers has also decreased, worsening the quality of the process of accumulation of human capital. Third, the lower labor supply has increased the demand for new workers, raising the opportunity cost of education for the young, thus reducing the need for human capital investments. Finally, the pandemic has also reduced the long-term demand for education through an increase in the number of orphans. Novella ( 2018 ) confirms the last link using a Zimbabwean survey for 2007–8. This revealed that orphans leave (secondary) school early and hence enter the labor market early compared with non-orphans. The worst effects emerge when both parents are dead or when the household is blended, i.e., when orphans and non-orphans live together. He also observes that this lower household income after a parent’s death only partially explains the lower investment in the orphans’ human capital. Evans and Miguel ( 2007 ) extend previous results for Kenya. Analyzing an extensive database of over 20,000 children, they observe that not only are orphans more likely to quit primary school, but the probability is higher in those cases where the mother dies and/or their academic performance was already weak. Therefore, they conclude that the inability to pay school fees and the need to find work seem less significant in the long term than the lack of emotional support and the presence of psychological trauma. Fortson ( 2011 ) models the schooling decision that maximizes the expected present value of lifetime utility, considering that HIV reduces its discount rate. He uses data of 15–49-year-olds covering the birth cohorts 1952–91 in 15 sub-Saharan African countries in order to confirm that HIV reduces longevity and human capital investments persistently in the long term. Moreover, the author suggests that both orphans and non-orphans are affected by pandemics, and that decreased schooling provision does not play a key role. Many scholars have analyzed the effects of HIV on educational achievements. Bell and Gersbach ( 2009 ) confirm all previous results by using an overlapping-generations model where both parents and children decide how much to invest in human capital. Moreover, they observe that (i) selective health and educational policies are more effective than comprehensive ones; and (ii) simultaneous health and educational policies are more (less) efficient than sequential ones if disease mortality is above (below) a threshold level.

Young ( 2005 ) combines two fertility effects with the orphan effect. First, if the virus is sexually transmitted, e.g. by HIV infection, then unprotected sexual activities and births are reduced. Second, the labor supply contraction, induced by the pandemic, improves wages and then reduces the mothers’ fertility. Third, lack of parental guidance reduces the human capital of orphans. These emerging long-term effects are mixed. Calibrating the model with South African microdata, he finds that: the female labor supply is more elastic than the male labor supply; fertility effects always prevail in the long term despite pessimistic assumptions; and per capita income tends to increase. Some scholars find that, in addition to human capital, pandemics depress other types of intangible capital. Aassve et al. ( 2021 ) show that the Spanish flu decreased social capital for many generations in the US. They use a long-term social trust survey and discover that: (i) the immigrants born after the Spanish flu and their heirs have lower social trust than those born before; and (ii) the effect is higher for those from countries with less uncensored information on pandemic effects. Using a behavioral experiment in Uganda, McCannon and Rodriguez ( 2019 ) find that grown-up orphans tend to have lower social capital. The probability of prosocial behavior is lower because orphans are more pessimistic about the community’s social contributions. McDonald and Roberts ( 2006 ) analyze data for 112 countries from 1960 to 1998 to determine how much HIV and malaria affect health capital and, consequently, income per capita growth in the long term. They observe that the degree of HIV prevalence in a country negatively affects health capital directly and economic growth indirectly. Moreover, they observe that this mechanism is significant in Africa, through both HIV and malaria, and in Latin America, only through HIV, but not in OECD and Asian Countries. Focusing on sub-Saharan Africa, Odugbesan and Rjoub ( 2019 ) confirm that income plays a key role in explaining the long-term effects of a pandemic. However, the direction of their results is reversed: the bidirectional link between a pandemic and human capital for upper-middle-, low-middle-, and low-income countries is, respectively, negative, positive, and insignificant.

Finally, the majority of effects described in this section are generally more severe in low-income countries. Here, reduced access to medical care, undernourishment, and the presence of other diseases could induce a poverty trap (Beach et al. 2021 ; Bloom et al. 2021 ; Lorentzen et al. 2008 ). A Malthusian equilibrium with low income, underinvestment in schooling and health, and high fertility emerge for tuberculosis (Delfino and Simmons 2005 ) but only partially for malaria (Bloom and Sachs 1998 ; Gallup and Sachs 2001 ). Moreover, the poverty trap is unclear for HIV, where both positive and negative pandemic effects on income distribution could emerge (Bloom and Mahal 1997 ; Bloom and Sachs 1998 ; Mahal 2004 ). Alfani ( 2021 ) suggests that high-mortality pandemics, like the plague, could reduce poverty by either exterminating the poor or redistributing income to the poor. Vice versa, Karlsson et al. ( 2014 ) suggest that low-mortality pandemics, like the Spanish flu, increase poverty due to pandemic-induced unemployment, inability to work for long periods, and general loss of income. As these effects are particularly severe and persistent for poor households, pandemics could aggravate inequality. Therefore, the long-term effects of pandemics on income distribution appear to depend on the medical profile of the disease.

Investments and Physical Capital

While pandemics affect the long-term dynamics of labor supply and human capital also through durable short-term mechanisms, their impact on capital and savings arise in the long term specifically. Acemoglu and Johnson ( 2007 ) argue that, since land and physical capital are not affected in the short term, the lower levels of labor supply and human capital reduce GDP but have an unclear effect on per capita income. Since pandemics reduce GDP and income growth, they also reduce physical capital accumulation, thereby triggering a long-term negative loop between GDP and capital. The authors hypothesize that, in the long term, GDP per capita should drop in high-income countries but not in low-income countries, where land is more relevant than physical and human capital, and the negative loop effect is weaker.

Bai et al. ( 2021 ) confirm that the long-term pandemic effect differs among countries. They show that infectious diseases in the last 15 years have increased permanent volatility in the US, UK, China, and Japan capital markets. However, public policies of correct timing and intensity could reduce the effect. Ru et al. ( 2021 ) find that countries that have already experienced similar pandemics react better and more readily to future pandemics, especially if past pandemics have led to deaths. Analyzing the 65 largest financial markets in the world, the authors note that countries with firsthand SARS experienced the deepest fall in the stock market during the COVID-19 pandemic. This reaction is positively correlated to the pandemic’s mortality. Donadelli et al. ( 2017 ) confirm that, from 2003 to 2014, disease-related news had adversely affected the returns of the pharmaceutical stock market. Analyzing 102 pharmaceutical firms listed on the US stock market, the authors note that investors were too optimistic about the future liquidity of pharmaceutical sector flows after the shock. This irrational behavior has a positive and persistent effect on the returns of the pharmaceutical stock portfolio. Cakici and Zaremba ( 2021 ) extend the previous results outside the pharmaceutical sector. They observe that pandemics induce irrationality among investors, impacting assets across countries and firms heterogeneously. Analyzing 19 international stock markets, they observe that the stock trend signals to investors the firms’ resilience and ability to react to negative shocks, leading to increased future share performance. Summarizing, the literature analyzing the effects of pandemics on equity markets concludes that these health shocks induce irrational behavior of investors, causing positive and negative long-term effects, heterogeneous among countries, sectors, and firms.

Consensus among scholars is lacking in regard to both the size and direction of the long-term pandemic effects on investments and physical capital. Cuddington ( 1993a ) observes that pandemics affect labor demand and capital markets. The total effect on wages is uncertain: supply shock increases wages, but the demand shock reduces them, because infected workers are less effective, as they need to take sick leave and are less productive. Pandemics also affect domestic capital accumulation because health care costs reduce savings. Therefore, the total impact on capital per capita, GDP, and GDP per capita is uncertain; however, calibrating the model with Tanzanian data, he finds that both GDP and GDP per capita sharply decreased from 1985 to 2010. Cuddington and Hancock ( 1994 ) confirm the result for Malawi, although the lower number of infected people reduced the long-term effects on the economy. Moreover, Cuddington (Cuddington 1993b ) observes that previously predicted effects also hold when formal and informal productive sectors coexist, and formal wages are sticky. Basco et al. ( 2021 ) affirm that the Spanish flu in Spain was primarily a demand shock but confirm that the pandemic impact on the real return of capital is ambiguous in the long term. Although at the theoretical level Karlsson et al. ( 2014 ) confirm the ambiguity of the long-term effect of the Spanish flu on the per capita return on capital, this ambiguity is not observed in the empirical analysis of the Swedish counties. Indeed, by analyzing the effects in the decade following the pandemic, the authors estimate no statistically significant effect on earnings per capita, but clear negative effects emerge on capital returns per capita. Finally, Jinjarak et al. ( 2021 ) show that the H3N2 pandemic reduces GDP, consumption, and the investments of 52 countries.

Other scholars demonstrate that the effect of pandemics is heterogeneous among sectors, a trait shared with most disasters (Halkos and Zisiadou 2019 ). In Egypt, pandemics depleted the rural workforce necessary for the maintenance of the crucial centralized irrigation system, which remained in a state of disrepair, hampering the well-being of the region for centuries (Borsch 2005 , 2015 ). Herlihy ( 1997 ) shows that the rise in wages following the Black Death increased demand for more nutritious and elaborate goods, diversifying consumption and improving welfare. Similarly, Pamuk ( 2007 ) shows that the Plague increased the demand for luxury goods in particular. Moreover, he observes a reduction in interest rates and increased investments, although with asymmetric components. Indeed, Alfani ( 2013 ) shows that the XVII century plague depressed Italian industries, in particular, the wool, flax, silk, and construction sectors, due to the loss of skills and the impossibility of procuring raw materials. Alfani and Percoco ( 2019 ) highlight that the shift of investments from urban to rural activities in this period reoriented the post-plague Italian manufacturing sector towards the production of semi-finished and low-quality goods. Summarizing, scholars observe that short-term changes in the relative composition of both demand and supply structures can lead to long-term sectoral effects.

Similar sectoral asymmetric effects have been recorded for more recent pandemics. Analyzing the potential effects of SARS in Asia, Lee and McKibbin ( 2004 ) find that countries specializing in trade and the tertiary sector are more damaged by both temporary and persistent pandemic shocks. Indeed, in these sectors, close contact with other people is often necessary. The retail and tourism sectors are particularly vulnerable. Gallup and Sachs ( 2001 ) provide further support by showing that Mediterranean and Caribbean countries benefited from the rapid and stable development of the tourism industry after the eradication of malaria. Finally, Mahal ( 2004 )‘s literature review on HIV effects shows a similar, although weaker, effect for sub-Saharan tourism. Moreover, the author shows that health, transport, and the primary sectors are also negatively affected by HIV. Pandemics affect the health sector by increasing costs for healthcare services and insurance. Moreover, he shows that workers in the transport and primary sectors belong to the social classes most affected by HIV. Oster ( 2012 ) finds that export is an essential explanation of the spread of HIV in Africa because more truckers and miners, among others, stay away from home for more extended and more numerous periods. As a result, they and their partners are more likely to engage in risky sexual intercourse, putting themselves and their stable partners in danger. She also affirms that trade could further aggravate the effect in the long term, as additional income could increase the amount of money spent on prostitution, or mitigate it, if money is spent on preventive measures. Using a quasi-experimental variation, Adda ( 2016 ) confirms that the new transportation networks and inter-regional trade accelerated disease diffusion in France from 1984 to 2010. Delfino and Simmons ( 2005 ) combine the effect of capital and labor, using a Lotka-Volterra predator-prey model where only healthy individuals are productive. The authors observe that the introduction of capital makes the path more complex, but that the economy still cyclically converges to a stationary equilibrium. Indeed, when labor supply decreases, GDP decreases. Therefore, both savings and investments are lower, and GDP per worker also decreases. Lower welfare reduces health services consumption, but the impact on the disease transmission is uncertain: it increases as the share of infected rises, but it decreases as the contagion period became shorter. When the labor supply increases again, the cycle restarts. Augier and Yaly ( 2013 ) show that complex growth paths could emerge even in a model where the pandemic affects only capital accumulation. The authors describe an overlapping-generations model where the pandemic increases premature deaths, and then only the survivors will use savings previously accumulated. The government proposes a funds system that redistributes rents among the survivors. Young people must decide how much to invest in this public fund, and how much to spend on health or other goods, knowing that better health reduces the chance of dying prematurely. They observe that the pandemic, capital, and health investments are linked in an articulated and recursive way: (i) the pandemic causes health investment to drop; but (ii) health investment reduces the diffusion of the pandemic; (iii) capital directly affects the investment; and then (iv) it indirectly affects the spread of the pandemic. Therefore, the economy converges to a long-term equilibrium only when contagion rates are low. Finally, Stiglitz and Guzman ( 2021 ) show that pandemics act as an unanticipated technology shock, generating unemployment that government intervention can effectively counteract. In the long term, uncertainty does not decline, thus further increasing the desirability of government intervention.

In Section 3 , we showed that, after a pandemic, life expectancy decreases because a healthy lifespan becomes more uncertain than before, leading to decreased investments in human capital. Similarly, scholars observe that the pandemic also reduces investments in physical capital. Lorentzen et al. ( 2008 ) show that the indirect effects of malaria on life expectancy are higher on physical rather than on human capital investments. Analyzing different databases and case studies, Gallup and Sachs ( 2001 ) conclude that the effects on per capita and total income are negative because both foreign investments and the revenues from tourist and business travelers are drastically lower in those countries affected by malaria. Analyzing the effects of HIV on 43 Asian countries from 1990 to 2015, Fawaz et al. ( 2019 ) conclude that investments and savings are usually inversely related to that pandemic. However, they show that both the sign and the intensity of the effect could differ depending on how far-sighted people are. Additionally, in low income countries, the negative effect of investment is independent of gender, but the pandemic affects men’s saving propensity more than women’s. Vice versa, in high income countries, when life expectancy decreases because of pandemic mortality, men save more but do not increase their investments, while women save less but invest more. Bloom and Mahal ( 1997 ) also focus on savings behavior, using it to explain the insignificant effect of HIV on the income per capita growth rate in 51 countries from 1980 to 1992. First, they observe that poor people are most affected by HIV, and that expensive medical treatments further aggravate their disadvantaged situation. However, social and economic mechanisms partially compensate for the high costs of official health services. Second, higher care costs cause both consumption and savings to drop. Moreover, lower life expectancy may increase precautionary savings in favor of surviving family members. Garrett ( 2008 ) studies the economic and social effects of the influenza pandemic 1918–9 in the US, analyzing newspaper articles and academic papers to draw lessons for modern pandemics. He observes that health care is relevant only with ideal health systems that certainly do not collapse after a pandemic, no matter how serious it is. Moreover, he concludes that, although a higher percentage of life insurance mitigates the adverse financial effects of a pandemic on households, the wealthiest households that will need it least will also be the more protected. Gustafsson-Wright et al. ( 2011 ) show that, in the case of pandemics, the private insurance system can be unfair and distortive, even in countries like Namibia, where the quality of public health care is relatively high, and most people have health insurance. The poor who cannot afford health insurance suffer from higher medical expenditure during a pandemic. There are no substantial effects on medical expenditure and family income until the virus starts affecting working capabilities; then, the economic consequences for the poorer strata worsen severely.

The comprehensive review from Hallegatte et al. ( 2020 ) confirms that poor people are disproportionately affected by natural hazards and disasters. Pandemics are no exception. Gaffeo ( 2003 ) provides additional support for the idea that pandemics can lead households into a poverty trap. Higher care costs and physical weakness reduce income capacity: for poor households, this leads to malnutrition, further reducing their physical capabilities, and increasing the pandemic’s morbidity and mortality. Physical and human capital trends reinforce this adverse and cumulative loop. Finally, he observes that pandemics worsen market failures for health insurance and local credit availability. Due to adverse selection and moral hazard, the higher uncertainty and information asymmetries inherent to pandemics lead to higher insurance premiums and reduced access to credit for the needy. Habyarimana et al. ( 2010 ) show that, while private firms could invest in their workers’ medical care, they are unlikely to do so. They describe the case of the pioneering firm Debswana Diamond Company in Botswana, which, since 2001, has invested in a program to improve the health of its workers affected by HIV. They observe that the treatment works, but the investment is unprofitable as the costs are too high, supporting the idea that African firms can only bear a small share of their workers’ health costs, if any.

While the previous literature shows that a pandemic increases income inequalities, Odugbesan and Rjoub ( 2019 ) argue that pandemics could hinder sustainable development. In this connection, these authors analyze the link between HIV and both public and private adjusted net savings, as an indicator of sustainable economic development, for 26 sub-Saharan African countries from 1990 to 2016. They show that HIV negatively and unidirectionally affects saving, and that the effect is particularly intense for upper-middle- and low-income countries. HIV also negatively affects the perception of government efficiency in low-middle-income countries. Odugbesan and Rjoub ( 2020 ) show that, for 23 sub-Saharan African countries from 1993 to 2016, the adverse relationship is bidirectional because the HIV control program and sustainable development compete for the same public spending budget. Keerthiratne and Tol ( 2017 ) show that the financial impact of disasters, pandemics included, is country- and time-specific. Moreover, Chakrabarty and Roy ( 2021 ) propose a model where the future pandemic uncertainty reduces government allocation of non-health expenditures in favor of the health ones. In 143 countries from 2000 to 2017, they found that higher-debt countries present a public misallocation and delay due to public constraints. A similar effect also emerges in low-income countries, but this is due to asymmetric information. Bai et al. ( 2021 ) show that, up to a point, the effects of pandemics could be efficiently mitigated with fiscal and monetary policies. Finally, Cavallo et al. ( 2013 ) confirm that governments and institutions could play a key role in the economic effects of a pandemic. Using a database from the Centre for Research on Epidemiology and Disasters, they observe that natural disasters, such as a pandemic, have a long-term negative economic impact only when they simultaneously cause a high number of deaths and are followed by institutional and political revolutions.

Knowledge and Innovation

Historians have identified numerous cases of pandemics being catalysts of significant, systemic change. In his comprehensive overview of the impact of the Black Death on Europe, Herlihy ( 1997 ) argues that it led to larger economic diversification, improved technology, and better lives, breaking the XIII century Malthusian deadlock by directing technological change towards the now cheaper input, i.e. capital. Although educational institutions were gravely hit, with one-sixth of European universities closed, as a long-term reaction to this short-term impact, a number of new educational institutions were built in reaction to the dearth of scholars. The new universities adopted more flexible curricula, contributing to the revival of classical studies. The need to face the Plague also forced the acceptance and diffusion of anatomical studies, fostering the development of the scientific approach in medicine. Epstein ( 2000 ) offers a similarly positive account, underlining how the Black Death brought much needed renewal. European feudalism was locked in a low-growth pattern, not because of lacking innovative capabilities, or market institutions, but rather due to the intensity of seigniorial rights, and the jurisdictional power of towns and lords, which were used to maximize the extraction of resources, mostly for military purposes, greatly hampering development. The scarcity of workforce caused by the Plague shock reduced the bargaining power of the landowner in favor of the worker. The resulting political and economic struggle is described as a process of “creative destruction”. The centralization process was greatly accelerated, leading to the consolidation of internal markets, the standardization of legal procedures and business norms, and the progressive rationalization of hierarchies. As a result, in the long term, transaction costs and economic uncertainty declined significantly, as testified by the structural decline in interest rates, which quickened the pace of innovation and trade growth. One of the long-lasting consequences of the pandemic for Europe was a more centralized, less predatory authority, able to support the process of economic development.

The institutionally “liquidationist” account of pandemics also applies to other centuries. For example, Alfani ( 2013 ) observes how plagues irrevocably affected the balance of power in Italy, favoring the rise of the House of Savoy, which eventually led to the Italian unification. Pamuk ( 2007 ) describes how the Plague created local skilled labor scarcity, incentivizing migration and fostering the dissemination of knowledge in the long term. Higher wages stimulated the substitution of land and capital for labor, creating conditions favorable to the implementation and diffusion of labor-saving innovations across all economic fields: the printing press, firearms, and high-capacity maritime transportation can all be linked to this general trend. Voigtländer and Voth ( 2013 ) offer what is perhaps the more optimistic view of the long-term impact of the Black Death, arguing that the positive impact of the persistently high European mortality rates dwarfed the effects of technological change for the entire 1500–1700 period. Clark ( 2007 ) provides a useful counterfactual, analyzing how the Far East, relatively less affected by plagues, maintained a growth regime characterized by both low income and low mortality. Not all plagues, however, are described in such a positive light.

Alfani and Percoco ( 2019 ) document the significant negative impact of the plague of 1629–30 on the long-term development of the Italian cities and the Italian economy. In addition to the mechanisms already explored in the previous sections, the authors argue that the significant losses suffered by the urban economic elite, who controlled most of the advanced manufacturing activities, caused an “ingenuity shock”, i.e. decreased both the availability and the willingness of the surviving elite to innovate in the urban industry, preferring agricultural investments instead. The latter took a dramatic hit in terms of production capabilities, which recovered only after decades. The exceptionally late recovery slowed the process of recovery and urbanization, weakening the Italian competitive position vis-à-vis Northern Europe in manufacturing. The almost uniform lack of wage increases signals how the long-term reduction in supply capabilities was not a consequence of lacking a skilled workforce, but rather a significant long-term change in the pattern of capitalist investments. This argument is important to underline how general, systemic renewal might encompass significant relative changes. The hypothesis that the plague did not damage, and perhaps even fostered, European development as a whole, is entirely consistent with the description of significant short- and long-term harm being wrought to large sections of the continental socioeconomic system. This is also consistent with Pamuk ( 2007 )‘s description of the divergence between North and South Europe, which emerged in response to the Plague as a consequence of the greater entrenchment of Southern political and economic elites, and the associated slower degree of institutional flexibility and, consequently, innovation and knowledge diffusion. In his recent overview of the subject, Alfani ( 2021 ) provides further evidence for the relevance of institutional change and policy choices on the long-term impact of pandemics on economic distribution and growth, illustrating how pandemics create opportunities for institutional change while also creating issues that, if not effectively tackled, can severely worsen the economic conditions of the poorer sections of the population.

On the negative side of the debate, Bar and Leukhina ( 2010 ) argue that epidemics have the capability to disrupt knowledge transfer across generations, leading to significant reductions in total factor productivity growth over time. They show that the long-term loss is moderated by the possibility of knowledge diffusion from regions that were spared negative health shocks, implying that the scope of this negative mechanism would be much greater in the case of a pandemic. Karlsson et al. ( 2014 ) document the impact of the Spanish flu on the Swedish economy, finding a long-term negative effect on capital income and a positive effect on the rate of poverty, both possibly driven by a significant persistent loss of skilled workers and consequently a decline in labor productivity. Jinjarak et al. ( 2021 ) show that the H3N2 epidemic can have permanent negative effects on productivity. Indeed, also when the productivity rate returns to its pre-shock level, some opportunities are lost or delayed forever, and then the innovation path will be always lower than without pandemics. Chen et al. ( 2021 ) even state that epidemics have the worst impact on innovation among natural disasters. Indeed, they affirm that epidemics reallocate public expenditure from innovation to health, reducing patent applications and innovation in 49 countries over 1985–2018. In Eastern Europe, feudal lords reacted to epidemics by re-enslaving the peasantry, greatly hampering the diffusion and implementation of new agricultural techniques, and locking the regions in a relative underdevelopment pattern called “second serfdom” (Domar 1970 ; Robinson and Acemoglu 2012 ). Similarly, the plagues affecting the Roman Empire and its successor states led to persistent socioeconomic degradation, aided by conservative political reforms introduced by the surviving elites (Duncan-Jones 1996 ; Sarris 2002 ; Little 2007 ; Harper 2016 ).

Yet pandemics are also great opportunities for the creation and diffusion of new knowledge. Bresalier ( 2012 ) documents how the Spanish influenza pandemic of 1918–9 was a turning point in the modernization of British medicine, leading to the establishment of key institutions and organizations that would shape the long-term development of medical research and healthcare, chief among them the Medical Research Council. The latter led to a wider active involvement of the state in sanitary matters. In general, the author shows that the pandemic’s effects were instrumental in developing the modern medical research system. Hopkins ( 1988 ) provides a description, similar in spirit, of how the successful smallpox eradication campaign conducted by the World Health Organization led to organizational learning, and the development and institutionalization of best practices, thereby greatly enhancing global medical response and prevention capabilities. Furthermore, large shocks, such as pandemics, can create windows of opportunity for change. This is echoed by Cohen ( 2019 )‘s review of the same episode, concluding that, while research and innovation activities played a key role in ensuring the campaign’s success, these efforts were at first greatly hindered by inappropriate practices and institutional routines. Only when the involved organizations implemented new and improved procedures did technological solutions become truly effective. While the scale differs, the argument echoes Pamuk ( 2007 )‘s. Wallace and Ràfols ( 2018 ) show that the avian flu highlighted how both excellence-based funding schemes and economic interests contribute to unduly restrict the field of active research as compared with the broad range of scientific opinions offered by experts, resulting in the development of a limited selection of techniques from the available knowledge base.

Analyzing the impact on the knowledge generation of vaccination subsidies, Finkelstein ( 2004 ) observes that, apart from the direct health impact from the eradication of illnesses, higher expected profitability might lead to socially wasteful competition for market share in the long term. Empirical evidence supports the hypothesis that the outcome depends on the state of the technological frontier and market conditions, as expressed by vaccination rates. In most cases, subsidies appear to lead to purely wasteful competition, but, in the case of the flu, there is evidence of increased product quality and demand, with the associated dynamic benefits outweighing static gains. Consistently, Kremer ( 2000 ) pointed out that market failures are endemic in the markets for both vaccine provision and vaccine research. While this opens up opportunities for policy intervention, it simultaneously underlines the challenges involved in the design of truly effective instruments. Similar challenges are described by Keohane ( 2016 ), in which innovative financial practices developed in the long term as a reaction to large shocks, including pandemics. He argues that “risk transfer for disease is a vital public good that the market has not otherwise provided” (ibid,130), and that new preventive and preparedness measures could be financed through the issue of catastrophe bonds. While these catastrophe bonds are expensive, the benefits of increased resilience in the face of health shocks might be a net gain, especially if the costs are somewhat lessened by pooled funds international initiatives. Although significant overlapping exists in terms of health-crisis preparedness and the organizational capacity of response, such an approach is probably more effective for estimating regional epidemic risks. Such instruments may be particularly useful in light of Confraria and Wang ( 2020 )‘s finding of persistent radical disparity between the disease burden carried by African countries and the amount of medical research dedicated to specifically African issues relative to global efforts.

The discussion so far has been focused on mechanisms that connect pandemics to the development of knowledge and practice, and from those to their economic and financial impact. Easterlin ( 1995 ) provides an original analysis based on a different viewpoint. Analyzing the steep decline in mortality that took place in northwestern Europe in the nineteenth century, he maintains that both the industrial and the health revolutions have a common root: the ascendancy of the scientific approach leading to technological change in both areas. The argument implies, on the one hand, that economic growth is not the main driver of life expectancy improvements, and, on the other, that improvements in health and life expectancy do not have a direct effect on economic outcomes, a position compatible with the relatively weak empirical evidence available (Acemoglu and Johnson 2007 ). The cause of structural change is argued to be found in the extraordinary stream of innovations implemented during the period, supported by a swarm of Schumpeterian “entrepreneurs”, only marginally motivated by profitability. Both those revolutions were triggered by the acceleration in the accumulation of usable empirical knowledge through the establishment and diffusion of the scientific method, the difference in timing to be imputed to the difficulty of developing and implementing the scientific solution. Deaton ( 2004 ) similarly argues that knowledge transfer, in the form of both effective practices and useful information, is key for explaining different national patterns of mortality decline and life expectancy increase, pointing out how globalization could benefit developing countries in this respect.

The argument is further expanded by Easterlin ( 1999 ), who showed that, while private firms have been crucial in fostering economic development, their role in improving health and especially infectious disease control practices has been marginal at best. Indeed, the preventive measures improve life expectancy more than the therapeutic ones, but firms rarely adopt them. However, the actions of households and governments are more important for disease prevention. The role of government is especially relevant because public action is necessary for both health education and prevention programs. Easterlin shows how irreplaceable effective knowledge and healthy practices are in the process of preventing and curing diseases, but how ineffective markets, contracts, and private property institutions have been in fostering their historical development, due to a number of related market failures. In fact, medical practitioners and public servants working towards the diffusion of salubrious norms have often found themselves hindered by economic actors defending their profitable, if deleterious, business. In his account of the US development, Gordon ( 2016 ) confirms both the decisive role of scientific advances and the importance of government intervention and regulation for the drastic improvement in health and life expectancy that took place in the nineteenth and twentieth centuries. However, Birchenall ( 2007 ) proposes an alternative explanation for the manifestly weak correlation between contemporaneous income growth and mortality, highlighting the significant long-term impact of income growth in terms of improved adult health and life expectancy, and subsequent mortality reduction. The argument is supported by a model illustrating how sustained economic growth, no matter the source, is sufficient to escape the Malthusian equilibrium, leading to drastically lower mortality in the process. Cervellati and Sunde ( 2015 ) provide further support by developing a model based on unified growth theory, also characterized by an inevitable take-off triggered by sufficient technological progress.

From the historical description emerges a complex interplay of negative and positive relations between health and business practices, driven by the contrast between short- and long-term interests, on the one hand, and private and public interests, on the other. This complexity is faced by Mokyr ( 2010 ) in his attempt to outline the principles of an evolutionary approach to the study of the development of useful medical knowledge. He begins by highlighting the two key idiosyncratic characteristics of such a knowledge field: the largely inelastic character of its demand, as all humans value their lives and health under all circumstances, and the relevance of negative exogenous shocks, such as the spread of pandemics. Medical knowledge maps to a set of instructions and recipes capable of guiding action, called techniques. According to context-specific selection criteria, only a subset of related techniques will actually be implemented for a given set of knowledge. While the actual usage of techniques is rival, knowledge can endlessly accumulate with only limited downsides. The evolutionary process of knowledge is mostly based on persuasion mechanisms; on the contrary, the related techniques are evaluated on their relative effectiveness. However, persistent empirical failure might not be sufficient as a selection mechanism, if no better technique is available on the basis of the socially accepted set of useful knowledge. This is particularly likely in the case of singleton techniques, based on the limited empirical knowledge that “this works”, and is therefore incapable of adaptation to sudden change. The shift towards scientific knowledge ensures that techniques are based on a more nuanced understanding of natural phenomena, enabling quicker and more efficient adaptation to exogenous shocks. Limits are provided by the path-dependency of knowledge development, which is only indirectly affected by the usefulness of related techniques. While this might result in the generation of “useless” knowledge, degrading response capacity in the present, sudden exogenous changes might lead to equally sudden revaluations. Summarizing, pandemics are a simultaneous shock to both practices and the underlying knowledge, as their often dramatic impact is sufficient to create opportunities for shifting entire development trajectories. The emergence of new knowledge and practices can be further amplified by diffuse and profound institutional change, which in turn may lead to significant upheavals, positive or negative. Owing to the complex nature of the outcomes, however, normative judgment lies beyond the capabilities of purely theoretical analysis.

The first key result of this review is that in the analysis of pandemics’ long-term economic consequences, historical and epidemiological characteristics are key (Donadelli et al. 2021 ; Meyers and Thomasson 2021 ). The extraordinary mortality associated with the Black Death is the most crucial factor in explaining its exceptional long-term consequences for European and global socioeconomic development (Pamuk 2007 ; Voigtländer and Voth 2013 ). Research on the consequences of HIV has rightly focused on its sexual transmission (Young 2005 ; Oster 2012 ; Fawaz et al. 2019 ; Gori et al. 2020 ) and the intergenerational consequences of increased mortality among working-age adults (Wobst and Arndt 2004 ; McDonald and Roberts 2006 ; McCannon and Rodriguez 2019 ). Therefore, the results offered by a general economic analysis of pandemics should be considered a wide collection of potential mechanisms, their empirical applicability and relative importance to be carefully weighed on a case-by-case basis. This does not imply that knowledge is not cumulative in this field, but rather that application of past knowledge should account for contextual factors in order to determine the likely long-term impact of a specific pandemic.

Our review of the literature goes one step further. By aggregating the pandemics by their effects on various economic factors, we observe some recurring trends, allowing some useful general conclusions to emerge. Table  1 provides a comprehensive overview of papers published in English focused on the relationship between pandemics and economic development. Following this paper’s structure, we organize the papers according to the mechanisms investigated into three broad categories: labor and human capital; physical capital and investments; and knowledge and innovation. We show that diseases can potentially affect all the productive factors of an economy in the long term. Most of the articles focus on the pandemic impacts on labor and human capital, all finding negative long-term impacts. However, some authors show that this effect could be partially mitigated in specific geographical areas, workers’ categories, and industrial sectors. The long-term effects on investment and physical capital are ambiguous: many papers show contrasting and complex mechanisms that do not allow us to know a priori the overall economic effects of pandemics on long-term investment trajectories. Notably, all papers which show long-term positive effects of pandemics on economic development focus on knowledge and innovation. However, negative cases also exist, leading many scholars to observe that the effect is potentially mixed, its direction dependent on necessary but not always implemented institutional changes.

The following general picture of the effects of pandemics on economic development emerges from our analysis. First, pandemics tend to reduce population and labor supply in both the short and the long term. This increases labor productivity, and therefore average wages. However, pandemics also hinder human capital accumulation, reducing productivity and per capita income growth. The negative effect is further compounded by the associated loss of knowledge, skills, experience, and innovative capabilities. Investments and savings are also negatively affected, leading to potential long-term hysteresis and the emergence of new, lower-income equilibria.

The pandemic shock can also break old patterns, opening new innovative trajectories previously inaccessible. The aggregate impact of these long-term mechanisms on the economic system is dependent on the relative relevance of, mostly harmful, adaptive mechanisms vis-à-vis potentially fruitful innovative responses. When the latter dominate the picture, negative long-term effects are overwhelmed by the benefits captured by radically new socioeconomic models of production, trade, and consumption. Therefore, the key factor determining the long-term impact of pandemics is identified with the innovation processes to which they give rise, and particularly the necessary accompanying processes of institutional change. While these effects are more difficult to capture using traditional economic methods, they are highlighted by historical analysis and should not be ignored by researchers and policymakers alike (Callegari and Feder 2021b ; Jena et al. 2021 ; Mandel and Veetil 2020 ).

Another important conclusion that can be drawn from this review is that most short-term outcomes, such as the immediate reduction in labor supply (Bloom and Sachs 1998 ; Alfani 2013 ), can bring, in the long term, significantly different consequences in both scope and quality when compared with the transient short-term effects (Delfino and Simmons 2005 ; Acemoglu and Johnson 2007 ; Basco et al. 2021 ). Several specific long-term mechanisms also emerge (e.g., Herlihy 1997 ; Young 2005 ; Augier and Yaly 2013 ), whose impact can hardly be overstated (Lorentzen et al. 2008 ; Voigtländer and Voth 2013 ). Therefore, it is unsurprising that attempts to produce comprehensive quantitative measurements of the economic consequences of pandemics appear to be affected by a significant downward bias (Lee and McKibbin 2004 ; Mahal 2004 ; Keogh-Brown et al. 2010 ). The exceptional nature of the shock brought by the Black Death of 1347–52 has obscured the economic consequences of the other late-medieval plagues, of which we know little. Lack of strong empirical evidence should be understood in the context of the complexity of the phenomena involved, and therefore not be interpreted at first sight as sufficient for falsification purposes. At the same time, however, the mechanisms at work in the most deadly pandemics should not be assumed to apply in exactly the same way to weaker, shorter, or smaller case episodes: the complexity of the phenomena under analysis cannot be reduced to a single formal model. Research on the long-term impacts of pandemics should be understood as a collaborative effort, with single researchers and teams focusing on different, yet compatible, mechanisms. A comprehensive picture can only emerge from subsequent efforts to produce cohesive overviews of the entirety of the debate rather than from a single model, no matter how ambitious.

Some lessons for policymakers also follow. The first is that, in light of the idiosyncratic characteristics of pandemics, precise and detailed analyses of their long-term effects are only possible ex-post. Therefore, preparations for such events should focus on reactive capabilities to ensure that: research efforts can be quickly and adequately supported; their results are credibly communicated to the authorities and the general public; and scientifically-founded counter-measures are rapidly implemented. These characteristics apply to both health measures and economic policy. Furthermore, when these dramatic events occur, the effective public intervention should be timely (Bai et al. 2021 ; Martin et al. 2020 ; Rodríguez-Caballero and Vera-Valdés 2020 ; Stiglitz and Guzman 2021 ) and designed starting from the general characteristics that emerged in this review, and then directed over time by distinctive challenges brought by the specific health shock. The second lesson is that symmetric health shocks will lead to asymmetric economic long-term consequences, as country-specific institutional settings mediate most effects. The tendencies towards the uncritical adoption of global solutions should be tempered by concern for the specific features of local socioeconomic systems, leading to a preliminary process of policy customization. Resistance and push-back from below should not be interpreted automatically as regressive tendencies, but rather as symptoms of the need for policy adaptation to local concerns. Finally, the third lesson is that, while pandemics require careful and extensive public intervention, what matters most in the long term is to avoid crushing the innovative response capabilities of the private sector. A virtuous process of creative destruction may emerge only if public intervention does not attempt to restore the old socioeconomic regime, potentially now unsustainable, at all costs, trampling adaptive bottom-up initiatives in the process. Consequently, while initial efforts should be aimed towards counteracting immediate shocks, they should eventually be complemented by measures aiming to support the positive qualitative developments triggered by the pandemic and curb emerging negative trends. Thus, a potential positive role for policy action can be expected to persist well beyond the outbreak period, focusing on enabling and supporting positive private responses through processes of institutional change.

Conclusions

The COVID-19 pandemic has made evident the need to study the overall economic effects of global health shocks. This literature review collects the main contributions that describe the long-term impact of a pandemic, in order to better understand the lessons from the current economic literature on this topic, and then to better address and analyze the effects on economic development of COVID-19 and of future risks of pandemics. The contributions are organized by discussing, in turn, the mechanisms affecting: labor and human capital; investments and physical capital; and knowledge and innovation. We conclude that pandemics could affect aggregate demand, aggregate supply, and productivity (Jinjarak et al. 2021 ). More precisely, we show that a pandemic reduces labor supply and human capital accumulation in the long term; that the complex interaction of these contrasting and idiosyncratic mechanisms on investments and savings is theoretically indeterminate; and that pandemics, when accompanied by supporting institutional change, can greatly benefit innovation and knowledge development. However, a detailed analysis of the pandemic’s specific characteristics, the affected economic systems, and their response remains necessary to understand which mechanisms can be expected to prevail and which policies should be implemented. The key factors determining their long-term impact are the associated processes of institutional change. We finally identify some general lessons for both researchers and policymakers. The research focuses on the theoretical plausibility and empirical significance of specific mechanisms that should be complemented by meta-analytic efforts aimed at reconstructing the resulting complexity. Policymakers should prioritize developing organizational learning and innovative capabilities, focusing on the ability to quickly adapt to emergencies, rather than developing rigid protocols calibrated over previous pandemics.

We expect the emergence of three new strands of literature in the near future. The first field of research will be on the long-term economic impact of the COVID-19 pandemic (Jordà et al. 2021 ; Poblete-Cazenave 2021 ; Tokic 2020 ). Such research will contribute to testing previously identified mechanisms reviewed here, while potentially also leading to the identification and theorization of new ones (Cacault et al. 2021 ; Silverio-Murillo et al. 2021 ; Costa Junior et al. 2021 ; Favilukis et al. 2021 ; Pagano et al. 2021 ). We also expect significant interest in comparing the impact of COVID-19 with previous pandemics, in order to highlight the relative importance of their respective defining features. The second field of research will be on the public and private responses to the effects of pandemics. The heterogeneity of both the method and timing of the institutional responses for the same health shock can be used to effectively test their efficiency and reduce the impact of future pandemic and epidemic waves (Adolph et al. 2021 ; Caserotti et al. 2021 ; Chakrabarty and Roy 2021 ; Croce et al. 2021 ; Martin et al. 2020 ). The ongoing debate on structural changes as a response to COVID-19 can be seen as a first step in increasing academic attention to the problem of the prediction of possible future pandemics and the precautionary measures to be taken in dealing with these events (Büscher et al. 2021 ; Dosi et al. 2020 ; Leach et al. 2021 ). Finally, we expect a more extensive interaction between, and cross-fertilization of, the medical and economic literatures (Avery et al. 2020 ; Murray 2020 ; Verikios 2020 ). This combination will be needed to better understand how a specific feature of the virus impacts economic development. A taxonomy of pandemics is necessary to group them correctly and then clarify how the different mechanisms move in and impact economic development. In general, we expect the academic debate on the long-term economic impact of pandemics to be renewed and reinforced in the coming years. This survey has the ultimate goal of preparing the basis for this inevitable and intellectually challenging new generation of scientific contributions on the long-term economic effects of pandemics.

Data Availability

Not applicable.

Aassve A, Alfani G, Gandolfi F, Le Moglie M (2021) Epidemics and trust: the case of the Spanish flu. Health Econ 30(4):840–857

Article   Google Scholar  

Acemoglu D, Johnson S (2007) Disease and development: the effect of life expectancy on economic growth. J Polit Econ 115(6):925–985

Adda J (2016) Economic activity and the spread of viral diseases: evidence from high frequency data. Q J Econ 131(2):891–941

Adolph C, Amano K, Bang-Jensen B, Fullman N, Wilkerson J (2021) Pandemic politics: timing state-level social distancing responses to COVID-19. J Health Polit Policy Law 46(2):211–233

Alfani G (2013) Plague in seventeenth-century Europe and the decline of Italy: an epidemiological hypothesis. Eur Rev Econ Hist 17(4):408–430

Alfani G (2021) Economic inequality in preindustrial times: Europe and beyond. J Econ Lit 59(1):3–44

Alfani G, Percoco M (2019) Plague and long-term development: the lasting effects of the 1629–30 epidemic on the Italian cities. Econ Hist Rev 72(4):1175–1201

Almond D (2006) Is the 1918 influenza pandemic over? Long-term effects of in utero influenza exposure in the post-1940 US population. J Polit Econ 114(4):672–712

Arora S (2001) Health, human productivity, and long-term economic growth. J Econ Hist 61(3):699–749

Augier L, Yaly A (2013) Economic growth and disease in the OLG model: the HIV/AIDS case. Econ Model 33:471–481

Avery C, Bossert W, Clark A, Ellison G, Ellison SF (2020) An economist’s guide to epidemiology models of infectious disease. J Econ Perspect 34(4):79–104

Bai L, Wei Y, Wei G, Li X, Zhang S (2021) Infectious disease pandemic and permanent volatility of international stock markets: a long-term perspective. Financ Res Lett 40:101709

Bar M, Leukhina O (2010) The role of mortality in the transmission of knowledge. J Econ Growth 15(4):291–321

Basco S, Domènech J, Rosés JR (2021) The redistributive effects of pandemics: evidence on the Spanish flu. World Dev 141:105389

Beach B, Clay K, Saavedra MH (2021) The 1918 influenza pandemic and its lessons for COVID-19. J Econ Lit forthcoming

Bell C, Gersbach H (2009) The macroeconomics of targeting: the case of an enduring epidemic. J Health Econ 28(1):54–72

Birchenall JA (2007) Escaping high mortality. J Econ Growth 12(4):351–387

Bleakley H (2010) Health, human capital, and development. Ann Rev Econ 2(1):283–310

Bloom DE, Kuhn M, Prettner K (2021) Modern infectious diseases: macroeconomic impacts and policy responses. J Econ Lit forthcoming

Bloom DE, Mahal AS (1997) Does the AIDS epidemic threaten economic growth? J Econ 77(1):105–124

Bloom DE, Sachs JD (1998) Geography, demography, and economic growth in Africa. Brook Pap Econ Act 1998(2):207–295

Borsch S (2005) The black death in Egypt and England: a comparative study. University of Texas Press, Austin

Google Scholar  

Borsch S (2015) Plague depopulation and irrigation decay in medieval Egypt. Med Globe 1(1):125–156

Boucekkine R, Diene B, Azomahou T (2008) Growth economics of epidemics: a review of the theory. Math Popul Stud 15(1):1–26

Bresalier M (2012) Uses of a pandemic: forging the identities of influenza and virus research in interwar Britain. Soc Hist Med 25(2):400–424

Büscher B, Feola G, Fischer AM, Fletcher R, Gerber JF, Harcourt W et al (2021) Planning for a world beyond COVID-19: five pillars for post-neoliberal development. World Dev 140:105357

Cacault MP, Hildebrand C, Laurent-Lucchetti J, Pellizzari M (2021) Distance learning in higher education: evidence from a randomized experiment. J Eur Econ Assoc 19(4):2322–2372

Cakici N, Zaremba A (2021) Who should be afraid of infections? Pandemic exposure and the cross-section of stock returns. J Int Financ Mark Inst Money 72:101333

Callegari B, Feder C (2021a) Entrepreneurship and the systemic consequences of epidemics: a literature review and emerging model. Int Entrepr Manag J. forthcoming

Callegari B, Feder C (2021b) The long-term economic effects of pandemics: toward an evolutionary approach. Industrial and corporate change; dtab064

Caserotti M, Girardi P, Rubaltelli E, Tasso A, Lotto L, Gavaruzzi T (2021) Associations of COVID-19 risk perception with vaccine hesitancy over time for Italian residents. Soc Sci Med 272:113688

Cavallo E, Galiani S, Noy I, Pantano J (2013) Catastrophic natural disasters and economic growth. Rev Econ Stat 95(5):1549–1561

Cervellati M, Sunde U (2011) Life expectancy and economic growth: the role of the demographic transition. J Econ Growth 16(2):99–133

Cervellati M, Sunde U (2015) The economic and demographic transition, mortality, and comparative development. Am Econ J Macroecon 7(3):189–225

Chakrabarty HS, Roy RP (2021) Pandemic uncertainties and fiscal procyclicality: a dynamic non-linear approach. Int Rev Econ Financ 72:664–671

Chen YE, Li C, Chang CP, Zheng M (2021) Identifying the influence of natural disasters on technological innovation. Econ Anal Pol 70:22–36

Cipolla CM (1974) The plague and the pre-Malthus Malthusians. J Eur Econ Hist 3(2):277–284

Clark G (2007) A farewell to alms: a brief economic history of the world. Princeton University Press, Princeton

Book   Google Scholar  

Cohen JM (2019) “Remarkable solutions to impossible problems”: lessons for malaria from the eradication of smallpox. Malar J 18(1):323

Confraria H, Wang L (2020) Medical research versus disease burden in Africa. Res Policy 49(3):103916

Costa DL (2015) Health and the economy in the United States from 1750 to the present. J Econ Lit 53(3):503–570

Costa Junior JC, Garcia-Cintado AC, Junior KM (2021) Macroeconomic policies and the pandemic-driven recession. Int Rev Econ Financ 72:438–465

Croce MM, Arteaga-Garavito MJ, Farroni P, Wolfskeil I (2021) When the markets get COVID: COntagion, viruses, and information diffusion. Working Papers

Cuddington JT (1993a) Modeling the macroeconomic effects of AIDS, with an application to Tanzania. World Bank Econ Rev 7(2):173–189

Cuddington JT (1993b) Further results on the macroeconomic effects of AIDS: the dualistic, labor-surplus economy. World Bank Econ Rev 7(3):403–417

Cuddington JT, Hancock JD (1994) Assessing the impact of AIDS on the growth path of the Malawian economy. J Dev Econ 43(2):363–368

Dauda RS (2019) HIV/AIDS and economic growth: evidence from West Africa. Int J Health Plann Manag 34(1):324–337

Deaton A (2004) Health in an age of globalization. National Bureau of Economic Research

Delfino D, Simmons PJ (2005) Dynamics of tuberculosis and economic growth. Environ Dev Econ 10(6):719–743

Dieppe A (2021) Global productivity: trends, drivers, and policies. World Bank Publications

Domar ED (1970) The causes of slavery or serfdom: a hypothesis. J Econ Hist 30(1):18–32

Donadelli M, Ferranna L, Gufler I, Paradiso A (2021) Using past epidemics to estimate the macroeconomic implications of COVID-19: a bad idea! Struct Chang Econ Dyn 57:214–224

Donadelli M, Kizys R, Riedel M (2017) Dangerous infectious diseases: bad news for main street, good news for wall street? J Financ Mark 35:84–103

Dosi G, Fanti L, Virgillito ME (2020) Unequal societies in usual times, unjust societies in pandemic ones. J Industr Bus Econ 47(3):371–389

Duncan-Jones RP (1996) The impact of the Antonine plague. J Roman Archaeol 9:108–136

Dunn SP (2006) Prolegomena to a post Keynesian health economics. Rev Soc Econ 64(3):273–299

Easterlin RA (1995) Industrial revolution and mortality revolution: two of a kind? J Evol Econ 5(4):393–408

Easterlin RA (1999) How beneficent is the market? A look at the modern history of mortality. Eur Rev Econ Hist 3(3):257–294

Epstein SR (2000) Freedom and growth: the rise of states and Markets in Europe, 1300–1750. Routledge, London

Evans DK, Miguel E (2007) Orphans and schooling in Africa: a longitudinal analysis. Demography 44(1):35–57

Favilukis, J., Lin, X., Sharifkhani, A. and Zhao X. (2021) Labor force telework flexibility and asset prices: evidence from the COVID-19 pandemic. Georgetown McDonough School of Business Research Paper, No. 3693239

Fawaz F, Frey E, Piscitiello D (2019) The effects of HIV mortality on saving and investment in Asia. Asian J Emp Res 9(1):1–15

Fiaschi D, Fioroni T (2019) Transition to modern growth in Great Britain: the role of technological progress, adult mortality and factor accumulation. Struct Chang Econ Dyn 51:472–490

Finkelstein A (2004) Static and dynamic effects of health policy: evidence from the vaccine industry. Q J Econ 119(2):527–564

Fortson JG (2011) Mortality risk and human capital investment: the impact of HIV/AIDS in sub-Saharan Africa. Rev Econ Stat 93(1):1–15

Gaffeo E (2003) The economics of HIV/AIDS: a survey. Dev Pol Rev 21(1):27–49

Gallup JL, Sachs JD (2001) The economic burden of malaria. Am J Trop Med Hyg 64(1):85–96

Garrett TA (2008) Pandemic economics: the 1918 influenza and its modern-day implications. Federal Reserve Bank of St. Louis. Review 90(2):75–93

Gordon RJ (2016) The rise and fall of American growth: the US standard of living since the civil war. Princeton University Press, Princeton and Oxford

Gori L, Lupi E, Manfredi P, Sodini M (2020) A contribution to the theory of economic development and the demographic transition: fertility reversal under the HIV epidemic. J Demogr Econ 86(2):125–155

Gries T, Naudé W (2021) Extreme events, entrepreneurial start-ups, and innovation: theoretical conjectures. Econ Disast Clim Chang 5:329–353

Guerrieri V, Lorenzoni G, Straub L, Werning I (2020) Macroeconomic implications of COVID-19: can negative supply shocks cause demand shortages? (no. w26918). Natl Bur Econ Res

Gustafsson-Wright E, Janssens W, Van Der Gaag J (2011) The inequitable impact of health shocks on the uninsured in Namibia. Health Policy Plan 26(2):142–156

Habyarimana J, Mbakile B, Pop-Eleches C (2010) The impact of HIV/AIDS and ARV treatment on worker absenteeism implications for African firms. J Hum Resour 45(4):809–839

Halkos G, Zisiadou A (2019) Examining the natural environmental hazards over the last century. Econ Disasters Clim Chang 3(2):119–150

Hallegatte S, Vogt-Schilb A, Rozenberg J, Bangalore M, Beaudet C (2020) From poverty to disaster and back: a review of the literature. Econ Disasters Clim Chang 4(1):223–247

Harper KN (2016) People, plagues, and prices in the Roman world: the evidence from Egypt. J Econ Hist 76(3):803–839

Herlihy D (1997) The black death and the transformation of the west. Harvard University Press, Cambridge

Hopkins JW (1988) The eradication of smallpox: organizational learning and innovation in international health administration. J Dev Areas 22(3):321–332

Jena PR, Majhi R, Kalli R, Managi S, Majhi B (2021) Impact of COVID-19 on GDP of major economies: application of the artificial neural network forecaster. Econ Anal Policy 69:324–339

Jinjarak Y, Noy I, Ta Q (2021) Pandemics and economic growth: evidence from the 1968 H3N2 influenza. Econ Disasters Clim Chang:1–21

Jordà Ò, Singh SR, Taylor AM (2021) Longer-run economic consequences of pandemics. Rev Econ Stat:1–29

Karlsson M, Nilsson T, Pichler S (2014) The impact of the 1918 Spanish flu epidemic on economic performance in Sweden: an investigation into the consequences of an extraordinary mortality shock. J Health Econ 36:1–19

Keerthiratne S, Tol RS (2017) Impact of natural disasters on financial development. Econ Disasters Clim Chang 1(1):33–54

Keogh-Brown MR, Wren-Lewis S, Edmunds WJ, Beutels P, Smith RD (2010) The possible macroeconomic impact on the UK of an influenza pandemic. Health Econ 19(11):1345–1360

Keohane GL (2016) Capital and the common good: how innovative finance is tackling the World’s Most urgent problems. Columbia University Press, New York

Kremer M (2000) Creating markets for new vaccines. Part I: Rationale. Innov Policy Econ 1:35–72

Lagerlöf NP (2003) From Malthus to modern growth: can epidemics explain the three regimes? Int Econ Rev 44(2):755–777

Leach M, MacGregor H, Scoones I, Wilkinson A (2021) Post-pandemic transformations: how and why COVID-19 requires us to rethink development. World Dev 138:105233

Lee JW, McKibbin WJ (2004) Globalization and disease: the case of SARS. Asian Econ Papers 3(1):113–131

Little LK (2007) Plague and the end of antiquity: the pandemic of 541–750. Cambridge University Press, Cambridge

Lorentzen P, McMillan J, Wacziarg R (2008) Death and development. J Econ Growth 13(2):81–124

Mahal A (2004) Economic implications of inertia on HIV/AIDS and benefits of action. Econ Polit Wkly 39(10):1049–1063

Malthus TR (1798) An essay on the principle of population as it affects the future improvement of society, with remarks on the speculations of Mr. Godwin, M. Condorcet, and other writers. J. Johnson in St. Paul’s churchyard, London

Mandel A, Veetil V (2020) The economic cost of COVID lockdowns: an out-of-equilibrium analysis. Econ Disasters Clim Chang 4(3):431–451

Martin A, Markhvida M, Hallegatte S, Walsh B (2020) Socio-economic impacts of COVID-19 on household consumption and poverty. Econ Disasters Clim Chang 4(3):453–479

McCannon BC, Rodriguez Z (2019) Orphans and pro-social behaviour: evidence from Uganda. J Int Dev 31(6):495–515

McDonald S, Roberts J (2006) AIDS and economic growth: a human capital approach. J Dev Econ 80(1):228–250

Meyers K, Thomasson MA (2021) Can pandemics affect educational attainment? Evidence from the polio epidemic of 1916. Cliometrica 15(2):231–265

Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6(7):e1000097

Mokyr J (2010) Induced technical innovation and medical history: an evolutionary approach. In: Grübler A, Nakicenovic N, Nordhaus WD (eds) Technological change and the environment (pp. 46–66). Routledge

Murray EJ (2020) Epidemiology’s time of need: COVID-19 calls for epidemic-related economics. J Econ Perspect 34(4):105–120

Novella R (2018) Orphanhood, household relationships, school attendance and child labor in Zimbabwe. J Int Dev 30(5):725–744

Noy I, Managi S (2020) It’s awful, why did nobody see it coming? Econ Disasters Clim Chang 4(3):429–430

Odugbesan JA, Rjoub H (2019) Relationship among HIV/AIDS prevalence, human capital, good governance, and sustainable development: empirical evidence from sub-Saharan Africa. Sustainability 11(5):1348

Odugbesan JA, Rjoub H (2020) Evaluating HIV/AIDS prevalence and sustainable development in sub-Saharan Africa: the role of health expenditure. Afr Health Sci 20(2):568–578

Oster E (2012) Routes of infection: exports and HIV incidence in sub-Saharan Africa. J Eur Econ Assoc 10(5):1025–1058

Pagano M, Wagner C, Zechner J (2021) Disaster resilience and asset prices. Working Paper

Pamuk Ş (2007) The black death and the origins of the ‘great divergence’ across Europe, 1300–1600. Eur Rev Econ Hist 11(3):289–317

Parman J (2015) Childhood health and sibling outcomes: nurture reinforcing nature during the 1918 influenza pandemic. Explor Econ Hist 58:22–43

Poblete-Cazenave M (2021) Simulating the long-term impacts of the COVID-19 pandemic on the sustainability of the population-economy-environment nexus. Econ Disasters Clim Chang:1–16

Rassy D, Smith RD (2013) The economic impact of H1N1 on Mexico's tourist and pork sectors. Health Econ 22(7):824–834

Robinson JA, Acemoglu D (2012) Why nations fail: the origins of power, prosperity and poverty. Profile, London

Rodríguez-Caballero CV, Vera-Valdés JE (2020) Long-lasting economic effects of pandemics: evidence on growth and unemployment. Econometrics 8(3):37

Ru H, Yang E, Zou K (2021) Combating the COVID-19 pandemic: the role of the SARS imprint. Manag Sci 67(9):5606–5615

Sarris P (2002) The Justinianic plague: origins and effects. Contin Chang 17(2):169–182

Schumpeter JA (1954) History of economic analysis. Oxford University Press, New York

Silverio-Murillo A, Hoehn-Velasco L, de la Miyar JRB, Rodríguez A (2021) COVID-19 and women’s health: examining changes in mental health and fertility. Econ Lett 199:109729

Stiglitz JE, Guzman MM (2021) The pandemic economic crisis, precautionary behavior, and mobility constraints: an application of the dynamic disequilibrium model with randomness. Ind Corp Chang 30(2):467–497

Tokic D (2020) Long-term consequences of the 2020 coronavirus pandemics: historical global-macro context. J Corp Account Financ 31(3):9–14

Verikios G (2020) The dynamic effects of infectious disease outbreaks: the case of pandemic influenza and human coronavirus. Socio Econ Plan Sci 71:100898

Voigtländer N, Voth H-J (2013) The three horsemen of riches: plague, war, and urbanization in early modern Europe. Rev Econ Stud 80(2):774–811

Wallace ML, Ràfols I (2018) Institutional shaping of research priorities: a case study on avian influenza. Res Policy 47(10):1975–1989

Weil DN (2014) Health and economic growth. In: Aghion P, Durlauf S (eds) Handbook of economic growth, edition 1, volume 2, chapter 3 (pp. 623–82). Elsevier

Wobst P, Arndt C (2004) HIV/AIDS and labor force upgrading in Tanzania. World Dev 32(11):1831–1847

World Bank (2020) Global economic prospects, June 2020. World Bank, Washington, DC

Young A (2005) The gift of the dying: the tragedy of AIDS and the welfare of future African generations. Q J Econ 120(2):423–466

Zinyemba TP, Pavlova M, Groot W (2020) Effects of HIV/AIDS on children’s educational attainment: a systematic literature review. J Econ Surv 34(1):35–84

Download references

Acknowledgment

The authors would like to thank the editor and the two referees for their useful comments. The usual disclaimer applies.

Author information

Authors and affiliations.

Oslo New University College, Oslo, Norway

Beniamino Callegari

Kristiania University College, Oslo, Norway

CT-TEM - Università della Valle d’Aosta, Aosta, Italy

Christophe Feder

BRICK - Collegio Carlo Alberto, Turin, Italy

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Christophe Feder .

Ethics declarations

Conflict of interest.

The Authors declare that there is no conflict of interest and that no funding has been associated with the writing of this article.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Economics of COVID-19

Rights and permissions

Reprints and permissions

About this article

Callegari, B., Feder, C. A Literature Review of Pandemics and Development: the Long-Term Perspective. EconDisCliCha 6 , 183–212 (2022). https://doi.org/10.1007/s41885-022-00106-w

Download citation

Received : 26 October 2021

Accepted : 09 January 2022

Published : 27 January 2022

Issue Date : March 2022

DOI : https://doi.org/10.1007/s41885-022-00106-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health Shock
• Human Capital

JEL classifications

• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 23 February 2021

Beta-blocker therapy in patients with COPD: a systematic literature review and meta-analysis with multiple treatment comparison

• Claudia Gulea   ORCID: orcid.org/0000-0001-9607-5901 1 , 2 ,
• Rosita Zakeri 3 ,
• Vanessa Alderman 4 ,
• Alexander Morgan 5 ,
• Jack Ross 6 &
• Jennifer K. Quint 1 , 2 , 7  

Respiratory Research volume  22 , Article number:  64 ( 2021 ) Cite this article

29k Accesses

29 Citations

38 Altmetric

Metrics details

Beta-blockers are associated with reduced mortality in patients with cardiovascular disease but are often under prescribed in those with concomitant COPD, due to concerns regarding respiratory side-effects. We investigated the effects of beta-blockers on outcomes in patients with COPD and explored within-class differences between different agents.

We searched the Cochrane Central Register of Controlled Trials, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Medline for observational studies and randomized controlled trials (RCTs) investigating the effects of beta-blocker exposure versus no exposure or placebo, in patients with COPD, with and without cardiovascular indications. A meta-analysis was performed to assess the association of beta-blocker therapy with acute exacerbations of COPD (AECOPD), and a network meta-analysis was conducted to investigate the effects of individual beta-blockers on FEV1. Mortality, all-cause hospitalization, and quality of life outcomes were narratively synthesized.

We included 23 observational studies and 14 RCTs. In pooled observational data, beta-blocker therapy was associated with an overall reduced risk of AECOPD versus no therapy (HR 0.77, 95%CI 0.70 to 0.85) . Among individual beta-blockers, only propranolol was associated with a relative reduction in FEV1 versus placebo, among 199 patients evaluated in RCTs. Narrative syntheses on mortality, all-cause hospitalization and quality of life outcomes indicated a high degree of heterogeneity in study design and patient characteristics but suggested no detrimental effects of beta-blocker therapy on these outcomes.

The class effect of beta-blockers remains generally positive in patients with COPD. Reduced rates of AECOPD, mortality, and improved quality of life were identified in observational studies, while propranolol was the only agent associated with a deterioration of lung function in RCTs.

COPD and cardiovascular disease (CVD) often co-occur, in an interaction characterized by complex biological mechanisms and risk factors such as smoking. Beta-blockers are recommended in treatment regimens of people with heart failure (HF), following myocardial infarction (MI), angina or hypertension, due to proven mortality benefits [ 1 , 2 , 3 , 4 ]. Seventeen years after the publication of the first robust meta-analysis demonstrating that beta-blockers do not impair lung function in patients with COPD [ 5 ], prescription rates remain lower than for people without COPD, among those with an indication for treatment. This treatment gap is thought to be, in part, due to concerns regarding adverse respiratory effects (such as a decrease in lung function) despite accumulating evidence to the contrary[ 6 ]. Concomitant CVD independently affects mortality and hospitalization in patients with COPD, further adding to the clinical burden and complexity of treatment pathways in these patients[ 7 , 8 ].

COPD guidelines recommend the use of cardioselective beta-blockers when appropriate, reinforced by evidence gathered in a Cochrane review [ 9 ]. Data regarding the association of beta-blocker therapy with mortality and acute exacerbations due to COPD (AECOPD) is derived mostly from observational data and previous reviews have aggregated results for cardio and non-cardioselective agents [ 10 , 11 ]. However, a recent single RCT [ 12 ] reported more hospitalizations due to AECOPD in patients treated with metoprolol as compared to placebo, though results on mortality and FEV1 were inconclusive.

Our study expands on previous literature by dissecting the effects of beta-blockers from both RCTs and observational studies, on a wide-range of clinically-relevant end points (mortality, AECOPD, FEV1, all-cause hospitalization and quality of life outcomes such as St. George’s Respiratory Questionnaire (SGRQ), 12 and 6MWT (12, 6 Minute Walking Test) and the Short-Form Health Survey Questionnaire (SF-36), thereby providing a comprehensive assessment of the effects of beta-blocker treatment in COPD. We have two overarching aims: (1) to identify and assess the class-effect of beta-blockers and (2) to compare within-class effects of beta-blockers on the aforementioned outcomes. If all studies have a minimum of one intervention in common with another, it will be possible to create a network of treatments, allowing both direct and indirect evidence to be used in deriving comparisons between beta-blockers not studied in a head-to-head manner, using a network-meta-analysis (NMA). Importantly, we also want to address a current gap in knowledge—we will investigate whether the potential benefits of beta-blockers are limited to those with CVD or may extend in the wider COPD population with or without undiagnosed CVD.

The protocol for this review was previously published [ 13 ]. Searches were conducted from inception to January 2021 in MEDLINE, Embase and CINAHL via Ovid and The Cochrane Collection Central Register of Clinical Trials to identify studies that examined the association between beta-blockers in patients with COPD (defined as post-bronchodilator FEV1/FVC of < 0.70, or as being in accordance with GOLD guidelines [ 6 ]; patients with a clinical diagnosis of COPD) and clinical, safety and quality of life outcomes. To ensure we captured all relevant evidence, we included prospective interventional trials (RCTs) and prospective observational studies (single-arm studies were excluded). At the screening stage, due to a scarcity of prospective observational studies, we decided to also include retrospective observational studies. We required all studies to report on mortality, AECOPD, all-cause hospitalization and quality of life outcomes. We also manually searched reference lists of previously published reviews. Abstracts were screened for inclusion by two independent reviewers, with any discrepancies resolved through discussion. Full texts of included abstracts were screened by a single investigator, and 25% of articles were additionally validated by a second investigator. Full inclusion/exclusion criteria applied at each stage are available in the Additional file 1 : Table S1.

Data extraction and quality assessment

For each accepted study, data was extracted on design, characteristics of study population including comorbidities, inclusion and exclusion criteria, treatment administered and the reported effect of beta-blocker on included outcomes. Details on planned data extraction are available in the protocol [ 13 ]. Authors were contacted to clarify ambiguously reported data from published reports. Included observational studies were assessed for risk of bias using the ROBIN-I [ 14 ] tool for cohort studies and RCTs were assessed using the ROB tool [ 15 ]. Bias domains evaluated include confounding, reporting, attrition, or measurement of outcomes. Each domain was assigned to a risk category such as “low”, “moderate”, “high” or “unclear” for observational studies and “low”, “high” or “some concerns” for RCTs. Additionally, we assessed the certainty of the evidence using the Grading of Recommendations Assessment Development and Evaluation (GRADE) framework [ 16 ].

Searches identified studies reporting on all-cause mortality, AECOPD, FEV1, all-cause hospitalization, SGRQ, the 12 and 6 MWT, and the SF-36. Four researchers extracted data from the included articles, and all were validated by a second researcher.

Data analysis

Where included studies were reasonably statistically and clinically similar, we pooled results using meta-analysis (to investigate class-effect of beta-blocker treatment), or NMA, where data on individual therapeutic compounds was available. Publication bias was assessed using funnel plots if there were at least 10 studies included in meta-analysis [ 17 ]. For binary outcomes we initially included studies that reported on outcomes in any format (Hazard ratio [HR], Odds Ratio [OR], Risk ratio, Incidence Rate); however, the final inclusion list contains only studies reporting HRs since this was the most common amongst included studies. Heterogeneity was assessed using I 2 [ 18 ].

FEV1—Network meta-analysis of RCTs

We performed a random-effects Bayesian NMA to estimate mean change in FEV1 between patients who received individual beta-blockers versus (vs.) placebo with 95% Credible intervals (CrI), using package gemtc [ 19 ] in R v3.6. CrIs represent the 95% probability that the true underlying effect lies in the interval specified. In cases where the standard deviation (SD) for the FEV1 measures was not reported, the SD was extrapolated by averaging the SDs from other studies with similar sample characteristics. Random-effect analyses are widely accepted as the appropriate, more conservative approach when there is heterogeneity across study methods. By contrast, fixed-effect models assume that effect size associated with an intervention does not vary from study to study, and they may be particularly appropriate when only few studies are available for analysis. The best model fit for each network was selected based on a review of the deviance information criterion (DIC) and an evaluation of the different model assumptions.

NMAs include direct and indirect evidence from trials to determine the best available treatment with respect to an outcome of interest. For the results to be valid, NMA assumptions need to be met, including the transitivity and consistency assumptions. For the transitivity assumption to be met, the studies that contribute direct evidence must be similar in distribution of covariates and effect modifiers across the trial populations. Inconsistency occurs when the indirect evidence in a network is different compared to the direct evidence. Assessing consistency of data in the network model is done implicitly in package “gemtc” which uses a decision rule to choose which comparisons may be potentially inconsistent—the “node-splitting” method. Small study effects were explored by conducting comparison-adjusted funnel plots [ 20 ] and publication bias was assessed by Egger’s test among comparisons of beta-blockers and placebo. A value of p < 0.1 indicated significant publication bias. To assess the probability that a treatment is the best within a network, rank probabilities were determined—the probability for each treatment to obtain each possible rank in terms of their relative effects. Interpretation needs to be made with caution, because a treatment may have a high probability of being first, or last treatment and its’ benefit over other treatments may be of little clinical value [ 21 ]. For this reason, we report a full ranking profile (where each treatment is assigned a probability of being first, second, and so on, best treatment in the network) which was derived using the surface under the cumulative ranking curve (SUCRA) [ 22 ].

Sensitivity analyses

We conducted two meta-regressions to establish whether FEV1 measurement at baseline or study duration influenced the main NMA results. These variables were added, separately, as covariates in the main NMA model; FEV1 as a continuous variable and follow-up dichotomised into short follow-up (less than 24 h) vs. long follow-up (more than 24 h). We compared model fit between models with and without covariates using the DIC. Where possible, we analyzed patients with and without CVD separately.

AECOPD—meta-analysis of observational studies

We pooled HRs denoting the association between beta-blocker exposure (vs. no exposure) amongst patients with COPD, using random-effects meta-analysis with the DerSimonian-Lard estimator in “metafor” [ 23 ] package in R v3.6.

Mortality; quality of life—narrative synthesis

If studies were too heterogeneous (I 2  > 75%), or where outcomes were reported in under three studies per treatment comparison, quantitative analysis was not reported, but summary results were graphed on forest plots without pooling the results (mortality) and/or synthesized qualitatively (quality of life outcomes).

The database search identified 2932 potentially relevant articles whilst other sources revealed six. After title and abstract screening, 187 articles underwent full-text review. We included 23 observational studies and 14 RCTs that reported on patients with COPD, in the systematic literature review. Out of a total of 23 observational studies, 21 reported on mortality [ 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ], five reported on AECOPD [ 24 , 33 , 35 , 45 , 46 ] three reported on all-cause hospitalization [ 47 , 48 , 49 ], one reported on SGRQ [ 45 ] and one reported on SF-36 [ 42 ]. From a total of 14 RCTs, 12 reported on FEV1 [ 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 ], two each reported on 12MWT [ 59 , 62 ] and 6MWT [ 12 , 56 ] and two reported on SGRQ [ 12 , 56 ] (Fig.  1 ).

According to our protocol, we intended to include data on effect of beta-blockers on AECOPD from RCTs, however our search strategy revealed only one study of this kind [ 12 ]. Based on a population of 532 patients with moderate to severe COPD, the authors reported no significant difference in time to first AECOPD (of any severity) between metoprolol and placebo, however the use of the beta-blocker was associated with a higher risk of severe exacerbation (requiring hospitalization). This study could not be included in the quantitative analysis, as there was no other RCT data to corroborate.

Quantitative analyses

There were five [ 24 , 33 , 35 , 45 , 46 ] observational studies reporting on the effect of beta-blockers on AECOPD in patients from at least five countries across Europe. Follow-up varied from 0.76 [ 46 ] to 7.2 years [ 33 ]. The average age of the patients ranged from 62.8 [24] to 74 [46] years old and the proportion of males from 49.8% [33] to 72.3% [45]. Only two studies reported on smoking status [ 33 , 45 ], which indicated the majority of patients were either current or former smokers. Comorbidities were frequent in all cohorts, specifically CVD, reported in all but one study [ 35 ]. Body mass index (BMI) was reported in only two studies and ranged between 25.5 [ 45 ] and 29. 9 kg/m 2 [ 24 ]. All study characteristics are available in Additional file 1 : Table S2 and Table S3.

In the presence of low statistical heterogeneity (< 25%), the random effects and fixed effects method for pooling effect estimates give identical results. Due to low heterogeneity (I 2  = 0, owing to the large weight attributed to one study only [ 46 ]) and the small overall number of studies, we report both random and fixed-effects meta-analyses of AECOPD. In random-effects analysis, the pooled estimate risk of AECOPD associated with beta-blocker use, from an total of 27,717 patients, was HR 0.78 [95%CI 0.74–0.82] suggesting a reduction in relative risk in the presence of beta-blockers (Fig.  2 , Additional file 1 : Table S4 for individual study outcomes). The fixed-effects meta-analysis yielded similar results (Additional File 1 : Figure S1). Due to low number of studies we could not formally assess the extent of publication bias. The GRADE assessment indicated the overall quality of evidence based on which the meta-analysis was conducted was low (Additional file 1 : Table S18).

Forest plot illustrating results of the meta-analysis evaluating the impact of beta-blocker therapy versus no beta-blocker therapy on AECOPD in patients with COPD (Estimate: HR hazard ratio, 95% CI confidence interval)

Data from 12 RCTs evaluating FEV1 in 199 patients and seven beta-blockers (atenolol, bisoprolol, carvedilol, celiprolol, metoprolol, propranolol, labetalol) were evaluated [ 50 , 51 , 52 , 53 , 54 , 55 , 57 , 58 , 59 , 60 , 61 , 63 ]. Duration of trials varied from 1 hour [ 53 , 59 ] to 3–4 months [ 57 ] and FEV1 measurement at baseline between 1.15 [ 59 ] and 2.41 L [l][ 61 ]. Most patients were over 40 years old except for one study where mean age was 39 [ 60 ]. Across all studies, over 50% of the patient population were male and four studies only included patients with CVD or hypertension explicitly [ 50 , 54 , 55 , 57 ] (Additional file 1 : Table S5). A comparison between studies enrolling patients with CVD and those enrolling patients with COPD only is difficult due to scarcity of reported data. BMI was available in two studies of COPD and CVD [ 55 , 57 ] and in one study only which excluded CVD [ 58 ]. Estimates were however similar and denoted overweight, but not obese patient populations. Celiprolol was the only treatment which was evaluated in patients without CVD exclusively, in one trial [ 61 ] only. Sample size, age and proportion of males were similar across all studies.

Figure  3 shows the network of eligible comparisons for FEV1 mean change from baseline to time-point, including seven treatments. All beta-blockers except carvedilol were evaluated in at least one placebo-controlled trial. Individual study FEV1 measurements are presented in Additional file 1 : Table S6. Figure  4 and Additional file 1 : Table S7 show the NMA results for FEV1. Consistency results are illustrated in Additional file 1 : Figure S2. Effects relative to placebo are presented separately for each treatment.

Network of beta-blockers used to treat patients with COPD, from RCTs assessing FEV1

Network meta-analysis results for mean difference in FEV1 (95 CrI), beta-blockers compared to placebo [measured in liters, CrI  credible intervals]

There was no significant difference in FEV1 amongst all beta-blockers except for propranolol, which was the only treatment associated with a decrease in FEV1 (mean difference [MD]:− 0.14 ml, 95% CrI,  0.28 to 0.016). Individual medications were ranked and are presented with estimates of the probability that each is the best treatment (i.e. probability that the treatment improves lung function). Figure  5 shows that celiprolol had the highest likelihood of being ranked best treatment, followed by labetalol. For the second rank, the same treatments appear the most likely. Overall, the SUCRA results based on the rankogram values appear to suggest labetalol (86.2%) and celiprolol (80%) are the most likely of being the best treatments to positively affect FEV1, whilst propranolol was the least likely (16.2% probability of being the best) (Additional file 1 : Table S9). According to the comparison-adjusted funnel plot, no publication bias was found for Egger’s test (p = 0.1286, Additional file 1 : Figure S3).

Rankogram illustrating probabilities that each treatment is first, second, third…eighth with regards to FEV1 improvement

The meta-regression analyses, with baseline FEV1 measurement, follow-up duration, respectively, added as covariates, showed similar results to the main analysis (model fit did not improve in either model with added covariates, Additional file 1 : Figure S4).

Beta-blocker therapy effect on FEV1 in patients with COPD with and without explicit CVD

Data from eight trials evaluating six beta-blockers (atenolol, bisoprolol, carvedilol, celiprolol, metoprolol, and propranolol) in 137 patients with COPD and no explicit CVD were evaluated [ 51 , 52 , 53 , 56 , 58 , 59 , 60 , 61 ]. No significant difference in FEV1 was detected when comparing each of the active treatments with placebo (Additional file 1 : Figure S5A). Additional file 1 : Figure S6 shows celiprolol was similarly likely to rank first in terms of increasing FEV1, while the second rank was surprisingly obtained by placebo, then celiprolol. There were four trials investigating six  beta-blockers (carvedilol, bisoprolol, atenolol, propranolol, metoprolol, labetalol) in patients with COPD and CVD [ 50 , 54 , 55 , 57 ]. No significant difference in FEV1 was detected when comparing each of the active treatments with placebo (Additional file 1 : Figure S5B, Additional file 1 : Figure S7).

Narrative synthesis

There were 21 observational studies reporting on mortality [ 24 , 25 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 43 , 44 , 64 ] which evaluated the effect of beta-blockers vs. no beta-blocker use, in an overall population of 422,552 patients from at least 11 countries (Additional file 1 : Table S2). According to inclusion criteria, 15 studies enrolled patients with COPD and a CVD indication [ 25 , 27 , 28 , 29 , 34 , 37 , 39 , 40 , 43 , 64 , 65 ], while the remaining six [ 24 , 26 , 32 , 33 , 35 , 44 ] did not specify whether those with CVD were specifically excluded; however, all studies had varying percentages of CVD comorbidities. Overall, patient characteristics varied: mean age ranged between 62.8 [ 24 ] and 84.6 [ 44 ] years old and the proportion of males between and 37% [ 26 ] and 100 [ 44 ]%. Distribution of comorbidities was mixed, with hypertension being the most widely reported and ranging between 27.5% [ 33 ] and 88.3% [ 37 ]. Smoking status was reported in seven studies [ 25 , 26 , 28 , 31 , 33 , 41 , 44 ] where most patients were recorded as being either current or former smokers, however data was not available consistently. BMI was reported in five studies [ 24 , 28 , 29 , 41 , 44 ] only, ranging between 20.4 [ 29 ] and 29.9 [ 24 ]. Follow-up time was also highly variable, ranging from 2 [ 30 ] to 112 months [ 40 ].

Individual adjusted study risk estimates for mortality associated with beta-blocker use vs. no beta-blocker use ranged from HR 0.46 (95%CI 0.19–1.11) [ 29 ] to 1.19 (95%CI 1.04 to 1.37) [ 26 ] (Fig.  6 ). While age and sex were the most common covariates adjusted for, the majority of studies used a variety of study-specific variables: medications for specific indications (such as hypertension, HF) and other comorbidities or clinical variables (Additional file 1 : Table S10). Two studies reported unadjusted analyses [ 27 , 28 ]. There was one study only reporting an increase in mortality risk associated with beta-blockers (HR: 1.19, 95% CI 1.04 to 1.37); however the population assessed in this report consisted of severe COPD patients who were undergoing long-term oxygen therapy [ 26 ]. There was a very high degree of heterogeneity amongst studies (I 2  = 99.3%). This was explored by conducting stratified analyses (i.e. stratifying by type of beta-blocker [cardioselective vs. non-cardioselective, Additional file 1 : Figure S8]; excluding unadjusted estimates; excluding the only study which exclusively included very severe COPD patients). However, due to heterogeneity remaining very high (I 2  > 75%), results from the outcome analysis are presented graphically (Fig.  6 ).

Forest plot illustrating the impact of beta-blocker therapy versus no beta-blocker therapy on mortality, in patients with COPD (Estimate: HR hazard ratio, 95% CI confidence interval)

All-cause hospitalization

All-cause hospitalization was reported in three observational studies [ 47 , 48 , 49 ]. One compared cardioselective beta-blockers to non-selective beta-blockers (and presented odds ratios [OR]) [ 48 ]; one compared non-cardioselective beta-blockers to selective beta-blockers (and presented HR) [ 49 ] and one compared cardioselective beta-blockers to lack of beta-blocker treatment (and presented relative risk) [ 47 ] therefore no class-effect comparison could be inferred. None of the studies found significant differences in all-cause hospitalization associated with the investigated treatments (Additional file 1 : Table S11).

Quality of life

SGRQ was assessed in two RCTs [ 12 , 56 ] and one observational study [ 45 ], but none reported mean change from baseline to follow-up per treatment arm. One RCT [ 66 ] compared metoprolol to placebo and one observational study [ 45 ] assessed any beta-blocker compared to lack of beta-blocker treatment; both reported no significant difference in SGRQ between the two treatment arms at one-year follow-up (Additional file 1 : Table S12).

12MWT was investigated in two RCTs [ 59 , 62 ]; one study investigated atenolol and metoprolol vs. placebo, which did not report a mean change in score at  four weeks follow-up [ 62 ]; the second study did not find a significant difference in distance walked between patients that received metoprolol vs. propranolol  six hours after treatment was administered [ 59 ] (Additional file 1 : Table S13).

Data on 6MTW was reported in two recent RCTs from 2017 [ 56 ], respectively 2019 [ 12 ]. The first evaluated the effect of bisoprolol compared to carvedilol and did not present mean change between treatment groups, however the calculated estimates suggest both agents decreased distance walked in patients with COPD with no difference apparent between the two; the second trial [ 12 ] did not identify a significant difference between the metoprolol and placebo on 6MWT (Additional file 1 : Table S14).

Data on SF-36 was available in one observational study [ 42 ]. Whilst overall scores were not available per treatment group, authors reported no significant association between beta-blocker treatment and individual domains of the quality of life assessment tool, either at baseline or 6.4 years follow-up (Additional file 1 : Table S15).

Risk of bias

Observational studies were mostly judged to have moderate risk of bias (23 studies [ 24 , 25 , 26 , 28 , 30 , 31 , 32 , 33 , 34 , 35 , 37 , 38 , 39 , 40 , 41 , 42 , 46 , 47 , 48 , 49 , 64 , 65 ]), two studies [ 25 , 45 ] were considered to be of low risk of bias, one [ 44 ] had serious risk of bias and one [ 27 ] did not provide enough information for a judgment to be made. The domains of bias which were mostly affected by a “moderate rating” were “bias due to confounding” and “bias in selection of participants into study” as the majority of studies included patients recruited from databases which did not provide clinical diagnoses and relied on ICD coding (without confirming validity of diagnosis) (Additional file 1 : Table S16). Ten RCTs [ 53 , 55 , 58 , 60 , 61 ] had moderate risk of bias, denoted by ratings of “some concerns”; two[ 56 , 57 ] studies were deemed of serious risk of bias, both due to the lack of blinding (Additional file 1 : Figure S9).

This comprehensive and up-to-date evaluation of the effects of beta-blockers in patients with COPD adds to the previous literature in several ways: we included all studies reporting on any type of beta-blocker treatment in patients with COPD, showing overall beneficial effects on AECOPD and mortality. For the first time, we used a probabilistic approach to evaluate the effect of beta-blockers on FEV1 using direct and indirect evidence from RCTs in an NMA, comparing seven treatments against placebo, and presented results for patients with COPD with and without CVD disease separately. No beta-blocker affected lung function significantly except propranolol, and the treatments less likely to have a detrimental effect on FEV1 were labetalol (in those with COPD and CVD) and celiprolol (in those with COPD without explicit CVD). Lastly, we found that data on all-cause hospitalization and quality of life endpoints such as SGRQ, 12 and 6MWT and SF-36 were scarcely reported across the literature and did not lend themselves to formal quantitative analysis—suggesting an area of focus for future studies.

Despite heterogeneous elements such as follow-up time, baseline characteristics including age, sex and comorbidities and geographical location, individual results from the 17 out of 21 studies reporting on mortality suggested beta-blocker therapy was associated with a diminished risk of death compared to those not prescribed beta-blockers, in patients with COPD. However, this quality of evidence was deemed “low” per GRADE assessment (Additional file 1 : Table S17) and we were not able to quantify the effect of beta-blockers on mortality due to considerable heterogeneity (I 2  > 75%). Previous reports [ 10 , 11 , 67 ] have provided pooled estimates of reductions in mortality risk associated with beta-blocker treatment, however all reported degrees of heterogeneity above the Cochrane I 2 threshold of 75%; 89.3% [ 10 ], 83% [ 11 ] and most recently 96% [ 67 ] bringing into question the validity and interpretability of these results as applied to the general COPD population. Reasons for very high heterogeneity in previous meta-analyses include: differences in study populations (i.e. including patients with differing degrees of severity), inaccurate risk of bias assessment and inclusion of different comparators for the intervention effect of interest (i.e. including studies where comparator arms received calcium channel blockers, despite aiming to assess the effect of beta-blocker treatment vs. lack of treatment) [ 67 ].

In our analysis, most studies were affected by bias, particularly due to confounding: two studies did not adjust for any covariate factors [ 27 , 55 ], whilst nine did not adjust for COPD severity either directly, or indirectly by including COPD medication regimen/exacerbation history in the final model [ 25 , 26 , 27 , 28 , 30 , 32 , 34 , 36 , 37 ]. Therefore, these studies may overestimate the prognostic effect of beta-blocker therapy on patients with COPD and may, in turn, skew results to show benefits. One of the reasons for the lack of adjustment for COPD-related variables may be due to using data from either existing drug-trials or CVD-specific registries which included data on subgroups of patients with COPD, reiterating the need for trials designed specifically for patients with COPD (with and without additional CVD) which may allow for reliable assessment of the true effect of beta-blockers in these patients. Furthermore, it is not surprising to observe a decrease in mortality, as this could be related to the effect of beta-blockers on other comorbid conditions of patients (i.e. CVD), which is established. A previous study [ 33 ] suggested long-term treatment with beta-blockers improved survival of patients with COPD without CVD, however future studies are needed to confirm this result and to assess whether beta-blockers provide non-CV mortality benefits.

We found evidence to suggest that patients with COPD who are given beta-blockers are at decreased risk of AECOPD (HR 0.78 [95%CI 0.74–0.82]), replicating findings from Du and colleagues [ 10 ] who report an even larger reduction in risk, of 37% (RR 0.63 [95% CI, 0.57–0.71]). However, this previous meta-analysis, had methodological limitations inherent to the observational nature of the pooled studies (i.e. residual confounding, immortal time bias), which may limit generalizability of results. However, the GRADE assessment revealed the body of observational evidence on which our estimate was derived was of “low” quality (Additional file 1 : Table S19). A recent RCT [ 12 ], less likely to be affected by the biases of previous observational studies, found no significant difference between metoprolol and placebo on the time to AECOPD of any severity, but revealed a significant increase in risk of AECOPD requiring hospitalization, in patients with COPD without an indication for beta-blocker treatment, bringing into question the protective effect of this specific beta-blocker agent.

However, this trial did not evaluate other beta-blockers, therefore future RCTs evaluating multiple regimens, are needed to confirm the benefit of these agents. Whether beta-blockers have an indirect effect on exacerbations of COPD could be assessed in clinical trials including patients with COPD and comorbid CVD, allowing assessment of these agents in a more representative COPD population.

FEV1 was assessed in 199 patients enrolled in 12 RCTs and we found that none of the individual cardioselective beta-blockers included in our NMA (atenolol, bisoprolol, celiprolol, metoprolol) were associated with significant effects on lung function in patients with COPD, regardless of baseline FEV1 or follow-up time. This is in line with a Cochrane review [ 9 ] which concluded that cardioselective beta-blockers given in either single dose or for longer durations, do not affect FEV1 in patients with COPD, even in those with the lowest baseline FEV1 measurements. Furthermore, our report extends to incorporate a lack of effect on FEV1 of non-selective beta-blockers such as carvedilol and labetalol. Propranolol was the only medication found to be associated with a reduction of 140 ml in FEV1 (95% CrI: -0.28, -0.016), which is larger than the threshold of 100 ml change deemed clinically significant by the American Thoracic Society and European Respiratory Society guidelines. This result is based on high quality evidence, according to the GRADE assessment (Additional file 1 : Table S19), and thus supports current recommendations to not use this medication in patients with COPD.

For the first time reported in the literature, we aimed to rank beta-blockers with respect to their effect on lung function. Propranolol had the lowest probability of being ranked first (suggesting worse impact on lung function), compared to all other individual treatments considered in our NMA, including placebo. Labetalol and celiprolol—drugs used in hypertension—were the least likely drugs to negatively impact FEV1, compared to all other beta-blockers; however, neither affected FEV1 with certainty compared to placebo and results were  inferred from very low quality evidence according to GRADE (Additional file 1 : Table S18), bringing into question their leading positions in the hierarchy. Since choice of beta-blocker may be influenced by CVD comorbidity (i.e. carvedilol, metoprolol and bisoprolol are recommended in stable HF; atenolol is more often prescribed in patients with asymptomatic hypertension, while bisoprolol is also used in atrial fibrillation, and propranolol is infrequently used to treat tachyarrhythmias), it is perhaps not surprising that we did not identify a clear “best” beta-blocker to be used in COPD. The fact that the beta-blockers less likely to decrease lung function are mainly used to treat hypertension may just reflect this subgroup of patients could be less prone to detrimental side-effects (i.e. indication bias), compared to others with COPD and more severe comorbidities. Indeed, the prescription of beta-blockers in COPD needs to consider clinically significant lung function alteration vs. mortality benefits in those with CVD, particularly MI [ 68 ] and HF [ 69 ].

Whilst CVD is diagnosed in 20 to 60% patients with COPD [ 70 ], our main analysis included primarily small trials and only three explicitly included patients with a cardiac comorbidity (one included angina [ 54 ], two included HF patients [ 55 , 57 ], and one included patients with hypertension, which is a common CVD risk factor [ 50 ]. In line with previous research [ 9 ], we report no significant FEV1 treatment effect in patients with COPD with CVD.

The remaining eight trials excluded those with CVD (or simply did not report whether this was present), and results mirrored those observed for patients with CVD. Whilst results from this subgroup analysis are encouraging, previous clinical data on in this subgroup is scarce. A recent single RCT including COPD patients without an indication for beta-blockers (therefore those with HF, previous MI or revascularization) failed to demonstrate clear benefits of metoprolol over placebo. Observational studies have included a more varied breadth of specific beta-blockers, however they do not present a clear picture: the population-based Rotterdam Study [ 71 ] reported significant decreases in FEV1 associated with both cardio and non-cardioselective beta-blockers, while two other studies, one from Scotland [ 35 ] and an one from Japan [ 72 ] reported no significant difference in FEV1. Yet, these results may be affected by confounding by indication, which may explain the variability of estimates. Additionally, the longer follow-up times in these studies (ranging from 4 to 6 years) may overlook effects of FEV1 decline which is documented in patients with COPD, regardless of CVD comorbidities.

Overall, our FEV1 analysis suggests the beta-blockers included in this review do not affect lung function in patients with COPD regardless of CVD disease status, and selectivity of agent does not appear to have an impact. However, the two treatment networks contained different medications (celiprolol was assessed in one trial excluding CVD, while labetalol in one trial including CVD) thus we cannot rule out any other potential differential results if a whole range of beta-blockers were included. Finally, we included evidence based on a relatively small population and some of the studies were conducted decades ago; therefore, large clinical studies are needed to assess other agents which may confer lung function benefits across contemporary COPD patients.

The effect of beta-blocker exposure on all-cause hospitalization and quality of life outcomes in patients with COPD could not be quantified, due to a paucity of data. Narrative results from the assessment of studies investigating quality of life outcomes, such as SGRQ, 12 and 6 MWT and SF-36 all suggest non-significant effect of beta-blockers, from both RCTs and observational studies, albeit the data was deemed to be of “very low” quality according to GRADE (Additional file 1 : Table S17). Currently, COPD management is focused on preventing exacerbations and improving functioning and health-related quality of life. Clinical studies of beta-blocker treatment in cardiac disease suggests improvements in exercise tolerance and functional status, so whether beta-blockers impair or improve these outcomes in patients with COPD also, is a topic of importance for clinical management. Both randomized trials and, importantly, prospective observational studies with longer follow-up times are needed.

Limitations

There are several limitations to our analysis: first, we included published, peer-reviewed literature only thus, results may affected by publication bias as it is more likely that studies reporting positive results (i.e. that did not find beta-blockers were associated with negative outcomes) are more often reported than negative studies. Nevertheless, our data is based on the most recent available evidence and portray a nuanced implication of specific beta-blocker treatment in patients with COPD, emphasizing the need for a targeted treatment of CVD comorbidity in these patients.

We only included stable COPD patients and whilst we showed that FEV1 reduction (or increase) was not significant according to beta-blocker exposure (apart for propranolol), we could not verify whether these therapeutic agents diminish the response to rescue COPD medication such as beta-agonists, administered during an exacerbation of COPD. We also did not verify long-term effects of co-administration of beta-blockers and beta-agonists and how their interaction may affect outcomes in patients receiving both types of medication.

Another issue is undiagnosed CVD in patients with COPD. Symptoms of ischemic heart disease or HF may be misattributed or overlapping with COPD, and thus not formally diagnosed, posing difficulties in disentangling possible non-cardiac effects of beta-blockers, independent of their proven cardiac benefits. One advantage of our FEV1 analysis is that we included RCTs only, where concomitant CVD is often ascertained more rigorously and therefore CVD status was known with a greater degree of confidence that may be the case in observational studies.

Furthermore, no statistically significant effect was detected in subgroup analyses stratified by CVD status, which may be due to limited sample size. Future, adequately powered RCTs are needed to assess the effect of beta-blockers in a diverse COPD population, allowing for accurate comparisons based on CVD status to be made.

A recent RCT [ 12 ] comparing metoprolol with placebo failed to find a significant effect on FEV1, but reported worsening of dyspnea and overall COPD symptoms, suggestive of respiratory effects not captured by spirometry. This confirms the need to evaluate a spectrum of respiratory outcomes to fully assess the implications of beta-blocker treatment in patients with COPD, which needs to be addressed in future studies

Confounding by contraindication is likely to affect interpretation of results—if we assume clinicians knowingly withheld treatment from patients due to concerns regarding breathlessness, this may have resulted in a reduced sample size of possible COPD patients who may have been eligible for beta-blocker therapy. Alternatively, doctors may prescribe beta-blockers to less severe patients, limiting generalizability.

Our AECOPD analysis is also limited by a low number of included studies, all of which were observational—we identified one RCT only (evaluating metoprolol). This reinstates the need of more carefully conducted RCTs to evaluate a range of beta-blockers and their effects of AECOPD, in order to validate observational data.

Findings from this analysis represent the most comprehensive and up-to-date available evidence synthesis to assess the effects of beta-blocker use in patients with COPD, spanning data published over four decades. A reduction in COPD exacerbation risk was inferred from observational data while clinical data were pooled to assess lung function. Mortality and quality of life were narratively described owing to high heterogeneity or sparsity of data, respectively. FEV1 was significantly impacted by propranolol, but not by atenolol, bisoprolol, carvedilol, celiprolol, labetalol or metoprolol. In the subset of individuals with CVD, no individual beta-blocker was associated with a reduction in lung function. Treatment choice in patients with COPD should be made according to CVD comorbidity guidelines on management.

Availability of data and materials

The datasets analyzed during this study are available from the corresponding author upon reasonable request.

Abbreviations

6-Minute walking test

12-Minute walking test

Acute exacerbation due to COPD

Body mass index

Chronic obstructive pulmonary disease

Confidence interval

Credible interval

Cardiovascular disease

Forced expiratory volume in one second

Heart failure

Hazard ratio

• Network meta-analysis

Myocardial infarction

Short-Form Health Survey Questionnaire

St. George’s Respiratory Questionnaire

Surface under the cumulative ranking

Randomized controlled trial

Heidenreich PAMK, Hastie T, Fadel B, Hagan V, Lee BK, Hlatky MA. Meta-analysis of trials comparing β-blockers, calcium antagonists, and nitrates for stable angina. JAMA. 1999;281(20):1927–36.

Article   CAS   PubMed   Google Scholar  

Ponikowski P, Voors AA, Anker SD, Bueno H, Cleland JGF, Coats AJS, et al. 2016 ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure: The Task Force for the diagnosis and treatment of acute and chronic heart failure of the European Society of Cardiology (ESC) Developed with the special contribution of the Heart Failure Association (HFA) of the ESC. Eur Heart J. 2016;37(27):2129–200.

Article   PubMed   Google Scholar  

Task Force on the management of STseamiotESoC, Steg PG, James SK, Atar D, Badano LP, Blomstrom-Lundqvist C, et al. ESC Guidelines for the management of acute myocardial infarction in patients presenting with ST-segment elevation. Eur Heart J. 2012;33(20):2569–619.

Article   CAS   Google Scholar  

Whelton PK, Carey RM, Aronow WS, Casey DE Jr, Collins KJ, Dennison Himmelfarb C, et al. 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA Guideline for the Prevention, Detection, Evaluation, and Management of High Blood Pressure in Adults: Executive Summary: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. Hypertension. 2018;71(6):1269–324.

Salpeter SR, Ormiston TM, Salpeter EE, Poole PJ, Cates CJ. Cardioselective beta-blockers for chronic obstructive pulmonary disease: a meta-analysis. Respir Med. 2003;97(10):1094–101.

Vestbo J, Hurd SS, Agusti AG, Jones PW, Vogelmeier C, Anzueto A, et al. Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease: GOLD executive summary. Am J Respir Crit Care Med. 2013;187(4):347–65.

Morgan AD, Zakeri R, Quint JK. Defining the relationship between COPD and CVD: what are the implications for clinical practice? Ther Adv Respir Dis. 2018;12:1753465817750524.

Article   PubMed   PubMed Central   Google Scholar  

Rabe KF, Hurst JR, Suissa S. Cardiovascular disease and COPD: dangerous liaisons? Eur Respir Rev 2018;27(149).

Salpeter S, Ormiston T, Salpeter E. Cardioselective beta-blockers for chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2016;4:CD003566.

Google Scholar  

Du Q, Sun Y, Ding N, Lu L, Chen Y. Beta-blockers reduced the risk of mortality and exacerbation in patients with COPD: a meta-analysis of observational studies. PLoS ONE. 2014;9(11):e113048.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Etminan MJS, Carleton B, FitzGerald JM. Beta-blocker use and COPD mortality: a systematic review and meta-analysis. BMC Pulm Med. 2012;12(1):48.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Dransfield MT, Voelker H, Bhatt SP, Brenner K, Casaburi R, Come CE, et al. Metoprolol for the Prevention of Acute Exacerbations of COPD. N Engl J Med. 2019;381(24):2304–14.

Gulea C, Zakeri R, Quint JK. Effect of beta-blocker therapy on clinical outcomes, safety, health-related quality of life and functional capacity in patients with chronic obstructive pulmonary disease (COPD): a protocol for a systematic literature review and meta-analysis with multiple treatment comparison. BMJ Open. 2018;8(11):e024736.

Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.

Puhan MA, Schunemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630.

Higgins JPTJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions. New Jersey: Wiley; 2019.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.

van Valkenhoef G, Lu G, de Brock B, Hillege H, Ades AE, Welton NJ. Automating network meta-analysis. Res Synth Methods. 2012;3(4):285–99.

Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PloS one. 2014;3;9(7):e99682.

Trinquart L, Attiche N, Bafeta A, Porcher R, Ravaud P. Uncertainty in treatment rankings: reanalysis of network meta-analyses of randomized trials. Ann Intern Med. 2016;164(10):666–73.

Furukawa TA, Salanti G, Atkinson LZ, Leucht S, Ruhe HG, Turner EH, et al. Comparative efficacy and acceptability of first-generation and second-generation antidepressants in the acute treatment of major depression: protocol for a network meta-analysis. BMJ Open. 2016;6(7):e010919.

Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3):1–48.

Article   Google Scholar  

Bhatt SP, Wells JM, Kinney GL, Washko GR Jr, Budoff M, Kim YI, et al. Beta-blockers are associated with a reduction in COPD exacerbations. Thorax. 2016;71(1):8–14.

Coiro S, Girerd N, Rossignol P, Ferreira JP, Maggioni A, Pitt B, et al. Association of beta-blocker treatment with mortality following myocardial infarction in patients with chronic obstructive pulmonary disease and heart failure or left ventricular dysfunction: a propensity matched-cohort analysis from the High-Risk Myocardial Infarction Database Initiative. Eur J Heart Fail. 2017;19(2):271–9.

Ekstrom MP, Hermansson AB, Strom KE. Effects of cardiovascular drugs on mortality in severe chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2013;187(7):715–20.

Gottlieb SSMR, Vogel RA. Effect of beta-blockade on mortality among high-risk and low-risk patients after myocardial infarction. N Engl J Med. 1998;339(8):489–97.

Hawkins NM, Huang Z, Pieper KS, Solomon SD, Kober L, Velazquez EJ, et al. Chronic obstructive pulmonary disease is an independent predictor of death but not atherosclerotic events in patients with myocardial infarction: analysis of the Valsartan in Acute Myocardial Infarction Trial (VALIANT). Eur J Heart Fail. 2009;11(3):292–8.

Kubota Y, Asai K, Furuse E, Nakamura S, Murai K, Tsukada YT, et al. Impact of beta-blocker selectivity on long-term outcomes in congestive heart failure patients with chronic obstructive pulmonary disease. Int J Chron Obstruct Pulmon Dis. 2015;10:515–23.

Mentz RJ, Wojdyla D, Fiuzat M, Chiswell K, Fonarow GC, O’Connor CM. Association of beta-blocker use and selectivity with outcomes in patients with heart failure and chronic obstructive pulmonary disease (from OPTIMIZE-HF). Am J Cardiol. 2013;111(4):582–7.

Quint JK, Herrett E, Bhaskaran K, Timmis A, Hemingway H, Wedzicha JA, et al. Effect of beta blockers on mortality after myocardial infarction in adults with COPD: population based cohort study of UK electronic healthcare records. BMJ. 2013;347:f6650.

Rodriguez-Manero M, Lopez-Pardo E, Cordero A, Ruano-Ravina A, Novo-Platas J, Pereira-Vazquez M, et al. A prospective study of the clinical outcomes and prognosis associated with comorbid COPD in the atrial fibrillation population. Int J Chron Obstruct Pulmon Dis. 2019;14:371–80.

Rutten FH, Hak Eelko ZNPA, Grobbee DE, Hoes AW. Blockers may reduce mortality and risk of exacerbations in patients with chronic obstructive pulmonary disease. Arch Intern Med. 2010;170(10):880–7.

Scrutinio D, Guida P, Passantino A, Ammirati E, Oliva F, Lagioia R, et al. Acutely decompensated heart failure with chronic obstructive pulmonary disease: clinical characteristics and long-term survival. Eur J Intern Med. 2019;60:31–8.

Short PM, Lipworth SI, Elder DH, Schembri S, Lipworth BJ. Effect of beta blockers in treatment of chronic obstructive pulmonary disease: a retrospective cohort study. BMJ. 2011;342:d2549.

Sin DD, McAlister FA. The effects of beta-blockers on morbidity and mortality in a population-based cohort of 11,942 elderly patients with heart failure. Am J Med. 2002;113(8):650–6.

Staszewsky L, Cortesi L, Tettamanti M, Dal Bo GA, Fortino I, Bortolotti A, et al. Outcomes in patients hospitalized for heart failure and chronic obstructive pulmonary disease: differences in clinical profile and treatment between 2002 and 2009. Eur J Heart Fail. 2016;18(7):840–8.

Su TH, Chang SH, Kuo CF, Liu PH, Chan YL. beta-blockers after acute myocardial infarction in patients with chronic obstructive pulmonary disease: a nationwide population-based observational study. PLoS ONE. 2019;14(3):e0213187.

Su VY, Chang YS, Hu YW, Hung MH, Ou SM, Lee FY, et al. Carvedilol, bisoprolol, and metoprolol use in patients with coexistent heart failure and chronic obstructive pulmonary disease. Medicine (Baltimore). 2016;95(5):e2427.

Article   CAS   PubMed Central   Google Scholar  

Su VYYY, Perng DW, Tsai YH, Chou KT, Su KC, Su WJ, Chen PC, Yang KY. Real-world effectiveness of medications on survival in patients with COPD-heart failure overlap. Aging. 2019;11(11):3650.

van Gestel YR, Hoeks SE, Sin DD, Welten GM, Schouten O, Witteveen HJ, et al. Impact of cardioselective beta-blockers on mortality in patients with chronic obstructive pulmonary disease and atherosclerosis. Am J Respir Crit Care Med. 2008;178(7):695–700.

van Gestel YR, Hoeks SE, Sin DD, Stam H, Mertens FW, Bax JJ, van Domburg RT, Poldermans D. Beta-blockers and health-related quality of life in patients with peripheral arterial disease and COPD. Int J Chronic Obstructive Pulm Dis. 2009;4:177.

Wang WH, Cheng CC, Mar GY, Wei KC, Huang WC, Liu CP. Improving outcomes in chronic obstructive pulmonary disease by taking beta-blockers after acute myocardial infarction: a nationwide observational study. Heart Vessels. 2019;34(7):1158–67.

Zeng LH, Hu YX, Liu L, Zhang M, Cui H. Impact of beta2-agonists, beta-blockers, and their combination on cardiac function in elderly male patients with chronic obstructive pulmonary disease. Clin Interv Aging. 2013;8:1157–65.

CAS   PubMed   PubMed Central   Google Scholar  

Maltais F, Buhl R, Koch A, Amatto VC, Reid J, Gronke L, et al. Beta-blockers in COPD: a cohort study from the TONADO Research Program. Chest. 2018;153(6):1315–25.

Rasmussen DB, Bodtger U, Lamberts M, Torp-Pedersen C, Gislason G, Lange P, et al. Beta-blocker use and acute exacerbations of COPD following myocardial infarction: a Danish nationwide cohort study. Thorax. 2020;75(11):928–33.

Brooks TW, Creekmore F, Young DC, Asche CV, Oberg B, Samuelson WM. Rates of hospitalizations and emergency department visits in patients with asthma and chronic obstructive pulmonary disease taking β-blockers. Pharmacotherapy. 2007;27(5):684–90.

Farland MZ, Peters CJ, Williams JD, Bielak KM, Heidel RE, Ray SM. beta-Blocker use and incidence of chronic obstructive pulmonary disease exacerbations. Ann Pharmacother. 2013;47(5):651–6.

Article   PubMed   CAS   Google Scholar  

Sessa M, Mascolo A, Mortensen RN, Andersen MP, Rosano GMC, Capuano A, et al. Relationship between heart failure, concurrent chronic obstructive pulmonary disease and beta-blocker use: a Danish nationwide cohort study. Eur J Heart Fail. 2018;20(3):548–56.

Adam WR, Meagher EJ, Barter CE. Labetalol, beta blockers, and acute deterioration of chronic airway obstruction. Clin Exp Hypertens A. 1982;4(8):1419–28.

CAS   PubMed   Google Scholar  

Chang CL, Mills GD, McLachlan JD, Karalus NC, Hancox RJ. Cardio-selective and non-selective beta-blockers in chronic obstructive pulmonary disease: effects on bronchodilator response and exercise. Intern Med J. 2010;40(3):193–200.

Chester EH, Schwartz HJ, Fleming GM. Adverse effect of propranolol on airway function in nonasthmatic chronic obstructive lung disease. Chest. 1981;79(5):540–4.

Sinclair DJ. Comparison of effects of propranolol and metoprolol on airways obstruction in chronic bronchitis. Br Med J. 1979;1(6157):168.

Dorow PBH, Tönnesmann U. Effects of single oral doses of bisoprolol and atenolol on airway function in nonasthmatic chronic obstructive lung disease and angina pectoris. Eur J Clin Pharmacol. 1986;31(2):143–7.

Hawkins NM, MacDonald MR, Petrie MC, Chalmers GW, Carter R, Dunn FG, et al. Bisoprolol in patients with heart failure and moderate to severe chronic obstructive pulmonary disease: a randomized controlled trial. Eur J Heart Fail. 2009;11(7):684–90.

Jabbal S, Anderson W, Short P, Morrison A, Manoharan A, Lipworth BJ. Cardiopulmonary interactions with beta-blockers and inhaled therapy in COPD. QJM. 2017;110(12):785–92.

Lainscak M, Podbregar M, Kovacic D, Rozman J, von Haehling S. Differences between bisoprolol and carvedilol in patients with chronic heart failure and chronic obstructive pulmonary disease: a randomized trial. Respir Med. 2011;105:S44–9.

Mainguy V, Girard D, Maltais F, Saey D, Milot J, Senechal M, et al. Effect of bisoprolol on respiratory function and exercise capacity in chronic obstructive pulmonary disease. Am J Cardiol. 2012;110(2):258–63.

McGavin CRWI. The effects of oral propranolol and metoprolol on lung function and exercise performance in chronic airways obstruction. Br J Dis Chest. 1978;72:327–32.

Ranchod SR. The effect of beta-blockers on ventilatory function in chronic bronchitis. South African Med J. 1982;61(12):423–4.

CAS   Google Scholar  

van der Woude HJZJ, Postma DS, Winter TH, van Hulst M, Aalbers R. Detrimental effects of β-blockers in COPD: a concern for nonselective β-blockers. Chest. 2005;127(3):818–24.

Butland RJPJ, Geddes DM. Effect of beta-adrenergic blockade on hyperventilation and exercise tolerance in emphysema. J Appl Physiol. 1983;54(5):1368–73.

Jabbal S, Lipworth BJ. Tolerability of bisoprolol on domiciliary spirometry in COPD. Lung. 2018;196(1):11–4.

Ellingsen J, Johansson G, Larsson K, Lisspers K, Malinovschi A, Stallberg B, et al. Impact of comorbidities and commonly used drugs on mortality in COPD—Real-world data from a primary care setting. Int J Chron Obstruct Pulmon Dis. 2020;15:235–45.

Sin DD, Anthonisen NR, Soriano JB, Agusti AG. Mortality in COPD: role of comorbidities. Eur Respir J. 2006;28(6):1245–57.

Dransfield MT, Rowe SM, Johnson JE, Bailey WC, Gerald LB. Use of beta blockers and the risk of death in hospitalised patients with acute exacerbations of COPD. Thorax. 2008;63(4):301–5.

Yang YL, Xiang ZJ, Yang JH, Wang WJ, Xu ZC, Xiang RL. Association of β-blocker use with survival and pulmonary function in patients with chronic obstructive pulmonary and cardiovascular disease: a systematic review and meta-analysis. Eur Heart J. 2020;41(46):4415–22.

Hjalmarson ÅHJ, Malek I, Ryden L, Vedin A, Waldenström A, Wedel H, Elmfeldt D, Holmberg S, Nyberg G, Swedberg K. Effect on mortality of metoprolol in acute myocardial infarction: a double-blind randomised trial. Lancet. 1981;17(318):823–7.

Hjalmarson ÅGS, Fagerberg B, Wedel H, Waagstein F, Kjekshus J, Wikstrand J, El Allaf D, Vítovec J, Aldershvile J, Halinen M. Effects of controlled-release metoprolol on total mortality, hospitalizations, and well-being in patients with heart failure: the Metoprolol CR/XL Randomized Intervention Trial in congestive heart failure (MERIT-HF). JAMA. 2000;283(10):1295–302.

Loth DW, Brusselle GG, Lahousse L, Hofman A, Leufkens HG, Stricker BH. beta-Adrenoceptor blockers and pulmonary function in the general population: the Rotterdam Study. Br J Clin Pharmacol. 2014;77(1):190–200.

Oda N, Miyahara N, Ichikawa H, Tanimoto Y, Kajimoto K, Sakugawa M, et al. Long-term effects of beta-blocker use on lung function in Japanese patients with chronic obstructive pulmonary disease. Int J Chron Obstruct Pulmon Dis. 2017;12:1119–24.

Download references

CG is funded by an NHLI studentship.

Author information

Authors and affiliations.

National Heart and Lung Institute, Imperial College London, Manresa Road, London, UK

Claudia Gulea & Jennifer K. Quint

NIHR Imperial Biomedical Research Centre, London, UK

British Heart Foundation Centre for Research Excellence, King’s College London, London, UK

Rosita Zakeri

Homerton University Hospital NHS Foundation Trust, London, UK

Vanessa Alderman

Epsom and St. Helier University Hospitals NHS Trust, Epsom, UK

Alexander Morgan

Guy’s & St Thomas’ NHS Foundation Trust, London, UK

Royal Brompton & Harefield NHS Foundation Trust, London, UK

Jennifer K. Quint

You can also search for this author in PubMed   Google Scholar

Contributions

CG, RZ and JKQ made substantial contributions to the conception and design of the study. CG, VA, AM and JR screened abstracts and full-texts and extracted the data. CG carried out statistical analyses and wrote the first draft. CG, RZ, JKQ, VA, AM and JR contributed to data interpretation and provided revisions to the final manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Claudia Gulea .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

CG, RZ, VA, AM and JR have no conflict of interest. JKQ’s research group has received funds from AZ, GSK, The Health Foundation, MRC, British Lung Foundation, IQVIA, Chiesi, and Asthma UK outside the submitted work; grants and personal fees from GlaxoSmithKline, Boehringer Ingelheim, AstraZeneca, Bayer, Insmed outside the submitted work.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: figure s1..

Forest plot illustrating results of the meta-analysis evaluating the impact of beta-blocker therapy vs. no beta-blocker therapy on AECOPD in patients with COPD. Figure S2 . Consistency results illustrating no significant difference between direct and indirect evidence across all comparisons that were assessed in the FEV1 network meta-analysis. Figure S3 . Comparison-adjusted funnel plot . Figure S4. Network meta-analysis with meta-regression results (long vs. short follow-up) . Figure S5 . Network meta-analysis results for patients without A) COPD with explicit cardiovascular disease; B) with cardiovascular disease. Figure S6. Rankogram illustrating probabilities of being 1st, 2nd, 3rd…7th with respect to improvement in lung function, for each beta-blocker (and placebo) in patients with COPD without explicit cardiovascular disease. Figure S7 . Rankogram illustrating probabilities of being 1st, 2nd, 3rd…7th with respect to improvement in lung function for each beta-blocker (and placebo) in patients with COPD with cardiovascular disease. Figure S8. Forest plot showing hazard ratios associated with A) Cardioselective beta-blockers and B) Non-cardioselective beta-blockers and mortality in patients with COPD . Figure S9. Risk of bias assessment, RCTs. Table S1 . Screening criteria. Table S2 . Summary of observational studies. Table S3. Patient characteristics—observational studies. Table S4 . AECOPD estimates for beta-blocker versus no beta-blocker use, from individual observational studies. Table S5 . Study characteristics—RCTs. Table S6 . Baseline  characteristics—RCTs. Table S7 . FEV1 measurements—RCTs. Table S8 . Network meta-analysis results—league table. Table S9 . SUCRA ranking probability of being the best treatment. Table S10. Mortality estimates for beta-blocker versus no beta-blocker use, from individual studies. Table S11 . All-cause hospitalization results. Table S12. SGRQ results. Table S13 .12MWT results. Table S14. 6MWT results. Table S15 . SF-36 results. Table S16 . Risk of bias assessment, observational studies. Table S17. GRADE assessment (mortality, quality of life). Table S18 . GRADE assessment (AECOPD). Table S19. GRADE assessment from each pair-wise comparison within the NMA network (FEV1 analysis)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gulea, C., Zakeri, R., Alderman, V. et al. Beta-blocker therapy in patients with COPD: a systematic literature review and meta-analysis with multiple treatment comparison. Respir Res 22 , 64 (2021). https://doi.org/10.1186/s12931-021-01661-8

Download citation

Received : 21 December 2020

Accepted : 10 February 2021

Published : 23 February 2021

DOI : https://doi.org/10.1186/s12931-021-01661-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Beta-blockers

Respiratory Research

ISSN: 1465-993X

• General enquiries: [email protected]

• Open access
• Published: 15 February 2023

Literature review of stroke assessment for upper-extremity physical function via EEG, EMG, kinematic, and kinetic measurements and their reliability

• Rene M. Maura   ORCID: orcid.org/0000-0001-6023-9038 1 ,
• Sebastian Rueda Parra 4 ,
• Richard E. Stevens 2 ,
• Douglas L. Weeks 3 ,
• Eric T. Wolbrecht 1 &
• Joel C. Perry 1  

Journal of NeuroEngineering and Rehabilitation volume  20 , Article number:  21 ( 2023 ) Cite this article

6011 Accesses

15 Citations

Metrics details

Significant clinician training is required to mitigate the subjective nature and achieve useful reliability between measurement occasions and therapists. Previous research supports that robotic instruments can improve quantitative biomechanical assessments of the upper limb, offering reliable and more sensitive measures. Furthermore, combining kinematic and kinetic measurements with electrophysiological measurements offers new insights to unlock targeted impairment-specific therapy. This review presents common methods for analyzing biomechanical and neuromuscular data by describing their validity and reporting their reliability measures.

This paper reviews literature (2000–2021) on sensor-based measures and metrics for upper-limb biomechanical and electrophysiological (neurological) assessment, which have been shown to correlate with clinical test outcomes for motor assessment. The search terms targeted robotic and passive devices developed for movement therapy. Journal and conference papers on stroke assessment metrics were selected using PRISMA guidelines. Intra-class correlation values of some of the metrics are recorded, along with model, type of agreement, and confidence intervals, when reported.

A total of 60 articles are identified. The sensor-based metrics assess various aspects of movement performance, such as smoothness, spasticity, efficiency, planning, efficacy, accuracy, coordination, range of motion, and strength. Additional metrics assess abnormal activation patterns of cortical activity and interconnections between brain regions and muscle groups; aiming to characterize differences between the population who had a stroke and the healthy population.

Range of motion, mean speed, mean distance, normal path length, spectral arc length, number of peaks, and task time metrics have all demonstrated good to excellent reliability, as well as provide a finer resolution compared to discrete clinical assessment tests. EEG power features for multiple frequency bands of interest, specifically the bands relating to slow and fast frequencies comparing affected and non-affected hemispheres, demonstrate good to excellent reliability for populations at various stages of stroke recovery. Further investigation is needed to evaluate the metrics missing reliability information. In the few studies combining biomechanical measures with neuroelectric signals, the multi-domain approaches demonstrated agreement with clinical assessments and provide further information during the relearning phase. Combining the reliable sensor-based metrics in the clinical assessment process will provide a more objective approach, relying less on therapist expertise. This paper suggests future work on analyzing the reliability of metrics to prevent biasedness and selecting the appropriate analysis.

Stroke is one of the leading causes of death and disability in developed countries. In the United States, a stroke occurs every 40 s, ranking stroke as the fifth leading cause of death and the first leading cause of disability in the country [ 1 ]. The high prevalence of stroke, coupled with increasing stroke survival rates, puts a growing strain on already limited healthcare resources; the cost of therapy is elevated [ 2 ] and restricted mostly to a clinical setting [ 3 ], leading to 50% of survivors that reach the chronic stage experiencing severe motor disability for upper extremities [ 4 ]. This highlights the need for refined (improved) assessment which can help pair person-specific impairment with appropriately targeted therapeutic strategies.

Rehabilitation typically starts with a battery of standardized tests to assess impairment and function. This initial evaluation serves as a baseline of movement capabilities and usually includes assessment of function during activities of daily living (ADL). Because these clinical assessments rely on trained therapists as raters, the scoring scale is designed to be discrete and, in some cases, bounded. While this improves the reliability of the metric [ 5 ] (i.e., raters more likely to agree), it also reduces the sensitivity of the scale. Furthermore, those assessment scales that are bounded, such as the Fugl-Meyer Assessment (FMA) [ 6 ], Ashworth or Modified Ashworth (MA) Scale [ 7 ], and Barthel Index [ 8 ], suffer from floor/ceiling effects where the limits of the scales become insensitive to the extremes of impairment and function. It is therefore important to develop new clinical assessment methods that are objective, quantifiable, reliable, and sensitive to change over the full range of function and impairment.

Over the last several decades, robotic devices have been designed and studied for administering post-stroke movement therapy. These devices have begun being adopted into clinical rehabilitation practice. More recently, researchers have proposed and studied the use of robotic devices to assess stroke-related impairments as an approach to overcome the limitations of existing clinical measures previously discussed [ 9 , 10 , 11 , 12 ]. Robots may be equipped with sensitive measurement devices that can be used to rate the person’s performance in a predefined task. These devices can include measuring kinematic (position/velocity), kinetic (force/torque), and/or neuromuscular (electromyography/electroencephalography) output from the subject during the task. Common sensor-based robotic metrics for post-stroke assessment included speed of response, planning time, movement planning, smoothness, efficiency, range, and efficacy [ 13 , 14 ]. Figure  1 demonstrates an example method for comprehensive assessment of a person who has suffered a stroke with data acquired during robotically administered tests. Furthermore, there is potential for new and more comprehensive knowledge to be gained from a wider array of assessment methods and metrics that combine the benefits of biomechanical (e.g., kinematic and kinetic) and neurological (e.g., electromyographic and electroencephalographic) measures [ 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 ].

Example of instrument for upper extremities bilateral biomechanical and neuromuscular assessment. From this data, a wide variety of measures and metrics for assessment of upper-extremity impairment and function may be reported

• Biomechanical assessment

Many classical methods of assessing impairment or function involve manual and/or instrumented quantification of performance through measures of motion (i.e., kinematic) and force (i.e., kinetic) capabilities. These classical methods rely on the training of the therapist to evaluate the capabilities of the person through keen observation (e.g., FMA [ 6 ] and MA [ 7 ]). The quality of kinematic and kinetic measures can be improved with the use of electronic-based measurements [ 23 ]. Robotic devices equipped with electronic sensors have the potential to improve the objectivity, sensitivity, and reliability of the assessment process by providing a means for more quantitative, precise, and accurate information [ 9 , 10 , 11 , 12 , 24 , 25 , 26 , 27 , 28 ]. Usually, the electronic sensors on a rehabilitation robotic device are used for control purposes [ 29 , 30 , 31 ]. Robotics can also measure movement outputs, such as force or joint velocities, which the clinician may not be able to otherwise measure as accurately (or simultaneously) using existing clinical assessment methods [ 23 ]. With accurate and repeatable measurement of forces and joint velocities, sensor-based assessments have the potential to assess the person’s movement in an objective and quantifiable way. This article reviews validity and reliability of biomechanical metrics in relationship to assessment of motor function for upper extremities.

Electrophysiological features for assessment

Neural signals that originate from the body can be measured using non-invasive methods. Among others, electroencephalograms (EEG) measure cortical electrical activity, and electromyograms (EMG) measure muscle electrical activity. The relative low cost, as well as the noninvasive nature of these technologies make them suitable for studying changes in cortical or muscle activation caused by conditions or injuries of the brain, such as the ones elicited by stroke lesions [ 32 ].

Initially, EMG/EEG were used strictly as clinical diagnostic tools [ 33 , 34 ]. Recent improvements in signal acquisition hardware and computational processing methods have increased their use as viable instruments for understanding and treating neuromuscular diseases and neural conditions [ 32 ]. Features extracted from these signals are being researched to assess their relationship to motor and cognitive deficits [ 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ] and delayed ischemia [ 34 , 43 ], as well as to identify different uses of the signals that could aid rehabilitation [ 44 ]. Applications of these features in the context of stroke include: (1) commanding robotic prostheses [ 45 , 46 ], exoskeletons [ 21 , 47 , 48 ], and brain-machine interfaces [ 44 , 49 , 50 , 51 ]; and (2) bedside monitoring for sub-acute patients and thrombolytic therapy [ 52 , 53 , 54 ]. Here we review the validity and reliability of metrics derived from electrophysiological signals in relationship to stroke motor assessment for upper extremity.

Reliability of metrics

Robotic or sensor-based assessment tools have not gained widespread clinical acceptance for stroke assessment. Numerous barriers to their clinical adoption remain, including demonstrating their reliability and providing sufficient validation of robotic metrics with respect to currently accepted assessment techniques [ 55 ]. In the assessment of motor function with sensor-based systems, several literature reviews reveal a wide spectrum of sensor-based metrics to use for stroke rehabilitation and demonstrate their validity [ 13 , 42 , 56 , 57 , 58 , 59 , 63 , 64 ]. However, in addition to demonstrating validity, new clinical assessments must also demonstrate good or excellent reliability in order to support their adoption in the clinical field. This is achieved by: (1) comparing multiple measurements on the same subject (test–retest reliability), and (2) checking agreement between multiple raters of the same subject (inter-rater reliability). Reliability quantifies an assessment’s ability to deliver scores that are free from measurement error [ 65 ]. Previous literature reviews have presented limited, if any, information on the reliability of the biomechanical robotic metrics. Murphy and Häger [ 66 ], Wang et al. [ 56 ], and Shishov et al. [ 67 ] reviewed reliability, but omitted some important aspects of intra-class correlation methods used in the study (e.g., the model type and/or the confidence interval), which are required when analyzing intra-class correlation methods for reliability [ 68 ]. If the reliability is not properly analyzed and reported, the study runs the risk of having a biased result. Murphy and Häger [ 66 ] also found a lack of studies determining the reliability of metrics in 2015. Since electronic-based assessments require the use of a therapist or an operator to administer the test, an inter-observer reliability test should be investigated to observe the effect of the test administrators on the assessment process. Therefore, both test–retest and inter-observer reliability in biomechanical and electrophysiological metrics are reviewed to provide updated information on the current findings of the metrics’ reliability.

Integrated metrics

Over the past 50 years, numerous examples of integrated metrics have provided valuable insight into the inner workings of human arm function. In the 1970s EMG was combined with kinematic data in patients with spasticity to understand muscle patterns during ballistic arm reach movements [ 69 ], the affects of pharmacological intervention on spastic stretch reflexes during passive vs. voluntary movement [ 70 ], and in the 1990s EMG was combined with kinetic data to understand the effects of abnormal synergy patterns on reach workspace when lifting the arm against gravity [ 71 ]. This work dispelled long-standing theories of muscular weakness and spasticity alone being the major contributors to arm impairment. More recently, quantified aspects of processed EEG and EMG signals are being combined with kinematic data to investigate the compensatory role, and relation to shoulder-related abnormal muscle synergies of the contralesional secondary sensorimotor cortex, in a group of chronic stroke survivors [ 72 ]. These and other works demonstrate convincingly the value of combined metrics and the insights they can uncover that isolated metrics cannot discover alone.

To provide further information on the stroke severity and the relearning process during stroke therapy, researchers are investigating a multi-modal approach using biomechanical and neuromuscular features [ 15 , 16 , 18 , 19 , 21 , 22 ]. Combining both neuromuscular and biomechanical metrics will provide a comprehensive assessment of the person’s movement starting from motor planning to the end of motor execution. Neuromuscular output provides valuable information on the feedforward control and the movement planning phase [ 22 ]. However, neuromuscular signals provides little information on the movement quality that is often investigated with movement function tests or biomechanical output [ 21 ]. Also, using neuromuscular data will provide information to therapist on the neurological status and nervous system reorganization of the person that biomechanical information cannot provide [ 73 ]. The additional information can assist in developing more personalized care for the person with stroke, as well as offer considerable information on the changes that occur at the physiological level.

Paper overview

This paper reviews published sensor-based methods, for biomechanical and neuromuscular assessment of impairment and function after neurological damage, and how the metrics resulting from the assessments, both alone and in combination, may be able to provide further information on the recovery process. Specifically, methods and metrics utilizing digitized kinematic, kinetic, EEG, and EMG data were considered. The “Methods” section explains how the literature review was performed. In “Measures and methods based on biomechanical performance” section, prevailing robotic assessment metrics are identified and categorized including smoothness, resistance, efficiency, accuracy, efficacy, planning, range-of-motion, strength, inter-joint coordination, and intra-joint coordination. In “Measures and methods based on neural activity using EEG/EMG” section, EEG- and EMG-derived measures are discussed by the primary category of analysis performed to obtain them, including frequency power and coherence analyses. The relationship of each method and metric to stroke impairment and/or function is also discussed. Section “Reliability of measures” discusses the reliability of sensor-based metrics and some of the complications in demonstrating the effectiveness of the metrics. Section “Integrated metrics” reviews previous studies on combining biomechanical and neuromuscular data to provide further information on the changes occurring during assessment and training. Finally, Section “Discussions and conclusions” concludes the paper with a discussion on the advantages of combining multi-domain data, which of the metrics from the earlier sections should be considered in future robotic applications, as well as the ones that still require more investigation for either validity and/or reliability.

A literature review was performed following PRISMA guidelines [ 74 ] on biomechanical and neuromuscular assessment in upper-limb stroke rehabilitation. The review was composed of two independent searches on (1) biomechanical robotic devices, and (2) electrophysiological digital signal processing. Figures  2 and 3 show the selection process of the electrophysiological and biomechanical papers, respectively. Each of these searches applied the following steps: In step 1, each researcher searched in Google Scholar for papers between 2000 and 2021 (see Table 1 for search terms and prompts). In step 2, resulting titles and abstracts were screened to remove duplicates, articles in other languages, and articles not related to the literature review. In step 3, researchers read the full texts of articles screened in step 2, papers qualifying for inclusion using the Literature Review Criteria in Table 1 were selected. Finally, in step 4, selected articles from independent review process were read by the other researcher. Uncertainties in determining if a paper should be included/excluded were discussed with the whole research group. Twenty-four papers focus on biomechanical measures (kinematic and kinetic), thirty-three focus on electrophysiological measures (EEG/EMG), and six papers on multimodal approaches combining biomechanical and neuromuscular measures to assess stroke. Three of the six multimodal papers are also reported in the biomechanical section and 3 papers were hand-picked. A total of 60 papers are reviewed and reported.

PRISMA flowchart on the selection for electrophysiological papers

PRISMA flow chart for the selection for biomechanical papers

Measures and methods based on biomechanical performance

This review presents common robotic metrics which have been previously used to assess impairment and function after stroke. Twenty-five biomechanical papers are reviewed, which used both sensor-based and traditional clinical metrics to assess upper-extremity impairment and function. The five common metrics included in the reviewed studies measured the number of velocity peaks (~ 9 studies), path-length ratio (~ 8 studies), the max speed of the arm (~ 7 studies), active range of motion (~ 7 studies), and movement time (~ 7 studies). The metrics are often compared to an established clinical assessment to determine validity of the metric. The sensor-based metrics can be categorized by the aspect in which they evaluate movement quality similar to De Los Reyes-Guzmán et al.: smoothness, efficiency, efficacy, accuracy, coordination, or range of motion [ 14 ]. Resistance, Movement Planning, Coordination, and Strength are included as additional categories since some of the reviewed sensor-based metrics best evaluate those movement aspects. Examples of common evaluation activities and specific metrics that have been computed to quantify movement quality are outlined in Table 2 .

Lack of arm movement smoothness is a key indicator of underlying impairment [ 79 ]. Traditional therapist-administered assessments do not computationally measure smoothness leaving therapists unable to determine the degree to which disruption to movement smoothness is compromising motor function and, therefore, ADL. Most metrics that have been developed to quantify smoothness are based on features of the velocity profile of an arm movement, such as speed [ 80 , 81 ], speed arc length [ 79 ], local minima of velocity [ 10 ], velocity peaks [ 75 , 76 , 81 ], tent [ 80 ], spectral [ 25 ], spectral arc length [ 25 , 81 ], modified spectral arc length [ 79 ], and mean arrest period ratio [ 76 ]. Table 3 summarizes the smoothness metrics and their corresponding equations with equation numbers for reference. The speed metric is expressed as a ratio between the mean speed and the peak speed (Eq. 1). The speed arc length is the temporal length of the velocity profile (Eq. 2). Local minima of velocity and the velocity peaks metrics are measured by counting the number of minimum (Eq. 3) or maximum (Eq. 4) peaks in the velocity profile, respectively. The tent metric is a graphical approach that divides the area under the velocity curve by the area of a single peak velocity curve (Eq. 5). The spectral metric is the summation of the maximal Fourier transformed velocity vector (Eq. 6). The spectral arc-length metric is calculated from the frequency spectrum of the velocity profile by performing a fast Fourier transform operation and then computing the length (Eq. 7). The modified spectral arc length adapts the cutoff frequency according to a given threshold velocity and an upper-bound cutoff frequency (Eq. 8). The modified spectral arc length is then independent of temporal movement scaling. The mean arrest period ratio is the time portion that movement speed exceeds a given percentage of peak speed (Eq. 9).

Another commonly used approach is to analyze the jerk (i.e., the derivative of acceleration) profile. The common ways to assess smoothness using the jerk profile are root mean square jerk, mean rectified jerk, normalized jerk, and the logarithm of dimensionless jerk. The root mean square jerk takes the root-mean-square of the jerk that is then normalized by the movement duration [ 82 ] (Eq. 10). The mean rectified jerk (normalized mean absolute jerk) is the mean of the magnitude jerk normalized or divided by the peak velocity [ 80 , 82 ] (Eq. 11). The normalized jerk (dimensionless-squared jerk) is the square of the jerk times the duration of the movement to the fifth power over the length squared (Eq. 12). It is then integrated over the duration and square rooted. The normalized jerk can be normalized by mean speed, max speed, or mean jerk [ 80 ]. The logarithm of dimensionless jerk (Eq. 13) is the logarithm of normalized jerk defined in Eq. 12 [ 81 ].

It has yet to be determined which smoothness metric is more effective for characterizing recovery of smooth movement. According to Rohrer et al. [ 80 ], the metrics of speed, local minima of velocity, peaks, tent, and mean arrest period ratio showed increases in smoothness for inpatient recovery from stroke, but the mean rectified jerk metric seemed to show a decrease in smoothness as survivors of stroke recovered. Rohrer et al. warned that a low smoothness factor in jerk does not always mean the person is highly impaired. The spectral arc-length metric showed a consistent increase in smoothness as the number of sub-movements decreased [ 25 ], whereas the other metrics showed sudden changes in smoothness. For example, the mean arrest period ratio and the speed metric showed an increase in smoothness with two or more sub-movements, but when two sub-movements started to merge, the smoothness decreased. As a result, the spectral arc-length metric appears to capture change over a wider range of movement conditions in recovery in comparison to other metrics.

The presence of a velocity-dependent hyperactive stretch reflex is referred to as spasticity [ 83 ]. Spasticity results in a lack of smoothness during both passive and active movements and is more pronounced with activities that involve simultaneous shoulder abduction loading and extension of the elbow, wrist, or fingers [ 83 ], which are unfortunately quite common in ADL. A standard approach to assessing spasticity by a therapist involves moving a subject’s passive arm at different velocities and checking for the level of resistance. While this manual approach is subjective, electronic sensors have the potential to assess severity of spasticity in much more objective ways. Centen et al. report a method to assess the spasticity of the elbow using an upper-limb exoskeleton [ 84 ] involving the measurement of peak velocity, final angle, and creep. Sin et al., similarly performed a comparison study between a therapist moving the arm versus a robot moving the arm. An EMG sensor was used to detect the catch and compared with a torque sensor to detect catch angle for the robotic motion [ 85 ]. The robot moving the arm seemed to perform better with the inclusion of either an EMG or a torque sensor than with the therapist moving the arm and the robot simply recording the movement. A related measure that may be correlated with spasticity is the assessment of joint resistance torques during passive movement [ 76 ]. This can provide an assessment of the velocity-dependent resistance to movement that arises following stroke.

Efficiency measures movement fluency in terms of both task completion times and spatial trajectories. In point-to-point reaching, people who have suffered a stroke commonly display inefficient paths in comparison to their healthy side or compared to subjects who are unimpaired [ 10 ]. During the early phases of recovery after stroke, subjects may show slow overall movement speed resulting in longer task times. As recovery progresses, overall speed tends to increase and task times decrease, indicating more effective and efficient motor planning and path execution. Therapists usually observe the person’s efficiency in completing a task and then rate the person’s ability in completing a task in a timely manner. Therefore, both task time (or movement time) [ 10 , 76 , 77 , 86 , 87 ] and mean speed [ 25 , 75 , 77 , 81 , 86 ] are effective ways to assess temporal efficiency. Similar measures used by Wagner et al. include peak-hand velocity and time to peak-hand velocity [ 87 ]. To measure spatial efficiency of movement, both Colombo et al. [ 75 ], Mostafavi [ 77 ], and Germanotta [ 86 ] calculated the movement path length and divided it by the straight-line distance between the start and end points. This is known as the path-length ratio.

Movement planning

Movement planning is associated with feedforward sensorimotor control, elements that occur before the initial phase of movement. A common approach is to use reaction time to assess the duration of the planning phase. In a typical clinical assessment, a therapist can only observe/quantify whether movement can be initiated or not, but has no way to quantify the lag between the signal to initiate movement and initiation of movement. Keller et al., Frisoli et al., and Mostafavi et al. quantified the reaction time to assess movement planning [ 10 , 76 , 77 ] in subjects who have suffered a stroke. Mostafavi assessed movement planning in three additional ways by assessing characteristics of the actual movement: change in direction, movement distance ratio, and maximum speed ratio [ 77 ]. The change in direction is the angular deviation between the initial movement vector and the straight line between the start and end points. The first-movement-distance ratio is the ratio between the distance the hand traveled during the initial movement and the total distance between start and end points. The first-movement-maximum speed ratio is the ratio of the maximum hand speed during the initial phase of the movement divided by the global hand speed for the entire movement task.

Movement efficacy 

Movement efficacy measures the person’s ability to achieve the desired task without assistance. While therapists can assess the number of completed repetitions, they have no means to kinetically quantify amount of assistance required to perform a given task. Movement efficacy is quantified by robot sensor systems that can measure: (a) person-generated movement, and/or (b) the amount of work performed by the robot to complete the movement (e.g., when voluntary person-generated movement fails to achieve a target). Hence, movement efficacy can involve both kinematic and kinetic measures. A kinematic metric that can be used to represent movement efficacy is the active movement index, which is calculated by dividing the portion of the distance the person is to complete by the total target distance for the task [ 75 ]. An example metric based on kinetic data is the amount of assistance metric, proposed by Balasubramanian et al. [ 25 ]. It is calculated by estimating the work performed by the robot to assist voluntary movement, and then dividing it by the work performed by the robot as if the person performs the task without assistance from the robot. A similar metric obtained by Germanotta et al. calculates the total work by using the movement’s path length, but Germanotta et al. also calculate the work generated towards the target [ 86 ].

Movement accuracy

Movement accuracy has been characterized by the error in the end-effector trajectory compared to a theoretical trajectory. It measures the person’s ability to follow a prescribed path, whereas movement efficiency assesses the person’s ability to find the most ideal path to reach a target. Colombo et al. measured movement accuracy in people after stroke by calculating the mean-absolute value of the distance, which is the mean absolute value of the distance between each point on the person’s path and the theoretical path [ 75 ]. Figure  4 demonstrates the difference between path-length ratio and mean-absolute value of the distance. The mean-absolute value of the distance computes the error between a desired trajectory and the actual, and the path-length ratio computes the total path length the person’s limb has traveled. Another similar metric is the average inter-quartile range, which quantifies the average “spread” among several trajectories [ 15 ]. Balasubramanian et al. characterized movement accuracy as a measure of the subject’s ability to achieve a target during active reaching. They refer to the metric as movement synergy [ 25 ], and calculate it by finding the distance between the end-effector’s final location and the target location.

Difference between path-length ratio and mean absolute value of the distance. A Path-length ratio. \(d_{ref}\) is the theoretical distance the hand should travel between the start and end point. \(d_{total}\) is the total distance the hand travelled from Start to End. B Mean absolute value of the distance. \(d_{i}\) is the distance between the theoretical path and the actual hand path

Intra-limb coordination

Intra-limb (inter-joint) coordination is a measure of the level of coordination achieved by individual joints of a limb or between multiple joints of the same limb (i.e., joint synergy) when performing a task. Since the upper limb consists of kinematic redundancies, the human arm can achieve a desired outcome in multiple ways. For example, a person might choose to move an atypical joint in order to compensate for a loss of mobility in another joint. Frisoli et al. and Bosecker et al. used the shoulder and elbow angle to find a linear correlation between the two angles in a movement task that required multi-joint movement [ 10 , 78 ]. In terms of clinical assessment, joint angle correlations can illustrate typical or atypical contribution of a joint while performing a multi-joint task.

Inter-limb coordination

Inter-limb coordination refers to a person’s ability to appropriately perform bilateral movements with affected and unaffected arms. Therapists observe the affected limb by often comparing to the unaffected limb during a matching task, such as position matching. Matching can either be accomplished with both limbs moving simultaneously or sequentially, and typically without the use of vision. Dukelow et al. used position matching to obtain measures of inter-limb coordination [ 24 ], including trial-to-trial variability, spatial contraction/expansion, and systematic shifts. Trial-to-trial variability is the standard deviation of the matching hand’s position for each location in the x (distal/proximal), y (anterior/posterior), and both in x and y in the transverse plane. Spatial contraction/expansion is the ratio of the 2D work area of the target hand to the 2D work area of the matching hand during a matching task. Systematic shifts were found by calculating the mean absolute position error between the target and matching hand for each target location.

Semrau et al. analyzed the performance of subjects in their ability to match their unaffected arm with the location of their affected arm [ 88 ]. In the experiment, a robot moved the affected arm to a position and the person then mirrored the position with the unaffected side. The researchers compared the data when the person was able to see the driven limb versus when they were unable to see the driven limb. The initial direction error, path length ratio, response latency, peak speed ratio, and their variabilities were calculated to assess the performance of the person’s ability to perform the task.

Range of motion

Range of motion is a measure of the extent of mobility in one or multiple joints. Traditionally, range of motion can be measured with the use of a goniometer [ 89 ]. The goniometer measures the individual joint range of motion, which takes considerable time. Range of motion can be expressed as a 1-DOF angular measure [ 76 , 89 ], a 2-DOF planar measure (i.e., work area) [ 82 ], or a 3-DOF spatial measure (i.e., workspace) [ 77 ]. Individual joints are commonly measured in joint space, whereas measures of area or volume are typically given in Cartesian space. In performing an assessment of work area or workspace with a robotic device, the measure can be estimated either by: (a) measuring individual joint angles with an exoskeleton device and then using these angles to compute the region swept out by the hand, or (b) directly measuring the hand or fingertips with a Cartesian (end-effector) device. The measurement of individual joint range of motion (ROM) as well as overall workspace have significant clinical importance in assessing both passive (pROM) and active (aROM) range of motion. To measure pROM, the robot drives arm movement while the person remains passive. The pROM is the maximum range of motion the person has with minimal or no pain. For aROM, a robot may place the arm in an initial position/orientation from which the person performs unassisted joint movements to determine the ROM of particular joints [ 76 ], or the area or volume swept by multiple joints. Lin et al. quantified the work area of the elbow and shoulder using potentiometers and derived test–retest reliability [ 89 ]. The potentiometer measurements were then compared to therapist measurements to determine validity.

Measures of strength evaluate a person’s ability to generate a force in a direction or a torque about a joint. Strength measurements may involve single or multiple joints. At the individual joint level, strength is typically measured from a predefined position of a person’s arm and/or hand. The person then applies a contraction to produce a torque at the assessed joint [ 76 , 78 ]. Multi-joint strength may also be measured by assessing strength and/or torque in various directions at distal locations along the arm, such as the hand. Lin et al. compared the grip strength obtained from load cells to a clinical method using precise weights, which showed excellent concurrent validity [ 89 ].

Measures and methods based on neural activity using EEG/EMG

Although much information can be captured and analyzed using the kinematic and kinetic measures listed above, their purview is limited. These measures provide insight into the functional outcomes of neurological system performance but provide limited perspective on potential contributing sources of measured impairment [ 90 ]. For a deeper look into the neuromuscular system, measures based on neurological activation are often pursued. As a complement to biomechanical measures, methods based on quantization of neural activity like EEG and EMG have been used to characterize the impact of stroke and its underlying mechanisms of impairments [ 91 , 92 ]. Over the past 20 years, numerous academic research studies have used these measures to explore the effects of stroke, therapeutic interventions, or time on the evolution of abnormal neural activity [ 91 ]. Groups with different levels of neurological health are commonly compared (e.g., chronic/acute/subacute stroke vs. non-impaired, or impairment level) or other specific experimental characteristics (e.g., different rehabilitation paradigms [ 93 , 94 ]). With this evidence, the validity of these metrics has been tested; however, the study of reliability of these metrics is needed to complete the jump from academic to clinical settings.

Extracting biomarkers from non-invasive neural activity requires careful decomposition and processing of raw EEG and EMG recordings [ 32 ]. Various methods have been used, and the results have produced a growing body of evidence for the validity of these biomarkers in providing insight on the current and future state of motor, cognitive, and language skills in people after stroke [ 38 , 95 ]. Some of the biomarkers derived from EEG signals include: power-related band-specific information [ 34 , 35 , 43 , 47 , 53 , 54 , 96 , 97 , 98 , 99 , 100 , 101 ], band frequency event-related synchronization and desynchronization (ERS/ERD) [ 22 , 51 , 102 , 103 ], intra-cortical coherence or functional connectivity [ 39 , 59 , 73 , 94 , 104 , 105 , 106 , 107 , 108 , 109 ], corticomuscular coherence (CMC) [ 37 , 110 , 111 , 112 , 113 ], among others [ 114 , 115 ]. Biomarkers extracted from EEG can be used to assess residual functional ability [ 38 , 54 , 73 , 97 , 98 , 99 ], derive prognostic indicators [ 34 , 43 , 104 ], or categorize people into groups (e.g., to better match impairments with therapeutic strategies) [ 39 , 47 , 58 , 116 ].

In the following subsections, valid biomarkers derived mostly from EEG signal features (relationship with motor outcome for a person after stroke) will be discussed and introduced theoretically. Distinctions will be made about the stage after stroke when signals were taken. Findings are reported from 33 studies that have examined the relationship between extracted neural features and motor function for different groups of people after stroke. These records are grouped by quantization methods used including approaches based on measures of frequency spectrum power (n = 9), inter-regional coherence (n = 10 for cortical coherence and n = 9 for CMC), and reliability (n = 5).

Frequency spectrum power

Power measures the amount of activity within a signal that occurs at a specific frequency or range of frequencies. Power can be computed in absolute or relative terms (i.e., with respect to other signals). It is often displayed as a power density spectrum where the magnitudes of signal power can be seen across a range of frequencies. In electro-cognitive research, the representation of power within specific frequency bands has been useful to explain brain activity and to characterize abnormal oscillatory activity due to regional neurological damage [ 32 , 117 ].

Frequency bands in EEG content

Electrical activity in the brain is dominated primarily by frequencies from 0–100 Hz where different frequency bands correspond with different states of activity: Delta (0–4 Hz) is associated with deep sleep, Theta (4–8 Hz) with drowsiness, Alpha (8–13 Hz) with relaxed alertness and important motor activity [ 117 ], and Beta (13–31 Hz) with focused alertness. Gamma waves (> 32 Hz) are also seen in EEG activity; however, their specific relationship to level of alertness or consciousness is still debated [ 32 , 117 ]. Important cognitive tasks have been found to trigger activity in these bands in different ways. Levels of both Alpha and Delta activity have also been shown to be affected by stroke and can therefore be examined as indicators of prognosis or impairment in sub-acute and chronic stroke [ 52 , 100 , 118 ].

Power in acute and sub-acute stroke

For individuals in the early post-stroke (i.e., sub-acute) phase, abnormal power levels can be an indicator of neurological damage [ 98 ]. Attenuation of activity in Alpha and Beta bands have been observed in the first hours after stroke [ 100 ] preceding the appearance of abnormally high Delta activity. Tolonen et al. reported a high correlation between Delta power and regional Cerebral Blood Flow (rCBF). This relationship appears during the sub-acute stroke phase and has been used to predict clinical, cognitive, and functional outcomes [ 119 ]. Delta activity has also been shown to positively correlate with 1-month National Institutes of Health Stroke Scale (NIHSS) [ 52 ] and 3-month Rankin scale [ 36 ] assessments.

Based on these findings, several QEEG (Quantitative Electroencephalography) metrics involving ratios of abnormal slow (Delta) and abnormal fast (Alpha and Beta) activity have been developed. The Delta-Alpha Ratio (DAR), Delta-Theta Ratio (DTR), and (Delta + Theta)/(Alpha + Beta) Ratio (DTABR also known as PRI for Power Ratio Index) relate amount of abnormal slow activity with the activity from faster bands and have been shown to provide valuable insight into prognosis of stroke outcome and thrombolytic therapy monitoring [ 98 ]. Increased DAR and DTABR have been repeatedly found to be the QEEG indices that best predict worse outcome for the following: comparing with the Functional Independence Measure and Functional Assessment Measure (FIM-FAM) at 105 days [ 53 ], Montreal Cognitive Assessment (MoCa) at 90 days [ 54 ], NIHSS at 1 month [ 35 ], modified ranking scale (mRS) at 6 months [ 105 ], NIHSS evolution at multiple times [ 120 ], and NIHSS at 12 months [ 96 ]. DAR was also used to classify people in the acute phase and healthy subjects with an accuracy of 100% [ 58 ].

The ability of basic EEG monitoring to derive useful metrics during the early stage of stroke has made EEG collection desirable for people who have suffered a stroke in intensive care settings. The derived QEEG indices have proven to be helpful to determine Delayed Cerebral Ischemia (DCI), increased DAR [ 43 ], and increased Delta power [ 34 , 118 ]. However, finding the electrode montage with the least number of electrodes that still reveals the necessary information for prognoses is one of the biggest challenges for this particular use of EEG. Comparing DAR from 19 electrodes on the scalp with 4 electrodes on the frontal cortex suggests that DAR from 4 frontal electrodes may be enough to detect early cognitive and functional deficits [ 53 ]. Studies explored the possibility of a single-electrode montage over the Fronto-Parietal area (FP1); the DAR and DTR from this electrode might be a valid predictor of cognitive function after stroke when correlated with the MoCA [ 54 ], relative power in Theta band correlated with mRS and modified Barthel Index (mBI) 30 and 90 days after stroke [ 121 ].

Power in chronic stroke

The role of power-related QEEG indices during chronic stroke and progression of motor functional performance have been examined with respect to rehabilitation therapies, since participants have recovered their motion to a certain degree [ 4 ]. Studies have shown that therapy and functional activity improvements correlate with changes of the shape and delay of event-related desynchronization and synchronization (ERD-ERS) for time–frequency power features when analyzing Alpha and Beta bands on the primary motor cortex for ipsilesional and contralesional hemispheres [ 21 , 22 , 122 ]. Therapies with better outcome tend to have reduced Delta rhythms and increased Alpha rhythms [ 122 ].

Bertolucci [ 47 ] compared starting power spectrum density in different bands for both hemispheres with changes in WMFT and FMA over time. Increased global Alpha and Beta activity was shown to correlate with better WMFT evolution while, increase in contralesional Beta activity was shown to be correlated with FMA evolution. Metrics combining slow and fast activity have also been tested in the chronic stage of stroke, significant negative correlation between DTABR (PRI) at the start of therapy was related to FMA change during robotic therapy [ 99 ]. This finding suggests that DTABR may have promise as prognostic indicators for all stages of stroke.

Brain Symmetry Index (BSI) is a generalized measure of “left to right” (affected to non-affected) power symmetry of mean spectral power per hemisphere. These inter-hemispheric relationships of power have been used as prognostic measures during all stages of stroke. Baseline BSI (during the sub-acute stage) was found to correlate with the FMA at 2 months [ 73 ], mRS at 6 months [ 123 ], and FM-UE predictor when using only theta band BSI for patients in the chronic stage [ 124 ]. BSI can be modified to account for the direction of asymmetry, the directed BSI at Delta and Theta bands proved meaningful to describe evolution from acute to chronic stages of upper limb impairment as measured by FM-UE [ 120 , 125 ]. Table 4 and Table 11 in Appendix 1 communicate power-derived metrics across different stages of stroke documented in this section and their main reported relationships with motor function. Findings are often reported in terms of correlation with clinical tests of motor function.

Brain connectivity (cortical coherence)

Brain connectivity is a measure of interaction and synchronization between distributed networks of the brain and allows for a clearer understanding of brain function. Although cortical damage from ischemic stroke is focal, cortical coherence can explain abnormalities in functionality of remote zones that share functional connections to the stroke-affected zone [ 59 ].

Several estimators of connectivity have been proposed in the literature. Coherency, partial coherence (pCoh) [ 125 ], multiple coherence (mCoh), imaginary part of coherence (iCoh) [ 126 ], Phase Lagged Index (PLI), weighted Phase Lagged Index (wPLI) [ 127 ], and simple ratios of power at certain frequency bands [ 73 ] describe synchronic symmetric activity between ROIs and are referred to as non-directed or functional connectivity [ 128 ]. Estimators based on Granger’s prediction such as partial directed coherence (PDC) [ 129 , 130 , 131 ], or directed transfer Function (DTF) [ 132 , 133 ] and any of their normalizations describe causal relationships between variables and are referred to as directed or effective connectivity [ 134 ]. Connectivity also allows the analysis of brain activity as network topologies, borrowing methods from graph theory [ 32 , 134 ]. Network features such as complexity, linearity, efficiency, clustering, path length, node hubs, and more can be derived from graphs [ 128 ]. Comparisons of these network features among groups with impairment and healthy controls have proven to be interesting tools to understand and characterize motor and functional deficits after stroke [ 108 ].

Studies have used intra- and inter-cortical coherence to expand the clinical understanding of the neural reorganization process [ 59 , 106 , 107 , 108 , 109 ], as a clinical motor and cognitive predictor [ 38 , 94 , 104 , 135 , 136 ], and as a tool to predict the efficacy of rehabilitation therapy [ 94 ]. Table 5 and Table 12 in Appendix 2 briefly summarize the main metrics discussed in this section and their results that are related with motor function assessment. In general, studies have shown that motor deficits in stroke survivors are related to less connectivity to main sensory motor areas [ 38 , 94 , 104 , 137 ], weak interhemispheric sensorimotor connectivity [ 109 , 138 ], less efficient networks [ 106 , 135 ], with less “small world” network patterns [ 108 , 134 ] (small-world networks are optimized to integrate specialized processes in the whole network and are known as an important feature of healthy brain networks).

Survivors of stroke tend to exhibit more modular (i.e., more clustered, less integrated) and less efficient networks than non-impaired controls with the biggest difference occurring in the Beta and Gamma bands [ 106 ]. Modular networks are less “small-world” [ 134 ]; small-world networks are optimized to integrate specialized processes in the whole network and are known as an important feature of healthy brain networks. Such a transition to a less small-world network was observed during the acute stage of stroke (first hours after stroke) and documented to be bilaterally decreased in the Delta band and bilaterally increased in the high Alpha band (also known as Alpha2: 10.5–13 Hz) [ 108 ].

Global connectivity with the ipsilesional primary motor cortex (M1) is the most researched biomarker derived from connectivity and has been studied in longitudinal experiments as a plasticity indicator leading to future outcome improvement [ 38 ], motor and therapy gains [ 94 ], upper limb gains during the sub-acute stage [ 137 ], and as a feature that characterizes stroke survivors’ cognitive deficits [ 104 ]. Pietro [ 38 ] used iCoh to test the weighted node degree (WND), a measure that quantifies the importance of a ROI in the brain, for M1 and reported that Beta-band features are linearly related with motor improvement as measured by FM-UE and Nine-Hole-Peg Test. Beta-band connectivity to ipsilesional M1, as measured by spectral coherence, can be used as a therapy outcome predictor, and more than that, results point heavily toward connectivity between M1 and ipsilesional frontal premotor area (PM) to be the most important variable as a therapy gain predictor; predictions can be further improved by using lesion-related information such as CST or MRI to yield more accurate results [ 94 ]. Comparisons between groups of people with impairment and controls showed significant differences on Alpha connectivity involving ipsilesional M1, this value showed a relation with FMA 3 months for the group with impairment due to stroke [ 104 ].

The relationship between interhemispheric ROI connectivity and motor impairment has been studied. The normalized interhemispheric strength (nIHS) from PDC was used to quantify the coupling between structures in the brain, Beta- and lower Gamma-band features of this quantity in sensorimotor areas exhibited linear relationships with the degree of motor impairment measured by CST [ 136 ]. A similar measure, also derived from PDC used to measure ROI interhemispheric importance named EEG-PDC was used in [ 109 ]; here the results show that Mu-band (10–12 Hz) and Beta-band features could be used to explain results for hand motor function from FM-UE. In another study, Beta debiased weighted phase lag index (dwPLI), correlated with outcome measured by Action Research Arm Test (ARAT) and FM-UE [ 138 ].

Global and local network efficiency for Beta and Gamma bands seem to be significantly decreased in the population who suffered from a stroke compared to healthy controls as reported in [ 106 ]. Newer results, such as the ones pointed out by [ 135 ] found statistically significant relationships between Beta network efficiency, network intradensity derived using a non-parametric method (named Generalized Measure of Association), and functional recovery results given by FM-UE. Global maximal coherence features in the Alpha band have been recently recognized as FM-UE predictors, where coherence was computed using PLI and related to motor outcome by means of linear regression [ 139 ].

Corticomuscular coherence

Corticomuscular coherence (CMC) is a measure of the amount of synchronous activity between signals in the brain (i.e., EEG or MEG) and associated musculature (i.e., EMG) of the body [ 92 ]. Typically measured during voluntary contractions [ 110 ], the presence of coherence demonstrates a direct relationship between cortical rhythms in the efferent motor commands and the discharge of neurons in the motor cortex [ 140 ]. CMC is computed as correlation between EEG and EMG signals at a given frequency. Early CMC research found synchronous (correlated) activity in Beta and low Gamma bands [ 40 , 41 , 42 ]. CMC is strongest in the contralateral motor cortex [ 141 ]. This metric seems to be affected by stroke-related lesions, and thus provides an interesting tool to assess motor recovery [ 111 , 142 , 143 , 144 ]. The level of CMC is lower in the chronic stage of stroke than in healthy subjects [ 112 , 145 ], with chronic stroke survivors showing lower peak CMC frequency [ 146 ], and topographical patterns that are more widespread than in healthy people; highlighting a connection to muscle synergies [ 142 , 147 , 148 ]. CMC has been shown to increase with training [ 37 , 112 , 144 ].

Corticomuscular coherence has been proposed as a tool to: (a) identify the functional contribution of reorganized cortical areas to motor recovery [ 37 , 112 , 141 , 144 , 146 ]; (b) understand functional remapping [ 93 , 142 , 145 ]; and (c) study the mechanisms underlying synergies [ 147 , 148 ]. CMC has shown increased abnormal correlation with deltoid EMG during elbow flexion for people who have motor impairment [ 147 ], and the best muscles to target with rehabilitative interventions [ 148 ]. Changes in CMC have been shown to correlate with motor improvement for different stages of stroke, although follow-up scores based on CMC have not shown statistically significant correlations when compared to clinical metrics [ 37 , 93 ]. Results summarizing CMC on stroke can be found in Table 6 and Table 13 in Appendix 3.

Reliability of measures

Each of the aforementioned measures have the potential to be integrated into robotic devices for upper-limb assessment. However, to improve the clinical acceptability of robotic-assisted assessment, the measurements and derived metrics must meet reliability standards in a clinical setting [ 55 ]. Reliability can be defined as the degree of consistency between measurements or the degree to which a measurement is free of error. A common method to represent the relative reliability of a measurement process is the intraclass correlation coefficient (ICC) [ 150 ]. Koo and Li suggest a guideline on reporting ICC values for reliability that includes the ICC value, analysis model (one-way random effects, two-way random effects, two-way fixed effects, or two-way mixed effects), the model type per Shrout and Fleiss (individual trials or mean of k trials), model definition (absolute agreement or consistency), and confidence interval [ 68 ]. Koo and Li also provide a flowchart in selecting the appropriate ICC based on the type of reliability and rater information. An ICC value below 0.5 indicates poor reliability, 0.5 to 0.75 moderate reliability, 0.75 to 0.9 good reliability, and above 0.9 excellent reliability. The reviewed papers will be evaluated based on these guidelines. For reporting the ICC, the Shrout and Fleiss convention is used [ 68 ]. The chosen reliability studies are included in the tables if the chosen ICC model, type, definition, and confidence interval are identifiable, and the metrics have previously been used in electronic-based metrics. For studies that report multiple ICC scores due to assessment of test–retest reliability for multiple raters, the lowest ICC reported is included to avoid bias in the reported results.

In the assessment of reliability of data from robotic sensors, common ways to assess reliability are to correlate multiple measurements in a single session (intra-session) and correlate multiple measurements between different sessions (inter-session) measurements (i.e., test–retest reliability) [ 151 ]. Checking for test–retest reliability determines the repeatability of the robotic metric. The repeatability is the ability to reproduce the same measurements under the same conditions. Table 7 shows the test–retest reliability of several robotic metrics. For metrics checking for test–retest reliability, a two-way mixed-effects model with either single or multiple measurements may be used [ 68 ]. Since the same set of sensors will be used to assess subjects, the two-way mixed model is used. The test–retest reliability should be checking for absolute agreement. Checking for absolute agreement (y = x) rather than consistency (y = x + b) determines the reliability without a bias or systematic error. For example, in Fig.  5 , for a two-way random effect with a single measurement checking for agreement gives a score of 0.18. When checking for consistency, the ICC score reaches to 1.00. In other words, the bias has no effect on the ICC score when checking for consistency. Therefore, when performing test–retest reliability, it is important to check for absolute agreement to prevent bias in the test–retest result.

Checking agreement versus consistency among ratings. For y = x, the absolute ICC score is 1 and the consistency ICC score is 1.00. For y = x + 1, the agreement ICC score is 0.18 and the consistency ICC score is 1.00. For y = 3x, the absolute ICC score is 0.32 and the consistency ICC score is 0.60. For y = 3x + 1, the absolute ICC score is 0.13 and the consistency ICC score is 0.60

Not only should a robotic metric demonstrate repeatability, it should also be reproducible when different operators are using the same device. Reproducibility evaluates the change in measurements when conditions have changed. Inter-rater reliability tests have been performed to determine the effect raters have when collecting measurements when two or more raters perform the same experimental protocol [ 68 ]. To prevent a biased result, raters should have no knowledge of the evaluations given by other raters, ensuring that raters’ measurements are independent from one another. Table 8 shows the reproducibility of several robotic biomechanical metrics. All the included studies have used two raters to check for reproducibility. The researchers performed a two-way random effects analysis with either a single measurement or multiple measurements to check for agreement.

Measurement reliability of robotic biomechanical assessment

Of the 24 papers reviewed for biomechanical metrics, 13 papers reported on reliability. 6 papers reported reproducibility and 9 papers reported on repeatability. Overall, the metrics seem to demonstrate good to moderate reliability for both repeatability and reproducibility. However, caution should be exercised in determining which robotic metric is more effective in assessing movement quality based on reliability studies. The quality of measurements is highly dependent on the quality of the robotic device and sensors [ 85 ]. Having a completely transparent robot with a sensitive and accurate sensor will further improve assessment of reliability. Also, the researchers have used different versions of the ICC, as seen in Tables 7 and 8 , which complicates direct comparisons of the metrics.

Reliability of electrophysiological signal features

Of the 33 papers reviewed for electrophysiological metrics, 5 papers reported on reliability. 6 papers reported on repeatability. Convenience of acquiring electrophysiological signals non-invasively is relatively new. Metrics for assessment of upper limb motor impairment in stroke, derived from these signals have shown to be valid in academic settings, but most of these valid metrics have yet to be tested for intra- and inter-session reliability to be used in clinical and rehabilitation settings. Few studies found as a result of our systematic search have looked at test–retest reliability of these metrics. Therefore, we found and manually added records reporting on intra- and inter-session reliability on metrics based on electrophysiological features described in section “Measures and methods based on neural activity using EEG/EMG”, even if reliability was not assessed on people with stroke. Relevant results are illustrated in Table 9 .

Spectral power features of EEG signals have been tested during rest [ 153 , 154 ] and task (cognitive and motor) conditions for different cohorts of subjects [ 102 , 103 ]. Some of the spectral features observed during these experiments are related to timed behavior of oscillatory activity due to cued experiments, such as event-related desynchronization of the Beta band (ERD and Beta rebound) [ 102 ] and topographical patterns of Alpha activity R = 0.9302, p < 0.001 [ 103 ].

Test–retest reliability for rest EEG functional connectivity has been explored for few of the estimators listed in section “Measures and methods based on neural activity using EEG/EMG”: (1) for a cohort of people with Alzheimer by means of the amplitude envelope correlation (AEC), phase lag index (PLI) and weighted phase lag index (wPLI) [ 155 ]; (2) in healthy subjects using iCoh and PLI [ 156 ]; and (3) in infants, by studying differences of inter-session PLI graph metrics such as path length, cluster coefficient, and network “small-worldness” [ 60 ]. Reliability for upper limb CMC has not yet been documented (at least to our knowledge). However, an experiment involving testing reliability of CMC for gait reports low CMC reliability in groups with different ages [ 61 ].

EEG and EMG measurements could be combined with kinematic and kinetic measurements to provide additional information about the severity of impairment and decrease the number of false positives from individual measurements [ 21 ]. This could further be used to explain abnormal relationships between brain activation, muscle activation and movement kinematics, as well as provide insight about subject motor performance during therapy [ 15 ]. The availability of EEG and EMG measures can also enhance aspects of biofeedback given during tests or be used to complement other assessments to provide a more holistic picture of an individual’s neurological function.

It has been shown that combining EEG, EMG, and kinematic data using a multi-domain approach can produce correlations to traditional clinical assessments, a summary of some of the reviewed studies is presented in Table 10 . Belfatto et al. have assessed people’s ROM for shoulder and elbow flexion, task time, and computed jerk to measure people’s smoothness, while the EMG was used to measure muscle synergies, and EEG detected ERD and a lateralization coefficient [ 21 ]. Comani et al. used task time, path length, normalized jerk, and speed to measure motor performance while observing ERD and ERS during motor training [ 22 ]. Pierella et al. gathered kinematic data from an upper-limb exoskeleton, which assessed the mean tangential velocity, path-length ratio, the number of speed peaks, spectral arc length, the amount of assistance, task time, and percentage of workspace, while observing EEG and EMG activity [ 18 ]. Mazzoleni et al. used the InMotion2 robot system to capture the movement accuracy, movement efficiency, mean speed, and the number of velocity peaks, while measuring brain activity with EEG [ 16 ]. However, further research is necessary to determine the effectiveness of the chosen metrics and methods compared to other more promising methods to assess function. Furthermore, greater consensus in literature is needed to support the clinical use of more reliable metrics. For example, newer algorithms to estimate smoothness such as spectral arc length have been shown to provide greater validity and reliability than the commonly used normalized jerk metric. Despite this evidence, normalized jerk remains a widely accepted measure of movement smoothness.

Discussions and conclusions

In this paper we reviewed studies that used different sensor-acquired biomechanical and electrophysiological signals to derive metrics related to neuromuscular impairment for stroke survivors; such metrics are of interest for robotic therapy and assessment applications. To assess the ability of a given measure to relate with impairment or motor outcome, we looked for metrics where results have been demonstrated to correlate or predict scores from established clinical assessment metrics for impairment and function (validity). Knowing that a metric has some relationship with impairment and function (i.e., that it is valid) is not enough for it to be used in clinical settings if those results are not repeatable (reliable). Thus, we also reviewed the reliability of metrics and related signal features looking for metrics which produce similar results for the same subject during different test sessions and for different raters. With this information, researchers can aim to use metrics that not only seem to be related with stroke, but also can be trusted, with less bias, and with a simpler interpretation. The main conclusions of this review paper are presented as answers to the following research questions.

Which biomechanical-based metrics show promise for valid assessment of function and impairment?

Metrics derived from kinematic (e.g., position & velocity) and kinetic (e.g., force & torque) sensors affixed to robotic and passive mechanical devices have successfully been used to measure biomechanical aspects of upper-extremity function and impairment in people after stroke. The five common metrics included in the reviewed studies measured the number of velocity peaks (~ 9 studies), path-length ratio (~ 8 studies), the maximum speed of the arm (~ 7 studies), active range of motion (~ 7 studies), and movement time (~ 7 studies). The metrics are often compared to an established clinical assessment to determine validity of the metric. According to the review study by Murphy and Häger, the Fugl-Meyer Assessment for Upper Extremity had significant correlation with movement time, movement smoothness, peak velocity, elbow extension, and shoulder flexion [ 66 ]. The movement time and smoothness showed strong correlation with the Action Research Arm Test, whereas speed, path-length ratio, and end-point error showed moderate correlation. Tran et al. reviewed specifically validation of robotic metrics with clinical assessments [ 57 ]. The review found mean speed, number of peak velocities, movement accuracy, and movement duration to be most promising metrics based on validation with clinical assessments. However, the review mentioned that some studies seem to conflict on the correlation between the robotic metric and clinical measures, which could be due to assessment task, subject characteristics, type of intervention, and robotic device. For further information about the validation of sensor-based metrics, please refer to the previously mentioned literature reviews [ 57 , 66 ].

Which biomechanical-based metrics show promise for repeatable assessment?

Repeatable measures, in which measurement taken by a single instrument and/or person produce low variation within a single task, are a critical requirement for assessment of impairment and function. The biomechanical based metrics that show the most promise for repeatability are range of motion, mean speed, mean distance, normal path length, spectral arc length, number of peaks, and task time. Two or more studies used these metrics and demonstrated good and excellent reliability, which implies the metric is robust against measurement noise and/or disturbances. Since the metrics have been used on different measuring instruments, the sensors’ resolution and signal-to-noise ratio appear to have a minimal impact on the reliability. However, more investigation is needed to confirm this robustness. In lieu of more evidence, it is recommended that investigators choose sensors similar or superior in quality to those used in the measuring devices presented in Tables 7 and 8 to achieve the same level of reliability.

What aspects of biomechanical-based metrics lack evidence or require more investigation?

Although many metrics (see previous section) demonstrate good or excellent repeatability across multiple studies, the evidence for reproducibility is limited to single studies. When developing a novel device capable of robotic assistance and assessment, researchers have typically focused their efforts to create a device capable of repeatable and reliable measurements. However, since the person administering the test is using the device to measure the subject’s performance, the reproducibility of the metric must also be considered. The reproducibility of a metric is affected by the ease-of-use of the device; if the device is too complicated to setup and use, there is an increased probability that different operators will observe different measurements. Also, the operator’s instructions to the subject affects the reproducibility, especially in the initial sessions, which may lead to different learning effects, and different assessment results. More studies are needed across multiple sites and operators to determine the reproducibility of the biomechanical metrics reviewed in this paper.

Which neural activity-based metrics (EEG & EMG) show the most promise for reliable assessment?

Electrical neurological signals such as EEG and EMG have successfully been used to understand changes in motor performance and outcome variability across all stages of post-stroke recovery including the first few hours after onset. Experimental results have shown that metrics derived from slow frequency power (delta power, relative delta power, and theta power), and power ratio between slow and fast EEG frequency bands like DAR and DTABR convey useful information both about current and future motor capabilities, as presented in Table 4 and Table 11 in Appendix 1. Multimodal studies using robotic tools for assessment of motor performance have expanded the study of power signal features in people who suffered a stroke in the chronic recovery stage by studying not only rest EEG activity but also task-related activity [ 19 , 21 , 122 ]; ERD-ERS features like amplitude and latency along with biomechanical measures have been shown to correlate with clinical measures of motor performance and to predict a person’s response to movement therapies. EEG power features in general have been found to have good to excellent reliability for test–retest conditions among different populations, across all frequency bands of interest (see Table 9 ).

Functional connectivity (i.e., non-directed connectivity) expands the investigative capacity of EEG measurements, enabling analyzing the brain as a network system by investigating the interactions between regions of interest in the brain while resting or during movement tasks. Inter-hemispheric interactions (interactions between the same ROI in both hemispheres) and global interactions (interactions between the entire brain and an ROI) reported as power or graph indices in Beta and Gamma bands have fruitfully been used to explain motor outcome scores. Although results seem promising, connectivity reliability is still debated with results ranging mostly between moderate to good reliability only for a few connectivity estimators ( PLI, wPLI and iCoh ).

Which neural activity-based metrics (EEG and EMG) lack evidence or require more investigation?

EEG and EMG provide useful non-invasive insight into the human neuromuscular system allowing researchers to make conjectures about its function and structure; however, interpretation of results based on these measures solely must be carefully analyzed within the frame of experimental conditions. Overall, the field needs more studies involving cohorts of stroke survivors to determine the reliability (test–retest) of metrics derived from EEG and EMG signal features that have already shown validity in academic studies.

Metrics calculated from power imbalance between interhemispheric activity like BSI , pwBSI and PRI [ 62 , 73 , 124 ] are a great premise to measure how the brain relies on foreign regions to accomplish tasks related with affected areas. A battery of diverse estimators for connectivity, especially those of effective (directed) connectivity, open the door to investigations into the relationship between abnormal communication of regions of interest and impairment (see Table 5 and Table 12 in Appendix 2). These metrics, although valid have yet to be tested in terms of reliability in clinical use. Reliability for connectivity metrics should specify which estimator was used to derive the metric.

CMC is another exciting neural-activity-based metric lacking sufficient evidence to support its significance. CMC considers and bridges two of the most affected domains for motor execution in neuromuscular system, making it a good candidate for robotic-based therapy and assessment of survivors of stroke [ 147 ]. Although features in the Beta and Gamma bands seem to be related to motor impairment, there is still not agreement about which one is most closely related to motor outcomes. Studies reviewed in this paper considered cortical spatial patterns of maximum coherence, peak frequency shift when compared to healthy controls, latency for peak coherence, among others (see Table 6 and Table 13 in Appendix 3). However, when comparing to motor outcomes, results are not always significant, and test–retest reliability for this metric is yet (to our knowledge) to be documented for the upper extremity (see [ 61 ] for a lower-extremity study).

What standards should be adopted for reporting biomechanical and neural activity-based metrics and their reliability?

For metrics to be accepted as reliable in the clinical field, researchers are asked to follow the guidelines presented in Koo and Li [ 68 ], which provide guidance on which ICC model to use depending on the type of reliability study and what should be reported (e.g., the software they used to compute the ICC and confidence interval). In the papers reviewed, some investigated the learning effects of the assessment task and checked for consistency rather than agreement (see Table 7 ). However, the learning effects should be minimal in a clinical setting between each session, and potential effects should be taken into consideration during protocol design; common practices to minimize the implications of learning effects is to allow practice runs by the patients [ 99 , 122 ] and to remove the first experimental runs [ 81 , 85 ]. By removing this information, signal analysis focuses performance of learned tasks with similar associated behaviors. Therefore, to demonstrate test–retest reliability (i.e., repeatability), the researcher should be checking for absolute agreement. Also, as can be seen in Tables 7 and 8 , there does not seem to be a standard on reporting ICC values. Some researchers report the confidence interval of the ICC value, while others do not. It was also difficult to determine the ICC model used in some of the studies. Therefore, a standard on reporting ICC values is needed to help readers understand the ICC used and prevent bias (see [ 68 ] for suggestive guideline on how to report ICC scores). Also, authors are asked to include the means of each individual session or rater would provide additional information on the variation of the means between the groups. The variation between groups can be shown with Bland–Altman plot, but readers are unable to perform other forms of analysis. To help with this, data from studies should be made publicly available to allow results to be verified and enable further analysis in the future.

When is it advantageous to combine biomechanical and neural activity-based metrics for assessment?

Biomechanical and neural activity provide distinct but complementary information about the neuro-musculoskeletal system, potentially offering a more complete picture of impairment and function after stroke. Metrics derived from kinematic/kinetic information assess motor performance based on motor execution; however, compensatory strategies related to stroke may mask underlying neural deficits (i.e., muscle synergies line up to complete a given task) [ 18 , 21 , 69 , 70 , 71 , 72 , 122 ]. Information relevant to these compensatory strategies can be obtained when analyzing electrophysiological activity, as has been done using connectivity [ 59 , 107 ], CMC [ 147 , 148 ] and brain cortical power [ 91 ].

Combining signals from multiple domains, although beneficial in the sense that it would allow a deeper understanding of a subject’s motor ability, is still a subject of exploration. Experimental paradigms play an important role that influences the decision of feature selection; increasing the dimensionality of signals may provide more useful information for analysis, but comes at the expense of experimental costs (e.g., hardware) and time (e.g., subject setup). With all this in mind, merging information from different domains in the hierarchy of the neuro-musculoskeletal system may provide a more comprehensive quantitative profile of a person’s impairment and performance. Examples of robotic multidomain methods such as the ones in [ 18 , 21 ], highlight the importance of this type of assessment for monitoring and understanding the impact of rehabilitation in chronic stroke survivors. In both cases, these methodologies allowed pairing of observed behavioral changes in task execution (i.e., biomechanical data) with corresponding functional recovery, instead of adopted compensation strategies.

What should be the focus of future investigations of biomechanical and/or neural activity-based metrics?

Determining the reliability and validity of sensor-based metrics requires carefully designed experiments. In future investigations, experiments should be conducted that calculate multiple metrics from multiple sensors and device combinations, allowing the effect of sensor type and quality on the measure’s reliability to be quantified. After the conclusion of such experiments, researchers are strongly encouraged to make their anonymized raw data public to allow other researchers to compute different ICCs. Performing comparison studies on the reliability of metrics will produce reliability data to expand Tables 7 , 8 , 9 and improve our ability to compare similar sensor-based metrics. Additional reliability studies should also be performed that include neural features of survivors of stroke, with increased focus on modeling the interactions between these domains (biomechanical and neural activity). It is also important to understand how to successfully combine data from multimodal experiments; many of the studies reviewed in this paper recorded multidimensional data, but performed analysis for each domain separately.

Availability of data and materials

Not applicable.

Abbreviations

Activities of daily living

Amplitude envelope correlation

Action research arm test

Active range of motion

Autism spectrum disorder

Box and Blocks test

Brain Symmetry Index

Canonical correlation analysis

Cortico-spinal tract

Delta-alpha ratio

Delayed cerebral ischemia

Direct directed transfer function

Degree of freedom

(Delta + Theta)/(Alpha + Beta)

Directed transfer function

Delta-theta ratio

• Electroencephalography

Electromyography

Event related desynchronization

Event related synchronization

Full frequency directed transfer function

Functional independence measure and functional assessment measure

Fugl-Meyer assessment for upper extremity

Generalized Measure of Association

Generalized partial directed coherence

Intra-class correlations

Imaginary part of coherence

Primary motor cortex

Modified Ashworth

Modified Barthel Index

Multiple coherence

Motricity Index

Montreal Cognitive Assessment

Movement related beta desynchronization

Magnetic resonance imaging

Modified Ranking Scale

Normalized interhemispheric strength

National Institutes of Health Stroke Scale

Non-negative matrix factorization algorithm

Principal component analysis

Partial coherence

Partial directed coherence

Phase lag index, weight phase lag index, debiased weighted phase lag index

Premotor area

Post movement beta rebound

Power Ratio Index

Passive range of motion

Quantitative EEG

Regional cerebral blood flow

Region of interest

Renormalized partial directed coherence

Singular value decomposition

Wolf motor function

Weighted Node Degree Index

Stroke Facts. 2020. https://www.cdc.gov/stroke/facts.htm . Accessed 26 Mar 2020.

Ottenbacher KJ, Smith PM, Illig SB, Linn RT, Ostir GV, Granger CV. Trends in length of stay, living setting, functional outcome, and mortality following medical rehabilitation. JAMA. 2004;292(14):1687–95. https://doi.org/10.1001/jama.292.14.1687 .

Article   CAS   PubMed   Google Scholar  

Lang CE, MacDonald JR, Gnip C. Counting repetitions: an observational study of outpatient therapy for people with hemiparesis post-stroke. J Neurol Phys Ther. 2007;31(1). https://journals.lww.com/jnpt/Fulltext/2007/03000/Counting_Repetitions__An_Observational_Study_of.4.aspx .

Gresham GE, Phillips TF, Wolf PA, McNamara PM, Kannel WB, Dawber TR. Epidemiologic profile of long-term stroke disability: the Framingham study. Arch Phys Med Rehabil. 1979;60(11):487–91.

CAS   PubMed   Google Scholar  

Duncan EA, Murray J. The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review. BMC Health Serv Res. 2012;12(1):96.

Article   PubMed   PubMed Central   Google Scholar  

Sullivan KJ, Tilson JK, Cen SY, Rose DK, Hershberg J, Correa A, et al. Fugl-meyer assessment of sensorimotor function after stroke: standardized training procedure for clinical practice and clinical trials. Stroke. 2011;42(2):427–32.

Article   PubMed   Google Scholar  

Ansari NN, Naghdi S, Arab TK, Jalaie S. The interrater and intrarater reliability of the Modified Ashworth Scale in the assessment of muscle spasticity: limb and muscle group effect. NeuroRehabilitation. 2008;23:231–7.

Wade DT, Collin C. The Barthel ADL Index: a standard measure of physical disability? Int Disabil Stud. 1988;10(2):64–7.

Maggioni S, Melendez-Calderon A, van Asseldonk E, Klamroth-Marganska V, Lünenburger L, Riener R, et al. Robot-aided assessment of lower extremity functions: a review. J Neuroeng Rehabil. 2016;13(1):72. https://doi.org/10.1186/s12984-016-0180-3 .

Frisoli A, Procopio C, Chisari C, Creatini I, Bonfiglio L, Bergamasco M, et al. Positive effects of robotic exoskeleton training of upper limb reaching movements after stroke. J Neuroeng Rehabil. 2012;9(1):36. https://doi.org/10.1186/1743-0003-9-36 .

Groothuis-Oudshoorn CGM, Prange GB, Hermens HJ, Ijzerman MJ, Jannink MJA. Systematic review of the effect of robot-aided therapy on recovery of the hemiparetic arm after stroke. J Rehabil Res Dev. 2006;43(2):171.

Harwin WS, Murgia A, Stokes EK. Assessing the effectiveness of robot facilitated neurorehabilitation for relearning motor skills following a stroke. Med Biol Eng Comput. 2011;49(10):1093–102.

Nordin N, Xie SQ, Wünsche B. Assessment of movement quality in robot- assisted upper limb rehabilitation after stroke: a review. J NeuroEngineering Rehabil. 2014;11:137. https://doi.org/10.1186/1743-0003-11-137 .

Article   Google Scholar  

De Los Reyes-Guzman A, Dimbwadyo-Terrer I, Trincado-Alonso F, Monasterio-Huelin F, Torricelli D, Gil-Agudo A. Quantitative assessment based on kinematic measures of functional impairments during upper extremity movements: a review. Clin Biomech. 2014;29(7):719–27. https://doi.org/10.1016/j.clinbiomech.2014.06.013 .

Molteni E, Preatoni E, Cimolin V, Bianchi AM, Galli M, Rodano R. A methodological study for the multifactorial assessment of motor adaptation: integration of kinematic and neural factors. 2010 Annu Int Conf IEEE Eng Med Biol Soc EMBC’10. 2010;4910–3.

Mazzoleni S, Coscia M, Rossi G, Aliboni S, Posteraro F, Carrozza MC. Effects of an upper limb robot-mediated therapy on paretic upper limb in chronic hemiparetic subjects: a biomechanical and EEG-based approach for functional assessment. 2009 IEEE Int Conf Rehabil Robot ICORR 2009. 2009;92–7.

Úbeda A, Azorín JM, Chavarriaga R, Millán JdR. Classification of upper limb center-out reaching tasks by means of EEG-based continuous decoding techniques. J Neuroeng Rehabil. 2017;14(1):1–14.

Pierella C, Pirondini E, Kinany N, Coscia M, Giang C, Miehlbradt J, et al. A multimodal approach to capture post-stroke temporal dynamics of recovery. J Neural Eng. 2020;17(4): 045002.

Steinisch M, Tana MG, Comani S. A post-stroke rehabilitation system integrating robotics, VR and high-resolution EEG imaging. IEEE Trans Neural Syst Rehabil Eng. 2013;21(5):849–59. https://doi.org/10.1596/978-1-4648-1002-2_Module14 .

Úbeda A, Hortal E, Iáñez E, Perez-Vidal C, Azorín JM. Assessing movement factors in upper limb kinematics decoding from EEG signals. PLoS ONE. 2015;10(5):1–12.

Belfatto A, Scano A, Chiavenna A, Mastropietro A, Mrakic-Sposta S, Pittaccio S, et al. A multiparameter approach to evaluate post-stroke patients: an application on robotic rehabilitation. Appl Sci. 2018;8(11):2248.

Comani S, Schinaia L, Tamburro G, Velluto L, Sorbi S, Conforto S, et al. Assessing Neuromotor Recovery in a stroke survivor with high resolution EEG, robotics and virtual reality. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2015. p. 3925–8.

Kwon HM, Yang IH, Lee WS, Yu ARL, Oh SY, Park KK. Reliability of intraoperative knee range of motion measurements by goniometer compared with robot-assisted arthroplasty. J Knee Surg. 2019;32(3):233–8.

Dukelow SP, Herter TM, Moore KD, Demers MJ, Glasgow JI, Bagg SD, et al. Quantitative assessment of limb position sense following stroke. Neurorehabil Neural Repair. 2010;24(2):178–87.

Balasubramanian S, Wei R, Herman R, He J. Robot-measured performance metrics in stroke rehabilitation. In: 2009 ICME International Conference on Complex Medical Engineering, CME 2009. 2009.

Otaka E, Otaka Y, Kasuga S, Nishimoto A, Yamazaki K, Kawakami M, et al. Clinical usefulness and validity of robotic measures of reaching movement in hemiparetic stroke patients. J Neuroeng Rehabil. 2015;12(1):66.

Singh H, Unger J, Zariffa J, Pakosh M, Jaglal S, Craven BC, et al. Robot-assisted upper extremity rehabilitation for cervical spinal cord injuries: a systematic scoping review. Disabil Rehabil Assist Technol. 2018;13(7):704–15. https://doi.org/10.1080/17483107.2018.1425747 .

Molteni F, Gasperini G, Cannaviello G, Guanziroli E. Exoskeleton and end-effector robots for upper and lower limbs rehabilitation: narrative review. PMR. 2018;10(9):174–88.

Jutinico AL, Jaimes JC, Escalante FM, Perez-Ibarra JC, Terra MH, Siqueira AAG. Impedance control for robotic rehabilitation: a robust markovian approach. Front Neurorobot. 2017;11(AUG):1–16.

Google Scholar  

Li Z, Huang Z, He W, Su CY. Adaptive impedance control for an upper limb robotic exoskeleton using biological signals. IEEE Trans Ind Electron. 2017;64(2):1664–74.

Marchal-Crespo L, Reinkensmeyer DJ. Review of control strategies for robotic movement training after neurologic injury. J NeuroEngineering Rehabil. 2009;6:20. https://doi.org/10.1186/1743-0003-6-20 .

Cohen MX. Analyzing neural time series data: theory and practice. Cambridge: MIT Press; 2014.

Book   Google Scholar  

Stafstrom CE, Carmant L. Seizures and epilepsy: an overview. Cold Spring Harb Perspect Med. 2015;5(6):65–77.

Machado C, Cuspineda E, Valdãs P, Virues T, Liopis F, Bosch J, et al. Assessing acute middle cerebral artery ischemic stroke by quantitative electric tomography. Clin EEG Neurosci. 2004;35(3):116–24.

Finnigan SP, Walsh M, Rose SE, Chalk JB. Quantitative EEG indices of sub-acute ischaemic stroke correlate with clinical outcomes. Clin Neurophysiol. 2007;118(11):2525–31.

Cuspineda E, Machado C, Galán L, Aubert E, Alvarez MA, Llopis F, et al. QEEG prognostic value in acute stroke. Clin EEG Neurosci. 2007;38(3):155–60.

Belardinelli P, Laer L, Ortiz E, Braun C, Gharabaghi A. Plasticity of premotor cortico-muscular coherence in severely impaired stroke patients with hand paralysis. NeuroImage Clin. 2017;14:726–33.

Di PM, Schnider A, Nicolo P, Rizk S, Guggisberg AG. Coherent neural oscillations predict future motor and language improvement after stroke. Brain. 2015;138(10):3048–60.

Chen CC, Lee SH, Wang WJ, Lin YC, Su MC. EEG-based motor network biomarkers for identifying target patients with stroke for upper limb rehabilitation and its construct validity. PLoS ONE. 2017;12(6):1–20. https://doi.org/10.1371/journal.pone.0178822 .

Article   CAS   Google Scholar  

Conway BA, Halliday DM, Farmer SF, Shahani U, Maas P, Weir AI, et al. Synchronization between motor cortex and spinal motoneuronal pool during the performance of a maintained motor task in man. J Physiol. 1995;489(3):917–24.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Salenius S, Portin K, Kajola M, Salmelin R, Hari R. Cortical control of human motoneuron firing during isometric contraction. J Neurophysiol. 1997;77(6):3401–5.

Mima T, Hallett M. Electroencephalographic analysis of cortico-muscular coherence: reference effect, volume conduction and generator mechanism. Clin Neurophysiol. 1999;110(11):1892–9.

Claassen J, Hirsch LJ, Kreiter KT, Du EY, Sander Connolly E, Emerson RG, et al. Quantitative continuous EEG for detecting delayed cerebral ischemia in patients with poor-grade subarachnoid hemorrhage. Clin Neurophysiol. 2004;115(12):2699–710.

Sullivan JL, Bhagat NA, Yozbatiran N, Paranjape R, Losey CG, Grossman RG, et al. Improving robotic stroke rehabilitation by incorporating neural intent detection: preliminary results from a clinical trial. In: 2017 International Conference on Rehabilitation Robotics (ICORR). IEEE; 2017. p. 122–7.

Muralidharan A, Chae J, Taylor DM. Extracting attempted hand movements from EEGs in people with complete hand paralysis following stroke. Front Neurosci. 2011. https://doi.org/10.3389/fnins.2011.00039 .

Nam C, Rong W, Li W, Xie Y, Hu X, Zheng Y. The effects of upper-limb training assisted with an electromyography-driven neuromuscular electrical stimulation robotic hand on chronic stroke. Front Neurol. 2017. https://doi.org/10.3389/fneur.2017.00679 .

Bertolucci F, Lamola G, Fanciullacci C, Artoni F, Panarese A, Micera S, et al. EEG predicts upper limb motor improvement after robotic rehabilitation in chronic stroke patients. Ann Phys Rehabil Med. 2018;61:e200–1.

Cantillo-Negrete J, Carino-Escobar RI, Carrillo-Mora P, Elias-Vinas D, Gutierrez-Martinez J. Motor imagery-based brain-computer interface coupled to a robotic hand orthosis aimed for neurorehabilitation of stroke patients. J Healthc Eng. 2018;3(2018):1–10.

Bhagat NA, Venkatakrishnan A, Abibullaev B, Artz EJ, Yozbatiran N, Blank AA, et al. Design and optimization of an EEG-based brain machine interface (BMI) to an upper-limb exoskeleton for stroke survivors. Front Neurosci. 2016;10(MAR):122.

PubMed   PubMed Central   Google Scholar  

Biasiucci A, Leeb R, Iturrate I, Perdikis S, Al-Khodairy A, Corbet T, et al. Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. Nat Commun. 2018;9(1):1–13. https://doi.org/10.1038/s41467-018-04673-z .

Ang KK, Guan C, Chua KSG, Ang BT, Kuah C, Wang C, et al. Clinical study of neurorehabilitation in stroke using EEG-based motor imagery brain-computer interface with robotic feedback. Annu Int Conf IEEE Eng Med Biol. 2010. pp. 5549–52.

Finnigan SP, Rose SE, Walsh M, Griffin M, Janke AL, Mcmahon KL, et al. Correlation of quantitative EEG in acute ischemic stroke with 30-day NIHSS score: comparison with diffusion and perfusion MRI. Stroke. 2004;35(4):899–903.

Schleiger E, Sheikh N, Rowland T, Wong A, Read S, Finnigan S. Frontal EEG delta / alpha ratio and screening for post-stroke cognitive de fi cits: the power of four electrodes. Int J Psychophysiol. 2014;94(1):19–24. https://doi.org/10.1016/j.ijpsycho.2014.06.012 .

Aminov A, Rogers JM, Johnstone SJ, Middleton S, Wilson PH. Acute single channel EEG predictors of cognitive function after stroke. PLoS ONE. 2017;12(10): e0185841.

Andresen EM. Criteria for assessing the tools of disability outcomes research. Arch Phys Med Rehabil. 2000. https://doi.org/10.1053/apmr.2000.20619 .

Wang Q, Markopoulos P, Yu B, Chen W, Timmermans A. Interactive wearable systems for upper body rehabilitation: a systematic review. J Neuroeng Rehabil. 2017;14(1):1–21.

Tran VD, Dario P, Mazzoleni S. Kinematic measures for upper limb robot-assisted therapy following stroke and correlations with clinical outcome measures: a review. Med Eng Phys. 2018;53:13–31. https://doi.org/10.1016/j.medengphy.2017.12.005 .

Finnigan S, Wong A, Read S. Defining abnormal slow EEG activity in acute ischaemic stroke: Delta/alpha ratio as an optimal QEEG index. Clin Neurophysiol. 2016;127(2):1452–9. https://doi.org/10.1016/j.clinph.2015.07.014 .

Carter AR, Shulman GL, Corbetta M. Why use a connectivity-based approach to study stroke and recovery of function? Neuroimage. 2012;62(4):2271–80.

van der Velde B, Haartsen R, Kemner C. Test-retest reliability of EEG network characteristics in infants. Brain Behav. 2019;9(5):1–10.

Gennaro F, de Bruin ED. A pilot study assessing reliability and age-related differences in corticomuscular and intramuscular coherence in ankle dorsiflexors during walking. Physiol Rep. 2020;8(4):1–12.

Brihmat N, Loubinoux I, Castel-Lacanal E, Marque P, Gasq D. Kinematic parameters obtained with the ArmeoSpring for upper-limb assessment after stroke: a reliability and learning effect study for guiding parameter use. J Neuroeng Rehabil. 2020;17(1):130. https://doi.org/10.1186/s12984-020-00759-2 .

Dewald JPA, Ellis MD, Acosta AM, McPherson JG, Stienen AHA. Implementation of impairment- based neurorehabilitation devices and technologies following brain injury. Neurorehabilitation technology, 2nd edn. 2016. 375–392 p.

Subramanian SK, Yamanaka J, Chilingaryan G, Levin MF. Validity of movement pattern kinematics as measures of arm motor impairment poststroke. Stroke. 2010;41(10):2303–8.

Fayers PM, Machin D. Quality of life: the assessment, analysis and reporting of patient‐reported outcomes . John Wiley & Sons, Incorporated. 2016;3:89-124.

Alt Murphy M, Häger CK. Kinematic analysis of the upper extremity after stroke—how far have we reached and what have we grasped? Phys Ther Rev. 2015;20(3):137–55.

Shishov N, Melzer I, Bar-Haim S. Parameters and measures in assessment of motor learning in neurorehabilitation; a systematic review of the literature. Front Hum Neurosci. 2017. https://doi.org/10.3389/fnhum.2017.00082 .

Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–63. https://doi.org/10.1016/j.jcm.2016.02.012 .

Angel RW. Electromyographic patterns during ballistic movement of normal and spastic limbs. Brain Res. 1975;99(2):387–92.

McLellan DL. C0-contraction and stretch reflexes in spasticity during treatment with baclofen. J Neurol Neurosurg Psychiatry. 1977;40(1):30–8.

Dewald JPA, Pope PS, Given JD, Buchanan TS, Rymer WZ. Abnormal muscle coactivation patterns during isometric torque generation at the elbow and shoulder in hemiparetic subjects. Brain. 1995;118(2):495–510. https://doi.org/10.1093/brain/118.2.495 .

Wilkins KB, Yao J, Owen M, Karbasforoushan H, Carmona C, Dewald JPA. Limited capacity for ipsilateral secondary motor areas to support hand function post-stroke. J Physiol. 2020;598(11):2153–67. https://doi.org/10.1113/JP279377 .

Agius Anastasi A, Falzon O, Camilleri K, Vella M, Muscat R. Brain symmetry index in healthy and stroke patients for assessment and prognosis. Stroke Res Treat. 2017;30(2017):1–9.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339: b2700.

Colombo R, Pisano F, Micera S, Mazzone A, Delconte C, Carrozza MC, et al. Assessing mechanisms of recovery during robot-aided neurorehabilitation of the upper limb. Neurorehabil Neural Repair. 2008;22(1):50–63. https://doi.org/10.1177/1545968307303401 .

Keller U, Schölch S, Albisser U, Rudhe C, Curt A, Riener R, et al. Robot-assisted arm assessments in spinal cord injured patients: a consideration of concept study. PLoS One. 2015;10(5):e0126948. https://doi.org/10.1371/journal.pone.0126948 .

Mostafavi SM. Computational models for improved diagnosis and prognosis of stroke using robot-based biomarkers. 2016. http://hdl.handle.net/1974/14563 .

Bosecker C, Dipietro L, Volpe B, Krebs HI. Kinematic robot-based evaluation scales and clinical counterparts to measure upper limb motor performance in patients with chronic stroke. Neurorehabil Neural Repair. 2010;24(1):62–9.

Balasubramanian S, Melendez-Calderon A, Roby-Brami A, Burdet E. On the analysis of movement smoothness. J Neuroeng Rehabil. 2015;12(1):112. https://doi.org/10.1186/s12984-015-0090-9 .

Rohrer B, Fasoli S, Krebs HI, Hughes R, Volpe B, Frontera WR, et al. Movement smoothness changes during stroke recovery. J Neurosci. 2002;22(18):8297–304.

Mobini A, Behzadipour S, Saadat M. Test-retest reliability of Kinect’s measurements for the evaluation of upper body recovery of stroke patients. Biomed Eng Online. 2015;14(1):1–14.

Zariffa J, Myers M, Coahran M, Wang RH. Smallest real differences for robotic measures of upper extremity function after stroke: implications for tracking recovery. J Rehabil Assist Technol Eng. 2018;5:205566831878803. https://doi.org/10.1177/2055668318788036 .

Elovic E, Brashear A. Spasticity : diagnosis and management. New York: Demos Medical; 2011. http://ida.lib.uidaho.edu:2048/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=e000xna&AN=352265&site=ehost-live&scope=site .

Centen A, Lowrey CR, Scott SH, Yeh TT, Mochizuki G. KAPS (kinematic assessment of passive stretch): a tool to assess elbow flexor and extensor spasticity after stroke using a robotic exoskeleton. J Neuroeng Rehabil. 2017;14(1):1–13.

Sin M, Kim WS, Cho K, Cho S, Paik NJ. Improving the test-retest and inter-rater reliability for stretch reflex measurements using an isokinetic device in stroke patients with mild to moderate elbow spasticity. J Electromyogr Kinesiol. 2017;2018(39):120–7. https://doi.org/10.1016/j.jelekin.2018.01.012 .

Germanotta M, Cruciani A, Pecchioli C, Loreti S, Spedicato A, Meotti M, et al. Reliability, validity and discriminant ability of the instrumental indices provided by a novel planar robotic device for upper limb rehabilitation. J Neuroeng Rehabil. 2018;15(1):1–14.

Wagner JM, Rhodes JA, Patten C. Reproducibility and minimal detectable change of three-dimensional kinematic analysis of reaching tasks in people with hemiparaesis after stroke. 2008. https://doi.org/10.2522/ptj.20070255 .

Semrau JA, Herter TM, Scott SH, Dukelow SP. Inter-rater reliability of kinesthetic measurements with the KINARM robotic exoskeleton. J Neuroeng Rehabil. 2017;14(1):1–10.

Lin CH, Chou LW, Wei SH, Lieu FK, Chiang SL, Sung WH. Validity and reliability of a novel device for bilateral upper extremity functional measurements. Comput Methods Programs Biomed. 2014;114(3):315–23. https://doi.org/10.1016/j.cmpb.2014.02.012 .

Wolf S, Butler A, Alberts J, Kim M. Contemporary linkages between EMG, kinetics and stroke. J Electromyogr Kinesiol. 2005;15(3):229–39.

Iyer KK. Effective assessments of electroencephalography during stroke recovery : contemporary approaches and considerations. J Neurophysiol. 2017;118(5):2521–5.

Liu J, Sheng Y, Liu H. Corticomuscular coherence and its applications: a review. Front Hum Neurosci. 2019;13(March):1–16.

Pan LLH, Yang WW, Kao CL, Tsai MW, Wei SH, Fregni F, et al. Effects of 8-week sensory electrical stimulation combined with motor training on EEG-EMG coherence and motor function in individuals with stroke. Sci Rep. 2018;8(1):1–10.

Wu J, Quinlan EB, Dodakian L, McKenzie A, Kathuria N, Zhou RJ, et al. Connectivity measures are robust biomarkers of cortical function and plasticity after stroke. Brain. 2015;138(8):2359–69.

Mrachacz-Kersting N, Jiang N, Thomas Stevenson AJ, Niazi IK, Kostic V, Pavlovic A, et al. Efficient neuroplasticity induction in chronic stroke patients by an associative brain-computer interface. J Neurophysiol. 2016;115(3):1410–21.

Bentes C, Peralta AR, Viana P, Martins H, Morgado C, Casimiro C, et al. Quantitative EEG and functional outcome following acute ischemic stroke. Clin Neurophysiol. 2018;129(8):1680–7.

Leon-carrion J, Martin-rodriguez JF, Damas-lopez J, Manuel J, Dominguez-morales MR. Delta–alpha ratio correlates with level of recovery after neurorehabilitation in patients with acquired brain injury. Clin Neurophysiol. 2009;120(6):1039–45. https://doi.org/10.1016/j.clinph.2009.01.021 .

Finnigan S, van Putten MJAM. EEG in ischaemic stroke: qEEG can uniquely inform (sub-)acute prognoses and clinical management. Clin Neurophysiol. 2013;124(1):10–9.

Trujillo P, Mastropietro A, Scano A, Chiavenna A, Mrakic-Sposta S, Caimmi M, et al. Quantitative EEG for predicting upper limb motor recovery in chronic stroke robot-assisted rehabilitation. IEEE Trans Neural Syst Rehabil Eng. 2017;25(7):1058–67.

Jordan K. Emergency EEG and continuous EEG monitoring in acute ischemic stroke. Clin Neurophysiol. 2004;21(5):341–52.

Comani S, Velluto L, Schinaia L, Cerroni G, Serio A, Buzzelli S, et al. Monitoring neuro-motor recovery from stroke with high-resolution EEG, robotics and virtual reality: a proof of concept. IEEE Trans Neural Syst Rehabil Eng. 2015;23(6):1106–16.

Espenhahn S, de Berker AO, van Wijk BCM, Rossiter HE, Ward NS. Movement-related beta oscillations show high intra-individual reliability. Neuroimage. 2017;147:175–85. https://doi.org/10.1016/j.neuroimage.2016.12.025 .

Vázquez-Marrufo M, Galvao-Carmona A, Benítez Lugo ML, Ruíz-Peña JL, Borges Guerra M, Izquierdo AG. Retest reliability of individual alpha ERD topography assessed by human electroencephalography. PLoS ONE. 2017;12(10):1–16.

Dubovik S, Ptak R, Aboulafia T, Magnin C, Gillabert N, Allet L, et al. EEG alpha band synchrony predicts cognitive and motor performance in patients with ischemic stroke. In: Behavioural Neurology. Hindawi Limited; 2013. p. 187–9.

Sheorajpanday RVAA, Nagels G, Weeren AJTMTM, Putten MJAMV, Deyn PPD, van Putten MJAM, et al. Quantitative EEG in ischemic stroke: correlation with functional status after 6 months. Clin Neurophysiol. 2011;122(5):874–83. https://doi.org/10.1016/j.clinph.2010.07.028 .

De Vico Fallani F, Astolfi L, Cincotti F, Mattia D, La Rocca D, Maksuti E, et al. Evaluation of the brain network organization from EEG signals: a preliminary evidence in stroke patient. In: Anatomical Record. 2009. p. 2023–31.

Westlake KP, Nagarajan SS. Functional connectivity in relation to motor performance and recovery after stroke. Front Syst Neurosci. 2011;18(5):8.

Caliandro P, Vecchio F, Miraglia F, Reale G, Della Marca G, La Torre G, et al. Small-world characteristics of cortical connectivity changes in acute stroke. Neurorehabil Neural Repair. 2017;31(1):81–94.

Eldeeb S, Akcakaya M, Sybeldon M, Foldes S, Santarnecchi E, Pascual-Leone A, et al. EEG-based functional connectivity to analyze motor recovery after stroke: a pilot study. Biomed Signal Process Control. 2019;49:419–26.

Myers LJ, O’Malley M. The relationship between human cortico-muscular coherence and rectified EMG. In: International IEEE/EMBS Conference on Neural Engineering, NER. IEEE Computer Society; 2003. p. 289–92.

Braun C, Staudt M, Schmitt C, Preissl H, Birbaumer N, Gerloff C. Crossed cortico-spinal motor control after capsular stroke. Eur J Neurosci. 2007;25(9):2935–45.

Larsen LH, Zibrandtsen IC, Wienecke T, Kjaer TW, Christensen MS, Nielsen JB, et al. Corticomuscular coherence in the acute and subacute phase after stroke. Clin Neurophysiol. 2017;128(11):2217–26.

Ang KK, Chua KSG, Phua KS, Wang C, Chin ZY, Kuah CWK, et al. A randomized controlled trial of EEG-based motor imagery brain-computer interface robotic rehabilitation for stroke. Clin EEG Neurosci. 2015;46(4):310–20.

Liu S, Guo J, Meng J, Wang Z, Yao Y, Yang J, et al. Abnormal EEG complexity and functional connectivity of brain in patients with acute thalamic ischemic stroke. Comput Math Methods Med. 2016;14(2016):1–9.

CAS   Google Scholar  

Sun R, Wong W, Wang J, Tong RK. Changes in electroencephalography complexity using a brain computer interface-motor observation training in chronic stroke patients : a fuzzy approximate entropy analysis. Front Hum Neurosci. 2017;5(11):444.

Auriat AM, Neva JL, Peters S, Ferris JK, Boyd LA. A review of transcranial magnetic stimulation and multimodal neuroimaging to characterize post-stroke neuroplasticity. Front Neurol. 2015;6:1–20.

Niedermeyer E, Schomer DL, Lopes da Silva FH. Niedermeyer’s electroencephalography: basic principles, clinical applications, and related fields, 6th edn. Philadelphia: Lippincott Williams & Wilkins.; 2011.

Foreman B, Claasen J. Update in intensive care and emergency medicine. Update in intensive care and emergency medicine. Springer Berlin Heidelberg; 2012.

Tolonen U, Ahonen A, Kallanranta T, Hokkanen E. Non-invasive external regional measurement of cerebral circulation time changes in supratentorial infarctions using pertechnetate. Stroke. 1981;12(4):437–44.

Saes M, Zandvliet SB, Andringa AS, Daffertshofer A, Twisk JWR, Meskers CGM, et al. Is resting-state EEG longitudinally associated with recovery of clinical neurological impairments early poststroke? A prospective cohort study. Neurorehabil Neural Repair. 2020;34(5):389–402.

Rogers J, Middleton S, Wilson PH, Johnstone SJ. Predicting functional outcomes after stroke: an observational study of acute single-channel EEG. Top Stroke Rehabil. 2020;27(3):161–72. https://doi.org/10.1080/10749357.2019.1673576 .

Sale P, Infarinato F, Lizio R, Babiloni C. Electroencephalographic markers of robot-aided therapy in stroke patients for the evaluation of upper limb rehabilitation. Rehabil Res. 2015;38(4):294–305.

Sheorajpanday RVA, Nagels G, Weeren AJTM, De Surgeloose D, De Deyn PP, De DPP. Additional value of quantitative EEG in acute anterior circulation syndrome of presumed ischemic origin. Clin Neurophysiol. 2010;121(10):1719–25.

Saes M, Meskers CGM, Daffertshofer A, van Wegen EEH, Kwakkel G. Are early measured resting-state EEG parameters predictive for upper limb motor impairment six months poststroke? Clin Neurophysiol. 2021;132(1):56–62. https://doi.org/10.1016/j.clinph.2020.09.031 .

Saes M, Meskers CGM, Daffertshofer A, de Munck JC, Kwakkel G, van Wegen EEH. How does upper extremity Fugl-Meyer motor score relate to resting-state EEG in chronic stroke? A power spectral density analysis. Clin Neurophysiol. 2019;130(5):856–62. https://doi.org/10.1016/j.clinph.2019.01.007 .

Nolte G, Bai O, Mari Z, Vorbach S, Hallett M. Identifying true brain interaction from EEG data using the imaginary part of coherency. Clin Neurophysiol. 2004;115(10):2292–307.

Stam CJ, Nolte G, Daffertshofer A. Phase lag index: assessment of functional connectivity from multi channel EEG and MEG with diminished bias from common sources. Hum Brain Mapp. 2007;28(11):1178–93.

Bullmore E, Sporns O. Complex brain networks: graph theoretical analysis of structural and functional systems. Nat Rev Neurosci. 2009;10(3):186–98.

Baccalá LA, Sameshima K. Partial directed coherence: a new concept in neural structure determination. Biol Cybern. 2001;84(6):463–74.

Baccalá LA, Sameshima K, Takahashi D. Generalized partial directed coherence. Int Conf Digit Signal Process. 2007;3:163–6.

Schelter B, Timmer J, Eichler M. Assessing the strength of directed influences among neural signals using renormalized partial directed coherence. J Neurosci Methods. 2009;179(1):121–30.

Kamiński M, Ding M, Truccolo WA, Bressler SL. Evaluating causal relations in neural systems: Granger causality, directed transfer function and statistical assessment of significance. Biol Cybern. 2001;85(2):145–57.

Korzeniewska A, Mańczak M, Kamiński M, Blinowska KJ, Kasicki S. Determination of information flow direction among brain structures by a modified directed transfer function (dDTF) method. J Neurosci Methods. 2003;125(1–2):195–207.

Fornito A, Bullmore ET, Zalesky A. Fundamentals of brain network analysis. Cambridge: Academic Press; 2016.

Philips GR, Daly JJ, Príncipe JC. Topographical measures of functional connectivity as biomarkers for post-stroke motor recovery. J Neuroeng Rehabil. 2017;14(1):67.

Pichiorri F, Petti M, Caschera S, Astolfi L, Cincotti F, Mattia D. An EEG index of sensorimotor interhemispheric coupling after unilateral stroke: clinical and neurophysiological study. Eur J Neurosci. 2018;47(2):158–63.

Hoshino T, Oguchi K, Inoue K, Hoshino A, Hoshiyama M. Relationship between upper limb function and functional neural connectivity among motor related-areas during recovery stage after stroke. Top Stroke Rehabil. 2020;27(1):57–66. https://doi.org/10.1080/10749357.2019.1658429 .

Hordacre B, Goldsworthy MR, Welsby E, Graetz L, Ballinger S, Hillier S. Resting state functional connectivity is associated with motor pathway integrity and upper-limb behavior in chronic stroke. Neurorehabil Neural Repair. 2020;34(6):547–57.

Riahi N, Vakorin VA, Menon C. Estimating Fugl-Meyer upper extremity motor score from functional-connectivity measures. IEEE Trans Neural Syst Rehabil Eng. 2020;28(4):860–8.

Gwin JT, Ferris DP. Beta- and gamma-range human lower limb corticomuscular coherence. Front Hum Neurosci. 2012;11(6):258.

Zheng Y, Peng Y, Xu G, Li L, Wang J. Using corticomuscular coherence to reflect function recovery of paretic upper limb after stroke: a case study. Front Neurol. 2018;10(8):728.

Rossiter HE, Eaves C, Davis E, Boudrias MH, Park CH, Farmer S, et al. Changes in the location of cortico-muscular coherence following stroke. NeuroImage Clin. 2013;2(1):50–5.

Mima T, Toma K, Koshy B, Hallett M. Coherence between cortical and muscular activities after subcortical stroke. Stroke. 2001;32(11):2597–601.

Krauth R, Schwertner J, Vogt S, Lindquist S, Sailer M, Sickert A, et al. Cortico-muscular coherence is reduced acutely post-stroke and increases bilaterally during motor recovery: a pilot study. Front Neurol. 2019;20(10):126.

Bao SC, Wong WW, Leung TW, Tong KY. Low gamma band cortico-muscular coherence inter-hemisphere difference following chronic stroke. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS. Institute of Electrical and Electronics Engineers Inc.; 2018. p. 247–50.

von Carlowitz-Ghori K, Bayraktaroglu Z, Hohlefeld FU, Losch F, Curio G, Nikulin VV. Corticomuscular coherence in acute and chronic stroke. Clin Neurophysiol. 2014;125(6):1182–91.

Chen X, Xie P, Zhang Y, Chen Y, Cheng S, Zhang L. Abnormal functional corticomuscular coupling after stroke. NeuroImage Clin. 2018;19:147–59. https://doi.org/10.1016/j.nicl.2018.04.004 .

Curado MR, Cossio EG, Broetz D, Agostini M, Cho W, Brasil FL, et al. Residual upper arm motor function primes innervation of paretic forearm muscles in chronic stroke after brain-machine interface (BMI) training. PLoS ONE. 2015;10(10):1–18.

Guo Z, Qian Q, Wong K, Zhu H, Huang Y, Hu X, et al. Altered corticomuscular coherence (CMCoh) pattern in the upper limb during finger movements after stroke. Front Neurol. 2020. https://doi.org/10.3389/fneur.2020.00410 .

Bruton A, Conway JH, Holgate ST. Reliability: what is it, and how is it measured? Physiotherapy. 2000;86(2):94–9.

Colombo R, Cusmano I, Sterpi I, Mazzone A, Delconte C, Pisano F. Test-retest reliability of robotic assessment measures for the evaluation of upper limb recovery. IEEE Trans Neural Syst Rehabil Eng. 2014;22(5):1020–9.

Costa V, Ramírez Ó, Otero A, Muñoz-García D, Uribarri S, Raya R. Validity and reliability of inertial sensors for elbow and wrist range of motion assessment. PeerJ. 2020;8: e9687.

Gasser T, Bächer P, Steinberg H. Test-retest reliability of spectral parameters of the EEG. Electroencephalogr Clin Neurophysiol. 1985;60(4):312–9.

Levin AR, Naples AJ, Scheffler AW, Webb SJ, Shic F, Sugar CA, et al. Day-to-day test-retest reliability of EEG profiles in children with autism spectrum disorder and typical development. Front Integr Neurosci. 2020;14:1–12.

Briels CT, Briels CT, Schoonhoven DN, Schoonhoven DN, Stam CJ, De Waal H, et al. Reproducibility of EEG functional connectivity in Alzheimer’s disease. Alzheimer’s Res Ther. 2020;12(1):1–14.

Marquetand J, Vannoni S, Carboni M, Li Hegner Y, Stier C, Braun C, et al. Reliability of magnetoencephalography and high-density electroencephalography resting-state functional connectivity metrics. Brain Connect. 2019;9(7):539–53.

Lowrey CR, Blazevski B, Marnet J-L, Bretzke H, Dukelow SP, Scott SH. Robotic tests for position sense and movement discrimination in the upper limb reveal that they each are highly reproducible but not correlated in healthy individuals. J Neuroeng Rehabil. 2020;17(1):103. https://doi.org/10.1186/s12984-020-00721-2 .

Simmatis LER, Early S, Moore KD, Appaqaq S, Scott SH. Statistical measures of motor, sensory and cognitive performance across repeated robot-based testing. J Neuroeng Rehabil. 2020;17(1):86. https://doi.org/10.1186/s12984-020-00713-2 .

Download references

Acknowledgements

The authors would like to thank Stephen Goodwin and Aaron I. Feinstein for their contributions to the collection and organization of references on robotic systems, measurements, and metrics.

This work was funded by the National Science Foundation (Award#1532239) and the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health (Award#K12HD073945). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Science Foundation nor the National Institutes of Health.

Author information

Authors and affiliations.

Mechanical Engineering Department, University of Idaho, Moscow, ID, USA

Rene M. Maura, Eric T. Wolbrecht & Joel C. Perry

Engineering and Physics Department, Whitworth University, Spokane, WA, USA

Richard E. Stevens

College of Medicine, Washington State University, Spokane, WA, USA

Douglas L. Weeks

Electrical Engineering Department, University of Idaho, ID, Moscow, USA

Sebastian Rueda Parra

You can also search for this author in PubMed   Google Scholar

Contributions

RM, and SRP drafted the manuscript and performed the literature search. EW, JP, RS, and DW provided concepts, edited, and revised the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Rene M. Maura .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table 11 .

See Table 12 .

See Table 13 .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Maura, R.M., Rueda Parra, S., Stevens, R.E. et al. Literature review of stroke assessment for upper-extremity physical function via EEG, EMG, kinematic, and kinetic measurements and their reliability. J NeuroEngineering Rehabil 20 , 21 (2023). https://doi.org/10.1186/s12984-023-01142-7

Download citation

Received : 27 May 2021

Accepted : 19 January 2023

Published : 15 February 2023

DOI : https://doi.org/10.1186/s12984-023-01142-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Reliability
• Robot-assisted therapy
• Exoskeleton
• Neurological assessment
• Rehabilitation
• Motor function

Journal of NeuroEngineering and Rehabilitation

ISSN: 1743-0003

• Submission enquiries: [email protected]

• Open access
• Published: 15 April 2024

Correction: Role of AI chatbots in education: systematic literature review

• Lasha Labadze 1 ,
• Maya Grigolia 2 &
• Lela Machaidze 3  

International Journal of Educational Technology in Higher Education volume  21 , Article number:  28 ( 2024 ) Cite this article

17 Accesses

Metrics details

The Original Article was published on 31 October 2023

Correction: Int J Educ Technol High Educ 20, 56 (2023)

https://doi.org/10.1186/s41239-023-00426-1

After publication of the original article (Labadze et al., 2023 ), the authors became aware that the use of a Large Language Model (LLM) has not been sufficiently documented in accordance with our editorial policy.

In view of this, the authors would like to add the following sentence to the Methodology section of the article:

“To improve the clarity of the discussion section, we employed Large Language Model (LLM) for stylistic suggestions.”

The original article has been updated.

Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. Int J Educ Technol High Educ , 20 , 56. https://doi.org/10.1186/s41239-023-00426-1 .

Article   Google Scholar  

Download references

Author information

Authors and affiliations.

Finance Department, American University of the Middle East, Block 6, Building 1, Egaila, Kuwait

Lasha Labadze

Statistics Department, American University of the Middle East, Block 6, Building 1, Egaila, Kuwait

Maya Grigolia

Caucasus School of Business, Caucasus University, 1 Paata Saakadze St, Tbilisi, 0102, Georgia

Lela Machaidze

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Lasha Labadze .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The online version of the original article can be found at https://doi.org/10.1186/s41239-023-00426-1 .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Labadze, L., Grigolia, M. & Machaidze, L. Correction: Role of AI chatbots in education: systematic literature review. Int J Educ Technol High Educ 21 , 28 (2024). https://doi.org/10.1186/s41239-024-00461-6

Download citation

Published : 15 April 2024

DOI : https://doi.org/10.1186/s41239-024-00461-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Loading metrics

Open Access

Ten Simple Rules for Writing a Literature Review

* E-mail: [email protected]

Affiliations Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France, Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France

• Marco Pautasso

Published: July 18, 2013

• https://doi.org/10.1371/journal.pcbi.1003149
• Reader Comments

Citation: Pautasso M (2013) Ten Simple Rules for Writing a Literature Review. PLoS Comput Biol 9(7): e1003149. https://doi.org/10.1371/journal.pcbi.1003149

Editor: Philip E. Bourne, University of California San Diego, United States of America

Copyright: © 2013 Marco Pautasso. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

Competing interests: The author has declared that no competing interests exist.

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications [1] . For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively [2] . Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests [3] . Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read [4] . For such summaries to be useful, however, they need to be compiled in a professional way [5] .

When starting from scratch, reviewing the literature can require a titanic amount of work. That is why researchers who have spent their career working on a certain research issue are in a perfect position to review that literature. Some graduate schools are now offering courses in reviewing the literature, given that most research students start their project by producing an overview of what has already been done on their research issue [6] . However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.

Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills [7] . In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.

Rule 1: Define a Topic and Audience

How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review. On the one hand, if you take several years to choose, several other people may have had the same idea in the meantime. On the other hand, only a well-considered topic is likely to lead to a brilliant literature review [8] . The topic must at least be:

• interesting to you (ideally, you should have come across a series of recent papers related to your line of work that call for a critical summary),
• an important aspect of the field (so that many readers will be interested in the review and there will be enough material to write it), and
• a well-defined issue (otherwise you could potentially include thousands of publications, which would make the review unhelpful).

Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9] , but also from serendipitous moments during desultory reading and discussions. In addition to choosing your topic, you should also select a target audience. In many cases, the topic (e.g., web services in computational biology) will automatically define an audience (e.g., computational biologists), but that same topic may also be of interest to neighbouring fields (e.g., computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

After having chosen your topic and audience, start by checking the literature and downloading relevant papers. Five pieces of advice here:

• keep track of the search items you use (so that your search can be replicated [10] ),
• keep a list of papers whose pdfs you cannot access immediately (so as to retrieve them later with alternative strategies),
• use a paper management system (e.g., Mendeley, Papers, Qiqqa, Sente),
• define early in the process some criteria for exclusion of irrelevant papers (these criteria can then be described in the review to help define its scope), and
• do not just look for research papers in the area you wish to review, but also seek previous reviews.

The chances are high that someone will already have published a literature review ( Figure 1 ), if not exactly on the issue you are planning to tackle, at least on a related topic. If there are already a few or several reviews of the literature on your issue, my advice is not to give up, but to carry on with your own literature review,

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33] .

https://doi.org/10.1371/journal.pcbi.1003149.g001

• discussing in your review the approaches, limitations, and conclusions of past reviews,
• trying to find a new angle that has not been covered adequately in the previous reviews, and
• incorporating new material that has inevitably accumulated since their appearance.

When searching the literature for pertinent papers and reviews, the usual rules apply:

• be thorough,
• use different keywords and database sources (e.g., DBLP, Google Scholar, ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science), and
• look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

If you read the papers first, and only afterwards start writing the review, you will need a very good memory to remember who wrote what, and what your impressions and associations were while reading each single paper. My advice is, while reading, to start writing down interesting pieces of information, insights about how to organize the review, and thoughts on what to write. This way, by the time you have read the literature you selected, you will already have a rough draft of the review.

Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11] , but you will have avoided the danger posed by staring at a blank document. Be careful when taking notes to use quotation marks if you are provisionally copying verbatim from the literature. It is advisable then to reformulate such quotes with your own words in the final draft. It is important to be careful in noting the references already at this stage, so as to avoid misattributions. Using referencing software from the very beginning of your endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

After having taken notes while reading the literature, you will have a rough idea of the amount of material available for the review. This is probably a good time to decide whether to go for a mini- or a full review. Some journals are now favouring the publication of rather short reviews focusing on the last few years, with a limit on the number of words and citations. A mini-review is not necessarily a minor review: it may well attract more attention from busy readers, although it will inevitably simplify some issues and leave out some relevant material due to space limitations. A full review will have the advantage of more freedom to cover in detail the complexities of a particular scientific development, but may then be left in the pile of the very important papers “to be read” by readers with little time to spare for major monographs.

There is probably a continuum between mini- and full reviews. The same point applies to the dichotomy of descriptive vs. integrative reviews. While descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, integrative reviews attempt to find common ideas and concepts from the reviewed material [12] . A similar distinction exists between narrative and systematic reviews: while narrative reviews are qualitative, systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias [13] , [14] . When systematic reviews analyse quantitative results in a quantitative way, they become meta-analyses. The choice between different review types will have to be made on a case-by-case basis, depending not just on the nature of the material found and the preferences of the target journal(s), but also on the time available to write the review and the number of coauthors [15] .

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16 , 17 . Including material just for the sake of it can easily lead to reviews that are trying to do too many things at once. The need to keep a review focused can be problematic for interdisciplinary reviews, where the aim is to bridge the gap between fields [18] . If you are writing a review on, for example, how epidemiological approaches are used in modelling the spread of ideas, you may be inclined to include material from both parent fields, epidemiology and the study of cultural diffusion. This may be necessary to some extent, but in this case a focused review would only deal in detail with those studies at the interface between epidemiology and the spread of ideas.

While focus is an important feature of a successful review, this requirement has to be balanced with the need to make the review relevant to a broad audience. This square may be circled by discussing the wider implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing the literature is not stamp collecting. A good review does not just summarize the literature, but discusses it critically, identifies methodological problems, and points out research gaps [19] . After having read a review of the literature, a reader should have a rough idea of:

• the major achievements in the reviewed field,
• the main areas of debate, and
• the outstanding research questions.

It is challenging to achieve a successful review on all these fronts. A solution can be to involve a set of complementary coauthors: some people are excellent at mapping what has been achieved, some others are very good at identifying dark clouds on the horizon, and some have instead a knack at predicting where solutions are going to come from. If your journal club has exactly this sort of team, then you should definitely write a review of the literature! In addition to critical thinking, a literature review needs consistency, for example in the choice of passive vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review has a number of telling features: it is worth the reader's time, timely, systematic, well written, focused, and critical. It also needs a good structure. With reviews, the usual subdivision of research papers into introduction, methods, results, and discussion does not work or is rarely used. However, a general introduction of the context and, toward the end, a recapitulation of the main points covered and take-home messages make sense also in the case of reviews. For systematic reviews, there is a trend towards including information about how the literature was searched (database, keywords, time limits) [20] .

How can you organize the flow of the main body of the review so that the reader will be drawn into and guided through it? It is generally helpful to draw a conceptual scheme of the review, e.g., with mind-mapping techniques. Such diagrams can help recognize a logical way to order and link the various sections of a review [21] . This is the case not just at the writing stage, but also for readers if the diagram is included in the review as a figure. A careful selection of diagrams and figures relevant to the reviewed topic can be very helpful to structure the text too [22] .

Rule 8: Make Use of Feedback

Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23] . As a rule, incorporating feedback from reviewers greatly helps improve a review draft. Having read the review with a fresh mind, reviewers may spot inaccuracies, inconsistencies, and ambiguities that had not been noticed by the writers due to rereading the typescript too many times. It is however advisable to reread the draft one more time before submission, as a last-minute correction of typos, leaps, and muddled sentences may enable the reviewers to focus on providing advice on the content rather than the form.

Feedback is vital to writing a good review, and should be sought from a variety of colleagues, so as to obtain a diversity of views on the draft. This may lead in some cases to conflicting views on the merits of the paper, and on how to improve it, but such a situation is better than the absence of feedback. A diversity of feedback perspectives on a literature review can help identify where the consensus view stands in the landscape of the current scientific understanding of an issue [24] .

Rule 9: Include Your Own Relevant Research, but Be Objective

In many cases, reviewers of the literature will have published studies relevant to the review they are writing. This could create a conflict of interest: how can reviewers report objectively on their own work [25] ? Some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. However, bias could also occur in the other direction: some scientists may be unduly dismissive of their own achievements, so that they will tend to downplay their contribution (if any) to a field when reviewing it.

In general, a review of the literature should neither be a public relations brochure nor an exercise in competitive self-denial. If a reviewer is up to the job of producing a well-organized and methodical review, which flows well and provides a service to the readership, then it should be possible to be objective in reviewing one's own relevant findings. In reviews written by multiple authors, this may be achieved by assigning the review of the results of a coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

Given the progressive acceleration in the publication of scientific papers, today's reviews of the literature need awareness not just of the overall direction and achievements of a field of inquiry, but also of the latest studies, so as not to become out-of-date before they have been published. Ideally, a literature review should not identify as a major research gap an issue that has just been addressed in a series of papers in press (the same applies, of course, to older, overlooked studies (“sleeping beauties” [26] )). This implies that literature reviewers would do well to keep an eye on electronic lists of papers in press, given that it can take months before these appear in scientific databases. Some reviews declare that they have scanned the literature up to a certain point in time, but given that peer review can be a rather lengthy process, a full search for newly appeared literature at the revision stage may be worthwhile. Assessing the contribution of papers that have just appeared is particularly challenging, because there is little perspective with which to gauge their significance and impact on further research and society.

Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from all quarters after the review has been published, so that there may soon be the need for an updated review. But this is the nature of science [27] – [32] . I wish everybody good luck with writing a review of the literature.

Acknowledgments

Many thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.

• 1. Rapple C (2011) The role of the critical review article in alleviating information overload. Annual Reviews White Paper. Available: http://www.annualreviews.org/userimages/ContentEditor/1300384004941/Annual_Reviews_WhitePaper_Web_2011.pdf . Accessed May 2013.
• View Article
• Google Scholar
• 7. Budgen D, Brereton P (2006) Performing systematic literature reviews in software engineering. Proc 28th Int Conf Software Engineering, ACM New York, NY, USA, pp. 1051–1052. doi: https://doi.org/10.1145/1134285.1134500 .
• 16. Eco U (1977) Come si fa una tesi di laurea. Milan: Bompiani.
• 17. Hart C (1998) Doing a literature review: releasing the social science research imagination. London: SAGE.
• 21. Ridley D (2008) The literature review: a step-by-step guide for students. London: SAGE.

REVIEW article

This article is part of the research topic.

Advances in Chronic Pain Treatment

Natural Biomimetic Nano-system for Drug Delivery in the Treatment of Rheumatoid Arthritis: A Literature Review of the Last 5 Years Provisionally Accepted

• 1 The First People Hospital of Mudanjiang City, China

The final, formatted version of the article will be published soon.

Rheumatoid arthritis (RA) is a chronic systemic autoimmune disease characterized primarily by synovitis, leading to the destruction of articular cartilage and bone and ultimately resulting in joint deformity, loss of function, and a significant impact on patients' quality of life. Currently, a combination of anti-rheumatic drugs, hormonal drugs, and biologics is used to mitigate disease progression. However, conventional drug therapy has limited bioavailability, and long-term use often leads to drug resistance and toxic side effects. Therefore, exploring new therapeutic approaches for RA is of great clinical importance. Nanodrug delivery systems offer promising solutions to overcome the limitations of conventional drugs. Among them, liposomes, the first nanodrug delivery system to be approved for clinical application and still widely studied, demonstrate the ability to enhance therapeutic efficacy with fewer adverse effects through passive or active targeting mechanisms. In this review, we provide a review of the research progress on the targeting mechanisms of various natural biomimetic nano-delivery systems in RA therapy. Additionally, we predict the development trends and application prospects of these systems, offering new directions for precision treatment of RA.

Keywords: Rheumatoid arthritis, natural biomimetic, Nano-system, Drug delivery, Treatment

Received: 12 Feb 2024; Accepted: 16 Apr 2024.

Copyright: © 2024 Li, Li and Zhuang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Mx. Liping Zhuang, The First People Hospital of Mudanjiang City, Mudanjiang, China

People also looked at

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Grad Med Educ
• v.8(3); 2016 Jul

The Literature Review: A Foundation for High-Quality Medical Education Research

a  These are subscription resources. Researchers should check with their librarian to determine their access rights.

Despite a surge in published scholarship in medical education 1 and rapid growth in journals that publish educational research, manuscript acceptance rates continue to fall. 2 Failure to conduct a thorough, accurate, and up-to-date literature review identifying an important problem and placing the study in context is consistently identified as one of the top reasons for rejection. 3 , 4 The purpose of this editorial is to provide a road map and practical recommendations for planning a literature review. By understanding the goals of a literature review and following a few basic processes, authors can enhance both the quality of their educational research and the likelihood of publication in the Journal of Graduate Medical Education ( JGME ) and in other journals.

The Literature Review Defined

In medical education, no organization has articulated a formal definition of a literature review for a research paper; thus, a literature review can take a number of forms. Depending on the type of article, target journal, and specific topic, these forms will vary in methodology, rigor, and depth. Several organizations have published guidelines for conducting an intensive literature search intended for formal systematic reviews, both broadly (eg, PRISMA) 5 and within medical education, 6 and there are excellent commentaries to guide authors of systematic reviews. 7 , 8

• A literature review forms the basis for high-quality medical education research and helps maximize relevance, originality, generalizability, and impact.
• A literature review provides context, informs methodology, maximizes innovation, avoids duplicative research, and ensures that professional standards are met.
• Literature reviews take time, are iterative, and should continue throughout the research process.
• Researchers should maximize the use of human resources (librarians, colleagues), search tools (databases/search engines), and existing literature (related articles).
• Keeping organized is critical.

Such work is outside the scope of this article, which focuses on literature reviews to inform reports of original medical education research. We define such a literature review as a synthetic review and summary of what is known and unknown regarding the topic of a scholarly body of work, including the current work's place within the existing knowledge . While this type of literature review may not require the intensive search processes mandated by systematic reviews, it merits a thoughtful and rigorous approach.

Purpose and Importance of the Literature Review

An understanding of the current literature is critical for all phases of a research study. Lingard 9 recently invoked the “journal-as-conversation” metaphor as a way of understanding how one's research fits into the larger medical education conversation. As she described it: “Imagine yourself joining a conversation at a social event. After you hang about eavesdropping to get the drift of what's being said (the conversational equivalent of the literature review), you join the conversation with a contribution that signals your shared interest in the topic, your knowledge of what's already been said, and your intention.” 9

The literature review helps any researcher “join the conversation” by providing context, informing methodology, identifying innovation, minimizing duplicative research, and ensuring that professional standards are met. Understanding the current literature also promotes scholarship, as proposed by Boyer, 10 by contributing to 5 of the 6 standards by which scholarly work should be evaluated. 11 Specifically, the review helps the researcher (1) articulate clear goals, (2) show evidence of adequate preparation, (3) select appropriate methods, (4) communicate relevant results, and (5) engage in reflective critique.

Failure to conduct a high-quality literature review is associated with several problems identified in the medical education literature, including studies that are repetitive, not grounded in theory, methodologically weak, and fail to expand knowledge beyond a single setting. 12 Indeed, medical education scholars complain that many studies repeat work already published and contribute little new knowledge—a likely cause of which is failure to conduct a proper literature review. 3 , 4

Likewise, studies that lack theoretical grounding or a conceptual framework make study design and interpretation difficult. 13 When theory is used in medical education studies, it is often invoked at a superficial level. As Norman 14 noted, when theory is used appropriately, it helps articulate variables that might be linked together and why, and it allows the researcher to make hypotheses and define a study's context and scope. Ultimately, a proper literature review is a first critical step toward identifying relevant conceptual frameworks.

Another problem is that many medical education studies are methodologically weak. 12 Good research requires trained investigators who can articulate relevant research questions, operationally define variables of interest, and choose the best method for specific research questions. Conducting a proper literature review helps both novice and experienced researchers select rigorous research methodologies.

Finally, many studies in medical education are “one-offs,” that is, single studies undertaken because the opportunity presented itself locally. Such studies frequently are not oriented toward progressive knowledge building and generalization to other settings. A firm grasp of the literature can encourage a programmatic approach to research.

Approaching the Literature Review

Considering these issues, journals have a responsibility to demand from authors a thoughtful synthesis of their study's position within the field, and it is the authors' responsibility to provide such a synthesis, based on a literature review. The aforementioned purposes of the literature review mandate that the review occurs throughout all phases of a study, from conception and design, to implementation and analysis, to manuscript preparation and submission.

Planning the literature review requires understanding of journal requirements, which vary greatly by journal ( table 1 ). Authors are advised to take note of common problems with reporting results of the literature review. Table 2 lists the most common problems that we have encountered as authors, reviewers, and editors.

Sample of Journals' Author Instructions for Literature Reviews Conducted as Part of Original Research Article a

Common Problem Areas for Reporting Literature Reviews in the Context of Scholarly Articles

Locating and Organizing the Literature

Three resources may facilitate identifying relevant literature: human resources, search tools, and related literature. As the process requires time, it is important to begin searching for literature early in the process (ie, the study design phase). Identifying and understanding relevant studies will increase the likelihood of designing a relevant, adaptable, generalizable, and novel study that is based on educational or learning theory and can maximize impact.

Human Resources

A medical librarian can help translate research interests into an effective search strategy, familiarize researchers with available information resources, provide information on organizing information, and introduce strategies for keeping current with emerging research. Often, librarians are also aware of research across their institutions and may be able to connect researchers with similar interests. Reaching out to colleagues for suggestions may help researchers quickly locate resources that would not otherwise be on their radar.

During this process, researchers will likely identify other researchers writing on aspects of their topic. Researchers should consider searching for the publications of these relevant researchers (see table 3 for search strategies). Additionally, institutional websites may include curriculum vitae of such relevant faculty with access to their entire publication record, including difficult to locate publications, such as book chapters, dissertations, and technical reports.

Strategies for Finding Related Researcher Publications in Databases and Search Engines

Search Tools and Related Literature

Researchers will locate the majority of needed information using databases and search engines. Excellent resources are available to guide researchers in the mechanics of literature searches. 15 , 16

Because medical education research draws on a variety of disciplines, researchers should include search tools with coverage beyond medicine (eg, psychology, nursing, education, and anthropology) and that cover several publication types, such as reports, standards, conference abstracts, and book chapters (see the box for several information resources). Many search tools include options for viewing citations of selected articles. Examining cited references provides additional articles for review and a sense of the influence of the selected article on its field.

Box Information Resources

• Web of Science a
• Education Resource Information Center (ERIC)
• Cumulative Index of Nursing & Allied Health (CINAHL) a
• Google Scholar

Once relevant articles are located, it is useful to mine those articles for additional citations. One strategy is to examine references of key articles, especially review articles, for relevant citations.

Getting Organized

As the aforementioned resources will likely provide a tremendous amount of information, organization is crucial. Researchers should determine which details are most important to their study (eg, participants, setting, methods, and outcomes) and generate a strategy for keeping those details organized and accessible. Increasingly, researchers utilize digital tools, such as Evernote, to capture such information, which enables accessibility across digital workspaces and search capabilities. Use of citation managers can also be helpful as they store citations and, in some cases, can generate bibliographies ( table 4 ).

Citation Managers

Knowing When to Say When

Researchers often ask how to know when they have located enough citations. Unfortunately, there is no magic or ideal number of citations to collect. One strategy for checking coverage of the literature is to inspect references of relevant articles. As researchers review references they will start noticing a repetition of the same articles with few new articles appearing. This can indicate that the researcher has covered the literature base on a particular topic.

Putting It All Together

In preparing to write a research paper, it is important to consider which citations to include and how they will inform the introduction and discussion sections. The “Instructions to Authors” for the targeted journal will often provide guidance on structuring the literature review (or introduction) and the number of total citations permitted for each article category. Reviewing articles of similar type published in the targeted journal can also provide guidance regarding structure and average lengths of the introduction and discussion sections.

When selecting references for the introduction consider those that illustrate core background theoretical and methodological concepts, as well as recent relevant studies. The introduction should be brief and present references not as a laundry list or narrative of available literature, but rather as a synthesized summary to provide context for the current study and to identify the gap in the literature that the study intends to fill. For the discussion, citations should be thoughtfully selected to compare and contrast the present study's findings with the current literature and to indicate how the present study moves the field forward.

To facilitate writing a literature review, journals are increasingly providing helpful features to guide authors. For example, the resources available through JGME include several articles on writing. 17 The journal Perspectives on Medical Education recently launched “The Writer's Craft,” which is intended to help medical educators improve their writing. Additionally, many institutions have writing centers that provide web-based materials on writing a literature review, and some even have writing coaches.

The literature review is a vital part of medical education research and should occur throughout the research process to help researchers design a strong study and effectively communicate study results and importance. To achieve these goals, researchers are advised to plan and execute the literature review carefully. The guidance in this editorial provides considerations and recommendations that may improve the quality of literature reviews.

• Open access
• Published: 10 April 2024

Development of an index system for the scientific literacy of medical staff: a modified Delphi study in China

• Shuyu Liang 2   na1 ,
• Ziyan Zhai 2   na1 ,
• Xingmiao Feng 2 ,
• Xiaozhi Sun 1 ,
• Jingxuan Jiao 1 ,
• Yuan Gao 1   na2 &
• Kai Meng   ORCID: orcid.org/0000-0003-1467-7904 2 , 3   na2  

BMC Medical Education volume  24 , Article number:  397 ( 2024 ) Cite this article

128 Accesses

Metrics details

Scientific research activity in hospitals is important for promoting the development of clinical medicine, and the scientific literacy of medical staff plays an important role in improving the quality and competitiveness of hospital research. To date, no index system applicable to the scientific literacy of medical staff in China has been developed that can effectively evaluate and guide scientific literacy. This study aimed to establish an index system for the scientific literacy of medical staff in China and provide a reference for improving the evaluation of this system.

In this study, a preliminary indicator pool for the scientific literacy of medical staff was constructed through the nominal group technique ( n  = 16) with medical staff. Then, two rounds of Delphi expert consultation surveys ( n  = 20) were conducted with clinicians, and the indicators were screened, revised and supplemented using the boundary value method and expert opinions. Next, the hierarchical analysis method was utilized to determine the weights of the indicators and ultimately establish a scientific literacy indicator system for medical staff.

Following expert opinion, the index system for the scientific literacy of medical staff featuring 2 first-level indicators, 9 second-level indicators, and 38 third-level indicators was ultimately established, and the weights of the indicators were calculated. The two first-level indicators were research literacy and research ability, and the second-level indicators were research attitude (0.375), ability to identify problems (0.2038), basic literacy (0.1250), ability to implement projects (0.0843), research output capacity (0.0747), professional capacity (0.0735), data-processing capacity (0.0239), thesis-writing skills (0.0217), and ability to use literature (0.0181).

Conclusions

This study constructed a comprehensive scientific literacy index system that can assess medical staff's scientific literacy and serve as a reference for evaluating and improving their scientific literacy.

Peer Review reports

Due to the accelerated aging of the population and the growing global demand for healthcare in the wake of epidemics, there is an urgent need for medicine to provide greater support and protection. Medical scientific research is a critical element in promoting medical science and technological innovation, as well as improving clinical diagnosis and treatment techniques. It is the main driving force for the development of healthcare [ 1 ].

Medical personnel are highly compatible with clinical research. Due to their close interaction with patients, medical staff are better equipped to identify pertinent clinical research issues and actually implement clinical research projects [ 2 ]. Countries have created favorable conditions for the research and development of medical personnel by providing financial support, developing policies, and offering training courses [ 3 , 4 ]. However, some clinical studies have shown that the ability of most medical staff does not match current health needs and cannot meet the challenges posed by the twenty-first century [ 5 ]. It is clear that highly skilled professionals with scientific literacy are essential for national and social development [ 6 ]. Given the importance of scientific research in countries and hospitals, it is crucial to determine the level of scientific research literacy that medical personnel should possess and how to train them to acquire the necessary scientific research skills. These issues have significant practical implications.

Scientific literacy refers to an individual's ability to engage in science-related activities [ 7 ]. Some scholars suggest that the scientific literacy of medical personnel encompasses the fundamental qualities required for scientific research work, encompassing three facets: academic moral accomplishment, scientific research theory accomplishment, and scientific research ability accomplishment [ 8 ]. The existing research has focused primarily on the research capabilities of medical staff. According to Rillero, problem-solving skills, critical thinking, communication skills, and the ability to interpret data are the four core components of scientific literacy [ 9 ]. The ability to perform scientific research in nursing encompasses a range of abilities, including identifying problems, conducting literature reviews, designing and conducting scientific research, practicing scientific research, processing data, and writing papers [ 10 ]. Moule and Goodman proposed a framework of skills that research-literate nurses should possess, such as critical thinking capacity, analytical skills, searching skills, research critique skills, the ability to read and critically appraise research, and an awareness of ethical issues [ 11 ]. Several researchers have developed self-evaluation questionnaires to assess young researchers' scientific research and innovative abilities in the context of university-affiliated hospitals (UHAs) [ 12 ]. The relevant indicators include sensitivity to problems, sensitivity to cutting-edge knowledge, critical thinking, and other aspects. While these indicators cover many factors, they do not consider the issue of scientific research integrity in the medical field. The lack of detailed and targeted indicators, such as clinical resource collection ability and interdisciplinary cooperation ability, hinders the effective measurement of the current status of scientific literacy among medical staff [ 12 ]. In conclusion, the current research on the evaluation indicators of scientific literacy among medical personnel is incomplete, overlooking crucial humanistic characteristics, attitudes, and other moral literacy factors. Therefore, there is an urgent need to establish a comprehensive and systematic evaluation index to effectively assess the scientific literacy of medical staff.

Therefore, this study utilized a literature search and nominal group technique to screen the initial evaluation index and subsequently constructed an evaluation index system for medical staff's scientific research literacy utilizing the Delphi method. This index system would serve as a valuable tool for hospital managers, aiding them in the selection, evaluation, and training of scientific research talent. Additionally, this approach would enable medical personnel to identify their own areas of weakness and implement targeted improvement strategies.

Patient and public involvement

Patients and the public were not involved in this research.

Study design and participants

In this study, an initial evaluation index system was developed through a literature review and nominal group technique. Subsequently, a more comprehensive and scientific index system was constructed by combining qualitative and quantitative analysis utilizing the Delphi method to consult with experts. Finally, the hierarchical analysis method and the percentage weight method were employed to empower the index system.

The program used for this study is shown in Fig.  1 .

Study design. AHP, analytic hierarchy process

Establishing the preliminary indicator pool

Search process.

A literature search was performed in the China National Knowledge Infrastructure (CNKI), WanFang, PubMed, Web of Science and Scopus databases to collect the initial evaluation indicators. The time span ranged from the establishment of the database to July 2022. We used a combination of several MeSH terms in our searches:(("Medical Staff"[Mesh] OR "Nurses"[Mesh] OR "Physicians"[Mesh])) AND (("Literacy"[Mesh]) OR "Aptitude"[Mesh]). We also used several Title/Abstract searches, including keywords such as: Evaluation, scientific literacy, research ability.

The inclusion criteria were as follows: (1)The subjects were nurses, medicial staff and other personnel engaged in the medical industry; (2) Explore topics related to scientific literacy, such as research ability, and literature that can clarify the structure or dependency between indicators of scientific literacy; (3) Select articles published in countries such as China, the United States, the United Kingdom, Australia and Canada; (4) Research published in English or Chinese is considered to be eligible. The exclusion criteria are as follows: (1) indicators not applicable to medical staff; (2) Conference abstracts, case reports or review papers; (3) Articles with repeated descriptions; (4) There are no full-text articles or grey literature. A total of 78 articles were retrieved and 60 were retained after screening according to inclusion and exclusion criteria.

The research was conducted by two graduate students and two undergraduate students who participated in the literature search and screening. The entire research process was supervised and guided by one professor. All five members were from the fields of social medicine and health management. The professor was engaged in hospital management and health policy research for many years.

Nominal group technique

The nominal group technique was introduced at Hospital H in Beijing in July 2022. This hospital, with over 2,500 beds and 3,000 doctors, is a leading comprehensive medical center also known for its educational and research achievements, including numerous national research projects and awards.

The interview questions were based on the research question: What research literacy should medical staff have? 16 clinicians and nurses from Hospital H were divided into 2 equal groups and asked to provide their opinions on important aspects of research literacy based on their positions and experiences. Once all participants had shared their thoughts, similar responses were merged and polished. If anyone had further inputs after this, a second round of interviews was held until no new inputs were given. The entire meeting, including both rounds, was documented by researchers with audio recordings on a tape recorder.

Scientific literacy dimensions

Based on the search process, the research group extracted 58 tertiary indicators. To ensure the practicality and comprehensiveness of the indicators, the Nominal group technique was used on the basis of the literature search. Panelists summarized the entries shown in the interviews and merged similar content to obtain 32 third-level indicators. The indicators obtained from the literature search were compared. Several indicators with similar meanings, such as capture information ability, language expression ability, communication ability, and scientific research integrity, were merged. Additionally, the indicators obtained from the literature search, such as scientific research ethics, database use ability, feasibility and analysis ability, were added to the 15 indicators. A total of 47 third-level indicators were identified.

Fengling Dai and colleagues developed an innovation ability index system with six dimensions covering problem discovery, information retrieval, research design, practice, data analysis, and report writing, which represents the whole of innovative activity. Additionally, the system includes an innovation spirit index focusing on motivation, thinking, emotion, and will, reflecting the core of the innovation process in terms of competence [ 13 ]. Liao et al. evaluated the following five dimensions in their study on scientific research competence: literature processing, experimental manipulation, statistical analysis, manuscript production, and innovative project design [ 14 ]. Mohan claimed that scientific literacy consists of four core components: problem solving, critical thinking, communication skills, and the ability to interpret data [ 15 ].

This study structured scientific literacy into 2 primary indicators (research literacy and research competence) and 9 secondary indicators (basic qualifications, research ethics, research attitude, problem identification, literature use, professional capacity, subject implementation, data processing, thesis writing, and research output).

Using the Delphi method to develop an index system

Expert selection.

This study used the Delphi method to distribute expert consultation questionnaires online, allowing experts to exchange opinions anonymously to ensure that the findings were more desirable and scientific. No fixed number of experts is required for a Delphi study, but the more experts involved, the more stable the results will be [ 16 ]; this method generally includes 15 to 50 experts [ 17 ]. We selected clinicians from several tertiary hospitals in the Beijing area to serve as Delphi study consultants based on the following inclusion criteria: (1) they had a title of senior associate or above; (2) they had more than 10 years of work experience in the field of clinical scientific research, and (3) they were presiding over national scientific research projects. The exclusion criteria were as follows: (1) full-time scientific researchers, and (2) personnel in hospitals who were engaged only in management. To ensure that the selected experts were representative, this study selected 20 experts from 14 tertiary hospitals affiliated with Capital Medical University, Peking University, the Chinese Academy of Medical Sciences and the China Academy of Traditional Chinese Medicine according to the inclusion criteria; the hospitals featured an average of 1,231 beds each, and 9 hospitals were included among the 77 hospitals in the domestic comprehensive hospital ranking (Fudan Hospital Management Institute ranking). The experts represented various specialties and roles from different hospitals, including cardiology, neurosurgery, neurology, ear and throat surgery, head and neck surgery, radiology, imaging, infection, vascular interventional oncology, pediatrics, general practice, hematology, stomatology, nephrology, urology, and other related fields. This diverse group included physicians, nurses, managers, and vice presidents. The selected experts had extensive clinical experience, achieved numerous scientific research accomplishments and possessed profound knowledge and experience in clinical scientific research. This ensured the reliability of the consultation outcomes.

Design of the expert consultation questionnaire

The Delphi survey for experts included sections on their background, familiarity with the indicator system, system evaluation, and opinions. Experts rated indicators on importance, feasibility, and sensitivity using a 1–10 scale and their own familiarity with the indicators on a 1–5 scale. They also scored their judgment basis and impact on a 1–3 scale, considering theoretical analysis, work experience, peer understanding, and intuition. Two rounds of Delphi surveys were carried out via email with 20 experts to evaluate and suggest changes to the indicators. Statistical coefficients were calculated to validate the Delphi process. Feedback from the first round led to modifications and the inclusion of an AHP questionnaire for the second round. After the second round, indicators deemed less important were removed, and expert discussion finalized the indicator weights based on their relative importance scores. This resulted in the development of an index system for medical staff scientific literacy. The questionnaire is included in Additional file 1 (first round) and Additional file 2 (second round).

Using the boundary value method to screen the indicators

In this study, the boundary value method was utilized to screen the indicators of medical staff's scientific literacy, and the importance, feasibility, and sensitivity of each indicator were measured using the frequency of perfect scores, the arithmetic mean, and the coefficient of variation, respectively. When calculating the frequency of perfect scores and arithmetic means, the boundary value was set as "mean-SD," and indicators with scores higher than this value were retained. When calculating the coefficient of variation, the cutoff value was set to "mean + SD," and indicators with values below this threshold were retained.

The principles of indicator screening are as follows:

To evaluate the importance of the indicators, if none of the boundary values of the three statistics met the requirements, the indicators were deleted.

If an indicator has two aspects, importance, feasibility, or sensitivity, and each aspect has two or more boundary values that do not meet the requirements, then the indicator is deleted.

If all three boundary values for an indicator meet the requirements, the research group discusses the modification feedback from the experts and determines whether the indicator should be used.

The results of the two rounds of boundary values are shown in Table  1 .

Using the AHP to assign weights

After the second round of Delphi expert consultations, the analytic hierarchy process (AHP) was used to determine the weights of the two first-level indicators and the nine second-level indicators. The weights of the 37 third-level indicators were subsequently calculated via the percentage weight method. The AHP, developed by Saaty in the 1980s, is used to determine the priority and importance of elements constituting the decision-making hierarchy. It is based on multicriteria decision-making (MCDM) and determines the importance of decision-makers' judgments based on weights derived from pairwise comparisons between elements. In the AHP, pairwise comparisons are based on a comparative evaluation in which each element's weight in the lower tier is compared with that of other lower elements based on the element in the upper tier [ 18 ].

AHP analysis involves the following steps:

Step 1: Establish a final goal and list related elements to construct a hierarchy based on interrelated criteria.

Step 2: Perform a pairwise comparison for each layer to compare the weights of each element. Using a score from 1 to 9, which is the basic scale of the AHP, each pair is compared according to the expert’s judgment, and the importance is judged [ 19 , 20 ].

Yaahp software was employed to analyze data by creating a judgment matrix based on the experts' scores and hierarchical model. The index system weights were obtained by combining the experts' scores. The percentage weight method used experts' importance ratings from the second round to calculate weights, ranking indicators by importance, calculating their scores based on frequency of ranking, and determining weighting coefficients by dividing these scores by the total of all third-level indicators' scores. The third-level indicator weighting coefficients were then calculated by multiplying the coefficients [ 21 ].

Data analysis

Expert positivity coefficient.

The expert positivity coefficient is indicated by the effective recovery rate of the expert consultation questionnaire, which represents the level of expert positivity toward this consultation and determines the credibility and scientific validity of the questionnaire results. Generally, a questionnaire with an effective recovery rate of 70% is considered very good [ 22 ].

In this study, 20 questionnaires were distributed in both rounds of Delphi expert counseling, and all 20 were effectively recovered, resulting in a 100% effective recovery rate. Consequently, the experts provided positive feedback on the Delphi counseling.

Expert authority coefficient (CR)

The expert authority coefficient (Cr) is the arithmetic mean of the judgment coefficient (Ca) and the familiarity coefficient (Cs), namely, Cr =  \(\frac{({\text{Ca}}+{\text{Cs}})}{2}\) . The higher the degree of expert authority is, the greater the predictive accuracy of the indicator. A Cr ≥ 0.70 was considered to indicate an acceptable level of confidence [ 23 ]. Ca represents the basis on which the expert makes a judgment about the scenario in question, while Cs represents the expert's familiarity with the relevant problem [ 24 ].

Ca is calculated on the basis of experts' judgments of each indicator and the magnitude of its influence. In this study, experts used "practical experience (0.4), "theoretical analysis (0.3), "domestic and foreign peers (0.2)" and "intuition (0.1)" as the basis for judgment and assigned points according to the influence of each basis for judgment on the experts' judgment. Ca = 1 when the basis for judgment has a large influence on the experts, and Ca = 0.5 when the influence of the experts' judgment is at a medium level. When no influence on expert judgment was evident, Ca = 0 [ 25 ] (Table  2 ).

Cs refers to the degree to which the expert was familiar with the question. This study used the Likert scale method to score experts’ familiarity with the question on a scale ranging from 0 to 1 (1 = very familiar, 0.75 = more familiar, 0.5 = moderately familiar, 0.25 = less familiar, 0 = unfamiliar). The familiarity coefficient for each expert (the average familiarity for each indicator) was calculated. The average familiarity coefficient was subsequently computed [ 26 ].

The Cr value of the primary indicator in this study was 0.83, and the Cr value of the secondary indicator was 0.82 (> 0.7); hence, the results of the expert consultation were credible and accurate, as shown in Table  3 .

The degree of expert coordination is an important indicator used to judge the consistency among various experts regarding indicator scores. This study used the Kendall W coordination coefficient test to determine the degree of expert coordination. A higher Kendall W coefficient indicates a greater degree of expert coordination and greater consistency in expert opinion, and P  <  0.05 indicates that the difference is significant [ 26 ]. The results of the three-dimensional harmonization coefficient test for each indicator in the two rounds of the expert consultation questionnaire were valid ( p  <  0.01 ), emphasizing the consistency of the experts' scores. The values of the Kendall W coordination coefficients for both rounds are shown in Table  4 .

Basic information regarding the participants

The 20 Delphi experts who participated in this study were predominantly male (80.0%) rather than female (20.0%). In addition, the participants’ ages were mainly concentrated in the range of 41–50 years old (60.0%). The majority of the experts were doctors by profession (85.0%), and their education and titles were mainly doctoral degree (90.0%) and full senior level (17.0%). The experts also exhibited high academic achievement in their respective fields and had many years of working experience, with the majority having between 21 and 25 years of experience (40.0%) (Table  5 ).

Index screening

The boundary value method was applied to eliminate indicators, leading to the removal of 6 third-level indicators in the first round. One of these, the ability to use statistical software, was associated with a more significant second-level indicator involving data processing, which was kept after expert review. Six indicators were merged into three indicators due to duplication, and 5 third-level indicators were added, resulting in 2 primary indicators, 10 secondary indicators, and 43 third-level indicators.

In the second round of Delphi expert consultation, 5 third-level indicators were deleted, as shown in Additional file 3 , and only one third-level indicator, "Scientific spirit", remained under the secondary indicator "research attitude". The secondary indicator "Research attitude" was combined with "Research ethics" and the third-level indicator "Scientific spirit" was also considered part of "Research ethics". After expert discussion, these were merged into a new secondary indicator "Research attitude" with three third-level indicators: "Research ethics", "Research integrity", and "Scientific spirit". The final index system included two primary indicators, nine secondary indicators, and thirty-eight third-level indicators, as shown in Additional File 3 .

Final index system with weights

The weights of the two primary indexes, research literacy and research ability, were equal. This was determined using the hierarchical analysis method and the percentage weight method based on the results of the second round of Delphi expert consultation (Table  6 ). The primary indicator of research literacy encompasses the fundamental qualities and attitudes medical staff develop over time, including basic qualifications and approach to research. The primary indicator of research ability refers to medical professionals' capacity to conduct scientific research in new areas using suitable methods, as well as their skills needed for successful research using scientific methods.

In this study, the Delphi method was employed, and after two rounds of expert consultation, in accordance with the characteristics and scientific research requirements of medical staff in China, an index system for the scientific literacy of medical staff in China was constructed. The index system for medical staff's scientific literacy in this study consists of 2 first-level indicators, 9 second-level indicators, and 38 third-level indicators. Medical institutions at all levels can use this index system to scientifically assess medical staff's scientific literacy.

In 2014, the Joint Task Force for Clinical Trial Competency (JTF) published its Core Competency Framework [ 27 ]. The Framework focuses more on the capacity to conduct clinical research. These include principles such as clinical research and quality practices for drug clinical trials. However, this framework does not apply to the current evaluation of scientific literacy in hospitals. Because these indicators do not apply to all staff members, there is a lack of practical scientific research, such as information about the final paper output. Therefore, the experts who constructed the index system in this study came from different specialties, and the indicators can be better applied to scientific researchers in all fields. This approach not only addresses clinical researchers but also addresses the concerns of hospital managers, and the indicators are more applicable.

The weighted analysis showed that the primary indicators "research literacy" and "research ability" had the same weight (0.50) and were two important components of scientific literacy. Research ability is a direct reflection of scientific literacy and includes the ability to identify problems, the ability to use literature, professional capacity, subject implementation capacity, data-processing capacity, thesis-writing skills, and research output capacity. Only by mastering these skills can medical staff carry out scientific research activities more efficiently and smoothly. The ability to identify problems refers to the ability of medical staff to obtain insights into the frontiers of their discipline and to identify and ask insightful questions. Ratten claimed that only with keen insight and sufficient sensitivity to major scientific issues can we exploit the opportunities for innovation that may lead to breakthroughs [ 28 ]. Therefore, it is suggested that in the process of cultivating the scientific literacy of medical staff, the ability to identify problems, including divergent thinking, innovative sensitivity, and the ability to produce various solutions, should be improved. Furthermore, this study included three subentries of the secondary indicator "research attitude", namely, research ethics, research integrity, and scientific spirit. This is likely because improper scientific research behavior is still prevalent. A study conducted in the United States and Europe showed that the rate of scientific research misconduct was 2% [ 13 ]. A small survey conducted in Indian medical schools and hospitals revealed that 57% of the respondents knew that someone had modified or fabricated data for publication [ 28 ]. The weight of this index ranked first in the secondary indicators, indicating that scientific attitude is an important condition for improving research quality, relevance, and reliability. Countries and hospitals should develop, implement, and optimize policies and disciplinary measures to combat academic misconduct.

In addition, the third-level indicator "scheduling ability" under the second-level indicator "basic qualification" has a high weight, indicating that medical staff attach importance to management and distribution ability in the context of scientific research. Currently, hospitals face several problems, such as a shortage of medical personnel, excessive workload, and an increase in the number of management-related documents [ 29 , 30 ]. These factors result in time conflicts between daily responsibilities and scientific research tasks, thereby presenting significant obstacles to the allocation of sufficient time for scientific inquiry [ 31 ]. Effectively arranging clinical work and scientific research time is crucial to improving the overall efficiency of scientific research. In the earlier expert interviews, most medical staff believed that scientific research work must be combined with clinical work rather than focused only on scientific research. Having the ability to make overall arrangements is essential to solving these problems. The high weight given to the second-level index of 'subject implementation capacity', along with its associated third-level indicators, highlights the challenges faced by young medical staff in obtaining research subjects. Before implementing a project, researchers must thoroughly investigate, analyze, and compare various aspects of the research project, including its technical, economic, and engineering aspects. Moreover, potential financial and economic benefits, as well as social impacts, need to be predicted to determine the feasibility of the project and develop a research plan [ 32 ]. However, for most young medical staff in medical institutions, executing such a project can be challenging due to their limited scientific research experience [ 33 ]. A researcher who possesses these skills can truly carry out independent scientific research.

The weights of the second-level index "research output capacity" cannot be ignored. In Chinese hospitals, the ability to produce scientific research output plays a certain role in employees’ ability to obtain rewards such as high pay, and this ability is also used as a reference for performance appraisals [ 34 ]. The general scientific research performance evaluation includes the number of projects, scientific papers and monographs, scientific and technological achievements, and patents. In particular, the publication of papers is viewed as an indispensable aspect of performance appraisal by Chinese hospitals [ 35 ]. Specifically, scientific research papers are the carriers of scientific research achievements and academic research and thus constitute an important symbol of the level of medical development exhibited by medical research institutions; they are thus used as recognized and important indicators of scientific research output [ 36 ]. This situation is consistent with the weight evaluation results revealed by this study.

The results of this study are important for the training and management of the scientific research ability of medical personnel. First, the index system focuses not only on external characteristics such as scientific knowledge and skills but also on internal characteristics such as individual traits, motivation, and attitudes. Therefore, when building a research team and selecting and employing researchers, hospital managers can use the index system to comprehensively and systematically evaluate the situation of researchers, which is helpful for optimizing the allocation of a research team, learning from each other's strengths, and strengthening the strength of the research team. Second, this study integrates the content of existing research to obtain useful information through in-depth interviews with medical staff and constructs an evaluation index system based on Delphi expert consultation science, which comprehensively includes the evaluation of the whole process of scientific research activities. These findings can provide a basis for medical institutions to formulate scientific research training programs, help medical personnel master and improve scientific research knowledge and skills, and improve their working ability and quality. Moreover, the effectiveness of the training can also be evaluated according to the system.

In China, with the emergence of STEM rankings, hospitals pay more and more attention to the scientific research performance of medical personnel. Scientific literacy not only covers the abilities of medical personnel engaged in scientific research, but also reflects their professional quality in this field. Having high quality medical personnel often means that they have excellent scientific research ability, and their scientific research performance will naturally rise. In view of this,,medical institutions can define the meaning of third-level indicators and create Likert scales to survey medical staff. Based on the weights assigned to each indicator, comprehensive scores can be calculated to evaluate the level of scientific literacy among medical staff. Through detailed data analysis, they can not only reveal their shortcomings in scientific research ability and quality, but also provide a strong basis for subsequent training and promotion. Through targeted inspection, we can not only promote the comprehensive improvement of the ability of medical staff, but also promote the steady improvement of their scientific research performance, and inject new vitality into the scientific research cause of hospitals.

Limitations

This study has several limitations that need to be considered. First, the participants were only recruited from Beijing (a city in China), potentially lacking geographical diversity. We plan to select more outstanding experts from across the country to participate. Second, the index system may be more suitable for countries with medical systems similar to those of China. When applying this system in other countries, some modifications may be necessary based on the local context. Last, While this study has employed scientific methods to establish the indicator system, the index system has yet to be implemented on a large sample of medical staff. Therefore, the reliability and validity of the index system must be confirmed through further research. In conclusion, it is crucial to conduct further detailed exploration of the effectiveness and practical application of the index system in the future.

This study developed an evaluation index system using the Delphi method to assess the scientific literacy of medical staff in China. The system comprises two primary indicators, nine secondary indicators, and thirty-eight third-level indicators, with each index assigned a specific weight. The index system emphasizes the importance of both attitudes and abilities in the scientific research process for medical staff and incorporates more comprehensive evaluation indicators. In the current era of medical innovation, enhancing the scientific literacy of medical staff is crucial for enhancing the competitiveness of individuals, hospitals, and overall medical services in society. This evaluation index system is universally applicable and beneficial for countries with healthcare systems similar to those of China. This study can serve as a valuable reference for cultivating highly qualified and capable research personnel and enhancing the competitiveness of medical research.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Coloma J, Harris E. From construction workers to architects: developing scientific research capacity in low-income countries. PLoS Biol. 2009;7(7):e1000156. https://doi.org/10.1371/journal.pbio.1000156 .

Article   Google Scholar  

Brauer SG, Haines TP, Bew PG. Fostering clinician-led research. Aust J Physiother. 2007;53(3):143–4. https://doi.org/10.1016/s0004-9514(07)70020-x .

The L. China’s research renaissance. Lancet. 2019;393(10179):1385. https://doi.org/10.1016/S0140-6736(19)30797-4 .

Hannay DR. Evaluation of a primary care research network in rural Scotland. Prim Health Care ResDevelop. 2006;7(3):194–200. https://doi.org/10.1191/1463423606pc296oa .

Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–58.

Xie Y, Wang J, Li S, Zheng Y. Research on the Influence Path of Metacognitive Reading Strategies on Scientific Literacy. J Intell. 2023;11(5):78. https://doi.org/10.3390/jintelligence11050078 . PMID: 37233327; PMCID: PMC10218841.

Pang YH, Cheng JL. Revise of scientific research ability self-evaluation rating scales of nursing staff. Chin Nurs Res. 2011;13:1205–8. https://doi.org/10.3969/j.issn.1009-6493.2011.13.040 .

Zhang J, Jianshan MAO, Gu Y. On the cultivation of scientific research literacy of medical graduate students. Continu Med Educ China. 2023;15(3):179–82. https://doi.org/10.3969/j.issn.1674-9308.2023.03.043 .

Rillero P. Process skills and content knowledge. Sci Act. 1998;3:3–4.

Google Scholar  

Liu RS. Study on reliability and validity of self rating scale for scientific research ability of nursing staff. Chinese J Pract Nurs. 2004;9:8–10. https://doi.org/10.3760/cma.j.issn.1672-7088.2004.09.005 .

Moule P, Goodman M. Nursing research: An introduction. London, UK: Sage; 2013.

Xue J, Chen X, Zhang Z, et al. Survey on status quo and development needs of research and innovation capabilities of young researchers at university-affiliated hospitals in China: a cross-sectional survey. Ann Transl Med. 2022;10(18):964. https://doi.org/10.21037/atm-22-3692 .

Fanelli D, Costas R, Fang FC, et al. Testing hypotheses on risk factors for scientific misconduct via matched-control analysis of papers containing problematic image duplications. Sci Eng Ethics. 2019;25(3):771–89. https://doi.org/10.1007/s11948-018-0023-7 .

Liao Y, Zhou H, Wang F, et al. The Impact of Undergraduate Tutor System in Chinese 8-Year Medical Students in Scientific Research. Front Med (Lausanne). 2022;9:854132. https://doi.org/10.3389/fmed.2022.854132 .

Mohan L, Singh Y, Kathrotia R, et al. Scientific literacy and the medical student: A cross-sectional study. Natl Med J India. 2020;33(1):35–7. https://doi.org/10.4103/0970-258X.308242 .

Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49(10):887–97. https://doi.org/10.1177/0004867415600891 .

Xinran S, Heping W, Yule H, et al. Defining the scope and weights of services of a family doctor service project for the functional community using Delphi technique and analytic hierarchy process. Chinese Gen Pract. 2021;24(34):4386–91.

Park S, Kim HK, Lee M. An analytic hierarchy process analysis for reinforcing doctor-patient communication. BMC Prim Care. 2023;24(1):24. https://doi.org/10.1186/s12875-023-01972-3 . Published 2023 Jan 21.

Zhou MLY, Yin H, et al. New screening tool for neonatal nutritional risk in China: a validation study. BMJ Open. 2021;11(4):e042467. https://doi.org/10.1136/bmjopen-2020-042467 .

Wang K, Wang Z, Deng J, et al. Study on the evaluation of emergency management capacity of resilient communities by the AHP-TOPSIS method. Int J Environ Res Public Health. 2022;19(23):16201. https://doi.org/10.3390/ijerph192316201 .

Yuwei Z, Chuanhui Y, Junlong Z, et al. Application of analytic Hierarchy Process and percentage weight method to determine the weight of traditional Chinese medicine appropriate technology evaluation index system. Chin J Tradit Chinese Med. 2017;32(07):3054–6.

Babbie E. The practice of social research. 10th Chinese language edition. Huaxia Publisher, 2005: 253–4.

Liu W, Hu M, Chen W. Identifying the Service Capability of Long-Term Care Facilities in China: an e-Delphi study. Front Public Health. 2022;10:884514. https://doi.org/10.3389/fpubh.2022.884514 .

Zeng G. Modern epidemiological methods and application. Pecking Union Medical College Union Press, 1996.

Geng Y, Zhao L, Wang Y, et al. Competency model for dentists in China: Results of a Delphi study. PLoS One. 2018;13(3):e0194411. https://doi.org/10.1371/journal.pone.0194411 .

Cong C, Liu Y, Wang R. Kendall coordination coefficient W test and its SPSS implementation. Journal of Taishan Medical College. 2010;31(7):487–490. https://doi.org/10.3969/j.issn.1004-7115.2010.07.002 .

Sonstein S, Seltzer J, Li R, et al. Moving from compliance to competency: a harmonized core competency framework for the clinical research professional. Clin Res. 2014;28(3):17–23.

Madan C, Kruger E, Tennant M. 30 Years of dental research in Australia and India: a comparative analysis of published peer review literature. Indian J Dent Res. 2012;23(2):293–4. https://doi.org/10.4103/0970-9290.100447 .

Siemens DR, Punnen S, Wong J, Kanji N. A survey on the attitudes towards research in medical school. BMC Med Educ. 2010;10:4. https://doi.org/10.1186/1472-6920-10-4 .

Solomon SS, Tom SC, Pichert J, Wasserman D, Powers AC. Impact of medical student research in the development of physician-scientists. J Investig Med. 2003;51(3):149–56. https://doi.org/10.1136/jim-51-03-17 .

Misztal-Okonska P, Goniewicz K, Hertelendy AJ, et al. How Medical Studies in Poland Prepare Future Healthcare Managers for Crises and Disasters: Results of a Pilot Study. Healthcare (Basel). 2020;8(3):202. https://doi.org/10.3390/healthcare8030202 .

Xu G. On the declaration of educational scientific research topics. Journal of Henan Radio & TV University. 2013;26(01):98–101.

Ju Y, Zhao X. Top three hospitals clinical nurse scientific research ability present situation and influence factors analysis. J Health Vocational Educ. 2022;40(17):125–8.

Zhu Q, Li T, Li X, et al. Industry gain public hospital medical staff performance distribution mode of integration, exploring. J Health Econ Res. 2022;33(11):6-82–6.

Sun YLL. Analysis of hospital papers published based on performance appraisal. China Contemp Med. 2015;22(31):161–3.

Jian Y, Wu J, Liu Y. Citation analysis of seven tertiary hospitals in Yunnan province from 2008 to 2012. Yunnan Medicine. 2014;(6):700–704.

Download references

Acknowledgements

The authors thank all who participated in the nominal group technique and two rounds of the Delphi study.

This study was supported by the National Natural Science Foundation of China (72074160) and the Natural Science Foundation Project of Beijing (9222004).

Author information

Shuyu Liang and Ziyan Zhai contributed equally to this work and joint first authors.

Kai Meng and Yuan Gao contributed equally to this work and share corresponding author.

Authors and Affiliations

Aerospace Center Hospital, No. 15 Yuquan Road, Haidian District, Beijing, 100049, China

Xiaozhi Sun, Jingxuan Jiao & Yuan Gao

School of Public Health, Capital Medical University, No.10 Xitoutiao, Youanmenwai Street, Fengtai District, Beijing, 100069, China

Shuyu Liang, Ziyan Zhai, Xingmiao Feng & Kai Meng

Beijing Tiantan Hospital, Capital Medical University, No. 119 South Fourth Ring West Road, Fengtai District, Beijing, 100070, China

You can also search for this author in PubMed   Google Scholar

Contributions

S.L. and Z.Z. contributed equally to this paper. S.L. took charge of the nominal group technique, data analysis, writing the first draft and revising the manuscript; Z.Z. was responsible for the Delphi survey, data analysis, and writing of the first draft of the manuscript; XF was responsible for the rigorous revision of Delphi methods; X.S. and J.J. were responsible for the questionnaire survey and data collection; Y.G. contributed to the questionnaire survey, organization of the nominal group interview, supervision, project administration and resources; and K.M. contributed to conceptualization, methodology, writing—review; editing, supervision, and project administration. All the authors read and approved the final manuscript.

Corresponding authors

Correspondence to Yuan Gao or Kai Meng .

Ethics declarations

Ethics approval and consent to participate.

This study involved human participants and was approved by the Ethical Review Committee of the Capital Medical University (No. Z2022SY089). Participation in the survey was completely voluntary, and written informed consent was obtained from the participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Liang, S., Zhai, Z., Feng, X. et al. Development of an index system for the scientific literacy of medical staff: a modified Delphi study in China. BMC Med Educ 24 , 397 (2024). https://doi.org/10.1186/s12909-024-05350-0

Download citation

Received : 25 October 2023

Accepted : 26 March 2024

Published : 10 April 2024

DOI : https://doi.org/10.1186/s12909-024-05350-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical staff
• Scientific literacy
• Evaluation indicators

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]