systematic literature review quantitative or qualitative

About Systematic Reviews

Are Systematic Reviews Qualitative or Quantitative?

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

A systematic review is designed to be transparent and replicable. Therefore, systematic reviews are considered reliable tools in scientific research and clinical practice. They synthesize the results using multiple primary studies by using strategies that minimize bias and random errors. Depending on the research question and the objectives of the research, the reviews can either be qualitative or quantitative. Qualitative reviews deal with understanding concepts, thoughts, or experiences. Quantitative reviews are employed when researchers want to test or confirm a hypothesis or theory. Let’s look at some of the differences between these two types of reviews.

To learn more about how long it takes to do a systematic review , you can check out the link to our full article on the topic.

Differences between Qualitative and Quantitative Reviews

The differences lie in the scope of the research, the methodology followed, and the type of questions they attempt to answer. Some of these differences include:

Research Questions

As mentioned earlier qualitative reviews attempt to answer open-ended research questions to understand or formulate hypotheses. This type of research is used to gather in-depth insights into new topics. Quantitative reviews, on the other hand, test or confirm existing hypotheses. This type of research is used to establish generalizable facts about a topic.

Type of Sample Data

The data collected for both types of research differ significantly. For qualitative research, data is collected as words using observations, interviews, and interactions with study subjects or from literature reviews. Quantitative studies collect data as numbers, usually from a larger sample size.

Data Collection Methods

To collect data as words for a qualitative study, researchers can employ tools such as interviews, recorded observations, focused groups, videos, or by collecting literature reviews on the same subject. For quantitative studies, data from primary sources is collected as numbers using rating scales and counting frequencies. The data for these studies can also be collected as measurements of variables from a well-designed experiment carried out under pre-defined, monitored conditions.

Data Analysis Methods

Data by itself cannot prove or demonstrate anything unless it is analyzed. Qualitative data is more challenging to analyze than quantitative data. A few different approaches to analyzing qualitative data include content analysis, thematic analysis, and discourse analysis. The goal of all of these approaches is to carefully analyze textual data to identify patterns, themes, and the meaning of words or phrases.

Quantitative data, since it is in the form of numbers, is analyzed using simple math or statistical methods. There are several software programs that can be used for mathematical and statistical analysis of numerical data.

Presentation of Results

Learn more about distillersr.

(Article continues below)

Final Takeaway – Qualitative or Quantitative?

3 reasons to connect.

• Quantitative vs. Qualitative Research

Research can be   quantitative or qualitative  or both:

• A quantitative systematic review will include studies that have numerical data.
• A qualitative systematic review derives data from observation, interviews, or verbal interactions and focuses on the meanings and interpretations of the participants. It may include focus groups, interviews, observations and diaries.

Video source:  UniversityNow: Quantitative vs. Qualitative Research

For more information on searching for qualitative evidence see:

Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: A structured methodological review.  Systematic Reviews, 5 (1), 1–23. https://doi.org/10.1186/S13643-016-0249-X/TABLES/5

• << Previous: Study Types & Terminology
• Next: Critical Appraisal of Studies >>
• Types of Questions
• Key Features and Limitations
• Is a Systematic Review Right for Your Research?
• Integrative Review
• Scoping Review
• Rapid Review
• Meta-Analysis/Meta-Synthesis
• Selecting a Review Type
• Reducing Bias
• Guidelines for Student Researchers
• Training Resources
• Register Your Protocol
• Handbooks & Manuals
• Reporting Guidelines
• PRESS 2015 Guidelines
• Search Strategies
• Selected Databases
• Grey Literature
• Handsearching
• Citation Searching
• Study Types & Terminology
• Critical Appraisal of Studies
• Broad Functionality Programs & Tools
• Search Strategy Tools
• Deduplication Tools
• CItation Screening
• Critical Appraisal Tools
• Quality Assessment/Risk of Bias Tools
• Data Collection/Extraction
• Meta Analysis Tools
• Books on Systematic Reviews
• Finding Systematic Review Articles in the Databases
• Systematic Review Journals
• More Resources
• Evidence-Based Practice Research in Nursing
• Citation Management Programs
• Last Updated: Apr 5, 2024 2:54 PM
• URL: https://libguides.adelphi.edu/Systematic_Reviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved April 9, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

• Open access
• Published: 15 December 2015

Qualitative and mixed methods in systematic reviews

• David Gough 1  

Systematic Reviews volume  4 , Article number:  181 ( 2015 ) Cite this article

22k Accesses

46 Citations

23 Altmetric

Metrics details

Expanding the range of methods of systematic review

The logic of systematic reviews is very simple. We use transparent rigorous approaches to undertake primary research, and so we should do the same in bringing together studies to describe what has been studied (a research map) or to integrate the findings of the different studies to answer a research question (a research synthesis). We should not really need to use the term ‘systematic’ as it should be assumed that researchers are using and reporting systematic methods in all of their research, whether primary or secondary. Despite the universality of this logic, systematic reviews (maps and syntheses) are much better known in health research and for answering questions of the effectiveness of interventions (what works). Systematic reviews addressing other sorts of questions have been around for many years, as in, for example, meta ethnography [ 1 ] and other forms of conceptual synthesis [ 2 ], but only recently has there been a major increase in the use of systematic review approaches to answer other sorts of research questions.

There are probably several reasons for this broadening of approach. One may be that the increased awareness of systematic reviews has made people consider the possibilities for all areas of research. A second related factor may be that more training and funding resources have become available and increased the capacity to undertake such varied review work.

A third reason could be that some of the initial anxieties about systematic reviews have subsided. Initially, there were concerns that their use was being promoted by a new managerialism where reviews, particularly effectiveness reviews, were being used to promote particular ideological and theoretical assumptions and to indirectly control research agendas. However, others like me believe that explicit methods should be used to enable transparency of perspectives driving research and to open up access to and participation in research agendas and priority setting [ 3 ] as illustrated, for example, by the James Lind Alliance (see http://www.jla.nihr.ac.uk/ ).

A fourth possible reason for the development of new approaches is that effectiveness reviews have themselves broadened. Some ‘what works’ reviews can be open to criticism for only testing a ‘black box’ hypothesis of what works with little theorizing or any logic model about why any such hypothesis should be true and the mechanisms involved in such processes. There is now more concern to develop theory and to test how variables combine and interact. In primary research, qualitative strategies are advised prior to undertaking experimental trials [ 4 , 5 ] and similar approaches are being advocated to address complexity in reviews [ 6 ], in order to ask questions and use methods that address theories and processes that enable an understanding of both impact and context.

This Special Issue of Systematic Reviews Journal is providing a focus for these new methods of review whether these use qualitative review methods on their own or mixed together with more quantitative approaches. We are linking together with the sister journal Trials for this Special Issue as there is a similar interest in what qualitative approaches can and should contribute to primary research using experimentally controlled trials (see Trials Special Issue editorial by Claire Snowdon).

Dimensions of difference in reviews

Developing the range of methods to address different questions for review creates a challenge in describing and understanding such methods. There are many names and brands for the new methods which may or may not withstand the changes of historical time, but another way to comprehend the changes and new developments is to consider the dimensions on which the approaches to review differ [ 7 , 8 ].

One important distinction is the research question being asked and the associated paradigm underlying the method used to address this question. Research assumes a particular theoretical position and then gathers data within this conceptual lens. In some cases, this is a very specific hypothesis that is then tested empirically, and sometimes, the research is more exploratory and iterative with concepts being emergent and constructed during the research process. This distinction is often labelled as quantitative or positivist versus qualitative or constructionist. However, this can be confusing as much research taking a ‘quantitative’ perspective does not have the necessary numeric data to analyse. Even if it does have such data, this might be explored for emergent properties. Similarly, research taking a ‘qualitative’ perspective may include implicit quantitative themes in terms of the extent of different qualitative findings reported by a study.

Sandelowski and colleagues’ solution is to consider the analytic activity and whether this aggregates (adds up) or configures (arranges) the data [ 9 ]. In a randomized controlled trial and an effectiveness review of such studies, the main analysis is the aggregation of data using a priori non-emergent strategies with little iteration. However, there may also be post hoc analysis that is more exploratory in arranging (configuring) data to identify patterns as in, for example, meta regression or qualitative comparative analysis aiming to identify the active ingredients of effective interventions [ 10 ]. Similarly, qualitative primary research or reviews of such research are predominantly exploring emergent patterns and developing concepts iteratively, yet there may be some aggregation of data to make statements of generalizations of extent.

Even where the analysis is predominantly configuration, there can be a wide variation in the dimensions of difference of iteration of theories and concepts. In thematic synthesis [ 11 ], there may be few presumptions about the concepts that will be configured. In meta ethnography which can be richer in theory, there may be theoretical assumptions underlying the review question framing the analysis. In framework synthesis, there is an explicit conceptual framework that is iteratively developed and changed through the review process [ 12 , 13 ].

In addition to the variation in question, degree of configuration, complexity of theory, and iteration are many other dimensions of difference between reviews. Some of these differences follow on from the research questions being asked and the research paradigm being used such as in the approach to searching (exhaustive or based on exploration or saturation) and the appraisal of the quality and relevance of included studies (based more on risk of bias or more on meaning). Others include the extent that reviews have a broad question, depth of analysis, and the extent of resultant ‘work done’ in terms of progressing a field of inquiry [ 7 , 8 ].

Mixed methods reviews

As one reason for the growth in qualitative synthesis is what they can add to quantitative reviews, it is not surprising that there is also growing interest in mixed methods reviews. This reflects similar developments in primary research in mixing methods to examine the relationship between theory and empirical data which is of course the cornerstone of much research. But, both primary and secondary mixed methods research also face similar challenges in examining complex questions at different levels of analysis and of combining research findings investigated in different ways and may be based on very different epistemological assumptions [ 14 , 15 ].

Some mixed methods approaches are convergent in that they integrate different data and methods of analysis together at the same time [ 16 , 17 ]. Convergent systematic reviews could be described as having broad inclusion criteria (or two or more different sets of criteria) for methods of primary studies and have special methods for the synthesis of the resultant variation in data. Other reviews (and also primary mixed methods studies) are sequences of sub-reviews in that one sub-study using one research paradigm is followed by another sub-study with a different research paradigm. In other words, a qualitative synthesis might be used to explore the findings of a prior quantitative synthesis or vice versa [ 16 , 17 ].

An example of a predominantly aggregative sub-review followed by a configuring sub-review is the EPPI-Centre’s mixed methods review of barriers to healthy eating [ 18 ]. A sub-review on the effectiveness of public health interventions showed a modest effect size. A configuring review of studies of children and young people’s understanding and views about eating provided evidence that the public health interventions did not take good account of such user views research, and that the interventions most closely aligned to the user views were the most effective. The already mentioned qualitative comparative analysis to identify the active ingredients within interventions leading to impact could also be considered a qualitative configuring investigation of an existing quantitative aggregative review [ 10 ].

An example of a predominantly configurative review followed by an aggregative review is realist synthesis. Realist reviews examine the evidence in support of mid-range theories [ 19 ] with a first stage of a configuring review of what is proposed by the theory or proposal (what would need to be in place and what casual pathways would have to be effective for the outcomes proposed by the theory to be supported?) and a second stage searching for empirical evidence to test for those necessary conditions and effectiveness of the pathways. The empirical testing does not however use a standard ‘what works’ a priori methods approach but rather a more iterative seeking out of evidence that confirms or undermines the theory being evaluated [ 20 ].

Although sequential mixed methods approaches are considered to be sub-parts of one larger study, they could be separate studies as part of a long-term strategic approach to studying an issue. We tend to see both primary studies and reviews as one-off events, yet reviews are a way of examining what we know and what more we want to know as a strategic approach to studying an issue over time. If we are in favour of mixing paradigms of research to enable multiple levels and perspectives and mixing of theory development and empirical evaluation, then we are really seeking mixed methods research strategies rather than simply mixed methods studies and reviews.

Noblit G. Hare RD: meta-ethnography: synthesizing qualitative studies. Newbury Park NY: Sage Publications; 1988.

Google Scholar  

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Article   PubMed   PubMed Central   Google Scholar  

Gough D, Elbourne D. Systematic research synthesis to inform policy, practice and democratic debate. Soc Pol Soc. 2002;2002:1.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance 2015. BMJ. 2015;350:h1258

Candy B, Jone L, King M, Oliver S. Using qualitative evidence to help understand complex palliative care interventions: a novel evidence synthesis approach. BMJ Support Palliat Care. 2014;4:Supp A41–A42.

Article   Google Scholar  

Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66:11.

Gough D, Oliver S, Thomas J. Introduction to systematic reviews. London: Sage; 2012.

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

Sandelowski M, Voils CJ, Leeman J, Crandlee JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6:4.

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Exp. 2008;11:72–84.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Qual Saf. 2015. 2014-003642.

Brannen J. Mixed methods research: a discussion paper. NCRM Methods Review Papers, 2006. NCRM/005.

Creswell J. Mapping the developing landscape of mixed methods research. In: Teddlie C, Tashakkori A, editors. SAGE handbook of mixed methods in social & behavioral research. New York: Sage; 2011.

Morse JM. Principles of mixed method and multi-method research design. In: Teddlie C, Tashakkori A, editors. Handbook of mixed methods in social and behavioural research. London: Sage; 2003.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.

Harden A, Thomas J. Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in the social and behavioral sciences. 2nd ed. London: Sage; 2010. p. 749–74.

Chapter   Google Scholar  

Pawson R. Evidenced-based policy: a realist perspective. London: Sage; 2006.

Book   Google Scholar  

Gough D. Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Med. 2013;11:22.

Download references

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, University College London, London, WC1H 0NR, UK

David Gough

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Gough .

Additional information

Competing interests.

The author is a writer and researcher in this area. The author declares that he has no other competing interests.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gough, D. Qualitative and mixed methods in systematic reviews. Syst Rev 4 , 181 (2015). https://doi.org/10.1186/s13643-015-0151-y

Download citation

Received : 13 October 2015

Accepted : 29 October 2015

Published : 15 December 2015

DOI : https://doi.org/10.1186/s13643-015-0151-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters & Hedges
• Grey Literature
• Managing & Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Apr 10, 2024 11:08 AM
• URL: https://guides.library.ucla.edu/systematicreviews

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Supplements
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 4, Issue Suppl 1
• Synthesising quantitative evidence in systematic reviews of complex health interventions
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Julian P T Higgins 1 ,
• José A López-López 1 ,
• Betsy J Becker 2 ,
• Sarah R Davies 1 ,
• Sarah Dawson 1 ,
• Jeremy M Grimshaw 3 , 4 ,
• Luke A McGuinness 1 ,
• Theresa H M Moore 1 , 5 ,
• Eva A Rehfuess 6 ,
• James Thomas 7 ,
• Deborah M Caldwell 1
• 1 Population Health Sciences , Bristol Medical School, University of Bristol , Bristol , UK
• 2 Department of Educational Psychology and Learning Systems, College of Education , Florida State University , Tallahassee , Florida , USA
• 3 Clinical Epidemiology Program , Ottawa Hospital Research Institute, The Ottawa Hospital , Ottawa , Ontario , Canada
• 4 Department of Medicine , University of Ottawa , Ottawa , Ontario , Canada
• 5 NIHR Collaboration for Leadership in Applied Health Care (CLAHRC) West , University Hospitals Bristol NHS Foundation Trust , Bristol , UK
• 6 Institute for Medical Information Processing , Biometry and Epidemiology, Pettenkofer School of Public Health, LMU Munich , Munich , Germany
• 7 EPPI-Centre, Department of Social Science , University College London , London , UK
• Correspondence to Professor Julian P T Higgins; julian.higgins{at}bristol.ac.uk

Public health and health service interventions are typically complex: they are multifaceted, with impacts at multiple levels and on multiple stakeholders. Systematic reviews evaluating the effects of complex health interventions can be challenging to conduct. This paper is part of a special series of papers considering these challenges particularly in the context of WHO guideline development. We outline established and innovative methods for synthesising quantitative evidence within a systematic review of a complex intervention, including considerations of the complexity of the system into which the intervention is introduced. We describe methods in three broad areas: non-quantitative approaches, including tabulation, narrative and graphical approaches; standard meta-analysis methods, including meta-regression to investigate study-level moderators of effect; and advanced synthesis methods, in which models allow exploration of intervention components, investigation of both moderators and mediators, examination of mechanisms, and exploration of complexities of the system. We offer guidance on the choice of approach that might be taken by people collating evidence in support of guideline development, and emphasise that the appropriate methods will depend on the purpose of the synthesis, the similarity of the studies included in the review, the level of detail available from the studies, the nature of the results reported in the studies, the expertise of the synthesis team and the resources available.

• meta-analysis
• complex interventions
• systematic reviews
• guideline development

Data availability statement

No additional data are available.

This is an open access article distributed under the terms of the Creative Commons Attribution IGO License ( CC BY NC 3.0 IGO ), which permits use, distribution,and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.Disclaimer: The author is a staff member of the World Health Organization. The author alone is responsible for the views expressed in this publication and they do not necessarily represent the views, decisions or policies of the World Health Organization.

https://doi.org/10.1136/bmjgh-2018-000858

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Summary box

Quantitative syntheses of studies on the effects of complex health interventions face high diversity across studies and limitations in the data available.

Statistical and non-statistical approaches are available for tackling intervention complexity in a synthesis of quantitative data in the context of a systematic review.

Appropriate methods will depend on the purpose of the synthesis, the number and similarity of studies included in the review, the level of detail available from the studies, the nature of the results reported in the studies, the expertise of the synthesis team and the resources available.

We offer considerations for selecting methods for synthesis of quantitative data to address important types of questions about the effects of complex interventions.

Public health and health service interventions are typically complex. They are usually multifaceted, with impacts at multiple levels and on multiple stakeholders. Also, the systems within which they are implemented may change and adapt to enhance or dampen their impact. 1 Quantitative syntheses ('meta-analyses’) of studies of complex interventions seek to integrate quantitative findings across multiple studies to achieve a coherent message greater than the sum of their parts. Interest is growing on how the standard systematic review and meta-analysis toolkit can be enhanced to address complexity of interventions and their impact. 2 A recent report from the Agency for Healthcare Research and Quality and a series of papers in the Journal of Clinical Epidemiology provide useful background on some of the challenges. 3–6

This paper is part of a series to explore the implications of complexity for systematic reviews and guideline development, commissioned by WHO. 7 Clearly, and as covered by other papers in this series, guideline development encompasses the consideration of many different aspects, 8 such as intervention effectiveness, economic considerations, acceptability 9 or certainty of evidence, 10 and requires the integration of different types of quantitative as well as qualitative evidence. 11 12 This paper is specifically concerned with methods available for the synthesis of quantitative results in the context of a systematic review on the effects of a complex intervention. We aim to point those collating evidence in support of guideline development to methodological approaches that will help them integrate the quantitative evidence they identify. A summary of how these methods link to many of the types of complexity encountered is provided in table 1 , based on the examples provided in a table from an earlier paper in the series. 1 An annotated list of the methods we cover is provided in table 2 .

• View inline

Quantitative synthesis possibilities to address aspects of complexity

Quantitative graphical and synthesis approaches mentioned in the paper, with their main strengths and weaknesses in the context of complex interventions

We begin by reiterating the importance of starting with meaningful research questions and an awareness of the purpose of the synthesis and any relevant background knowledge. An important issue in systematic reviews of complex interventions is that data available for synthesis are often extremely limited, due to small numbers of relevant studies and limitations in how these studies are conducted and their results are reported. Furthermore, it is uncommon for two studies to evaluate exactly the same intervention, in part because of the interventions’ inherent complexity. Thus, each study may be designed to provide information on a unique context or a novel intervention approach. Outcomes may be measured in different ways and at different time points. We therefore discuss possible approaches when data are highly limited or highly heterogeneous, including the use of graphical approaches to present very basic summary results. We then discuss statistical approaches for combining results and for understanding the implications of various kinds of complexity.

In several places we draw on an example of a review undertaken to inform a recent WHO guideline on protecting, promoting and supporting breast feeding. 13 The review seeks to determine the effects of interventions to promote breast feeding delivered in five types of settings (health services, home, community, workplace, policy context or a combination of settings). 8 The included interventions were predominantly multicomponent, and were implemented in complex systems across multiple contexts. The review included 195 studies, including many from low-income and middle-income countries, and concluded that interventions should be delivered in a combination of settings to achieve high breastfeeding rates.

The importance of the research question

The starting point in any synthesis of quantitative evidence is a clear purpose. The input of stakeholders is critical to ensure that questions are framed appropriately, addressing issues important to those commissioning, delivering and affected by the intervention. Detailed discussion of the development of research questions is provided in an earlier paper in the series, 1 and a subsequent paper explains the importance of taking context into account. 9 The first of these papers describes two possible perspectives. A complex interventions perspective emphasises the complexities involved in conceptualising, specifying and implementing the intervention per se, including the array of possibly interacting components and the behaviours required to implement it. A complex systems perspective emphasises the complexity of the systems into which the intervention is introduced, including possible interactions between the intervention and the system, interactions between individuals within the system and how the whole system responds to the intervention.

The simplest purpose of a systematic review is to determine whether a particular type of complex intervention (or class of interventions) is effective compared with a ‘usual practice’ alternative. The familiar PICO framework is helpful for framing the review: 14 in the PICO framework, a broad research question about effectiveness is uniquely specified by describing the participants (‘P’, including the setting and prevailing conditions) to which the intervention is to be applied; the intervention (‘I’) and comparator (‘C’) of interest, and the outcomes (‘O’, including their time course) that might be impacted by the intervention. In the breastfeeding review, the primary synthesis approach was to combine all available studies, irrespective of setting, and perform separate meta-analyses for different outcomes. 15

More useful than a review that asks ‘does a complex intervention work?’ is one that determines the situations in which a complex intervention has a larger or smaller effect. Indeed, research questions targeted by syntheses in the presence of complexity often dissect one or more of the PICO elements to explore how intervention effects vary both within and across studies (ie, treating the PICO elements as ‘moderators’). For instance, analyses may explore variation across participants, settings and prevailing conditions (including context); or across interventions (including different intervention components that may be present or absent in different studies); or across outcomes (including different outcome measures, at different levels of the system and at different time points) on which effects of the intervention occur. In addition, there may be interest in how aspects of the underlying system or the intervention itself mediate the effects, or in the role of intermediate outcomes on the pathway from intervention to impact. 16 In the breastfeeding review, interest moved from the overall effects across interventions to investigations of how effects varied by such factors as intervention delivery setting, high-income versus low-income country, and urban versus rural setting. 15

The role of logic models to inform a synthesis

An earlier paper describes the benefits of using system-based logic models to characterise a priori theories about how the system operates. 1 These provide a useful starting point for most syntheses since they encourage consideration of all aspects of complexity in relation to the intervention or the system (or both). They can help identify important mediators and moderators, and inform decisions about what aspects of the intervention and system need to be addressed in the synthesis. As an example, a protocol for a review of the health effects of environmental interventions to reduce the consumption of sugar-sweetened beverages included a system-based logic model, detailing how the characteristics of the beverages, and the physiological characteristics and psychological characteristics of individuals, are thought to impact on outcomes such as weight gain and cardiovascular disease. 17 The logic model informs the selection of outcomes and the general plans for synthesis of the findings of included studies. However, system-based models do not usually include details of how implementation of an intervention into the system is likely to affect subsequent outcomes. They therefore have a limited role in informing syntheses that seek to explain mechanisms of action.

A quantitative synthesis may draw on a specific proposed framework for how an intervention might work; these are sometimes referred to as process-orientated logic models, and may be strongly driven by qualitative research evidence. 12 They represent causal processes, describing what components or aspects of an intervention are thought to impact on what behaviours and actions, and what the further consequences of these impacts are likely to be. 18 They may encompass mediators of effect and moderators of effect. A synthesis may simply adopt the proposed causal model at face value and attempt to quantify the relationships described therein. Where more than one possible causal model is available, a synthesis may explore which of the models is better supported by the data, for example, by examining the evidence for specific links within the model or by identifying a statistical model that corresponds to the overall causal model. 18 19

A systematic review on community-level interventions for improving access to food in low-income and middle-income countries was based on a logic model that depicts how interventions might lead to improved health status. 20 The model includes direct effects, such as increased financial resources of individuals and decreased food prices; intermediate effects, such as increased quantity of food available and increase in intake; and main outcomes of interest, such as nutritional status and health indicators. The planned statistical synthesis, however, was to tackle these one at a time.

Considering the types of studies available

Studies of the effects of complex interventions may be randomised or non-randomised, and often involve clustering of participants within social or organisational units. Randomised trials, if sufficiently large, provide the most convincing evidence about the effects of interventions because randomisation should result in intervention and comparator groups with similar distributions of both observed and unobserved baseline characteristics. However, randomised trials of complex interventions may be difficult or impossible to undertake, or may be performed only in specific contexts, yielding results that are not generalisable. Non-randomised study designs include so-called ‘quasi-experiments’ and may be longitudinal studies, including interrupted time series and before-after studies, with or without a control group. Non-randomised studies are at greater risk of bias, sometimes substantially so, although may be undertaken in contexts that are more relevant to decision making. Analyses of non-randomised studies often use statistical controls for confounders to account for differences between intervention groups, and challenges are introduced when different sets of confounders are used in different studies. 21 22

Randomised trials and non-randomised studies might both be included in a review, and analysts may have to decide whether to combine these in one synthesis, and whether to combine results from different types of non-randomised studies in a single analysis. Studies may differ in two ways: by answering different questions, or by answering similar questions with different risks of bias. The research questions must be sufficiently similar and the studies sufficiently free of bias for a synthesis to be meaningful. In the breastfeeding review, randomised, quasi-experimental and observational studies were combined; no evidence suggested that the effects differed across designs. 15 In practice, many methodologists generally recommend against combining randomised with non-randomised studies. 23

Preparing for a quantitative synthesis

Before undertaking a quantitative synthesis of complex interventions, it can be helpful to begin the synthesis non-quantitatively, looking at patterns and characteristics of the data identified. Systematic tabulation of information is recommended, and this might be informed by a prespecified logic model. The most established framework for non-quantitative synthesis is that proposed by Popay et al . 24 The Cochrane Consumers and Communication group succinctly summarise the process as an 'investigation of the similarities and the differences between the findings of different studies, as well as exploration of patterns in the data’. 25 Another useful framework was described by Petticrew and Roberts. 26 They identify three stages in the initial narrative synthesis: (1) Organisation of studies into logical categories, the structure of which will depend on the purpose of the synthesis, possibly relating to study design, outcome or intervention types. (2) Within-study analysis, involving the description of findings within each study. (3) Cross-study synthesis, in which variations in study characteristics and potential biases are integrated and the range of effects described. Aspects of this process are likely to be implemented in any systematic review, even when a detailed quantitative synthesis is undertaken.

In some circumstances the available data are too diverse, too non-quantitative or too sparse for a quantitative synthesis to be meaningful even if it is possible. The best that can be achieved in many reviews of complex interventions is a non-quantitative synthesis following the guidance given in the above frameworks.

Options when effect size estimates cannot be obtained or studies are too diverse to combine

Graphical approaches.

Graphical displays can be very valuable to illustrate patterns in results of studies. 27 We illustrate some options in figure 1 . Forest plots are the standard illustration of the results of multiple studies (see figure 1 , panel A), but require a similar effect size estimate from each study. For studies of complex interventions, the diversity of approaches to the intervention, the context, 1 evaluation approaches and reporting differences can lead to considerable variation across studies in what results are available. Some novel graphical approaches have been proposed for such situations. A recent development is the albatross plot, which plots p values against sample sizes, with approximate effect-size contours superimposed (see figure 1 , panel B). 28 The contours are computed from the p values and sample sizes, based on an assumption about the type of analysis that would have given rise to the p values. Although these plots are designed for situations when effect size estimates are not available, the contours can be used to infer approximate effect sizes from studies that are analysed and reported in highly diverse ways. Such an advantage may prove to be a disadvantage, however, if the contours are overinterpreted.

• Download figure
• Open in new tab
• Download powerpoint

Example graphical displays of data from a review of interventions to promote breast feeding, for the outcome of continued breast feeding up to 23 months. 15 Panel A: Forest plot for relative risk (RR) estimates from each study. Panel B: Albatross plot of p value against sample size (effect contours drawn for risk ratios assuming a baseline risk of 0.15; sample sizes and baseline risks extracted from the original papers by the current authors); Panel C: Harvest plot (heights reflect design: randomised trials (tall), quasi-experimental studies (medium), observational studies (short); bar shading reflects follow-up: longest follow-up (black) to shortest follow-up (light grey) or no information (white)). Panel D: Bubble plot (bubble sizes and colours reflect design: randomised trials (large, green), quasi-experimental studies (medium, red), observational studies (small, blue); precision defined as inverse of the SE of each effect estimate (derived from the CIs); categories are: “Potential Harm”: RR <0.8; “No Effect”: RRs between 0.8 and 1.25; “Potential Benefit”: RR >1.25 and CI includes RR=1; “Benefit”: RR >1.25 and CI excludes RR=1).

Harvest plots have been proposed by Ogilvie et al as a graphical extension of a vote counting approach to synthesis (see figure 1 , panel C). 29 However, approaches based on vote counting of statistically significant results have been criticised on the basis of their poor statistical properties, and because statistical significance is an outdated and unhelpful notion. 30 The harvest plot is a matrix of small illustrations, with different outcome domains defining rows and different qualitative conclusions (negative effect, no effect, positive effect) defining columns. Each study is represented by a bar that is positioned according to its measured outcome and qualitative conclusion. Bar heights and shadings can depict features of the study, such as objectivity of the outcome measure, suitability of the study design and study quality. 29 31 A similar idea to the harvest plot is the effect direction plot proposed by Thomson and Thomas. 32

A device to plot the findings from a large and complex collection of evidence is a bubble plot (see figure 1 , panel D). A bubble plot illustrates the direction of each finding (or whether the finding was unclear) on a horizontal scale, using a vertical scale to indicate the volume of evidence, and with bubble sizes to indicate some measure of credibility of each finding. Such an approach can also depict findings of collections of studies rather than individual studies, and was used successfully, for example, to summarise findings from a review of systematic reviews of the effects of acupuncture on various indications for pain. 33

Statistical methods not based on effect size estimates

We have mentioned that a frequent problem is that standard meta-analysis methods cannot be used because data are not available in a similar format from every study. In general, the core principles of meta-analysis can be applied even in this situation, as is highlighted in the Cochrane Handbook , by addressing the questions: ‘What is the direction of effect?’; 'What is the size of effect?’; ‘Is the effect consistent across studies?’; and 'What is the strength of evidence for the effect?’. 34

Alternatives to the estimation of effect sizes could be used more often than they are in practice, allowing some basic statistical inferences despite diversely reported results. The most fundamental analysis is to test the overall null hypothesis of no effect in any of the studies. Such a test can be undertaken using only minimally reported information from each study. At its simplest, a binomial test can be performed using only the direction of effect observed in each study, irrespective of its CI or statistical significance. 35 Where exact p values are available as well as the direction of effect, a more powerful test can be performed by combining these using, for example, Fisher’s combination of p values. 36 It is important that these p values are computed appropriately, however, accounting for clustering or matching of participants within the studies. Rejecting the null model based on such tests provides no information about the magnitude of the effect, providing information only on whether at least one study shows an effect is present, and if so, its direction. 37

Standard synthesis methods

Meta-analysis for overall effect.

Probably the most familiar approach to meta-analysis is that of estimating a single summary effect across similar studies. This simple approach lends itself to the use of forest plots to display the results of individual studies as well as syntheses, as illustrated for the breastfeeding studies in figure 1 (panel A). This analysis addresses the broad question of whether evidence from a collection of studies supports an impact of the complex intervention of interest, and requires that every study makes a comparison of a relevant intervention against a similar alternative. In the context of complex interventions, this is described by Caldwell and Welton as the ‘lumping’ approach, 38 and by Guise et al as the ‘holistic’ approach. 5 6 One key limitation of the simple approach is that it requires similar types of data from each study. A second limitation is that the meta-analysis result may have limited relevance when the studies are diverse in their characteristics. Fixed-effect models, for instance, are unlikely to be appropriate for complex interventions because they ignore between-studies variability in underlying effect sizes. Results based on random-effects models will need to be interpreted by acknowledging the spread of effects across studies, for example, using prediction intervals. 39

A common problem when undertaking a simple meta-analysis is that individual studies may report many effect sizes that are correlated with each other, for example, if multiple outcomes are measured, or the same outcome variable is measured at several time points. Numerous approaches are available for dealing with such multiplicity, including multivariate meta-analysis, multilevel modelling, and strategies for selecting effect sizes. 40 A very simple strategy that has been used in systematic reviews of complex interventions is to take the median effect size within each study, and to summarise these using the median of these effect sizes across studies. 41

Exploring heterogeneity

Diversity in the types of participants (and contexts), interventions and outcomes are key to understanding sources of complexity. 9 Many of these important sources of heterogeneity are most usefully examined—to the extent that they can reliably be understood—using standard approaches for understanding variability across studies, such as subgroup analyses and meta-regression.

A simple strategy to explore heterogeneity is to estimate the overall effect separately for different levels of a factor using subgroup analyses (referring to subgrouping studies rather than participants). 42 As an example, McFadden et al conducted a systematic review and meta-analysis of 73 studies of support for healthy breastfeeding mothers with healthy term babies. 43 They calculated separate average effects for interventions delivered by a health professional, a lay supporter or with mixed support, and found that the effect on cessation of exclusive breast feeding at up to 6 months was greater for lay support compared with professionals or mixed support (p=0.02). Guise et al provide several ways of grouping studies according to their interventions, for example, grouping studies by key components, by function or by theory. 5 6

Meta-regression provides a flexible generalisation to subgroup analyses, whereby study-level covariates are included in a regression model using effect size estimates as the dependent variable. 44 45 Both continuous and categorical covariates can be included in such models; with a single categorical covariate, the approach is essentially equivalent to subgroup analyses. Meta-regression with continuous covariates in theory allows the extrapolation of relationships to contexts that were not examined in any of the studies, but this should generally be avoided. For example, if the effect of an interventional approach appears to increase as the size of the group to which it is applied decreases, this does not mean that it will work even better when applied to a single individual. More generally, the mathematical form of the relationship modelled in a meta-regression requires careful selection. Most often a linear relationship is assumed, but a linear relationship does not permit step changes such as might occur if an interventional approach requires a particular level of some feature of the underlying system before it has an effect.

Several texts provide guidance for using subgroup analysis and meta-regression in a general context 45 46 and for complex interventions. 3 4 47 In principle, many aspects of complexity in interventions can be addressed using these strategies, to create an understanding of the ‘response surface’. 48–50 However, in practice, the number of studies is often too small for reliable conclusions to be drawn. In general, subgroup analysis and meta-regression are fraught with dangers associated with having few studies, many sources of variation across study features and confounding of these features with each other as well as with other, often unobserved, variables. It is therefore important to prespecify a small number of plausible sources of diversity so as to reduce the danger of reaching spurious conclusions based on study characteristics that correlate with the effects of the interventions but are not the cause of the variation. The ability of statistical analyses to identify true sources of heterogeneity will depend on the number of studies, the sizes of the studies and the true differences between effects in studies with different characteristics.

Synthesis methods for understanding components of the intervention

When interventions comprise distinct components, it is attractive to separate out the individual effects of these components. 51 Meta-regression can be used for this, using covariates to code the presence of particular features in each intervention implementation. As an example, Blakemore et al analysed 39 intervention comparisons from 33 independent studies aiming to reduce urgent healthcare use in adults with asthma. 52 Effect size estimates were coded according to components used in the interventions, and the authors found that multicomponent interventions including skills training, education and relapse prevention appeared particularly effective. In another example, of interventions to support family caregivers of people with Alzheimer’s disease, 53 the authors used methods for decomposing complex interventions proposed by Czaja et al , 54 and created covariates that reduced the complexity of the interventions to a small number of features about the intensity of the interventions. More sophisticated models for examining components have been described by Welton et al , 55 Ivers et al 56 and Madan et al . 57

A component-level approach may be useful when there is a need to disentangle the ‘active ingredients’ of an intervention, for example, when adapting an existing intervention for a new setting. However, components-based approaches require assumptions, such as whether individual components are additive or interact with each other. Furthermore, the effects of components can be difficult to estimate if they are used only in particular contexts or populations, or are strongly correlated with use of other components. An alternative approach is to treat each combination of components as a separate intervention. These separate interventions might then be compared in a single analysis using network meta-analysis. A network meta-analysis combines results from studies comparing two or more of a larger set of interventions, using indirect comparisons via common comparators to rank-order all interventions. 47 58 59 As an example, Achana et al examined the effectiveness of safety interventions on the uptake of three poisoning prevention practices in households with children. Each singular combination of intervention components was defined as a separate intervention in the network. 60 Network meta-analysis may also be useful when there is a need to compare multiple interventions to answer an ‘in principle’ question of which intervention is most effective. Consideration of the main goals of the synthesis will help those aiming to prepare guidelines to decide which of these approaches is most appropriate to their needs.

A case study exploring components is provided in box 1 , and an illustration is provided in figure 2 . The component-based analysis approach can be likened to a factorial trial, in that it attempts to separate out the effects of individual components of the complex interventions, and the network meta-analysis approach can be likened to a multiarm trial approach, where each complex intervention in the set of studies is a different arm in the trial. 47 Deciding between the two approaches can leave the analyst caught between the need to ‘split’ components to reflect complexity (and minimise heterogeneity) and ‘lump’ to make an analysis feasible. Both approaches can be used to examine other features of interventions, including interventions designed for delivery at different levels. For example, a review of the effects of interventions for children exposed to domestic violence and abuse included studies of interventions targeted at children alone, parents alone, children and parents together, and parents and children separately. 61 A network meta-analysis approach was taken to the synthesis, with the people targeted by the intervention used as a distinguishing feature of the interventions included in the network.

Example of understanding components of psychosocial interventions for coronary heart disease

Welton et al reanalysed data from a Cochrane review 89 of randomised controlled trials assessing the effects of psychological interventions on mortality and morbidity reduction for people with coronary heart disease. 55 The Cochrane review focused on the effectiveness of any psychological intervention compared with usual care, and found evidence that psychological interventions reduced non-fatal reinfarctions and depression and anxiety symptoms. The Cochrane review authors highlighted the large heterogeneity among interventions as an important limitation of their review.

Welton et al were interested in the effects of the different intervention components. They classified interventions according to which of five key components were included: educational, behavioural, cognitive, relaxation and psychosocial support ( figure 2 ). Their reanalysis examined the effect of each component in three different ways: (1) An additive model assuming no interactions between components. (2) A two-factor interaction model, allowing for interactions between pairs of components. (3) A network meta-analysis, defining each combination of components as a separate intervention, therefore allowing for full interaction between components. Results suggested that interventions with behavioural components were effective in reducing the odds of all-cause mortality and non-fatal myocardial infarction, and that interventions with behavioural and/or cognitive components were effective for reducing depressive symptoms.

Intervention components in the studies integrated by Welton et al (a sample of 18 from 56 active treatment arms). EDU, educational component; BEH, behavioural component; COG, cognitive component; REL, relaxation component; SUP, psychosocial support component.

A common limitation when implementing these quantitative methods in the context of complex interventions is that replication of the same intervention in two or more studies is rare. Qualitative comparative analysis (QCA) might overcome this problem, being designed to address the ’small N; many variables’ problem. 62 QCA involves: (1) Identifying theoretically driven thresholds for determining intervention success or failure. (2) Creating a 'truth table’, which takes the form of a matrix, cross-tabulating all possible combinations of conditions (eg, participant and intervention characteristics) against each study and its associated outcomes. (3) Using Boolean algebra to eliminate redundant conditions and to identify configurations of conditions that are necessary and/or sufficient to trigger intervention success or failure. QCA can usefully complement quantitative integration, sometimes in the context of synthesising diverse types of evidence.

Synthesis methods for understanding mechanisms of action

An alternative purpose of a synthesis is to gain insight into the mechanisms of action behind an intervention, to inform its generalisability or applicability to a particular context. Such syntheses of quantitative data may complement syntheses of qualitative data, 11 and the two forms might be integrated. 12 Logic models, or theories of action, are important to motivate investigations of mechanism. The synthesis is likely to focus on intermediate outcomes reflecting intervention processes, and on mediators of effect (factors that influence how the intervention affects an outcome measure). Two possibilities for analysis are to use these intermediate measurements as predictors of main outcomes using meta-regression methods, 63 or to use multivariate meta-analysis to model the intermediate and main outcomes simultaneously, exploiting and estimating the correlations between them. 64 65 If the synthesis suggests that hypothesised chains of outcomes hold, this lends weight to the theoretical model underlying the hypothesis.

An approach to synthesis closely identified with this category of interventions is model-driven meta-analysis, in which different sources of evidence are integrated within a causal path model akin to a directed acyclic graph. A model-driven meta-analysis is an explanatory analysis. 66 It attempts to go further than a standard meta-analysis or meta-regression to explore how and why an intervention works, for whom it works, and which aspects of the intervention (factors) are driving overall effect. Such syntheses have been described in frequentist 19 67–70 and Bayesian 71 72 frameworks and are variously known as model-driven meta-analysis, linked meta-analysis, meta-mediation analysis and meta-analysis of structural equation models. In their simplest form, standard meta-analyses estimate a summary correlation independently for each pair of variables in the model. The approach is inherently multivariate, requiring the estimation of multiple correlations (which, if obtained from a single study, are also not independent). 73–75 Each study is likely to contribute fragments of the correlation matrix. A summary correlation matrix, combined either by fixed-effects or random-effects methods, then serves as the input for subsequent analysis via a standardised regression or structural equation model.

An example is provided in box 2 . The model in figure 3 postulates that the effect of ‘Dietary adherence’ on ‘Diabetes complications’ is not direct but is mediated by ‘Metabolic control’. 76 The potential for model-driven meta-analysis to incorporate such indirect effects also allows for mediating effects to be explicitly tested and in so doing allows the meta-analyst to identify and explore the mechanisms underpinning a complex intervention. 77

Theoretical diabetes care model (adapted from Brown et al 68 ).

Example of a model-driven meta-analysis for type 2 diabetes

Brown et al present a model-driven meta-analysis of correlational research on psychological and motivational predictors of diabetes outcomes, with medication and dietary adherence factors as mediators. 76 In a linked methodological paper, they present the a priori theoretical model on which their analysis is based. 68 The model is simplified in figure 3 , and summarised for the dietary adherence pathway only. The aim of their full analysis was to determine the predictive relationships among psychological factors and motivational factors on metabolic control and body mass index (BMI), and the role of behavioural factors as possible mediators of the associations among the psychological and motivational factors and metabolic control and BMI outcomes.

The analysis is based on a comprehensive systematic review. Due to the number of variables in their full model, 775 individual correlational or predictive studies reported across 739 research papers met eligibility criteria. Correlations between each pair of variables in the model were summarised using an overall average correlation, and homogeneity assessed. Multivariate analyses were used to estimate a combined correlation matrix. These results were used, in turn, to estimate path coefficients for the predictive model and their standard errors. For the simplified model illustrated here, the results suggested that coping and self-efficacy were strongly related to dietary adherence, which was strongly related to improved glycaemic control and, in turn, a reduction in diabetic complications.

Synthesis approaches for understanding complexities of the system

Syntheses may seek to address complexities of the system to understand either the impact of the system on the effects of the intervention or the effects of the intervention on the system. This may start by modelling the salient features of the system’s dynamics, rather than focusing on interventions. Subgroup analysis and meta-regression are useful approaches for investigating the extent to which an intervention’s effects depend on baseline features of the system, including aspects of the context. Sophisticated meta-regression models might investigate multiple baseline features, using similar approaches to the component-based meta-analyses described earlier. Specifically, aspects of context or population characteristics can be regarded as ‘components’ of the system into which the intervention is introduced, and similar statistical modelling strategies used to isolate effects of individual factors, or interactions between them.

When interventions act at multiple levels, it may be important to understand the effects at these different levels. Outcomes may be measured at different levels (eg, at patient, clinician and clinical practice levels) and analysed separately. Qualitative research plays a particularly important role in identifying the outcomes that should be assessed through quantitative synthesis. 12 Care is needed to ensure that the unit of analysis issues are addressed. For example, if clinics are the unit of randomisation, then outcomes measured at the clinic level can be analysed using standard methods, whereas outcomes measured at the level of the patient within the clinic would need to account for clustering. In fact, multiple dependencies may arise in such data, when patients receive care in small groups. Detailed investigations of effect at different levels, including interactions between the levels, would lend themselves to multilevel (hierarchical) models for synthesis. Unfortunately, individual participant data at all levels of the hierarchy are needed for such analyses.

Model-based approaches also offer possibilities for addressing complex systems; these include economic models, mathematical models and systems science methods generally. 78–80 Broadly speaking, these provide mathematical representations of logic models, and analyses may involve incorporation of empirical data (eg, from systematic reviews), computer simulation, direct computation or a mixture of these. Multiparameter evidence synthesis methods might be used. 81 82 Approaches include models to represent systems (eg, systems dynamics models) and approaches that simulate individuals within the system (eg, agent-based models). 79 Models can be particularly useful when empirical evidence does not address all important considerations, such as ‘real-world’ contexts, long-term effects, non-linear effects and complexities such as feedback loops and threshold effects. An example of a model-based approach to synthesis is provided in box 3 . The challenge when adopting these approaches is often in the identification of system components, and accurately estimating causes and effects (and uncertainties). There are few examples of the use of these analytical tools in systematic reviews, but they may be useful when the focus of analysis is on understanding the causes of complexity in a given system rather than on the impact of an intervention.

Example of a mathematical modelling approach for soft drinks industry levy

Briggs et al examined the potential impact of a soft drinks levy in the UK, considering possible different types of response to the levy by industry. 90 Various scenarios were posited, with effects on health outcomes informed by empirical data from randomised trials and cohort studies of association between sugar intake and body weight, diabetes and dental caries. Figure 4 provides a simple characterisation of how the empirical data were fed into the model. Inputs into the model included levels of consumption of various types of drinks (by age and sex), volume of drinks sales, and baseline levels of obesity, diabetes and dental caries (by age and sex). The authors concluded that health gains would be greatest if industry reacted by reformulating their products to include less sugar.

Simplified version of the conceptual model used by Briggs et al ( a dapted from Briggs et al 90 ).

Considerations of bias and relevance

It is always important to consider the extent to which (1) The findings from each study have internal validity, particularly for non-randomised studies which are typically at higher risk of bias. (2) Studies may have been conducted but not reported because of unexciting findings. (3) Each study is applicable to the purposes of the review, that is, has external validity (or ‘directness’), in the language of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group. 83 At minimum, internal and external validity should be examined and reported, and the risk of publication bias assessed, and these can be achieved through the GRADE framework. 10 With sufficient studies, information collected might be used in meta-regression analyses to evaluate empirically whether studies with and without specific sources of bias or indirectness differ in their results.

It may be desirable to learn about a specific setting, intervention type or outcome measure more directly than others. For example, to inform a decision for a low-income setting, emphasis should be placed on results of studies performed in low-income countries. One option is to restrict the synthesis to these studies. An alternative is to model the dependence of an intervention’s effect on some feature(s) related to the income setting, and extract predictions from the model that are most relevant to the setting of interest. This latter approach makes fuller use of available data, but relies on stronger assumptions.

Often, however, the accumulated studies are too few or too disparate to draw conclusions about the impact of bias or relevance. On rare occasions, syntheses might implement formal adjustments of individual study results for likely biases. Such adjustments may be made by imposing prior distributions to depict the magnitude and direction of any biases believed to exist. 84 85 The choice of a prior distribution may be informed by formal assessments of risk of bias, by expert judgement, or possibly by empirical data from meta-epidemiological studies of biases in randomised and/or non-randomised studies. 86 For example, Wolf et al implemented a prior distribution based on findings of a meta-epidemiological study 87 to adjust for lack of blinding in studies of interventions to improve quality of point-of-use water sources in low-income and middle-income settings. 88 Unfortunately, empirical evidence of bias is mostly limited to clinical trials, is weak for trials of public health and social care interventions, and is largely non-existent for non-randomised studies.

Our review of quantitative synthesis methods for evaluating the effects of complex interventions has outlined many possible approaches that might be considered by those collating evidence in support of guideline development. We have described three broad categories: (1) Non-quantitative methods, including tabulation, narrative and graphical approaches. (2) Standard meta-analysis methods, including meta-regression to investigate study-level moderators of effect. (3) More advanced synthesis methods, in which models allow exploration of intervention components, investigation of both moderators and mediators, examination of mechanisms, and exploration of complexities of the system.

The choice among these approaches will depend on the purpose of the synthesis, the similarity of the studies included in the review, the level of detail available from the studies, the nature of the results reported in the studies, the expertise of the synthesis team, and the resources available. Clearly the advanced methods require more expertise and resources than the simpler methods. Furthermore, they require a greater level of detail and typically a sizeable evidence base. We therefore expect them to be used seldomly; our aim here is largely to articulate what they can achieve so that they can be adopted when they are appropriate. Notably, the choice among these approaches will also depend on the extent to which guideline developers and users at global, national or local levels understand and are willing to base their decisions on different methods. Where possible, it will thus be important to involve concerned stakeholders during the early stages of the systematic review process to ensure the relevance of its findings.

Complexity is common in the evaluation of public health interventions at individual, organisational or community levels. To help systematic review and guideline development teams decide how to address this complexity in syntheses of quantitative evidence, we summarise considerations and methods in tables 1 and 2 . We close with the important remark that quantitative synthesis is not always a desirable feature of a systematic review. Whereas some sophisticated methods are available to deal with a variety of complex problems, on many occasions—perhaps even the majority in practice—the studies may be too different from each other, too weak in design or have data too sparse, for statistical methods to provide insight beyond a commentary on what evidence has been identified.

Acknowledgments

The authors thank the following for helpful comments on earlier drafts of the paper: Philippa Easterbrook, Matthias Egger, Anayda Portela, Susan L Norris, Mark Petticrew.

• Petticrew M ,
• Thomas J , et al
• Anderson L ,
• Elder R , et al
• Rehfuess E ,
• Noyes J , et al
• Viswanathan M , et al
• World Health Organization
• Rehfuess EA ,
• Stratil JM ,
• Scheel IB , et al
• Flemming K , et al
• Montgomery P ,
• Movsisyan A ,
• Grant SP , et al
• Flemming K ,
• Garside R , et al
• Moore G , et al
• Richardson WS ,
• Wilson MC ,
• Nishikawa J , et al
• Chowdhury R ,
• Sankar MJ , et al
• Collins D ,
• Johnson K ,
• von Philipsborn P ,
• Burns J , et al
• Schoonees A ,
• Becker BJ ,
• Duvendack M , et al
• Reeves BC ,
• Higgins JPT , et al
• Roberts H ,
• Ryan R , Cochrane Consumers and Communication Review Group
• Anzures-Cabrera J ,
• Harrison S ,
• Martin RM , et al
• Ogilvie D ,
• Petticrew M , et al
• Sterne JA ,
• Davey Smith G
• Crowther M ,
• Avenell A ,
• MacLennan G , et al
• Thomson HJ ,
• Taylor SL ,
• Higgins JPT ,
• Bushman BJ ,
• Caldwell DM ,
• Thompson SG ,
• Spiegelhalter DJ
• López-López JA ,
• Lipsey MW , et al
• Grimshaw JM ,
• Thomas RE ,
• Borenstein M ,
• McFadden A ,
• Renfrew MJ , et al
• van Houwelingen HC ,
• Arends LR ,
• Melendez-Torres GJ ,
• Ioannidis JP ,
• Steckelberg A ,
• Richter B , et al
• Presseau J ,
• Newham JJ , et al
• Blakemore A ,
• Dickens C ,
• Anderson R , et al
• Schulz R , et al
• Lee CC , et al
• Welton NJ ,
• Adamopoulos E , et al
• Tricco AC ,
• Trikalinos TA , et al
• Aveyard P , et al
• Calderbank-Batista T
• Achana FA ,
• Cooper NJ ,
• Bujkiewicz S , et al
• Howarth E ,
• Moore THM ,
• Welton NJ , et al
• O’Mara-Eves A ,
• Thompson SG
• Jackson D ,
• Raudenbush SW ,
• García AA , et al
• Cheung MW ,
• Watson SI ,
• Stewart GB ,
• Mengersen K ,
• García AA ,
• Brown A , et al
• Liu H , et al
• Johnson L ,
• Althaus C , et al
• Stamatakis KA
• Greenwood-Lee J ,
• Nettel-Aguirre A , et al
• Caldwell D , et al
• Colbourn T ,
• Asseburg C ,
• Bojke L , et al
• Guyatt GH ,
• Oxman AD , GRADE Working Group
• Turner RM ,
• Spiegelhalter DJ ,
• Smith GCS , et al
• Carlin JB , et al
• Schulz KF , et al
• Savović J ,
• Altman DG , et al
• Prüss-Ustün A ,
• Cumming O , et al
• Bennett P ,
• West R , et al
• Briggs ADM ,
• Mytton OT ,
• Kehlbacher A , et al

Handling editor Soumyadeep Bhaumik

Contributors JPTH co-led the project, conceived the paper, led discussions and wrote the first draft. JAL-L undertook analyses, contributed to discussions and contributed to writing the manuscript. BJB drafted material on mechanisms, contributed to discussions and contributed extensively to writing the manuscript. SRD screened and categorised the results of the literature searches, collated examples and contributed to discussions. SD undertook searches to identify relevant literature and contributed to discussions. JMG contributed to discussions and commented critically on drafts. LAM undertook analyses, contributed to discussions and commented critically on drafts. THMM contributed examples, contributed to discussions and commented critically on drafts. EAR and JT contributed to discussions and commented critically on drafts. DMC co-led the project, contributed to discussions and drafted extensive parts of the paper. All authors approved the final version of the manuscript.

Funding Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation. JPTH was funded in part by Medical Research Council (MRC) grant MR/M025209/1, by the MRC Integrative Epidemiology Unit at the University of Bristol (MC_UU_12013/9) and by the MRC ConDuCT-II Hub (Collaboration and innovation for Difficult and Complex randomised controlled Trials In Invasive procedures – MR/K025643/1). BJB was funded in part by grant DRL-1252338 from the US National Science Foundation (NSF). JMG holds a Canada Research Chair in Health Knowledge Transfer and Uptake. LAM is funded by a National Institute for Health Research (NIHR) Systematic Review Fellowship (RM-SR-2016-07 26). THMM was funded by the NIHR Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West). JT is supported by the NIHR Collaboration for Leadership in Applied Health Research and Care North Thames at Bart’s Health NHS Trust. DMC was funded in part by NIHR grant PHR 15/49/08 and by the Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer –MR/KO232331/1).

Disclaimer The views expressed are those of the authors and not necessarily those of the CRC program, the MRC, the NSF, the NHS, the NIHR or the UK Department of Health.

Competing interests JMG reports personal fees from the Campbell Collaboration. EAR reports being a Methods Editor with Cochrane Public Health.

Patient consent Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

Griffith University

Popular sites

Home > Griffith Sciences > School of Environment and Science > Research > Systematic Quantitative Literature Review

Systematic Quantitative Literature Review

School of Environment and Science

• Learning and teaching

A smart and effective method for undertaking literature reviews particularly for research students and others new to a discipline.

Narrative methods that are commonly used in many research theses, rely on the expertise and experience of the author, making them challenging for novices. In contrast, the method we use and recommend involves systematically searching the literature using online database and other sources to find all relevant papers that fit specific criteria (systematically identifying the literature), entering information about each study into a personal database, then compiling tables that summarise the current status of the literature (quantifying the literature). The results are reliable, quantifiable and reproducible.

Using this method, it’s also possible to determine if there are suitable datasets for meta-analysis. By mapping the literature we can not only identify what is known, but also, but where there are gaps: a critical issue in advancing research and designing PhD research programs.

Reliable, quantifiable and reproducible

The method works well for specific topics, but also for summarising diverse inter-disciplinary research. Using this method many of our students and others have gone on to publish their reviews. Importantly for PhD students, the database can be updated during the PhD thesis allowing them to easily identify relevant papers and produce their final thesis without having to re-read all the literature.

• Slides from workshop on the method (PDF 4MB)
• Slides from the advanced workshop on the method (PDF 5MB)
• More resources on doing your PhD as a series of papers

The method and its benefits

• Pickering, C.M. and Byrne, J. (2014). The benefits of publishing systematic quantitative literature reviews for PhD candidates and other early career researchers. Higher Education Research and Development. 33: 534-548
• Pickering, C., Grignon, J., Steven, R., Guitart, D. and Byrne. J. (2015). Publishing not perishing: How research students transition from novice to knowledgeable using systematic quantitative literature reviews. Studies in Higher Education. 40:10, 1756-1769, DOI: 10.1080/03075079.2014.914907. A pre-print copy of the paper is available here , or the final published version from the publisher's website.

Research study

Videos about the method

Overview of method

Being systematic

Creating your own review database

Writing the review

Why publish during your PhD?

Rochele Steven discusses using the method

Julien Grignon discusses using the method

Advanced SQLR 1 - Challenges in being systematic

Advanced SQLR 2 - Coding challenges

Advanced SQLR 3 - Advanced data analysis

Advanced SQLR 4 - Reviewers comments

Three circles for structuring a literature review

Eloise Stephenson - Ross River virus ecology

There are now hundreds of papers published using this method. A full list of them is available from google scholar.

Some select examples showing how they have been done, including searching strategies, ways to analysis the data and address some concerns regarding use/non-use of grey literature, factors affecting demand for, and supply of research by country etc, addressed in the advanced videos include:

• Guitart, D., Pickering, C.M. and Byrne, J. (2012). Past results and future directions in urban community gardens research. Urban Forestry and Urban Greening. 11: 364-373 — This was one of the original reviews using our methods. It highlights the importance of gap analysis and dealing with reviewing a very diverse literature including disciplines and methods used, and the capacity to review both quantitative and qualitative literature.
• Steven, R. Pickering, C.M. and Castley, G. (2011). A review of the impacts of nature based recreation on birds. Journal of Environmental Management. 92: 2287-2294 — This early SQLR provides a detailed example of why gap analysis is important and ways of quantifying differences between the supply of literature and need for research by calculating bird diversity vs number of studies per region.
• Pickering, C.M., Rossi, S.D., Hernando, A. and Barros, A. (2018). Current knowledge and future research directions for monitoring and management of visitors in recreational and protected areas. Journal of Outdoor Recreation and Tourism. 21: 10-18 — This SQLR of abstracts from a conference, includes a detailed examination of factors affecting the supply of research including why research is often dominated by literature from North America and Europe, and why that can be an issue. It also includes more sophisticated ways to access the results by using multi-dimensional analysis.
• Verrall, B. and Pickering, C.M. (2020). Alpine vegetation in the context of climate change: A global review of past research and future directions. Science of the Total Environment. 748:141344 . This is an example of a bibliometric/scientometric literature review – an alternative method using vosViewer to visuals patterns in keywords over time in the literature and changes in who is publishing on a topic and where.
• Thomas, S. (2014). Blue carbon: Knowledge gaps, critical issues and novel approaches — This review also uses Leximancer analysis (text analysis) of themes to map concepts in the literature based on keywords and phrases in the papers.
• Riebe, L., Girardi, A. and Whitsed, C. (2016). A systematic literature review of teamwork pedagogy in higher education. Small Group Research. 47: 619-664 — This paper from the education area includes an excellent example of how to use keywords and search terms in a stepped down way to identify a complex literature.
• Liao, Y., Deschamps, F., de Freitas Rocha Loures, E. and Ramos, L.F.P. (2017). Past, present and future of Industry 4.0- a systematic literature review and research agenda proposal. International Journal of Production Research. 55: 3609-3629 — This paper includes a good example about how to set out the justification for which papers to include and exclude including less clear-cut topics. It also includes an excellent way to quantify the disciplines represented by papers using well recognised (SCImago) data for each journal.
• Turner, J.A., Babcock, R.C., Hovey, R., and Kendrick, G.A. (2017). Deep thinking: a systematic review of mesophotic coral ecosystems. ICES Journal of Marine Science — This review uses column diagrams to clearly illustrate how over time the methods used in the discipline and the types of analysis conducted become more complex as the literature develops.
• Pahlevan-Sharif, S. Mura, P., and Wijesinghe, S.N.R. (2019). A systematic review of systematic reviews in tourism. Journal of Hospitality and Tourism Management. 39: 158.165 — This recent paper provides an overview review of other literature reviews. It highlights the types of methods used in reviews in tourism including recommendations for future reviews.
• Bezerra, M.F., Lacerda, L.D. and Lai, C-T. (2019). Trace metals and persistent organic pollutants contamination in batoids (Chondrichthyes: Batoidea): A systematic review. Environmental Pollution. 248: 684-695 — This recent review includes a broad SQLR and meta-analysis of a science/quantitative area.
• Nikulina, V., Simon, D., Ny, H. and Baumann, H. (2019). Context-adapted urban planning for rapid transitioning of personal mobility towards sustainability: A systematic literature review. Sustainability: 11: — This paper combines a SQLR, with narrative commentary on themes, and a bibliometric analysis.

Further resources and contacts

• Griffith University Research Centres
• Professor Catherine Pickering
• Pickering, C.M. (2012). Writing Ecology Research Papers. Environment Futures Research Centre. Griffith University, Gold Coast

Article in The Conversation:

• Pickering, C.M. and Byrne, J. (2014). How to find the knowns and unknowns in any research. The Conversation.

Important reference for how to report systematic literature reviews required by many journals:

• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71 - with more details available here.
• Boote, B.N. and Beile P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher. 34: 3-15.
• Crisp, B.R. (2015) Systematic reviews: A social work perspective. Australian Social Work, 68:3, 284-295.
• Murray, R. (2011). How to Write a Thesis. McGraw Hill Open University Press. Maidenhead, England (Chapter on writing a literature reviews).
• Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing, Oxford, England.
• Randolph J.J. (2009). A guide to writing the dissertation literature review. Practical Assessment, Research and Evaluation. 14: 1-13.

Here are examples of the types of excel databases used in some Systematic Quantitative Literature Reviews:

• Steven et al. 2011 database of papers on impacts nature based tourism on birds (XLSX 16KB)
• Ballantyne and Pickering In review databases of papers on environmental impacts of recreation trails (XLSX 32KB)
• Byrne and Portanger 2014 database of papers climate change, energy policy and justice (XLSX 48KB)

Some of the journals publishing SQLR include:

• Analyse und Kritik
• ASEE Annual Conference and Exposition, Conference Proceedings
• Asia and the Pacific Policy Studies
• Asian Journal of Criminology
• Austral Entomology
• Australian Social Work
• Behaviour Change
• Biological Conservation
• BMC Health Services Research
• CIRP Journal of Manufacturing Science and Technology
• Climatic Change
• Conservation Biology
• Corruption in Sport: Causes, Consequences, and Reform
• Crop Protection
• Crystal Research and Technology
• Cuadernos de Desarrollo Rural
• Cuadernos de Turismo
• Current Issues in Tourism
• Documents d'Analisi Geografica
• Ecologia Austral
• Ecological Economics
• Ecology and Society
• Education Sciences
• Educational Media International
• Energy and Buildings
• Environment International
• Environment Systems and Decisions
• Environmental Modelling and Software
• Environmental Pollution
• Environmental Science and Policy
• European Journal of Higher Education
• European Journal of Information Systems
• Evaluation and Program Planning
• Event Management
• Fish and Fisheries
• Frontiers in Ecology and the Environment
• Global Environmental Change
• Habitat International
• Higher Education Research and Development
• ICES Journal of Marine Science
• IEEE Internet of Things Journal
• IFAC-PapersOnLine
• International Journal of Disaster Risk Reduction
• International Journal of Language and Communication Disorders
• International Journal of Managing Projects in Business
• International Journal of Mental Health Nursing
• International Journal of Production Research
• International Journal of the Commons
• International Journal of Tourism Research
• International Social Work
• Issues in Educational Research
• Journal of Business Research
• Journal of Cleaner Production
• Journal of Environmental Management
• Journal of Healthcare Leadership
• Journal of Hospitality and Tourism Research
• Journal of Hospitality Marketing and Management
• Journal of Information Technology in Construction
• Journal of Outdoor Recreation and Tourism
• Journal of Place Management and Development
• Journal of Reproductive and Infant Psychology
• Journal of Sustainable Tourism
• Journal of Technical Education and Training
• Journal of the Medical Library Association
• Journal of Travel and Tourism Marketing
• Journal of Urbanism
• Journal of Vocational Rehabilitation
• Landscape and Urban Planning
• Local Environment
• Managing Sport and Leisure
• Natural Hazards and Earth System Sciences
• Nurse Education in Practice
• Ocean and Coastal Management
• Procedia CIRP
• Procedia Engineering
• Procedia Manufacturing
• Proceedings of AISB Annual Convention 2018
• Proceedings of the International Conference on e-Learning, ICEL
• Progress in Transplantation
• Reliability Engineering and System Safety
• Renewable and Sustainable Energy Reviews
• Restoration Ecology
• Scientometrics
• Sex Education
• Small Group Research
• Solar Energy
• South African Computer Journal
• Speech, Language and Hearing
• Studies in Higher Education
• Sustainability (Switzerland)
• Teaching and Learning in Medicine
• Teaching and Teacher Education
• The Routledge Companion to Management Information Systems
• Tourism Management
• Tourism Management Perspectives
• Training and Education in Professional Psychology
• Transportation Research Record
• Urban Ecosystems
• Urban Forestry and Urban Greening
• Water Research
• Wildlife Research
• Wiley Interdisciplinary Reviews: Climate Change
• World Leisure Journal

Connect with us

If you would like to work, study or collaborate with us, get in touch

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 24, Issue 2
• Five tips for developing useful literature summary tables for writing review articles
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0003-0157-5319 Ahtisham Younas 1 , 2 ,
• http://orcid.org/0000-0002-7839-8130 Parveen Ali 3 , 4
• 1 Memorial University of Newfoundland , St John's , Newfoundland , Canada
• 2 Swat College of Nursing , Pakistan
• 3 School of Nursing and Midwifery , University of Sheffield , Sheffield , South Yorkshire , UK
• 4 Sheffield University Interpersonal Violence Research Group , Sheffield University , Sheffield , UK
• Correspondence to Ahtisham Younas, Memorial University of Newfoundland, St John's, NL A1C 5C4, Canada; ay6133{at}mun.ca

https://doi.org/10.1136/ebnurs-2021-103417

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Literature reviews offer a critical synthesis of empirical and theoretical literature to assess the strength of evidence, develop guidelines for practice and policymaking, and identify areas for future research. 1 It is often essential and usually the first task in any research endeavour, particularly in masters or doctoral level education. For effective data extraction and rigorous synthesis in reviews, the use of literature summary tables is of utmost importance. A literature summary table provides a synopsis of an included article. It succinctly presents its purpose, methods, findings and other relevant information pertinent to the review. The aim of developing these literature summary tables is to provide the reader with the information at one glance. Since there are multiple types of reviews (eg, systematic, integrative, scoping, critical and mixed methods) with distinct purposes and techniques, 2 there could be various approaches for developing literature summary tables making it a complex task specialty for the novice researchers or reviewers. Here, we offer five tips for authors of the review articles, relevant to all types of reviews, for creating useful and relevant literature summary tables. We also provide examples from our published reviews to illustrate how useful literature summary tables can be developed and what sort of information should be provided.

Tip 1: provide detailed information about frameworks and methods

• Download figure
• Open in new tab
• Download powerpoint

Tabular literature summaries from a scoping review. Source: Rasheed et al . 3

The provision of information about conceptual and theoretical frameworks and methods is useful for several reasons. First, in quantitative (reviews synthesising the results of quantitative studies) and mixed reviews (reviews synthesising the results of both qualitative and quantitative studies to address a mixed review question), it allows the readers to assess the congruence of the core findings and methods with the adapted framework and tested assumptions. In qualitative reviews (reviews synthesising results of qualitative studies), this information is beneficial for readers to recognise the underlying philosophical and paradigmatic stance of the authors of the included articles. For example, imagine the authors of an article, included in a review, used phenomenological inquiry for their research. In that case, the review authors and the readers of the review need to know what kind of (transcendental or hermeneutic) philosophical stance guided the inquiry. Review authors should, therefore, include the philosophical stance in their literature summary for the particular article. Second, information about frameworks and methods enables review authors and readers to judge the quality of the research, which allows for discerning the strengths and limitations of the article. For example, if authors of an included article intended to develop a new scale and test its psychometric properties. To achieve this aim, they used a convenience sample of 150 participants and performed exploratory (EFA) and confirmatory factor analysis (CFA) on the same sample. Such an approach would indicate a flawed methodology because EFA and CFA should not be conducted on the same sample. The review authors must include this information in their summary table. Omitting this information from a summary could lead to the inclusion of a flawed article in the review, thereby jeopardising the review’s rigour.

Tip 2: include strengths and limitations for each article

Critical appraisal of individual articles included in a review is crucial for increasing the rigour of the review. Despite using various templates for critical appraisal, authors often do not provide detailed information about each reviewed article’s strengths and limitations. Merely noting the quality score based on standardised critical appraisal templates is not adequate because the readers should be able to identify the reasons for assigning a weak or moderate rating. Many recent critical appraisal checklists (eg, Mixed Methods Appraisal Tool) discourage review authors from assigning a quality score and recommend noting the main strengths and limitations of included studies. It is also vital that methodological and conceptual limitations and strengths of the articles included in the review are provided because not all review articles include empirical research papers. Rather some review synthesises the theoretical aspects of articles. Providing information about conceptual limitations is also important for readers to judge the quality of foundations of the research. For example, if you included a mixed-methods study in the review, reporting the methodological and conceptual limitations about ‘integration’ is critical for evaluating the study’s strength. Suppose the authors only collected qualitative and quantitative data and did not state the intent and timing of integration. In that case, the strength of the study is weak. Integration only occurred at the levels of data collection. However, integration may not have occurred at the analysis, interpretation and reporting levels.

Tip 3: write conceptual contribution of each reviewed article

While reading and evaluating review papers, we have observed that many review authors only provide core results of the article included in a review and do not explain the conceptual contribution offered by the included article. We refer to conceptual contribution as a description of how the article’s key results contribute towards the development of potential codes, themes or subthemes, or emerging patterns that are reported as the review findings. For example, the authors of a review article noted that one of the research articles included in their review demonstrated the usefulness of case studies and reflective logs as strategies for fostering compassion in nursing students. The conceptual contribution of this research article could be that experiential learning is one way to teach compassion to nursing students, as supported by case studies and reflective logs. This conceptual contribution of the article should be mentioned in the literature summary table. Delineating each reviewed article’s conceptual contribution is particularly beneficial in qualitative reviews, mixed-methods reviews, and critical reviews that often focus on developing models and describing or explaining various phenomena. Figure 2 offers an example of a literature summary table. 4

Tabular literature summaries from a critical review. Source: Younas and Maddigan. 4

Tip 4: compose potential themes from each article during summary writing

While developing literature summary tables, many authors use themes or subthemes reported in the given articles as the key results of their own review. Such an approach prevents the review authors from understanding the article’s conceptual contribution, developing rigorous synthesis and drawing reasonable interpretations of results from an individual article. Ultimately, it affects the generation of novel review findings. For example, one of the articles about women’s healthcare-seeking behaviours in developing countries reported a theme ‘social-cultural determinants of health as precursors of delays’. Instead of using this theme as one of the review findings, the reviewers should read and interpret beyond the given description in an article, compare and contrast themes, findings from one article with findings and themes from another article to find similarities and differences and to understand and explain bigger picture for their readers. Therefore, while developing literature summary tables, think twice before using the predeveloped themes. Including your themes in the summary tables (see figure 1 ) demonstrates to the readers that a robust method of data extraction and synthesis has been followed.

Tip 5: create your personalised template for literature summaries

Often templates are available for data extraction and development of literature summary tables. The available templates may be in the form of a table, chart or a structured framework that extracts some essential information about every article. The commonly used information may include authors, purpose, methods, key results and quality scores. While extracting all relevant information is important, such templates should be tailored to meet the needs of the individuals’ review. For example, for a review about the effectiveness of healthcare interventions, a literature summary table must include information about the intervention, its type, content timing, duration, setting, effectiveness, negative consequences, and receivers and implementers’ experiences of its usage. Similarly, literature summary tables for articles included in a meta-synthesis must include information about the participants’ characteristics, research context and conceptual contribution of each reviewed article so as to help the reader make an informed decision about the usefulness or lack of usefulness of the individual article in the review and the whole review.

In conclusion, narrative or systematic reviews are almost always conducted as a part of any educational project (thesis or dissertation) or academic or clinical research. Literature reviews are the foundation of research on a given topic. Robust and high-quality reviews play an instrumental role in guiding research, practice and policymaking. However, the quality of reviews is also contingent on rigorous data extraction and synthesis, which require developing literature summaries. We have outlined five tips that could enhance the quality of the data extraction and synthesis process by developing useful literature summaries.

• Aromataris E ,
• Rasheed SP ,

Twitter @Ahtisham04, @parveenazamali

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• For authors
• New editors
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 58, Issue 8
• ‘You can change your life through sports’—physical activity interventions to improve the health and well-being of adults experiencing homelessness: a mixed-methods systematic review
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0003-0248-4160 Jo Dawes 1 ,
• Raphael Rogans-Watson 2 ,
• Julie Broderick 3
• 1 Department of Epidemiology & Public Health , UCL , London , UK
• 2 Elderly Medicine , University Hospitals Sussex NHS Foundation Trust , Worthing , UK
• 3 Discipline of Physiotherapy, School of Medicine , Trinity College Dublin , Dublin , Ireland
• Correspondence to Jo Dawes, Department of Epidemiology & Public Health, UCL, London, London, UK; joanna.dawes{at}ucl.ac.uk

Objectives Systematically synthesise evidence of physical activity interventions for people experiencing homelessness (PEH).

Design Mixed-methods systematic review.

Data sources EMBASE, Web of Science, CINAHL, PubMed (MEDLINE), PsycINFO, SPORTDiscus and Cochrane Library, searched from inception to October 2022.

Eligibility criteria PICO framework: population (quantitative/qualitative studies of PEH from high-income countries); intervention (physical activity); comparison (with/without comparator) and outcome (any health/well-being-related outcome). The risk of bias was assessed using Joanna Briggs Institute critical appraisal tools.

Results 3615 records were screened, generating 18 reports (17 studies, 11 qualitative and 6 quantitative (1 randomised controlled trial, 4 quasi-experimental, 1 analytical cross-sectional)) from the UK, USA, Denmark and Australia, including 554 participants (516 PEH, 38 staff). Interventions included soccer (n=7), group exercise (indoor (n=3), outdoor (n=5)) and individual activities (n=2). The risk of bias assessment found study quality to vary; with 6 being high, 6 moderate, 4 low and 1 very low. A mixed-methods synthesis identified physical and mental health benefits. Qualitative evidence highlighted benefits carried into wider life, the challenges of participating and the positive impact of physical activity on addiction. Qualitative and quantitative evidence was aligned demonstrating the mental health benefits of outdoor exercise and increased physical activity from indoor group exercise. Quantitative evidence also suggests improved musculoskeletal health, cardiovascular fitness, postural balance and blood lipid markers (p<0.05).

Conclusion Qualitative evidence suggests that physical activity interventions for PEH can benefit health and well-being with positive translation to wider life. There was limited positive quantitative evidence, although most was inconclusive. Although the evidence suggests a potential recommendation for physical activity interventions for PEH, results may not be transferable outside high-income countries. Further research is required to determine the effectiveness and optimal programme design.

• Physical activity
• Physical fitness
• Public health

Data availability statement

Data are available upon reasonable request.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bjsports-2023-107562

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

People experiencing homelessness suffer a higher burden of physical and mental health conditions than housed populations.

Limited studies suggest that regular physical activity may address many health conditions prevalent among people experiencing homelessness, although the evidence has not been systematically reviewed.

WHAT THIS STUDY ADDS

A variety of physical activity interventions have been designed and provided to engage people experiencing homelessness, including soccer, outdoor and indoor group activities, and individual activities.

The synthesis of qualitative and quantitative evidence suggests that physical activity can benefit the mental and physical health of people experiencing homelessness with positive translation of benefits to wider life.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

Group physical activity interventions seemed to be the most benefitical to people experiencing homelessness, perhaps due to its facilitation of social support and connection.

Qualitative data highlighted the pressure some participants felt in competetive tournament settings. Organisers should recognise this and consider support to ameliorate impacts of pressure experienced.

Consideration should be given to the intensity level of physical activity interventions for this population. Given the high prevalence and poor health of many people experiencing homelessness, lower threshhold activities are likely to be more inclusive for the population.

Introduction

Homelessness is an extreme form of social exclusion 1 2 related to poverty in high-income countries. 3 People experiencing homelessness (PEH) are defined as those who are ‘roofless’ (eg, no fixed abode) and ‘houseless’ (eg, living in hostel, shelter, temporary accommodation) in accordance with the European Federation of National Organisations Working with the Homeless. 4 Prior to the COVID-19 pandemic, homelessness in the UK had increased annually since 2010 5 with estimates of all categories of homelessness in England standing at 280 000 people, 6 of which 4266 were estimated to be sleeping on the streets. 7 The Organisation for Economic Co-operation and Development (OECD) estimates that almost 2 million people are experiencing homelessness in 35 OECD countries. 8

PEH have poorer health than the general population, 9 10 often characterised by a tri-morbidity of mental health diagnoses, chronic physical health conditions and addiction. 9 Poor health is thought to be both precipitated and exacerbated by poor living conditions, lack of resources, social exclusion, stigmatisation and difficulty accessing suitable health services. 11

Physical activity is beneficial for people with disabilities and chronic health conditions, both from a physical health and a social perspective. Guidance suggests that the type and amount of physical activity should be determined by a person’s abilities and the severity of their condition or disability, which may change over time. 12 PEH live with a high burden of physical deficits, 13 falls and frailty, 14 respiratory disease, cardiac problems, stroke and diabetes, 15 which could be positively influenced by physical activity. A recent scoping review found that among PEH, overall levels of physical activity appeared to be low, though the authors recognised that across studies reviewed, physical activity levels varied. 16 Low levels of physical activity could be due to limited opportunities or barriers to accessing physical activity, rather than through choice. Consequently, PEH may miss out on health gains and a reduced risk of harm that physical activity affords people with these conditions. It is important that this population has opportunities for physical activity to stabilise or reverse physical declines associated with homelessness. Given the multiple barriers PEH face accessing services, it may be important that physical activity interventions are specifically tailored to their needs to optimise reach and participation. This perspective is consistent with public engagement activities with PEH and staff who care for them, which took place prior to the commencement of this research. This research poses two research questions: what is the range of physical activity interventions provided to PEH? And, what is the evidence supporting the effectiveness of these interventions?

This review aims to summarise the available evidence for physical activity interventions intended to improve health outcomes of adults experiencing homelessness, focusing on physical activity interventions and their effectiveness in improving health outcomes.

A preliminary scoping review revealed that published literature in the field of physical activity for PEH comprised both quantitative and qualitative research. Therefore, a mixed-methods systematic review was adopted. This allowed for the findings of effectiveness (quantitative evidence) and participant experiences (qualitative evidence) to be brought together, to facilitate a broader understanding of whether and how interventions worked. 17 18 This systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) 2020 guidelines 19 and checklist, 20 and the protocol was registered a priori in PROSPERO database (reference number: CRD42020216716).

Identification

Defining search terms.

Initial search terms were generated by reviewers (JFD, RR-W and JB), who between them, have extensive clinical and research expertise and experience in the health of PEH, physical activity and systematic review methodology. The search terms were refined and tailored for a preliminary search of MEDLINE, and used to test the proof of concept and search strategy. The search syntax ( online supplemental file 1 ) was designed by a professional librarian in collaboration with two reviewers (JFD and RR-W).

Supplemental material

Search terms were refined, adapted and run in MEDLINE, EMBASE, Web of Science, CINAHL, PsycINFO, SPORTDiscus and the Cochrane Library. The searches were conducted on 17 February 2021, including literature from the previous 30 years (1991–2021) and restricted to English language only. The searches were re-run using the original search terms by a specialist librarian at Trinity College Dublin on 19 October 2022 to identify any new reports published between 21 October 2021 and 19 October 2022. All previous databases were searched, except SPORTDiscus, as it was unavailable in the institution’s library databases. Duplicates were removed at this stage. The reference lists of relevant systematic reviews and all included studies were hand-searched for reports to be added for screening. Corresponding authors of records that comprised an abstract only were contacted, where possible, to request full-text reports. Additionally, an expert reviewer suggested a study unidentified by searches, but met the inclusion criteria, so it was put forward for screening.

Title and abstract screening

On completion of the identification process, all report titles and abstracts were uploaded to the online systematic reviewing management system, Covidence. Two pairs of reviewers (JFD/RR-W and JFD/JB) independently performed (a) title and abstract screening and (b) full-text screening, judged against predetermined protocol criteria. In the event of disagreement, the third reviewer (JB or RR-W) was consulted for an additional opinion.

The PICO framework was used to identify inclusion criteria. For inclusion, all the following criteria were to be met:

Studies that included adults who were homeless under the European Typology of Homelessness and housing Exclusion (ETHOS) criteria for homelessness, 4 that is rooflessness, houselessness, living in insecure housing or living in inadequate housing. Age >18 years.

Intervention

Studies that included any physical activity intervention delivered as a stand-alone intervention or part of multimodal intervention, in any setting. Studies undertaken in high-income countries 21 were included, where there is assumed consistency in health and social care infrastructure as well as in family and community support systems, which impact how homelessness is perceived and managed. 22

This mixed-methods review included quantitative studies reporting any measures demonstrating health outcomes, including but not limited to primary measures such as cardiovascular fitness and strength, and qualitative findings describing participant perceptions linking physical activity intervention to health and/or well-being outcomes.

The presence of a comparison group was not required as an inclusion criterion.

Study types

This review considered quantitative, qualitative and mixed-methods studies.

Risk of bias assessment

In recognition of the diverse study designs included in this review, the Joanna Briggs Institute (JBI) critical appraisal tool portfolio was a key resource for judging quality and risk of bias. 23 24 These tools provide a criterion-based checklist for determining presence (yes), absence (no), a lack of clarity (unclear) or a lack of applicability (not applicable) of quality in studies across a variety of methods. 25 To determine the dependability and credibility of qualitative reports, their ConQual ratings were calculated. 26 Although Munn et al discourage cut-off values in determining the quality level in quantitative studies, for clarity and consistency of this mixed-methods review, a pragmatic decision was made to select cut-offs of <25% (very low), <50% (low), <75% (moderate) and >75% (high). Munn et al state that if cut-offs are preferred, these thresholds are best decided by the reviewers themselves. 25 A summary of the quality assessment of all reports is given in online supplemental file 2 .

Protocol deviation

This review was registered on PROSPERO, registration number: CRD42020216716. Found at https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=216716 . In the PROSPERO protocol, we stated we would use Cochrane and Downs and Black risk of bias tools. However, once the diversity of the final studies was identified, the review team recognised that JBI risk of bias tools were more suited to the studies within our review.

Data extraction

The following data were extracted to an excel spreadsheet: study design, inclusion criteria, participants (description, number, accommodation, age, education, employment, ethnicity, race, biological sex, mental health and physical health), intervention (setting, frequency, intensity, time, type, group or individual, presence of other non-physical activity intervention components), quantitative outcome measures and qualitative themes.

Initially, JFD carried out and collated data extraction from five reports. This was reviewed by RR-W and JB to ensure accuracy and consistency. Once all three team members agreed on the data extraction process, the remaining reports were divided among the team for completion of data extraction. Data from each report were checked for accuracy by another member of the research team. Any inconsistencies in interpretation or reporting were discussed, and consensus was reached.

Strategy for mixed-methods data synthesis

The synthesis followed the JBI methodology for mixed-methods systematic reviews, 27 whereby established convergent, segregated, results-based mixed-methods frameworks for systematic reviewing were employed. 28 29 First, qualitative and quantitative data were meaningfully categorised by JFD and JB, respectively. Each reviewer conducted their analysis separately, independently and concurrently. JFD adopted a reflexive thematic analysis approach to synthesise the qualitative data, by extracting all qualitative results into an excel spreadsheet and following the six processes of thematic analysis, namely: familiarisation; coding; generating initial themes; reviewing and developing themes; refining, defining and naming themes; and writing up. 30 Details of themes are outlined in online supplemental file 3 . Due to the heterogeneity of quantitative studies, it was not possible for JB to carry out a meta-analysis. So narrative synthesis was used. Quantitative findings were then ‘qualitized’ to transform them into a qualitative, descriptive format. Next, quantitative and qualitative evidence were linked and organised to produce an overall ‘configured analysis’ 27 and reported as a series of tables and combined narrative synthesis.

Equality and diversity statement

Our author and librarian team consisted of three women and two men. The author team included early and mid-career researchers and clinicians across two disciplines (medicine and physiotherapy) from two countries (UK and Ireland). This research explores physical activity interventions for PEH, an under-served, often marginalised and excluded population who experience extreme socioeconomic disadvantage. This population is known to have complex and chronic health needs and is an often-overlooked group in physical activity research.

Study selection

13 737 records were identified through searches. After the removal of duplicates (n=10 122), 3615 records were screened by title and abstract, with 3496 records excluded at this stage. 119 reports were sought for full-text review, 4 could not be found, so 115 full-text reports were reviewed. Of these, 97 records were excluded at this stage (exclusions based on: 1 duplicate, 9 population, 59 intervention, 8 non-English language, 19 insufficient data, 1 protocol only). Finally, 18 reports were included for quality checking. Two reports described different aspects of a single study. Therefore, data were extracted from 18 reports describing 17 studies. The full identification, screening and inclusion process are outlined in a PRISMA diagram ( figure 1 ).

• Download figure
• Open in new tab
• Download powerpoint

PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-analyses.

Quality assessment

The majority of the 11 qualitative studies were high quality, with 8 reporting at least 7 out of 10 quality criteria on the JBI checklist for qualitative studies ( online supplemental file 2 ). One study was of very low quality, 31 with only the statement of researcher positionality being clear, and all other criteria either unreported or unclear. Of the quantitative studies, the one randomised controlled trial (RCT) 32 was assessed as moderate quality due to methodological limitations, for example, lack of clarity regarding blinding of assessor and whether treatment groups were concealed. The analytical cross-sectional study was of moderate quality, and in general, quasi-experimental studies were of high quality.

Description of studies

Eighteen reports, describing 17 studies, were included ( table 1 ). Of these studies, 7 were from the USA, 5 from the UK, 3 from Denmark and 2 from Australia. The variety of designs across these studies comprised 11 qualitative and 6 quantitative reports (4 quasi-experimental, 1 RCT and 1 analytical cross-sectional). The interventions addressed varied, including soccer (n=7); group outdoor exercise (n=5); group indoor multimodal exercise (n=3) and individual multimodal interventions (n=2) ( online supplemental file 4 ).

• View inline

Summary of included studies

Study populations

Online supplemental file 5 provides detail of each study population included in this systematic review. Across the 17 studies, 516 PEH were participants. Some studies included women only (n=5), men only (n=5) or mixed cohorts (n=7). Three qualitative studies reported staff/coaches’ perspectives (n=38). The age range of participants who were homeless was 16–65 years. It was specified in the review protocol that only studies with participants >18 years would be included. However, for pragmatic reasons, several studies 33–37 were included despite containing participants from the age of 16 years. In these studies, proportions of participants <18 years were not specified, although one study 38 stated that the ‘majority’ of participants were between the ages of 20 and 24 years. Descriptions of study participants’ experiences of homelessness varied but were mainly focused on: street homeless, living in hostel/shelter, transitional/social service accommodation or ‘homeless at time of intervention’. Studies that focused on Street Soccer and the Homeless World Cup invited participation from PEH and other socially excluded groups, for example, people attending unemployment offices or drug rehabilitation services. Although these studies did not define proportions of participants experiencing homelessness, for pragmatic reasons they were included, as the intervention had been specifically designed for PEH. Only one study specified exclusion criteria, 32 which were based on reading ability and length of time staying in the shelter. In two studies, several participants were eligible but chose not to participate, 39 40 the reasons for which was not specified. The number of study drop-outs was described in three reports/two studies, 32 36 37 but the reasons for drop-out were not specified.

Physical activity interventions and their components

Online supplemental file 4 provides a description of all included interventions. The studies included seven soccer interventions (tournament focused (n=2), group training focused (n=3) and combining group training and tournament participation (n=2)); five group outdoor exercise (adventure training (n=3), running (n=1) and gardening (n=1)); three group indoor multimodal exercise (aerobic-based circuits (n=2) and dance (n=1)) and two individual multimodal interventions (pedometer with step goals and earn-a-bike scheme). Online supplemental file 4 also provides programming variables, including: setting; frequency; intensity; time; type and the presence of other non-physical activity components of multimodal interventions.

Seven studies investigated the impact of soccer for PEH. These studies (eight reports) explored soccer group training (n=4), 39 41–43 tournament participation (n=2) 33 40 and interventions of training for and participating in tournaments (n=2). 44 45 The studies involving tournaments were focused around national or international tournaments such as the Homeless World Cup or Street Soccer USA Cup. 33 40 44 45

Group outdoor exercise

Five studies provided evidence of the value of group outdoor exercise. These included group outdoor adventure (n=3), 34 35 46  women’s running groups (n=1) 38 and women’s gardening groups (n=1). 47 These studies described multimodal interventions, including outdoor adventure interventions which contained multiple activities (eg, archery, rock climbing, hiking), and all studies reported additional support, such as the provision of education, debriefing, opportunities for reflection, childcare, food or clothing.

Group indoor multimodal exercise

All group indoor multimodal exercise studies (n=3) were instructor-led interventions provided to small groups in settings such as leisure centres 36 or shelter recreation rooms. 31 All studies were multimodal as they combined different types of activity, for example, stretching, cardiovascular exercise, dance, aerobic circuits, strength-based exercise to music and meditation.

Individual multimodal interventions

Two studies reported interventions for individuals. 32 48 One involved participants wearing a pedometer and working towards a step goal. This was provided along with an educational newsletter and fruit/vegetable snacks. 32 The other study described cycle training to learn road safety and cycle maintenance, alongside earning a bicycle for individual use. 48

Intervention and outcomes

Findings are described across four tables ( tables 2–5 ). Table 2 shows all synthesised findings relating to mental health and table 3 shows all synthesised findings relating to physical health where the configured analysis identified qualitative and quantitative evidence supporting matched themes. Table 4 shows evidence that was identified in either quantitative or qualitative reports alone. For example, findings, where only quantitative data existed, were related to bone health and blood markers. Whereas qualitative evidence only was identified relating to other important aspects of physical activity, not specifically or directly health-related, such as the benefits carried into wider life, challenges of participation and addiction.

Summary of synthesised findings relating to mental health benefits of physical activity participation

Summary of synthesised findings relating to physical health benefits of physical activity participation

Outcomes where quantitative only or qualitative only findings exist, no mixed-methods synthesis

Summary of available evidence for physical activity interventions categorised by intervention type, findings and evidence quality

The impact of physical activity interventions on the mental health of PEH

There were several domains within mental health where both quantitative and qualitative evidence was synthesised, suggesting physical activity was beneficial (summarised in table 2 ). These included enhanced confidence, empowerment and self-esteem; resilience, coping and hope; independence, self-regulation and personal development; stress and anxiety; and mood and state of mind.

Enhanced confidence, empowerment and self-esteem

There was high quality qualitative evidence that group running, soccer and indoor group exercise, and moderate quality qualitative evidence that group outdoor adventure and earn-a-bike enhanced confidence, empowerment and self-esteem. However, the only quantitative study to assess outcomes in this domain used the Hope scale (agency subscale), finding no significant differences between groups. One soccer player suggested:

… Football gave me confidence and took away feelings of depression as it made me more social. 44

Resilience, coping and hope

There was high quality qualitative evidence that group running, and group outdoor adventure enhanced resilience, coping and hope. However, the only quantitative study to measure relevant outcomes using the Hope Scale (pathway domain) found no significant difference between intervention and control groups. A member of staff involved in delivering group outdoor adventure described changes in a participant’s ability to cope:

… when we went to Coniston, not even 10 min, we was there she wanted to come home, but she didn’t and she learned how to cope… she really enjoyed herself. 35

Independence, self-regulation and personal development

Qualitative evidence suggested that group running, and soccer (both high quality) and group outdoor adventure and earn-a-bike (both moderate quality) enhanced independence, self-regulation and personal development. This was supported by moderate quality quantitative evidence that outdoor adventure improved life functioning. An outdoor adventure participant describes how it impacted them:

when I leave here, I face any challenges… in my life, then I know that I will be able to do them because I’ve become a stronger person from coming here. 35

Stress and anxiety

There was high quality qualitative evidence that group running, indoor group exercise and outdoor adventure and moderate quality qualitative evidence that soccer and earn-a-bike had a positive effect on stress and anxiety. The studies that used quantitative measures to assess stress/anxiety in soccer (moderate quality) and indoor group exercise (low quality) did not conclusively support the qualitative evidence. A participant at a gym-based programme said:

I… didn’t have the confidence to go outside, I felt a lot of like anxiety and this, the gym and stuff helps me with my anxiety really well. 36

Mood and state of mind

There was high quality qualitative evidence that soccer, group running and indoor group exercise and moderate quality qualitative evidence that earn-a-bike enhanced mood and state of mind. This was supported by moderate quality quantitative evidence that group outdoor adventure improved well-being.

The impact of physical activity interventions on the physical health of PEH

Changes were shown in the following physical health domains: body shape and weight loss; fitness levels; physical skills development and physical activity levels. The synthesised findings are summarised in table 3 . Quantitative findings not corroborated by qualitative findings are summarised in table 4 .

Body shape and weight loss

Synthesised findings showed that indoor group exercise and group running (both high quality qualitative evidence) were perceived as improving body shape and facilitating weight loss, while soccer was shown to significantly decrease weight-bearing fat mass and total fat mass (high quality quantitative evidence).

I took my measurements when I started street fit, and I took my measurements now, and I’m a lot more buff. 36

Fitness levels

Synthesised findings for fitness levels showed that group running, group indoor training and earn-a-bike (all high-quality qualitative evidence) significantly improved fitness and endurance levels, a finding backed up by a high-quality quantitative study of soccer. A person who cycled with earn-a-bike described trying to increase fitness:

… after riding, you know, for an hour, two hours, and sometimes I’ll ride for four hours. You know, I really want to make sure that my body is fit. 48

Physical skill development

While moderate quality qualitative evidence for group outdoor adventure was suggestive of positive changes in physical skills development, the quantitative research exploring this domain through measuring postural balance showed no significant difference between intervention and control groups. However, when comparing pre to post values in the intervention group, postural balance improved by 39% (p=0.004) in the right leg and 45% (p=0.006) in the left leg.

Physical activity levels

Synthesised findings showed that group indoor exercise and running groups (both high quality qualitative evidence) and earn-a-bike (moderate qualitative evidence) positively influenced physical activity levels. This was supported by a moderate quality quantitative pedometer and set a step count study. A woman from a running group described how since joining the group she now runs on her own:

I feel so much more body confident … I can actually run for the whole session without nearly dying. I also go out for runs on my own and I definitely think I’ve got faster. 38

Bone health and cholesterol

A high quality study measured markers of bone health 41 and cholesterol levels 39 in PEH who played soccer. Although not all bone markers improved, increases in osteocalcin from pre-intervention to post-intervention were reported and this change was significantly different between controls and intervention groups. With regards to cholesterol markers (low-density lipoprotein-lipid (LDL)/high-density lipoprotein (HDL)) cholesterol was lowered and LDL:HDL ratios increased in the intervention group after 12 weeks of soccer—findings which were significantly different (p=0.05) from the control group.

Other considerations relevant to physical activity interventions for PEH

There were some findings relevant which described the impact of physical activity for PEH described in qualitative literature only. Themes include addiction, self-medication and medication; benefits carried into wider life and challenges to participation in physical activity when homeless (outlined in table 4 ).

Addiction, self-medication and medication

Across several qualitative studies of soccer (high quality) and earn-a-bike (moderate quality), physical activity positively influencing addiction was described. One person who played football stated:

I’m drinking less and do not think I need alcohol as much now… It’s great to feel this way and football is a focus for us. 44

Benefits of physical activity participation carried into wider life

Most of the qualitative studies, including soccer, running groups, earn-a-bike, outdoor adventure, gardening and dance, described benefits to wider life. Subthemes included: development of life and interpersonal skills, improved social connectedness and relationships with others, practical and functional benefits, and physical activity as a catalyst for positive healthy life change. A participant who undertook leisure centre-based group indoor training said:

I’ve noticed a massive improvement in my fitness, and it’s definitely keeping me motivated to live a healthy lifestyle, because you don’t put in all that hard work and then want to ruin it, you know what I mean? 36

… and similarly, how a participant of soccer described life change:

We can go back there and show that homelessness isn’t permanent and that you can change your life through sports. 33

Challenges to participation in physical activity when homeless

Qualitative evidence demonstrated the importance of acknowledging specific challenges related to physical activity PEH faced, which impacted uptake and dropout rates across a variety of interventions. Those who participated in soccer tournaments described heavy defeats impacting on self-worth. 44 Women who participated in running groups described lack of funds for transport or the unpredictability of homelessness as a barrier to attending. 38 There was also worry about loss of donated kit (eg, running clothes) 38 and equipment (eg, bicycle) 48 through theft and staff who led dance groups reported inconsistent attendance among shelter-dwellers. 31

An overall summary of available evidence for physical activity interventions categorised by intervention type, findings and evidence quality is provided in table 5 .

This review identified evidence for diverse physical activity interventions for PEH. The mixed-methods methodology enabled a meaningfully configured synthesis of the breadth of available evidence. This review demonstrated positive impacts of physical activity for PEH in relation to mental and physical health outcomes with translation of benefits to wider life.

Physical activity interventions were heterogeneous, grouped into broad categories of soccer, group outdoor exercise, group indoor multimodal exercise and individualised multimodal interventions. In terms of specific sports, soccer predominated (7/17). This is unsurprising considering its global resonance. 37 The mental health benefits of physical activity participation identified in our review align with research carried out in non-homeless populations, for example, the psychological state of ‘flow’ (where a person feels simultaneously cognitively efficient, motivated and happy) has been found to be increased by soccer training and running. 49 However, the majority (4/7) of soccer interventions included in our review included tournament participation. While benefits to tournament participation exist, negative experiences of pressure and fear of letting down teammates were qualitatively reported. Organisers of soccer tournaments for PEH should consider support to ameliorate impacts of possible pressure experienced, which could negatively impact mental health or self-management of addiction. Moreover, our review highlights that comparing the nuances of benefits and challenges of tournament participation and training warrants further research.

Group exercise appeared to be most beneficial for PEH. It is likely that group activities facilitated social support, which is especially pertinent for PEH whose social networks are often fragmented. 50 Configured qualitative and quantitative findings highlighted most evidence for mental health benefits in group outdoor exercise. Specifically, these benefits related to an improvement in mood and state of mind and increased independence, focus, personal development and ability to foster relationships. This may be related to emerging evidence for optimised benefits of outdoor exercise. 51 Corroboration of qualitative and quantitative evidence indicated that PEH who participated in physical activity interventions increased their physical activity levels. There is inherent difficulty comparing types of interventions for levels of benefit, as interventions and outcome measures were heterogeneous. Many physical activity interventions included additional intervention components such as counselling, food or sports kit. Consequently, it is not known if these additional components, enhanced or diluted the effect of physical activity. Moreover, descriptions of physical activity programme variables such as dosage were often lacking, limiting judgment of interventions.

Programme intensity deserves consideration. Soccer, which predominated, is a vigorous intensity sport (10 metabolic equivalent of task (METs) for competitive soccer and 7 METs for casual soccer) 52 so it is likely this high entry level may be exclusionary for some PEH. It should also be considered that some participants may be content to participate on the field while exerting minimal energy, so a diversity in intensity levels is also possible. Given the high prevalence and early manifestation of non-communicable diseases and poor general physical health in many PEH, 53 specifically focused lower threshold physical activity interventions should be also considered. Some low threshold programmes were identified such as gardening and dance. People designing physical activity interventions for PEH should consider a range of abilities and likely poor physical condition, perhaps offering a choice of low threshold activity, as well as higher intensity options, depending on ability and interest.

Qualitative studies dominated the evidence base, justifying the methodological decision of a mixed-methods review. The quality of evidence of most qualitative studies was judged to be high, with perspectives of staff enhancing credibility to the understanding of intervention impact. Significant changes were reported for the outcomes of weightbearing fat mass and overall fat mass in one soccer study, 41 although changes in muscle mass were not reported. Cardiovascular fitness and endurance also improved significantly in soccer studies. 39 42 While these findings were in small, uncontrolled studies, the implication of even minor changes to outcomes such as cardiovascular fitness and endurance may be of importance to PEH, as this group is significantly more likely than housed people to be hospitalised due to acute trauma. 54 Although not specifically explored in this population, it is likely that higher baseline fitness and strength levels may aid recovery post-hospitalisation, so multifaceted programmes addressing cardiovascular endurance and strength may be most beneficial for this population. A limitation of the evidence identified was that only one quantitative study was an RCT. While RCTs are considered the highest evidence level, this review attests to the usefulness of other study designs in this novel and emerging topic. It is acknowledged that RCTs may be especially difficult to undertake due to possible implementation barriers and complexities within this cohort. We propose that to build the evidence base, forms of controlled trials should be conducted where possible, with a view to including more randomised trials in the future. A further limitation was that feasibility outcomes such as adherence and retention rates were not well described, though challenges to participation were described in several qualitative studies. Feasibility analysis, including assessment of adherence and retention, should be included in future studies. Outcome measures employed were not consistent, for example, cardiovascular fitness was measured in three different ways: the Yo-Yo endurance test, cycle ergometry and maximal treadmill testing. The evidence base is limited in terms of the most suitable outcome measures 55 to use in physical activity interventions for PEH. Future studies should explore the most suitable outcome measures with a view to improving consistency in their use to enable future evidence syntheses.

Strengths of this review were its mixed-methods design and the global spread of identified research including studies from the UK and Europe, North America and Australia. However, only high-income countries were included, as low-income and middle-income settings were considered to have different structural influences on homelessness. So a limitation of this systematic review is that the translation of findings to other settings is not known. With regards to descriptions of exclusion criteria, included studies appeared to be pragmatic with minimal reporting of these. For example, no studies listed addiction status or gender diversity as pre-specified barriers to inclusion. Notably, most studies described the outcomes of ‘real world’ established programmes for PEH. In these cases, study eligibility criteria were dependent on those who engaged with the specific programme in the first instance, the eligibility criteria for which were not described, and were most likely self-selection. Only a small number of drop-outs were reported and there was minimal detail about their characteristics. We recognise that a level of stability in addiction and overall socioeconomic status is required to enable engagement in any type of physical activity intervention. Consequently, conclusions drawn from our review may not be applicable to the full diversity of PEH.

A final strength was that the review team capitalised on expertise in inclusion health, physical activity interventions and evidence synthesis with input from expert medical librarians. Studies were quality assessed using a consistent ‘family’ of critical analysis tools from JBI.

This mixed-methods systematic review demonstrates the value in exploring literature across a wide variety of methodological domains to gain insights into the existence and impact of a variety of physical activity interventions for PEH. To confidently inform policy, more research in this topic is required, however, from a practice and research perspective, our results provide initial justification for the inclusion of this typically under-represented group in targeted physical activity interventions with benefits to multiple aspects of physical and mental health, and positive translation into wider life demonstrated. Future research should include larger-scale high quality quantitative research to provide more robust evidence regarding objective impact.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

Acknowledgments.

The authors would like to thank Jacqui Smith, clinical librarian at UCL for sharing her extensive knowledge and supporting the team with their protocol design, searching strategy and carrying out the searches. Thanks also to and David Mockler, librarian, Trinity College Dublin, Dr Cliona Ni Cheallaigh and Professor Andrew Hayward for their advice and support of this work.

• Marmott M ,
• Wilkinson RG
• Van Straaten B ,
• Rodenburg G ,
• Van der Laan J , et al
• Fitzpatrick S ,
• Bramley G ,
• Ministry of Housing CLG
• Geddes JR ,
• Arnold EM ,
• Strenth CR ,
• Hedrick LP , et al
• Armstrong M ,
• Shulman C ,
• Hudson B , et al
• Piercy KL ,
• Troiano RP ,
• Ballard RM , et al
• Kiernan S ,
• Ní Cheallaigh C ,
• Murphy N , et al
• Rogans-Watson R ,
• Lewer D , et al
• Aldridge RW ,
• Menezes D , et al
• Mockler D ,
• Ní Cheallaigh C , et al
• Lizarondo L ,
• Carrier J , et al
• Bressan V ,
• Bagnasco A ,
• Aleo G , et al
• McKenzie JE ,
• Bossuyt PM , et al
• Mathew CM ,
• Mendonca O , et al
• Vardell E ,
• Barker TH ,
• Moola S , et al
• Porritt K ,
• Lockwood C , et al
• Lizarondo LSC ,
• Carrier J ,
• Godfrey C , et al
• Bujold M , et al
• Sandelowski M ,
• Knestaur M ,
• Devine MA ,
• Kendzor DE ,
• Allicock M ,
• Businelle MS , et al
• Peachey J ,
• Borland J , et al
• Norton CL ,
• Pelletier A , et al
• Quinton ML ,
• Holland MJG , et al
• Laird Y , et al
• Collins J ,
• Maughan RJ ,
• Gleeson M , et al
• Sanders C ,
• Randers MB ,
• Petersen J ,
• Andersen LJ , et al
• Marschall J ,
• Nielsen T-T , et al
• Hornstrup T , et al
• Petersen J , et al
• Strybosch V
• Thompson JL ,
• Goldstein A
• Strahler K ,
• Krustrup P , et al
• Tucker JS ,
• Golinelli D , et al
• Binsted G , et al
• Ainsworth BE ,
• Haskell WL ,
• Herrmann SD , et al
• Hwang SW , et al
• Silver CM ,
• Thomas AC ,
• Reddy S , et al
• Broderick J ,
• Bates ME , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1
• Data supplement 2
• Data supplement 3
• Data supplement 4
• Data supplement 5

Twitter @DawesJo

Contributors JFD, RR-W and JB all contributed to the planning, conduct and reporting of the work described in the manuscript. JFD is responsible for the overall content as guarantor. JFD and RR-W designed and contributed to the initial registration of the research and the identification of literature at the search stage. JFD, RR-W and JB all contributed to the screening, data extraction and reporting. All authors contributed to the writing up, review, editing and finalising of the manuscript.

Funding JFD was funded by a pre-doctoral fellowship from the National Institute for Health and Care Research (NIHR) School for Public Health Research (SPHR), Grant Reference Number PD-SPH-2015. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

• Open access
• Published: 11 April 2024

The role of champions in the implementation of technology in healthcare services: a systematic mixed studies review

• Sissel Pettersen 1 ,
• Hilde Eide 2 &
• Anita Berg 1  

BMC Health Services Research volume  24 , Article number:  456 ( 2024 ) Cite this article

Metrics details

Champions play a critical role in implementing technology within healthcare services. While prior studies have explored the presence and characteristics of champions, this review delves into the experiences of healthcare personnel holding champion roles, as well as the experiences of healthcare personnel interacting with them. By synthesizing existing knowledge, this review aims to inform decisions regarding the inclusion of champions as a strategy in technology implementation and guide healthcare personnel in these roles.

A systematic mixed studies review, covering qualitative, quantitative, or mixed designs, was conducted from September 2022 to March 2023. The search spanned Medline, Embase, CINAHL, and Scopus, focusing on studies published from 2012 onwards. The review centered on health personnel serving as champions in technology implementation within healthcare services. Quality assessments utilized the Mixed Methods Appraisal Tool (MMAT).

From 1629 screened studies, 23 were included. The champion role was often examined within the broader context of technology implementation. Limited studies explicitly explored experiences related to the champion role from both champions’ and health personnel’s perspectives. Champions emerged as promoters of technology, supporting its adoption. Success factors included anchoring and selection processes, champions’ expertise, and effective role performance.

The specific tasks and responsibilities assigned to champions differed across reviewed studies, highlighting that the role of champion is a broad one, dependent on the technology being implemented and the site implementing it. Findings indicated a correlation between champion experiences and organizational characteristics. The role’s firm anchoring within the organization is crucial. Limited evidence suggests that volunteering, hiring newly graduated health personnel, and having multiple champions can facilitate technology implementation. Existing studies predominantly focused on client health records and hospitals, emphasizing the need for broader research across healthcare services.

Conclusions

With a clear mandate, dedicated time, and proper training, health personnel in champion roles can significantly contribute professional, technological, and personal competencies to facilitate technology adoption within healthcare services. The review finds that the concept of champions is a broad one and finds varied definitions of the champion role concept. This underscores the importance of describing organizational characteristics, and highlights areas for future research to enhance technology implementation strategies in different healthcare settings with support of a champion.

Peer Review reports

Digital health technologies play a transformative role in healthcare service systems [ 1 , 2 ]. The utilization of technology and digitalization is essential for ensuring patient safety, delivering high quality, cost-effective, and sustainable healthcare services [ 3 , 4 ]. The implementation of technology in healthcare services is a complex process that demands systematic changes in roles, workflows, and service provision [ 5 , 6 ].

The successful implementation of new technologies in healthcare services relies on the adaptability of health professionals [ 7 , 8 , 9 ]. Champions have been identified as a key factor in the successful implementation of technology among health personnel [ 10 , 11 , 12 ]. However, they have rarely been studied as an independent strategy; instead, they are often part of a broader array of strategies in implementation studies (e.g., Hudson [ 13 ], Gullslett and Bergmo [ 14 ]). Prior research has frequently focused on determining the presence or absence of champions [ 10 , 12 , 15 ], as well as investigating the characteristics of individuals assuming the champion role (e.g., George et al. [ 16 ], Shea and Belden [ 17 ]).

Recent reviews on champions [ 18 , 19 , 20 ] have studied their effects on adherence to guidelines, implementation of innovations and facilitation of evidence-based practice. While these reviews suggest that having champions yields positive effects, they underscore the importance for studies that offer detailed insights into the champion’s role concerning specific types of interventions.

There is limited understanding of the practical role requirements and the actual experiences of health personnel in performing the champion role in the context of technology implementation within healthcare services. Further, this knowledge is needed to guide future research on the practical, professional, and relational prerequisites for health personnel in this role and for organizations to successfully employ champions as a strategy in technology implementation processes.

This review seeks to synthesize the existing empirical knowledge concerning the experiences of those in the champion role and the perspectives of health personnel involved in technology implementation processes. The aim is to contribute valuable insights that enhance our understanding of practical role requirements, the execution of the champion role, and best practices in this domain.

The term of champions varies [ 10 , 19 ] and there is a lack of explicit conceptualization of the term ‘champion’ in the implementation literature [ 12 , 18 ]. Various terms for individuals with similar roles also exist in the literature, such as implementation leader, opinion leader, facilitator, change agent, superuser and facilitator. For the purpose of this study, we have adopted the terminology utilized in the recent review by Rigby, Redley and Hutchinson [ 21 ] collectively referring to these roles as ‘champions’. This review aims to explore the experiences of health personnel in their role as champions and the experiences of health personnel interacting with them in the implementation of technology in the healthcare services.

Prior review studies on champions in healthcare services have employed various designs [ 10 , 18 , 19 , 20 ]. In this review, we utilized a comprehensive mixed studies search to identify relevant empirical studies [ 22 ]. The search was conducted utilizing the Preferred Reporting Items for Systematic and Meta-Analysis (PRISMA) guidelines, ensuring a transparent and comprehensive overview that can be replicated or updated by others [ 23 ]. The study protocol is registered in PROSPERO (ID CRD42022335750), providing a more comprehensive description of the methods [ 24 ]. A systematic mixed studies review, examining research using diverse study designs, is well-suited for synthesizing existing knowledge and identifying gaps by harnessing the strengths of both qualitative and quantitative methods [ 22 ]. Our search encompassed qualitative, quantitative, and mixed methods design to capture experiences with the role of champions in technology implementation.

Search strategy and study selection

Search strategy.

The first author, in collaboration with a librarian, developed the search strategy based on initial searches to identify appropriate terms and truncations that align with the eligibility criteria. The search was constructed utilizing a combination of MeSH terms and keywords related to technology, implementation, champion, and attitudes/experiences. Conducted in August/September 2022, the search encompassed four databases: Medline, Embase, CINAHL, and Scopus, with an updated search conducted in March 2023. The full search strategy for Medline is provided in Appendix  1 . The searches in Embase, CINAHL and Scopus employed the same strategy, with adopted terms and phrases to meet the requirements of each respective database.

Eligibility criteria

We included all empirical studies employing qualitative, quantitative, and mixed methods designs that detailed the experiences and/or attitudes of health personnel regarding the champions role in the implementation of technology in healthcare services. Articles in the English language published between 2012 and 2023 were considered. The selected studies involved technology implemented or adapted within healthcare services.

Conference abstract and review articles were excluded from consideration. Articles published prior 2012 were excluded as a result of the rapid development of technology, which could impact the experiences reported. Furthermore, articles involving surgical technology and pre-implementation studies were also excluded, as the focus was on capturing experiences and attitudes from the adoption and daily use of technology. The study also excluded articles that involved champions without clinical health care positions.

Study selection

A total of 1629 studies were identified and downloaded from the selected databases, with Covidence [ 25 ] utilized as a software platform for screening. After removing 624 duplicate records, all team members collaborated to calibrate the screening process utilizing the eligibility criteria on the initial 50 studies. Subsequently, the remaining abstracts were independently screened by two researchers, blinded to each other, to ensure adherence to the eligibility criteria. Studies were included if the title and abstract included the term champion or its synonyms, along with technology in healthcare services, implementation, and health personnel’s experiences or attitudes. Any discrepancies were resolved through consensus among all team members. A total of 949 abstracts were excluded for not meeting this inclusion condition. During the initial search, 56 remaining studies underwent full-text screening, resulting in identification of 22 studies qualified for review.

In the updated search covering the period September 2022 to March 2023, 64 new studies were identified. Of these, 18 studies underwent full-text screening, and one study was included in our review. The total number of included studies is 23. The PRISMA flowchart (Fig.  1 ) illustrates the process.

Flow Chart illustrating the study selection and screening process

Data extraction

The research team developed an extraction form for the included studies utilizing an Excel spreadsheet. Following data extraction, the information included the Name of Author(s) Year of publication, Country/countries, Title of the article, Setting, Aim, Design, Participants, and Sample size of the studies, Technology utilized in healthcare services, name/title utilized to describe the Champion Role, how the studies were analyzed and details of Attitude/Experience with the role of champion. Data extraction was conducted by SP, and the results were deliberated in a workshop with the other researchers AB, and HE until a consensus was reached. Any discrepancies were resolved through discussions. The data extraction was categorized into three categories: qualitative, quantitative, and mixed methods, in preparation for quality appraisal.

Quality appraisal

The MMAT [ 26 ] was employed to assess the quality of the 23 included studies. Specifically designed for mixed studies reviews, the MMAT allows for the appraisal of the methodological quality of studies falling into five categories. The studies in our review encompassed qualitative, quantitative descriptive, and mixed methods studies. The MMAT begins with two screening questions to confirm the empirical nature of this study. Subsequently, all studies were categorized by type and evaluated utilizing specific criteria based on their research methods, with ratings of ‘Yes,’ ‘No’ or ‘Can’t tell.’ The MMAT discourages overall scores in favor of providing a detailed explanation for each criterion. Consequently, we did not rely on the MMAT’s overall methodical quality scores and continued to include all 23 studies for our review. Two researchers independently scored the studies, and any discrepancies were discussed among all team members until a consensus was reached. The results of the MMAT assessments are provided in Appendix  2 .

Data synthesis

Based on discussions of this material, additional tables were formulated to present a comprehensive overview of the study characteristics categorized by study design, study settings, technology included, and descriptions/characteristics of the champion role. To capture attitudes and experiences associated with the champion role, the findings from the included studies were translated into narrative texts [ 22 ]. Subsequently, the reviewers worked collaboratively to conduct a thematic analysis, drawing inspiration from Braun and Clarke [ 27 ]. Throughout the synthesis process, multiple meetings were conducted to discern and define the emerging themes and subthemes.

The adopting of new technology in healthcare services can be perceived as both an event and a process. According to Iqbal [ 28 ], experience is defined as the knowledge and understanding gained after an event or the process of living through or undergoing an event. This review synthesizes existing empirical knowledge regarding the experiences of occupying the champion role, and the perspectives of health personnel interacting with champions in technology implementation processes.

Study characteristics

The review encompassed a total of 23 studies, and an overview of these studies is presented in Table  1 . Of these, fourteen studies employed a qualitative design, four had quantitative design, and five utilized a mixed method design. The geographical distribution revealed that the majority of studies were conducted in the USA (8), followed by Australia (5), England (4), Canada (2), Norway (2), Ireland (1), and Malaysia (1). In terms of settings, 11 studies were conducted in hospitals, five in primary health care, three in home-based care settings, and four in a mixed settings where two or more settings collaborated. Various technologies were employed across these studies, with client health records (7) and telemedicine (5) being the most frequently utilized. All studies included experiences from champions or health personnel collaborating with champions in their respective healthcare services. Only three studies had the champion role as a main objective [ 29 , 30 , 31 ]. The remaining studies described champions as one of the strategies in technology implementation processes, including 10 evaluation studies (including feasibility studies [ 32 , 33 , 34 ] and one cost-benefit study [ 30 ]).

Several studies underscored the importance of champions for successful implementation [ 29 , 30 , 31 , 34 , 35 , 36 , 37 , 38 , 40 , 41 , 42 , 43 , 49 ]. Four studies specifically highlighted champions as a key factor for success [ 34 , 36 , 37 , 43 ], and one study went further to describe champions as the most important factor for successful implementation [ 39 ]. Additionally, one study associated champions with reduced labor cost [ 30 ].

Thin descriptions, yet clear expectations for technology champions’ role and -attributes

The analyses revealed that the concept of champions in studies pertaining to technology implementation in healthcare services varies, primarily as a result of the diversity of terms utilized to describe the role combined with short role descriptions. Nevertheless, the studies indicated clear expectations for the champion’s role and associated attributes.

The term champion

The term champion was expressed in 20 different forms across the 23 studies included in our review. Three studies utilized multiple terms within the same study [ 32 , 47 , 48 ] and 15 different authors [ 29 , 32 , 33 , 35 , 36 , 37 , 39 , 40 , 41 , 42 , 43 , 44 , 46 , 47 , 50 ] employed the term with different compositions (Table  1 ). Furthermore, four authors utilized the term Super user [ 30 , 31 , 49 , 51 ], while four authors employed the terms Facilitator [ 38 ], IT clinician [ 48 ], Leader [ 45 ], and Manager [ 34 ], each in combination with more specific terms (such as local opinion leaders, IT nurse, or practice manager).

Most studies associated champion roles with specific professions. In seven studies, the professional title was explicitly linked to the concept of champions, such as physician champions or clinical nurse champions, or through the strategic selection of specific professions [ 29 , 33 , 36 , 40 , 43 , 47 , 50 ]. Additionally, some studies did not specify professions, but utilized terms like clinicians [ 45 ] or health professionals [ 41 ].

All included articles portray the champion’s role as facilitating implementation and daily use of technology among staff. In four studies, the champion’s role was not elaborated beyond indicating that the individual holding the role is confident with an interest in technology [ 35 , 41 , 42 , 44 ]. The champion’s role was explicitly examined in six studies [ 29 , 30 , 31 , 33 , 46 , 50 ]. Furthermore, seven studies described the champion in both the methods and results [ 32 , 36 , 38 , 47 , 48 , 49 , 51 ]. In ten of the studies, champions were solely mentioned in the results [ 34 , 35 , 37 , 39 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ].

Eight studies provided a specific description or definition of the champion [ 29 , 30 , 31 , 32 , 38 , 48 , 49 , 50 ]. The champion’s role was described as involving training in the specific technology, being an expert on the technology, providing support and assisting peers when needed. In some instance, the champion had a role in leading the implementation [ 50 ], while in other situations, the champion operated as a mediator [ 48 ].

The champions tasks

In the included studies, the champion role encompassed two interrelated facilitators tasks: promoting the technology and supporting others in adopting the technology in their daily practice. Promoting the technology involved encouraging staff adaptation [ 32 , 34 , 35 , 37 , 40 , 41 , 49 ], generally described as being enthusiastic about the technology [ 32 , 35 , 37 , 41 , 48 ], influencing the attitudes and beliefs of colleagues [ 42 , 45 ] and legitimizing the introduction of the technology [ 42 , 46 , 48 ]. Supporting others in technology adaption involved training and teaching [ 31 , 35 , 38 , 40 , 51 ], as well as providing technical support [ 30 , 31 , 39 , 43 , 49 ] and social support [ 49 ]. Only four studies reported that the champions received their own training to enable them able to support their colleagues [ 30 , 31 , 39 , 48 ]. Furthermore, eight studies [ 32 , 34 , 38 , 40 , 48 , 49 , 50 , 51 ], specified that the champion role included leadership and management responsibilities, mentioning tasks such as planning, organizing, coordinating, and mediating technology adaption without providing further details.

Desirable champion attributes

To effectively fulfill their role, champions should ideally possess clinical expertise and experience [ 29 , 35 , 38 , 40 , 48 ], stay professionally updated [ 37 , 48 ], and possess knowledge of the organization and workflows [ 29 , 34 , 46 ]. They should have the ability to understand and communicate effectively with healthcare personnel [ 31 , 32 , 46 , 49 ] and be proficient in IT language [ 51 ]. Moreover, champions should demonstrate a general technological interest and competence, and competence, along with specific knowledge of the technology to be implemented [ 32 , 37 , 49 ]. It is also emphasized that they should command formal and/or informal respect and authority in the organization [ 36 , 45 ], be accessible to others [ 39 , 43 ], possess leadership qualities [ 34 , 37 , 38 , 46 ], and understand and balance the needs of stakeholders [ 43 ]. Lastly, the champions should be enthusiastic promoters of the technology, engaging and supporting others [ 31 , 32 , 33 , 34 , 37 , 39 , 40 , 41 , 43 , 49 ], while also effectively coping with cultural resistance to change [ 31 , 46 ].

Anchoring and recruiting for the champion role

The champions were organized differently within services, holding various positions in the organizations, and being recruited for the role in different ways.

Anchoring the champion role

The champion’s role is primarily anchored at two levels: the management level and/or the clinical level, with two studies having champions at both levels [ 34 , 49 ]. Those working with the management actively participated in the planning of the technology implementation [ 29 , 36 , 40 , 41 , 45 ]. Serving as advisors to management, they leveraged their clinical knowledge to guide the implementation in alignment with the necessities and possibilities of daily work routines in the clinics. Champions in this capacity experienced having a clear formal position that enabled them to fulfil their role effectively [ 29 , 40 ]. Moreover, these champions served as bridge builders between the management and department levels [ 36 , 45 ], ensuring the necessary flow of information in both directions.

Champions anchored at the clinic level played a pivotal role in the practical implementation and facilitation of the daily use of technology [ 31 , 33 , 35 , 37 , 38 , 43 , 48 , 51 ]. Additionally, these champions actively participated in meetings with senior management to discuss the technology and its implementation in the clinic. This position conferred potential influence over health personnel [ 33 , 35 ]. Champions at the clinic level facilitated collaboration between employees, management, and suppliers [ 48 ]. Fontaine et al. [ 36 ] identified respected champions at the clinical level, possessing authority and formal support from all leadership levels, as the most important factor for success.

Only one study reported that the champions received additional compensation for their role [ 36 ], while another study mentioned champions having dedicated time to fulfil their role [ 46 ]. The remaining studies did not provide this information.

Recruiting for the role as champion

Several studies have reported different experiences regarding the management’s selection of champions. A study highlighted the distinctions between a volunteered role and an appointed champion’s role [ 31 ]. Some studies underscored that appointed champions were chosen based on technological expertise and skills [ 41 , 48 , 51 ]. Moreover, the selection criteria included champions’ interest in the specific technology [ 42 ] or experiential skills [ 40 ]. The remaining studies did not provide this information.

While the champion role was most frequently held by health personnel with clinical experience, one study deviated by hiring 150 newly qualified nurses as champions [ 30 ] for a large-scale implementation of an Electronic Health Record (EHR). Opting for clinical novices assisted in reducing implementation costs, as it avoided disrupting daily tasks and interfering with daily operations. According to Bullard [ 30 ], these super-user nurses became highly sought after post-implementation as a result of their technological confidence and competence.

Reported experiences of champions and health personnel

Drawing from the experiences of both champions and health personnel, it is essential for a champion to possess a combination of general knowledge and specific champion characteristics. Furthermore, champions are required to collaborate with individuals both within and outside the organization. The subsequent paragraphs delineate these experiences, categorizing them into four subsets: champions’ contextual knowledge and expertise, preferred performance of the champion role, recognizing that a champion alone is insufficient, and distinguishing between reactive and proactive champions.

Champions’ contextual knowledge and know-how

Health personnel with experience interacting with champions emphasized that a champion must be familiar with the department and its daily work routines [ 35 , 40 ]. Knowledge of the department’s daily routines made it easier for champions to facilitate the adaptation of technology. However, there was a divergence of opinions on whether champions were required to possess extensive clinical experience to fulfil their role. In most studies, having an experienced and competent clinician as a champion instilled a sense of confidence among health personnel. Conversely, Bullard’s study [ 30 ] exhibited that health personnel were satisfied with newly qualified nurses in the role of champion, despite their initial skepticism.

It is a generally expected that champions should possess technological knowledge beyond that of other health professionals [ 37 , 41 ]. Some health personnel perceived the champions as uncritical promoters of technology, with the impression that health personnel were being compelled to utilize technology [ 46 ]. Champions could also overestimate the readiness of health personnel to implement a technology, especially during the early phases of the implementation process [ 32 ]. Regardless of whether the champion is at the management level or the clinic level, champions themselves have acknowledged the importance of providing time and space for innovation. Moreover, the recruitment of champions should span all levels of the organization [ 34 , 46 ]. Furthermore, champions must be familiar with daily work routines, work tools, and work surfaces [ 38 , 40 , 43 ].

Preferable performance of the champion role

The studies identified several preferable characteristics of successful champions. Health personnel favored champions utilizing positive words when discussing technology and exhibiting positive attitudes while facilitating and adapting it [ 33 , 34 , 37 , 38 , 41 , 46 ]. Additionally, champions who were enthusiastic and engaging were considered good role models for the adoption of technology. Successful champions were perceived as knowledgeable and adept problem solvers who motivated and supported health personnel [ 41 , 43 , 44 , 48 ]. They were also valued for being available and responding promptly when contacted [ 42 ]. Health professionals noted that champions perceived as competent garnered respect in the organization [ 40 ]. Moreover, some health personnel felt that some certain champions wielded a greater influence based on how they encouraged the use of the system [ 48 ]. It was also emphasized that health personnel needed to feel it was safe to provide feedback to champions, especially when encountering difficulties or uncertainties [ 49 ].

A champion is not enough

The role of champions proved to be more demanding than expected [ 29 , 31 , 38 ], involving tasks such as handling an overwhelming number of questions or actively participating in the installation process to ensure the technology functions effectively in the department [ 29 ]. Regardless of the organizational characteristics or the champion’s profile, appointing the champion as a “solo implementation agent” is deemed unsuitable. If the organization begins with one champion, it is recommended that this individual promptly recruits others into the role [ 42 ].

Health personnel, reliant on champions’ expertise, found it beneficial to have champions in all departments, and these champions had to be actively engaged in day-to-day operations [ 31 , 33 , 34 , 37 ]. Champions themselves also noted that health personnel increased their technological expertise through their role as champions in the department [ 39 ].

Furthermore, the successful implementation of technology requires the collaboration of various professions and support functions, a task that cannot be solely addressed by a champion [ 29 , 43 , 48 ]. In Orchard et. al.‘s study [ 34 ], champions explicitly emphasized the necessity of support from other personnel in the organization, such as those responsible for the technical aspects and archiving routines, to provide essential assistance.

According to health personnel, the role of champions is vulnerable in case they become sick or leave their position [ 42 , 51 ]. In some of the included studies, only one or a few hold the position of champion [ 37 , 38 , 42 , 48 ]. Two studies observed that their implementations were not completed because champions left or reassigned for various reasons [ 32 , 51 ]. The health professionals in the study by Owens and Charles [ 32 ] expressed that champions must be replaced in such cases. Further, the study of Olsen et al., 2021 [ 42 ] highlights the need for quicky building a champion network within the organization.

Reactive and proactive champions

Health personnel and champions alike noted that champions played both a reactive and proactive role. The proactive role entailed facilitating measures such as training and coordination [ 31 , 32 , 33 , 34 , 37 , 39 , 40 , 41 , 43 , 48 , 49 ] as initiatives to generate enthusiasm for the technology [ 31 , 32 , 33 , 34 , 35 , 37 , 39 , 40 , 41 , 43 , 49 ]. On the other hand, the reactive role entailed hands-on support and troubleshooting [ 30 , 31 , 39 , 43 , 49 ].

In a study presenting experiences from both health personnel and champions, Yuan et al. [ 31 ] found that personnel observed differences in the assistance provided by appointed and self-chosen champions. Appointed champions demonstrated the technology, answered questions from health personnel, but quickly lost patience and track of employees who had received training [ 31 ]. Health personnel perceived that self-chosen champions were proactive and well-prepared to facilitate the utilization of technology, communicating with the staff as a group and being more competent in utilizing the technology in daily practice [ 31 ]. Health personnel also noted that volunteer champions were supportive, positive, and proactive in promoting the technology, whereas appointed champions acted on request and had a more reactive approach [ 31 ].

This review underscores the breadth of the concept of champion and the significant variation in the champion’s role in implementation of technology in healthcare services. This finding supports the results from previous reviews [ 10 , 18 , 19 , 20 ]. The majority of studies meeting our inclusion criteria did not specifically focus on the experiences of champions and health personnel regarding the champion role, with the exception of studies by Bullard [ 30 ], Gui et al. [ 29 ], Helmer-Smith et al. [ 33 ], Hogan-Murphy et al. [ 46 ], Rea et al. [ 50 ], and Yuan et al. [ 31 ].

The 23 studies encompassed in this review utilized 20 different terms for the champion role. In most studies, the champion’s role was briefly described in terms of the duties it entailed or should entail. This may be linked to the fact that the role of champions was not the primary focus of the study, but rather one of the strategies in the implementation process being investigated. This result reinforces the conclusions drawn by Miech et al. [ 10 ] and Shea et al. [ 12 ] regarding the lack of united understandings of the concept. Furthermore, in Santos et al.‘s [ 19 ] review, champions were only operationalized through presence or absence in 71.4% of the included studies. However, our review finds that there is a consistent and shared understanding that champions should promote and support technology implementation.

Several studies advocate for champions as an effective and recommended strategy for implementing technology [ 30 , 31 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 42 , 43 , 45 , 46 ]. However, we identified that few studies exclusively explore health personnel`s experiences within the champion role when implementing technology in healthcare services.

This suggests a general lack of information essential for understanding the pros, cons, and prerequisites for champions as a strategy within this field of knowledge. However, this review identifies, on a general basis, the types of support and structures required for champions to perform their role successfully from the perspectives of health personnel, contributing to Shea’s conceptual model [ 12 ].

Regarding the organization of the role, this review identified champions holding both formal appointed and informal roles, working in management or clinical settings, being recruited for their clinical and/or technological expertise, and either volunteering or being hired with specific benefits for the role. Regardless of these variations, anchoring the role is crucial for both the individuals holding the champion role and the health personnel interacting with them. Anchoring, in this context, is associated with the clarity of the role’s content and a match between role expectations and opportunities for fulfilment. Furthermore, the role should be valued by the management, preferably through dedicated time and/or salary support [ 34 , 36 , 46 ]. Additionally, our findings indicate that relying on a “solo champion” is vulnerable to issues such as illness, turnover, excessive workload, and individual champion performance [ 32 , 37 ]. Based on these insights, it appears preferable to appoint multiple champions, with roles at both management and clinical levels [ 33 ].

Some studies have explored the selection of champions and its impact on role performance, revealing diverse experiences [ 30 , 31 ]. Notably, Bullard [ 30 ], stands out for emphasizing long clinical experience, and hiring newly trained nurses as superusers to facilitate the use of electronic health records. Despite facing initial reluctance, these newly trained nurses gradually succeeded in their roles. This underscores the importance of considering contextual factors in the champion selection [ 30 , 52 ]. In Bullard’s study [ 30 ], the collaboration between newly trained nurses as digital natives and clinical experienced health personnel proved beneficial, highlighting the need to align champion selection with the organization’s needs based on personal characteristics. This finding aligns with Melkas et al.‘s [ 9 ] argument that implementing technology requires a deeper understanding of users, access to contextual know-how, and health personnel’s tacit knowledge.

To meet role expectations and effectively leverage their professional and technological expertise, champions should embody personal qualities such as the ability to engage others, take a leadership role, be accessible, supportive, and communicate clearly. These qualities align with the key attributes for change in healthcare champions described by Bonawitz et al. [ 15 ]. These attributes include influence, ownership, physical presence, persuasiveness, grit, and a participative leadership style (p.5). These findings suggest that the active performance of the role, beyond mere presence, is crucial for champions to be a successful strategy in technology implementation. Moreover, the recruitment process is not inconsequential. Identifying the right person for the role and providing them with adequate training, organizational support, and dedicated time to fulfill their responsibilities emerge as an important factor based on the insights from champions and health personnel.

Strengths and limitations

While this study benefits from identifying various terms associated with the role of champions, it acknowledges the possibility of missing some studies as a result of diverse descriptions of the role. Nonetheless, a notable strength of the study lies in its specific focus on the health personnel’s experiences in holding the champion role and the broader experiences of health personnel concerning champions in technology implementation within healthcare services. This approach contributes valuable insights into the characteristics of experiences and attitudes toward the role of champions in implementing technology. Lastly, the study emphasizes the relationship between the experiences with the champion role and the organizational setting’s characteristics.

The champion role was frequently inadequately defined [ 30 , 33 , 34 , 35 , 36 , 37 , 39 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 51 ], aligning with previous reviews [ 17 , 19 , 21 ]. As indicated by van Laere and Aggestam [ 52 ], this lack of clarity complicates the identification and comparison of champions across studies. Studies that lacking a distinct definition of the champion’s role were consequently excluded. Only studies written in English were included, introducing the possibility of overlooking relevant studies based on our chosen terms for identifying the champion’s role. Most of the included studies focused on technology implementation in a general context, with champions being just one of several measures. This approach resulted in scant descriptions, as champions were often discussed in the results, discussion, or implications sections rather than being the central focus of the research.

As highlighted by Hall et al. [ 18 ]., methodological issues and inadequate reporting in studies of the champion role create challenges for conducting high-quality reviews, introducing uncertainty around the findings. We have adopted a similar approach to Santos et al. [ 19 ], including all studies even when some issues were identified during the quality assessment. Our review shares the same limitations as previous review by Santos et al. [ 19 ] on the champion role.

Practical implications, policy, and future research

The findings emphasize the significance of the relationship between experiences with the champion role and characteristics of organizational settings as crucial factors for success in the champion role. Clear anchoring of the role within the organization is vital and may impact routines, workflows, staffing, and budgets. Despite limited evidence on the experience of the champion’s role, volunteering, hiring newly graduated health personnel, and appointing more than one champion are identified as facilitators of technology implementation. This study underscores the need for future empirical research including clear descriptions of the champion roles, details on study settings and the technologies to be adopted. This will enable the determination of outcomes and success factors in holding champions in technology implementation processes, transferability of knowledge between contexts and technologies as well as enhance the comparability of studies. Furthermore, there is a need for studies to explore experiences with the champion role, preferably from the perspective of multiple stakeholders, as well as focus on the champion role within various healthcare settings.

This study emphasizes that champions can hold significant positions when provided with a clear mandate, dedicated time, and training, contributing their professional, technological, and personal competencies to expedite technology adoption within services. It appears to be an advantage if the health personnel volunteer or apply for the role to facilitate engaged and proactive champions. The implementation of technology in healthcare services demands efforts from the entire service, and the experiences highlighted in this review exhibits that champions can play an important role. Consequently, empirical studies dedicated to the champion role, employing robust designs based current knowledge, are still needed to provide solid understanding of how champions can be a successful initiative when implementing technology in healthcare services.

Data availability

This review relies exclusively on previously published studies. The datasets supporting the conclusions of this article are included within the article and its supplementary files: Description and characteristics of included studies in Table  1 , Study characteristics. The search strategy is provided in Appendix  1 , and the Critical Appraisal Summary of included studies utilizing MMAT is presented in Appendix  2 .

Abbreviations

Electronic Health Record

Implementation Outcomes Framework

Preferred Reporting Items for Systematics and Meta-Analysis

Meskó B, Drobni Z, Bényei É, Gergely B, Győrffy Z. Digital health is a cultural transformation of traditional healthcare. mHealth. 2017;3:38. https://doi.org/10.21037/mhealth.2017.08.07 .

Article   PubMed   PubMed Central   Google Scholar  

Pérez Sust P, Solans O, Fajardo JC, Medina Peralta M, Rodenas P, Gabaldà J, et al. Turning the crisis into an opportunity: Digital health strategies deployed during the COVID-19 outbreak. JMIR Public Health Surveill. 2020;6:e19106. https://doi.org/10.2196/19106 .

Alotaibi YK, Federico F. The impact of health information technology on patient safety. Saudi MedJ. 2017;38:117380. https://doi.org/10.15537/smj.2017.12.20631 .

Article   Google Scholar  

Kuoppamäki S. The application and deployment of welfare technology in Swedish municipal care: a qualitative study of procurement practices among municipal actors. BMC Health Serv Res. 2021;21:918. https://doi.org/10.1186/s12913-021-06944-w .

Kraus S, Schiavone F, Pluzhnikova A, Invernizzi AC. Digital transformation in healthcare: analyzing the current state-of-research. J Bus Res. 2021;123:557–67. https://doi.org/10.1016/j.jbusres.2020.10.030 .

Frennert S. Approaches to welfare technology in municipal eldercare. JTechnolHum. 2020;38:22646. https://doi.org/10.1080/15228835.2020.1747043 .

Konttila J, Siira H, Kyngäs H, Lahtinen M, Elo S, Kääriäinen M, et al. Healthcare professionals’ competence in digitalisation: a systematic review. Clin Nurs. 2019;28:74561. https://doi.org/10.1111/jocn.14710 .

Jacob C, Sanchez-Vazquez A, Ivory C. Social, organizational, and technological factors impacting clinicians’ adoption of mobile health tools: systematic literature review. JMIR mHealth uHealth. 2020;8:e15935. https://doi.org/10.2196/15935 .

Melkas H, Hennala L, Pekkarinen S, Kyrki V. Impacts of robot implementation on care personnel and clients in elderly-care institutions. Int J Med Inf. 2020;134:104041. https://doi.org/10.1016/j.ijmedinf.2019.104041 .

Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: an integrative review of champions in healthcare-related implementation. SAGE Open Med. 2018;6. https://doi.org/10.1177/2050312118773261 .

Foong HF, Kyaw BM, Upton Z, Tudor Car L. Facilitators and barriers of using digital technology for the management of diabetic foot ulcers: a qualitative systematic review. Int Wound J. 2020;17:126681. https://doi.org/10.1111/iwj.13396 .

Shea CM. A conceptual model to guide research on the activities and effects of innovation champions. Implement Res Pract. 2021;2. https://doi.org/10.1177/2633489521990443 .

Hudson D. Physician engagement strategies in health information system implementations. Healthc Manage Forum. 2023;36:86–9. https://doi.org/10.1177/08404704221131921 .

Article   PubMed   Google Scholar  

Gullslett MK, Strand Bergmo T. Implementation of E-prescription for multidose dispensed drugs: qualitative study of general practitioners’’ experiences. JMIR HumFactors. 2022;9:e27431. https://doi.org/10.2196/27431 .

Bonawitz K, Wetmore M, Heisler M, Dalton VK, Damschroder LJ, Forman J, et al. Champions in context: which attributes matter for change efforts in healthcare? Implement Sci. 2020;15:62. https://doi.org/10.1186/s13012-020-01024-9 .

George ER, Sabin LL, Elliott PA, Wolff JA, Osani MC, McSwiggan Hong J, et al. Examining health care champions: a mixed-methods study exploring self and peer perspectives of champions. Implement Res Pract. 2022;3. https://doi.org/10.1177/26334895221077880 .

Shea CM, Belden CM. What is the extent of research on the characteristics, behaviors, and impacts of health information technology champions? A scoping review. BMC Med Inf Decis Mak. 2016;16:2. https://doi.org/10.1186/s12911-016-0240-4 .

Hall AM, Flodgren GM, Richmond HL, Welsh S, Thompson JY, Furlong BM, Sherriff A. Champions for improved adherence to guidelines in long-term care homes: a systematic review. Implement Sci Commun. 2021;2(1):85–85. https://doi.org/10.1186/s43058-021-00185-y .

Santos WJ, Graham ID, Lalonde M, Demery Varin M, Squires JE. The effectiveness of champions in implementing innovations in health care: a systematic review. Implement Sci Commun. 2022;3(1):1–80. https://doi.org/10.1186/s43058-022-00315-0 .

Wood K, Giannopoulos V, Louie E, Baillie A, Uribe G, Lee KS, Haber PS, Morley KC. The role of clinical champions in facilitating the use of evidence-based practice in drug and alcohol and mental health settings: a systematic review. Implement Res Pract. 2020;1:2633489520959072–2633489520959072. https://doi.org/10.1177/2633489520959072 .

Rigby K, Redley B, Hutchinson AM. Change agent’s role in facilitating use of technology in residential aged care: a systematic review. Int J Med Informatics. 2023;105216. https://doi.org/10.1016/j.ijmedinf.2023.105216 .

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45. https://doi.org/10.1146/annurev-publhealth-032013-182440 .

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71 .

Pettersen S, Berg A, Eide H. Experiences and attitudes to the role of champions in implementation of technology in health services. A systematic review. PROSPERO. 2022. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022335750 . Accessed [15 Feb 2023].

Covidence. Better Syst Rev Manag. https://www.covidence.org/ . 2023;26.

Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The mixed methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Inf. 2018;34:285–91. https://doi.org/10.3233/EFI-180221 .

Braun V, Clarke V. Thematic analysis: a practical guide. 1st ed. SAGE; 2022.

Iqbal MP, Manias E, Mimmo L, Mears S, Jack B, Hay L, Harrison R. Clinicians’ experience of providing care: a rapid review. BMC Health Serv Res. 2020;20:1–10. https://doi.org/10.1186/s12913-020-05812-3 .

Gui X, Chen Y, Zhou X, Reynolds TL, Zheng K, Hanauer DA. Physician champions’ perspectives and practices on electronic health records implementation: challenges and strategies. JAMIA open. 2020;3:5361. https://doi.org/10.1093/jamiaopen/ooz051 .

Bullard KL. Cost effective staffing for an EHR implementation. Nurs Econ. 2016;34:726.

Google Scholar  

Yuan CT, Bradley EH, Nembhard IM. A mixed methods study of how clinician ‘super users’ influence others during the implementation of electronic health records. BMC Med Inf Decis Mak. 2015;15:26. https://doi.org/10.1186/s12911-015-0154-6 .

Owens C, Charles N. Implementation of a text-messaging intervention for adolescents who self-harm (TeenTEXT): a feasibility study using normalisation process theory. Child Adolesc Psychiatry Ment Health. 2016;10:14. https://doi.org/10.1186/s13034-016-0101-z .

Helmer-Smith M, Fung C, Afkham A, Crowe L, Gazarin M, Keely E, et al. The feasibility of using electronic consultation in long-term care homes. JAm Med Dir Assoc. 2020;21:11661170e2. https://doi.org/10.1016/j.jamda.2020.03.003 .

Orchard J, Lowres N, Freedman SB, Ladak L, Lee W, Zwar N, et al. Screening for atrial fibrillation during influenza vaccinations by primary care nurses using a smartphone electrocardiograph (iECG): a feasibility study. Eur J Prev Cardiol. 2016;23:13–20. https://doi.org/10.1177/2047487316670255 .

Bee P, Lovell K, Airnes Z, Pruszynska A. Embedding telephone therapy in statutory mental health services: a qualitative, theory-driven analysis. BMC Psychiatry. 2016;16:56. https://doi.org/10.1186/s12888-016-0761-5 .

Fontaine P, Whitebird R, Solberg LI, Tillema J, Smithson A, Crabtree BF. Minnesota’s early experience with medical home implementation: viewpoints from the front lines. J Gen Intern Med. 2015;30(7):899–906. https://doi.org/10.1007/s11606-014-3136-y .

Kolltveit B-CH, Gjengedal E, Graue M, Iversen MM, Thorne S, Kirkevold M. Conditions for success in introducing telemedicine in diabetes foot care: a qualitative inquiry. BMC Nurs. 2017;16:2. https://doi.org/10.1186/s12912-017-0201-y .

Salbach NM, McDonald A, MacKay-Lyons M, Bulmer B, Howe JA, Bayley MT, et al. Experiences of physical therapists and professional leaders with implementing a toolkit to advance walking assessment poststroke: a realist evaluation. Phys Ther. 2021;101:1–11. https://doi.org/10.1093/ptj/pzab232 .

Schwarz M, Coccetti A, Draheim M, Gordon G. Perceptions of allied health staff of the implementation of an integrated electronic medical record across regional and metropolitan settings. Aust Health Rev. 2020;44:965–72. https://doi.org/10.1071/AH19024 .

Stewart J, McCorry N, Reid H, Hart N, Kee F. Implementation of remote asthma consulting in general practice in response to the COVID-19 pandemic: an evaluation using extended normalisation process theory. BJGP Open. 2022;6:1–10. https://doi.org/10.3399/BJGPO.2021.0189 .

Bennett-Levy J, Singer J, DuBois S, Hyde K. Translating mental health into practice: what are the barriers and enablers to e-mental health implementation by aboriginal and Torres Strait Islander health professionals? JMed. Internet Res. 2017;19:e1. https://doi.org/10.2196/jmir.6269 .

Olsen J, Peterson S, Stevens A. Implementing electronic health record-based National Diabetes Prevention Program referrals in a rural county. Public Health Nurs (Boston Mass). 2021;38(3):464–9. https://doi.org/10.1111/phn.12860 .

Yang L, Brown-Johnson CG, Miller-Kuhlmann R, Kling SMR, Saliba-Gustafsson EA, Shaw JG, et al. Accelerated launch of video visits in ambulatory neurology during COVID-19: key lessons from the Stanford experience. Neurology. 2020;95:305–11. https://doi.org/10.1212/WNL.0000000000010015 .

Article   CAS   PubMed   Google Scholar  

Buckingham SA, Sein K, Anil K, Demain S, Gunn H, Jones RB, et al. Telerehabilitation for physical disabilities and movement impairment: a service evaluation in South West England. JEval Clin Pract. 2022;28:108495. https://doi.org/10.1111/jep.13689 .

Chung OS, Robinson T, Johnson AM, Dowling NL, Ng CH, Yücel M, et al. Implementation of therapeutic virtual reality into psychiatric care: clinicians’ and service managers’’ perspectives. Front Psychiatry. 2022;12:791123. https://doi.org/10.3389/fpsyt.2021.791123 .

Hogan-Murphy D, Stewart D, Tonna A, Strath A, Cunningham S. Use of normalization process theory to explore key stakeholders’ perceptions of the facilitators and barriers to implementing electronic systems for medicines management in hospital settings. Res SocialAdm Pharm. 2021;17:398405. https://doi.org/10.1016/j.sapharm.2020.03.005 .

Moss SR, Martinez KA, Nathan C, Pfoh ER, Rothberg MB. Physicians’ views on utilization of an electronic health record-embedded calculator to assess risk for venous thromboembolism among medical inpatients: a qualitative study. TH Open. 2022;6:e33–9. https://doi.org/10.1055/s-0041-1742227 .

Yusof MM. A case study evaluation of a critical Care Information System adoption using the socio-technical and fit approach. Int J Med Inf. 2015;84:486–99. https://doi.org/10.1016/j.ijmedinf.2015.03.001 .

Dugstad J, Sundling V, Nilsen ER, Eide H. Nursing staff’s evaluation of facilitators and barriers during implementation of wireless nurse call systems in residential care facilities. A cross-sectional study. BMC Health Serv Res. 2020;20:163. https://doi.org/10.1186/s12913-020-4998-9 .

Rea K, Le-Jenkins U, Rutledge C. A technology intervention for nurses engaged in preventing catheter-associated urinary tract infections. Comput Inf Nurs. 2018;36:305–13. https://doi.org/10.1097/CIN.0000000000000429 .

Bail K, Davey R, Currie M, Gibson J, Merrick E, Redley B. Implementation pilot of a novel electronic bedside nursing chart: a mixed-methods case study. Aust Health Rev. 2020;44:672–6. https://doi.org/10.1071/AH18231 .

van Laere J, Aggestam L. Understanding champion behaviour in a health-care information system development project – how multiple champions and champion behaviours build a coherent whole. Eur J Inf Syst. 2016;25:47–63. https://doi.org/10.1057/ejis.2015.5 .

Download references

Acknowledgements

We would like to thank the librarian Malin E. Norman, at Nord university, for her assistance in the development of the search, as well as guidance regarding the scientific databases.

This study is a part of a PhD project undertaken by the first author, SP, and funded by Nord University, Norway. This research did not receive any specific grant from funding agencies in the public, commercial, as well as not-for-profit sectors.

Open access funding provided by Nord University

Author information

Authors and affiliations.

Faculty of Nursing and Health Sciences, Nord university, P.O. Box 474, N-7801, Namsos, Norway

Sissel Pettersen & Anita Berg

Centre for Health and Technology, Faculty of Health Sciences, University of South-Eastern Norway, PO Box 7053, N-3007, Drammen, Norway

You can also search for this author in PubMed   Google Scholar

Contributions

The first author/SP has been the project manager and was mainly responsible for all phases of the study. The second and third authors HE and AB have contributed to screening, quality assessment, analysis and discussion of findings. Drafting of the final manuscript has been a collaboration between the first/SP and third athor/AB. The final manuscript has been approved by all authors.

Corresponding author

Correspondence to Sissel Pettersen .

Ethics declarations

Ethics approval and consent to participate.

This review does not involve the processing of personal data, and given the nature of this study, formal consent is not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Pettersen, S., Eide, H. & Berg, A. The role of champions in the implementation of technology in healthcare services: a systematic mixed studies review. BMC Health Serv Res 24 , 456 (2024). https://doi.org/10.1186/s12913-024-10867-7

Download citation

Received : 19 June 2023

Accepted : 14 March 2024

Published : 11 April 2024

DOI : https://doi.org/10.1186/s12913-024-10867-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Technology implementation
• Healthcare personnel
• Healthcare services
• Mixed methods
• Organizational characteristics
• Technology adoption
• Role definitions
• Healthcare settings
• Systematic review

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Retirement planning – a systematic review of literature and future research directions

• Published: 28 October 2023

Cite this article

• Kavita Karan Ingale   ORCID: orcid.org/0000-0003-3570-4211 1 &
• Ratna Achuta Paluri 2  

510 Accesses

Explore all metrics

Rising life expectancy and an aging population across nations are leading to an increased need for long-term financial savings and a focus on the financial well-being of retired individuals amidst changing policy framework. This study is a systematic review based on a scientific way of producing high-quality evidence based on 191 articles from the Scopus and Web of Science databases. It adopts the Theory, Context, Characteristics, and Method (TCCM) framework to analyze literature. This study provides collective insights into financial decision-making for retirement savings and identifies constructs for operationalizing and measuring financial behavior for retirement planning. Further, it indicates the need for an interdisciplinary approach. Though cognitive areas were studied extensively, the non-cognitive areas received little attention. Qualitative research design is gaining prominence in research over other methods, with the sparse application of mixed methods design. The study’s TCCM framework explicates several areas for further research. Furthermore, it guides the practice and policy by integrating empirical evidence and concomitant findings. Coherent synthesis of the extant literature reconciles the highly fragmented field of retirement planning. No research reports prospective areas for further analysis based on the TCCM framework on retirement planning, which highlights the uniqueness of the study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

A Research Proposal to Examine Psychological Factors Influence on Financial Planning for Retirement in China

Domains and determinants of retirement timing: A systematic review of longitudinal studies

Micky Scharn, Ranu Sewdas, … Allard J. van der Beek

Reinventing Retirement

Deanna L. Sharpe

Data Availability

The research data will be made available on request.

Acknowledgment.

Elderly population is defined as a population aged 65 years and over.

Defined benefit plan guarantees benefits to the employee, while defined contribution plan requires employees to decide on their own investment and bear the financial risks identified with it.

“The old-age dependency ratio is defined as the number of individuals aged 65 and over per 100 people of working age defined as those at ages 20 to 64”(OECD 2023 ).

Adams GA, Rau BL (2011) Putting off tomorrow to do what you want today: planning for Retirement. Am Psychol 66(3):180–192. https://doi.org/10.1037/a0022131

Article   Google Scholar  

Aegon Cfor, Longevity, Retirement ICR (2016) The Aegon Retirement Readiness Survey 2016. In The Aegon Retirement Readiness Survey 2016 . https://www.aegon.com/contentassets/c6a4b1cdded34f1b85a4f21d4c66e5d3/2016-aegon-retirement-readiness-report-india.pdf

Agarwalla SK, Barua SK, Jacob J, Varma JR (2015) Financial Literacy among Working Young in Urban India. World Development , 67 (2013), 101–109. https://doi.org/10.1016/j.worlddev.2014.10.004

Ajzen I (1991) The theory of Planned Behavior. Organ Behav Hum Decis Process 50:179–211. https://doi.org/10.47985/dcidj.475

Anderson A, Baker F, Robinson DT (2017) Precautionary savings, retirement planning, and misperceptions of financial literacy. J Financ Econ 126(2):383–398. https://doi.org/10.1016/j.jfineco.2017.07.008

Atkinson A, Messy FA (2011) Assessing financial literacy in 12 countries: an OECD/INFE international pilot exercise. J Pension Econ Finance 10(4):657–665. https://doi.org/10.1017/S1474747211000539`

Aydin AE, Akben Selcuk E (2019) An investigation of financial literacy, money ethics, and time preferences among college students: a structural equation model. Int J Bank Mark 37(3):880–900. https://doi.org/10.1108/IJBM-05-2018-0120

Bapat D (2020) Antecedents to responsible financial management behavior among young adults: the moderating role of financial risk tolerance. Int J Bank Mark 38(5):1177–1194. https://doi.org/10.1108/IJBM-10-2019-0356

Beckett A, Hewer P, Howcroft B (2000) An exposition of consumer behaviour in the financial services industry. Int J Bank Mark 18(1):15–26. https://doi.org/10.1108/02652320010315325

Białowolski P (2019) Economic sentiment as a driver for household financial behavior. J Behav Experimental Econ 80(August 2017):59–66. https://doi.org/10.1016/j.socec.2019.03.006

Binswanger J, Carman KG (2012) How real people make long-term decisions: the case of retirement preparation. J Economic Behav Organ 81(1):39–60. https://doi.org/10.1016/j.jebo.2011.08.010

Brounen D, Koedijk KG, Pownall RAJ (2016) Household financial planning and savings behavior. J Int Money Finance 69:95–107. https://doi.org/10.1016/j.jimonfin.2016.06.011

Brown R, Jones M (2015) Mapping and exploring the topography of contemporary financial accounting research. Br Acc Rev 47(3):237–261. https://doi.org/10.1016/j.bar.2014.08.006

Brown S, Gray D (2016) Household finances and well-being in Australia: an empirical analysis of comparison effects. J Econ Psychol 53:17–36. https://doi.org/10.1016/j.joep.2015.12.006

Brown S, Taylor K (2014) Household finances and the big five personality traits. J Econ Psychol 45:197–212. https://doi.org/10.1016/j.joep.2014.10.006

Brown S, Taylor K (2016) Early influences on saving behaviour: analysis of British panel data. J Bank Finance 62:1–14. https://doi.org/10.1016/j.jbankfin.2015.09.011

Brüggen EC, Post T, Schmitz K (2019) Interactivity in online pension planners enhances engagement with retirement planning – but not for everyone. J Serv Mark 33(4):488–501. https://doi.org/10.1108/JSM-02-2018-0082

Bruggen E, Post T, Katharina S (2019) Interactivity in online pension planners enhances engagement with retirement planning but not for everyone. J Serv Mark 33(4):488–501

Calcagno R, Monticone C (2015) Financial literacy and the demand for financial advice. J Bank Finance 50:363–380. https://doi.org/10.1016/j.jbankfin.2014.03.013

Campbell JY (2006) Household finance. J Finance 61(4):1553–1604. https://doi.org/10.1111/j.1540-6261.2006.00883.x

Choudhury K (2015) Service quality and customers’ behavioural intentions: class and mass banking and implications for the consumer and society. Asia Pac J Mark Logistics 27(5):735–757

Chowdhry N, Jung J, Dholakia U (2018) Association for consumer research. Adv Consum Res 42:42–46

Google Scholar  

Clark GL, Knox-Hayes J, Strauss K (2009) Financial sophistication, salience, and the scale of deliberation in UK retirement planning. Environ Plann A 41(10):2496–2515. https://doi.org/10.1068/a41265

Clark R, Lusardi A, Mitchell OS (2017) Employee Financial Literacy and Retirement Plan Behavior: a case study. Econ Inq 55(1):248–259. https://doi.org/10.1111/ecin.12389

Collins JM, Urban C (2016) The role of information on Retirement Planning: evidence from a field study. Econ Inq 54(4):1860–1872. https://doi.org/10.1111/ecin.12349

Creswell J (2009) Research Design Qualitative Quantitative and Mixed Methods Approaches. In Sage Publishing: Vol. Third edit . https://doi.org/10.1002/tl.20234

Csorba L (2020) The determining factors of financial culture, financial literacy, and financial behavior. Public Finance Q 65:67–83. https://doi.org/10.35551/PFQ_2020_1_6

Davidoff T, Gerhard P, Post T (2017) Reverse mortgages: what homeowners (don’t) know and how it matters. J Economic Behav Organ 133:151–171. https://doi.org/10.1016/j.jebo.2016.11.007

Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly: Management Information Systems 13(3):319–339. https://doi.org/10.2307/249008

Devlin J (2001) Consumer evaluation and competitive advantage in retail financial services - a research agenda. Eur J Mark 35(5/6):639–660

Dholakia U, Tam L, Yoon S, Wong N (2016) The ant and the grasshopper: understanding personal saving orientation of consumers. J Consum Res 43(1):134–155. https://doi.org/10.1093/jcr/ucw004

Dolls M, Doerrenberg P, Peichl A, Stichnoth H (2018) Do retirement savings increase in response to information about retirement and expected pensions? J Public Econ 158(July 2017):168–179. https://doi.org/10.1016/j.jpubeco.2017.12.014

Dragos SL, Dragos CM, Muresan GM (2020) From intention to the decision in purchasing life insurance and private pensions: different effects of knowledge and behavioural factors. J Behav Experimental Econ 87(March):101555. https://doi.org/10.1016/j.socec.2020.101555

Drever AI, Odders-white E, Kalish CW, Hoagland EM, Nelms EN, Drever AI, Odders-white E, Charles W, Else-quest NM, Hoagland EM, Nelms EN (2015) Foundations of Financial Weil-Being: Insights into the Role of Executive Function, Financial Socialization, and Experience-Based Learning in Childhood and Youth Source : The Journal of Consumer Affairs, Vol. 49, No. 1, Special Issue on Starting Ea. The Journal of Consumer Affairs , 49 (1)

Duflo E, Saez E (2002) Participation and investment decisions in a retirement plan: the influence of colleagues’ choices. J Public Econ 85(1):121–148. https://doi.org/10.1016/S0047-2727(01)00098-6

Duxbury D, Summers B, Hudson R, Keasey K (2013) How people evaluate defined contribution, annuity-based pension arrangements: a behavioral exploration. J Econ Psychol 34:256–269. https://doi.org/10.1016/j.joep.2012.10.008

Earl J, Bednall T, Muratore A (2015) A matter of time: why some people plan for retirement and others do not. Work Aging and Retirement 1(2):181–189. https://doi.org/10.1093/workar/wau005

Employees Benefits Research Institute (2020) EBRI Retirement Confidence Survey Report (Issue 202)

Engel JF, Kollat DT, Blackwell RD (1968) A model of consumer motivation and behavior. In: Research in consumer behavior. Holt, Rinehart and Winston, Inc., New York, pp 3–20

Erasmus A, Boshoff E, Rousseau G (2001) Consumer decision-making models within the discipline of consumer science: a critical approach. J Family Ecol Consumer Sci /Tydskrif Vir Gesinsekologie En Verbruikerswetenskappe 29(1):82–90. https://doi.org/10.4314/jfecs.v29i1.52799

Farrell L, Fry TRL, Risse L (2016) The significance of financial self-efficacy in explaining women’s personal finance behaviour. J Econ Psychol 54:85–99

Fernandes D, Lynch JG, Netemeyer RG (2014) Financial literacy, financial education, and downstream financial behaviors. Manage Sci 60(8):1861–1883. https://doi.org/10.1287/mnsc.2013.1849

Filbec G, Ricciardi V, Evensky H, Fan S, Holzhauer H, Spieler A (2017) Behavioral finance: a panel discussion. J Behav Experimental Finance 15:52–58. https://doi.org/10.1016/j.jbef.2015.07.003

Fishbein M (1979) A theory of reasoned action: some applications and implications. Nebraska Symposium on Motivation 27:65–116

Fisher PJ, Montalto CP (2010) Effect of saving motives and horizon on saving behaviors. J Econ Psychol 31(1):92–105. https://doi.org/10.1016/j.joep.2009.11.002

Flores SAM, Vieira KM (2014) Propensity toward indebtedness: an analysis using behavioral factors. J Behav Exp Finance 3:1–10

Foxall GR, Pallister JG (1998) Measuring purchase decision involvement for financial services: comparison of the Zaichkowsky and Mittal scales. Int J Bank Mark 16(5):180–194. https://doi.org/10.1108/02652329810228181

Friedman M (1957) Introduction to “A theory of the consumption function”. In: A theory of the consumption function. Princeton University Press, pp 1–6

Frydman C, Camerer CF (2016) The psychology and neuroscience of financial decision making. Trends Cogn Sci 20(9):661–675. https://doi.org/10.1016/j.tics.2016.07.003

Gardarsdóttir RB, Dittmar H (2012) The relationship of materialism to debt and financial well-being: the case of Iceland’s perceived prosperity. J Econ Psychol 33(3):471–481. https://doi.org/10.1016/j.joep.2011.12.008

Gathergood J (2012) Self-control, financial literacy and consumer over-indebtedness. J Econ Psychol 33(3):590–602

Gerhard P, Gladstone JJ, Hoffmann AOI (2018) Psychological characteristics and household savings behavior: the importance of accounting for latent heterogeneity. J Economic Behav Organ 148:66–82. https://doi.org/10.1016/j.jebo.2018.02.013

Gibbs PT (2009) Time, temporality, and

Goedde-Menke M, Lehmensiek-Starke M, Nolte S (2014) An empirical test of competing hypotheses for the annuity puzzle. J Econ Psychol 43:75–91

Gough O, Nurullah M (2009) Understanding what drives the purchase decision in pension and investment products. J Financial Serv Mark 14(2):152–172. https://doi.org/10.1057/fsm.2009.14

Griffin B, Loe D, Hesketh B (2012) Using Proactivity, Time Discounting, and the theory of Planned Behavior to identify predictors of Retirement Planning. Educ Gerontol 38(12):877–889. https://doi.org/10.1080/03601277.2012.660857

Gritten A (2011) New insights into consumer confidence in financial services. Int J Bank Mark 29(2):90–106. https://doi.org/10.1108/02652321111107602

Grohmann A (2018) Financial literacy and financial behavior: Evidence from the emerging Asian middle class. Pacific Basin Finance Journal , 48 (November 2017), 129–143. https://doi.org/10.1016/j.pacfin.2018.01.007

Grohmann A, Kouwenberg R, Menkhoff L (2015) Childhood roots of financial literacy. J Econ Psychol 51:114–133. https://doi.org/10.1016/j.joep.2015.09.002

Hair JF, Sarstedt M, Ringle CM, Mena JA (2012) An assessment of the use of partial least squares structural equation modeling in marketing research . 414–433. https://doi.org/10.1007/s11747-011-0261-6

Hanna SD, Kim KT, Chen SCC (2016) Retirement savings. In: Handbook of consumer finance research, pp 33–43

Harrison T, Waite K, White P (2006) Analysis by paralysis: the pension purchase decision process. Int J Bank Mark 24(1):5–23. https://doi.org/10.1108/02652320610642317

Hastings J, Mitchell O (2011) How financial literact and impatience shape retirement wealth and investment behaviors. Pengaruh Harga Diskon Dan Persepsi Produk Terhadap Nilai Belanja Serta Perilaku Pembelian Konsumen, NBER Working paper, 1–28

Hauff J, Carlander A, Amelie G, Tommy G, Holmen M (2016) Breaking the ice of low financial involvement: does narrative information format from a trusted sender increase savings in mutual funds? Int J Bank Mark 34(2):151–170

Hentzen JK, Hoffmann A, Dolan R, Pala E (2021) Artificial intelligence in customer-facing financial services: a systematic literature review and agenda for future research. Int J Bank Mark. https://doi.org/10.1108/IJBM-09-2021-0417

Hershey DA, Mowen JC (2000) Psychological determinants of financial preparedness for retirement. Gerontologist 40(6):687–697. https://doi.org/10.1093/geront/40.6.687

Hershey DA, Henkens K, Van Dalen HP (2007) Mapping the minds of retirement planners: a cross-cultural perspective. J Cross-Cult Psychol 38(3):361–382. https://doi.org/10.1177/0022022107300280

Hershey DA, Jacobs-Lawson JM, McArdle JJ, Hamagami F (2007b) Psychological foundations of financial planning for retirement. J Adult Dev 14(1–2):26–36. https://doi.org/10.1007/s10804-007-9028-1

Hershey DA, Jacobs-Lawson JM, McArdle JJ, Hamagami F (2008) Psychological foundations of financial planning for retirement. J Adult Dev 14(1–2):26–36. https://doi.org/10.1007/s10804-007-9028-1

Hershfield H, Goldstein D, Sharpe W, Fox J, Yeykelis L, Carstensen L, Bailenson J (2011) Increasing saving behavior through age-progressed renderings of the future self. J Mark Res 48:23–37

Hoffmann AOI, Broekhuizen TLJ (2009) Susceptibility to and impact of interpersonal influence in an investment context. J Acad Mark Sci 37:488–503

Hoffmann AOI, Broekhuizen TLJ (2010) Understanding investors’ decisions to purchase innovative products: drivers of adoption timing and range. Int J Res Mark 27(4):342–355. https://doi.org/10.1016/j.ijresmar.2010.08.002

Hoffmann AOI, Plotkina D (2020a) Positive framing when assessing the personal resources to manage one’s finances increases consumers’ retirement self-efficacy and improves retirement goal clarity. Psychol Mark 38(12):2286–2304. https://doi.org/10.1002/mar.21563

Hoffmann AOI, Plotkina D (2020b) Why and when does financial information affect retirement planning intentions and which consumers are more likely to act on them? Journal of Business Research , 117 (September 2019), 411–431. https://doi.org/10.1016/j.jbusres.2020.06.023

Hoffmann AOI, Plotkina D (2021) Let your past define your future. How recalling successful financial experiences can increase beliefs of self-efficacy in financial planning. J Consum Aff 55(3):847–871. https://doi.org/10.1111/joca.12378

Hoffmann AOI, Risse L (2020) Do good things come in pairs? How personality traits help explain individuals’ simultaneous pursuit of a healthy lifestyle and financially responsible behavior. J Consum Aff 54(3):1082–1120. https://doi.org/10.1111/joca.12317

Hsiao YJ, Tsai WC (2018) Financial literacy and participation in the derivatives markets. J Bank Finance 88:15–29

Huhmann BA, McQuitty S (2009) A model of consumer financial numeracy. Int J Bank Mark 27(4):270–293. https://doi.org/10.1108/02652320910968359

Huston SJ (2010) Measuring financial literacy. J Consum Aff 44(2):296–316. https://doi.org/10.1111/j.1745-6606.2010.01170.x

Ijevleva K, Arefjevs I (2014) Analysis of the Aggregate Financial Behaviour of customers using the Transtheoretical Model of Change. Procedia - Social and Behavioral Sciences 156(April):435–438. https://doi.org/10.1016/j.sbspro.2014.11.217

Ingale KK, Paluri RA (2020) Financial literacy and financial behavior: a bibliometric analysis. Rev Behav Finance. https://doi.org/10.1108/RBF-06-2020-0141

Jacobs-Lawson J, Hershey D (2005) Influence of future time perspective, financial knowledge, and financial risk tolerance on retirement savings behavior. Financial Serv Rev 14:331–344. https://doi.org/10.1088/1751-8113/44/8/085201

Jappelli T, Padula M (2013) Investment in financial literacy and saving decisions. J Bank Finance 37(8):2779–2792. https://doi.org/10.1016/j.jbankfin.2013.03.019

Kadoya Y, Rahim Khan MS (2020) Financial literacy in Japan: new evidence using financial knowledge, behavior, and attitude. Sustain (Switzerland) 12(9). https://doi.org/10.3390/su12093683

Kamil NSSN, Musa R, Sahak SZ (2014) Examining the Role of Financial Intelligence Quotient (FiQ) in explaining credit card usage behavior: a conceptual Framework. Procedia - Social and Behavioral Sciences 130:568–576. https://doi.org/10.1016/j.sbspro.2014.04.066

Kerry MJ (2018) Psychological antecedents of retirement planning: a systematic review. Front Psychol 9(OCT). https://doi.org/10.3389/fpsyg.2018.01870

Kerry MJ, Embretson SE (2018) An experimental evaluation of competing age predictions of future time perspective between workplace and retirement domains. Front Psychol 8(JAN):1–9. https://doi.org/10.3389/fpsyg.2017.02316

Kiliyanni AL, Sivaraman S (2016) The perception-reality gap in financial literacy: evidence from the most literate state in India. Int Rev Econ Educ 23:47–64. https://doi.org/10.1016/j.iree.2016.07.001

Kimiyaghalam F, Mansori S, Safari M, Yap S (2017) Parents’ influence on retirement planning in Malaysia. Family Consumer Sci Res J 45(3):315–325

Klapper L, Lusardi A, Panos GA (2013) Financial literacy and its consequences: evidence from Russia during the financial crisis. J Bank Finance 37(10):3904–3923

Koehler DJ, Langstaff J, Liu WQ (2015) A simulated financial savings task for studying consumption and retirement decision-making. J Econ Psychol 46:89–97. https://doi.org/10.1016/j.joep.2014.12.004

Kramer MM (2016) Financial literacy, confidence, and financial advice seeking. Journal of Economic Behavior and Organization , 131 (June 2015), 198–217. https://doi.org/10.1016/j.jebo.2016.08.016

Kumar S, Tomar S, Verma D (2019) Women’s financial planning for retirement: systematic literature review and future research agenda. Int J Bank Mark 37(1):120–141. https://doi.org/10.1108/IJBM-08-2017-0165

Kwon KN, Lee J (2009) The effects of reference point, knowledge, and risk propensity on the evaluation of financial products. J Bus Res 62(7):719–725. https://doi.org/10.1016/j.jbusres.2008.07.002

Landerretche OM, Martínez C (2013) Voluntary savings, financial behavior, and pension finance literacy: evidence from Chile. J Pension Econ Finance 12(3):251–297. https://doi.org/10.1017/S1474747212000340

Lee T (2017) (David). Clear, conspicuous, and improving: US corporate websites for critical financial literacy in retirement. International Journal of Bank Marketing , 35 (5), 761–780. https://doi.org/10.1108/IJBM-01-2016-0010

Liang C-J, Wang Wen‐Hung, Farquhar JD (2009) (2009). The influence of customer perceptions on financial performance in financial services. International Journal of Bank Marketing , 27 (2), 129–149

Liberman N, Trope Y (2003) Construal level theory of intertemporal judgment and decision. In: Loewenstein G, Read D, Baumeister R (eds) Time and decision: economic and psychological perspectives on intertemporal choice, pp 245–276

Lim KL, Soutar GN, Lee JA (2013) Factors affecting investment intentions: a consumer behaviour perspective. J Financ Serv Mark 18:301–315

Lin C, Hsiao YJ, Yeh CY (2017) Financial literacy, financial advisors, and information sources on demand for life insurance. Pac Basin Finance J 43(March):218–237. https://doi.org/10.1016/j.pacfin.2017.04.002

Lown JM (2011) Development and validation of a Financial Self-Efficacy Scale. J Financial Couns Plann 22(2):54–63

Lusardi A, Mitchell OS (2007) Baby Boomer retirement security: the roles of planning, financial literacy, and housing wealth. J Monet Econ 54(1):205–224. https://doi.org/10.1016/j.jmoneco.2006.12.001

Maloney M, McCarthy A (2017) Understanding pension communications at the organizational level: insights from bounded rationality theory & implications for HRM. Hum Resource Manage Rev 27(2):338–352. https://doi.org/10.1016/j.hrmr.2016.08.001

Marjanovic Z, Fiksenbaum L, Greenglass E (2018) Financial threat correlates with acute economic hardship and behavioral intentions that can improve one’s personal finances and health. J Behav Experimental Econ 77(April):151–157. https://doi.org/10.1016/j.socec.2018.09.012

Marques S, Mariano J, Lima ML, Abrams D (2018) Are you talking to the future me? The moderator role of future self-relevance on the effects of aging salience in retirement savings. J Appl Soc Psychol 48(7):360–368. https://doi.org/10.1111/jasp.12516

McKechnie S (1992) Consumer buying behaviour in financial services: an overview. Int J Bank Mark 10(5):5–39. https://doi.org/10.1108/02652329210016803

Milner T, Rosenstreich D (2013a) A review of consumer decision-making models and development of a new model for financial services. J Financial Serv Mark 18(2):106–120. https://doi.org/10.1057/fsm.2013.7

Milner T, Rosenstreich D (2013b) Insights into mature consumers of financial services. J Consumer Mark 30(3):248–257. https://doi.org/10.1108/07363761311328919

Mitchell OS, Mukherjee A (2017) Assessing the demand for micro pensions among India’s poor. J Econ Ageing 9:30–40. https://doi.org/10.1016/j.jeoa.2016.05.004

Mitchell O, Utkus S (2003) Lessons from Behavioral Finance for Retirement Plan Design (PRC WP 2003-6). http://prc.wharton.upenn.edu/prc/prc.html

Modigliani F, Brumberg RH (1954) Utility analysis and the consumption function: an interpretation of cross-section data. In: Kurihara KK (ed) Post-Keynesian economics. Rutgers University Press, New Brunswick, pp 388–436

Moher D, Liberati A, Tetzlaff J, Altman DG, Altman D, Antes G, Atkins D, Barbour V, Barrowman N, Berlin JA, Clark J, Clarke M, Cook D, D’Amico R, Deeks JJ, Devereaux PJ, Dickersin K, Egger M, Ernst E, …, Tugwell P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6(7). https://doi.org/10.1371/journal.pmed.1000097

Monti M, Pelligra V, Martignon L, Berg N (2014) Retail investors and financial advisors: new evidence on trust and advice taking heuristics. J Bus Res 67(8):1749–1757. https://doi.org/10.1016/j.jbusres.2014.02.022

Mouna A, Anis J (2017) Financial literacy in Tunisia: its determinants and its implications on investment behavior. Res Int Bus Finance 39:568–577

Mullainathan S, Thaler R (2000) Massachusetts Institute of Technology Department of Economics Working Paper Series . September

Nga KH, Yeoh KK (2018) An exploratory model on retirement savings behaviour: a Malaysian study. Int J Bus Soc 19(3):637–659

OECD (2023) Old-age dependency ratio (indicator). https://doi.org/10.1787/e0255c98-en . Accessed 13 Oct 2023

Onwuegbuzie AJ, Collins KM (2007) A typology of mixed methods sampling designs in social science research. Qualitative Rep 12(2):474–498

Pallister JG, Wang HC, Foxall GR (2007) An application of the style/involvement model to financial services. Technovation 27(1–2):78–88. https://doi.org/10.1016/j.technovation.2005.10.001

Pan L, Pezzuti T, Lu W, Pechmann C (2019) Hyperopia and frugality: different motivational drivers and yet similar effects on consumer spending. J Bus Res 95(August 2018):347–356

Parise G, Peijnenburg K (2017) Understanding the Determinants of Financial Outcomes and Choices: The Role of Noncognitive Abilities. BIS Working Papers

Paul J, Rosado-Serrano A (2019) Gradual internationalization vs Born-Global/International new venture models: a review and research agenda. Int Mark Rev 36(6):830–858. https://doi.org/10.1108/IMR-10-2018-0280

Paul J, Criado AR (2020) The art of writing literature review: what do we know and what do we need to know? Int Bus Rev 29(4):101717. https://doi.org/10.1016/j.ibusrev.2020.101717

Paul J, Khatri P, Kaur Duggal H (2023) Frameworks for developing impactful systematic literature reviews and theory building: what, why and how? J Decis Syst 00(00):1–14. https://doi.org/10.1080/12460125.2023.2197700

Petkoska J, Earl JK (2009) Understanding the influence of demographic and psychological variables on Retirement Planning. Psychol Aging 24(1):245–251. https://doi.org/10.1037/a0014096

Piotrowska M (2019) The importance of personality characteristics and behavioral constraints for retirement saving. Econ Anal Policy 64:194–220

Plath DA, Stevenson TH (2005) Financial services consumption behavior across Hispanic American consumers. J Bus Res 58(8):1089–1099. https://doi.org/10.1016/j.jbusres.2004.03.003

Poterba JM (2015) Saver heterogeneity and the challenge of assessing retirement saving adequacy. Natl Tax J 68(2):377–388. https://doi.org/10.17310/ntj.2015.2.06

Potrich ACG, Vieira KM, Kirch G (2018) How well do women do when it comes to financial literacy? Proposition of an indicator and analysis of gender differences. J Behav Experimental Finance 17:28–41. https://doi.org/10.1016/j.jbef.2017.12.005

Rai D, Lin CW (2019) (Wilson). The influence of implicit self-theories on consumer financial decision making. Journal of Business Research , 95 (August 2018), 316–325. https://doi.org/10.1016/j.jbusres.2018.08.016

Ramalho TB, Forte D (2019) Financial literacy in Brazil – do knowledge and self-confidence relate with behavior? RAUSP Manage J 54(1):77–95. https://doi.org/10.1108/RAUSP-04-2018-0008

Rana J, Paul J (2017) Consumer behavior and purchase intention for organic food: a review and research agenda. J Retailing Consumer Serv 38(June):157–165. https://doi.org/10.1016/j.jretconser.2017.06.004

Ranyard R, McNair S, Nicolini G, Duxbury D (2020) An item response theory approach to constructing and evaluating brief and in-depth financial literacy scales. J Consum Aff 54(3):1121–1156. https://doi.org/10.1111/joca.12322

RBI Household Finance Committee (2017) Indian household finance. Reserve Bank of India, Mumbai

Ruefenacht M, Schlager T, Maas P, Puustinen P (2015) Drivers of long-term savings behavior from consumer’s perspective. Electron Libr 34(1):1–5

Scholz JK, Seshadri A, Khitatrakun S (2006) Are Americans saving “optimally” for retirement? J Polit Econ 114(4):607–643

Schuabb T, França LH, Amorim SM (2019) Retirement savings model tested with Brazilian private health care workers. Front Psychol 10(JULY):1–11. https://doi.org/10.3389/fpsyg.2019.01701

Schuhen M, Schurkmann S (2014) International Review of Economics Education. Int Rev Econ Educ 16:1–11

Segel-Karpas D, Werner P (2014) Perceived financial retirement preparedness and its correlates: a national study in Israel. Int J Aging Hum Dev 79(4):279–301. https://doi.org/10.1177/0091415015574177

Seth H, Talwar S, Bhatia A, Saxena A, Dhir A (2020) Consumer resistance and inertia of retail investors: Development of the resistance adoption inertia continuance (RAIC) framework. Journal of Retailing and Consumer Services , 55 (August 2019), 102071. https://doi.org/10.1016/j.jretconser.2020.102071

Sewell M (2008) Behavioural finance. Economist 389(8604):1–13. https://doi.org/10.1057/9780230280786_5

Shefrin HM, Thaler RH (1988) The behavioral life‐cycle hypothesis. Econ Inq 26(4):609–643

Shim S, Serido J, Tang C (2012) The ant and the grasshopper revisited: the present psychological benefits of saving and future oriented financial behavior. J Econ Psychol 33(1):155–165

Simon HA (1978) Information-processing theory of human problem solving. In: Handbook of learning and cognitive processes, vol 5, pp 271–295

Sivaramakrishnan S, Srivastava M, Rastogi A (2017) Attitudinal factors, financial literacy, and stock market participation. Int J Bank Mark 34(1):1–5

Snyder H (2019) Literature review as a research methodology: an overview and guidelines. J Bus Res 104(August):333–339. https://doi.org/10.1016/j.jbusres.2019.07.039

Stawski RS, Hershey DA, Jacobs-Lawson JM (2007) Goal clarity and financial planning activities as determinants of retirement savings contributions. Int J Aging Hum Dev 64(1):13–32. https://doi.org/10.2190/13GK-5H72-H324-16P2

Steinert JI, Zenker J, Filipiak U, Movsisyan A, Cluver LD, Shenderovich Y (2018) Do saving promotion interventions increase household savings, consumption, and investments in Sub-saharan Africa? A systematic review and meta-analysis. World Dev 104:238–256. https://doi.org/10.1016/j.worlddev.2017.11.018

Steinhart Y, Mazursky D (2010) Purchase availability and involvement antecedents among financial products. Int J Bank Mark 28(2):113–135. https://doi.org/10.1108/02652321011018314

Strömbäck C, Lind T, Skagerlund K, Västfjäll D, Tinghög G (2017) Does self-control predict financial behavior and financial well-being? J Behav Experimental Finance 14:30–38. https://doi.org/10.1016/j.jbef.2017.04.002

Strömbäck C, Skagerlund K, Västfjäll D, Tinghög G (2020) Subjective self-control but not objective measures of executive functions predict financial behavior and well-being. Journal of Behavioral and Experimental Finance , 27 . https://doi.org/10.1016/j.jbef.2020.100339

Tam L, Dholakia U (2014) Saving in cycles: how to get people to save more money. Psychol Sci 25(2):531–537. https://doi.org/10.1177/0956797613512129

Tang N, Baker A (2016) Self-esteem, financial knowledge and financial behavior. J Econ Psychol 54:164–176

Tate M, Evermann J, Gable G (2015) An integrated framework for theories of individual attitudes toward technology. Inform Manage 52(6):710–727. https://doi.org/10.1016/j.im.2015.06.005

Taylor MP, Jenkins SP, Sacker A (2011) Financial capability and psychological health. J Econ Psychol 32(5):710–723. https://doi.org/10.1016/j.joep.2011.05.006

Tennyson S, Yang HK (2014) The role of life experience in long-term care insurance decisions. Journal of Economic Psychology , 42 (2014), 175–188. https://doi.org/10.1016/j.joep.2014.04.002

Thaler BRH (1994) Psychology and savings policies. Am Econ Rev 84(2):175–179. http://www.jstor.org/stable/3132220

Thaler R (1980) Toward a positive theory of consumer choice. J Econ Behav Organ 1:39–60

Thaler RH (2005) Advances in behavioral finance. Adv Behav Finance 2:1–694. https://doi.org/10.2307/2329257

Thaler R, Shefrin H (1981) An economic theory of self-control. J Polit Econ 89(2):392–406

Tomar S, Kent Baker H, Kumar S, Hoffmann AOI (2021) Psychological determinants of retirement financial planning behavior. Journal of Business Research , 133 (November 2020), 432–449. https://doi.org/10.1016/j.jbusres.2021.05.007

Topa G, Moriano JA, Depolo M, Alcover CM, Morales JF (2009) Antecedents and consequences of retirement planning and decision-making: a meta-analysis and model. J Vocat Behav 75(1):38–55. https://doi.org/10.1016/j.jvb.2009.03.002

Topa G, Moriano JA, Depolo M, Alcover CM, Moreno A (2011) Retirement and wealth relationships: Meta-analysis and SEM. Res Aging 33(5):501–528. https://doi.org/10.1177/0164027511410549

Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14(3):207–222. https://doi.org/10.1111/1467-8551.00375

Ülkümen G, Cheema A (2011) Framing goals to influence personal savings: the role of specificity and construal level. J Mark Res 48(6):958–969. https://doi.org/10.1509/jmr.09.0516

United Nations, Department of Economic and Social, Affairs PD (2020) (2019). World Population Ageing 2019. In United Nations . http://link.springer.com/chapter/10.1007/ 978-94-007-5204-7_6

Utkarsh, Pandey A, Ashta A, Spiegelman E, Sutan A (2020) Catch them young: impact of financial Socialization, financial literacy and attitude towards money on the financial well-being of young adults. Int J Consumer Stud 44(6):531–541. https://doi.org/10.1111/ijcs.12583

Valente TW, Paredes P, Poppe P (1998) Matching the message to the process: the relative ordering of knowledge, attitudes, and practices in behavior change research. Hum Commun Res 24(3):366–385

Van Rooij M, Teppa F (2014) Personal traits and individual choices: taking action in economic and non-economic decisions. J Econ Behav Organ 100:33–43

van Rooij M, Lusardi A, Alessie R (2011) Financial literacy and stock market participation. J Financ Econ 101(2):449–472. https://doi.org/10.1016/j.jfineco.2011.03.006

Van Rooij MCJ, Lusardi A, Alessie RJM (2011a) Financial literacy and retirement planning in the Netherlands. J Econ Psychol 32(4):593–608. https://doi.org/10.1016/j.joep.2011.02.004

van Schie RJG, Dellaert BGC, Donkers B (2015) Promoting later planned retirement: construal level intervention impact reverses with age. J Econ Psychol 50:124–131. https://doi.org/10.1016/j.joep.2015.06.010

Venkatesh V, Morris M, Davis G, Davis F (2003) Factors influencing the Use of M-Banking by academics: Case Study sms-based M-Banking. MIS Q 27(3):425–478

Vitt LA (2004) Consumers’ financial decisions and the psychology of values. J Financial Service Professionals 58(November):68–77. http://search.ebscohost.com/login.aspx?direct=true &db=bth&AN=14888952&site=ehost-live

Wang L, Lu W, Malhotra NK (2011) Demographics, attitude, personality, and credit card features correlate with credit card debt: a view from China. J Econ Psychol 32(1):179–193. https://doi.org/10.1016/j.joep.2010.11.006

World Economic Forum (2019) Investing in (and for) our future. Issue June. www.weforum.org

Xia T, Wang Z, Li K (2014) Financial literacy overconfidence and stock market participation. Soc Indic Res 119(3):1233–1245. https://doi.org/10.1007/s11205-013-0555-9

Xiao JJ, Chen C, Chen F (2014) Consumer financial capability and financial satisfaction. Soc Indic Res 118(1):415–432. https://doi.org/10.1007/s11205-013-0414-8

Yeung DY, Zhou X (2017) Planning for retirement: longitudinal effect on retirement resources and post-retirement well-being. Front Psychol 8:1300

Zhou R, Pham MT (2004) Promotion and prevention across mental accounts: when financial products dictate consumers’ investment goals. J Consum Res 31(1):125–135. https://doi.org/10.1086/383429

Download references

Acknowledgements

Authors would like to acknowledge the academicians and researchers who guided the search of the article and would like to thank the experts for the valuable inputs to refine the work.

There is no funding received for this research.

Author information

Authors and affiliations.

Symbiosis International (Deemed University), Pune, India

Kavita Karan Ingale

Symbiosis Institute of Operations Management, Symbiosis International (Deemed) University, Pune, India

Ratna Achuta Paluri

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors contributed to the conceptualization, research design, methodology, analysis of the data,writing of the manuscript and its revision.

Corresponding author

Correspondence to Kavita Karan Ingale .

Ethics declarations

Conflict of interest.

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Ingale, K.K., Paluri, R.A. Retirement planning – a systematic review of literature and future research directions. Manag Rev Q (2023). https://doi.org/10.1007/s11301-023-00377-x

Download citation

Received : 14 December 2022

Accepted : 04 October 2023

Published : 28 October 2023

DOI : https://doi.org/10.1007/s11301-023-00377-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Retirement planning
• Systematic literature review
• Financial behavior
• Household finance
• Long-term savings
• Pension plan
• Financial literacy
• TCCM framework
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Korean J Anesthesiol
• v.71(2); 2018 Apr

Introduction to systematic review and meta-analysis

1 Department of Anesthesiology and Pain Medicine, Inje University Seoul Paik Hospital, Seoul, Korea

2 Department of Anesthesiology and Pain Medicine, Chung-Ang University College of Medicine, Seoul, Korea

Systematic reviews and meta-analyses present results by combining and analyzing data from different studies conducted on similar research topics. In recent years, systematic reviews and meta-analyses have been actively performed in various fields including anesthesiology. These research methods are powerful tools that can overcome the difficulties in performing large-scale randomized controlled trials. However, the inclusion of studies with any biases or improperly assessed quality of evidence in systematic reviews and meta-analyses could yield misleading results. Therefore, various guidelines have been suggested for conducting systematic reviews and meta-analyses to help standardize them and improve their quality. Nonetheless, accepting the conclusions of many studies without understanding the meta-analysis can be dangerous. Therefore, this article provides an easy introduction to clinicians on performing and understanding meta-analyses.

Introduction

A systematic review collects all possible studies related to a given topic and design, and reviews and analyzes their results [ 1 ]. During the systematic review process, the quality of studies is evaluated, and a statistical meta-analysis of the study results is conducted on the basis of their quality. A meta-analysis is a valid, objective, and scientific method of analyzing and combining different results. Usually, in order to obtain more reliable results, a meta-analysis is mainly conducted on randomized controlled trials (RCTs), which have a high level of evidence [ 2 ] ( Fig. 1 ). Since 1999, various papers have presented guidelines for reporting meta-analyses of RCTs. Following the Quality of Reporting of Meta-analyses (QUORUM) statement [ 3 ], and the appearance of registers such as Cochrane Library’s Methodology Register, a large number of systematic literature reviews have been registered. In 2009, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [ 4 ] was published, and it greatly helped standardize and improve the quality of systematic reviews and meta-analyses [ 5 ].

Levels of evidence.

In anesthesiology, the importance of systematic reviews and meta-analyses has been highlighted, and they provide diagnostic and therapeutic value to various areas, including not only perioperative management but also intensive care and outpatient anesthesia [6–13]. Systematic reviews and meta-analyses include various topics, such as comparing various treatments of postoperative nausea and vomiting [ 14 , 15 ], comparing general anesthesia and regional anesthesia [ 16 – 18 ], comparing airway maintenance devices [ 8 , 19 ], comparing various methods of postoperative pain control (e.g., patient-controlled analgesia pumps, nerve block, or analgesics) [ 20 – 23 ], comparing the precision of various monitoring instruments [ 7 ], and meta-analysis of dose-response in various drugs [ 12 ].

Thus, literature reviews and meta-analyses are being conducted in diverse medical fields, and the aim of highlighting their importance is to help better extract accurate, good quality data from the flood of data being produced. However, a lack of understanding about systematic reviews and meta-analyses can lead to incorrect outcomes being derived from the review and analysis processes. If readers indiscriminately accept the results of the many meta-analyses that are published, incorrect data may be obtained. Therefore, in this review, we aim to describe the contents and methods used in systematic reviews and meta-analyses in a way that is easy to understand for future authors and readers of systematic review and meta-analysis.

Study Planning

It is easy to confuse systematic reviews and meta-analyses. A systematic review is an objective, reproducible method to find answers to a certain research question, by collecting all available studies related to that question and reviewing and analyzing their results. A meta-analysis differs from a systematic review in that it uses statistical methods on estimates from two or more different studies to form a pooled estimate [ 1 ]. Following a systematic review, if it is not possible to form a pooled estimate, it can be published as is without progressing to a meta-analysis; however, if it is possible to form a pooled estimate from the extracted data, a meta-analysis can be attempted. Systematic reviews and meta-analyses usually proceed according to the flowchart presented in Fig. 2 . We explain each of the stages below.

Flowchart illustrating a systematic review.

Formulating research questions

A systematic review attempts to gather all available empirical research by using clearly defined, systematic methods to obtain answers to a specific question. A meta-analysis is the statistical process of analyzing and combining results from several similar studies. Here, the definition of the word “similar” is not made clear, but when selecting a topic for the meta-analysis, it is essential to ensure that the different studies present data that can be combined. If the studies contain data on the same topic that can be combined, a meta-analysis can even be performed using data from only two studies. However, study selection via a systematic review is a precondition for performing a meta-analysis, and it is important to clearly define the Population, Intervention, Comparison, Outcomes (PICO) parameters that are central to evidence-based research. In addition, selection of the research topic is based on logical evidence, and it is important to select a topic that is familiar to readers without clearly confirmed the evidence [ 24 ].

Protocols and registration

In systematic reviews, prior registration of a detailed research plan is very important. In order to make the research process transparent, primary/secondary outcomes and methods are set in advance, and in the event of changes to the method, other researchers and readers are informed when, how, and why. Many studies are registered with an organization like PROSPERO ( http://www.crd.york.ac.uk/PROSPERO/ ), and the registration number is recorded when reporting the study, in order to share the protocol at the time of planning.

Defining inclusion and exclusion criteria

Information is included on the study design, patient characteristics, publication status (published or unpublished), language used, and research period. If there is a discrepancy between the number of patients included in the study and the number of patients included in the analysis, this needs to be clearly explained while describing the patient characteristics, to avoid confusing the reader.

Literature search and study selection

In order to secure proper basis for evidence-based research, it is essential to perform a broad search that includes as many studies as possible that meet the inclusion and exclusion criteria. Typically, the three bibliographic databases Medline, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) are used. In domestic studies, the Korean databases KoreaMed, KMBASE, and RISS4U may be included. Effort is required to identify not only published studies but also abstracts, ongoing studies, and studies awaiting publication. Among the studies retrieved in the search, the researchers remove duplicate studies, select studies that meet the inclusion/exclusion criteria based on the abstracts, and then make the final selection of studies based on their full text. In order to maintain transparency and objectivity throughout this process, study selection is conducted independently by at least two investigators. When there is a inconsistency in opinions, intervention is required via debate or by a third reviewer. The methods for this process also need to be planned in advance. It is essential to ensure the reproducibility of the literature selection process [ 25 ].

Quality of evidence

However, well planned the systematic review or meta-analysis is, if the quality of evidence in the studies is low, the quality of the meta-analysis decreases and incorrect results can be obtained [ 26 ]. Even when using randomized studies with a high quality of evidence, evaluating the quality of evidence precisely helps determine the strength of recommendations in the meta-analysis. One method of evaluating the quality of evidence in non-randomized studies is the Newcastle-Ottawa Scale, provided by the Ottawa Hospital Research Institute 1) . However, we are mostly focusing on meta-analyses that use randomized studies.

If the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) system ( http://www.gradeworkinggroup.org/ ) is used, the quality of evidence is evaluated on the basis of the study limitations, inaccuracies, incompleteness of outcome data, indirectness of evidence, and risk of publication bias, and this is used to determine the strength of recommendations [ 27 ]. As shown in Table 1 , the study limitations are evaluated using the “risk of bias” method proposed by Cochrane 2) . This method classifies bias in randomized studies as “low,” “high,” or “unclear” on the basis of the presence or absence of six processes (random sequence generation, allocation concealment, blinding participants or investigators, incomplete outcome data, selective reporting, and other biases) [ 28 ].

The Cochrane Collaboration’s Tool for Assessing the Risk of Bias [ 28 ]

Data extraction

Two different investigators extract data based on the objectives and form of the study; thereafter, the extracted data are reviewed. Since the size and format of each variable are different, the size and format of the outcomes are also different, and slight changes may be required when combining the data [ 29 ]. If there are differences in the size and format of the outcome variables that cause difficulties combining the data, such as the use of different evaluation instruments or different evaluation timepoints, the analysis may be limited to a systematic review. The investigators resolve differences of opinion by debate, and if they fail to reach a consensus, a third-reviewer is consulted.

Data Analysis

The aim of a meta-analysis is to derive a conclusion with increased power and accuracy than what could not be able to achieve in individual studies. Therefore, before analysis, it is crucial to evaluate the direction of effect, size of effect, homogeneity of effects among studies, and strength of evidence [ 30 ]. Thereafter, the data are reviewed qualitatively and quantitatively. If it is determined that the different research outcomes cannot be combined, all the results and characteristics of the individual studies are displayed in a table or in a descriptive form; this is referred to as a qualitative review. A meta-analysis is a quantitative review, in which the clinical effectiveness is evaluated by calculating the weighted pooled estimate for the interventions in at least two separate studies.

The pooled estimate is the outcome of the meta-analysis, and is typically explained using a forest plot ( Figs. 3 and ​ and4). 4 ). The black squares in the forest plot are the odds ratios (ORs) and 95% confidence intervals in each study. The area of the squares represents the weight reflected in the meta-analysis. The black diamond represents the OR and 95% confidence interval calculated across all the included studies. The bold vertical line represents a lack of therapeutic effect (OR = 1); if the confidence interval includes OR = 1, it means no significant difference was found between the treatment and control groups.

Forest plot analyzed by two different models using the same data. (A) Fixed-effect model. (B) Random-effect model. The figure depicts individual trials as filled squares with the relative sample size and the solid line as the 95% confidence interval of the difference. The diamond shape indicates the pooled estimate and uncertainty for the combined effect. The vertical line indicates the treatment group shows no effect (OR = 1). Moreover, if the confidence interval includes 1, then the result shows no evidence of difference between the treatment and control groups.

Forest plot representing homogeneous data.

Dichotomous variables and continuous variables

In data analysis, outcome variables can be considered broadly in terms of dichotomous variables and continuous variables. When combining data from continuous variables, the mean difference (MD) and standardized mean difference (SMD) are used ( Table 2 ).

Summary of Meta-analysis Methods Available in RevMan [ 28 ]

The MD is the absolute difference in mean values between the groups, and the SMD is the mean difference between groups divided by the standard deviation. When results are presented in the same units, the MD can be used, but when results are presented in different units, the SMD should be used. When the MD is used, the combined units must be shown. A value of “0” for the MD or SMD indicates that the effects of the new treatment method and the existing treatment method are the same. A value lower than “0” means the new treatment method is less effective than the existing method, and a value greater than “0” means the new treatment is more effective than the existing method.

When combining data for dichotomous variables, the OR, risk ratio (RR), or risk difference (RD) can be used. The RR and RD can be used for RCTs, quasi-experimental studies, or cohort studies, and the OR can be used for other case-control studies or cross-sectional studies. However, because the OR is difficult to interpret, using the RR and RD, if possible, is recommended. If the outcome variable is a dichotomous variable, it can be presented as the number needed to treat (NNT), which is the minimum number of patients who need to be treated in the intervention group, compared to the control group, for a given event to occur in at least one patient. Based on Table 3 , in an RCT, if x is the probability of the event occurring in the control group and y is the probability of the event occurring in the intervention group, then x = c/(c + d), y = a/(a + b), and the absolute risk reduction (ARR) = x − y. NNT can be obtained as the reciprocal, 1/ARR.

Calculation of the Number Needed to Treat in the Dichotomous table

Fixed-effect models and random-effect models

In order to analyze effect size, two types of models can be used: a fixed-effect model or a random-effect model. A fixed-effect model assumes that the effect of treatment is the same, and that variation between results in different studies is due to random error. Thus, a fixed-effect model can be used when the studies are considered to have the same design and methodology, or when the variability in results within a study is small, and the variance is thought to be due to random error. Three common methods are used for weighted estimation in a fixed-effect model: 1) inverse variance-weighted estimation 3) , 2) Mantel-Haenszel estimation 4) , and 3) Peto estimation 5) .

A random-effect model assumes heterogeneity between the studies being combined, and these models are used when the studies are assumed different, even if a heterogeneity test does not show a significant result. Unlike a fixed-effect model, a random-effect model assumes that the size of the effect of treatment differs among studies. Thus, differences in variation among studies are thought to be due to not only random error but also between-study variability in results. Therefore, weight does not decrease greatly for studies with a small number of patients. Among methods for weighted estimation in a random-effect model, the DerSimonian and Laird method 6) is mostly used for dichotomous variables, as the simplest method, while inverse variance-weighted estimation is used for continuous variables, as with fixed-effect models. These four methods are all used in Review Manager software (The Cochrane Collaboration, UK), and are described in a study by Deeks et al. [ 31 ] ( Table 2 ). However, when the number of studies included in the analysis is less than 10, the Hartung-Knapp-Sidik-Jonkman method 7) can better reduce the risk of type 1 error than does the DerSimonian and Laird method [ 32 ].

Fig. 3 shows the results of analyzing outcome data using a fixed-effect model (A) and a random-effect model (B). As shown in Fig. 3 , while the results from large studies are weighted more heavily in the fixed-effect model, studies are given relatively similar weights irrespective of study size in the random-effect model. Although identical data were being analyzed, as shown in Fig. 3 , the significant result in the fixed-effect model was no longer significant in the random-effect model. One representative example of the small study effect in a random-effect model is the meta-analysis by Li et al. [ 33 ]. In a large-scale study, intravenous injection of magnesium was unrelated to acute myocardial infarction, but in the random-effect model, which included numerous small studies, the small study effect resulted in an association being found between intravenous injection of magnesium and myocardial infarction. This small study effect can be controlled for by using a sensitivity analysis, which is performed to examine the contribution of each of the included studies to the final meta-analysis result. In particular, when heterogeneity is suspected in the study methods or results, by changing certain data or analytical methods, this method makes it possible to verify whether the changes affect the robustness of the results, and to examine the causes of such effects [ 34 ].

Heterogeneity

Homogeneity test is a method whether the degree of heterogeneity is greater than would be expected to occur naturally when the effect size calculated from several studies is higher than the sampling error. This makes it possible to test whether the effect size calculated from several studies is the same. Three types of homogeneity tests can be used: 1) forest plot, 2) Cochrane’s Q test (chi-squared), and 3) Higgins I 2 statistics. In the forest plot, as shown in Fig. 4 , greater overlap between the confidence intervals indicates greater homogeneity. For the Q statistic, when the P value of the chi-squared test, calculated from the forest plot in Fig. 4 , is less than 0.1, it is considered to show statistical heterogeneity and a random-effect can be used. Finally, I 2 can be used [ 35 ].

I 2 , calculated as shown above, returns a value between 0 and 100%. A value less than 25% is considered to show strong homogeneity, a value of 50% is average, and a value greater than 75% indicates strong heterogeneity.

Even when the data cannot be shown to be homogeneous, a fixed-effect model can be used, ignoring the heterogeneity, and all the study results can be presented individually, without combining them. However, in many cases, a random-effect model is applied, as described above, and a subgroup analysis or meta-regression analysis is performed to explain the heterogeneity. In a subgroup analysis, the data are divided into subgroups that are expected to be homogeneous, and these subgroups are analyzed. This needs to be planned in the predetermined protocol before starting the meta-analysis. A meta-regression analysis is similar to a normal regression analysis, except that the heterogeneity between studies is modeled. This process involves performing a regression analysis of the pooled estimate for covariance at the study level, and so it is usually not considered when the number of studies is less than 10. Here, univariate and multivariate regression analyses can both be considered.

Publication bias

Publication bias is the most common type of reporting bias in meta-analyses. This refers to the distortion of meta-analysis outcomes due to the higher likelihood of publication of statistically significant studies rather than non-significant studies. In order to test the presence or absence of publication bias, first, a funnel plot can be used ( Fig. 5 ). Studies are plotted on a scatter plot with effect size on the x-axis and precision or total sample size on the y-axis. If the points form an upside-down funnel shape, with a broad base that narrows towards the top of the plot, this indicates the absence of a publication bias ( Fig. 5A ) [ 29 , 36 ]. On the other hand, if the plot shows an asymmetric shape, with no points on one side of the graph, then publication bias can be suspected ( Fig. 5B ). Second, to test publication bias statistically, Begg and Mazumdar’s rank correlation test 8) [ 37 ] or Egger’s test 9) [ 29 ] can be used. If publication bias is detected, the trim-and-fill method 10) can be used to correct the bias [ 38 ]. Fig. 6 displays results that show publication bias in Egger’s test, which has then been corrected using the trim-and-fill method using Comprehensive Meta-Analysis software (Biostat, USA).

Funnel plot showing the effect size on the x-axis and sample size on the y-axis as a scatter plot. (A) Funnel plot without publication bias. The individual plots are broader at the bottom and narrower at the top. (B) Funnel plot with publication bias. The individual plots are located asymmetrically.

Funnel plot adjusted using the trim-and-fill method. White circles: comparisons included. Black circles: inputted comparisons using the trim-and-fill method. White diamond: pooled observed log risk ratio. Black diamond: pooled inputted log risk ratio.

Result Presentation

When reporting the results of a systematic review or meta-analysis, the analytical content and methods should be described in detail. First, a flowchart is displayed with the literature search and selection process according to the inclusion/exclusion criteria. Second, a table is shown with the characteristics of the included studies. A table should also be included with information related to the quality of evidence, such as GRADE ( Table 4 ). Third, the results of data analysis are shown in a forest plot and funnel plot. Fourth, if the results use dichotomous data, the NNT values can be reported, as described above.

The GRADE Evidence Quality for Each Outcome

N: number of studies, ROB: risk of bias, PON: postoperative nausea, POV: postoperative vomiting, PONV: postoperative nausea and vomiting, CI: confidence interval, RR: risk ratio, AR: absolute risk.

When Review Manager software (The Cochrane Collaboration, UK) is used for the analysis, two types of P values are given. The first is the P value from the z-test, which tests the null hypothesis that the intervention has no effect. The second P value is from the chi-squared test, which tests the null hypothesis for a lack of heterogeneity. The statistical result for the intervention effect, which is generally considered the most important result in meta-analyses, is the z-test P value.

A common mistake when reporting results is, given a z-test P value greater than 0.05, to say there was “no statistical significance” or “no difference.” When evaluating statistical significance in a meta-analysis, a P value lower than 0.05 can be explained as “a significant difference in the effects of the two treatment methods.” However, the P value may appear non-significant whether or not there is a difference between the two treatment methods. In such a situation, it is better to announce “there was no strong evidence for an effect,” and to present the P value and confidence intervals. Another common mistake is to think that a smaller P value is indicative of a more significant effect. In meta-analyses of large-scale studies, the P value is more greatly affected by the number of studies and patients included, rather than by the significance of the results; therefore, care should be taken when interpreting the results of a meta-analysis.

When performing a systematic literature review or meta-analysis, if the quality of studies is not properly evaluated or if proper methodology is not strictly applied, the results can be biased and the outcomes can be incorrect. However, when systematic reviews and meta-analyses are properly implemented, they can yield powerful results that could usually only be achieved using large-scale RCTs, which are difficult to perform in individual studies. As our understanding of evidence-based medicine increases and its importance is better appreciated, the number of systematic reviews and meta-analyses will keep increasing. However, indiscriminate acceptance of the results of all these meta-analyses can be dangerous, and hence, we recommend that their results be received critically on the basis of a more accurate understanding.

1) http://www.ohri.ca .

2) http://methods.cochrane.org/bias/assessing-risk-bias-included-studies .

3) The inverse variance-weighted estimation method is useful if the number of studies is small with large sample sizes.

4) The Mantel-Haenszel estimation method is useful if the number of studies is large with small sample sizes.

5) The Peto estimation method is useful if the event rate is low or one of the two groups shows zero incidence.

6) The most popular and simplest statistical method used in Review Manager and Comprehensive Meta-analysis software.

7) Alternative random-effect model meta-analysis that has more adequate error rates than does the common DerSimonian and Laird method, especially when the number of studies is small. However, even with the Hartung-Knapp-Sidik-Jonkman method, when there are less than five studies with very unequal sizes, extra caution is needed.

8) The Begg and Mazumdar rank correlation test uses the correlation between the ranks of effect sizes and the ranks of their variances [ 37 ].

9) The degree of funnel plot asymmetry as measured by the intercept from the regression of standard normal deviates against precision [ 29 ].

10) If there are more small studies on one side, we expect the suppression of studies on the other side. Trimming yields the adjusted effect size and reduces the variance of the effects by adding the original studies back into the analysis as a mirror image of each study.

Drivers of inappropriate use of antimicrobials in South Asia: A systematic review of qualitative literature

Affiliations.

• 1 Department of Population Health Sciences, University of Utah, Salt Lake City, Utah, United States of America.
• 2 School of Medicine, University of Utah, Salt Lake City, Utah, United States of America.
• 3 Department of Internal Medicine, University of Utah, Salt Lake City, Utah, United States of America.
• 4 International Centre for Diarrhoeal Disease Research, Bangladesh (icddr,b), Dhaka, Bangladesh.
• 5 Department of Internal Medicine, Intermountain Health, Murray, Utah, United States of America.
• PMID: 38573955
• PMCID: PMC10994369
• DOI: 10.1371/journal.pgph.0002507

Antimicrobial resistance is a global public health crisis. Effective antimicrobial stewardship requires an understanding of the factors and context that contribute to inappropriate use of antimicrobials. The goal of this qualitative systematic review was to synthesize themes across levels of the social ecological framework that drive inappropriate use of antimicrobials in South Asia. In September 2023, we conducted a systematic search using the electronic databases PubMed and Embase. Search terms, identified a priori, were related to research methods, topic, and geographic location. We identified 165 articles from the initial search and 8 upon reference review (n = 173); after removing duplicates and preprints (n = 12) and excluding those that did not meet eligibility criteria (n = 115), 46 articles were included in the review. We assessed methodological quality using the qualitative Critical Appraisal Skills Program checklist. The studies represented 6 countries in South Asia, and included data from patients, health care providers, community members, and policy makers. For each manuscript, we wrote a summary memo to extract the factors that impede antimicrobial stewardship. We coded memos using NVivo software; codes were organized by levels of the social ecological framework. Barriers were identified at multiple levels including the patient (self-treatment with antimicrobials; perceived value of antimicrobials), the provider (antimicrobials as a universal therapy; gaps in knowledge and skills; financial or reputational incentives), the clinical setting (lack of resources; poor regulation of the facility), the community (access to formal health care; informal drug vendors; social norms), and policy (absence of a regulatory framework; poor implementation of existing policies). This study is the first to succinctly identify a range of norms, behaviors, and policy contexts driving inappropriate use of antimicrobials in South Asia, emphasizing the importance of working across multiple sectors to design and implement approaches specific to the region.

Copyright: © 2024 Murray et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Grants and funding

• K24 AI166087/AI/NIAID NIH HHS/United States
• R01 AI135114/AI/NIAID NIH HHS/United States
• R21 HD109819/HD/NICHD NIH HHS/United States