what are the benefits of a systematic literature review

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved April 9, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

Systematic reviews: the good, the bad, and the ugly

Affiliation.

• 1 Division of Gastroenterology, Department of Medicine, McMaster University Health Science Centre, Hamilton, Ontario, Canada.
• PMID: 19417748
• DOI: 10.1038/ajg.2009.118

Systematic reviews systematically evaluate and summarize current knowledge and have many advantages over narrative reviews. Meta-analyses provide a more reliable and enhanced precision of effect estimate than do individual studies. Systematic reviews are invaluable for defining the methods used in subsequent studies, but, as retrospective research projects, they are subject to bias. Rigorous research methods are essential, and the quality depends on the extent to which scientific review methods are used. Systematic reviews can be misleading, unhelpful, or even harmful when data are inappropriately handled; meta-analyses can be misused when the difference between a patient seen in the clinic and those included in the meta-analysis is not considered. Furthermore, systematic reviews cannot answer all clinically relevant questions, and their conclusions may be difficult to incorporate into practice. They should be reviewed on an ongoing basis. As clinicians, we need proper methodological training to perform good systematic reviews and must ask the appropriate questions before we can properly interpret such a review and apply its conclusions to our patients. This paper aims to assist in the reading of a systematic review.

Publication types

• Comparative Study
• Systematic Review
• Evidence-Based Medicine / standards*
• Evidence-Based Medicine / trends
• Gastroenterology*
• Meta-Analysis as Topic*
• Randomized Controlled Trials as Topic
• Reproducibility of Results
• Research Design
• Review Literature as Topic*
• Sensitivity and Specificity

• Open access
• Published: 14 August 2018

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

• Chris Cooper   ORCID: orcid.org/0000-0003-0864-5607 1 ,
• Andrew Booth 2 ,
• Jo Varley-Campbell 1 ,
• Nicky Britten 3 &
• Ruth Garside 4  

BMC Medical Research Methodology volume  18 , Article number:  85 ( 2018 ) Cite this article

200k Accesses

202 Citations

118 Altmetric

Metrics details

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before.

The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

A literature review.

Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through ‘pearl growing’, citation chasing, a search of PubMed using the systematic review methods filter, and the authors’ topic knowledge.

The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of ‘key stages’ in the process of literature searching.

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

Conclusions

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Peer Review reports

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving review stakeholders clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. This is in contrast to the information science literature, which has developed information processing models as an explicit basis for dialogue and empirical testing. Without an explicit model, research in the process of systematic literature searching will remain immature and potentially uneven, and the development of shared information models will be assumed but never articulated.

One way of developing such a conceptual model is by formally examining the implicit “programme theory” as embodied in key methodological texts. The aim of this review is therefore to determine if a shared model of the literature searching process in systematic reviews can be detected across guidance documents and, if so, how this process is reported and supported.

Identifying guidance

Key texts (henceforth referred to as “guidance”) were identified based upon their accessibility to, and prominence within, United Kingdom systematic reviewing practice. The United Kingdom occupies a prominent position in the science of health information retrieval, as quantified by such objective measures as the authorship of papers, the number of Cochrane groups based in the UK, membership and leadership of groups such as the Cochrane Information Retrieval Methods Group, the HTA-I Information Specialists’ Group and historic association with such centres as the UK Cochrane Centre, the NHS Centre for Reviews and Dissemination, the Centre for Evidence Based Medicine and the National Institute for Clinical Excellence (NICE). Coupled with the linguistic dominance of English within medical and health science and the science of systematic reviews more generally, this offers a justification for a purposive sample that favours UK, European and Australian guidance documents.

Nine guidance documents were identified. These documents provide guidance for different types of reviews, namely: reviews of interventions, reviews of health technologies, reviews of qualitative research studies, reviews of social science topics, and reviews to inform guidance.

Whilst these guidance documents occasionally offer additional guidance on other types of systematic reviews, we have focused on the core and stated aims of these documents as they relate to literature searching. Table  1 sets out: the guidance document, the version audited, their core stated focus, and a bibliographical pointer to the main guidance relating to literature searching.

Once a list of key guidance documents was determined, it was checked by six senior information professionals based in the UK for relevance to current literature searching in systematic reviews.

Identifying supporting studies

In addition to identifying guidance, the authors sought to populate an evidence base of supporting studies (henceforth referred to as “studies”) that contribute to existing search practice. Studies were first identified by the authors from their knowledge on this topic area and, subsequently, through systematic citation chasing key studies (‘pearls’ [ 1 ]) located within each key stage of the search process. These studies are identified in Additional file  1 : Appendix Table 1. Citation chasing was conducted by analysing the bibliography of references for each study (backwards citation chasing) and through Google Scholar (forward citation chasing). A search of PubMed using the systematic review methods filter was undertaken in August 2017 (see Additional file 1 ). The search terms used were: (literature search*[Title/Abstract]) AND sysrev_methods[sb] and 586 results were returned. These results were sifted for relevance to the key stages in Fig.  1 by CC.

The key stages of literature search guidance as identified from nine key texts

Extracting the data

To reveal the implicit process of literature searching within each guidance document, the relevant sections (chapters) on literature searching were read and re-read, with the aim of determining key methodological stages. We defined a key methodological stage as a distinct step in the overall process for which specific guidance is reported, and action is taken, that collectively would result in a completed literature search.

The chapter or section sub-heading for each methodological stage was extracted into a table using the exact language as reported in each guidance document. The lead author (CC) then read and re-read these data, and the paragraphs of the document to which the headings referred, summarising section details. This table was then reviewed, using comparison and contrast to identify agreements and areas of unique guidance. Consensus across multiple guidelines was used to inform selection of ‘key stages’ in the process of literature searching.

Having determined the key stages to literature searching, we then read and re-read the sections relating to literature searching again, extracting specific detail relating to the methodological process of literature searching within each key stage. Again, the guidance was then read and re-read, first on a document-by-document-basis and, secondly, across all the documents above, to identify both commonalities and areas of unique guidance.

Results and discussion

Our findings.

We were able to identify consensus across the guidance on literature searching for systematic reviews suggesting a shared implicit model within the information retrieval community. Whilst the structure of the guidance varies between documents, the same key stages are reported, even where the core focus of each document is different. We were able to identify specific areas of unique guidance, where a document reported guidance not summarised in other documents, together with areas of consensus across guidance.

Unique guidance

Only one document provided guidance on the topic of when to stop searching [ 2 ]. This guidance from 2005 anticipates a topic of increasing importance with the current interest in time-limited (i.e. “rapid”) reviews. Quality assurance (or peer review) of literature searches was only covered in two guidance documents [ 3 , 4 ]. This topic has emerged as increasingly important as indicated by the development of the PRESS instrument [ 5 ]. Text mining was discussed in four guidance documents [ 4 , 6 , 7 , 8 ] where the automation of some manual review work may offer efficiencies in literature searching [ 8 ].

Agreement between guidance: Defining the key stages of literature searching

Where there was agreement on the process, we determined that this constituted a key stage in the process of literature searching to inform systematic reviews.

From the guidance, we determined eight key stages that relate specifically to literature searching in systematic reviews. These are summarised at Fig. 1 . The data extraction table to inform Fig. 1 is reported in Table  2 . Table 2 reports the areas of common agreement and it demonstrates that the language used to describe key stages and processes varies significantly between guidance documents.

For each key stage, we set out the specific guidance, followed by discussion on how this guidance is situated within the wider literature.

Key stage one: Deciding who should undertake the literature search

The guidance.

Eight documents provided guidance on who should undertake literature searching in systematic reviews [ 2 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [ 6 ]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [ 2 , 7 , 8 , 9 , 10 , 11 ].

How the guidance corresponds to the published studies

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [ 13 ] in a variety of roles [ 12 , 16 , 20 , 21 , 24 , 25 , 26 ]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [ 17 , 18 , 27 , 28 ] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [ 29 ] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [ 29 ]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [ 30 ].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [ 2 , 6 ]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [ 12 , 15 , 16 , 31 ].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [ 31 ] have questioned if the necessary involvement is actually happening [ 31 ].

Key stage two: Determining the aim and purpose of a literature search

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [ 6 , 7 , 8 , 9 , 10 ]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [ 4 ] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [ 2 ] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [ 3 ]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [ 6 , 8 , 10 ]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

The need for thorough and comprehensive literature searches appears as uniform within the eight guidance documents that describe approaches to literature searching in systematic reviews of effectiveness. Reviews of effectiveness (of intervention or cost), accuracy and prognosis, require thorough and comprehensive literature searches to transparently produce a reliable estimate of intervention effect. The belief that all relevant studies have been ‘comprehensively’ identified, and that this process has been ‘transparently’ reported, increases confidence in the estimate of effect and the conclusions that can be drawn [ 32 ]. The supporting literature exploring the need for comprehensive literature searches focuses almost exclusively on reviews of intervention effectiveness and meta-analysis. Different ‘styles’ of review may have different standards however; the alternative, offered by purposive sampling, has been suggested in the specific context of qualitative evidence syntheses [ 33 ].

What is a comprehensive literature search?

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table 2 should be used to identify studies. Egger et al. [ 34 ], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

a search not restricted to English language;

where Cochrane CENTRAL or at least two other electronic databases had been searched (such as MEDLINE or EMBASE); and

at least one of the following search methods has been used to identify unpublished trials: searches for (I) conference abstracts, (ii) theses, (iii) trials registers; and (iv) contacts with experts in the field [ 34 ].

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [ 35 ].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [ 34 ]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [ 9 ]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [ 36 ].

What is the point of comprehensive literature searching?

The purpose of thorough and comprehensive literature searches is to avoid missing key studies and to minimize bias [ 6 , 8 , 10 , 34 , 37 , 38 , 39 ] since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size [ 35 ]. Felson (1992) sets out potential biases that could affect the estimate of effect in a meta-analysis [ 40 ] and Tricco et al. summarize the evidence concerning bias and confounding in systematic reviews [ 35 ]. Egger et al. point to non-publication of studies, publication bias, language bias and MEDLINE bias, as key biases [ 34 , 35 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. Comprehensive searches are not the sole factor to mitigate these biases but their contribution is thought to be significant [ 2 , 32 , 34 ]. Fehrmann (2011) suggests that ‘the search process being described in detail’ and that, where standard comprehensive search techniques have been applied, increases confidence in the search results [ 32 ].

Does comprehensive literature searching work?

Egger et al., and other study authors, have demonstrated a change in the estimate of intervention effectiveness where relevant studies were excluded from meta-analysis [ 34 , 47 ]. This would suggest that missing studies in literature searching alters the reliability of effectiveness estimates. This is an argument for comprehensive literature searching. Conversely, Egger et al. found that ‘comprehensive’ searches still missed studies and that comprehensive searches could, in fact, introduce bias into a review rather than preventing it, through the identification of low quality studies then being included in the meta-analysis [ 34 ]. Studies query if identifying and including low quality or grey literature studies changes the estimate of effect [ 43 , 48 ] and question if time is better invested updating systematic reviews rather than searching for unpublished studies [ 49 ], or mapping studies for review as opposed to aiming for high sensitivity in literature searching [ 50 ].

Aim and purpose beyond reviews of effectiveness

The need for comprehensive literature searches is less certain in reviews of qualitative studies, and for reviews where a comprehensive identification of studies is difficult to achieve (for example, in Public health) [ 33 , 51 , 52 , 53 , 54 , 55 ]. Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al. (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ]. Different approaches to literature searching [ 37 , 38 , 59 , 60 , 61 , 62 ] and developing the concept of when to stop searching are important areas for further study [ 36 , 59 ].

The study by Nussbaumer-Streit et al. has been published since the submission of this literature review [ 63 ]. Nussbaumer-Streit et al. (2018) conclude that abbreviated literature searches are viable options for rapid evidence syntheses, if decision-makers are willing to trade the certainty from a comprehensive literature search and systematic review, but that decision-making which demands detailed scrutiny should still be based on comprehensive literature searches [ 63 ].

Key stage three: Preparing for the literature search

Six documents provided guidance on preparing for a literature search [ 2 , 3 , 6 , 7 , 9 , 10 ]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [ 9 ].

Two key tasks were perceptible in preparing for a literature searching [ 2 , 6 , 7 , 10 , 11 ]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [ 6 , 11 ]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [ 10 ]) and indicate the resources required for literature searching and the review of the studies that follows [ 7 , 10 ].

Three documents summarised guidance on where to search to determine if a new review was justified [ 2 , 6 , 11 ]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [ 6 , 11 ]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [ 64 ]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [ 2 ].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [ 7 , 10 ]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [ 65 ]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [ 66 , 67 ]. Preparing for literature searches and scoping constitutes an area for future research.

Key stage four: Designing the search strategy

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [ 1 , 4 , 7 , 8 , 9 ]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [ 4 ].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [ 10 ].

The use of limits, such as language or date limits, were discussed in all documents [ 2 , 3 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ].

Search strategy structure

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [ 68 ]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [ 6 , 68 ]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [ 69 ] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [ 55 ]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [ 69 ] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [ 70 ]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [ 68 ].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [ 4 ]. Glanville developed this idea in a study by Whitting et al. [ 71 ] and a worked example of this approach is included in the development of a search filter by Cooper et al. [ 72 ].

Writing search strategies: Conceptual and objective approaches

Hausner et al. [ 73 ] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [ 73 ] and describe what may be done to validate it [ 74 ].

The use of limits

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [ 34 ] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [ 75 ].

Key stage five: Determining the process of literature searching and deciding where to search (bibliographic database searching)

Table 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [ 4 , 9 , 11 ]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [ 2 , 3 , 4 , 6 , 7 , 8 , 10 ], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [ 2 , 6 ].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [ 76 ], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [ 77 ], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [ 78 ]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [ 79 ] report differing search returns from the same database (AMED) but where the ‘host’ was different [ 79 ].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [ 80 ] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [ 48 ]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [ 81 ]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [ 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ].

When to database search is another question posed in the literature. Beyer et al. [ 91 ] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [ 91 ]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [ 92 , 93 ]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [ 94 ].

Key stage six: Determining the process of literature searching and deciding where to search (supplementary search methods)

Table 2 also summaries the process of literature searching which follows bibliographic database searching. As Table 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [ 95 ]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [ 9 ]. Four guidance documents [ 2 , 3 , 6 , 9 ] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [ 2 ]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [ 37 , 61 , 62 , 96 , 97 , 98 , 99 , 100 , 101 ] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [ 95 ]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [ 95 ]. The guidance is limited in this regard and, as Table 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Key stage seven: Managing the references

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [ 2 , 4 , 6 , 8 , 10 ]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [ 2 , 4 , 6 , 8 ]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [ 10 ].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [ 102 ]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [ 103 , 104 ] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [ 105 ]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [ 106 , 107 ]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

Key stage eight: Documenting the search

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 9 ]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [ 3 , 4 , 6 , 8 , 9 , 10 ]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [ 6 ])). Three guidance documents reported that the number of studies identified should be recorded [ 3 , 6 , 10 ]. The number of duplicates identified [ 10 ], the screening decisions [ 3 ], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [ 8 ], and an annotation of search terms tested but not used [ 4 ] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [ 9 ].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [ 108 ]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [ 108 ]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [ 109 ] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [ 110 ]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [ 52 , 109 ]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [ 31 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 ]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [ 120 ] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [ 120 ]. Atkinson et al. [ 121 ] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [ 121 ].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [ 5 , 122 , 123 ]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [ 120 ]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [ 108 ]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [ 124 ]) and ROBIS (Whiting et al. [ 125 ]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [ 124 , 125 ]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [ 120 ].

Limitations

Other handbooks exist.

A potential limitation of this literature review is the focus on guidance produced in Europe (the UK specifically) and Australia. We justify the decision for our selection of the nine guidance documents reviewed in this literature review in section “ Identifying guidance ”. In brief, these nine guidance documents were selected as the most relevant health care guidance that inform UK systematic reviewing practice, given that the UK occupies a prominent position in the science of health information retrieval. We acknowledge the existence of other guidance documents, such as those from North America (e.g. the Agency for Healthcare Research and Quality (AHRQ) [ 126 ], The Institute of Medicine [ 127 ] and the guidance and resources produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 128 ]). We comment further on this directly below.

The handbooks are potentially linked to one another

What is not clear is the extent to which the guidance documents inter-relate or provide guidance uniquely. The Cochrane Handbook, first published in 1994, is notably a key source of reference in guidance and systematic reviews beyond Cochrane reviews. It is not clear to what extent broadening the sample of guidance handbooks to include North American handbooks, and guidance handbooks from other relevant countries too, would alter the findings of this literature review or develop further support for the process model. Since we cannot be clear, we raise this as a potential limitation of this literature review. On our initial review of a sample of North American, and other, guidance documents (before selecting the guidance documents considered in this review), however, we do not consider that the inclusion of these further handbooks would alter significantly the findings of this literature review.

This is a literature review

A further limitation of this review was that the review of published studies is not a systematic review of the evidence for each key stage. It is possible that other relevant studies could help contribute to the exploration and development of the key stages identified in this review.

This literature review would appear to demonstrate the existence of a shared model of the literature searching process in systematic reviews. We call this model ‘the conventional approach’, since it appears to be common convention in nine different guidance documents.

The findings reported above reveal eight key stages in the process of literature searching for systematic reviews. These key stages are consistently reported in the nine guidance documents which suggests consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews.

In Table 2 , we demonstrate consensus regarding the application of literature search methods. All guidance documents distinguish between primary and supplementary search methods. Bibliographic database searching is consistently the first method of literature searching referenced in each guidance document. Whilst the guidance uniformly supports the use of supplementary search methods, there is little evidence for a consistent process with diverse guidance across documents. This may reflect differences in the core focus across each document, linked to differences in identifying effectiveness studies or qualitative studies, for instance.

Eight of the nine guidance documents reported on the aims of literature searching. The shared understanding was that literature searching should be thorough and comprehensive in its aim and that this process should be reported transparently so that that it could be reproduced. Whilst only three documents explicitly link this understanding to minimising bias, it is clear that comprehensive literature searching is implicitly linked to ‘not missing relevant studies’ which is approximately the same point.

Defining the key stages in this review helps categorise the scholarship available, and it prioritises areas for development or further study. The supporting studies on preparing for literature searching (key stage three, ‘preparation’) were, for example, comparatively few, and yet this key stage represents a decisive moment in literature searching for systematic reviews. It is where search strategy structure is determined, search terms are chosen or discarded, and the resources to be searched are selected. Information specialists, librarians and researchers, are well placed to develop these and other areas within the key stages we identify.

This review calls for further research to determine the suitability of using the conventional approach. The publication dates of the guidance documents which underpin the conventional approach may raise questions as to whether the process which they each report remains valid for current systematic literature searching. In addition, it may be useful to test whether it is desirable to use the same process model of literature searching for qualitative evidence synthesis as that for reviews of intervention effectiveness, which this literature review demonstrates is presently recommended best practice.

Abbreviations

Behaviour of interest; Health context; Exclusions; Models or Theories

Cochrane Database of Systematic Reviews

The Cochrane Central Register of Controlled Trials

Database of Abstracts of Reviews of Effects

Enhancing transparency in reporting the synthesis of qualitative research

Institute for Quality and Efficiency in Healthcare

National Institute for Clinical Excellence

Population, Intervention, Comparator, Outcome

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Setting, Perspective, Intervention, Comparison, Evaluation

Sample, Phenomenon of Interest, Design, Evaluation, Research type

STrengthening the Reporting of OBservational studies in Epidemiology

Trial Search Co-ordinators

Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Information & Libraries Journal. 2008;25(4):313–7.

Article   Google Scholar  

Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing Ltd; 2006.

Book   Google Scholar  

Institute for Quality and Efficiency in Health Care (IQWiG). IQWiG Methods Resources. 7 Information retrieval 2014 [Available from: https://www.ncbi.nlm.nih.gov/books/NBK385787/ .

NICE: National Institute for Health and Care Excellence. Developing NICE guidelines: the manual 2014. Available from: https://www.nice.org.uk/media/default/about/what-we-do/our-programmes/developing-nice-guidelines-the-manual.pdf .

Sampson M. MJ, Lefebvre C, Moher D, Grimshaw J. Peer Review of Electronic Search Strategies: PRESS; 2008.

Google Scholar  

Centre for Reviews & Dissemination. Systematic reviews – CRD’s guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009.

eunetha: European Network for Health Technology Assesment Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness 2016. Available from: http://www.eunethta.eu/sites/default/files/Guideline_Information_Retrieval_V1-1.pdf .

Kugley SWA, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: Campbell Collaboration. 2017; Available from: https://www.campbellcollaboration.org/library/searching-for-studies-information-retrieval-guide-campbell-reviews.html

Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions; 2011.

Collaboration for Environmental Evidence. Guidelines for Systematic Review and Evidence Synthesis in Environmental Management.: Environmental Evidence:; 2013. Available from: http://www.environmentalevidence.org/wp-content/uploads/2017/01/Review-guidelines-version-4.2-final-update.pdf .

The Joanna Briggs Institute. Joanna Briggs institute reviewers’ manual. 2014th ed: the Joanna Briggs institute; 2014. Available from: https://joannabriggs.org/assets/docs/sumari/ReviewersManual-2014.pdf

Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Inf Libr J. 2003;20(2):65–74.

Article   CAS   Google Scholar  

Harris MR. The librarian's roles in the systematic review process: a case study. Journal of the Medical Library Association. 2005;93(1):81–7.

PubMed   PubMed Central   Google Scholar  

Egger JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015;10(5):e0125931.

Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.

Article   PubMed   Google Scholar  

McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26.

Weller AC. Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports. J Med Libr Assoc. 2004;92(2):163–4.

Swinkels A, Briddon J, Hall J. Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Info Libr J. 2006;23(4):248–56.

Foster M. An overview of the role of librarians in systematic reviews: from expert search to project manager. EAHIL. 2015;11(3):3–7.

Lawson L. OPERATING OUTSIDE LIBRARY WALLS 2004.

Vassar M, Yerokhin V, Sinnett PM, Weiher M, Muckelrath H, Carr B, et al. Database selection in systematic reviews: an insight through clinical neurology. Health Inf Libr J. 2017;34(2):156–64.

Townsend WA, Anderson PF, Ginier EC, MacEachern MP, Saylor KM, Shipman BL, et al. A competency framework for librarians involved in systematic reviews. Journal of the Medical Library Association : JMLA. 2017;105(3):268–75.

Cooper ID, Crum JA. New activities and changing roles of health sciences librarians: a systematic review, 1990-2012. Journal of the Medical Library Association : JMLA. 2013;101(4):268–77.

Crum JA, Cooper ID. Emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. Journal of the Medical Library Association : JMLA. 2013;101(4):278–86.

Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301–15.

Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440–8.

Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Academic medicine : journal of the Association of American Medical Colleges. 2011;86(8):1049–54.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association : JMLA. 2016;104(4):267–77.

Morris M, Boruff JT, Gore GC. Scoping reviews: establishing the role of the librarian. Journal of the Medical Library Association : JMLA. 2016;104(4):346–54.

Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016;11(9):e0163309.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods. 2011;2(1):15–32.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews. 2016;5(1):74.

Article   PubMed   PubMed Central   Google Scholar  

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76.

Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol. 2008;61(5):422–34.

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010;26(4):431–5.

Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010;27(2):114–22.

Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Systematic Reviews. 2015;4(1):36.

Betrán AP, Say L, Gülmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5

Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45(8):885–92.

Article   PubMed   CAS   Google Scholar  

Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: unlocking the file drawer. Science. 2014;345(6203):1502–5.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12(4):e0176210.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet (London, England). 1997;350(9074):326–9.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76.

Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki A-A, Ioannidis JPA. The effects of excluding treatments from network meta-analyses: survey. BMJ : British Medical Journal. 2013;347

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127.

van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–44.e3.

Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches. German medical science : GMS e-journal. 2016;14:Doc09.

Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Research Synthesis Methods. 2012;3(1):1–10.

Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Res Pap Educ. 2007;22(2):213–28.

Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25(2):153–78.

Britten N, Garside R, Pope C, Frost J, Cooper C. Asking more of qualitative synthesis: a response to Sally Thorne. Qual Health Res. 2017;27(9):1370–6.

Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015;32(3):220–35.

Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014;3:135.

Nussbaumer-Streit B, Klerings I, Wagner G, Titscher V, Gartlehner G. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Systematic Reviews. 2016;5:197.

Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17(1):121.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603.

Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Eval Health Prof. 2011;34(3):297–308.

Levay P, Raynor M, Tuvey D. The Contributions of MEDLINE, Other Bibliographic Databases and Various Search Techniques to NICE Public Health Guidance. 2015. 2015;10(1):19.

Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, et al. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018;102:1–11.

Briscoe S, Cooper C, Glanville J, Lefebvre C. The loss of the NHS EED and DARE databases and the effect on evidence synthesis and evaluation. Res Synth Methods. 2017;8(3):256–7.

Stansfield C, O'Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods.n/a-n/a.

Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: a word frequency analysis study. Journal of the American Medical Informatics Association : JAMIA. 2012;19(3):479–88.

Stansfield C, Thomas J, Kavanagh J. 'Clustering' documents automatically to support scoping reviews of research: a case study. Res Synth Methods. 2013;4(3):230–41.

PubMed   Google Scholar  

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Andrew B. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355–68.

Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43.

Whiting P, Westwood M, Bojke L, Palmer S, Richardson G, Cooper J, et al. Clinical effectiveness and cost-effectiveness of tests for the diagnosis and investigation of urinary tract infection in children: a systematic review and economic model. Health technology assessment (Winchester, England). 2006;10(36):iii-iv, xi-xiii, 1–154.

Cooper C, Levay P, Lorenc T, Craig GM. A population search filter for hard-to-reach populations increased search efficiency for a systematic review. J Clin Epidemiol. 2014;67(5):554–9.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Systematic Reviews. 2012;1(1):19.

Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016;77:118–24.

Craven J, Levay P. Recording database searches for systematic reviews - what is the value of adding a narrative to peer-review checklists? A case study of nice interventional procedures guidance. Evid Based Libr Inf Pract. 2011;6(4):72–87.

Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev. 2015;4:104.

Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, et al. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013;66(9):1051–7.

Cooper C, Rogers M, Bethel A, Briscoe S, Lowe J. A mapping review of the literature on UK-focused health and social care databases. Health Inf Libr J. 2015;32(1):5–22.

Younger P, Boddy K. When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Inf Libr J. 2009;26(2):126–35.

Lam MT, McDiarmid M. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. Journal of the Medical Library Association : JMLA. 2016;104(4):284–9.

Bethel A, editor Search summary tables for systematic reviews: results and findings. HLC Conference 2017a.

Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16(1):161.

Adams CE, Frederick K. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24(3):741–8.

Kelly L, St Pierre-Hansen N. So many databases, such little clarity: searching the literature for the topic aboriginal. Canadian family physician Medecin de famille canadien. 2008;54(11):1572–3.

Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention. 2008;14(6):401–4.

Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine. 2004;12(4):228–32.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87.

Taylor B, Wylie E, Dempster M, Donnelly M. Systematically retrieving research: a case study evaluating seven databases. Res Soc Work Pract. 2007;17(6):697–706.

Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58.

Duffy S, de Kock S, Misso K, Noake C, Ross J, Stirk L. Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE in-process searches via Ovid. Journal of the Medical Library Association : JMLA. 2016;104(4):309–12.

Katchamart W, Faulkner A, Feldman B, Tomlinson G, Bombardier C. PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. J Clin Epidemiol. 2011;64(7):805–7.

Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement (in Press). Research Synthesis Methods. 2017;

Cooper C, Booth, A., Britten, N., Garside, R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: A methodological review. (In Press). BMC Systematic Reviews. 2017.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ (Clinical research ed). 2005;331(7524):1064–5.

Article   PubMed Central   Google Scholar  

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. PharmacoEconomics. 2015;33(1):5–11.

Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with web of science and Google scholar. Res Synth Methods. 2016;7(1):34–45.

McManus RJ, Wilson S, Delaney BC, Fitzmaurice DA, Hyde CJ, Tobias RS, et al. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ (Clinical research ed). 1998;317(7172):1562–3.

Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Journal of public health research. 2014;3(2):177.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.

Bethel A. Endnote Training (YouTube Videos) 2017b [Available from: http://medicine.exeter.ac.uk/esmi/workstreams/informationscience/is_resources,_guidance_&_advice/ .

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240–3.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association : JMLA. 2017;105(1):84–7.

Gall C, Brahmi FA. Retrieval comparison of EndNote to search MEDLINE (Ovid and PubMed) versus searching them directly. Medical reference services quarterly. 2004;23(3):25–32.

Ahmed KK, Al Dhubaib BE. Zotero: a bibliographic assistant to researcher. J Pharmacol Pharmacother. 2011;2(4):303–5.

Coar JT, Sewell JP. Zotero: harnessing the power of a personal bibliographic manager. Nurse Educ. 2010;35(5):205–7.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748–54.

Toews LC. Compliance of systematic reviews in veterinary journals with preferred reporting items for systematic reviews and meta-analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association : JMLA. 2017;105(3):233–9.

Booth A. "brimful of STARLITE": toward standards for reporting literature searches. Journal of the Medical Library Association : JMLA. 2006;94(4):421–9. e205

Faggion CM Jr, Wu YC, Tu YK, Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery. Br J Radiol. 2016;89(1062):20150878.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods. 2014;5(2):116–30.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association : JMLA. 2009;97(1):21–9.

Bigna JJ, Um LN, Nansseu JR. A comparison of quality of abstracts of systematic reviews including meta-analysis of randomized controlled trials in high-impact general medicine journals before and after the publication of PRISMA extension for abstracts: a systematic review and meta-analysis. Syst Rev. 2016;5(1):174.

Akhigbe T, Zolnourian A, Bulters D. Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature. Journal of clinical neuroscience : official journal of the Neurosurgical Society of Australasia. 2017;39:45–8.

Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011;6(11):e27611.

Wasiak J, Tyack Z, Ware R. Goodwin N. Jr. Poor methodological quality and reporting standards of systematic reviews in burn care management. International wound journal: Faggion CM; 2016.

Tam WW, Lo KK, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7(2):e013905.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98–115.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods. 2015;6(1):87–95.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

Relevo R, Balshem H. Finding evidence for comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. 2011;64(11):1168–77.

Medicine Io. Standards for Systematic Reviews 2011 [Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx .

CADTH: Resources 2018.

Download references

Acknowledgements

CC acknowledges the supervision offered by Professor Chris Hyde.

This publication forms a part of CC’s PhD. CC’s PhD was funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The open access fee for this publication was paid for by Exeter Medical School.

RG and NB were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula.

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Authors and affiliations.

Institute of Health Research, University of Exeter Medical School, Exeter, UK

Chris Cooper & Jo Varley-Campbell

HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

Andrew Booth

Nicky Britten

European Centre for Environment and Human Health, University of Exeter Medical School, Truro, UK

Ruth Garside

You can also search for this author in PubMed   Google Scholar

Contributions

CC conceived the idea for this study and wrote the first draft of the manuscript. CC discussed this publication in PhD supervision with AB and separately with JVC. CC revised the publication with input and comments from AB, JVC, RG and NB. All authors revised the manuscript prior to submission. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chris Cooper .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Appendix tables and PubMed search strategy. Key studies used for pearl growing per key stage, working data extraction tables and the PubMed search strategy. (DOCX 30 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cooper, C., Booth, A., Varley-Campbell, J. et al. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18 , 85 (2018). https://doi.org/10.1186/s12874-018-0545-3

Download citation

Received : 20 September 2017

Accepted : 06 August 2018

Published : 14 August 2018

DOI : https://doi.org/10.1186/s12874-018-0545-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Literature Search Process
• Citation Chasing
• Tacit Models
• Unique Guidance
• Information Specialists

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• Open access
• Published: 15 December 2015

Qualitative and mixed methods in systematic reviews

• David Gough 1  

Systematic Reviews volume  4 , Article number:  181 ( 2015 ) Cite this article

22k Accesses

46 Citations

23 Altmetric

Metrics details

Expanding the range of methods of systematic review

The logic of systematic reviews is very simple. We use transparent rigorous approaches to undertake primary research, and so we should do the same in bringing together studies to describe what has been studied (a research map) or to integrate the findings of the different studies to answer a research question (a research synthesis). We should not really need to use the term ‘systematic’ as it should be assumed that researchers are using and reporting systematic methods in all of their research, whether primary or secondary. Despite the universality of this logic, systematic reviews (maps and syntheses) are much better known in health research and for answering questions of the effectiveness of interventions (what works). Systematic reviews addressing other sorts of questions have been around for many years, as in, for example, meta ethnography [ 1 ] and other forms of conceptual synthesis [ 2 ], but only recently has there been a major increase in the use of systematic review approaches to answer other sorts of research questions.

There are probably several reasons for this broadening of approach. One may be that the increased awareness of systematic reviews has made people consider the possibilities for all areas of research. A second related factor may be that more training and funding resources have become available and increased the capacity to undertake such varied review work.

A third reason could be that some of the initial anxieties about systematic reviews have subsided. Initially, there were concerns that their use was being promoted by a new managerialism where reviews, particularly effectiveness reviews, were being used to promote particular ideological and theoretical assumptions and to indirectly control research agendas. However, others like me believe that explicit methods should be used to enable transparency of perspectives driving research and to open up access to and participation in research agendas and priority setting [ 3 ] as illustrated, for example, by the James Lind Alliance (see http://www.jla.nihr.ac.uk/ ).

A fourth possible reason for the development of new approaches is that effectiveness reviews have themselves broadened. Some ‘what works’ reviews can be open to criticism for only testing a ‘black box’ hypothesis of what works with little theorizing or any logic model about why any such hypothesis should be true and the mechanisms involved in such processes. There is now more concern to develop theory and to test how variables combine and interact. In primary research, qualitative strategies are advised prior to undertaking experimental trials [ 4 , 5 ] and similar approaches are being advocated to address complexity in reviews [ 6 ], in order to ask questions and use methods that address theories and processes that enable an understanding of both impact and context.

This Special Issue of Systematic Reviews Journal is providing a focus for these new methods of review whether these use qualitative review methods on their own or mixed together with more quantitative approaches. We are linking together with the sister journal Trials for this Special Issue as there is a similar interest in what qualitative approaches can and should contribute to primary research using experimentally controlled trials (see Trials Special Issue editorial by Claire Snowdon).

Dimensions of difference in reviews

Developing the range of methods to address different questions for review creates a challenge in describing and understanding such methods. There are many names and brands for the new methods which may or may not withstand the changes of historical time, but another way to comprehend the changes and new developments is to consider the dimensions on which the approaches to review differ [ 7 , 8 ].

One important distinction is the research question being asked and the associated paradigm underlying the method used to address this question. Research assumes a particular theoretical position and then gathers data within this conceptual lens. In some cases, this is a very specific hypothesis that is then tested empirically, and sometimes, the research is more exploratory and iterative with concepts being emergent and constructed during the research process. This distinction is often labelled as quantitative or positivist versus qualitative or constructionist. However, this can be confusing as much research taking a ‘quantitative’ perspective does not have the necessary numeric data to analyse. Even if it does have such data, this might be explored for emergent properties. Similarly, research taking a ‘qualitative’ perspective may include implicit quantitative themes in terms of the extent of different qualitative findings reported by a study.

Sandelowski and colleagues’ solution is to consider the analytic activity and whether this aggregates (adds up) or configures (arranges) the data [ 9 ]. In a randomized controlled trial and an effectiveness review of such studies, the main analysis is the aggregation of data using a priori non-emergent strategies with little iteration. However, there may also be post hoc analysis that is more exploratory in arranging (configuring) data to identify patterns as in, for example, meta regression or qualitative comparative analysis aiming to identify the active ingredients of effective interventions [ 10 ]. Similarly, qualitative primary research or reviews of such research are predominantly exploring emergent patterns and developing concepts iteratively, yet there may be some aggregation of data to make statements of generalizations of extent.

Even where the analysis is predominantly configuration, there can be a wide variation in the dimensions of difference of iteration of theories and concepts. In thematic synthesis [ 11 ], there may be few presumptions about the concepts that will be configured. In meta ethnography which can be richer in theory, there may be theoretical assumptions underlying the review question framing the analysis. In framework synthesis, there is an explicit conceptual framework that is iteratively developed and changed through the review process [ 12 , 13 ].

In addition to the variation in question, degree of configuration, complexity of theory, and iteration are many other dimensions of difference between reviews. Some of these differences follow on from the research questions being asked and the research paradigm being used such as in the approach to searching (exhaustive or based on exploration or saturation) and the appraisal of the quality and relevance of included studies (based more on risk of bias or more on meaning). Others include the extent that reviews have a broad question, depth of analysis, and the extent of resultant ‘work done’ in terms of progressing a field of inquiry [ 7 , 8 ].

Mixed methods reviews

As one reason for the growth in qualitative synthesis is what they can add to quantitative reviews, it is not surprising that there is also growing interest in mixed methods reviews. This reflects similar developments in primary research in mixing methods to examine the relationship between theory and empirical data which is of course the cornerstone of much research. But, both primary and secondary mixed methods research also face similar challenges in examining complex questions at different levels of analysis and of combining research findings investigated in different ways and may be based on very different epistemological assumptions [ 14 , 15 ].

Some mixed methods approaches are convergent in that they integrate different data and methods of analysis together at the same time [ 16 , 17 ]. Convergent systematic reviews could be described as having broad inclusion criteria (or two or more different sets of criteria) for methods of primary studies and have special methods for the synthesis of the resultant variation in data. Other reviews (and also primary mixed methods studies) are sequences of sub-reviews in that one sub-study using one research paradigm is followed by another sub-study with a different research paradigm. In other words, a qualitative synthesis might be used to explore the findings of a prior quantitative synthesis or vice versa [ 16 , 17 ].

An example of a predominantly aggregative sub-review followed by a configuring sub-review is the EPPI-Centre’s mixed methods review of barriers to healthy eating [ 18 ]. A sub-review on the effectiveness of public health interventions showed a modest effect size. A configuring review of studies of children and young people’s understanding and views about eating provided evidence that the public health interventions did not take good account of such user views research, and that the interventions most closely aligned to the user views were the most effective. The already mentioned qualitative comparative analysis to identify the active ingredients within interventions leading to impact could also be considered a qualitative configuring investigation of an existing quantitative aggregative review [ 10 ].

An example of a predominantly configurative review followed by an aggregative review is realist synthesis. Realist reviews examine the evidence in support of mid-range theories [ 19 ] with a first stage of a configuring review of what is proposed by the theory or proposal (what would need to be in place and what casual pathways would have to be effective for the outcomes proposed by the theory to be supported?) and a second stage searching for empirical evidence to test for those necessary conditions and effectiveness of the pathways. The empirical testing does not however use a standard ‘what works’ a priori methods approach but rather a more iterative seeking out of evidence that confirms or undermines the theory being evaluated [ 20 ].

Although sequential mixed methods approaches are considered to be sub-parts of one larger study, they could be separate studies as part of a long-term strategic approach to studying an issue. We tend to see both primary studies and reviews as one-off events, yet reviews are a way of examining what we know and what more we want to know as a strategic approach to studying an issue over time. If we are in favour of mixing paradigms of research to enable multiple levels and perspectives and mixing of theory development and empirical evaluation, then we are really seeking mixed methods research strategies rather than simply mixed methods studies and reviews.

Noblit G. Hare RD: meta-ethnography: synthesizing qualitative studies. Newbury Park NY: Sage Publications; 1988.

Google Scholar  

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Article   PubMed   PubMed Central   Google Scholar  

Gough D, Elbourne D. Systematic research synthesis to inform policy, practice and democratic debate. Soc Pol Soc. 2002;2002:1.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance 2015. BMJ. 2015;350:h1258

Candy B, Jone L, King M, Oliver S. Using qualitative evidence to help understand complex palliative care interventions: a novel evidence synthesis approach. BMJ Support Palliat Care. 2014;4:Supp A41–A42.

Article   Google Scholar  

Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66:11.

Gough D, Oliver S, Thomas J. Introduction to systematic reviews. London: Sage; 2012.

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

Sandelowski M, Voils CJ, Leeman J, Crandlee JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6:4.

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Exp. 2008;11:72–84.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Qual Saf. 2015. 2014-003642.

Brannen J. Mixed methods research: a discussion paper. NCRM Methods Review Papers, 2006. NCRM/005.

Creswell J. Mapping the developing landscape of mixed methods research. In: Teddlie C, Tashakkori A, editors. SAGE handbook of mixed methods in social & behavioral research. New York: Sage; 2011.

Morse JM. Principles of mixed method and multi-method research design. In: Teddlie C, Tashakkori A, editors. Handbook of mixed methods in social and behavioural research. London: Sage; 2003.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.

Harden A, Thomas J. Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in the social and behavioral sciences. 2nd ed. London: Sage; 2010. p. 749–74.

Chapter   Google Scholar  

Pawson R. Evidenced-based policy: a realist perspective. London: Sage; 2006.

Book   Google Scholar  

Gough D. Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Med. 2013;11:22.

Download references

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, University College London, London, WC1H 0NR, UK

David Gough

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Gough .

Additional information

Competing interests.

The author is a writer and researcher in this area. The author declares that he has no other competing interests.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gough, D. Qualitative and mixed methods in systematic reviews. Syst Rev 4 , 181 (2015). https://doi.org/10.1186/s13643-015-0151-y

Download citation

Received : 13 October 2015

Accepted : 29 October 2015

Published : 15 December 2015

DOI : https://doi.org/10.1186/s13643-015-0151-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Advertisement

Toward a framework for selecting indicators of measuring sustainability and circular economy in the agri-food sector: a systematic literature review

• LIFE CYCLE SUSTAINABILITY ASSESSMENT
• Published: 02 March 2022

Cite this article

• Cecilia Silvestri   ORCID: orcid.org/0000-0003-2528-601X 1 ,
• Luca Silvestri   ORCID: orcid.org/0000-0002-6754-899X 2 ,
• Michela Piccarozzi   ORCID: orcid.org/0000-0001-9717-9462 1 &
• Alessandro Ruggieri 1  

2853 Accesses

11 Citations

9 Altmetric

Explore all metrics

A Correction to this article was published on 24 March 2022

This article has been updated

The implementation of sustainability and circular economy (CE) models in agri-food production can promote resource efficiency, reduce environmental burdens, and ensure improved and socially responsible systems. In this context, indicators for the measurement of sustainability play a crucial role. Indicators can measure CE strategies aimed to preserve functions, products, components, materials, or embodied energy. Although there is broad literature describing sustainability and CE indicators, no study offers such a comprehensive framework of indicators for measuring sustainability and CE in the agri-food sector.

Starting from this central research gap, a systematic literature review has been developed to measure the sustainability in the agri-food sector and, based on these findings, to understand how indicators are used and for which specific purposes.

The analysis of the results allowed us to classify the sample of articles in three main clusters (“Assessment-LCA,” “Best practice,” and “Decision-making”) and has shown increasing attention to the three pillars of sustainability (triple bottom line). In this context, an integrated approach of indicators (environmental, social, and economic) offers the best solution to ensure an easier transition to sustainability.

Conclusions

The sample analysis facilitated the identification of new categories of impact that deserve attention, such as the cooperation among stakeholders in the supply chain and eco-innovation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Source: Authors’ elaboration. Notes: The graph shows the temporal distribution of the articles under analysis

Source: Authors’ elaborations. Notes: The graph shows the time distribution of articles from the three major journals

Source: Authors’ elaboration. Notes: The graph shows the composition of the sample according to the three clusters identified by the analysis

Source: Authors’ elaboration. Notes: The graph shows the distribution of articles over time by cluster

Source: Authors’ elaboration. Notes: The graph shows the network visualization

Source: Authors’ elaboration. Notes: The graph shows the overlay visualization

Source: Authors’ elaboration. Notes: The graph shows the classification of articles by scientific field

Source: Authors’ elaboration. Notes: Article classification based on their cluster to which they belong and scientific field

Source: Authors’ elaboration

Source: Authors’ elaboration. Notes: The graph shows the distribution of items over time based on TBL

Source: Authors’ elaboration. Notes: The graph shows the Pareto diagram highlighting the most used indicators in literature for measuring sustainability in the agri-food sector

Source: Authors’ elaboration. Notes: The graph shows the distribution over time of articles divided into conceptual and empirical

Source: Authors’ elaboration. Notes: The graph shows the classification of articles, divided into conceptual and empirical, in-depth analysis

Source: Authors’ elaboration. Notes: The graph shows the geographical distribution of the authors

Source: Authors’ elaboration. Notes: The graph shows the distribution of authors according to the continent from which they originate

Source: Authors’ elaboration. Notes: The graph shows the time distribution of publication of authors according to the continent from which they originate

Source: Authors’ elaboration. Notes: Sustainability measurement indicators and impact categories of LCA, S-LCA, and LCC tools should be integrated in order to provide stakeholders with best practices as guidelines and tools to support both decision-making and measurement, according to the circular economy approach

Similar content being viewed by others

Common Methods and Sustainability Indicators

Transition heuristic frameworks in research on agro-food sustainability transitions

Hamid El Bilali

Research on agro-food sustainability transitions: where are food security and nutrition?

Change history, 24 march 2022.

A Correction to this paper has been published: https://doi.org/10.1007/s11367-022-02038-9

Acero AP, Rodriguez C, Ciroth A (2017) LCIA methods: impact assessment methods in life cycle assessment and their impact categories. Version 1.5.6. Green Delta 1–23

Accorsi R, Versari L, Manzini R (2015) Glass vs. plastic: Life cycle assessment of extra-virgin olive oil bottles across global supply chains. Sustain 7:2818–2840. https://doi.org/10.3390/su7032818

Adjei-Bamfo P, Maloreh-Nyamekye T, Ahenkan A (2019) The role of e-government in sustainable public procurement in developing countries: a systematic literature review. Resour Conserv Recycl 142:189–203. https://doi.org/10.1016/j.resconrec.2018.12.001

Article   Google Scholar  

Aivazidou E, Tsolakis N, Vlachos D, Iakovou E (2015) Water footprint management policies for agrifood supply chains: a critical taxonomy and a system dynamics modelling approach. Chem Eng Trans 43:115–120. https://doi.org/10.3303/CET1543020

Alhaddi H (2015) Triple bottom line and sustainability: a literature review. Bus Manag Stud 1:6–10

Allaoui H, Guo Y, Sarkis J (2019) Decision support for collaboration planning in sustainable supply chains. J Clean Prod 229:761–774. https://doi.org/10.1016/j.jclepro.2019.04.367

Alshqaqeeq F, Amin Esmaeili M, Overcash M, Twomey J (2020) Quantifying hospital services by carbon footprint: a systematic literature review of patient care alternatives. Resour Conserv Recycl 154:104560. https://doi.org/10.1016/j.resconrec.2019.104560

Anwar F, Chaudhry FN, Nazeer S et al (2016) Causes of ozone layer depletion and its effects on human: review. Atmos Clim Sci 06:129–134. https://doi.org/10.4236/acs.2016.61011

Aquilani B, Silvestri C, Ruggieri A (2016). A Systematic Literature Review on Total Quality Management Critical Success Factors and the Identification of New Avenues of Research. https://doi.org/10.1108/TQM-01-2016-0003

Aramyan L, Hoste R, Van Den Broek W et al (2011) Towards sustainable food production: a scenario study of the European pork sector. J Chain Netw Sci 11:177–189. https://doi.org/10.3920/JCNS2011.Qpork8

Arfini F, Antonioli F, Cozzi E et al (2019) Sustainability, innovation and rural development: the case of Parmigiano-Reggiano PDO. Sustain 11:1–17. https://doi.org/10.3390/su11184978

Assembly UG (2005) Resolution adopted by the general assembly. New York, NY

Avilés-Palacios C, Rodríguez-Olalla A (2021) The sustainability of waste management models in circular economies. Sustain 13:1–19. https://doi.org/10.3390/su13137105

Azevedo SG, Silva ME, Matias JCO, Dias GP (2018) The influence of collaboration initiatives on the sustainability of the cashew supply chain. Sustain 10:1–29. https://doi.org/10.3390/su10062075

Bajaj S, Garg R, Sethi M (2016) Total quality management: a critical literature review using Pareto analysis. Int J Product Perform Manag 67:128–154

Banasik A, Kanellopoulos A, Bloemhof-Ruwaard JM, Claassen GDH (2019) Accounting for uncertainty in eco-efficient agri-food supply chains: a case study for mushroom production planning. J Clean Prod 216:249–256. https://doi.org/10.1016/j.jclepro.2019.01.153

Barth H, Ulvenblad PO, Ulvenblad P (2017) Towards a conceptual framework of sustainable business model innovation in the agri-food sector: a systematic literature review. Sustain 9. https://doi.org/10.3390/su9091620

Bastas A, Liyanage K (2018) Sustainable supply chain quality management: a systematic review

Beckerman W (1992) Economic growth and the environment: whose growth? Whose environment? World Dev 20:481–496. https://doi.org/10.1016/0305-750X(92)90038-W

Belaud JP, Prioux N, Vialle C, Sablayrolles C (2019) Big data for agri-food 4.0: application to sustainability management for by-products supply chain. Comput Ind 111:41–50. https://doi.org/10.1016/j.compind.2019.06.006

Bele B, Norderhaug A, Sickel H (2018) Localized agri-food systems and biodiversity. Agric 8. https://doi.org/10.3390/agriculture8020022

Bilali H El, Calabrese G, Iannetta M et al (2020) Environmental sustainability of typical agro-food products: a scientifically sound and user friendly approach. New Medit 19:69–83. https://doi.org/10.30682/nm2002e

Blanc S, Massaglia S, Brun F et al (2019) Use of bio-based plastics in the fruit supply chain: an integrated approach to assess environmental, economic, and social sustainability. Sustain 11. https://doi.org/10.3390/su11092475

Bloemhof JM, van der Vorst JGAJ, Bastl M, Allaoui H (2015) Sustainability assessment of food chain logistics. Int J Logist Res Appl 18:101–117. https://doi.org/10.1080/13675567.2015.1015508

Bonisoli L, Galdeano-Gómez E, Piedra-Muñoz L (2018) Deconstructing criteria and assessment tools to build agri-sustainability indicators and support farmers’ decision-making process. J Clean Prod 182:1080–1094. https://doi.org/10.1016/j.jclepro.2018.02.055

Bonisoli L, Galdeano-Gómez E, Piedra-Muñoz L, Pérez-Mesa JC (2019) Benchmarking agri-food sustainability certifications: evidences from applying SAFA in the Ecuadorian banana agri-system. J Clean Prod 236. https://doi.org/10.1016/j.jclepro.2019.07.054

Bornmann L, Haunschild R, Hug SE (2018) Visualizing the context of citations referencing papers published by Eugene Garfield: a new type of keyword co-occurrence analysis. Scientometrics 114:427–437. https://doi.org/10.1007/s11192-017-2591-8

Boulding KE (1966) The economics of the coming spaceship earth. New York, 1-17

Bracquené E, Dewulf W, Duflou JR (2020) Measuring the performance of more circular complex product supply chains. Resour Conserv Recycl 154:104608. https://doi.org/10.1016/j.resconrec.2019.104608

Burck J, Hagen U, Bals C et al (2021) Climate Change Performance Index

Calisto Friant M, Vermeulen WJV, Salomone R (2020) A typology of circular economy discourses: navigating the diverse visions of a contested paradigm. Resour Conserv Recycl 161:104917. https://doi.org/10.1016/j.resconrec.2020.104917

Campbell BM, Beare DJ, Bennett EM et al (2017) Agriculture production as a major driver of the earth system exceeding planetary boundaries. Ecol Soc 22. https://doi.org/10.5751/ES-09595-220408

Capitanio F, Coppola A, Pascucci S (2010) Product and process innovation in the Italian food industry. Agribusiness 26:503–518. https://doi.org/10.1002/agr.20239

Caputo P, Zagarella F, Cusenza MA et al (2020) Energy-environmental assessment of the UIA-OpenAgri case study as urban regeneration project through agriculture. Sci Total Environ 729:138819. https://doi.org/10.1016/j.scitotenv.2020.138819

Article   CAS   Google Scholar  

Chabowski BR, Mena JA, Gonzalez-Padron TL (2011) The structure of sustainability research in marketing, 1958–2008: a basis for future research opportunities. J Acad Mark Sci 39:55–70. https://doi.org/10.1007/s11747-010-0212-7

Chadegani AA, Salehi H, Yunus M et al (2017) A comparison between two main academic literature collections : Web of Science and Scopus databases. Asian Soc Sci 9:18–26. https://doi.org/10.5539/ass.v9n5p18

Chams N, Guesmi B, Gil JM (2020) Beyond scientific contribution: assessment of the societal impact of research and innovation to build a sustainable agri-food sector. J Environ Manage 264. https://doi.org/10.1016/j.jenvman.2020.110455

Chandrakumar C, McLaren SJ, Jayamaha NP, Ramilan T (2019) Absolute sustainability-based life cycle assessment (ASLCA): a benchmarking approach to operate agri-food systems within the 2°C global carbon budget. J Ind Ecol 23:906–917. https://doi.org/10.1111/jiec.12830

Chaparro-Africano AM (2019) Toward generating sustainability indicators for agroecological markets. Agroecol Sustain Food Syst 43:40–66. https://doi.org/10.1080/21683565.2019.1566192

Colicchia C, Strozzi F (2012) Supply chain risk management: a new methodology for a systematic literature review

Conca L, Manta F, Morrone D, Toma P (2021) The impact of direct environmental, social, and governance reporting: empirical evidence in European-listed companies in the agri-food sector. Bus Strateg Environ 30:1080–1093. https://doi.org/10.1002/bse.2672

Coppola A, Ianuario S, Romano S, Viccaro M (2020) Corporate social responsibility in agri-food firms: the relationship between CSR actions and firm’s performance. AIMS Environ Sci 7:542–558. https://doi.org/10.3934/environsci.2020034

Corona B, Shen L, Reike D et al (2019) Towards sustainable development through the circular economy—a review and critical assessment on current circularity metrics. Resour Conserv Recycl 151:104498. https://doi.org/10.1016/j.resconrec.2019.104498

Correia MS (2019) Sustainability: An overview of the triple bottom line and sustainability implementation. Int J Strateg Eng 2:29–38.  https://doi.org/10.4018/IJoSE.2019010103

Coteur I, Marchand F, Debruyne L, Lauwers L (2019) Structuring the myriad of sustainability assessments in agri-food systems: a case in Flanders. J Clean Prod 209:472–480. https://doi.org/10.1016/j.jclepro.2018.10.066

CREA (2020) L’agricoltura italiana conta 2019

Crenna E, Sala S, Polce C, Collina E (2017) Pollinators in life cycle assessment: towards a framework for impact assessment. J Clean Prod 140:525–536. https://doi.org/10.1016/j.jclepro.2016.02.058

D’Eusanio M, Serreli M, Zamagni A, Petti L (2018) Assessment of social dimension of a jar of honey: a methodological outline. J Clean Prod 199:503–517. https://doi.org/10.1016/j.jclepro.2018.07.157

Dania WAP, Xing K, Amer Y (2018) Collaboration behavioural factors for sustainable agri-food supply chains: a systematic review. J Clean Prod 186:851–864

De Pascale A, Arbolino R, Szopik-Depczyńska K et al (2021) A systematic review for measuring circular economy: the 61 indicators. J Clean Prod 281. https://doi.org/10.1016/j.jclepro.2020.124942

De Schoenmakere M, Gillabel J (2017) Circular by design: products in the circular economy

Del Borghi A, Gallo M, Strazza C, Del Borghi M (2014) An evaluation of environmental sustainability in the food industry through life cycle assessment: the case study of tomato products supply chain. J Clean Prod 78:121–130. https://doi.org/10.1016/j.jclepro.2014.04.083

Del Borghi A, Strazza C, Magrassi F et al (2018) Life cycle assessment for eco-design of product–package systems in the food industry—the case of legumes. Sustain Prod Consum 13:24–36. https://doi.org/10.1016/j.spc.2017.11.001

Denyer D, Tranfield D (2009) Producing a systematic review. In: Buchanan B (ed) The sage handbook of organization research methods. Sage Publications Ltd, Cornwall, pp 671–689

Google Scholar  

Dietz T, Grabs J, Chong AE (2019) Mainstreamed voluntary sustainability standards and their effectiveness: evidence from the Honduran coffee sector. Regul Gov. https://doi.org/10.1111/rego.12239

Dixon-Woods M (2011) Using framework-based synthesis for conducting reviews of qualitative studies. BMC Med 9:9–10. https://doi.org/10.1186/1741-7015-9-39

do Canto NR, Bossle MB, Marques L, Dutra M, (2020) Supply chain collaboration for sustainability: a qualitative investigation of food supply chains in Brazil. Manag Environ Qual an Int J. https://doi.org/10.1108/MEQ-12-2019-0275

dos Santos RR, Guarnieri P (2020) Social gains for artisanal agroindustrial producers induced by cooperation and collaboration in agri-food supply chain. Soc Responsib J. https://doi.org/10.1108/SRJ-09-2019-0323

Doukidis GI, Matopoulos A, Vlachopoulou M, Manthou V, Manos B (2007) A conceptual framework for supply chain collaboration: empirical evidence from the agri‐food industry. Supply Chain Manag an Int Journal 12:177–186. https://doi.org/10.1108/13598540710742491

Durach CF, Kembro J, Wieland A (2017) A new paradigm for systematic literature reviews in supply chain management. J Supply Chain Manag 53:67–85. https://doi.org/10.1111/jscm.12145

Durán-Sánchez A, Álvarez-García J, Río-Rama D, De la Cruz M (2018) Sustainable water resources management: a bibliometric overview. Water 10:1–19. https://doi.org/10.3390/w10091191

Duru M, Therond O (2015) Livestock system sustainability and resilience in intensive production zones: which form of ecological modernization? Reg Environ Chang 15:1651–1665. https://doi.org/10.1007/s10113-014-0722-9

Edison Fondazione (2019) Le eccellenze agricole italiane. I primati europei e mondiali dell’Italia nei prodotti vegetali. Milan (IT)

Ehrenfeld JR (2005) The roots of sustainability. MIT Sloan Manag Rev 46(2)46:23–25

Elia V, Gnoni MG, Tornese F (2017) Measuring circular economy strategies through index methods: a critical analysis. J Clean Prod 142:2741–2751. https://doi.org/10.1016/j.jclepro.2016.10.196

Elkington J (1997) Cannibals with forks : the triple bottom line of 21st century business. Capstone, Oxford

Esposito B, Sessa MR, Sica D, Malandrino O (2020) Towards circular economy in the agri-food sector. A systematic literature review. Sustain 12. https://doi.org/10.3390/SU12187401

European Commission (2018) Agri-food trade in 2018

European Commission (2019) Monitoring EU agri-food trade: development until September 2019

Eurostat (2018) Small and large farms in the EU - statistics from the farm structure survey

FAO (2011) Biodiversity for food and agriculture. Italy, Rome

FAO (2012) Energy-smart food at FAO: an overview. Italy, Rome

FAO (2014) Food wastage footprint: fool cost-accounting

FAO (2016) The state of food and agriculture climate change, agriculture and food security. Italy, Rome

FAO (2017) The future of food and agriculture: trends and challenges. Italy, Rome

FAO (2020) The state of food security and nutrition in the world. Transforming Food Systems for Affordable Healthy Diets. Rome, Italy

Fassio F, Tecco N (2019) Circular economy for food: a systemic interpretation of 40 case histories in the food system in their relationships with SDGs. Systems 7:43. https://doi.org/10.3390/systems7030043

Fathollahi A, Coupe SJ (2021) Life cycle assessment (LCA) and life cycle costing (LCC) of road drainage systems for sustainability evaluation: quantifying the contribution of different life cycle phases. Sci Total Environ 776:145937. https://doi.org/10.1016/j.scitotenv.2021.145937

Ferreira VJ, Arnal ÁJ, Royo P et al (2019) Energy and resource efficiency of electroporation-assisted extraction as an emerging technology towards a sustainable bio-economy in the agri-food sector. J Clean Prod 233:1123–1132. https://doi.org/10.1016/j.jclepro.2019.06.030

Fiksel J (2006) A framework for sustainable remediation. JOM 8:15–22. https://doi.org/10.1021/es202595w

Flick U (2014) An introduction to qualitative research

Franciosi C, Voisin A, Miranda S et al (2020) Measuring maintenance impacts on sustainability of manufacturing industries : from a systematic literature review to a framework proposal. J Clean Prod 260:1–19. https://doi.org/10.1016/j.jclepro.2020.121065

Gaitán-Cremaschi D, Meuwissen MPM, Oude AGJML (2017) Total factor productivity: a framework for measuring agri-food supply chain performance towards sustainability. Appl Econ Perspect Policy 39:259–285. https://doi.org/10.1093/aepp/ppw008

Galdeano-Gómez E, Zepeda-Zepeda JA, Piedra-Muñoz L, Vega-López LL (2017) Family farm’s features influencing socio-economic sustainability: an analysis of the agri-food sector in southeast Spain. New Medit 16:50–61

Gallopín G, Herrero LMJ, Rocuts A (2014) Conceptual frameworks and visual interpretations of sustainability. Int J Sustain Dev 17:298–326. https://doi.org/10.1504/IJSD.2014.064183

Gallopín GC (2003) Sostenibilidad y desarrollo sostenible: un enfoque sistémico. Cepal, LATIN AMERICA

Garnett T (2013) Food sustainability: problems, perspectives and solutions. Proc Nutr Soc 72:29–39. https://doi.org/10.1017/S0029665112002947

Garofalo P, D’Andrea L, Tomaiuolo M et al (2017) Environmental sustainability of agri-food supply chains in Italy: the case of the whole-peeled tomato production under life cycle assessment methodology. J Food Eng 200:1–12. https://doi.org/10.1016/j.jfoodeng.2016.12.007

Gava O, Bartolini F, Venturi F et al (2018) A reflection of the use of the life cycle assessment tool for agri-food sustainability. Sustain 11. https://doi.org/10.3390/su11010071

Gazzola P, Querci E (2017) The connection between the quality of life and sustainable ecological development. Eur Sci J 7881:1857–7431

Geissdoerfer M, Savaget P, Bocken N, Hultink EJ (2017) The circular economy – a new sustainability paradigm ? The circular economy – a new sustainability paradigm ? J Clean Prod 143:757–768. https://doi.org/10.1016/j.jclepro.2016.12.048

Georgescu-Roegen N (1971) The entropy low and the economic process. Harward University Press, Cambridge Mass

Book   Google Scholar  

Gerbens-Leenes PW, Moll HC, Schoot Uiterkamp AJM (2003) Design and development of a measuring method for environmental sustainability in food production systems. Ecol Econ 46:231–248. https://doi.org/10.1016/S0921-8009(03)00140-X

Gésan-Guiziou G, Alaphilippe A, Aubin J et al (2020) Diversity and potentiality of multi-criteria decision analysis methods for agri-food research. Agron Sustain Dev 40. https://doi.org/10.1007/s13593-020-00650-3

Ghisellini P, Cialani C, Ulgiati S (2016) A review on circular economy: the expected transition to a balanced interplay of environmental and economic systems. J Clean Prod 114:11–32. https://doi.org/10.1016/j.jclepro.2015.09.007

Godoy-Durán Á, Galdeano- Gómez E, Pérez-Mesa JC, Piedra-Muñoz L (2017) Assessing eco-efficiency and the determinants of horticultural family-farming in southeast Spain. J Environ Manage 204:594–604. https://doi.org/10.1016/j.jenvman.2017.09.037

Gold S, Kunz N, Reiner G (2017) Sustainable global agrifood supply chains: exploring the barriers. J Ind Ecol 21:249–260. https://doi.org/10.1111/jiec.12440

Goucher L, Bruce R, Cameron DD et al (2017) The environmental impact of fertilizer embodied in a wheat-to-bread supply chain. Nat Plants 3:1–5. https://doi.org/10.1038/nplants.2017.12

Green A, Nemecek T, Chaudhary A, Mathys A (2020) Assessing nutritional, health, and environmental sustainability dimensions of agri-food production. Glob Food Sec 26:100406. https://doi.org/10.1016/j.gfs.2020.100406

Guinée JB, Heijungs R, Huppes G et al (2011) Life cycle assessment: past, present, and future †. Environ Sci Technol 45:90–96. https://doi.org/10.1021/es101316v

Guiomar N, Godinho S, Pinto-Correia T et al (2018) Typology and distribution of small farms in Europe: towards a better picture. Land Use Policy 75:784–798. https://doi.org/10.1016/j.landusepol.2018.04.012

Gunasekaran A, Patel C, McGaughey RE (2004) A framework for supply chain performance measurement. Int J Prod Econ 87:333–347. https://doi.org/10.1016/j.ijpe.2003.08.003

Gunasekaran A, Patel C, Tirtiroglu E (2001) Performance measures and metrics in a supply chain environment. Int J Oper Prod Manag 21:71–87. https://doi.org/10.1108/01443570110358468

Hamam M, Chinnici G, Di Vita G et al (2021) Circular economy models in agro-food systems: a review. Sustain 13

Harun SN, Hanafiah MM, Aziz NIHA (2021) An LCA-based environmental performance of rice production for developing a sustainable agri-food system in Malaysia. Environ Manage 67:146–161. https://doi.org/10.1007/s00267-020-01365-7

Harvey M, Pilgrim S (2011) The new competition for land: food, energy, and climate change. Food Policy 36:S40–S51. https://doi.org/10.1016/j.foodpol.2010.11.009

Hawkes C, Ruel MT (2006) Understanding the links between agriculture and health. DC: International Food Policy Research Institute. Washington, USA

Hellweg S, Milà i Canals L (2014) Emerging approaches, challenges and opportunities in life cycle assessment. Science (80)344:1109LP–1113. https://doi.org/10.1126/science.1248361

Higgins V, Dibden J, Cocklin C (2015) Private agri-food governance and greenhouse gas abatement: constructing a corporate carbon economy. Geoforum 66:75–84. https://doi.org/10.1016/j.geoforum.2015.09.012

Hill T (1995) Manufacturing strategy: text and cases., Macmillan

Hjeresen DD, Gonzales R (2020) Green chemistry promote sustainable agriculture?The rewards are higher yields and less environmental contamination. Environemental Sci Techonology 103–107

Horne R, Grant T, Verghese K (2009) Life cycle assessment: principles, practice, and prospects. Csiro Publishing, Collingwood, Australia

Horton P, Koh L, Guang VS (2016) An integrated theoretical framework to enhance resource efficiency, sustainability and human health in agri-food systems. J Clean Prod 120:164–169. https://doi.org/10.1016/j.jclepro.2015.08.092

Hospido A, Davis J, Berlin J, Sonesson U (2010) A review of methodological issues affecting LCA of novel food products. Int J Life Cycle Assess 15:44–52. https://doi.org/10.1007/s11367-009-0130-4

Huffman T, Liu J, Green M et al (2015) Improving and evaluating the soil cover indicator for agricultural land in Canada. Ecol Indic 48:272–281. https://doi.org/10.1016/j.ecolind.2014.07.008

Ilbery B, Maye D (2005) Food supply chains and sustainability: evidence from specialist food producers in the Scottish/English borders. Land Use Policy 22:331–344. https://doi.org/10.1016/j.landusepol.2004.06.002

Ingrao C, Faccilongo N, Valenti F et al (2019) Tomato puree in the Mediterranean region: an environmental life cycle assessment, based upon data surveyed at the supply chain level. J Clean Prod 233:292–313. https://doi.org/10.1016/j.jclepro.2019.06.056

Iocola I, Angevin F, Bockstaller C et al (2020) An actor-oriented multi-criteria assessment framework to support a transition towards sustainable agricultural systems based on crop diversification. Sustain 12. https://doi.org/10.3390/su12135434

Irabien A, Darton RC (2016) Energy–water–food nexus in the Spanish greenhouse tomato production. Clean Technol Environ Policy 18:1307–1316. https://doi.org/10.1007/s10098-015-1076-9

ISO 14040:2006 (2006) Environmental management — life cycle assessment — principles and framework

ISO 14044:2006 (2006) Environmental management — life cycle assessment — requirements and guidelines

ISO 15392:2008 (2008) Sustainability in building construction–general principles

Istat (2019) Andamento dell’economia agricola

Jaakkola E (2020) Designing conceptual articles : four approaches. AMS Rev 1–9. https://doi.org/10.1007/s13162-020-00161-0

Jin R, Yuan H, Chen Q (2019) Science mapping approach to assisting the review of construction and demolition waste management research published between 2009 and 2018. Resour Conserv Recycl 140:175–188. https://doi.org/10.1016/j.resconrec.2018.09.029

Johnston P, Everard M, Santillo D, Robèrt KH (2007) Reclaiming the definition of sustainability. Environ Sci Pollut Res Int 14:60–66. https://doi.org/10.1065/espr2007.01.375

Jorgensen SE, Burkhard B, Müller F (2013) Twenty volumes of ecological indicators-an accounting short review. Ecol Indic 28:4–9. https://doi.org/10.1016/j.ecolind.2012.12.018

Joshi S, Sharma M, Kler R (2020) Modeling circular economy dimensions in agri-tourism clusters: sustainable performance and future research directions. Int J Math Eng Manag Sci 5:1046–1061. https://doi.org/10.33889/IJMEMS.2020.5.6.080

Kamilaris A, Gao F, Prenafeta-Boldu FX, Ali MI (2017) Agri-IoT: a semantic framework for Internet of Things-enabled smart farming applications. In: 2016 IEEE 3rd World Forum on Internet of Things, WF-IoT 2016. pp 442–447

Karuppusami G, Gandhinathan R (2006) Pareto analysis of critical success factors of total quality management: a literature review and analysis. TQM Mag 18:372–385. https://doi.org/10.1108/09544780610671048

Kates RW, Parris TM, Leiserowitz AA (2005) What is sustainable development? Goals, indicators, values, and practice. Environ Sci Policy Sustain Dev 47:8–21. https://doi.org/10.1080/00139157.2005.10524444

Khounani Z, Hosseinzadeh-Bandbafha H, Moustakas K et al (2021) Environmental life cycle assessment of different biorefinery platforms valorizing olive wastes to biofuel, phosphate salts, natural antioxidant, and an oxygenated fuel additive (triacetin). J Clean Prod 278:123916. https://doi.org/10.1016/j.jclepro.2020.123916

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering version 2.3. Engineering 45. https://doi.org/10.1145/1134285.1134500

Korhonen J, Nuur C, Feldmann A, Birkie SE (2018) Circular economy as an essentially contested concept. J Clean Prod 175:544–552. https://doi.org/10.1016/j.jclepro.2017.12.111

Kuisma M, Kahiluoto H (2017) Biotic resource loss beyond food waste: agriculture leaks worst. Resour Conserv Recycl 124:129–140. https://doi.org/10.1016/j.resconrec.2017.04.008

Laso J, Hoehn D, Margallo M et al (2018) Assessing energy and environmental efficiency of the Spanish agri-food system using the LCA/DEA methodology. Energies 11. https://doi.org/10.3390/en11123395

Lee KM (2007) So What is the “triple bottom line”? Int J Divers Organ Communities Nations Annu Rev 6:67–72. https://doi.org/10.18848/1447-9532/cgp/v06i06/39283

Lehmann RJ, Hermansen JE, Fritz M et al (2011) Information services for European pork chains - closing gaps in information infrastructures. Comput Electron Agric 79:125–136. https://doi.org/10.1016/j.compag.2011.09.002

León-Bravo V, Caniato F, Caridi M, Johnsen T (2017) Collaboration for sustainability in the food supply chain: a multi-stage study in Italy. Sustainability 9:1253

Lepage A (2009) The quality of life as attribute of sustainability. TQM J 21:105–115. https://doi.org/10.1108/17542730910938119

Li CZ, Zhao Y, Xiao B et al (2020) Research trend of the application of information technologies in construction and demolition waste management. J Clean Prod 263. https://doi.org/10.1016/j.jclepro.2020.121458

Lo Giudice A, Mbohwa C, Clasadonte MT, Ingrao C (2014) Life cycle assessment interpretation and improvement of the Sicilian artichokes production. Int J Environ Res 8:305–316. https://doi.org/10.22059/ijer.2014.721

Lueddeckens S, Saling P, Guenther E (2020) Temporal issues in life cycle assessment—a systematic review. Int J Life Cycle Assess 25:1385–1401. https://doi.org/10.1007/s11367-020-01757-1

Luo J, Ji C, Qiu C, Jia F (2018) Agri-food supply chain management: bibliometric and content analyses. Sustain 10. https://doi.org/10.3390/su10051573

Lynch J, Donnellan T, Finn JA et al (2019) Potential development of Irish agricultural sustainability indicators for current and future policy evaluation needs. J Environ Manage 230:434–445. https://doi.org/10.1016/j.jenvman.2018.09.070

MacArthur E (2013) Towards the circular economy. J Ind Ecol 2:23–44

MacArthur E (2017) Delivering the circular economy a toolkit for policymakers, The Ellen MacArthur Foundation

MacInnis DJ (2011) A framework for conceptual. J Mark 75:136–154. https://doi.org/10.1509/jmkg.75.4.136

Mangla SK, Luthra S, Rich N et al (2018) Enablers to implement sustainable initiatives in agri-food supply chains. Int J Prod Econ 203:379–393. https://doi.org/10.1016/j.ijpe.2018.07.012

Marotta G, Nazzaro C, Stanco M (2017) How the social responsibility creates value: models of innovation in Italian pasta industry. Int J Glob Small Bus 9:144–167. https://doi.org/10.1504/IJGSB.2017.088923

Martucci O, Arcese G, Montauti C, Acampora A (2019) Social aspects in the wine sector: comparison between social life cycle assessment and VIVA sustainable wine project indicators. Resources 8. https://doi.org/10.3390/resources8020069

Mayring P (2004) Forum : Qualitative social research Sozialforschung 2. History of content analysis. A Companion to Qual Res 1:159–176

McKelvey B (2002) Managing coevolutionary dynamics. In: 18th EGOS Conference. Barcelona, Spain, pp 1–21

McMichael AJ, Butler CD, Folke C (2003) New visions for addressing sustainability. Science (80- ) 302:1191–1920

Mehmood A, Ahmed S, Viza E et al (2021) Drivers and barriers towards circular economy in agri-food supply chain: a review. Bus Strateg Dev 1–17. https://doi.org/10.1002/bsd2.171

Mella P, Gazzola P (2011) Sustainability and quality of life: the development model. In: Kapounek S (ed) Enterprise and competitive environment. Mendel University: Brno, Czechia. 542–551

Merli R, Preziosi M, Acampora A (2018) How do scholars approach the circular economy ? A systematic literature review. J Clean Prod 178:703–722. https://doi.org/10.1016/j.jclepro.2017.12.112

Merli R, Preziosi M, Acampora A et al (2020) Recycled fibers in reinforced concrete: a systematic literature review. J Clean Prod 248:119207. https://doi.org/10.1016/j.jclepro.2019.119207

Miglietta PP, Morrone D (2018) Managing water sustainability: virtual water flows and economic water productivity assessment of the wine trade between Italy and the Balkans. Sustain 10. https://doi.org/10.3390/su10020543

Mitchell MGE, Chan KMA, Newlands NK, Ramankutty N (2020) Spatial correlations don’t predict changes in agricultural ecosystem services: a Canada-wide case study. Front Sustain Food Syst 4:1–17. https://doi.org/10.3389/fsufs.2020.539892

Moraga G, Huysveld S, Mathieux F et al (2019) Circular economy indicators: what do they measure?. Resour Conserv Recycl 146:452–461. https://doi.org/10.1016/j.resconrec.2019.03.045

Morrissey JE, Dunphy NP (2015) Towards sustainable agri-food systems: the role of integrated sustainability and value assessment across the supply-chain. Int J Soc Ecol Sustain Dev 6:41–58. https://doi.org/10.4018/IJSESD.2015070104

Moser G (2009) Quality of life and sustainability: toward person-environment congruity. J Environ Psychol 29:351–357. https://doi.org/10.1016/j.jenvp.2009.02.002

Muijs D (2010) Doing quantitative research in education with SPSS. London

Muller MF, Esmanioto F, Huber N, Loures ER (2019) A systematic literature review of interoperability in the green Building Information Modeling lifecycle. J Clean Prod 223:397–412. https://doi.org/10.1016/j.jclepro.2019.03.114

Muradin M, Joachimiak-Lechman K, Foltynowicz Z (2018) Evaluation of eco-efficiency of two alternative agricultural biogas plants. Appl Sci 8. https://doi.org/10.3390/app8112083

Naseer MA, ur R, Ashfaq M, Hassan S, et al (2019) Critical issues at the upstream level in sustainable supply chain management of agri-food industries: evidence from Pakistan’s citrus industry. Sustain 11:1–19. https://doi.org/10.3390/su11051326

Nattassha R, Handayati Y, Simatupang TM, Siallagan M (2020) Understanding circular economy implementation in the agri-food supply chain: the case of an Indonesian organic fertiliser producer. Agric Food Secur 9:1–16. https://doi.org/10.1186/s40066-020-00264-8

Nazari-Sharabian M, Ahmad S, Karakouzian M (2018) Climate change and eutrophication: a short review. Eng Technol Appl Sci Res 8:3668–3672. https://doi.org/10.5281/zenodo.2532694

Nazir N (2017) Understanding life cycle thinking and its practical application to agri-food system. Int J Adv Sci Eng Inf Technol 7:1861–1870. https://doi.org/10.18517/ijaseit.7.5.3578

Negra C, Remans R, Attwood S et al (2020) Sustainable agri-food investments require multi-sector co-development of decision tools. Ecol Indic 110:105851. https://doi.org/10.1016/j.ecolind.2019.105851

Newsham KK, Robinson SA (2009) Responses of plants in polar regions to UVB exposure: a meta-analysis. Glob Chang Biol 15:2574–2589. https://doi.org/10.1111/j.1365-2486.2009.01944.x

Niemeijer D, de Groot RS (2008) A conceptual framework for selecting environmental indicator sets. Ecol Indic 8:14–25. https://doi.org/10.1016/j.ecolind.2006.11.012

Niero M, Kalbar PP (2019) Coupling material circularity indicators and life cycle based indicators: a proposal to advance the assessment of circular economy strategies at the product level. Resour Conserv Recycl 140:305–312. https://doi.org/10.1016/j.resconrec.2018.10.002

Nikolaou IE, Tsagarakis KP (2021) An introduction to circular economy and sustainability: some existing lessons and future directions. Sustain Prod Consum 28:600–609. https://doi.org/10.1016/j.spc.2021.06.017

Notarnicola B, Hayashi K, Curran MA, Huisingh D (2012) Progress in working towards a more sustainable agri-food industry. J Clean Prod 28:1–8. https://doi.org/10.1016/j.jclepro.2012.02.007

Notarnicola B, Tassielli G, Renzulli PA, Monforti F (2017) Energy flows and greenhouses gases of EU (European Union) national breads using an LCA (life cycle assessment) approach. J Clean Prod 140:455–469. https://doi.org/10.1016/j.jclepro.2016.05.150

Opferkuch K, Caeiro S, Salomone R, Ramos TB (2021) Circular economy in corporate sustainability reporting: a review of organisational approaches. Bus Strateg Environ 1–22. https://doi.org/10.1002/bse.2854

Padilla-Rivera A, do Carmo BBT, Arcese G, Merveille N, (2021) Social circular economy indicators: selection through fuzzy delphi method. Sustain Prod Consum 26:101–110. https://doi.org/10.1016/j.spc.2020.09.015

Pagotto M, Halog A (2016) Towards a circular economy in Australian agri-food industry: an application of input-output oriented approaches for analyzing resource efficiency and competitiveness potential. J Ind Ecol 20:1176–1186. https://doi.org/10.1111/jiec.12373

Parent G, Lavallée S (2011) LCA potentials and limits within a sustainable agri-food statutory framework. Global food insecurity. Springer, Netherlands, Dordrecht, pp 161–171

Chapter   Google Scholar  

Pattey E, Qiu G (2012) Trends in primary particulate matter emissions from Canadian agriculture. J Air Waste Manag Assoc 62:737–747. https://doi.org/10.1080/10962247.2012.672058

Pauliuk S (2018) Critical appraisal of the circular economy standard BS 8001:2017 and a dashboard of quantitative system indicators for its implementation in organizations. Resour Conserv Recycl 129:81–92. https://doi.org/10.1016/j.resconrec.2017.10.019

Peano C, Migliorini P, Sottile F (2014) A methodology for the sustainability assessment of agri-food systems: an application to the slow food presidia project. Ecol Soc 19. https://doi.org/10.5751/ES-06972-190424

Peano C, Tecco N, Dansero E et al (2015) Evaluating the sustainability in complex agri-food systems: the SAEMETH framework. Sustain 7:6721–6741. https://doi.org/10.3390/su7066721

Pearce DW, Turner RK (1990) Economics of natural resources and the environment. Harvester Wheatsheaf, Hemel Hempstead, Herts

Pelletier N (2018) Social sustainability assessment of Canadian egg production facilities: methods, analysis, and recommendations. Sustain 10:1–17. https://doi.org/10.3390/su10051601

Peña C, Civit B, Gallego-Schmid A et al (2021) Using life cycle assessment to achieve a circular economy. Int J Life Cycle Assess 26:215–220. https://doi.org/10.1007/s11367-020-01856-z

Perez Neira D (2016) Energy sustainability of Ecuadorian cacao export and its contribution to climate change. A case study through product life cycle assessment. J Clean Prod 112:2560–2568. https://doi.org/10.1016/j.jclepro.2015.11.003

Pérez-Neira D, Grollmus-Venegas A (2018) Life-cycle energy assessment and carbon footprint of peri-urban horticulture. A comparative case study of local food systems in Spain. Landsc Urban Plan 172:60–68. https://doi.org/10.1016/j.landurbplan.2018.01.001

Pérez-Pons ME, Plaza-Hernández M, Alonso RS et al (2021) Increasing profitability and monitoring environmental performance: a case study in the agri-food industry through an edge-iot platform. Sustain 13:1–16. https://doi.org/10.3390/su13010283

Petti L, Serreli M, Di Cesare S (2018) Systematic literature review in social life cycle assessment. Int J Life Cycle Assess 23:422–431. https://doi.org/10.1007/s11367-016-1135-4

Pieroni MPP, McAloone TC, Pigosso DCA (2019) Business model innovation for circular economy and sustainability: a review of approaches. J Clean Prod 215:198–216. https://doi.org/10.1016/j.jclepro.2019.01.036

Polit DF, Beck CT (2004) Nursing research: principles and methods. Lippincott Williams & Wilkins, Philadelphia, PA

Porkka M, Gerten D, Schaphoff S et al (2016) Causes and trends of water scarcity in food production. Environ Res Lett 11:015001. https://doi.org/10.1088/1748-9326/11/1/015001

Prajapati H, Kant R, Shankar R (2019) Bequeath life to death: state-of-art review on reverse logistics. J Clean Prod 211:503–520. https://doi.org/10.1016/j.jclepro.2018.11.187

Priyadarshini P, Abhilash PC (2020) Policy recommendations for enabling transition towards sustainable agriculture in India. Land Use Policy 96:104718. https://doi.org/10.1016/j.landusepol.2020.104718

Pronti A, Coccia M (2020) Multicriteria analysis of the sustainability performance between agroecological and conventional coffee farms in the East Region of Minas Gerais (Brazil). Renew Agric Food Syst. https://doi.org/10.1017/S1742170520000332

Rabadán A, González-Moreno A, Sáez-Martínez FJ (2019) Improving firms’ performance and sustainability: the case of eco-innovation in the agri-food industry. Sustain 11. https://doi.org/10.3390/su11205590

Raut RD, Luthra S, Narkhede BE et al (2019) Examining the performance oriented indicators for implementing green management practices in the Indian agro sector. J Clean Prod 215:926–943. https://doi.org/10.1016/j.jclepro.2019.01.139

Recanati F, Marveggio D, Dotelli G (2018) From beans to bar: a life cycle assessment towards sustainable chocolate supply chain. Sci Total Environ 613–614:1013–1023. https://doi.org/10.1016/j.scitotenv.2017.09.187

Redclift M (2005) Sustainable development (1987–2005): an oxymoron comes of age. Sustain Dev 13:212–227. https://doi.org/10.1002/sd.281

Rezaei M, Soheilifard F, Keshvari A (2021) Impact of agrochemical emission models on the environmental assessment of paddy rice production using life cycle assessment approach. Energy Sources. Part A Recover Util Environ Eff 1–16

Rigamonti L, Mancini E (2021) Life cycle assessment and circularity indicators. Int J Life Cycle Assess. https://doi.org/10.1007/s11367-021-01966-2

Risku-Norja H, Mäenpää I (2007) MFA model to assess economic and environmental consequences of food production and consumption. Ecol Econ 60:700–711. https://doi.org/10.1016/j.ecolecon.2006.05.001

Ritzén S, Sandström GÖ (2017) Barriers to the circular economy – integration of perspectives and domains. Procedia CIRP 64:7–12. https://doi.org/10.1016/j.procir.2017.03.005

Rockström J, Steffen W, Noone K et al (2009) A safe operating space for humanity. Nature 461:472–475. https://doi.org/10.1038/461472a

Roos Lindgreen E, Mondello G, Salomone R et al (2021) Exploring the effectiveness of grey literature indicators and life cycle assessment in assessing circular economy at the micro level: a comparative analysis. Int J Life Cycle Assess. https://doi.org/10.1007/s11367-021-01972-4

Roselli L, Casieri A, De Gennaro BC et al (2020) Environmental and economic sustainability of table grape production in Italy. Sustain 12.  https://doi.org/10.3390/su12093670

Ross RB, Pandey V, Ross KL (2015) Sustainability and strategy in U.S. agri-food firms: an assessment of current practices. Int Food Agribus Manag Rev 18:17–48

Royo P, Ferreira VJ, López-Sabirón AM, Ferreira G. (2016) Hybrid diagnosis to characterise the energy and environmental enhancement of photovoltaic modules using smart materials. Energy 101:174–189. https://doi.org/10.1016/j.energy.2016.01.101

Ruggerio CA (2021) Sustainability and sustainable development: a review of principles and definitions. Sci Total Environ 786:147481. https://doi.org/10.1016/j.scitotenv.2021.147481

Ruiz-Almeida A, Rivera-Ferre MG (2019) Internationally-based indicators to measure agri-food systems sustainability using food sovereignty as a conceptual framework. Food Secur 11:1321–1337. https://doi.org/10.1007/s12571-019-00964-5

Ryan M, Hennessy T, Buckley C et al (2016) Developing farm-level sustainability indicators for Ireland using the Teagasc National Farm Survey. Irish J Agric Food Res 55:112–125. https://doi.org/10.1515/ijafr-2016-0011

Saade MRM, Yahia A, Amor B (2020) How has LCA been applied to 3D printing ? A systematic literature review and recommendations for future studies. J Clean Prod 244:118803. https://doi.org/10.1016/j.jclepro.2019.118803

Saitone TL, Sexton RJ (2017) Agri-food supply chain: evolution and performance with conflicting consumer and societal demands. Eur Rev Agric Econ 44:634–657. https://doi.org/10.1093/erae/jbx003

Salim N, Ab Rahman MN, Abd Wahab D (2019) A systematic literature review of internal capabilities for enhancing eco-innovation performance of manufacturing firms. J Clean Prod 209:1445–1460. https://doi.org/10.1016/j.jclepro.2018.11.105

Salimi N (2021) Circular economy in agri-food systems BT - strategic decision making for sustainable management of industrial networks. In: International S (ed) Rezaei J. Publishing, Cham, pp 57–70

Salomone R, Ioppolo G (2012) Environmental impacts of olive oil production: a life cycle assessment case study in the province of Messina (Sicily). J Clean Prod 28:88–100. https://doi.org/10.1016/j.jclepro.2011.10.004

Sánchez AD, Río DMDLC, García JÁ (2017) Bibliometric analysis of publications on wine tourism in the databases Scopus and WoS. Eur Res Manag Bus Econ 23:8–15. https://doi.org/10.1016/j.iedeen.2016.02.001

Saputri VHL, Sutopo W, Hisjam M, Ma’aram A (2019) Sustainable agri-food supply chain performance measurement model for GMO and non-GMO using data envelopment analysis method. Appl Sci 9. https://doi.org/10.3390/app9061199

Sassanelli C, Rosa P, Rocca R, Terzi S (2019) Circular economy performance assessment methods : a systematic literature review. J Clean Prod 229:440–453. https://doi.org/10.1016/j.jclepro.2019.05.019

Schiefer S, Gonzalez C, Flanigan S (2015) More than just a factor in transition processes? The role of collaboration in agriculture. In: Sutherland LA, Darnhofer I, Wilson GA, Zagata L (eds) Transition pathways towards sustainability in agriculture: case studies from Europe, CPI Group. Croydon, UK, pp. 83

Seuring S, Muller M (2008) From a literature review to a conceptual framework for sustainable supply chain management. J Clean Prod 16:1699–1710. https://doi.org/10.1016/j.jclepro.2008.04.020

Silvestri C, Silvestri L, Forcina A, et al (2021) Green chemistry contribution towards more equitable global sustainability and greater circular economy: A systematic literature review. J Clean Prod 294. https://doi.org/10.1016/j.jclepro.2021.126137

Smetana S, Schmitt E, Mathys A (2019) Sustainable use of Hermetia illucens insect biomass for feed and food: attributional and consequential life cycle assessment. Resour Conserv Recycl 144:285–296. https://doi.org/10.1016/j.resconrec.2019.01.042

Sonesson U, Berlin J, Ziegler F (2010) Environmental assessment and management in the food industry: life cycle assessment and related approaches. Woodhead Publishing, Cambridge

Soussana JF (2014) Research priorities for sustainable agri-food systems and life cycle assessment. J Clean Prod 73:19–23. https://doi.org/10.1016/j.jclepro.2014.02.061

Soylu A, Oruç C, Turkay M et al (2006) Synergy analysis of collaborative supply chain management in energy systems using multi-period MILP. Eur J Oper Res 174:387–403. https://doi.org/10.1016/j.ejor.2005.02.042

Spaiser V, Ranganathan S, Swain RB, Sumpter DJ (2017) The sustainable development oxymoron: quantifying and modelling the incompatibility of sustainable development goals. Int J Sustain Dev World Ecol 24:457–470. https://doi.org/10.1080/13504509.2016.1235624

Stewart R, Niero M (2018) Circular economy in corporate sustainability strategies: a review of corporate sustainability reports in the fast-moving consumer goods sector. Bus Strateg Environ 27:1005–1022. https://doi.org/10.1002/bse.2048

Stillitano T, Spada E, Iofrida N et al (2021) Sustainable agri-food processes and circular economy pathways in a life cycle perspective: state of the art of applicative research. Sustain 13:1–29. https://doi.org/10.3390/su13052472

Stone J, Rahimifard S (2018) Resilience in agri-food supply chains: a critical analysis of the literature and synthesis of a novel framework. Supply Chain Manag 23:207–238. https://doi.org/10.1108/SCM-06-2017-0201

Strazza C, Del Borghi A, Gallo M, Del Borghi M (2011) Resource productivity enhancement as means for promoting cleaner production: analysis of co-incineration in cement plants through a life cycle approach. J Clean Prod 19:1615–1621. https://doi.org/10.1016/j.jclepro.2011.05.014

Su B, Heshmati A, Geng Y, Yu X (2013) A review of the circular economy in China: moving from rhetoric to implementation. J Clean Prod 42:215–227. https://doi.org/10.1016/j.jclepro.2012.11.020

Suárez-Eiroa B, Fernández E, Méndez-Martínez G, Soto-Oñate D (2019) Operational principles of circular economy for sustainable development: linking theory and practice. J Clean Prod 214:952–961. https://doi.org/10.1016/j.jclepro.2018.12.271

Svensson G, Wagner B (2015) Implementing and managing economic, social and environmental efforts of business sustainability. Manag Environ Qual an Int Journal 26:195–213. https://doi.org/10.1108/MEQ-09-2013-0099

Tasca AL, Nessi S, Rigamonti L (2017) Environmental sustainability of agri-food supply chains: an LCA comparison between two alternative forms of production and distribution of endive in northern Italy. J Clean Prod 140:725–741. https://doi.org/10.1016/j.jclepro.2016.06.170

Tassielli G, Notarnicola B, Renzulli PA, Arcese G (2018) Environmental life cycle assessment of fresh and processed sweet cherries in southern Italy. J Clean Prod 171:184–197. https://doi.org/10.1016/j.jclepro.2017.09.227

Teixeira R, Pax S (2011) A survey of life cycle assessment practitioners with a focus on the agri-food sector. J Ind Ecol 15:817–820. https://doi.org/10.1111/j.1530-9290.2011.00421.x

Tobergte DR, Curtis S (2013) ILCD Handbook. J Chem Info Model. https://doi.org/10.278/33030

Tortorella MM, Di Leo S, Cosmi C et al (2020) A methodological integrated approach to analyse climate change effects in agri-food sector: the TIMES water-energy-food module. Int J Environ Res Public Health 17:1–21. https://doi.org/10.3390/ijerph17217703

Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidenceinformed management knowledge by means of systematic review. Br J Manag 14:207–222

Trivellas P, Malindretos G, Reklitis P (2020) Implications of green logistics management on sustainable business and supply chain performance: evidence from a survey in the greek agri-food sector. Sustain 12:1–29. https://doi.org/10.3390/su122410515

Tsangas M, Gavriel I, Doula M et al (2020) Life cycle analysis in the framework of agricultural strategic development planning in the Balkan region. Sustain 12:1–15. https://doi.org/10.3390/su12051813

Ülgen VS, Björklund M, Simm N (2019) Inter-organizational supply chain interaction for sustainability : a systematic literature review.

UNEP S (2020) Guidelines for social life cycle assessment of products and organizations 2020.

UNEP/SETAC (2009) United Nations Environment Programme-society of Environmental Toxicology and Chemistry. Guidelines for social life cycle assessment of products. France

United Nations (2011) Guiding principles on business and human rights. Implementing the United Nations “protect, respect and remedy” framework

United Nations (2015) Transforming our world: the 2030 agenda for sustainable development. sustainabledevelopment.un.org

Van Asselt ED, Van Bussel LGJ, Van Der Voet H et al (2014) A protocol for evaluating the sustainability of agri-food production systems - a case study on potato production in peri-urban agriculture in the Netherlands. Ecol Indic 43:315–321. https://doi.org/10.1016/j.ecolind.2014.02.027

Van der Ploeg JD (2014) Peasant-driven agricultural growth and food sovereignty. J Peasant Stud 41:999–1030. https://doi.org/10.1080/03066150.2013.876997

van Eck NJ, Waltman L (2010) Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 84:523–538. https://doi.org/10.1007/s11192-009-0146-3

Van Eck NJ, Waltman L (2019) Manual for VOSviwer version 1.6.10. CWTS Meaningful metrics 1–53

Vasa L, Angeloska A, Trendov NM (2017) Comparative analysis of circular agriculture development in selected Western Balkan countries based on sustainable performance indicators. Econ Ann 168:44–47. https://doi.org/10.21003/ea.V168-09

Verdecho MJ, Alarcón-Valero F, Pérez-Perales D et al (2020) A methodology to select suppliers to increase sustainability within supply chains. Cent Eur J Oper Res. https://doi.org/10.1007/s10100-019-00668-3

Vergine P, Salerno C, Libutti A et al (2017) Closing the water cycle in the agro-industrial sector by reusing treated wastewater for irrigation. J Clean Prod 164:587–596. https://doi.org/10.1016/j.jclepro.2017.06.239

WCED (1987) Our common future - call for action

Webster K (2013) What might we say about a circular economy? Some temptations to avoid if possible. World Futures 69:542–554

Wheaton E, Kulshreshtha S (2013) Agriculture and climate change: implications for environmental sustainability indicators. WIT Trans Ecol Environ 175:99–110. https://doi.org/10.2495/ECO130091

Wijewickrama MKCS, Chileshe N, Rameezdeen R, Ochoa JJ (2021) Information sharing in reverse logistics supply chain of demolition waste: a systematic literature review. J Clean Prod 280:124359. https://doi.org/10.1016/j.jclepro.2020.124359

Woodhouse A, Davis J, Pénicaud C, Östergren K (2018) Sustainability checklist in support of the design of food processing. Sustain Prod Consum 16:110–120. https://doi.org/10.1016/j.spc.2018.06.008

Wu R, Yang D, Chen J (2014) Social Life Cycle Assessment Revisited Sustain 6:4200–4226. https://doi.org/10.3390/su6074200

Yadav S, Luthra S, Garg D (2021) Modelling Internet of things (IoT)-driven global sustainability in multi-tier agri-food supply chain under natural epidemic outbreaks. Environ Sci Pollut Res 16633–16654. https://doi.org/10.1007/s11356-020-11676-1

Yee FM, Shaharudin MR, Ma G et al (2021) Green purchasing capabilities and practices towards Firm’s triple bottom line in Malaysia. J Clean Prod 307:127268. https://doi.org/10.1016/j.jclepro.2021.127268

Yigitcanlar T (2010) Rethinking sustainable development: urban management, engineering, and design. IGI Global

Zamagni A, Amerighi O, Buttol P (2011) Strengths or bias in social LCA? Int J Life Cycle Assess 16:596–598. https://doi.org/10.1007/s11367-011-0309-3

Download references

Author information

Authors and affiliations.

Department of Economy, Engineering, Society and Business Organization, University of “Tuscia, ” Via del Paradiso 47, 01100, Viterbo, VT, Italy

Cecilia Silvestri, Michela Piccarozzi & Alessandro Ruggieri

Department of Engineering, University of Rome “Niccolò Cusano, ” Via Don Carlo Gnocchi, 3, 00166, Rome, Italy

Luca Silvestri

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Cecilia Silvestri .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Communicated by Monia Niero

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: a number of ill-placed paragraph headings were removed and the source indication "Authors' elaborations" was added to Tables 1-3.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 31 KB)

Rights and permissions.

Reprints and permissions

About this article

Silvestri, C., Silvestri, L., Piccarozzi, M. et al. Toward a framework for selecting indicators of measuring sustainability and circular economy in the agri-food sector: a systematic literature review. Int J Life Cycle Assess (2022). https://doi.org/10.1007/s11367-022-02032-1

Download citation

Received : 15 June 2021

Accepted : 16 February 2022

Published : 02 March 2022

DOI : https://doi.org/10.1007/s11367-022-02032-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Agri-food sector
• Sustainability
• Circular economy
• Triple bottom line
• Life cycle assessment
• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 08 April 2024

A systematic review and multivariate meta-analysis of the physical and mental health benefits of touch interventions

• Julian Packheiser   ORCID: orcid.org/0000-0001-9805-6755 2   na1   nAff1 ,
• Helena Hartmann 2 , 3 , 4   na1 ,
• Kelly Fredriksen 2 ,
• Valeria Gazzola   ORCID: orcid.org/0000-0003-0324-0619 2 ,
• Christian Keysers   ORCID: orcid.org/0000-0002-2845-5467 2 &
• Frédéric Michon   ORCID: orcid.org/0000-0003-1289-2133 2  

Nature Human Behaviour ( 2024 ) Cite this article

18k Accesses

1838 Altmetric

Metrics details

• Human behaviour
• Paediatric research
• Randomized controlled trials

Receiving touch is of critical importance, as many studies have shown that touch promotes mental and physical well-being. We conducted a pre-registered (PROSPERO: CRD42022304281) systematic review and multilevel meta-analysis encompassing 137 studies in the meta-analysis and 75 additional studies in the systematic review ( n  = 12,966 individuals, search via Google Scholar, PubMed and Web of Science until 1 October 2022) to identify critical factors moderating touch intervention efficacy. Included studies always featured a touch versus no touch control intervention with diverse health outcomes as dependent variables. Risk of bias was assessed via small study, randomization, sequencing, performance and attrition bias. Touch interventions were especially effective in regulating cortisol levels (Hedges’ g  = 0.78, 95% confidence interval (CI) 0.24 to 1.31) and increasing weight (0.65, 95% CI 0.37 to 0.94) in newborns as well as in reducing pain (0.69, 95% CI 0.48 to 0.89), feelings of depression (0.59, 95% CI 0.40 to 0.78) and state (0.64, 95% CI 0.44 to 0.84) or trait anxiety (0.59, 95% CI 0.40 to 0.77) for adults. Comparing touch interventions involving objects or robots resulted in similar physical (0.56, 95% CI 0.24 to 0.88 versus 0.51, 95% CI 0.38 to 0.64) but lower mental health benefits (0.34, 95% CI 0.19 to 0.49 versus 0.58, 95% CI 0.43 to 0.73). Adult clinical cohorts profited more strongly in mental health domains compared with healthy individuals (0.63, 95% CI 0.46 to 0.80 versus 0.37, 95% CI 0.20 to 0.55). We found no difference in health benefits in adults when comparing touch applied by a familiar person or a health care professional (0.51, 95% CI 0.29 to 0.73 versus 0.50, 95% CI 0.38 to 0.61), but parental touch was more beneficial in newborns (0.69, 95% CI 0.50 to 0.88 versus 0.39, 95% CI 0.18 to 0.61). Small but significant small study bias and the impossibility to blind experimental conditions need to be considered. Leveraging factors that influence touch intervention efficacy will help maximize the benefits of future interventions and focus research in this field.

Similar content being viewed by others

Touching the social robot PARO reduces pain perception and salivary oxytocin levels

Nirit Geva, Florina Uzefovsky & Shelly Levy-Tzedek

The impact of mindfulness apps on psychological processes of change: a systematic review

Natalia Macrynikola, Zareen Mir, … John Torous

The why, who and how of social touch

Juulia T. Suvilehto, Asta Cekaite & India Morrison

The sense of touch has immense importance for many aspects of our life. It is the first of all the senses to develop in newborns 1 and the most direct experience of contact with our physical and social environment 2 . Complementing our own touch experience, we also regularly receive touch from others around us, for example, through consensual hugs, kisses or massages 3 .

The recent coronavirus pandemic has raised awareness regarding the need to better understand the effects that touch—and its reduction during social distancing—can have on our mental and physical well-being. The most common touch interventions, for example, massage for adults or kangaroo care for newborns, have been shown to have a wide range of both mental and physical health benefits, from facilitating growth and development to buffering against anxiety and stress, over the lifespan of humans and animals alike 4 . Despite the substantial weight this literature gives to support the benefits of touch, it is also characterized by a large variability in, for example, studied cohorts (adults, children, newborns and animals), type and duration of applied touch (for example, one-time hug versus repeated 60-min massages), measured health outcomes (ranging from physical health outcomes such as sleep and blood pressure to mental health outcomes such as depression or mood) and who actually applies the touch (for example, partner versus stranger).

A meaningful tool to make sense of this vast amount of research is through meta-analysis. While previous meta-analyses on this topic exist, they were limited in scope, focusing only on particular types of touch, cohorts or specific health outcomes (for example, refs. 5 , 6 ). Furthermore, despite best efforts, meaningful variables that moderate the efficacy of touch interventions could not yet be identified. However, understanding these variables is critical to tailor touch interventions and guide future research to navigate this diverse field with the ultimate aim of promoting well-being in the population.

In this Article, we describe a pre-registered, large-scale systematic review and multilevel, multivariate meta-analysis to address this need with quantitative evidence for (1) the effect of touch interventions on physical and mental health and (2) which moderators influence the efficacy of the intervention. In particular, we ask whether and how strongly health outcomes depend on the dynamics of the touching dyad (for example, humans or robots/objects, familiarity and touch directionality), demographics (for example, clinical status, age or sex), delivery means (for example, type of touch intervention or touched body part) and procedure (for example, duration or number of sessions). We did so separately for newborns and for children and adults, as the health outcomes in newborns differed substantially from those in the other age groups. Despite the focus of the analysis being on humans, it is widely known that many animal species benefit from touch interactions and that engaging in touch promotes their well-being as well 7 . Since animal models are essential for the investigation of the mechanisms underlying biological processes and for the development of therapeutic approaches, we accordingly included health benefits of touch interventions in non-human animals as part of our systematic review. However, this search yielded only a small number of studies, suggesting a lack of research in this domain, and as such, was insufficient to be included in the meta-analysis. We evaluate the identified animal studies and their findings in the discussion.

Touch interventions have a medium-sized effect

The pre-registration can be found at ref. 8 . The flowchart for data collection and extraction is depicted in Fig. 1 .

Animal outcomes refer to outcomes measured in non-human species that were solely considered as part of a systematic review. Included languages were French, Dutch, German and English, but our search did not identify any articles in French, Dutch or German. MA, meta-analysis.

For adults, a total of n  = 2,841 and n  = 2,556 individuals in the touch and control groups, respectively, across 85 studies and 103 cohorts were included. The effect of touch overall was medium-sized ( t (102) = 9.74, P  < 0.001, Hedges’ g  = 0.52, 95% confidence interval (CI) 0.42 to 0.63; Fig. 2a ). For newborns, we could include 63 cohorts across 52 studies comprising a total of n  = 2,134 and n  = 2,086 newborns in the touch and control groups, respectively, with an overall effect almost identical to the older age group ( t (62) = 7.53, P  < 0.001, Hedges’ g  = 0.56, 95% CI 0.41 to 0.71; Fig. 2b ), suggesting that, despite distinct health outcomes, touch interventions show comparable effects across newborns and adults. Using these overall effect estimates, we conducted a power sensitivity analysis of all the included primary studies to investigate whether such effects could be reliably detected 9 . Sufficient power to detect such effect sizes was rare in individual studies, as investigated by firepower plots 10 (Supplementary Figs. 1 and 2 ). No individual effect size from either meta-analysis was overly influential (Cook’s D  < 0.06). The benefits were similar for mental and physical outcomes (mental versus physical; adults: t (101) = 0.79, P  = 0.432, Hedges’ g difference of −0.05, 95% CI −0.16 to 0.07, Fig. 2c ; newborns: t (61) = 1.08, P  = 0.284, Hedges’ g difference of −0.19, 95% CI −0.53 to 0.16, Fig. 2d ).

a , Orchard plot illustrating the overall benefits across all health outcomes for adults/children across 469 in part dependent effect sizes from 85 studies and 103 cohorts. b , The same as a but for newborns across 174 in part dependent effect sizes from 52 studies and 63 cohorts. c , The same as a but separating the results for physical versus mental health benefits across 469 in part dependent effect sizes from 85 studies and 103 cohorts. d , The same as b but separating the results for physical versus mental health benefits across 172 in part dependent effect sizes from 52 studies and 63 cohorts. Each dot reflects a measured effect, and the number of effects ( k ) included in the analysis is depicted in the bottom left. Mean effects and 95% CIs are presented in the bottom right and are indicated by the central black dot (mean effect) and its error bars (95% CI). The heterogeneity Q statistic is presented in the top left. Overall effects of moderator impact were assessed via an F test, and post hoc comparisons were done using t tests (two-sided test). Note that the P values above the mean effects indicate whether an effect differed significantly from a zero effect. P values were not corrected for multiple comparisons. The dot size reflects the precision of each individual effect (larger indicates higher precision). Small-study bias for the overall effect was significant ( F test, two-sided test) in the adult meta-analysis ( F (1, 101) = 21.24, P  < 0.001; Supplementary Fig. 3 ) as well as in the newborn meta-analysis ( F (1, 61) = 5.25, P  = 0.025; Supplementary Fig. 4 ).

Source data

On the basis of the overall effect of both meta-analyses as well as their median sample sizes, the minimum number of studies necessary for subgroup analyses to achieve 80% power was k  = 9 effects for adults and k  = 8 effects for newborns (Supplementary Figs. 5 and 6 ). Assessing specific health outcomes with sufficient power in more detail in adults (Fig. 3a ) revealed smaller benefits to sleep and heart rate parameters, moderate benefits to positive and negative affect, diastolic blood and systolic blood pressure, mobility and reductions of the stress hormone cortisol and larger benefits to trait and state anxiety, depression, fatigue and pain. Post hoc tests revealed stronger benefits for pain, state anxiety, depression and trait anxiety compared with respiratory, sleep and heart rate parameters (see Fig. 3 for all post hoc comparisons). Reductions in pain and state anxiety were increased compared with reductions in negative affect ( t (83) = 2.54, P  = 0.013, Hedges’ g difference of 0.31, 95% CI 0.07 to 0.55; t (83) = 2.31, P  = 0.024, Hedges’ g difference of 0.27, 95% CI 0.03 to 0.51). Benefits to pain symptoms were higher compared with benefits to positive affect ( t (83) = 2.22, P  = 0.030, Hedges’ g difference of 0.29, 95% CI 0.04 to 0.54). Finally, touch resulted in larger benefits to cortisol release compared with heart rate parameters ( t (83) = 2.30, P  = 0.024, Hedges’ g difference of 0.26, 95% CI 0.04–0.48).

a , b , Health outcomes in adults analysed across 405 in part dependent effect sizes from 79 studies and 97 cohorts ( a ) and in newborns analysed across 105 in part dependent effect sizes from 46 studies and 56 cohorts ( b ). The type of health outcomes measured differed between adults and newborns and were thus analysed separately. Numbers on the right represent the mean effect with its 95% CI in square brackets and the significance level estimating the likelihood that the effect is equal to zero. Overall effects of moderator impact were assessed via an F test, and post hoc comparisons were done using t tests (two-sided test). The F value in the top right represents a test of the hypothesis that all effects within the subpanel are equal. The Q statistic represents the heterogeneity. P values of post hoc tests are depicted whenever significant. P values above the horizontal whiskers indicate whether an effect differed significantly from a zero effect. Vertical lines indicate significant post hoc tests between moderator levels. P values were not corrected for multiple comparisons. Physical outcomes are marked in red. Mental outcomes are marked in blue.

In newborns, only physical health effects offered sufficient data for further analysis. We found no benefits for digestion and heart rate parameters. All other health outcomes (cortisol, liver enzymes, respiration, temperature regulation and weight gain) showed medium to large effects (Fig. 3b ). We found no significant differences among any specific health outcomes.

Non-human touch and skin-to-skin contact

In some situations, a fellow human is not readily available to provide affective touch, raising the question of the efficacy of touch delivered by objects and robots 11 . Overall, we found humans engaging in touch with other humans or objects to have medium-sized health benefits in adults, without significant differences ( t (99) = 1.05, P  = 0.295, Hedges’ g difference of 0.12, 95% CI −0.11 to 0.35; Fig. 4a ). However, differentiating physical versus mental health benefits revealed similar benefits for human and object touch on physical health outcomes, but larger benefits on mental outcomes when humans were touched by humans ( t (97) = 2.32, P  = 0.022, Hedges’ g difference of 0.24, 95% CI 0.04 to 0.44; Fig. 4b ). It must be noted that touching with an object still showed a significant effect (see Supplementary Fig. 7 for the corresponding orchard plot).

a , Forest plot comparing humans versus objects touching a human on health outcomes overall across 467 in part dependent effect sizes from 85 studies and 101 cohorts. b , The same as a but separately for mental versus physical health outcomes across 467 in part dependent effect sizes from 85 studies and 101 cohorts. c , Results with the removal of all object studies, leaving 406 in part dependent effect sizes from 71 studies and 88 cohorts to identify whether missing skin-to-skin contact is the relevant mediator of higher mental health effects in human–human interactions. Numbers on the right represent the mean effect with its 95% CI in square brackets and the significance level estimating the likelihood that the effect is equal to zero. Overall effects of moderator impact were assessed via an F test, and post hoc comparisons were done using t tests (two-sided test). The F value in the top right represents a test of the hypothesis that all effects within the subpanel are equal. The Q statistic represents the heterogeneity. P values of post hoc tests are depicted whenever significant. P values above the horizontal whiskers indicate whether an effect differed significantly from a zero effect. Vertical lines indicate significant post hoc tests between moderator levels. P values were not corrected for multiple comparisons. Physical outcomes are marked in red. Mental outcomes are marked in blue.

We considered the possibility that this effect was due to missing skin-to-skin contact in human–object interactions. Thus, we investigated human–human interactions with and without skin-to-skin contact (Fig. 4c ). In line with the hypothesis that skin-to-skin contact is highly relevant, we again found stronger mental health benefits in the presence of skin-to-skin contact that however did not achieve nominal significance ( t (69) = 1.95, P  = 0.055, Hedges’ g difference of 0.41, 95% CI −0.00 to 0.82), possibly because skin-to-skin contact was rarely absent in human–human interactions, leading to a decrease in power of this analysis. Results for skin-to-skin contact as an overall moderator can be found in Supplementary Fig. 8 .

Influences of type of touch

The large majority of touch interventions comprised massage therapy in adults and kangaroo care in newborns (see Supplementary Table 1 for a complete list of interventions across studies). However, comparing the different types of touch explored across studies did not reveal significant differences in effect sizes based on touch type, be it on overall health benefits (adults: t (101) = 0.11, P  = 0.916, Hedges’ g difference of 0.02, 95% CI −0.32 to 0.29; Fig. 5a ) or comparing different forms of touch separately for physical (massage therapy versus other forms: t (99) = 0.99, P  = 0.325, Hedges’ g difference 0.16, 95% CI −0.15 to 0.47) or for mental health benefits (massage therapy versus other forms: t (99) = 0.75, P  = 0.458, Hedges’ g difference of 0.13, 95% CI −0.22 to 0.48) in adults (Fig. 5c ; see Supplementary Fig. 9 for the corresponding orchard plot). A similar picture emerged for physical health effects in newborns (massage therapy versus kangaroo care: t (58) = 0.94, P  = 0.353, Hedges’ g difference of 0.15, 95% CI −0.17 to 0.47; massage therapy versus other forms: t (58) = 0.56, P  = 0.577, Hedges’ g difference of 0.13, 95% CI −0.34 to 0.60; kangaroo care versus other forms: t (58) = 0.07, P  = 0.947, Hedges’ g difference of 0.02, 95% CI −0.46 to 0.50; Fig. 5d ; see also Supplementary Fig. 10 for the corresponding orchard plot). This suggests that touch types may be flexibly adapted to the setting of every touch intervention.

a , Forest plot of health benefits comparing massage therapy versus other forms of touch in adult cohorts across 469 in part dependent effect sizes from 85 studies and 103 cohorts. b , Forest plot of health benefits comparing massage therapy, kangaroo care and other forms of touch for newborns across 174 in part dependent effect sizes from 52 studies and 63 cohorts. c , The same as a but separating mental and physical health benefits across 469 in part dependent effect sizes from 85 studies and 103 cohorts. d , The same as b but separating mental and physical health outcomes where possible across 164 in part dependent effect sizes from 51 studies and 62 cohorts. Note that an insufficient number of studies assessed mental health benefits of massage therapy or other forms of touch to be included. Numbers on the right represent the mean effect with its 95% CI in square brackets and the significance level estimating the likelihood that the effect is equal to zero. Overall effects of moderator impact were assessed via an F test, and post hoc comparisons were done using t tests (two-sided test). The F value in the top right represents a test of the hypothesis that all effects within the subpanel are equal. The Q statistic represents heterogeneity. P values of post hoc tests are depicted whenever significant. P values above the horizontal whiskers indicate whether an effect differed significantly from a zero effect. Vertical lines indicate significant post hoc tests between moderator levels. P values were not corrected for multiple comparisons. Physical outcomes are marked in red. Mental outcomes are marked in blue.

The role of clinical status

Most research on touch interventions has focused on clinical samples, but are benefits restricted to clinical cohorts? We found health benefits to be significant in clinical and healthy populations (Fig. 6 ), whether all outcomes are considered (Fig. 6a,b ) or physical and mental health outcomes are separated (Fig. 6c,d , see Supplementary Figs. 11 and 12 for the corresponding orchard plots). In adults, however, we found higher mental health benefits for clinical populations compared with healthy ones (Fig. 6c ; t (99) = 2.11, P  = 0.037, Hedges’ g difference of 0.25, 95% CI 0.01 to 0.49).

a , Health benefits for clinical cohorts of adults versus healthy cohorts of adults across 469 in part dependent effect sizes from 85 studies and 103 cohorts. b , The same as a but for newborn cohorts across 174 in part dependent effect sizes from 52 studies and 63 cohorts. c , The same as a but separating mental versus physical health benefits across 469 in part dependent effect sizes from 85 studies and 103 cohorts. d , The same as b but separating mental versus physical health benefits across 172 in part dependent effect sizes from 52 studies and 63 cohorts. Numbers on the right represent the mean effect with its 95% CI in square brackets and the significance level estimating the likelihood that the effect is equal to zero. Overall effects of moderator impact were assessed via an F test, and post hoc comparisons were done using t tests (two-sided test).The F value in the top right represents a test of the hypothesis that all effects within the subpanel are equal. The Q statistic represents the heterogeneity. P values of post hoc tests are depicted whenever significant. P values above the horizontal whiskers indicate whether an effect differed significantly from a zero effect. Vertical lines indicate significant post hoc tests between moderator levels. P values were not corrected for multiple comparisons. Physical outcomes are marked in red. Mental outcomes are marked in blue.

A more detailed analysis of specific clinical conditions in adults revealed positive mental and physical health benefits for almost all assessed clinical disorders. Differences between disorders were not found, with the exception of increased effectiveness of touch interventions in neurological disorders (Supplementary Fig. 13 ).

Familiarity in the touching dyad and intervention location

Touch interventions can be performed either by familiar touchers (partners, family members or friends) or by unfamiliar touchers (health care professionals). In adults, we did not find an impact of familiarity of the toucher ( t (99) = 0.12, P  = 0.905, Hedges’ g difference of 0.02, 95% CI −0.27 to 0.24; Fig. 7a ; see Supplementary Fig. 14 for the corresponding orchard plot). Similarly, investigating the impact on mental and physical health benefits specifically, no significant differences could be detected, suggesting that familiarity is irrelevant in adults. In contrast, touch applied to newborns by their parents (almost all studies only included touch by the mother) was significantly more beneficial compared with unfamiliar touch ( t (60) = 2.09, P  = 0.041, Hedges’ g difference of 0.30, 95% CI 0.01 to 0.59) (Fig. 7b ; see Supplementary Fig. 15 for the corresponding orchard plot). Investigating mental and physical health benefits specifically revealed no significant differences. Familiarity with the location in which the touch was applied (familiar being, for example, the participants’ home) did not influence the efficacy of touch interventions (Supplementary Fig. 16 ).

a , Health benefits for being touched by a familiar (for example, partner, family member or friend) versus unfamiliar toucher (health care professional) across 463 in part dependent effect sizes from 83 studies and 101 cohorts. b , The same as a but for newborn cohorts across 171 in part dependent effect sizes from 51 studies and 62 cohorts. c , The same as a but separating mental versus physical health benefits across 463 in part dependent effect sizes from 83 studies and 101 cohorts. d , The same as b but separating mental versus physical health benefits across 169 in part dependent effect sizes from 51 studies and 62 cohorts. Numbers on the right represent the mean effect with its 95% CI in square brackets and the significance level estimating the likelihood that the effect is equal to zero. Overall effects of moderator impact were assessed via an F test, and post hoc comparisons were done using t tests (two-sided test). The F value in the top right represents a test of the hypothesis that all effects within the subpanel are equal. The Q statistic represents the heterogeneity. P values of post hoc tests are depicted whenever significant. P values above the horizontal whiskers indicate whether an effect differed significantly from a zero effect. Vertical lines indicate significant post hoc tests between moderator levels. P values were not corrected for multiple comparisons. Physical outcomes are marked in red. Mental outcomes are marked in blue.

Frequency and duration of touch interventions

How often and for how long should touch be delivered? For adults, the median touch duration across studies was 20 min and the median number of touch interventions was four sessions with an average time interval of 2.3 days between each session. For newborns, the median touch duration across studies was 17.5 min and the median number of touch interventions was seven sessions with an average time interval of 1.3 days between each session.

Delivering more touch sessions increased benefits in adults, whether overall ( t (101) = 4.90, P  < 0.001, Hedges’ g  = 0.02, 95% CI 0.01 to 0.03), physical ( t (81) = 3.07, P  = 0.003, Hedges’ g  = 0.02, 95% CI 0.01–0.03) or mental benefits ( t (72) = 5.43, P  < 0.001, Hedges’ g  = 0.02, 95% CI 0.01–0.03) were measured (Fig. 8a ). A closer look at specific outcomes for which sufficient data were available revealed that positive associations between the number of sessions and outcomes were found for trait anxiety ( t (12) = 7.90, P  < 0.001, Hedges’ g  = 0.03, 95% CI 0.02–0.04), depression ( t (20) = 10.69, P  < 0.001, Hedges’ g  = 0.03, 95% CI 0.03–0.04) and pain ( t (37) = 3.65, P  < 0.001, Hedges’ g  = 0.03, 95% CI 0.02–0.05), indicating a need for repeated sessions to improve these adverse health outcomes. Neither increasing the number of sessions for newborns nor increasing the duration of touch per session in adults or newborns increased health benefits, be they physical or mental (Fig. 8b–d ). For continuous moderators in adults, we also looked at specific health outcomes as sufficient data were generally available for further analysis. Surprisingly, we found significant negative associations between touch duration and reductions of cortisol ( t (24) = 2.71, P  = 0.012, Hedges’ g  = −0.01, 95% CI −0.01 to −0.00) and heart rate parameters ( t (21) = 2.35, P  = 0.029, Hedges’ g  = −0.01, 95% CI −0.02 to −0.00).

a , Meta-regression analysis examining the association between the number of sessions applied and the effect size in adults, either on overall health benefits (left, 469 in part dependent effect sizes from 85 studies and 103 cohorts) or for physical (middle, 245 in part dependent effect sizes from 69 studies and 83 cohorts) or mental benefits (right, 224 in part dependent effect sizes from 60 studies and 74 cohorts) separately. b , The same as a for newborns (overall: 150 in part dependent effect sizes from 46 studies and 53 cohorts; physical health: 127 in part dependent effect sizes from 44 studies and 51 cohorts; mental health: 21 in part dependent effect sizes from 11 studies and 12 cohorts). c , d the same as a ( c ) and b ( d ) but for the duration of the individual sessions. For adults, 449 in part dependent effect sizes across 80 studies and 96 cohorts were included in the overall analysis. The analysis of physical health benefits included 240 in part dependent effect sizes across 67 studies and 80 cohorts, and the analysis of mental health benefits included 209 in part dependent effect sizes from 56 studies and 69 cohorts. For newborns, 145 in part dependent effect sizes across 45 studies and 52 cohorts were included in the overall analysis. The analysis of physical health benefits included 122 in part dependent effect sizes across 43 studies and 50 cohorts, and the analysis of mental health benefits included 21 in part dependent effect sizes from 11 studies and 12 cohorts. Each dot represents an effect size. Its size indicates the precision of the study (larger indicates better). Overall effects of moderator impact were assessed via an F test (two-sided test). The P values in each panel represent the result of a regression analysis testing the hypothesis that the slope of the relationship is equal to zero. P values are not corrected for multiple testing. The shaded area around the regression line represents the 95% CI.

Demographic influences of sex and age

We used the ratio between women and men in the single-study samples as a proxy for sex-specific effects. Sex ratios were heavily skewed towards larger numbers of women in each cohort (median 83% women), and we could not find significant associations between sex ratio and overall ( t (62) = 0.08, P  = 0.935, Hedges’ g  = 0.00, 95% CI −0.00 to 0.01), mental ( t (43) = 0.55, P  = 0.588, Hedges’ g  = 0.00, 95% CI −0.00 to 0.01) or physical health benefits ( t (51) = 0.15, P  = 0.882, Hedges’ g  = −0.00, 95% CI −0.01 to 0.01). For specific outcomes that could be further analysed, we found a significant positive association of sex ratio with reductions in cortisol secretion ( t (18) = 2.31, P  = 0.033, Hedges’ g  = 0.01, 95% CI 0.00 to 0.01) suggesting stronger benefits in women. In contrast to adults, sex ratios were balanced in samples of newborns (median 53% girls). For newborns, there was no significant association with overall ( t (36) = 0.77, P  = 0.447, Hedges’ g  = −0.01, 95% CI −0.02 to 0.01) and physical health benefits of touch ( t (35) = 0.93, P  = 0.359, Hedges’ g  = −0.01, 95% CI −0.02 to 0.01). Mental health benefits did not provide sufficient data for further analysis.

The median age in the adult meta-analysis was 42.6 years (s.d. 21.16 years, range 4.5–88.4 years). There was no association between age and the overall ( t (73) = 0.35, P  = 0.727, Hedges’ g = 0.00, 95% CI −0.01 to 0.01), mental ( t (53) = 0.94, P  = 0.353, Hedges’ g  = 0.01, 95% CI −0.01 to 0.02) and physical health benefits of touch ( t (60) = 0.16, P  = 0.870, Hedges’ g  = 0.00, 95% CI −0.01 to 0.01). Looking at specific health outcomes, we found significant positive associations between mean age and improved positive affect ( t (10) = 2.54, P  = 0.030, Hedges’ g  = 0.01, 95% CI 0.00 to 0.02) as well as systolic blood pressure ( t (11) = 2.39, P  = 0.036, Hedges’ g  = 0.02, 95% CI 0.00 to 0.04).

A list of touched body parts can be found in Supplementary Table 1 . For the touched body part, we found significantly higher health benefits for head touch compared with arm touch ( t (40) = 2.14, P  = 0.039, Hedges’ g difference of 0.78, 95% CI 0.07 to 1.49) and torso touch ( t (40) = 2.23, P  = 0.031; Hedges’ g difference of 0.84, 95% CI 0.10 to 1.58; Supplementary Fig. 17 ). Touching the arm resulted in lower mental health compared with physical health benefits ( t (37) = 2.29, P  = 0.028, Hedges’ g difference of −0.35, 95% CI −0.65 to −0.05). Furthermore, we found a significantly increased physical health benefit when the head was touched as opposed to the torso ( t (37) = 2.10, P  = 0.043, Hedges’ g difference of 0.96, 95% CI 0.06 to 1.86). Thus, head touch such as a face or scalp massage could be especially beneficial.

Directionality

In adults, we tested whether a uni- or bidirectional application of touch mattered. The large majority of touch was applied unidirectionally ( k  = 442 of 469 effects). Unidirectional touch had higher health benefits ( t (101) = 2.17, P  = 0.032, Hedges’ g difference of 0.30, 95% CI 0.03 to 0.58) than bidirectional touch. Specifically, mental health benefits were higher in unidirectional touch ( t (99) = 2.33, P  = 0.022, Hedges’ g difference of 0.46, 95% CI 0.06 to 0.66).

Study location

For adults, we found significantly stronger health benefits of touch in South American compared with North American cohorts ( t (95) = 2.03, P  = 0.046, Hedges’ g difference of 0.37, 95% CI 0.01 to 0.73) and European cohorts ( t (95) = 2.22, P  = 0.029, Hedges’ g difference of 0.36, 95% CI 0.04 to 0.68). For newborns, we found weaker effects in North American cohorts compared to Asian ( t (55) = 2.28, P  = 0.026, Hedges’ g difference of −0.37, 95% CI −0.69 to −0.05) and European cohorts ( t (55) = 2.36, P  = 0.022, Hedges’ g difference of −0.40, 95% CI −0.74 to −0.06). Investigating the interaction with mental and physical health benefits did not reveal any effects of study location in both meta-analyses (Supplementary Fig. 18 ).

Systematic review of studies without effect sizes

All studies where effect size data could not be obtained or that did not meet the meta-analysis inclusion criteria can be found on the OSF project 12 in the file ‘Study_lists_final_revised.xlsx’ (sheet ‘Studies_without_effect_sizes’). Specific reasons for exclusion are furthermore documented in Supplementary Table 2 . For human health outcomes assessed across 56 studies and n  = 2,438 individuals, interventions mostly comprised massage therapy ( k  = 86 health outcomes) and kangaroo care ( k  = 33 health outcomes). For datasets where no effect size could be computed, 90.0% of mental health and 84.3% of physical health parameters were positively impacted by touch. Positive impact of touch did not differ between types of touch interventions. These results match well with the observations of the meta-analysis of a highly positive benefit of touch overall, irrespective of whether a massage or any other intervention is applied.

We also assessed health outcomes in animals across 19 studies and n  = 911 subjects. Most research was conducted in rodents. Animals that received touch were rats (ten studies, k  = 16 health outcomes), mice (four studies, k  = 7 health outcomes), macaques (two studies, k  = 3 health outcomes), cats (one study, k  = 3 health outcomes), lambs (one study, k  = 2 health outcomes) and coral reef fish (one study, k  = 1 health outcome). Touch interventions mostly comprised stroking ( k  = 13 health outcomes) and tickling ( k  = 10 health outcomes). For animal studies, 71.4% of effects showed benefits to mental health-like parameters and 81.8% showed positive physical health effects. We thus found strong evidence that touch interventions, which were mostly conducted by humans (16 studies with human touch versus 3 studies with object touch), had positive health effects in animal species as well.

The key aim of the present study was twofold: (1) to provide an estimate of the effect size of touch interventions and (2) to disambiguate moderating factors to potentially tailor future interventions more precisely. Overall, touch interventions were beneficial for both physical and mental health, with a medium effect size. Our work illustrates that touch interventions are best suited for reducing pain, depression and anxiety in adults and children as well as for increasing weight gain in newborns. These findings are in line with previous meta-analyses on this topic, supporting their conclusions and their robustness to the addition of more datasets. One limitation of previous meta-analyses is that they focused on specific health outcomes or populations, despite primary studies often reporting effects on multiple health parameters simultaneously (for example, ref. 13 focusing on neck and shoulder pain and ref. 14 focusing on massage therapy in preterms). To our knowledge, only ref. 5 provides a multivariate picture for a large number of dependent variables. However, this study analysed their data in separate random effects models that did not account for multivariate reporting nor for the multilevel structure of the data, as such approaches have only become available recently. Thus, in addition to adding a substantial amount of new data, our statistical approach provides a more accurate depiction of effect size estimates. Additionally, our study investigated a variety of moderating effects that did not reach significance (for example, sex ratio, mean age or intervention duration) or were not considered (for example, the benefits of robot or object touch) in previous meta-analyses in relation to touch intervention efficacy 5 , probably because of the small number of studies with information on these moderators in the past. Owing to our large-scale approach, we reached high statistical power for many moderator analyses. Finally, previous meta-analyses on this topic exclusively focused on massage therapy in adults or kangaroo care in newborns 15 , leaving out a large number of interventions that are being carried out in research as well as in everyday life to improve well-being. Incorporating these studies into our study, we found that, in general, both massages and other types of touch, such as gentle touch, stroking or kangaroo care, showed similar health benefits.

While it seems to be less critical which touch intervention is applied, the frequency of interventions seems to matter. More sessions were positively associated with the improvement of trait outcomes such as depression and anxiety but also pain reductions in adults. In contrast to session number, increasing the duration of individual sessions did not improve health effects. In fact, we found some indications of negative relationships in adults for cortisol and blood pressure. This could be due to habituating effects of touch on the sympathetic nervous system and hypothalamic–pituitary–adrenal axis, ultimately resulting in diminished effects with longer exposure, or decreased pleasantness ratings of affective touch with increasing duration 16 . For newborns, we could not support previous notions that the duration of the touch intervention is linked to benefits in weight gain 17 . Thus, an ideal intervention protocol does not seem to have to be excessively long. It should be noted that very few interventions lasted less than 5 min, and it therefore remains unclear whether very short interventions have the same effect.

A critical issue highlighted in the pandemic was the lack of touch due to social restrictions 18 . To accommodate the need for touch in individuals with small social networks (for example, institutionalized or isolated individuals), touch interventions using objects/robots have been explored in the past (for a review, see ref. 11 ). We show here that touch interactions outside of the human–human domain are beneficial for mental and physical health outcomes. Importantly, object/robot touch was not as effective in improving mental health as human-applied touch. A sub-analysis of missing skin-to-skin contact among humans indicated that mental health effects of touch might be mediated by the presence of skin-to-skin contact. Thus, it seems profitable to include skin-to-skin contact in future touch interventions, in line with previous findings in newborns 19 . In robots, recent advancements in synthetic skin 20 should be investigated further in this regard. It should be noted that, although we did not observe significant differences in physical health benefits between human–human and human–object touch, the variability of effect sizes was higher in human–object touch. The conditions enabling object or robot interactions to improve well-being should therefore be explored in more detail in the future.

Touch was beneficial for both healthy and clinical cohorts. These data are critical as most previous meta-analytic research has focused on individuals diagnosed with clinical disorders (for example, ref. 6 ). For mental health outcomes, we found larger effects in clinical cohorts. A possible reason could relate to increased touch wanting 21 in patients. For example, loneliness often co-occurs with chronic illnesses 22 , which are linked to depressed mood and feelings of anxiety 23 . Touch can be used to counteract this negative development 24 , 25 . In adults and children, knowing the toucher did not influence health benefits. In contrast, familiarity affected overall health benefits in newborns, with parental touch being more beneficial than touch applied by medical staff. Previous studies have suggested that early skin-to-skin contact and exposure to maternal odour is critical for a newborn’s ability to adapt to a new environment 26 , supporting the notion that parental care is difficult to substitute in this time period.

With respect to age-related effects, our data further suggest that increasing age was associated with a higher benefit through touch for systolic blood pressure. These findings could potentially be attributed to higher basal blood pressure 27 with increasing age, allowing for a stronger modulation of this parameter. For sex differences, our study provides some evidence that there are differences between women and men with respect to health benefits of touch. Overall, research on sex differences in touch processing is relatively sparse (but see refs. 28 , 29 ). Our results suggest that buffering effects against physiological stress are stronger in women. This is in line with increased buffering effects of hugs in women compared with men 30 . The female-biased primary research in adults, however, begs for more research in men or non-binary individuals. Unfortunately, our study could not dive deeper into this topic as health benefits broken down by sex or gender were almost never provided. Recent research has demonstrated that sensory pleasantness is affected by sex and that this also interacts with the familiarity of the other person in the touching dyad 29 , 31 . In general, contextual factors such as sex and gender or the relationship of the touching dyad, differences in cultural background or internal states such as stress have been demonstrated to be highly influential in the perception of affective touch and are thus relevant to maximizing the pleasantness and ultimately the health benefits of touch interactions 32 , 33 , 34 . As a positive personal relationship within the touching dyad is paramount to induce positive health effects, future research applying robot touch to promote well-being should therefore not only explore synthetic skin options but also focus on improving robots as social agents that form a close relationship with the person receiving the touch 35 .

As part of the systematic review, we also assessed the effects of touch interventions in non-human animals. Mimicking the results of the meta-analysis in humans, beneficial effects of touch in animals were comparably strong for mental health-like and physical health outcomes. This may inform interventions to promote animal welfare in the context of animal experiments 36 , farming 37 and pets 38 . While most studies investigated effects in rodents, which are mostly used as laboratory animals, these results probably transfer to livestock and common pets as well. Indeed, touch was beneficial in lambs, fish and cats 39 , 40 , 41 . The positive impact of human touch in rodents also allows for future mechanistic studies in animal models to investigate how interventions such as tickling or stroking modulate hormonal and neuronal responses to touch in the brain. Furthermore, the commonly proposed oxytocin hypothesis can be causally investigated in these animal models through, for example, optogenetic or chemogenetic techniques 42 . We believe that such translational approaches will further help in optimizing future interventions in humans by uncovering the underlying mechanisms and brain circuits involved in touch.

Our results offer many promising avenues to improve future touch interventions, but they also need to be discussed in light of their limitations. While the majority of findings showed robust health benefits of touch interventions across moderators when compared with a null effect, post hoc tests of, for example, familiarity effects in newborns or mental health benefit differences between human and object touch only barely reached significance. Since we computed a large number of statistical tests in the present study, there is a risk that these results are false positives. We hope that researchers in this field are stimulated by these intriguing results and target these questions by primary research through controlled experimental designs within a well-powered study. Furthermore, the presence of small-study bias in both meta-analyses is indicative that the effect size estimates presented here might be overestimated as null results are often unpublished. We want to stress however that this bias is probably reduced by the multivariate reporting of primary studies. Most studies that reported on multiple health outcomes only showed significant findings for one or two among many. Thus, the multivariate nature of primary research in this field allowed us to include many non-significant findings in the present study. Another limitation pertains to the fact that we only included articles in languages mostly spoken in Western countries. As a large body of evidence comes from Asian countries, it could be that primary research was published in languages other than specified in the inclusion criteria. Thus, despite the large and inclusive nature of our study, some studies could have been missed regardless. Another factor that could not be accounted for in our meta-analysis was that an important prerequisite for touch to be beneficial is its perceived pleasantness. The level of pleasantness associated with being touched is modulated by several parameters 34 including cultural acceptability 43 , perceived humanness 44 or a need for touch 45 , which could explain the observed differences for certain moderators, such as human–human versus robot–human interaction. Moreover, the fact that secondary categorical moderators could not be investigated with respect to specific health outcomes, owing to the lack of data points, limits the specificity of our conclusions in this regard. It thus remains unclear whether, for example, a decreased mental health benefit in the absence of skin-to-skin contact is linked mostly to decreased anxiolytic effects, changes in positive/negative affect or something else. Since these health outcomes are however highly correlated 46 , it is likely that such effects are driven by multiple health outcomes. Similarly, it is important to note that our conclusions mainly refer to outcomes measured close to the touch intervention as we did not include long-term outcomes. Finally, it needs to be noted that blinding towards the experimental condition is essentially impossible in touch interventions. Although we compared the touch intervention with other interventions, such as relaxation therapy, as control whenever possible, contributions of placebo effects cannot be ruled out.

In conclusion, we show clear evidence that touch interventions are beneficial across a large number of both physical and mental health outcomes, for both healthy and clinical cohorts, and for all ages. These benefits, while influenced in their magnitude by study cohorts and intervention characteristics, were robustly present, promoting the conclusion that touch interventions can be systematically employed across the population to preserve and improve our health.

Open science practices

All data and code are accessible in the corresponding OSF project 12 . The systematic review was registered on PROSPERO (CRD42022304281) before the start of data collection. We deviated from the pre-registered plan as follows:

Deviation 1: During our initial screening for the systematic review, we were confronted with a large number of potential health outcomes to look at. This observation of multivariate outcomes led us to register an amendment during data collection (but before any effect size or moderator screening). In doing so, we aimed to additionally extract meta-analytic effects for a more quantitative assessment of our review question that can account for multivariate data reporting and dependencies of effects within the same study. Furthermore, as we noted a severe lack of studies with respect to health outcomes for animals during the inclusion assessment for the systematic review, we decided that the meta-analysis would only focus on outcomes that could be meaningfully analysed on the meta-analytic level and therefore only included health outcomes of human participants.

Deviation 2: In the pre-registration, we did not explicitly exclude non-randomized trials. Since an explicit use of non-randomization for group allocation significantly increases the risk of bias, we decided to exclude them a posteriori from data analysis.

Deviation 3: In the pre-registration, we outlined a tertiary moderator level, namely benefits of touch application versus touch reception. This level was ignored since no included study specifically investigated the benefits of touch application by itself.

Deviation 4: In the pre-registration, we suggested using the RoBMA function 47 to provide a Bayesian framework that allows for a more accurate assessment of publication bias beyond small-study bias. Unfortunately, neither multilevel nor multivariate data structures are supported by the RoBMA function, to our knowledge. For this reason, we did not further pursue this analysis, as the hierarchical nature of the data would not be accounted for.

Deviation 5: Beyond the pre-registered inclusion and exclusion criteria, we also excluded dissertations owing to their lack of peer review.

Deviation 6: In the pre-registration, we stated to investigate the impact of sex of the person applying the touch. This moderator was not further analysed, as this information was rarely given and the individuals applying the touch were almost exclusively women (7 males, 24 mixed and 85 females in studies on adults/children; 3 males, 17 mixed and 80 females in studied on newborns).

Deviation 7: The time span of the touch intervention as assessed by subtracting the final day of the intervention from the first day was not investigated further owing to its very high correlation with the number of sessions ( r (461) = 0.81 in the adult meta-analysis, r (145) = 0.84 in the newborn meta-analysis).

Inclusion and exclusion criteria

To be included in the systematic review, studies had to investigate the relationship between at least one health outcome (physical and/or mental) in humans or animals and a touch intervention, include explicit physical touch by another human, animal or object as part of an intervention and include an experimental and control condition/group that are differentiated by touch alone. Of note, as a result of this selection process, no animal-to-animal touch intervention study was included, as they never featured a proper no-touch control. Human touch was always explicit touch by a human (that is, no brushes or other tools), either with or without skin-to-skin contact. Regarding the included health outcomes, we aimed to be as broad as possible but excluded parameters such as neurophysiological responses or pleasantness ratings after touch application as they do not reflect health outcomes. All included studies in the meta-analysis and systematic review 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141 , 142 , 143 , 144 , 145 , 146 , 147 , 148 , 149 , 150 , 151 , 152 , 153 , 154 , 155 , 156 , 157 , 158 , 159 , 160 , 161 , 162 , 163 , 164 , 165 , 166 , 167 , 168 , 169 , 170 , 171 , 172 , 173 , 174 , 175 , 176 , 177 , 178 , 179 , 180 , 181 , 182 , 183 , 184 , 185 , 186 , 187 , 188 , 189 , 190 , 191 , 192 , 193 , 194 , 195 , 196 , 197 , 198 , 199 , 200 , 201 , 202 , 203 , 204 , 205 , 206 , 207 , 208 , 209 , 210 , 211 , 212 , 213 , 214 , 215 , 216 , 217 , 218 , 219 , 220 , 221 , 222 , 223 , 224 , 225 , 226 , 227 , 228 , 229 , 230 , 231 , 232 , 233 , 234 , 235 , 236 , 237 , 238 , 239 , 240 , 241 , 242 , 243 , 244 , 245 , 246 , 247 , 248 , 249 , 250 , 251 , 252 , 253 , 254 , 255 , 256 , 257 , 258 , 259 , 260 , 261 , 262 , 263 are listed in Supplementary Table 2 . All excluded studies are listed in Supplementary Table 3 , together with a reason for exclusion. We then applied a two-step process: First, we identified all potential health outcomes and extracted qualitative information on those outcomes (for example, direction of effect). Second, we extracted quantitative information from all possible outcomes (for example, effect sizes). The meta-analysis additionally required a between-subjects design (to clearly distinguish touch from no-touch effects and owing to missing information about the correlation between repeated measurements 264 ). Studies that explicitly did not apply a randomized protocol were excluded before further analysis to reduce risk of bias. The full study lists for excluded and included studies can be found in the OSF project 12 in the file ‘Study_lists_final_revised.xlsx’. In terms of the time frame, we conducted an open-start search of studies until 2022 and identified studies conducted between 1965 and 2022.

Data collection

We used Google Scholar, PubMed and Web of Science for our literature search, with no limitations regarding the publication date and using pre-specified search queries (see Supplementary Information for the exact keywords used). All procedures were in accordance with the updated Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines 265 . Articles were assessed in French, Dutch, German or English. The above databases were searched from 2 December 2021 until 1 October 2022. Two independent coders evaluated each paper against the inclusion and exclusion criteria. Inconsistencies between coders were checked and resolved by J.P. and H.H. Studies excluded/included for the review and meta-analysis can be found on the OSF project.

Search queries

We used the following keywords to search the chosen databases. Agents (human versus animal versus object versus robot) and touch outcome (physical versus mental) were searched separately together with keywords searching for touch.

TOUCH: Touch OR Social OR Affective OR Contact OR Tactile interaction OR Hug OR Massage OR Embrace OR Kiss OR Cradling OR Stroking OR Haptic interaction OR tickling

AGENT: Object OR Robot OR human OR animal OR rodent OR primate

MENTAL OUTCOME: Health OR mood OR Depression OR Loneliness OR happiness OR life satisfaction OR Mental Disorder OR well-being OR welfare OR dementia OR psychological OR psychiatric OR anxiety OR Distress

PHYSICAL OUTCOME: Health OR Stress OR Pain OR cardiovascular health OR infection risk OR immune response OR blood pressure OR heart rate

Data extraction and preparation

Data extraction began on 10 October 2022 and was concluded on 25 February 2023. J.P. and H.H. oversaw the data collection process, and checked and resolved all inconsistencies between coders.

Health benefits of touch were always coded by positive summary effects, whereas adverse health effects of touch were represented by negative summary effects. If multiple time points were measured for the same outcome on the same day after a single touch intervention, we extracted the peak effect size (in either the positive or negative direction). If the touch intervention occurred multiple times and health outcomes were assessed for each time point, we extracted data points separately. However, we only extracted immediate effects, as long-term effects not controlled through the experimental conditions could be due to influences other than the initial touch intervention. Measurements assessing long-term effects without explicit touch sessions in the breaks were excluded for the same reason. Common control groups for touch interventions comprised active (for example, relaxation therapy) as well as passive control groups (for example, standard medical care). In the case of multiple control groups, we always contrasted the touch group to the group that most closely matched the touch condition (for example, relaxation therapy was preferred over standard medical care). We extracted information from all moderators listed in the pre-registration (Supplementary Table 4 ). A list of included and excluded health outcomes is presented in Supplementary Table 5 . Authors of studies with possible effects but missing information to calculate those effects were contacted via email and asked to provide the missing data (response rate 35.7%).

After finalizing the list of included studies for the systematic review, we added columns for moderators and the coding schema for our meta-analysis per our updated registration. Then, each study was assessed for its eligibility in the meta-analysis by two independent coders (J.P., H.H., K.F. or F.M.). To this end, all coders followed an a priori specified procedure: First, the PDF was skimmed for possible effects to extract, and the study was excluded if no PDF was available or the study was in a language different from the ones specified in ‘ Data collection ’. Effects from studies that met the inclusion criteria were extracted from all studies listing descriptive values or statistical parameters to calculate effect sizes. A website 266 was used to convert descriptive and statistical values available in the included studies (means and standard deviations/standard errors/confidence intervals, sample sizes, F values, t values, t test P values or frequencies) into Cohen’s d , which were then converted in Hedges’ g . If only P value thresholds were reported (for example, P  < 0.01), we used this, most conservative, value as the P value to calculate the effect size (for example, P  = 0.01). If only the total sample size was given but that number was even and the participants were randomly assigned to each group, we assumed equal sample sizes for each group. If delta change scores (for example, pre- to post-touch intervention) were reported, we used those over post-touch only scores. In case frequencies were 0 when frequency tables were used to determine effect sizes, we used a value of 0.5 as a substitute to calculate the effect (the default setting in the ‘metafor’ function 267 ). From these data, Hedges’ g and its variance could be derived. Effect sizes were always computed between the experimental and the control group.

Statistical analysis and risk of bias assessment

Owing to the lack of identified studies, health benefits to animals were not included as part of the statistical analysis. One meta-analysis was performed for adults, adolescents and children, as outcomes were highly comparable. We refer to this meta-analysis as the adult meta-analysis, as children/adolescent cohorts were only targeted in a minority of studies. A separate meta-analysis was performed for newborns, as their health outcomes differed substantially from any other age group.

Data were analysed using R (version 4.2.2) with the ‘rma.mv’ function from the ‘metafor’ package 267 in a multistep, multivariate and multilevel fashion.

We calculated an overall effect of touch interventions across all studies, cohorts and health outcomes. To account for the hierarchical structure of the data, we used a multilevel structure with random effects at the study, cohort and effects level. Furthermore, we calculated the variance–covariance matrix of all data points to account for the dependencies of measured effects within each individual cohort and study. The variance–covariance matrix was calculated by default with an assumed correlation of effect sizes within each cohort of ρ  = 0.6. As ρ needed to be assumed, sensitivity analyses for all computed effect estimates were conducted using correlations between effects of 0, 0.2, 0.4 and 0.8. The results of these sensitivity analyses can be found in ref. 12 . No conclusion drawn in the present manuscript was altered by changing the level of ρ . The sensitivity analyses, however, showed that higher assumed correlations lead to more conservative effect size estimates (see Supplementary Figs. 19 and 20 for the adult and newborn meta-analyses, respectively), reducing the type I error risk in general 268 . In addition to these procedures, we used robust variance estimation with cluster-robust inference at the cohort level. This step is recommended to more accurately determine the confidence intervals in complex multivariate models 269 . The data distribution was assumed to be normal, but this was not formally tested.

To determine whether individual effects had a strong influence on our results, we calculated Cook’s distance D . Here, a threshold of D  > 0.5 was used to qualify a study as influential 270 . Heterogeneity in the present study was assessed using Cochran’s Q , which determines whether the extracted effect sizes estimate a common population effect size. Although the Q statistic in the ‘rma.mv’ function accounts for the hierarchical nature of the data, we also quantified the heterogeneity estimator σ ² for each random-effects level to provide a comprehensive overview of heterogeneity indicators. These indicators for all models can be found on the OSF project 12 in the Table ‘Model estimates’. To assess small study bias, we visually inspected the funnel plot and used the standard error as a moderator in the overarching meta-analyses.

Before any sub-group analysis, the overall effect size was used as input for power calculations. While such post hoc power calculations might be limited, we believe that a minimum number of effects to be included in subgroup analyses was necessary to allow for meaningful conclusions. Such medium effect sizes would also probably be the minimum effect sizes of interest for researchers as well as clinical practitioners. Power calculation for random-effects models further requires a sample size for each individual effect as well as an approximation of the expected heterogeneity between studies. For the sample size input, we used the median sample size in each of our studies. For heterogeneity, we assumed a value between medium and high levels of heterogeneity ( I ² = 62.5% 271 ), as moderator analyses typically aim at reducing heterogeneity overall. Subgroups were only further investigated if the number of observed effects achieved ~80% power under these circumstances, to allow for a more robust interpretation of the observed effects (see Supplementary Figs. 5 and 6 for the adult and newborn meta-analysis, respectively). In a next step, we investigated all pre-registered moderators for which sufficient power was detected. We first looked at our primary moderators (mental versus physical health) and how the effect sizes systematically varied as a function of our secondary moderators (for example, human–human or human–object touch, duration, skin-to-skin presence, etc.). We always included random slopes to allow for our moderators to vary with the random effects at our clustering variable, which is recommended in multilevel models to reduce false positives 272 . All statistical tests were performed two-sided. Significance of moderators was determined using omnibus F tests. Effect size differences between moderator levels and their confidence intervals were assessed via t tests.

Post hoc t tests were performed comparing mental and physical health benefits within each interacting moderator (for example, mental versus physical health benefits in cancer patients) and mental or physical health benefits across levels of the interacting moderator (for example, mental health benefits in cancer versus pain patients). The post hoc tests were not pre-registered. Data were visualized using forest plots and orchard plots 273 for categorical moderators and scatter plots for continuous moderators.

For a broad overview of prior work and their biases, risk of bias was assessed for all studies included in both meta-analyses and the systematic review. We assessed the risk of bias for the following parameters:

Bias from randomization, including whether a randomization procedure was performed, whether it was a between- or within-participant design and whether there were any baseline differences for demographic or dependent variables.

Sequence bias resulting from a lack of counterbalancing in within-subject designs.

Performance bias resulting from the participants or experiments not being blinded to the experimental conditions.

Attrition bias resulting from different dropout rates between experimental groups.

Note that four studies in the adult meta-analysis did not explicitly mention randomization as part of their protocol. However, since these studies never showed any baseline differences in all relevant variables (see ‘Risk of Bias’ table on the OSF project ) , we assumed that randomization was performed but not mentioned. Sequence bias was of no concern for studies for the meta-analysis since cross-over designs were excluded. It was, however, assessed for studies within the scope of the systematic review. Importantly, performance bias was always high in the adult/children meta-analysis, as blinding of the participants and experimenters to the experimental conditions was not possible owing to the nature of the intervention (touch versus no touch). For studies with newborns and animals, we assessed the performance bias as medium since neither newborns or animals are likely to be aware of being part of an experiment or specific group. An overview of the results is presented in Supplementary Fig. 21 , and the precise assessment for each study can be found on the OSF project 12 in the ‘Risk of Bias’ table.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

All data are available via Open Science Framework at https://doi.org/10.17605/OSF.IO/C8RVW (ref. 12 ). Source data are provided with this paper.

Code availability

All code is available via Open Science Framework at https://doi.org/10.17605/OSF.IO/C8RVW (ref. 12 ).

Fulkerson, M. The First Sense: a Philosophical Study of Human Touch (MIT Press, 2013).

Farroni, T., Della Longa, L. & Valori, I. The self-regulatory affective touch: a speculative framework for the development of executive functioning. Curr. Opin. Behav. Sci. 43 , 167–173 (2022).

Article   Google Scholar  

Ocklenburg, S. et al. Hugs and kisses—the role of motor preferences and emotional lateralization for hemispheric asymmetries in human social touch. Neurosci. Biobehav. Rev. 95 , 353–360 (2018).

Ardiel, E. L. & Rankin, C. H. The importance of touch in development. Paediatr. Child Health 15 , 153–156 (2010).

Article   PubMed   PubMed Central   Google Scholar  

Moyer, C. A., Rounds, J. & Hannum, J. W. A meta-analysis of massage therapy research. Psychol. Bull. 130 , 3–18 (2004).

Article   PubMed   Google Scholar  

Lee, S. H., Kim, J. Y., Yeo, S., Kim, S. H. & Lim, S. Meta-analysis of massage therapy on cancer pain. Integr. Cancer Ther. 14 , 297–304 (2015).

LaFollette, M. R., O’Haire, M. E., Cloutier, S. & Gaskill, B. N. A happier rat pack: the impacts of tickling pet store rats on human–animal interactions and rat welfare. Appl. Anim. Behav. Sci. 203 , 92–102 (2018).

Packheiser, J., Michon, F. Eva, C., Fredriksen, K. & Hartmann H. The physical and mental health benefits of social touch: a comparative systematic review and meta-analysis. PROSPERO https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022304281 (2023).

Lakens, D. Sample size justification. Collabra. Psychol. 8 , 33267 (2022).

Quintana, D. S. A guide for calculating study-level statistical power for meta-analyses. Adv. Meth. Pract. Psychol. Sci. https://doi.org/10.1177/25152459221147260 (2023).

Eckstein, M., Mamaev, I., Ditzen, B. & Sailer, U. Calming effects of touch in human, animal, and robotic interaction—scientific state-of-the-art and technical advances. Front. Psychiatry 11 , 555058 (2020).

Packheiser, J. et al. The physical and mental health benefits of affective touch: a comparative systematic review and multivariate meta-analysis. Open Science Framework https://doi.org/10.17605/OSF.IO/C8RVW (2023).

Kong, L. J. et al. Massage therapy for neck and shoulder pain: a systematic review and meta-analysis. Evid. Based Complement. Altern. Med. 2013 , 613279 (2013).

Wang, L., He, J. L. & Zhang, X. H. The efficacy of massage on preterm infants: a meta-analysis. Am. J. Perinatol. 30 , 731–738 (2013).

Field, T. Massage therapy research review. Complement. Ther. Clin. Pract. 24 , 19–31 (2016).

Bendas, J., Ree, A., Pabel, L., Sailer, U. & Croy, I. Dynamics of affective habituation to touch differ on the group and individual level. Neuroscience 464 , 44–52 (2021).

Article   CAS   PubMed   Google Scholar  

Charpak, N., Montealegre‐Pomar, A. & Bohorquez, A. Systematic review and meta‐analysis suggest that the duration of Kangaroo mother care has a direct impact on neonatal growth. Acta Paediatr. 110 , 45–59 (2021).

Packheiser, J. et al. A comparison of hugging frequency and its association with momentary mood before and during COVID-19 using ecological momentary assessment. Health Commun. https://doi.org/10.1080/10410236.2023.2198058 (2023).

Whitelaw, A., Heisterkamp, G., Sleath, K., Acolet, D. & Richards, M. Skin to skin contact for very low birthweight infants and their mothers. Arch. Dis. Child. 63 , 1377–1381 (1988).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Yogeswaran, N. et al. New materials and advances in making electronic skin for interactive robots. Adv. Robot. 29 , 1359–1373 (2015).

Durkin, J., Jackson, D. & Usher, K. Touch in times of COVID‐19: touch hunger hurts. J. Clin. Nurs. https://doi.org/10.1111/jocn.15488 (2021).

Rokach, A., Lechcier-Kimel, R. & Safarov, A. Loneliness of people with physical disabilities. Soc. Behav. Personal. Int. J. 34 , 681–700 (2006).

Palgi, Y. et al. The loneliness pandemic: loneliness and other concomitants of depression, anxiety and their comorbidity during the COVID-19 outbreak. J. Affect. Disord. 275 , 109–111 (2020).

Heatley-Tejada, A., Dunbar, R. I. M. & Montero, M. Physical contact and loneliness: being touched reduces perceptions of loneliness. Adapt. Hum. Behav. Physiol. 6 , 292–306 (2020).

Article   CAS   Google Scholar  

Packheiser, J. et al. The association of embracing with daily mood and general life satisfaction: an ecological momentary assessment study. J. Nonverbal Behav. 46 , 519–536 (2022).

Porter, R. The biological significance of skin-to-skin contact and maternal odours. Acta Paediatr. 93 , 1560–1562 (2007).

Hawkley, L. C., Masi, C. M., Berry, J. D. & Cacioppo, J. T. Loneliness is a unique predictor of age-related differences in systolic blood pressure. Psychol. Aging 21 , 152–164 (2006).

Russo, V., Ottaviani, C. & Spitoni, G. F. Affective touch: a meta-analysis on sex differences. Neurosci. Biobehav. Rev. 108 , 445–452 (2020).

Schirmer, A. et al. Understanding sex differences in affective touch: sensory pleasantness, social comfort, and precursive experiences. Physiol. Behav. 250 , 113797 (2022).

Berretz, G. et al. Romantic partner embraces reduce cortisol release after acute stress induction in women but not in men. PLoS ONE 17 , e0266887 (2022).

Gazzola, V. et al. Primary somatosensory cortex discriminates affective significance in social touch. Proc. Natl Acad. Sci. USA 109 , E1657–E1666 (2012).

Sorokowska, A. et al. Affective interpersonal touch in close relationships: a cross-cultural perspective. Personal. Soc. Psychol. Bull. 47 , 1705–1721 (2021).

Ravaja, N., Harjunen, V., Ahmed, I., Jacucci, G. & Spapé, M. M. Feeling touched: emotional modulation of somatosensory potentials to interpersonal touch. Sci. Rep. 7 , 40504 (2017).

Saarinen, A., Harjunen, V., Jasinskaja-Lahti, I., Jääskeläinen, I. P. & Ravaja, N. Social touch experience in different contexts: a review. Neurosci. Biobehav. Rev. 131 , 360–372 (2021).

Huisman, G. Social touch technology: a survey of haptic technology for social touch. IEEE Trans. Haptics 10 , 391–408 (2017).

Lewejohann, L., Schwabe, K., Häger, C. & Jirkof, P. Impulse for animal welfare outside the experiment. Lab. Anim. https://doi.org/10.17169/REFUBIUM-26765 (2020).

Sørensen, J. T., Sandøe, P. & Halberg, N. Animal welfare as one among several values to be considered at farm level: the idea of an ethical account for livestock farming. Acta Agric. Scand. A 51 , 11–16 (2001).

Google Scholar  

Verga, M. & Michelazzi, M. Companion animal welfare and possible implications on the human–pet relationship. Ital. J. Anim. Sci. 8 , 231–240 (2009).

Coulon, M. et al. Do lambs perceive regular human stroking as pleasant? Behavior and heart rate variability analyses. PLoS ONE 10 , e0118617 (2015).

Soares, M. C., Oliveira, R. F., Ros, A. F. H., Grutter, A. S. & Bshary, R. Tactile stimulation lowers stress in fish. Nat. Commun. 2 , 534 (2011).

Gourkow, N., Hamon, S. C. & Phillips, C. J. C. Effect of gentle stroking and vocalization on behaviour, mucosal immunity and upper respiratory disease in anxious shelter cats. Prev. Vet. Med. 117 , 266–275 (2014).

Oliveira, V. E. et al. Oxytocin and vasopressin within the ventral and dorsal lateral septum modulate aggression in female rats. Nat. Commun. 12 , 2900 (2021).

Burleson, M. H., Roberts, N. A., Coon, D. W. & Soto, J. A. Perceived cultural acceptability and comfort with affectionate touch: differences between Mexican Americans and European Americans. J. Soc. Personal. Relatsh. 36 , 1000–1022 (2019).

Wijaya, M. et al. The human ‘feel’ of touch contributes to its perceived pleasantness. J. Exp. Psychol. Hum. Percept. Perform. 46 , 155–171 (2020).

Golaya, S. Touch-hunger: an unexplored consequence of the COVID-19 pandemic. Indian J. Psychol. Med. 43 , 362–363 (2021).

Ng, T. W. H., Sorensen, K. L., Zhang, Y. & Yim, F. H. K. Anger, anxiety, depression, and negative affect: convergent or divergent? J. Vocat. Behav. 110 , 186–202 (2019).

Maier, M., Bartoš, F. & Wagenmakers, E.-J. Robust Bayesian meta-analysis: addressing publication bias with model-averaging. Psychol. Methods 28 , 107–122 (2022).

Ahles, T. A. et al. Massage therapy for patients undergoing autologous bone marrow transplantation. J. Pain. Symptom Manag. 18 , 157–163 (1999).

Albert, N. M. et al. A randomized trial of massage therapy after heart surgery. Heart Lung 38 , 480–490 (2009).

Ang, J. Y. et al. A randomized placebo-controlled trial of massage therapy on the immune system of preterm infants. Pediatrics 130 , e1549–e1558 (2012).

Arditi, H., Feldman, R. & Eidelman, A. I. Effects of human contact and vagal regulation on pain reactivity and visual attention in newborns. Dev. Psychobiol. 48 , 561–573 (2006).

Arora, J., Kumar, A. & Ramji, S. Effect of oil massage on growth and neurobehavior in very low birth weight preterm neonates. Indian Pediatr. 42 , 1092–1100 (2005).

PubMed   Google Scholar  

Asadollahi, M., Jabraeili, M., Mahallei, M., Asgari Jafarabadi, M. & Ebrahimi, S. Effects of gentle human touch and field massage on urine cortisol level in premature infants: a randomized, controlled clinical trial. J. Caring Sci. 5 , 187–194 (2016).

Basiri-Moghadam, M., Basiri-Moghadam, K., Kianmehr, M. & Jani, S. The effect of massage on neonatal jaundice in stable preterm newborn infants: a randomized controlled trial. J. Pak. Med. Assoc. 65 , 602–606 (2015).

Bauer, B. A. et al. Effect of massage therapy on pain, anxiety, and tension after cardiac surgery: a randomized study. Complement. Ther. Clin. Pract. 16 , 70–75 (2010).

Beijers, R., Cillessen, L. & Zijlmans, M. A. C. An experimental study on mother-infant skin-to-skin contact in full-terms. Infant Behav. Dev. 43 , 58–65 (2016).

Bennett, S. et al. Acute effects of traditional Thai massage on cortisol levels, arterial blood pressure and stress perception in academic stress condition: a single blind randomised controlled trial. J. Bodyw. Mov. Therapies 20 , 286–292 (2016).

Bergman, N., Linley, L. & Fawcus, S. Randomized controlled trial of skin-to-skin contact from birth versus conventional incubator for physiological stabilization in 1200- to 2199-gram newborns. Acta Paediatr. 93 , 779–785 (2004).

Bigelow, A., Power, M., MacLellan‐Peters, J., Alex, M. & McDonald, C. Effect of mother/infant skin‐to‐skin contact on postpartum depressive symptoms and maternal physiological stress. J. Obstet. Gynecol. Neonatal Nurs. 41 , 369–382 (2012).

Billhult, A., Bergbom, I. & Stener-Victorin, E. Massage relieves nausea in women with breast cancer who are undergoing chemotherapy. J. Altern. Complement. Med. 13 , 53–57 (2007).

Billhult, A., Lindholm, C., Gunnarsson, R. & Stener-Victorin, E. The effect of massage on cellular immunity, endocrine and psychological factors in women with breast cancer—a randomized controlled clinical trial. Auton. Neurosci. 140 , 88–95 (2008).

Braun, L. A. et al. Massage therapy for cardiac surgery patients—a randomized trial. J. Thorac. Cardiovasc. Surg. 144 , 1453–1459 (2012).

Cabibihan, J.-J. & Chauhan, S. S. Physiological responses to affective tele-touch during induced emotional stimuli. IEEE Trans. Affect. Comput. 8 , 108–118 (2017).

Campeau, M.-P. et al. Impact of massage therapy on anxiety levels in patients undergoing radiation therapy: randomized controlled trial. J. Soc. Integr. Oncol. 5 , 133–138 (2007).

Can, Ş. & Kaya, H. The effects of yakson or gentle human touch training given to mothers with preterm babies on attachment levels and the responses of the baby: a randomized controlled trial. Health Care Women Int. 43 , 479–498 (2021).

Carfoot, S., Williamson, P. & Dickson, R. A randomised controlled trial in the north of England examining the effects of skin-to-skin care on breast feeding. Midwifery 21 , 71–79 (2005).

Castral, T. C., Warnock, F., Leite, A. M., Haas, V. J. & Scochi, C. G. S. The effects of skin-to-skin contact during acute pain in preterm newborns. Eur. J. Pain. 12 , 464–471 (2008).

Cattaneo, A. et al. Kangaroo mother care for low birthweight infants: a randomized controlled trial in different settings. Acta Paediatr. 87 , 976–985 (1998).

Charpak, N., Ruiz-Peláez, J. G. & Charpak, Y. Rey-Martinez kangaroo mother program: an alternative way of caring for low birth weight infants? One year mortality in a two cohort study. Pediatrics 94 , 804–810 (1994).

Chermont, A. G., Falcão, L. F. M., de Souza Silva, E. H. L., de Cássia Xavier Balda, R. & Guinsburg, R. Skin-to-skin contact and/or oral 25% dextrose for procedural pain relief for term newborn infants. Pediatrics 124 , e1101–e1107 (2009).

Chi Luong, K., Long Nguyen, T., Huynh Thi, D. H., Carrara, H. P. O. & Bergman, N. J. Newly born low birthweight infants stabilise better in skin-to-skin contact than when separated from their mothers: a randomised controlled trial. Acta Paediatr. 105 , 381–390 (2016).

Cho, E.-S. et al. The effects of kangaroo care in the neonatal intensive care unit on the physiological functions of preterm infants, maternal–infant attachment, and maternal stress. J. Pediatr. Nurs. 31 , 430–438 (2016).

Choi, H. et al. The effects of massage therapy on physical growth and gastrointestinal function in premature infants: a pilot study. J. Child Health Care 20 , 394–404 (2016).

Choudhary, M. et al. To study the effect of Kangaroo mother care on pain response in preterm neonates and to determine the behavioral and physiological responses to painful stimuli in preterm neonates: a study from western Rajasthan. J. Matern. Fetal Neonatal Med. 29 , 826–831 (2016).

Christensson, K. et al. Temperature, metabolic adaptation and crying in healthy full-term newborns cared for skin-to-skin or in a cot. Acta Paediatr. 81 , 488–493 (1992).

Cloutier, S. & Newberry, R. C. Use of a conditioning technique to reduce stress associated with repeated intra-peritoneal injections in laboratory rats. Appl. Anim. Behav. Sci. 112 , 158–173 (2008).

Cloutier, S., Wahl, K., Baker, C. & Newberry, R. C. The social buffering effect of playful handling on responses to repeated intraperitoneal injections in laboratory rats. J. Am. Assoc. Lab. Anim. Sci. 53 , 168–173 (2014).

CAS   PubMed   PubMed Central   Google Scholar  

Cloutier, S., Wahl, K. L., Panksepp, J. & Newberry, R. C. Playful handling of laboratory rats is more beneficial when applied before than after routine injections. Appl. Anim. Behav. Sci. 164 , 81–90 (2015).

Cong, X. et al. Effects of skin-to-skin contact on autonomic pain responses in preterm infants. J. Pain. 13 , 636–645 (2012).

Cong, X., Ludington-Hoe, S. M., McCain, G. & Fu, P. Kangaroo care modifies preterm infant heart rate variability in response to heel stick pain: pilot study. Early Hum. Dev. 85 , 561–567 (2009).

Cong, X., Ludington-Hoe, S. M. & Walsh, S. Randomized crossover trial of kangaroo care to reduce biobehavioral pain responses in preterm infants: a pilot study. Biol. Res. Nurs. 13 , 204–216 (2011).

Costa, R. et al. Tactile stimulation of adult rats modulates hormonal responses, depression-like behaviors, and memory impairment induced by chronic mild stress: role of angiotensin II. Behav. Brain Res. 379 , 112250 (2020).

Cutshall, S. M. et al. Effect of massage therapy on pain, anxiety, and tension in cardiac surgical patients: a pilot study. Complement. Ther. Clin. Pract. 16 , 92–95 (2010).

Dalili, H., Sheikhi, S., Shariat, M. & Haghnazarian, E. Effects of baby massage on neonatal jaundice in healthy Iranian infants: a pilot study. Infant Behav. Dev. 42 , 22–26 (2016).

Diego, M. A., Field, T. & Hernandez-Reif, M. Vagal activity, gastric motility, and weight gain in massaged preterm neonates. J. Pediatr. 147 , 50–55 (2005).

Diego, M. A., Field, T. & Hernandez-Reif, M. Temperature increases in preterm infants during massage therapy. Infant Behav. Dev. 31 , 149–152 (2008).

Diego, M. A. et al. Preterm infant massage elicits consistent increases in vagal activity and gastric motility that are associated with greater weight gain. Acta Paediatr. 96 , 1588–1591 (2007).

Diego, M. A. et al. Spinal cord patients benefit from massage therapy. Int. J. Neurosci. 112 , 133–142 (2002).

Diego, M. A. et al. Aggressive adolescents benefit from massage therapy. Adolescence 37 , 597–607 (2002).

Diego, M. A. et al. HIV adolescents show improved immune function following massage therapy. Int. J. Neurosci. 106 , 35–45 (2001).

Dieter, J. N. I., Field, T., Hernandez-Reif, M., Emory, E. K. & Redzepi, M. Stable preterm infants gain more weight and sleep less after five days of massage therapy. J. Pediatr. Psychol. 28 , 403–411 (2003).

Ditzen, B. et al. Effects of different kinds of couple interaction on cortisol and heart rate responses to stress in women. Psychoneuroendocrinology 32 , 565–574 (2007).

Dreisoerner, A. et al. Self-soothing touch and being hugged reduce cortisol responses to stress: a randomized controlled trial on stress, physical touch, and social identity. Compr. Psychoneuroendocrinol. 8 , 100091 (2021).

Eaton, M., Mitchell-Bonair, I. L. & Friedmann, E. The effect of touch on nutritional intake of chronic organic brain syndrome patients. J. Gerontol. 41 , 611–616 (1986).

Edens, J. L., Larkin, K. T. & Abel, J. L. The effect of social support and physical touch on cardiovascular reactions to mental stress. J. Psychosom. Res. 36 , 371–382 (1992).

El-Farrash, R. A. et al. Longer duration of kangaroo care improves neurobehavioral performance and feeding in preterm infants: a randomized controlled trial. Pediatr. Res. 87 , 683–688 (2020).

Erlandsson, K., Dsilna, A., Fagerberg, I. & Christensson, K. Skin-to-skin care with the father after cesarean birth and its effect on newborn crying and prefeeding behavior. Birth 34 , 105–114 (2007).

Escalona, A., Field, T., Singer-Strunck, R., Cullen, C. & Hartshorn, K. Brief report: improvements in the behavior of children with autism following massage therapy. J. Autism Dev. Disord. 31 , 513–516 (2001).

Fattah, M. A. & Hamdy, B. Pulmonary functions of children with asthma improve following massage therapy. J. Altern. Complement. Med. 17 , 1065–1068 (2011).

Feldman, R. & Eidelman, A. I. Skin-to-skin contact (kangaroo care) accelerates autonomic and neurobehavioural maturation in preterm infants. Dev. Med. Child Neurol. 45 , 274–281 (2003).

Feldman, R., Eidelman, A. I., Sirota, L. & Weller, A. Comparison of skin-to-skin (kangaroo) and traditional care: parenting outcomes and preterm infant development. Pediatrics 110 , 16–26 (2002).

Feldman, R., Singer, M. & Zagoory, O. Touch attenuates infants’ physiological reactivity to stress. Dev. Sci. 13 , 271–278 (2010).

Feldman, R., Weller, A., Sirota, L. & Eidelman, A. I. Testing a family intervention hypothesis: the contribution of mother–infant skin-to-skin contact (kangaroo care) to family interaction, proximity, and touch. J. Fam. Psychol. 17 , 94–107 (2003).

Ferber, S. G. et al. Massage therapy by mothers and trained professionals enhances weight gain in preterm infants. Early Hum. Dev. 67 , 37–45 (2002).

Ferber, S. G. & Makhoul, I. R. The effect of skin-to-skin contact (kangaroo care) shortly after birth on the neurobehavioral responses of the term newborn: a randomized, controlled trial. Pediatrics 113 , 858–865 (2004).

Ferreira, A. M. & Bergamasco, N. H. P. Behavioral analysis of preterm neonates included in a tactile and kinesthetic stimulation program during hospitalization. Rev. Bras. Fisioter. 14 , 141–148 (2010).

Fidanza, F., Polimeni, E., Pierangeli, V. & Martini, M. A better touch: C-tactile fibers related activity is associated to pain reduction during temporal summation of second pain. J. Pain. 22 , 567–576 (2021).

Field, T. et al. Leukemia immune changes following massage therapy. J. Bodyw. Mov. Ther. 5 , 271–274 (2001).

Field, T. et al. Benefits of combining massage therapy with group interpersonal psychotherapy in prenatally depressed women. J. Bodyw. Mov. Ther. 13 , 297–303 (2009).

Field, T., Delage, J. & Hernandez-Reif, M. Movement and massage therapy reduce fibromyalgia pain. J. Bodyw. Mov. Ther. 7 , 49–52 (2003).

Field, T. et al. Fibromyalgia pain and substance P decrease and sleep improves after massage therapy. J. Clin. Rheumatol. 8 , 72–76 (2002).

Field, T., Diego, M., Gonzalez, G. & Funk, C. G. Neck arthritis pain is reduced and range of motion is increased by massage therapy. Complement. Ther. Clin. Pract. 20 , 219–223 (2014).

Field, T., Diego, M., Hernandez-Reif, M., Deeds, O. & Figueiredo, B. Pregnancy massage reduces prematurity, low birthweight and postpartum depression. Infant Behav. Dev. 32 , 454–460 (2009).

Field, T. et al. Insulin and insulin-like growth factor-1 increased in preterm neonates following massage therapy. J. Dev. Behav. Pediatr. 29 , 463–466 (2008).

Field, T. et al. Yoga and massage therapy reduce prenatal depression and prematurity. J. Bodyw. Mov. Ther. 16 , 204–209 (2012).

Field, T., Diego, M., Hernandez-Reif, M., Schanberg, S. & Kuhn, C. Massage therapy effects on depressed pregnant women. J. Psychosom. Obstet. Gynecol. 25 , 115–122 (2004).

Field, T., Diego, M., Hernandez-Reif, M. & Shea, J. Hand arthritis pain is reduced by massage therapy. J. Bodyw. Mov. Ther. 11 , 21–24 (2007).

Field, T., Gonzalez, G., Diego, M. & Mindell, J. Mothers massaging their newborns with lotion versus no lotion enhances mothers’ and newborns’ sleep. Infant Behav. Dev. 45 , 31–37 (2016).

Field, T. et al. Children with asthma have improved pulmonary functions after massage therapy. J. Pediatr. 132 , 854–858 (1998).

Field, T., Hernandez-Reif, M., Diego, M. & Fraser, M. Lower back pain and sleep disturbance are reduced following massage therapy. J. Bodyw. Mov. Ther. 11 , 141–145 (2007).

Field, T. et al. Effects of sexual abuse are lessened by massage therapy. J. Bodyw. Mov. Ther. 1 , 65–69 (1997).

Field, T. et al. Pregnant women benefit from massage therapy. J. Psychosom. Obstet. Gynecol. 20 , 31–38 (1999).

Field, T. et al. Juvenilerheumatoid arthritis: benefits from massage therapy. J. Pediatr. Psychol. 22 , 607–617 (1997).

Field, T., Hernandez-Reif, M., Taylor, S., Quintino, O. & Burman, I. Labor pain is reduced by massage therapy. J. Psychosom. Obstet. Gynecol. 18 , 286–291 (1997).

Field, T. et al. Massage therapy reduces anxiety and enhances EEG pattern of alertness and math computations. Int. J. Neurosci. 86 , 197–205 (1996).

Field, T. et al. Brief report: autistic children’s attentiveness and responsivity improve after touch therapy. J. Autism Dev. Disord. 27 , 333–338 (1997).

Field, T. M. et al. Tactile/kinesthetic stimulation effects on preterm neonates. Pediatrics 77 , 654–658 (1986).

Field, T. et al. Massage reduces anxiety in child and adolescent psychiatric patients. J. Am. Acad. Child Adolesc. Psychiatry 31 , 125–131 (1992).

Field, T. et al. Burn injuries benefit from massage therapy. J. Burn Care Res. 19 , 241–244 (1998).

Filho, F. L. et al. Effect of maternal skin-to-skin contact on decolonization of methicillin-oxacillin-resistant Staphylococcus in neonatal intensive care units: a randomized controlled trial. BMC Pregnancy Childbirth https://doi.org/10.1186/s12884-015-0496-1 (2015).

Forward, J. B., Greuter, N. E., Crisall, S. J. & Lester, H. F. Effect of structured touch and guided imagery for pain and anxiety in elective joint replacement patients—a randomized controlled trial: M-TIJRP. Perm. J. 19 , 18–28 (2015).

Fraser, J. & Ross Kerr, J. Psychophysiological effects of back massage on elderly institutionalized patients. J. Adv. Nurs. 18 , 238–245 (1993).

Frey Law, L. A. et al. Massage reduces pain perception and hyperalgesia in experimental muscle pain: a randomized, controlled trial. J. Pain. 9 , 714–721 (2008).

Gao, H. et al. Effect of repeated kangaroo mother care on repeated procedural pain in preterm infants: a randomized controlled trial. Int. J. Nurs. Stud. 52 , 1157–1165 (2015).

Garner, B. et al. Pilot study evaluating the effect of massage therapy on stress, anxiety and aggression in a young adult psychiatric inpatient unit. Aust. N. Z. J. Psychiatry 42 , 414–422 (2008).

Gathwala, G., Singh, B. & Singh, J. Effect of kangaroo mother care on physical growth, breastfeeding and its acceptability. Trop. Dr. 40 , 199–202 (2010).

Geva, N., Uzefovsky, F. & Levy-Tzedek, S. Touching the social robot PARO reduces pain perception and salivary oxytocin levels. Sci. Rep. 10 , 9814 (2020).

Gitau, R. et al. Acute effects of maternal skin-to-skin contact and massage on saliva cortisol in preterm babies. J. Reprod. Infant Psychol. 20 , 83–88 (2002).

Givi, M. Durability of effect of massage therapy on blood pressure. Int. J. Prev. Med. 4 , 511–516 (2013).

PubMed   PubMed Central   Google Scholar  

Glover, V., Onozawa, K. & Hodgkinson, A. Benefits of infant massage for mothers with postnatal depression. Semin. Neonatol. 7 , 495–500 (2002).

Gonzalez, A. et al. Weight gain in preterm infants following parent-administered vimala massage: a randomized controlled trial. Am. J. Perinatol. 26 , 247–252 (2009).

Gray, L., Watt, L. & Blass, E. M. Skin-to-skin contact is analgesic in healthy newborns. Pediatrics 105 , e14 (2000).

Grewen, K. M., Anderson, B. J., Girdler, S. S. & Light, K. C. Warm partner contact is related to lower cardiovascular reactivity. Behav. Med. 29 , 123–130 (2003).

Groër, M. W., Hill, J., Wilkinson, J. E. & Stuart, A. Effects of separation and separation with supplemental stroking in BALB/c infant mice. Biol. Res. Nurs. 3 , 119–131 (2002).

Gürol, A. P., Polat, S. & Nuran Akçay, M. Itching, pain, and anxiety levels are reduced with massage therapy in burned adolescents. J. Burn Care Res. 31 , 429–432 (2010).

Haley, S. et al. Tactile/kinesthetic stimulation (TKS) increases tibial speed of sound and urinary osteocalcin (U-MidOC and unOC) in premature infants (29–32 weeks PMA). Bone 51 , 661–666 (2012).

Harris, M., Richards, K. C. & Grando, V. T. The effects of slow-stroke back massage on minutes of nighttime sleep in persons with dementia and sleep disturbances in the nursing home: a pilot study. J. Holist. Nurs. 30 , 255–263 (2012).

Hart, S. et al. Anorexia nervosa symptoms are reduced by massage therapy. Eat. Disord. 9 , 289–299 (2001).

Hattan, J., King, L. & Griffiths, P. The impact of foot massage and guided relaxation following cardiac surgery: a randomized controlled trial. Issues Innov. Nurs. Pract. 37 , 199–207 (2002).

Haynes, A. C. et al. A calming hug: design and validation of a tactile aid to ease anxiety. PLoS ONE 17 , e0259838 (2022).

Henricson, M., Ersson, A., Määttä, S., Segesten, K. & Berglund, A.-L. The outcome of tactile touch on stress parameters in intensive care: a randomized controlled trial. Complement. Ther. Clin. Pract. 14 , 244–254 (2008).

Hernandez-Reif, M., Diego, M. & Field, T. Preterm infants show reduced stress behaviors and activity after 5 days of massage therapy. Infant Behav. Dev. 30 , 557–561 (2007).

Hernandez-Reif, M., Dieter, J. N. I., Field, T., Swerdlow, B. & Diego, M. Migraine headaches are reduced by massage therapy. Int. J. Neurosci. 96 , 1–11 (1998).

Hernandez-Reif, M. et al. Natural killer cells and lymphocytes increase in women with breast cancer following massage therapy. Int. J. Neurosci. 115 , 495–510 (2005).

Hernandez-Reif, M. et al. Children with cystic fibrosis benefit from massage therapy. J. Pediatr. Psychol. 24 , 175–181 (1999).

Hernandez-Reif, M., Field, T., Krasnegor, J. & Theakston, H. Lower back pain is reduced and range of motion increased after massage therapy. Int. J. Neurosci. 106 , 131–145 (2001).

Hernandez-Reif, M. et al. High blood pressure and associated symptoms were reduced by massage therapy. J. Bodyw. Mov. Ther. 4 , 31–38 (2000).

Hernandez-Reif, M. et al. Parkinson’s disease symptoms are differentially affected by massage therapy vs. progressive muscle relaxation: a pilot study. J. Bodyw. Mov. Ther. 6 , 177–182 (2002).

Hernandez-Reif, M., Field, T. & Theakston, H. Multiple sclerosis patients benefit from massage therapy. J. Bodyw. Mov. Ther. 2 , 168–174 (1998).

Hernandez-Reif, M. et al. Breast cancer patients have improved immune and neuroendocrine functions following massage therapy. J. Psychosom. Res. 57 , 45–52 (2004).

Hertenstein, M. J. & Campos, J. J. Emotion regulation via maternal touch. Infancy 2 , 549–566 (2001).

Hinchcliffe, J. K., Mendl, M. & Robinson, E. S. J. Rat 50 kHz calls reflect graded tickling-induced positive emotion. Curr. Biol. 30 , R1034–R1035 (2020).

Hodgson, N. A. & Andersen, S. The clinical efficacy of reflexology in nursing home residents with dementia. J. Altern. Complement. Med. 14 , 269–275 (2008).

Hoffmann, L. & Krämer, N. C. The persuasive power of robot touch. Behavioral and evaluative consequences of non-functional touch from a robot. PLoS ONE 16 , e0249554 (2021).

Holst, S., Lund, I., Petersson, M. & Uvnäs-Moberg, K. Massage-like stroking influences plasma levels of gastrointestinal hormones, including insulin, and increases weight gain in male rats. Auton. Neurosci. 120 , 73–79 (2005).

Hori, M. et al. Tickling during adolescence alters fear-related and cognitive behaviors in rats after prolonged isolation. Physiol. Behav. 131 , 62–67 (2014).

Hori, M. et al. Effects of repeated tickling on conditioned fear and hormonal responses in socially isolated rats. Neurosci. Lett. 536 , 85–89 (2013).

Hucklenbruch-Rother, E. et al. Delivery room skin-to-skin contact in preterm infants affects long-term expression of stress response genes. Psychoneuroendocrinology 122 , 104883 (2020).

Im, H. & Kim, E. Effect of yakson and gentle human touch versus usual care on urine stress hormones and behaviors in preterm infants: a quasi-experimental study. Int. J. Nurs. Stud. 46 , 450–458 (2009).

Jain, S., Kumar, P. & McMillan, D. D. Prior leg massage decreases pain responses to heel stick in preterm babies. J. Paediatr. Child Health 42 , 505–508 (2006).

Jane, S.-W. et al. Effects of massage on pain, mood status, relaxation, and sleep in Taiwanese patients with metastatic bone pain: a randomized clinical trial. Pain 152 , 2432–2442 (2011).

Johnston, C. C. et al. Kangaroo mother care diminishes pain from heel lance in very preterm neonates: a crossover trial. BMC Pediatr. 8 , 13 (2008).

Johnston, C. C. et al. Kangaroo care is effective in diminishing pain response in preterm neonates. Arch. Pediatr. Adolesc. Med. 157 , 1084–1088 (2003).

Jung, M. J., Shin, B.-C., Kim, Y.-S., Shin, Y.-I. & Lee, M. S. Is there any difference in the effects of QI therapy (external QIGONG) with and without touching? a pilot study. Int. J. Neurosci. 116 , 1055–1064 (2006).

Kapoor, Y. & Orr, R. Effect of therapeutic massage on pain in patients with dementia. Dementia 16 , 119–125 (2017).

Karagozoglu, S. & Kahve, E. Effects of back massage on chemotherapy-related fatigue and anxiety: supportive care and therapeutic touch in cancer nursing. Appl. Nurs. Res. 26 , 210–217 (2013).

Karbasi, S. A., Golestan, M., Fallah, R., Golshan, M. & Dehghan, Z. Effect of body massage on increase of low birth weight neonates growth parameters: a randomized clinical trial. Iran. J. Reprod. Med. 11 , 583–588 (2013).

Kashaninia, Z., Sajedi, F., Rahgozar, M. & Noghabi, F. A. The effect of kangaroo care on behavioral responses to pain of an intramuscular injection in neonates . J. Pediatr. Nurs. 3 , 275–280 (2008).

Kelling, C., Pitaro, D. & Rantala, J. Good vibes: The impact of haptic patterns on stress levels. In Proc. 20th International Academic Mindtrek Conference 130–136 (Association for Computing Machinery, 2016).

Khilnani, S., Field, T., Hernandez-Reif, M. & Schanberg, S. Massage therapy improves mood and behavior of students with attention-deficit/hyperactivity disorder. Adolescence 38 , 623–638 (2003).

Kianmehr, M. et al. The effect of massage on serum bilirubin levels in term neonates with hyperbilirubinemia undergoing phototherapy. Nautilus 128 , 36–41 (2014).

Kim, I.-H., Kim, T.-Y. & Ko, Y.-W. The effect of a scalp massage on stress hormone, blood pressure, and heart rate of healthy female. J. Phys. Ther. Sci. 28 , 2703–2707 (2016).

Kim, M. A., Kim, S.-J. & Cho, H. Effects of tactile stimulation by fathers on physiological responses and paternal attachment in infants in the NICU: a pilot study. J. Child Health Care 21 , 36–45 (2017).

Kim, M. S., Sook Cho, K., Woo, H.-M. & Kim, J. H. Effects of hand massage on anxiety in cataract surgery using local anesthesia. J. Cataract Refr. Surg. 27 , 884–890 (2001).

Koole, S. L., Tjew A Sin, M. & Schneider, I. K. Embodied terror management: interpersonal touch alleviates existential concerns among individuals with low self-esteem. Psychol. Sci. 25 , 30–37 (2014).

Krohn, M. et al. Depression, mood, stress, and Th1/Th2 immune balance in primary breast cancer patients undergoing classical massage therapy. Support. Care Cancer 19 , 1303–1311 (2011).

Kuhn, C. et al. Tactile-kinesthetic stimulation effects sympathetic and adrenocortical function in preterm infants. J. Pediatr. 119 , 434–440 (1991).

Kumar, J. et al. Effect of oil massage on growth in preterm neonates less than 1800 g: a randomized control trial. Indian J. Pediatr. 80 , 465–469 (2013).

Lee, H.-K. The effects of infant massage on weight, height, and mother–infant interaction. J. Korean Acad. Nurs. 36 , 1331–1339 (2006).

Leivadi, S. et al. Massage therapy and relaxation effects on university dance students. J. Dance Med. Sci. 3 , 108–112 (1999).

Lindgren, L. et al. Touch massage: a pilot study of a complex intervention. Nurs. Crit. Care 18 , 269–277 (2013).

Lindgren, L. et al. Physiological responses to touch massage in healthy volunteers. Auton. Neurosci. Basic Clin. 158 , 105–110 (2010).

Listing, M. et al. Massage therapy reduces physical discomfort and improves mood disturbances in women with breast cancer. Psycho-Oncol. 18 , 1290–1299 (2009).

Ludington-Hoe, S. M., Cranston Anderson, G., Swinth, J. Y., Thompson, C. & Hadeed, A. J. Randomized controlled trial of kangaroo care: cardiorespiratory and thermal effects on healthy preterm infants. Neonatal Netw. 23 , 39–48 (2004).

Lund, I. et al. Corticotropin releasing factor in urine—a possible biochemical marker of fibromyalgia. Neurosci. Lett. 403 , 166–171 (2006).

Ma, Y.-K. et al. Lack of social touch alters anxiety-like and social behaviors in male mice. Stress 25 , 134–144 (2022).

Massaro, A. N., Hammad, T. A., Jazzo, B. & Aly, H. Massage with kinesthetic stimulation improves weight gain in preterm infants. J. Perinatol. 29 , 352–357 (2009).

Mathai, S., Fernandez, A., Mondkar, J. & Kanbur, W. Effects of tactile-kinesthetic stimulation in preterms–a controlled trial. Indian Pediatr. 38 , 1091–1098 (2001).

CAS   PubMed   Google Scholar  

Matsunaga, M. et al. Profiling of serum proteins influenced by warm partner contact in healthy couples. Neuroenocrinol. Lett. 30 , 227–236 (2009).

CAS   Google Scholar  

Mendes, E. W. & Procianoy, R. S. Massage therapy reduces hospital stay and occurrence of late-onset sepsis in very preterm neonates. J. Perinatol. 28 , 815–820 (2008).

Mirnia, K., Arshadi Bostanabad, M., Asadollahi, M. & Hamid Razzaghi, M. Paternal skin-to-skin care and its effect on cortisol levels of the infants. Iran. J. Pediatrics 27 , e8151 (2017).

Mitchell, A. J., Yates, C., Williams, K. & Hall, R. W. Effects of daily kangaroo care on cardiorespiratory parameters in preterm infants. J. Neonatal-Perinat. Med. 6 , 243–249 (2013).

Mitchinson, A. R. et al. Acute postoperative pain management using massage as an adjuvant therapy: a randomized trial. Arch. Surg. 142 , 1158–1167 (2007).

Modrcin-Talbott, M. A., Harrison, L. L., Groer, M. W. & Younger, M. S. The biobehavioral effects of gentle human touch on preterm infants. Nurs. Sci. Q. 16 , 60–67 (2003).

Mok, E. & Pang Woo, C. The effects of slow-stroke back massage on anxiety and shoulder pain in elderly stroke patients. Complement. Ther. Nurs. Midwifery 10 , 209–216 (2004).

Mokaberian, M., Noripour, S., Sheikh, M. & Mills, P. J. Examining the effectiveness of body massage on physical status of premature neonates and their mothers’ psychological status. Early Child Dev. Care 192 , 2311–2325 (2021).

Mori, H. et al. Effect of massage on blood flow and muscle fatigue following isometric lumbar exercise. Med. Sci. Monit. Int. Med. J. Exp. Clin. Res. 10 , CR173–CR178 (2004).

Moyer-Mileur, L. J., Haley, S., Slater, H., Beachy, J. & Smith, S. L. Massage improves growth quality by decreasing body fat deposition in male preterm infants. J. Pediatr. 162 , 490–495 (2013).

Moyle, W. et al. Foot massage and physiological stress in people with dementia: a randomized controlled trial. J. Altern. Complement. Med. 20 , 305–311 (2014).

Muntsant, A., Shrivastava, K., Recasens, M. & Giménez-Llort, L. Severe perinatal hypoxic-ischemic brain injury induces long-term sensorimotor deficits, anxiety-like behaviors and cognitive impairment in a sex-, age- and task-selective manner in C57BL/6 mice but can be modulated by neonatal handling. Front. Behav. Neurosci. 13 , 7 (2019).

Negahban, H., Rezaie, S. & Goharpey, S. Massage therapy and exercise therapy in patients with multiple sclerosis: a randomized controlled pilot study. Clin. Rehabil. 27 , 1126–1136 (2013).

Nelson, D., Heitman, R. & Jennings, C. Effects of tactile stimulation on premature infant weight gain. J. Obstet. Gynecol. Neonatal Nurs. 15 , 262–267 (1986).

Griffin, J. W. Calculating statistical power for meta-analysis using metapower. Quant. Meth. Psychol . 17 , 24–39 (2021).

Nunes, G. S. et al. Massage therapy decreases pain and perceived fatigue after long-distance Ironman triathlon: a randomised trial. J. Physiother. 62 , 83–87 (2016).

Ohgi, S. et al. Comparison of kangaroo care and standard care: behavioral organization, development, and temperament in healthy, low-birth-weight infants through 1 year. J. Perinatol. 22 , 374–379 (2002).

O′Higgins, M., St. James Roberts, I. & Glover, V. Postnatal depression and mother and infant outcomes after infant massage. J. Affect. Disord. 109 , 189–192 (2008).

Okan, F., Ozdil, A., Bulbul, A., Yapici, Z. & Nuhoglu, A. Analgesic effects of skin-to-skin contact and breastfeeding in procedural pain in healthy term neonates. Ann. Trop. Paediatr. 30 , 119–128 (2010).

Oliveira, D. S., Hachul, H., Goto, V., Tufik, S. & Bittencourt, L. R. A. Effect of therapeutic massage on insomnia and climacteric symptoms in postmenopausal women. Climacteric 15 , 21–29 (2012).

Olsson, E., Ahlsén, G. & Eriksson, M. Skin-to-skin contact reduces near-infrared spectroscopy pain responses in premature infants during blood sampling. Acta Paediatr. 105 , 376–380 (2016).

Pauk, J., Kuhn, C. M., Field, T. M. & Schanberg, S. M. Positive effects of tactile versus kinesthetic or vestibular stimulation on neuroendocrine and ODC activity in maternally-deprived rat pups. Life Sci. 39 , 2081–2087 (1986).

Pinazo, D., Arahuete, L. & Correas, N. Hugging as a buffer against distal fear of death. Calid. Vida Salud 13 , 11–20 (2020).

Pope, M. H. et al. A prospective randomized three-week trial of spinal manipulation, transcutaneous muscle stimulation, massage and corset in the treatment of subacute low back pain. Spine 19 , 2571–2577 (1994).

Preyde, M. Effectiveness of massage therapy for subacute low-back pain: a randomized controlled trial. Can. Med. Assoc. J. 162 , 1815–1820 (2000).

Ramanathan, K., Paul, V. K., Deorari, A. K., Taneja, U. & George, G. Kangaroo mother care in very low birth weight infants. Indian J. Pediatr. 68 , 1019–1023 (2001).

Reddan, M. C., Young, H., Falkner, J., López-Solà, M. & Wager, T. D. Touch and social support influence interpersonal synchrony and pain. Soc. Cogn. Affect. Neurosci. 15 , 1064–1075 (2020).

Rodríguez-Mansilla, J. et al. The effects of ear acupressure, massage therapy and no therapy on symptoms of dementia: a randomized controlled trial. Clin. Rehabil. 29 , 683–693 (2015).

Rose, S. A., Schmidt, K., Riese, M. L. & Bridger, W. H. Effects of prematurity and early intervention on responsivity to tactual stimuli: a comparison of preterm and full-term infants. Child Dev. 51 , 416–425 (1980).

Scafidi, F. A. et al. Massage stimulates growth in preterm infants: a replication. Infant Behav. Dev. 13 , 167–188 (1990).

Scafidi, F. A. et al. Effects of tactile/kinesthetic stimulation on the clinical course and sleep/wake behavior of preterm neonates. Infant Behav. Dev. 9 , 91–105 (1986).

Scafidi, F. & Field, T. Massage therapy improves behavior in neonates born to HIV-positive mothers. J. Pediatr. Psychol. 21 , 889–897 (1996).

Scarr-Salapatek, S. & Williams, M. L. A stimulation program for low birth weight infants. Am. J. Public Health 62 , 662–667 (1972).

Serrano, B., Baños, R. M. & Botella, C. Virtual reality and stimulation of touch and smell for inducing relaxation: a randomized controlled trial. Comput. Hum. Behav. 55 , 1–8 (2016).

Seyyedrasooli, A., Valizadeh, L., Hosseini, M. B., Asgari Jafarabadi, M. & Mohammadzad, M. Effect of vimala massage on physiological jaundice in infants: a randomized controlled trial. J. Caring Sci. 3 , 165–173 (2014).

Sharpe, P. A., Williams, H. G., Granner, M. L. & Hussey, J. R. A randomised study of the effects of massage therapy compared to guided relaxation on well-being and stress perception among older adults. Complement. Therap. Med. 15 , 157–163 (2007).

Sherman, K. J., Cherkin, D. C., Hawkes, R. J., Miglioretti, D. L. & Deyo, R. A. Randomized trial of therapeutic massage for chronic neck pain. Clin. J. Pain. 25 , 233–238 (2009).

Shiloh, S., Sorek, G. & Terkel, J. Reduction of state-anxiety by petting animals in a controlled laboratory experiment. Anxiety, Stress Coping 16 , 387–395 (2003).

Shor-Posner, G. et al. Impact of a massage therapy clinical trial on immune status in young Dominican children infected with HIV-1. J. Altern. Complement. Med. 12 , 511–516 (2006).

Simpson, E. A. et al. Social touch alters newborn monkey behavior. Infant Behav. Dev. 57 , 101368 (2019).

Smith, S. L., Haley, S., Slater, H. & Moyer-Mileur, L. J. Heart rate variability during caregiving and sleep after massage therapy in preterm infants. Early Hum. Dev. 89 , 525–529 (2013).

Smith, S. L. et al. The effect of massage on heart rate variability in preterm infants. J. Perinatol. 33 , 59–64 (2013).

Solkoff, N. & Matuszak, D. Tactile stimulation and behavioral development among low-birthweight infants. Child Psychiatry Hum. Dev. 6 , 3337 (1975).

Srivastava, S., Gupta, A., Bhatnagar, A. & Dutta, S. Effect of very early skin to skin contact on success at breastfeeding and preventing early hypothermia in neonates. Indian J. Public Health 58 , 22–26 (2014).

Stringer, J., Swindell, R. & Dennis, M. Massage in patients undergoing intensive chemotherapy reduces serum cortisol and prolactin: massage in oncology patients reduces serum cortisol. Psycho-Oncol. 17 , 1024–1031 (2008).

Suman Rao, P. N., Udani, R. & Nanavati, R. Kangaroo mother care for low birth weight infants: a randomized controlled trial. Indian Pediatr. 45 , 17–23 (2008).

Sumioka, H. et al. A huggable device can reduce the stress of calling an unfamiliar person on the phone for individuals with ASD. PLoS ONE 16 , e0254675 (2021).

Sumioka, H., Nakae, A., Kanai, R. & Ishiguro, H. Huggable communication medium decreases cortisol levels. Sci. Rep. 3 , 3034 (2013).

Suzuki, M. et al. Physical and psychological effects of 6-week tactile massage on elderly patients with severe dementia. Am. J. Alzheimer’s Dis. Other Dement. 25 , 680–686 (2010).

Thomson, L. J. M., Ander, E. E., Menon, U., Lanceley, A. & Chatterjee, H. J. Quantitative evidence for wellbeing benefits from a heritage-in-health intervention with hospital patients. Int. J. Art. Ther. 17 , 63–79 (2012).

Triplett, J. L. & Arneson, S. W. The use of verbal and tactile comfort to alleviate distress in young hospitalized children. Res. Nurs. Health 2 , 17–23 (1979).

Walach, H., Güthlin, C. & König, M. Efficacy of massage therapy in chronic pain: a pragmatic randomized trial. J. Altern. Complement. Med. 9 , 837–846 (2003).

Walker, S. C. et al. C‐low threshold mechanoafferent targeted dynamic touch modulates stress resilience in rats exposed to chronic mild stress. Eur. J. Neurosci. 55 , 2925–2938 (2022).

Weinrich, S. P. & Weinrich, M. C. The effect of massage on pain in cancer patients. Appl. Nurs. Res. 3 , 140–145 (1990).

Wheeden, A. et al. Massage effects on cocaine-exposed preterm neonates. Dev. Behav. Pediatr. 14 , 318–322 (1993).

White, J. L. & Labarba, R. C. The effects of tactile and kinesthetic stimulation on neonatal development in the premature infant. Dev. Psychobiol. 9 , 569–577 (1976).

Wilkie, D. J. et al. Effects of massage on pain intensity, analgesics and quality of life in patients with cancer pain: a pilot study of a randomized clinical trial conducted within hospice care delivery. Hosp. J. 15 , 31–53 (2000).

Willemse, C. J. A. M., Toet, A. & van Erp, J. B. F. Affective and behavioral responses to robot-initiated social touch: toward understanding the opportunities and limitations of physical contact in human–robot interaction. Front. ICT 4 , 12 (2017).

Willemse, C. J. A. M. & van Erp, J. B. F. Social touch in human–robot interaction: robot-initiated touches can induce positive responses without extensive prior bonding. Int. J. Soc. Robot. 11 , 285–304 (2019).

Woods, D. L., Beck, C. & Sinha, K. The effect of therapeutic touch on behavioral symptoms and cortisol in persons with dementia. Res. Complement. Med. 16 , 181–189 (2009).

Yamaguchi, M., Sekine, T. & Shetty, V. A salivary cytokine panel discriminates moods states following a touch massage intervention. Int. J. Affect. Eng. 19 , 189–198 (2020).

Yamazaki, R. et al. Intimacy in phone conversations: anxiety reduction for Danish seniors with hugvie. Front. Psychol. 7 , 537 (2016).

Yang, M.-H. et al. Comparison of the efficacy of aroma-acupressure and aromatherapy for the treatment of dementia-associated agitation. BMC Complement. Altern. Med. 15 , 93 (2015).

Yates, C. C. et al. The effects of massage therapy to induce sleep in infants born preterm. Pediatr. Phys. Ther. 26 , 405–410 (2014).

Yu, H. et al. Social touch-like tactile stimulation activates a tachykinin 1-oxytocin pathway to promote social interactions. Neuron 110 , 1051–1067 (2022).

Lakens, D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t -tests and ANOVAs. Front. Psychol. 4 , 863 (2013).

Page, M. J., et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst. Rev. https://doi.org/10.1186/s13643-021-01626-4 (2021).

Wilson, D. B. Practical meta-analysis effect size calculator (Version 2023.11.27). https://campbellcollaboration.org/research-resources/effect-size-calculator.html (2023).

Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw https://doi.org/10.18637/jss.v036.i03 (2010).

Scammacca, N., Roberts, G. & Stuebing, K. K. Meta-analysis with complex research designs: dealing with dependence from multiple measures and multiple group comparisons. Rev. Educ. Res. 84 , 328–364 (2014).

Pustejovsky, J. E. & Tipton, E. Meta-analysis with robust variance estimation: expanding the range of working models. Prev. Sci. Off. J. Soc. Prev. Res. 23 , 425–438 (2022).

Cook, R. D. in International Encyclopedia of Statistical Science (ed. M. Lovric) S. 301–302 (Springer, 2011).

Higgins, J. P. T., Thompson, S. & Deeks, J. Measuring inconsistency in meta-analyses. BMJ https://doi.org/10.1136/bmj.327.7414.557 (2003).

Oberauer, K. The importance of random slopes in mixed models for Bayesian hypothesis testing. Psychol. Sci. 33 , 648–665 (2022).

Nakagawa, S. et al. The orchard plot: cultivating a forest plot for use in ecology, evolution, and beyond. Res. Synth. Methods 12 , 4–12 (2021).

Download references

Acknowledgements

We thank A. Frick and E. Chris for supporting the initial literature search and coding. We also thank A. Dreisoerner, T. Field, S. Koole, C. Kuhn, M. Henricson, L. Frey Law, J. Fraser, M. Cumella Reddan, and J. Stringer, who kindly responded to our data requests and provided additional information or data with respect to single studies. J.P. was supported by the German National Academy of Sciences Leopoldina (LPDS 2021-05). H.H. was supported by the Marietta-Blau scholarship of the Austrian Agency for Education and Internationalisation (OeAD) and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation, project ID 422744262 – TRR 289). C.K. received funding from OCENW.XL21.XL21.069 and V.G. from the European Research Council (ERC) under European Union’s Horizon 2020 research and innovation programme, grant ‘HelpUS’ (758703) and from the Dutch Research Council (NWO) grant OCENW.XL21.XL21.069. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Open access funding provided by Ruhr-Universität Bochum.

Author information

Julian Packheiser

Present address: Social Neuroscience, Faculty of Medicine, Ruhr University Bochum, Bochum, Germany

These authors contributed equally: Julian Packheiser, Helena Hartmann.

Authors and Affiliations

Social Brain Lab, Netherlands Institute for Neuroscience, Royal Netherlands Academy of Art and Sciences, Amsterdam, the Netherlands

Julian Packheiser, Helena Hartmann, Kelly Fredriksen, Valeria Gazzola, Christian Keysers & Frédéric Michon

Center for Translational and Behavioral Neuroscience, University Hospital Essen, Essen, Germany

Helena Hartmann

Clinical Neurosciences, Department for Neurology, University Hospital Essen, Essen, Germany

You can also search for this author in PubMed   Google Scholar

Contributions

J.P. contributed to conceptualization, methodology, formal analysis, investigation, data curation, writing the original draft, review and editing, visualization, supervision and project administration. HH contributed to conceptualization, methodology, formal analysis, investigation, data curation, writing the original draft, review and editing, visualization, supervision and project administration. K.F. contributed to investigation, data curation, and review and editing. C.K. and V.G. contributed to conceptualization, and review and editing. F.M. contributed to conceptualization, methodology, formal analysis, investigation, writing the original draft, and review and editing.

Corresponding author

Correspondence to Julian Packheiser .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Human Behaviour thanks Ville Harjunen, Rebecca Boehme and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary Figs. 1–21 and Tables 1–4.

Reporting Summary

Peer review file, supplementary table 1.

List of studies included in and excluded from the meta-analyses/review.

Supplementary Table 2

PRISMA checklist, manuscript.

Supplementary Table 3

PRISMA checklist, abstract.

Source Data Fig. 2

Effect size/error (columns ‘Hedges_g’ and ‘variance’) information for each study/cohort/effect included in the analysis. Source Data Fig. 3 Effect size/error (columns ‘Hedges_g’ and ‘variance’) together with moderator data (column ‘Outcome’) for each study/cohort/effect included in the analysis. Source Data Fig. 4 Effect size/error (columns ‘Hedges_g’ and ‘variance’) together with moderator data (columns ‘dyad_type’ and ‘skin_to_skin’) for each study/cohort/effect included in the analysis. Source Data Fig. 5 Effect size/error (columns ‘Hedges_g’ and ‘variance’) together with moderator data (column ‘touch_type’) for each study/cohort/effect included in the analysis. Source Data Fig. 6 Effect size/error (columns ‘Hedges_g’ and ‘variance’) together with moderator data (column ‘clin_sample’) for each study/cohort/effect included in the analysis. Source Data Fig. 7 Effect size/error (columns ‘Hedges_g’ and ‘variance’) together with moderator data (column ‘familiarity’) for each study/cohort/effect included in the analysis. Source Data Fig. 7 Effect size/error (columns ‘Hedges_g’ and ‘variance’) together with moderator data (columns ‘touch_duration’ and ‘sessions’) for each study/cohort/effect included in the analysis.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Packheiser, J., Hartmann, H., Fredriksen, K. et al. A systematic review and multivariate meta-analysis of the physical and mental health benefits of touch interventions. Nat Hum Behav (2024). https://doi.org/10.1038/s41562-024-01841-8

Download citation

Received : 16 August 2023

Accepted : 29 January 2024

Published : 08 April 2024

DOI : https://doi.org/10.1038/s41562-024-01841-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Open access
• Published: 06 December 2022

What improves access to primary healthcare services in rural communities? A systematic review

• Zemichael Gizaw 1 ,
• Tigist Astale 2 &
• Getnet Mitike Kassie 2  

BMC Primary Care volume  23 , Article number:  313 ( 2022 ) Cite this article

13k Accesses

9 Citations

1 Altmetric

Metrics details

To compile key strategies from the international experiences to improve access to primary healthcare (PHC) services in rural communities. Different innovative approaches have been practiced in different parts of the world to improve access to essential healthcare services in rural communities. Systematically collecting and combining best experiences all over the world is important to suggest effective strategies to improve access to healthcare in developing countries. Accordingly, this systematic review of literature was undertaken to identify key approaches from international experiences to enhance access to PHC services in rural communities.

All published and unpublished qualitative and/or mixed method studies conducted to improvement access to PHC services were searched from MEDLINE, Scopus, Web of Science, WHO Global Health Library, and Google Scholar. Articles published other than English language, citations with no abstracts and/or full texts, and duplicate studies were excluded. We included all articles available in different electronic databases regardless of their publication years. We assessed the methodological quality of the included studies using mixed methods appraisal tool (MMAT) version 2018 to minimize the risk of bias. Data were extracted using JBI mixed methods data extraction form. Data were qualitatively analyzed using emergent thematic analysis approach to identify key concepts and coded them into related non-mutually exclusive themes.

Our analysis of 110 full-text articles resulted in ten key strategies to improve access to PHC services. Community health programs or community-directed interventions, school-based healthcare services, student-led healthcare services, outreach services or mobile clinics, family health program, empanelment, community health funding schemes, telemedicine, working with traditional healers, working with non-profit private sectors and non-governmental organizations including faith-based organizations are the key strategies identified from international experiences.

This review identified key strategies from international experiences to improve access to PHC services in rural communities. These strategies can play roles in achieving universal health coverage and reducing disparities in health outcomes among rural communities and enabling them to get healthcare when and where they want.

Peer Review reports

Introduction

Universal health coverage (UHC) is used to provide expanding services to eliminate access barriers. Universal health coverage is defined by the world health organization (WHO) as access to key promotional, preventive, curative and rehabilitative health services for all at an affordable rate and ensuring equity in access. The term universal has been described as the State's legal obligation to provide healthcare to all its citizens, with particular attention to ensuring that all poor and excluded groups are included [ 1 , 2 , 3 ].

Strengthening primary healthcare (PHC) is the most comprehensive, reliable and productive approach to improving people's physical and mental wellbeing and social well-being, and that PHC is a pillar of a sustainable health system for UHC and health-related sustainable development goals [ 4 , 5 ]. Despite tremendous progress over the last decades, there are still unaddressed health needs of people in all parts of the world [ 6 , 7 ]. Many people, particularly the poor and people living in rural areas and those who are in vulnerable circumstances, face challenges to remain healthy [ 8 ].

Geographical and financial inaccessibility, inadequate funding, inconsistent medication supply and equipment and personnel shortages have left the reach, availability and effect of PHC services in many countries disappointingly limited [ 9 , 10 ]. A recent Astana Declaration recognized those aspects of PHC need to be changed to adapt adequately to current and emerging threats to the healthcare system. This declaration discussed that implementation of a need-based, comprehensive, cost-effective, accessible, efficient and sustainable healthcare system is needed for disadvantaged and rural populations in more local and convenient settings to provide care when and where they want it [ 8 ].

Different innovative approaches have been practiced in different parts of the world to improve access to essential healthcare services in rural communities. Systematically collecting and combining best experiences all over the world is important to suggest effective strategies to improve access to healthcare in developing countries. Accordingly, this systematic review of literature was undertaken to identify key approaches from international experiences to enhance access to PHC services in rural communities. The findings of this systematic literature review can be used by healthcare professionals, researchers and policy makers to improve healthcare service delivery in rural communities.

Methodology

Research question.

What improves access to PHC services in rural communities? We used the PICO (population, issue/intervention, comparison/contrast, and outcome) construct to develop the search question [ 11 ]. The population is rural communities or remote communities in developing countries who have limited access to healthcare services. Moreover, we extended the population to developed countries to capture experiences of both developing and developed countries. The issue/intervention is implementation of different community-based health interventions to access to essential healthcare services. In this systematic review, we focused on PHC health services, mainly essential or basic healthcare services, community or public health services, and health promotion or health education. Primary healthcare is “a health care system that addressed social, economic, and political causes of poor health promotes health though health services at the primary care level enhances health of the community” [ 12 ]. Comparison/contrast is not appropriate for this review. The outcome is improved access to essential healthcare services.

Outcome measures

The outcome of this review is access to PHC services, such as preventive, promotive, curative, rehabilitative, and palliative health services which are affordable, convenient or acceptable, and available to all who need care.

Criteria for considering studies for this review

All published and unpublished qualitative and/or mixed method studies conducted to improve access to PHC services were included. Government and international or national organizations reports were also included. Different organizations whose primary mission is health or promotion of community health were selected. We included articles based on these eligibility criteria: context or scope of studies (access to PHC services), article type (primary studies), and publication language (English). Articles published other than English language, citations with no abstracts and/or full texts, reviews, and duplicate studies were excluded. We included all articles available in different electronic databases regardless of their publication years. We didn’t use time of publication for screening.

Information sources and search strategy

We searched relevant articles from MEDLINE, Scopus, Web of Science, WHO Global Health Library, and Google Scholar to access all forms of evidence. An initial search of MEDLINE was undertaken followed by analysis of the text words contained in the title and abstract, and of the index terms used to describe articles. We used the aforementioned performance indicators of PHC delivery and the PICO as we described above to choose keywords. A second search using all identified keywords and index terms was undertaken across all included databases. Thirdly, references of all identified articles were searched to get additional studies. The full electronic search strategy for MEDLINE, a major database we used for this review is included as a supplementary file (Additional file 1 : Appendix 1).

Study selection and assessment of methodological quality

Search results from different electronic databases were exported to Endnote reference manager version 7 to remove duplication. Two independent reviewers (ZG and BA) screened out records. An initial screening of titles and abstracts was done based on the PICO criteria and language of publication. Secondary screening of full-text papers was done for studies we included at the initial screening phase. We further investigated and assessed records included in the full-text articles against the inclusion and exclusion criteria. We sat together and discussed the eligibility assessment. The interrater agreement was 90%. We resolved disagreements by consensus for points we had different rating. We used the PRISMA flow diagram to summarize the study selection processes.

Methodological quality of the included studies was assessed using mixed methods appraisal tool (MMAT) version 2018 [ 13 ]. As it is clearly indicated in the user guide of the MMAT tool, it is discouraged to calculate an overall score from the ratings of each criterion. Instead, it is advised to provide a more detailed presentation of the ratings of each criterion to better inform quality of the included studies. The rating of each criterion was, therefore, done as per the detail explanations included in the guideline. Almost all the included full text articles fulfilled the criteria and all the included full text articles were found to be better quality.

Data extraction

We independently extracted data from papers included in the review using JBI mixed methods data extraction form. This form is only used for reviews that follow a convergent integrated approach, i.e. integration of qualitative data and qualitative data [ 14 ]. The data extraction form was piloted on randomly selected papers and modified accordingly. One reviewer extracted the data from the included studies and the second reviewer checked the extracted data. Disagreements were resolved by discussion between the two reviewers. Information was extracted from each included study on: list of authors, year of publication, study area, population of interest, study type, methods, focus of the studies, main findings, authors’ conclusion, and limitations of the study.

Synthesis of findings

The included full-text articles were qualitatively analyzed using emergent thematic analysis approach to identify key concepts and coded them into related non-mutually exclusive themes. Themes are strategies mentioned or discussed in the included records to improve access to PHC services. Themes were identified manually by reading the included records again and again. We then synthesized each theme by comparing the discussion and conclusion of the included articles.

Systematic review registration number

The protocol of this review is registered in PROSPERO (the registration number is: CRD42019132592) to avoid unplanned duplication and to enable comparison of reported review methods with what was planned in the protocol. It is available at https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42019132592 .

Schematic of the systematic review and reporting of the search

We used PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2009 checklist [ 15 ] for reporting of this systematic review.

Study selection

The search strategy identified 1148 titles and abstracts [914 from PubMed (Table 1 ) and 234 from other sources] as of 10 March 2022. We obtained 900 after we removed duplicated articles. Following assessment by title and abstract, 485 records were excluded because these records did not meet the criteria as mentioned in the method section. Additional 256 records were discarded because the records did not discuss the outcome of interest well and some records were systematic reviews. The full text of the remaining 159 records was examined in more detail. It appeared that 49 studies did not meet the inclusion criteria as described in the method section. One hundred ten records met the inclusion criteria and were included in the systematic review or synthesis (Fig.  1 ).

Study selection flow diagram

Of 900 articles resulting from the search term, 110 (12.2%) met the inclusion criteria. The included full-text articles were published between 1993 and 2021. Ninety-two (83.6%) of the included full-text articles were research articles, 5(4.5%) were technical reports, 3 (2.7%) were perspective, 4 (3.6%) was discussion paper, 3(2.7%) were dissertation or thesis, 2 (1.8%) were commentary, and 1 (0.9) was a book. Thirty-six (33%) and 29 (26%) of the included full-text articles were conducted in Africa and North America, respectively (Fig.  2 ).

Regions where the included full-test articles conducted

Key strategies identified

The analysis of 110 full-text articles resulted in 10 themes. The themes are key strategies to improve access to PHC services in rural communities. The key strategies identified are community health programs or community-directed healthcare interventions, school-based healthcare services, student-led healthcare services, outreach services or mobile clinics, family health program, empanelment, community health funding schemes, telemedicine, promoting the role of traditional medicine, working with non-profit private sectors and non-governmental organizations (NGOs) including faith-based organizations (Table 2 ).

Description of strategies

a. Community health programs or community-directed healthcare interventions

Twenty-four (21.8%) of the full-text articles included in this review discussed that community health programs (CHPs) or community-directed healthcare interventions are best strategies to provide basic health and medical care close to the community to increase access and coverage of essential health services. Community health programs are locally based health promotion, disease prevention, and treatment programs available typically to communities in need and community-directed intervention strategy is an approach in which communities themselves direct the planning and implementation of intervention delivery. Rural communities, especially, in developing countries have no access to healthcare facilities in the near distance and have less chance to receive healthcare from doctors, health officers, nurses or midwives. In response to this critical problems, many countries have been investing heavily in community based primary health care to bring services to rural and remote areas where most of the population lives. Community health programs include construction of health posts or community health centers close to the community and deployment of community health workers (CHWs), such as health extension workers, to reach-out every village, who play a prominent role as the gatekeepers of healthcare in rural communities. Community-directed healthcare intervention is an approach in which communities themselves direct the planning and implementation of healthcare interventions. Community participation remains crucial in the identification of health problems, planning or designing of health interventions and implementation of the interventions, which enhances need-based and demand-driven provision of health services while promoting sustainability and ownership (Additional file 2 : Appendix 2, Table A1).

b. School-based primary healthcare

In this review, 9 of 110 (8.2%) of the included full-text articles pointed out that school-based healthcare services can be effective to improve access to PHC services. School-based health services are health programs that offer health care to children and youth either in a school or on school grounds and usually staffed according to school community needs and resources. School-based health services provide a variety of healthcare services to underserved children, youth and vulnerable populations in a convenient and accessible environment. Access to comprehensive health services via schools leads to improved access to healthcare (Additional file 3 : Appendix 3, Table A2).

c. Student-led healthcare services

In this review, 5 of 110 (4.5%) of the full-text articles discussed that the use of medical and health science students as healthcare service providers can minimize problems related with shortage of health professionals in rural healthcare system and can play appreciable roles to minimize healthcare service access problems in rural communities. Student-led healthcare services are developed through consultation between universities and local health providers and are purposefully designed clinical placements with a focus on clinical educational activities for pre-registration students. Student-led clinics link students, healthcare professionals, community-based organizations, universities, and communities. In this approach, students can gain practical experience in an interdisciplinary setting and through exposure to a community with unique and severe needs (Additional file 4 : Appendix 4, Table A3).

d. Outreach services or mobile clinics

In this systematic literature review, 18 of 110 (16.4%) of the included studies discussed that outreach services or mobile clinics in primary care and rural hospital settings can improve access to PHC services in rural communities. Mobile outreach service is defined as healthcare services provided by a mobile team of trained providers, from a higher-level health facility to a lower-level health facilities or locally available community facilities that are not used for clinical services, such as schools, health posts, or other community structures. Outreach services improve access to specialists and hospital-based services, strengthen connections between specialists and PHC providers, and give the benefits of consultations in primary care settings. Specialist outreach services have the potential to overcome access barriers faced by disadvantaged rural and remote communities. Furthermore, a community-based mobile clinics can be effective in uncovering illness and in directing patients to a healthcare home (Additional file 5 : Appendix 5, Table A4).

e. Family health program

Four (3.6%) of the included full-text articles discussed that family health program (FHP) is highly cost-effective tool for improving access to healthcare services for deprived areas (such as rural communities). Family health program means the program is a program designed to provide primary care as well as the prevention and early treatment of communicable and non-communicable diseases in defined populations by deploying interdisciplinary healthcare teams include physicians, nurses, nurse assistants, and full-time community health agents. It has evolved into a robust approach to providing primary care for defined populations by deploying interdisciplinary healthcare teams. The nucleus of each team includes a physician, a nurse, a nurse assistant, and full-time community health agents. This approach is effective on improving access to healthcare and eliminating health disparities (Additional file 6 : Appendix 6, Table A5).

f. Empanelment

This systematic review of literature identified that empanelment (also known as rostering) is a best strategy to proactively provide coordinated primary healthcare towards achieving universal health coverage. Empanelment is a continuous, iterative set of processes that identify and assign populations to facilities, care teams, or primary care providers who have a responsibility to know their assigned population. It enables health systems to improve health outcomes and to reduce costs. Empanelment establishes a point of care for individuals and simultaneously holds primary healthcare providers and care teams accountable for actively managing care for a specific group of individuals (Additional file 7 : Appendix 7, Table A6).

g. Community health funding schemes

In this systematic review of literature, 11 (10%) of the included articles discussed that community health funding schemes such as community-based health insurance (CBHI) increases access to healthcare services in low-income rural communities. Community-based health insurance schemes are usually voluntary and characterized by community members pooling funds to offset the cost of healthcare. Moreover, this approach is effective to mobilize domestic resources for health at low income levels. For low-income countries, community health financing has modest ability to increase the total amount of funds for healthcare. Properly structured community health financing system can significantly improve efficiency, reduce the cost of healthcare, improve quality and health outcomes, and pool risks. Community-financing schemes could improve preventive services and reduce the incidence of diseases. It could also improve people’s access to healthcare and the quality of services, thus improving their health status. Community health financing could also improve risk pooling and reduce health-induced impoverishment. Community health insurance has potential positive impacts on health and social security (Additional file 8 : Appendix 8, Table A7).

h. Telemedicine

In this review, 13 of 110 (11.8%) articles discussed that telemedicine is one of the solutions for rural subspecialty healthcare delivery. Telemedicine can be defined as the use of technology (computers, video, phone, messaging) by a medical professional to diagnose and treat patients in a remote location. The provision of subspecialty services using telemedicine to a remote and medically underserved population provides improved access to subspecialty care. Telemedicine brings sustainable healthcare to rural populations. Use of information and communication technologies in support of health and health-related fields, including healthcare services, health surveillance, health education, and health research has the potential to greatly improve health service efficiency, expand or scale up treatment delivery to thousands of patients in the rural populations (Additional file 9 : Appendix 9, Table A8).

i. Promoting the role of traditional medicine

Seven (6.4%) of the included articles showed that incorporating traditional healers into public health system addresses healthcare needs of people with limited access to allopathic medicine. Traditional medicine is the sum total of the knowledge, skill, and practices based on the theories, beliefs, and experiences indigenous to different cultures, whether explicable or not, used in the maintenance of health as well as in the prevention, diagnosis, improvement or treatment of physical and mental illness. Knowledge about traditional medicine has a catalyzing effect in meeting health sector development objectives. Integrating traditional medicine into national health systems in combination with national policy and regulation for products, practices and providers can enhance access to PHC services in remote populations (Additional file 10 : Appendix 10, Table A9).

j. Working with non-profit private sectors and non-governmental organizations

In this systematic review, 15 of 110 (13.6%) of the included articles revealed that working with non-profit private sectors and NGOs strengthens the healthcare system. Involving the non-profit private sectors, faith-based organizations (FBOs), and NGOs for health system strengthening eventually contributes to create a healthcare system reflecting an increased efficiency, more equity and good governance in health. International and local NGOs have endeavored to fill the gaps in access to healthcare services, research and advocacy. Non-profit private sectors and NGOs have a key role in improving health in low- and middle-income countries. With networks that reach even the most remote communities, many FBOs are well positioned to promote demand and access for healthcare services. Partnership among FBOs is critical in increasing access to healthcare services, and ensuring sustainability by influencing behaviors at the community, family and individual level. Faith-based organizations play an integral role in the healthcare system by increasing health seeking behaviors and delivering supportive services that address common access and cultural barriers (Additional file 11 : Appendix 11, Table A10).

This systematic literature review found that community health programs or community-directed healthcare interventions, school-based healthcare services, student-led healthcare services, outreach services or mobile clinics, family health program, empanelment, community health funding schemes, telehealth, integrative medicine, and working with non-profit private sectors and NGOs are key strategies to improve access to PHC services in rural communities. The identified strategies address the four major pillars of primary healthcare (i.e., community participation, inter-sectoral coordination, appropriate technology, and support mechanism made available) [ 126 ]. Moreover, the identified strategies are effective to improve access to healthcare services to rural communities. Moreover, the identified strategies are effective to solve shortage of manpower and to build knowledge and skill of the local health workforces in rural healthcare system. The ability of a healthcare system to meet health needs of the population depends largely on the knowledge, skills, motivation and deployment of the people responsible for organizing and delivering health services. The results of this review can strengthen the health information system, which are core elements of the healthcare system that ensure community engagement through dissemination and use of timely and reliable health information to rural populations. This review also suggests strategies to narrow down the health disparities among rural populations, which is wide in most Least and Middle Income Countries (LMICs). Healthcare services are usually disproportionately concentrated in major urban areas. As a result, rural communities face growing health disparities, largely attributed to weak policies, inefficiencies, poor leadership, and governance in healthcare system.

This review identified that community health programs or community-directed healthcare interventions address health disparities by ensuring equitable access to health resources in communities where health equity is limited by socioeconomic and geographical factors. Community health programs include identifying and prioritizing public health problems in a specific geographic area; designing and implementing public health interventions (such as establishing community health centers, mobile clinics, and outreach programs); providing services (such as health education, screenings, social support, and counseling), and deploying community health workers to promote healthy behaviors; advocating for improved care for populations at risk; and working with stakeholders to address community healthcare needs [ 16 , 17 , 18 , 127 , 128 , 129 , 130 ]. The community-oriented PHC model which is socially responsive medicine makes a healthcare system more rational, accountable, appropriate, and socially relevant to the public. Consequently, this model serves as a paradigm for reforming healthcare systems. Community-directed interventions can be considered as a realistic means to increase accessibility of interventions at community-level in rural areas [ 32 , 33 , 34 , 35 , 36 , 37 , 38 ]. This approach is best in situations where there are cultural barriers to implement interventions because this strategy is effective to develop ownership in the community. In-service and on-the-job training for community health workers, close supervision and government support, and program evaluation is very important to strengthen the community health program [ 131 , 132 , 133 ].

This review identified that school-based PHC services are effective strategies to improve access to PHC services. School-based health services provide a variety of healthcare services to children, youth and vulnerable populations in a convenient and accessible environment which indirectly improve leadership and governance. Science teachers and home room teachers play important roles to implement this strategy. It impacts on delivering preventive care such as immunizations, managing chronic illnesses and providing reproductive health services for adolescents. Comprehensive health services via schools improve access to healthcare information [ 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 ]. Access to school around the world increased drastically in the last century [ 134 ]. This high schooling rate is a good opportunity to provide healthcare services to school learners in accessible places and to disseminate health messages to families. Prior researches suggest that school-based healthcare services increase access to healthcare by increasing utilization of primary care, prevention services, and health maintenance visits [ 135 , 136 ]. Including science teachers, home room teachers, school principals, students, communities, community health workers, and other interested parties in the school-based healthcare system as main actors or promoters must be considered to sustain the impact. Health and education sectors should work in collaboration with the above-mentioned actors to plan, implement and monitor the progress. School-based healthcare services are preferable in situations when there is high schooling rate and limited access to healthcare institutions. This strategy is also an alternative way in areas where the health seeking behavior of the community is low.

The use of medical and health science students in rural healthcare system was identified as a key strategy to minimize health inequalities in rural communities due to shortages in health workforce and distribution of healthcare resources [ 49 , 50 , 51 , 52 , 53 ]. Student-led health intervention is an alternative approach to provide essential healthcare services to the community where there is shortage of healthcare workers [ 137 , 138 ]. Students will have opportunities to learn professional skills and competencies while they are providing healthcare services to the community. Moreover, benefits for student learning include increased communication, collaboration, and leadership skills [ 53 , 139 ]. Student-led health intervention also enables increased access to services, more time for assessments and treatments, increased depth of health teaching, holistic and integrated healthcare, and free health supports [ 140 , 141 , 142 , 143 ]. However, the use of medical and health science students in the rural healthcare system may have ethical and competency issues. Supporting strategies such as close supervision, preparing clear protocols, and including senior experts in the team should be considered.

This systematic review of literature found that outreach services or mobile clinics can improve access to PHC service delivery in rural populations [ 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 ]. In developing countries, the highest proportion of people lives in rural areas where doctor services are not available. Rural communities travel to major cities to get specialist services. This reflects a desire for closer integration between primary and secondary care. Specialist outreach services or mobile clinics have become one of the effective solution to solve health disparities, to improve access to healthcare services, and to build capacity of local healthcare workforces. This strategy is preferable in situations when there are high loads in tertiary or referral level hospitals and when there is high patient leakage in the referral system [ 63 , 64 , 65 , 66 , 67 , 68 , 69 ]. However, the implementation may not be easy. It needs well established healthcare system and budget. Moreover, the efficiency of care may be lower compared with hospital-based cares and the effect on patients’ health outcomes might be small [ 56 , 57 , 61 ] . Irregular specialist visits in rural areas may not have real impacts unless the services are sustainable with a strong commitment at national and local levels. Outreach activities should be included in health policies with strong leadership, healthcare financing, and private initiatives must be encouraged to maintain the activities over time.

This review revealed that FHP is highly effective tool for improving health for rural communities. The FHP has provided a new, more robust model of primary healthcare services designed to provide accessible, first contact, comprehensive, and whole person care that is coordinated with other healthcare services. It has positive results to improved availability, access to, and use of health services, and improved health indicators, such as reduced infant mortality, improved detection of cases of neglected diseases, and reduced health disparities [ 73 , 144 , 145 , 146 ]. The FHP deploys interdisciplinary healthcare teams. The team includes a physician, a nurse, a nurse assistant, and full-time community health agents. Family health teams are organized geographically. The teams are responsible for delivering public health interventions [ 72 , 74 ]. Family health program is an alternative strategy in rural healthcare system in situations when there are inequities in access to care; when there is high hospitalization rate; when there is low health seeking behavior in the community; and when there is poor case detecting and reporting system. Despite these remarkable achievements, the FHP has some challenges include difficulties in the recruitment and retention of doctors trained appropriately to deliver primary healthcare, large variations in quality of local care, patchy integration of primary care services with existing secondary and tertiary care, and slow adoption of FHP in large population [ 147 ].

In this review, empanelment has been identified as a best strategy to deliver coordinated primary healthcare towards achieving universal health coverage [ 76 , 77 , 78 , 79 ]. The goal of empanelment is provide people-centered healthcare services based on their needs to ensure that every established patient receives optimal care, whether he/she regularly visits healthcare centers. Major activities in this approach include assignment of all patients to a healthcare provider panel; update panel assignments on a regular basis; and use panel data to educate, and track patients [ 79 ]. Empanelment enables healthcare systems to improve patient experiences, reduce costs, and improve health outcomes. Empanelment is an effective strategy to deliver four key functions: first-contact accessibility, continuity, comprehensiveness, and coordination [ 148 ]. Effective empanelment requires responsibility for the health of a target population, including providing healthcare services based on their health status, which is an important step in moving towards people-centered integrated healthcare [ 79 ].

This review identified that community health funding schemes such as community-based health insurance (CBHI) increases access to healthcare in low-income rural communities. Moreover, this approach is effective to mobilize domestic resources for health at low income levels [ 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ]. Community-based health insurance is an emerging strategy to provide financial protection against the cost of illness. It is an effective strategy to improve access to quality health services for low-income rural households [ 149 ]. Existence of social capital in the community is a determinant factor for the effectiveness of CBHI as social capital has a positive effect on the community's demand for insurance [ 150 , 151 ]. Moreover, solidarity and trust between the members are the key principles for the good functioning of a CBHI. Solidarity and trust stir-up members who are susceptible to risk to put together their resources for common use [ 149 , 152 , 153 ]. Affordability of premiums or contributions, technical arrangements made by the scheme management, timing of collecting the contributions, trust in the integrity and competence of the managers of the CBHI, The quality of care offered through the CBHI, accessible across different population groups are some of the determinant factors to be considered to increase people’s decision to join the CBHI schemes [ 154 , 155 ].

In this review, telemedicine has been identified as one of the many possible solutions for rural subspecialty healthcare delivery. Telemedicine is a vital technological tool to increase healthcare access, improve care delivery systems, engage in culturally competent outreach, health workforce development, and health information system [ 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 ]. Telemedicine can be a great alternative to the traditional healthcare system in situations like diagnoses of common medical problems; inquiries about various medical issues for home treatments; post-treatment check-ins or follow-up for chronic care; holidays, weekends, late night or any other situation when regular medical care is not possible; patient inability to leave the house; patients who lack regular access to relevant medical expertise in their geographic area ; and etc. However, technological issues are challenges when dealing with telemedicine, especially in developing countries. General problems of Internet connectivity and access to infrastructure can minimize benefits of this strategy. Costs associated with technology can also be a barrier. Furthermore, health technology requires human capacity to use it. Therefore, strengthening the information communication technologies (ICT) and human capacity building on ICT are important to address the health needs of the rural communities.

This systematic review of literature identified that promoting the role of TM solves problems of access to allopathic medicine. Integration of TM in health system will result in increased coverage and access to healthcare services. The role of complementary and alternative medicine for health is undisputed particularly in light of its role in health promotion and well-being. It also supports local health workforces [ 104 , 105 , 106 , 107 , 108 , 109 ]. Incorporating traditional healers into the public health system addresses healthcare needs [ 156 , 157 ]. However, integrating TM to the public healthcare system is challenging. It is a general belief that TM defies scientific procedures in terms of objectivity, measurement, codification and classification [ 157 ]. If integrated, who provides training to medical doctors on the ontology, epistemology and the efficacies of TM in modern medicine [ 157 ]. Due to these, some scholars suggest that both TM and modern medicine be allowed to operate and develop independent of one another [ 158 , 159 ]. Another fundamental challenge to TM is the widespread reported cases of fake healers and healings [ 157 ]. Generally, this strategy is more of feasible in areas where formal trainings on integrative medicine are available. Even though the integration is challenging, the health sector can use traditional healers as health educators or health promoters by providing training and continuous support. It can be also possible to use traditional healers as facilitators in the community-directed approaches. In general TM can be used in the primary healthcare system where no access to allopathic medicine and when conventional medicine is ineffective in treatment of disease [ 160 ].

Working with non-profit private sectors and NGOs has been identified as effective strategies to strengthen the healthcare system in developing countries [ 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 ]. Since governments in developing countries are challenged to meet the health needs of their populations because of financial constraints, limited human resources, and weak health infrastructure; the private sector (especially the non-profit private sectors) and non-governmental organizations can help expand access to healthcare services through its resources, expertise, and infrastructure. However, the presence of an NGO in the operation, may contribute to unrealistic expectations of health services, affecting perceptions of the latter negatively [ 113 ]. Moreover, reports have it that besides other issues in many instances NGOs allocated funds only to disease specific projects (vertical programming) rather than to broad based investments (horizontal programming) [ 161 ]. There are also concerns that donor expenditures in developing countries are not only unsustainable but may be considered as inadequate considering the enormous healthcare burden [ 161 , 162 , 163 , 164 ]. To avoid unrealistic expectations and dissatisfaction, and to increase and sustain the population’s trust in the organization, NGOs should operate in a manner that is as integrated as possible within the existing structure and should work close to the population it serves, with services anchored in the community. Moreover, faith-based organizations contribute in health such as disease prevention, health education or promotion, and community health development beyond psychological and spiritual care [ 119 , 120 , 121 , 122 , 123 , 124 ]. Religious organizations can reach all segments of rural populations. Therefore, integrating PHC services, especially health education and promotion, diseases prevention and community health development with religious organizations intensifies delivery of healthcare services. Working with FBOs is a best way in situations where cultural and faith-based barriers are common and in areas, where access problems are often related to lack of providers. However, religious organizations need intensive training on health promotion and health system to enable them to respond to local contexts within the framework of national policies. Moreover, there should be strong partnership with government agenesis to sustain the effort [ 165 , 166 , 167 , 168 ].

Contribution of this review

Various studies reported one or more strategies to improve access to primary healthcare services. However, the strategies reported by individual studies are not compiled together and there is lack of pooled evidence on effective strategies to improve access to healthcare system. This systematic literature review was, therefore, conducted to compile effective strategies to improve access to healthcare services in rural communities. The review suggests key strategies to improve access to PHC services in rural communities. These suggested strategies are implementable in countries that suffer from shortage of health workers and healthcare financing because all the strategies used locally available opportunities. The local healthcare system needs, therefore, scan the available opportunities in the locality for implementing the suggested strategies and needs to integrate the strategies in the healthcare system to sustain the impacts. Healthcare providers, researchers and policy makers could use the results of this systematic literature review to increase access to healthcare services in hard-to-reach areas. As the strategies are compiled from experiences of different countries (developed and least developed countries), there might be contextual differences like socio-economic, cultural, institutional, and geographical challenges to adopt the identified strategies. Moreover, some of the experiences only come from one or two countries. Therefore, strategy developers and implementers need to consider these contextual challenges or variation during adopting and implementing different strategies.

Strengths and limitations of the study

As a strength, this systematic review explores international (both developed and developing countries) best experiences on primary healthcare service delivery and identified ten key approaches to improve access to PHC services in rural communities. We also searched relevant published or unpublished articles, dissertations or theses, discussion papers, and perspectives from a wide range of sources, such as MEDLINE, Scopus, Web of Science, WHO Global Health Library, and Google Scholar.

As a limitation, we entirely relied on electronic databases to search relevant articles. We didn’t include locally available printed out records. We also applied limits for language. We excluded articles published other than English language. We believed we could get more relevant articles if we had access to records available in prints and if we include articles published other than English language. Furthermore, since the strategies are compiled from experiences of different countries (developed and least developed countries), there might be contextual differences like socio-economic, cultural, institutional and geographical challenges to adopt the identified strategies. There was also limited evidence for some articles, especially reports to rate their methodological quality. Readers should also note that our review might missed some important work in improving access to PHC services and the identified strategies are not the only strategies to improve access to PHC services. There might be other effective strategies which are not included in this review. In addition generalizability might be affected since some of the experiences only come from one or two countries. Moreover, this review focuses on access not quality of care delivered.

This review identified key strategies from international experiences to improve access to PHC services in rural communities. These strategies are effective to improve access to healthcare services in rural or remote communities. They can also play roles in achieving UHC and reducing disparities in health outcomes and increase access to rural communities to get healthcare when and where they want. Therefore, incorporating these key strategies suggested by this review in to the healthcare system is useful to enhance PHC services and to minimize impacts of health disparity in rural communities. However, the identified strategies may not be easy to implement. Increasing number and capacity of human resource for health; strengthening the healthcare financing system; improving medicine and supplies; working in different partners and communities; establishing monitoring and evaluation system; strong and committed leadership; and encouraging private initiatives must be considered to implement and maintain these strategies over time. Moreover, policy makers, program planners and implementers who want to utilize findings of this review should be aware that these are not the only effective strategies to improve access to primary healthcare services.

Availability of data and materials

All the extracted data are included in the manuscript.

Abbreviations

Community-based health insurance

Faith-based organizations

Family health program

Information communication technologies

Mixed methods appraisal tool

Non-governmental organizations

• Primary healthcare

Primary Health Care Performance Initiative

Population, phenomena of interest and context)

Traditional medicine

Universal health coverage

Hampton MB, Kettle AJ, Winterbourn CC. Inside the neutrophil phagosome: oxidants, myeloperoxidase, and bacterial killing. Blood. 1998;92(9):3007–17.

Article   CAS   Google Scholar  

Kirby M. The right to health fifty years on: Still skeptical? Health Hum Rights. 1999;4(1):6–25.

O’Connell T, Rasanathan K, Chopra M. What does universal health coverage mean? The Lancet. 2014;383(9913):277–9.

White F. Primary health care and public health: foundations of universal health systems. Med Princ Pract. 2015;24(2):103–16.

Article   Google Scholar  

Sanders D, Nandi S, Labonté R, Vance C, Van Damme W. From primary health care to universal health coverage—one step forward and two steps back. The Lancet. 2019;394(10199):619–21.

Brezzi M, Luongo P. Regional Disparities In Access To Health Care. 2016.

Google Scholar  

Hartley D. Rural health disparities, population health, and rural culture. Am J Public Health. 2004;94(10):1675–8.

Walraven G. The 2018 Astana declaration on primary health care, is it useful? J Glob Health. 2019;9(1).

Gillam S. Is the declaration of Alma Ata still relevant to primary health care? BMJ (Clinical research ed). 2008;336(7643):536–8.

Tollman S, Doherty J, Mulligan JA. General Primary Care. In: Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A, Musgrove P, editors. Disease Control Priorities in Developing Countries. Washington: World Bank The International Bank for Reconstruction and Development/The World Bank Group; 2006. Available at https://www.ncbi.nlm.nih.gov/books/NBK11789/pdf/Bookshelf_NBK11789.pdf .

Stern C, Jordan Z, McArthur A. Developing the review question and inclusion criteria. AJN The Am J Nurs. 2014;114(4):53–6.

World Health Organization. losing the gap in a generation. Commission on Social Determinants of Health FINAL REPORT. 2008. Available at https://www.who.int/social_determinants/final_report/csdh_finalreport_2008.pdf . Accessed on 22 March 2022.

Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, GagnonM-P GF, Nicolau B, O’Cathain A. Mixed methods appraisal tool (MMAT), version 2018. Canada: IC Canadian Intellectual Property Office, Industry; 2018. Available at https://mixedmethodsappraisaltoolpublicpbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf .

JBI Manual for Evidence Synthesis. Appendix 8.1 JBI Mixed Methods Data Extraction Form following a Convergent Integrated Approach. Available at https://jbi-global-wiki.refined.site/space/MANUAL/3318284375/Appendix+8.1+JBI+Mixed+Methods+Data+Extraction+Form+following+a+Convergent+Integrated+Approach . Accessed on 12 August 2021. 

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Assefa Y, Gelaw YA, Hill PS, Taye BW, Van Damme W. Community health extension program of Ethiopia, 2003–2018: successes and challenges toward universal coverage for primary healthcare services. Glob Health. 2019;15(1):1–11.

Admassie A, Abebaw D, Woldemichael AD. Impact evaluation of the Ethiopian health services extension programme. J Dev Eff. 2009;1(4):430–49.

Yitayal M, Berhane Y, Worku A, Kebede Y. The community-based Health extension Program significantly improved contraceptive utilization in West gojjam Zone, ethiopia. J Multidiscip Healthc. 2014;7:201.

Croke K, Mengistu AT, O’Connell SD, Tafere K. The impact of a health facility construction campaign on health service utilisation and outcomes: analysis of spatially linked survey and facility location data in Ethiopia. BMJ Glob Health. 2020;5(8):e002430.

Arwal S. Health Posts in Afghanistan. J Gen Practice. 2015;3(213):2.

Negussie A, Girma G. Is the role of Health Extension Workers in the delivery of maternal and child health care services a significant attribute? The case of Dale district, southern Ethiopia. BMC Health Serv Res. 2017;17(1):1–8.

Than KK, Mohamed Y, Oliver V, Myint T, La T, Beeson JG, Luchters S. Prevention of postpartum haemorrhage by community-based auxiliary midwives in hard-to-reach areas of Myanmar: a qualitative inquiry into acceptability and feasibility of task shifting. BMC Pregnancy Childbirth. 2017;17(1):1–10.

Medhanyie A, Spigt M, Kifle Y, Schaay N, Sanders D, Blanco R, GeertJan D, Berhane Y. The role of health extension workers in improving utilization of maternal health services in rural areas in Ethiopia: a cross sectional study. BMC Health Serv Res. 2012;12(1):1–9.

Sakeah E, McCloskey L, Bernstein J, Yeboah-Antwi K, Mills S, Doctor HV. Can community health officer-midwives effectively integrate skilled birth attendance in the community-based health planning and services program in rural Ghana? Reprod Health. 2014;11(1):1–13.

Sarmento DR. Traditional birth attendance (TBA) in a health system: what are the roles, benefits and challenges: a case study of incorporated TBA in Timor-Leste. Asia Pac Fam Med. 2014;13(1):1–9.

Rahmawati R, Bajorek B. Peer Reviewed: A Community Health Worker-Based Program for Elderly People with Hypertension in Indonesia: A Qualitative Study, 2013. Prev Chronic Dis. 2015;12:E175.

Feltner FJ, Ely GE, Whitler ET, Gross D, Dignan M. Effectiveness of community health workers in providing outreach and education for colorectal cancer screening in Appalachian Kentucky. Soc Work Health Care. 2012;51(5):430–40.

Hughes MM, Yang E, Ramanathan D, Benjamins MR. Community-based diabetes community health worker intervention in an underserved Chicago population. J Community Health. 2016;41(6):1249–56.

Panday S, Bissell P, Van Teijlingen E, Simkhada P. The contribution of female community health volunteers (FCHVs) to maternity care in Nepal: a qualitative study. BMC Health Serv Res. 2017;17(1):1–11.

Datiko DG, Lindtjørn B. Health extension workers improve tuberculosis case detection and treatment success in southern Ethiopia: a community randomized trial. PLoS ONE. 2009;4(5):e5443.

le Roux KW, Almirol E, Rezvan PH, Le Roux IM, Mbewu N, Dippenaar E, Stansert-Katzen L, Baker V, Tomlinson M, Rotheram-Borus M. Community health workers impact on maternal and child health outcomes in rural South Africa–a non-randomized two-group comparison study. BMC Public Health. 2020;20(1):1–14.

Witmer A, Seifer SD, Finocchio L, Leslie J, O’Neil EH. Community health workers: integral members of the health care work force. Am J Public Health. 1995;85(8 Pt 1):1055–8.

Wright RA. Community-oriented primary care. The cornerstone of health care reform. Jama. 1993;269(19):2544–7.

Makaula P, Bloch P, Banda HT, Mbera GB, Mangani C, de Sousa A, Nkhono E, Jemu S, Muula AS. Primary Health Care in rural Malawi - a qualitative assessment exploring the relevance of the community-directed interventions approach. BMC Health Serv Res. 2012;12:328.

Katabarwa MN, Habomugisha P, Richards FO Jr, Hopkins D. Community-directed interventions strategy enhances efficient and effective integration of health care delivery and development activities in rural disadvantaged communities of Uganda. Trop Med Int Health : TM & IH. 2005;10(4):312–21.

Madon S, Malecela MN, Mashoto K, Donohue R, Mubyazi G, Michael E. The role of community participation for sustainable integrated neglected tropical diseases and water, sanitation and hygiene intervention programs: A pilot project in Tanzania. Soc Sci Med. 1982;2018(202):28–37.

Okeibunor JC, Orji BC, Brieger W, Ishola G, Otolorin E, Rawlins B, Ndekhedehe EU, Onyeneho N, Fink G. Preventing malaria in pregnancy through community-directed interventions: evidence from Akwa Ibom State, Nigeria. Malaria J. 2011;10:227.

Brieger WR, Sommerfeld JU, Amazigo UV. The Potential for Community-Directed Interventions: Reaching Underserved Populations in Africa. Int Q Community Health Educ. 2015;35(4):295–316.

Braimah JA, Sano Y, Atuoye KN, Luginaah I. Access to primary health care among women: the role of Ghana’s community-based health planning and services policy. Prim Health Care Res Dev. 2019;20:e82.

Kaplan DW, Brindis CD, Phibbs SL, Melinkovich P, Naylor K, Ahlstrand K. A comparison study of an elementary school–based health center: effects on health care access and use. Arch Pediatr Adolesc Med. 1999;153(3):235–43.

Allison MA, Crane LA, Beaty BL, Davidson AJ, Melinkovich P, Kempe A. School-based health centers: improving access and quality of care for low-income adolescents. Pediatrics. 2007;120(4):e887–94.

Keeton V, Soleimanpour S, Brindis CD. School-based health centers in an era of health care reform: Building on history. Curr Probl Pediatr Adolesc Health Care. 2012;42(6):132–56.

Brindis CD, Klein J, Schlitt J, Santelli J, Juszczak L, Nystrom RJ. School-based health centers: Accessibility and accountability. J Adolesc Health. 2003;32(6):98–107.

Hutchinson P, Carton TW, Broussard M, Brown L, Chrestman S. Improving adolescent health through school-based health centers in post-Katrina New Orleans. Child Youth Serv Rev. 2012;34(2):360–8.

Paschall MJ, Bersamin M. School-based health centers, depression, and suicide risk among adolescents. Am J Prev Med. 2018;54(1):44–50.

Minguez M, Santelli JS, Gibson E, Orr M, Samant S. Reproductive health impact of a school health center. J Adolesc Health. 2015;56(3):338–44.

Gibson EJ, Santelli JS, Minguez M, Lord A, Schuyler AC. Measuring school health center impact on access to and quality of primary care. J Adolesc Health. 2013;53(6):699–705.

Bozigar M. A Cross-Sectional Survey to Evaluate Potential for Partnering With School Nurses to Promote Human Papillomavirus Vaccination. Prev Chronic Dis. 2020;17:E111.

Suen J, Attrill S, Thomas JM, Smale M, Delaney CL, Miller MD. Effect of student-led health interventions on patient outcomes for those with cardiovascular disease or cardiovascular disease risk factors: a systematic review. BMC Cardiovasc Disord. 2020;20(1):1–10.

Atuyambe LM, Baingana RK, Kibira SP, Katahoire A, Okello E, Mafigiri DK, Ayebare F, Oboke H, Acio C, Muggaga K. Undergraduate students’ contributions to health service delivery through communitybased education. BMC Med Educ. 2016;16:123.

Stuhlmiller CM, Tolchard B. Developing a student-led health and wellbeing clinic in an underserved community: collaborative learning, health outcomes and cost savings. BMC Nurs. 2015;14(1):1–8.

Campbell DJ, Gibson K, O’Neill BG, Thurston WE. The role of a student-run clinic in providing primary care for Calgary’s homeless populations: a qualitative study. BMC Health Serv Res. 2013;13(1):1–6.

Simpson SA, Long JA. Medical student-run health clinics: important contributors to patient care and medical education. J Gen Intern Med. 2007;22(3):352–6.

Gruen RL, O’Rourke IC, Bailie RS, d’Abbs PH, O’Brien MM, Verma N. Improving access to specialist care for remote Aboriginal communities: evaluation of a specialist outreach service. Med J Aust. 2001;174(10):507–11.

Gruen RL, Weeramanthri T, Bailie R. Outreach and improved access to specialist services for indigenous people in remote Australia: the requirements for sustainability. J Epidemiol Community Health. 2002;56(7):517–21.

Gruen RL, Bailie RS, Wang Z, Heard S, O’Rourke IC. Specialist outreach to isolated and disadvantaged communities: a population-based study. The Lancet. 2006;368(9530):130–8.

Bond M, Bowling A, Abery A, McClay M, Dickinson E. Evaluation of outreach clinics held by specialists in general practice in England. J Epidemiol Community Health. 2000;54(2):149–56.

Irani M, Dixon M, Dean JD. Care closer to home: past mistakes, future opportunities. J R Soc Med. 2007;100(2):75–7.

Bailey JJ, Black ME, Wilkin D. Specialist outreach clinics in general practice. BMJ (Clinical research ed). 1994;308(6936):1083–6.

De Roodenbeke E, Lucas S, Rouzaut A, Bana F. Outreach services as a strategy to increase access to health workers in remote and rural areas. Geneva: WHO; 2011.

Bowling A, Stramer K, Dickinson E, Windsor J, Bond M. Evaluation of specialists’ outreach clinics in general practice in England: process and acceptability to patients, specialists, and general practitioners. J Epidemiol Community Health. 1997;51(1):52–61.

Spencer N. Consultant paediatric outreach clinics–a practical step in integration. Arch Dis Child. 1993;68(4):496–500.

Aljasir B, Alghamdi MS. Patient satisfaction with mobile clinic services in a remote rural area of Saudi Arabia. East Mediterr Health J. 2010;16(10):1085–90.

Lee EJ, O’Neal S. A mobile clinic experience: nurse practitioners providing care to a rural population. J Pediatr Health Care. 1994;8(1):12–7.

Cone PH, Haley JM. Mobile clinics in Haiti, part 1: Preparing for service-learning. Nurse Educ Pract. 2016;21:1–8.

Diaz-Perez Mde J, Farley T, Cabanis CM. A program to improve access to health care among Mexican immigrants in rural Colorado. J Rural Health. 2004;20(3):258–64.

Hill C, Zurakowski D, Bennet J, Walker-White R, Osman JL, Quarles A, Oriol N. Knowledgeable Neighbors: a mobile clinic model for disease prevention and screening in underserved communities. Am J Public Health. 2012;102(3):406–10.

Edgerley LP, El-Sayed YY, Druzin ML, Kiernan M, Daniels KI. Use of a community mobile health van to increase early access to prenatal care. Matern Child Health J. 2007;11(3):235–9.

Peters G, Doctor H, Afenyadu G, Findley S, Ager A. Mobile clinic services to serve rural populations in Katsina State, Nigeria: perceptions of services and patterns of utilization. Health Policy Plan. 2014;29(5):642–9.

Neke NM, Gadau G, Wasem J. Policy makers’ perspective on the provision of maternal health services via mobile health clinics in Tanzania—Findings from key informant interviews. PLoS ONE. 2018;13(9):e0203588.

Padmadas SS, Johnson FA, Leone T, Dahal GP. Do mobile family planning clinics facilitate vasectomy use in Nepal? Contraception. 2014;89(6):557–63.

Macinko J, Harris MJ. Brazil’s family health strategy—delivering community-based primary care in a universal health system. N Engl J Med. 2015;372(23):2177–81.

Macinko J, Lima Costa MF. Access to, use of and satisfaction with health services among adults enrolled in Brazil’s Family Health Strategy: evidence from the 2008 National Household Survey. Tropical Med Int Health. 2012;17(1):36–42.

Dourado I, Oliveira VB, Aquino R, Bonolo P, Lima-Costa MF, Medina MG, Mota E, Turci MA, Macinko J. Trends in primary health care-sensitive conditions in Brazil: the role of the Family Health Program (Project ICSAP-Brazil). Medical care. 2011;49:577–84.

Aquino R, De Oliveira NF, Barreto ML. Impact of the family health program on infant mortality in Brazilian municipalities. Am J Public Health. 2009;99(1):87–93.

Chong P-N, Tang WE. Transforming primary care—the way forward with the TEAMS2 approach. Fam Pract. 2019;36(3):369–70.

Primary Health Care Performance Initiatives (phcpi). Improvement strategies model: Population health management: Empanelment. Available at https://improvingphc.org/sites/default/files/Empanelment%20-%20v1.2%20-%20last%20updated%2012.13.2019.pdf . Accessed on 18 March 2022. 

McGough P, Chaudhari V, El-Attar S, Yung P. A health system’s journey toward better population health through empanelment and panel management. Healthcare. 2018;6(66):1–9.

Bearden T, Ratcliffe HL, Sugarman JR, Bitton A, Anaman LA, Buckle G, Cham M, Quan DCW, Ismail F, Jargalsaikhan B. Empanelment: A foundational component of primary health care. Gates Open Res. 2019;3:1654.

Hsiao WC. Unmet health needs of two billion: is community financing a solution? 2001.

Wang W, Temsah G, Mallick L. The impact of health insurance on maternal health care utilization: evidence from Ghana, Indonesia and Rwanda. Health Policy Plan. 2017;32(3):366–75.

Atnafu DD, Tilahun H, Alemu YM. Community-based health insurance and healthcare service utilisation, North-West, Ethiopia: a comparative, cross-sectional study. BMJ Open. 2018;8(8):e019613.

USAID. Ethiopia’s Community-based Health Insurance: A Step on the Road to Universal Health Coverage. Available at https://www.hfgproject.org/ethiopias-community-based-health-insurance-step-road-universal-health-coverage/ . Accessed on 18 March 2022.

Blanchet NJ, Fink G, Osei-Akoto I. The effect of Ghana’s National Health Insurance Scheme on health care utilisation. Ghana Med J. 2012;46(2):76–84.

CAS   Google Scholar  

Nshakira-Rukundo E, Mussa EC, Nshakira N, Gerber N, von Braun J. Impact of community-based health insurance on utilisation of preventive health services in rural Uganda: a propensity score matching approach. Int J Health Econ Manag. 2021;21(2):203–27.

Mwaura JW, Pongpanich S. Access to health care: the role of a community based health insurance in Kenya. Pan Afr Med J. 2012;12(1):35.

Jutting JP. The Impact Of Health Insurance On The Access To Health Care And Financial Protection In Rural Developing Countries: The Example of Senegal. HNP discussion paper series;. World Bank, Washington, DC. © World Bank. 2011. https://openknowledge.worldbank.org/handle/10986/13774 . License: CC BY 3.0 IGO.

Balamiento NC. The impact of social health insurance on healthcare utilization outcomes: evidence from the indigent program of the Philippine National Health Insurance. International Institute of Social Studies. 2018. Available at https://thesis.eur.nl/pub/46445/Balamiento,%20Neeanne_MA_2017_18%20_ECD.pdf . Accessed 30 Nov 2022.

Farrell CM, Gottlieb A. The effect of health insurance on health care utilization in the justice-involved population: United States, 2014–2016. Am J Public Health. 2020;110(S1):S78–84.

Thuong NTT. Impact of health insurance on healthcare utilisation patterns in Vietnam: a survey-based analysis with propensity score matching method. BMJ Open. 2020;10(10):e040062.

Custodio R, Gard AM, Graham G. Health information technology: addressing health disparity by improving quality, increasing access, and developing workforce. J Health Care Poor Underserved. 2009;20(2):301–7.

Meier CA, Fitzgerald MC, Smith JM. eHealth: extending, enhancing, and evolving health care. Annu Rev Biomed Eng. 2013;15:359–82.

Anstey Watkins JOT, Goudge J, Gomez-Olive FX, Griffiths F. Mobile phone use among patients and health workers to enhance primary healthcare: A qualitative study in rural South Africa. Soc Sci Med. 1982;2018(198):139–47.

Kuntalp M, Akar O. A simple and low-cost Internet-based teleconsultation system that could effectively solve the health care access problems in underserved areas of developing countries. Comput Methods Programs Biomed. 2004;75(2):117–26.

Price M, Yuen EK, Goetter EM, Herbert JD, Forman EM, Acierno R, Ruggiero KJ. mHealth: a mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother. 2014;21(5):427–36.

Bashshur RL, Shannon GW, Krupinski EA, Grigsby J, Kvedar JC, Weinstein RS, Sanders JH, Rheuban KS, Nesbitt TS, Alverson DC, et al. National telemedicine initiatives: essential to healthcare reform. Telemed J E Health. 2009;15(6):600–10.

Norton SA, Burdick AE, Phillips CM, Berman B. Teledermatology and underserved populations. Arch Dermatol. 1997;133(2):197–200.

Raza T, Joshi M, Schapira RM, Agha Z. Pulmonary telemedicine–a model to access the subspecialist services in underserved rural areas. Int J Med Informatics. 2009;78(1):53–9.

Shouneez YH. Smartphone hearing screening in mHealth assisted community-based primary care. UPSpace Institutional Repository, Department of Liberary Service. Dissertation (MCommPath)--University of Pretoria. 2016. Available at http://hdl.handle.net/2263/53477 . Accessed 17 Mar 2022.

Marcin JP, Ellis J, Mawis R, Nagrampa E, Nesbitt TS, Dimand RJ. Using telemedicine to provide pediatric subspecialty care to children with special health care needs in an underserved rural community. Pediatrics. 2004;113(1 Pt 1):1–6.

Olu O, Muneene D, Bataringaya JE, Nahimana M-R, Ba H, Turgeon Y, Karamagi HC, Dovlo D. How can digital health technologies contribute to sustainable attainment of universal health coverage in Africa? A perspective. Front Public Health. 2019;7:341.

Ryan MH, Yoder J, Flores SK, Soh J, Vanderbilt AA. Using health information technology to reach patients in underserved communities: A pilot study to help close the gap with health disparities. Global J Health Sci. 2016;8(6):86.

Buckwalter KC, Davis LL, Wakefield BJ, Kienzle MG, Murray MA. Telehealth for elders and their caregivers in rural communities. Fam Community Health. 2002;25(3):31–40.

WHO Regional Committee for Africa. Promoting the role of traditional medicine in health systems: a strategy for the African Region. World Health Organization. Regional Office for Africa. Available at http://www.who.int/iris/handle/10665/95467. .

Mishra SR, Neupane D, Kallestrup P. Integrating complementary and alternative medicine into conventional health care system in developing countries: an example of Amchi. J Evid-Based Complementary Altern Med. 2015;20(1):76–9.

Mbwambo ZH, Mahunnah RL, Kayombo EJ. Traditional health practitioner and the scientist: bridging the gap in contemporary health research in Tanzania. Tanzan Health Res Bull. 2007;9(2):115–20.

Poudyal AK, Jimba M, Murakami I, Silwal RC, Wakai S, Kuratsuji T. A traditional healers’ training model in rural Nepal: strengthening their roles in community health. Trop Med Int Health : TM & IH. 2003;8(10):956–60.

Payyappallimana U. Role of Traditional Medicine in Primary Health Care: An Overview of Perspectives and Challenges. Yokohama J Social Sciences. 2009;14(6):723–43.

Kange’ethe SM. Traditional healers as caregivers to HIV/AIDS clients and other terminally challenged persons in Kanye community home-based care programme (CHBC), Botswana. SAHARA J. 2009;6(2):83–91.

Habtom GK. Integrating traditional medical practice with primary healthcare system in Eritrea. J Complement Integr Med. 2015;12(1):71–87.

Ejaz I, Shaikh BT, Rizvi N. NGOs and government partnership for health systems strengthening: a qualitative study presenting viewpoints of government, NGOs and donors in Pakistan. BMC Health Serv Res. 2011;11(1):1–7.

Wu FS. International non-governmental actors in HIV/AIDS prevention in China. Cell Res. 2005;15(11):919–22.

Biermann O, Eckhardt M, Carlfjord S, Falk M, Forsberg BC. Collaboration between non-governmental organizations and public services in health–a qualitative case study from rural Ecuador. Glob Health Action. 2016;9(1):32237.

Mercer A, Khan MH, Daulatuzzaman M, Reid J. Effectiveness of an NGO primary health care programme in rural Bangladesh: evidence from the management information system. Health Policy Plan. 2004;19(4):187–98.

Baqui AH, Rosecrans AM, Williams EK, Agrawal PK, Ahmed S, Darmstadt GL, Kumar V, Kiran U, Panwar D, Ahuja RC. NGO facilitation of a government community-based maternal and neonatal health programme in rural India: improvements in equity. Health Policy Plan. 2008;23(4):234–43.

Ricca J, Kureshy N, LeBan K, Prosnitz D, Ryan L. Community-based intervention packages facilitated by NGOs demonstrate plausible evidence for child mortality impact. Health Policy Plan. 2014;29(2):204–16.

Ahmed N, DeRoeck D, Sadr-Azodi N. Private sector engagement and contributions to immunisation service delivery and coverage in Sudan. BMJ Glob Health. 2019;4(2):e001414.

Edimond BJ. The Contribution of Non-Governmental Organizations in Delivery of Basic Health Services in Partnership with Local Government. Doctoral Dissertation, Uganda Martyrs University. 2014.

Chand S, Patterson J: Faith-Based Models for Improving Maternal and Newborn Health. IMA World Health and ActionAid International USA, 2007 Available at https://imaworldhealthorg/wp-content/uploads/2014/06/faith_based_models_for_improving_maternal_and_newborn_health.pdf

Magezi V. Churchdriven primary health care: Models for an integrated church and community primary health care in Africa (a case study of the Salvation Army in East Africa). HTS Teologiese Studies/ Theological Studies. 2018;74(2):4365.

Villatoro AP, Dixon E, Mays VM. Faith-based organizations and the Affordable Care Act: Reducing Latino mental health care disparities. Psychol Serv. 2016;13(1):92–104.

Levin J. Faith-based initiatives in health promotion: history, challenges, and current partnerships. American journal of health promotion : AJHP. 2014;28(3):139–41.

Green A, Shaw J, Dimmock F, Conn C. A shared mission? Changing relationships between government and church health services in Africa. Int J Health Plann Manage. 2002;17(4):333–53.

Bandy G, Crouch A. Building from common foundations : the World Health Organization and faith-based organizations in primary healthcare. World Health Organization; 2008. Available at https://apps.who.int/iris/handle/10665/43884 . Accessed 16 Mar 2022.

Zahnd WE, Jenkins WD, Shackelford J, Lobb R, Sanders J, Bailey A. Rural cancer screening and faith community nursing in the era of the Affordable Care Act. J Health Care Poor Underserved. 2018;29(1):71–80.

Wagle K. Primary Health Care (PHC): History, Principles, Pillars, Elements & Challenges. Global Health, 2020. Available at https://www.publichealthnotes.com/primary-health-care-phc-history-principles-pillars-elements-challenges/ . Accessed 4 June 2022.

Bhatt J, Bathija P. Ensuring access to quality health care in vulnerable communities. Acad Med. 2018;93(9):1271.

Arvey SR, Fernandez ME. Identifying the core elements of effective community health worker programs: a research agenda. Am J Public Health. 2012;102(9):1633–7.

Pennel CL, McLeroy KR, Burdine JN, Matarrita-Cascante D, Wang J. Community health needs assessment: potential for population health improvement. Popul Health Manag. 2016;19(3):178–86.

Chudgar RB, Shirey LA, Sznycer-Taub M, Read R, Pearson RL, Erwin PC. Local health department and academic institution linkages for community health assessment and improvement processes: a national overview and local case study. J Public Health Manag Pract. 2014;20(3):349–55.

Desta FA, Shifa GT, Dagoye DW, Carr C, Van Roosmalen J, Stekelenburg J, Nedi AB, Kols A, Kim YM. Identifying gaps in the practices of rural health extension workers in Ethiopia: a task analysis study. BMC Health Serv Res. 2017;17(1):1–9.

Lehmann U, Sanders D. Community health workers: what do we know about them. The state of the evidence on programmes, activities, costs and impact on health outcomes of using community health workers Geneva: World Health Organization; 2007. Available at https://www.hrhresourcecenter.org/node/1587.html . Accessed 17 Mar 2022.

Chen N, Raghavan M, Albert J, McDaniel A, Otiso L, Kintu R, West M, Jacobstein D. The community health systems reform cycle: strengthening the integration of community health worker programs through an institutional reform perspective. Global Health: Sci Practice. 2021;9(Supplement 1):S32–46.

Roser M, Ortiz-Ospina E: Global rise of education. Our World in Data 2017. Available at https://ourworldindata.org/global-rise-of-education . Accessed on 29 May 2019.

Santelli J, Morreale M, Wigton A, Grason H. School health centers and primary care for adolescents: a review of the literature. J Adolesc Health. 1996;18(5):357–66.

Wade TJ, Mansour ME, Guo JJ, Huentelman T, Line K, Keller KN. Access and utilization patterns of school-based health centers at urban and rural elementary and middle schools. Public Health Reports. 2008;123(6):739–50.

Johnson I, Hunter L, Chestnutt IG. Undergraduate students’ experiences of outreach placements in dental secondary care settings. Eur J Dent Educ. 2012;16(4):213–7.

Ndira S, Ssebadduka D, Niyonzima N, Sewankambo N, Royall J. Tackling malaria, village by village: a report on a concerted information intervention by medical students and the community in Mifumi Eastern Uganda. Afr Health Sci. 2014;14(4):882–8.

Frakes K-a, Brownie S, Davies L, Thomas JB, Miller M-E, Tyack Z. Capricornia Allied Health Partnership (CAHP): a case study of an innovative model of care addressing chronic disease through a regional student-assisted clinic. Aust Health Rev. 2014;38(5):483–6.

Frakes KA, Brownie S, Davies L, Thomas J, Miller ME, Tyack Z. The sociodemographic and health-related characteristics of a regional population with chronic disease at an interprofessional student-assisted clinic in Q ueensland C apricornia A llied H ealth P artnership. Aust J Rural Health. 2013;21(2):97–104.

Frakes K-A, Tyzack Z, Miller M, Davies L, Swanston A, Brownie S. The Capricornia Project: Developing and implementing an interprofessional student-assisted allied health clinic. 2011.

Frakes K-A, Brownie S, Davies L, Thomas J, Miller M-E, Tyack Z. Experiences from an interprofessional student-assisted chronic disease clinic. J Interprof Care. 2014;28(6):573–5.

Schutte T, Tichelaar J, Dekker RS, van Agtmael MA, de Vries TP, Richir MC. Learning in student-run clinics: A systematic review. Med Educ. 2015;49(3):249–63.

Paim J, Travassos C, Almeida C, Bahia L, Macinko J. The Brazilian health system: history, advances, and challenges. The Lancet. 2011;377(9779):1778–97.

Rocha R, Soares RR. Evaluating the impact of community-based health interventions: evidence from Brazil’s Family Health Program. Health Econ. 2010;19(S1):126–58.

Rasella D, Harhay MO, Pamponet ML, Aquino R, Barreto ML. Impact of primary health care on mortality from heart and cerebrovascular diseases in Brazil: a nationwide analysis of longitudinal data. BMJ (Clinical research ed). 2014;349:g4014.

Harris M. Brazil’s Family Health Programme: A cost effective success that higher income countries could learn from. BMJ: Br Med J. 2010;341(7784):1171–2.

Starfield B. Is primary care essential? The lancet. 1994;344(8930):1129–33.

Donfouet HPP, Mahieu P-A. Community-based health insurance and social capital: a review. Heal Econ Rev. 2012;2(1):1–5.

Zhang L, Wang H, Wang L, Hsiao W. Social capital and farmer’s willingness-to-join a newly established community-based health insurance in rural China. Health Policy. 2006;76(2):233–42.

Donfouet HPP. Essombè J-RE, Mahieu P-A, Malin E: Social capital and willingness-to-pay for community-based health insurance in rural Cameroon. Global J Health Sci. 2011;3(1):142.

Grunau J. Exploring people’s motivation to join or not to join the community-based health insurance’Sina Passenang’in Sotouboua, Togo. 2013.

Gitahi JW. Innovative Healthcare Financing and Equity through Community Based Health Insurance Schemes (CBHHIS) In Kenya. United States International University-Africa Digital Repository. Available at http://erepo.usiu.ac.ke/11732/3654 . Accessed 18 May 2022.

Carrin G, Waelkens MP, Criel B. Community-based health insurance in developing countries: a study of its contribution to the performance of health financing systems. Tropical Med Int Health. 2005;10(8):799–811.

Umeh CA, Feeley FG. Inequitable access to health care by the poor in community-based health insurance programs: a review of studies from low-and middle-income countries. Global Health: Science And Practice. 2017;5(2):299–314.

Odebiyi AI. Western trained nurses’ assessment of the different categories of traditional healers in southwestern Nigeria. Int J Nurs Stud. 1990;27(4):333–42.

Abdullahi AA. Trends and challenges of traditional medicine in Africa. Afr J Tradit Complement Altern Med : AJTCAM. 2011;8(5 Suppl):115–23.

Taye OR. Yoruba Traditional Medicine and the Challenge of Integration. The J Pan Afr Studies. 2009;3(3):73–90.

Konadu K. Medicine and Anthropology in Twentieth Century Africa: Akan Medicine and Encounters with (Medical) Anthropology. African Studies Quarterly. 2008;10(2 & 3).

Benzie IF, Wachtel-Galor S: Herbal medicine: biomolecular and clinical aspects. 2nd Ed. 2011. Available at https://www.crcpress.com/Herbal-Medicine-Biomolecular-and-Clinical-Aspects-Second-Edition/Benzie-Wachtel-Galor/p/book/9781439807132 . Accessed 21 May 2022.

Ejughemre U. Donor support and the impacts on health system strengthening in sub-saharan africa: assessing the evidence through a review of the literature. Am J Public Health Res. 2013;1(7):146–51.

Seppey M, Ridde V, Touré L, Coulibaly A. Donor-funded project’s sustainability assessment: a qualitative case study of a results-based financing pilot in Koulikoro region. Mali Globalization and health. 2017;13(1):1–15.

Shaw RP, Wang H, Kress D, Hovig D. Donor and domestic financing of primary health care in low income countries. Health Systems & Reform. 2015;1(1):72–88.

Gotsadze G, Chikovani I, Sulaberidze L, Gotsadze T, Goguadze K, Tavanxhi N. The challenges of transition from donor-funded programs: results from a theory-driven multi-country comparative case study of programs in Eastern Europe and Central Asia supported by the Global Fund. Global Health: Science and Practice. 2019;7(2):258–72.

Ascroft J, Sweeney R, Samei M, Semos I, Morgan C. Strengthening church and government partnerships for primary health care delivery in Papua New Guinea: Lessons from the international experience. Health policy and health finance knowledge hub Working paper series. 2011(16).

Campbell MK, Hudson MA, Resnicow K, Blakeney N, Paxton A, Baskin M. Church-based health promotion interventions: evidence and lessons learned. Annu Rev Public Health. 2007;28:213–34.

Olivier J, Wodon Q. The role of faith-inspired health care providers in Sub-Saharan Africa and public private partnerships: Strengthening the Evidence for faith-inspired health engagement in Africa, Volume 1. Health, Nutrition and Population (HNP) Discussion Paper Series 76223v1. Available at https://documents1.worldbank.org/curated/en/851911468203673017 . Accessed 20 May 2022.

Schumann C, Stroppa A, Moreira-Almeida A. The contribution of faith-based health organisations to public health. Int Psychiatry. 2011;8(3):62–4.

Download references

Acknowledgements

The author would like to thank IPHC- E for funding this review.

This review was funded by International Institute for Primary Health Care- Ethiopia (IPHC- E).

Author information

Authors and affiliations.

Department of Environmental and Occupational Health and Safety, Institute of Public Health, College of Medicine and Health Sciences, University of Gondar, Gondar, Ethiopia

Zemichael Gizaw

International Institute for Primary Health Care- Ethiopia, Ethiopian Public Health Institute, Addis Ababa, Ethiopia

Tigist Astale & Getnet Mitike Kassie

You can also search for this author in PubMed   Google Scholar

Contributions

ZG prepared the manuscript. TA and GMK critically reviewed the protocol and manuscript. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Zemichael Gizaw .

Ethics declarations

Ethics approval and consent to participate.

Systematic review does not required ethics approval.

Consent for publication

This manuscript does not contain any individual person’s data.

Competing interests

The authors declared that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: .

Searchstrategy. MEDLINE (PubMed).

Additional file 2: Appendix 2: Table A1.

Description of full-text articles which discussed community health programs or community-directed interventions as a strategy to improve PHC service delivery in ruralcommunities.

Additional file 3:

Appendix 3: Table A2. Description of full-text articles which discussed school-based healthcareservices as a strategy to improve PHCservice delivery in rural communities.

Additional file 4:

Appendix 4: Table A3. Description of full-text articles which discussed student-led healthcareservices as a strategy to improve PHC service delivery in ruralcommunities.

Additional file 5: Appendix 5: Table A4

. Descriptionof full-text articles which discussed outreach services or mobile clinics as astrategy to improve PHC service delivery in ruralcommunities.

Additional file 6:

  Appendix 6: Table A5. Description of full-text articles which discussed family health program as astrategy to improve PHC service delivery in rural,communities.

Additional file 7:

  Appendix 7: Table A6. Description of full-text articles whichdiscussed empanelment as a strategy to improve PHC service delivery in ruralcommunities.

Additional file 8:

  Appendix 9: Table A8. Description of full-text articles which discussed telemedicine or mobile healthas a strategy to improve PHC service delivery in ruralcommunities.

Additional file 9:

  Appendix 8: Table A7. Description of full-text articles which discussed community health funding schemes as a strategy to improve PHC service delivery in ruralcommunities.

Additional file 10:

  Appendix 10: Table A9. Description of full-text articles which discussed promoting the role of workingwith traditional healers as a strategy toimprove PHC service delivery in rural communities.

Additional file 11:

  Appendix 11: Table A10. Description of full-text articles which discussed working with non-profitprivate sectors and non-governmental organizations as a strategy to improve PHC service delivery in rural communities.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gizaw, Z., Astale, T. & Kassie, G.M. What improves access to primary healthcare services in rural communities? A systematic review. BMC Prim. Care 23 , 313 (2022). https://doi.org/10.1186/s12875-022-01919-0

Download citation

Received : 09 August 2022

Accepted : 18 November 2022

Published : 06 December 2022

DOI : https://doi.org/10.1186/s12875-022-01919-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Access to PHC services
• Rural communities
• Key strategies to improve access to PHC services

BMC Primary Care

ISSN: 2731-4553

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley-Blackwell Online Open

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure  1 . A list of excluded studies with reasons can be found in Appendix B .

Study selection flowchart

3.1. Characteristics of included reviews

Table  1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n  = 3), literature search ( n  = 3), study selection ( n  = 2), data extraction ( n  = 1), and RoB assessment ( n  = 2) (Table  2 ).

Summary of findings from review evaluating systematic review methods

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n  = 4), Hopewell et al. 20 and Crumley et al. 19 ( n  = 30), and Robson et al. 14 and Morissette et al. 23 ( n  = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table  2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration  or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]