literature review impact analysis

• Open access
• Published: 18 March 2015

A narrative review of research impact assessment models and methods

• Andrew J Milat 1 , 2 ,
• Adrian E Bauman 2 &
• Sally Redman 2 , 3  

Health Research Policy and Systems volume  13 , Article number:  18 ( 2015 ) Cite this article

24k Accesses

92 Citations

36 Altmetric

Metrics details

Research funding agencies continue to grapple with assessing research impact. Theoretical frameworks are useful tools for describing and understanding research impact. The purpose of this narrative literature review was to synthesize evidence that describes processes and conceptual models for assessing policy and practice impacts of public health research.

The review involved keyword searches of electronic databases, including MEDLINE, CINAHL, PsycINFO, EBM Reviews, and Google Scholar in July/August 2013. Review search terms included ‘research impact’, ‘policy and practice’, ‘intervention research’, ‘translational research’, ‘health promotion’, and ‘public health’. The review included theoretical and opinion pieces, case studies, descriptive studies, frameworks and systematic reviews describing processes, and conceptual models for assessing research impact. The review was conducted in two phases: initially, abstracts were retrieved and assessed against the review criteria followed by the retrieval and assessment of full papers against review criteria.

Thirty one primary studies and one systematic review met the review criteria, with 88% of studies published since 2006. Studies comprised assessments of the impacts of a wide range of health-related research, including basic and biomedical research, clinical trials, health service research, as well as public health research. Six studies had an explicit focus on assessing impacts of health promotion or public health research and one had a specific focus on intervention research impact assessment. A total of 16 different impact assessment models were identified, with the ‘payback model’ the most frequently used conceptual framework. Typically, impacts were assessed across multiple dimensions using mixed methodologies, including publication and citation analysis, interviews with principal investigators, peer assessment, case studies, and document analysis. The vast majority of studies relied on principal investigator interviews and/or peer review to assess impacts, instead of interviewing policymakers and end-users of research.

Conclusions

Research impact assessment is a new field of scientific endeavour and there are a growing number of conceptual frameworks applied to assess the impacts of research.

Peer Review reports

There is increasing recognition that health research investment should lead to improvements in policy [ 1 - 3 ], practice, resource allocation, and, ultimately, the health of the community [ 4 , 5 ]. However, research impacts are complex, non-linear, and unpredictable in nature and there is a propensity to ‘count what can be easily measured’, rather than measuring what ‘counts’ in terms of significant, enduring changes [ 6 ].

Traditional academic-oriented indices of research productivity, such as number of papers, impact factors of journals, citations, research funding, and esteem measures, are well established and widely used by research granting bodies and academic institutions [ 7 ], but they do not always relate well to the ultimate goals of applied health research [ 6 , 8 , 9 ]. Governments are signaling that research metrics of research quality and productivity are insufficient to determine research value because they say little about the real world benefits of research [ 10 - 12 ]. At the same time, research funders continue to grapple with the fundamental problem of assessing broader impacts of research. This task is made more challenging because there are currently no agreed systematic approaches to measuring broader research impacts, particularly impacts on health policy and practice [ 13 , 14 ].

Recent years have seen the development of a number of frameworks that can assist in better describing and understanding the impact of research. Conceptual frameworks can help organize data collection, analysis, and reporting to promote clarity and consistency in the impact assessments made. In the context of this review, research impact is defined as: “… any type of output of research activities which can be considered a ‘positive return’ for the scientific community, health systems, patients, and the society in general ” [ 13 ], p. 2.

In light of these gaps in the literature, the purpose of this narrative literature review was to synthesize evidence that describes processes and conceptual models for assessing research impacts, with a focus on policy and practice impacts of public health research.

Literature review search strategy

The review involved keyword searches of electronic databases including MEDLINE (general medicine), CINAHL (nursing and allied health), PsycINFO (psychology and related behavioural and social sciences), EBM Reviews, Cochrane Database of Systematic Reviews 2005 to May 2013, and Google Scholar. Review search terms included ‘research impact’ OR ‘policy and practice’ AND ‘intervention research’ AND ‘translational research’ AND ‘health promotion’ AND ‘public health’.

The review included theoretical and opinion pieces, case studies, descriptive studies, frameworks and systematic reviews describing processes, and conceptual models for assessing research impact.

The review was conducted in two phases in July/August 2013. In phase 1, abstracts were retrieved and assessed against the review criteria. For abstracts that met the review criteria in phase 1, full papers were retrieved and were assessed for inclusion in the final review. Studies included in the review met the following criteria: i) published in English from January 1990 to June 2013; ii) described processes, theories, or frameworks associated with the assessment of research impact; and iii) were theoretical and opinion pieces, case studies, descriptive studies, frameworks, or systematic reviews.

Due the dearth of public health and health promotion-specific research impact assessment, papers with a focus on clinical or health services research impact assessment were included. The reference lists of the final papers were checked to ensure inclusion of further relevant papers; where such articles were considered relevant, they were included in the review. The search process is shown in Figure  1 .

Literature search process and numbers of papers identified, excluded, and included in the review of research impact assessment.

Findings of the literature review

An initial review of abstracts in electronic databases against the inclusion criteria yielded 431 abstracts and searches of reference lists and the grey literature identified a further 9 documents. Of the 434 abstracts and documents reviewed, 39 met the inclusion criteria and full papers were retrieved. Upon review of the full publications against the review criteria, a further 7 papers were excluded as they did not meet the review criteria, leaving 32 publications in the review [ 8 , 9 , 13 , 15 - 44 ]. A summary of characteristics of studies included in the review that have a focus on processes, theories, or frameworks associated with the assessment of research impact including reference details, study type, domains of impact, methods and indicators, frameworks applied or proposed, and key lessons learned is provided in Additional file 1 : Table S1.

Study characteristics

The review identified 31 primary studies and 1 systematic review that met the review criteria. Six of the studies were reports found in the grey literature. Interestingly, 88% of studies that met the review criteria were published since 2006. The studies in the review included assessments of the impacts of a wide range of health-related research, including basic and biomedical research, clinical trials, health service research, as well as public health research. Six studies [ 22 , 23 , 34 , 36 , 40 , 43 ] had an explicit focus on assessing impacts of health promotion or public health research and 1 had a specific focus on intervention research impact assessment [ 36 ].

The majority of studies were conducted in Australia, United Kingdom, and North America, noting that the review was limited to studies published in English. The unit of assessment varied greatly from researchers (research teams [ 22 ] to whole institutions [ 15 ]) to research disciplines (e.g., prevention research [ 23 ], cancer research [ 41 ], tobacco control research [ 43 ]) or type of grants, for example, from public funding bodies [ 17 , 24 ]. The most frequently applied research methods across studies in rank order were publication and citation analysis, interviews with principal investigators, peer assessment, case studies, and document analysis. The nature of frameworks and methods used to measure research impacts will now be examined in greater detail.

Frameworks and methods for measuring research impacts

Indices of traditional research productivity such as number of papers, impact factors of journals, and citations figured prominently in studies in the literature review [ 18 , 23 , 41 ].

Across the majority of studies in this review, research impact was assessed using multiple dimensions and methodological approaches. A total of 16 different impact assessment models were identified, with the ‘payback model’ being the most frequently used conceptual framework [ 15 , 24 , 29 , 31 , 44 ]. Other frequently used models included health economics frameworks [ 19 , 21 , 37 ], variants of Research Program Logic Models [ 9 , 35 , 42 ], and the Research Impact Framework [ 8 , 30 ]. A number of recent frameworks, including the Health Services Research Impact Framework [ 20 ] and the Banzi Health Research Impact Framework [ 13 , 34 , 36 ], are hybrids of previous conceptual approaches and categorize impacts and benefits in many dimensions, trying to integrate them. Commonly applied frameworks identified in the review, including the Payback model, Research Impact Framework, health economics models, and the new hybrid Health Research Impact Framework, will now be examined in greater detail.

The payback model was developed by Buxton and Hanney [ 45 ] and takes into account resources, research processes, primary outputs, dissemination, secondary outputs and applications, and benefits or final outcomes provided by the research. Categories of outcome in the ‘payback’ framework include i) knowledge production (journal articles, books/book chapters, conference proceeding, reports); ii) use of research in the research system (acquisition of formal qualifications by members of the research team, career advancement, and use of project findings for methodology in subsequent research); iii) use of research project findings in health system policy/decision making (findings used in policy/decision making at any level of the health service such as geographic level and organisation level); iv) application of the research findings through changed behaviour (changes in behaviour observed or expected through the application of findings to research-informed policies at a geographical, organisation and population level); v) factors influencing the utilization of research (impact of research dissemination in terms of policy/decision making/behavioural change); and vi) health/health service/economic benefits (improved service delivery, cost savings, improved health, or increased equity).

The model is usually applied as a semi-structured interview guide for researchers to identify the impact of their research and is often accompanied by bibliometric analysis and verification processes. The payback categories have been found to be applicable to assessing impact of research [ 15 , 24 , 29 ], especially the more proximal impacts on knowledge production, research targeting, capacity building and absorption, and informing practice, policy, and product development. The model has been found to be less effective in eliciting information about the longer term categories of impact on health and health sector benefits and economics [ 29 ].

The Research Impact Framework was developed in the UK by Kuruvilla et al. [ 8 , 30 ], and draws upon both the research impact literature and UK research assessment criteria for publically funded research, and was validated through empirical analysis of research projects at the London School of Hygiene & Tropical Medicine. The framework is built around four categories of impact, namely i) research related, ii) policy, iii) service, and iv) societal. Within each of these areas, further descriptive categories are identified. For example, the nature of research impact on policy can be described using the Weiss categorisation of ‘instrumental use’, where research findings drive policy-making; ‘mobilisation of support’, where research provides support for policy proposals; ‘conceptual use’, where research influences the concepts and language of policy deliberations; and ‘redefining/wider influence’, where research leads to rethinking and changing established practices and beliefs [ 30 ]. The framework is applied as a semi-structured interview guide for researchers to identify the impact of their research. Users of the framework have reported that it enables the systematic identification of a range of specific and verifiable impacts and allows consideration of the unintended effects of research [ 30 ].

The framework proposed by Banzi et al. [ 13 ] is an adaption of the Canadian Academy of Health Science impact model [ 25 ] in light of a systematic review and includes five broad categories of research impact, namely i) advancing knowledge, ii) capacity building, iii) informing decision-making, iv) health and other sector benefits, and v) broad socio-economic benefits. The Banzi framework proposes a set of indicators for each domain. To illustrate, indicators for informing decision making include citation in guidelines, policy documents, and plans; references used as background for successful funding proposals; consulting, support activity, and contributing to advisory committees; patents and industrial collaboration; packages of material and communication to key target audiences about findings. This multidimensional framework takes into account several aspects of research impact and use, as well as comprehensive analytical approaches including bibliometric analysis, surveys, audit, document review, case studies, and panel assessment. Panel assessments generally involve a process asking experts to assess the merits of research against impact criteria.

Economic models used to assess impacts of research varied from cost benefit analysis to return on investment and employed a variety of methods for determining economic benefits of research. The National Institutes of Medicine study in 1993 was among the first studies to attempt to systematically monetize the benefits of medical research. It provided estimates of savings for health care systems (direct costs) and savings for the community as a whole (indirect costs), and quantified benefits in terms of quality adjusted life years. On the other hand, the Deloitte Access Economics study [ 21 ] built on the foundations of the 1993 analysis to estimate the returns on investment in research in Australia for the main disease areas and employed of health system expenditure modelling and monetised total quality adjusted life years gained. According to Buxton et al. [ 19 ], measuring only health care savings is generally seen as too narrow a focus, and their analysis considered the benefits, or indirect cost savings, in avoiding lost production and the further activity stimulated by research.

The aforementioned models all attempted to quantify a mix of more proximal research and policy and practice impacts, as well as more distal societal and economic benefits of research. It is also interesting to note that across the studies in this review, only four [ 16 , 29 , 34 , 36 ] interviewed non-academic end-users of research in impact assessment processes, with the vast majority of studies relying on principal investigator interviews and/or peer review processes to assess impacts.

Comprehensive monitoring and measurement of research impact is a complex undertaking requiring the involvement of many actors within the research pipeline [ 13 ]. Interestingly, 90% of studies that met the review criteria were published since 2006, indicating that this is a new field of research. Given the dearth of literature on public health research impact assessment, this review included assessments of the impacts of a wide range of health-related research, including basic and biomedical research, clinical trials, and health service research as well as public health research.

The review of both the published and grey literature also revealed that there are a number of conceptual frameworks currently being applied that describe processes of assessing research impact. These frameworks differ in their terminology and approaches. The lack of a common understanding of terminology and metrics makes the task of quantifying research efforts, outputs, and, ultimately, performance in this area more difficult.

Most of the models identified in the review used multidimensional conceptualization and categorization of research impact. These multidimensional models, such as the Payback model, Research Impact Framework, and Banzi Health Research Impact Framework, shared common features including assessment of traditional research outputs, such as publication and research funding, but also a broader range of potential benefits, including capacity, building, policy and product development, and service development, as well as broader societal and economic impacts. Assessments that considered more than one category were valued for their ability to capture multifaceted impact processes [ 13 , 36 , 44 ]. Interestingly, these frameworks recognised that research often impacts not only in the country within which research is conducted, but also internationally. However, for practical reasons, most studies limited assessment and verification of impacts to a single country [ 19 , 34 , 36 ].

Several methods were used to practically assess research impact, including desk analysis, bibliometrics, panel assessments, interviews, and case studies. A number of studies highlighted the utility of case study methods noting that a considerable range of research paybacks and perspectives would not have been identified without employing a structured case study approach [ 13 , 36 , 44 ]. However, it was noted that case studies can be at risk of ‘conceptualization bias’ and ‘reporting bias’ especially when they are designed or carried out retrospectively [ 13 ]. The costs of conducting case studies can also be a barrier when assessing large volumes of research [ 13 , 36 ].

Despite recent efforts, little is known about the nature and mechanisms that underpin the influence that health research has on health policy or practice. This review suggests that, to date, most primary studies of health research impacts have been small scale case studies or reviews of medical and health services research funding [ 27 , 31 , 35 , 39 , 41 ], with only two studies offering comprehensive assessments of the policy and practice impacts of public health research, with both focusing on prevention research in Australia.

The first of these aforementioned studies examined impact of population health surveillance studies on obesity prevention policy and practice [ 34 ], while the second [ 36 ] examined the policy and practice impacts of intervention research funded through the NSW Health Promotion Demonstration Research Grants Scheme 2000–2006. Both of these studies utilised comprehensive mixed methods to assess impacts that included semi-structured interviews with both investigators and end-users, bibliometric analysis, document review, verification processes, and case studies. These studies concluded that research projects can achieve the greatest policy and practice impacts if they address proximal needs of the policy context by engaging end-users from the inception of research projects and utilizing existing policy networks and structures, as well as using a range of strategies to disseminate findings that go beyond traditional peer review publications.

This review suggests that the research sector often still uses bibliometric indices to assess research impacts, rather than measuring more enduring and arguably more important policy and practice outcomes [ 6 ]. However, governments are increasingly signaling that research metrics of research quality are insufficient to determine research value because they say little about real world benefits of research [ 10 - 12 ]. The Australian Excellence in Innovation trial [ 26 ] and the UK’s Research Excellence Framework trials [ 28 , 46 ] were commissioned by governments to determine the public benefit from research spending [ 10 , 16 , 47 ].

These attempts raise an important question of how to construct an impact assessment process that can assess multi-dimensional impacts while being feasible to implement on a system level. For example, can 28 indicators across 4 domains of Research Impact Framework be realistically measured in practice? This could also be said of the Research Impact Model [ 13 ], which has 26 indicators, and the Research Excellent Framework by Ovseiko et al. [ 38 ], which has a total of 20 impact indicators. If such methods are to be widely used in practice by research funders and academic institutions to assess research impacts, the right balance between comprehensiveness and feasibility must be struck.

Though a number of studies suggest it is difficult to determine longer-term societal and economic benefits of research as part of multi-dimensional research impact assessment processes [ 13 , 36 , 44 ], the health economic impact models presented in this review and the broader literature demonstrate that it is feasible to undertake these analyses, particularly if the right methods are used [ 19 , 21 , 37 , 48 ].

The review revealed that, where broader policy and practice impacts of research have been assessed in the literature, the vast majority of studies have relied on principal investigator interviews and/or peer review to assess impacts, instead of interviewing policymakers and other important end-users of research. This would seem to be a methodological weakness of previous research, as solely relying on principal investigator assessments, particularly of impacts of their own research, has an inherent bias, leaving the research impact assessment process open to ‘gilding the lily’. In light of this, future impact assessment processes should routinely engage end-users of research in interviews and assessment processes, but also include independent documentary verification, thus addressing methodological limitations of previous research.

One of the greatest practical issues in measuring research impact, including the impact of public health research, are the long lag times before impacts manifest. It has been observed that, on average, it takes over 6 years for research evidence to reach reviews, papers, and textbooks, and a further 9 years for this evidence to be implemented into practice [ 49 ]. In light of this, it is important to allow sufficient time for impacts to manifest, while not waiting so long that these impacts cannot be verified by stakeholders involved in the production and use of the research. Studies in this review have addressed this issue by only assessing studies that had been completed for at least 24 months [ 36 ].

As identified in previous research [ 13 ], a major challenge is attribution of impacts and understanding what would have happened without individual research activity or what some describe as the ‘counterfactual’. Creating a control situation for this type of research is difficult, but, where possible, identification of baseline measures and contextual factors is important in understanding what counterfactual situations may have arisen. Confidence in attribution of effects can be improved by undertaking independent verification of processes and engaging end-users in assessments instead of solely relying on investigators accounts of impacts [ 36 ].

The research described in this review has some limitations that merit closer examination. Given the paucity of research in this area, review criteria had to be adjusted to include assessment of impacts beyond public health research to include all health research. It was also challenging to make direct comparisons across studies mostly due to the heterogeneity of studies and the lack of a standard terminology, hence the broad definition of ‘research impact’ finally applied in the review criteria. Although the majority of studies were found in the traditional biomedical databases (i.e., Medline, etc.), 18% were found in the grey literature highlighting the importance of using multiple data sources in future review processes. Another methodological limitation also identified in previous reviews [ 13 ], is that we did not estimate the level of publication bias and selective publication in this emerging field. Finally, as our analysis included studies published up to June 2013, we may not have captured more recent approaches to impact assessment.

Research impact assessment is a new field of scientific endeavour and typically impacts are assessed using mixed methodologies, including publication and citation analysis, interviews with principal investigators, peer assessment, case studies, and document analysis. The literature is characterised by an over reliance on bibliometric methods to assess research impact. Future impact assessment processes could be strengthened by routinely engaging the end-users of research in interviews and assessment processes. If multidimensional research impact assessment methods are to be widely used in practice by research funders and academic institutions, the right balance between comprehensiveness and feasibility must be determined.

Anderson W, Papadakis E. Research to improve health practice and policy. Med J Aust. 2009;191(11/12):646–7.

PubMed   Google Scholar  

Cooksey D. A review of UK health research funding. London: HMSO; 2006.

Google Scholar  

Health and Medical Research Strategic Review Committee. The virtuous cycle: working together for health and medical research. Canberra: Commonwealth of Australia; 1998.

National Health and Medical Research Council Public Health Advisory Committee. Report of the Review of Public Health Research Funding in Australia. Canberra: NHMRC; 2008.

Campbell DM. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009;6:21.

PubMed   PubMed Central   Google Scholar  

Wells R, Whitworth JA. Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Aust New Zealand Health Policy. 2007;4:14.

Australian Government Australian Research Council. Excellence in Research in Australia 2012. Canberra: Australian Research Council; 2012.

Kuruvilla S, Mays N, Walt G. Describing the impact of health services and policy research. J Health Serv Res Policy. 2007;12 Suppl 1:S1. -23-31.

Weiss AP. Measuring the impact of medical research: moving from outputs to outcomes. Am J Psychiatr. 2007;164(2):206–14.

Bornmann L. Measuring the societal impact of research. Eur Mol Biol Organ. 2012;13(8):673–6.

CAS   Google Scholar  

Holbrook JB. Re-assessing the science–society relation: The case of the US National Science Foundation’s broader impacts merit review criterion (1997–2011). In: Frodeman R, Holbrook JB, Mitcham C, Xiaonan H, editors. Peer Review, Research Integrity, and the Governance of Science–Practice, Theory, and Current Discussions. Dalian: People’s Publishing House and Dalian University of Technology; 2012. p. 328–62.

Holbrook JB, Frodeman R. Science’s social effects. Issues in Science and Technology. 2007. http://issues.org/23-3/p_frodeman-3/ .

Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. health Res Policy Syst. 2011;9:26.

Boaz A, Fitzpatrick S, Shaw B. Assessing the impact of research on policy: A review of the literature for a project on bridging research and policy through outcome evaluation. London: Policy Studies Institute London; 2008.

Aymerich M, Carrion C, Gallo P, Garcia M, López-Bermejo A, Quesada M, et al. Measuring the payback of research activities: a feasible ex-post evaluation methodology in epidemiology and public health. Soc Sci Med. 2012;75(3):505–10.

Barber R, Boote JD, Parry GD, Cooper CL, Yeeles P, Cook S. Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expect. 2012;15(3):229–41.

Barker K, The UK. Research Assessment Exercise: the evolution of a national research evaluation system. Res Eval. 2007;16(1):3–12.

Boyack KW, Jordan P. Metrics associated with NIH funding: a high-level view. J Am Med Inform Assoc. 2011;18(4):423–31.

Buxton M, Hanney S, Morris S, Sundmacher L, Mestre-Ferrandiz J, Garau M, et al. Medical research: what’s it worth. Estimating the economic benefits from medical research in the UK. Report for MRC, Wellcome Trust and the Academy of Medical Sciences. 2008. http://www.wellcome.ac.uk/stellent/groups/corporatesite/@sitestudioobjects/documents/web_document/wtx052110.pdf .

Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, et al. ‘Making evidence count’: A framework to monitor the impact of health services research. Aust J Rural Health. 2012;20(2):51–8.

Deloitte Access Economics. Extrapolated returns on investment in NHMRC medical research. Canberra: Australian Society for Medical Research; 2012.

Derrick GE, Haynes A, Chapman S, Hall WD. The association between four citation metrics and peer rankings of research influence of Australian researchers in six fields of public health. PLoS One. 2011;6(4):e18521.

CAS   PubMed   PubMed Central   Google Scholar  

Franks AL, Simoes EJ, Singh R, Gray BS. Assessing prevention research impact: a bibliometric analysis. Am J Prev Med. 2006;30(3):211–6.

Graham KE, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: development and implementation of the Alberta Innovates–Health Solutions impact framework. Res Eval. 2012;21(5):354–67.

Canadian Institutes of Health Research. Developing a CIHR framework to measure the impact of health research. Ottawa: Canadian Institutes of Health Research; 2005.

Group of Eight. Excellence in innovation: research impacting our nation’s future – assessing the benefits. Adelaide: Australian Technology Network of Universities; 2012.

Hanney S. An assessment of the impact of the NHS Health Technology Assessment Programme. Southampton: National Coordinating Centre for Health Technology Assessment, University of Southampton; 2007.

Higher Education Funding Council for England. Panel criteria and working methods. London: Higher Education Funding Council for England; 2012.

Kalucy EC, Jackson-Bowers E, McIntyre E, Reed R. The feasibility of determining the impact of primary health care research projects using the Payback Framework. Health Res Policy Syst. 2009;7:11.

Kuruvilla S, Mays N, Pleasant A, Walt G. Describing the impact of health research: a Research Impact Framework. BMC Health Serv Res. 2006;6(1):134.

Kwan P, Johnston J, Fung AYK, Chong DSY, Collins RA, Lo SV. A systematic evaluation of payback of publicly funded health and health services research in Hong Kong. BMC Health Serv Res. 2007;7(1):121.

Landry R, Amara N, Lamari M. Climbing the ladder of research utilization: Evidence from social science research. Sci Commun. 2001;22:396–422.

Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy. 2003;8(3):165–70.

Laws R, King L, Hardy LL, Milat AJ, Rissel C, Newson R, et al. Utilization of a population health survey in policy and practice: a case study. Health Res Policy Syst. 2013;11:4.

Liebow E, Phelps J, Van Houten B, Rose S, Orians C, Cohen J, et al. Toward the assessment of scientific and public health impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environ Health Perspect. 2009;117(7):1147.

Milat AJ, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Syst. 2013;11:5.

National Institutes of Health. Cost savings resulting from NIH research support. Bethesda, MD: United States Department of Health and Human Services National Institute of Health; 1993.

Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators. BMC Health Serv Res. 2012;12:478.

Schapper CC, Dwyer T, Tregear GW, Aitken M, Clay MA. Research performance evaluation: the experience of an independent medical research institute. Aust Health Rev. 2012;36(2):218–23.

Spoth RL, Schainker LM, Hiller-Sturmhöefel S. Translating family-focused prevention science into public health impact: illustrations from partnership-based research. Alcohol Res Health. 2011;34(2):188.

Sullivan R, Lewison G, Purushotham AD. An analysis of research activity in major UK cancer centres. Eur J Cancer. 2011;47(4):536–44.

CAS   PubMed   Google Scholar  

Taylor J, Bradbury-Jones C. International principles of social impact assessment: lessons for research? J Res Nurs. 2011;16(2):133–45.

Warner KE, Tam J. The impact of tobacco control research on policy: 20 years of progress. Tob Control. 2012;21(2):103–9.

Wooding S, Hanney S, Buxton M, Grant J. The returns from arthritis research. Volume 1: Approach analysis and recommendations. Netherlands: RAND Europe; 2004.

Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy. 1996;1(1):35–43.

Higher Education Funding Council for England. Decisions on assessing research impact. Bristol: Higher Education Funding Council for England; 2011.

Grant J, Brutscher P-B, Kirk SE, Butler L, Wooding S. Capturing research impacts: a review of international practice. Documented Briefing. RAND Corporation; 2010. http://www.rand.org/pubs/documented_briefings/DB578.html .

Murphy KM, Topel RH. Measuring the gains from medical research: an economic approach. Chicago: University of Chicago Press; 2010.

Balas EA, Boren SA. Managing clinical knowledge for health care improvement. In: Bemmel J, McCray AT, editors. Yearbook of Medical Informatics 2000: Patient-Centered Systems. Stuttgart, Germany: Schattauer Verlagsgesellschaft mbH; 2000. p. 65–70.

Download references

Author information

Authors and affiliations.

New South Wales Ministry of Health, 73 Miller St North, Sydney, NSW, 2060, Australia

Andrew J Milat

School of Public Health, University of Sydney, Level 2, Medical Foundation, Building, K25, Sydney, NSW, 2006, Australia

Andrew J Milat, Adrian E Bauman & Sally Redman

Sax Institute, Sydney, Level 2, 10 Quay, St Haymarket, NSW, 2000, Australia

Sally Redman

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Andrew J Milat .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

AJM conceived the study, designed the methods, and conducted the literature searches. AJM drafted the manuscript and all authors contributed to data interpretation and have read and approved the final manuscript.

Additional file

Additional file 1: table s1..

Characteristics of studies focusing on processes, theories, or frameworks assessing research impact.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/4.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Milat, A.J., Bauman, A.E. & Redman, S. A narrative review of research impact assessment models and methods. Health Res Policy Sys 13 , 18 (2015). https://doi.org/10.1186/s12961-015-0003-1

Download citation

Received : 07 November 2014

Accepted : 16 February 2015

Published : 18 March 2015

DOI : https://doi.org/10.1186/s12961-015-0003-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Policy and practice impact
• Research impact
• Research returns

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Skip to main content
• Accessibility help

Information

We use cookies to collect anonymous data to help us improve your site browsing experience.

Click 'Accept all cookies' to agree to all cookies that collect anonymous data. To only allow the cookies that make the site work, click 'Use essential cookies only.' Visit 'Set cookie preferences' to control specific cookies.

Your cookie preferences have been saved. You can change your cookie settings at any time.

Impact assessment in governments: literature review

This report reviews literature regarding five types of policy level impact assessments (environment, equity, health, regulatory, rural) in five countries (Ireland, Netherlands, New Zealand, Sweden & Wales). It was commissioned by the Scottish Government to inform their approach to impact assessment.

Key messages from this review

This literature review was commissioned by the Scottish Government to inform their approach to impact assessment. We reviewed the literature regarding five types of policy level impact assessments (environment, equity, health, regulatory, rural) in five countries (Ireland, Netherlands, New Zealand, Sweden and Wales). These countries were most likely to require regulatory impact assessment, and least likely to require rural proofing.

More than 1000 potentially useful documents were identified using search engines. Of these, more than 110 plus legislation and guidance informed this report. Much of the literature is somewhat dated; relies on a limited number of case studies; and is carried out by academics who may be testing a hypothesis rather than presenting a balanced view. As such, the findings of this research need to be taken with caution.

What types of assessments are carried out? Scotland has more different types of impact assessment than any other countries studied. New Zealand has climate impact assessment and Wales has wellbeing of future generations assessment, neither of which is carried out in Scotland. Several countries have integrated impact assessments ( e.g. Ireland's RIA , Wales' wellbeing assessment).

What assessment systems are particularly interesting? Welsh wellbeing assessment is interesting because it covers a wide range of impacts, is clearly future-looking, and seems to have strong government support. US 'environmental justice' assessment brings together environmental, health and equality dimensions, and seems effective at leading to changes. These assessment systems apply at the programme or plan level, rather than at policy level.

Are assessments actually carried out? Legally-required assessments are generally carried out, but based on evidence from this review many seem to be a formality, carried out late and/or with little influence on policy-making. However, gaps in the literature were identified relating to the timing of actual assessments and how their findings are used in policy-making.

How effective are the assessments that are carried out? In terms of:

• changes to policies – assessment effectiveness is mixed/limited
• public participation – this is very important for transparency and policy improvement. In practice public engagement is limited, but stakeholder engagement is more common.
• knowledge and learning – there is often learning by policy-makers, with consequent long-term organisational change
• costs v. benefits – not enough information exists to be able to come to a conclusion

In particular, even where an impact assessment does not lead to changes in a policy, it can have benefits in terms of improved transparency and accountability of decision-making, increased awareness of the public, and increased trust between stakeholders.

Is integration of impact assessments advisable? Integration of impact assessments – for instance bringing together environmental, social and economic impact assessment into a 'sustainability assessment' - may promote a more holistic approach to assessment, but care needs to be taken in terms of which elements get the most emphasis. Integration is not just a matter of new legal requirements and guidance: it involves issues of data availability, the number of indicators to use, terminology and frames of reference, build-up of expertise, and intersectoral cooperation. The level of integration depends on issues like what minimum standards or thresholds must be achieved and what trade-offs are permitted.

There is also the 'detail paradox', which states that the power of each objective diminishes with the addition of other objectives: in other words, the more detailed the assessment is, the less significance, on average, is attached to each detail.

What are preconditions for effective assessment? In rough order of importance:

• High-level commitment and supportive organisations
• Policy-makers' willingness to learn and change in response to the assessment findings
• Legal requirement for the impact assessment to be carried out
• Oversight and quality review of the assessments
• Fitting the assessment to the decision in terms of timing, types of alternatives considered, recommendations etc.
• Involvement of the public/stakeholders
• Starting the impact assessment early in the policy-making process
• Adequate funding
• Adequate data and expertise
• Collaboration and information sharing between assessors and government departments
• Follow up to check whether the policy incorporated the assessment recommendations, whether the assessment adequately identified impacts, and how the assessment process can be improved

Email: [email protected]

There is a problem

Thanks for your feedback

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Health Res Policy Syst

Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment

Catherine r. hanna.

1 CRUK Clinical Trials Unit, Institute of Cancer Sciences, University of Glasgow, Glasgow, United Kingdom

Kathleen A. Boyd

2 Health Economics and Health Technology Assessment, Institute of Health and Wellbeing, University of Glasgow, Glasgow, United Kingdom

Robert J. Jones

Associated data.

Additional files included. No primary research data analysed.

Performing cancer research relies on substantial financial investment, and contributions in time and effort from patients. It is therefore important that this research has real life impacts which are properly evaluated. The optimal approach to cancer research impact evaluation is not clear. The aim of this study was to undertake a systematic review of review articles that describe approaches to impact assessment, and to identify examples of cancer research impact evaluation within these reviews.

In total, 11 publication databases and the grey literature were searched to identify review articles addressing the topic of approaches to research impact assessment. Information was extracted on methods for data collection and analysis, impact categories and frameworks used for the purposes of evaluation. Empirical examples of impact assessments of cancer research were identified from these literature reviews. Approaches used in these examples were appraised, with a reflection on which methods would be suited to cancer research  impact evaluation going forward.

In total, 40 literature reviews were identified. Important methods to collect and analyse data for impact assessments were surveys, interviews and documentary analysis. Key categories of impact spanning the reviews were summarised, and a list of frameworks commonly used for impact assessment was generated. The Payback Framework was most often described. Fourteen examples of impact evaluation for cancer research were identified. They ranged from those assessing the impact of a national, charity-funded portfolio of cancer research to the clinical practice impact of a single trial. A set of recommendations for approaching cancer research impact assessment was generated.

Conclusions

Impact evaluation can demonstrate if and why conducting cancer research  is worthwhile. Using a mixed methods, multi-category assessment organised within a framework, will provide a robust evaluation, but the ability to perform this type of assessment may be constrained by time and resources. Whichever approach is used, easily measured, but inappropriate metrics should be avoided. Going forward, dissemination of the results of cancer research impact assessments will allow the cancer research community to learn how to conduct these evaluations.

Supplementary information

Supplementary information accompanies this paper at 10.1186/s12961-020-00658-x.

Cancer research attracts substantial public funding globally. For example, the National Cancer Institute (NCI) in the United States of America (USA) had a 2020 budget of over $6 billion United States (US) dollars. In addition to public funds, there is also huge monetary investment from private pharmaceutical companies, as well as altruistic investment of time and effort to participate in cancer research from patients and their families. In the United Kingdom (UK), over 25,000 patients were recruited to cancer trials funded by one charity (Cancer Research UK (CRUK)) alone in 2018 [ 1 ]. The need to conduct research within the field of oncology is an ongoing priority because cancer is highly prevalent, with up to one in two people now having a diagnosis of cancer in their lifetime [ 2 , 3 ], and despite current treatments, mortality and morbidity from cancer are still high [ 2 ].

In the current era of increasing austerity, there is a desire to ensure that the money and effort to conduct any type of research delivers tangible downstream benefits for society with minimal waste [ 4 – 6 ]. These wider, real-life benefits from research are often referred to as research impact. Given the significant resources required to conduct cancer research in particular, it is reasonable to question if this investment is leading to the longer-term benefits expected, and to query the opportunity cost of not spending the same money directly within other public sectors such as health and social care, the environment or education.

The interest in evaluating research impact has been rising, partly driven by the actions of national bodies and governments. For example, in 2014, the UK government allocated its £2 billion annual research funding to higher education institutions, in part based on an assessment of the impact of research performed by each institution in an assessment exercise known as the Research Excellence Framework (REF). The proportion of funding dependent on impact assessment will increase from 20% in 2014, to 25% in 2021[ 7 ].

Despite the clear rationale and contemporary interest in research impact evaluation, assessing the impact of research comes with challenges. First, there is no single definition of what research impact encompasses, with potential differences in the evaluation approach depending on the definition. Second, despite the recent surge of interest, knowledge of how best to perform assessments and the infrastructure for, and experience in doing so, are lagging [ 6 , 8 , 9 ]. For the purposes of this review, the definition of research impact given by the UK Research Councils is used (see Additional file 1 for full definition). This definition was chosen because it takes a broad perspective, which includes academic, economic and societal views of research impact [ 10 ].

There is a lack of clarity on how to perform research impact evaluation, and this extends to cancer research. Although there is substantial interest from cancer funders and researchers [ 11 ], this interest is not accompanied by instruction or reflection on which approaches would be suited to assessing the impact of cancer research specifically. In a survey of Australian cancer researchers, respondents indicated that they felt a responsibility to deliver impactful research, but that evaluating and communicating this impact to stakeholders was difficult. Respondents also suggested that the types of impact expected from research, and the approaches used, should be discipline specific [ 12 ]. Being cognisant of the discipline specific nature of impact assessment, and understanding the uniqueness of cancer research in approaching such evaluations, underpins the rationale for this study.

The aim of this study was to explore approaches to research impact assessment, identify those approaches that have been used previously for cancer research, and to use this information to make recommendations for future evaluations. For the purposes of this study, cancer research included both basic science and applied research, research into any malignant disease, concerning paediatric or adult cancer, and studies spanning nursing, medical, public health elements of cancer research.

The study objectives were to:

• i. Identify existing literature reviews that report approaches to research impact assessment and summarise these approaches.
• ii. Use these literature reviews to identify examples of cancer research impact evaluations, describe the approaches to evaluation used within these studies, and compare them to those described in the broader literature.

This approach was taken because of the anticipated challenge of conducting a primary review of empirical examples of cancer research impact evaluation, and to allow a critique of empirical studies in the context of lessons learnt from the wider literature. A primary review would have been difficult because examples of cancer research impact evaluation, for example, the assessment of research impact on clinical guidelines [ 13 ], or clinical practice [ 14 – 16 ], are often not categorised in publication databases under the umbrella term of research impact. Reasons for this are the lack of medical subject heading (MeSH) search term relating to research impact assessment and the differing definitions for research impact. In addition, many authors do not recognise their evaluations as sitting within the discipline of research impact assessment, which is a novel and emerging field of study.

General approach

A systematic search of the literature was performed to identify existing reviews of approaches to assess the impact of research. No restrictions were placed on the discipline, field, or scope (national/global) of research for this part of the study. In the second part of this study, the reference lists of the literature reviews identified were searched to find empirical examples of the evaluation of the impact of cancer research specifically.

Data sources and searches

For the first part of the study, 11 publication databases and the grey literature from January 1998 to May 2019 were searched. The electronic databases were Medline, Embase, Health Management and Policy Database, Education Resources Information Centre, Cochrane, Cumulative Index of Nursing and Allied Health Literature, Applied Social Sciences Index and Abstract, Social Services Abstracts, Sociological Abstracts, Health Business Elite and Emerald. The search strategy specified that article titles must contain the word “impact”, as well as a second term indicating that the article described the evaluation of impact, such as “model” or “measurement” or “method”. Additional file 1 provides a full list of search terms. The grey literature was searched using a proforma. Keywords were inserted into the search function of websites listed on the proforma and the first 50 results were screened. Title searches were performed by either a specialist librarian or the primary researcher (Dr. C Hanna). All further screening of records was performed by the primary researcher.

Following an initial title screen, 800 abstracts were reviewed and 140 selected for full review. Articles were kept for final inclusion in the study by assessing each article against specific inclusion criteria (Additional file 1 ). There was no assessment of the quality of the included reviews other than to describe the search strategy used. If two articles drew primarily on the same review but contributed a different critique of the literature or methods to evaluate impact, both were kept. If a review article was part of a grey literature report, for example a thesis, but was also later published in a journal, the journal article only was kept. Out of 140 articles read in full, 27 met the inclusion criteria and a further 13 relevant articles were found through reference list searching from the included reviews [ 17 ].

For the second part of the study, the reference lists from the literature reviews were manually screened [ 17 ] ( n  = 4479 titles) by the primary researcher to identify empirical examples of assessment of the impact of cancer research. Summary tables and diagrams from the reviews were also searched using the words “cancer” and “oncology” to identify relevant articles that may have been missed by reference list searching. After removal of duplicates, 57 full articles were read and assessed against inclusion criteria (Additional file 1 ). Figure  1 shows the search strategy for both parts of the study according to the guidelines for preferred reporting items for systematic reviews and meta-analysis (PRISMA) [ 18 ].

Search strategies for this study

Data extraction and analysis

A data extraction form produced in Microsoft ® Word 2016 was used to collect details for each literature review. This included year of publication, location of primary author, research discipline, aims of the review as described by the authors and the search strategy (if any) used. Information on approaches to impact assessment was extracted under three specific themes which had been identified from a prior scoping review as important factors when planning and conducting research impact evaluation. These themes were: (i) categorisation of impact into different types depending on who or what is affected by the research (the individuals, institutions, or parts of society, the environment), and how they are affected (for example health, monetary gain, sustainability) (ii) methods of data collection and analysis for the purposes of evaluation, and (iii) frameworks to organise and communicate research impact. There was space to document any other key findings the researcher deemed important. After data extraction, lists of commonly described categories, methods of data collection and analysis, and frameworks were compiled. These lists were tabulated or presented graphically and narrative analysis was used to describe and discuss the approaches listed.

For the second part of the study, a separate data extraction form produced in Microsoft ® Excel 2016 was used. Basic information on each study was collected, such as year of publication, location of primary authors, research discipline, aims of evaluation as described by the authors and research type under assessment. Data was also extracted from these empirical examples using the same three themes as outlined above, and the approaches used in these studies were compared to those identified from the literature reviews. Finally, a set of recommendations for future evaluations of cancer research impact were developed by identifying the strengths of the empirical examples and using the lists generated from the first part of the study to identify improvements that could be made.

Part one: Identification and analysis of literature reviews describing approaches to research impact assessment

Characteristics of included literature reviews.

Forty literature reviews met the pre-specified inclusion criteria and the characteristics of each review are outlined in Table ​ Table1. 1 . A large proportion (20/40; 50%) were written by primary authors based in the UK, followed by the USA (5/40; 13%) and Australia (5/40; 13%), with the remainder from Germany (3/40; 8%), Italy (3/40; 8%), the Netherlands (1/40; 3%), Canada (1/40; 3%), France (1/40; 3%) and Iran (1/40; 3%). All reviews were published since 2003, despite the search strategy dating from 1998. Raftery et al. 2016 [ 19 ] was an update to Hanney et al. 2007 [ 20 ] and both were reviews of studies assessing research impact relevant to a programme of health technology assessment research. The narrative review article by Greenhalgh et al. [ 21 ] was based on the same search strategy used by Raftery et al. [ 19 ].

Summary of literature reviews

Approximately half of the reviews (19/40; 48%) described approaches to evaluate research impact without focusing on a specific discipline and nearly the same amount (16/40; 40%) focused on evaluating the impact of health or biomedical research. Two reviews looked at approaches to impact evaluation for environmental research and one focused on social sciences and humanities research. Finally, two reviews provided a critique of impact evaluation methods used by different countries at a national level [ 22 , 23 ]. None of these reviews focused specifically on cancer research.

Twenty-five reviews (25/40; 63%) specified search criteria and 11 of these included a PRISMA diagram. The articles that did not outline a search strategy were often expert reviews of the approaches to impact assessment methods and the authors stated they had chosen the articles included based on their prior knowledge of the topic. Most reviews were found by searching traditional publication databases, however seven (7/40; 18%) were found from the grey literature. These included four reports written by an independent, not-for-profit research institution (Research and Development (RAND) Europe) [ 23 – 26 ], one literature review which was part of a Doctor of Philosophy (Ph.D) thesis [ 27 ], a literature review informing a quantitative study [ 28 ] and a review that provided background information for a report to the UK government on the best use of impact metrics [ 29 ].

Key findings from the reviews: approaches to research impact evaluation

Nine reviews attempted to categorise the type of research impact being assessed according to who or what is affected by research, and how they are affected. In Fig.  2 , colour coding was used to identify overlap between impact types identified in these reviews to produce a summary list of seven main impact categories.

Categories of impact identified in the included literature reviews

The first two categories of impact refer to the immediate knowledge produced from research and the contribution research makes to driving innovation and building capacity for future activities within research institutions. The former is often referred to as the academic impact of research. The academic impact of cancer research may include the knowledge gained from conducting experiments or performing clinical trials that is subsequently disseminated via journal publications. The latter may refer to securing future funding for cancer research, providing knowledge that allows development of later phase clinical trials or training cancer researchers of the future.

The third category identified was the impact of research on policy. Three of the review articles included in this overview specifically focused policy impact evaluation [ 30 – 32 ]. In their review, Hanney et al. [ 30 ] suggested that policy impact (of health research) falls into one of three sub-categories: impact on national health policies from the government, impact on clinical guidelines from professional bodies, and impact on local health service policies. Cruz Rivera and colleagues [ 33 ] specifically distinguished impact on policy making from impact on clinical guidelines, which they described under health impact. This shows that the lines between categories will often blur.

Impact on health was the next category identified and several of the reviews differentiated health sector impact from impact on health gains. For cancer research, both types of health impact will be important given that it is a health condition which is a major burden for healthcare systems and the patients they treat. Economic impact of research was the fifth category. For cancer research, there is likely to be close overlap between healthcare system and economic impacts because of the high cost of cancer care for healthcare services globally.

In their 2004 article, Buxton et al. [ 34 ] searched the literature for examples of the evaluation of economic return on investment in health research and found four main approaches, which were referenced in several later reviews [ 19 , 25 , 35 , 36 ]. These were (i) measuring direct cost savings to the health-care system, (ii) estimating benefits to the economy from a healthy workforce, (iii) evaluating benefits to the economy from commercial development and, (iv) measuring the intrinsic value to society of the health gain from research. In a later review [ 25 ], they added an additional approach of estimating the spill over contribution of research to the Gross Domestic Product (GDP) of a nation.

The final category was social impact. This term was commonly used in a specific sense to refer to research improving human rights, well-being, employment, education and social inclusion [ 33 , 37 ]. Two of the reviews which included this category focused on the impact of non-health related research (social sciences and agriculture), indicating that this type of impact may be less relevant or less obvious for health related disciplines such as oncology. Social impact is distinct from the term societal impact, which was used in a wider sense to describe impact that is external to traditional academic benefits [ 38 , 39 ]. Other categories of impact identified that did not show significant overlap between the reviews included cultural and technological impact. In two of the literature reviews [ 33 , 40 ], the authors provided a list of indicators of impact within each of their categories. In the review by Thonon et al. [ 40 ], only one (1%) of these indicators was specific to evaluating the impact of cancer research.

In total, 36 (90%) reviews discussed methods to collect or analyse the data required to conduct an impact evaluation. The common methods described, and the  strengths and weaknesses of each approach, are shown in Additional file 2 : Table S1. Many authors advocated using a mixture of methods and in particular, the triangulation of surveys, interviews (of researchers or research users), and documentary analysis [ 20 , 30 – 32 ]. A large number of reviews cautioned against the use of quantitative metrics, such as bibliometrics, alone [ 29 , 30 , 41 – 48 ]. Concerns included that these metrics were often not designed to be comparable between research programmes [ 49 ], their use may incentivise researchers to focus on quantity rather than quality [ 42 ], and these metrics could be gamed and used in the wrong context to make decisions about researcher funding, employment and promotion [ 41 , 43 , 45 ].

Several reviews explained that the methods for data collection and analysis chosen for impact evaluation depended on both the unit of research under analysis and the rationale for the impact analysis [ 23 , 24 , 26 , 31 , 36 , 50 , 51 ]. Specific to cancer research, the unit of analysis may be a single clinical trial or a programme of trials, research performed at a cancer centre or research funded by a specific institution or charity. The rationale for research impact assessment was categorised in multiple reviews under four headings (“the 4 As”): advocacy, accountability, analysis and allocation [ 19 , 20 , 23 , 24 , 30 – 33 , 36 , 46 , 52 , 53 ]. Finally, Boaz and colleagues found that there was a lack of information on the cost-effectiveness of research impact evaluation methods but suggested that pragmatic, but often cheaper approaches to evaluation, such as surveys, were least likely to give in depth insights into the processes through which research impact occurred [ 31 ].

Applied to research impact evaluation, a framework provides a way of organising collected data, which encourages a more objective and structured evaluation than would be possible with an ad hoc analysis. In total, 27 (68%) reviews discussed the use of a framework in this context. Additional file 2 : Table S2 lists the frameworks mentioned in three or more of the included reviews. The most frequently described framework was the Payback Framework, developed by Buxton and Hanney in 1996 [ 54 ], and many of the other frameworks identified reported that they were developed by adapting key elements of the Payback framework. None of the frameworks identified were specifically developed to assess the impact of cancer research, however several were specific to health research. The unit of cancer research being evaluated will dictate the most suitable framework to use in any evaluation. The unit of research most suited to each framework is outlined in Additional file 2 : Table S2.

Additional findings from the included reviews

The challenges of research impact evaluation were commonly discussed in these reviews. Several mentioned that the time lag [ 24 , 25 , 33 , 35 , 38 , 46 , 50 , 53 , 55 ] between research completion and impact occurring should influence when an impact evaluation is carried out: too early and impact will not have occurred, too late and it is difficult to link impact to the research in question. This overlapped with the challenge of attributing impact to a particular piece of research [ 24 , 26 , 33 – 35 , 37 – 39 , 46 , 50 , 56 ]. Many authors argued that the ability to show attribution was inversely related to the time since the research was carried out [ 24 , 25 , 31 , 46 , 53 ].

Part II: Empirical examples of cancer research impact evaluation

Study characteristics.

In total, 14 empirical impact evaluations relevant to cancer research were identified from the references lists of the literature reviews included in the first part of this study. These empirical studies were published between 1994–2015 by primary authors located in the UK (7/14; 50%), USA (2/14; 14%), Italy (2/14; 14%), Canada (2/14; 14%) and Brazil (1/14; 14%). Table ​ Table2 2 lists these studies with the rationale for each assessment (defined using the “4As”), the unit of analysis of cancer research evaluated and the main findings from each evaluation. The categories of impact evaluated, methods of data collection and analysis, and impact frameworks utilised are also summarised in Table ​ Table2 2 and discussed in more detail below.

Examples of primary studies assessing the impact of cancer research

Approaches to cancer research impact evaluation used in empirical studies

Several of the empirical studies focused on academic impact. For example, Ugolini and colleagues evaluated scholarly outputs from one cancer research centre in Italy [ 57 ] and in a second study looked at the academic impact of cancer research from European countries [ 58 ]. Saed et al. [ 59 ] used submissions to an international cancer conference (American Society of Clinical Oncology (ASCO)) to evaluate the dissemination of cancer research to the academic community, and Lewison and colleagues [ 60 – 63 ] assessed academic, as well as policy impact and dissemination of cancer research findings to the lay media.

The category of the health impact was also commonly evaluated, with particular focus on the assessment of survival gains. Life years gained or deaths averted [ 64 ], life expectancy gains [ 65 ] and years of extra survival [ 66 ] were all used as indicators of the health impact attributable to cancer research. Glover and colleagues [ 67 ] used a measure of health utility, the quality adjusted life year (QALY), which combines both survival and quality of life assessments. Lakdawalla and colleagues [ 66 ] considered the impact of both research on cancer screening and treatments, and concluded that survival gains were 80% attributable to treatment improvement. In contrast, Glover and colleagues [ 67 ] acknowledged the importance of improved cancer therapies due to research but also highlight the major impacts from research around smoking cessation, as well as cervical and bowel cancer screening. Several of these studies that assessed health impact, also used the information on health gains to assess the economic impact of the same research [ 64 – 67 ].

Finally, two studies [ 68 , 69 ] performed multi-dimensional research impact assessments, which incorporated nearly all of the seven categories of impact identified from the previous literature (Fig.  2 ). In their assessment of the impact of research funded by one breast cancer charity in Australia, Donovan and colleagues [ 69 ] evaluated academic, capacity building, policy, health, and wider economic impacts. Montague and Valentim [ 68 ] assessed the impact of one randomised clinical trial (MA17) which investigated the use of a hormonal medication as an adjuvant treatment for patients with breast cancer. In their study, they assessed the dissemination of research findings, academic impact, capacity building for future trials and international collaborations, policy citation, and the health impact of decreased breast cancer recurrence attributable to the clinical trial.

Methods for data collection and analysis used in these studies aligned with the categories of impact assessed. For example, studies assessing academic impact used traditional bibliometric searching of publication databases and associated metrics. Ugolini et al. [ 57 ] applied a normalised journal impact factor to publications from a cancer research centre as an indicator of the research quality and productivity from that centre. This analysis was adjusted for the number of employees within each department and the scores were used to apportion 20% of future research funding. The same bibliometric method of analysis was used in a second study by the same authors to compare and contrast national level, cancer research efforts across Europe [ 58 ]. They assessed the quantity and the mean impact factor of the journals for publications from each country and compared this to the location-specific population and GDP. A similar approach was used for the manual assessment of 10% of cancer research abstracts submitted to an international conference (ASCO) between 2001–2003 and 2006–2008 [ 59 ]. These authors examined if the location of authors affected the likelihood of the abstract being presented orally, as a face-to-face poster or online only.

Lewison and colleagues, who performed four of the studies identified [ 60 – 63 ], used a different bibliometric method of publication citation count to analyse the dissemination, academic, and policy impact of cancer research. The authors also assigned a research level to publications to differentiate if the research was a basic science or clinical cancer study by coding the words in the title of each article or the journal in which the paper was published. The cancer research types assessed by these authors included cancer research at a national level for two different countries (UK and Russia) and research performed by cancer centres in the UK.

To assess policy impact these authors extracted journal publications from cancer clinical guidelines and for media impact they looked at publications cited in articles stored within an online repository from a well-known UK media organisation (British Broadcasting Co-operation). Interestingly, most of the cancer research publications contained in guidelines and cited in the UK media were clinical studies whereas a much higher proportion published by UK cancer centres were basic science studies. These authors also identified that funders of cancer research played an critical role as commentators to explain the importance of the research in the lay media. The top ten most frequent commentators (commenting on > 19 media articles (out of 725) were all representatives from the UK charity CRUK.

A combination of clinical trial findings and documentary analysis of large data repositories were used to estimate health system or health impact. In their study, Montague and Valentim [ 68 ] cited the effect size for a decrease in cancer recurrence from a clinical trial and implied the same health gains would be expected in real life for patients with breast cancer living in Canada. In their study of the impact of charitable and publicly funded cancer research in the UK, Glover et al. [ 67 ] used CRUK and Office for National Statistics (ONS) cancer incidence data, as well as national hospital databases listing episodes of radiotherapy delivered, number of cancer surgeries performed and systemic anti-cancer treatments prescribed, to evaluate changes in practice attributable to cancer research. In their USA perspective study, Lakdawalla et al. [ 66 ] used the population-based Surveillance, Epidemiology and End Results Program (SEER) database to evaluate the number of patients likely to be affected by the implementation of cancer research findings [ 66 ]. Survival calculations from clinical trials were also applied to population incidence estimates to predict the scale of survival gain attributable to cancer research [ 64 , 66 ].

The methods of data collection and analysis used for economic evaluations aligned with the categories of assessment identified by Buxton in their 2004 literature review [ 34 ]. For example, three studies [ 65 – 67 ] estimated direct healthcare cost savings from implementation of cancer research. This was particularly relevant in one ex-ante assessment of the potential impact of a clinical trial testing the equivalence of using less intensive follow up for patients following cancer surgery [ 65 ]. These authors assessed the number of years it would take (“years to payback”) of implementing the hypothetical clinical trial findings to outweigh the money spent developing and running the trial. The return on investment calculation was performed by estimating direct cost savings to the healthcare system by using less intensive follow up without any detriment to survival.

The second of Buxton’s categories was an estimation of productivity loss using the human capital approach. In this method, the economic value of survival gains from cancer research are calculated by measuring the monetary contribution from patients surviving longer who are of working age. This approach was used in two studies [ 64 , 66 ] and in both, estimates of average income (USA) were utilised. Buxton’s fourth category, an estimation of an individual’s willingness to pay for a statistical life, was used in two assessments [ 65 , 66 ], and Glover and colleagues [ 67 ] adapted this method, placing a monetary value on the opportunity cost of QALYs forgone in the UK health service within a fixed budget [ 70 ]. One of the studies that used this method identified that there may be differences in how patients diagnosed with distinct cancer types value the impact of research on cancer specific survival [ 66 ]. In particular, individuals with pancreatic cancer seemed to be willing to spend up to 80% of their annual income for the extra survival attributable to implementation of cancer research findings, whereas this fell to below 50% for breast and colorectal cancer. Only one of the studies considered Buxton’s third category of benefits to the economy from commercial development [ 66 ]. These authors calculated the gain to commercial companies from sales of on-patent pharmaceuticals and concluded that economic gains to commercial producers were small relative to gains from research experienced by cancer patients.

The cost estimates used in these impact evaluations came from documentary analysis, clinical trial publications, real-life data repositories, surveys, and population average income estimates. For example, in one study, cost information from NCI trials was supplemented by using a telephone phone survey to pharmacies, historical Medicare documents and estimates of the average income from the 1986 US Bureau of the Census Consumer Income [ 64 ]. In their study, Coyle et al. [ 65 ] costed annual follow up and treatment for cancer recurrence based on the Ontario Health Insurance plan, a cost model relevant to an Ottawa hospital and cost estimates from Statistics Canada [ 71 ]. The data used to calculate the cost of performing cancer research was usually from funding bodies and research institutions. For example, charity reports and Canadian research institution documents were used to estimate that it costs the National Cancer Institute in Canada $1500 per patient accrued to a clinical trial [ 65 ]. Government research investment outgoings were used to calculate that $300 billion was spent on cancer research in the USA from 1971 to 2000, 25% of which was contributed by the NCI [ 66 ] and that the NCI spent over $10 million USD in the 1980s to generate the knowledge that adjuvant chemotherapy was beneficial to colorectal cancer patients [ 64 ]. Charity and research institution spending reports, along with an estimation of the proportion of funds spent specifically on cancer research, were used to demonstrate £15 billion of UK charity and public money was spent on cancer research between 1970 and 2009 [ 67 ].

Lastly, the two studies [ 68 , 69 ] which adopted a multi-category approach to impact assessment used the highest number and broadest range of methods identified from the previous literature (Additional file 2 : Table S1). The methods utilised included surveys and semi-structured telephone interviews with clinicians, documentary analysis of funding and project reports, case studies, content analysis of media release, peer review, bibiliometrics, budget analysis, large data repository review, and observations of meetings.

Only two of the empirical studies identified used an impact framework. Unsurprisingly, these were also the studies that performed a multi-category assessment and used the broadest range of methods within their analyses. Donovan et al. [ 69 ] used the Payback framework (Additional file 2 : Table S2) to guide the categories of impact assessed and the questions in their researcher surveys and interviews. They also reported the results of their evaluation using the same categories: from knowledge production, through capacity building to health and wider economic impacts. Montague and Valentim [ 68 ] used the Canadian Academy Health Services (CAHS) Framework (Additional file 2 : Table S2). Rather than using the framework in it is original form, they arranged impact indicators from the CAHS framework within a hierarchy to illustrate impacts occurring over time. The authors distinguished short term, intermediate and longer-term changes resulting from one clinical cancer trial, aligning with the concept of categorising impacts based on when they occur, which was described in one of the literature reviews identified in the first part of this study [ 33 ].

Lastly, the challenges of time lags and attribution of impact were identified and addressed by several of these empirical studies. Lewison and colleagues tracked the citation of over 3000 cancer publications in UK cancer clinical guidelines over time [ 61 ], and in their analysis Donovan et al. [ 69 ] explicitly acknowledged that the short time frame between their analysis and funding of the research projects under evaluations was likely to under-estimate the impact achieved. Glover et al. [ 67 ] used bibliometric analysis of citations in clinical cancer guidelines to estimate the average time from publication to clinical practice change (8 years). They added 7 years to account for the time between funding allocation and publication of research results giving an overall time lag from funding cancer research to impact of 15 years. The challenge of attribution was addressed in one study by using a time-line to describe impacts occurring at different time-points but linking back to the original research in question [ 68 ]. The difficultly of estimating time lags and attributing impact to cancer research were both specifically addressed in a companion study [ 72 ] to the one conducted by Glover and colleagues. In this study, instead of quantifying the return on cancer research investment, qualitative methods of assessment were used. This approach identified factors that enhanced and accelerated the process of impact occurring and helped to provide a narrative to link impacts to research.

This study has identified several examples of the evaluation of the impact of cancer research. These evaluations  were performed over three decades, and mostly assessed research performed in high-income countries. Justification for the approach to searching the literature used  in this study is given by looking at the titles of the articles identified. In only 14% (2/14) was the word “impact” included, suggesting that performing a search for empirical examples of cancer research impact evaluation using traditional publication databases would have been challenging. Furthermore, all the studies identified were included within reviews of approaches to research impact evaluation, which negated the subjective decision of whether the studies complied with a particular definition of research impact.

Characteristics of research that were specifically relevant to cancer studies can be identified from these impact assessments. Firstly, many of these evaluations acknowledged the contribution of both basic and applied studies to the body of cancer research, and several studies categorised research publications based on this distinction. Second, the strong focus on health impact and the expectation that cancer research will improve health was not surprising. The focus on survival in particular, especially in economic studies looking at the value of health gains, reflects the high mortality of cancer as a disease entity. This contrasts with similar evaluations of musculoskeletal or mental health research, which have focused on improvements in morbidity [ 73 , 74 ]. Third, several studies highlighted the distinction between research looking at different aspects of the cancer care continuum; from screening, prevention and diagnosis, to treatment and end of life care. The division of cancer as a disease entity by the site of disease was also recognised. Studies that analysed the number of patients diagnosed with cancer, or population-level survival gains, often used site-specific cancer incidence and other studies evaluated research relating to only one type of cancer [ 64 , 65 , 68 , 69 ]. Lastly, the empirical examples of cancer research impact identified in this study confirm the huge investment into cancer research that exists, and the desire by many research organisations and funders to quantify a rate of return on that investment. Most of these studies concluded that cancer research investment far exceeded expectations of the return on investment. Even using the simple measure of future research grants attracted by researchers funded by one cancer charity, the monetary value of these grants outweighed the initial investment [ 69 ].

There were limitations in the approaches to impact evaluation used in these studies which were recognised by reflecting on the findings from the broader literature. Several studies assessed academic impact in isolation, and studies using the journal impact factor or the location of authors on publications were limited in the information they provided. In particular, using the journal impact factor (JIF) to allocate funding research which was used in one study, is now outdated and controversial. The policy impact of cancer research was commonly evaluated by using clinical practice guidelines, but other policy types that could be used in impact assessment [ 30 ], such as national government reports or local guidelines, were rarely used. In addition, using cancer guidelines as a surrogate for clinical practice change and health service impact could have drawbacks. For example, guidelines can often be outdated, irrelevant or simply not used by cancer clinicians and in addition, local hospitals often have their own local clinical guidelines, which may take precedent over national documents. Furthermore, the other aspects of policy impact described in the broader literature [ 30 ], such as impact on policy agenda setting and implementation, were rarely assessed. There were also no specific examples of social, environmental or cultural impacts and very few of the studies mentioned wider economic benefits from cancer research, such as spin out companies and patents. It may be that these types of impact were less relevant to cancer research being assessed, however unexpected impacts may have be identified if they were considered at the time of impact evaluation.

Reflecting on how the methods of data collection and analysis used in these studies aligned with those listed in Additional file 2 : Table S1 bibliometrics, alternative metrics (media citation), documentary analysis, surveys and economic approaches were often used. Methods less commonly adopted were interviews, using a scale and focus groups. This may have been due to the time and resource implications of using qualitative techniques and more in depth analysis, or a lack of awareness by authors regarding the types of scales that could be used. An example of a scale that could be used to assess the impact of research on policy is provided in one of the literature reviews identified [ 30 ]. The method of collecting expert testimony from researchers was utilised in the studies identified, but there were no obvious examples of testimony about the impact of cancer research from stakeholders such as cancer patients or their families.

Lastly, despite the large number of examples identified from the previous literature, a minority of the empirical assessments used an impact framework. The Payback Framework, and an iteration of the CAHS Framework were used with success and these studies are excellent examples of how frameworks can be used for cancer research impact evaluation in future. Other frameworks identified from the literature (Additional file 2 : Table S2) that may be appropriate for the assessment of cancer research impact in future include Anthony Weiss’s logic model [ 75 ], the research impact framework [ 76 ] and the research utilisation ladder [ 77 ]. Weiss’s model is specific to medical research and encourages evaluation of how clinical trial publication results are implemented in practice and lead to health gain. He describes an efficacy-efficiency gap [ 75 ] between clinical decision makers becoming aware of research findings, changing their practice and this having impact on health. The Research Impact Framework, developed by the Department of Public Health and Policy at the UK London School of Hygiene and Tropical Medicine [ 76 ], is an aid for researchers to self-evaluate their research impact, and offers an extensive list of categories and indicators of research which could be applied to evaluating the impact of cancer research. Finally, Landry’s Research Utilisation Ladder [ 77 ] has similarities to the hierarchy used in the empirical study by Montegue and Valentim [ 68 ], and focuses on the role of the individual researcher in determining how research is utilised and its subsequent impact.

Reflecting on the strengths and limitations of the empirical approaches to cancer research impact identified in this study, Fig.  3 outlines recommendations for the future. One of these recommendations refers to improving the use of real-life data to assess the actual impact of research on incidence, treatment, and outcomes, rather than predicting these impacts by using clinical trial results. Databases for cancer incidence, such as SEER (USA) and the Office of National Statistics (UK), are relatively well established. However, those that collect data on treatments delivered and patient outcomes are less so, and when they do exist, they have been difficult to establish and maintain and often have large quantities of missing data [ 78 , 79 ]. In their study, Glover et al. [ 67 ] specifically identified the lack of good quality data documenting radiotherapy use in the UK in 2012.

Suggestions for approaching cancer research impact evaluation.

1 Thonon F, Boulkedid R, Teixeira M, Gottot S, Saghatchian M, Alberti C. Identifying potential indicators to measure the outcome of translational cancer research: a mixed methods approach. Health Res Policy Syst. 2015;13:72

The recommendations also suggest that impact assessment for cancer and other health research could be made more robust by giving researchers access to cost data linked to administrative datasets. This type of data was used in empirical impact assessments performed in the USA [ 64 , 66 ] because the existing Medicare and Medicaid health service infrastructure collects and provides access to this data. In the UK, hospital cost data is collected for accounting purposes but this could be unlocked as a resource for research impact assessments going forward. A good example of where attempts are being made to link resource use to cost data for cancer care in the UK is through the UK Colorectal Cancer Intelligence Hub [ 80 ].

Lastly, several empirical examples highlighted that impact from cancer research can be increased when researchers or research organisations advocate, publicise and help to interpret research findings for a wider audience [ 60 , 72 ]. In addition, it is clear from these studies that organisations that want to evaluate the impact of their cancer research must also appreciate that research impact evaluation is a multi-disciplinary effort, requiring the skills and input from individuals with different skill sets, such as basic scientists, clinicians, social scientists, health economists, statisticians, and information technology analysts. Furthermore, the users and benefactors from cancer research, such as patients and their families, should not be forgotten, and asking them which impacts from cancer research are important will help direct and improve future evaluations.

The strengths of this study are the broad, yet systematic approach used to identify existing reviews within the research impact literature. This allowed a more informed assessment of cancer research evaluations than would have been possible if a primary review of these empirical examples had been undertaken. Limitations of the study include the fact that the review protocol was not registered in advance and that one researcher screened the full articles for review. The later was partly mitigated by using pre-defined inclusion criteria.

Impact assessment is a way of communicating to funders and patients the merits of undertaking cancer research and learning from previous research to develop better studies that will have positive impacts on society in the future. To the best of our knowledge, this is the first review to consider how to approach evaluation of the impact of cancer research. At the policy level, a lesson learned from this study for institutions, governments, and funders of cancer research, is that an exact prescription for how to conduct cancer research impact evaluation cannot be provided, but a multi-disciplinary approach and sufficient resources are required if a meaningful assessment can be achieved. The approach to impact evaluation used to assess cancer research will depend on the type of research being assessed, the unit of analysis, rationale for the assessment and the resources available. This study has added to an important dialogue for cancer researchers, funders and patients about how cancer research can be evaluated and ultimately how future cancer research impact can be improved.

Acknowledgements

We would like to acknowledge the help of Ms Lorraine MacLeod, specialist librarian from the Beatson West of Scotland Cancer Network in NHS Greater Glasgow and Clyde for her assistance in formulating the search strategy. We would like to acknowledge that Professor Stephen Hanney provided feedback on an earlier version of this review.

Abbreviations

Authors’ contributions.

All authors contributed to the concept and design of the study. CH was responsible for the main data analysis and writing of the manuscript. KAB and RJJ responsible for writing, editing and final approval of the manuscript. All authors read and approved the final manuscript.

Dr. Catherine Hanna has a CRUK and University of Glasgow grant. Grant ID: {"type":"entrez-nucleotide","attrs":{"text":"C61974","term_id":"2420679"}} C61974 /A2429.

Availability of data and materials

Ethics approval and consent to participate.

No ethical approval required.

Consent for publication

No patient level or third party copyright material used.

Competing interests

Nil declared.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Catherine R. Hanna, Email: [email protected] .

Kathleen A. Boyd, Email: [email protected] .

Robert J. Jones, Email: [email protected] .

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Sustainability in project management: A literature review and impact analysis

Related Papers

Wasit Journal of Engineering Science

Hussaen A H Kahachi

This research is looking at the concept of sustainable project management and its importance in real life project management. Firstly, the research will examine the meaning of sustainability and sustainable management. Next, it will examine, evaluate and classify the different drivers for the integration of sustainability in project management as social, economic, legislative and environmental drivers. Then, the research will discuss the potential benefits of implementing sustainability in project management throughout project's lifecycle. The researcher concluded that there are many environmental, social/ethical, legislative and economic benefits derived from integrating sustainability into project management throughout project's lifecycle, however, there are some restrictions as the tools for integration are still under development.

collins egbu

In the recent years, lots of debates have been going on concerning sustainability and its implications on project management success. Sustainability is one of the challenges we face in this generation. Sustainable development has acquired an increasing awareness of business, social and management literatures. The ideas of sustainability are now being incorporated by many companies and organizations in their marketing, annual reports, in their commercial communications and in many other deeds. Some of these actions are project organized. This paper delves into sustainability integration/ application in the context of project management. But the question is how is sustainability incorporated into projects and project management? It is expected that sustainability have some effects on the management of project, initiation and organization. Sustainability integration into project and project management is rapidly taking part as a good industrial exercise and a research topic. The original meaning of successful project management is to deliver a project or service as at when due, in the bracket of the estimated budget and to the quality agreed upon. But recently, the increasing environmental issues awareness, especially the global climatic change has given birth to more attention to sustainable development. Many companies are trying to imbibe the practice of sustainability. In this case, the managers of project are being tasked with integrating sustainable development principles into projects and to all the project teams. The actions of sustainability are limited due to some regulations/ laws. Another problem of sustainability is the insufficiency in the knowledge of how it can be integrated into the methodologies of project management. Here, we are going to look into the debates on sustainability, the different perspectives of sustainability and also evaluate sustainable general objectives and in the professional practice context as applied to projects.

Hradec Economic Days

Jana Kostalova

Project management maturity influences the success of project management and project implementation. Based on project management international standards and other project management methodologies, various project management maturity models have developed. These models evaluate the level of project management maturity from many lenses; holistically, or in specific project management capabilities or specific to certain industry sectors. Projects are implemented in all human endeavors. Research and development, innovation initiatives, new ventures all harness projects as tool of change. Sustainability has three dimensions – social, economic and environmental. Project management is as much as life skill as it is a management science, relevant to business and society., It is therefore fundamental that sustainability is interwoven into project management tools, techniques and practices. Consequently, project management maturity models have to also evaluate how sustainability is included and applied in project management. The article defines a framework to assess sustainable project management in the form of the most important indicators based on the three pillars of sustainability: People, Planet and Profit. This approach could be used for evaluation of sustainability in project management generally or for the evaluation of a specific project management maturity model.

Academy of Management Proceedings

Robert Moehler

Sustainability

Danijela Toljaga-Nikolić

The incorporation of the social, environmental, and economic dimensions of sustainability in different aspects of human life and business provides a guarantee for our future. Organizations have shown a great interest in incorporating sustainability into managerial concepts, both at the strategic and operational levels. Sustainable business strategies are being implemented in many projects, which has led to a recent expansion of interest in exploring the potential of integrating sustainability dimensions in project management. With the intention of contributing to a better understanding of sustainable project management, this paper examines whether project management methodologies, applied in different sectors, support the introduction of sustainability dimensions. It also surveys the level of integration of sustainability dimensions in groups of project management processes. Considering that the incorporation of sustainability in project management poses numerous challenges for proj...

Journal of Cleaner Production

Mauro Luiz Martens

Elumah Lucas , shobayo peter babatunde

Sustainable development is an important challenge currently facing the economy. It deals with how to develop the economy without jeopardizing the life of future generations. Organizations have incorporated the concept of sustainability in their overall organizational structures. In recent times the concept of sustainable development has been associated with project management; Projects are seen as a mechanism of change both within an organization, and the economy at large, and this mechanism play a crucial role in achieving both sustainable business process and practices. Thus, project managers are seen as change agents who play important role on the sustainability of an organization. However, the standard of project management in Nigeria have failed to realize the crucial role project managers play in ensuring sustainable development, and the project managers as well lack the capability to achieve sustainability in their project. This study aims to identify the role of project management in ensuring sustainable development. The study further aims to make specific recommendations on developing the standards of the project management capabilities so as to prepare the project managers for the crucial role of achieving sustainable development in an organization and the economy at large.

IRJET Journal

RELATED PAPERS

Journal of the Academy of Nutrition and Dietetics

Ethan Bergman

Ahmad Fizam

Atmospheric Research

VAROTSOS COSTAS

The Proceedings of Conference of Kanto Branch

Wits Journal of Clinical Medicine

Zibya Barday

Galicia Clínica

Mário Pires

Jurnal INTEKNA : Informasi Teknik dan Niaga

yuswinda febrita

Brain Research Bulletin

Raul Heredia

Campos: Revista de Antropologia

Bárbara Rossin

Pacific Academy of Higher Education & Research University)

Dr. MOHD A B A S S BHAT

Ilha do Desterro A Journal of English Language, Literatures in English and Cultural Studies

Henning Wrage

mohamed al-ibrahim

Rania J Qutieshat

Biology of Blood and Marrow Transplantation

David Rizzieri

Journal of Thermal Analysis and Calorimetry

rania sabry

World Resources Institute

Felipe Carazo

Journal of the American Society of Nephrology

Ismail Khalid

Tarja Mäkeläinen

Journal of Nanoelectronics and Optoelectronics

Sudipta Biswas

Zenodo (CERN European Organization for Nuclear Research)

Charles Tobin-West

Verimlilik dergisi

Sebahattin Devecioglu

Revista do CNMP

Leonardo Granato

Enfoque Derecho

Josefina Miró Quesada Gayoso

Science Education

Lynn Dierking

See More Documents Like This

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• Open access
• Published: 28 March 2024

Impact of obesity on outcomes of extracorporeal membrane oxygenation support: a systematic review and meta-analysis

• Xinhua Huang 1 &
• Xiaoqing Lin 1  

BMC Pulmonary Medicine volume  24 , Article number:  157 ( 2024 ) Cite this article

Metrics details

Extracorporeal membrane oxygenation (ECMO) is used when standard methods of standard treatment methods are not successful. Obese patients present unique challenges during ECMO due to large body size hindering sufficient flows, difficulties with patient positioning and anatomical landmark identification, and restricted radiology scans. This meta-analysis aims to investigate the impact of obesity on the outcomes of patients undergoing ECMO.

Databases (PubMed, Embase, and Scopus databases) were searched to identify relevant studies published until July 2023. Data were reported as odds ratios (OR) with 95% confidence interval (CI), and the descriptive data were reported as standard difference of means (SDM) by a random effects model.

A literature search identified 345 studies. Of them, 18 studies met the inclusion criteria. The findings from the meta-analysis revealed no significant association between obesity and survival outcomes after ECMO (odds ratio (OR): 0.91, 95% confidence interval (CI): 0.70–1.17, p: 0.46). Moreover, no comparative significant differences were found between obese and non-obese individuals on the duration of ECMO procedure (standardized mean difference (SMD): 0.07, -0.03–0.17), length of hospital stay (-0.03, -0.19 to 0.12), and duration of ventilation support (-0.10, -0.44 to 0.24).

The meta-analysis findings suggest no significant impact of obesity on the survival outcomes after the ECMO procedure. There was no significant impact of obesity on the duration of ECMO procedures, length of hospital stay, and duration of ventilation support.

Peer Review reports

Introduction

Obesity is a global health issue that presents formidable challenges to healthcare providers, particularly during life-saving interventions such as extracorporeal membrane oxygenation (ECMO) procedures [ 1 , 2 , 3 ]. Obesity, as defined by the Centers for Disease Control and Prevention, is characterized by a Body Mass Index (BMI) of 30.0 or higher. BMI is calculated by dividing a person’s weight in kilograms by the square of their height in meters, serving as a screening tool for evaluating body fatness. The CDC classifies BMI into four categories: underweight (BMI less than 18.5), healthy weight (BMI 18.5 to < 25), overweight (BMI 25.0 to < 30), and obesity (BMI 30.0 or higher) [ 4 ]. ECMO is used in patients with severe respiratory or cardiac failure when other conventional treatments fail [ 1 , 5 , 6 , 7 ]. However, for obese patients, the management of ECMO remains particularly complex due to the unique anatomical and physiological characteristics of this group of patients [ 8 , 9 ], the presence of various comorbidities, such as diabetes mellitus and hypertension [ 10 , 11 ], or difficulties in diagnosing and monitoring [ 3 , 8 , 10 , 11 , 12 ]. However, despite these potential issues, obesity has not been identified as a significant risk factor for hospital mortality in patients with acute lung failure and cardiac diseases [ 3 , 10 , 11 , 13 ]. Therefore, existing guidelines do not categorize obesity as an absolute contraindication for ECMO support [ 3 , 10 , 11 ]. There is still a lack of comprehensive reviews summarizing the overall evidence in this area [ 9 , 13 , 14 , 15 , 16 ].

This review and meta-analysis aims to summarize all existing data and evaluate the impact of obesity on outcomes, such as mortality rates post-ECMO, the duration of ECMO procedures, lengths of hospital stays, and ventilation requirements, in patients undergoing ECMO. Our study is particularly relevant to the nursing field due to the integral role nurses play in the care of ECMO patients and may contribute to development of evidence-based guidelines, optimizing the management of obese patients on ECMO and potentially improving their survival and recovery rates in critical care settings.

The review was performed in adherence to the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 17 ]. The paper was registered at PROSPERO, No. CRD42023448406.

Search strategy

PubMed, Embase, and Scopus databases were searched by using appropriate keywords. We initially compared the complications with ECMO in between obese and non-obese individuals. Search items were as follows: “Obesity,” “Extracorporeal membrane oxygenation,“, “ECMO”, “Body mass index”, “BMI”, “Impact,” “Outcomes,” “Systematic review,” “Meta-analysis,” “ECMO outcomes,” “ECMO complications,” “ECMO survival,” “ECMO mortality,” “Obesity and critical care,” “ECMO and obesity,” “Extracorporeal life support,” “ECMO effectiveness,” “ECMO complications in obese patients.” These terms were combined using the OR operator to ensure comprehensive coverage of relevant literature.

Inclusion criteria (as per the PECOS criteria)

Population: Obese individuals who underwent ECMO procedure.

Exposure: Extracorporeal membrane oxygenation (ECMO) procedures.

Comparison: Non-obese who underwent ECMO procedure.

Outcome: Mortality, ECMO procedure duration, length of hospital stay, and length of ventilation.

Study Design: Randomized controlled trials (RCTs), cohort studies, case-control studies, and observational studies.

Language: English.

Exclusion criteria

Studies are not reporting comparative outcomes between obese and non-obese individuals who underwent ECMO procedures.

Non-English languages studies.

Incomplete studies, unavailable data, case reports, editorials, commentaries, and letters.

The eligibility of the identified studies was independently assessed by two reviewers. Each study underwent a thorough evaluation based on predetermined criteria to ensure its relevance to the research question. The literature search was also independently conducted by the same two reviewers to minimize bias and improve the accuracy of study selection.

In instances where discrepancies or differences of opinion arose between the reviewers regarding the inclusion or exclusion of a particular study. The outcome of the discussion between the two reviewers was to address any conflicts and facilitate consensus on the final selection of studies. This consensus-based approach among the reviewers ensured a comprehensive and unbiased selection of studies for inclusion in the meta-analysis.

Quality assessment

To assess the potential bias in the cohort trials included in this study, ROBINS-I tool was used [ 18 ]. Two independent reviewers conducted a thorough evaluation of the methodological quality of the included studies. In cases where discrepancies arose, the outcome of the discussion between the two reviewers was to resolve any disagreements and reach a consensus.

Data extraction

Data were systematically extracted from the selected studies, and included study type, groups involved, sample size, and age of obese and non-obese cohorts, BMIs of the respective groups, type of ECMO procedures, length of hospital stay, duration of ECMO procedure, mortality events, and period of ventilation.

Statistical analysis

The Meta-analysis version 3.0 was employed for the statistical analysis. Only a random effects model was implemented [ 19 ]. The mortality outcomes between obese and non-obese individuals undergoing ECMO procedures was analyzed. Odds ratios (OR) were computed on the basis of the number of events reported in the included studies. The analysis further examined various factors, including the duration of the ECMO procedure, length of hospital stay, and duration of ventilation. For quantifying these changes standard difference of means (SDM) were computed based on the descriptive data provided in the included studies. Heterogeneity among the studies was assessed by I 2 statistics. I 2 values between 0 and 25% indicated negligible heterogeneity, 25–75% indicated moderate heterogeneity, and ≥ 75% indicated substantial heterogeneity [ 20 ]. Publication bias was assessed by Duval and Tweedy’s trim and fill procedure [ 21 ]. Additionally, as we had used both adjusted and unadjusted values for the primary outcome of mortality in our analysis due to paucity of data present in the included studies, we conducted a leave-one-out sensitivity analysis to strengthen the interpretation of our primary outcome. All analyses conducted in this study adhered to a significance level of 5%.

Study selection and characteristics

Among the 345 papers initially retrieved, 18 papers were eligible, as shown in Fig.  1 . All of the included studies were retrospective cohort studies [ 3 , 5 , 8 , 9 , 10 , 11 , 14 , 15 , 16 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 ]. The extracted data from these studies are summarized in Table  1 .

PRISMA flowchart

Participant information

The analysis incorporated data from 21,361 patients undergoing ECMO. A total of 15,447 patients (10,075 males, 5129 females) were non-obese, and 5914 (3405 males, 2323 females) were obese. Sex distribution was not reported by four of the included studies [ 5 , 15 , 16 , 24 ]. The mean age of the non-obese and obsess patients was 51.3 ± 10.6 years and 50.1 ± 9.4 years, respectively.

Assessment of study quality

The methodological quality of the cohort studies included in the analysis was evaluated using the ROBINS-I tool [ 18 ]. As summarized in Table  2 , there was a high risk of bias across the included studies. However, it was noted that several studies had missing data, showed signs of deviation from intervention, and selection bias suggesting possible sources of bias (Table  2 ).

Our analysis of 17 cohort studies (Fig.  2 A) indicates a non-significant effect of obesity on overall mortality in patients who underwent ECMO procedure (odds ratio [OR]: 0.91, 95% confidence interval [CI]: 0.70–1.17, p  = 0.46), with moderate heterogeneity (I 2 : 38.5%). Figure  2 B indicates leave one out sensitivity analysis for this analysis.

( A ) The comparative outcome between obese and non-obese patients who underwent ECMO procedure on overall mortality. ( B ) Leave one out sensitivity analysis

Publication bias

To assess publication bias, we employed Duval and Tweedy’s trim and fill method, which estimates the number of missing studies on either side of the mean effect of the funnel plot. According to this method, three studies were missing on the right side of the mean effect. The overall random effect models provided a point estimate and 95% confidence interval for the studies as 0.90 (95% CI 0.85–0.96). The trim and fill imputed point estimates were calculated as 0.92 (95% CI 0.86–0.98). The results of the publication bias assessment are presented in Fig.  3 .

The trim and fill method by Duval and Tweedy provides a visualization of how publication bias can impact study results

ECMO procedure duration

Our analysis of 18 cohort studies (Fig.  4 ) indicates no difference in the duration of ECMO procedure in obese individuals as compared to non-obese individuals (Standardized difference in means: 0.07, 95% CI: -0.03–0.17, p  = 0.16) with negligible heterogeneity (I 2 : 24.5%).

The comparative outcome between obese and non-obese individuals who underwent ECMO procedure on the overall duration of the procedure

Length of hospital stay

Our analysis of 11 cohort studies (Fig.  5 ) indicates a non-significant difference in the length of hospital stay for non-obese individuals as compared to the obese individuals who underwent ECMO procedure (SDM: -0.03, 95% CI: -0.19–0.12, p  = 0.67) with negligible heterogeneity (I 2 : 8.8%).

The comparative outcome between obese and non-obese individuals who underwent ECMO procedure on the length of hospital stay

Duration of ventilation

Our analysis of 4 cohort studies (Fig.  6 ) indicates a non-significant difference in the duration of ventilation for non-obese individuals as compared to the obese individuals who underwent ECMO procedure (SDM: -0.10, 95% CI: -0.44–0.24, p  = 0.57) with negligible heterogeneity (I 2 : 12.9%).

The comparative outcome between obese and non-obese individuals who underwent ECMO procedures during the period of ventilation

The results of our meta-analysis showed no substantial correlation between obesity and post-ECMO mortality outcomes. No difference was found between obese and non-obese patients in terms of ECMO procedure duration, the length of hospital stay, and the duration of ventilation required. Our results are particulary important for nursing professionals who are providing holistic care during ECMO interventions, and may serve to inform nursing practices to tailor their care strategies effectively.

Obesity may lead to complications during ECMO procedures through several possible mechanisms [ 19 , 20 , 23 ]. The increased adipose tissue in obese patients can pose challenges during cannulation, as it obscures the underlying vessels and makes it difficult to identify suitable cannulation sites [ 1 , 3 , 6 , 7 , 8 , 9 , 12 , 19 ]. This may result in suboptimal cannula positioning or vascular injuries during insertion, leading to hemorrhage or improper ECMO flow [ 23 , 29 , 31 ]. Altered respiratory mechanics in obese patients, characterized by reduced lung compliance and increased airway resistance, can impact efficient oxygenation and carbon dioxide removal by the ECMO system [ 23 , 29 ]. As a result, inadequate oxygenation and perfusion can cause hypoxemia and inadequate organ support [ 5 ]. For instance, in obese individuals, changes in chest wall resistance and lung compliance can complicate respiratory function [ 32 ]. With respect to ECMO, oxygenation challenges may arise due to ventilation-perfusion mismatch and atelectasis [ 33 ]. The removal of CO2 is impeded by these altered mechanics, potentially resulting in hypercapnia. While ECMO provides extracorporeal support for both oxygenation and CO2 removal, its efficacy is influenced by various external factors, including pump speed, blood flow rate, and the efficiency of the oxygenator and sweep gas [ 34 ]. It is essential to recognize that the effectiveness of ECMO is not solely determined by the inherent nature of the lung disease; rather, it is a complex interplay of external parameters. In situations where lung mechanics are altered, patients may require a prolonged duration of ECMO support, allowing for sufficient time for the lungs to recover to a point where they can adequately provide oxygenation and ventilation. Therefore, tailoring ECMO configurations and closely monitoring patients for dynamic adjustments are crucial aspects for optimizing outcomes in these cases.Furthermore, obese patients with pre-existing cardiac issues may experience additional strain on the heart due to the hemodynamic load imposed by the ECMO circuit [ 5 , 14 ]. This can lead to myocardial ischemia, arrhythmias, or cardiac arrest [ 17 , 18 ]. Moreover, obesity increases the risk of thromboembolic events, making obese patients more susceptible to clot formation within the ECMO circuit or embolization of clots to vital organs [ 6 , 8 , 13 , 21 ]. Impaired immune function in obese patients also heightens the risk of potentially life-threatening ECMO-related infections [ 3 , 9 , 10 , 12 ].

In our study, we evaluated the comparative mortality outcomes between obese and non-obese patients, undergoing ECMO. A study by Lu, Ortoleva et al. (2022) [ 16 ] also reported lack of association between obesity and mortality in their cohort, and provided several plausible explanations for that observation. Firstly, 944 patients in their cohort had missing BMI data, and the reported in-hospital mortality rate was high (89.1%). Secondly, while comorbidities were considered in the analysis, the severity of illness before cannulation for VA-ECMO was not accounted for. Thirdly, sample size in Lu, Ortoleva et al. (2022) study was small [ 16 ], which might have hindered the detection of existing differences. Interestingly Djordjevic, Ivanov et al. (2022) and Merritt-Genore, Lyden et al. (2020) reported reduced mortality outcomes in their obese cohort as compared to the non-obese cohort who underwent ECMO procedure [ 15 , 29 ]. Here, the authors suggested that perhaps the reduction in the mortality outcomes could be due to the obesity paradox in individuals undergoing ECMO due to the potential benefits provided by the increased adipose tissue. Obese individuals may have greater nutritional and metabolic reserves, which could help them withstand the stress of ECMO support better [ 35 ]. Additionally, adipose tissue produces certain protective hormones that may have favorable effects on cardiovascular function and immune response during critical illness [ 36 , 37 ].

We also did not observe any significant difference in ECMO procedure duration between obese and non-obese individuals. This lack of difference can be explained by recent advancements in ECMO technology and expertise in managing obese patients which minimized any potential procedural delays related specifically to obesity. Additionally, the ECMO circuit could have been adapted to accommodate the larger body size of obese patients, allowing for adequate flow and gas exchange, which could also reduce the differences in procedure duration. Finally, healthcare providers could have developed standardized protocols and tailored strategies for obese patients, optimizing patient positioning and cannula placement, leading to comparable procedure durations. All these reasons can also explain the lack of differences in the duration of hospitalization and ventilation support between obese and non-obese individuals observed in our meta-analysis. Besides, as our findings indicate that obesity does not significantly impact ECMO outcomes. These findings can empower nurses to tailor care strategies for obese patients, address specific challenges, and contribute to collaborative decision-making. Based on this information, nurses can focus on meticulous monitoring, educate and support patients based on evidence, and actively participate in refining protocols to optimize care for obese individuals on ECMO.

Our study, despite its rigorous methodology, is not immune to certain limitations inherent in non-randomized observational studies. Firstly, confounding bias, a common concern in such research designs, arises from the potential influence of unmeasured or unaccounted variables on the observed associations. Although we employed careful study selection criteria and adjusted our analyses where feasible, the diversity in methodologies, patient characteristics, and reporting practices across the included studies introduces the possibility of residual confounding. Addressing confounding bias in the context of our research question is complex. While some studies may have adjusted for relevant factors, the variability in the control of confounders across the literature is a notable limitation. Secondly, there was variation in the reported parameters across the studies included in the review. For instance, mortality and ECMO procedure duration were the most widely evaluated outcomes (i.e., reported by 17 and 18 studies), whereas the duration of ventilation support was only reported by four studies. This variation in unreported data could be a source of heterogeneity in our analyses and could also introduce bias in our results, making generalizability of our findings difficult. Furthermore, while some studies in the literature have employed propensity score matching to address selection bias and confounding, our decision to utilize unadjusted values was pragmatic. The diverse methodologies and variable reporting across studies made it challenging to uniformly apply propensity score matching. This limitation underscores the importance of interpreting our results with caution, recognizing the potential impact of unmeasured confounders. Therefore, we strongly recommend future studies to replicate our findings in large-scale trials with consistent data reporting for obese and non-obese individuals undergoing ECMO procedures to help develop more robust evidence to guide clinicians in selecting appropriate strategies to improve outcomes following ECMO.

Conclusions

In conclusion, this meta-analysis provides valuable insights into the impact of obesity on ECMO outcomes. It indicates that obesity alone does not significantly affect mortality outcomes or the duration of the ECMO procedure. However, healthcare providers should be aware of the challenges that obese patients may face during ECMO and consider tailored management approaches to optimize their care. This study contributes to a better understanding of the role of obesity in ECMO and can aid in improving the clinical decision-making process for this specific patient population.

Data availability

Data is provided within the manuscript or supplementary information files.

Nair P, Davies AR, Beca J, Bellomo R, Ellwood D, Forrest P, et al. Extracorporeal membrane oxygenation for severe ARDS in pregnant and postpartum women during the 2009 H1N1 pandemic. Intensive Care Med. 2011;37:648–54.

Article   PubMed   PubMed Central   Google Scholar  

Nordan T, Critsinelis AC, Vest A, Zhang Y, Chen FY, Couper GS, et al. Prolonged waitlisting is associated with mortality in extracorporeal membrane oxygenation-supported heart transplantation candidates. JTCVS Open. 2022;12:234–54.

Prasad NR, Elkholey K, Patel NR, Junqueira E, Cohen ES, Whitmore SP. Obesity associated with improved mortality of extracorporeal membrane oxygenation for severe COVID-19 pneumonia. Perfusion. 2023;:2676591231178896.

May AL, Freedman D, Sherry B, Blanck HM. Centers for Disease Control and Prevention (CDC). Obesity - United States, 1999–2010. MMWR Suppl. 2013;62:120–8.

PubMed   Google Scholar  

Lazzeri C, Bonizzoli M, Cianchi G, Batacchi S, Terenzi P, Cozzolino M, et al. Body mass index and echocardiography in refractory ARDS treated with veno-venous extracorporeal membrane oxygenation. J Artif Organs. 2017;20:50–6.

Article   CAS   PubMed   Google Scholar  

Azouz V, Hill S, Firstenberg M. Extracorporeal membrane oxygenation in Extreme obesity: a Case Report and Review of the literature. Heart Surg Forum. 2018;21:E485–8.

Article   PubMed   Google Scholar  

Diaz RA, Graf J, Zambrano JM, Ruiz C, Espinoza JA, Bravo SI, et al. Extracorporeal membrane oxygenation for COVID-19-associated severe Acute Respiratory Distress Syndrome in Chile: a Nationwide Incidence and Cohort Study. Am J Respir Crit Care Med. 2021;204:34–43.

Powell EK, Haase DJ, Lankford A, Boswell K, Esposito E, Hamera J, et al. Body mass index does not impact survival in COVID-19 patients requiring veno-venous extracorporeal membrane oxygenation. Perfusion. 2023;38:1174–81.

Peetermans M, Guler I, Meersseman P, Wilmer A, Wauters J, Meyns B, et al. Impact of BMI on outcomes in respiratory ECMO: an ELSO registry study. Intensive Care Med. 2023;49:37–49.

Salna M, Chicotka S, Biscotti M, Agerstrand C, Liou P, Brodie D, et al. Morbid obesity is not a contraindication to transport on extracorporeal support. Eur J Cardiothorac Surg. 2018;53:793–8.

Salna M, Fried J, Kaku Y, Brodie D, Sayer G, Uriel N, et al. Obesity is not a contraindication to veno-arterial extracorporeal life support. Eur J Cardiothorac Surg. 2021;60:831–8.

Weinberg A, Tapson VF, Ramzy D. Massive pulmonary embolism: extracorporeal membrane oxygenation and Surgical Pulmonary Embolectomy. Semin Respir Crit Care Med. 2017;38:66–72.

Kosmopoulos M, Kalra R, Alexy T, Gaisendrees C, Jaeger D, Chahine J, et al. The impact of BMI on arrest characteristics and survival of patients with out-of-hospital cardiac arrest treated with extracorporeal cardiopulmonary resuscitation. Resuscitation. 2023;188:109842.

Lee SN, Moon D, Jo MS, Yoo K-D. Associations between Body Mass Index and Clinical outcomes in Acute Myocardial Infarction supported with extracorporeal membrane oxygenation. Acta Cardiol Sin. 2020;36:448–55.

PubMed   PubMed Central   Google Scholar  

Merritt-Genore H, Lyden E, Ryan T, Kwapnoski Z. The effect of patient obesity on extracorporeal membrane oxygenator outcomes and ventilator dependency. J Card Surg. 2020;35:1283–6.

Lu SY, Ortoleva J, Colon K, Mueller A, Laflam A, Shelton K, et al. Association between Body Mass Index and outcomes in Venoarterial extracorporeal membrane oxygenation. Anesth Analg. 2022;134:341–7.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Borenstein M. Comprehensive Meta-Analysis Software. 2022. p. 535–48.

Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21:1539–58.

Duval S, Tweedie R. A nonparametric Trim and fill Method of Accounting for Publication Bias in Meta-Analysis. J Am Stat Assoc. 2000;95:89–98.

Google Scholar  

Kon ZN, Dahi S, Evans CF, Byrnes KA, Bittle GJ, Wehman B, et al. Class III obesity is not a contraindication to venovenous extracorporeal membrane oxygenation support. Ann Thorac Surg. 2015;100:1855–60.

Cho WH, Oh JY, Yeo HJ, Han J, Kim J, Hong S-B, et al. Obesity survival paradox in pneumonia supported with extracorporeal membrane oxygenation: analysis of the national registry. J Crit Care. 2018;48:453–7.

Galvagno SM, Pelekhaty S, Cornachione CR, Deatrick KB, Mazzeffi MA, Scalea TM, et al. Does Weight Matter? Outcomes in adult patients on venovenous extracorporeal membrane oxygenation when stratified by obesity class. Anesth Analg. 2020;131:754–61.

Verkerk BS, Dzierba AL, Muir J, Der-Nigoghossian C, Brodie D, Bacchetta M, et al. Opioid and benzodiazepine requirements in obese adult patients receiving extracorporeal membrane oxygenation. Ann Pharmacother. 2020;54:144–50.

Alvarez NH, O’Malley TJ, Abai B, Salvatore DM, DiMuzio PJ, Hirose H. Complications of Peripheral Cannulation Site in obese patients on adult extracorporeal membrane oxygenation. ASAIO J. 2021;67:1294–300.

Mongero LB, Stammers AH, Tesdahl EA, Petersen C, Patel K, Jacobs JP. The use of extracorporeal membrane oxygenation in COVID-19 patients with severe cardiorespiratory failure: the influence of obesity on outcomes. J Extra Corpor Technol. 2021;53:293–8.

Balik M, Svobodova E, Porizka M, Maly M, Brestovansky P, Volny L, et al. The impact of obesity on the outcome of severe SARS-CoV-2 ARDS in a high volume ECMO centre: ECMO and corticosteroids support the obesity paradox. J Crit Care. 2022;72:154162.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Djordjevic I, Ivanov B, Sabashnikov A, Gaisendrees C, Gerfer S, Suhr L, et al. Impact of obesity on In-Hospital outcomes in veno-arterial ECMO patients. Heart Lung Circ. 2022;31:1393–8.

Javidfar J, Zaaqoq AM, Labib A, Barnett AG, Hayanga JA, Eschun G et al. Morbid obesity’s impact on COVID-19 patients requiring venovenous extracorporeal membrane oxygenation: the covid-19 critical care consortium database review. Perfusion. 2023;:2676591231156487.

Heuts S, Mariani S, van Bussel BCT, Boeken U, Samalavicius R, Bounader K, et al. The relation between obesity and mortality in Postcardiotomy Venoarterial membrane oxygenation. Ann Thorac Surg. 2023;116:147–54.

Parameswaran K, Todd DC, Soth M. Altered respiratory physiology in obesity. Can Respir J. 2006;13:203–10.

Spinelli E, Colussi G, Dal Santo G, Scotti E, Marongiu I, Garbelli E, et al. Atelectasis, Shunt, and worsening oxygenation following reduction of respiratory rate in healthy pigs undergoing ECMO: an experimental lung imaging study. Front Physiol. 2021;12:663313.

Rabah H, Rabah A. Extracorporeal membrane oxygenation (ECMO): what we need to know. Cureus. 2022;14:e26735.

Pickkers P. The obesity Paradox in patients in need of extracorporeal membrane oxygenation. Am J Respir Crit Care Med. 2023;208:649–50.

Chait A, den Hartigh LJ. Adipose tissue distribution, inflammation and its metabolic consequences, including Diabetes and Cardiovascular Disease. Front Cardiovasc Med. 2020;7:22.

Ghigliotti G, Barisione C, Garibaldi S, Fabbi P, Brunelli C, Spallarossa P, et al. Adipose tissue immune response: novel triggers and consequences for chronic inflammatory conditions. Inflammation. 2014;37:1337–53.

Download references

Acknowledgements

Not applicable.

Author information

Authors and affiliations.

Department of Geriatric, HuZhou Third Municipal Hospital, the Affiliated Hospital of Huzhou University, 2088 Tiaoxi East Road, Wuxing District, Huzhou City, Zhejiang Province, China

Xinhua Huang & Xiaoqing Lin

You can also search for this author in PubMed   Google Scholar

Contributions

XH and XL developed the concept for this review. XH designed the study. Data collection and analysis were completed by XH and XL interpreted the results. Creation of figures was completed by XH and XL wrote the article. All authors approved the final manuscript.

Corresponding author

Correspondence to Xiaoqing Lin .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Huang, X., Lin, X. Impact of obesity on outcomes of extracorporeal membrane oxygenation support: a systematic review and meta-analysis. BMC Pulm Med 24 , 157 (2024). https://doi.org/10.1186/s12890-024-02971-5

Download citation

Received : 09 December 2023

Accepted : 18 March 2024

Published : 28 March 2024

DOI : https://doi.org/10.1186/s12890-024-02971-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Extracorporeal membrane oxygenation

BMC Pulmonary Medicine

ISSN: 1471-2466

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Systematic Literature Review of Guidelines on Budget Impact Analysis for Health Technology Assessment

Affiliations.

• 1 Department of Community Medicine and School of Public Health, Post Graduate Institute of Medical Education and Research, Sector-12, Chandigarh, 160012, India.
• 2 International Decision Support Initiative (iDSI), Imperial College, London, United Kingdom.
• 3 Department of Community Medicine and School of Public Health, Post Graduate Institute of Medical Education and Research, Sector-12, Chandigarh, 160012, India. [email protected].
• PMID: 33956308
• DOI: 10.1007/s40258-021-00652-6

Objectives: The objective of this systematic review was to review the recommendations for the conduct of a budget impact analysis in national or organisational guidelines globally.

Methods: We searched several databases including MELDINE, EMBASE, The Cochrane Library, National Guideline Clearinghouse, HTA Database (International Network of Agencies for Health Technology Assessment), Econlit and IDEAS Database (RePEc, Research Papers in Economics). The OVID platform was used to run the search in all databases simultaneously. In addition, a search of the grey literature was also conducted. The timeframe was set from 2000 to 2020 with language of publication restricted to English.

Results: A total of 13 publications were selected. All the countries where financing of health is predominantly tax funded with public provisioning recommend a healthcare payer (government) perspective. However, countries where a healthcare payer includes a mix of federal government, communities, hospital authorities and patient communities recommend a complementary analysis with a wider societal perspective. While four guidelines prefer a simple cost calculator for costing, the rest rely on a decision modelling approach. None of the guidelines recommend discounting except the Polish guidelines, which recommend discounting at 5%. Only two countries, Belgium and Poland, mention that indirect costs, if significant, should be included in addition to direct costs.

Conclusions: The comparative cross-country analysis shows that a standard set of recommendations cannot be directly useful for all as there are contextual differences. Thus, budget impact analysis guidelines must be carefully contextualised in the policy environment of a country so as to reflect the dynamics of health systems.

© 2021. The Author(s), under exclusive licence to Springer Nature Switzerland AG.

Publication types

• Systematic Review
• Technology Assessment, Biomedical*

• Study Guides
• Homework Questions

Analysis of Zombie Scholarship Literature Review

• Arts & Humanities