health economics literature review

• Systematic review
• Open access
• Published: 15 July 2019

Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review

• Sarah Louise Elin Roberts   ORCID: orcid.org/0000-0002-6807-9830 1 ,
• Andy Healey 1 , 2 &
• Nick Sevdalis 2  

Implementation Science volume  14 , Article number:  72 ( 2019 ) Cite this article

22k Accesses

71 Citations

50 Altmetric

Metrics details

Economic evaluation can inform whether strategies designed to improve the quality of health care delivery and the uptake of evidence-based practices represent a cost-effective use of limited resources. We report a systematic review and critical appraisal of the application of health economic methods in improvement/implementation research.

A systematic literature search identified 1668 papers across the Agris, Embase, Global Health, HMIC, PsycINFO, Social Policy and Practice, MEDLINE and EconLit databases between 2004 and 2016. Abstracts were screened in Rayyan database, and key data extracted into Microsoft Excel. Evidence was critically appraised using the Quality of Health Economic Studies (QHES) framework.

Thirty studies were included—all health economic studies that included implementation or improvement as a part of the evaluation. Studies were conducted mostly in Europe (62%) or North America (23%) and were largely hospital-based (70%). The field was split between improvement ( N  = 16) and implementation ( N  = 14) studies. The most common intervention evaluated (43%) was staffing reconfiguration, specifically changing from physician-led to nurse-led care delivery. Most studies ( N  = 19) were ex-post economic evaluations carried out empirically—of those, 17 were cost effectiveness analyses. We found four cost utility analyses that used economic modelling rather than empirical methods. Two cost-consequence analyses were also found. Specific implementation costs considered included costs associated with staff training in new care delivery pathways, the impacts of new processes on patient and carer costs and the costs of developing new care processes/pathways. Over half (55%) of the included studies were rated ‘good’ on QHES. Study quality was boosted through inclusion of appropriate comparators and reporting of incremental analysis (where relevant); and diminished through use of post-hoc subgroup analysis, limited reporting of the handling of uncertainty and justification for choice of discount rates.

Conclusions

The quantity of published economic evaluations applied to the field of improvement and implementation research remains modest; however, quality is overall good. Implementation and improvement scientists should work closely with health economists to consider costs associated with improvement interventions and their associated implementation strategies. We offer a set of concrete recommendations to facilitate this endeavour.

Peer Review reports

Both improving health care and implementation of evidence-based practices are receiving increasing attention within the wider applied health research field. A recent editorial in Implementation Science [ 1 ] discussed the importance implementation science places on the robustness and validity of health economic evaluations and the benefits gained by properly evaluating both implementation and improvement interventions. We define improvement science as the scientific approach to achieving better patient experience and outcomes through changing provider behaviour and organisation, using systematic change methods and strategies [ 2 ]. We define implementation science as the scientific study of methods to promote the uptake of research findings into routine health care practice or policy [ 2 ].

This paper presents a review of the application of economic evaluation to evaluative studies of service improvement initiatives and interventions focused on facilitating the implementation of evidence into practice. The aim of economic evaluation is to present evidence on the costs and consequences (in terms of patient outcomes) of quality improvement strategies and methods for increasing the uptake of evidence-based practices compared to the ‘status quo’. In doing so, it informs whether specific initiatives are (or have been) a worthwhile (or ‘cost-effective’) use of the limited resources of health systems.

Depending on the service and population context, the methods used in economic evaluations can vary depending on the perspective taken. This can range from a narrow assessment of patient outcomes alongside immediate health care provider cost impacts through to the quantification of costs and consequences affecting other (non-health related) sectors, organisations and wider society. In health programme evaluation, economic evaluations are most frequently carried out ‘ex-post’ or ‘after the fact’, using empirical methods applied to cost and outcome data extracted from trials or other research designs used to evaluate initiatives being tested in specific populations and settings. Economic evaluations can also be applied ‘ex-ante’—to inform option appraisal and pre-implementation decision making using available evidence and modelling to simulate the costs and outcomes of alternatives, e.g. in relation to population scale up or geographical spread of strategies and methods for improvement and evidence uptake.

While economic evaluation has become an integral part of health technology assessment, its application within improvement and implementation evaluative research remains relatively limited [ 1 ]. In two earlier reviews (Hoomans et al. in 2007 [ 3 ] and earlier Grimshaw et al. in 2004 [ 4 ]), the use of economic methods in evaluating the implementation of evidence-based guidelines was examined, and the authors found evidence of limited quality and scope for understanding the cost-effectiveness of implementation strategies. It is now over a decade since these reviews were published, hence a fresh evidence review, synthesis and appraisal is required.

The aim of this study was to examine what advances have been made in the use of economic analysis within implementation and improvement science research, specifically in relation to the quantity and quality of published economic evidence in this field; and to what extent economic evaluations have considered implementation and improvement as part of a holistic approach to evaluating interventions or programmes within the applied health arena.

Search strategy

A systematic review methodology was undertaken. A search strategy was developed to capture evidence published after 2003 (the date of most recent evidence review) and the last searches were performed on 16th March 2016. The searches were performed on the following databases: Agris, Embase, Global Health, HMIC, PsycINFO, Social Policy and Practice, MEDLINE and EconLit. These databases were chosen to attempt to capture the widest range of health improvement, social scientific and health economic studies.

The search strategy (Table  1 ) was designed to capture studies that had a quantitative economic element (i.e. costs and outcomes based on randomised trial data, observational study data or synthesis of the wider empirical evidence base to support economic modelling). The search was conducted to be inclusive of studies whereby behavioural interventions for quality improvement and implementation of evidence into practice were evaluated as well as initiatives around re-design or adjustment to care pathways or reconfiguration of staffing inputs for the purpose of quality improvement.

We searched across a wide range of clinical settings, including primary, secondary and tertiary care and public health.

The completed search results were downloaded into Endnote X6 for citation management and deduplication. Screening was done in Rayyan, a web-based literature screening program [ 6 ]. Rayyan allows for easy abstract and full text screening of studies, custom inclusion and exclusion criteria, as well as custom tags or labels that can be added to each entry. Studies were initially screened using the inclusion/exclusion criteria outlined in the next section, on title and abstract only (by SLER); studies that were borderline for inclusion were more thoroughly screened by examining their full text. The reference lists of the studies were checked for any related studies that were not picked up by the search.

Inclusion and exclusion criteria

Studies were included if they:

Were published in the English language

Reported on a completed study

Study protocols, methodological papers or conference abstracts were excluded (after additional searches had been performed to ensure that full papers had not been subsequently published).

Were published after 2003 and before 16th March 2016

Were conducted in public health, primary, secondary or tertiary care

Further, studies were included if they covered aspects of:

Implementation

Quality/service improvement

Health or clinical service delivery

Staff behaviour change

Patient behaviour change

And they also:

Had patient focused outcomes or outcomes as overall service improvement that would improve patient outcomes or care, expressed as quantifiable outcomes

Had economic elements, expressed as quantifiable outcomes

Reported one of the following health economic methodologies:

Cost effectiveness analysis

Cost-utility analysis

Cost-benefit analysis

Cost-consequence analysis

Burden of disease

The following study designs were included:

Randomised controlled trials

Hybrid effectiveness-implementation trials

Comparative controlled trials without random assignment

Before and after studies

Systematic reviews

Time series study design

Studies or papers that did not fall within the above criteria were excluded. No geographical exclusions were applied. Cost-only studies were not included as the aim of this review was to establish the extent that both costs and benefits were being considered as part of a holistic approach to evaluation of implementation and improvement interventions.

To mitigate for potential selection bias after screening, keyword searching was done in Rayyan for the main keywords within the excluded categories (primarily, those that were deemed to be topic-relevant but not containing economic methods). These were then re-screened by the first author. Studies that included only minimal discussion of costs or costing with no evidence of application of appropriate, standard costing methods (as per the criteria above) were excluded.

Data extraction

Screened studies were downloaded from Rayyan and transferred into a template developed in Microsoft Excel 2016 for detailed data extraction. During screening, each included study was tagged in Rayyan with the reasons for inclusion, type of economic evaluation (see Table  2 ), which economic modelling method used (if applicable), whether improvement or implementation study, the health condition covered, the focus of the reported intervention and health care setting. These were cross-checked for accuracy during the data extraction stage. The next stage of the extraction added the country of the study, perspective of the study (healthcare only or ‘societal’), and more detailed information about the economic methods. The latter included whether the evaluation included appropriate comparators (e.g. status quo/the standard care practice), patient outcome measures used, whether costs and outcomes were analysed and reported in the form of incremental cost-effectiveness ratios (ICERs) for cost-effectiveness or cost-utility analyses, how uncertainty was handled and what conclusions were made regarding the cost-effectiveness of the interventions under evaluation.

Quality appraisal

Each paper’s methodological quality was assessed using the Quality of Health Economic Studies (QHES) standardised framework [ 4 ]. The QHES instrument was designed to more easily tell the difference between high-quality and low-quality studies [ 5 ]. Each study was scored out of 100 based on 16 criteria, with points allocated for full and partial assessments against each item (see Appendix in Table  7 for the framework and scoring system). As per standard practice using this framework, the studies were deemed to be of good quality if they attained a score of 75/100 or higher [ 5 ].

Studies included

Figure  1 shows the flow of studies through the screening stages of the systematic review.

Consort diagram

In total, the initial search strategy identified 1668 articles, of which 1566 were excluded, 1525 during the initial screen and 41 following full text screening. Reasons for exclusion were as follows: the study did not include implementation or quality improvement research aspects (575); it did not include economic aspects (447); was not within a health care/public health setting (437); it was in a language other than English (22); it was incomplete (19); or it was not a full refereed publication (e.g. conference abstracts, doctoral theses) (37).

Thirty studies were included in the final evidence review and synthesis.

Descriptive analysis of the evidence base

Table  3 provides a descriptive overview of the evidence base reported in the 30 reviewed studies. Seventeen of the studies (62%) were European-based (mostly from the UK—12 studies), six studies (23%) were based in either the USA or Canada, four from Australia and one each from Ethiopia, a subset of African countries (Uganda, Kenya and South Africa) and Malaysia. In terms of health care settings, 21 studies were hospital-based, approximately half in inpatient wards and departments, including cardiology, oncology, rheumatology, gastroenterology, geriatrics, endocrinology, orthopaedics and respiratory medicine, or specifically concerning ward management or discharge protocols.

Sixteen of the included studies were identified as ‘improvement’ studies (see Table  3 , panel 1a) and 14 were identified as ‘implementation’ studies (see Table  3 , panel 1b). The definitions from Batalden and Davidoff (2007) that are cited in the introduction were used to stratify the studies. The most common focus of the reviewed improvement studies was staff reconfigurations within a clinical area from medical to nursing staff; for implementation studies, the most common focus was on implementation strategies of new care pathways or novel services.

Table  4 summarises the types of intervention evaluated. The most common intervention type, evaluated in 13 (43%) of the included studies, was staffing reconfiguration for service quality improvement, specifically changing from physician-led to nurse-led delivery of interventions to patients. More broadly, interventions involving general service reorganisation or changes to existing systems of care were the primary focus in ten (33%) of studies reviewed.

Nineteen studies were ex-post economic evaluations of which 17 were CEAs with one CUA [ 7 , 12 , 14 , 15 , 17 ] [ 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 33 ]. All these evaluations compared a new intervention against current practice. There were also four further CUAs that used economic modelling rather than empirical methods [ 8 , 9 , 10 , 34 ], and two cost-consequence analyses [ 16 , 35 ]. Three of the included studies were literature reviews [ 11 , 13 , 36 ].

Specific implementation costs, such as those associated with training staff in new care delivery pathways, the impacts of new processes on patient and carer costs and the costs of developing the new processes were considered by six of the reviewed studies. Scenario analysis for rollout or scaling up was included in three of the studies, and potential funding sources were considered by one study.

Twenty-two of the papers were included in the QHES economic quality appraisal: as the quality scale is designed to evaluate cost-minimisation, cost-effectiveness and cost-utility studies [ 5 ], the literature reviews, meta-analyses or commentaries were excluded for this component. Of the excluded papers, four were systematic reviews and four were papers that did not report on specific studies. The QHES instrument contains 16 dimensions and an outline of the dimensions, the average score and the percentage of the papers reaching the perfect score for each dimension can be found in Table  5 . While most of the papers in this study reached the threshold of being ‘good’ studies, the scores are gained mostly in the same areas in each paper. The average quality score was 76 out of a possible 100 (Fig.  2 ). Thirteen of the studies (62%) attained a ‘good’ score of over 75. Only one study [ 33 ] obtained a ‘perfect’ score of 100 points. Improvement studies performed overall better than implementation studies on the QHES.

Quality appraisal of economic evidence—distribution of QHES instrument scores

The best performing QHES dimensions were the methodological dimensions. Incremental analysis with a relevant comparator (dimension 6) was used in all but one study, and in 81% of studies the data sources for the analysis were from randomised controlled trials, the highest scoring type of evidence in the QHES instrument (Table 6 ). The costing element, covered by dimension 9, performed poorly overall. While three quarters of studies gave details of what methodology was used to quantify service inputs (such as use of self-report service use schedules) and the sources and methods used for estimating unit costs, only two gave justification for why they chose that method. By comparison, there was justification for the use of effectiveness measures and study outcomes given in two-thirds of studies.

Discount rates were correctly applied and stated when adjusting for timing of costs and benefits in all cases where measured costs and outcomes extended beyond 1 year.

A little over a quarter of the included studies declared the perspective of their analysis and gave a justification for the perspective used. Only a third gave details of how parameter uncertainty was addressed in relation to the study conclusions. Justification for chosen discount rates was not provided in around half the studies that used them. Where subgroup analysis was carried out, this was done post-hoc rather than being pre-planned with a clear a priori justification for the use of the chosen subgroups.

Reflections on the evidence

The aim of this review was to critically evaluate the application of economic analysis within implementation and service improvement evaluative research in recent years. The results of evaluating the 30 included papers paint a picture of an area of research that is still developing. The reviewed studies were generally of good quality. However, we found that there were aspects of improvement and implementation that were not adequately covered in many studies. These reflect particularly project costs relating to managerial and clinical time allocated to preparatory work and training and education as well as ongoing costs linked to monitoring care quality and outcomes—all of which are known strategies for successful implementation [ 37 ]. Only six out of 30 studies included an explicit assessment of these type of ‘hidden’ costs of improvement and implementation strategies. This risks underestimating the cost impacts of change and could represent a missed opportunity to develop evidence about the likely comparative magnitude and importance of fixed and recurrent costs that are integral to the scale up and spread of improvement- and implementation-focussed initiatives.

A further reflection: many of the economic studies picked up in our review were linked to wider studies built around more traditional evaluative research designs, specifically randomised controlled trials. There was no evidence that economic methods have as yet been integrated into more advanced evaluative designs within the fields of improvement and implementation design, particularly ‘hybrid’ designs [ 38 , 39 ] that aim to jointly test clinical effectiveness of the evaluated health intervention on patient outcomes and, simultaneously, effectiveness of implementation strategies in embedding the clinical intervention within an organisation or service. This may reflect the fact that hybrid designs are a more recent methodological development, which requires further integration into traditional health care evaluations.

Furthermore, and in relation to the wider role of health economic evaluations within the improvement and implementation science arena, we found that all of the studies included in our review were empirical and ex-post in nature. The studies evaluated costs and outcomes retrospectively using data over a period of time following the introduction of a specific improvement or implementation initiative. This is certainly valuable information for decision makers in making decisions about already applied interventions and in building up an economic evidence base around these interventions. However, it also suggests that economic analysis, and particularly economic modelling, currently at least appears to have a less important role in informing decisions over which options to pursue at earlier stages of implementing change, and in the appraisal of spread and scale up within wider populations. Such earlier phase economic analyses were simply not found in our review. We reflect that either this type of economic analysis is not happening—hence there is a significant gap in the application of economic considerations in improvement and implementation policy decisions; or that such analyses may indeed be undertaken but being less likely to be reported in academic publications and thus under-represented in our review. We cannot rule out either possibility based on this review. Our collective experience suggests that more nuanced economic analyses than simply consideration of ‘costs’ should be carried out in early phases of implementation and improvement programme planning; prospective economic modelling offers a way forward for health care improvers and policy makers planning scale up of evidence interventions.

Quality of the evidence

Comparison between economic studies identified in a previous review carried out by Hoomans et al. (covering the immediately preceding period 1998 to 2004) with those identified in this review (2004 to 2016) shows evidence of a general improvement in quality over the past two decades, with the caveat that the two reviews used different quality appraisal frameworks. For example, only 42% of studies reviewed by Hoomans et al. included evaluation of costs and outcomes against ‘standard practice/status quo’ comparators, compared to 95% of studies in our review. Likewise, costing methodology was only deemed adequate in 11% of cases included in the Hoomans et al. review, compared to 76% of the studies in this review. Justification for the outcome measures used was not reported in any of the studies included in Hoomans et al. but reported in 68% of studies included here. This is a welcome improvement of applied economics within health care implementation and improvement research. We attribute it at least partly to improvements in reporting economic analyses over time, which would appear to have made an impact on the studies we captured. Additionally, the expanding application of health economic evaluations within the improvement and implementation sphere where high-quality study reporting has been a major recent focus has also plausibly contributed to improved reporting. Future evidence reviews will confirm whether this pattern is sustained over time.

Strengths and limitations

This review offers an updated synthesis of an emerging field of economics evaluations of health care intervention evaluations covering both implementation and improvement science studies. The strict inclusion criteria mean that the reviewed evidence is cohesive. The systematic appraisal we carried out also allows us a longitudinal critique of the quality of economic studies in this field. Despite not being able to directly compare the quality assessment from the previous reviews, we would argue that the QHES used here is based on Drummond’s guidelines (used in prior reviews) and is designed to cover the same topics, but offers a simpler, quantifiable format that is easier to apply. [ 32 ]

This review has some limitations. First, while our search strategy was quite broad, our inclusion criteria were strict, which may have limited the number of studies that we identified and synthesised. We aimed to clearly demarcate the economic analyses carried out within healthcare implementation and improvement interventions research—and to explicitly include papers that included both costs and benefits, and so did not include cost-only studies. We also only considered papers reported in English. Taken together, these criteria are stricter than those applied to prior reviews, which were more inclusive of qualitative outcomes and costing studies.

Implications for implementation and improvement research and future directions

Our review demonstrates an increasing number of health economic evaluations nested within implementation and improvement research studies, which further appear to be improving in methodological quality in recent years. Based on our review, we offer the following recommendations and areas for improvement in the continued application of health economic methods to improvement and implementation science evaluative research:

Utilise published guidance on conducting economic evaluation in implementation research and quality improvement projects. Existing implementation frameworks [ 40 ] make reference to the need to consider costs as part of an evaluative research strategy, but do not specify how this is to be done. The relationship between implementation outcomes, service outcomes and patient outcomes is central to understanding the benefits and costs and overall cost-effectiveness of an intervention.

Include detailed consideration of the measurement of the resource implications and ‘hidden’ costs relating to wider support activities required to initiate service improvement or to implement evidence into practice (e.g. costs of manualising an intervention; costs of developing and delivering train-the-trainers interventions as implementation strategies and so on).

Ensure that economic methods become fully integrated into the application of more recent methodological advancements in the evaluative design of improvement and implementation strategies, including ‘hybrid’ designs that seek to jointly test impact on implementation and patient outcomes. This would also provide an opportunity to explore the inter-linkages and relationships between implementation outcomes and economic measures of impact and the cost-effectiveness of improvement and implementation strategies.

While most of the economic studies included in this review were both ex-post and empirical, we would also highlight the value of ex-ante economic evaluation in policy-making contexts. This could be informative either at the early phase of an improvement or implementation project, to guide choices over which options are most likely to yield a cost-effective use of resources (and to rule out those that are likely to be excessively costly compared to expected benefits), or for quantifying the benefits and costs of spread of best practice and delivery at scale.

Finally, we would strongly recommend use of published guidelines and quality assurance frameworks to guide both the design and reporting of economic evaluations. Examples include the QHES framework (used here), the Consolidated Health Economic Reporting Standards (CHEERS) guidance [ 32 ] or the Drummond criteria [ 31 ].

Economic evaluation can inform choices over whether and how resources should be allocated to improve services and for implementing evidence into health care practice. Our systematic review of the recent literature has shown that the quality of economic evidence in the field of improvement and implementation science has improved over time, though there remains scope for continued improvement in key areas and for increased collaboration between health economics and implementation science.

Hoomans T, Severens JL. Economic evaluation of implementation strategies in health care. Implement Sci. 2014;9(1):168.

Article   PubMed   PubMed Central   Google Scholar  

Batalden PB, Davidoff F. What is “quality improvement” and how can it transform healthcare? Qual Saf Health Care. 2007;16(1):2.

Hoomans T, Evers SMAA, Ament AJHA, Hübben MWA, Van Der Weijden T, Grimshaw JM, et al. The methodological quality of economic evaluations of guideline implementation into clinical practice: a systematic review of empiric studies. Value Health. 2007;10(4):305–16.

Article   PubMed   Google Scholar  

Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Int J Technol Assess Health Care. 2005;21(1):149.

Article   Google Scholar  

Ofman JJ, Sullivan SD, Neumann PJ, Chiou C-F, Henning JM, Wade SW, et al. Examining the value and quality of health economic analyses: implications of utilizing the QHES. J Manag Care Pharm. 2003;9(1):53–61.

PubMed   Google Scholar  

Elmagarmid A, Fedorowicz Z, Hammady H, Ilyas I, Khabsa M, Ouzzani M. Rayyan: a systematic reviews web app for exploring and filtering searches for eligible studies for Cochrane reviews. In: Evidence-informed public health: opportunities and challenges. Abstracts of the 22nd Cochrane colloquium. Hyderabad: Wiley; 2014. p. 21–6.

Google Scholar  

Latour CHM, Bosmans JE, van Tulder MW, de Vos R, Huyse FJ, de Jonge P, et al. Cost-effectiveness of a nurse-led case management intervention in general medical outpatients compared with usual care: an economic evaluation alongside a randomized controlled trial. J Psychosom Res. 2007;62(3):363–70.

Burr JM, Mowatt G, Hernández R, Siddiqui MA, Cook J, Lourenco T, Ramsay C, Vale L, Fraser C, Azuara-Blanco A, Deeks J. The clinical effectiveness and cost-effectiveness of screening for open angle glaucoma: a systematic review and economic evaluation. Health Technol Assess (Winch. Eng.). 2007;11(41):1–90.

Burr JM, Botello-Pinzon P, Takwoingi Y, Hernández R, Vazquez-Montes M, Elders A, Asaoka R, Banister K, van der Schoot J, Fraser C, King A, Lemij H, Sanders R, Vernon S, Tuulonen A, Kotecha A, Glasziou P, Garway-Heath D, Crabb D, Vale L, Azuara-Blanco A, Perera R, Ryan M, Deeks J, Cook J. Surveillance for ocular hypertension : an evidence synthesis and economic evaluation. Health Technol Assess. 2012;16(29):1–272.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Robertson C, Arcot Ragupathy SK, Boachie C, Dixon JM, Fraser C, Hernández R, et al. The clinical effectiveness and cost-effectiveness of different surveillance mammography regimens after the treatment for primary breast cancer: systematic reviews registry database analyses and economic evaluation. Health Technol Assess (Winch. Eng.). 2011;15(34):v–322.

CAS   Google Scholar  

Umscheid CA, Williams K, Brennan PJ. Hospital-based comparative effectiveness centers: translating research into practice to improve the quality, safety and value of patient care. J Gen Intern Med. 2010;25(12):1352–5.

Albers-Heitner CP, Joore MA, Winkens RA, Lagro-Janssen AL, Severens JL, Berghmans LC. Cost-effectiveness of involving nurse specialists for adult patients with urinary incontinence in primary care compared to care-as-usual: an economic evaluation alongside a pragmatic randomized controlled trial. Neurourol Urodyn. 2012;31(4):526–34.

Article   CAS   PubMed   Google Scholar  

Faulkner A, Mills N, Bainton D, Baxter K, Kinnersley P, Peters TJ. Sharp D. a systematic review of the effect of primary care-based service innovations on quality and patterns of referral to specialist secondary care. Br J Gen Pract. 2003;53(496):878–84.

PubMed   PubMed Central   Google Scholar  

Bauer JC. Nurse practitioners as an underutilized resource for health reform: evidence-based demonstrations of cost-effectiveness. J Am Acad Nurse Pract. 2010;22(4):228–31.

Brunenberg DE, van Steyn MJ, Sluimer JC, Bekebrede LL, Bulstra SK, Joore MA. Joint recovery programme versus usual care: an economic evaluation of a clinical pathway for joint replacement surgery. Med Care. 2005:1018–26.

Dawes HA, Docherty T, Traynor I, Gilmore DH, Jardine AG, Knill-Jones R. Specialist nurse supported discharge in gynaecology: a randomised comparison and economic evaluation. Eur J Obstet Gynecol Reprod Biol. 2007;130(2):262–70.

Furze G, Cox H, Morton V, Chuang LH, Lewin RJ, Nelson P, Carty R, Norris H, Patel N, Elton P. Randomized controlled trial of a lay-facilitated angina management programme. J Adv Nurs. 2012;68(10):2267–79.

Judd WR, Stephens DM, Kennedy CA. Clinical and economic impact of a quality improvement initiative to enhance early recognition and treatment of sepsis. Ann Pharmacother. 2014;48(10):1269–75.

Kifle YA, Nigatu TH. Cost-effectiveness analysis of clinical specialist outreach as compared to referral system in Ethiopia: an economic evaluation. Cost Eff Resource Allocation. 2010;8(1):13.

Kilpatrick K, Kaasalainen S, Donald F, Reid K, Carter N, Bryant-Lukosius D, Martin-Misener R, Harbman P, Marshall DA, Charbonneau-Smith R, DiCenso A. The effectiveness and cost-effectiveness of clinical nurse specialists in outpatient roles: a systematic review. J Eval Clin Pract. 2014;20(6):1106–23.

Maloney S, Haas R, Keating JL, Molloy E, Jolly B, Sims J, Morgan P, Haines T. Breakeven, cost benefit, cost effectiveness, and willingness to pay for web-based versus face-to-face education delivery for health professionals. J Med Internet Res. 2012;14(2):e47.

Mortimer D, French SD, McKenzie JE, Denise AO, Green SE. Economic evaluation of active implementation versus guideline dissemination for evidence-based care of acute low-back pain in a general practice setting. PLoS One. 2013;8(10):e75647.

Purshouse RC, Brennan A, Rafia R, Latimer NR, Archer RJ, Angus CR, Preston LR, Meier PS. Modelling the cost-effectiveness of alcohol screening and brief interventions in primary care in England. Alcohol Alcohol. 2012;48(2):180–8.

Rachev BT. The economics of health service transformation: a business model for care coordination for chronic condition patients in the UK and US. Clin Gov An Int J. 2015;20(3):113–22.

Tappenden P, Campbell F, Rawdin A, Wong R, Kalita N. The clinical effectiveness and cost-effectiveness of home-based, nurse-led health promotion for older people: a systematic review. Health Technol Assess (Winch. Eng.). 2012;16(20):1.

Tappenden P, Chilcott J, Brennan A, Squires H, Glynne-Jones R, Tappenden J. Using whole disease modeling to inform resource allocation decisions: economic evaluation of a clinical guideline for colorectal cancer using a single model. Value Health. 2013;16(4):542–53.

Vestergaard AS, Ehlers LH. A health economic evaluation of stroke prevention in atrial fibrillation: guideline adherence versus the observed treatment strategy prior to 2012 in Denmark. Pharmacoeconomics. 2015;33(9):967–79.

Williams KS, Assassa RP, Cooper NJ, Turner DA, Shaw C, Abrams KR, Mayne C, Jagger C, Matthews R, Clarke M, McGrother CW. Clinical and cost-effectiveness of a new nurse-led continence service: a randomised controlled trial. Br J Gen Pract. 2005;55(518):696–703.

Williams J, Russell I, Durai D, Cheung WY, Farrin A, Bloor K, Coulton S, Richardson G. What are the clinical outcome and cost-effectiveness of endoscopy undertaken by nurses when compared with doctors? A multi-institution nurse endoscopy trial (MINuET). Health Technol Assess. 2006;10(40):1–93.

Yarbrough PM, Kukhareva PV, Spivak ES, Hopkins C, Kawamoto K. Evidence-based care pathway for cellulitis improves process, clinical, and cost outcomes. J Hosp Med. 2015;10(12):780–6.

Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ. 1996;313(7052):275–83.

Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, Augustovski F, Briggs AH, Mauskopf J, Loder E. Consolidated health economic evaluation reporting standards (CHEERS) statement. Cost Eff Resource Allocation. 2013;11(1):6.

Afzali HH, Gray J, Beilby J, Holton C, Karnon J. A model-based economic evaluation of improved primary care management of patients with type 2 diabetes in Australia. Appl Health Econ Health Policy. 2013;11(6):661–70.

Hernández RA, Jenkinson D, Vale L, Cuthbertson BH. Economic evaluation of nurse-led intensive care follow-up programmes compared with standard care: the PRaCTICaL trial. Eur J Health Econ. 2014;15(3):243–52.

Karnon J, Partington A, Horsfall M, Chew D. Variation in clinical practice: a priority setting approach to the staged funding of quality improvement. Appl Health Econ Health Policy. 2016;14(1):21–7.

Mdege ND, Chindove S, Ali S. The effectiveness and cost implications of task-shifting in the delivery of antiretroviral therapy to HIV-infected patients: a systematic review. Health Policy Plan. 2012;28(3):223–36.

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. 2015;10(1):21.

Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217.

Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB, Jones L, Collins LM, Duan N, Mittman BS, Wallace A, Tabak RG. An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38:1–22.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38(2):65–76.

Download references

Acknowledgements

Not applicable.

This research is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South London at King’s College Hospital NHS Foundation Trust. The authors are members of King’s Improvement Science, which is part of the NIHR CLAHRC South London and comprises a specialist team of improvement scientists and senior researchers based at King’s College London. Its work is funded by King’s Health Partners (Guy’s and St Thomas’ NHS Foundation Trust, King’s College Hospital NHS Foundation Trust, King’s College London and South London and Maudsley NHS Foundation Trust), Guy’s and St Thomas’ Charity, the Maudsley Charity and the Health Foundation. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Author information

Authors and affiliations.

King’s Health Economics, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, David Goldberg Centre, De Crespigny Park, London, SE5 8AF, UK

Sarah Louise Elin Roberts & Andy Healey

Centre for Implementation Science, King’s College London, London, UK

Andy Healey & Nick Sevdalis

You can also search for this author in PubMed   Google Scholar

Contributions

SLER, AH and NS conceptualised the literature review. SLER performed literature searches, screening and analysis. SLER, AH and NS contributed to the writing of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Sarah Louise Elin Roberts .

Ethics declarations

Ethics approval and consent to participate.

Not required.

Consent for publication

Competing interests.

Sevdalis is the Director of London Safety and Training Solutions Ltd., which provides quality and safety training and advisory services on a consultancy basis to healthcare organisation globally. The other authors have no interests to declare.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Roberts, S.L.E., Healey, A. & Sevdalis, N. Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review. Implementation Sci 14 , 72 (2019). https://doi.org/10.1186/s13012-019-0901-7

Download citation

Received : 29 August 2018

Accepted : 29 April 2019

Published : 15 July 2019

DOI : https://doi.org/10.1186/s13012-019-0901-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.

Springer Medicine

Health economics review, latest issues, health economics review 1/2024, health economics review 1/2023, health economics review 1/2022, health economics review 1/2021, health economics review 1/2020, health economics review 1/2019, health economics review 1/2018, health economics review 1/2017, health economics review 1/2016, health economics review 1/2015.

scroll for more

use your arrow keys for more

scroll or use arrow keys for more

About this journal

Health Economics Review is an open-access journal covering all aspects of health economics, from a micro- to macro-scale. We welcome a broad range of theoretical contributions, empirical studies and analyses of health policy with a health economic focus, including the growing importance of health care in developing countries. Research topics include health care finance, health insurance and reimbursement, economics-focused policy analysis and health care management. Our research and authors come from a range of high- middle- and low-income countries.

• Medical Journals
• Webcasts & Webinars
• CME & eLearning
• Newsletters
• ESMO Congress 2023
• 2023 ERS Congress
• ESC Congress 2023
• Advances in Alzheimer’s
• About Springer Medicine
• Diabetology
• Endocrinology
• Gastroenterology
• Geriatrics and Gerontology
• Gynecology and Obstetrics
• Infectious Disease
• Internal Medicine
• Respiratory Medicine
• Rheumatology

• Open access
• Published: 02 December 2017

Systematic reviews of health economic evaluations: a protocol for a systematic review of characteristics and methods applied

• Miriam Luhnen 1 , 2 ,
• Barbara Prediger 3 ,
• Edmund A. M. Neugebauer 4 , 5 &
• Tim Mathes 3  

Systematic Reviews volume  6 , Article number:  238 ( 2017 ) Cite this article

6151 Accesses

9 Citations

2 Altmetric

Metrics details

The number of systematic reviews of economic evaluations is steadily increasing. This is probably related to the continuing pressure on health budgets worldwide which makes an efficient resource allocation increasingly crucial. In particular in recent years, the introduction of several high-cost interventions presents enormous challenges regarding universal accessibility and sustainability of health care systems. An increasing number of health authorities, inter alia, feel the need for analyzing economic evidence.

Economic evidence might effectively be generated by means of systematic reviews. Nevertheless, no standard methods seem to exist for their preparation so far.

The objective of this study was to analyze the methods applied for systematic reviews of health economic evaluations (SR-HE) with a focus on the identification of common challenges.

Methods/design

The planned study is a systematic review of the characteristics and methods actually applied in SR-HE. We will combine validated search filters developed for the retrieval of economic evaluations and systematic reviews to identify relevant studies in MEDLINE (via Ovid, 2015-present). To be eligible for inclusion, studies have to conduct a systematic review of full economic evaluations. Articles focusing exclusively on methodological aspects and secondary publications of health technology assessment (HTA) reports will be excluded. Two reviewers will independently assess titles and abstracts and then full-texts of studies for eligibility. Methodological features will be extracted in a standardized, beforehand piloted data extraction form. Data will be summarized with descriptive statistical measures and systematically analyzed focusing on differences/similarities and methodological weaknesses.

The systematic review will provide a detailed overview of characteristics of SR-HE and the applied methods. Differences and methodological shortcomings will be detected and their implications will be discussed. The findings of our study can improve the recommendations on the preparation of SR-HE. This can increase the acceptance and usefulness of systematic reviews in health economics for researchers and medical decision makers.

Systematic review registration

The review will not be registered with PROSPERO as it does not meet the eligibility criterion of dealing with clinical outcomes.

Peer Review reports

Continuing pressure on health budgets worldwide makes an efficient resource allocation increasingly crucial. In recent years, the introduction of several high-cost interventions presents enormous challenges regarding accessibility and sustainability of health care systems [ 1 , 2 ]. This makes economic considerations more important for health authorities and their decision-making process regarding pricing and reimbursement especially of new interventions.

Systematic reviews of health economic evaluations (SR-HE) can provide evidence about the cost-effectiveness of an intervention within a limited time frame. They are valuable (1) to inform the development of an own economic model, (2) to identify the most relevant studies for a particular decision, and (3) to identify the implicated economic trade-offs [ 3 ]. Moreover, provided that high-quality economic evaluations that exist are sufficiently transferable and demonstrate similar results regarding cost-effectiveness, SR-HE might indicate the most cost-effective intervention.

Jefferson et al. [ 4 ] found that SR-HE show fundamental methodological flaws, especially regarding their search strategy and the application of an appropriate quality assessment tool. Nevertheless, little research has been performed to further develop the methods for SR-HE in the meantime. Standards for the preparation of SR-HE do not seem to exist so far: More recent studies focusing on the available methodological guidelines found that the recommendations still vary widely and are partly imprecise [ 5 , 6 , 7 , 8 ]. It is therefore to be expected that the conduct of SR-HE still varies widely and still shows methodological shortcomings. The aim of this paper is

To provide a detailed overview of the characteristics and applied methods in recently published SR-HE

To identify similarities and differences between the characteristics and methods of SR-HE

To identify common challenges

Methods/Design

We used the PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) 2015 checklist to develop the methods for this systematic review protocol [ 9 ] (please see Additional file 1 ).

Should protocol amendments be necessary, these will be documented including details of the date, changes made, and the rationale for changes.

Literature search

A systematic search in Ovid MEDLINE(R) Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily, and Ovid MEDLINE(R) 1946 to Present will be performed. We will limit the publication date of our search to the period 2015/01/01 to present. A validated search filter for economic evaluations (Emory University [Grady] [ 10 ]) will be combined with a validated filter for the retrieval of systematic reviews (Lee [ 11 ]), as presented in Table  1 . This strategy was chosen as it provides an optimal balance between sensitivity and precision. Search results will be downloaded to EndNote version X7 where duplicates will be identified and removed.

Inclusion criteria

We will include articles available as full-text and written in English, German, French, or Spanish if they fulfill all of the following criteria:

Systematic literature search in at least one electronic database and transparent description of study selection. We will exclude articles applying abbreviated review methods (e.g., scoping reviews and short reviews) as judged by the authors of the SR-HE.

Inclusion of full economic evaluations (i.e., cost-effectiveness/cost-utility/cost-benefit-analyses [ 12 ]) and/or the cost-effectiveness of an intervention was reviewed. Articles reviewing solely partial economic evaluations (like cost-of-illness studies or budget impact analyses) will be excluded.

Objective to answer a cost-effectiveness research question, i.e., we will exclude articles focusing exclusively on methodological aspects (e.g., analysis of methods applied in health economic modeling studies).

Full-text journal article. Protocols, commentaries, editorials, and conference proceedings will be excluded. Likewise, secondary publications of HTA reports will be excluded as the focus of our study will be on the scientific literature instead of documents stemming from regulatory processes within a certain jurisdiction in a health care system.

Study selection

Two reviewers will independently assess the titles and abstracts retrieved in the electronic literature search against the inclusion criteria. Possible eligible full-text articles will be retrieved and screened by two reviewers to reach a final decision about inclusion. Any disagreements will be resolved through discussion or involvement of a third reviewer.

We will prepare a PRISMA flowchart to illustrate the selection process.

Data abstraction

Methodological features will be extracted in a standardized, beforehand piloted data extraction form (Table  2 ). We developed an electronical extraction form in Microsoft Excel 2010 for a previous study (not published yet) in which we analyzed HTA reports of international HTA organizations for the methods applied for SR-HE and adapted it for the purpose of the present study. This approach for data abstraction and data presentation was inspired by the publication of Page et al. [ 13 ] which provides an overview of epidemiology and reporting characteristics of systematic reviews of biomedical research. Data items presented in the included articles will be classified according to the categories depicted in Table  3 . Data will be extracted each by a single reviewer. After extraction of the first articles, a 10% random sample will be verified for accuracy and correctness of data entries by a second reviewer. Discrepancies will be resolved through discussion or third party, if necessary. In case of frequent and/or substantial disagreements, a verification of 100% is intended.

Data analysis and presentation

We will analyze all data using Microsoft Excel 2010. Results for each data item extracted will be presented in tables. For nominal data, we will provide numbers and percentages. We will provide median and ranges for ordinal data.

In order to allow an estimation of the number of SR-HE published per year and to analyze possible changes over time, we will present the number of hits resulting from our search strategy for the years 2015 to 2017.

Since no tool for the critical appraisal of SR-HE exists (comparable e.g., to AMSTAR [A Measurement Tool to Assess Systematic Reviews] [ 14 ]), we will not critically appraise included articles by means of a certain tool but focus on similarities, differences, and methodological shortcomings.

As far as possible, the results of our study will be reported in accordance with the PRISMA guidelines [ 15 ].

Abbreviations

A Measurement Tool to Assess Systematic Reviews

Cost-effectiveness analysis

Patient, intervention, comparison, outcome, setting

Preferred Reporting Items for Systematic review and Meta-Analysis

Prospective Register of Systematic reviews

Systematic reviews of health economic evaluations

European Commission. Inception impact assessment—strengthening of the EU cooperation on. Health Technology Assessment (HTA) Available from: http://ec.europa.eu/smart-regulation/roadmaps/docs/2016_sante_144_health_technology_assessments_en.pdf . Accessed April 19, 2017

OECD. Fiscal sustainability of health systems: bridging health and finance perspectives. Paris: OECD Publishing; 2015.

Google Scholar  

Anderson R. Systematic reviews of economic evaluations: utility or futility? Health Econ. 2010 Mar;19(3):350–64.

Article   PubMed   Google Scholar  

Jefferson T, Demicheli V, Vale L. Quality of systematic reviews of economic evaluations in health care. JAMA. 2002 Jun 5;287(21):2809–12.

Mathes T, Walgenbach M, Antoine SL, et al. Methods for systematic reviews of health economic evaluations: a systematic review, comparison, and synthesis of method literature. Med Decis Mak. 2014 Oct;34(7):826–40.

Article   Google Scholar  

Thielen FW, Van Mastrigt G, Burgers LT, et al. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Expert Rev Pharmacoecon Outcomes Res. 2016 Dec;16(6):705–21.

Article   CAS   PubMed   Google Scholar  

van Mastrigt GA, Hiligsmann M, Arts JJ, et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3). Expert Rev Pharmacoecon Outcomes Res. 2016 Dec;16(6):689–704.

Wijnen B, Van Mastrigt G, Redekop WK, et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3). Expert Rev Pharmacoecon Outcomes Res. 2016 Dec;16(6):723–32.

Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015 Jan 02;349:g7647.

Glanville J, Fleetwood K, Yellowlees A, et al. Development and testing of search filters to identify economic evaluations in MEDLINE and EMBASE. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2009.

Lee E, Dobbins M, Decorby K, et al. An optimal search filter for retrieving systematic reviews and meta-analyses. BMC Med Res Methodol. 2012 Apr 18;12:51.

Article   PubMed   PubMed Central   Google Scholar  

Drummond MF, Sculpher MJ, Claxton K, et al. Methods for the economic evaluation of health care programmes. Oxford: Oxford University Press; 2015.

Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016 May;13(5):e1002028.

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007 Feb 15;7:10.

Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9. w64

Download references

Acknowledgements

Not applicable

No funding will be received for the proposed study.

Availability of data and materials

The datasets generated and analyzed during the current study will be available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Department Health Care and Health Economics, Institute for Quality and Efficiency in Health Care (IQWiG), Im Mediapark 8, 50670, Cologne, Germany

Miriam Luhnen

Faculty of Health, Department of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Haus 38, 51109, Cologne, Germany

Institute for Research in Operative Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Haus 38, 51109, Cologne, Germany

Barbara Prediger & Tim Mathes

Faculty of Health, Brandenburg Medical School – Theodor Fontane, Campus Neuruppin, Fehrbelliner Str. 38, 16816, Neuruppin, Germany

Edmund A. M. Neugebauer

Interdisciplinary Centre for Health Services Research, Witten/Herdecke University, Alfred-Herrhausen-Straße 50, 58448, Witten, Germany

You can also search for this author in PubMed   Google Scholar

Contributions

ML and TM developed and piloted the data extraction form. ML developed the search strategy for the proposed systematic review and drafted the manuscript. TM and BP commented on the manuscript. EAMN supported the conceptualization of the systematic review. All authors reviewed the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Tim Mathes .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) 2015 checklist: recommended items to address in a systematic review protocol. (DOCX 36 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Luhnen, M., Prediger, B., Neugebauer, E.A.M. et al. Systematic reviews of health economic evaluations: a protocol for a systematic review of characteristics and methods applied. Syst Rev 6 , 238 (2017). https://doi.org/10.1186/s13643-017-0639-8

Download citation

Received : 05 May 2017

Accepted : 23 November 2017

Published : 02 December 2017

DOI : https://doi.org/10.1186/s13643-017-0639-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic review
• Economic evaluation
• Reimbursement
• Medical decision making

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Systematic reviews of health economic evaluations: a protocol for a systematic review of characteristics and methods applied

Affiliations.

• 1 Department Health Care and Health Economics, Institute for Quality and Efficiency in Health Care (IQWiG), Im Mediapark 8, 50670, Cologne, Germany.
• 2 Faculty of Health, Department of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Haus 38, 51109, Cologne, Germany.
• 3 Institute for Research in Operative Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Haus 38, 51109, Cologne, Germany.
• 4 Faculty of Health, Brandenburg Medical School - Theodor Fontane, Campus Neuruppin, Fehrbelliner Str. 38, 16816, Neuruppin, Germany.
• 5 Interdisciplinary Centre for Health Services Research, Witten/Herdecke University, Alfred-Herrhausen-Straße 50, 58448, Witten, Germany.
• 6 Institute for Research in Operative Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Haus 38, 51109, Cologne, Germany. [email protected].
• PMID: 29197411
• PMCID: PMC5712099
• DOI: 10.1186/s13643-017-0639-8

Background: The number of systematic reviews of economic evaluations is steadily increasing. This is probably related to the continuing pressure on health budgets worldwide which makes an efficient resource allocation increasingly crucial. In particular in recent years, the introduction of several high-cost interventions presents enormous challenges regarding universal accessibility and sustainability of health care systems. An increasing number of health authorities, inter alia, feel the need for analyzing economic evidence. Economic evidence might effectively be generated by means of systematic reviews. Nevertheless, no standard methods seem to exist for their preparation so far. The objective of this study was to analyze the methods applied for systematic reviews of health economic evaluations (SR-HE) with a focus on the identification of common challenges.

Methods/design: The planned study is a systematic review of the characteristics and methods actually applied in SR-HE. We will combine validated search filters developed for the retrieval of economic evaluations and systematic reviews to identify relevant studies in MEDLINE (via Ovid, 2015-present). To be eligible for inclusion, studies have to conduct a systematic review of full economic evaluations. Articles focusing exclusively on methodological aspects and secondary publications of health technology assessment (HTA) reports will be excluded. Two reviewers will independently assess titles and abstracts and then full-texts of studies for eligibility. Methodological features will be extracted in a standardized, beforehand piloted data extraction form. Data will be summarized with descriptive statistical measures and systematically analyzed focusing on differences/similarities and methodological weaknesses.

Discussion: The systematic review will provide a detailed overview of characteristics of SR-HE and the applied methods. Differences and methodological shortcomings will be detected and their implications will be discussed. The findings of our study can improve the recommendations on the preparation of SR-HE. This can increase the acceptance and usefulness of systematic reviews in health economics for researchers and medical decision makers.

Systematic review registration: The review will not be registered with PROSPERO as it does not meet the eligibility criterion of dealing with clinical outcomes.

Keywords: Economic evaluation; Medical decision making; Reimbursement; Systematic review.

• Cost-Benefit Analysis*
• Delivery of Health Care
• Economics, Medical*
• Review Literature as Topic*
• Systematic Reviews as Topic
• Technology Assessment, Biomedical

• Research article
• Open access
• Published: 24 October 2013

Public health economics: a systematic review of guidance for the economic evaluation of public health interventions and discussion of key methodological issues

• Rhiannon Tudor Edwards 1 ,
• Joanna Mary Charles 1 &
• Huw Lloyd-Williams 1  

BMC Public Health volume  13 , Article number:  1001 ( 2013 ) Cite this article

37k Accesses

83 Citations

31 Altmetric

Metrics details

If Public Health is the science and art of how society collectively aims to improve health, and reduce inequalities in health, then Public Health Economics is the science and art of supporting decision making as to how society can use its available resources to best meet these objectives and minimise opportunity cost. A systematic review of published guidance for the economic evaluation of public health interventions within this broad public policy paradigm was conducted.

Electronic databases and organisation websites were searched using a 22 year time horizon (1990–2012). References of papers were hand searched for additional papers for inclusion. Government reports or peer-reviewed published papers were included if they; referred to the methods of economic evaluation of public health interventions, identified key challenges of conducting economic evaluations of public health interventions or made recommendations for conducting economic evaluations of public health interventions. Guidance was divided into three categories UK guidance, international guidance and observations or guidance provided by individual commentators in the field of public health economics. An assessment of the theoretical frameworks underpinning the guidance was made and served as a rationale for categorising the papers.

We identified 5 international guidance documents, 7 UK guidance documents and 4 documents by individual commentators. The papers reviewed identify the main methodological challenges that face analysts when conducting such evaluations. There is a consensus within the guidance that wider social and environmental costs and benefits should be looked at due to the complex nature of public health. This was reflected in the theoretical underpinning as the majority of guidance was categorised as extra-welfarist.

Conclusions

In this novel review we argue that health economics may have come full circle from its roots in broad public policy economics. We may find it useful to think in this broader paradigm with respect to public health economics. We offer a 12 point checklist to support government, NHS commissioners and individual health economists in their consideration of economic evaluation methodology with respect to the additional challenges of applying health economics to public health.

Peer Review reports

We live in an unequal world and know that inequalities in health and lifetime opportunity are fundamentally linked to inequalities in income. A paradox is emerging–in order to affect health gain in those groups of society that face most socioeconomic challenges, we must trade off our more familiar, if implicit, Western goal of health gain maximisation. In the UK the principle of health gain maximisation underpins the technical appraisal approach taken by the National Institute of Health and Care Excellence (NICE), [ 1 ]. NICE updated its “Guide to the methods of technology appraisal” in 2013 [ 2 ]. The updated version does not appropriate directly to public health; however, the guide maintains its position from the previous version with regards to economic evaluations. The use of cost-effectiveness, especially cost-utility analysis is preferred [ 2 ]. Cost per Quality Adjusted Life Year (QALY) estimates are favoured and appraised in accordance with the £20,000-30,000 threshold [ 2 ]. This is justified by the institute’s focus on maximising health gains [ 2 ]. In health economics, three papers have begun to shape thinking about public health economics. Kelly et al. [ 3 ] set out additional challenges of applying tools of economic evaluation to public health interventions as compared with the evaluation of clinical interventions. These challenges span multiple versus single outcomes, the effect of individual behaviour change upon the successful uptake of interventions, the difficulty in establishing cause and effect due to the multi-faceted nature of public health interventions and the high level of social variation involved in public health interventions. Weatherly et al. [ 4 ] offered additional considerations that health economists should build into their evaluations of public health interventions. These considerations being the need for other approaches due to the limited availability of randomised controlled trials, measurement of a range of outcomes beyond Quality Adjusted Life Years (QALYs), consideration of inter-sectoral costs and consequences which may include wider benefits and spill over effects, and a focus on equity. Payne et al. [ 5 ], introduced the idea of some public health interventions having the characteristics of “complex” interventions, and the subsequent need to measure a much broader range of outcomes than focus on QALYs, suggesting that capability theory may offer one way forward as a means for better capturing such wider benefits [ 6 ]. These three papers focus on, and critique, the traditional toolbox of methods of economic evaluation applied to the evaluation of Public Health interventions [ 7 ]. Looking back, in the UK, Derek Wanless challenged health economists to apply their methods of economic evaluation in a public health setting [ 8 ]. More recently, the NICE Centre for Public Health Excellence has called for health economists to think more broadly about how economics as a parent discipline in its widest sense, can help support those responsible for resource allocation decisions in Public Health. This was taken forward in a Medical Research Council (MRC) Population Health Sciences Research Network (PHSRN) funded workshop on population health economics in Glasgow, May 2012. Though consensus seems to be from the authors above, that the QALY is inadequate in a public health setting; Owen et al. [ 9 ] provide a powerful message that many public health interventions are indeed cost-effective, well below the NICE threshold of £20,000-30,000 per QALY. We observe a growing interest and expectation that public health interventions should be “cost saving” [ 10 ], hence an interest by government, local government and the NHS in return on investment analysis as an alternative to cost-effectiveness analysis [ 11 ].

However, economic evaluations of public health economics are not without challenges; therefore, there is a need of guidance in this field. To examine what guidance currently exists in the field of economic evaluations of public health economics we conducted a systematic review of UK guidance, international guidance and searched for papers offering observations from key commentators of the economic evaluation of public health interventions.

Literature search

We followed a PRISMA [ 12 ] approach to reporting the findings of the systematic review of published guidance for the economic evaluation of public health interventions.

PubMed, MedLine, CRD database, EconLit were searched between September and October 2012 for published guidance of economic evaluation methods for public health interventions. In addition, The Medical Research Council, Joseph Rowntree Foundation, National Institute for Health and Clinical Excellence, World Health Organisation and World Bank websites were also searched for relevant guidance and reference lists of published reviews were scrutinized (e.g. Owen et al. [ 9 ]).

Databases were searched for literature for the period 1990–2012. It was deemed appropriate to narrow the search to this time period as we wanted to include the more recent contributions and the studies found did not generally refer to articles before 1990. We restricted our search to papers published in the English language. Searches were conducted in October 2012.

The search terms used were: public health, public health economics, guidance for economic/econometric evaluation of public health interventions, challenges of public health economics, methods of public health economics, world health organisation, and health economics [see Additional file 1 ].

Study selection

The following exclusion and inclusion criteria were employed during the searches.

Papers were included if:

The paper had a reference in the title and/or abstract to the methods of economic evaluation of public health interventions.

The paper identified key challenges of conducting economic evaluations of public health interventions.

The paper made recommendations for conducting economic evaluations of public health interventions.

The paper was from a national source (e.g. Government or Advisory Group policy documents and reports) or published in a peer-review journal.

Papers were excluded if;

They were not specifically related to the economic evaluation of public health interventions.

They did not provide guidance on the economic evaluation of public health interventions.

They were published in a language other than English.

Papers identified by the searches were screened by reading the abstracts. Articles that matched the inclusion criteria above were obtained and read by RTE, JMC and HLW. It was also necessary to search literature by hand e.g., lists of references of guidance meeting the inclusion criteria. Information was extracted from each paper on the challenges and recommendations of methods to employ when conducting economic evaluations of public health interventions. We wanted to look as widely as possible at relevant health economics specific and public policy guidance on the evaluation of public health interventions.

Data collection process

We developed a summary of guidance and key observations, providing the following information; author, publication date, source of published guidance (e.g., UK or International) and key points. As this review was undertaken in a public policy context with reports the more prominent type of published guidance the authors were unable to adhere to PICO guidelines, instead we provided a summary.

Results will be presented as a narrative review as the search strategy aims to identify UK and international guidance and observations from key commentators. Therefore, results are likely to contain a high level of heterogeneity, which may not permit meta-analysis.

Assessment of theoretical underpinning

To strengthen the narrative review an assessment will be made of the theoretical underpinning of the included guidance. The theoretical paradigms will be categorised based upon whether they related to a macro/micro [ 13 ], welfarist/extra welfarist [ 14 ], capabilities [ 15 ] or behavioural economics approach [ 16 ]. These theoretical underpinnings will serve as a rationale or framework for categorising the papers. It must be noted however that no specific mention was made in the guidance as to its theoretical basis, but rather it is a judgment that we the authors have made. The first paradigm delineates between macro and micro economics. It is suggested that if a set of guidance relates to the evaluation of individual programmes then it belongs in a micro framework [ 13 ]. Otherwise if the guidance looks at outcomes on a population, whole economy level then a macro framework would apply [ 13 ]. Research on macroeconomic modelling uses information fed into general equilibrium models which connect health expenditure growth to its impact on the overall economy [ 17 ]. Health impact assessments conducted by the World Bank can be said to have a strong macro basis. Further a differentiation is made based on whether the theoretical underpinnings of the guidance relates to welfarist or extra-welfarist theory. Welfarism relates to the assumption that it is a measure of utility that’s important when measuring health, that is the utility received from the consumption of healthcare goods and services relative to other goods and services [ 14 ]. It is based on individualism and consequentialism [ 18 ]. Cost-benefit analysis is a good example of a welfarist perspective [ 19 ]. Extra-welfarism on the other hand goes beyond the study of utility and takes health itself and other non-utility measures as the unit of outcome [ 14 ]. Extra-welfarism is based on the underlying idea that rational choice and utility maximising behaviour, the underpinnings of welfarist theory are irrelevant to health behaviours [ 18 ]. Health is the maxim and not welfare. The EQ-5D [ 20 ] and concurrent measurement of QALYs provide a good example of extra-welfarist thinking). The third type of theoretical basis for guidance is the capability approach. Capabilities, according to Sen [ 15 ], are concerned with the ability to achieve functionings such as attachment, role, enjoyment, security and control. Ill health does not reduce quality of life on its own but insofar as it reduces the ability to achieve, for example, independence [ 21 ]. We assess the guidance based upon whether or not it promotes the capability approach [ 15 ] in a robust way, as an alternative to the QALY approach [ 5 ]. Finally, an underlying theory based on behavioural economics studies the behavioural aspects of economic agents and how this affects their decision making [ 16 ]. Behavioural economics does not suggest a rejection of the neoclassical approach to economics, but does advocate the psychological underpinnings of economic analysis. In this way it is argued that the assumptions that underlie theory are adapted to reflect a more realistic view of the world. Here we assess whether or not the guidance is based in behavioural economic theory.

The initial database search provided a total of 36 citations. 36 remained after removing duplicates. Of these, 13 were excluded based upon their executive summary or abstract as they did not meet the criteria. Of the remaining 23, a total of 16 papers met the inclusion criteria (See Figure  1 ).

Flowchart outlining paper selection process for the systematic review.

Due to the heterogeneity of the results yielded by the search, we provide a narrative review below. Guidance relevant to the economic evaluation of public health interventions was divided into the following categories; UK guidance, international guidance and observations of key commentators relevant to the economic evaluation of public health interventions.

Guidance relevant to the economic evaluation of public health interventions

Uk guidance.

Looking beyond the health economics literature to broad economic and public policy guidance, which may be directly relevant to the evaluation of public health interventions, the HM Treasury Green Book [ 22 ] outlines various points related to the treatment of costs and benefits when conducting economic appraisals of public programmes, points which relate well to the area of public health. According to the Green Book guidance, relevant costs and benefits, i.e. those which can be attributed or influenced by the intervention in question should be measured and then valued and care should be taken to avoid spuriously accurate figures [ 22 ]. Costs and benefits should be based on market prices as these are usually the best reflection of their opportunity cost i.e. best alternative use [ 22 ]. Wider social and environmental costs and benefits should be considered although there may not be a market price for these costs and benefits. However, they should not be ignored simply because it is difficult to quantify them.

When it is not possible to express costs and benefits in market prices the ‘willingness to accept measure’ should be used [ 22 ]. This is where patients state the minimum amount they would need to be compensated in order to forgo a good or service. In terms of valuing benefits the preferred measure is also market prices unless the market is distorted and does not reflect truly opportunity costs of resource use. If this is the case techniques such as ‘willingness to pay’ surveys can be used or one can infer a price from consumer behaviour through their ‘revealed preference’.

The Joseph Rowntree Foundation [ 23 ] has published a practical guide to conducting economic evaluation in the social welfare field but many of the principles can be applied to the evaluation of public health interventions. Although the study excludes health care it draws heavily upon health economics to address questions in the social welfare field. Examples of health care economic evaluation are provided but only insofar as to facilitate thinking about applying these methods in a social welfare context. There is a stronger focus on the evaluation of complex public health interventions which, according to the authors, have much more in common with the social welfare field. They outline the main methods of economic evaluation before going on to discuss methods of evaluating specific costs and benefits of different types of interventions. By providing information on how to synthesise costs and benefits they arm the reader with the necessary tools to conduct cost-effectiveness analysis.

A set of guidance has been put forward by Drummond et al. [ 7 ]. In this they acknowledge the additional challenges of applying traditional techniques of economic evaluation to a public health context over and above that of a clinical context. They add that a study by Tarn and Smith [ 24 ] concludes that there is widespread agreement on the main methods of economic evaluation of clinical interventions, whilst there is still some disagreement over specific methodological issues relating to public health. Drummond et al. [ 7 ], reiterated in Weatherly et al. [ 4 ], discussed below, identify four specific methodological challenges in economic evaluation of public health interventions:

Firstly, attributing outcomes to interventions; most published guidelines, including NICE, prefer the use of Randomised Controlled Trials (RCTs) to compare alternatives [ 1 , 2 ]. There are likely to be fewer controlled trials of public health programmes due to the very large sample size required to power pragmatic trials of public health interventions adequately. Follow up in clinical trials is often limited to one or two years at the most, whilst public health programmes could be expected to have an impact over the longer term. Secondly, measuring and valuing outcomes; in economic evaluations of clinical interventions, outcomes are usually measured in natural units or QALYs [ 1 , 2 ]. In the economic evaluation of public health interventions, other outcomes must be considered including effects on individuals not directly targeted by the programme and non-health related outcomes such as education. Some of these outcomes can be incorporated into QALYs, some not. Thirdly, incorporating equity considerations; in many cases the aim of the public health intervention is to reduce inequalities. The normal assumption in economic evaluation methods is that a QALY is of the same value to everyone who receives it. It is possible to look at the distribution of QALY gains between population sub-groups in order to tackle inequalities in health. Fourth, identifying inter-sectoral costs and consequences; the impact of public health interventions can be wide-ranging. The costs and benefits may fall on parts of the public sector not confined to health alone, such as the judicial system, education and housing.

The main guidance for the UK is set out by NICE in its 2009 document “Methods for the development of NICE public health guidance” [ 25 ]. This guidance outlines the conditions under which an economic analysis should be carried out. If a high-quality economic study has already been published which relates to the public health intervention under consideration then no further economic modelling should be attempted. However, when there is paucity of evidence in the literature and when there is expected net benefit from a public health intervention then the guidance indicates that we should proceed with economic evaluation. This guidance is underpinned by an extra-welfarist approach in that, it builds on and adapts NICE guidance for the appraisal of health care technologies [ 1 , 2 ], adhering to the assumption that the aim of public health interventions is to maximise QALYs. Although NICE have not explicitly recommended an ICER above which public health interventions should not be recommended and below which they should, in general, interventions with an ICER of less than £20,000 per QALY gained are considered cost-effective [ 25 ].

The Cabinet Office has published guidance on ‘Social Return on Investment’ (SROI) in public sector settings but the techniques suggested relate well to the area of public health [ 11 ]. Social Return on Investment is a method used to capture value beyond that of financial return. There are two types, evaluative and forecast. Forecast is potentially useful in the planning stage of an intervention and may show how investment could maximise impact, while evaluative SROI is retrospective. In terms of measuring costs and outcomes, this may be done using market prices. The price mechanism is an allocative concept that relates the relative valuations of a good or service by two parties. Where a cost or outcome can be measured using price there is enough information contained within that price to enable a transaction to take place where both parties maximise the benefits from the transaction. However when it is not possible to measure costs and outcomes using price, other methods have to be considered. Here the authors discuss such valuation methods, which include ‘contingent valuation’, that is the individual’s ‘willingness to pay’ (benefits) or ‘willingness to accept’ (costs) and ‘revealed preference’. After identifying and measuring the relevant costs and benefits an SROI ratio can in theory be calculated as the total present value divided by the value of inputs which gives a ratio of £x value per £1 of investment. Of course the challenge of placing monetary value on public health outcomes is not to be underestimated. It may or may not be more straightforward than in the economic evaluation of a clinical intervention, as wider impacts spilling over across sectors may in fact have market prices.

In addition to the NICE guidance referred to above there is another set of guidance published by NICE which deals with cost impact and projected return on investment of public health interventions [ 26 ]. This approach aims to measure the potential savings that can be achieved by implementing one intervention over another. The team provide tools for financial planning, over a 3–5 year time horizon to support local implementation of NICE’s recommendations. One of the main steps in the cost impact assessment is assessing interventions and identifying areas that are most likely to have a resource impact. In addition to costs of delivering services the potential savings arising from recommendations are also estimated.

In terms of the theoretical paradigms underlying UK guidance we found the majority of the guidance was classed as extra-welfarist. This guidance discussed going beyond QALYs to measure wider costs and benefits. The guidance by the Cabinet Office on SROI was guided by a micro-economic theory in that it discussed one-by-one evaluations and no attempt was made to aggregate results into a macro framework.

See Table  1 for a summary of the UK guidance above.

International guidance

We found a range of published guidance on applying economic evaluation methods to public health interventions from international organisations. For example the World Bank guidance [ 27 ] is focused on using techniques of economic evaluation to select a cost-effective package of healthcare interventions in developing countries. Total healthcare costs are expected to rise in the future due to the ageing population needing more hospital and long term care. This has implications for the financing and organisation of health services. The need for economic analysis of public health interventions has therefore never been greater. This manual provides guidelines for health planners on how to collect, analyse and interpret cost and effectiveness data to evaluate a package of health services. It cannot however take into account the inherent differences between countries in terms of different health care systems having different features and so it recommends that the approach be customised in order to accommodate country specific studies.

The World Health Organisation (WHO) [ 28 ] who advocate use of a generalised cost-effectiveness analysis approach, where the comparator is “doing nothing”, as opposed to the “usual care” allowing decision makers to see what could be achieved if all health care resources were re-allocated. So instead of advocating an “incremental” approach to evaluation, the WHO approach encourages analysis of a portfolio of interventions to see whether they provide an efficient use of resources. The WHO guidance focuses on the need to be able to generalise results across different international settings. They argue that generalised cost-effectiveness analysis can be used for a wide range of health technologies, including public health interventions.

Honeycutt et al. [ 29 ] provide guidance from the US which takes the form of a practical ‘step-by-step’ guide to conducting cost-effectiveness analyses of public health interventions. It begins by identifying the study question which is essential in order to know what costs and benefits to include and what kind of methods to employ. The guidance goes on to identify the study perspective, that is, from whose perspective is the study based? It distinguishes between a societal perspective, where all costs and benefits are considered even if no actual payment is made for some of these resources, and a narrower stakeholder perspective. The next step is to consider the time frame for the intervention. They argue that this is straightforward for initiatives that involve one-on-one contact over the course of, say, a year but more difficult if we are considering such interventions as media campaigns for example. The last step in the process is to choose the relevant method of economic evaluation. They list the different types of economic evaluation and suggest the context in which each should be applied. For example, cost analysis can be useful for choosing alternative resource mixtures that limit programme costs. Cost-effectiveness analysis answers questions about whether interventions produce outcomes that are worth the investment and should be used when comparing two or more strategies.

According to the WHO a Disability Adjusted Life Year (DALY) is a measurement of a year of ‘healthy’ life lost due to disease or premature death, or lived at a lower quality due to a disability or disease [ 28 ]. However, DALYs only reflect the presence of a medical condition which relates to certain functional limitations and do not give credit to interventions that improve the ability of individuals to live with a certain disability or disease. The WHO proposes to address this gap in the DALY approach by offering two public health relevant outcome measures 1) the activity limitation score and 2) participation limitation score, based on data from Zambia (Mont and Loeb [ 30 ]).

The Organisation for Economic Co-operation and Development (OECD) [ 31 ] has produced guidance which aims to give an economic framework to the prevention of chronic diseases. Their premise is that the prevention of chronic diseases may increase social welfare and enhance health equity. Their approach links chronic disease, caused mainly by lifestyle choices, with the performance of markets and rationality failures. Priority is given to areas of potential failure such as externalities, with the effects of individual consumption of tobacco, for example, having an effect on others (negative externality) thus justifying interventions. Information failure is another type of market failure whereby individuals lack sufficient knowledge on what constitutes an overall healthy diet for example. Rationality itself, the underpinning of well-functioning markets, can be said to have failed. They argue that this is the case when inconsistent time-preferences and lack of self-control lead to addictive behaviour thus making lifestyle changes difficult. The OECD suggests influencing choices available to individuals and seeks preventative measures where these market failures exist.

In terms of a theoretical paradigm all international guidance was classed as extra-welfarist, apart from the OECD (2008) which looked at the problem from a vantage of behavioural economics. The majority of guidance advocated cost-effectiveness analysis, whilst Mont and Loeb [ 30 ] proposed the use of DALYs rather than mortality on its own.

See Table  2 for a summary of the international guidance above.

Observations of key commentators relevant to the economic evaluation of public health interventions

There have been a few publications that discuss the methodology of economic evaluations of public health interventions, most notably Kelly et al. [ 3 ], Weatherly et al. [ 4 ] and Payne et al. [ 5 ]. Kelly et al. [ 3 ] set out seven ways in which the economic evaluation of public health interventions is potentially different from the evaluation of clinical interventions. Among these is the fact that, firstly, unlike clinical interventions most public health interventions require a change in individual or population behaviour in order to ensure uptake of the intervention in the first place. This behavioural change must be modelled into the economic analysis. Another problem concerns biological and social variation, in that clinical trials are conducted in a narrowly defined area and within a known span of biological variation. Public health interventions, on the other hand, occur in a much wider socio-economic context which is not as easily defined or as generalisable. Finally, public health interventions can change during their implementation which can complicate the interpretation of results. This is because public health interventions are usually implemented without any pre-trial development and are thus susceptible to change during their implementation.

Weatherly et al. [ 4 ] suggest four key methodological challenges that face the economic analyst when conducting evaluations of public health interventions. These comprise firstly, the problem of attributing effects to a specific public health intervention and the fact that there will inevitably be less use of randomised controlled trials in the analysis of public health programmes due to power and sample size requirements. Secondly, measuring and valuing outcomes; QALYs may not be the best way to measure effects of individuals not targeted by the intervention and other non-health effects. Thirdly, equity considerations; i.e. it is possible to look at the distribution of QALY gains between population sub-groups, but that there will be a trade off or sacrifice of moving from an efficiency goal of overall QALY maximisation. Fourth, inter-sectoral costs and consequences; costs and benefits may fall on parts of the public sector not confined to health alone.

Payne et al. [ 5 ] distinguish between ‘simple’ and ‘complex’ interventions by recourse to the 2008 MRC guidance [ 32 ]. Accordingly complex interventions have the properties of involving; more than one group or organisational level that is targeted by the intervention, numerous and variable outcomes and a degree of flexibility or tailoring of the intervention being permitted. They do not talk specifically about public health interventions although these can also be deemed as complex. Rather the authors discuss complex interventions in the context of genetic services. These are regarded as similar to other types of complex interventions in that they take into account non-health outcomes as well as health related outcomes. The aim of this paper is two-fold. Payne et al. [ 5 ] seek to understand if it is reasonable to measure health status alone in the context of complex interventions. They come to the conclusion that public health guidance is more pragmatic than for clinical interventions and should include alternative outcome measures, such as life-years gained and cases averted, as well as QALYs. Therefore, maximising QALY gains should not be the only aim of complex/public interventions. It should also be looking at Sen’s capability theory [ 15 ] as a possible component of outcome measures. In so far as the capability approach (defined as the ability to function given the choice) looks to distribute capability equally across society this is fundamentally different to the idea of maximising QALYs with no regard to equity issues. Secondly, the authors aim to find out if it is possible to evaluate complex interventions using the current NICE guidelines. They conclude that outcome measures associated with a complex intervention need to be extended to include; the values of the process of delivering healthcare, non-health outcomes and capability (first proposed by Sen, [ 15 ]) to participate equally in life [ 33 ]. All these are relevant arguments to how best we approach the evaluation of public health interventions.

Marsh et al. [ 34 ] concede that economic evaluation methods developed for Health Technology Assessments (technical interventions), do not capture all the costs and benefits associated with public health interventions. They argue that the trend of using modelling techniques, and specifically a broad range of outcomes, to assess public health interventions should continue. They argue for the development of a number of ‘valuation paradigms’ such as Sen’s capability approach [ 15 ] and ‘subjective well-being’ approach [ 35 ] both of which can provide the decision maker with broader measures of value than the approaches currently on offer.

Apart from the capability approach employed in Payne et al. [ 5 ] most of the documents from key commentators are underpinned by extra-welfarism. The majority of key commentators in this field see the need and importance to go beyond a utility measure of health, and this theme is recurrent in the literature.

See Table  3 for a summary of observation by the key commentators.

This systematic review yielded a total of 16 relevant papers. A paucity of material is therefore apparent in this field, especially if we compare with the economic evaluation of clinical interventions. Economic evaluations of public health interventions can be seen as a nascent academic endeavour, but the literature does identify the major methodological challenges that analysts will face when conducting such evaluations.

The main theme, reflected throughout, concerns the holistic nature of public health interventions. That is, complex public health interventions, by their very nature, deal with wider social and environmental costs and benefits than do clinical interventions and therefore there is a need to consider a much broader range of outcomes than a focus on QALYs alone [ 3 – 5 , 11 , 22 , 23 ]. This is reflected in the common theme of extra-welfarism that underpins the majority of guidance found in the review. Another salient issue identified remains on the theme of QALYs. It is apparent, from the literature [ 4 , 5 ], that the efficiency goal of overall QALY maximisation is not sufficient in the realms of public health interventions. The reasons for this are two-fold. First public health interventions must deal with equity considerations–a theme which is recurrent in the literature. Insofar as this is true there is a trade-off between the traditional Western notion of utility maximisation on one hand and equity issues that come to the fore on the other. Indeed Payne et al. [ 5 ] suggest Sen’s capability theory [ 15 ] as a means of including equity considerations in economic analysis of public health interventions since capability theory considers the distribution of capability across society. Secondly as the public health guidance is more pragmatic than is the case for clinical interventions then alternative outcome measures such as life-years gained and cases averted should be employed in addition to QALYs. However, although the argument that we should be looking beyond QALYs is indeed compelling it should also be made clear that QALYs, and their common currency, do provide an useful way of deeming if public health interventions are cost-effective in relation to NICE thresholds [ 9 ]–an argument that could be difficult to make without them.

Health economics coming full circle

In the UK, health economics in the 1960s grew out of more general public policy economics, responding to a demand from the NHS to address resource allocation issues within a culture of evidence based medicine [ 36 , 37 ]. In the 1990s and 2000s, health economics remained within a “medical model” of healthcare [ 38 – 40 ], with a focus on the standardisation of methods of evaluation [ 7 ]. However, paradigms were beginning to change with a focus on “health economics” rather than health services economics [ 41 ], which has continued through the decade, with growing interest in how to apply techniques of economic evaluation to the evaluation of public health interventions. Arguably this developed from the Wanless 2004 [ 8 ] report and a growing awareness of pressures on the NHS attributable to lifestyle choices. Recent commentators (Kelly et al. [ 3 ], Weatherly et al., [ 4 ] and Payne et al., [ 5 ]) have argued for health economists to measure a full range of outcome measures, going beyond QALYs; take account of stakeholders, and to acknowledge potential impacts on the need to address inequalities in health. We argue here that health economics may have come full circle from its roots in broad public policy economics and that we may find it useful to think in this broader paradigm with respect to public health economics. Thinking within a broader paradigm also applies to the theoretical underpinnings of guidance within this field. The majority of guidance found was categorised as extra-welfarist; however, we argue that perhaps this should be widened to include more than utility such as QALYs. It should include capability theory, behavioural economics theory and in some cases return full circle to more welfarist principles.

The QALY/SROI dilemma

There seems to be two schools of thought emerging in the literature relating to the economic evaluation of public health interventions. At this point it is worth restating that public health spans the environment, public policy, control of infectious diseases, screening, supporting behaviour change relating to consumption and lifestyle choices, and supporting government regulation in order to foster a society that promotes better health. There has been much more literature on economic evaluations of public health interventions aimed at the individual level than the potential much larger gains from environmental or legislative adaptation [ 42 ]. The extra-welfarist theoretical consensus of the guidance seems to be that QALYs are inadequate as a lone measure of outcome of public health interventions, as they fail in any way to capture the potential broad range of benefits to an individual, their family, community and society as a whole. However, the benefits of using a common accepted currency such as the QALY are illustrated by Owen et al. [ 9 ] who provide a powerful message that many public health interventions are indeed cost effective, well below the NICE threshold of £20,000-30,000 per QALY. Without the use of QALYs this argument could not be effectively put forward. Proponents of a SROI approach, which has its roots in welfarism, seem to have brushed away the 40 years that health economists have wrestled with assigning a monetary value to health outcomes. Why should this be any easier in a public health setting than in a clinical setting?

Should we expect public health interventions to be cost saving?

The SROI approach perpetuates a growing expectation in our experience that public health interventions “need” to demonstrate that they will save money for society in the long run. This is interesting as decisions to fund surgical interventions or a new drug are not routinely subjected to this expectation of “invest to save”. Woolf et al. [ 10 ] in a US context, argue that there is a need to distinguish between measuring the outcome of a public health intervention e.g., the number of people who stop smoking in response to a particular intervention, as opposed to the wider long term social benefits of reduced prevalence of smoking in society and associated economic benefits. Woolf et al. [ 10 ] argue that there is a need to look at the cost-effectiveness of a public health prevention intervention in the same way as a clinical intervention–mindful that it may well have much wider benefits, not just expect it to be cost saving, hence levelling the playing field for resources between public health and clinical interventions.

Design conduct and reporting issues of economic evaluations of public health interventions

The checklist below was inspired by the Drummond et al. checklist for a sound economic evaluation [ 43 ] and our own experiences of designing, undertaking and reporting economic evaluations alongside trials of public health interventions [ 44 – 47 ]. None of the guidance above, in our view, answers these questions systematically, to this extent our checklist is novel in distilling out key questions for consideration. The sets of guidance reviewed in this paper do provide a wide range of frameworks within which to approach the whole task of conducting economic evaluations of public health interventions. These questions, together with the sets of guidance reviewed in this paper highlight issues that need addressing in the design conduct and reporting of economic evaluations of public health interventions see Table  4 . We have not tried to be prescriptive, rather to raise pertinent questions.

Strengths and limitations

This paper is the first to summarise the range of public health guidance available for the economic evaluation of public health interventions. We offer a checklist that highlights issues that need addressing in order to conduct economic evaluations of public health interventions and provide the most up to date summary of guidance in this field, which has the potential to be of wide benefit to health economists and other public health services researchers.

The narrative review methodology was chosen due to a high level of heterogeneity between each of the guidance documents. A potential limitation of this method is that it may have resulted in a limited review. However, we tried to minimise this by expanding our methodology and exploring the theoretical frameworks underpinning the guidance and discussing their implications.

Another potential limitation of this review is the language restriction placed in the inclusion criteria. The review identified guidance published only in English; this decision was based upon financial constraints of translation as the review was unfunded. Additionally, we did not find any guidance in languages other than English when conducting an initial scoping search of the literature.

This paper is in no way prescriptive–rather it offers the first comprehensive source of published guidelines for those embarking on an economic evaluation of a public health intervention. The message from our public health colleagues is very clear–think multidisciplinary. We argue here that health economics has come, with respect to public health, full circle from its origins in wider public policy. If we approach the evaluation of public health interventions as “public policy economists”, measuring health outcomes as part of the full range of outcomes, rather than trying to stretch the “medical model” with the limitations of the health economics toolbox, we may make greater strides to inform public policy to improve health.

Abbreviations

Disability adjusted life year

Medical research council

National institute of health and care excellence

Organisation for economic co-operation and development

Population health sciences research network

Quality adjusted life year

Social return on investment’

United Kingdom

United States

World Health Organisation.

National Institute for Health and Care Excellence (NICE): Guide to the methods of technology appraisal. http://www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf ,

National Institute for Health and Care Excellence (NICE): Guide to the methods of technology appraisal. http://www.nice.org.uk/media/D45/1E/GuideToMethodsTechnologyAppraisal2013.pdf ,

Kelly MP, McDaid D, Ludbrook A, Powell J: Economic appraisal of public health interventions. http://www.cawt.com/Site/11/Documents/Publications/Population%20Health/Economics%20of%20Health%20Improvement/Economic_appraisal_of_public_health_interventions.pdf ,

Weatherly H, Drummond M, Claxton K, Cookson R, Ferguson B, Godfrey C, Rice N, Sculpher M, Sowden A: Methods for assessing the cost-effectiveness of public health interventions: key challenges and recommendations. Health Policy. 2009, 93: 85-92. 10.1016/j.healthpol.2009.07.012.

Article   PubMed   Google Scholar  

Payne K, McAllister M, Davies LM: Valuing the economic benefits of complex interventions: when maximising health is not sufficient. Health Econ. 2012, doi:10.1002/hec.2795 [Published Online First]

Google Scholar  

Coast J, Smith R, Lorgelly P: Should the capability approach be applied in health economics?. Health Econ. 2008, 17: 667-670. 10.1002/hec.1359.

Drummond M, Weatherly H, Claxton K, Cookson R, Ferguson B, Godfrey C, Rice N, Sculpher M, Sowden A: Assessing the Challenge of Applying Standard Methods of Economic Evaluation to Public Health Interventions. http://phrc.lshtm.ac.uk/papers/PHRC_D1-05_Final_Report.pdf ,

Wanless D: Securing good health for the whole population: Final report. http://webarchive.nationalarchives.gov.uk/+/http:/www.hm-treasury.gov.uk/media/D/3/Wanless04_summary.pdf ,

Owen L, Morgan A, Fischer A, Ellis S, Hoy A, Kelly M: The cost-effectiveness of public health interventions. J Public Health. 2011, doi: 10.1093/pubmed/fdr075 [Published Online First]

Woolf SH, Husten CG, Lewin LS, Marks JS, Fielding JE, Sanchez EJ: The economic argument for disease prevention: distinguishing between value and savings. http://www.prevent.org/data/files/initiatives/economicargumentfordiseaseprevention.pdf ,

The Cabinet Office: A Guide to Social Return on Investment. http://www.thesroinetwork.org/publications/cat_view/29-the-sroi-guide-2009?orderby=dmdate_published&ascdesc=DESC ,

Liberati A, Altman DG, Tetzlaff J, Mulrows C, Gøtzsche PC, Ioannidis JPA, Clarke M, Deveraux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009, 6: e1000100-10.1371/journal.pmed.1000100.

Article   PubMed   PubMed Central   Google Scholar  

MacDonald G, O’Hara K: Ten elements of mental health, its promotion and demotion: implications for practice. 1998, London: Society of Health Education and Promotion Specialists

Culyer AJ: The normative economics of health care finance and provision. Oxf Rev Econ Policy. 1989, 5: 34-58. 10.1093/oxrep/5.1.34.

Article   Google Scholar  

Sen AK: Commodities and Capabilities. 1985, Oxford: Oxford University Press

Camerer CF, Lowenstein G: Behavioural economics: past present and future. http://www.hss.caltech.edu/~camerer/ribe239.pdf ,

Astolfi R, Lorenzoni L, Oderkirk J: A Comparative Analysis of Health Forecasting Methods. 2012, : OECD Health Working Papers, No. 59, OECD Publishing, http://dx.doi.org/10.1787/5k912j389bf0-en ,

Book   Google Scholar  

Morris S, Devlin N, Parkin D: Economic Analysis in Healthcare. 2007, Chichester, UK: Wiley and Sons

Gyrd-Hansen D: Willingness to pay for a QALY: theoretical and methodological issues. Pharmacoeconomics. 2005, 23: 423-432. 10.2165/00019053-200523050-00002.

Group EQL: EuroQoL—a new facility for the measurement of health related quality of life. Health Policy. 1990, 16: 199-208.

Grewal I, Lewis J, Flynn T, Brown J, Bond J, Coast J: Developing attributes for a generic Quality of Life Measure for Older People: preferences or capabilities?. Soc Sci Med. 2006, 62: 1891-1901. 10.1016/j.socscimed.2005.08.023.

HM Treasury: The Green Book: Appraisal and Evaluation in Central Government. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220541/green_book_complete.pdf ,

Byford S, McDaid D, Sefton T: Because it’s worth it: a practical guide to conducting economic evaluations in the social welfare field. http://www.jrf.org.uk/sites/files/jrf/1859351123.pdf ,

Tarn TY, Smith MD: Pharmacoeconomic guidelines around the world. ISPOR Connections. 2004, 10: 4-5.

National Institute for Health and Clinical Excellence (NICE): Methods for the development of NICE public health guidance. http://www.nice.org.uk/media/CE1/F7/CPHE_Methods_manual_LR.pdf ,

National Institute for Health and Clinical Excellence (NICE): Supporting Investment in Public Health: Review of Methods for Assessing Cost Effectiveness, Cost Impact and Return on Investment. http://www.nice.org.uk/media/664/AC/cost_impact_proof_of_concept.pdf ,

Brenzel L: Selecting an Essential Package of Health Services using Cost-Effectiveness Analysis: A Manual for Professionals in Developing Countries. http://pdf.usaid.gov/pdf_docs/PNACJ346.pdf ,

Tan-Torres Edejer T, Baltussen R, Adam T, Hutubessy R, Acharya A, Evans DB, Murray CJL: World Health Organisation guidance to cost-effectiveness analysis. http://www.who.int/choice/publications/p_2003_generalised_cea.pdf ,

Honeycutt AA, Clayton L, Khavjou O, Finkelstein EA, Prabhu A, Blitstein JL, Evans WD, Renaud JM: Guide to Analysing the Cost-Effectiveness of Community Public Health Prevention Approaches. http://aspe.hhs.gov/health/reports/06/cphpa/report.pdf ,

Mont D, Loeb M: Beyond DALYs: Developing Indicators to Assess the Impact of Public Health Interventions on the Lives of People with Disabilities. http://siteresources.worldbank.org/SOCIALPROTECTION/Resources/SP-Discussion-papers/Disability-DP/0815.pdf ,

Sassi F, Hurst J: The Prevention of Lifestyle-Related Chronic Diseases: An Economic Framework. http://www.oecd.org/health/healthpoliciesanddata/40324263.pdf ,

Medical Research Council: Guidance on the development, evaluation and implementation of complex interventions to improve health. http://www.mrc.ac.uk/Utilities/Documentrecord/index.htm?d=MRC004871 ,

Sen AK: Capability and Well-being. The Quality of Life. Edited by: Nussbaum MC, Sen AK. 1993, Oxford: Clarendon Press

Marsh K, Phillips CJ, Fordham R, Bertranou E, Hale J: Estimating cost-effectiveness in public health: a summary of modelling and valuation methods. Health Econ Rev. 2012, 2: doi:10.1186/2191-1991-2-17 [Published Online First]

Diener E: Subjective well-being. Psychol Bull. 1984, 95: 542-575.

Article   CAS   PubMed   Google Scholar  

Williams A: Cost-benefit analysis: bastard science? And/or insidious poison in the body politick?. J Public Econ. 1972, 1: 199-225. 10.1016/0047-2727(72)90002-3.

Sackett DL, Rosenberg WMC, Muir Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312: 71-72. 10.1136/bmj.312.7023.71.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Williams A: Priorities and research strategy in health economics for the 1990. Health Econ. 1993, 2: 295-302. 10.1002/hec.4730020402.

Phelps CE: Perspectives in health economics. Health Econ. 1995, 4: 335-353. 10.1002/hec.4730040501.

Fuchs VR: The future of health economics. J Health Econ. 2000, 19: 141-157. 10.1016/S0167-6296(99)00033-8.

Edwards RT: Paradigms and research programmes: is it time to move from health care economics to health economics?. Health Econ. 2001, 10: 635-649. 10.1002/hec.610.

Chokshi DA, Farley TA: The cost-effectiveness of environmental approaches to disease prevention. N Engl J Med. 2012, 367: 295-297. 10.1056/NEJMp1206268.

Drummond MF, Stoddart GL, Torrance GW: Methods for the Economic Evaluation of Health Care Programmes. 2005, Oxford: Oxford University Press, 3

Edwards RT, Céilleachair A, Bywater T, Hughes DA, Hutchings J: A parenting programme for children at risk of developing conduct disorder: a cost-effectiveness analysis. BMJ. 2007, 334: 682-10.1136/bmj.39126.699421.55.

Edwards RT, Hounsome B, Linck P, Russell IT: Economic evaluation alongside pragmatic randomised trials: developing a Standard Operating Procedure for clinical trials units. Trials. 2008, 14: 64-65.

Edwards RT, Neal RD, Linck P, Bruce N, Mullock L, Nelhans N, Pasterfield D, Russell D, Russell I, Woodfine L, on behalf of the CHARISMA study group: Enhancing ventilation in homes of children with asthma: cost-effectiveness study alongside randomised trial. Br J Gen Pract. 2011, 61: 733-741. 10.3399/bjgp11X606645.

Murphy S, Edwards RT, Williams N, Raisenen L, Moore G, Linck P, Hounsome N, Ud Din N, Moore L: An evaluation of the effectiveness and cost effectiveness of the National Exercise Referral Scheme in Wales, UK: a randomised controlled trial of a public health policy initiative. J Epidemiol Community Health. 2012, 66: 745-753. 10.1136/jech-2011-200689.

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2458/13/1001/prepub

Download references

Acknowledgements

We would like to acknowledge Alison Shaw for providing reader support to the first author. RTE post is funded by Public Health Wales, JMC post is partly-funded by Public Health Wales and HLW post is funded by NISCHR.

Author information

Authors and affiliations.

Centre for Health Economics and Medicines Evaluation, Institute of Medical and Social Care Research, Bangor University, Dean Street Building, Bangor, Gwynedd, LL57 1UT, UK

Rhiannon Tudor Edwards, Joanna Mary Charles & Huw Lloyd-Williams

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rhiannon Tudor Edwards .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

HLW conducted the initial electronic searches, articles matching the inclusion criteria were obtained and read by RTE, JMC and HLW. All authors contributed to the drafting of the manuscript and approved the final manuscript.

Electronic supplementary material

12889_2013_5962_moesm1_esm.doc.

Additional file 1: Search terms used for the systematic review presented in hierarchical and combination order.(DOC 22 KB)

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, authors’ original file for figure 4, authors’ original file for figure 5, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Edwards, R.T., Charles, J.M. & Lloyd-Williams, H. Public health economics: a systematic review of guidance for the economic evaluation of public health interventions and discussion of key methodological issues. BMC Public Health 13 , 1001 (2013). https://doi.org/10.1186/1471-2458-13-1001

Download citation

Received : 01 February 2013

Accepted : 16 October 2013

Published : 24 October 2013

DOI : https://doi.org/10.1186/1471-2458-13-1001

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health economics
• Public health
• Published guidelines
• Public sector policy
• Inequalities in health
• Outcome measurement

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Systematic reviews of health economic evaluations: a protocol for a systematic review of characteristics and methods applied

Miriam luhnen.

1 Department Health Care and Health Economics, Institute for Quality and Efficiency in Health Care (IQWiG), Im Mediapark 8, 50670 Cologne, Germany

2 Faculty of Health, Department of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Haus 38, 51109 Cologne, Germany

Barbara Prediger

3 Institute for Research in Operative Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Haus 38, 51109 Cologne, Germany

Edmund A. M. Neugebauer

4 Faculty of Health, Brandenburg Medical School – Theodor Fontane, Campus Neuruppin, Fehrbelliner Str. 38, 16816 Neuruppin, Germany

5 Interdisciplinary Centre for Health Services Research, Witten/Herdecke University, Alfred-Herrhausen-Straße 50, 58448 Witten, Germany

Associated Data

The datasets generated and analyzed during the current study will be available from the corresponding author on reasonable request.

The number of systematic reviews of economic evaluations is steadily increasing. This is probably related to the continuing pressure on health budgets worldwide which makes an efficient resource allocation increasingly crucial. In particular in recent years, the introduction of several high-cost interventions presents enormous challenges regarding universal accessibility and sustainability of health care systems. An increasing number of health authorities, inter alia, feel the need for analyzing economic evidence.

Economic evidence might effectively be generated by means of systematic reviews. Nevertheless, no standard methods seem to exist for their preparation so far.

The objective of this study was to analyze the methods applied for systematic reviews of health economic evaluations (SR-HE) with a focus on the identification of common challenges.

Methods/design

The planned study is a systematic review of the characteristics and methods actually applied in SR-HE. We will combine validated search filters developed for the retrieval of economic evaluations and systematic reviews to identify relevant studies in MEDLINE (via Ovid, 2015-present). To be eligible for inclusion, studies have to conduct a systematic review of full economic evaluations. Articles focusing exclusively on methodological aspects and secondary publications of health technology assessment (HTA) reports will be excluded. Two reviewers will independently assess titles and abstracts and then full-texts of studies for eligibility. Methodological features will be extracted in a standardized, beforehand piloted data extraction form. Data will be summarized with descriptive statistical measures and systematically analyzed focusing on differences/similarities and methodological weaknesses.

The systematic review will provide a detailed overview of characteristics of SR-HE and the applied methods. Differences and methodological shortcomings will be detected and their implications will be discussed. The findings of our study can improve the recommendations on the preparation of SR-HE. This can increase the acceptance and usefulness of systematic reviews in health economics for researchers and medical decision makers.

Systematic review registration

The review will not be registered with PROSPERO as it does not meet the eligibility criterion of dealing with clinical outcomes.

Electronic supplementary material

The online version of this article (10.1186/s13643-017-0639-8) contains supplementary material, which is available to authorized users.

Continuing pressure on health budgets worldwide makes an efficient resource allocation increasingly crucial. In recent years, the introduction of several high-cost interventions presents enormous challenges regarding accessibility and sustainability of health care systems [ 1 , 2 ]. This makes economic considerations more important for health authorities and their decision-making process regarding pricing and reimbursement especially of new interventions.

Systematic reviews of health economic evaluations (SR-HE) can provide evidence about the cost-effectiveness of an intervention within a limited time frame. They are valuable (1) to inform the development of an own economic model, (2) to identify the most relevant studies for a particular decision, and (3) to identify the implicated economic trade-offs [ 3 ]. Moreover, provided that high-quality economic evaluations that exist are sufficiently transferable and demonstrate similar results regarding cost-effectiveness, SR-HE might indicate the most cost-effective intervention.

Jefferson et al. [ 4 ] found that SR-HE show fundamental methodological flaws, especially regarding their search strategy and the application of an appropriate quality assessment tool. Nevertheless, little research has been performed to further develop the methods for SR-HE in the meantime. Standards for the preparation of SR-HE do not seem to exist so far: More recent studies focusing on the available methodological guidelines found that the recommendations still vary widely and are partly imprecise [ 5 – 8 ]. It is therefore to be expected that the conduct of SR-HE still varies widely and still shows methodological shortcomings. The aim of this paper is

• To provide a detailed overview of the characteristics and applied methods in recently published SR-HE
• To identify similarities and differences between the characteristics and methods of SR-HE
• To identify common challenges

Methods/Design

We used the PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) 2015 checklist to develop the methods for this systematic review protocol [ 9 ] (please see Additional file 1 ).

Should protocol amendments be necessary, these will be documented including details of the date, changes made, and the rationale for changes.

Literature search

A systematic search in Ovid MEDLINE(R) Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily, and Ovid MEDLINE(R) 1946 to Present will be performed. We will limit the publication date of our search to the period 2015/01/01 to present. A validated search filter for economic evaluations (Emory University [Grady] [ 10 ]) will be combined with a validated filter for the retrieval of systematic reviews (Lee [ 11 ]), as presented in Table  1 . This strategy was chosen as it provides an optimal balance between sensitivity and precision. Search results will be downloaded to EndNote version X7 where duplicates will be identified and removed.

Details of the bibliographic database search strategy

Limit: publication year 2015–present

Inclusion criteria

We will include articles available as full-text and written in English, German, French, or Spanish if they fulfill all of the following criteria:

• Systematic literature search in at least one electronic database and transparent description of study selection. We will exclude articles applying abbreviated review methods (e.g., scoping reviews and short reviews) as judged by the authors of the SR-HE.
• Inclusion of full economic evaluations (i.e., cost-effectiveness/cost-utility/cost-benefit-analyses [ 12 ]) and/or the cost-effectiveness of an intervention was reviewed. Articles reviewing solely partial economic evaluations (like cost-of-illness studies or budget impact analyses) will be excluded.
• Objective to answer a cost-effectiveness research question, i.e., we will exclude articles focusing exclusively on methodological aspects (e.g., analysis of methods applied in health economic modeling studies).
• Full-text journal article. Protocols, commentaries, editorials, and conference proceedings will be excluded. Likewise, secondary publications of HTA reports will be excluded as the focus of our study will be on the scientific literature instead of documents stemming from regulatory processes within a certain jurisdiction in a health care system.

Study selection

Two reviewers will independently assess the titles and abstracts retrieved in the electronic literature search against the inclusion criteria. Possible eligible full-text articles will be retrieved and screened by two reviewers to reach a final decision about inclusion. Any disagreements will be resolved through discussion or involvement of a third reviewer.

We will prepare a PRISMA flowchart to illustrate the selection process.

Data abstraction

Methodological features will be extracted in a standardized, beforehand piloted data extraction form (Table  2 ). We developed an electronical extraction form in Microsoft Excel 2010 for a previous study (not published yet) in which we analyzed HTA reports of international HTA organizations for the methods applied for SR-HE and adapted it for the purpose of the present study. This approach for data abstraction and data presentation was inspired by the publication of Page et al. [ 13 ] which provides an overview of epidemiology and reporting characteristics of systematic reviews of biomedical research. Data items presented in the included articles will be classified according to the categories depicted in Table  3 . Data will be extracted each by a single reviewer. After extraction of the first articles, a 10% random sample will be verified for accuracy and correctness of data entries by a second reviewer. Discrepancies will be resolved through discussion or third party, if necessary. In case of frequent and/or substantial disagreements, a verification of 100% is intended.

Data extraction form

CEA cost-effectiveness analysis; PICOS patient, intervention, comparison, outcome, setting; PRISMA Preferred Reporting Items for Systematic review and Meta-Analysis; SR-HE systematic review of health economic evaluations

Categories for the classification of data items extracted in the included reviews

Data analysis and presentation

We will analyze all data using Microsoft Excel 2010. Results for each data item extracted will be presented in tables. For nominal data, we will provide numbers and percentages. We will provide median and ranges for ordinal data.

In order to allow an estimation of the number of SR-HE published per year and to analyze possible changes over time, we will present the number of hits resulting from our search strategy for the years 2015 to 2017.

Since no tool for the critical appraisal of SR-HE exists (comparable e.g., to AMSTAR [A Measurement Tool to Assess Systematic Reviews] [ 14 ]), we will not critically appraise included articles by means of a certain tool but focus on similarities, differences, and methodological shortcomings.

As far as possible, the results of our study will be reported in accordance with the PRISMA guidelines [ 15 ].

Acknowledgements

Not applicable

No funding will be received for the proposed study.

Availability of data and materials

Abbreviations, additional file.

Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) 2015 checklist: recommended items to address in a systematic review protocol. (DOCX 36 kb)

Authors’ contributions

ML and TM developed and piloted the data extraction form. ML developed the search strategy for the proposed systematic review and drafted the manuscript. TM and BP commented on the manuscript. EAMN supported the conceptualization of the systematic review. All authors reviewed the manuscript and approved the final manuscript.

Ethics approval and consent to participate

Consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Miriam Luhnen, Email: [email protected] .

Barbara Prediger, Email: [email protected] .

Edmund A. M. Neugebauer, Email: ed.enatnoF-bhm@tanaked .

Tim Mathes, Email: [email protected] .

Health Economics: A Review

Cite this chapter.

• Keya Sengupta 2  

521 Accesses

A good number of studies on health economics has been conducted, mostly during the last 50 years. In the Indian context however, a lot more studies in this sector needs to be undertaken, in view of the dearth of adequate research work in this area. Most of the literature surveyed for the purpose of the present work relates to the workdone by health economists of the west. All the work reviewed has been divided broadly into four categories as represented in the diagram below (Diagram 3.1):

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Arrow KJ (1963) Uncertainty and the welfare economics of medical care. Am Econ Rev 53:941–973

Google Scholar  

Bloom DE, Canning D (2000) The health and wealth of nations. Science 287(5456):1207–1209

Article   CAS   PubMed   Google Scholar  

Bloom DE, Canning D, Sevilla J (2001) The effect of health on economic growth: theory and evidence, Working paper. NBER, Cambridge

Book   Google Scholar  

Brunello G, Fort M, Schneeweis N, Ebmer R (2011) The casual effect of education on health: what is the role of health behaviour, Working paper. Institute for the Study of Labour, Austria

Culyer AJ (1971) Nature of the commodity “health care” and its efficient allocation. Oxf Econ Pap 23:189–211

Friedman M, Kuznets S (1945) Income from independent professional practice. National Bureau of Economic Research, New York

Fuch VR (1966) The output of the health industry. In: Cooper MH, Culyer T (eds) Health economics. Penguin, London

Gellego R (2000) Introducing purchaser/provider separation in the Catalon health administration: a budget analysis. Public Adm 78:423–442

Article   Google Scholar  

Grossman M (1972) On the concept of health capital and the demand for health. J Polit Econ 80(2):223–255

Higgins C, Lavin T, Metcalfe O (2008) Health impacts of education. Institute of Public Health, Ireland

Jack W (1999) Principles of health economics for developing countries. World Bank Institute of Development Studies, Washington, DC

Lindsay CM (1969) Medical care and the economics of sharing. Economica 36(144):351–362

Luft H (1978) Poverty and health: economic causes and consequences of health problems. Ballinger Publishing Company, Cambridge, MA

Muskin SJ (1962) Health as an investment. J Polit Econ 70, 129 pp

Smith J (1999) Healthy bodies and Thich wallets: the dual relation between health and economic status. J Econ Perspect 13(2):145–166

Sorkin AL (1997) Health economics in developing countries. Lexington Books, Lexington

Strauss J, Duncan J (1998) Health, nutrition and economic development. J Econ Lit 36(2):766–817

Download references

Author information

Authors and affiliations.

Indian Institute of Management, Nongthymmai, Shillong, India

Keya Sengupta

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer India

About this chapter

Sengupta, K. (2016). Health Economics: A Review. In: Determinants of Health Status in India. Springer, New Delhi. https://doi.org/10.1007/978-81-322-2535-5_3

Download citation

DOI : https://doi.org/10.1007/978-81-322-2535-5_3

Publisher Name : Springer, New Delhi

Print ISBN : 978-81-322-2534-8

Online ISBN : 978-81-322-2535-5

eBook Packages : Economics and Finance Economics and Finance (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Aims and scope

Benefits of publishing with bmc, high visibility.

Health Economics Review 's open access policy allows maximum visibility of articles published in the journal as they are available to a wide, global audience. 

Speed of publication

Health Economics Review offers a fast publication schedule whilst maintaining rigorous peer review; all articles must be submitted online, and peer review is managed fully electronically (articles are distributed in PDF form, which is automatically generated from the submitted files). Articles will be published with their final citation after acceptance, in both fully browsable web form, and as a formatted PDF.

Flexibility

Online publication in Health Economics Review gives you the opportunity to publish large datasets, large numbers of color illustrations and moving pictures, to display data in a form that can be read directly by other software packages so as to allow readers to manipulate the data for themselves, and to create all relevant links (for example, to PubMed , to sequence and other databases, and to other articles).

Promotion and press coverage

Articles published in Health Economics Review are included in article alerts and regular email updates. Some may be highlighted on Health Economics Review ’s pages and on the BMC homepage.

In addition, articles published in Health Economics Review may be promoted by press releases to the general or scientific press. These activities increase the exposure and number of accesses for articles published in Health Economics Review . A list of articles recently press-released by journals published by BMC is available here .

As an author of an article published in Health Economics Review you retain the copyright of your article and you are free to reproduce and disseminate your work (for further details, see the BMC license agreement ).

For further information about the advantages of publishing in a journal from BMC, please click here .

Open access

All articles published by Health Economics Review are made freely and permanently accessible online immediately upon publication, without subscription charges or registration barriers. Further information about open access can be found here .

As authors of articles published in Health Economics Review you are the copyright holders of your article and have granted to any third party, in advance and in perpetuity, the right to use, reproduce or disseminate your article, according to the BMC license agreement .

For those of you who are US government employees or are prevented from being copyright holders for similar reasons, BMC can accommodate non-standard copyright lines. Please contact us if further information is needed.

Article processing charges (APC)

Authors who publish open access in Health Economics Review are required to pay an article processing charge (APC). The APC price will be determined from the date on which the article is accepted for publication.

The current APC, subject to VAT or local taxes where applicable, is: £2090.00/$2790.00/€2390.00

Visit our open access support portal and our Journal Pricing FAQs for further information.

Open access funding

Visit Springer Nature’s open access funding & support services for information about research funders and institutions that provide funding for APCs.

Springer Nature offers agreements that enable institutions to cover open access publishing costs. Learn more about our open access agreements to check your eligibility and discover whether this journal is included.

Springer Nature offers APC waivers and discounts for articles published in our fully open access journals whose corresponding authors are based in the world’s lowest income countries (see our APC waivers and discounts policy for further information). Requests for APC waivers and discounts from other authors will be considered on a case-by-case basis, and may be granted in cases of financial need (see our open access policies for journals for more information). All applications for discretionary APC waivers and discounts should be made at the point of manuscript submission; requests made during the review process or after acceptance are unable to be considered.

Indexing services

All articles published in Health Economics Review are included in:

• Current Contents - Social & Behavioral Sciences
• Emerging Sources Citation Index
• PubMed Central
• Research Papers in Economics
• Social Sciences Citation Index
• Summon by Serial Solutions

The full text of all articles is deposited in digital archives around the world to guarantee long-term digital preservation. You can also access all articles published by BMC on SpringerLink . Health Economics Review has an Impact Factor of 2.306.

Peer-review policy

Peer-review is the system used to assess the quality of a manuscript before it is published. Independent researchers in the relevant research area assess submitted manuscripts for originality, validity and significance to help editors determine whether the manuscript should be published in their journal. You can read more about the peer-review process here .

Health Economics Review operates a double-blind peer-review system, where the reviewers do not know the names or affiliations of the authors and the reviewer reports provided to the authors are anonymous. Submitted manuscripts will generally be reviewed by two to three experts who will be asked to evaluate whether the manuscript is scientifically sound and coherent, whether it duplicates already published work, and whether or not the manuscript is sufficiently clear for publication. Reviewers will also be asked to indicate how interesting and significant the research is. The Editors will reach a decision based on these reports and, where necessary, they will consult with members of the Editorial Board.

Editorial policies

All manuscripts submitted to Health Economics Review should adhere to BMC's editorial policies .

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Citing articles in Health Economics Review

Articles in Health Economics Review  should be cited in the same way as articles in a traditional journal. Because articles are not printed, they do not have page numbers; instead, they are given a unique article number.

Article citations follow this format:

Authors: Title. Health Econ Rev [year], [volume number]:[article number].

e.g. Roberts LD, Hassall DG, Winegar DA, Haselden JN, Nicholls AW, Griffin JL: Increased hepatic oxidative metabolism distinguishes the action of Peroxisome Proliferator-Activated Receptor delta from Peroxisome Proliferator-Activated Receptor gamma in the Ob/Ob mouse. Health Econ Rev 2009, 1 :115.

refers to article 115 from Volume 1 of the journal.

Appeals and complaints

Authors who wish to appeal a rejection or make a complaint should follow the procedure outlined in the BMC Editorial Policies .

• Editorial Board
• Manuscript editing services
• Instructions for Editors
• Sign up for article alerts and news from this journal

Annual Journal Metrics

2022 Citation Impact 2.4 - 2-year Impact Factor 2.8 - 5-year Impact Factor 1.204 - SNIP (Source Normalized Impact per Paper) 0.836 - SJR (SCImago Journal Rank)

2023 Speed 11 days submission to first editorial decision for all manuscripts (Median) 220 days submission to accept (Median)

2023 Usage  566,888 downloads 483 Altmetric mentions

• More about our metrics
• ISSN: 2191-1991 (electronic)

Health Economics Review

ISSN: 2191-1991

• Submission enquiries: [email protected]
• General enquiries: [email protected]