open innovation literature review

Open innovation: Literature review and outlook

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 15 May 2024

Learning together for better health using an evidence-based Learning Health System framework: a case study in stroke

• Helena Teede 1 , 2   na1 ,
• Dominique A. Cadilhac 3 , 4   na1 ,
• Tara Purvis 3 ,
• Monique F. Kilkenny 3 , 4 ,
• Bruce C.V. Campbell 4 , 5 , 6 ,
• Coralie English 7 ,
• Alison Johnson 2 ,
• Emily Callander 1 ,
• Rohan S. Grimley 8 , 9 ,
• Christopher Levi 10 ,
• Sandy Middleton 11 , 12 ,
• Kelvin Hill 13 &
• Joanne Enticott   ORCID: orcid.org/0000-0002-4480-5690 1  

BMC Medicine volume  22 , Article number:  198 ( 2024 ) Cite this article

254 Accesses

2 Altmetric

Metrics details

In the context of expanding digital health tools, the health system is ready for Learning Health System (LHS) models. These models, with proper governance and stakeholder engagement, enable the integration of digital infrastructure to provide feedback to all relevant parties including clinicians and consumers on performance against best practice standards, as well as fostering innovation and aligning healthcare with patient needs. The LHS literature primarily includes opinion or consensus-based frameworks and lacks validation or evidence of benefit. Our aim was to outline a rigorously codesigned, evidence-based LHS framework and present a national case study of an LHS-aligned national stroke program that has delivered clinical benefit.

Current core components of a LHS involve capturing evidence from communities and stakeholders (quadrant 1), integrating evidence from research findings (quadrant 2), leveraging evidence from data and practice (quadrant 3), and generating evidence from implementation (quadrant 4) for iterative system-level improvement. The Australian Stroke program was selected as the case study as it provides an exemplar of how an iterative LHS works in practice at a national level encompassing and integrating evidence from all four LHS quadrants. Using this case study, we demonstrate how to apply evidence-based processes to healthcare improvement and embed real-world research for optimising healthcare improvement. We emphasize the transition from research as an endpoint, to research as an enabler and a solution for impact in healthcare improvement.

Conclusions

The Australian Stroke program has nationally improved stroke care since 2007, showcasing the value of integrated LHS-aligned approaches for tangible impact on outcomes. This LHS case study is a practical example for other health conditions and settings to follow suit.

Peer Review reports

Internationally, health systems are facing a crisis, driven by an ageing population, increasing complexity, multi-morbidity, rapidly advancing health technology and rising costs that threaten sustainability and mandate transformation and improvement [ 1 , 2 ]. Although research has generated solutions to healthcare challenges, and the advent of big data and digital health holds great promise, entrenched siloes and poor integration of knowledge generation, knowledge implementation and healthcare delivery between stakeholders, curtails momentum towards, and consistent attainment of, evidence-and value-based care [ 3 ]. This is compounded by the short supply of research and innovation leadership within the healthcare sector, and poorly integrated and often inaccessible health data systems, which have crippled the potential to deliver on digital-driven innovation [ 4 ]. Current approaches to healthcare improvement are also often isolated with limited sustainability, scale-up and impact [ 5 ].

Evidence suggests that integration and partnership across academic and healthcare delivery stakeholders are key to progress, including those with lived experience and their families (referred to here as consumers and community), diverse disciplines (both research and clinical), policy makers and funders. Utilization of evidence from research and evidence from practice including data from routine care, supported by implementation research, are key to sustainably embedding improvement and optimising health care and outcomes. A strategy to achieve this integration is through the Learning Health System (LHS) (Fig.  1 ) [ 2 , 6 , 7 , 8 ]. Although there are numerous publications on LHS approaches [ 9 , 10 , 11 , 12 ], many focus on research perspectives and data, most do not demonstrate tangible healthcare improvement or better health outcomes. [ 6 ]

Monash Learning Health System: The Learn Together for Better Health Framework developed by Monash Partners and Monash University (from Enticott et al. 2021 [ 7 ]). Four evidence quadrants: Q1 (orange) is evidence from stakeholders; Q2 (green) is evidence from research; Q3 (light blue) is evidence from data; and, Q4 (dark blue) is evidence from implementation and healthcare improvement

In developed nations, it has been estimated that 60% of care provided aligns with the evidence base, 30% is low value and 10% is potentially harmful [ 13 ]. In some areas, clinical advances have been rapid and research and evidence have paved the way for dramatic improvement in outcomes, mandating rapid implementation of evidence into healthcare (e.g. polio and COVID-19 vaccines). However, healthcare improvement is challenging and slow [ 5 ]. Health systems are highly complex in their design, networks and interacting components, and change is difficult to enact, sustain and scale up. [ 3 ] New effective strategies are needed to meet community needs and deliver evidence-based and value-based care, which reorients care from serving the provider, services and system, towards serving community needs, based on evidence and quality. It goes beyond cost to encompass patient and provider experience, quality care and outcomes, efficiency and sustainability [ 2 , 6 ].

The costs of stroke care are expected to rise rapidly in the next decades, unless improvements in stroke care to reduce the disabling effects of strokes can be successfully developed and implemented [ 14 ]. Here, we briefly describe the Monash LHS framework (Fig.  1 ) [ 2 , 6 , 7 ] and outline an exemplar case in order to demonstrate how to apply evidence-based processes to healthcare improvement and embed real-world research for optimising healthcare. The Australian LHS exemplar in stroke care has driven nationwide improvement in stroke care since 2007.

An evidence-based Learning Health System framework

In Australia, members of this author group (HT, AJ, JE) have rigorously co-developed an evidence-based LHS framework, known simply as the Monash LHS [ 7 ]. The Monash LHS was designed to support sustainable, iterative and continuous robust benefit of improved clinical outcomes. It was created with national engagement in order to be applicable to Australian settings. Through this rigorous approach, core LHS principles and components have been established (Fig.  1 ). Evidence shows that people/workforce, culture, standards, governance and resources were all key to an effective LHS [ 2 , 6 ]. Culture is vital including trust, transparency, partnership and co-design. Key processes include legally compliant data sharing, linkage and governance, resources, and infrastructure [ 4 ]. The Monash LHS integrates disparate and often siloed stakeholders, infrastructure and expertise to ‘Learn Together for Better Health’ [ 7 ] (Fig.  1 ). This integrates (i) evidence from community and stakeholders including priority areas and outcomes; (ii) evidence from research and guidelines; (iii) evidence from practice (from data) with advanced analytics and benchmarking; and (iv) evidence from implementation science and health economics. Importantly, it starts with the problem and priorities of key stakeholders including the community, health professionals and services and creates an iterative learning system to address these. The following case study was chosen as it is an exemplar of how a Monash LHS-aligned national stroke program has delivered clinical benefit.

Australian Stroke Learning Health System

Internationally, the application of LHS approaches in stroke has resulted in improved stroke care and outcomes [ 12 ]. For example, in Canada a sustained decrease in 30-day in-hospital mortality has been found commensurate with an increase in resources to establish the multifactorial stroke system intervention for stroke treatment and prevention [ 15 ]. Arguably, with rapid advances in evidence and in the context of an ageing population with high cost and care burden and substantive impacts on quality of life, stroke is an area with a need for rapid research translation into evidence-based and value-based healthcare improvement. However, a recent systematic review found that the existing literature had few comprehensive examples of LHS adoption [ 12 ]. Although healthcare improvement systems and approaches were described, less is known about patient-clinician and stakeholder engagement, governance and culture, or embedding of data informatics into everyday practice to inform and drive improvement [ 12 ]. For example, in a recent review of quality improvement collaborations, it was found that although clinical processes in stroke care are improved, their short-term nature means there is uncertainty about sustainability and impacts on patient outcomes [ 16 ]. Table  1 provides the main features of the Australian Stroke LHS based on the four core domains and eight elements of the Learning Together for Better Health Framework described in Fig.  1 . The features are further expanded on in the following sections.

Evidence from stakeholders (LHS quadrant 1, Fig.  1 )

Engagement, partners and priorities.

Within the stroke field, there have been various support mechanisms to facilitate an LHS approach including partnership and broad stakeholder engagement that includes clinical networks and policy makers from different jurisdictions. Since 2008, the Australian Stroke Coalition has been co-led by the Stroke Foundation, a charitable consumer advocacy organisation, and Stroke Society of Australasia a professional society with membership covering academics and multidisciplinary clinician networks, that are collectively working to improve stroke care ( https://australianstrokecoalition.org.au/ ). Surveys, focus groups and workshops have been used for identifying priorities from stakeholders. Recent agreed priorities have been to improve stroke care and strengthen the voice for stroke care at a national ( https://strokefoundation.org.au/ ) and international level ( https://www.world-stroke.org/news-and-blog/news/world-stroke-organization-tackle-gaps-in-access-to-quality-stroke-care ), as well as reduce duplication amongst stakeholders. This activity is built on a foundation and culture of research and innovation embedded within the stroke ‘community of practice’. Consumers, as people with lived experience of stroke are important members of the Australian Stroke Coalition, as well as representatives from different clinical colleges. Consumers also provide critical input to a range of LHS activities via the Stroke Foundation Consumer Council, Stroke Living Guidelines committees, and the Australian Stroke Clinical Registry (AuSCR) Steering Committee (described below).

Evidence from research (LHS quadrant 2, Fig.  1 )

Advancement of the evidence for stroke interventions and synthesis into clinical guidelines.

To implement best practice, it is crucial to distil the large volume of scientific and trial literature into actionable recommendations for clinicians to use in practice [ 24 ]. The first Australian clinical guidelines for acute stroke were produced in 2003 following the increasing evidence emerging for prevention interventions (e.g. carotid endarterectomy, blood pressure lowering), acute medical treatments (intravenous thrombolysis, aspirin within 48 h of ischemic stroke), and optimised hospital management (care in dedicated stroke units by a specialised and coordinated multidisciplinary team) [ 25 ]. Importantly, a number of the innovations were developed, researched and proven effective by key opinion leaders embedded in the Australian stroke care community. In 2005, the clinical guidelines for Stroke Rehabilitation and Recovery [ 26 ] were produced, with subsequent merged guidelines periodically updated. However, the traditional process of periodic guideline updates is challenging for end users when new research can render recommendations redundant and this lack of currency erodes stakeholder trust [ 27 ]. In response to this challenge the Stroke Foundation and Cochrane Australia entered a pioneering project to produce the first electronic ‘living’ guidelines globally [ 20 ]. Major shifts in the evidence for reperfusion therapies (e.g. extended time-window intravenous thrombolysis and endovascular clot retrieval), among other advances, were able to be converted into new recommendations, approved by the Australian National Health and Medical Research Council within a few months of publication. Feedback on this process confirmed the increased use and trust in the guidelines by clinicians. The process informed other living guidelines programs, including the successful COVID-19 clinical guidelines [ 28 ].

However, best practice clinical guideline recommendations are necessary but insufficient for healthcare improvement and nesting these within an LHS with stakeholder partnership, enables implementation via a range of proven methods, including audit and feedback strategies [ 29 ].

Evidence from data and practice (LHS quadrant 3, Fig.  1 )

Data systems and benchmarking : revealing the disparities in care between health services. A national system for standardized stroke data collection was established as the National Stroke Audit program in 2007 by the Stroke Foundation [ 30 ] following various state-level programs (e.g. New South Wales Audit) [ 31 ] to identify evidence-practice gaps and prioritise improvement efforts to increase access to stroke units and other acute treatments [ 32 ]. The Audit program alternates each year between acute (commencing in 2007) and rehabilitation in-patient services (commencing in 2008). The Audit program provides a ‘deep dive’ on the majority of recommendations in the clinical guidelines whereby participating hospitals provide audits of up to 40 consecutive patient medical records and respond to a survey about organizational resources to manage stroke. In 2009, the AuSCR was established to provide information on patients managed in acute hospitals based on a small subset of quality processes of care linked to benchmarked reports of performance (Fig.  2 ) [ 33 ]. In this way, the continuous collection of high-priority processes of stroke care could be regularly collected and reviewed to guide improvement to care [ 34 ]. Plus clinical quality registry programs within Australia have shown a meaningful return on investment attributed to enhanced survival, improvements in quality of life and avoided costs of treatment or hospital stay [ 35 ].

Example performance report from the Australian Stroke Clinical Registry: average door-to-needle time in providing intravenous thrombolysis by different hospitals in 2021 [ 36 ]. Each bar in the figure represents a single hospital

The Australian Stroke Coalition endorsed the creation of an integrated technological solution for collecting data through a single portal for multiple programs in 2013. In 2015, the Stroke Foundation, AuSCR consortium, and other relevant groups cooperated to design an integrated data management platform (the Australian Stroke Data Tool) to reduce duplication of effort for hospital staff in the collection of overlapping variables in the same patients [ 19 ]. Importantly, a national data dictionary then provided the common data definitions to facilitate standardized data capture. Another important feature of AuSCR is the collection of patient-reported outcome surveys between 90 and 180 days after stroke, and annual linkage with national death records to ascertain survival status [ 33 ]. To support a LHS approach, hospitals that participate in AuSCR have access to a range of real-time performance reports. In efforts to minimize the burden of data collection in the AuSCR, interoperability approaches to import data directly from hospital or state-level managed stroke databases have been established (Fig.  3 ); however, the application has been variable and 41% of hospitals still manually enter all their data.

Current status of automated data importing solutions in the Australian Stroke Clinical Registry, 2022, with ‘ n ’ representing the number of hospitals. AuSCR, Australian Stroke Clinical Registry; AuSDaT, Australian Stroke Data Tool; API, Application Programming Interface; ICD, International Classification of Diseases; RedCAP, Research Electronic Data Capture; eMR, electronic medical records

For acute stroke care, the Australian Commission on Quality and Safety in Health Care facilitated the co-design (clinicians, academics, consumers) and publication of the national Acute Stroke Clinical Care Standard in 2015 [ 17 ], and subsequent review [ 18 ]. The indicator set for the Acute Stroke Standard then informed the expansion of the minimum dataset for AuSCR so that hospitals could routinely track their performance. The national Audit program enabled hospitals not involved in the AuSCR to assess their performance every two years against the Acute Stroke Standard. Complementing these efforts, the Stroke Foundation, working with the sector, developed the Acute and Rehabilitation Stroke Services Frameworks to outline the principles, essential elements, models of care and staffing recommendations for stroke services ( https://informme.org.au/guidelines/national-stroke-services-frameworks ). The Frameworks are intended to guide where stroke services should be developed, and monitor their uptake with the organizational survey component of the Audit program.

Evidence from implementation and healthcare improvement (LHS quadrant 4, Fig.  1 )

Research to better utilize and augment data from registries through linkage [ 37 , 38 , 39 , 40 ] and to ensure presentation of hospital or service level data are understood by clinicians has ensured advancement in the field for the Australian Stroke LHS [ 41 ]. Importantly, greater insights into whole patient journeys, before and after a stroke, can now enable exploration of value-based care. The LHS and stroke data platform have enabled focused and time-limited projects to create a better understanding of the quality of care in acute or rehabilitation settings [ 22 , 42 , 43 ]. Within stroke, all the elements of an LHS culminate into the ready availability of benchmarked performance data and support for implementation of strategies to address gaps in care.

Implementation research to grow the evidence base for effective improvement interventions has also been a key pillar in the Australian context. These include multi-component implementation interventions to achieve behaviour change for particular aspects of stroke care, [ 22 , 23 , 44 , 45 ] and real-world approaches to augmenting access to hyperacute interventions in stroke through the use of technology and telehealth [ 46 , 47 , 48 , 49 ]. The evidence from these studies feeds into the living guidelines program and the data collection systems, such as the Audit program or AuSCR, which are then amended to ensure data aligns to recommended care. For example, the use of ‘hyperacute aspirin within the first 48 h of ischemic stroke’ was modified to be ‘hyperacute antiplatelet…’ to incorporate new evidence that other medications or combinations are appropriate to use. Additionally, new datasets have been developed to align with evidence such as the Fever, Sugar, and Swallow variables [ 42 ]. Evidence on improvements in access to best practice care from the acute Audit program [ 50 ] and AuSCR is emerging [ 36 ]. For example, between 2007 and 2017, the odds of receiving intravenous thrombolysis after ischemic stroke increased by 16% 9OR 1.06 95% CI 1.13–1.18) and being managed in a stroke unit by 18% (OR 1.18 95% CI 1.17–1.20). Over this period, the median length of hospital stay for all patients decreased from 6.3 days in 2007 to 5.0 days in 2017 [ 51 ]. When considering the number of additional patients who would receive treatment in 2017 in comparison to 2007 it was estimated that without this additional treatment, over 17,000 healthy years of life would be lost in 2017 (17,786 disability-adjusted life years) [ 51 ]. There is evidence on the cost-effectiveness of different system-focussed strategies to augment treatment access for acute ischemic stroke (e.g. Victorian Stroke Telemedicine program [ 52 ] and Melbourne Mobile Stroke Unit ambulance [ 53 ]). Reciprocally, evidence from the national Rehabilitation Audit, where the LHS approach has been less complete or embedded, has shown fewer areas of healthcare improvement over time [ 51 , 54 ].

Within the field of stroke in Australia, there is indirect evidence that the collective efforts that align to establishing the components of a LHS have had an impact. Overall, the age-standardised rate of stroke events has reduced by 27% between 2001 and 2020, from 169 to 124 events per 100,000 population. Substantial declines in mortality rates have been reported since 1980. Commensurate with national clinical guidelines being updated in 2007 and the first National Stroke Audit being undertaken in 2007, the mortality rates for men (37.4 deaths per 100,000) and women (36.1 deaths per 100,0000 has declined to 23.8 and 23.9 per 100,000, respectively in 2021 [ 55 ].

Underpinning the LHS with the integration of the four quadrants of evidence from stakeholders, research and guidelines, practice and implementation, and core LHS principles have been addressed. Leadership and governance have been important, and programs have been established to augment workforce training and capacity building in best practice professional development. Medical practitioners are able to undertake courses and mentoring through the Australasian Stroke Academy ( http://www.strokeacademy.com.au/ ) while nurses (and other health professionals) can access teaching modules in stroke care from the Acute Stroke Nurses Education Network ( https://asnen.org/ ). The Association of Neurovascular Clinicians offers distance-accessible education and certification to develop stroke expertise for interdisciplinary professionals, including advanced stroke co-ordinator certification ( www.anvc.org ). Consumer initiative interventions are also used in the design of the AuSCR Public Summary Annual reports (available at https://auscr.com.au/about/annual-reports/ ) and consumer-related resources related to the Living Guidelines ( https://enableme.org.au/resources ).

The important success factors and lessons from stroke as a national exemplar LHS in Australia include leadership, culture, workforce and resources integrated with (1) established and broad partnerships across the academic-clinical sector divide and stakeholder engagement; (2) the living guidelines program; (3) national data infrastructure, including a national data dictionary that provides the common data framework to support standardized data capture; (4) various implementation strategies including benchmarking and feedback as well as engagement strategies targeting different levels of the health system; and (5) implementation and improvement research to advance stroke systems of care and reduce unwarranted variation in practice (Fig.  1 ). Priority opportunities now include the advancement of interoperability with electronic medical records as an area all clinical quality registry’s programs needs to be addressed, as well as providing more dynamic and interactive data dashboards tailored to the need of clinicians and health service executives.

There is a clear mandate to optimise healthcare improvement with big data offering major opportunities for change. However, we have lacked the approaches to capture evidence from the community and stakeholders, to integrate evidence from research, to capture and leverage data or evidence from practice and to generate and build on evidence from implementation using iterative system-level improvement. The LHS provides this opportunity and is shown to deliver impact. Here, we have outlined the process applied to generate an evidence-based LHS and provide a leading exemplar in stroke care. This highlights the value of moving from single-focus isolated approaches/initiatives to healthcare improvement and the benefit of integration to deliver demonstrable outcomes for our funders and key stakeholders — our community. This work provides insight into strategies that can both apply evidence-based processes to healthcare improvement as well as implementing evidence-based practices into care, moving beyond research as an endpoint, to research as an enabler, underpinning delivery of better healthcare.

Availability of data and materials

Not applicable

Abbreviations

Australian Stroke Clinical Registry

Confidence interval

• Learning Health System

World Health Organization. Delivering quality health services . OECD Publishing; 2018.

Enticott J, Braaf S, Johnson A, Jones A, Teede HJ. Leaders’ perspectives on learning health systems: A qualitative study. BMC Health Serv Res. 2020;20:1087.

Article   PubMed   PubMed Central   Google Scholar  

Melder A, Robinson T, McLoughlin I, Iedema R, Teede H. An overview of healthcare improvement: Unpacking the complexity for clinicians and managers in a learning health system. Intern Med J. 2020;50:1174–84.

Article   PubMed   Google Scholar  

Alberto IRI, Alberto NRI, Ghosh AK, Jain B, Jayakumar S, Martinez-Martin N, et al. The impact of commercial health datasets on medical research and health-care algorithms. Lancet Digit Health. 2023;5:e288–94.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Dixon-Woods M. How to improve healthcare improvement—an essay by Mary Dixon-Woods. BMJ. 2019;367: l5514.

Enticott J, Johnson A, Teede H. Learning health systems using data to drive healthcare improvement and impact: A systematic review. BMC Health Serv Res. 2021;21:200.

Enticott JC, Melder A, Johnson A, Jones A, Shaw T, Keech W, et al. A learning health system framework to operationalize health data to improve quality care: An Australian perspective. Front Med (Lausanne). 2021;8:730021.

Dammery G, Ellis LA, Churruca K, Mahadeva J, Lopez F, Carrigan A, et al. The journey to a learning health system in primary care: A qualitative case study utilising an embedded research approach. BMC Prim Care. 2023;24:22.

Foley T, Horwitz L, Zahran R. The learning healthcare project: Realising the potential of learning health systems. 2021. Available from https://learninghealthcareproject.org/wp-content/uploads/2021/05/LHS2021report.pdf . Accessed Jan 2024.

Institute of Medicine. Best care at lower cost: The path to continuously learning health care in America. Washington: The National Academies Press; 2013.

Google Scholar  

Zurynski Y, Smith CL, Vedovi A, Ellis LA, Knaggs G, Meulenbroeks I, et al. Mapping the learning health system: A scoping review of current evidence - a white paper. 2020:63

Cadilhac DA, Bravata DM, Bettger J, Mikulik R, Norrving B, Uvere E, et al. Stroke learning health systems: A topical narrative review with case examples. Stroke. 2023;54:1148–59.

Braithwaite J, Glasziou P, Westbrook J. The three numbers you need to know about healthcare: The 60–30-10 challenge. BMC Med. 2020;18:1–8.

Article   Google Scholar  

King D, Wittenberg R, Patel A, Quayyum Z, Berdunov V, Knapp M. The future incidence, prevalence and costs of stroke in the UK. Age Ageing. 2020;49:277–82.

Ganesh A, Lindsay P, Fang J, Kapral MK, Cote R, Joiner I, et al. Integrated systems of stroke care and reduction in 30-day mortality: A retrospective analysis. Neurology. 2016;86:898–904.

Lowther HJ, Harrison J, Hill JE, Gaskins NJ, Lazo KC, Clegg AJ, et al. The effectiveness of quality improvement collaboratives in improving stroke care and the facilitators and barriers to their implementation: A systematic review. Implement Sci. 2021;16:16.

Australian Commission on Safety and Quality in Health Care. Acute stroke clinical care standard. 2015. Available from https://www.safetyandquality.gov.au/our-work/clinical-care-standards/acute-stroke-clinical-care-standard . Accessed Jan 2024.

Australian Commission on Safety and Quality in Health Care. Acute stroke clinical care standard. Sydney: ACSQHC; 2019. Available from https://www.safetyandquality.gov.au/publications-and-resources/resource-library/acute-stroke-clinical-care-standard-evidence-sources . Accessed Jan 2024.

Ryan O, Ghuliani J, Grabsch B, Hill K, G CC, Breen S, et al. Development, implementation, and evaluation of the Australian Stroke Data Tool (AuSDaT): Comprehensive data capturing for multiple uses. Health Inf Manag. 2022:18333583221117184.

English C, Bayley M, Hill K, Langhorne P, Molag M, Ranta A, et al. Bringing stroke clinical guidelines to life. Int J Stroke. 2019;14:337–9.

English C, Hill K, Cadilhac DA, Hackett ML, Lannin NA, Middleton S, et al. Living clinical guidelines for stroke: Updates, challenges and opportunities. Med J Aust. 2022;216:510–4.

Cadilhac DA, Grimley R, Kilkenny MF, Andrew NE, Lannin NA, Hill K, et al. Multicenter, prospective, controlled, before-and-after, quality improvement study (Stroke123) of acute stroke care. Stroke. 2019;50:1525–30.

Cadilhac DA, Marion V, Andrew NE, Breen SJ, Grabsch B, Purvis T, et al. A stepped-wedge cluster-randomized trial to improve adherence to evidence-based practices for acute stroke management. Jt Comm J Qual Patient Saf. 2022.

Elliott J, Lawrence R, Minx JC, Oladapo OT, Ravaud P, Jeppesen BT, et al. Decision makers need constantly updated evidence synthesis. Nature. 2021;600:383–5.

Article   CAS   PubMed   Google Scholar  

National Stroke Foundation. National guidelines for acute stroke management. Melbourne: National Stroke Foundation; 2003.

National Stroke Foundation. Clinical guidelines for stroke rehabilitation and recovery. Melbourne: National Stroke Foundation; 2005.

Phan TG, Thrift A, Cadilhac D, Srikanth V. A plea for the use of systematic review methodology when writing guidelines and timely publication of guidelines. Intern Med J . 2012;42:1369–1371; author reply 1371–1362

Tendal B, Vogel JP, McDonald S, Norris S, Cumpston M, White H, et al. Weekly updates of national living evidence-based guidelines: Methods for the Australian living guidelines for care of people with COVID-19. J Clin Epidemiol. 2021;131:11–21.

Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:50.

Harris D, Cadilhac D, Hankey GJ, Hillier S, Kilkenny M, Lalor E. National stroke audit: The Australian experience. Clin Audit. 2010;2:25–31.

Cadilhac DA, Purvis T, Kilkenny MF, Longworth M, Mohr K, Pollack M, et al. Evaluation of rural stroke services: Does implementation of coordinators and pathways improve care in rural hospitals? Stroke. 2013;44:2848–53.

Cadilhac DA, Moss KM, Price CJ, Lannin NA, Lim JY, Anderson CS. Pathways to enhancing the quality of stroke care through national data monitoring systems for hospitals. Med J Aust. 2013;199:650–1.

Cadilhac DA, Lannin NA, Anderson CS, Levi CR, Faux S, Price C, et al. Protocol and pilot data for establishing the Australian Stroke Clinical Registry. Int J Stroke. 2010;5:217–26.

Ivers N, Jamtvedt G, Flottorp S, Young J, Odgaard-Jensen J, French S, et al. Audit and feedback: Effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev . 2012

Australian Commission on Safety and Quality in Health Care. Economic evaluation of clinical quality registries. Final report. . 2016:79

Cadilhac DA, Dalli LL, Morrison J, Lester M, Paice K, Moss K, et al. The Australian Stroke Clinical Registry annual report 2021. Melbourne; 2022. Available from https://auscr.com.au/about/annual-reports/ . Accessed 6 May 2024.

Kilkenny MF, Kim J, Andrew NE, Sundararajan V, Thrift AG, Katzenellenbogen JM, et al. Maximising data value and avoiding data waste: A validation study in stroke research. Med J Aust. 2019;210:27–31.

Eliakundu AL, Smith K, Kilkenny MF, Kim J, Bagot KL, Andrew E, et al. Linking data from the Australian Stroke Clinical Registry with ambulance and emergency administrative data in Victoria. Inquiry. 2022;59:469580221102200.

PubMed   Google Scholar  

Andrew NE, Kim J, Cadilhac DA, Sundararajan V, Thrift AG, Churilov L, et al. Protocol for evaluation of enhanced models of primary care in the management of stroke and other chronic disease (PRECISE): A data linkage healthcare evaluation study. Int J Popul Data Sci. 2019;4:1097.

CAS   PubMed   PubMed Central   Google Scholar  

Mosalski S, Shiner CT, Lannin NA, Cadilhac DA, Faux SG, Kim J, et al. Increased relative functional gain and improved stroke outcomes: A linked registry study of the impact of rehabilitation. J Stroke Cerebrovasc Dis. 2021;30: 106015.

Ryan OF, Hancock SL, Marion V, Kelly P, Kilkenny MF, Clissold B, et al. Feedback of aggregate patient-reported outcomes (PROs) data to clinicians and hospital end users: Findings from an Australian codesign workshop process. BMJ Open. 2022;12:e055999.

Grimley RS, Rosbergen IC, Gustafsson L, Horton E, Green T, Cadigan G, et al. Dose and setting of rehabilitation received after stroke in Queensland, Australia: A prospective cohort study. Clin Rehabil. 2020;34:812–23.

Purvis T, Middleton S, Craig LE, Kilkenny MF, Dale S, Hill K, et al. Inclusion of a care bundle for fever, hyperglycaemia and swallow management in a national audit for acute stroke: Evidence of upscale and spread. Implement Sci. 2019;14:87.

Middleton S, McElduff P, Ward J, Grimshaw JM, Dale S, D’Este C, et al. Implementation of evidence-based treatment protocols to manage fever, hyperglycaemia, and swallowing dysfunction in acute stroke (QASC): A cluster randomised controlled trial. Lancet. 2011;378:1699–706.

Middleton S, Dale S, Cheung NW, Cadilhac DA, Grimshaw JM, Levi C, et al. Nurse-initiated acute stroke care in emergency departments. Stroke. 2019:STROKEAHA118020701.

Hood RJ, Maltby S, Keynes A, Kluge MG, Nalivaiko E, Ryan A, et al. Development and pilot implementation of TACTICS VR: A virtual reality-based stroke management workflow training application and training framework. Front Neurol. 2021;12:665808.

Bladin CF, Kim J, Bagot KL, Vu M, Moloczij N, Denisenko S, et al. Improving acute stroke care in regional hospitals: Clinical evaluation of the Victorian Stroke Telemedicine program. Med J Aust. 2020;212:371–7.

Bladin CF, Bagot KL, Vu M, Kim J, Bernard S, Smith K, et al. Real-world, feasibility study to investigate the use of a multidisciplinary app (Pulsara) to improve prehospital communication and timelines for acute stroke/STEMI care. BMJ Open. 2022;12:e052332.

Zhao H, Coote S, Easton D, Langenberg F, Stephenson M, Smith K, et al. Melbourne mobile stroke unit and reperfusion therapy: Greater clinical impact of thrombectomy than thrombolysis. Stroke. 2020;51:922–30.

Purvis T, Cadilhac DA, Hill K, Reyneke M, Olaiya MT, Dalli LL, et al. Twenty years of monitoring acute stroke care in Australia from the national stroke audit program (1999–2019): Achievements and areas of future focus. J Health Serv Res Policy. 2023.

Cadilhac DA, Purvis T, Reyneke M, Dalli LL, Kim J, Kilkenny MF. Evaluation of the national stroke audit program: 20-year report. Melbourne; 2019.

Kim J, Tan E, Gao L, Moodie M, Dewey HM, Bagot KL, et al. Cost-effectiveness of the Victorian Stroke Telemedicine program. Aust Health Rev. 2022;46:294–301.

Kim J, Easton D, Zhao H, Coote S, Sookram G, Smith K, et al. Economic evaluation of the Melbourne mobile stroke unit. Int J Stroke. 2021;16:466–75.

Stroke Foundation. National stroke audit – rehabilitation services report 2020. Melbourne; 2020.

Australian Institute of Health and Welfare. Heart, stroke and vascular disease: Australian facts. 2023. Webpage https://www.aihw.gov.au/reports/heart-stroke-vascular-diseases/hsvd-facts/contents/about (accessed Jan 2024).

Download references

Acknowledgements

The following authors hold National Health and Medical Research Council Research Fellowships: HT (#2009326), DAC (#1154273), SM (#1196352), MFK Future Leader Research Fellowship (National Heart Foundation #105737). The Funders of this work did not have any direct role in the design of the study, its execution, analyses, interpretation of the data, or decision to submit results for publication.

Author information

Helena Teede and Dominique A. Cadilhac contributed equally.

Authors and Affiliations

Monash Centre for Health Research and Implementation, 43-51 Kanooka Grove, Clayton, VIC, Australia

Helena Teede, Emily Callander & Joanne Enticott

Monash Partners Academic Health Science Centre, 43-51 Kanooka Grove, Clayton, VIC, Australia

Helena Teede & Alison Johnson

Stroke and Ageing Research, Department of Medicine, School of Clinical Sciences at Monash Health, Monash University, Level 2 Monash University Research, Victorian Heart Hospital, 631 Blackburn Rd, Clayton, VIC, Australia

Dominique A. Cadilhac, Tara Purvis & Monique F. Kilkenny

Stroke Theme, The Florey Institute of Neuroscience and Mental Health, University of Melbourne, Heidelberg, VIC, Australia

Dominique A. Cadilhac, Monique F. Kilkenny & Bruce C.V. Campbell

Department of Neurology, Melbourne Brain Centre, Royal Melbourne Hospital, Parkville, VIC, Australia

Bruce C.V. Campbell

Department of Medicine, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Victoria, Australia

School of Health Sciences, Heart and Stroke Program, University of Newcastle, Hunter Medical Research Institute, University Drive, Callaghan, NSW, Australia

Coralie English

School of Medicine and Dentistry, Griffith University, Birtinya, QLD, Australia

Rohan S. Grimley

Clinical Excellence Division, Queensland Health, Brisbane, Australia

John Hunter Hospital, Hunter New England Local Health District and University of Newcastle, Sydney, NSW, Australia

Christopher Levi

School of Nursing, Midwifery and Paramedicine, Australian Catholic University, Sydney, NSW, Australia

Sandy Middleton

Nursing Research Institute, St Vincent’s Health Network Sydney and and Australian Catholic University, Sydney, NSW, Australia

Stroke Foundation, Level 7, 461 Bourke St, Melbourne, VIC, Australia

Kelvin Hill

You can also search for this author in PubMed   Google Scholar

Contributions

HT: conception, design and initial draft, developed the theoretical formalism for learning health system framework, approved the submitted version. DAC: conception, design and initial draft, provided essential literature and case study examples, approved the submitted version. TP: revised the manuscript critically for important intellectual content, approved the submitted version. MFK: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. BC: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. CE: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. AJ: conception, design and initial draft, developed the theoretical formalism for learning health system framework, approved the submitted version. EC: revised the manuscript critically for important intellectual content, approved the submitted version. RSG: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. CL: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. SM: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. KH: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. JE: conception, design and initial draft, developed the theoretical formalism for learning health system framework, approved the submitted version. All authors read and approved the final manuscript.

Authors’ Twitter handles

@HelenaTeede

@DominiqueCad

@Coralie_English

@EmilyCallander

@EnticottJo

Corresponding authors

Correspondence to Helena Teede or Dominique A. Cadilhac .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Teede, H., Cadilhac, D.A., Purvis, T. et al. Learning together for better health using an evidence-based Learning Health System framework: a case study in stroke. BMC Med 22 , 198 (2024). https://doi.org/10.1186/s12916-024-03416-w

Download citation

Received : 23 July 2023

Accepted : 30 April 2024

Published : 15 May 2024

DOI : https://doi.org/10.1186/s12916-024-03416-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Evidence-based medicine
• Person-centred care
• Models of care
• Healthcare improvement

BMC Medicine

ISSN: 1741-7015

• Submission enquiries: [email protected]
• General enquiries: [email protected]

To read this content please select one of the options below:

Please note you do not have access to teaching notes, a comprehensive review of open innovation literature.

Journal of Science and Technology Policy Management

ISSN : 2053-4620

Article publication date: 7 March 2016

The purpose of this study is to pursue a comprehensive review of the progress of open innovation literature.

Design/methodology/approach

Using a wide range of literature sources, altogether 293 articles relevant to the study’s objective were identified for statistical analysis. Moreover, contributory articles published from 2003 to June 2015 were included for content analysis.

The study contributes in two ways. First, based on content analysis of the selected contributory articles, the authors shed light on the overall development of the open innovation literature and highlight the findings of significant studies. Second, the authors provide a detailed picture of the progress of open innovation literature by analyzing the comprehensive set of articles. Total yearly publication activity was calculated, and publication activity in different disciplines was addressed. The study unveils most influential articles, authors and journals that have discussed open innovation. The geographical locations of influential articles and authors are revealed. Additionally, frequently used keywords are listed.

Originality/value

The authors present a new framework of open innovation research, highlight the progress of existing research and suggest avenues for future research.

• Literature review
• Open innovation
• Content analysis
• Research and development
• Bibliometric study

Acknowledgements

The authors are grateful to the Finnish Cultural Foundation for financial support.

Hossain, M. , Islam, K.M.Z. , Sayeed, M.A. and Kauranen, I. (2016), "A comprehensive review of open innovation literature", Journal of Science and Technology Policy Management , Vol. 7 No. 1, pp. 2-25. https://doi.org/10.1108/JSTPM-02-2015-0009

Emerald Group Publishing Limited

Copyright © 2016, Emerald Group Publishing Limited

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

Mapping Payment and Pricing Schemes for Health Innovation: Protocol of a Scoping Literature Review

• Open access
• Published: 21 May 2024

Cite this article

You have full access to this open access article

• Vittoria Ardito   ORCID: orcid.org/0000-0003-2034-716X 1 ,
• Ludovico Cavallaro   ORCID: orcid.org/0009-0002-3426-2570 1 ,
• Michael Drummond   ORCID: orcid.org/0000-0002-6126-0944 1 , 2 &
• Oriana Ciani   ORCID: orcid.org/0000-0002-3607-0508 1  

Introduction

Innovative pricing and payment/reimbursement schemes have been proposed as one part of the solution to the problem of patient access to new health technologies or to the uncertainty about their long-term effectiveness. As part of a Horizon Europe research project on health innovation next generation pricing and payment models (HI-PRIX), this protocol illustrates the conceptual and methodological steps related to a scoping review aiming at investigating nature and scope of pricing and payment/reimbursement schemes applied to, or proposed for, existing or new health technologies.

A scoping review of literature will be performed according to the PRISMA guidelines for scoping reviews (PRISMA-ScR) guidelines. The search will be conducted in three scientific databases (i.e., PubMed, Web of Science, and Scopus), over a 2010–2023 timeframe. The search strategy is structured around two blocks of keywords, namely “pricing and payment/reimbursement schemes,” and “innovativeness” (of the scheme type or scheme use). A simplified search will be replicated in the gray literature. Studies illustrating pricing and payment/reimbursement schemes with a sufficient level of details to explain their characteristics and functioning will be deemed eligible to be considered for data synthesis. Pricing and payment/reimbursement schemes will be classified according to several criteria, such as their purpose, nature, governance, data collection needs, and foreseen distribution of risk. The results will populate a publicly available online tool, the Pay-for-Innovation Observatory.

The findings of this review have the potential to offer a comprehensive toolkit with a variety of pricing and payment schemes to policymakers and manufacturers facing reimbursement and access decisions.

Avoid common mistakes on your manuscript.

1 Introduction

1.1 background.

Medical innovation is advancing rapidly, but it is often characterized by clinical and economic uncertainty at the time of entry to the health care system. For medicinal products, clinical uncertainty is linked to the fact that often pivotal studies used for marketing approval do not follow the “gold standard” [i.e., blinded, two-arm, phase III randomized controlled trials (RCTs)] [ 1 ], or rely on surrogate endpoints as predictors of clinical effectiveness [ 2 ]. Regulatory agencies such as the European Medicines Agency (EMA) and the Federal Drug Administration (FDA) are, therefore, granting marketing authorization on the basis of incomplete or limited evidence, sometimes with the commitment for the manufacturer to conduct postapproval clinical studies [ 2 , 3 , 4 , 5 ]. For medical devices (MDs), the quantity, type, and quality of evidence required for their approval has been traditionally considered to be weaker than for drugs [ 6 , 7 ]. RCTs can often be not viable for MDs, characterized by unique features such as the incremental innovation or the learning curve associated with reiterated use. Similar considerations arise for drug–device combinations, including software-incorporating devices and digital medical devices, for which the incremental improvements of the device/software components can rapidly make past RCT results outdated. This scenario is often coupled with extremely high prices of certain health innovations, often resulting in large upfront payments, that occur before accrual of any clinical benefits and generate large budgetary impacts associated with reimbursement or coverage [ 8 ]. Considered together, this situation is posing significant challenges to authorities, payers, providers, and, ultimately, patients.

On top of this, access to health innovations can be further challenged by operational complexities raised by the distinctive features of certain technologies. These characteristics include new modes of administration (e.g., single-administration therapies), high treatment personalization, the need for a highly-skilled workforce, sophistication of the logistics of treatment delivery (e.g., transportation of lab-treated specimens), and difficulties in scaling up the manufacturing capacity due to the above. Consider, for example, Advanced Therapy Medicinal Products (ATMPs), often cited as paradigmatic examples of technologies that may combine all those challenges together [ 9 , 10 ].

In this context, new pricing and payment/reimbursement models have been proposed as practical solutions to ensure timely patient access to promising innovations, while simultaneously addressing coverage problems. Innovative payment models (IPM) are agreements between manufacturers, governmental bodies, and payers defined to act as a bridge to access, reward research and Advanced therapy medicinal products: Overview.development (R&D) efforts adequately, and balance the financial sustainability of healthcare systems [ 11 ]. New payment models have been termed differently and might be referred to as risk-sharing agreements (RSA) [ 12 ], managed entry agreements (MEAs) [ 13 ], or innovative contracting [ 14 ]. They might have a wide variety of formulations, with outcome-based and/or financial-based components or with payments split over time (e.g., instalments or annuities). For instance, a taxonomy developed by Carlson and colleagues categorized performance-based reimbursement schemes in terms of timing, execution, and health outcomes, distinguishing between outcome-based versus nonoutcome-based schemes [ 15 ]. In addition, Towse et al. distinguished between the agreements that specified how evidence would be translated into revisions of price, revenues, and/or use, and those that instead specified an evidence review point where renegotiation would take place [ 12 ]. Other frameworks have focused on coverage options more generally, distinguishing between schemes with objectives of evidence generation from those of price reduction [ 16 ], on different types of performance-based RSAs [ 17 ], or on the key reasons for using MEAs [ 18 , 19 ]. More recently, Horrow and Kesselheim developed a taxonomy of possible payment arrangements for gene therapies, that include, among others, installments, subscriptions, expenditure caps, and others [ 8 ].

For clarity, we specify that this work will be focused on health technologies at large, including medicinal products, MDs, and drug–device combinations, and that these will be referred to interchangeably as “health technologies” or “health innovations.” We also specify in this context the distinction between pricing and payment/reimbursement schemes. “Pricing schemes” refer to any approach or methodology to calculate, measure, or quantify a fair price for health technologies. An example is rate of return pricing, namely a scheme in which a prespecified rate of return is ensured to manufacturers, after covering the costs of developing and marketing the product [ 20 ]. On the other hand, “payment/reimbursement schemes” or arrangements refer to formulation of any aspect that has to be defined to govern the payment of health innovations, including, but not limited to, the types and number of stakeholders involved, the moment in which the payment occurs, the split of payments over time, or the linkage to an outcome component. Examples here include the subscription model, that delinks reimbursements from volumes of sales, offering manufacturers a fixed monetary amount [ 21 ], or the conditional treatment continuation agreement, where coverage is continued only for patients who achieve a prespecified response to treatment [ 13 , 22 , 23 ]. While the two approaches might capture the same value from different perspectives (i.e., manufacturers and payers), this is not always the case.

1.2 Objectives

Given the contemporary challenges experienced by healthcare systems globally in ensuring access to the latest available health technologies, the objective of this study is to perform an extensive mapping of the pricing and payment/reimbursement schemes that are currently used, or have been proposed, to allow for timely and widespread use of potentially innovative health technologies. Specifically, a scoping literature review will be conducted to respond to the following three research objectives:

To generate a comprehensive and updated catalogue of innovative pricing and payment/reimbursement schemes for health technologies;

To develop a conceptual framework that characterizes any pricing and payment/reimbursement schemes for health technologies, ultimately contributing to cluster them through a newly defined taxonomy;

To investigate which pricing and payment/reimbursement schemes are better suited to address a given coverage or reimbursement challenge, by accounting for the distinctive features of different technology classes, therapeutic areas, settings and healthcare systems, and ultimately clarifying which scheme best serves a given policy objective.

2.1 Protocol and Registration

This protocol was developed based on the PRISMA protocol guidelines and written in accordance with the PRISMA-P statement [ 24 , 25 ]. The protocol has been registered in the International Prospective Register of Systematic Reviews (PROSPERO; registration number: CRD42023444824). The review will be conducted according to the updated methodological guidance and the PRISMA guidelines for scoping reviews (PRISMA-ScR) [ 26 , 27 ]. Scoping reviews are a type of knowledge synthesis that follow a systematic approach to map relevant concepts, theories, sources, and knowledge gaps in a given area by extensively identifying, reviewing, and synthetizing the evidence available in literature [ 28 ].

2.2 Intervention

This scoping review will be focused both on pricing and payment/reimbursement schemes for health technologies as described above. Such schemes will be investigated across several dimensions relevant to their application, including technology classes, therapeutic areas, setting of care, healthcare systems, and geographies. These dimensions are described in more detail below.

2.3 Setting

Any pricing and payment/reimbursement schemes/strategy/arrangements that are used or that have been proposed for health technologies delivered either in-hospital or outpatient settings will be included. Within this perimeter, the focus is on technologies for which a pricing arrangement has to be established and negotiated with a manufacturer (i.e., external innovation). Conversely, innovations in services originated directly by health care providers (e.g., hospital-based innovation or innovation embedded in healthcare service delivery processes) will not be considered (i.e., internal innovation).

2.4 Timeframe

The timeframe of the current study will extend from 2010 onwards. Our search started in 2010 to build on the previously conducted study by Carlson et al. published in 2010, knowing that at the same time several countries started experimenting new schemes [ 15 ]. The literature search was performed in the first quarter of 2024 and will be updated in April 2024.

2.5 Eligibility Criteria

Studies illustrating pricing and payment/reimbursement schemes of health technologies with a level of detail that is sufficient to explain their functioning across different health technologies will be deemed eligible to be included in this review. Theoretical schemes (i.e., schemes that have only been proposed) and implemented schemes (i.e., schemes that have practical applications) will be equally considered in the analyses. Pricing and payment/reimbursement schemes will not be excluded based on their perceived innovativeness, as not only the scheme per se could be innovative but also the application or use in a given context. Furthermore, no exclusions will be made based on the country of implementation of the schemes, nor on the type of study design. For this reason, editorials, commentaries, and perspectives will be included when a given scheme is proposed and discussed. Search records will be extracted with no exclusions on the publication language, but the language expertise of the research team (e.g., English or Italian) will guide the study selection.

2.6 Information Sources

Literature searches will be conducted through different sources, and both scientific and gray literature will be considered.

Scientific publications will be searched in three databases, namely PubMed (Medline), Web of Science, and Scopus. In addition, the reference list of the studies included and of the reviews identified will be scanned to ensure that no relevant important work has been missed. In case relevant papers are not retrieved by our search, it will be replicated in top-tier journals in the area of pharmaceutical policy (i.e., Journal of Pharmaceutical Policy and Practice; Expert Review of Medical Devices; Expert Review of Pharmacoeconomics and Outcome Research; Value in Health, European Journal of Health Economics; PharmacoEconomics; PharmacoEconomics—Open; Health Economics; Applied Health Economics; Health Policy; Health Affairs; Applied Health Services Research and Policy; Cost-effectiveness and Resource Allocation).

As for the gray literature, reports, white papers, and websites of a range of relevant institutions will be searched. Key institutions include, but are not limited to, international organizations, industry-oriented organizations, HTA agencies, patient associations, and consulting and research companies, such as European Commission (EC), Organization for Economic Cooperation and Development (OECD), European Federation of Pharmaceutical Industries and Associations (EFPIA), International HTA Database, Pharmaceutical Pricing and Reimbursement Information (PPRI), European Patient Forum (EPF), European Patients Academy for Therapeutic Innovation (EUPATI), and others. As a subsequent step, the list of schemes identified will be circulated to the relevant individuals in HTA bodies and other relevant institutions mentioned above, in case they are aware of any that have not been identified.

2.7 Search Strategy

The structure of the search strategy is developed around two main concepts: (1) “pricing and payment/reimbursement schemes” and (2) “innovativeness” (of the scheme type or scheme use). Particularly, it is built using combinations of the following terms: performance-based, value-based, evidence-based, risk-sharing, reimbursement, rebate, pricing, contract, scheme, guarantee, and health system. To restrict the number of retrieved records, database-specific addendums are used to filter the two main search blocks, namely Mesh Terms in PubMed (Medline), Web of Science categories in Web of Science, and index terms in Scopus. The complete search for each database is presented in Table 1 .

2.8 Study Records

2.8.1 study selection.

The records retrieved through the database search will be imported in RefWorks, a tool for reference management that is used to detect and remove duplicated studies. The final list of records will be exported into a structured Microsoft Excel spreadsheet, where they will be screened based on title and abstract, and assessed against eligibility criteria. Two members of the research team (V.A. and L.C.) will assess the first 200 records based on title and abstract, and the interrater agreement will be measured using kappa statistics [ 29 ]. The remaining papers will be first screened based on title and abstract and then read full/text by two researchers. Disagreement over final inclusions will be solved by an arbitrator (O.C.). The entire research team will read all the studies eventually included in the analysis.

2.8.2 Data Collection Process

Data collection will be performed by two independent researchers (V.A. and L.C.). Data will be extracted using an ad hoc Microsoft spreadsheet, developed by the research team after preliminarily reading a pool of seminal papers. To ensure consistency across reviewers, the extraction sheet will be tested by each reviewer and possibly recalibrated before starting the data collection process. Information on the pricing and payment/reimbursement schemes will be collected, as specified in the following section.

2.8.3 Data Extraction

Data items will be collected at the individual scheme level, although different studies may contribute to the definition of a single scheme. Data items to be extracted may include general information on the scheme, and information on one or more examples of implementation, if available, as indicated in the following Table 2 .

2.8.4 Data Synthesis

The study findings will be synthetized using narrative synthesis. Descriptive statistics on the pricing and payment/reimbursement schemes identified through this review will be provided, according to the most relevant dimensions of the data collection. Given the exploratory nature of this scoping review and the variety of the types of studies (expected to be predominantly studies with qualitative designs), a quantitative synthesis of the results will not be performed. Furthermore, given the foreseen high variety of the studies, a risk of bias assessment will not be performed. In parallel, the catalogue of pricing and payment/reimbursement schemes mapped through the review will be made accessible online to the scientific community in the form of a freely available repository called the Pay-for-Innovation Observatory, that different stakeholders could use for a variety of purposes. This broad availability of the findings of the review will also facilitate constructive comments and feedback.

2.9 Machine Learning-Powered Updates of the Scoping Review

Considering the rapidly evolving landscape of health innovations and the ensuing pricing and payment challenges, our work will be periodically updated with ASReview ( https://asreview.nl/ ), an open-source machine learning (ML) software that allows to streamline the screening process for titles and abstracts within systematic reviews. In addition to the primary search, the ML-based software will be employed to perform periodic updates of the scoping review. ASReview utilizes an active researcher-in-the-loop ML algorithm, employing text mining to rank articles in terms of their likelihood for inclusion. This approach involves prior human input from the research team to guide the ML screening process and decision. ASReview offers various classifier models to determine the relevance of included articles. In a simulation study using six comprehensive systematic review datasets covering diverse topics, it was observed that the naive Bayes (NB) and term frequency-inverse document frequency (TF-IDF) models outperformed other settings [ 30 ]. The NB classifier estimates an article’s relevance probability based on TF-IDF measurements, which gauge the uniqueness of specific words within an article relative to their frequency across all articles [ 31 ]. Consequently, the combination of NB and TF-IDF has been selected for use in our work.

The software will be trained using at least one relevant and one irrelevant article to establish a foundational knowledge base, with the expectation that performance will be enhanced as prior knowledge increases.

ASReview will conduct an initial ranking of all unlabeled articles, sorting them based on descending probabilities of relevance. The top-ranked article will undergo assessment of its title and abstract against the predetermined eligibility criteria, thereby determining its relevance. Following this assessment, the ML tool will assimilate the acquired knowledge and recalibrate the article rankings, with the subsequent highest-ranked article being presented for evaluation against the eligibility criteria. This iterative interplay between the ML tool’s ranking and the reviewers’ decision making continues until reaching a data-driven stopping criterion previously defined by the research team, i.e., the sampling criterion (which entails screening a set proportion of the highest-ranked articles) and the heuristic criterion (which prompts screening cessation upon encountering n consecutive predefined irrelevant articles).

3 Discussion

This scoping review of literature aims at investigating innovative pricing and payment/reimbursement schemes for health technologies, as well as at exploring innovative ways of using established schemes (e.g., price–volume agreements). This work will be conducted as part of the larger Horizon Europe research project Health Innovation Next Generation Payment and Pricing Model (HI-PRIX; grant agreement number 101095593), which aims at fostering access to health innovations by promoting the adoption of new pricing and payment models, in an effort to balance sustainability of health innovation with sustainability of healthcare systems. The findings of this review will be made freely accessible to the scientific community that includes governmental bodies, payers, HTA agencies, and policy makers, through an online tool, which will be termed the Pay-for-Innovation Observatory. Other databases already exist, such as the Performance Based Risk Sharing Database, proprietary of the University of Washington [ 32 ], or the repository on medical devices produced as part of the EU’s Horizon 2020 research project Pushing the Boundaries of Cost and Outcome Analysis of Medical Technologies (COMED). Our Pay-for-Innovation Observatory will build on these prior examples, expanding on the dimensions investigated (e.g., classes of health technologies covered) and making the database openly accessible.

Previously published taxonomies have classified pricing or payment/reimbursement schemes with a siloed approach, typically focusing separately on clusters of schemes, such as performance-based risk sharing agreements RSAs only [ 17 ], MEAs only [ 18 , 19 ], or coverage with evidence development (CED) schemes only. Furthermore, prior taxonomies have been predominantly developed using the lens of the public authorities or payers [ 15 , 16 , 23 ], as these mostly categorize the available coverage options as opposed to the strategies available to manufacturers to price health technologies. Lastly, these previous frameworks were published mostly in the early 2010s (i.e., the majority before 2014) and might fail at accounting for some of the innovative contracting schemes that have been designed to address the distinctive features of new health technologies, such gene therapies, that have now become available.

All in all, this work will inform on the different schemes available to promote access to potentially innovative, new or expensive health technologies in the area of medicinal products, medical devices, and drug–device combinations.

Data Availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Code Availability

Not applicable.

Zhang AD, Puthumana J, Downing NS, Shah ND, Krumholz HM, Ross JS. Assessment of clinical trials supporting US food and drug administration approval of novel therapeutic agents, 1995–2017. JAMA Netw Open. 2020;3(4): e203284. https://doi.org/10.1001/jamanetworkopen.2020.3284 .

Article   PubMed   PubMed Central   Google Scholar  

Schuster Bruce C, Brhlikova P, Heath J, McGettigan P. The use of validated and nonvalidated surrogate endpoints in two European Medicines Agency expedited approval pathways: A cross-sectional study of products authorised 2011–2018. PLOS Med. 2019;16(9): e1002873. https://doi.org/10.1371/journal.pmed.1002873 .

Vokinger KN, Kesselheim AS, Glaus CEG, Hwang TJ. Therapeutic value of drugs granted accelerated approval or conditional marketing authorization in the US and Europe From 2007 to 2021. JAMA Health Forum. 2022;3(8): e222685. https://doi.org/10.1001/jamahealthforum.2022.2685 .

Gyawali B, Kesselheim AS, Ross JS. the accelerated approval program for cancer drugs — finding the right balance. N Engl J Med. 2023;389(11):968–71. https://doi.org/10.1056/NEJMp2306872 .

Article   PubMed   Google Scholar  

Pease AM, Krumholz HM, Downing NS, Aminawung JA, Shah ND, Ross JS. Postapproval studies of drugs initially approved by the FDA on the basis of limited evidence: systematic review. BMJ. 2017;357: j1680. https://doi.org/10.1136/bmj.j1680 .

Drummond M, Griffin A, Tarricone R. Economic evaluation for devices and drugs—same or different? Value Health. 2009;12(4):402–4. https://doi.org/10.1111/j.1524-4733.2008.00476_1.x .

Tarricone R, et al. Lifecycle evidence requirements for high-risk implantable medical devices: a European perspective. Expert Rev Med Devices. 2020;17(10):993–1006. https://doi.org/10.1080/17434440.2020.1825074 .

Article   CAS   PubMed   Google Scholar  

Horrow C, Kesselheim AS. Confronting high costs and clinical uncertainty: innovative payment models for gene therapies: study examines costs, clinical uncertainties, and payment models for gene therapies. Health Aff (Millwood). 2023;42(11):1532–40. https://doi.org/10.1377/hlthaff.2023.00527 .

Advanced therapy medicinal products: Overview. European Medicines Agency. [Online]. https://www.ema.europa.eu/en/human-regulatory/overview/advanced-therapy-medicinal-products-overview . Accessed 10 Apr 2024.

Drummond M, et al. How are health technology assessment bodies responding to the assessment challenges posed by cell and gene therapy? BMC Health Serv Res. 2023;23(1):484. https://doi.org/10.1186/s12913-023-09494-5 .

European Commission. Directorate General for Health and Food Safety. and Expert Panel on effective ways of investing in Health (EXPH)., Opinion on innovative payment models for high-cost innovative-medicines. LU: Publications Office, 2018. Accessed: Jun. 08, 2023. [Online]. https://doi.org/10.2875/049700 .

Towse A, Garrison LP. Canʼt get no satisfaction? Will pay for performance help?: toward an economic framework for understanding performance-based risk-sharing agreements for innovative medical Products. Pharmacoeconomics. 2010;28(2):93–102. https://doi.org/10.2165/11314080-000000000-00000 .

Performance-based managed entry agreements for new medicines in OECD countries and EU member states: How they work and possible improvements going forward. OECD Health Working Papers 115. 2019. https://doi.org/10.1787/6e5e4c0f-en .

‘INNOVATIVE CONTRACTING FOR ATMPS IN EUROPE: Recent learnings from the manufacturer experience’. Alliance for Regenerative Medicine-Dolon, Aug. 2023. [Online]. https://dolon.com/wp-content/uploads/2023/08/Innovative-contracting-for-ATMPs-in-Europe-1.pdf?x23572 . Accessed 10 Apr 2024.

Carlson JJ, Sullivan SD, Garrison LP, Neumann PJ, Veenstra DL. Linking payment to health outcomes: a taxonomy and examination of performance-based reimbursement schemes between healthcare payers and manufacturers. Health Policy Amst Neth. 2010;96(3):179–90. https://doi.org/10.1016/j.healthpol.2010.02.005 .

Article   Google Scholar  

Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, risk sharing, or patient access scheme? A framework for coverage decisions. Value Health. 2012;15(3):570–9. https://doi.org/10.1016/j.jval.2011.12.013 .

Garrison LP, et al. Performance-based risk-sharing arrangements—good practices for design, implementation, and evaluation: report of the ISPOR Good Practices for Performance-Based Risk-Sharing Arrangements Task Force. Value Health. 2013;16(5):703–19. https://doi.org/10.1016/j.jval.2013.04.011 .

Ferrario A, Kanavos P. Managed entry agreements for pharmaceuticals: The European experience. EMiNet, Brussels, 2013. [Online]. Available: http://eprints.lse.ac.uk/50513/1/__Libfile_repository_Content_Ferrario . Accessed 10 Apr 2024.

Ferrario A, Kanavos P. Dealing with uncertainty and high prices of new medicines: a comparative analysis of the use of managed entry agreements in Belgium, England, the Netherlands and Sweden. Soc Sci Med. 2015;124:39–47. https://doi.org/10.1016/j.socscimed.2014.11.003 .

Drummond M, Towse A. Is rate of return pricing a useful approach when value-based pricing is not appropriate? Eur J Health Econ. 2019;20(7):945–8. https://doi.org/10.1007/s10198-019-01032-7 .

Leonard C, et al. Can the UK “Netflix” Payment Model Boost the Antibacterial Pipeline? Appl Health Econ Health Policy. 2023;21(3):365–72. https://doi.org/10.1007/s40258-022-00786-1 .

Carlson JJ, Gries KS, Yeung K, Sullivan SD, Garrison LP. Current status and trends in performance-based risk-sharing arrangements between healthcare payers and medical product manufacturers. Appl Health Econ Health Policy. 2014;12(3):231–8. https://doi.org/10.1007/s40258-014-0093-x .

Launois R, Navarrete LF, Ethgen O, Le Moine J-G, Gatsinga R. Health economic value of an innovation: delimiting the scope and framework of future market entry agreements. J Mark Access Health Policy. 2014;2(1):24988. https://doi.org/10.3402/jmahp.v2.24988 .

Shamseer L, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349(jan021):g7647–g7647. https://doi.org/10.1136/bmj.g7647 .

PRISMA-P Group, et al. ‘Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. https://doi.org/10.1186/2046-4053-4-1 .

Article   PubMed Central   Google Scholar  

Peters MDJ, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18(10):2119–26. https://doi.org/10.11124/JBIES-20-00167 .

Tricco AC, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73. https://doi.org/10.7326/M18-0850 .

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32. https://doi.org/10.1080/1364557032000119616 .

Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–3.

PubMed   Google Scholar  

Ferdinands G, et al. Active learning for screening prioritization in systematic reviews—a simulation study. Open Sci Framework. 2020. https://doi.org/10.31219/osf.io/w6qbg .

Havrlant L, Kreinovich V. A simple probabilistic explanation of term frequency-inverse document frequency (tf-idf) heuristic (and variations motivated by this explanation). Int J Gen Syst. 2017;46(1):27–36. https://doi.org/10.1080/03081079.2017.1291635 .

Performance Based Risk Sharing Database. University of Washington. [Online]. Available: https://sop.washington.edu/department-of-pharmacy/research/performance-based-risk-sharing-database/ . Accessed 8 Jan 2024

Download references

Author information

Authors and affiliations.

Center for Research on Health and Social Care Management, SDA Bocconi School of Management, Via Sarfatti, 10, 20136, Milan, MI, Italy

Vittoria Ardito, Ludovico Cavallaro, Michael Drummond & Oriana Ciani

Centre for Health Economics, University of York, York, UK

Michael Drummond

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Vittoria Ardito .

Ethics declarations

This project has received funding from the European Union’s Horizon Europe research and innovation program under Grant agreement number 101095593.

Conflict of Interest

V.A., L.C., M.D., and O.C. declare that they have no conflict of interest.

Authors’ Contribution

V.A. and O.C. contributed to the study conception. V.A. drafted the first version of the manuscript. V.A. and L.C. designed the preliminary search strategy and performed the initial screening of the records. All authors (V.A., L.C., M.D., and O.C.) contributed to developing the data extraction form and revised and approved the final version of this manuscript.

Ethics Approval

Consent to participate, consent for publication (from patients/participants), rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/ .

Reprints and permissions

About this article

Ardito, V., Cavallaro, L., Drummond, M. et al. Mapping Payment and Pricing Schemes for Health Innovation: Protocol of a Scoping Literature Review. PharmacoEconomics Open (2024). https://doi.org/10.1007/s41669-024-00496-5

Download citation

Accepted : 06 May 2024

Published : 21 May 2024

DOI : https://doi.org/10.1007/s41669-024-00496-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us
• Track your research