policy making research papers

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

Policy capacities and effective policy design: a review

Ishani mukherjee.

1 School of Social Sciences, Singapore Management University, 90 Stamford Road, Level 4, Singapore, 178903 Singapore

M. Kerem Coban

2 GLODEM, Koc University, Rumelifeneri Yolu, 34450 Istanbul, Turkey

Azad Singh Bali

3 The School of Politics & International Relations; The Crawford School of Public Policy, The Australian National University, University Avenue, Acton, ACT 2600 Australia

Associated Data

Upon request.

Not applicable.

Effectiveness has been understood at three levels of analysis in the scholarly study of policy design. The first is at the systemic level indicating what entails effective formulation environments or spaces making them conducive to successful design. The second reflects more program level concerns, surrounding how policy tool portfolios or mixes can be effectively constructed to address complex policy objectives. The third is a more specific instrument level, focusing on what accounts for and constitutes the effectiveness of particular types of policy tools. Undergirding these three levels of analysis are comparative research concerns that concentrate on the capacities of government and political actors to devise and implement effective designs. This paper presents a systematic review of a largely scattered yet quickly burgeoning body of knowledge in the policy sciences, which broadly asks what capacities engender effectiveness at the multiple levels of policy design? The findings bring to light lessons about design effectiveness at the level of formulation spaces, policy mixes and policy programs. Further, this review points to a future research agenda for design studies that is sensitive to the relative orders of policy capacity, temporality and complementarities between the various dimensions of policy capacity.

Introduction: capacity considerations for effective policy design

The heart of policy design resides in the act of devising policy alternatives that meet stated government goals. While it is understood that not all policies can be carefully crafted, the policy sciences have been motivated by questions about why some policy alternatives are often developed well, while others are less so. Why do some policy choices, once formulated, effectively go forth through subsequent policymaking processes while others do not? How do some policies arise from meticulously crafted modes of formulation while others are shaped by partisan processes such as electoral or legislative bargaining (Howlett, 2011 ). Understanding factors that enable how deliberate designing of policy occurs and how superior designs can be achieved in complex issue-areas is central to the research agenda of the modern policy sciences (Howlett, 2014a , 2014b ; Howlett et al., 2017 ). The critical need to acknowledge, engage with and fully understand the capabilities underlying this exercise of good design, is also constantly escalating, especially in the face of widespread public crises.

Over the last few decades, a growing curiosity about the feasibility of formulation processes and the context within which policy choices unfold, has allowed policy scholars to gain a comparative perspective on policy design realities. Policy design is now generally defined as the purposive action of linking policy instruments with distinctly stated policy goals (Bobrow, 2006 ; Linder & Peters, 1984 ; Majone, 1975 ; May, 2003 ), stemming from the systematic endeavor to analyze how targets react or change their behaviors in response to instruments of governance. Effective design subsequently involves applying the knowledge gained about instrument-target relationships, to the creation of policies that can then predictably lead to desired policy outcomes (Bobrow & Dryzek, 1987 ; Gilabert & Lawford-Smith, 2012 ; Peters, 2018 ; Sidney, 2007 ; Weaver, 2009a , 2009b ). These activities are prefaced on the assumption that feasible polices can be realistically generated through effective design processes only when, firstly, contradictions internal to the substantive content of policy are resolved or minimized, and secondly, when the necessary capacities and capabilities to enact design procedures are in place (Bali et al., 2019 ; Mukherjee & Bali, 2019 ).

The recent scholarship in the policy sciences recognizes the first of these two emphases. For instance, studies anchored in the new design orientation explicitly focus on policy tools, how they are sequenced and assembled in mixes, how these mixes are calibrated, and their relative efficacies in meeting policy goals (del Río & Howlett, 2013 ; Howlett & Lejano, 2013 ). However, these studies have to a lesser degree raised issues about the capacity that is essential for effective policy design. In other words, experience from a variety of sectors and jurisdictions have alluded to what ‘effectiveness’ or ‘best practices’ imply for the activity of policy design, but lesser so about what capacities enable effectiveness.

Discussion of this latter topic is a largely scattered body of knowledge in the theoretical and empirical contribution of policy studies scholarship. For instance, the contemporary frameworks and theories of the policy process do not explicitly operationalize capacity as an independent variable in explaining policy outcomes (see for example Howlett et al., 2020a , 2020b for a recent review of the theories of the policy process). Here, we do not claim a ceteris paribus condition in which policy capacity is the only explanatory factor determining policy design effectiveness. While recognizing that many different determinants of policy design effectiveness exist, the article surveys the extant literature to specifically highlight the state of the knowledge on policy capacity requisites of policy design effectiveness. In doing so, the article brings to light the capacity ‘gap’ that exists in the policy design literature and draws lessons on not only what ‘effectiveness’ means at multiple levels of design but what is known to date about the capacities necessary for its enabling. The central question thus motivating this review asks what types of capacity are needed for effective policy design ? And to this aim, the article presents findings of a critical review synthesizing the existing scholarship on policy capacity and design in the policy sciences.

The article follows with an examination of the conceptual correspondence between the literatures on policy design effectiveness and policy capacity. The methodology informing this review is outlined next. In the fifth section that forms the core of our review, we consolidate the findings of our research on effective policy design spaces and instrument mixes and critically analyze these in the context of four emerging yet under-theorized themes from the scholarship on policy capacity, namely (1) the potential hierarchies in types of policy capacity, (2) the temporal dynamics within policy capacity, (3) task and agency-specific capabilities, and (4) complementarities among different types of capacities. We conclude by discussing avenues to advance a research agenda on effective design spaces and policy instrument mixes, which rigorously engages with these four themes of policy capacity.

Through this process, the paper makes two novel contributions focusing on the intersection of the policy design and policy capacity literatures. Firstly, it synthesizes the growing body of research in the policy sciences on effective policy design in terms of how particularly it discusses the necessary policy capacities that enable it. And secondly, by anchoring the review in the policy design orientation, the paper is able to identify four themes arising from the scholarly work on policy capacity that have yet to receive requisite theoretical and empirical scrutiny in the policy sciences. In doing so, we respond to repeated calls in the literature on the need to advance the scholarship and develop meaningful research questions on policy design effectiveness and the capacities that it necessitates. (Howlett & Lejano, 2013 ; Howlett et al., 2015a , 2015b ).

Understanding policy effectiveness

Policy effectiveness can be understood at three nested levels (Peters et al., 2018 ). The first relates to creating a conducive design space or an environment for policy formulation, which allows for effective policy design to occur (Howlett & Mukherjee, 2018a , 9). The second refers to developing effective policy mixes that are capable of addressing problems, and the third involves effectively designing and deploying individual policy instruments .

Effectiveness in design spaces

The essential idea is that the nature of the overall policy design space can significantly influence how effectively intended design activities occur and thus upon the likely resulting effectiveness of policy designs that emerge from them. These spaces reflect existing policy styles within a sector, are shaped by political conditions, reflect policy legacies (Howlett & Tosun, 2021 ), and therefore constrain (or enable) options available for designers. Developing policymaking spaces that are amenable to design activities involves a constant and concurrent stock-taking exercise of potential public capacities that might be pertinent in any problem-solving situation (Anderson, 1975 ). However, having an intention to be formal and analytical in designing and evaluating policy alternatives is not enough in itself to promote a design-centered process, since this also depends on the government’s ability to undertake such an analysis and to alter the status quo (Howlett & Mukherjee, 2018b ). Capacity challenges plaguing a design situation can lead to the generation of alternatives which are tenuously ‘patched’ together rather than deliberately packaged to uphold coherence and consistency (Howlett & Rayner, 2013 ).

Effectiveness in instrument mixes

While considerations for the design environment’s bearing on effective formulation have occupied the research agenda of policy tool studies in recent years, the new design orientation has contributed to a discourse on how to effectively incorporate policy mixes of policy goals and means (Briassoulis, 2005 ; Doremus, 2003 ; Gunningham et al., 1998 ; Hood, 2007 ; Howlett, 2011 ; Jordan et al., 2011 , 2012 ; Peters et al., 2005 , 2018 ; Yi & Feiock, 2012 ).

Selecting and deploying multiple instruments in the context of dedicated policy mixes ‘are all about constrained efforts to match goals and expectations both within and across categories of policy elements’ (Howlett, 2009a , 74). Achieving effectiveness with respect to deploying such mixes or policy portfolios relies on ensuring that mechanisms, calibrations, objectives and settings display ‘coherence’, ‘consistency’ and ‘congruence’ with each other (Howlett & Rayner, 2007 ). Scholars steeped in the new design orientation who are concerned with effectiveness have cautioned about how some policy mixes that are not designed in a planned fashion, can be plagued by internal inconsistencies, whereas others can be more successful in creating an internally supportive combination (del Río, 2010 ; Grabosky, 1994 ; Gunningham et al., 1998 ; Howlett & Rayner, 2007 ). This depends on how well they are able to adapt and support changing policy circumstances, as Thelen ( 2004 ) noted how the organization of macro-institutions has usually not resulted through calculated planning but rather has emerged out of processes of incremental adjustments such as ‘layering’ or ‘drift’ (Sewerin et al., 2020 ).

Effectiveness at the instrument level

While most of the research in the contemporary policy sciences have focused on issues around design spaces and instrument mixes, these has been limited, if any, comparative research on the efficacy of individual instruments and how they are calibrated (Capano and Howlett, 2020 ). At the most granular level, this third level of effectiveness focusses on the efficacy of individual policy tools and how these individual instruments are calibrated. Within this, we also need to differentiate between substantive instruments such as taxes, licenses, and subsidies; and the more indirect procedural instruments (such as competition, network structure, and royal commissions) which include administrative processes for selecting and deploying substantive tools (Capano and Howlett, 2020 ; Howlett, 2000 ).

There are at least three factors that condition the effectiveness of individual instruments and how they are calibrated. First, the extent to which substantive policy tools is supported by their procedural counterparts. Second, the extent to which critical institutional pre-requisites that condition the performance of instruments are present in policy mixes. Third, the extent of how far particular components of instruments or their calibrations can be easily adjusted in the short run and long run. This refers to changes in the settings of instruments such as adjusting tax rates or contribution rates for a pension fund. In some cases, there are sufficient ‘degrees of freedom’ to make these changes, or for them to be auto adjusting such as cost of living stabilizers, but in many cases calibrating instruments are difficult thereby undermining the effectiveness of an instrument.

Policy capacity: a brief review

Policy capacity, defined as a set of skills, competencies, and resources across government agencies to design and pursue policy goals (Rotberg, 2014 ; Howlett, 2015 ; Tiernan & Wanna, 2006 ; Wu et al., 2010 , 2015 ), has been a central research theme in public policy in recent years (Howlett and Ramesh, 2015 ; Newman et al., 2017 ; Karo & Kattel, 2018 ; Daugbjerg et al., 2018 ; Bali & Ramesh, 2019 ). In a notable first contribution, Wu et al. ( 2015 ) offer a framework to conceptualize policy capacity at multiple levels of governance. They argue that capacity can be understood as skills and competencies existing across government agencies at three nested levels: the individual (e.g., policymakers, decision-makers), the organization (e.g., an agency or a program), and at the systemic level (e.g., the whole of government or the macro level institutional, structural contexts) (Table ​ (Table1). 1 ).

Dimensions and levels of policy capacity.

Source : Adapted from Wu et al. ( 2015 ) and Howlett and Ramesh ( 2015 )

At the level of individuals occupied with policy formulation, those striving for effective design require technical know-how to conduct practical policy analysis and disseminate knowledge, while leadership and negotiation abilities are additionally relevant for those in managerial positions. Analysts also need political savvy and acumen for incorporating and accounting for various stakeholder interests and assessing political feasibility. At the level of government organizations , information mobilization capabilities to enable timely and relevant policy analysis, administrative capital for ongoing coordination between policymaking agencies, and political backing all fundamentally build overall policy capacity. At the system level, effective policy design requires institutions for knowledge creation and utilization, alongside mechanisms to coordinate across different levels of government, and overall trust and political legitimacy (Mukherjee & Howlett, 2016 ).

Howlett and Ramesh ( 2015 ) extend Wu et al.’s ( 2015 ) work on capacity drawing on the metaphor of an ‘Achilles’ Heel.’ That is, how certain types of capacities can become critical to the sustaining policy efforts and outcomes in specific modes of governance, and how any weaknesses in these ‘critical’ capacities can undermine policy efforts (Menaheim and Stein 2013 ).

Technical knowledge, for example, is a critical capacity required for the sustainable functioning of policy systems based on market-based governance. Analytical skills at the level of individual analysts and policy workers are key, and the ‘policy analytical capacity’ (Rayner et al., 2013 ; Wellstead et al., 2011 ) of government needs to be especially high to deal with complex quantitative economic and financial issues involved in regulating and steering the sector and preventing crises (Bakır & Çoban, 2019 ; Rayner et al., 2013 ; Woo et al., 2016 ). Similarly, undertaking policy design within legal systems of governance relying heavily on high levels of managerial capacities that can deter against diminishing returns of compliance or mounting non-compliance with government directives (Coban, 2020a ; May, 2005 ). Capacities at the systemic level can be especially critical in this case as governments find it difficult to enact traditional command-and-control instruments in the absence of overall public trust.

The appeal of Wu et al.’s ( 2015 ) framework lies in its inherent simplicity. Each of the nine capabilities lend themselves to, in principle, being empirically operationalized and allows analysts to assess strengths and weaknesses of governments across different types of capabilities (e.g., Bajpai and Chong, 2019 ; Saguin et al., 2018 ). Yet such simplicity also generates concerns.

First, the contribution by Wu et al. ( 2015 ) does not lend itself to drawing causal inference or developing a theory of policy capacity. Moreover, as our review demonstrates below, the mechanisms that connect indicators with specific types of capacities are not explicitly mentioned. Secondly, the current literature seems to adopt a benevolent approach to incumbents relying on or mobilizing policy capacity. 1 That is, policy capacity could also facilitate the ‘dark side’ of policymaking (Howlett, 2020 ), by advancing policymakers’ self-interested, political and/or economic ‘rent-seeking’ objectives (see Chindarkar et al., 2017 ; Howlett and Mukherjee, 2016 ). Furthermore, it can be instrumental for developing ‘placebo policies’ as ‘agenda management safety valves’ (McConnell, 2020 , 965) or for ‘hidden agendas’ (McConnell, 2018 ) to further political goals rather than addressing the core of policy problems. These represent unchartered areas, especially if we consider the challenges generated by the rise of populism and autocratization around the world (Kelemen, 2017 ; Maerz, 2020 ; Norris & Inglehart, 2019 ).

This review relies on building and scrutinizing a database of peer-reviewed journal articles that are located at the intersection of policy capacity, policy design, and effectiveness. A keyword search based on these themes was conducted on Scopus, and Thomson Reuters’ Web of Science (WoS). Scopus and WoS are two major repositories of scientific knowledge published in various forms: conference proceedings, edited book chapters, peer-reviewed journal articles. The search protocol was conducted similarly on both databases to cross-check for any duplicate journal articles, and avoided selection bias that can result from extracting data from a single database. The search covered three collections of WoS citation indexes: Social Sciences Citation Index (SSCI), Emerging Sources Citation Index (ESCI), Arts and Humanities Citation Index (A&HCI). We explicitly included ESCI and A&HCI along with SSCI given our concerns for inclusivity.

The data collection and sample selection process had four steps. The first involved searching for, ‘policy design’, ‘capacity’, ‘effectiveness’, as keywords for the topic of an article. In this focused search, we omitted a set of alternative keywords such as ‘capability’, which are mostly used in public management scholarship. More importantly, the focused search as conducted through these keywords allowed us to capture a range of terms, such as ‘governance capacity’ and ‘administrative capacity’, in which capacity has been used in the context of policy design and/or design effectiveness. As such, it should be noted that articles that incorporated such varieties of capacity, but did not directly discuss policy design were excluded from the final database. In this light, we are aware that the search focused on a designated subset in the existing policy design literature. However, this scope allowed us to fully capture the dispersed attempts made so far to deliberately link policy capacity and design effectiveness and address our express interest in showcasing the current state of the literature that is located at the intersection of policy capacity, policy design, and design effectiveness. Additionally, the search was designed to be as inclusive as possible given the time period, disciplines, and multiple databases that it incorporates.

While acknowledging the limitations of the search logic described herein, we maintain that additional keywords would result in extra layers that dilute the task of specifically exploring the policy capacity requisites for policy design effectiveness. We also note that detecting journal articles on WoS and Scopus required us to run the search several times with various combinations of these keywords. This is because research that is positioned at the intersection of policy capacity, policy design, and effectiveness is in its adolescence. We therefore combined the results of multiple searches while removing duplicate entries. Our search covered the period between 1900 and May 17, 2020, the date we ran the search on WoS and Scopus. This time period allowed for construction of an inclusive database. This search yielded a sample of 9382 sources. The second step involved filtering our initial search for journal articles that are published in English. 2 The result of this process reduced the sample to 7441 articles. In the third step, we further refined our search by filtering the articles according to various relevant (inter)disciplinary areas: ‘political science’, ‘public administration’, ‘economics’, ‘management’, ‘international relations’, ‘sociology’, ‘social sciences interdisciplinary.’ In so doing, we included articles that are not only published in political science and public administration but also in other main social science disciplines and those that were classified in the interdisciplinary social sciences category. This choice was mainly driven by inclusivity concerns and an expectation of capturing articles that may empirically or conceptually refer to policy design, policy capacity, and/or effectiveness. The result of the second stage to limit our search to relevant fields yielded 1431 journal articles.

Following the above-mentioned steps, we read titles, abstracts, and full texts to further refine the most relevant articles. Articles that had the main keywords in the topic, but were not directly related to our research questions were omitted based on a reading of their introductory sections and research questions. We omitted articles that used different forms of capacity without an explicit interest in operationalizing capacity for design effectiveness. We also omitted articles that attempted to measure or evaluate effectiveness of an instrument or program. In this regard, as our interest in this article is to make sense of what capacity for ‘effectiveness’ means at multiple levels of design, our exclusion criteria meant that we eliminated articles which presented only nominal links between policy design and policy capacity. Consequently, the final sample included 146 articles. As for coding, the sample included articles that discuss policy design as well as effectiveness. Therefore, coding had to sort according to levels of policy design and dimensions of policy capacity. This process involved two tracks. First, we coded articles to capture dimensions of policy capacity according to parameters suggested by Wu et al. ( 2018 , 6–14). Second, reading the articles served to code an article whether it did examined design space, discussing design effectiveness of a policy instrument, policy mixes reading the articles led us to code articles whether it was about design space, discussing design effectiveness of a policy instrument, policy mixes/programs, or combinations levels of policy design.

Table ​ Table2 2 and Fig.  1 summarize the results of the coding process. Articles on design space, policy mixes and programs have the highest share among those referring to level of policy design. A main observation at the outset is that there is a significant gap in the literature on studies discussing policy design and capacity at the level of individual instruments. The review included explicitly those scholarly contributions that engage with capacity considerations. Undoubtedly, the field of environmental policy (and for that matter social policy and financial policy) is replete with the discussion of singular instrument types such as taxes, social security schemes, emissions trading schemes, among others. But this review could not identify articles that expressly deal with the question of capacity and what is needed on the part of policy designers to formulate these instruments, which is a significant void that needs to be filled in future studies. Even the studies that distill the state of knowledge on effective program design, rarely discuss individual constituent policy tools.

Levels of policy design and dimensions of policy capacity

Dimensions of policy capacity and levels of policy design

On the dimensions of policy capacity, articles address analytical and political capacity more so than operational, while there is a more equal distribution of articles referring to individual, organizational, or systemic scales. In addition, our observations point to a limited number of studies that look at both organizational and individual policy capacity, as well as both political and operational policy capacities. Finally, we note that only a few studies attempt to relate policy capacity with effective design space for global public policies, instruments, and mixes/programs (Bernstein & Cashore, 2012 ; Cashore et al., 2019 ; Dare, 2018 ; Dorsch & Flachsland, 2017 ; Jordan & Huitema, 2014 ; Stone & Ladi, 2015 ; Vince & Nursey-Bray, 2016 ).

Policy capacity requisites for effective policy design: emerging trends and existing gaps

In this section, we discuss the main findings on the link between policy design effectiveness and policy capacity as revealed through the review of the literature. While these findings are discussed at the level of effective policy design spaces and effective instrument mixes, we critically examine them through the perspective of four overarching emphases that that are developing within the scholarship on policy capacity and policy design (Bali & Ramesh, 2018 , 340–341; Howlett and Ramesh, 2015 ; Capano & Howlett, 2020 ; Howlett et al., 2015a , 2015b ). These are, namely:

• Hierarchies or ‘orders’ among specific types of capacities, which indicate what kinds of capabilities are more pre-requisite and foundational to others that are more ‘second-tier’ and aspirational.
• Temporality of policy capacity endowments, or the time needed for policy capacity investments to achieve actual effectiveness outcomes.
• The distinction between task-specific and agency-specific policy capacities and how to reconcile between them; and
• The synergies and complementarities between different policy capacities.

Capacities for effective policy design spaces

Developing effective design spaces is fundamentally about ensuring that policy tools are anticipated to fit or cohere with broad governance arrangements, while delivering a means to address certain policy goals. It is argued, for example, that several variables are critical for effectiveness within collaborative modes of governance, including reconciling with ‘prior history of conflict or cooperation, the incentives for stakeholders to participate, power and resource imbalances, leadership and institutional design’ (Ansell & Gash, 2008 , 543). Similarly, the absence of clear property rights and mechanisms to enforce contracts stymie the effectiveness of hybrid governance arrangements to design suitable public–private partnerships (PPPs) in service delivery (Virani, 2019 ).

An enabling design space that is able to support the design of its constituent policy instruments signifies an environment that is marked by high analytical, operational and political capacity (Capano, 2018 ; Chindarkar et al., 2017 ). Determining exactly what capacities are required in order to develop the political and administrative spaces needed to carry out complex policy design processes is currently a subject of much interest in the field (Considine, 2012 ). In order to address these issues, it is recognized that policy designers need to be cognizant about the internal mechanisms of their polity and constituent policy sectors which can boost or undermine their ability to think systematically about policy and develop effective policies (Braathen, 2007 ; Braathen & Croci, 2005 ; Grant, 2010 ; Skodvin et al., 2010 ).

In this vein, organizations and individual policymakers need political support from the policy design spaces or environments that they occupy. For this, they derive legitimacy and authority from system-level political capacity, which subsequently creates a favorable milieu for the application of individual and organizational political capacities during the design process (Woo et al., 2015 ; Xiarchogiannopoulou, 2015 ). Political support to policymakers and interactions between policymakers and politicians have been argued as being non-substitutable when it comes to overcoming ambiguous goals and promoting managerial effectiveness, by supplying organizations with a clear understanding of their overall mandate (Meckling & Nahm, 2018 ; Stazyk & Goerdel, 2011 ).

At more individual and organizational levels, political capacity is essential for maneuvering effectively within the constraints of the design space (Hartley et al., 2015 ) and is embodied in the levels of trust, especially political trust and legitimacy within the public sector. Individual and organizational political capacity is also necessary to garner strategic stakeholder support that is vital both before and during the design process, as well as in subsequent stages of policy implementation (Bali & Ramesh, 2019 ). For example, in the case of macro-prudential policy, a suitable financial policy mix is possible in an enabling design space that is characterized by capable, analytically skilled individual central bankers that have coalition-building skills, a government committed to evidence-informed policy, and presence of inter-organizational collaboration mechanisms at system level (Bakır & Çoban, 2019 ).

This example also highlights the importance of ‘legitimation capacity’ in effective design environments (Woo et al., 2015 ; see also Pal & Clark, 2015 ). Policymakers and organizations that are highly regarded by key societal actors and receive sustained political support are able design effective policies with more accountability (Busuioc & Lodge, 2016 ; Rimkuté, 2018 ). For example, as is visible in the case of health and safety regulation in the UK, the regulatory agency’s outreach and engagement with policy targets increases its political acumen by helping to overcoming citizens’ biases, and furthering its legitimacy by shoring up societal support for future policy design (Dunlop, 2015 ). This case also underscores the dilemma that may exist between expertise-led, technocratic, and less accountable design on the one hand; and participatory, more accountable design processes on the other, and the relative effectiveness of either situation (Montpetit, 2008 ). Yet, overall high levels of trust and political support at the system level are shown in most cases to allow the design process to be endowed with necessary information and access to critical resources at the outset (Chindarkar et al., 2017 ; Hartley et al., 2015 ).

An example of this latter context is the rise of ‘big data’ analytics that has also necessitated a parallel emphasis on big data readiness at all three levels of capacity (Clarke & Craft, 2017 ; Giest, 2017 ; Giest & Mukherjee, 2018 ; Golan et al., 2017 ). For example, policy responses to the Covid-19 pandemics in countries like Singapore have included combining mobile-phone-tower data and machine learning to develop social graphs that track propinquity to improve contact-tracing (The Economist, 2020 ; see also Woo, 2020 on the Singapore case). Big data has also been used for network analysis in policy formulation (Giest, 2017 ). But, the availability of data, network analysis and modeling necessitate complex skills such as making use of software, models to produce insights that inform policy design. Moreover, related studies have repeatedly underlined that policymakers should take into consideration behavioral dimensions of policies, which becomes more likely when organizational infrastructure allows for the participatory collection as well as engagement with behavioral data and analysis (Leong & Howlett, 2020 ; Mukherjee & Mukherjee, 2018 ).

Hierarchies within types of capacities

Studies in this review of effective design spaces implicitly operationalize specific types of capacities as a spectrum of independent variables and argue that they shape policy outcomes. While this advances our understanding of how capacity is connected with notions of effectiveness, the causal mechanisms that undergird such links have not always been made clear. This can be explained to some extent by the tendency in the literature to operationalize capacity in a straightforward, often univariate manner, while ignoring possible orders or hierarchies among specific types of capacities. In other words, policy capacity can be multi-dimensional with notable interaction between foundational, first-order and more aspirational or ‘second-order’ capacities. Lodhi ( 2018 ) and Hartley and Zhang ( 2018 ), for instance, suggest a comprehensive measurement of policy capacity. Such efforts can then allow for multiple orders of capacities to be observed while and better locate the interactions between them. A focus on how policy capacities at one level can enable, prevail over those, or constrain capacities at the other two levels are neglected factors when theorizing the link between policy capacity and policy design effectiveness.

For example, if system-level policy capacity is more crucial as it constitutes the environment in which an organization or an individual policymaker operates, can it be postulated that without the acquisition of system-level capacities, even high individual or organizational policy capacity might not be sufficient for effective policy design? More research along this vein is warranted to advance our understanding about any hierarchy or orders of policy capacity and the role they play in developing effective design spaces.

Along the same lines, most studies in this review focus on operationalizing a specific type of capacity rather than considering how combinations or interactions between different types of capacities shape policy outcomes. For instance, in a context wherein system-level policy capacities are high but individual policy capacities cannot uphold organizational capacities, one may observe sub-optimal design or even non-design. Such a case could indicate that while we may consider the presence of system-level policy capacity to be detrimental for on-the-ground mobilization of organizational and/or individual policy capacity, the reverse dynamic may also be important for effective policy design.

Further, while most studies in the review have considered political capacity to play a more critical role than operational and analytical capacities, they have stopped short of developing hypothesis or propositions to attribute plausible reasons for its significance. This, in turn, stagnates any advancement in how specific types of capacities can explain and beget design effectiveness.

Temporal dynamics of capacity

There is a gap in our understanding on the temporal dynamics and change within the policy design literature (see, e.g., Capano & Howlett, 2020 ; Bali & Ramesh, 2018 ), and this lacuna is also evident in this review of necessary capacities for effect design. Temporality in the context of capacities for effective design explores changes in specific types of capacity endowments over time, to their sustained or ultimate impacts on policy outcomes. It also includes a consideration of how investments in capacity building have a latent gestation period before which they begin to affect outcomes. None of the studies in this review explicitly dwelled on the temporal dimensions of capacities, echoing the popular refrain on the largely atheoretic discussion on policy tools and capacity (Howlett & Ramesh, 2015 ; Howlett et al., 2015a , 2015b ).

Temporality in the context of effective policy design can be conceptualized in two ways. The first is to consider the impact and scope of changes in capacity on effectiveness at different stages of design process. For example, what are the causal mechanisms by which changes in capacities contribute to changes in policy outcomes? That is, do interventions at time t 0 affect outcomes by time t n . Is the lag between t n –t 0 standardized across different types of capacities? Such lines of enquiry can inform about how individual, organizational, or system-level policy capacities change over time and result in fluctuations in the effectiveness of policy designs. For instance, the National Sample Survey Organizations of India in the 1950s was recognized globally as a center for excellence and pioneering statistical sampling techniques and methodologies, but in recent years has become mired in controversy on the quality of its statistical estimates (Banerjee et al., 2017 ).

Secondly, a discussion on temporality also implicates concerns about robustness and resilience of policy design. Robustness over time can enable policymakers, organizations or a system to endure shocks, policy surprises, and turbulence, while allowing them flexibility (Ansell et al., 2016 ; Capano & Woo, 2017 ; Howlett et al., 2018 ; Mergel et al., 2021 ). Endurance could be achieved with adaptability to structural, institutional and actor-level changes and/or evolution of existing policy capacities over time (e.g., Alaerts, 2020 ; Capano & Pavan, 2019 ; Van Der Steen et al., 2018 ). And subsequent adaptability could arise on improvements in complementarities among different types and levels of critical capacity requisites. These are particularly relevant to anticipatory policy design (Bali et al., 2019 ; Huitema et al., 2018 ; Kimbell & Vesnić-Alujević, 2020 ), especially in cases of high contextual uncertainty, as is exemplified by numerous examples of climate change impacts on agriculture or water policy domains (Nair & Howlett, 2017 ). While such a conceptualization seems plausible, the existing literature lacks a systematic understanding of what types of capacities enable design spaces to endure substantial changes in the structural and institutional contexts of policies as, for example, the Covid-19 crisis has already demonstrated (Walter, 2020 ; Weible, 2020 ).

These considerations also call for a discussion on the temporal nature of acquiring or engendering policy capacities and which of these are necessary earlier on in the design process. For example, effective policy design could be the outcome of initial improvements in individual and organizational capacities, which may later require the build-up and/or mobilization of system-level capacities. These are propositions that need to be examined to advance our understanding of whether or not individuals, organizations, or systems need to build particular capacities first for effective policy design to subsequently unfold.

Capacities for effective instrument mixes and programs

The growing intractability of contemporary challenges that governments face in areas such as health and urban planning among others has necessitated the use of multiple policy tools to be carefully and deliberately assembled in policy mixes or portfolios (Howlett & Lejano, 2013 ). This has made the task of effective policy design more challenging, as designers have to match not only policy goals and aims, but also instrument mixes and governance modes (Peters & Pierre, 2015 ; Tosun & Lang, 2017 ; Wen, 2017 ). In turn, this effort towards striving for compatibility requires a spectrum of analytical capacities that enables policymakers, organizations and political systems to employ skills pertaining to the accurate articulation of operational objectives, which in turn require an accurate interpretation of context relevant information and data. These analytical skills become fundamental to the success of sector-wide programs that may otherwise suffer from a mismatch between stated objectives and the policy tool collections that are constructed as a response. In other words, and as reported in many program-level studies, the more (or less) policymakers resemble analytically capable policy designers, the more (or less) likely they are to construct an effective mix of policies through a program. For instance , Siwale and Okoye ( 2017 ) argue that microfinance program initiatives in Zambia were ineffective largely due to limitations in policymakers’ analytical capabilities.

Besides individual and organizational policy capacities, reforms buttressed on the tenets of New Public Management (NPM) marked administrative changes in the late 1990s, which embodied a large, albeit skewed, emphasis on the kinds of capabilities that are necessary for policy success. With this transformation, policy capacity to design and steer policies became truncated, as states increasingly contracted out the delivery of public services to the private sector and civil society. This has been argued to have resulted in loss of policy capacity within government, in the reform era, in the form of declining skilled human resources which affect both organizational and system-level analytical and operational capacity within the state apparatus (Bakvis, 2002 ; Baskoy et al., 2011 ; Craft & Daku, 2017 ; Donahue et al., 2000 ; Howlett, 2000 , 2009b ; Lodge, 2013 ). Put differently, with the ‘hollowing out’ of the state, the changing role of the state as the primary actor in the design process has evolved into that of a policy navigator that steers the policy process and coordinates the interactions between non-state actors and those between the state and non-state actors (Lindquist, 1992 ). Policy capacity in this sense has been often supplemented by external expertise, knowledge, know-how supplied by variegated epistemic communities, think tanks, business, international organizations, scientists, non-governmental organizations, or civil society groups among others can supply (Haas, 1992 ; Stone, 2003 ).

With the externalization of knowledge and related capacities, many studies have alluded to greater participation being fundamental for effective program design that needs to be shaped in a way that is more notably open to stakeholder input and learning from that input (Borrás, 2011 ; Hoppe, 2011 , 2018 ; Jordan & Huitema, 2014 ; Vince & Nursey-Bray, 2016 ). The water quality program in the European Union (EU) is a case in point. Brown ( 2000 ) examines the EU’s operational and analytical capacity to design effective directives when it faces scientific uncertainty in the given policy area, and most importantly fluid number and quality of staff (see also Jensen, 2018 on policy capacity requisites for effective water policy in developing countries).

This case and others demonstrate that input from international organizations and local stakeholders generally tend to increase the supranational organization’s operational and analytical capacity. Echoing the call for greater participation, Mukherjee and Mukherjee ( 2018 ) determine citizen participation to be fundamental in co-production in rural sanitation programs in India, Bangladesh, and Indonesia. Lang ( 2014 ) studies analytical capacity in PPPs in which the private sector brings its own expertise to complement goals set out by policymakers. Similarly, Bengston et al. ( 2004 ) sheds light on participation of citizens and other stakeholders in urban policy in making formulation more effective. These studies all suggest that when policymakers have a tendency to underestimate or even ignore stakeholder participation and input, the effectiveness of policy design and implemented policies can decline considerably. While a few recent studies have now begun to look at particular types of capacities that different stakeholders, especially interest groups, can contribute (see Coban, 2020b ; Daugbjerg et al., 2018 ) they still fall short of addressing the benefits or challenges they can bring specifically to policy design effectiveness, thus calling for further research in this area.

Additionally, when non-state actors participate actively in the design process, this understandably has implications for the governance capacities that are available for effective policy formulation. Studies highlighting polycentric policy design processes have emphasized policy capabilities for enabling the coordination and collaboration of multiple actors. Political capacity to manage collaboration and coordination has also been called ‘collaborative capacity’ in some public management literature, within organizations or specific programs (Ansell & Gash, 2008 ; Braun, 2008a , 2008b ; Schout & Andrew, 2008 ; Weber et al., 2007 ).

In a multi-level design situation, such as policy programs, horizontal and vertical coordination of parties similarly demand high political capacity (Peters, 2015 ). Golan et al. ( 2017 ), for example, show that lack of effective coordination between the central authorities and the local authorities in the design of rural cash transfer programs that omit a considerable share of the target population, lead to reduced effectiveness of the program’s objectives. Similarly, Wen’s ( 2017 ) study on social policy in China indicates that when the central state does not coordinate policy design with the local authorities that lack policy capacity, policy design effectiveness faces substantial challenges at all levels.

Collaboration and coordination challenges have been significant in developing countries as well as in advanced economies. Williams and McNutt ( 2013 ), focusing on policy programs for climate change adaptation in the Canadian finance sector, assert network management capacities for aligning the targets of local and federal and provincial agencies to be built into the design of the programs and well before their implementation. Additionally, Skeete ( 2017 ) examines policy instrument mixes that regulate carbon emissions emanating from diesel use in the European Union (EU). The author finds that lack of coordination between member states and EU authorities, besides leading to inherent flexibilities of the regulatory framework, also leads to fuel taxes failing to achieve original climate policy goals. Similarly, Spendzharova ( 2016 ) maintains that disconnect between EU member states and EU authorities in the design of banking structure reforms after the global financial crisis leads to a mismatch in design processes in terms of prioritizing domestic reforms vis-à-vis EU level financial reforms.

Complementarities in policy capacities

Such studies on policy instrument mixes and programs highlight the primary role of analytical capacity in developing and deploying effective instrument mixes. However, it can be insufficient if not operating alongside suitable organizational and political capacities, which ultimately determine how successfully they are implemented (Bali et al., 2019 ; Mukherjee & Bali, 2019 ). In other words, analytical capabilities are enhanced or sharpened by operational and political capacity endowments at the level of organizations. This is not surprising as policy design is ultimately a political activity and requires individual policymakers to strategically operate within a broad community of policy stakeholders and organizations (Peters, 2015 ).

For example, Mukherjee and Giest ( 2019 ) show how individual policy entrepreneurs’ capacity to form and maintain coalitions has enabled effective use of individual, organizational and system-level capacities in digital transformation in the EU. Similarly, Ramesh and Bali ( 2019 ) demonstrate how operational capabilities in Singapore’s health system were amplified by sustained political capacity and trust in government. However, these studies and others in this review do not develop generalizable propositions that can be empirically examined on the complementarities and synergies among different types of capacities in different contexts. That is, the aggregate impact of a series of specific capacity endowments is larger than their individual impacts (Wu et al., 2015 ). Similarly, do critical deficits in capacities affect outcomes? (Howlett & Ramesh, 2015 ). These theoretical gaps are particularly visible given that developing policy designs that harness synergies and complementarities among tools is a central theme in the new design orientation (Howlett et al., 2015a , Howlett et al., 2015b ).

One way to address this missing link is to canvass the recent advances around policy success in the public management literature. For instance, design effectiveness is intrinsically related to policy success, as ‘successful policy often resides in policy design and the diligent work undertaken’ (McConnell, 2017 , 17). These themes have been interrogated further in a series of studies that aim to advance what is described as ‘positive public administration’ (Compton & ‘t Hart, 2019 ; Luetjens et al., 2019 ; Douglas et al., 2019 ), which define success across four broad dimensions: if it achieves its goals (i.e., programmatic success), produces largely supported socially appropriate outcomes (i.e., process success), contributes to problem-solving capacity and enhance legitimacy (i.e., political success), and is robust (i.e., endurance) (Ibid, 5).

Connecting groups of capacities with specific dimensions of success can allow analysts to develop proposals around complementarities in capacities to be then examined empirically. For example, policy success could be less likely when operational capacity at system level in the form of coordination mechanisms both within the state and between the state and non-state actors is not established and/or mobilized. Testable claims that emerge from this debate are that if these conditions are not met, enabling political and processual success may not emerge leading to incongruent policy goals and tools. Cumulatively, these outcomes may result in failures in programmatic and endurance terms, bringing about policy (instrument) fiascos (Bovens & ‘t Hart, 2016 ). This in turn provides a richer understanding of the types of capacities required for developing and deploying effective mixes.

Task and agency-specific capacities

There is a tendency in the literature and in contemporary debates to use ‘policy capacity’ as a catchall phrase (Wu et al., 2015 ). An avenue to overcome this simplification is to engage rigorously with the ‘capacity for what’ question (Bali & Ramesh, 2018 ). That is, to identify, ex ante, and theorize task-specific and agency-specific capacities needed for routine but complex tasks in contemporary service delivery such as contracting, managing PPPs, and administering pension funds; and accomplishing these effectively during periods of extreme uncertainties and volatility such as crises (Capano et al., 2020 ; Stirling, 2010 ).

The new design orientation has set up a tall order for effectiveness in program designs whereby designs must be coordinated, coherent, reduce contingent liabilities, and avoid Type 1&2 errors, among others (Bali & Ramesh,  2017 ; 2018 ; Howlett, 2018 ). For example, while network governance may be well suited to policy design for sensitive issues such as elderly care or parental supervision (Pestoff et al., 2012 ) in other situations, civil society may not be well enough organized or endowed in order to generate beneficial network modes of governance off-the-ground and without initial regulatory support (Tunzelmann, 2010 ). Networks, for example, ‘will fail when governments encounter capability problems at the organizational level such as a lack of societal leadership, poor associational structures and weak state steering capacities which make adoption of network governance modes problematic’ (Howlett & Ramesh, 2014 , 324).

However, in our review there is limited, if any, theoretical discussion on the types of capacities needed to achieve these outcomes. That is, the range of capacities required to accomplish tasks such as contracting, commissioning, and collaboration while all under the umbrella of network governance require a variety of distinct capabilities and skillsets (O’Flynn, 2019 ). Failing to recognize these variations and invest in task-specific capabilities has played a key role in failed social policy reforms in many developing economies (Maurya & Ramesh, 2019 ; Virani, 2019 ). Along the same lines, variations in the capacities of agencies within government to pursue such tasks must be recognized (Bardhan, 2016 ).

Conclusion: avenues to advance the research agenda on capacity and design

This paper addresses a scattered body of knowledge in the policy sciences and aims to advance our understanding of the relationship between policy capacity and effective policy design. To this end, this paper presents a review of the existing literature that studies effective policy design through the lens of policy capacity, and argues that such a perspective offers an important starting point for scrutinizing the role of complementarities among organizational, individual, and system-level analytical, operational, and political capacities, within the broader policy sciences.

Clarifying the relationship between design effectiveness and policy capacities is central to advancing the research agenda of the new design orientation in the policy sciences. The theoretical union of these two bodies of literature, at its core, is about reiterating the problem-solving approach in the policy sciences. That is, it inspires building on the research questions surrounding how specific policy interventions are devised to address specific types of problems, with notions of what is fundamentally needed to enable these designs. The most well-intentioned efforts at policy design can be constrained by the capabilities of governments, and those involved in the design process (Mukherjee & Bali, 2019 ). Forwarding such a research agenda can further refine the generalizable hypotheses to investigate and improve policy deliberations regarding effective policy formulation, which already inform the policy sciences (Howlett & Lejano, 2013 ; Howlett et al., 2015a , Howlett et al., 2015b ). To this end, this review has provided several starting points for infusing policy design research with policy capacity concerns.

Our central thesis is that the growing body of research on policy design effectiveness, which is synthesized in this paper, remains largely descriptive and tends to confound rather than clarify the relationship between policy capacity and effective policy design. Our review points to several outstanding questions that need to be highlighted: Do individual, organizational, or system-level policy capacity change over time? Does effectiveness of policy designs and success of policies vary over time with changes in policy capacity of various types ‘spilling over’ and at different levels? Thirdly, do orders of policy capacity exist? And can we distinguish between hierarchies or levels of policy capacity, which have serious implications for effective policy design and thereby policy success (or failure). Specifically, this strand of reasoning can help distil those capacities that are fundamental at the start of policy design (t 0) before successive ones are developed at subsequent stages of policy design (at t 1 and expectedly later at t n ). Is there a hierarchy among levels of policy capacity? If yes, then what is the nature of that hierarchy and are there causal inferences that can be drawn between more fundamental ‘enabling’ capacities and more aspirational ‘second-tier’ capacities? And, how does such a hierarchy impact effectiveness of policy design and determine policy success (or failure)? Finally, given the lack of focus on policy capacity requisites for effective individual policy instrument design, does, and if so, how policy capacity enable effective policy instruments?

Scholarly efforts to engage with these questions can be a generative exercise, signposting new areas for theoretical exploration and empirical testing. In this concluding section, we briefly comment on two avenues to synthesize our critique, by engaging with the two respective levels of policy effectiveness that have been explored in this paper.

Effective policy design spaces: situating capacity in theories of the policy process

A central theme in the policy design literature, which pervades all studies covered in this review, is that an enabling design space provides a platform for successful policy design, as such spaces are supported by significant capacity endowments, which not only improve policy deliberation but also allows designers to best navigate changing and often volatile design contexts (Howlett & Mukherjee, 2018a , 2018b ; Howlett et al., 2018 ; Peters et al., 2018 ; Rahman et al., 2019 ). However, most of this discussion remains largely divorced from mainstream theories and frameworks of the policy process, especially those that explain policy formulation and deliberation styles of governments. If our goal is to advance our understanding of effective design spaces, and what capacities engender them, we need to locate capacity within frameworks and theories of the policy process that are focused on them. For instance, interrogating the role of capacity in incrementalism, the policy narrative framework, or the advocacy coalition framework can generate theoretically grounded propositions and empirical testing on specific mechanisms through which capacity shapes design spaces.

Effective instrument mixes and programs: developing capacity as an independent variable

Another avenue to engage with questions relating to hierarchies, complementarities and temporal dynamics of specific types of capacities raised earlier in this paper is to explicitly canvass policy capacity as a system of independent variables, and to examine its causal impact on policy outcomes. However, as Peters ( 2020 ) states, this is challenging to do especially in the context of policy design as its impact is intermediated by many exogenous factors (Peters, 2020 ). And, as has been noted earlier, the links between specific types of capacities and how they are empirically operationalized are not always clear. Nonetheless, these methodological shortcomings can be managed to some extent by through in-depth critical case studies (see Yee & Liu, 2021 ), or focusing on comparisons among most similar cases (see Yan & Saguin, 2021 ), and avoiding sweeping comparisons that are characteristic in studies of comparative public policy. Similarly, limitations around how capacity is empirically operationalized can be managed by encouraging problem or policy-specific capacity studies. For example, Bajpai and Chong ( 2019 ) extend Wu et al.’s ( 2015 ) framework to study foreign policy capacity. Similarly, Bali and Ramesh ( 2021 ) operationalize different types of capacities to sustain health reform.

Dealing with capacity as explanatory variables would allow analysts to engage with questions around hierarchies, complementarities, and temporal dynamics raised in this review. Specifically, studies can test claims that without system-level political capacity (i.e., trust in government, accountability, legitimacy), having high operational and analytical capacities at individual and/or organizational levels may have less impact on design since mobilization of these capacities might not deliver legitimate, widely supported policies at later stages of policy design. Forthcoming research could also explore whether or not system-level political capacity is indeed the most fundamental type of capacity, while the remaining are more secondary or complementary. It may also be the case that any ‘secondary’ capacities at individual or managerial levels can be observed to contribute to solidifying political capacity at system level, and research on these directional relationships between different orders of policy capacity would greatly enrich the discussion on policy process and more specifically policy design.

These questions reveal a certain degree of agitation and urgency with wanting to find critical answers about how to match publicly salient goals with means that are effective, durable, equitable and also flexible in erratic policy contexts. Joining together concerns about capacity and how to design policy answers effectively signifies a promising, and perhaps also a vital avenue of further academic enquiry, and especially so in times marked by unprecedented public crises.

Acknowledgements

The authors thank Kidjie Saguin for his support in preparation of earlier versions of this paper. Kerem gratefully acknowledges the organizational support of Sabanci University and GLODEM, Koc University, as part of the paper was written during his Part-time lectureship at Sabanci University.

Availability of data and material

Code availability, declarations.

The authors declare that they have no conflict of interest.

1 We thank an anonymous referee for raising this essential point.

2 We are aware of two major caveats. Firstly, our database only covers journal articles written in English. In addition, our database excluded monographs and edited book chapters. Studies that are written in other languages and those published as monographs and edited book chapters are likely to offer additional insights to the findings in the article, which demands further research.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Ishani Mukherjee, Email: gs.ude.ums@minahsi .

M. Kerem Coban, Email: [email protected] .

Azad Singh Bali, Email: [email protected] .

• Alaerts GJ. Adaptive policy implementation: Process and impact of Indonesia’s national irrigation reform 1999–2018. World Development. 2020; 129 :104880. [ Google Scholar ]
• Anderson JE. Public policymaking. Praeger; 1975. [ Google Scholar ]
• Ansell C, Gash A. Collaborative governance in theory and practice. Journal of Public Administration Research and Theory. 2008; 18 (4):543–571. [ Google Scholar ]
• Ansell C, Trondal J, Øgård M, editors. Governance in turbulent times. Oxford University Press; 2016. [ Google Scholar ]
• Bajpai K, Chong B. India's foreign policy capacity. Policy Design and Practice. 2019; 2 (2):137–162. [ Google Scholar ]
• Bakır C, Çoban MK. How can a seemingly weak state in the financial services industry act strong? The role of organizational policy capacity in monetary and macroprudential policy. New Perspectives on Turkey. 2019; 61 :71–96. [ Google Scholar ]
• Bakvis H. Rebuilding policy capacity in the era of the fiscal divident: A report from Canada. Governance. 2002; 13 (1):71–103. [ Google Scholar ]
• Bali AS, Capano G, Ramesh M. Anticipating and designing for policy effectiveness. Policy and Society. 2019; 38 (1):1–13. [ Google Scholar ]
• Bali AS, Ramesh M. Designing effective healthcare: Matching policy tools to problems in China. Public Administration and Development. 2017; 37 (1):40–508. [ Google Scholar ]
• Bali AS, Ramesh M. Policy capacity: A design perspective. In: Howlett M, Mukherjee I, editors. Routledge handbook of policy design. Routledge; 2018. pp. 331–344. [ Google Scholar ]
• Bali AS, Ramesh M. Assessing health reform: Studying tool appropriateness and critical capacities. Policy and Society. 2019; 38 (1):148–166. [ Google Scholar ]
• Bali AS, Ramesh M. Governing healthcare in India: A policy capacity perspective. International Review of Administrative Sciences. 2021 doi: 10.1177/00208523211002605. [ CrossRef ] [ Google Scholar ]
• Banerjee AV, Bardhan PK, Somanathan R, editors. Poverty and income distribution in India. Juggernaut; 2017. [ Google Scholar ]
• Bardhan P. State and development: The need for a reappraisal of the current literature. Journal of Economic Literature. 2016; 54 (3):862–892. [ Google Scholar ]
• Baskoy T, Evans B, Shields J. Assessing policy capacity in Canada’s public services: Perspectives of deputy and assistant deputy ministers. Canadian Public Administration. 2011; 54 (2):217–234. [ Google Scholar ]
• Bengston DN, Fletcher JO, Nelson KC. Public policies for managing urban growth and protecting open space: Policy instruments and lessons learned in the United States. Landscape and Urban Planning. 2004; 69 (2–3):271–286. [ Google Scholar ]
• Bernstein S, Cashore B. Complex global governance and domestic policies: Four pathways of influence. International Affairs. 2012; 88 (3):585–604. [ Google Scholar ]
• Bobrow DB. Policy design: Ubiquitous, necessary and difficult. In: Peters BG, Pierre J, editors. Handbook of public policy. Sage; 2006. pp. 75–96. [ Google Scholar ]
• Bobrow DB, Dryzek JS. Policy analysis by design. University of Pittsburgh Press; 1987. [ Google Scholar ]
• Borrás S. Policy learning and organizational capacities in innovation policies. Science and Public Policy. 2011; 38 (9):725–734. [ Google Scholar ]
• Bovens M, ‘t Hart P. Revisiting the study of policy failures. Journal of European Public Policy. 2016; 23 (5):653–666. [ Google Scholar ]
• Braathen NA. Instrument mixes for environmental policy: How many stones should be used to kill a bird? International Review of Environmental and Resource Economics. 2007; 1 (2):185–235. [ Google Scholar ]
• Braathen NA, Croci E. Environmental agreements used in combination with other policy instruments. In: Croci E, editor. The handbook of environmental voluntary agreements: Design, implementation and evaluation issues. Springer; 2005. pp. 335–364. [ Google Scholar ]
• Braun D. Organising the political coordination of knowledge and innovation policies. Science and Public Policy. 2008; 34 (4):227–239. [ Google Scholar ]
• Braun D. Lessons on the political coordination of knowledge and innovation policies. Science and Public Policy. 2008; 34 (4):289–298. [ Google Scholar ]
• Briassoulis H. Policy integration for complex environmental problems. Ashgate; 2005. [ Google Scholar ]
• Brown ML. Scientific uncertainty ad learning in European Union environmental policymaking. Policy Studies Journal. 2000; 28 (3):576–596. [ Google Scholar ]
• Busuioc EM, Lodge M. The reputational basis of public accountability. Governance. 2016; 29 (2):247–263. [ Google Scholar ]
• Capano G. Policy design spaces in reforming governance in higher education: The dynamics in Italy and the Netherlands. Higher Education. 2018; 75 (4):675–694. [ Google Scholar ]
• Capano G, Howlett H, Jarvis DSL, Ramesh M, Goyal N. Mobilizing policy (in)capacity to fight COVID-19: Understanding variations in national responses. Policy and Society. 2020; 39 (3):285–308. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Capano, G., & Howlett, M. (2020). The knowns and unknowns of policy instrument analysis: Policy tools and the current research agenda on policy mixes. Sage Open . 10.1177/2158244019900568.
• Capano G, Pavan E. Designing anticipatory policies through the use of ICTs. Policy and Society. 2019; 38 (1):96–117. [ Google Scholar ]
• Capano G, Woo JJ. Resilience and robustness in policy design: A critical appraisal. Policy Sciences. 2017; 50 (3):399–426. [ Google Scholar ]
• Cashore B, Bernstein S, Humphreys D, Visseren-Hamakers I, Rietig K. Designing stakeholder learning dialogues for effective global governance. Policy and Society. 2019; 38 (1):118–147. [ Google Scholar ]
• Chindarkar N, Howlett M, Ramesh M. Introduction to the special issue: Conceptualizing effective social policy design: Design spaces and capacity challenges. Public Administration and Developmentv. 2017; 37 (1):3–14. [ Google Scholar ]
• Clarke A, Craft J. The vestiges and vanguards of policy design in a digital context. Canadian Public Administration. 2017; 60 (4):476–497. [ Google Scholar ]
• Coban MK. Diffuse interest groups and regulatory policy change: Financial consumer protection in Turkey. Interest Groups an Advocacy. 2020; 9 (2):220–243. [ Google Scholar ]
• Coban MK. Compliance forces, domestic policy process, and international regulatory standards: Compliance with Basel III. Business and Politics. 2020; 22 (1):161–195. [ Google Scholar ]
• Compton M, ‘t Hart P. How to ‘see’ great policy successes. In: Compton M, ‘t Hart P, editors. Great policy successes. Oxford University Press; 2019. pp. 1–20. [ Google Scholar ]
• Considine M. Thinking outside the box? Applying design theory to public policy. Politics and Policy. 2012; 40 (4):704–724. [ Google Scholar ]
• Craft J, Daku M. A comparative assessment of elite policy recruits in Canada. Journal of Comparative Policy Analysis: Research and Practice. 2017; 19 (3):207–226. [ Google Scholar ]
• Dare L. From global forests to local politics: Unwrapping the boundaries within forest certification. Australian Journal of Political Science. 2018; 53 (4):529–547. [ Google Scholar ]
• Daugbjerg C, Fraussen B, Halpin D. Interest group and policy capacity: Models of engagement, policy goods and networks. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 243–261. [ Google Scholar ]
• del Río P. Analysing the interactions between renewable energy promotion and energy efficiency support schemes: The impact of different instruments and design elements. Energy Policy. 2010; 38 (9):4978–4989. [ Google Scholar ]
• del Río, P., & Howlett, M. (2013). “Beyond the ‘Tinbergen rule’ in policy design: Matching tools and goals in policy portfolios.”SSRN Scholarly Paper. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2247238 . Last accessed 11 June 2020).
• Donahue AK, Selden SC, Ingraham PW. Measuring government management capacity: A comparative analysis of city human resources management systems. Journal of Public Administration Research and Theory. 2000; 10 (2):381–412. [ Google Scholar ]
• Doremus H. A policy portfolio approach to biodiversity protection on private lands. Environmental Science & Policy. 2003; 6 (3):217–232. [ Google Scholar ]
• Dorsch MJ, Flachsland C. A polycentric approach to global climate governance. Global Environmental Politics. 2017; 17 (2):45–64. [ Google Scholar ]
• Douglas, S., Hart, P. T., Ansell, C., Anderson, L., Flinders, M., Head, B., & Moynihan, D. (2019). Towards positive public administration: A manifesto . Available online at https://www.researchgate.net/publication/336362499_Towards_Positive_Public_Administration_A_Manifesto .
• Dunlop CA. Organizational political capacity as learning. Policy and Society. 2015; 34 (4–5):259–270. [ Google Scholar ]
• Giest S. Big data for policymaking: Fad or fasttrack? Policy Sciences. 2017; 50 (3):367–382. [ Google Scholar ]
• Giest S, Mukherjee I. Behavioral instruments in renewable energy and the role of big data: A policy perspective. Energy Policy. 2018; 123 :360–366. [ Google Scholar ]
• Gilabert P, Lawford-Smith H. Political feasibility: A conceptual exploration. Political Studies. 2012; 60 (4):809–825. [ Google Scholar ]
• Golan J, Sicular T, Umapathi N. Unconditional cash transfers in China: Who benefits from the rural minimum living standard guarantee (Dibao) program? World Development. 2017; 93 :316–336. [ Google Scholar ]
• Grabosky PN. Green markets: Environmental regulation by the private sector. Law and Policy. 1994; 16 (4):419–448. [ Google Scholar ]
• Grant W. Policy instruments in the common agricultural policy. West European Politics. 2010; 33 (1):22–38. [ Google Scholar ]
• Gunningham N, Grabosky PN, Sinclair D. Smart regulation: Designing environmental policy. Clarendon Press; 1998. [ Google Scholar ]
• Haas PM. Introduction: Epistemic communities and international policy coordination. International Organization. 1992; 46 (1):1–35. [ Google Scholar ]
• Hartley J, Alford J, Hughes O, Yates S. Public value and political astuteness in the work of public managers: The art of the possible. Public Administration. 2015; 93 (1):195–211. [ Google Scholar ]
• Hartley K, Zhang J. Measuring policy capacity through governance indices. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 67–97. [ Google Scholar ]
• Hood C. Intellectual obsolescence and intellectual makeovers: Reflections on the tools of government after two decades. Governance. 2007; 20 (1):127–144. [ Google Scholar ]
• Hoppe R. Institutional constraints and practical problems in deliberative and participatory policy making. Policy and Politics. 2011; 39 (2):163–186. [ Google Scholar ]
• Hoppe R. Heuristics for practitioners of design: Rule-of-thumb for structuring unstructured problems. Public Policy and Administration. 2018; 33 (4):384–408. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Howlett M. Managing the “hollow state”: Procedural policy instruments and modern governance. Canadian Public Administration. 2000; 43 (4):412–431. [ Google Scholar ]
• Howlett M. Governance modes, policy regimes and operational plans: A multi-level nested model of policy instrument choice and policy design. Policy Sciences. 2009; 42 (1):73–89. [ Google Scholar ]
• Howlett M. Policy analytical capacity and evidence-based policymaking: Lessons from Canada. Canadian Public Administration. 2009; 52 (2):153–172. [ Google Scholar ]
• Howlett M. Designing public policies: Principles and instruments. Routledge; 2011. [ Google Scholar ]
• Howlett M. From the ‘old’ to the ‘new’ policy design: Design thinking beyond markets and collaborative governance. Policy Sciences. 2014; 47 (3):187–207. [ Google Scholar ]
• Howlett M. Policy design: What, who, how and why? In: Halpern C, Lascoumes P, Patrick LG, editors. L’instrumentation et ses effets. Presses de Sciences Po; 2014. pp. 281–315. [ Google Scholar ]
• Howlett M. Policy analytical capacity: The supply and demand for policy analysis in government. Policy and Society. 2015; 34 (3–4):173–182. [ Google Scholar ]
• Howlett M. The criteria for effective policy design: Character and context in policy instrument choice. Journal of Asian Public Policy. 2018; 11 (3):245–266. [ Google Scholar ]
• Howlett M. Dealing with the dark side of policy-making: Managing behavioural risk and volatility in policy designs. Journal of Comparative Policy Analysis. 2020; 22 (6):612–625. [ Google Scholar ]
• Howlett M, Capano G, Ramesh M. Designing for robustness: Surprise, agility and improvisation in policy design. Policy and Design. 2018; 37 (4):405–421. [ Google Scholar ]
• Howlett M, Lejano RP. Tales from the Crypt: The rise and fall (and rebirth?) of policy design. Administration and Society. 2013; 45 (3):357–381. [ Google Scholar ]
• Howlett M, Mukherjee I. An Asian perspective on policy instruments: Policy styles, governance modes and critical capacity challenges. Asia Pacific Journal of Public Administration. 2016; 38 (1):24–42. [ Google Scholar ]
• Howlett M, Mukherjee I. The importance of policy design: Effective processes, tools and outcomes. In: Howlett M, Mukherjee I, editors. Routledge handbook of policy design. Routledge; 2018. pp. 3–19. [ Google Scholar ]
• Howlett M, Mukherjee I. The contribution of comparative policy analysis to policy design: Articulating principles of effectiveness and clarifying design spaces. Journal of Comparative Policy Analysis. 2018; 20 (1):72–87. [ Google Scholar ]
• Howlett M, Mukherjee I, Rayner J. The elements of effective program design: A two-level analysis. In: Howlett M, Mukherjee I, editors. Handbook of policy formulation. Edward Elgar Publishing; 2017. pp. 129–144. [ Google Scholar ]
• Howlett M, Mukherjee I, Woo JJ. From tools to toolkits in policy design studies: The new design orientation towards policy formulation research. Policy and Politics. 2015; 43 (2):291–311. [ Google Scholar ]
• Howlett M, Ramesh M. The two orders of governance failure: Design mismatches and policy capacity issues in modern governance. Policy and Society. 2014; 33 (4):317–327. [ Google Scholar ]
• Howlett M, Ramesh M. Achilles’ heels of governance: Critical capacity deficits and their role in governance failures. Regulation and Governance. 2015; 10 (4):301–313. [ Google Scholar ]
• Howlett M, Ramesh M, Capano G. Policy-makers, policy-takers and policy tools: Dealing with behavioural issues in policy design. Journal of Comparative Policy Analysis. 2020; 22 (6):487–497. [ Google Scholar ]
• Howlett M, Ramesh M, Perl A. Studying public policy: Principles and processes. 4. Oxford University Press; 2020. [ Google Scholar ]
• Howlett M, Ramesh M, Wu X. Understanding the persistence of policy failures: The role of politics, governance and uncertainty. Public Policy and Administration. 2015; 30 (3–4):209–220. [ Google Scholar ]
• Howlett M, Rayner J. Design principles for policy mixes: Cohesion and coherence in ‘new governance arrangements’ Policy and Society. 2007; 26 (4):1–18. [ Google Scholar ]
• Howlett M, Rayner J. Patching vs packaging in policy formulation: Assessing policy portfolio design. Politics and Governance. 2013; 1 (2):170–182. [ Google Scholar ]
• Howlett M, Tosun J, editors. Routledge handbook of policy styles. Routledge; 2021. [ Google Scholar ]
• Huitema D, Jordan A, Munaretto S, Hildén M. Policy experimentation: Core concepts, political dynamics, governance and impacts. Policy Sciences. 2018; 51 (2):143–159. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Jensen O. Designing effective water policy: Capacity and effectiveness of reforms in developing countries. Policy and Society. 2018; 38 (1):77–95. [ Google Scholar ]
• Jordan A, Benson D, Wurzel R, Zito A. Policy instruments in practice. In: Dryzek JS, Norgaard RB, Schlosberg D, editors. Oxford handbook of climate change and society. Oxford University Press; 2011. pp. 537–549. [ Google Scholar ]
• Jordan A, Benson D, Wurzel R, Zito A. Environmental policy: Governing by multiple policy instruments? In: Richardson JJ, editor. Constructing a policy state? Policy dynamics in the EU. Oxford University Press; 2012. pp. 105–124. [ Google Scholar ]
• Jordan A, Huitema D. Innovations in climate policy: The politics of invention, diffusion, and evaluation. Environmental Politics. 2014; 23 (5):715–734. [ Google Scholar ]
• Karo E, Kattel R. Innovation and the state: Towards an evolutionary theory of policy capacity. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 123–150. [ Google Scholar ]
• Kelemen RD. Europe’s other democratic deficit: National authoritarianism in Europe’s Democratic Union. Government and Opposition. 2017; 52 (2):211–238. [ Google Scholar ]
• Kimbell L, Vesnić-Alujević L. After the toolkit: Anticipatory logics and the future of government. Policy Design and Practice. 2020; 3 (2):95–108. [ Google Scholar ]
• Lang A. Government capacities and stakeholders: What facilitates ehealth legislation? Globalization and Health. 2014; 10 (4):1–14. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Leong Ch, Howlett M. Theorizing the behavioral state: Resolving the theory-practice paradox of policy sciences. Public Policy and Administration. 2020 doi: 10.1177/0952076720977588. [ CrossRef ] [ Google Scholar ]
• Linder SH, Peters BG. From social theory to policy design. Journal of Public Policy. 1984; 4 (3):237–259. [ Google Scholar ]
• Lindquist EA. Public managers and policy communities: Learning to meet new challenges. Canadian Public Administration. 1992; 35 (2):127–159. [ Google Scholar ]
• Lodge M. Crisis, resources and the state: Executive politics in the age of the depleted state. Political Studies Review. 2013; 11 (3):378–390. [ Google Scholar ]
• Lodhi, I. (2018). Policy capacity: Introducing a comprehensive index. Paper prepared for the international workshop on public policy—Pittsburgh 2018 , June 26–28, Pittsburgh, United States.
• Luetjens J, Mintrom M, ‘t Hart P, editors. Successful public policy: Lessons from Australia and New Zealand. Australian National University Press; 2019. [ Google Scholar ]
• Maerz S, et al. State of the world 2019: Autocratization surges–resistance grows. Democratization. 2020; 27 (6):909–927. [ Google Scholar ]
• Majone G. On the notion of political feasibility. European Journal of Political Research. 1975; 3 (2):259–274. [ Google Scholar ]
• Maurya D, Ramesh M. Program design, implementation and performance: The case of social health insurance in India. Health Economics, Policy and Law. 2019; 14 (4):487–508. [ PubMed ] [ Google Scholar ]
• May PJ. Policy design and implementation. In: Peters BG, Pierre J, editors. Handbook of public administration. Sage Publications; 2003. pp. 223–233. [ Google Scholar ]
• May PJ. Regulation and compliance motivations: Examining different approaches. Public Administration Review. 2005; 65 (1):31–44. [ Google Scholar ]
• McConnell A. Policy success and failure. Oxford Research Encyclopedia of Politics. 2017 doi: 10.1093/acrefore/9780190228637.013.137. [ CrossRef ] [ Google Scholar ]
• McConnell A. Hidden agendas: Shining a light on the dark side of public policy. European Journal of Public Policy. 2018; 25 (12):1739–1758. [ Google Scholar ]
• McConnell A. The use of placebo policies to escape from policy traps. Journal of European Public Policy. 2020; 27 (7):957–976. [ Google Scholar ]
• Meckling J, Nahm J. The power of process: State capacity and climate policy. Governance. 2018; 31 (4):741–757. [ Google Scholar ]
• Menahem G, Stein R. High-capacity and low-capacity governance networks in welfare services delivery: A typology and empirical examination of the case of Israeli municipalities. Public Administration. 2013; 91 (1):211–231. [ Google Scholar ]
• Mergel I, Ganapati S, Whitford AB. Agile: A new way of governing. Public Administration Review. 2021; 81 (1):161–165. [ Google Scholar ]
• Montpetit É. Policy design for legitimacy: Expert knowledge, citizens, time and inclusion in the United Kingdom’s biotechnology sector. Public Administration. 2008; 86 (1):259–277. [ Google Scholar ]
• Mukherjee I, Bali AS. Policy effectiveness and capacity: Two sides of the design coin. Policy Design and Practice. 2019; 2 (2):103–114. [ Google Scholar ]
• Mukherjee I, Giest S. Designing policies in uncertain contexts: Entrepreneurial capacity and the case of the European emission trading scheme. Public Policy and Administration. 2019; 34 (3):262–286. [ Google Scholar ]
• Mukherjee I, Howlett M. An Asian perspective on policy instruments: Policy styles, governance modes and critical capacity challenges. AsiaPacific Journal of Public Administration. 2016; 38 (1):24–42. [ Google Scholar ]
• Mukherjee I, Mukherjee N. Designing for sustainable outcomes: Espousing behavioural change into co-production programmes. Policy and Society. 2018; 37 (3):326–346. [ Google Scholar ]
• Nair S, Howlett M. Policy myopia as a source of policy failure: Adaptation and policy learning under deep uncertainty. Policy and Politics. 2017; 45 (1):103–118. [ Google Scholar ]
• Newman J, Cherney A, Head BW. Policy capacity and evidence-based policy in the public service. Public Management Review. 2017; 19 (2):157–174. [ Google Scholar ]
• Norris P, Inglehart R. Cultural backlash: Trump, Brexit, and authoritarian populism. Cambridge University Press; 2019. [ Google Scholar ]
• O’Flynn J. Rethinking relationships: Clarity, contingency, and capabilities. Policy Design and Practice. 2019; 2 (2):115–136. [ Google Scholar ]
• Pal LA, Clark ID. Making reform stick: Political acumen as an element of political capacity for policy change and innovation. Policy and Society. 2015; 34 (3–4):247–257. [ Google Scholar ]
• Pestoff VA, Brandsen T, Verschuere B. New public governance, the third sector and co-production. Routledge; 2012. [ Google Scholar ]
• Peters BG. Policy capacity in public administration. Policy and Society. 2015; 34 (3–4):219–228. [ Google Scholar ]
• Peters BG. Can we be casual about being causal? Journal of Comparative Policy Analysis. 2020 doi: 10.1080/13876988.2020.1793327. [ CrossRef ] [ Google Scholar ]
• Peters BG, Eliadis P, Hill M, Howlett M. Conclusion: The future of instruments research. In: Eliadis P, Hill MM, Howlett M, editors. Designing government: From instruments to governance. McGill-Queen’s University Press; 2005. pp. 353–363. [ Google Scholar ]
• Peters BG, et al. Designing for policy effectiveness: Defining and understanding a concept. Cambridge University Press; 2018. [ Google Scholar ]
• Peters BG, Pierre J. Governance and policy problems: Instruments as unitary and mixed modes of policy intervention. Asia Pacific Journal of Public Administration. 2015; 37 (4):224–235. [ Google Scholar ]
• Rahman M, Wellstead A, Howlett M. From adaptive capacity to policy capacity adaptation assessments: Insights from the policy sciences. In: Keskitalo ECH, Preston BL, editors. Research handbook on climate change adaptation policy. Edward Elgar; 2019. pp. 291–309. [ Google Scholar ]
• Ramesh M, Bali AS. The remarkable healthcare performance in Singapore. In: Compton M, ‘t Hart P, editors. Great Policy Successes. Oxford University Press; 2019. pp. 42–62. [ Google Scholar ]
• Rayner J, McNutt K, Wellstead AM. Dispersed capacity and weak coordination: The challenge of climate change adaptation in Canada’s forest policy sector. Review of Policy Research. 2013; 30 (1):66–90. [ Google Scholar ]
• Rimkuté D. Organizational reputation and risk regulation: The effect of reputational threats on agency scientific outputs. Public Administration. 2018; 96 (1):70–83. [ Google Scholar ]
• Rotberg RI. Good governance means performance and results. Governance. 2014; 27 (3):511–518. [ Google Scholar ]
• Saguin K, Ramesh M, Howlett M. Policy work and capacities in a developing country: Evidence from the Philippines. Asia Pacific Journal of Public Administration. 2018; 40 (1):1–22. [ Google Scholar ]
• Schout A, Andrew J. The European Union’s governance ambitions and its administrative capacities. Journal of European Public Policy. 2008; 15 (4):957–974. [ Google Scholar ]
• Sewerin S, Béland D, Cashore B. Designing policy for the long term: Agency, policy feedback and policy change. Policy Sciences. 2020; 53 (2):243–252. [ Google Scholar ]
• Sidney MS. Policy formulation: Design and tools. In: Fischer F, Miller GJ, Sidney MS, editors. Handbook of public policy analysis: Theory, politics and methods. CRC Taylor & Francis; 2007. pp. 79–87. [ Google Scholar ]
• Siwale J, Okoye O. Microfinance regulation and social sustainability of microfinance institutions: The case of Nigeria and Zambia. Annals of Public and Cooperative Economics. 2017; 88 (4):611–632. [ Google Scholar ]
• Skeete J. Examining the role of policy design and policy interaction in EU automotive emissions performance gaps. Energy Policy. 2017; 104 :373–381. [ Google Scholar ]
• Skodvin T, Gullberg AT, Aakre S. Target-group influence and political feasibility: The case of climate policy design in Europe. Journal of European Public Policy. 2010; 17 (6):854–873. [ Google Scholar ]
• Spendzharova AB. Regulatory cascading: Limitations of policy design in European banking structural reforms. Policy and Society. 2016; 35 (3):227–237. [ Google Scholar ]
• Stazyk EC, Goerdel HT. The benefits of bureaucracy: Public managers’ perceptions of political support, goal ambiguity, and organizational effectiveness. Journal of Public Administration Research and Theory. 2011; 21 (4):645–672. [ Google Scholar ]
• Stirling A. Keep it complex. Nature. 2010; 468 (7327):1029–1031. [ PubMed ] [ Google Scholar ]
• Stone D. The “knowledge bank” and the global development network. Global Governance. 2003; 9 (1):43–61. [ Google Scholar ]
• Stone D, Ladi S. Global public policy and transnational administration. Public Administration. 2015; 93 (4):839–855. [ Google Scholar ]
• The Economist. (2020). Countries are using apps and data networks to keep tabs on the pandemic . March 26.
• Thelen K. How institutions evolve: The political economy of skills in Germany, Britain, the United States, and Japan. Cambridge University Press; 2004. [ Google Scholar ]
• Tiernan, A., & Wanna, J. (2006). Competence, capacity, capability: Towards conceptual clarity in the discourse of declining policy skills. Paper presented at the govnet international conference . Australian National University, Canberra, Australia.
• Tosun J, Lang A. Policy integration: Mapping the different concepts. Policy Studies. 2017; 38 (6):553–570. [ Google Scholar ]
• Van Der Steen M, Scherpenisse J, Van Twist M. Anticipating surprise: The case of the early warning system of Rijkswaterstaat in the Netherlands. Policy and Society. 2018; 37 (4):473–490. [ Google Scholar ]
• Vince J, Nursey-Bray M. Policy capacity in oceans governance: Rio+20 and Australia’s outcomes. Australian Journal of Maritime and Ocean Affairs. 2016; 8 (3):165–179. [ Google Scholar ]
• Virani A. Notions of policy effectiveness and implications for policy design: Insights from public–private partnerships in India. Policy Design and Practice. 2019; 2 (2):198–214. [ Google Scholar ]
• von Tunzelmann N. Technology and technology policy in the postwar UK: Market failure or“network failure?” Revue D’économie Industrielle. 2010; 129–130 :237–258. [ Google Scholar ]
• Walter J, et al. Policy capacity in disruptive times. Australian Journal of Political Science. 2020; 55 (1):72–85. [ Google Scholar ]
• Weaver, K. (2009a). If you build it, will they come? Overcoming unforeseen obstacles to program effectiveness . The Tansley Lecture, University of Saskatchewan.
• Weaver K. Target compliance: The final frontier of policy implementation. Brookings Institution Press; 2009. [ Google Scholar ]
• Weber EP, Lovrich NP, Gaffney MJ. Assessing collaborative capacity in a multidimensional world. Administration and Society. 2007; 39 (2):194–220. [ Google Scholar ]
• Weible CM, et al. COVID-19 and the policy sciences: Initial reactions and perspectives. Policy Sciences. 2020; 53 (2):225–241. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Wellstead AM, Stedman RC, Howlett M. Policy analytical capacity in changing governance contexts: A structural equation model (SEM) study of contemporary Canadian policy work. Public Policy and Administration. 2011; 26 (3):353–373. [ Google Scholar ]
• Wen Z. Government purchase of services in China: Similar intentions, different policy designs. Public Administration and Development. 2017; 37 (1):65–78. [ Google Scholar ]
• Williams RA, McNutt K. Climate change adaptation and policy capacity in the Canadian finance sector: A meso analysis. Review of Policy Research. 2013; 30 (1):91–113. [ Google Scholar ]
• Woo JJ. Policy capacity and Singapore’s response to the COVID-19 pandemic. Policy and Society. 2020; 39 (3):345–362. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Woo JJ, Ramesh M, Howlett M. Legitimation capacity: System-level resources and political skills in public. Policy and Society. 2015; 34 (3–4):271–283. [ Google Scholar ]
• Woo JJ, Ramesh M, Howlett M, Coban MK. Dynamics of global financial governance: Constraints, opportunities, and capacities in Asia. Policy and Society. 2016; 35 (3):269–282. [ Google Scholar ]
• Wu X, Ramesh M, Howlett M. Policy capacity: A conceptual framework for understanding policy competences and capabilities. Policy and Society. 2015; 34 (3–4):165–171. [ Google Scholar ]
• Wu X, Ramesh M, Howlett M. Policy capacity: Conceptual framework and essential components. In: Wu X, Howlett M, Ramesh M, editors. Policy capacity and governance: Assessing governmental competences and capabilities in theory and practice. Springer; 2018. pp. 1–25. [ Google Scholar ]
• Wu X, Ramesh M, Howlett M, Fritzen S. The public policy primer: Managing public policy. Routledge; 2010. [ Google Scholar ]
• Xiarchogiannopoulou E. Reform technology and consensus in building in discursive institutionalism: The integration of knowledge in policy formulation in Greek pension reform, 1990–2002. Critical Policy Studies. 2015; 9 (4):416–433. [ Google Scholar ]
• Yan Y, Saguin K. Policy capacity matters for capacity development. International Review of Administrative Sciences. 2021 doi: 10.1177/0020852320983867. [ CrossRef ] [ Google Scholar ]
• Yee W, Liu P. Governance capacity and regulatory enforcement: The case of food safety regulatory reform in Beijing. International Review of Administrative Sciences; 2021. [ Google Scholar ]
• Yi H, Feiock RC. Policy tool interactions and the adoption of state renewable portfolio standards. Review of Policy Research. 2012; 29 (2):193–206. [ Google Scholar ]

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

40 The Unique Methodology of Policy Research

Amitai Etzioni is a university professor and Professor of International Relations at The George Washington University. He served as a Senior Advisor at the Carter White House; taught at Columbia University, Harvard University, and University of California, Berkeley; and served as president of the Society for the Advancement of Socio-Economics (SASE). A study by Richard Posner ranked him among the top 100 American intellectuals. Etzioni is the author of many books, including Security First (2007), Foreign Policy: Thinking Outside the Box (2016), and Avoiding War with China (2017). His most recent book, Happiness is the Wrong Metric: A Liberal Communitarian Response to Populism, was published by Springer in January 2018.

• Published: 02 September 2009
• Cite Icon Cite
• Permissions Icon Permissions

This article provides a unique methodology of policy research, focusing on the various factors that differentiate policy research from basic research. It identifies malleability as a key variable of policy research, and this is defined as the amount of resources that would have to be expended to cause change in a given variable or variables. The scope of analysis/factors of policy research is shown to encompass all the major facets of the social phenomenon it is trying to deal with. Basic research, on the other hand, fragments the world into abstract and analytical slices, which are then studied individually. The last two differentiating factors of policy research and basic research, which are privacy and communication, are studied in the last two sections of the article.

Policy research requires a profoundly different methodology from that on which basic research relies, because policy research is always dedicated to changing the world while basic research seeks to understand it as it is. 1 The notion that if one merely understands the world better, then one will in turn know how to better it, is not supported by the evidence.

Typical policy goals are the reduction of poverty, curbing crime, cutting pollution, or changing some other condition (Mitchell and Mitchell 1969, 393) . Even those policies whose purpose is to maintain the status quo are promoting change—they aim to slow down or even reverse processes of deterioration, for instance that of natural monuments or historical documents. When no change is sought, say, when no one is concerned with changing the face of the moon, then there is no need for policy research in that particular area.

Moreover, although understanding the causes of a phenomenon, which successful basic research allows, is helpful in formulating policy, often a large amount of other information that is structured in a different manner best serves policy makers. 2 Policy researchers draw on a large amount of information that has no particular analytical base or theoretical background (of the kind that basic research provides). 3 In this sense medical science, which deals with changing bodies and minds, is a protypical policy science. It is estimated that about half of the information physicians employ has no basis in biology, chemistry, or any other science; but rather it is based on an accumulation of experience. 4 This knowledge is passed on from one medical cohort to another, as “these are the way things are done” and “they work.”

The same holds true for other policy sciences. For instance, criminologists who inform a local government that studies show that rehabilitation works more effectively in minimum security prisons than in maximum security prisons (a fact that can be explained by sociological theoretical concepts based on basic research) 5 know from long experience that they had better also alert the local authorities that such a reduction in security could potentially lead some inmates to escape and commit crimes in surrounding areas. Without being willing to accept such a “side effect” of the changed security policy, those governments who introduced it may well lose the next election and security in the prison will be returned to its previously high level. There is no particular sociological theoretical reason for escapes to rise when security is lowered. It is an observation based on common sense and experience; however it is hardly an observation that policy makers, let alone policy researchers should ignore. (They may though explore ways of coping with this “side effect,” for instance by either preparing the public ahead of time, introducing an alert system when inmates escape, or some other such measure.)

The examples just given seek to illustrate the difference between the information that basic research generates versus information that plays a major role in policy research. That is, there are important parts of the knowledge on which policy research draws that are based on distilled practice and are not derivable from basic research. Much of what follows deals with major differences in the ways that information and analysis are structured in sound policy research in contrast to the ways basic research is carried out.

One clarification before I can proceed: Policy research should not be confused with applied research. Applied research presumes that a policy decision has already been made and those responsible are now looking for the most efficient ways to implement it. Policy research helps to determine what the policy decision ought to be.

1. Malleability

A major difference between basic and policy research is that malleability is a key variable for the latter though not the former (Weimer and Vining 1989; 4) . Indeed for policy researchers it is arguably the single most important variable. Malleability for the purposes at hand ought to be defined as the amount of resources (including time, energy, and political capital) that would have to be expended to cause change in a given variable or variables. For policy research, malleability is a cardinal consideration because resources always fall short of what is required to implement given policy goals. Hence, to employ resources effectively requires determining the relative results to be generated from different patterns of allocation (Dunn 1981, 334– 402) . In contrast, basic research has no principled reason to favor some factors (or variables) over others. For basic research, it matters little if at all whether a condition under study can be modified and if it can how much it would cost. To illustrate, many sociological studies compare people by gender and age and although these variables may seem relevant, they are of limited value to policy research. Other variables used, such as the levels of income of various populations, the extent of education of various racial and ethnic groups, and the average size of cities, are somewhat more malleable but still not highly so. In contrast, perceptions are much more malleable.

One may say that basic research should reveal a preference for variables that have been less studied; however, such a consideration concerns the economics and politics of science rather than methodology. Because all scientific findings are conditional and temporary and often subject to profound revision and recasting, for basic researchers, retesting old findings can be just as valuable as covering new variables. In short, although in principle for basic research the study of all variables is legitimate, in a given period of time or amongst a given group of scientists, some may consider certain variables as more “interesting” or “promising” than others. In contrast, to reiterate, for policy research, malleability is the most important variable as it is directly related to its core reason for being: Promoting change.

Given the dominance of basic research methodology in the ways policy research is taught, it is not surprising to find that the question of which variables are more malleable than others is rarely studied in any systematic way. Due to the importance of this issue for policy research, some elaboration and illustrations are called for. Economic feasibility is a good case in point. Many policy researchers' final reports do not include any, not even crude estimates of the costs involved in what they are recommending. 6 Even less common is any consideration of the question of whether such changes can be made acceptable to elected representatives and the public at large; that is, political feasibility (Weimer and Vining 1989, 292– 324) . For instance, over the last decades several groups favored advancing their policy goals through constitutional amendments, ignoring the fact that these are extremely difficult to get passed.

In other cases, feasibility is treated as a secondary “applied” question to be studied later, after policy makers adopt the recommended policy. However, the issue runs much deeper than the assessments of feasibility of one kind or another. The challenge to policy research is to determine the relative resistance to change according to the different variables that are to be tackled. And this question must be tackled not on an ad hoc basis, but rather as a major part of systematic policy research. Moreover, if the variables involved are studied from this viewpoint, they themselves may be changed; that is, feasibility is enhanced rather than treated as a given.

Another example of the cardinal need to take malleability into account when conducting policy research concerns changing public attitudes. Policy makers often favor a “public education' campaign when they desire to affect people's beliefs and conduct. Policy makers tend to assume that it is feasible to change such predispositions through a way that might be called the Madison Avenue approach, which entails running a series of commercials (or public service announcements), mounting billboards, obtaining celebrity endorsements, and so on.

For example, the United States engaged in such a campaign in 2003 and 2004 to change the hearts and minds of “the Arab street” through what has also been termed “public diplomacy.” 7 The way this was carried out provides a vivid example of lack of attention to feasibility issues. American public diplomacy, developed by the State Department, included commercials, websites, and speakers programs that sought to “reconnect the world's billion Muslims with the United States the way McDonald's highlights its billion customers served” (Satloff 2003, 18) . It was based on the premiss that “blitzing Arab and Muslim countries with Britney Spears videos and Arabic‐language sitcoms will earn Washington millions of new Muslim sympathizers” (Satloff 2003, 18) . A study found that the results were “disastrous” (Satloff 2003, 18) . Some countries declined to air the messages and many Muslims who did see the material viewed it as blatant propaganda and offensive rather than compelling.

Actually, policy researchers bent on studying feasibility report that the Madison Avenue approach works only when large amounts of money are spent to shift people from one product to another when there are next to no differences between them (e.g. two brands of toothpaste) and when there is an inclination to use the product in the first place. However, when these methods are applied to changing attitudes about matters as different as condom use, 8 the United Nations, 9 electoral reform, and so forth, they are much less successful. Changing people's behavior—say to conserve energy, drive slower, cease smoking—is many hundreds of times more difficult. This is a major reason why totalitarian regimes, despite intensive public education campaigns, usually fail. The question of what is most feasible is determined by fiat by policy makers and their staffs rather than by studies that are reported to the policy makers by policy researchers. Hence decisions are often based on a fly‐by‐the‐seat‐ of‐your‐pants sense of what can be changed rather than on empirical evidence. 10 One of the few exceptions is studies of nation building in which several key policy researchers presented the reasons why such endeavors can be carried out at best only slowly while at the same time many policy makers claimed that it could be achieved in short order and at low cost. 11

In a preliminary stab at outlining the relative malleability of various factors, one may note that as a rule the laws of nature are not malleable; social relations, including patterns of asset distribution and power, are of limited malleability; and symbolic relations are highly malleable. Thus any policy‐making body that would seek to modify the level of gravity, for example, not for a particular situation (for instance a space travel simulator) but in general, will find this task at best extremely difficult to advance. In contrast, those who seek to change a flag, a national motto, the ways people refer to one another (e.g. Ms Instead of girl or broad), have a relatively easy time of doing so. Changes in the distribution of wealth among the classes or races—by public policy—are easier than changes involving the laws of nature, but more difficult than changing hearts and minds.

When policy researchers or policy makers ignore these observations and enact laws that seek grand and quick changes in power relations and economic patterns, the laws are soon reversed. A case in point is the developments that ensued when a policy researcher inserted into legislation the phrase “maximum feasible participation of the poor.” This Act was used to try to circumvent prevailing local power structures by directing federal funds to voluntary groups that included the poor on their advisory boards, which thus helped “empower the poor.” The law was nullified shortly thereafter. Similarly, when a constitutional amendment was enacted that banned the consumption of alcohol in the United States, it had some severely distorted effects on the American justice and law enforcement systems and did little actually to reduce the consumption of alcohol. It was also the only constitutional amendment ever to be repealed.

Among social changes, often legal and political reduction in inequality is relatively easier to come by than are socioeconomic changes along similar lines. Thus, African‐Americans and women gained de jure and de facto voting rights long before the differences in their income and representation in the seats of power moved closer to those of whites (in the case of African‐Americans) and of men (in the case of women). Nor have socioeconomic differences been reduced nearly as much as legal and political differences, although in both realms considerable inequalities remain. The same is true not just for the United States, but for other free societies and those that have been recently liberated.

In short, there are important differences in which dedication of resources, commitment of political capital, and public education are needed in order to bring about change. Sound policy research best makes the determination of which factors are more malleable than others, which is a major subject of study.

2. Scope of Analysis

Another particularly important difference between basic research and policy research methodology concerns the scope of factors that are best encompassed. Policy research at its best encompasses all the major facets of the social phenomenon it is trying to deal with. 12 In contrast, basic research proceeds by fragmenting the world into abstract, analytical slices which are then studied individually.

A wit has suggested that in economics everything has a price; in sociology, nothing has a price. Policy makers and hence researchers are at a disadvantage when they formulate preferred policy alternatives without paying attention to the longer‐run economic and budgetary effects—or the effect of such policy on social relations including families (e.g. tax preferences for singles), socioeconomic classes (e.g. estate taxes), and so on.

To put it in elementary terms, a basic researcher may well study only the prices of flowers (together with other economic factors); a physiologist the wilting processes; a social psychologist the symbolic meaning of flowers; and so forth. But a community that plans to grow flowers in its public gardens must deal with most, if not all of these elements and the relations between them. Flowers that are quick to wilt will not be suitable for its public gardens; the community will be willing to pay more for flowers that have a longer life or those that command a positive symbolic meaning, and so on.

Medicine provides another model of a policy science. It cannot be based only on biology, chemistry, anatomy, or any one science that studies a subset of variables relating to the body. Instead physicians draw on all these sciences and add observations of interaction effects among the variables. This forms a medical knowledge base and drives “policy” recommendations (i.e. medical prescriptions). Indeed doctors have often been chastised when they do not take into account still other variables, such as those studied by psychologists and anthropologists. Similarly, international relations is a policy science that best combines variables studied by economists, political scientists, law professors, and many others.

In short, the scope of variables that basic research encompasses can be quite legitimate and effective but also rather narrow. Policy researchers must be more eclectic and include at least all the variables that account for a significant degree of variance in the phenomenon that the policy aims to change.

3. Private and Confidential

Basic research is a public endeavor. As a rule its results are published so that others can critically assess them and piece them together with their findings and those of still others in order to build ever more encompassing and robust bodies of knowledge. Unpublished work is often not considered when scientists are evaluated for hiring and promoting, for prizes, or for some other reason, especially not if the work is kept secret for commercial or public security reasons. Historically, scientific findings were published in monographs, books, and articles in suitable journals. These served as the main outlets for the findings of basic research both because only by making scientific findings public could they become part of the cumulative scientific knowledge base and also because publication indicates that they have already passed some measure of peer review. It is only through peer review that evidence can be critically scrutinized. In recent years findings are still made public but increasingly they are often posted on websites, most of which lack peer review foundations, which is one reason why they are less trusted and not treated as a full‐fledged publication. Publication is still considered an essential element of basic research.

In contrast, the findings of policy research are often not published—they are provided in private to one policy maker or another (Radin 1997, 204– 18) . The main purpose of policy research is not to contribute to the cumulative process of building knowledge but rather to put to service available knowledge. In that profound sense policy research is often not public but client oriented. 13 Although some policy research is conducted in think tanks and public policy schools that may treat it similarly to basic research, more often than not it is conducted in specialized units in government agencies, the White House, corporate associations, and labor unions. And often tools of policy research are memos and briefings, not publications.

Often the findings of policy researchers are considered confidential or are governed by state secret acts (which is the case in many nations that have a less strong view of civil liberties than does the United States). That is, the findings are merely aimed at a specific client or a group of clients, and sharing them with the public is considered an offense. 14

4. Communication

Basic researchers, as a rule, are much less concerned with communicating, especially with a larger, “secular” public than are policy researchers. This may at first seem a contradiction to the previously made point that science (in the basic research sense) is public while policy research is often “private” (even when conducted for public officials). The seeming contradiction vanishes once one notes that basic researchers are obligated to share their findings with their colleagues , often a small group, and that they seek feedback from this group for both scientific and psychological validation. However, as a rule basic researchers have little interest in the public at large. Indeed, they tend to be highly critical of those who seek to reach such an audience—as did scholars such as Jay Gould and Carl Sagan (Etzioni 2003, 57– 60) .

In contrast, policy researchers often recognize the need to mobilize public support for the policies that their findings favor and hence they tend to help policy makers to mobilize such support by communicating with the public. James Fishkin developed a policy idea he called “deliberative democracy,” which entailed bringing together a group of people who constitute a living sample of the population for a period of time during which they are exposed to public education and presentations by public figures, and they are given a chance to have a dialogue. By measuring the changes in the views of this living sample, Fishkin found that one is able to learn how to change the public's mind. Fishkin did not just develop the concept and publish his ideas, but conducted a long and intensive campaign through radio, TV, newspapers, visits with public leaders, and much more, until his living sample was implemented in several locations (Fishkin 1997) . Indeed, according to Eugene Bardach, policy researchers must prepare themselves for “a long campaign potentially involving many players, including the mass public” (Bardach 2002, 115– 17) .

Hence, basic researchers are more likely to use technical terms (which may sound like jargon to outsiders), mathematical notations, extensive footnotes, and other such scientific features. On the other hand, policy researchers are more likely to express themselves in the vernacular and avoid technical terms.

One can readily show numerous publications of professors at schools of public policy and even think tanks that are rather similar if not indistinguishable from those of basic researchers. 15 But this is the case because these schools conduct mostly basic, and surprisingly little policy research. For example, on 28 April 2004 Google search found only 210 entries for “policy research methodology,” the good part of which referred to university classes by that name. But on closer examination, most entries were referring to basic, not policy research methodology. For instance, a course titled “Cultural Policy Research Methodology” at Griffith University in Australia includes in its course description “basic research techniques, particularly survey methodologies, qualitative methods and a more in depth approach to statistics.” 16 Many other entries were for classes in policy or research methodology (usually basic). The main reasons for this are ( a ) because few places train people in the special methodologies that policy research requires and ( b ) the reward structure is closely tied to basic research. Typically, promotions (especially tenure) at public policy schools are determined by evaluations and votes by senior colleagues from the basic research departments at the same universities or at other ones. Thus the future of an economist at the Harvard Business School may depend on what her colleagues in the Harvard Economics department think of her work. More informally, being invited to become a member of a basic research department is considered a source of prestige and an opportunity to shore up one's training and research. Conversely, only being affiliated with a policy school (like other professional schools) indicates a lack of recognition, which may translate into objective disadvantages. This pecking order, which favors basic over policy (considered “applied”) research, is of considerable psychological importance to researchers in practically all universities. Even in think tanks dedicated to policy research, many respect basic research more than policy research and hope to conduct it one day or regret that they are not suited to carry it out. 17

People who work for think tanks, which are largely dedicated to policy research, often seek to move to universities, in which tenure is more common and there is a greater sense of prestige. Hence many such researchers are keen to keep their “basic” credentials, although often they are unaware of the special methodology that policy research requires or are untutored in carrying it out in the first place because they were trained in basic research modes instead.

At annual meetings of one's discipline, in which findings are presented and evaluated, jobs are negotiated and information about them shared, and prestige scoring is rearranged, policy researchers will typically attend those dominated by their basic research colleagues. And attendance at policy research associations (such as the Association for Public Policy Analysis and Management) is meager. Most prizes and other awards available to researchers go to those who conduct basic research.

In short, although the logic of policy research favors it to be more communicative than basic research, this is often not the case because the training and institutional formations in which policy research is largely conducted favor basic research.

Bardach, E. ( 2002 ). Educating the client: an introduction.   Journal of Policy Analysis and Management , 21 (1): 115–17. 10.1002/pam.1044

Google Scholar

Berelson, B. , and Steiner, G.   1964 . Human Behavior: An Inventory of Scientific Findings . New York: Harcourt Brace and World.

Google Preview

Carothers, T.   1999 . Aiding Democracy Abroad: The Learning Curve. Washington, DC: Carnegie Endowment for International Peace.

DeJong, W. , Wolf, R. C. , and Austin, S. B.   2001 . US federally funded television public service announcements (PSAs) to prevent HIV/AIDS: a content analysis.   Journal of Health Communication , 6: 249–63. 10.1080/108107301752384433

Dunn, W. N.   1981 . Public Policy Analysis: An Introduction . Englewood Cliffs, NJ: Prentice Hall.

Etzioni, A.   1968 . The Active Society: A Theory of Societal and Political Processes . New York: Free Press.

——  1971 a . A Comparative Analysis of Complex Organizations , rev. edn. New York: Free Press.

——  1971 b . Policy research.   American Sociologist , 6 (supplementary issue: June): 8–12.

——  2003 . My Brother's Keeper: A Memoir and a Message . Lanham, Md.: Rowman and Littlefield.

——  2004 . A self‐restrained approach to nation‐building by foreign powers.   International Affairs , 80: 1–17. 10.1111/j.1468-2346.2004.00362.x

Fishkin, J. S.   1997 . The Voice of the People: Public Opinion and Democracy . New Haven, Conn.: Yale University Press.

Free Expression Project   2003 . The Progress of Science and Useful Arts: Why Copyright Today Threatens Intellectual Freedom , 2nd edn. New York: Free Expression Project; available at: www.fepproject.org/policyreports/copyright2dconc.html (accessed 27 Apr. 2004).

Inglefinger, F. J. , Relman, A. S. , and Findland, M.   1966 . Controversy in Internal Medicine . Philadelphia: W. B. Saunder.

Lasswell, H. , and Lerner, D.   1951 . The Policy Sciences . Stanford, Calif.: Stanford University Press.

Miller, D. W. 2001. Poking holes in the theory of broken windows. Chronicle of Higher Education (Feb.): A14.

Miller, J. 2004. Censored study on bioterror doubts U.S. preparedness. New York Times (29 Mar.): A15.

Mitchell, J. , and Mitchell, W.   1969 . Policy‐Making and Human Welfare . Chicago: Rand McNally.

Nelson, B.   1999 . Diversity and public problem solving: ideas and practice in policy education.   Journal of Policy Analysis and Management , 18: 134–55. 10.1002/(SICI)1520-6688(199924)18:1<134::AID-PAM9>3.0.CO;2-6

Radin, B. A.   1997 . The evolution of the policy analysis field: from conversation to conversations.   Journal of Policy Analysis and Management , 16: 204–18. 10.1002/(SICI)1520-6688(199721)16:2<204::AID-PAM1>3.0.CO;2-M

Raver, C.   2002 . Emotions matter: making the case for the role of young children's emotional development for early school readiness.   Social Policy Report , 16 (3): 3–19.

Roe, E.   1998 . Taking Complexity Seriously: Policy Analysis, Triangulation, and Sustainable Development . Boston: Kluwer Academic.

Satloff, R.   2003 . How to win friends and influence Arabs.   Weekly Standard , 18 Aug: 18–19.

Schön, D.   1983 . The Reflective Practitioner . New York: Basic Books.

Scott, J. 1994. Condom ads get direct: use them and get sex. Atlanta Journal and Constitution (3 Oct.): B1.

Star, S. A. , and Hughes, H. M.   1950 . Report on an educational campaign: the Cincinnati plan for the United Nations. American   Journal of Sociology , 55: 389–400.

Weimer, D. L. , and Vining, A. R.   1989 . Policy Analysis: Concepts and Practice . Englewood Cliffs, NJ: Prentice Hall.

Weiss, C.   1983 . Ideology, interests and information: the basis of policy positions. Pp. 213–45 in Ethics, the Social Sciences and Policy Analysis , ed. D. Callahan and B. Jennings . New York: Plenum.

Wilson, J. Q. , and Kelling, G.   1982 . Broken windows: the police and neighborhood safety.   Atlantic Monthly , 249 (3: Mar.): 29–38.

The first book to deal with policy sciences and consequently often cited is Lasswell and Lerner's The Policy Sciences (1951) . However this book does not address the methodological issues at hand. For an early treatment of these issues, see Etzioni 1971 b , 1968 .

For an example of how to structure and present policy research and analysis, see Dunn 1981, 322 .

For example many policy makers subscribe to George L. Kelling and James Q. Wilson's criminology theories because they make sense, despite the fact that they are not grounded in academic research. See Wilson and Kelling 1982 . For criticisms of this approach to criminology, see Miller 2001 .

“Much” of medicine is not scientifically supported (Inglefinger, Relman, and Findland 1966) . “85 percent of the problems a doctor sees in his office are not in the book” (quoted from a physician in Schön 1983, 16) .

See Etzioni 1971 a , 246– 7 .

See for example Free Expression Project 2003 ; Raver 2002, 3– 19 .

See, for instance, The Advisory Group on Public Diplomacy in the Arab and Muslim World, “Changing minds, winning peace: a new strategic direction for U.S. public diplomacy in the Arab and Muslim world,” Oct, 2003, Edward P. Djerejian, chair.

For instance, the Centers for Disease Control conducted a ten‐year ad campaign to educate Americans about condoms and to encourage their use to prevent HIV transmission. After spending millions of dollars on these ads, a CDC study found that only 45 % of sexually active high school students used a condom the last time they had sex: see Scott 1994 . A recent evaluation of the program issued an unqualified “no” in answer to the question, “Has the U.S. federal government's HIV /AIDS television [public service announcement] campaign been designed not only to make the public aware of HIV /AIDS but also to provide appropriate messages to motivate and reinforce behavior change?” See DeJong, Wolf, and Austin 2001, 256 . Of the fifty‐six ads reviewed, fifty were created by the CDC, the other six were created by the National Institute on Drug Abuse.

Star and Hughes 1950 , quoted in Berelson and Steiner 1964, 530 .

Indeed unlike science, Carol Weiss has argued that in the policy field it may be impossible to separate objective knowledge from ideology or interests: see Weiss 1983 .

See Carothers 1999 ; Etzioni 2004 .

Roe 1998 . For an academic policy research perspective, see Nelson 1999 .

See “Professional practice symposium: educating the client,” Journal of Policy Analysis and Management , 21 (1: 2002): 115– 36.

For instance, the Defense Department has prohibited a Washington think tank from publishing a complete report about the lack of government preparedness for bioterror attacks: see Miller 2004 .

See for instance the reports of the family research division of the Heritage Foundation, available at www.heritage.org/research/family/issues2004.cfm (accessed 29 Apr. 2004). See also “The war on drugs: addicted to failure,” Recommendations of the Citizens' Commission on US Drug Policy, available at www.ips‐dc.org/projects /drugpolicy.htm (accessed 29 Apr. 2004).

See Griffith University course catalog. Available at: www22.gu.edu.au/STIP/servlet/STIP?s=7319AMC (accessed 28 Apr. 2004).

This section is based on my personal observations of organizations such as the John F. Kennedy School of Government, the American Enterprise Institute, RAND, CATO, the Heritage Foundation, and many others.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Policy Studies and Regional Public Policy-Making

• First Online: 10 October 2018

Cite this chapter

• Anne Marie Hoffmann 4  

Part of the book series: Governance, Development, and Social Inclusion in Latin America ((GDSILA))

230 Accesses

The objective of this chapter is to develop a model for the analysis of regional public policy-making. This model connects the macro-level, which is determined by the structures and the normativity of the respective region, with the micro-level of policy-making, represented by transgovernmental networks. Concepts of policy studies complement regional integration theory and will be used to identify factors that influence these networks. Policy studies traditionally consider the national sphere of public policy-making. Also, approaches for policy-making at the global level have increasingly been developed. Until now, however, a regional level of public policy-making has not yet been theorized. This chapter, therefore, relies on findings of both levels: the national and the global. The analytical dimensions of actors, interests, structures, and ideas are discussed individually in view of the regional level of policy-making. Finally, the analytical model is designed based on the findings for these dimensions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

“proceso social tejido alrededor de un tema o cuestión.”

“cristalizan o malogran la representatividad, constitucionalidad, publicidad, racionalidad de los gobiernos.”

Adler, Emanuel, and Peter M. Haas. 1990. Conclusion: Epistemic Communities, World Order, and the Creation of a Reflective Research Program. In When Knowledge is Power: Three Models of Change in International Organizations , ed. Ernst B. Haas, vol. 22, 367–390. Studies in International Political Economy. Berkeley: University of California Press.

Google Scholar  

Aguilar Villanueva, Luis F. 1992. La Hechura de Las Políticas . Vol. 2. Mexico: Miguel Ángel Porrúa.

Birkland, Thomas A. 2006. Agenda Setting in Public Policy. In Handbook of Public Policy Analysis: Theory, Politics, and Methods , ed. Frank Fischer, Gerald J. Miller, and Mara S. Sidney, 63–78. Boca Raton, FL: CRC Press.

Chapter   Google Scholar  

Bouckaert, Geert, Guy Peters, and Koen Verhoest. 2010. The Coordination of Public Sector Organizations . Hampshire: Palgrave Macmillan UK.

Book   Google Scholar  

Cross, Mai’a K. Davis. 2013. Rethinking Epistemic Communities Twenty Years Later. Review of International Studies 39 (1): 137–160.

Article   Google Scholar  

Cuervo, Jorge I., ed. 2007. Ensayos Sobre Politicas Publicas . Bogotá: Universidad Externado de Colombia.

———., ed. 2015. Ensayos sobre políticas públicas II . Bogotá: Universidad Externado de Colombia.

deLeon, Peter, and Danielle M. Vogenbeck. 2006. The Policy Sciences at the Crossroad. In Handbook of Public Policy Analysis: Theory, Politics, and Methods , ed. Frank Fischer, Gerald J. Miller, and Mara S. Sidney, 3–14. Boca Raton, FL: CRC Press.

Drezner, Daniel W. 2008. All Politics Is Global: Explaining International Regulatory Regimes . Rev. ed. Princeton, NJ: Princeton University Press.

Fischer, Frank. 2006. Deliberative Policy Analysis as Practical Reason: Integrating Empirical and Normative Arguments. In Handbook of Public Policy Analysis: Theory, Politics, and Methods , ed. Frank Fischer, Gerald J. Miller, and Mara S. Sidney, 223–236. Boca Raton, FL: CRC Press.

Fischer, Frank, and John Forester. 1993. The Argumentative Turn in Policy Analysis and Planning . Durham, NC: Duke University Press.

Fischer, Frank, and Herbert Gottweis. 2012. Introduction: The Argumentative Turn Revisited. In The Argumentative Turn Revisited: Public Policy as Communicative Practice , ed. Frank Fischer and Herbert Gottweis, 1–30. Durham, NC: Duke University Press.

Graeme, Currie, Suzana Grubnic, and Ron Hodges. 2011. Leadership in Public Services Networks: Antecedents, Process and Outcome. Public Administration 89 (2): 242–264.

Haas, Peter M. 1992. Introduction: Epistemic Communities and International Policy Coordination. International Organization 46 (1): 1–35.

Holzner, Burkart. 1972. Reality Construction in Society . Rev. ed. Cambridge, MA: Schenkman Publication.

Howlett, Michael, Allan McConnell, and Anthony D. Perl. 2013. Reconciling Streams and Stages: Avoiding Mixed Metaphors in the Characterization of Policy Processes. In APSA 2013 Annual Meeting Paper . http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2300442 .

Howlett, Michael, Anthony Perl, and M. Ramesh. 2009. Studying Public Policy: Policy Cycles & Policy Subsystems . Toronto, ON: Oxford University Press.

Kaul, Inge. 2013. Global Public Policy: A Policy in the Making. Accessed July 9, 2015. http://www.icpublicpolicy.org/IMG/pdf/panel73_s1_kaul.pdf .

Keohane, Robert O., and Joseph S. Nye. 1974. Transgovernmental Relations and International Organizations. World Politics 27 (1): 39–62.

———. 2011. Power and Interdependence . 4th ed. Stoughton: Longman.

Lasswell, Harold D. 1956. The Political Science of Science: An Inquiry into the Possible Reconciliation of Mastery and Freedom. The American Political Science Review 50 (4): 961–979.

Majone, Giandomenico. 1992. Evidence, Argument, and Persuasion in the Policy Process . New Haven, CT: Yale University Press.

Mayntz, Renate. 1993. Modernization and the Logic of Interorganizational Networks. Knowledge and Policy 6 (1): 3–16.

Messner, Dirk. 1995. Die Netzwerkgesellschaft: wirtschaftliche Entwicklung und internationale Wettbewerbsfähigkeit als Probleme gesellschaftlicher Steuerung . Köln: Weltforum-Verlag.

Miller, Hugh T., and Tansu Demir. 2006. Policy Communities. In Handbook of Public Policy Analysis: Theory, Politics, and Methods , ed. Frank Fischer, Gerald J. Miller, and Mara S. Sidney, 137–148. Boca Raton, FL: CRC Press.

de Oliveira, José Antônio Puppim. 2006. Desafios Do Planejamento Em Políticas Públicas: Diferentes Visões e Práticas. RAP Rio de Janeiro 40 (1): 273–288.

Oszlak, Oscar, and Guillermo A. O’Donnell. 1976. Estado y Políticas Estatales En América Latina: Hacia Una Estrategia de Investigación . Buenos Aires: Cedes.

Pallares, Francesc. 1988. Las Políticas Públicas: El Sistema Político En Acción. Revista de Estudios Políticos 62: 141–162.

Pattberg, Philipp. 2013. The Emerging Transnational Policy Sphere: Lessons Learned from the Sustainability Arena. Accessed August 14, 2015. http://www.icpublicpolicy.org/IMG/pdf/panel_73_s2__pattberg.pdf .

Peters, Guy. 2013. Individuals, Institutions and Public Policy: Making the Linkages. Accessed June 2, 2015. http://www.icpublicpolicy.org/IMG/pdf/panel_84_s1_peters.pdf .

Radaelli, Claudio M. 1999a. The Public Policy of the European Union: Whither Politics of Expertise? Journal of European Public Policy 6: 757–774.

———. 1999b. Technocracy in the European Union . London: Longman.

Raustalia, Kal. 2002. The Architecture of International Cooperation: Transgovernmental Networks and the Future of International Law. Virginia Journal of International Law Association 43 (1): 1–92.

Ruggie, John Gerard. 2004. Reconstituting the Global Public Domain—Issues, Actors, and Practices. European Journal of International Relations 10 (4): 499–531.

Schlappa, Hans, and Yasmin Imani. 2013. Leadership in the Co-Production of Public Services: An Initial Conceptual Framework. Accessed October 17, 2015. http://www.icpublicpolicy.org/IMG/pdf/panel_43_h_schlappa_y_imani.pdf .

Schmidt, Vivian A. 2012. Discursive Institutionalism: Scope, Dynamics, and Philosophical Underpinnings. In The Argumentative Turn Revisited: Public Policy as Communicative Practice , ed. Frank Fischer and Herbert Gottweis, 85–113. Durham, NC: Duke University Press.

Slaughter, Anne-Marie. 2009. A New World Order . Princeton, NJ: Princeton University Press.

Souza, Celina. 2006. Políticas Públicas: Uma Revisão Da Literatura. Sociologias 8: 20–45.

Stone, Diane. 2013. Knowledge Actors and Transnational Governance: The Private-Public Policy Nexus in the Global Agora . Basingstoke: Palgrave Macmillan.

Torfing, Jacob. 2012. Interactive Governance: Advancing the Paradigm . Oxford: Oxford University Press.

Velasco, Javier Torres. 2015. Prefacio. In Ensayos sobre políticas públicas II , ed. Jorge Iván Cuervo Restrepo, 9–15. Bogotá: U. Externado de Colombia.

Wagner, Peter. 2006. Public Policy, Social Science, and the State: An Historical Perspective. In Handbook of Public Policy Analysis: Theory, Politics, and Methods , ed. Frank Fischer, Gerald J. Miller, and Mara S. Sidney, 29–40. Boca Raton, FL: CRC Press.

Weible, Christopher M., and Paul A. Sabatier. 2006. A Guide to Advocacy Coalition Framework. In Handbook of Public Policy Analysis: Theory, Politics, and Methods , ed. Frank Fischer, Gerald J. Miller, and Mara S. Sidney, 123–136. Boca Raton, FL: CRC Press.

Wolin, Sheldon S. 1960. Politics and Vision: Continuity and Innovation in Western Political Thought . Boston: Princeton University Press.

Yanow, Dvora. 2000. Conducting Interpretive Policy Analysis. Qualitative Research Methods . Beverly Hills, CA [u.a.], 1986–47. Thousand Oaks, CA [u.a.]: Sage.

Download references

Author information

Authors and affiliations.

Hamburg University of Applied Sciences, Hamburg, Germany

Anne Marie Hoffmann

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2019 The Author(s)

About this chapter

Hoffmann, A.M. (2019). Policy Studies and Regional Public Policy-Making. In: Regional Governance and Policy-Making in South America. Governance, Development, and Social Inclusion in Latin America. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-98068-3_3

Download citation

DOI : https://doi.org/10.1007/978-3-319-98068-3_3

Published : 10 October 2018

Publisher Name : Palgrave Macmillan, Cham

Print ISBN : 978-3-319-98067-6

Online ISBN : 978-3-319-98068-3

eBook Packages : Political Science and International Studies Political Science and International Studies (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Open access
• Published: 06 March 2021

How to bring research evidence into policy? Synthesizing strategies of five research projects in low-and middle-income countries

• Séverine Erismann 1 , 2 ,
• Maria Amalia Pesantes 3 ,
• David Beran 4 ,
• Andrea Leuenberger 1 , 2 ,
• Andrea Farnham 1 , 2 ,
• Monica Berger Gonzalez de White 1 , 2 , 5 ,
• Niklaus Daniel Labhardt 1 , 2 , 6 ,
• Fabrizio Tediosi 1 , 2 ,
• Patricia Akweongo 7 ,
• August Kuwawenaruwa 1 , 2 , 8 ,
• Jakob Zinsstag 1 , 2 ,
• Fritz Brugger 9 ,
• Claire Somerville 10 ,
• Kaspar Wyss 1 , 2 &
• Helen Prytherch 1 , 2  

Health Research Policy and Systems volume  19 , Article number:  29 ( 2021 ) Cite this article

24k Accesses

17 Citations

13 Altmetric

Metrics details

Addressing the uptake of research findings into policy-making is increasingly important for researchers who ultimately seek to contribute to improved health outcomes. The aims of the Swiss Programme for Research on Global Issues for Development (r4d Programme) initiated by the Swiss National Science Foundation and the Swiss Agency for Development and Cooperation are to create and disseminate knowledge that supports policy changes in the context of the 2030 Agenda for Sustainable Development. This paper reports on five r4d research projects and shows how researchers engage with various stakeholders, including policy-makers, in order to assure uptake of the research results.

Eleven in-depth interviews were conducted with principal investigators and their research partners from five r4d projects, using a semi-structured interview guide. The interviews explored the process of how stakeholders and policy-makers were engaged in the research project.

Three key strategies were identified as fostering research uptake into policies and practices: (S1) stakeholders directly engaged with and sought evidence from researchers; (S2) stakeholders were involved in the design and throughout the implementation of the research project; and (S3) stakeholders engaged in participatory and transdisciplinary research approaches to coproduce knowledge and inform policy. In the first strategy, research evidence was directly taken up by international stakeholders as they were actively seeking new evidence on a very specific topic to up-date international guidelines. In the second strategy, examples from two r4d projects show that collaboration with stakeholders from early on in the projects increased the likelihood of translating research into policy, but that the latter was more effective in a supportive and stable policy environment. The third strategy adopted by two other r4d projects demonstrates the benefits of promoting colearning as a way to address potential power dynamics and working effectively across the local policy landscape through robust research partnerships.

Conclusions

This paper provides insights into the different strategies that facilitate collaboration and communication between stakeholders, including policy-makers, and researchers. However, it remains necessary to increase our understanding of the interests and motivations of the different actors involved in the process of influencing policy, identify clear policy-influencing objectives and provide more institutional support to engage in this complex and time-intensive process.

Peer Review reports

Increasingly, research funders are asking their grantees to address the uptake of research findings into decision-making processes and policy-making [ 1 , 2 ]. This growing trend is a response to a need for real-world and context-sensitive evidence to respond to and address complex health systems and health service delivery bottlenecks faced by policy-makers, health practitioners, communities and other actors that require more than single interventions to induce large-scale change [ 3 ]. Moreover, there is growing pressure for applied and implementation research to be relevant, demonstrate value for money and result in high-impact publications. The relevance of ensuring the translation of research into practice is also reflected in growing support for research projects with concrete requirements regarding the evaluation of their impact of science on society [ 4 ].

One example of the above is the Swiss Programme for Research on Global Issues for Development (r4d Programme) initiated by the Swiss National Science Foundation (SNSF) and the Swiss Agency for Development and Cooperation (SDC) covering the period 2012–2022. The r4d Programme is aimed at researchers in Switzerland and low-and middle-income countries (LMICs) conducting projects that specifically focus on poverty reduction and the protection of public goods in developing countries. Its specific objectives are to create and disseminate knowledge that supports policy-making in the area of global development and foster research on global issues in the context of the 2030 Agenda for Sustainable Development [ 5 , 6 ].

While the linkage of research to policy is strongly encouraged by research funding agencies, the uptake of research evidence by policy-makers to establish new laws and regulations or to improve policies to solve a problem or enhance implementation effectiveness, especially in LMICs, remains weak [ 2 , 7 ]. This is often referred to as the gap between research and policy [ 8 ]. One of the factors that was identified with the dearth of research uptake in previous studies is a lack of evidence that is context sensitive, timely and relevant for policy-makers; other factors include difficulties in accessing existing evidence, challenges with correctly interpreting and using existing evidence [ 7 , 9 ] and also a lack of interest from policy-makers in the use and uptake of evidence [ 10 ]. Using the SNSF r4d funding scheme, our aim is to show how researchers have engaged with stakeholders, including policy-makers, from the onset of a research project, in order to identify strategies for evidence uptake and use.

As part of the r4d Programme, several synthesis initiatives have been launched to disseminate the research evidence from the r4d projects and increase its impact ( http://www.r4d.ch/r4d programme/synthesis ). The aim of one of these synthesis initiatives is to support knowledge translation and exchange, as well as knowledge diffusion and dissemination among 15 r4d projects focusing on public health. More specifically, the aim is to facilitate the uptake of findings for the benefit of societies in LMICs, especially with regards to social inclusion and gender equity in the drive towards universal health coverage (UHC) and the 2030 Agenda for Sustainable Development [ 6 ]. The present study and resulting article are part of this synthesis initiative.

In this article, we present—through five case studies—strategies to translate and bridge evidence emerging from research into policy-making and decision-making. We rely on the experiences of five public health projects within the r4d research initiative. This paper describes these experiences, reports on the lessons learnt and outlines important features and challenges of engaging in this process using the researchers’ perspectives. This paper contributes to the body of literature on research translation by highlighting concrete examples and successful strategies for the uptake of research evidence in policy formulation.

Invitations were sent out to researchers working on projects within the r4d Programme to share their experiences with the project. Based on the interest shown by researchers, five projects were selected by the authors to demonstrate the different approaches and strategies used in the r4d projects with the aim to influence policy. Researchers were asked to share descriptions of the different approaches used in seeking to influence the uptake of research results by policy-makers. Each project represents a case study with emphasis on the main features of their translational approaches and the challenges, enablers and successes encountered.

The different research–policy engagement strategies were identified through data analysis of the interviews conducted within the framework of the five r4d case studies and were inspired by the work conducted by Uzochukwu and colleagues in Nigeria [ 2 ], who described four detailed strategies to support evidence-informed policy-making: (1) policy-makers and stakeholders seeking evidence from researchers; (2) involving stakeholders in designing objectives of a research project and throughout the research period; (3) facilitating policy-maker–researcher engagement in optimizing ways of using research findings to influence policy and practice; (4) active dissemination of own research findings to relevant stakeholders and policy-makers (see Table 1 ).

In using the term stakeholder, we apply the following definition by Brinkerhoff and Crosby [ 11 ]: “A stakeholder is an individual or group that makes a difference or that can affect or be affected by the achievement of the organization’s objectives”. Hence, individual stakeholders can include politicians (heads of state and legislators), government bureaucrats and technocrats from various sectors (e.g. health), but also representatives of civil society organizations and support groups [ 12 ].

Data collection

Eleven in-depth interviews with principal investigators and their research partners from five r4d projects were conducted by the first author, using a semi-structured interview guide. The interview guide covered the following themes: (1) How were stakeholders involved in the research project? (2) Was there uptake of research evidence in national/international policies? (3) How were research results disseminated? (4) What were the challenges or obstacles encountered in disseminating and translating evidence from research to policy? The interview duration was between 30 and 45 min. Seven interviews were conducted with researchers based in Switzerland and four with researchers in LMICs. At least two interviews were conducted for each r4d case study.

Data management and analysis

Of the 11 interviews, nine were audio recorded and notes taken. Audio files were transcribed verbatim by the same researcher. Two interviews were not recorded, but detailed notes were taken during the interview.

A qualitative content analysis method was used in order to organize and structure both the manifest and latent content [ 13 ]. Aligned to overall study questions, essential content was identified by the first author, which involved a process of generating a provisional list of themes of interest that were based on the study objectives, including stakeholder involvement in the generation of research questions, research process, generation of results and dissemination of research findings, as well as challenges to research dissemination and policy uptake. In a next step, the transcripts were sorted and grouped by the first author according to the coding scheme for analysis. This involved using the content summary analysis method, which consists of reducing the textual content and preserving only the essential content in order to produce a short text [ 14 ]. As several co-authors were interviewed, they validated that their perspective was not misinterpreted or misrepresented.

Three key strategies were identified for research uptake into policy and practice throughout the data collection of this synthesis initiative: (S1) stakeholders directly engaged with and sought evidence from researchers; (S2) stakeholders were involved in the design and throughout the implementation of the research project; and (S3) stakeholders engaged in participatory and transdisciplinary research approaches to co-produce knowledge and inform policy. The first two strategies (S1, S2) are in line with Uzochukwu and colleagues’ work [ 2 ], and the third strategy (S3) is an additional category based on the experiences of researchers in r4d projects [ 2 ]. Each r4d project is described in more detail as a case study in one of these three strategies (Table 2 ).

S1: stakeholders directly engaged with and sought evidence from researchers

In this strategy, international stakeholders requested evidence from the research team. This is a unique (and rare) strategy, as stated by Uzochukwu et al. [ 2 ], and often involves a policy window of opportunity in which stakeholders, including policy-makers, are looking to solve a particular problem, which coincides with the publishing of a scientific report or paper and the interests of these same groups [ 15 , 16 ].

Improving the HIV care cascade in Lesotho: towards 90-90-90—a research collaboration with the Ministry of Health of Lesotho

In this r4d project, the research team was approached by the International Aids Society (IAS) and the World Health Organization (WHO) in Geneva, based on the publication of their study protocol [ 17 ], introducing their innovative research approach of same-day antiretroviral therapy (ART) initiation in rural communities in Lesotho:

“They [international stakeholders] were all keen of getting the results out and requested evidence of the randomized controlled trials. We shared the results confidentially with WHO as soon as we had the data and thereafter published the results in a journal with a wide reach. WHO as well as other international guidelines and policy committees took up the recommendation of same-day ART initiation and informed global guidelines” (Researcher 1).

As a result, many HIV programmes in sub-Saharan Africa as well as in the global north have adopted the practice of offering rapid-start ART to persons who test HIV positive even outside a health facility. In this example, the policy window and direct stakeholder engagement was crucial for the effective translation and uptake of research evidence.

Furthermore, by closely collaborating with national policy-makers, the research team advocated for the setting up of a research database and of knowledge management units within the Ministry of Health (MoH) of Lesotho, which have been successfully established. The members of the research project consortia have also initiated a national research symposium on a bi-annual basis, which is chaired by the MoH with the aim of facilitating the dissemination and uptake of research findings.

S2: Stakeholders were involved in the design and throughout the implementation of the research project

In this strategy, policy uptake is facilitated through stakeholder engagement from the beginning as well as during the conduct of research activities, through participating at workshops or functioning in the governance of the projects. Two r4d projects illustrate this strategy.

Health system governance for an inclusive and sustainable social health protection in Ghana and Tanzania

This project established a Country Advisory Group (CAG) at the start that included representatives of the main stakeholders of the social health protection systems. The CAGs were involved in all phases of the project, from the definition of the research plans to the dissemination of the results. The specific research questions addressed by the project emerged from the interactions with these main stakeholders, i.e. national policy-makers, healthcare providers and members of the social health protection schemes (the NHIS and the Livelihood Empowerment Against Poverty schemes in Ghana; and the National Health Insurance Fund, the Community Health Funds and the Tanzania Social Action Fund in Tanzania). Specifically in Ghana, the following stakeholders played a major role in shaping the research plan: the Ministry of Gender Children and Social Protection (MGCSP), the Ghana Health Service (Policy Planning and Monitoring and Evaluation Division; Research and Development Division), the National Health Insurance Authority (NHIA) and the Associations of Private Health Care Providers. In Tanzania, a major role was played by the Ministry of Health, Community, Development, Gender, Elderly and Children, the President’s Office—Regional Administration and Local Government, by representatives of civil society organizations, such as Sikika, by the SDC (Swiss Agency for Development and Cooperation) Health Promotion and System Strengthening project and by the SDC-supported development programme.

These stakeholders were subsequently involved in steering the research, as captured by a researcher:

“In Ghana, it was a balanced relationship. They were involved since the very beginning of the project in articulating what the information gap at policy level is, formulating the research questions and understanding the methods/what is feasible. In Tanzania, where the policy landscape is more fragmented, it was very important to listen to the voices of several different stakeholders” (Researcher 2).

The stakeholder consultations in Ghana and Tanzania initially involved discussions on the relevance of the research plans to address the existing gaps in strengthening the social health protection scheme, the synergies with other research initiatives and the feasibility of implementing the proposed research. Later on in the project, the consultation process involved reviewing and discussing the focus of the research and the appropriateness of the research aims in light of decisions and reforms that were under discussion by the government but not in the public domain. This led to revision of the research questions as they would have become redundant when such reforms were made public, especially in Ghana. These consultation processes were more formal in Ghana and more informal in Tanzania, but they were very informative and had a tangible impact on the research plans, which were revised according to the feedback received. However, the research teams were always independent in deciding on the research methodology and in interpreting the results. The in-country dissemination of the results at the end of the first phase of the project informed the decisions to be made on the research plan for the second phase and provided the opportunity to discuss policy implications based on the results of the first phase. Because of this close collaboration and engagement with stakeholders, the results of the studies were widely disseminated in Ghana. Two of the main findings of the project were particularly considered by these stakeholders. According to the researcher:

“First, the study results showed that even though people registered with the NHIS they continued to pay out of pocket for health services. The reasons for this were delays in reimbursement by NHIS, escalating prices of drugs and medical products, low tariffs, lack of trust between providers and NHIA and inefficiencies. Secondly, the results showed that the current system of targeting the poor is not working properly, with more than half of people registered in the NHIS as indigents being in the non-poor socio-economic groups. These results contributed to inform decisions regarding the revision of the NHIA reimbursement tariffs, and to improve the identification of the poor to be exempted from paying the NHIS premium, in collaboration with the MGCSP” (Researcher 3).

In Tanzania, research was conducted to assess the effects of the public private partnership, referred as the Jazia Prime Vendor System (Jazia PVS), on improving access to medicines in the Dodoma and Morogoro regions in Tanzania. This is one of the reforms in the area of supply chain management taking place in the country. Results showed that a number of accountability mechanisms (inventory and financial auditing, close monitoring of standard operating procedures) implemented in conjunction with Jazia PVS contributed positively to the performance of Jazia PVS. Participants’ acceptability of Jazia PVS was influenced by the increased availability of essential medicines at the facilities, higher-order fulfilment rates and timely delivery of the consignment [ 18 , 19 , 20 ].

The findings from this study were disseminated during the national meeting attended by various stakeholders, including CAG members, government officials and policy-makers. In addition, the findings were used to inform the national scale-up of the Jazia PVS intervention as the government of Tanzania decided to scale up the Jazia PVS to all the 23 regions in 2018. Moreover, the results/manuscripts were published or submitted to peer-reviewed journals [ 18 , 19 , 20 ], enabling other countries intending to adopt such innovate public–private partnerships for improvement of the in-country pharmaceutical supply chain to learn from Jazia PVS in Tanzania.

Health impact assessment for engaging natural resource extraction projects in sustainable development in producer regions (HIA4SD)

In this r4d project, stakeholders were involved from the outset through their participation in the project launch meeting and in regular consortium meetings. The project is a collaboration between the Swiss Tropical and Public Health Institute (Swiss TPH), the Center for Development and Cooperation (NADEL) at the Swiss Federal Institute of Technology in Zurich/Switzerland and national research institutes, namely the Institut de Recherches en Sciences de la Santé in Burkina Faso, the University of Health and Allied Sciences in Ghana, the Centro de Investigação em Saúde de Manhiça in Mozambique and the Ifakara Health Institute in Tanzania [ 21 ]. The involvement of key stakeholders from the government, civil society, private sector and research community in an engaged dialogue from the beginning iss of central importance in this project, as in most cases mining is a highly politicized topic. To promote the immediate integration of research findings into policy, the project is organized into two streams, namely an “impact research stream” and a “governance stream”, that work in parallel. While the impact research stream is focused on evidence generation to support the uptake of health impact assessment (HIA) in Africa, the governance stream is focused on understanding the policy terrain and consequently the pathways that need to be utilized to support translation of the evidence into policy and practice. The second phase of the study is devoted to the dissemination of research findings into policy at the national and local levels, including capacity-building activities for national stakeholders. As the HIA4SD project examines operational questions of relevance for guiding both policy-making and decision-making, team members sought to regularly engage with and inform the national stakeholders. According to the researcher:

“Strategies employed to influence policy vary according to the country, but included regular stakeholder workshops, participation in a new national platform launched to discuss issues around mining in Mozambique, development of policy briefs, strengthened collaborations with national ministries of health, discussion of results and advocacy with policy makers, and conference presentation of findings” (Researcher 4).

In these two case examples, continuous stakeholder engagement was considered essential to translate and disseminate research evidence. Thus, beyond the stage of setting the objectives, contact with stakeholders was active and maintained on a regular basis through regular exchanges with stakeholder groups during workshops or meetings, which facilitated the dissemination and uptake of the research results. While the time and level of meaningful interaction varied across the countries and workshops, all meetings were well attended by participants from varied levels of government, MoHs, nongovernmental organizations and private industry, prompting spirited discussion and insight from these groups. All stakeholders were willing to attend these workshops as part of the scope of their professional duties.

S3: stakeholders engaged in participatory and transdisciplinary research approaches to co-produce knowledge and inform policy

In the two examples presented in this section, the research questions and approaches arose through community and stakeholder participation in the research and intervention design itself. The methodology adopted allowed them to engage, design research, act, share and sustain partnerships between the communities, the involved stakeholders and researchers [ 22 ]. These participatory research approaches facilitated grassroot-level policy and practice changes which were not researcher nor policy maker led, and that show promising approaches for developing culturally aligned solutions [ 23 ]. Policy makers at both the regional and national levels were invited to be part of the participatory research approach: they were interviewed during the initial stage, then the research results were presented and discussed with them; thereafter, we had several meetings to co-create potential interventions to address the identified problems, with the aim to directly engage in the research and intervention design itself in partnerships with the community stakeholders, including local leaders, and the researchers.

Surveillance and response to zoonotic diseases in Maya communities of Guatemala: a case for OneHealth

The research was embedded in a collaboration between the Universidad del Valle in Guatemala, the MoH, the Ministry of Animal Production and Health, the Maya Qéqchi’ Council of Elders, TIGO Telecommunications Foundation and the community development councils. The objective of this r4d programme was to set up integrated animal–human disease surveillance (OneHealth) in Maya communities in Guatemala. The research approach arose from a context of medical pluralism, where communities have access to and use two different medical systems: (1) the modern Western medical system and (2) traditional Maya medicine [ 24 ].

Researchers and community members collaborated at all stages of the research process, including the planning stage. Even before the grant proposal was finalized, researchers met with the communities that, should the funding come through, would be invited to participate in the research. According to the researchers:

“The project was set up through a transdisciplinary process, with academic and non-academic actors—including national, local and traditional authorities—involved in the problem through a collaborative design, analysis, dissemination and research translation. It was a co-producing transformative process—transferring knowledge between academic and non-academic stakeholders in plenary sessions and through group work. These meetings were held every year to continuously follow up the progress of the process” (Researcher 7).

The active engagement and collaboration by the community and stakeholders facilitated the acceptability of the study results and hence its dissemination, captured by the researchers as follows:

“The main result was that they allowed a frank discussion between Maya medical exponents in human–animal health and Western medicine, which allowed the patients and the animal holders to avoid the cognitive dissonance and so that the patients or the animal holders can choose freely what they want. Cognitive dissonance exists if one system dominates the other—or refutes the other” (Researcher 7).

“After all stakeholders discussed the research evidence produced jointly, an unprecedented process of collaboration between Government authorities and communities followed to develop three joint responses: a) education campaigns led by local teachers in tandem with the Ministry of Education, b) communication strategies at regional levels led by the Human and Animal Health authorities along with traditional Maya Ajilonel (medicine specialists), and c) a policy framework for producing a OneHealth approach led by Central Government authorities” (Researcher 8).

The process of mutual learning throughout the project produced a new level of awareness, facilitating culturally pertinent and socially robust responses that overcame a historical tendency of unilateral policy making based solely on Western values and preferences. As the project implemented a new approach to monitoring animal and human populations, the involvement of regional teams from the different ministries (Health, Livestock and Agriculture) throughout all the phases of methodological design, data collection, posterior data analysis and design of specific interventions for the local population (transformation of scientific results into actions for public health improvement) was essential to ensuring that the approach used secured the regional authorities’ commitment to defining new policies for immediate application in their territory. Accordingly, this also contributed towards the development of a OneHealth national strategy for Guatemala in which ministries start to cooperate to take up priority issues.

Addressing the double burden of disease: improving health systems for non-communicable and neglected tropical diseases (Community Health System Innovation [COHESION])

Together with three Swiss academic partners, this r4d project examined the challenges of a double burden of non-communicable and neglected tropical diseases at the primary healthcare level in vulnerable populations in Mozambique, Nepal and Peru. Community participation and co-creation were key elements of the project’s approach. The work conducted in Peru illustrates this approach:

“At the beginning, the people who were involved were respondents, but then they became active participants. So it was this active engagement and the changing of roles, giving feedback not just from the research responses but also from being involved in the process, which helped to design and create interventions together with the research team” (Researcher 5).

This participatory approach to co-creation actively sought a diverse range of stakeholders, including community members, primary healthcare workers, and regional and national health authorities. The co-creation approach to participatory research enables context-specific variation in methodological design, a critical element when studying three very different countries and health systems. Central to all aspects was a feedback loop whereby early findings were shared with research participants for further elaboration and iteration.

As active co-creators of the research process, local communities developed high levels of trust in the methodology and data, with the result that researchers achieved deeper “buy-in” which in turn is known to enhance the uptake of findings by decision-makers [ 25 ] as communities in which research is being undertaken play a central role in the decision-making process [ 26 ].

Challenges to research uptake in health policy identified by r4d researchers

During the interviews, r4d researchers identified several challenges to research utilization and uptake into policy. These challenges are summarized and highlighted in Table 3 .

Three key strategies identified for research uptake in policy and practice are described in this paper, namely: (S1) stakeholders directly engaged with and sought evidence from researchers; (S2) stakeholders were involved in the design and throughout the implementation of the research project; and (S3) stakeholders engaged in participatory and transdisciplinary research approaches to co-produce knowledge and inform policy. These strategies are in line with the overall objectives of the r4d projects, which are to generate scientific knowledge and research-based solutions to reduce poverty and global risks in LMICs, and also to offer national and international stakeholders integrated approaches to solving problems [ 5 ]. In the course of our synthesis work, we found that several lessons could be learned from the three strategies identified for research uptake in policy and practice.

S1: raising awareness of planned research to attract stakeholder involvement

The actual uptake of research findings in policy was most direct in the case of the first strategy (S1), in which IAS and WHO stakeholders were wanting new knowledge on HIV and same-day ART initiation, and were actively seeking new evidence on these specific topics. The findings published in peer-reviewed journals were then taken up by these stakeholders to update international policies and guidelines on rapid ART initiation [ 27 ]. This was also found in other studies, highlighting the importance of the timeliness and relevance of findings and the production of credible and trustworthy reports, among others, as key factors in promoting the use of research evidence in policy [ 2 , 28 ].

S2: sustainable collaborations in a supportive policy environment with stakeholder engagement from early on and throughout the research process

With regards to the second strategy (S2), we found that constant collaboration with an advisory and steering group composed of diverse stakeholders, including policy-makers, from early on promotes the uptake and use of research evidence. In line with findings from other studies [ 2 ], the experiences encountered in the r4d public health projects show that early involvement of stakeholders in the processes to identify the research problem and set the priorities facilitated the continuous exchange of information that might ultimately influence policy. The r4d project on social governance mechanisms in Ghana highlight that the evidence produced influenced policy documents (identification of the poor and tariff adjustments), but that frequent changes government officials made it difficult to maintain a close relationship between the researchers and the governmental agencies/policy stakeholders. From this, we draw the conclusion that research approaches need to be more adaptive and flexible to be successful in an unsupportive or unstable policy environment to ensure continuity in promoting the dissemination and uptake of research evidence in policy-making. One possible manner to secure this transformation is for researchers to apply for additional funding after the grant is finished. Other studies have also come to this conclusion, thereby demonstrating the key role of a supportive and effective policy environment that includes some degree of independence in governance and financing, strong links to stakeholders that facilitate trust and influence and also the capacity within the government workforce to process and apply policy advice developed by the research findings [ 29 ]. By involving stakeholders in the process of identifying research objectives and designing the project, as seen particularly in the r4d case studies on social health protection in Ghana and Tanzania and the HI4SD, but also in the HIV care cascade in Lesotho, the research approach responded to the need of locally led and demand-driven research in these countries, strengthening local research capacities and institutions, but also investing in research that is aligned with the national research priorities. As highlighted by other authors, advantages of this “demand-driven” approach is that it tailors research questions to local needs, helps to strengthen local individual and organizational capacities and provides a realized stringent framework on which a research project should deliver outcomes [ 30 , 31 ].

S3: co-creation and equal partnerships

The third strategy with a strong participatory approach, such as that adopted by two r4d projects, OneHealth in Guatemala and COHESION, demonstrates benefits to promoting co-learning as a way to minimize the impact of unequal power dynamics and to work effectively across the local policy landscape through equal partnerships. It also facilitates identifying solutions that are culturally pertinent, socially more robust and implementable.

The approaches of co-creation, equal participation and stakeholder involvement used in the research projects raise questions of ‘governance’, that is the way rules, norms and actions are structured, sustained and regulated by public and para-public actors to condition the engagement and impact of public involvement activities [ 32 , 33 ]. Through stakeholder involvement in setting the agenda and designing the research projects, as shown in the case studies on social protection in Ghana and Tanzania and the HI4SD project, but particularly in the two projects using a co-creation approach, the engagement of a range of stakeholders serves to make the health research systems a participaant in the endeavor that then has the capacity to promote changes in the healthcare system it aims to serve. By establishing a shared vision with a public involvement agenda and through the collaborative efforts of various stakeholders, as we found particularly in the co-creation approach, supportive health research systems are established. This leads to greater public advancement through collaborative actions, thereby tackling the stated problems of the health systems [ 34 ].

There were four key challenges mentioned by the respondents during the interviews to research uptake in policy making. The first was the necessary time investment by researchers to translate the result and develop policy advocacy products for the different audiences. This challenge is all the more difficult because research evidence and tangible products only become available towards the end of a research project, leaving only a short window of opportunity for exchange and engagement. There seems to be a need for wider discussion on the role of researchers in influencing policy. The concerns raised included whether influencing policy is actually a role for researchers and whether researchers have the right skills to be effective in persuasion or network formation [ 35 ]. Conversely, researchers may be in a good position to engage in the policy process if they enjoy finding solutions to complex problems while working with diverse and collaborative groups in partnerships [ 36 , 37 ]. The rationale for engaging in such a process needs to be clarified in advance: is the aim to frame an existing problem, or is it to simply measure the issues at stake and provide sound evidence according to an existing frame? Regarding the the former, how far should researchers go to be useful and influential in the policy process or to present challenges faced by vulnerable populations [ 37 ]? While fully engaging in the policy process may be the best approach for researchers to achieve credibility and impact, there may also be significant consequences, such as the risk of political interests undermining the methodological rigour of academic research (being considered as academic ‘lightweight’ among one’s peer group) [ 38 , 39 , 40 , 41 ]. For researchers there is also considerable opportunity costs because engaging in the policy-influencing process is a time-consuming activity [ 35 ], with no clear guarantee of the impact of success [ 37 ]. It is therefore crucial to consider the investment and overall time researchers may have to spend to engage [ 35 ], and how this time and investment can be distributed between actual research and the production of outreach products, such as policy briefs, presentation of research findings as policy narratives [ 35 ] and the setting-up of alliances, building of networks and exploitation of windows of opportunity for policy change [ 37 ].

The second challenge included the issue of scale and objectivity, as most of the projects are not scaled or national-level studies and thus are highly context specific. The difficulty to measure the contributions of a single research project or study in terms of policy outcomes was also highlighted, particularly in view of the different understandings among researchers and funders on the possible policy impacts of the research, which can range from guiding policy-makers to understand a situation or problem (awareness raising) to influencing a particular course of action by establishing new or revising existing policies. This has also been emphasized in the Evidence Peter Principle [ 42 ], showing that single studies are often inappropriately used to make global policy statements for which they are not suitable. To make global policy statements, an assessment of the global evidence in systematic reviews is needed [ 42 , 43 ].

The third challenge mentioned was the frequent changes in staff at the governmental level, which demanded continuous interactions between r4d researchers and stakeholders, highlighting the need for more adaptive and flexible research approaches. These should include a thorough analytical process prior to implementation in historical, sociopolitical and economic aspects, power differentials and context; backward planning exercises to check assumptions; and conflict transformation and negotiation skills in order to be able to constantly adapt to changing contexts. In line with our research findings, when researchers make the time investment needed to engage in the policy-influencing process, an opportunity is provided to getting know the involved stakeholders better and improve their understanding of the policy world in practice, but also to build diverse and longer-term networks [ 37 , 44 ] and to identify policy problems and the appropriate stakeholders to work with [ 45 , 46 ]. Engaging a diverse range of stakeholders through co-designing the research is widely held to be practically the best way to guarantee the uptake and use of evidence in policy through a more dynamic research approach [ 47 ]. However, the development of networks and contacts for collaboration, as well as the skills to do so, takes time and effort and is an ongoing process [ 48 ], factors which need to be acknowledged more widely.

Lastly, the fourth challenge related to research uptake was the diverging interests between researchers, research funding bodies and stakeholders. Time was identified as a limiting factor from the perspective of the design of the research project. Most research projects, including the r4d projects, are funded for 3–4 years [ 5 ]. It takes a considerable amount of time to generate new research results, and often these are more likely to be produced for further use at the end of a project. If researchers should engage more fully in the policy process to secure meaningful impact, it is critical to discuss the extent to which they have the skills, resources and institutional support to do so [ 37 ], as well as how projects could be set up differently. This could be done either by the funders in providing the necessary support that allows researchers to have the means to impact policy, or by the researchers in the design of their project to take on board the different strategies to influence evidence use and uptake. In moving forward, defining shared goals from the outset between funders and the researchers might translate to more achievable milestones in terms of which policy issue, theme or process a research project aims to change in order to effectively influence policy [ 49 ]. This would help to identify the resources and budget needed by the funders in order for the researchers to engage with more resources over a longer time span in this process.

Limitations

Interviews were limited to researchers of the r4d projects and did not include local stakeholders. Therefore, the synthesis work, including the analysis and results, reflects solely the perspective of researchers. We are aware that had we included a range of stakeholders, including policy-makers, in the sample, we would have potentially been able to identify additional factors relating to social, cultural and political barriers to the use and uptake of research findings in politics and practice. However, constraints such as access to local stakeholders, language barriers and time zones drove our decision to focus on researchers. A future synthesis effort would need to include the other voices.

There is ever growing awareness of how critical it is to close the gap between policy-makers, practitioners and researchers. Using the researchers’ perspectives, in this article we give insight into three different strategies that can facilitate this process, with the first strategy requiring proactive searching for the latest findings on the part of well-informed policy-makers, the second requiring researchers to take steps to ensure an active exchange of ideas and information with diverse stakeholders when designing the research project and ensuring the latter’s involvement throughout; and the third using a transdisciplinary and/or co-creation approach to establish equal partnerships and trust among all involved stakeholders.

The five case studies reported here also show some of the difficulties that prevail for research to be taken up into policy and practice, despite everyone’s best intentions and efforts. Researchers may not always be best placed for communication, dissemination and advocacy work, all activities which are very time intensive or become important only towards the end of a research project when clear and high-quality evidence is produced. Moreover, it takes a strong body of evidence, advocacy and coalition building with appropriate stakeholders to influence policy, and then a further major effort of resources to see policy followed through into practice. It is through experiences such as this synthesis initiative that precious insights and learning can be gained for the common good of all involved moving forward, and it is crucial that funders continue to support and/or adapt their funding schemes to ensure some of these strategies are implemented.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Antiretroviral therapy

Country Advisory Group

Community Health System Innovation

Health impact assessment

Health impact assessment for engaging natural resource extraction projects in sustainable development in producer regions

Human immunodeficiency virus

International Aids Society

Jazia Prime Vendor System

Low- and middle-income countries

Ministry of Health

Ministry of Gender Children and Social Protection

Center for Development and Cooperation at the Swiss Federal Institute of Technology

National Health Insurance Authority

National Health Insurance Scheme

Swiss Programme for Research on Global Issues for Development

Swiss Agency for Development and Cooperation

Swiss National Science Foundation

Swiss Tropical and Public Health Institute

Universal health coverage

World Health Organization

Court J, Young J. Bridging research and policy in international development: an analytical and practical framework. J Dev Pract. 2006;16(1):85–90.

Article   Google Scholar  

Uzochukwu B, Onwujekwe O, Mbachu C, Okwuosa C, Etiaba E, Nyström ME, Gilson L. The challenge of bridging the gap between researchers and policy makers: experiences of a Health Policy Research Group in engaging policy makers to support evidence informed policy making in Nigeria. Glob Health. 2016;12(1):67.

Di Ruggiero E, Edwards N. The interplay between participatory health research and implementation research: Canadian research funding perspectives. Biomed Res Int. 2018;2018:1519402.

Rau H, Goggins G, Fahy F. From invisibility to impact: recognising the scientific and societal relevance of interdisciplinary sustainability research. Res Policy. 2018;47(1):266–76.

Swiss National Science Foundation: Swiss Programme for Research on Global Issues for Development (r4d programme). http://www.r4d.ch/r4d programme/portrait . Accessed 20 Jan 2020.

United Nations: Sustainable Development Goals. https://sustainabledevelopment.un.org/ . Accessed 17 Dec 2019.

Shroff ZC, Javadi D, Gilson L, Kang R, Ghaffar A. Institutional capacity to generate and use evidence in LMICs: current state and opportunities for HPSR. Health Res Policy Syst. 2017;15(1):94.

McKee M. Bridging the gap between research and policy and practice Comment on “CIHR health system impact fellows: reflections on ‘driving change’ within the health system.” Int J Health Policy Manag. 2019;8(9):557–9.

World Health Organization. Sound choices: enhancing capacity for evidence-informed health policy. In: Bennett S, Green A, editors. Geneva: WHO; 2007.

Stoker G, Evans M. Evidence-based policy making in the social sciences: methods that matter. Bristol: Policy Press; 2016.

Book   Google Scholar  

Brinkerhoff DW, Crosby B. Managing policy reform: concepts and tools for decision-makers in developing and transitioning countries. Sterling: Kumarian Press; 2002.

Google Scholar  

Hardee KFI, Boezwinkle J, Clark B. A framework for analyzing the components of family planning, reproductive health, maternal health, and HIV/AIDS policies. Wilmette: The Policy Circle; 2004.

Mayring P. Qualitative Inhaltsanalyse. In: Mey G, Mruck K, editors. Handbuch qualitative Forschung in der Psychologie. Wiesbaden: Springer; 2010. pp. 601–613.

Mayring P, Fenzl T. Qualitative Inhaltsanalyse. In: Baur N, Blasius J, editors. Handbuch Methoden der empirischen Sozialforschung. Wiesbaden: Springer; 2014. pp. 543–556.

Rose DC, Amano T, González-Varo JP, Mukherjee N, Robertson RJ, Simmons BI, Wauchope HS, Sutherland WJ. Calling for a new agenda for conservation science to create evidence-informed policy. Biol Conserv. 2019;238:108222.

Shiffman J, Smith S. Generation of political priority for global health initiatives: a framework and case study of maternal mortality. Lancet. 2007;370(9595):1370–9.

Labhardt ND, Ringera I, Lejone TI, Klimkait T, Muhairwe J, Amstutz A, Glass TR. Effect of offering same-day ART vs usual health facility referral during home-based HIV testing on linkage to care and viral suppression among adults with HIV in Lesotho: the CASCADE randomized clinical trial. JAMA. 2018;319(11):1103–12.

Kuwawenaruwa A, Wyss K, Wiedenmayer K, Metta E, Tediosi F. The effects of medicines availability and stock-outs on household’s utilization of healthcare services in Dodoma region, Tanzania. Health Policy Plan. 2020;35(3):323–33.

Kuwawenaruwa A TF, Metta E, Obrist B, Wiedenmayer K, Msamba V, Wyss K. Acceptability of a prime vendor system in public healthcare facilities in Tanzania. Int J Health Policy Manag. 2020. (in press).

Kuwawenaruwa ATF, Obrist B, Metta E, Chiluda F, Wiedenmayer K, Wyss K. The role of accountability in the performance of Jazia prime vendor system in Tanzania. J Pharm Policy Pract. 2020;2020(13):25.

Farnham A, Cossa H, Dietler D, Engebretsen R, Leuenberger A, Lyatuu I, Nimako B, Zabre HR, Brugger F, Winkler MS. Investigating health impacts of natural resource extraction projects in Burkina Faso, Ghana, Mozambique, and Tanzania: protocol for a mixed methods study. JMIR Res Protoc. 2020;9(4):e17138.

Beran D, Lazo-Porras M, Cardenas MK, Chappuis F, Damasceno A, Jha N, Madede T, Lachat S, Perez Leon S, Aya Pastrana N, Pesantes MA, Singh SB, Sharma S, Somerville C, Suggs LS, Miranda JJ. Moving from formative research to co-creation of interventions: insights from a community health system project in Mozambique, Nepal and Peru. BMJ Gob Health. 2018;3(6):e001183.

Mertens DM. Advancing social change in South Africa through transformative research. S Afr Rev Sociol. 2016;47(1):5–17.

Berger-González M, Stauffacher M, Zinsstag J, Edwards P, Krütli P. Transdisciplinary research on cancer-healing systems between biomedicine and the Maya of Guatemala: a tool for reciprocal reflexivity in a multi-epistemological setting. Qual Health Res. 2016;26(1):77–91.

Theron LC. Using research to influence policy and practice: the case of the pathways-to-resilience study (South Africa). In: Abubakar A, van de Vijver FJR, editors. Handbook of applied developmental science in sub-Saharan Africa. New York: Springer; 2017. pp. 373–87.

Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health. 2006;60(10):854–7.

WHO. Guidelines for managing advanced HIV disease and rapid initiation of antiretroviral therapy. Geneva: World Health Organization; 2017.

Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health policymaking (STP) 3: setting priorities for supporting evidence-informed policymaking. Health Res Policy Syst. 2009;7(S1):I3.

Bennett S, Corluka A, Doherty J, Tangcharoensathien V, Patcharanarumol W, Jesani A, Kyabaggu J, Namaganda G, Hussain AMZ, de-Graft Aikins A. Influencing policy change: the experience of health think tanks in low- and middle-income countries. Health Policy Plan. 2011;27(3):194–203.

Kok MO, Gyapong JO, Wolffers I, Ofori-Adjei D, Ruitenberg EJ. Towards fair and effective North–South collaboration: realising a programme for demand-driven and locally led research. Health Res Policy Syst. 2017;15(1):96.

Wolffers I, Adjei S. Research-agenda setting in developing countries. Lancet. 1999;353(9171):2248–9.

Article   CAS   Google Scholar  

Dodgson R, Lee K, Drager N. Global health governance: a conceptual review. London: Centre on Global Change and Health, London School of Hygiene & Tropical Medicine/World Health Organization; 2018.

Saltman RB, Ferroussier-Davis O. The concept of stewardship in health policy. Bull World Health Organ. 2000;78(6):732–9.

CAS   PubMed   PubMed Central   Google Scholar  

Miller FA, Patton SJ, Dobrow M, Marshall DA, Berta W. Public involvement and health research system governance: a qualitative study. Health Res Policy Syst. 2018;16(1):87.

Lloyd J. Should academics be expected to change policy? Six reasons why it is unrealistic for research to drive policy change. https://blogs.lse.ac.uk/impactofsocialsciences/2016/05/25/should-academics-be-expected-to-change-policy-six-reasons-why-it-is-unrealistic/ . Accessed 28 May 2020.

Petes LE, Meyer MD. An ecologist’s guide to careers in science policy advising. Front Ecol Environ. 2018;16(1):53–4.

Oliver KCP. The dos and don’ts of influencing policy: a systematic review of advice to academics. Palgrave Commun. 2019;5(1):21.

Hutchings JA, Stenseth NC. Communication of science advice to government. Trends Ecol Evol. 2016;31(1):7–11.

Maynard A. Is public engagement really career limiting?. https://www.timeshighereducation.com/blog/public-engagement-really-career-limiting . Accessed 27 May 2020.

Haynes AS, Derrick GE, Chapman S, Redman S, Hall WD, Gillespie J, Sturk H. From “our world” to the “real world”: exploring the views and behaviour of policy-influential Australian public health researchers. Soc Sci Med. 2011;72(7):1047–55.

Crouzat E, Arpin I, Brunet L, Colloff MJ, Turkelboom F, Lavorel S. Researchers must be aware of their roles at the interface of ecosystem services science and policy. Ambio. 2018;47(1):97–105.

White H. The Evidence Peter Principle: the misuse and abuse of evidence Reflections on the evidence architecture. https://www.campbellcollaboration.org/blog/the-evidence-peter-principle-the-misuse-and-abuse-of-evidence.html?utm_source=Campbell+Collaboration+newsletters&utm_campaign=4dfa01ec7d-Newsletter+September+2019&utm_medium=email&utm_term=0_ab55bacb0c-4dfa01ec7d-199138457 . Accessed 29 Jan 2020.

Caird J, Sutcliffe K, Kwan I, Dickson K, Thomas J. Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evid Policy J Res Debate Pract. 2015;11(1):81–97.

Evans MC, Cvitanovic C. An introduction to achieving policy impact for early career researchers. Palgrave Commun. 2018;4(1):88.

Echt L. Context matters:” a framework to help connect knowledge with policy in government institutions. LSE Impact Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2017/12/19/context-matters-a-framework-to-help-connect-knowledge-with-policy-in-government-institutions/ . Accessed 13 July 2020.

Lucey JM, Palmer G, Yeong KL, Edwards DP, Senior MJM, Scriven SA, Reynolds G, Hill JK. Reframing the evidence base for policy-relevance to increase impact: a case study on forest fragmentation in the oil palm sector. J Appl Ecol. 2017;54(3):731–6.

Green D: How academics and NGOs can work together to influence policy: insights from the InterAction report. LSE Impact blog https://blogs.lse.ac.uk/impactofsocialsciences/2016/09/23/how-academics-and-ngos-can-work-together-to-influence-policy-insights-from-the-interaction-report/ . Accessed 13 July 2020.

Boaz A, Hanney S, Borst R, O’Shea A, Kok M. How to engage stakeholders in research: design principles to support improvement. Health Res Policy Syst. 2018;16(1):60.

Tilley HSL, Rea J, Ball L, Young J. 10 things to know about how to influence policy with research. London: Overseas Development Institute; 2017.

Download references

Acknowledgements

The authors would like to acknowledge the contribution of Dr Claudia Rutte from the r4d programme/SNSF for her inputs to the history and background of the r4d programme.

The r4d synthesis initiative is implemented by the Swiss Tropical and Public Health Institute, which funded the costs of publishing this paper.

Author information

Authors and affiliations.

Swiss Tropical and Public Health Institute, Basel, Switzerland

Séverine Erismann, Andrea Leuenberger, Andrea Farnham, Monica Berger Gonzalez de White, Niklaus Daniel Labhardt, Fabrizio Tediosi, August Kuwawenaruwa, Jakob Zinsstag, Kaspar Wyss & Helen Prytherch

University of Basel, Basel, Switzerland

CRONICAS Centre of Excellence in Chronic Diseases, Universidad Peruana Cayetano Heredia, Lima, Peru

Maria Amalia Pesantes

Division of Tropical and Humanitarian Medicine, University of Geneva and Geneva University Hospitals, Geneva, Switzerland

David Beran

Centro de Estudios en Salud, Universidad del Valle de Guatemala, Guatemala, Guatemala

Monica Berger Gonzalez de White

Department of Infectious Diseases and Hospital Epidemiology, University Hospital Basel, Basel, Switzerland

Niklaus Daniel Labhardt

School of Public Health, College of Health Sciences, University of Ghana, Accra, Ghana

Patricia Akweongo

Ifakara Health Institute, Plot 463, Kiko Avenue Mikocheni, Dar es Salaam, Tanzania

August Kuwawenaruwa

Swiss Federal Institute of Technology Zurich, Zurich, Switzerland

Fritz Brugger

Gender Centre, Graduate Institute of International and Development Studies, Geneva, Switzerland

Claire Somerville

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the writing of this manuscript. Each author contributed with synthesizing their project experiences and with the discussion and recommendations. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Séverine Erismann or Helen Prytherch .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Erismann, S., Pesantes, M.A., Beran, D. et al. How to bring research evidence into policy? Synthesizing strategies of five research projects in low-and middle-income countries. Health Res Policy Sys 19 , 29 (2021). https://doi.org/10.1186/s12961-020-00646-1

Download citation

Received : 15 July 2020

Accepted : 15 October 2020

Published : 06 March 2021

DOI : https://doi.org/10.1186/s12961-020-00646-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Evidence-based policy-making
• Research for development

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Research Engagement with Policy Makers: a practical guide to writing policy briefs

Effective communication between academics and policy makers plays an important role in informing political decision making and creating impact for researchers. Policy briefs are short evidence summaries written by researchers to inform the development or implementation of policy. This guide has been developed to support researchers to write effective policy briefs. It is jointly produced by the NIHR Policy Research Unit in Behavioural Science (BehSciPRU) and the UCL Centre for Behaviour Change (CBC). It has been written in consultation with policy advisers and synthesises current evidence and expert opinion on what makes an effective policy brief. It is for any researcher who wishes to increase the impact of their work by activity that may influence the process of policy formation, implementation or evaluation. Whilst the guide has been written primarily for a UK audience, it is hoped that it will be useful to researchers in other countries.

Related Papers

Toronto, Ontario: Policy Bench, Fraser Mustard Institute of Human Development, University of Toront

Sima Sajedinejad

Objective: This toolkit provides a guide for researchers on the development of effective policy briefs to communicate research findings to policymakers to support evidence-informed decision-making. Key Points: • A policy brief is one component of a comprehensive policy impact plan. It typically provides a concise summary of a specific issue, the policy options to address it, and some recommendations for action. • Policy briefs are often based on a larger evidence synthesis or research study which provides more technical details. • Policy briefs generally target an informed, non-specialist audience. Therefore, language should be clear and concise; academic and technical jargon should be drilled down into plain and nonacademic language. • There are two types of policy briefs: an objective brief that gives balanced information about the policy options and allows policymakers to make their own decision, and an advocacy brief favouring one suggested policy option. • The process of developing a policy brief includes planning, writing, designing, and revising, and disseminating. Typically, visual aids are used to present the results in a simplified way to capture readers' attention with clear takeaways. • The outline of a policy brief usually includes the following sections: title, background, research results, policy options, implications, recommendations, and conclusions. • Results and proposed solutions or policy options to address the issue should be presented in a neutral, objective manner and should consider important dimensions such as feasibility, costs, and other pros and cons. Make sure your argument flows clearly based on appropriate structure and data. • The implications section draws from the results of the research to discuss the broader implications of the findings from a policy perspective. • Recommendations should be easy to find, clear and easy to understand, short, specific, realistic, relevant, attainable, and usually start with action words.

Kizito Ndihokubwayo

Lindsey Pike

Access to reliable and timely information ensures that decision-makers can operate effectively. The motivations and challenges of parliamentarians and policy-makers in accessing evidence have been well documented in the policy literature. However, there has been little focus on research-providers. Understanding both the demand- and the supply-side of research engagement is imperative to enhancing impactful interactions. This study reports on a recent online consultation with research professionals on their policy experience, including motivations and barriers to engage with decision-makers.

Policy Design and Practice

Shannon Guillot-Wright, PhD

Bridging the "Know–Do" Gap: Knowledge brokering to improve child wellbeing

Meredith Edwards

Health Economics & Decision Science (HEDS) Discussion Paper Series

Erika E Atienzo

Over the last two decades, there has been an emphasis on the concept of evidence-based policy. However, evidence-based policy remains a major challenge and a gap exists in the systematic translation of scientific knowledge into policies. The awareness of this evidence-policy gap has led to a proliferation of research. As the demand for evidence-informed policy-making escalates, so does the need to unveil the mechanisms by which we can influence the process of research uptake. In this paper, we present a protocol for a systematic review. We aim to conduct an umbrella review/overview of reviews about factors affecting the use of research by policy-makers and/or decision-makers in health, education and social services areas. The results of this review could contribute to improving the utilisation of research in the policy-making process, by identifying factors that are most important in influencing the uptake of research by policy-makers.

International Journal of Behavioral Nutrition and Physical Activity

Heather Yeatman

Australia and New Zealand Health Policy

Anthony Zwi

Tamlin Gorter

Heaven Crawley

The concept of ‘evidence-based policy’ making has become something of a mantra within government circles. Within academia too, there is a growing emphasis on the ‘relevance’ of research to ‘real world’ issues and problems. For those of us who have been directly engaged in what might be described as ‘policy’ or ‘applied’ research for many years, this shift in emphasis is welcome and very much overdue. But the increased recognition afforded to research evidence in the policy making process belies a complex and difficult relationship between academics and policy-makers whose modus operandi is very different and who may have widely divergent motivations, objectives, methods and measures of ‘success’. Attempts to bring these two worlds together are not without their problems, which are explored in this short chapter.

RELATED PAPERS

Bárbara Barbosa

Technical Physics

Victor Kats

mahir M ozmen

Konstantina Chatzara

Nicola Mitchell

Jurnal Riset dan Inovasi Pembelajaran

ummiati rauf

Ciencia E Cultura

Dilce Rossetti

Electronic Journal of Electronic Government

Joseph Feller

Research Outreach

Stefanie Harsch

Ishak Turan

Journal of Polymer Science Part B: Polymer Physics

Frontiers in Sustainable Food Systems

Marilyn Sitaker

Dieli Vesaro Palma

Lecture Notes in Computer Science

Ana Maria Paulino Perez

Molecular pharmacology

Thorsten Keller

INTERNATIONAL JOURNAL OF ENDORSING HEALTH SCIENCE RESEARCH (IJEHSR)

Czech Journal of International Relations

Jaroslav Weinfurter

Proceedings of Fourth IEEE International Conference on Software Reuse

David Garlan

Cellular Physiology and Biochemistry

Tony Muslin

Sustainable environment research

Askwar Hilonga

Eduardo Calmon de Almeida Cezar

Bijoyendra Narayan

Jurnal Riset Veteriner Indonesia (Journal of The Indonesian Veterinary Research)

Srihadi Agungpriyono

hgkfgj rstgef

International journal of environmental research and public health

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

The video above is decorative and not illustrative.

Policy Topics

• Social Policy
• Gender, Race & Identity

The past shapes the present when it comes to Southern roots, race, and marriage trends

Professor Deirdre Bloome and coauthor Garrett Pace from the University of Nevada found that Black and white people whose grandparents grew up in the southern U.S. are more likely to marry than people with geographic lineages elsewhere in the U.S.

• Globalization

When it comes to global economic governance, less can be more if you want increased cooperation, stability, and equitable growth

An unexpected benefit from globalization: Improved corporate tax collection for developing countries

• Environment & Energy

Jane Gilbert MPA 1994: The heat is on to cool down Miami-Dade County

Patterns of benzodiazepine initiation among older acute ischemic stroke survivors (p1-10.006).

Objective: Describe temporal changes in outpatient Benzodiazepine (BDZ) initiation rates following Acute Ischemic Stroke (AIS) discharge among Medicare beneficiaries. Background: Despite recommendati

Evaluating the Concordance Between International Classification of Diseases, Tenth Revision Code, and Stroke Severity as Measured by the National Institutes of Health Stroke Scale (P1-5.013)

Objective: To use the National Institutes of Health Stroke Scale (NIHSS) as reported in Paul Coverdell Stroke Registry (PCSR) to assess the accuracy of the ICD-10-based NIHSS as a measure of Acute Isc

• Development & Economic Growth
• Poverty, Inequality & Opportunity

Who benefits? Experimental evidence on Saudi women’s take-up of unemployment assistance

Does a woman’s take-up of government benefits vary with her perception of how they will be shared within the household?

• Science, Technology & Data
• Education, Training & Labor

Auditing the Use of Language Models to Guide Hiring Decisions

Regulatory efforts to protect against algorithmic bias have taken on increased urgency with rapid advances in large language models (LLMs), which are machine learning models that can achieve performan

Watch & Listen

• Human Rights

Indigenous Sovereignty and Human Rights in the United States

• Public Finance

The Ghost Budget: How U.S. war spending went rogue, wasted billions, and how to fix it

A Human Rights-Based Approach to Mental Health

Human Rights and Indigenous Rights in New Zealand

Get smart & reliable public policy insights right in your inbox. 

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 15 April 2024

Revealed: the ten research papers that policy documents cite most

• Dalmeet Singh Chawla 0

Dalmeet Singh Chawla is a freelance science journalist based in London.

You can also search for this author in PubMed   Google Scholar

Policymakers often work behind closed doors — but the documents they produce offer clues about the research that influences them. Credit: Stefan Rousseau/Getty

When David Autor co-wrote a paper on how computerization affects job skill demands more than 20 years ago, a journal took 18 months to consider it — only to reject it after review. He went on to submit it to The Quarterly Journal of Economics , which eventually published the work 1 in November 2003.

Autor’s paper is now the third most cited in policy documents worldwide, according to an analysis of data provided exclusively to Nature . It has accumulated around 1,100 citations in policy documents, show figures from the London-based firm Overton (see ‘The most-cited papers in policy’), which maintains a database of more than 12 million policy documents, think-tank papers, white papers and guidelines.

“I thought it was destined to be quite an obscure paper,” recalls Autor, a public-policy scholar and economist at the Massachusetts Institute of Technology in Cambridge. “I’m excited that a lot of people are citing it.”

The most-cited papers in policy

Economics papers dominate the top ten papers that policy documents reference most.

Data from Sage Policy Profiles as of 15 April 2024

The top ten most cited papers in policy documents are dominated by economics research. When economics studies are excluded, a 1997 Nature paper 2 about Earth’s ecosystem services and natural capital is second on the list, with more than 900 policy citations. The paper has also garnered more than 32,000 references from other studies, according to Google Scholar. Other highly cited non-economics studies include works on planetary boundaries, sustainable foods and the future of employment (see ‘Most-cited papers — excluding economics research’).

These lists provide insight into the types of research that politicians pay attention to, but policy citations don’t necessarily imply impact or influence, and Overton’s database has a bias towards documents published in English.

Interdisciplinary impact

Overton usually charges a licence fee to access its citation data. But last year, the firm worked with the London-based publisher Sage to release a free web-based tool that allows any researcher to find out how many times policy documents have cited their papers or mention their names. Overton and Sage said they created the tool, called Sage Policy Profiles, to help researchers to demonstrate the impact or influence their work might be having on policy. This can be useful for researchers during promotion or tenure interviews and in grant applications.

Autor thinks his study stands out because his paper was different from what other economists were writing at the time. It suggested that ‘middle-skill’ work, typically done in offices or factories by people who haven’t attended university, was going to be largely automated, leaving workers with either highly skilled jobs or manual work. “It has stood the test of time,” he says, “and it got people to focus on what I think is the right problem.” That topic is just as relevant today, Autor says, especially with the rise of artificial intelligence.

Most-cited papers — excluding economics research

When economics studies are excluded, the research papers that policy documents most commonly reference cover topics including climate change and nutrition.

Walter Willett, an epidemiologist and food scientist at the Harvard T.H. Chan School of Public Health in Boston, Massachusetts, thinks that interdisciplinary teams are most likely to gain a lot of policy citations. He co-authored a paper on the list of most cited non-economics studies: a 2019 work 3 that was part of a Lancet commission to investigate how to feed the global population a healthy and environmentally sustainable diet by 2050 and has accumulated more than 600 policy citations.

“I think it had an impact because it was clearly a multidisciplinary effort,” says Willett. The work was co-authored by 37 scientists from 17 countries. The team included researchers from disciplines including food science, health metrics, climate change, ecology and evolution and bioethics. “None of us could have done this on our own. It really did require working with people outside our fields.”

Sverker Sörlin, an environmental historian at the KTH Royal Institute of Technology in Stockholm, agrees that papers with a diverse set of authors often attract more policy citations. “It’s the combined effect that is often the key to getting more influence,” he says.

Has your research influenced policy? Use this free tool to check

Sörlin co-authored two papers in the list of top ten non-economics papers. One of those is a 2015 Science paper 4 on planetary boundaries — a concept defining the environmental limits in which humanity can develop and thrive — which has attracted more than 750 policy citations. Sörlin thinks one reason it has been popular is that it’s a sequel to a 2009 Nature paper 5 he co-authored on the same topic, which has been cited by policy documents 575 times.

Although policy citations don’t necessarily imply influence, Willett has seen evidence that his paper is prompting changes in policy. He points to Denmark as an example, noting that the nation is reformatting its dietary guidelines in line with the study’s recommendations. “I certainly can’t say that this document is the only thing that’s changing their guidelines,” he says. But “this gave it the support and credibility that allowed them to go forward”.

Broad brush

Peter Gluckman, who was the chief science adviser to the prime minister of New Zealand between 2009 and 2018, is not surprised by the lists. He expects policymakers to refer to broad-brush papers rather than those reporting on incremental advances in a field.

Gluckman, a paediatrician and biomedical scientist at the University of Auckland in New Zealand, notes that it’s important to consider the context in which papers are being cited, because studies reporting controversial findings sometimes attract many citations. He also warns that the list is probably not comprehensive: many policy papers are not easily accessible to tools such as Overton, which uses text mining to compile data, and so will not be included in the database.

The top 100 papers

“The thing that worries me most is the age of the papers that are involved,” Gluckman says. “Does that tell us something about just the way the analysis is done or that relatively few papers get heavily used in policymaking?”

Gluckman says it’s strange that some recent work on climate change, food security, social cohesion and similar areas hasn’t made it to the non-economics list. “Maybe it’s just because they’re not being referred to,” he says, or perhaps that work is cited, in turn, in the broad-scope papers that are most heavily referenced in policy documents.

As for Sage Policy Profiles, Gluckman says it’s always useful to get an idea of which studies are attracting attention from policymakers, but he notes that studies often take years to influence policy. “Yet the average academic is trying to make a claim here and now that their current work is having an impact,” he adds. “So there’s a disconnect there.”

Willett thinks policy citations are probably more important than scholarly citations in other papers. “In the end, we don’t want this to just sit on an academic shelf.”

doi: https://doi.org/10.1038/d41586-024-00660-1

Autor, D. H., Levy, F. & Murnane, R. J. Q. J. Econ. 118 , 1279–1333 (2003).

Article   Google Scholar  

Costanza, R. et al. Nature 387 , 253–260 (1997).

Willett, W. et al. Lancet 393 , 447–492 (2019).

Article   PubMed   Google Scholar  

Steffen, W. et al. Science 347 , 1259855 (2015).

Rockström, J. et al. Nature 461 , 472–475 (2009).

Download references

Reprints and permissions

Related Articles

We must protect the global plastics treaty from corporate interference

World View 17 APR 24

UN plastics treaty: don’t let lobbyists drown out researchers

Editorial 17 APR 24

Smoking bans are coming: what does the evidence say?

News 17 APR 24

CERN’s impact goes way beyond tiny particles

Spotlight 17 APR 24

The economic commitment of climate change

Article 17 APR 24

Last-mile delivery increases vaccine uptake in Sierra Leone

Article 13 MAR 24

Head of the Thrust of Robotics and Autonomous Systems

Reporting to the Dean of Systems Hub, the Head of ROAS is an executive assuming overall responsibility for the academic, student, human resources...

Guangzhou, Guangdong, China

The Hong Kong University of Science and Technology (Guangzhou)

Head of Biology, Bio-island

Head of Biology to lead the discovery biology group.

BeiGene Ltd.

Research Postdoctoral Fellow - MD (Cardiac Surgery)

Houston, Texas (US)

Baylor College of Medicine (BCM)

Director of Mass Spectrometry

Loyola University Chicago, Stritch School of Medicine (SSOM) seeks applicants for a full-time Director of Mass Spectrometry.

Chicago, Illinois

Loyola University of Chicago - Cell and Molecular Physiology Department

Associate or Senior Editor, Nature Biomedical Engineering

Associate Editor or Senior Editor, Nature Biomedical Engineering Location: London, Shanghai and Madrid — Hybrid office and remote working Deadline:...

London (Central), London (Greater) (GB)

Springer Nature Ltd

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Short report
• Open access
• Published: 12 April 2024

A modified action framework to develop and evaluate academic-policy engagement interventions

• Petra Mäkelä   ORCID: orcid.org/0000-0002-0938-1175 1 ,
• Annette Boaz   ORCID: orcid.org/0000-0003-0557-1294 2 &
• Kathryn Oliver   ORCID: orcid.org/0000-0002-4326-5258 1  

Implementation Science volume  19 , Article number:  31 ( 2024 ) Cite this article

594 Accesses

19 Altmetric

Metrics details

There has been a proliferation of frameworks with a common goal of bridging the gap between evidence, policy, and practice, but few aim to specifically guide evaluations of academic-policy engagement. We present the modification of an action framework for the purpose of selecting, developing and evaluating interventions for academic-policy engagement.

We build on the conceptual work of an existing framework known as SPIRIT (Supporting Policy In Health with Research: an Intervention Trial), developed for the evaluation of strategies intended to increase the use of research in health policy. Our aim was to modify SPIRIT, (i) to be applicable beyond health policy contexts, for example encompassing social, environmental, and economic policy impacts and (ii) to address broader dynamics of academic-policy engagement. We used an iterative approach through literature reviews and consultation with multiple stakeholders from Higher Education Institutions (HEIs) and policy professionals working at different levels of government and across geographical contexts in England, alongside our evaluation activities in the Capabilities in Academic Policy Engagement (CAPE) programme.

Our modifications expand upon Redman et al.’s original framework, for example adding a domain of ‘Impacts and Sustainability’ to capture continued activities required in the achievement of desirable outcomes. The modified framework fulfils the criteria for a useful action framework, having a clear purpose, being informed by existing understandings, being capable of guiding targeted interventions, and providing a structure to build further knowledge.

The modified SPIRIT framework is designed to be meaningful and accessible for people working across varied contexts in the evidence-policy ecosystem. It has potential applications in how academic-policy engagement interventions might be developed, evaluated, facilitated and improved, to ultimately support the use of evidence in decision-making.

Peer Review reports

Contributions to the literature

There has been a proliferation of theories, models and frameworks relating to translation of research into practice. Few specifically relate to engagement between academia and policy.

Challenges of evidence-informed policy-making are receiving increasing attention globally. There is a growing number of academic-policy engagement interventions but a lack of published evaluations.

This article contributes a modified action framework that can be used to guide how academic-policy engagement interventions might be developed, evaluated, facilitated, and improved, to support the use of evidence in policy decision-making.

Our contribution demonstrates the potential for modification of existing, useful frameworks instead of creating brand-new frameworks. It provides an exemplar for others who are considering when and how to modify existing frameworks to address new or expanded purposes while respecting the conceptual underpinnings of the original work.

Academic-policy engagement refers to ways that Higher Education Institutions (HEIs) and their staff engage with institutions responsible for policy at national, regional, county or local levels. Academic-policy engagement is intended to support the use of evidence in decision-making and in turn, improve its effectiveness, and inform the identification of barriers and facilitators in policy implementation [ 1 , 2 , 3 ]. Challenges of evidence-informed policy-making are receiving increasing attention globally, including the implications of differences in cultural norms and mechanisms across national contexts [ 4 , 5 ]. Although challenges faced by researchers and policy-makers have been well documented [ 6 , 7 ], there has been less focus on actions at the engagement interface. Pragmatic guidance for the development, evaluation or comparison of structured responses to the challenges of academic-policy engagement is currently lacking [ 8 , 9 ].

Academic-policy engagement exists along a continuum of approaches from linear (pushing evidence out from academia or pulling evidence into policy), relational (promoting mutual understandings and partnerships), and systems approaches (addressing identified barriers and facilitators) [ 4 ]. Each approach is underpinned by sets of beliefs, assumptions and expectations, and each raises questions for implementation and evaluation. Little is known about which academic-policy engagement interventions work in which settings, with scarce empirical evidence to inform decisions about which interventions to use, when, with whom, or why, and how organisational contexts can affect motivation and capabilities for such engagement [ 10 ]. A deeper understanding through the evaluation of engagement interventions will help to identify inhibitory and facilitatory factors, which may or may not transfer across contexts [ 11 ].

The intellectual technologies [ 12 ] of implementation science have proliferated in recent decades, including models, frameworks and theories that address research translation and acknowledge difficulties in closing the gap between research, policy and practice [ 13 ]. Frameworks may serve overlapping purposes of describing or guiding processes of translating knowledge into practice (e.g. the Quality Implementation Framework [ 14 ]); or helping to explain influences on implementation outcomes (e.g. the Theoretical Domains Framework [ 15 ]); or guiding evaluation (e.g. the RE-AIM framework [ 16 , 17 ]. Frameworks can offer an efficient way to look across diverse settings and to identify implementation differences [ 18 , 19 ]. However, the abundance of options raises its own challenges when seeking a framework for a particular purpose, and the use of a framework may mean that more weight is placed on certain aspects, leading to a partial understanding [ 13 , 17 ].

‘Action frameworks’ are predictive models that intend to organise existing knowledge and enable a logical approach for the selection, implementation and evaluation of intervention strategies, thereby facilitating the expansion of that knowledge [ 20 ]. They can guide change by informing and clarifying practical steps to follow. As flexible entities, they can be adapted to accommodate new purposes. Framework modification may include the addition of constructs or changes in language to expand applicability to a broader range of settings [ 21 ].

We sought to identify one organising framework for evaluation activities in the Capabilities in Academic-Policy Engagement (CAPE) programme (2021–2023), funded by Research England. The CAPE programme aimed to understand how best to support effective and sustained engagement between academics and policy professionals across the higher education sector in England [ 22 ]. We first searched the literature and identified an action framework that was originally developed between 2011 and 2013, to underpin a trial known as SPIRIT (Supporting Policy In health with Research: an Intervention Trial) [ 20 , 23 ]. This trial evaluated strategies intended to increase the use of research in health policy and to identify modifiable points for intervention.

We selected the SPIRIT framework due to its potential suitability as an initial ‘road map’ for our evaluation of academic-policy interventions in the CAPE programme. The key elements of the original framework are catalysts, organisational capacity, engagement actions, and research use. We wished to build on the framework’s embedded conceptual work, derived from literature reviews and semi-structured interviews, to identify policymakers’ views on factors that assist policy agencies’ use of research [ 20 ]. The SPIRIT framework developers defined its “locus for change” as the policy organisation ( [ 20 ], p. 151). They proposed that it could offer the beginning of a process to identify and test pathways in policy agencies’ use of evidence.

Our goal was to modify SPIRIT to accommodate a different locus for change: the engagement interface between academia and policy. Instead of imagining a linear process in which knowledge comes from researchers and is transmitted to policy professionals, we intended to extend the framework to multidirectional relational and system interfaces. We wished to include processes and influences at individual, organisational and system levels, to be relevant for HEIs and their staff, policy bodies and professionals, funders of engagement activities, and facilitatory bodies. Ultimately, we seek to address a gap in understanding how engagement strategies work, for whom, how they are facilitated, and to improve the evaluation of academic-policy engagement.

We aimed to produce a conceptually guided action framework to enable systematic evaluation of interventions intending to support academic-policy engagement.

We used a pragmatic combination of processes for framework modification during our evaluation activities in the CAPE programme [ 22 ]. The CAPE programme included a range of interventions: seed funding for academic and policy professional collaboration in policy-focused projects, fellowships for academic placements in policy settings, or for policy professionals with HEI staff, training for policy professionals, and a range of knowledge exchange events for HEI staff and policy professionals. We modified the SPIRIT framework through iterative processes shown in Table  1 , including reviews of literature; consultations with HEI staff and policy professionals across a range of policy contexts and geographic settings in England, through the CAPE programme; and piloting, refining and seeking feedback from stakeholders in academic-policy engagement.

A number of characteristics of the original SPIRIT framework could be applied to academic-policy engagement. While keeping the core domains, we modified the framework to capture dynamics of engagement at multiple academic and policy levels (individuals, organisations and system), extending beyond the original unidirectional focus on policy agencies’ use of research. Components of the original framework, the need for modifications, and their corresponding action-oriented implications are shown in Table  2 . We added a new domain, ‘Impacts and Sustainability’, to consider transforming and enduring aspects at the engagement interface. The modified action framework is shown in Fig.  1 .

SPIRIT Action Framework Modified for Academic-Policy Engagement Interventions (SPIRIT-ME), adapted with permission from the Sax Institute. Legend: The framework acknowledges that elements in each domain may influence other elements through mechanisms of action and that these do not necessarily flow through the framework in a ‘pipeline’ sequence. Mechanisms of action are processes through which engagement strategies operate to achieve desired outcomes. They might rely on influencing factors, catalysts, an aspect of an intervention action, or a combination of elements

Identifying relevant theories or models for missing elements

Catalysts and capacity.

Within our evaluation of academic-policy interventions, we identified a need to develop the original domain of catalysts beyond ‘policy/programme need for research’ and ‘new research with potential policy relevance’. Redman et al. characterised a catalyst as “a need for information to answer a particular problem in policy or program design, or to assist in supporting a case for funding” in the original framework (p. 149). We expanded this “need for information” to a perceived need for engagement, by either HEI staff or policy professionals, linking to the potential value they perceived in engaging. Specifically, there was a need to consider catalysts at the level of individual engagement, for example HEI staff wanting research to have real-world impact, or policy professionals’ desires to improve decision-making in policy, where productive interactions between academic and policy stakeholders are “necessary interim steps in the process that lead to societal impact” ( [ 24 ], p. 214). The catalyst domain expands the original emphasis on a need for research, to take account of challenges to be overcome by both the academic and policy communities in knowing how, and with whom, to engage and collaborate with [ 25 ].

We used a model proposing that there are three components for any behaviour: capability, opportunity and motivation, which is known as the COM-B model [ 26 ]. Informed by CAPE evaluation activities and our discussions with stakeholders, we mapped the opportunity and motivation constructs into the ‘catalysts’ domain of the original framework. Opportunity is an attribute of the system that can facilitate engagement. It may be a tangible factor such as the availability of seed funding, or a perceived social opportunity such as institutional support for engagement activities. Opportunity can act at the macro level of systems and organisational structures. Motivation acts at the micro level, deriving from an individual’s mental processes that stimulate and direct their behaviours; in this case, taking part in academic-policy engagement actions. The COM-B model distinguishes between reflective motivation through conscious planning and automatic motivation that may be instinctive or affective [ 26 ].

We presented an early application of the COM-B model to catalysts for engagement at an academic conference, enabling an informal exploration of attendees’ subjective views on the clarity and appropriateness, when developing the framework. This application introduces possibilities for intervention development and support by highlighting ‘opportunities’ and ‘motivations’ as key catalysts in the modified framework.

Within the ‘capacity’ domain, we retained the original levels of individuals, organisations and systems. We introduced individual capability as a construct from the COM-B model, describing knowledge, skills and abilities to generate behaviour change as a precursor of academic-policy engagement. This reframing extends the applicability to HEI staff as well as policy professionals. It brings attention to different starting conditions for individuals, such as capabilities developed through previous experience, which can link with social opportunity (for example, through training or support) as a catalyst.

Engagement actions

We identified a need to modify the original domain ‘engagement actions’ to extend the focus beyond the use of research. We added three categories of engagement actions described by Best and Holmes [ 27 ]: linear, relational, and systems. These categories were further specified through a systematic mapping of international organisations’ academic-policy engagement activities [ 5 ]. This framework modification expands the domain to encompass: (i) linear ‘push’ of evidence from academia or ‘pull’ of evidence into policy agencies; (ii) relational approaches focused on academic-policy-maker collaboration; and (iii) systems’ strategies to facilitate engagement for example through strategic leadership, rewards or incentives [ 5 ].

We retained the elements in the original framework’s ‘outcomes’ domain (instrumental, tactical, conceptual and imposed), which we found could apply to outcomes of engagement as well as research use. For example, discussions between a policy professional and a range of academics could lead to a conceptual outcome by considering an issue through different disciplinary lenses. We expanded these elements by drawing on literature on engagement outcomes [ 28 ] and through sense-checking with stakeholders in CAPE. We added capacity-building (changes to skills and expertise), connectivity (changes to the number and quality of relationships), and changes in organisational culture or attitude change towards engagement.

Impacts and sustainability

The original framework contained endpoints described as: ‘Better health system and health outcomes’ and ‘Research-informed health policy and policy documents’. For modification beyond health contexts and to encompass broader intentions of academic-policy engagement, we replaced these elements with a new domain of ‘Impacts and sustainability’. This domain captures the continued activities required in achievement of desirable outcomes [ 29 ]. The modification allows consideration of sustainability in relation to previous stages of engagement interventions, through the identification of beneficial effects that are sustained (or not), in which ways, and for whom. Following Borst [ 30 ], we propose a shift from the expectation that ‘sustainability’ will be a fixed endpoint. Instead, we emphasise the maintenance work needed over time, to sustain productive engagement.

Influences and facilitators

We modified the overarching ‘Policy influences’ (such as public opinion and media) in the original framework, to align with factors influencing academic-policy engagement beyond policy agencies’ use of research. We included influences at the level of the individual (for example, individual moral discretion [ 31 ]), the organisation (for example, managerial practices [ 31 ]) and the system (for example, career incentives [ 32 ]). Each of these processes takes place in the broader context of social, policy and financial environments (that is, potential sources of funding for engagement actions) [ 29 ].

We modified the domain ‘Reservoir of relevant and reliable research’ underpinning the original framework, replacing it with ‘Reservoir of people skills’, to emphasise intangible facilitatory work at the engagement interface, in place of concrete research outputs. We used the ‘Promoting Action on Research Implementation in Health Services’ (PARiHS) framework [ 33 , 34 ], which gives explicit consideration to facilitation mechanisms for researchers and policy-makers [ 13 ] . Here, facilitation expertise includes mechanisms that focus on particular goals (task-oriented facilitation) or enable changes in ways of working (holistic-oriented facilitation). Task-orientated facilitation skills might include, for example, the provision of contacts, practical help or project management skills, while holistic-oriented facilitation involves building and sustaining partnerships or support skills’ development across a range of capabilities. These conceptualisations aligned with our consultations with facilitators of engagement in CAPE. We further extended these to include aspects identified in our evaluation activities: strategic planning, contextual awareness and entrepreneurial orientation.

Piloting and refining the modified framework through stakeholder engagement

We piloted an early version of the modified framework to develop a survey for all CAPE programme participants. During this pilot stage, we sought feedback from the CAPE delivery team members across HEI and policy contexts in England. CAPE delivery team members are based at five collaborating universities with partners in the Parliamentary Office for Science and Technology (POST) and Government Office for Science (GO-Science), and Nesta (a British foundation that supports innovation). The HEI members include academics and professional services knowledge mobilisation staff, responsible for leading and coordinating CAPE activities. The delivery team comprised approximately 15–20 individuals (with some fluctuations according to individual availabilities).

We assessed appropriateness and utility, refined terminology, added domain elements and explored nuances. For example, stakeholders considered the multi-layered possibilities within the domain ‘capacity’, where some HEI or policy departments may demonstrate a belief that it is important to use research in policy, but this might not be the perception of the organisation as a whole. We also sought stakeholders’ views on the utility of the new domains, for example, the identification of facilitator expertise such as acting as a knowledge broker or intermediary; providing training, advice or guidance; facilitating engagement opportunities; creating engagement programmes; and sustainability of engagement that could be conceptualised at multiple levels: personally, in processes or through systems.

Testing against criteria for useful action framework

The modified framework fulfils the properties of a useful action framework [ 20 ]:

It has a clearly articulated purpose: development and evaluation of academic-policy engagement interventions through linear, relational and/or system approaches. It has identified loci for change, at the level of the individual, the organisation or system.

It has been informed by existing understandings, including conceptual work of the original SPIRIT framework, conceptual models identified from the literature, published empirical findings, understandings from consultation with stakeholders, and evaluation activities in CAPE.

It can be applied to the development, implementation and evaluation of targeted academic-policy engagement actions, the selection of points for intervention and identification of potential outcomes, including the work of sustaining them and unanticipated consequences.

It provides a structure to build knowledge by guiding the generation of hypotheses about mechanisms of action in academic-policy engagement interventions, or by adapting the framework further through application in practice.

The proliferation of frameworks to articulate processes of research translation reveals a need for their adaptation when applied in specific contexts. The majority of models in implementation science relate to translation of research into practice. By contrast, our focus was on engagement between academia and policy. There are a growing number of academic-policy engagement interventions but a lack of published evaluations [ 10 ].

Our framework modification provides an exemplar for others who are considering how to adapt existing conceptual frameworks to address new or expanded purposes. Field et al. identified the multiple, idiosyncratic ways that the Knowledge to Action Framework has been applied in practice, demonstrating its ‘informal’ adaptability to different healthcare settings and topics [ 35 ]. Others have reported on specific processes for framework refinement or extension. Wiltsey Stirman et al. adopted a framework that characterised forms of intervention modification, using a “pragmatic, multifaceted approach” ( [ 36 ], p.2). The authors later used the modified version as a foundation to build a further framework to encompass implementation strategies in a range of settings [ 21 ]. Oiumet et al. used the approach of borrowing from a different disciplinary field for framework adaptation, by using a model of absorptive capacity from management science to develop a conceptual framework for civil servants’ absorption of research knowledge [ 37 ].

We also took the approach of “adapting the tools we think with” ( [ 38 ], p.305) during our evaluation activities on the CAPE programme. Our conceptual modifications align with the literature on motivation and entrepreneurial orientation in determining policy-makers’ and researchers’ intentions to carry out engagement in addition to ‘usual’ roles [ 39 , 40 ]. Our framework offers an enabler for academic-policy engagement endeavours, by providing a structure for approaches beyond the linear transfer of information, emphasising the role of multidirectional relational activities, and the importance of their facilitation and maintenance. The framework emphasises the relationship between individuals’ and groups’ actions, and the social contexts in which these are embedded. It offers additional value by capturing the organisational and systems level factors that influence evidence-informed policymaking, incorporating the dynamic features of contexts shaping engagement and research use.

Conclusions

Our modifications extend the original SPIRIT framework’s focus on policy agencies’ use of research, to encompass dynamic academic-policy engagement at the levels of individuals, organisations and systems. Informed by the knowledge and experiences of policy professionals, HEI staff and knowledge mobilisers, it is designed to be meaningful and accessible for people working across varied contexts and functions in the evidence-policy ecosystem. It has potential applications in how academic-policy engagement interventions might be developed, evaluated, facilitated and improved, and it fulfils Redman et al.’s criteria as a useful action framework [ 20 ].

We are testing the ‘SPIRIT-Modified for Engagement’ framework (SPIRIT-ME) through our ongoing evaluation of academic-policy engagement activities. Further empirical research is needed to explore how the framework may capture ‘additionality’, that is, to identify what is achieved through engagement actions in addition to what would have happened anyway, including long-term changes in strategic behaviours or capabilities [ 41 , 42 , 43 ]. Application of the modified framework in practice will highlight its strengths and limitations, to inform further iterative development and adaptation.

Availability of data and materials

Not applicable.

Stewart R, Dayal H, Langer L, van Rooyen C. Transforming evidence for policy: do we have the evidence generation house in order? Humanit Soc Sci Commun. 2022;9(1):1–5.

Article   Google Scholar  

Sanderson I. Complexity, ‘practical rationality’ and evidence-based policy making. Policy Polit. 2006;34(1):115–32.

Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gülmezoglu M, et al. Using Qualitative Evidence in Decision Making for Health and Social Interventions: An Approach to Assess Confidence in Findings from Qualitative Evidence Syntheses (GRADE-CERQual). PLOS Med. 2015;12(10):e1001895.

Article   PubMed   PubMed Central   Google Scholar  

Bonell C, Meiksin R, Mays N, Petticrew M, McKee M. Defending evidence informed policy making from ideological attack. BMJ. 2018;10(362):k3827.

Hopkins A, Oliver K, Boaz A, Guillot-Wright S, Cairney P. Are research-policy engagement activities informed by policy theory and evidence? 7 challenges to the UK impact agenda. Policy Des Pract. 2021;4(3):341–56.

Google Scholar  

Head BW. Toward More “Evidence-Informed” Policy Making? Public Adm Rev. 2016;76(3):472–84.

Walker LA, Lawrence NS, Chambers CD, Wood M, Barnett J, Durrant H, et al. Supporting evidence-informed policy and scrutiny: A consultation of UK research professionals. PLoS ONE. 2019;14(3):e0214136.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Graham ID, Tetroe J, Group the KT. Planned action theories. In: Knowledge Translation in Health Care. John Wiley and Sons, Ltd; 2013. p. 277–87. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118413555.ch26 Cited 2023 Nov 1

Davies HT, Powell AE, Nutley SM. Mobilising knowledge to improve UK health care: learning from other countries and other sectors – a multimethod mapping study. Southampton (UK): NIHR Journals Library; 2015. (Health Services and Delivery Research). Available from: http://www.ncbi.nlm.nih.gov/books/NBK299400/ Cited 2023 Nov 1

Oliver K, Hopkins A, Boaz A, Guillot-Wright S, Cairney P. What works to promote research-policy engagement? Evid Policy. 2022;18(4):691–713.

Nelson JP, Lindsay S, Bozeman B. The last 20 years of empirical research on government utilization of academic social science research: a state-of-the-art literature review. Adm Soc. 2023;28:00953997231172923.

Bell D. Technology, nature and society: the vicissitudes of three world views and the confusion of realms. Am Sch. 1973;42:385–404.

Milat AJ, Li B. Narrative review of frameworks for translating research evidence into policy and practice. Public Health Res Pract. 2017; Available from: https://apo.org.au/sites/default/files/resource-files/2017-02/apo-nid74420.pdf Cited 2023 Nov 1

Meyers DC, Durlak JA, Wandersman A. The quality implementation framework: a synthesis of critical steps in the implementation process. Am J Community Psychol. 2012;50(3–4):462–80.

Article   PubMed   Google Scholar  

Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7(1):37.

Glasgow RE, Battaglia C, McCreight M, Ayele RA, Rabin BA. Making implementation science more rapid: use of the RE-AIM framework for mid-course adaptations across five health services research projects in the veterans health administration. Front Public Health. 2020;8. Available from: https://www.frontiersin.org/articles/10.3389/fpubh.2020.00194 Cited 2023 Jun 13

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci IS. 2015 Apr 21 10. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4406164/ Cited 2020 May 4

Sheth A, Sinfield JV. An analytical framework to compare innovation strategies and identify simple rules. Technovation. 2022;1(115):102534.

Birken SA, Powell BJ, Shea CM, Haines ER, Alexis Kirk M, Leeman J, et al. Criteria for selecting implementation science theories and frameworks: results from an international survey. Implement Sci. 2017;12(1):124.

Redman S, Turner T, Davies H, Williamson A, Haynes A, Brennan S, et al. The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015;136:147–55.

Miller CJ, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implement Sci. 2021;16(1):36.

CAPE. CAPE. 2021. CAPE Capabilities in Academic Policy Engagement. Available from: https://www.cape.ac.uk/ Cited 2021 Aug 3

CIPHER Investigators. Supporting policy in health with research: an intervention trial (SPIRIT)—protocol for a stepped wedge trial. BMJ Open. 2014;4(7):e005293.

Spaapen J, Van Drooge L. Introducing ‘productive interactions’ in social impact assessment. Res Eval. 2011;20(3):211–8.

Williams C, Pettman T, Goodwin-Smith I, Tefera YM, Hanifie S, Baldock K. Experiences of research-policy engagement in policymaking processes. Public Health Res Pract. 2023. Online early publication. https://doi.org/10.17061/phrp33232308 .

Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(1):42.

Best A, Holmes B. Systems thinking, knowledge and action: towards better models and methods. Evid Policy J Res Debate Pract. 2010;6(2):145–59.

Edwards DM, Meagher LR. A framework to evaluate the impacts of research on policy and practice: A forestry pilot study. For Policy Econ. 2020;1(114):101975.

Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. 2011;101(11):2059–67.

Borst RAJ, Wehrens R, Bal R, Kok MO. From sustainability to sustaining work: What do actors do to sustain knowledge translation platforms? Soc Sci Med. 2022;1(296):114735.

Zacka B. When the state meets the street: public service and moral agency. Harvard university press; 2017. Available from: https://books.google.co.uk/books?hl=en&lr=&id=3KdFDwAAQBAJ&oi=fnd&pg=PP1&dq=zacka+when+the+street&ots=x93YEHPKhl&sig=9yXKlQiFZ0XblHrbYKzvAMwNWT4 Cited 2023 Nov 28

Torrance H. The research excellence framework in the United Kingdom: processes, consequences, and incentives to engage. Qual Inq. 2020;26(7):771–9.

Rycroft-Malone J. The PARIHS framework—a framework for guiding the implementation of evidence-based practice. J Nurs Care Qual. 2004;19(4):297–304.

Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011;6(1):99.

Field B, Booth A, Ilott I, Gerrish K. Using the knowledge to action framework in practice: a citation analysis and systematic review. Implement Sci. 2014;9(1):172.

Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58.

Ouimet M, Landry R, Ziam S, Bédard PO. The absorption of research knowledge by public civil servants. Evid Policy. 2009;5(4):331–50.

Martin D, Spink MJ, Pereira PPG. Multiple bodies, political ontologies and the logic of care: an interview with Annemarie Mol. Interface - Comun Saúde Educ. 2018;22:295–305.

Sajadi HS, Majdzadeh R, Ehsani-Chimeh E, Yazdizadeh B, Nikooee S, Pourabbasi A, et al. Policy options to increase motivation for improving evidence-informed health policy-making in Iran. Health Res Policy Syst. 2021;19(1):91.

Athreye S, Sengupta A, Odetunde OJ. Academic entrepreneurial engagement with weak institutional support: roles of motivation, intention and perceptions. Stud High Educ. 2023;48(5):683–94.

Bamford D, Reid I, Forrester P, Dehe B, Bamford J, Papalexi M. An empirical investigation into UK university–industry collaboration: the development of an impact framework. J Technol Transf. 2023 Nov 13; Available from: https://doi.org/10.1007/s10961-023-10043-9 Cited 2023 Dec 20

McPherson AH, McDonald SM. Measuring the outcomes and impacts of innovation interventions assessing the role of additionality. Int J Technol Policy Manag. 2010;10(1–2):137–56.

Hind J. Additionality: a useful way to construct the counterfactual qualitatively? Eval J Australas. 2010;10(1):28–35.

Download references

Acknowledgements

We are very grateful to the CAPE Programme Delivery Group members, for many discussions throughout this work. Our thanks also go to the Sax Institute, Australia (where the original SPIRIT framework was developed), for reviewing and providing helpful feedback on the article. We also thank our reviewers who made very constructive suggestions, which have strengthened and clarified our article.

The evaluation of the CAPE programme, referred to in this report, was funded by Research England. The funding body had no role in the design of the study, analysis, interpretation or writing the manuscript.

Author information

Authors and affiliations.

Department of Health Services Research and Policy, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, 15-17 Tavistock Place, Kings Cross, London, WC1H 9SH, UK

Petra Mäkelä & Kathryn Oliver

Health and Social Care Workforce Research Unit, The Policy Institute, Virginia Woolf Building, Kings College London, 22 Kingsway, London, WC2B 6LE, UK

Annette Boaz

You can also search for this author in PubMed   Google Scholar

Contributions

PM conceptualised the modification of the framework reported in this work. All authors made substantial contributions to the design of the work. PM drafted the initial manuscript. AB and KO contributed to revisions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Petra Mäkelä .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval was granted for the overarching CAPE evaluation by the London School of Hygiene and Tropical Medicine Research Ethics Committee (reference 26347).

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mäkelä, P., Boaz, A. & Oliver, K. A modified action framework to develop and evaluate academic-policy engagement interventions. Implementation Sci 19 , 31 (2024). https://doi.org/10.1186/s13012-024-01359-7

Download citation

Received : 09 January 2024

Accepted : 20 March 2024

Published : 12 April 2024

DOI : https://doi.org/10.1186/s13012-024-01359-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Evidence-informed policy
• Academic-policy engagement
• Framework modification

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 18 April 2024

Research ethics and artificial intelligence for global health: perspectives from the global forum on bioethics in research

• James Shaw 1 , 13 ,
• Joseph Ali 2 , 3 ,
• Caesar A. Atuire 4 , 5 ,
• Phaik Yeong Cheah 6 ,
• Armando Guio Español 7 ,
• Judy Wawira Gichoya 8 ,
• Adrienne Hunt 9 ,
• Daudi Jjingo 10 ,
• Katherine Littler 9 ,
• Daniela Paolotti 11 &
• Effy Vayena 12  

BMC Medical Ethics volume  25 , Article number:  46 ( 2024 ) Cite this article

Metrics details

The ethical governance of Artificial Intelligence (AI) in health care and public health continues to be an urgent issue for attention in policy, research, and practice. In this paper we report on central themes related to challenges and strategies for promoting ethics in research involving AI in global health, arising from the Global Forum on Bioethics in Research (GFBR), held in Cape Town, South Africa in November 2022.

The GFBR is an annual meeting organized by the World Health Organization and supported by the Wellcome Trust, the US National Institutes of Health, the UK Medical Research Council (MRC) and the South African MRC. The forum aims to bring together ethicists, researchers, policymakers, research ethics committee members and other actors to engage with challenges and opportunities specifically related to research ethics. In 2022 the focus of the GFBR was “Ethics of AI in Global Health Research”. The forum consisted of 6 case study presentations, 16 governance presentations, and a series of small group and large group discussions. A total of 87 participants attended the forum from 31 countries around the world, representing disciplines of bioethics, AI, health policy, health professional practice, research funding, and bioinformatics. In this paper, we highlight central insights arising from GFBR 2022.

We describe the significance of four thematic insights arising from the forum: (1) Appropriateness of building AI, (2) Transferability of AI systems, (3) Accountability for AI decision-making and outcomes, and (4) Individual consent. We then describe eight recommendations for governance leaders to enhance the ethical governance of AI in global health research, addressing issues such as AI impact assessments, environmental values, and fair partnerships.

Conclusions

The 2022 Global Forum on Bioethics in Research illustrated several innovations in ethical governance of AI for global health research, as well as several areas in need of urgent attention internationally. This summary is intended to inform international and domestic efforts to strengthen research ethics and support the evolution of governance leadership to meet the demands of AI in global health research.

Peer Review reports

Introduction

The ethical governance of Artificial Intelligence (AI) in health care and public health continues to be an urgent issue for attention in policy, research, and practice [ 1 , 2 , 3 ]. Beyond the growing number of AI applications being implemented in health care, capabilities of AI models such as Large Language Models (LLMs) expand the potential reach and significance of AI technologies across health-related fields [ 4 , 5 ]. Discussion about effective, ethical governance of AI technologies has spanned a range of governance approaches, including government regulation, organizational decision-making, professional self-regulation, and research ethics review [ 6 , 7 , 8 ]. In this paper, we report on central themes related to challenges and strategies for promoting ethics in research involving AI in global health research, arising from the Global Forum on Bioethics in Research (GFBR), held in Cape Town, South Africa in November 2022. Although applications of AI for research, health care, and public health are diverse and advancing rapidly, the insights generated at the forum remain highly relevant from a global health perspective. After summarizing important context for work in this domain, we highlight categories of ethical issues emphasized at the forum for attention from a research ethics perspective internationally. We then outline strategies proposed for research, innovation, and governance to support more ethical AI for global health.

In this paper, we adopt the definition of AI systems provided by the Organization for Economic Cooperation and Development (OECD) as our starting point. Their definition states that an AI system is “a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy” [ 9 ]. The conceptualization of an algorithm as helping to constitute an AI system, along with hardware, other elements of software, and a particular context of use, illustrates the wide variety of ways in which AI can be applied. We have found it useful to differentiate applications of AI in research as those classified as “AI systems for discovery” and “AI systems for intervention”. An AI system for discovery is one that is intended to generate new knowledge, for example in drug discovery or public health research in which researchers are seeking potential targets for intervention, innovation, or further research. An AI system for intervention is one that directly contributes to enacting an intervention in a particular context, for example informing decision-making at the point of care or assisting with accuracy in a surgical procedure.

The mandate of the GFBR is to take a broad view of what constitutes research and its regulation in global health, with special attention to bioethics in Low- and Middle- Income Countries. AI as a group of technologies demands such a broad view. AI development for health occurs in a variety of environments, including universities and academic health sciences centers where research ethics review remains an important element of the governance of science and innovation internationally [ 10 , 11 ]. In these settings, research ethics committees (RECs; also known by different names such as Institutional Review Boards or IRBs) make decisions about the ethical appropriateness of projects proposed by researchers and other institutional members, ultimately determining whether a given project is allowed to proceed on ethical grounds [ 12 ].

However, research involving AI for health also takes place in large corporations and smaller scale start-ups, which in some jurisdictions fall outside the scope of research ethics regulation. In the domain of AI, the question of what constitutes research also becomes blurred. For example, is the development of an algorithm itself considered a part of the research process? Or only when that algorithm is tested under the formal constraints of a systematic research methodology? In this paper we take an inclusive view, in which AI development is included in the definition of research activity and within scope for our inquiry, regardless of the setting in which it takes place. This broad perspective characterizes the approach to “research ethics” we take in this paper, extending beyond the work of RECs to include the ethical analysis of the wide range of activities that constitute research as the generation of new knowledge and intervention in the world.

Ethical governance of AI in global health

The ethical governance of AI for global health has been widely discussed in recent years. The World Health Organization (WHO) released its guidelines on ethics and governance of AI for health in 2021, endorsing a set of six ethical principles and exploring the relevance of those principles through a variety of use cases. The WHO guidelines also provided an overview of AI governance, defining governance as covering “a range of steering and rule-making functions of governments and other decision-makers, including international health agencies, for the achievement of national health policy objectives conducive to universal health coverage.” (p. 81) The report usefully provided a series of recommendations related to governance of seven domains pertaining to AI for health: data, benefit sharing, the private sector, the public sector, regulation, policy observatories/model legislation, and global governance. The report acknowledges that much work is yet to be done to advance international cooperation on AI governance, especially related to prioritizing voices from Low- and Middle-Income Countries (LMICs) in global dialogue.

One important point emphasized in the WHO report that reinforces the broader literature on global governance of AI is the distribution of responsibility across a wide range of actors in the AI ecosystem. This is especially important to highlight when focused on research for global health, which is specifically about work that transcends national borders. Alami et al. (2020) discussed the unique risks raised by AI research in global health, ranging from the unavailability of data in many LMICs required to train locally relevant AI models to the capacity of health systems to absorb new AI technologies that demand the use of resources from elsewhere in the system. These observations illustrate the need to identify the unique issues posed by AI research for global health specifically, and the strategies that can be employed by all those implicated in AI governance to promote ethically responsible use of AI in global health research.

RECs and the regulation of research involving AI

RECs represent an important element of the governance of AI for global health research, and thus warrant further commentary as background to our paper. Despite the importance of RECs, foundational questions have been raised about their capabilities to accurately understand and address ethical issues raised by studies involving AI. Rahimzadeh et al. (2023) outlined how RECs in the United States are under-prepared to align with recent federal policy requiring that RECs review data sharing and management plans with attention to the unique ethical issues raised in AI research for health [ 13 ]. Similar research in South Africa identified variability in understanding of existing regulations and ethical issues associated with health-related big data sharing and management among research ethics committee members [ 14 , 15 ]. The effort to address harms accruing to groups or communities as opposed to individuals whose data are included in AI research has also been identified as a unique challenge for RECs [ 16 , 17 ]. Doerr and Meeder (2022) suggested that current regulatory frameworks for research ethics might actually prevent RECs from adequately addressing such issues, as they are deemed out of scope of REC review [ 16 ]. Furthermore, research in the United Kingdom and Canada has suggested that researchers using AI methods for health tend to distinguish between ethical issues and social impact of their research, adopting an overly narrow view of what constitutes ethical issues in their work [ 18 ].

The challenges for RECs in adequately addressing ethical issues in AI research for health care and public health exceed a straightforward survey of ethical considerations. As Ferretti et al. (2021) contend, some capabilities of RECs adequately cover certain issues in AI-based health research, such as the common occurrence of conflicts of interest where researchers who accept funds from commercial technology providers are implicitly incentivized to produce results that align with commercial interests [ 12 ]. However, some features of REC review require reform to adequately meet ethical needs. Ferretti et al. outlined weaknesses of RECs that are longstanding and those that are novel to AI-related projects, proposing a series of directions for development that are regulatory, procedural, and complementary to REC functionality. The work required on a global scale to update the REC function in response to the demands of research involving AI is substantial.

These issues take greater urgency in the context of global health [ 19 ]. Teixeira da Silva (2022) described the global practice of “ethics dumping”, where researchers from high income countries bring ethically contentious practices to RECs in low-income countries as a strategy to gain approval and move projects forward [ 20 ]. Although not yet systematically documented in AI research for health, risk of ethics dumping in AI research is high. Evidence is already emerging of practices of “health data colonialism”, in which AI researchers and developers from large organizations in high-income countries acquire data to build algorithms in LMICs to avoid stricter regulations [ 21 ]. This specific practice is part of a larger collection of practices that characterize health data colonialism, involving the broader exploitation of data and the populations they represent primarily for commercial gain [ 21 , 22 ]. As an additional complication, AI algorithms trained on data from high-income contexts are unlikely to apply in straightforward ways to LMIC settings [ 21 , 23 ]. In the context of global health, there is widespread acknowledgement about the need to not only enhance the knowledge base of REC members about AI-based methods internationally, but to acknowledge the broader shifts required to encourage their capabilities to more fully address these and other ethical issues associated with AI research for health [ 8 ].

Although RECs are an important part of the story of the ethical governance of AI for global health research, they are not the only part. The responsibilities of supra-national entities such as the World Health Organization, national governments, organizational leaders, commercial AI technology providers, health care professionals, and other groups continue to be worked out internationally. In this context of ongoing work, examining issues that demand attention and strategies to address them remains an urgent and valuable task.

The GFBR is an annual meeting organized by the World Health Organization and supported by the Wellcome Trust, the US National Institutes of Health, the UK Medical Research Council (MRC) and the South African MRC. The forum aims to bring together ethicists, researchers, policymakers, REC members and other actors to engage with challenges and opportunities specifically related to research ethics. Each year the GFBR meeting includes a series of case studies and keynotes presented in plenary format to an audience of approximately 100 people who have applied and been competitively selected to attend, along with small-group breakout discussions to advance thinking on related issues. The specific topic of the forum changes each year, with past topics including ethical issues in research with people living with mental health conditions (2021), genome editing (2019), and biobanking/data sharing (2018). The forum is intended to remain grounded in the practical challenges of engaging in research ethics, with special interest in low resource settings from a global health perspective. A post-meeting fellowship scheme is open to all LMIC participants, providing a unique opportunity to apply for funding to further explore and address the ethical challenges that are identified during the meeting.

In 2022, the focus of the GFBR was “Ethics of AI in Global Health Research”. The forum consisted of 6 case study presentations (both short and long form) reporting on specific initiatives related to research ethics and AI for health, and 16 governance presentations (both short and long form) reporting on actual approaches to governing AI in different country settings. A keynote presentation from Professor Effy Vayena addressed the topic of the broader context for AI ethics in a rapidly evolving field. A total of 87 participants attended the forum from 31 countries around the world, representing disciplines of bioethics, AI, health policy, health professional practice, research funding, and bioinformatics. The 2-day forum addressed a wide range of themes. The conference report provides a detailed overview of each of the specific topics addressed while a policy paper outlines the cross-cutting themes (both documents are available at the GFBR website: https://www.gfbr.global/past-meetings/16th-forum-cape-town-south-africa-29-30-november-2022/ ). As opposed to providing a detailed summary in this paper, we aim to briefly highlight central issues raised, solutions proposed, and the challenges facing the research ethics community in the years to come.

In this way, our primary aim in this paper is to present a synthesis of the challenges and opportunities raised at the GFBR meeting and in the planning process, followed by our reflections as a group of authors on their significance for governance leaders in the coming years. We acknowledge that the views represented at the meeting and in our results are a partial representation of the universe of views on this topic; however, the GFBR leadership invested a great deal of resources in convening a deeply diverse and thoughtful group of researchers and practitioners working on themes of bioethics related to AI for global health including those based in LMICs. We contend that it remains rare to convene such a strong group for an extended time and believe that many of the challenges and opportunities raised demand attention for more ethical futures of AI for health. Nonetheless, our results are primarily descriptive and are thus not explicitly grounded in a normative argument. We make effort in the Discussion section to contextualize our results by describing their significance and connecting them to broader efforts to reform global health research and practice.

Uniquely important ethical issues for AI in global health research

Presentations and group dialogue over the course of the forum raised several issues for consideration, and here we describe four overarching themes for the ethical governance of AI in global health research. Brief descriptions of each issue can be found in Table  1 . Reports referred to throughout the paper are available at the GFBR website provided above.

The first overarching thematic issue relates to the appropriateness of building AI technologies in response to health-related challenges in the first place. Case study presentations referred to initiatives where AI technologies were highly appropriate, such as in ear shape biometric identification to more accurately link electronic health care records to individual patients in Zambia (Alinani Simukanga). Although important ethical issues were raised with respect to privacy, trust, and community engagement in this initiative, the AI-based solution was appropriately matched to the challenge of accurately linking electronic records to specific patient identities. In contrast, forum participants raised questions about the appropriateness of an initiative using AI to improve the quality of handwashing practices in an acute care hospital in India (Niyoshi Shah), which led to gaming the algorithm. Overall, participants acknowledged the dangers of techno-solutionism, in which AI researchers and developers treat AI technologies as the most obvious solutions to problems that in actuality demand much more complex strategies to address [ 24 ]. However, forum participants agreed that RECs in different contexts have differing degrees of power to raise issues of the appropriateness of an AI-based intervention.

The second overarching thematic issue related to whether and how AI-based systems transfer from one national health context to another. One central issue raised by a number of case study presentations related to the challenges of validating an algorithm with data collected in a local environment. For example, one case study presentation described a project that would involve the collection of personally identifiable data for sensitive group identities, such as tribe, clan, or religion, in the jurisdictions involved (South Africa, Nigeria, Tanzania, Uganda and the US; Gakii Masunga). Doing so would enable the team to ensure that those groups were adequately represented in the dataset to ensure the resulting algorithm was not biased against specific community groups when deployed in that context. However, some members of these communities might desire to be represented in the dataset, whereas others might not, illustrating the need to balance autonomy and inclusivity. It was also widely recognized that collecting these data is an immense challenge, particularly when historically oppressive practices have led to a low-trust environment for international organizations and the technologies they produce. It is important to note that in some countries such as South Africa and Rwanda, it is illegal to collect information such as race and tribal identities, re-emphasizing the importance for cultural awareness and avoiding “one size fits all” solutions.

The third overarching thematic issue is related to understanding accountabilities for both the impacts of AI technologies and governance decision-making regarding their use. Where global health research involving AI leads to longer-term harms that might fall outside the usual scope of issues considered by a REC, who is to be held accountable, and how? This question was raised as one that requires much further attention, with law being mixed internationally regarding the mechanisms available to hold researchers, innovators, and their institutions accountable over the longer term. However, it was recognized in breakout group discussion that many jurisdictions are developing strong data protection regimes related specifically to international collaboration for research involving health data. For example, Kenya’s Data Protection Act requires that any internationally funded projects have a local principal investigator who will hold accountability for how data are shared and used [ 25 ]. The issue of research partnerships with commercial entities was raised by many participants in the context of accountability, pointing toward the urgent need for clear principles related to strategies for engagement with commercial technology companies in global health research.

The fourth and final overarching thematic issue raised here is that of consent. The issue of consent was framed by the widely shared recognition that models of individual, explicit consent might not produce a supportive environment for AI innovation that relies on the secondary uses of health-related datasets to build AI algorithms. Given this recognition, approaches such as community oversight of health data uses were suggested as a potential solution. However, the details of implementing such community oversight mechanisms require much further attention, particularly given the unique perspectives on health data in different country settings in global health research. Furthermore, some uses of health data do continue to require consent. One case study of South Africa, Nigeria, Kenya, Ethiopia and Uganda suggested that when health data are shared across borders, individual consent remains necessary when data is transferred from certain countries (Nezerith Cengiz). Broader clarity is necessary to support the ethical governance of health data uses for AI in global health research.

Recommendations for ethical governance of AI in global health research

Dialogue at the forum led to a range of suggestions for promoting ethical conduct of AI research for global health, related to the various roles of actors involved in the governance of AI research broadly defined. The strategies are written for actors we refer to as “governance leaders”, those people distributed throughout the AI for global health research ecosystem who are responsible for ensuring the ethical and socially responsible conduct of global health research involving AI (including researchers themselves). These include RECs, government regulators, health care leaders, health professionals, corporate social accountability officers, and others. Enacting these strategies would bolster the ethical governance of AI for global health more generally, enabling multiple actors to fulfill their roles related to governing research and development activities carried out across multiple organizations, including universities, academic health sciences centers, start-ups, and technology corporations. Specific suggestions are summarized in Table  2 .

First, forum participants suggested that governance leaders including RECs, should remain up to date on recent advances in the regulation of AI for health. Regulation of AI for health advances rapidly and takes on different forms in jurisdictions around the world. RECs play an important role in governance, but only a partial role; it was deemed important for RECs to acknowledge how they fit within a broader governance ecosystem in order to more effectively address the issues within their scope. Not only RECs but organizational leaders responsible for procurement, researchers, and commercial actors should all commit to efforts to remain up to date about the relevant approaches to regulating AI for health care and public health in jurisdictions internationally. In this way, governance can more adequately remain up to date with advances in regulation.

Second, forum participants suggested that governance leaders should focus on ethical governance of health data as a basis for ethical global health AI research. Health data are considered the foundation of AI development, being used to train AI algorithms for various uses [ 26 ]. By focusing on ethical governance of health data generation, sharing, and use, multiple actors will help to build an ethical foundation for AI development among global health researchers.

Third, forum participants believed that governance processes should incorporate AI impact assessments where appropriate. An AI impact assessment is the process of evaluating the potential effects, both positive and negative, of implementing an AI algorithm on individuals, society, and various stakeholders, generally over time frames specified in advance of implementation [ 27 ]. Although not all types of AI research in global health would warrant an AI impact assessment, this is especially relevant for those studies aiming to implement an AI system for intervention into health care or public health. Organizations such as RECs can use AI impact assessments to boost understanding of potential harms at the outset of a research project, encouraging researchers to more deeply consider potential harms in the development of their study.

Fourth, forum participants suggested that governance decisions should incorporate the use of environmental impact assessments, or at least the incorporation of environment values when assessing the potential impact of an AI system. An environmental impact assessment involves evaluating and anticipating the potential environmental effects of a proposed project to inform ethical decision-making that supports sustainability [ 28 ]. Although a relatively new consideration in research ethics conversations [ 29 ], the environmental impact of building technologies is a crucial consideration for the public health commitment to environmental sustainability. Governance leaders can use environmental impact assessments to boost understanding of potential environmental harms linked to AI research projects in global health over both the shorter and longer terms.

Fifth, forum participants suggested that governance leaders should require stronger transparency in the development of AI algorithms in global health research. Transparency was considered essential in the design and development of AI algorithms for global health to ensure ethical and accountable decision-making throughout the process. Furthermore, whether and how researchers have considered the unique contexts into which such algorithms may be deployed can be surfaced through stronger transparency, for example in describing what primary considerations were made at the outset of the project and which stakeholders were consulted along the way. Sharing information about data provenance and methods used in AI development will also enhance the trustworthiness of the AI-based research process.

Sixth, forum participants suggested that governance leaders can encourage or require community engagement at various points throughout an AI project. It was considered that engaging patients and communities is crucial in AI algorithm development to ensure that the technology aligns with community needs and values. However, participants acknowledged that this is not a straightforward process. Effective community engagement requires lengthy commitments to meeting with and hearing from diverse communities in a given setting, and demands a particular set of skills in communication and dialogue that are not possessed by all researchers. Encouraging AI researchers to begin this process early and build long-term partnerships with community members is a promising strategy to deepen community engagement in AI research for global health. One notable recommendation was that research funders have an opportunity to incentivize and enable community engagement with funds dedicated to these activities in AI research in global health.

Seventh, forum participants suggested that governance leaders can encourage researchers to build strong, fair partnerships between institutions and individuals across country settings. In a context of longstanding imbalances in geopolitical and economic power, fair partnerships in global health demand a priori commitments to share benefits related to advances in medical technologies, knowledge, and financial gains. Although enforcement of this point might be beyond the remit of RECs, commentary will encourage researchers to consider stronger, fairer partnerships in global health in the longer term.

Eighth, it became evident that it is necessary to explore new forms of regulatory experimentation given the complexity of regulating a technology of this nature. In addition, the health sector has a series of particularities that make it especially complicated to generate rules that have not been previously tested. Several participants highlighted the desire to promote spaces for experimentation such as regulatory sandboxes or innovation hubs in health. These spaces can have several benefits for addressing issues surrounding the regulation of AI in the health sector, such as: (i) increasing the capacities and knowledge of health authorities about this technology; (ii) identifying the major problems surrounding AI regulation in the health sector; (iii) establishing possibilities for exchange and learning with other authorities; (iv) promoting innovation and entrepreneurship in AI in health; and (vi) identifying the need to regulate AI in this sector and update other existing regulations.

Ninth and finally, forum participants believed that the capabilities of governance leaders need to evolve to better incorporate expertise related to AI in ways that make sense within a given jurisdiction. With respect to RECs, for example, it might not make sense for every REC to recruit a member with expertise in AI methods. Rather, it will make more sense in some jurisdictions to consult with members of the scientific community with expertise in AI when research protocols are submitted that demand such expertise. Furthermore, RECs and other approaches to research governance in jurisdictions around the world will need to evolve in order to adopt the suggestions outlined above, developing processes that apply specifically to the ethical governance of research using AI methods in global health.

Research involving the development and implementation of AI technologies continues to grow in global health, posing important challenges for ethical governance of AI in global health research around the world. In this paper we have summarized insights from the 2022 GFBR, focused specifically on issues in research ethics related to AI for global health research. We summarized four thematic challenges for governance related to AI in global health research and nine suggestions arising from presentations and dialogue at the forum. In this brief discussion section, we present an overarching observation about power imbalances that frames efforts to evolve the role of governance in global health research, and then outline two important opportunity areas as the field develops to meet the challenges of AI in global health research.

Dialogue about power is not unfamiliar in global health, especially given recent contributions exploring what it would mean to de-colonize global health research, funding, and practice [ 30 , 31 ]. Discussions of research ethics applied to AI research in global health contexts are deeply infused with power imbalances. The existing context of global health is one in which high-income countries primarily located in the “Global North” charitably invest in projects taking place primarily in the “Global South” while recouping knowledge, financial, and reputational benefits [ 32 ]. With respect to AI development in particular, recent examples of digital colonialism frame dialogue about global partnerships, raising attention to the role of large commercial entities and global financial capitalism in global health research [ 21 , 22 ]. Furthermore, the power of governance organizations such as RECs to intervene in the process of AI research in global health varies widely around the world, depending on the authorities assigned to them by domestic research governance policies. These observations frame the challenges outlined in our paper, highlighting the difficulties associated with making meaningful change in this field.

Despite these overarching challenges of the global health research context, there are clear strategies for progress in this domain. Firstly, AI innovation is rapidly evolving, which means approaches to the governance of AI for health are rapidly evolving too. Such rapid evolution presents an important opportunity for governance leaders to clarify their vision and influence over AI innovation in global health research, boosting the expertise, structure, and functionality required to meet the demands of research involving AI. Secondly, the research ethics community has strong international ties, linked to a global scholarly community that is committed to sharing insights and best practices around the world. This global community can be leveraged to coordinate efforts to produce advances in the capabilities and authorities of governance leaders to meaningfully govern AI research for global health given the challenges summarized in our paper.

Limitations

Our paper includes two specific limitations that we address explicitly here. First, it is still early in the lifetime of the development of applications of AI for use in global health, and as such, the global community has had limited opportunity to learn from experience. For example, there were many fewer case studies, which detail experiences with the actual implementation of an AI technology, submitted to GFBR 2022 for consideration than was expected. In contrast, there were many more governance reports submitted, which detail the processes and outputs of governance processes that anticipate the development and dissemination of AI technologies. This observation represents both a success and a challenge. It is a success that so many groups are engaging in anticipatory governance of AI technologies, exploring evidence of their likely impacts and governing technologies in novel and well-designed ways. It is a challenge that there is little experience to build upon of the successful implementation of AI technologies in ways that have limited harms while promoting innovation. Further experience with AI technologies in global health will contribute to revising and enhancing the challenges and recommendations we have outlined in our paper.

Second, global trends in the politics and economics of AI technologies are evolving rapidly. Although some nations are advancing detailed policy approaches to regulating AI more generally, including for uses in health care and public health, the impacts of corporate investments in AI and political responses related to governance remain to be seen. The excitement around large language models (LLMs) and large multimodal models (LMMs) has drawn deeper attention to the challenges of regulating AI in any general sense, opening dialogue about health sector-specific regulations. The direction of this global dialogue, strongly linked to high-profile corporate actors and multi-national governance institutions, will strongly influence the development of boundaries around what is possible for the ethical governance of AI for global health. We have written this paper at a point when these developments are proceeding rapidly, and as such, we acknowledge that our recommendations will need updating as the broader field evolves.

Ultimately, coordination and collaboration between many stakeholders in the research ethics ecosystem will be necessary to strengthen the ethical governance of AI in global health research. The 2022 GFBR illustrated several innovations in ethical governance of AI for global health research, as well as several areas in need of urgent attention internationally. This summary is intended to inform international and domestic efforts to strengthen research ethics and support the evolution of governance leadership to meet the demands of AI in global health research.

Data availability

All data and materials analyzed to produce this paper are available on the GFBR website: https://www.gfbr.global/past-meetings/16th-forum-cape-town-south-africa-29-30-november-2022/ .

Clark P, Kim J, Aphinyanaphongs Y, Marketing, Food US. Drug Administration Clearance of Artificial Intelligence and Machine Learning Enabled Software in and as Medical devices: a systematic review. JAMA Netw Open. 2023;6(7):e2321792–2321792.

Article   Google Scholar  

Potnis KC, Ross JS, Aneja S, Gross CP, Richman IB. Artificial intelligence in breast cancer screening: evaluation of FDA device regulation and future recommendations. JAMA Intern Med. 2022;182(12):1306–12.

Siala H, Wang Y. SHIFTing artificial intelligence to be responsible in healthcare: a systematic review. Soc Sci Med. 2022;296:114782.

Yang X, Chen A, PourNejatian N, Shin HC, Smith KE, Parisien C, et al. A large language model for electronic health records. NPJ Digit Med. 2022;5(1):194.

Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023;6(1):120.

Jobin A, Ienca M, Vayena E. The global landscape of AI ethics guidelines. Nat Mach Intell. 2019;1(9):389–99.

Minssen T, Vayena E, Cohen IG. The challenges for Regulating Medical Use of ChatGPT and other large Language models. JAMA. 2023.

Ho CWL, Malpani R. Scaling up the research ethics framework for healthcare machine learning as global health ethics and governance. Am J Bioeth. 2022;22(5):36–8.

Yeung K. Recommendation of the council on artificial intelligence (OECD). Int Leg Mater. 2020;59(1):27–34.

Maddox TM, Rumsfeld JS, Payne PR. Questions for artificial intelligence in health care. JAMA. 2019;321(1):31–2.

Dzau VJ, Balatbat CA, Ellaissi WF. Revisiting academic health sciences systems a decade later: discovery to health to population to society. Lancet. 2021;398(10318):2300–4.

Ferretti A, Ienca M, Sheehan M, Blasimme A, Dove ES, Farsides B, et al. Ethics review of big data research: what should stay and what should be reformed? BMC Med Ethics. 2021;22(1):1–13.

Rahimzadeh V, Serpico K, Gelinas L. Institutional review boards need new skills to review data sharing and management plans. Nat Med. 2023;1–3.

Kling S, Singh S, Burgess TL, Nair G. The role of an ethics advisory committee in data science research in sub-saharan Africa. South Afr J Sci. 2023;119(5–6):1–3.

Google Scholar  

Cengiz N, Kabanda SM, Esterhuizen TM, Moodley K. Exploring perspectives of research ethics committee members on the governance of big data in sub-saharan Africa. South Afr J Sci. 2023;119(5–6):1–9.

Doerr M, Meeder S. Big health data research and group harm: the scope of IRB review. Ethics Hum Res. 2022;44(4):34–8.

Ballantyne A, Stewart C. Big data and public-private partnerships in healthcare and research: the application of an ethics framework for big data in health and research. Asian Bioeth Rev. 2019;11(3):315–26.

Samuel G, Chubb J, Derrick G. Boundaries between research ethics and ethical research use in artificial intelligence health research. J Empir Res Hum Res Ethics. 2021;16(3):325–37.

Murphy K, Di Ruggiero E, Upshur R, Willison DJ, Malhotra N, Cai JC, et al. Artificial intelligence for good health: a scoping review of the ethics literature. BMC Med Ethics. 2021;22(1):1–17.

Teixeira da Silva JA. Handling ethics dumping and neo-colonial research: from the laboratory to the academic literature. J Bioethical Inq. 2022;19(3):433–43.

Ferryman K. The dangers of data colonialism in precision public health. Glob Policy. 2021;12:90–2.

Couldry N, Mejias UA. Data colonialism: rethinking big data’s relation to the contemporary subject. Telev New Media. 2019;20(4):336–49.

Organization WH. Ethics and governance of artificial intelligence for health: WHO guidance. 2021.

Metcalf J, Moss E. Owning ethics: corporate logics, silicon valley, and the institutionalization of ethics. Soc Res Int Q. 2019;86(2):449–76.

Data Protection Act - OFFICE OF THE DATA PROTECTION COMMISSIONER KENYA [Internet]. 2021 [cited 2023 Sep 30]. https://www.odpc.go.ke/dpa-act/ .

Sharon T, Lucivero F. Introduction to the special theme: the expansion of the health data ecosystem–rethinking data ethics and governance. Big Data & Society. Volume 6. London, England: SAGE Publications Sage UK; 2019. p. 2053951719852969.

Reisman D, Schultz J, Crawford K, Whittaker M. Algorithmic impact assessments: a practical Framework for Public Agency. AI Now. 2018.

Morgan RK. Environmental impact assessment: the state of the art. Impact Assess Proj Apprais. 2012;30(1):5–14.

Samuel G, Richie C. Reimagining research ethics to include environmental sustainability: a principled approach, including a case study of data-driven health research. J Med Ethics. 2023;49(6):428–33.

Kwete X, Tang K, Chen L, Ren R, Chen Q, Wu Z, et al. Decolonizing global health: what should be the target of this movement and where does it lead us? Glob Health Res Policy. 2022;7(1):3.

Abimbola S, Asthana S, Montenegro C, Guinto RR, Jumbam DT, Louskieter L, et al. Addressing power asymmetries in global health: imperatives in the wake of the COVID-19 pandemic. PLoS Med. 2021;18(4):e1003604.

Benatar S. Politics, power, poverty and global health: systems and frames. Int J Health Policy Manag. 2016;5(10):599.

Download references

Acknowledgements

We would like to acknowledge the outstanding contributions of the attendees of GFBR 2022 in Cape Town, South Africa. This paper is authored by members of the GFBR 2022 Planning Committee. We would like to acknowledge additional members Tamra Lysaght, National University of Singapore, and Niresh Bhagwandin, South African Medical Research Council, for their input during the planning stages and as reviewers of the applications to attend the Forum.

This work was supported by Wellcome [222525/Z/21/Z], the US National Institutes of Health, the UK Medical Research Council (part of UK Research and Innovation), and the South African Medical Research Council through funding to the Global Forum on Bioethics in Research.

Author information

Authors and affiliations.

Department of Physical Therapy, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada

Berman Institute of Bioethics, Johns Hopkins University, Baltimore, MD, USA

Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, USA

Department of Philosophy and Classics, University of Ghana, Legon-Accra, Ghana

Caesar A. Atuire

Centre for Tropical Medicine and Global Health, Nuffield Department of Medicine, University of Oxford, Oxford, UK

Mahidol Oxford Tropical Medicine Research Unit, Faculty of Tropical Medicine, Mahidol University, Bangkok, Thailand

Phaik Yeong Cheah

Berkman Klein Center, Harvard University, Bogotá, Colombia

Armando Guio Español

Department of Radiology and Informatics, Emory University School of Medicine, Atlanta, GA, USA

Judy Wawira Gichoya

Health Ethics & Governance Unit, Research for Health Department, Science Division, World Health Organization, Geneva, Switzerland

Adrienne Hunt & Katherine Littler

African Center of Excellence in Bioinformatics and Data Intensive Science, Infectious Diseases Institute, Makerere University, Kampala, Uganda

Daudi Jjingo

ISI Foundation, Turin, Italy

Daniela Paolotti

Department of Health Sciences and Technology, ETH Zurich, Zürich, Switzerland

Effy Vayena

Joint Centre for Bioethics, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

JS led the writing, contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. JA contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. CA contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. PYC contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. AE contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. JWG contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. AH contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. DJ contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. KL contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. DP contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. EV contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper.

Corresponding author

Correspondence to James Shaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Shaw, J., Ali, J., Atuire, C.A. et al. Research ethics and artificial intelligence for global health: perspectives from the global forum on bioethics in research. BMC Med Ethics 25 , 46 (2024). https://doi.org/10.1186/s12910-024-01044-w

Download citation

Received : 31 October 2023

Accepted : 01 April 2024

Published : 18 April 2024

DOI : https://doi.org/10.1186/s12910-024-01044-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Machine learning
• Research ethics
• Global health

BMC Medical Ethics

ISSN: 1472-6939

• General enquiries: [email protected]