the process of literature review in research

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• ~[123]~: Jan 4, 2024 10:52 AM
• ~[124]~: https://guides.library.yale.edu/YSNDoctoral

Research Process :: Step by Step

• Introduction
• Select Topic
• Identify Keywords
• Background Information
• Develop Research Questions
• Refine Topic
• Search Strategy
• Popular Databases
• Evaluate Sources
• Types of Periodicals
• Reading Scholarly Articles
• Primary & Secondary Sources
• Organize / Take Notes
• Writing & Grammar Resources
• Annotated Bibliography
• Literature Review
• Citation Styles
• Paraphrasing
• Privacy / Confidentiality
• Research Process
• Selecting Your Topic
• Identifying Keywords
• Gathering Background Info
• Evaluating Sources

Organize the literature review into sections that present themes or identify trends, including relevant theory. You are not trying to list all the material published, but to synthesize and evaluate it according to the guiding concept of your thesis or research question.  

What is a literature review?

A literature review is an account of what has been published on a topic by accredited scholars and researchers. Occasionally you will be asked to write one as a separate assignment, but more often it is part of the introduction to an essay, research report, or thesis. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries

A literature review must do these things:

• be organized around and related directly to the thesis or research question you are developing
• synthesize results into a summary of what is and is not known
• identify areas of controversy in the literature
• formulate questions that need further research

Ask yourself questions like these:

• What is the specific thesis, problem, or research question that my literature review helps to define?
• What type of literature review am I conducting? Am I looking at issues of theory? methodology? policy? quantitative research (e.g. on the effectiveness of a new procedure)? qualitative research (e.g., studies of loneliness among migrant workers)?
• What is the scope of my literature review? What types of publications am I using (e.g., journals, books, government documents, popular media)? What discipline am I working in (e.g., nursing psychology, sociology, medicine)?
• How good was my information seeking? Has my search been wide enough to ensure I've found all the relevant material? Has it been narrow enough to exclude irrelevant material? Is the number of sources I've used appropriate for the length of my paper?
• Have I critically analyzed the literature I use? Do I follow through a set of concepts and questions, comparing items to each other in the ways they deal with them? Instead of just listing and summarizing items, do I assess them, discussing strengths and weaknesses?
• Have I cited and discussed studies contrary to my perspective?
• Will the reader find my literature review relevant, appropriate, and useful?

Ask yourself questions like these about each book or article you include:

• Has the author formulated a problem/issue?
• Is it clearly defined? Is its significance (scope, severity, relevance) clearly established?
• Could the problem have been approached more effectively from another perspective?
• What is the author's research orientation (e.g., interpretive, critical science, combination)?
• What is the author's theoretical framework (e.g., psychological, developmental, feminist)?
• What is the relationship between the theoretical and research perspectives?
• Has the author evaluated the literature relevant to the problem/issue? Does the author include literature taking positions she or he does not agree with?
• In a research study, how good are the basic components of the study design (e.g., population, intervention, outcome)? How accurate and valid are the measurements? Is the analysis of the data accurate and relevant to the research question? Are the conclusions validly based upon the data and analysis?
• In material written for a popular readership, does the author use appeals to emotion, one-sided examples, or rhetorically-charged language and tone? Is there an objective basis to the reasoning, or is the author merely "proving" what he or she already believes?
• How does the author structure the argument? Can you "deconstruct" the flow of the argument to see whether or where it breaks down logically (e.g., in establishing cause-effect relationships)?
• In what ways does this book or article contribute to our understanding of the problem under study, and in what ways is it useful for practice? What are the strengths and limitations?
• How does this book or article relate to the specific thesis or question I am developing?

Text written by Dena Taylor, Health Sciences Writing Centre, University of Toronto

http://www.writing.utoronto.ca/advice/specific-types-of-writing/literature-review

• ~[123]~: May 13, 2024 11:24 AM
• ~[124]~: https://libguides.uta.edu/researchprocess

University of Texas Arlington Libraries 702 Planetarium Place · Arlington, TX 76019 · 817-272-3000

• Internet Privacy
• Accessibility
• Problems with a guide? Contact Us.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 13 May 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• ~[123]~: May 2, 2024 10:39 AM
• ~[124]~: https://libguides.northwestern.edu/literaturereviews

News alert: UC Berkeley has announced its next university librarian

Secondary menu

• Log in to your Library account
• Hours and Maps
• Connect from Off Campus
• UC Berkeley Home

Search form

Conducting a literature review: why do a literature review, why do a literature review.

• How To Find "The Literature"
• Found it -- Now What?

Besides the obvious reason for students -- because it is assigned! -- a literature review helps you explore the research that has come before you, to see how your research question has (or has not) already been addressed.

You identify:

• core research in the field
• experts in the subject area
• methodology you may want to use (or avoid)
• gaps in knowledge -- or where your research would fit in

It Also Helps You:

• Publish and share your findings
• Justify requests for grants and other funding
• Identify best practices to inform practice
• Set wider context for a program evaluation
• Compile information to support community organizing

Great brief overview, from NCSU

Want To Know More?

• ~[123]~: Apr 25, 2024 1:10 PM
• ~[124]~: https://guides.lib.berkeley.edu/litreview

Literature Reviews

What is a literature review.

• Literature Review Process

Purpose of a Literature Review

• Choosing a Type of Review
• Developing a Research Question
• Searching the Literature
• Searching Tips
• ChatGPT [beta]
• Documenting your Search
• Using Citation Managers
• Concept Mapping
• Writing the Review
• Further Resources

The Library's Subject Specialists are happy to help with your literature reviews!  Find your Subject Specialist here . 

If you have questions about this guide, contact Librarian  Jamie Niehof ([email protected]).

A literature review is an overview of the available research for a specific scientific topic. Literature reviews summarize existing research to answer a review question, provide context for new research, or identify important gaps in the existing body of literature.

An incredible amount of academic literature is published each year, by estimates over two million articles .

Sorting through and reviewing that literature can be complicated, so this Research Guide provides a structured approach to make the process more manageable.

THIS GUIDE IS AN OVERVIEW OF THE LITERATURE REVIEW PROCESS:

• Getting Started (asking a research question | defining scope)
• Organizing the Literature
• Writing the Literature Review (analyzing | synthesizing)

A  literature search  is a systematic search of the scholarly sources in a particular discipline. A  literature review   is the analysis, critical evaluation and synthesis of the results of that search. During this process you will move from a review  of  the literature to a review  for   your research.   Your synthesis of the literature is your unique contribution to research.

WHO IS THIS RESEARCH GUIDE FOR?

— those new to reviewing the literature

— those that need a refresher or a deeper understanding of writing literature reviews

You may need to do a literature review as a part of a course assignment, a capstone project, a master's thesis, a dissertation, or as part of a journal article. No matter the context, a literature review is an essential part of the research process. 

WHAT IS THE PURPOSE OF A LITERATURE REVIEW?

A literature review is typically performed for a specific reason. Even when assigned as an assignment, the goal of the literature review will be one or more of the following:

• To communicate a project's novelty by identifying a research gap

• An overview of research issues , methodologies or results relevant to field
• To explore the  volume and types of available studies
• To establish familiarity with current research before carrying out a new project
• To resolve conflicts amongst contradictory previous studies

Reviewing the literature helps you understand a research topic and develop your own perspective.

A LITERATURE REVIEW IS NOT :

• An annotated bibliography – which is a list of annotated citations to books, articles and documents that includes a brief description and evaluation for each entry
• A literary review – which is a critical discussion of the merits and weaknesses of a literary work
• A book review – which is a critical discussion of the merits and weaknesses of a particular book
• ~[123]~: May 9, 2024 11:44 AM
• ~[124]~: https://guides.lib.umich.edu/litreview

How to write a Literature Review: Literature review process

• Literature review process
• Purpose of a literature review
• Evaluating sources
• Managing sources
• Request a literature search
• Selecting the approach to use
• Quantitative vs qualitative method
• Summary of different research methodologies
• Research design vs research methodology
• Diagram: importance of research
• Attributes of a good research scholar

Step 1: Select a topic

• Select a topic you can manage in the time frame you have to complete your project.
• Establish your research questions and organize your literature into logical categories around the subject/ topic areas of your questions.  Your research questions must be specific enou gh to guide you to the relevant literature.
• Make sure you understand the concept of ‘broader’ and ‘narrower’ terms.  The narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good survey of the literature.

Step 2: Identify the most relevant sources on your topic

Use a variety of resources - locate books , journals , and documents that contain useful information and ideas on your topic. Internet sites , theses & dissertations , conference papers , ePrints and government or industry reports can also be included. Do not rely solely on electronic full-text material which is more easily available. Reference sources such as dictionaries can assist in defining terminology, and encyclopaedias may provide useful introductions to your topic by experts in the field and will list key references.

Step 3 : Search and refine

• Unisa has a number of databases that provide full text access to articles, that allow you to refine your search to ‘peer reviewed’ journals.  These are scholarly journals which go through a rigorous process of quality assessment by several researchers or subject specialists in the academic community before they are accepted for publication. 
• Use the And, Or, Not operators, Wildcards and Logical Brackets when searching in the databases.  For instance, you can use And to narrow your search while the operator OR expands your search.  Not, on the other hand, helps to exclude irrelevant information from your search results.  Please click here for more information on searching.

Literature review process - an overview

Step 3: search and refine.

• Unisa has a number of  databases  that provide full text access to articles, that allow you to refine your search to ‘peer reviewed’ journals.  These are scholarly journals which go through a rigorous process of quality assessment by several researchers or subject specialists in the academic community before they are accepted for publication. 
• Use the  And, Or, Not  operators,  Wildcards  and  Logical Brackets  when searching in the databases.  For instance, you can use  And  to narrow your search while  the  operator  OR  expands your search.   Not,  on the other hand,   helps to exclude   irrelevant information from your search results.  Please click  here  for more information on searching.

How do I write a literature review

See the chapter below for a helpful overview of the literature review process, especially the sections on how to analyse the literature you have gathered and how to write up your literature review:

Literature Reviews and Bibliographic Searches. 2006. In V. Desai, & R. Potter (Eds.),  Doing Development Research.  (pp. 209-222). London, England: SAGE Publications, Ltd. Available at:  http://0-dx.doi.org.oasis.unisa.ac.za/10.4135/9781849208925.n22     (A student will be prompted at some stage for his/ her student number and myUnisa password. A staff member will be prompted at some stage for his/ her Unisa Network username and login password).

This book is available in the  Sage Research Methods Online  database.

Step 4: Read and analyse

Group the sources into the  themes  and  sub-themes  of your topic.  As you read widely but selectively in your topic area, consider what themes or issues connect your sources together.

• Do they present one or different solutions?
• Is there an aspect of the field that is missing?
• How well do they present the material and do they portray it according to an appropriate theory?
• Do they reveal a trend in the field?
• A raging debate?
• Pick one of these themes to focus the organization of your review.

Step 5: Write the literature review

You can organize the review in many ways; for example, you can center the review  historically  (how the topic has been dealt with over time); or center it on the  theoretical positions  surrounding your topic (those for a position vs. those against, for example); or you can focus on how each of your sources contributes to your understanding of your project.

Your literature review should include:

• an  introduction  which explains how your review is organized.
• a  body  which contains the  headings  and  subheadings  that provide a map to show the various perspectives of your argument. In other words the body contains the evaluation of the materials you want to include on your topic.
• a  summary .

Some of the information on this page is indebted to the sources below:

Caldwell College Library

Monmouth University Library

University of Cape Town Libraries

• ~[123]~: Apr 30, 2024 1:19 PM
• ~[124]~: https://libguides.unisa.ac.za/literature_review

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• ~[123]~: Oct 26, 2022 2:49 PM
• ~[124]~: https://guides.lib.utexas.edu/literaturereviews

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• ~[123]~: May 9, 2024 11:05 AM
• ~[124]~: https://libguides.usc.edu/writingguide

• General Education Courses
• School of Business
• School of Design
• School of Education
• School of Health Sciences
• School of Justice Studies
• School of Nursing
• School of Technology
• CBE Student Guide
• Online Library
• Ask a Librarian
• Learning Express Library
• Interlibrary Loan Request Form
• Library Staff
• Databases A-to-Z
• Discovery Search
• Publication Finder
• Video Databases
• NoodleTools
• Library Guides
• Course Guides
• Writing Lab
• Rasmussen Technical Support (PSC)
• Copyright Toolkit
• Faculty Toolkit
• Suggest a Purchase
• Refer a Student Tutor
• Live Lecture/Peer Tutor Scheduler
• Faculty Interlibrary Loan Request Form
• Professional Development Databases
• Publishing Guide
• Professional Development Guides (AAOPD)
• Literature Review Guide
• Back to Research Help

The Literature Review

• What is a Literature Review?
• Plan Your Literature Review
• Identify a Research Gap
• Define Your Research Question
• Search the Literature
• Analyze Your Research Results
• Manage Research Results
• Write the Literature Review

What is a Literature Review?  What is its purpose?

The purpose of a literature review is to offer a  comprehensive review of scholarly literature on a specific topic along with an  evaluation of the strengths and weaknesses of authors' arguments . In other words, you are summarizing research available on a certain topic and then drawing conclusions about researchers' findings. To make gathering research easier, be sure to start with a narrow/specific topic and then widen your topic if necessary.

A thorough literature review provides an accurate description of current knowledge on a topic and identifies areas for future research.  Are there gaps or areas that require further study and exploration? What opportunities are there for further research? What is missing from my collection of resources? Are more resources needed?

It is important to note that conclusions described in the literature you gather may contradict each other completely or in part.  Recognize that knowledge creation is collective and cumulative.  Current research is built upon past research findings and discoveries.  Research may bring previously accepted conclusions into question.  A literature review presents current knowledge on a topic and may point out various academic arguments within the discipline.

What a Literature Review is not

• A literature review is not an annotated bibliography .  An annotated bibliography provides a brief summary, analysis, and reflection of resources included in the bibliography.  Often it is not a systematic review of existing research on a specific subject.  That said, creating an annotated bibliography throughout your research process may be helpful in managing the resources discovered through your research.
• A literature review is not a research paper .  A research paper explores a topic and uses resources discovered through the research process to support a position on the topic.  In other words, research papers present one side of an issue.  A literature review explores all sides of the research topic and evaluates all positions and conclusions achieved through the scientific research process even though some conclusions may conflict partially or completely.

From the Online Library

SAGE Research Methods is a web-based research methods tool that covers quantitative, qualitative and mixed methods. Researchers can explore methods and concepts to help design research projects, understand a particular method or identify a new method, and write up research. Sage Research Methods focuses on methodology rather than disciplines, and is of potential use to researchers from the social sciences, health sciences and other research areas.

• Sage Research Methods Project Planner - Reviewing the Literature View the resources and videos for a step-by-step guide to performing a literature review.

The Literature Review: Step by Step

Follow this step-by-step process by using the related tabs in this Guide.

• Define your Research question
• Analyze the material you’ve found
• Manage the results of your research
• Write your Review

Getting Started

Consider the following questions as you develop your research topic, conduct your research, and begin evaluating the resources discovered in the research process:

• What is known about the subject?
• Are there any gaps in the knowledge of the subject?
• Have areas of further study been identified by other researchers that you may want to consider?
• Who are the significant research personalities in this area?
• Is there consensus about the topic?
• What aspects have generated significant debate on the topic?
• What methods or problems were identified by others studying in the field and how might they impact your research?
• What is the most productive methodology for your research based on the literature you have reviewed?
• What is the current status of research in this area?
• What sources of information or data were identified that might be useful to you?
• How detailed? Will it be a review of ALL relevant material or will the scope be limited to more recent material, e.g., the last five years.
• Are you focusing on methodological approaches; on theoretical issues; on qualitative or quantitative research?

What is Academic Literature?

What is the difference between popular and scholarly literature?

To better understand the differences between popular and scholarly articles, comparing characteristics and purpose of the publications where these articles appear is helpful.

Popular Article (Magazine)

• Articles are shorter and are written for the general public
• General interest topics or current events are covered
• Language is simple and easy to understand
• Source material is not cited
• Articles often include glossy photographs, graphics, or visuals
• Articles are written by the publication's staff of journalists
• Articles are edited and information is fact checked

Examples of magazines that contain popular articles:

Scholarly Article (Academic Journal)

• Articles are written by scholars and researchers for academics, professionals, and experts in the field
• Articles are longer and report original research findings
• Topics are narrower in focus and provide in-depth analysis
• Technical or scholarly language is used
• Source material is cited
• Charts and graphs illustrating research findings are included
• Many are  "peer reviewed"  meaning that panels of experts review articles submitted for publication to ensure that proper research methods were used and research findings are contributing something new to the field before selecting for publication.

Examples of academic journals that contain scholarly articles:

Define your research question

Selecting a research topic can be overwhelming.  Consider following these steps:

1.  Brainstorm  research topic ideas

      - Free write: Set a timer for five minutes and write down as many ideas as you can in the allotted time

      -  Mind-Map  to explore how ideas are related

2.  Prioritize  topics based on personal interest and curiosity

3.  Pre-research

      - Explore encyclopedias and reference books for background information on the topic

      - Perform a quick database or Google search on the topic to explore current issues. 

4.  Focus the topic  by evaluating how much information is available on the topic

         - Too much information?  Consider narrowing the topic by focusing on a specific issue 

         - Too little information?  Consider broadening the topic 

5.  Determine your purpose  by considering whether your research is attempting to:

         - further the research on this topic

         - fill a gap in the research

         - support existing knowledge with new evidence

         - take a new approach or direction

         - question or challenge existing knowledge

6.  Finalize your research question

NOTE:  Be aware that your initial research question may change as you conduct research on your topic.

Searching the Literature

Research on your topic should be conducted in the academic literature.  The  Rasmussen University Online Library contains subject-focused databases that contain the leading academic journals in your programmatic area.

Consult the  Using the Online Library video tutorials  for information about how to effectively search library databases.

Watch the video below for tips on how to create a search statement that will provide relevant results

Need help starting your research?  Make a  research appointment with a Rasmussen Librarian .

TIP:  Document as you research.  Begin building your references list using the citation managers in one of these resources:

• APA Academic Writer

Recommended programmatic databases include:

Data Science

Coverage includes computer engineering, computer theory & systems, research and development, and the social and professional implications of new technologies. Articles come from more than 1,900 academic journals, trade magazines, and professional publications.

Provides access to full-text peer-reviewed journals, transactions, magazines, conference proceedings, and published standards in the areas of electrical engineering, computer science, and electronics. It also provides access to the IEEE Standards Dictionary Online. Full-text available.

Computing, telecommunications, art, science and design databases from ProQuest.

Healthcare Management

Articles from scholarly business journals back as far as 1886 with content from all disciplines of business, including marketing, management, accounting, management information systems, production and operations management, finance, and economics. Contains 55 videos from the Harvard Faculty Seminar Series, on topics such as leadership, sustaining competitive advantage, and globalization. To access the videos, click "More" in the blue bar at the top. Select "Images/ Business Videos." Uncheck "Image Quick View Collection" to indicate you only wish to search for videos. Enter search terms.

Provides a truly comprehensive business research collection. The collection consists of the following databases and more: ABI/INFORM Complete, ProQuest Entrepreneurship, ProQuest Accounting & Tax, International Bibliography of Social Sciences (IBSS), ProQuest Asian Business and Reference, and Banking Information Source.

The definitive research tool for all areas of nursing and allied health literature. Geared towards the needs of nurses and medical professionals. Covers more than 750 journals from 1937 to present.

HPRC provides information on the creation, implementation and study of health care policy and the health care system. Topics covered include health care administration, economics, planning, law, quality control, ethics, and more.

PolicyMap is an online mapping site that provides data on demographics, real estate, health, jobs, and other areas across the U.S. Access and visualize data from Census and third-party records.

Human Resources

Articles from all subject areas gathered from more than 11,000 magazines, journals, books and reports. Subjects include astronomy, multicultural studies, humanities, geography, history, law, pharmaceutical sciences, women's studies, and more. Coverage from 1887 to present. Start your research here.

Cochrane gathers and summarizes the best evidence from research to help you make informed choices about treatments. Whether a doctor or nurse, patient, researcher or student, Cochrane evidence provides a tool to enhance your healthcare knowledge and decision making on topics ranging from allergies, blood disorders, and cancer, to mental health, pregnancy, urology, and wounds.

Health sciences, biology, science, and pharmaceutical information from ProQuest. Includes articles from scholarly, peer-reviewed journals, practical and professional development content from professional journals, and general interest articles from magazines and newspapers.

Joanna Briggs Institute Academic Collection contains evidence-based information from across the globe, including evidence summaries, systematic reviews, best practice guidelines, and more. Subjects include medical, nursing, and healthcare specialties.

Comprehensive source of full-text articles from more than 1,450 scholarly medical journals.

Articles from more than 35 nursing journals in full text, searchable as far back as 1995.

Analyzing Your Research Results

You have completed your research and discovered many, many academic articles on your topic.  The next step involves evaluating and organizing the literature found in the research process.

As you review, keep in mind that there are three types of research studies:

• Quantitative
• Qualitative 
• Mixed Methods

Consider these questions as you review the articles you have gathered through the research process:

1. Does the study relate to your topic?

2. Were sound research methods used in conducting the study?

3. Does the research design fit the research question? What variables were chosen? Was the sample size adequate?

4. What conclusions were drawn?  Do the authors point out areas for further research?

Reading Academic Literature

Academic journals publish the results of research studies performed by experts in an academic discipline.  Articles selected for publication go through a rigorous peer-review process.  This process includes a thorough evaluation of the research submitted for publication by journal editors and other experts or peers in the field.  Editors select articles based on specific criteria including the research methods used, whether the research contributes new findings to the field of study, and how the research fits within the scope of the academic journal.  Articles selected often go through a revision process prior to publication.

Most academic journal articles include the following sections:

• Abstract    (An executive summary of the study)
• Introduction  (Definition of the research question to be studied)
• Literature Review  (A summary of past research noting where gaps exist)
• Methods  (The research design including variables, sample size, measurements)
• Data   (Information gathered through the study often displayed in tables and charts)
• Results   (Conclusions reached at the end of the study)
• Conclusion   (Discussion of whether the study proved the thesis; may suggest opportunities for further research)
• Bibliography  (A list of works cited in the journal article)

TIP:  To begin selecting articles for your research, read the   highlighted sections   to determine whether the academic journal article includes information relevant to your research topic.

Step 1: Skim the article

When sorting through multiple articles discovered in the research process, skimming through these sections of the article will help you determine whether the article will be useful in your research.

1.  Article title   and subject headings assigned to the article

2.   Abstract

3.   Introduction

4.  Conclusion

If the article fits your information need, go back and  read the article thoroughly.

TIP:  Create a folder on your computer to save copies of articles you plan to use in your thesis or research project.  Use  NoodleTools  or  APA Academic Writer  to save APA references.

Step 2: Determine Your Purpose

Think about how you will evaluate the academic articles you find and how you will determine whether to include them in your research project.  Ask yourself the following questions to focus your search in the academic literature:

• ​Are you looking for an overview of a topic? an explanation of a specific concept, idea, or position?
• Are you exploring gaps in the research to identify a new area for academic study?
• Are you looking for research that supports or disagrees with your thesis or research question?
• Are you looking for examples of a research design and/or research methods you are considering for your own research project?

Step 3: Read Critically

Before reading the article, ask yourself the following:

• What is my research question?  What position am I trying to support?
• What do I already know about this topic?  What do I need to learn?
• How will I evaluate the article?  Author's reputation? Research design? Treatment of topic? 
• What are my biases about the topic?

As you read the article make note of the following:

• Who is the intended audience for this article?
• What is the author's purpose in writing this article?
• What is the main point?
• How was the main point proven or supported?  
• Were scientific methods used in conducting the research?
• Do you agree or disagree with the author? Why?
• How does this article compare or connect with other articles on the topic?
• Does the author recommend areas for further study?
• How does this article help to answer your research question?

Managing your Research

Tip:  Create APA references for resources as you discover them in the research process

Use APA Academic Writer or NoodleTools to generate citations and manage your resources.  Find information on how to use these resources in the Citation Tools Guide .

Writing the Literature Review

Once research has been completed, it is time to structure the literature review and begin summarizing and synthesizing information.  The following steps may help with this process:

• Chronological
• By research method used
• Explore contradictory or conflicting conclusions
• Read each study critically
• Critique methodology, processes, and conclusions
• Consider how the study relates to your topic

• Description of public health nursing nutrition assessment and interventions for home‐visited women. This article provides a nice review of the literature in the article introduction. You can see how the authors have used the existing literature to make a case for their research questions. more... less... Horning, M. L., Olsen, J. M., Lell, S., Thorson, D. R., & Monsen, K. A. (2018). Description of public health nursing nutrition assessment and interventions for home‐visited women. Public Health Nursing, 35(4), 317–326. https://doi.org/10.1111/phn.12410
• Improving Diabetes Self-Efficacy in the Hispanic Population Through Self-Management Education Doctoral papers are a good place to see how literature reviews can be done. You can learn where they searched, what search terms they used, and how they decided which articles were included. Notice how the literature review is organized around the three main themes that came out of the literature search. more... less... Robles, A. N. (2023). Improving diabetes self-efficacy in the hispanic population through self-management education (Order No. 30635901). Available from ProQuest Dissertations & Theses Global: The Sciences and Engineering Collection. https://www.proquest.com/dissertations-theses/improving-diabetes-self-efficacy-hispanic/docview/2853708553/se-2
• Exploring mediating effects between nursing leadership and patient safety from a person-centred perspective: A literature review Reading articles that publish the results of a systematic literature review is a great way to see in detail how a literature review is conducted. These articles provide an article matrix, which provides you an example of how you can document information about the articles you find in your own search. To see more examples, include "literature review" or "systematic review" as a search term. more... less... Wang, M., & Dewing, J. (2021). Exploring mediating effects between nursing leadership and patient safety from a person‐centred perspective: A literature review. Journal of Nursing Management, 29(5), 878–889. https://doi.org/10.1111/jonm.13226

Database Search Tips

• Boolean Operators
• Keywords vs. Subjects
• Creating a Search String
• Library databases are collections of resources that are searchable, including full-text articles, books, and encyclopedias.
• Searching library databases is different than searching Google. Best results are achieved when using Keywords linked with Boolean Operators . 
• Applying Limiters such as full-text, publication date, resource type, language, geographic location, and subject help to refine search results.
• Utilizing Phrases or Fields , in addition to an awareness of Stop Words , can focus your search and retrieve more useful results.
• Have questions? Ask a Librarian

Boolean Operators connect keywords or concepts logically to retrieve relevant articles, books, and other resources.  There are three Boolean Operators:

Using AND 

• Narrows search results
• Connects two or more keywords/concepts
• All keywords/concepts connected with "and" must be in an article or resource to appear in the search results list

Venn diagram of the AND connector

Example: The result list will include resources that include both keywords -- "distracted driving" and "texting" -- in the same article or resource, represented in the shaded area where the circles intersect (area shaded in purple).

• Broadens search results ("OR means more!")
• Connects two or more synonyms or related keywords/concepts
• Resources appearing in the results list will include any of the terms connected with the OR connector

Venn diagram of the OR connector

Example:  The result list will include resources that include the keyword "texting" OR the keyword "cell phone" (entire area shaded in blue); either is acceptable.

• Excludes keywords or concepts from the search
• Narrows results by removing resources that contain the keyword or term connected with the NOT connector
• Use sparingly

Venn diagram of the NOT connector

Example: The result list will include all resources that include the term "car" (green area) but will exclude any resource that includes the term "motorcycle" (purple area) even though the term car may be present in the resource.

A library database searches for keywords throughout the entire resource record including the full-text of the resource, subject headings, tags, bibliographic information, etc.

• Natural language words or short phrases that describe a concept or idea
• Can retrieve too few or irrelevant results due to full-text searching (What words would an author use to write about this topic?)
• Provide flexibility in a search
• Must consider synonyms or related terms to improve search results
• TIP: Build a Keyword List

Example:  The keyword list above was developed to find resources that discuss how texting while driving results in accidents.  Notice that there are synonyms (texting and "text messaging"), related terms ("cell phones" and texting), and spelling variations ("cell phone" and cellphone).  Using keywords when searching full text requires consideration of various words that express an idea or concept.

• Subject Headings
• Predetermined "controlled vocabulary" database editors apply to resources to describe topical coverage of content
• Can retrieve more precise search results because every article assigned that subject heading will be retrieved.
• Provide less flexibility in a search
• Can be combined with a keyword search to focus search results.
• TIP: Consult database subject heading list or subject headings assigned to relevant resources

Example 1: In EBSCO's Academic Search Complete, clicking on the "Subject Terms" tab provides access to the entire subject heading list used in the database.  It also allows a search for specific subject terms.

Example 2:  A subject term can be incorporated into a keyword search by clicking on the down arrow next to "Select a Field" and selecting "Subject Terms" from the dropdown list.  Also, notice how subject headings are listed below the resource title, providing another strategy for discovering subject headings used in the database.

When a search term is more than one word, enclose the phrase in quotation marks to retrieve more precise and accurate results.  Using quotation marks around a term will search it as a "chunk," searching for those particular words together in that order within the text of a resource. 

"cell phone"

"distracted driving"

"car accident"

TIP: In some databases, neglecting to enclose phrases in quotation marks will insert the AND Boolean connector between each word resulting in unintended search results.

Truncation provides an option to search for a root of a keyword in order to retrieve resources that include variations of that word.  This feature can be used to broaden search results, although some results may not be relevant.  To truncate a keyword, type an asterisk (*) following the root of the word.

For example:

Library databases provide a variety of tools to limit and refine search results.  Limiters provide the ability to limit search results to resources having specified characteristics including:

• Resource type
• Publication date
• Geographic location

In both the EBSCO and ProQuest databases, the limiting tools are located in the left panel of the results page.

                                                 EBSCO                                                     ProQuest

The short video below provides a demonstration of how to use limiters to refine a list of search results.

Each resource in a library database is stored in a record.  In addition to the full-text of the resources, searchable Fields are attached that typically include:

• Journal title
• Date of Publication

Incorporating Fields into your search can assist in focusing and refining search results by limiting the results to those resources that include specific information in a particular field.

In both EBSCO and ProQuest databases, selecting the Advanced Search option will allow Fields to be included in a search.

For example, in the Advanced Search option in EBSCO's Academic Search Complete database, clicking on the down arrow next to "Select a Field" provides a list of fields that can be searched within that database.  Select the field and enter the information in the text box to the left to use this feature.

Stop words are short, commonly used words--articles, prepositions, and pronouns-- that are automatically dropped from a search.  Typical stop words include:

In library databases, a stop word will not be searched even if it is included in a phrase enclosed in quotation marks.  In some instances, a word will be substituted for the stop word to allow for the other words in the phrase to be searched in proximity to one another within the text of the resource.

For example, if you searched company of America, your result list will include these variatons:

• company in America
• company of America
• company for America

Creating an Search String

This short video demonstrates how to create a search string -- keywords connected with Boolean operators -- to use in a library database search to retrieve relevant resources for any research assignment.

• Database Search Menu Template Use this search menu template to plan a database search.
• ~[123]~: May 7, 2024 2:33 PM
• ~[124]~: https://guides.rasmussen.edu/LitReview

Browse Econ Literature

• Working papers
• Software components
• Book chapters
• JEL classification

More features

• Subscribe to new research

RePEc Biblio

Author registration.

• Economics Virtual Seminar Calendar NEW!

How to Undertake an Impactful Literature Review: Understanding Review Approaches and Guidelines for High-impact Systematic Literature Reviews

• Author & abstract
• Related works & more

Corrections

• Amrita Chakraborty
• Arpan Kumar Kar

Suggested Citation

Download full text from publisher.

Follow serials, authors, keywords & more

Public profiles for Economics researchers

Various research rankings in Economics

RePEc Genealogy

Who was a student of whom, using RePEc

Curated articles & papers on economics topics

Upload your paper to be listed on RePEc and IDEAS

New papers by email

Subscribe to new additions to RePEc

EconAcademics

Blog aggregator for economics research

Cases of plagiarism in Economics

About RePEc

Initiative for open bibliographies in Economics

News about RePEc

Questions about IDEAS and RePEc

RePEc volunteers

Participating archives

Publishers indexing in RePEc

Privacy statement

Found an error or omission?

Opportunities to help RePEc

Get papers listed

Have your research listed on RePEc

Open a RePEc archive

Have your institution's/publisher's output listed on RePEc

Get RePEc data

Use data assembled by RePEc

This paper is in the following e-collection/theme issue:

Published on 13.5.2024 in Vol 26 (2024)

Suitability of the Current Health Technology Assessment of Innovative Artificial Intelligence-Based Medical Devices: Scoping Literature Review

Authors of this article:

• Line Farah 1, 2 * , PharmD   ; 
• Isabelle Borget 2, 3, 4 , PharmB, PhD   ; 
• Nicolas Martelli 2, 5 , PharmD, PhD   ; 
• Alexandre Vallee 6 * , MD, PhD  

1 Innovation Center for Medical Devices Department, Foch Hospital, Suresnes, France

2 Groupe de Recherche et d'accueil en Droit et Economie de la Santé Department, University Paris-Saclay, Orsay, France

3 Department of Biostatistics and Epidemiology, Gustave Roussy, University Paris-Saclay, Villejuif, France

4 Oncostat U1018, Inserm, Équipe Labellisée Ligue Contre le Cancer, University Paris-Saclay, Villejuif, France

5 Pharmacy Department, Georges Pompidou European Hospital, Paris, France

6 Department of Epidemiology and Public Health, Foch Hospital, Suresnes, France

*these authors contributed equally

Corresponding Author:

Line Farah, PharmD

Innovation Center for Medical Devices Department

Foch Hospital

40 rue Worth

Suresnes, 92150

Phone: 33 952329655

Email: [email protected]

Background: Artificial intelligence (AI)–based medical devices have garnered attention due to their ability to revolutionize medicine. Their health technology assessment framework is lacking.

Objective: This study aims to analyze the suitability of each health technology assessment (HTA) domain for the assessment of AI-based medical devices.

Methods: We conducted a scoping literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We searched databases (PubMed, Embase, and Cochrane Library), gray literature, and HTA agency websites.

Results: A total of 10.1% (78/775) of the references were included. Data quality and integration are vital aspects to consider when describing and assessing the technical characteristics of AI-based medical devices during an HTA process. When it comes to implementing specialized HTA for AI-based medical devices, several practical challenges and potential barriers could be highlighted and should be taken into account (AI technological evolution timeline, data requirements, complexity and transparency, clinical validation and safety requirements, regulatory and ethical considerations, and economic evaluation).

Conclusions: The adaptation of the HTA process through a methodological framework for AI-based medical devices enhances the comparability of results across different evaluations and jurisdictions. By defining the necessary expertise, the framework supports the development of a skilled workforce capable of conducting robust and reliable HTAs of AI-based medical devices. A comprehensive adapted HTA framework for AI-based medical devices can provide valuable insights into the effectiveness, cost-effectiveness, and societal impact of AI-based medical devices, guiding their responsible implementation and maximizing their benefits for patients and health care systems.

Introduction

Artificial intelligence (AI) has emerged as a transformative technology with vast potential across various sectors, including health care [ 1 , 2 ]. In this field, AI-based medical devices have garnered significant attention due to their ability to revolutionize diagnosis, treatment, and patient monitoring [ 3 ]. These devices use advanced algorithms and machine learning techniques to analyze complex medical data sets, thereby providing valuable insights and support to health care professionals in their decision-making. To meet the ever-increasing demand for integration of AI-based medical devices into clinical practice, their efficient evaluation through adapted health technology assessment (HTA) is now crucial [ 4 ]. Several frameworks have been published showing an adaptation of items for reporting clinical trials related to AI-based medical devices ( Table 1 ). However, there is still a lack of a much needed adaptation of the standard HTA framework to better suit the assessment of AI-based health care technologies.

a HTA: health technology assessment.

b CONSORT-AI: Consolidated Standards of Reporting Trials–Artificial Intelligence.

c CONSORT: Consolidated Standards of Reporting Trials.

d SPIRIT-AI: Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence.

e SPIRIT: Standard Protocol Items: Recommendations for Interventional Trials.

f TRIPOD-AI: Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis-Artificial Intelligence.

g TRIPOD: Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis.

An HTA involves a systematic evaluation of a medical technology’s clinical, economic, ethical, and social implications to determine its overall value and impact on health care delivery [ 9 ]. The European Network for Health Technology Assessment has designed an HTA Core Model that provides a methodological framework for production and sharing of HTA information [ 5 ]. It evaluates the following nine domains: (1) health problem and current use of technology, (2) description and technical characteristics of the technology, (3) safety, (4) clinical effectiveness, (5) costs and economic evaluation, (6) ethical analysis, (7) organizational aspects, (8) patients and social aspects, and (9) legal aspects ( Multimedia Appendix 1 ).

A full understanding of the capabilities, limitations, and specificities of AI-based medical devices is paramount to complete these HTA domain assessments and, thereby, inform evidence-based decision-making and allow for policy development and the responsible integration of these technologies into health care systems [ 10 , 11 ]. Much uncertainty remains with regard to the reliability of AI-based medical devices, data issues, and regulatory processes, resulting in multiple challenges faced by HTA agencies assessing new technologies and delivering their approval [ 4 ]. Nevertheless, AI-based medical devices require strict regulations and specific legislations [ 12 - 14 ]. Over the last few decades, the evaluation of AI-based medical devices through an HTA process has received growing interest, as shown by the increase from 1 published article in 1990 to 94 in December 2023, with 484 articles in total during the period 1990-2023 ( Table 2 ).

The objective of this review was to critically assess the comprehensive suitability of the current HTA process for AI-based medical devices. By evaluating the performance and capabilities of AI-based medical devices across multiple dimensions, this review aimed to inform health care professionals, policy makers, and researchers about the challenges and opportunities associated with these technologies. Ultimately, this review sought to facilitate evidence-based decision-making; promote responsible implementation; and maximize the potential benefits of AI-based medical devices in improving health care quality, accessibility, and outcomes.

To this end, we analyzed the suitability of each HTA domain for the assessment of AI-based medical devices and proposed an adapted HTA framework.

Search Strategies

A scoping literature review was conducted following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology [ 15 , 16 ]. All articles related to HTA methods for AI-based medical devices were selected. Data extraction focused on assessment criteria, methodological evaluations, and results.

The search terms and strategy outlined in Textbox 1 were used for the scoping review.

Database and search strategy

• “technology assessment, biomedical” [MeSH term] OR (“technology” [all fields] AND “assessment” [all fields] AND “biomedical” [all fields]) OR “biomedical technology assessment” [all fields] OR (“health” [all fields] AND “technology” [all fields] AND “assessment” [all fields]) OR “health technology assessment” [all fields]
• “equipment and supplies” [MeSH term] OR (“equipment” [all fields] AND “supplies” [all fields]) OR “equipment and supplies” [all fields] OR (“medical” [all fields] AND “device” [all fields]) OR “medical device” [all fields]
• “artificial intelligence” [MeSH term] OR (“artificial” [all fields] AND “intelligence” [all fields]) OR “artificial intelligence” [all fields]
• (“artificial intelligence” OR “artificial intelligence-based medical device”) AND “technology assessment”
• (technology assessment) and (artificial intelligence)

We searched multiple databases: PubMed, Embase, and the Cochrane Library. Additional articles were retrieved manually from the gray literature and from HTA agency websites (Bundesinstitut für Arzneimittel und Medizinprodukte [Federal Institute for Drugs and Medical Devices in Germany], Haute Autorité de Santé [French Health Authority], National Health Service, National Institute for Health and Care Excellence, National Institute of Public Health, International Network of Agencies for Health Technology Assessment, and the European Network for Health Technology Assessment).

Study Selection Process

The studies were selected by 2 reviewers (LF and AV). After removing duplicates, both reviewers independently screened the abstracts to select eligible articles and then analyzed full-text reports for eligibility. A third party (NM) resolved the possible discrepancies highlighted during the selection process should a consensus not be reached. An extraction database was used to list the selected articles meeting the inclusion criteria to ensure that all eligible articles were included.

The inclusion and exclusion criteria were related to (1) language (this review was limited to the English and French languages), (2) article type (reviews and primary research were included, whereas other article types were excluded, such as abstracts, commentaries, editorials, letters to the editor, case reports, case series, animal studies, phase-1 and phase-2 studies, pilot studies, duplicate studies, irrelevant studies, studies with the wrong aim, and studies available in abstract form only), (3) type of technology (only AI-based medical devices were eligible), and (4) type of evaluation (HTA articles were included).

Data Extraction

In total, 2 analysts (LF and AV) independently extracted data items from the selected articles. A third party was involved to resolve any discrepancies highlighted during the selection process. The following data were extracted from each article: (1) general characteristics of the studies (authors, journal, and publication date), (2) study objective, and (3) HTA assessment domain for AI-based medical devices related to the article.

Methodological Quality Appraisal

Neither the methodological quality nor the risk of bias of the included articles were assessed, consistent with scoping review guidance [ 15 ].

Application Example of the HTA of an AI-Based Medical Device

Concerning each HTA domain, we added a case study of the HTA of several AI-based medical devices in diabetes to illustrate our recommendations.

Scoping Review Results

The literature search resulted in 775 citations summarized in the PRISMA flowchart ( Figure 1 ). After removal of 32.6% (253/775) of duplicates, we carried out an initial screening of the remaining 522 publications, resulting in 129 (24.7%) potentially relevant full-text papers. After further screening, we assessed 77 reports for eligibility, based on which we excluded 11 (14%) for not referring to AI-based medical devices and 2 (3%) for not being HTAs. We then included a further 14 records from 59 potential citations available from HTA agency websites and gray literature.

This gave a total of 78 included articles covering one or multiple HTA domains, the distribution of which is provided at the bottom of Figure 1 .

We then summarized the data collection for each selected article in Multimedia Appendix 2 [ 1 , 2 , 4 , 9 , 10 , 17 - 89 ].

Suitability of Each HTA Domain for the Assessment of AI-Based Medical Devices

As domain 1, health problem and current use of technology, is suitable for any type of medical technology and systematically addressed these technologies, we focused on the suitability of the HTA domains (from 1 to 9).

Domains 1 and 2 (Health Problem, Current Use, Description, and Technical Characteristics of the Technology)

Data quality and integration are vital aspects to consider when describing and assessing the technical characteristics of AI-based medical devices during an HTA process [ 17 ]. These devices often rely on accessing and analyzing diverse health care data sources, including electronic health records, medical images, genetic data, and wearable device data. Therefore, it is essential to evaluate their ability to seamlessly integrate and exchange data with existing health care systems [ 18 ].

First, interoperability refers to the ability of AI-based medical devices to interact and communicate with other health care technologies and systems [ 10 ]. This includes the ability to access and use data from different sources, such as laboratory systems, imaging archives, and patient health records. The assessment should consider whether the devices adhere to relevant data standards and protocols, ensuring efficient data exchange and compatibility with existing health care infrastructures [ 19 ].

Second, data integration involves the ability of AI-based medical devices to aggregate and analyze data from multiple sources to provide comprehensive and accurate insights [ 20 ]. The HTA should assess whether the devices can handle different data types, formats, and resolutions and whether they can effectively integrate and harmonize data from disparate sources.

In addition, data privacy and security considerations are crucial when evaluating AI-based medical devices [ 21 ]. The assessment should examine whether the devices comply with relevant data protection regulations, use appropriate data anonymization and encryption techniques, and have robust security measures in place to protect patient information [ 22 ].

Therefore, a comprehensive HTA should address both the interoperability and data integration capabilities of AI-based medical devices, ensuring that they can seamlessly interact with existing health care systems and integrate data from multiple sources and that they adhere to data privacy and security standards. By evaluating all these aspects, the HTA could determine the devices’ feasibility, scalability, and potential impact on health care delivery.

Domain 3 (Safety) and Domain 4 (Clinical Effectiveness)

The assessment of clinical effectiveness and the impact on patient outcomes is a crucial aspect when evaluating AI-based medical devices through a comprehensive HTA [ 4 ]. While accuracy and performance metrics are important, it is essential to determine how these devices translate into tangible benefits for patients and health care delivery [ 23 ].

Clinical utility refers to the extent to which the AI-based medical device improves clinical decision-making, patient outcomes, and health care processes [ 24 ]. The HTA should examine whether the device provides actionable and reliable information that potentially helps health care professionals make more accurate diagnoses or improve treatment plans or monitoring strategies [ 25 , 26 ]. It should also evaluate the potential impact of the device on patient outcomes by assessing, for example, improved survival rates, reduced complications, or enhanced quality of life [ 27 ].

To assess clinical utility, the HTA should consider the device’s performance in relevant clinical scenarios and its ability to address specific clinical questions or challenges. This may involve evaluating the device’s performance against established clinical guidelines or expert opinions as well as considering the device’s potential to fill gaps in clinical practice or enhance existing diagnostic or treatment methods.

Furthermore, the HTA should examine the broader impact of AI-based medical devices on health care systems and resource allocation [ 21 ]. This includes evaluating the devices’ potential to optimize resource use, reduce health care costs, or improve workflow efficiency. Economic evaluations such as cost-effectiveness analyses can provide insights into the value for money and long-term cost savings associated with the adoption of these devices [ 28 , 29 ].

A comprehensive HTA should thoroughly assess the clinical effectiveness and impact on patient outcomes of AI-based medical devices, examining their performance in relevant clinical scenarios, their alignment with clinical guidelines, and their potential to improve health care processes and resource allocation. By evaluating these aspects, the HTA can provide a holistic understanding of the AI devices’ effectiveness and their potential to positively transform health care delivery [ 21 ].

AI-based medical devices must undergo rigorous clinical validation to assess their performance and reliability in real-world health care settings [ 30 , 31 ]. Clinical validation involves evaluating the device’s accuracy, sensitivity, specificity, and overall diagnostic or prognostic performance. In addition, the devices should be tested across diverse patient populations and compared against gold-standard reference methods or expert opinions [ 32 ].

Furthermore, the use of real-world evidence (RWE) is crucial for a comprehensive HTA [ 33 ]. RWE involves gathering data from routine clinical practice, electronic health records, and other sources to evaluate the device’s effectiveness and safety in real-world settings [ 34 ]. These data can help assess the device’s performance in a broader patient population and identify any potential limitations or biases that may arise in specific clinical scenarios.

Robust clinical validation studies and the integration of RWE provide critical evidence for the evaluation of AI-based medical devices in an HTA process [ 4 ]. These studies should include a sufficient sample size, appropriate study design, and statistical analysis to ensure the validity and generalizability of the results [ 35 - 38 ]. By considering clinical validation and RWE adapted to AI-based medical devices, HTA can provide valuable insights into their clinical utility and impact in real-world health care settings, facilitating evidence-based decision-making for their adoption and use.

A lack of safety evaluation was highlighted in a systematic review, showing that only 9% of AI-based medical device studies evaluated safety criteria [ 4 ]. However, safety is crucial for the confidence in and adoption of AI-based technologies for both patients and health care professionals [ 39 ]. While these devices have the potential to improve diagnosis and treatment outcomes, their integration into clinical practice must be accompanied by robust safety evaluations [ 2 , 40 ]. Identifying and mitigating potential risks, such as algorithmic bias, data privacy breaches, and algorithm failures, is essential to protect patient well-being and maintain trust in these innovative technologies [ 41 ].

To ensure the safety of AI-based medical devices, it is crucial to establish standardized evaluation frameworks and guidelines [ 39 ]. These frameworks should encompass rigorous testing methodologies, validation procedures, and continuous monitoring of AI performance. Collaboration among stakeholders, including manufacturers, regulatory agencies, health care providers, and researchers, is essential to develop and implement comprehensive safety evaluation protocols. Addressing the safety gap in AI-based medical devices not only ensures patient welfare but also instills confidence in health care professionals to embrace and use these technologies effectively [ 40 ]. By prioritizing safety evaluation, we can unleash the full potential of AI-based medical devices, leading to transformative advancements in health care delivery [ 1 ].

Domain 5 (Costs and Economic Evaluation)

Costs and economic evaluation play a crucial role in the comprehensive assessment of health technology adoption, particularly in the context of the HTA of AI-based medical devices [ 4 , 9 , 42 ]. While AI technologies show huge potential to improve health care delivery, reduce time to diagnosis, and save money, their economic impact requires careful consideration to ensure their successful integration and sustainability [ 23 ].

When assessing AI-based medical devices through the HTA process, economic evaluations involve analyzing the costs and benefits associated with the adoption and use of the technology [ 4 ]. These evaluations go beyond the upfront costs of acquiring the AI-based medical device; they encompass various factors, such as training, infrastructure modifications, maintenance, and ongoing operational costs [ 43 ]. Cost analysis also includes improved patient outcomes, reduced hospital readmissions, and shortened hospital stays. By conducting economic evaluations, decision makers can gauge the cost-effectiveness and cost utility to weigh the affordability of AI-based medical devices for their particular use against the potential mid- and long-term cost savings considered [ 44 ].

HTA evaluates the suitability of AI-based medical devices by considering their potential economic benefits, risks, and ethical implications [ 45 , 46 ]. By incorporating economic evaluation into the HTA process, decision makers can make evidence-based choices about the allocation of limited health care resources and prioritize interventions that provide the greatest value for money [ 4 , 47 ].

Finally, considering cost savings and conducting economic evaluations as part of the HTA process helps facilitate the adoption of AI-based medical devices [ 37 ]. It provides decision makers with the necessary information to determine the financial feasibility and potential return on investment associated with implementing these technologies. HTA ensures that AI-based medical devices are suitable for integration into the health care system, fostering confidence in their effectiveness, efficiency, and cost utility [ 4 , 43 , 48 ].

Domain 6 (Ethical Analysis) and Domain 8 (Patients and Social Aspects)

Ethical and societal implications are critical aspects that must be considered when evaluating the suitability of AI-based medical devices for a comprehensive HTA [ 45 ]. The integration of AI technologies into health care raises important ethical concerns that need to be addressed to ensure responsible and equitable use [ 49 ].

Data privacy and patient consent are primary ethical considerations [ 50 ]. The HTA should evaluate whether AI-based medical devices adhere to strict data protection regulations, maintain patient confidentiality, and obtain appropriate informed consent for data use. It is essential to assess whether the devices have mechanisms in place to handle sensitive patient information securely and protect it against unauthorized access or data breaches.

Algorithmic fairness and bias are additional ethical concerns [ 51 ]. AI algorithms can inadvertently perpetuate biases present in the data used for training, resulting in unequal treatment or access to health care resources [ 52 ]. The HTA should assess whether the devices have been evaluated for fairness and bias and consider the steps taken to mitigate any identified biases [ 53 ].

Moreover, the HTA should examine the impact of AI-based medical devices on health care disparities and access to care [ 54 ]. It is crucial to assess whether the devices have the potential to exacerbate existing inequalities or whether they can contribute to reducing disparities by improving health care access, particularly for underserved populations.

Accountability and transparency in AI decision-making processes are also important ethical considerations [ 55 ]. The HTA should evaluate whether the AI devices provide clear explanations for their outputs and ensure that health care professionals and patients can understand and challenge the device’s recommendations [ 21 ].

A comprehensive HTA should thoroughly examine the ethical and societal implications of AI-based medical devices, ensuring that they prioritize patient privacy, fairness, and equitable access to care; fostering trust; and ensuring that these devices align with societal values and goals.

To ensure the adoption of AI-based medical devices through a truthfully AI concept, explainability, interpretability, and transparency are important considerations for their comprehensive HTA [ 56 ]. Indeed, these devices often use complex algorithms and machine and deep learning techniques, resulting in their operating as “black boxes” that reach decisions and recommendations that are challenging to understand [ 55 ]. However, explainability and transparency are essential to ensure trust, accountability, and acceptance of AI-based technologies in health care. First, Explainability refers to the ability to understand and interpret the reasoning behind the device’s outputs [ 56 ]. It involves providing clinicians and users with transparent explanations of how the AI algorithm processes input data and generates results. This enables health care professionals to trust the device’s recommendations and make informed decisions based on the provided information. The HTA should evaluate the extent to which AI-based medical devices can provide interpretable and understandable explanations of their decision-making processes [ 17 , 46 ]. Second, Transparency involves disclosing important information about the AI-based medical devices, including the data used for training, algorithmic methodologies used, and potential limitations or biases [ 57 ]. Transparency promotes trust and allows stakeholders to assess the device’s reliability and potential risks. The HTA should assess whether the device manufacturers provide clear documentation and information to health care professionals and patients about the device’s capabilities, limitations, and potential errors [ 21 , 58 , 59 ]. Transparency is closely linked to regulatory considerations [ 60 , 61 ]. The HTA should consider whether the device complies with relevant regulatory standards and whether the manufacturers have provided the necessary documentation and evidence to support their ethical claims [ 21 , 62 , 63 ].

A comprehensive HTA process should address concerns regarding the explainability and transparency of AI-based medical devices, trust, accountability, and ethical implications related to the use of AI in health care [ 45 , 46 ].

Domain 7 (Organizational Aspects)

Organizational aspects play a crucial role in the effective integration of AI-based medical devices into health care systems [ 64 ]. To ensure consistency, transparency, and comparability in the evaluation process, there is a growing need for a robust methodological framework that provides standardized guidelines for assessing the organizational impact of the implementation of AI-based medical devices [ 65 ]. According to a descriptive analysis led in German hospitals, the main barriers to AI-based medical device adoption were lack of resources (staff, knowledge, and financial) [ 66 ].

Clear indicators are needed to measure the organizational readiness for and impact of AI-based medical devices [ 4 , 10 , 67 ]. Several criteria have been highlighted, such as (1) health care workplace readiness and stakeholder acceptance, (2) AI-based medical device organization alignment assessment, and (3) business plan (financing and investments) [ 64 ].

Domain 9 (Legal Aspects)

The roles and responsibilities of health care professionals are also impacted by these AI solutions [ 68 ]. It is necessary to evaluate whether they complement or replace health care professionals’ expertise and whether additional training, supervision, or support is required for their optimal use [ 69 , 70 ]. As AI technologies could evolve and become more prevalent in health care, it is crucial to ensure that these devices comply with existing legal frameworks and regulations [ 49 , 71 ]. HTA plays a pivotal role in evaluating the legal implications of AI-based medical devices by assessing factors such as data privacy, security, liability, and regulatory compliance [ 72 ].

One key legal aspect to consider is data privacy and protection [ 73 , 74 ]. AI-based medical devices often rely on vast amounts of patient data for training and decision-making. Therefore, it is essential to evaluate whether these devices adhere to relevant data protection laws, such as the General Data Protection Regulation in the European Union [ 75 , 76 ]. HTA examines the measures taken by AI-based medical device manufacturers to safeguard patient privacy, including data anonymization, encryption, and secure data storage practices [ 37 ]. These issues are highlighted in the Artificial Intelligence Act published in June 2023 by the European Commission [ 77 , 78 ].

Cybersecurity is another critical consideration [ 21 , 79 ]. As AI-based medical devices handle sensitive patient data and make critical health care decisions, it is crucial to assess the security measures implemented to prevent unauthorized access, data breaches, or tampering [ 80 ]. HTA evaluates the robustness of the security protocols implemented by device manufacturers and their compliance with industry standards and regulations [ 21 , 81 ].

Liability is also a significant legal aspect to be addressed in the context of AI-based medical devices [ 82 , 83 ]. When errors or adverse events occur due to the use of these devices, determining liability can be complex [ 84 ]. HTA examines the legal frameworks and liability guidelines pertaining to AI technologies, including whether clear guidelines exist regarding the responsibility of manufacturers, health care providers, and users in case of device malfunctions or errors [ 85 ]. Assessing liability aspects within the HTA process helps establish accountability and ensures that legal frameworks adequately address potential risks [ 86 ].

Regulatory compliance is a crucial consideration when assessing the suitability of AI-based medical devices for HTA [ 87 ]. Depending on the jurisdiction, AI devices need to undergo regulatory approval processes before being introduced into the market [ 88 ]. HTA examines whether AI-based medical device manufacturers have obtained the necessary regulatory approvals, such as clearance from the relevant health authorities or certification from regulatory bodies such as the Food and Drug Administration (FDA) in the United States or notified bodies in Europe [ 89 ]. This evaluation ensures that AI-based medical devices comply with existing regulations and are fit for clinical use [ 61 ].

Recommendations to Adapt the 9 HTA Domains for AI-Based Medical Devices

Taking into account the previous considerations, some recommendations could be proposed to adapt and personalize these standard HTA domains to AI-based medical devices’ specificities. Therefore, we suggest 4 main recommendations by domain in Table 3 .

a ML: machine learning.

b DL: deep learning.

c DMS: diabetes management system.

d FDA: Food and Drug Administration.

The adaptation of the HTA process through a methodological framework for AI-based medical devices enhances the comparability of results across different evaluations and jurisdictions [ 90 - 92 ]. It promotes consistency in the assessment methodologies, reporting formats, and presentation of findings, enabling decision makers to make informed choices based on reliable and comparable evidence. This standardization contributes to the overall credibility and acceptance of HTA outcomes related to AI-based medical devices [ 93 ]. In addition, a methodological framework would address the need for appropriate expertise and skills to conduct the HTA of AI-based medical devices [ 94 - 99 ]. It would outline the qualifications and competencies required for the individuals involved in the assessment, including knowledge of AI technologies [ 46 ]. By defining the necessary expertise, the framework supports the development of a skilled workforce capable of conducting robust and reliable HTAs of AI-based medical devices ( Figure 2 ).

Evaluating the trade-offs and weighing different features of AI-based medical devices is indeed a complex and important task, especially in domains such as health care where the impact on human lives is significant. The acceptability of AI-based medical devices should be assessed on a case-by-case basis considering various factors, including, for instance, performance, accuracy, cost, explainability, and the specific context in which they are being used.

Trade-offs between accuracy and cost are common in AI. It may be acceptable for AI to increase follow-up care costs if it significantly improves accuracy and patient outcomes. For example, if an AI system can detect diseases at an earlier stage, it might lead to more effective treatment and, ultimately, lower overall health care costs in the long run.

The balance between explainability and performance is a critical consideration. While explainable algorithms are preferred for safety and transparency reasons, there may be cases in which a highly complex, unexplainable algorithm outperforms explainable ones. In such cases, the trade-off between transparency and performance should be carefully evaluated based on the specific use case and potential risks.

The minimum performance for adding this technology to the tools available to human clinicians depends on the specific task and the level of trust that patients, health care providers, and regulators have in the technology. Some key factors to consider include the following: (1) The complexity of the task—AI-based medical devices should excel in tasks that are well defined and data driven, but they may not replace human clinicians in tasks requiring complex decision-making, empathy, or ethical considerations. (2) Safety and reliability—AI-based medical devices should demonstrate a high level of safety and reliability, ideally surpassing the performance of human clinicians in terms of avoiding errors. (3) Ethical considerations—AI-based medical devices should adhere to ethical standards, including patient privacy, informed consent, and unbiased decision-making, which are often considered even more important than performance metrics. (4) Regulatory approval—regulatory bodies often establish performance thresholds for AI-based medical devices. Compliance with these thresholds is essential for market acceptance.

In general, AI should aim to complement and enhance the capabilities of human clinicians rather than completely replacing them. The specific threshold for acceptable performance will vary across applications and contexts, and it should be determined through a combination of rigorous testing, peer-reviewed studies, and input from health care professionals and patients.

It is important to note that ethical considerations, patient safety, and the potential for bias should always be at the forefront of these discussions, and AI-based medical devices should not be adopted solely for the sake of automation or cost reduction if they compromise these critical aspects of health care.

Use Case of the Application of the Aforementioned HTA Recommendations for AI: AI-Based Medical Devices in Pathways for Patients With Diabetes

Concerning the HTA domain 3 on the safety of an AI-based medical device, the potential risk of patient injury of an insulin delivery AI-based medical device system should be taken into account in the risk management process of algorithm development [ 96 ]. In the case of an evolutive deep learning–based medical device without continuous ongoing safety assessment, a wrong dosage administration due to an AI error, for instance, could provoke a serious adverse event such as an acid ketosis coma for a patient with diabetes.

A risk management plan should be available for users and regularly updated with the impact of the AI changes on the safety plan.

In relation to the HTA domain 4, which focuses on the effectiveness of an AI-based medical device, the FDA in the United States has proposed a regulatory framework for modifications to evolutive AI- and machine learning–based software as a medical device with modification guidance focused on the risk to users or patients resulting from the AI changes [ 100 ]. They have asked for an Algorithm Change Protocol with specific methods in place to achieve and control the risks of the anticipated types of modifications related to performance, use, or inputs. A continuous clinical effectiveness assessment could be an interesting approach for an AI-based medical device detection system for diabetes foot ulcer for patients needing adaptative treatment modifications to prevent ulcer development [ 101 ].

Concerning the HTA domain 5, to evaluate a diabetic retinopathy screening AI-based medical device, clinical effectiveness is not sufficient. The cost-effectiveness of different types of AI diabetic retinopathy screening should be compared with no screening and ophthalmologist screening. A recent study demonstrated that AI-based screening was the most cost-effective, not only saving costs but also improving the quality of life of patients with diabetes [ 44 ]. In this case, the long-term assessment of the economic impact of AI introduction in diabetic retinopathy screening highlighted the added value of this technology.

The HTA domain 7 on organizational impact has to assess how the AI-based medical device can be effectively integrated into the health care pathway and prevent wasteful spending. More thorough attention must be paid to the following aspects: (1) evaluating needs and determining the added value of the implementation of the AI-based medical device; (2) assessing workplace preparedness, including stakeholder acceptance of the introduction of the AI-based medical device and involvement; and (3) analyzing the alignment between AI technology and organizational structure [ 64 ]. Decision makers and technology advocates need to address the complexities of AI more comprehensively and understand the systemic challenges that its adoption poses to health care organizations and systems. As an example, consider an AI tool used for diagnosing diabetic retinopathy in a primary care setting, such as by a family physician or nurse [ 102 ]. In theory, this could lead to shorter waiting times for patients. However, if the health care organization faces challenges such as a shortage of specialized staff (eg, ophthalmologists), insufficient organization of care pathways, and lack of specialized facilities for proper management and follow-up after diagnosis, the introduction of AI might adversely affect the quality of care and patient experience. In such a scenario, the AI application might merely transfer the delay from primary to secondary care, failing to address the fundamental issue.

Finally, concerning the last domains (6, 8, and 9) about ethical, patient, social, and legal aspects, AI-based continuous glucose monitoring and insulin pumps should also be assessed on ethical and legal aspects [ 103 ]. While citizen juries have generally shown support for AI in research and treatment, concerns remain. The risks of data theft and privacy breaches necessitate careful consideration of ethical and legal issues for patients. Although AI can aid in decision-making, it cannot wholly substitute a physician’s expertise. Effective regulations and systems designed to ensure safety, reduce bias, and enhance transparency are essential.

By implementing these recommendations, stakeholders can foster the responsible integration, regulation, and evaluation of AI-based medical devices. These measures can enhance the evidence base, address ethical concerns, and maximize the potential benefits of these AI technologies in improving health care outcomes while protecting patient safety, privacy, and equity.

Practical Challenges and Potential Barriers to Implementing HTAs Specific for AI-Based Medical Devices

When it comes to implementing specialized HTAs for AI-based medical devices, several practical challenges and potential barriers could be encountered and should be taken into account: First, AI technological evolution timeline—AI technologies evolve at a much faster pace compared to traditional medical devices, making it challenging for HTA frameworks to keep up with the latest developments and assess their long-term impact effectively. AI-based medical devices have a short product lifetime, between 12 and 18 months, in contrast to drug products. This shorter life cycle highlights the need for evolutive and fast-track HTA processes for AI-based medical devices. Second, data requirements and quality—AI systems rely heavily on large data sets for training and validation. Ensuring the availability of high-quality, representative data is a significant challenge. There is also the issue of data privacy and security, which must be addressed. The availability and quality of data and evidence required for conducting HTAs on AI-based medical devices present a complex and evolving landscape. Assessing the feasibility and challenges associated with gathering such data is crucial for robust evaluations. First, the availability of data can vary significantly depending on the AI-based medical device. While some devices may have access to vast amounts of high-quality real-world patient data, others might face limitations due to the novelty of the technology or issues related to data privacy. Second, the quality of data is paramount as inaccurate or biased data can lead to flawed assessments. Ensuring data accuracy, representativeness, and relevance is a constant challenge in AI-based medical device evaluations [ 104 ]. The rapid pace of AI development can result in limited long-term data, making it difficult to assess the device’s sustained performance and safety [ 105 ]. Balancing the need for robust evidence with the dynamic nature of AI technologies is a significant challenge that HTA organizations must address to provide valuable insights for informed decision-making in health care. Third, complexity and transparency—the complex algorithms used in AI-based medical devices can be difficult to understand and assess, leading to issues with transparency and explainability [ 17 ]. This complexity can pose a challenge for regulators and assessors in HTA processes. Fourth, clinical validation and safety requirements—generating robust clinical evidence to demonstrate the safety, efficacy, and effectiveness of AI-based medical devices can be challenging. This includes proving that these devices perform consistently across diverse patient populations. Moreover, to account for a life cycle that would include updates that may improve performance, one solution to investigate could be the proposition of the FDA in the United States in 2019 to experiment with a “dynamic certification” [ 100 ]. It allows for the re-evaluation of the AI-based medical device in case of a substantial modification of the indication of the medical device or the way to deliver the diagnosis, for instance. Fifth, regulatory and ethical considerations—adapting existing regulatory frameworks to accommodate AI-based medical devices, addressing ethical concerns such as bias, and ensuring equitable access are critical challenges [ 106 ]. Sixth, economic evaluation—determining the cost-effectiveness of AI-based medical devices, especially when benefits might be indirect or long term, poses a unique challenge for HTA [ 93 ]. Seventh, stakeholder engagement and trust—building trust among health care providers, patients, and policy makers regarding the reliability, trustworthiness, and usefulness of AI-based medical devices is crucial but challenging [ 107 ]. Eighth, integration into health care systems and interoperability—the integration of AI-based medical devices into existing health care workflows and systems can be complex and resource intensive [ 18 ]. Ninth, global and local applicability—ensuring that AI-based medical devices are effective and appropriate for use in different global and local contexts, considering varying health care systems and population needs, is another significant barrier.

Practically, one suggestion to implement such frameworks could be to implement these recommendations in the future European clinical joint assessment guidelines. As they are being currently discussed in a European project on a common HTA process for connected medical devices that includes AI-based medical devices, it could be the opportunity to tackle these challenges at the European level [ 108 ].

Addressing these challenges requires a concerted effort from regulators, health care providers, technology developers, and other stakeholders in the health care ecosystem.

Conclusions

AI-based medical devices have the potential to transform health care delivery, but the suitability of the current comprehensive HTA requires careful adaptation of the evaluation across the 9 dimensions. While these AI devices show promise in improving accuracy, safety, and efficiency, there is a need for robust clinical validation, integration into workflows, economic evaluation, and addressing of ethical and legal implications. A comprehensive adapted HTA framework for AI-based medical devices can provide valuable insights into their effectiveness, cost-effectiveness, and societal impact, guiding their responsible implementation and maximizing their benefits for patients and health care systems.

Data Availability

The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

Authors' Contributions

LF and AV contributed to the conceptualization. LF and AV contributed to the methodology. LF, AV, NM, and IB contributed to the article selection. LF and AV contributed to the writing and original draft preparation. NM and IB contributed to the writing and review and editing. LF and AV contributed to the validation. The authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

None declared.

The 9 health technology assessment (HTA) domains for the assessment of artificial intelligence–based medical devices adapted from the HTA Core Model proposed by the European Network for Health Technology Assessment.

Summary results of the scoping review and general characteristics of the studies included in the scoping review.

• Bajwa J, Munir U, Nori A, Williams B. Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J. Jul 2021;8(2):e188-e194. [ CrossRef ] [ Medline ]
• Bohr A, Memarzadeh K. The rise of artificial intelligence in healthcare applications. In: Artificial Intelligence in Healthcare. Cambridge, MA. Academic Press; 2020.
• Park CW, Seo SW, Kang N, Ko B, Choi BW, Park CM, et al. Artificial intelligence in health care: current applications and issues. J Korean Med Sci. Nov 02, 2020;35(42):e379. [ CrossRef ] [ Medline ]
• Farah L, Davaze-Schneider J, Martin T, Nguyen P, Borget I, Martelli N. Are current clinical studies on artificial intelligence-based medical devices comprehensive enough to support a full health technology assessment? A systematic review. Artif Intell Med. Jun 2023;140:102547. [ CrossRef ] [ Medline ]
• HTA core model. European Network for Health Technology Assessment. URL: https://www.eunethta.eu/hta-core-model/ [accessed 2021-10-20]
• Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med. Sep 2020;26(9):1364-1374. [ CrossRef ] [ Medline ]
• Rivera SC, Liu X, Chan AW, Denniston AK, Calvert MJ, SPIRIT-AICONSORT-AI Working Group. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension. BMJ. Sep 09, 2020;370:m3210. [ CrossRef ] [ Medline ]
• Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma JB, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. Jul 09, 2021;11(7):e048008. [ CrossRef ] [ Medline ]
• Chen Y. Health technology assessment and economic evaluation: is it applicable for the traditional medicine? Integr Med Res. Mar 2022;11(1):100756. [ CrossRef ] [ Medline ]
• He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nat Med. Jan 2019;25(1):30-36. [ CrossRef ] [ Medline ]
• Giordano C, Brennan M, Mohamed B, Rashidi P, Modave F, Tighe P. Accessing artificial intelligence for clinical decision-making. Front Digit Health. Jun 25, 2021;3:645232. [ CrossRef ] [ Medline ]
• Vayena E, Blasimme A, Cohen IG. Machine learning in medicine: addressing ethical challenges. PLoS Med. Nov 6, 2018;15(11):e1002689. [ CrossRef ] [ Medline ]
• Dzobo K, Adotey S, Thomford NE, Dzobo W. Integrating artificial and human intelligence: a partnership for responsible innovation in biomedical engineering and medicine. OMICS. May 2020;24(5):247-263. [ CrossRef ] [ Medline ]
• Zawati M, Lang M. What's in the box?: uncertain accountability of machine learning applications in healthcare. Am J Bioeth. Nov 2020;20(11):37-40. [ CrossRef ] [ Medline ]
• Peters MD, Godfrey C, McInerney P, Khalil H, Larsen P, Marnie C, et al. Best practice guidance and reporting items for the development of scoping review protocols. JBI Evid Synth. Apr 01, 2022;20(4):953-968. [ CrossRef ] [ Medline ]
• Pollock D, Peters MD, Khalil H, McInerney P, Alexander L, Tricco AC, et al. Recommendations for the extraction, analysis, and presentation of results in scoping reviews. JBI Evid Synth. Mar 01, 2023;21(3):520-532. [ CrossRef ] [ Medline ]
• Farah L, Murris JM, Borget I, Guilloux A, Martelli NM, Katsahian SI. Assessment of performance, interpretability, and explainability in artificial intelligence–based health technologies: what healthcare stakeholders need to know. Mayo Clin Proc Digit Health. Jun 2023;1(2):120-138. [ CrossRef ]
• Lehne M, Sass J, Essenwanger A, Schepers J, Thun S. Why digital medicine depends on interoperability. NPJ Digit Med. Aug 20, 2019;2:79. [ CrossRef ] [ Medline ]
• Esmaeilzadeh P. Use of AI-based tools for healthcare purposes: a survey study from consumers' perspectives. BMC Med Inform Decis Mak. Jul 22, 2020;20(1):170. [ CrossRef ] [ Medline ]
• Zia A, Aziz M, Popa I, Khan SA, Hamedani AF, Asif AR. Artificial intelligence-based medical data mining. J Pers Med. Aug 24, 2022;12(9):1359. [ CrossRef ] [ Medline ]
• Ming J, He Y, Yang Y, Hu M, Zhao X, Liu J, et al. Health technology assessment of medical devices: current landscape, challenges, and a way forward. Cost Eff Resour Alloc. Oct 05, 2022;20(1):54. [ CrossRef ] [ Medline ]
• Filkins BL, Kim JY, Roberts B, Armstrong W, Miller MA, Hultner ML, et al. Privacy and security in the era of digital health: what should translational researchers know and do about it? Am J Transl Res. Mar 15, 2016;8(3):1560-1580. [ Medline ]
• Johnson KB, Wei WQ, Weeraratne D, Frisse ME, Misulis K, Rhee K, et al. Precision medicine, AI, and the future of personalized health care. Clin Transl Sci. Jan 2021;14(1):86-93. [ CrossRef ] [ Medline ]
• Sloane EJ, Silva RJ. Artificial intelligence in medical devices and clinical decision support systems. In: Clinical Engineering Handbook (Second Edition). Cambridge, MA. Academic Press; 2020.
• Steuten LM. Early stage health technology assessment for precision biomarkers in oral health and systems medicine. OMICS. Jan 2016;20(1):30-35. [ CrossRef ] [ Medline ]
• Javaid M, Haleem A, Pratap Singh R, Suman R, Rab S. Significance of machine learning in healthcare: features, pillars and applications. Int J Intell Networks. 2022;3:58-73. [ CrossRef ]
• Rowland SP, Fitzgerald JE, Holme T, Powell J, McGregor A. What is the clinical value of mHealth for patients? NPJ Digit Med. Jan 13, 2020;3:4. [ CrossRef ] [ Medline ]
• Voets MM, Veltman J, Slump CH, Siesling S, Koffijberg H. Systematic review of health economic evaluations focused on artificial intelligence in healthcare: the tortoise and the cheetah. Value Health. Mar 2022;25(3):340-349. [ CrossRef ] [ Medline ]
• Kirisits A, Redekop WK. The economic evaluation of medical devices: challenges ahead. Appl Health Econ Health Policy. Feb 2013;11(1):15-26. [ CrossRef ] [ Medline ]
• Park SH, Choi J, Byeon JS. Key principles of clinical validation, device approval, and insurance coverage decisions of artificial intelligence. Korean J Radiol. Mar 2021;22(3):442-453. [ CrossRef ] [ Medline ]
• Tsopra R, Fernandez X, Luchinat C, Alberghina L, Lehrach H, Vanoni M, et al. A framework for validating AI in precision medicine: considerations from the European ITFoC consortium. BMC Med Inform Decis Mak. Oct 02, 2021;21(1):274. [ CrossRef ] [ Medline ]
• Bolboacă SD. Medical diagnostic tests: a review of test anatomy, phases, and statistical treatment of data. Comput Math Methods Med. May 28, 2019;2019:1891569. [ CrossRef ] [ Medline ]
• Hogervorst MA, Pontén J, Vreman RA, Mantel-Teeuwisse AK, Goettsch WG. Real world data in health technology assessment of complex health technologies. Front Pharmacol. Feb 10, 2022;13:837302. [ CrossRef ] [ Medline ]
• Simon GE, Bindman AB, Dreyer NA, Platt R, Watanabe JH, Horberg M, et al. When can we trust real-world data to evaluate new medical treatments? Clin Pharmacol Ther. Jan 2022;111(1):24-29. [ CrossRef ] [ Medline ]
• Pongiglione B, Torbica A, Blommestein H, de Groot S, Ciani O, Walker S, et al. Do existing real-world data sources generate suitable evidence for the HTA of medical devices in Europe? Mapping and critical appraisal. Int J Technol Assess Health Care. Apr 26, 2021;37(1):e62. [ CrossRef ] [ Medline ]
• Pongiglione B, Torbica A. How real can we get in generating real world evidence? Exploring the opportunities of routinely collected administrative data for evaluation of medical devices. Health Econ. Sep 2022;31 Suppl 1(Suppl 1):25-43. [ CrossRef ] [ Medline ]
• Zemplényi A, Tachkov K, Balkanyi L, Németh B, Petykó ZI, Petrova G, et al. Recommendations to overcome barriers to the use of artificial intelligence-driven evidence in health technology assessment. Front Public Health. Apr 26, 2023;11:1088121. [ CrossRef ] [ Medline ]
• Daubner-Bendes R, Kovács S, Niewada M, Huic M, Drummond M, Ciani O, et al. Quo Vadis HTA for medical devices in central and eastern Europe? Recommendations to address methodological challenges. Front Public Health. Jan 08, 2021;8:612410. [ CrossRef ] [ Medline ]
• Larson DB, Harvey H, Rubin DL, Irani N, Tse JR, Langlotz CP. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: summary and recommendations. J Am Coll Radiol. Mar 2021;18(3 Pt A):413-424. [ CrossRef ] [ Medline ]
• Choudhury A, Asan O. Role of artificial intelligence in patient safety outcomes: systematic literature review. JMIR Med Inform. Jul 24, 2020;8(7):e18599. [ CrossRef ] [ Medline ]
• Belenguer L. AI bias: exploring discriminatory algorithmic decision-making models and the application of possible machine-centric solutions adapted from the pharmaceutical industry. AI Ethics. 2022;2(4):771-787. [ CrossRef ] [ Medline ]
• Binder L, Ghadban M, Sit C, Barnard K. Health technology assessment process for oncology drugs: impact of CADTH changes on public payer reimbursement recommendations. Curr Oncol. Mar 01, 2022;29(3):1514-1526. [ CrossRef ] [ Medline ]
• Wolff J, Pauling J, Keck A, Baumbach J. The economic impact of artificial intelligence in health care: systematic review. J Med Internet Res. Feb 20, 2020;22(2):e16866. [ CrossRef ] [ Medline ]
• Abràmoff MD, Roehrenbeck C, Trujillo S, Goldstein J, Graves AS, Repka MX, et al. A reimbursement framework for artificial intelligence in healthcare. NPJ Digit Med. Jun 09, 2022;5(1):72. [ CrossRef ] [ Medline ]
• Bélisle-Pipon JC, Couture V, Roy MC, Ganache I, Goetghebeur M, Cohen IG. What makes artificial intelligence exceptional in health technology assessment? Front Artif Intell. Nov 02, 2021;4:736697. [ CrossRef ] [ Medline ]
• Alami H, Lehoux P, Auclair Y, de Guise M, Gagnon MP, Shaw J, et al. Artificial intelligence and health technology assessment: anticipating a new level of complexity. J Med Internet Res. Jul 07, 2020;22(7):e17707. [ CrossRef ] [ Medline ]
• Love-Koh J, Peel A, Rejon-Parrilla JC, Ennis K, Lovett R, Manca A, et al. The future of precision medicine: potential impacts for health technology assessment. Pharmacoeconomics. Dec 2018;36(12):1439-1451. [ CrossRef ] [ Medline ]
• Gomez Rossi J, Rojas-Perilla N, Krois J, Schwendicke F. Cost-effectiveness of artificial intelligence as a decision-support system applied to the detection and grading of melanoma, dental caries, and diabetic retinopathy. JAMA Netw Open. Mar 01, 2022;5(3):e220269. [ CrossRef ] [ Medline ]
• Naik N, Hameed BM, Shetty DK, Swain D, Shah M, Paul R, et al. Legal and ethical consideration in artificial intelligence in healthcare: who takes responsibility? Front Surg. Mar 14, 2022;9:862322. [ CrossRef ] [ Medline ]
• Haynes CL, Cook GA, Jones MA. Legal and ethical considerations in processing patient-identifiable data without patient consent: lessons learnt from developing a disease register. J Med Ethics. May 2007;33(5):302-307. [ CrossRef ] [ Medline ]
• McCradden MD, Joshi S, Mazwi M, Anderson JA. Ethical limitations of algorithmic fairness solutions in health care machine learning. Lancet Digit Health. May 2020;2(5):e221-e223. [ CrossRef ] [ Medline ]
• Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. Nov 01, 2018;178(11):1544-1547. [ CrossRef ] [ Medline ]
• Fletcher RR, Nakeshimana A, Olubeko O. Addressing fairness, bias, and appropriate use of artificial intelligence and machine learning in global health. Front Artif Intell. Apr 15, 2021;3:561802. [ CrossRef ] [ Medline ]
• Tachkov K, Zemplenyi A, Kamusheva M, Dimitrova M, Siirtola P, Pontén J, et al. Barriers to use artificial intelligence methodologies in health technology assessment in central and east European countries. Front Public Health. Jul 14, 2022;10:921226. [ CrossRef ] [ Medline ]
• Durán JM, Jongsma KR. Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI. J Med Ethics. Mar 18, 2021. (forthcoming). [ CrossRef ] [ Medline ]
• Amann J, Blasimme A, Vayena E, Frey D, Madai VI, Precise4Q consortium. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Med Inform Decis Mak. Nov 30, 2020;20(1):310. [ CrossRef ] [ Medline ]
• Kiseleva A, Kotzinos D, De Hert P. Transparency of AI in healthcare as a multilayered system of accountabilities: between legal requirements and technical limitations. Front Artif Intell. May 30, 2022;5:879603. [ CrossRef ] [ Medline ]
• Baltaxe E, Hsieh HW, Roca J, Cano I. The assessment of medical device software supporting health care services for chronic patients in a tertiary hospital: overarching study. J Med Internet Res. Jan 04, 2023;25:e40976. [ CrossRef ] [ Medline ]
• Garfield S, Polisena J, S Spinner D, Postulka A, Y Lu C, Tiwana SK, et al. Health technology assessment for molecular diagnostics: practices, challenges, and recommendations from the medical devices and diagnostics special interest group. Value Health. 2016;19(5):577-587. [ CrossRef ] [ Medline ]
• Fraser AG, Biasin E, Bijnens B, Bruining N, Caiani EG, Cobbaert K, et al. Artificial intelligence in medical device software and high-risk medical devices - a review of definitions, expert recommendations and regulatory initiatives. Expert Rev Med Devices. Jun 2023;20(6):467-491. [ CrossRef ] [ Medline ]
• Beckers R, Kwade Z, Zanca F. The EU medical device regulation: implications for artificial intelligence-based medical device software in medical physics. Phys Med. Mar 2021;83:1-8. [ CrossRef ] [ Medline ]
• Melvin T, Torre M. New medical device regulations: the regulator's view. EFORT Open Rev. Jun 03, 2019;4(6):351-356. [ CrossRef ] [ Medline ]
• Fleetcroft C, McCulloch P, Campbell B. IDEAL as a guide to designing clinical device studies consistent with the new European Medical Device Regulation. BMJ Surg Interv Health Technol. Mar 04, 2021;3(1):e000066. [ CrossRef ] [ Medline ]
• Alami H, Lehoux P, Denis JL, Motulsky A, Petitgand C, Savoldelli M, et al. Organizational readiness for artificial intelligence in health care: insights for decision-making and practice. J Health Organ Manag. Dec 03, 2020. (forthcoming). [ CrossRef ] [ Medline ]
• Segur-Ferrer J, Moltó-Puigmartí C, Pastells-Peiró R, Vivanco-Hidalgo RM. Methodological frameworks and dimensions to be taken into consideration in digital health technology assessment: protocol for a scoping review. JMIR Res Protoc. Oct 11, 2022;11(10):e39905. [ CrossRef ] [ Medline ]
• Weinert L, Müller J, Svensson L, Heinze O. Perspective of information technology decision makers on factors influencing adoption and implementation of artificial intelligence technologies in 40 German hospitals: descriptive analysis. JMIR Med Inform. Jun 15, 2022;10(6):e34678. [ CrossRef ] [ Medline ]
• de Hond AA, Leeuwenberg AM, Hooft L, Kant IM, Nijman SW, van Os HJ, et al. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. NPJ Digit Med. Jan 10, 2022;5(1):2. [ CrossRef ] [ Medline ]
• Ahuja AS. The impact of artificial intelligence in medicine on the future role of the physician. PeerJ. 2019;7:e7702. [ CrossRef ] [ Medline ]
• Widrig D, Tag B. HTA and its legal issues: a framework for identifying legal issues in health technology assessment. Int J Technol Assess Health Care. Dec 2014;30(6):587-594. [ CrossRef ] [ Medline ]
• Vella Bonanno P, Bucsics A, Simoens S, Martin AP, Oortwijn W, Gulbinovič J, et al. Proposal for a regulation on health technology assessment in Europe - opinions of policy makers, payers and academics from the field of HTA. Expert Rev Pharmacoecon Outcomes Res. Jun 2019;19(3):251-261. [ CrossRef ] [ Medline ]
• McKee M, Wouters OJ. The challenges of regulating artificial intelligence in healthcare comment on "clinical decision support and new regulatory frameworks for medical devices: are we ready for it? - a viewpoint paper". Int J Health Policy Manag. 2023;12:7261. [ CrossRef ] [ Medline ]
• Hordern V. Data protection compliance in the age of digital health. Eur J Health Law. Jun 2016;23(3):248-264. [ CrossRef ] [ Medline ]
• Stanberry B. The legal and ethical aspects of telemedicine. 2: data protection, security and European law. J Telemed Telecare. 1998;4(1):18-24. [ CrossRef ] [ Medline ]
• Poullet Y. Legal aspects of data protection in medical informatics. Stud Health Technol Inform. 1991;1:138-160. [ Medline ]
• Dove ES, Chen J. To what extent does the EU general data protection regulation (GDPR) apply to citizen scientist-led health research with mobile devices? J Law Med Ethics. Mar 2020;48(1_suppl):187-195. [ CrossRef ] [ Medline ]
• Marovic B, Curcin V. Impact of the European general data protection regulation (GDPR) on health data management in a European Union candidate country: a case study of Serbia. JMIR Med Inform. Apr 17, 2020;8(4):e14604. [ CrossRef ] [ Medline ]
• EU Artificial Intelligence Act. URL: https://artificialintelligenceact.eu/ [accessed 2023-07-24]
• Proposal for a regulation of the European parliament and of the council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain union legislative acts. European Union. 2021. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206 [accessed 2023-07-24]
• Camara C, Peris-Lopez P, Tapiador JE. Security and privacy issues in implantable medical devices: a comprehensive survey. J Biomed Inform. Jun 2015;55:272-289. [ CrossRef ] [ Medline ]
• Migliore A, Ratti M, Cerbo M, Jefferson T. Health Technology Assessment: managing the introduction and use of medical devices in clinical practice in Italy. Expert Rev Med Devices. May 2009;6(3):251-257. [ CrossRef ] [ Medline ]
• Pisapia A, Banfi G, Tomaiuolo R. The novelties of the regulation on health technology assessment, a key achievement for the European Union health policies. Clin Chem Lab Med. Jul 26, 2022;60(8):1160-1163. [ CrossRef ] [ Medline ]
• Maliha G, Gerke S, Cohen IG, Parikh RB. Artificial intelligence and liability in medicine: balancing safety and innovation. Milbank Q. Sep 2021;99(3):629-647. [ CrossRef ] [ Medline ]
• Jassar S, Adams SJ, Zarzeczny A, Burbridge BE. The future of artificial intelligence in medicine: medical-legal considerations for health leaders. Healthc Manage Forum. May 2022;35(3):185-189. [ CrossRef ] [ Medline ]
• Samore MH, Evans RS, Lassen A, Gould P, Lloyd J, Gardner RM, et al. Surveillance of medical device-related hazards and adverse events in hospitalized patients. JAMA. Jan 21, 2004;291(3):325-334. [ CrossRef ] [ Medline ]
• Bleher H, Braun M. Diffused responsibility: attributions of responsibility in the use of AI-driven clinical decision support systems. AI Ethics. 2022;2(4):747-761. [ CrossRef ] [ Medline ]
• Street J, Stafinski T, Lopes E, Menon D. Defining the role of the public in Health Technology Assessment (HTA) and HTA-informed decision-making processes. Int J Technol Assess Health Care. Apr 2020;36(2):87-95. [ CrossRef ] [ Medline ]
• Massella M, Dri DA, Gramaglia D. Regulatory considerations on the use of machine learning based tools in clinical trials. Health Technol (Berl). 2022;12(6):1085-1096. [ CrossRef ] [ Medline ]
• Marcus HJ, Payne CJ, Hughes-Hallett A, Marcus AP, Yang GZ, Darzi A, et al. Regulatory approval of new medical devices: cross sectional study. BMJ. May 20, 2016;353:i2587. [ CrossRef ] [ Medline ]
• Milam ME, Koo CW. The current status and future of FDA-approved artificial intelligence tools in chest radiology in the United States. Clin Radiol. Feb 2023;78(2):115-122. [ CrossRef ] [ Medline ]
• Vreman RA, Mantel-Teeuwisse AK, Hövels AM, Leufkens HG, Goettsch WG. Differences in health technology assessment recommendations among European jurisdictions: the role of practice variations. Value Health. Jan 2020;23(1):10-16. [ CrossRef ] [ Medline ]
• Allen N, Walker SR, Liberti L, Salek S. Health technology assessment (HTA) case studies: factors influencing divergent HTA reimbursement recommendations in Australia, Canada, England, and Scotland. Value Health. Mar 2017;20(3):320-328. [ CrossRef ] [ Medline ]
• Agboola F, Wright AC, Herron-Smith S, Mathur D, Rind D. Evaluation of diversity of clinical trials informing health technology assessments in the United States: a 5-year analysis of institute for clinical and economic review assessments. Value Health. Sep 2023;26(9):1345-1352. [ CrossRef ] [ Medline ]
• Hendrix N, Veenstra DL, Cheng M, Anderson NC, Verguet S. Assessing the economic value of clinical artificial intelligence: challenges and opportunities. Value Health. Mar 2022;25(3):331-339. [ CrossRef ] [ Medline ]
• Nielsen CP, Lauritsen SW, Kristensen FB, Bistrup ML, Cecchetti A, Turk E. Involving stakeholders and developing a policy for stakeholder involvement in the European network for health technology assessment, EUnetHTA. Int J Technol Assess Health Care. Dec 2009;25 Suppl 2:84-91. [ CrossRef ] [ Medline ]
• Scott AM, Wale JL, HTAi PatientCitizen Involvement in HTA Interest Group‚ Patient InvolvementEducation Working Group. Patient advocate perspectives on involvement in HTA: an international snapshot. Res Involv Engagem. Jan 10, 2017;3:2. [ CrossRef ] [ Medline ]
• Gagnon MP, Desmartis M, Gagnon J, St-Pierre M, Gauvin FP, Rhainds M, et al. Introducing the patient's perspective in hospital health technology assessment (HTA): the views of HTA producers, hospital managers and patients. Health Expect. Dec 2014;17(6):888-900. [ CrossRef ] [ Medline ]
• Gagnon M, Desmartis M, Gagnon J, St-Pierre M, Rhainds M, Coulombe M, et al. Framework for user involvement in health technology assessment at the local level: views of health managers, user representatives, and clinicians. Int J Technol Assess Health Care. Jan 2015;31(1-2):68-77. [ CrossRef ] [ Medline ]
• Angelis A, Kanavos P. Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics. May 2016;34(5):435-446. [ CrossRef ] [ Medline ]
• Wale JL, Thomas S, Hamerlijnck D, Hollander R. Patients and public are important stakeholders in health technology assessment but the level of involvement is low - a call to action. Res Involv Engagem. Jan 05, 2021;7(1):1. [ CrossRef ] [ Medline ]
• Artificial intelligence and machine learning (AI/ML)-enabled medical devices. U.S. Food & Drug Administration. Oct 19, 2023. URL: https://tinyurl.com/m8ba5aw4 [accessed 2023-11-25]
• Cassidy B, Hoon Yap M, Pappachan JM, Ahmad N, Haycocks S, O'Shea C, et al. Artificial intelligence for automated detection of diabetic foot ulcers: a real-world proof-of-concept clinical evaluation. Diabetes Res Clin Pract. Nov 2023;205:110951. [ CrossRef ] [ Medline ]
• Mackenzie SC, Sainsbury CA, Wake DJ. Diabetes and artificial intelligence beyond the closed loop: a review of the landscape, promise and challenges. Diabetologia. Feb 2024;67(2):223-235. [ CrossRef ] [ Medline ]
• Sridhar GR, Lakshmi G. Ethical issues of artificial intelligence in diabetes mellitus. Med Res Arch. Aug 2023;11(8):1-8. [ CrossRef ]
• Zhang L, Zhang L. Artificial intelligence for remote sensing data analysis: a review of challenges and opportunities. IEEE Geosci Remote Sens Mag. Apr 13, 2022;10(2):270-294. [ FREE Full text ] [ CrossRef ]
• Challen R, Denny J, Pitt M, Gompels L, Edwards T, Tsaneva-Atanasova K. Artificial intelligence, bias and clinical safety. BMJ Qual Saf. Mar 2019;28(3):231-237. [ CrossRef ] [ Medline ]
• Beil M, Proft I, van Heerden D, Sviri S, van Heerden PV. Ethical considerations about artificial intelligence for prognostication in intensive care. Intensive Care Med Exp. Dec 10, 2019;7(1):70. [ CrossRef ] [ Medline ]
• Bærøe K, Miyata-Sturm A, Henden E. How to achieve trustworthy artificial intelligence for health. Bull World Health Organ. Apr 01, 2020;98(4):257-262. [ CrossRef ] [ Medline ]
• Julian E, Gianfrate F, Sola-Morales O, Mol P, Bergmann JF, Salmonson T, et al. How can a joint European health technology assessment provide an 'additional benefit' over the current standard of national assessments? : insights generated from a multi-stakeholder survey in hematology/oncology. Health Econ Rev. Jun 02, 2022;12(1):30. [ CrossRef ] [ Medline ]

Abbreviations

Edited by Y Zhuang; submitted 02.08.23; peer-reviewed by A de Hond, E Vashishtha, I Cano, N Hendrix; comments to author 28.10.23; revised version received 17.12.23; accepted 28.12.23; published 13.05.24.

©Line Farah, Isabelle Borget, Nicolas Martelli, Alexandre Vallee. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 13.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Open access
• Published: 13 May 2024

Sexual and reproductive health implementation research in humanitarian contexts: a scoping review

• Alexandra Norton 1 &
• Hannah Tappis 2  

Reproductive Health volume  21 , Article number:  64 ( 2024 ) Cite this article

Metrics details

Meeting the health needs of crisis-affected populations is a growing challenge, with 339 million people globally in need of humanitarian assistance in 2023. Given one in four people living in humanitarian contexts are women and girls of reproductive age, sexual and reproductive health care is considered as essential health service and minimum standard for humanitarian response. Despite growing calls for increased investment in implementation research in humanitarian settings, guidance on appropriate methods and analytical frameworks is limited.

A scoping review was conducted to examine the extent to which implementation research frameworks have been used to evaluate sexual and reproductive health interventions in humanitarian settings. Peer-reviewed papers published from 2013 to 2022 were identified through relevant systematic reviews and a literature search of Pubmed, Embase, PsycInfo, CINAHL and Global Health databases. Papers that presented primary quantitative or qualitative data pertaining to a sexual and reproductive health intervention in a humanitarian setting were included.

Seven thousand thirty-six unique records were screened for inclusion, and 69 papers met inclusion criteria. Of these, six papers explicitly described the use of an implementation research framework, three citing use of the Consolidated Framework for Implementation Research. Three additional papers referenced other types of frameworks used in their evaluation. Factors cited across all included studies as helping the intervention in their presence or hindering in their absence were synthesized into the following Consolidated Framework for Implementation Research domains: Characteristics of Systems, Outer Setting, Inner Setting, Characteristics of Individuals, Intervention Characteristics, and Process.

This review found a wide range of methodologies and only six of 69 studies using an implementation research framework, highlighting an opportunity for standardization to better inform the evidence for and delivery of sexual and reproductive health interventions in humanitarian settings. Increased use of implementation research frameworks such as a modified Consolidated Framework for Implementation Research could work toward both expanding the evidence base and increasing standardization.

Plain English summary

Three hundred thirty-nine million people globally were in need of humanitarian assistance in 2023, and meeting the health needs of crisis-affected populations is a growing challenge. One in four people living in humanitarian contexts are women and girls of reproductive age, and provision of sexual and reproductive health care is considered to be essential within a humanitarian response. Implementation research can help to better understand how real-world contexts affect health improvement efforts. Despite growing calls for increased investment in implementation research in humanitarian settings, guidance on how best to do so is limited. This scoping review was conducted to examine the extent to which implementation research frameworks have been used to evaluate sexual and reproductive health interventions in humanitarian settings. Of 69 papers that met inclusion criteria for the review, six of them explicitly described the use of an implementation research framework. Three used the Consolidated Framework for Implementation Research, a theory-based framework that can guide implementation research. Three additional papers referenced other types of frameworks used in their evaluation. This review summarizes how factors relevant to different aspects of implementation within the included papers could have been organized using the Consolidated Framework for Implementation Research. The findings from this review highlight an opportunity for standardization to better inform the evidence for and delivery of sexual and reproductive health interventions in humanitarian settings. Increased use of implementation research frameworks such as a modified Consolidated Framework for Implementation Research could work toward both expanding the evidence base and increasing standardization.

Peer Review reports

Over the past few decades, the field of public health implementation research (IR) has grown as a means by which the real-world conditions affecting health improvement efforts can be better understood. Peters et al. put forward the following broad definition of IR for health: “IR is the scientific inquiry into questions concerning implementation – the act of carrying an intention into effect, which in health research can be policies, programmes, or individual practices (collectively called interventions)” [ 1 ].

As IR emphasizes real-world circumstances, the context within which a health intervention is delivered is a core consideration. However, much IR implemented to date has focused on higher-resource settings, with many proposed frameworks developed with particular utility for a higher-income setting [ 2 ]. In recognition of IR’s potential to increase evidence across a range of settings, there have been numerous reviews of the use of IR in lower-resource settings as well as calls for broader use [ 3 , 4 ]. There have also been more focused efforts to modify various approaches and frameworks to strengthen the relevance of IR to low- and middle-income country settings (LMICs), such as the work by Means et al. to adapt a specific IR framework for increased utility in LMICs [ 2 ].

Within LMIC settings, the centrality of context to a health intervention’s impact is of particular relevance in humanitarian settings, which present a set of distinct implementation challenges [ 5 ]. Humanitarian responses to crisis situations operate with limited resources, under potential security concerns, and often under pressure to relieve acute suffering and need [ 6 ]. Given these factors, successful implementation of a particular health intervention may require different qualities than those that optimize intervention impact under more stable circumstances [ 7 ]. Despite increasing recognition of the need for expanded evidence of health interventions in humanitarian settings, the evidence base remains limited [ 8 ]. Furthermore, despite its potential utility, there is not standardized guidance on IR in humanitarian settings, nor are there widely endorsed recommendations for the frameworks best suited to analyze implementation in these settings.

Sexual and reproductive health (SRH) is a core aspect of the health sector response in humanitarian settings [ 9 ]. Yet, progress in addressing SRH needs has lagged far behind other services because of challenges related to culture and ideology, financing constraints, lack of data and competing priorities [ 10 ]. The Minimum Initial Service Package (MISP) for SRH in Crisis Situations is the international standard for the minimum set of SRH services that should be implemented in all crisis situations [ 11 ]. However, as in other areas of health, there is need for expanded evidence for planning and implementation of SRH interventions in humanitarian settings. Recent systematic reviews of SRH in humanitarian settings have focused on the effectiveness of interventions and service delivery strategies, as well as factors affecting utilization, but have not detailed whether IR frameworks were used [ 12 , 13 , 14 , 15 ]. There have also been recent reviews examining IR frameworks used in various settings and research areas, but none have explicitly focused on humanitarian settings [ 2 , 16 ].

Given the need for an expanded evidence base for SRH interventions in humanitarian settings and the potential for IR to be used to expand the available evidence, a scoping review was undertaken. This scoping review sought to identify IR approaches that have been used in the last ten years to evaluate SRH interventions in humanitarian settings.

This review also sought to shed light on whether there is a need for a common framework to guide research design, analysis, and reporting for SRH interventions in humanitarian settings and if so, if there are any established frameworks already in use that would be fit-for-purpose or could be tailored to meet this need.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews was utilized to guide the elements of this review [ 17 ]. The review protocol was retrospectively registered with the Open Science Framework ( https://osf.io/b5qtz ).

Search strategy

A two-fold search strategy was undertaken for this review, which covered the last 10 years (2013–2022). First, recent systematic reviews pertaining to research or evaluation of SRH interventions in humanitarian settings were identified through keyword searches on PubMed and Google Scholar. Four relevant systematic reviews were identified [ 12 , 13 , 14 , 15 ] Table 1 .

Second, a literature search mirroring these reviews was conducted to identify relevant papers published since the completion of searches for the most recent review (April 2017). Additional file 1 includes the search terms that were used in the literature search [see Additional file 1 ].

The literature search was conducted for papers published from April 2017 to December 2022 in the databases that were searched in one or more of the systematic reviews: PubMed, Embase, PsycInfo, CINAHL and Global Health. Searches were completed in January 2023 Table 2 .

Two reviewers screened each identified study for alignment with inclusion criteria. Studies in the four systematic reviews identified were considered potentially eligible if published during the last 10 years. These papers then underwent full-text review to confirm satisfaction of all inclusion criteria, as inclusion criteria were similar but not fully aligned across the four reviews.

Literature search results were exported into a citation manager (Covidence), duplicates were removed, and a step-wise screening process for inclusion was applied. First, all papers underwent title and abstract screening. The remaining papers after abstract screening then underwent full-text review to confirm satisfaction of all inclusion criteria. Title and abstract screening as well as full-text review was conducted independently by both authors; disagreements after full-text review were resolved by consensus.

Data extraction and synthesis

The following content areas were summarized in Microsoft Excel for each paper that met inclusion criteria: publication details including author, year, country, setting [rural, urban, camp, settlement], population [refugees, internally displaced persons, general crisis-affected], crisis type [armed conflict, natural disaster], crisis stage [acute, chronic], study design, research methods, SRH intervention, and intervention target population [specific beneficiaries of the intervention within the broader population]; the use of an IR framework; details regarding the IR framework, how it was used, and any rationale given for the framework used; factors cited as impacting SRH interventions, either positively or negatively; and other key findings deemed relevant to this review.

As the focus of this review was on the approach taken for SRH intervention research and evaluation, the quality of the studies themselves was not assessed.

Twenty papers underwent full-text review due to their inclusion in one or more of the four systematic reviews and meeting publication date inclusion criteria. The literature search identified 7,016 unique papers. After full-text screening, 69 met all inclusion criteria and were included in the review. Figure  1 illustrates the search strategy and screening process.

Flow chart of paper identification

Papers published in each of the 10 years of the review timeframe (2013–2022) were included. 29% of the papers originated from the first five years of the time frame considered for this review, with the remaining 71% papers coming from the second half. Characteristics of included publications, including geographic location, type of humanitarian crisis, and type of SRH intervention, are presented in Table  3 .

A wide range of study designs and methods were used across the papers, with both qualitative and quantitative studies well represented. Twenty-six papers were quantitative evaluations [ 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ], 17 were qualitative [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ], and 26 used mixed methods [ 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 ]. Within the quantitative evaluations, 15 were observational, while five were quasi-experimental, five were randomized controlled trials, and one was an economic evaluation. Study designs as classified by the authors of this review are summarized in Table  4 .

Six papers (9%) explicitly cited use of an IR framework. Three of these papers utilized the Consolidated Framework for Implementation Research (CFIR) [ 51 , 65 , 70 ]. The CFIR is a commonly used determinant framework that—in its originally proposed form in 2009—is comprised of five domains, each of which has constructs to further categorize factors that impact implementation. The CFIR domains were identified as core content areas influencing the effectiveness of implementation, and the constructs within each domain are intended to provide a range of options for researchers to select from to “guide diagnostic assessments of implementation context, evaluate implementation progress, and help explain findings.” [ 87 ] To allow for consistent terminology throughout this review, the original 2009 CFIR domains and constructs are used.

Guan et al. conducted a mixed methods study to assess the feasibility and effectiveness of a neonatal hepatitis B immunization program in a conflict-affected rural region of Myanmar. Guan et al. report mapping data onto the CFIR as a secondary analysis step. They describe that “CFIR was used as a comprehensive meta-theoretical framework to examine the implementation of the Hepatitis B Virus vaccination program,” and implementation themes from multiple study data sources (interviews, observations, examination of monitoring materials) were mapped onto CFIR constructs. They report their results in two phases – Pre-implementation training and community education, and Implementation – with both anchored in themes that they had mapped onto CFIR domains and constructs. All but six constructs were included in their analysis, with a majority summarized in a table and key themes explored further in the narrative text. They specify that most concerns were identified within the Outer Setting and Process domains, while elements identified within the Inner Setting domain provided strength to the intervention and helped mitigate against barriers [ 70 ].

Sarker et al. conducted a qualitative study to assess provision of maternal, newborn and child health services to Rohingya refugees residing in camps in Cox’s Bazar, Bangladesh. They cite using CFIR as a guide for thematic analysis, applying it after a process of inductive and deductive coding to index these codes into the CFIR domains. They utilized three of the five CFIR domains (Outer Setting, Inner Setting, and Process), stating that the remaining two domains (Intervention Characteristics and Characteristics of Individuals) were not relevant to their analysis. They then proposed two additional CFIR domains, Context and Security, for use in humanitarian contexts. In contrast to Guan et al., CFIR constructs are not used nor mentioned by Sarker et al., with content under each domain instead synthesized as challenges and potential solutions. Regarding the CFIR, Sarker et al. write, “The CFIR guided us for interpretative coding and creating the challenges and possible solutions into groups for further clarification of the issues related to program delivery in a humanitarian crisis setting.” [ 51 ]

Sami et al. conducted a mixed methods case study to assess the implementation of a package of neonatal interventions at health facilities within refugee and internally displaced persons camps in South Sudan. They reference use of the CFIR earlier in the study than Sarker et al., basing their guides for semi-structured focus group discussions on the CFIR framework. They similarly reference a general use of the CFIR framework as they conducted thematic analysis. Constructs are referenced once, but they do not specify whether their application of the CFIR framework included use of domains, constructs, or both. This may be in part because they then applied an additional framework, the World Health Organization (WHO) Health System Framework, to present their findings. They describe a nested approach to their use of these frameworks: “Exploring these [CFIR] constructs within the WHO Health Systems Framework can identify specific entry points to improve the implementation of newborn interventions at critical health system building blocks.” [ 65 ]

Three papers cite use of different IR frameworks. Bolan et al. utilized the Theoretical Domains Framework in their mixed methods feasibility study and pilot cluster randomized trial evaluating pilot use of the Safe Delivery App by maternal and newborn health workers providing basic emergency obstetric and newborn care in facilities in the conflict-affected Maniema province of the Democratic Republic of the Congo (DRC). They used the Theroetical Domains Framework in designing interview questions, and further used it as the coding framework for their analysis. Similar to the CFIR, the Theoretical Domains Framework is a determinant framework that consists of domains, each of which then includes constructs. Bolan et al. utilized the Theoretical Domains Framework at the construct level in interview question development and at the domain level in their analysis, mapping interview responses to eight of the 14 domains [ 83 ]. Berg et al. report using an “exploratory design guided by the principles of an evaluation framework” developed by the Medical Research Council to analyze the implementation process, mechanisms of impact, and outcomes of a three-pillar training intervention to improve maternal and neonatal healthcare in the conflict-affected South Kivu province of the DRC [ 67 , 88 ]. Select components of this evaluation framework were used to guide deductive analysis of focus group discussions and in-depth interviews [ 67 ]. In their study of health workers’ knowledge and attitudes toward newborn health interventions in South Sudan, before and after training and supply provision, Sami et al. report use of the Conceptual Framework of the Role of Attitudes in Evidence-Based Practice Implementation in their analysis process. The framework was used to group codes following initial inductive coding analysis of in-depth interviews [ 72 ].

Three other papers cite use of specific frameworks in their intervention evaluation [ 19 , 44 , 76 ]. As a characteristic of IR is the use of an explicit framework to guide the research, the use of the frameworks in these three papers meets the intention of IR and serves the purpose that an IR framework would have in strengthening the analytical rigor. Castle et al. cite use of their program’s theory of change as a framework for a mixed methods evaluation of the provision of family planning services and more specifically uptake of long-acting reversible contraception use in the DRC. They describe use of the theory of change to “enhance effectiveness of [long-acting reversible contraception] access and uptake.” [ 76 ] Thommesen et al. cite use of the AAAQ (Availability, Accessibility, Acceptability and Quality) framework in their qualitative study assessing midwifery services provided to pregnant women in Afghanistan. This framework is focused on the “underlying elements needed for attainment of optimum standard of health care,” but the authors used it in this paper to evaluate facilitators and barriers to women accessing midwifery services [ 44 ]. Jarrett et al. cite use of the Centers for Disease Control and Prevention’s (CDC) Guidelines for Evaluating Public Health Surveillance Systems to explore the characteristics of a population mobility, mortality and birth surveillance system in South Kivu, DRC. Use of these CDC guidelines is cited as one of four study objectives, and commentary is included in the Results section pertaining to each criteria within these guidelines, although more detail regarding use of these guidelines or the authors’ experience with their use in the study is not provided [ 19 ].

Overall, 22 of the 69 papers either explicitly or implicitly identified IR as relevant to their work. Nineteen papers include a focus on feasibility (seven of which did not otherwise identify the importance of exploring questions concerning implementation), touching on a common outcome of interest in implementation research [ 89 ].

While a majority of papers did not explicitly or implicitly use an IR framework to evaluate their SRH intervention of focus, most identified factors that facilitated implementation when they were present or served as a barrier when absent. Sixty cite factors that served as facilitators and 49 cite factors that served as barriers, with just three not citing either. Fifty-nine distinct factors were identified across the papers.

Three of the six studies that explicitly used an IR framework used the CFIR, and the CFIR is the only IR framework that was used by multiple studies. As previously mentioned, Means et al. put forth an adaptation of the CFIR to increase its relevance in LMIC settings, proposing a sixth domain (Characteristics of Systems) and 11 additional constructs [ 2 ]. Using the expanded domains and constructs as proposed by Means et al., the 59 factors cited by papers in this review were thematically grouped into the six domains: Characteristics of Systems, Outer Setting, Inner Setting, Characteristics of Individuals, Intervention Characteristics, and Process. Within each domain, alignment with CFIR constructs was assessed for, and alignment was found with 29 constructs: eight of Means et al.’s 11 constructs, and 21 of the 39 standard CFIR constructs. Three factors did not align with any construct (all fitting within the Outer Setting domain), and 14 aligned with a construct label but not the associated definition. Table 5 synthesizes the mapping of factors affecting SRH intervention implementation to CFIR domains and constructs, with the construct appearing in italics if it is considered to align with that factor by label but not by definition.

Table 6 lists the CFIR constructs that were not found to have alignment with any factor cited by the papers in this review.

This scoping review sought to assess how IR frameworks have been used to bolster the evidence base for SRH interventions in humanitarian settings, and it revealed that IR frameworks, or an explicit IR approach, are rarely used. All four of the systematic reviews identified with a focus on SRH in humanitarian settings articulate the need for more research examining the effectiveness of SRH interventions in humanitarian settings, with two specifically citing a need for implementation research/science [ 12 , 13 ]. The distribution of papers across the timeframe included in this review does suggest that more research on SRH interventions for crisis-affected populations is taking place, as a majority of relevant papers were published in the second half of the review period. The papers included a wide range of methodologies, which reflect the differing research questions and contexts being evaluated. However, it also invites the question of whether there should be more standardization of outcomes measured or frameworks used to guide analysis and to facilitate increased comparison, synthesis and application across settings.

Three of the six papers that used an IR framework utilized the CFIR. Guan et al. used the CFIR at both a domain and construct level, Sarker et al. used the CFIR at the domain level, and Sami et al. did not specify which CFIR elements were used in informing the focus group discussion guide [ 51 , 65 , 70 ]. It is challenging to draw strong conclusions about the applicability of CFIR in humanitarian settings based on the minimal use of CFIR and IR frameworks within the papers reviewed, although Guan et al. provides a helpful model for how analysis can be structured around CFIR domains and constructs. It is worth considering that the minimal use of IR frameworks, and more specifically CFIR constructs, could be in part because that level of prescriptive categorization does not allow for enough fluidity in humanitarian settings. It also raises questions about the appropriate degree of standardization to pursue for research done in these settings.

The mapping of factors affecting SRH intervention implementation provides an example of how a modified CFIR framework could be used for IR in humanitarian contexts. This mapping exercise found factors that mapped to all five of the original CFIR domains as well as the sixth domain proposed by Means et al. All factors fit well within the definition for the selected domain, indicating an appropriate degree of fit between these existing domains and the factors identified as impacting SRH interventions in humanitarian settings. On a construct level, however, the findings were more variable, with one-quarter of factors not fully aligning with any construct. Furthermore, over 40% of the CFIR constructs (including the additional constructs from Means et al.) were not found to align with any factors cited by the papers in this review, also demonstrating some disconnect between the parameters posed by the CFIR constructs and the factors cited as relevant in a humanitarian context.

It is worth noting that while the CFIR as proposed in 2009 was used in this assessment, as well as in the included papers which used the CFIR, an update was published in 2022. Following a review of CFIR use since its publication, the authors provide updates to construct names and definitions to “make the framework more applicable across a range of innovations and settings.” New constructs and subconstructs were also added, for a total of 48 constructs and 19 subconstructs across the five domains [ 90 ]. A CFIR Outcomes Addendum was also published in 2022, based on recommendations for the CFIR to add outcomes and intended to be used as a complement to the CFIR determinants framework [ 91 ]. These expansions to the CFIR framework may improve applicability of the CFIR in humanitarian settings. Several constructs added to the Outer Setting domain could be of particular utility – critical incidents, local attitudes, and local conditions, each of which could help account for unique challenges faced in contexts of crisis. Sub-constructs added within the Inner Setting domain that seek to clarify structural characteristics and available resources would also be of high utility based on mapping of the factors identified in this review to the original CFIR constructs. As outcomes were not formally included in the CFIR until the 2022 addendum, a separate assessment of implementation outcomes was not undertaken in this review. However, analysis of the factors cited by papers in this review as affecting implementation was derived from the full text of the papers and thus captures content relevant to implementation determinants that is contained within the outcomes.

Given the demonstrated need for additional flexibility within an IR framework for humanitarian contexts, while not a focus of this review, it is worth considering whether a different framework could provide a better fit than the CFIR. Other frameworks have differing points of emphasis that would create different opportunities for flexibility but that do not seem to resolve the challenges experienced in applying the CFIR to a humanitarian context. As one example, the EPIS (Exploration, Preparation, Implementation, Sustainment) Framework considers the impact of inner and outer context on each of four implementation phases; while the constructs within this framework are broader than the CFIR, an emphasis on the intervention characteristics is missing, a domain where stronger alignment within the CFIR is also needed [ 92 ]. Alternatively, the PRISM (Practical, Robust Implementation and Sustainability Model) framework is a determinant and evaluation framework that adds consideration of context factors to the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) outcomes framework. It has a stronger emphasis on intervention aspects, with sub-domains to account for both organization and patient perspectives within the intervention. While PRISM does include aspects of context, external environment considerations are less robust and intentionally less comprehensive in scope, which would not provide the degree of alignment possible between the Characteristics of Systems and Outer Setting CFIR domains for the considerations unique to humanitarian environments [ 93 ].

Reflecting on their experience with the CFIR, Sarker et al. indicate that it can be a “great asset” in both evaluating current work and developing future interventions. They also encourage future research of humanitarian health interventions to utilize the CFIR [ 51 ]. The other papers that used the CFIR do not specifically reflect on their experience utilizing it, referring more generally to having felt that it was a useful tool [ 65 , 70 ]. On their use of an evaluation framework, Berg et al. reflected that it lent useful structure and helped to identify aspects affecting implementation that otherwise would have gone un-noticed [ 67 ]. The remaining studies that utilized an IR framework did not specifically comment on their experience with its use [ 72 , 83 ]. While a formal IR framework was not engaged by other studies, a number cite a desire for IR to contribute further detail to their findings [ 21 , 37 ].

In their recommendations for strengthening the evidence base for humanitarian health interventions, Ager et al. speak to the need for “methodologic innovation” to develop methodologies with particular applicability in humanitarian settings [ 7 ]. As IR is not yet routinized for SRH interventions, this could be opportune timing for the use of a standardized IR framework to gauge its utility. Using an IR framework to assess factors influencing implementation of the MISP in initial stages of a humanitarian response, and interventions to support more comprehensive SRH service delivery in protracted crises, could lend further rigor and standardization to SRH evaluations, as well as inform strategies to improve MISP implementation over time. Based on categorizing factors identified by these papers as relevant for intervention evaluation, there does seem to be utility to a modified CFIR approach. Given the paucity of formal IR framework use within SRH literature, it would be worth conducting similar scoping exercises to assess for explicit use of IR frameworks within the evidence base for other health service delivery areas in humanitarian settings. In the interim, the recommended approach from this review for future IR on humanitarian health interventions would be a modified CFIR approach with domain-level standardization and flexibility for constructs that may standardize over time with more use. This would enable use of a common analytical framework and vocabulary at the domain level for stakeholders to describe interventions and the factors influencing the effectiveness of implementation, with constructs available to use and customize as most appropriate for specific contexts and interventions.

This review had a number of limitations. As this was a scoping review and a two-part search strategy was used, the papers summarized here may not be comprehensive of those written pertaining to SRH interventions over the past 10 years. Papers from 2013 to 2017 that would have met this scoping review’s inclusion criteria may have been omitted due to being excluded from the systematic reviews. The review was limited to papers available in English. Furthermore, this review did not assess the quality of the papers included or seek to assess the methodology used beyond examination of the use of an IR framework. It does, however, serve as a first step in assessing the extent to which calls for implementation research have been addressed, and identify entry points for strengthening the science and practice of SRH research in humanitarian settings.

With one in 23 people worldwide in need of humanitarian assistance, and financing required for response plans at an all-time high, the need for evidence to guide resource allocation and programming for SRH in humanitarian settings is as important as ever [ 94 ]. Recent research agenda setting initiatives and strategies to advance health in humanitarian settings call for increased investment in implementation research—with priorities ranging from research on effective strategies for expanding access to a full range of contraceptive options to integrating mental health and psychosocial support into SRH programming to capturing accurate and actionable data on maternal and perinatal mortality in a wide range of acute and protracted emergency contexts [ 95 , 96 ]. To truly advance guidance in these areas, implementation research will need to be conducted across diverse humanitarian settings, with clear and consistent documentation of both intervention characteristics and outcomes, as well as contextual and programmatic factors affecting implementation.

Conclusions

Implementation research has potential to increase impact of health interventions particularly in crisis-affected settings where flexibility, adaptability and context-responsive approaches are highlighted as cornerstones of effective programming. There remains significant opportunity for standardization of research in the humanitarian space, with one such opportunity occurring through increased utilization of IR frameworks such as a modified CFIR approach. Investing in more robust sexual and reproductive health research in humanitarian contexts can enrich insights available to guide programming and increase transferability of learning across settings.

Availability of data and materials

The datasets analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Availability, Accessibility, Acceptability and Quality

Centers for Disease Control and Prevention

Consolidated Framework for Implementation Research

Democratic Republic of the Congo

Exploration, Preparation, Implementation, Sustainment

• Implementation research

Low and middle income country

Minimum Initial Service Package

Practical, Robust Implementation and Sustainability Model

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Reach, Effectiveness, Adoption, Implementation, Maintenance

• Sexual and reproductive health

World Health Organization

Peters DH, et al. Implementation research: what it is and how to do it. RESEARCH METHODS. 2013;347:7.

Means AR, et al. Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: a systematic review. Implement Sci. 2020;15(1):17.

Article   PubMed   PubMed Central   Google Scholar  

Alonge O, et al. How is implementation research applied to advance health in low-income and middle-income countries? BMJ Glob Health. 2019;4(2):e001257.

Ridde V, Pérez D, Robert E. Using implementation science theories and frameworks in global health. BMJ Glob Health. 2020;5(4):e002269.

Gaffey MF, et al. Delivering health and nutrition interventions for women and children in different conflict contexts: a framework for decision making on what, when, and how. Lancet (London, England). 2021;397(10273):543–54.

Article   PubMed   Google Scholar  

Singh NS, et al. Delivering health interventions to women, children, and adolescents in conflict settings: what have we learned from ten country case studies? The Lancet. 2021;397(10273):533–42.

Article   Google Scholar  

Ager A, et al. Strengthening the evidence base for health programming in humanitarian crises. Science. 2014;345(6202):1290–2.

Article   CAS   PubMed   Google Scholar  

Blanchet K, et al. Evidence on public health interventions in humanitarian crises. The Lancet. 2017;390(10109):2287–96.

Sphere A. The Sphere Handbook | Standards for quality humanitarian response. 2018.

Google Scholar  

Barot S. In a State of Crisis: Meeting the Sexual and Reproductive Health Needs of Women in Humanitarian Situations. Guttmacher Policy Rev. 2017;20:7.

Crisis, I.-A.W.G.f.R.H.i., Minimum Initial Service Package. 2020: https://www.unfpa.org/resources/minimum-initial-service-package-misp-srh-crisis-situations .

Casey SE. Evaluations of reproductive health programs in humanitarian settings: a systematic review. Confl Heal. 2015;9(1):S1.

Singh NS, et al. A long way to go: a systematic review to assess the utilisation of sexual and reproductive health services during humanitarian crises. BMJ Glob Health. 2018;3(2):e000682.

Singh NS, et al. Evaluating the effectiveness of sexual and reproductive health services during humanitarian crises: A systematic review. PLoS ONE. 2018;13(7):e0199300.

Warren E, et al. Systematic review of the evidence on the effectiveness of sexual and reproductive health interventions in humanitarian crises. BMJ Open. 2015;5(12):e008226.

Dadich A, Piper A, Coates D. Implementation science in maternity care: a scoping review. Implement Sci. 2021;16(1):16.

Tricco AC, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73.

Devine A, et al. Strategies for the prevention of perinatal hepatitis B transmission in a marginalized population on the Thailand-Myanmar border: a cost-effectiveness analysis. BMC Infect Dis. 2017;17(1):552.

Jarrett P, et al. Evaluation of a population mobility, mortality, and birth surveillance system in South Kivu. Democratic Republic of the Congo Disasters. 2020;44(2):390–407.

PubMed   Google Scholar  

Logie CH, et al. A Psycho-Educational HIV/STI Prevention Intervention for Internally Displaced Women in Leogane, Haiti: Results from a Non-Randomized Cohort Pilot Study. PLoS ONE. 2014;9(2):e89836.

O’Laughlin KN, et al. A cohort study to assess a communication intervention to improve linkage to HIV care in Nakivale Refugee Settlement. Uganda Glob Public Health. 2021;16(12):1848–55.

Adam I. The influence of maternal health education on the place of delivery in conflict settings of Darfur. Sudan Conflict and Health. 2015;9:31.

Adam IF, et al. Relationship between implementing interpersonal communication and mass education campaigns in emergency settings and use of reproductive healthcare services: evidence from Darfur, Sudan. BMJ Open. 2015;5(9):e008285.

Edmond K, et al. Mobile outreach health services for mothers and children in conflict-affected and remote areas: a population-based study from Afghanistan. Arch Dis Child. 2020;105(1):18–25.

Nasir S, et al. Dissemination and implementation of the e-MCHHandbook, UNRWA’s newly released maternal and child health mobile application: a cross-sectional study. BMJ Open. 2020;10(3):e034885.

O’Laughlin KN, et al. Feasibility and acceptability of home-based HIV testing among refugees: a pilot study in Nakivale refugee settlement in southwestern Uganda. BMC Infect Dis. 2018;18(1):332.

Adam I. Evidence from cluster surveys on the association between home-based counseling and use of family planning in conflict-affected Darfur. Int J Gynecol Obstet. 2016;133(2):221–5.

Casey S, et al. Availability of long-acting and permanent family-planning methods leads to increase in use in conflict-affected northern Uganda: Evidence from cross-sectional baseline and endline cluster surveys. Glob Public Health. 2013;8(3):284–97.

Corna F, et al. Supporting maternal mental health of Rohingya refugee women during the perinatal period to promote child health and wellbeing: a field study in Cox’s Bazar. Intervention, the Journal of Mental Health & Psychosocial Support in Conflict Affected Areas. 2019;17(2):160–8.

Glass N, et al. Effectiveness of the Communities Care programme on change in social norms associated with gender-based violence (GBV) with residents in intervention compared with control districts in Mogadishu, Somalia. BMJ Open. 2019;9(3):e023819.

James LE, et al. Development and Testing of a Community-Based Intervention to Address Intimate Partner Violence among Rohingya and Syrian Refugees: A Social Norms-Based Mental Health-Integrated Approach. Int J Environ Res Public Health. 2021;18(21):11674.

Le Roux E, et al. Engaging with faith groups to prevent VAWG in conflict-affected communities: results from two community surveys in the DRC. BMC Int Health Hum Rights. 2020;20(1):27.

Morris CN, et al. When political solutions for acute conflict in Yemen seem distant, demand for reproductive health services is immediate: a programme model for resilient family planning and post-abortion care services. Sex Reprod Health Matters. 2019;27(2):1610279.

Anibueze AU, et al. Impact of counseling visual multimedia on use of family planning methods among displaced Nigerian families. Health Promot Int. 2022;37(3):daac060.

Doocy S, et al. Cash-based assistance and the nutrition status of pregnant and lactating women in the Somalia food crisis: A comparison of two transfer modalities. PLoS ONE. 2020;15(4):e0230989.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Draiko CV, et al. The effect of umbilical cord cleansing with chlorhexidine gel on neonatal mortality among the community births in South Sudan: a quasi-experimental study. Pan Afr Med J. 2021;38:78.

Edmond KM, et al. Can community health worker home visiting improve care-seeking and maternal and newborn care practices in fragile states such as Afghanistan? A population-based intervention study. BMC Med. 2018;16(1):106.

Edmond KM, et al. Conditional cash transfers to improve use of health facilities by mothers and newborns in conflict affected countries, a prospective population based intervention study from Afghanistan. BMC Pregnancy Childbirth. 2019;19(1):193.

Bakesiima R, et al. Effect of peer counselling on acceptance of modern contraceptives among female refugee adolescents in northern Uganda: A randomised controlled trial. PLoS ONE. 2021;16(9):e0256479.

Greene MC, et al. Evaluation of an integrated intervention to reduce psychological distress and intimate partner violence in refugees: Results from the Nguvu cluster randomized feasibility trial. PLoS ONE. 2021;16(6):e0252982.

Gupta J, et al. Gender norms and economic empowerment intervention to reduce intimate partner violence against women in rural Côte d’Ivoire: a randomized controlled pilot study. BMC Int Health Hum Rights. 2013;13(1):46.

Hossain M, et al. Working with men to prevent intimate partner violence in a conflict-affected setting: a pilot cluster randomized controlled trial in rural Côte d’Ivoire. BMC Public Health. 2014;14(1):339.

Vaillant J, et al. Engaging men to transform inequitable gender attitudes and prevent intimate partner violence: a cluster randomised controlled trial in North and South Kivu, Democratic Republic of Congo. BMJ Glob Health. 2020;5(5):e002223.

Thommesen T, et al. “The midwife helped me … otherwise I could have died”: women’s experience of professional midwifery services in rural Afghanistan - a qualitative study in the provinces Kunar and Laghman. BMC Pregnancy Childbirth. 2020;20(1):140.

Awasom-Fru A, et al. Doctors’ experiences providing sexual and reproductive health care at Catholic Hospitals in the conflict-affected North-West region of Cameroon: a qualitative study. Reprod Health. 2022;19(1):126.

Kabakian-Khasholian T, Makhoul J, Ghusayni A. “A person who does not have money does not enter”: a qualitative study on refugee women’s experiences of respectful maternity care. BMC Pregnancy and Childbirth. 2022;22(1):748.

Lilleston P, et al. Evaluation of a mobile approach to gender-based violence service delivery among Syrian refugees in Lebanon. Health Policy Plan. 2018;33(7):767–76.

Mugo NS, et al. Barriers Faced by the Health Workers to Deliver Maternal Care Services and Their Perceptions of the Factors Preventing Their Clients from Receiving the Services: A Qualitative Study in South Sudan. Matern Child Health J. 2018;22(11):1598–606.

Persson M, et al. A qualitative study on health care providers’ experiences of providing comprehensive abortion care in Cox’s Bazar, Bangladesh. Conflict and Health. 2021;15(1):6.

Phanwichatkul T, et al. The perceptions and practices of Thai health professionals providing maternity care for migrant Burmese women: An ethnographic study. Women Birth. 2022;35(4):e356–68.

Sarker M, et al. Effective maternal, newborn and child health programming among Rohingya refugees in Cox’s Bazar, Bangladesh: Implementation challenges and potential solutions. PLoS ONE. 2020;15(3):e0230732.

Tousaw E, et al. “Without this program, women can lose their lives”: migrant women’s experiences with the Safe Abortion Referral Programme in Chiang Mai. Thailand Reprod Health Matters. 2017;25(51):58–68.

Tousaw E, et al. “It is just like having a period with back pain”: exploring women’s experiences with community-based distribution of misoprostol for early abortion on the Thailand-Burma border. Contraception. 2018;97(2):122–9.

West L, et al. Factors in use of family planning services by Syrian women in a refugee camp in Jordan. Journal of Family Planning and Reproductive Health Care. 2017;43(2):96–102.

O’Connell KA, et al. Meeting the Sexual and Reproductive Health Needs of Internally Displaced Persons in Ethiopia’s Somali Region: A Qualitative Process Evaluation. Glob Health Sci Pract. 2022;10(5):e2100818.

Orya E, et al. Strengthening close to community provision of maternal health services in fragile settings: an exploration of the changing roles of TBAs in Sierra Leone and Somaliland. BMC Health Serv Res. 2017;17(1):460.

Perera SM, et al. Barriers to seeking post-abortion care in Paktika Province, Afghanistan: a qualitative study of clients and community members. BMC Womens Health. 2021;21(1):390.

Tanabe M, et al. Piloting community-based medical care for survivors of sexual assault in conflict-affected Karen State of eastern Burma. Confl Heal. 2013;7(1):12.

Tran NT, et al. Clinical outreach refresher trainings in crisis settings (S-CORT): clinical management of sexual violence survivors and manual vacuum aspiration in Burkina Faso, Nepal, and South Sudan. Reprod Health Matters. 2017;25(51):103–13.

Yankah E, et al. Feasibility and acceptability of mobile phone platforms to deliver interventions to address gender-based violence among Syrian adolescent girls and young women in Izmir. Turkey Vulnerable Children and Youth Studies. 2020;15(2):133–43.

Muuo S, et al. Barriers and facilitators to care-seeking among survivors of gender-based violence in the Dadaab refugee complex. Sex Reprod Health Matters. 2020;28(1):1722404.

Amsalu R, et al. Essential newborn care practice at four primary health facilities in conflict affected areas of Bossaso, Somalia: a cross-sectional study. Conflict and Health. 2019;13(13):27.

Myers A, et al. Facilitators and barriers in implementing the Minimum Initial Services Package (MISP) for reproductive health in Nepal post-earthquake. Conflict and Health. 2018;12:35.

Santo L.C.d, et al. Feasibility and acceptability of a video library tool to support community health worker counseling in rural Afghan districts: a cross-sectional assessment. Conflict and Health. 2020;14:56.

Sami S, et al. Understanding health systems to improve community and facility level newborn care among displaced populations in South Sudan: a mixed methods case study. BMC Pregnancy Childbirth. 2018;18(1):325.

Amsalu R, et al. Effectiveness of clinical training on improving essential newborn care practices in Bossaso, Somalia: a pre and postintervention study. BMC Pediatr. 2020;20(1):215.

Berg M, Mwambali SN, Bogren M. Implementation of a three-pillar training intervention to improve maternal and neonatal healthcare in the Democratic Republic Of Congo: a process evaluation study in an urban health zone. Glob Health Action. 2022;15(1):2019391.

Castillo M, et al. Turning Disaster into an Opportunity for Quality Improvement in Essential Intrapartum and Newborn Care Services in the Philippines: Pre- to Posttraining Assessments. Biomed Res Int. 2016;2016:1–9.

Foster AM, Arnott G, Hobstetter M. Community-based distribution of misoprostol for early abortion: evaluation of a program along the Thailand-Burma border. Contraception. 2017;96(4):242–7.

Guan TH, et al. Implementation of a neonatal hepatitis B immunization program in rural Karenni State, Myanmar: A mixed-methods study. PLoS ONE. 2021;16(12):e0261470.

Logie, C.H., et al., Mixed-methods findings from the Ngutulu Kagwero (agents of change) participatory comic pilot study on post-rape clinical care and sexual violence prevention with refugee youth in a humanitarian setting in Uganda. Global Public Health, 2022((Logie C.H., [email protected]) Factor-Inwentash Faculty of Social Work, University of Toronto, Toronto, Canada(Logie C.H., [email protected]) Women’s College Research Institute, Women’s College Hospital, Toronto, Canada(Logie C.H., carmen.l).

Sami S, et al. “You have to take action”: changing knowledge and attitudes towards newborn care practices during crisis in South Sudan. Reprod Health Matters. 2017;25(51):124–39.

Smith JR, et al. Clinical care for sexual assault survivors multimedia training: a mixed-methods study of effect on healthcare providers’ attitudes, knowledge, confidence, and practice in humanitarian settings. Confl Heal. 2013;7(1):14.

Stevens A, et al. Folate supplementation to prevent birth abnormalities: evaluating a community-based participatory action plan for refugees and migrant workers on the Thailand-Myanmar border. Public Health. 2018;161:83–9.

Nguyen Toan T, et al. Strengthening healthcare providers’ capacity for safe abortion and postabortion care services in humanitarian settings: lessons learned from the clinical outreach refresher training model (S-CORT) in Uganda, Nigeria, and the Democratic Republic of Congo. Conflict and Health. 2021;15(1):20.

Castle S, et al. Successful programmatic approaches to facilitating IUD uptake: CARE’s experience in DRC. BMC Womens Health. 2019;19(1):104.

Deitch J, et al. “They Love Their Patients”: Client Perceptions of Quality of Postabortion Care in North and South Kivu, the Democratic Republic of the Congo. Global health, science and practice. 2019;7(Suppl 2):S285–98.

Ferreyra C, et al. Evaluation of a community-based HIV test and start program in a conflict affected rural area of Yambio County, South Sudan. PLoS ONE. 2021;16(7):e0254331.

Ho LS, Wheeler E. Using Program Data to Improve Access to Family Planning and Enhance the Method Mix in Conflict-Affected Areas of the Democratic Republic of the Congo. Glob Health Sci Pract. 2018;6(1):161–77.

Klabbers RE, et al. Health Worker Perspectives on Barriers and Facilitators of Assisted Partner Notification for HIV for Refugees and Ugandan Nationals: A Mixed Methods Study in West Nile Uganda. AIDS Behav. 2021;25(10):3206–22.

Turner C, et al. Neonatal Intensive Care in a Karen Refugee Camp: A 4 Year Descriptive Study. PLoS ONE. 2013;8(8):e72721.

Vries Id, et al. Key lessons from a mixed-method evaluation of a postnatal home visit programme in the humanitarian setting of Gaza. Eastern Mediterr Health J. 2021;27(6):546–52.

Bolan NE, et al. mLearning in the Democratic Republic of the Congo: A Mixed-Methods Feasibility and Pilot Cluster Randomized Trial Using the Safe Delivery App. Global health, science and practice. 2018;6(4):693–710.

Khan MN, et al. Evaluating feasibility and acceptability of a local psycho-educational intervention for pregnant women with common mental problems affected by armed conflict in Swat, Pakistan: A parallel randomized controlled feasibility trial. Int J Soc Psychiatry. 2017;63(8):724–35.

Hynes M, et al. Using a quality improvement approach to improve maternal and neonatal care in North Kivu, Democratic Republic of Congo. Reprod Health Matters. 2017;25(51):140–50.

Gibbs A, et al. The impacts of combined social and economic empowerment training on intimate partner violence, depression, gender norms and livelihoods among women: an individually randomised controlled trial and qualitative study in Afghanistan. BMJ Glob Health. 2020;5(3):e001946.

Damschroder L, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science: IS; 2009.

Moore GF, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

Proctor E, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Damschroder LJ, et al. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

Damschroder LJ, et al. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implement Sci. 2022;17(1):7.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):4–23.

Feldstein AC, Glasgow RE. A Practical, Robust Implementation and Sustainability Model (PRISM) for Integrating Research Findings into Practice. The Joint Commission Journal on Quality and Patient Safety. 2008;34(4):228–43.

OCHA. Global Humanitarian Overview 2023. 2022 [cited 2023 8/3/2023]; Available from: https://humanitarianaction.info/node/13073/article/glance-0 . Accessed 8 Mar 2023.

Kobeissi L, et al. Setting research priorities for sexual, reproductive, maternal, newborn, child and adolescent health in humanitarian settings. Confl Heal. 2021;15(1):16.

Save the, C., et al. Roadmap to Accelerate Progress for Every Newborn in Humanitarian Settings 2020 – 2024. 2020. p. 52.

Inter-Agency Working Group on Reproductive Health in, C. Inter-Agency Field Manual on Reproductive Health in Humanitarian Settings. 2018.

Download references

Acknowledgements

Not applicable.

The authors received no funding for this study.

Author information

Authors and affiliations.

Duke University School of Medicine, 40 Duke Medicine Circle, Durham, NC, 27710, USA

Alexandra Norton

Johns Hopkins Bloomberg School of Public Health, 615 N. Wolfe St, Baltimore, MD, 21205, USA

Hannah Tappis

You can also search for this author in PubMed   Google Scholar

Contributions

AN and HT designed the scoping review. AN conducted the literature search. AN and HT screened records for inclusion. AN extracted data from included studies. Both authors contributed to synthesis of results. AN drafted the manuscript and both authors contributed to editorial changes.

Corresponding author

Correspondence to Alexandra Norton .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

. Literature search terms: Exact search terms used in literature search, with additional detail on the methodology to determine search terms and definitions used for each component of the search

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Norton, A., Tappis, H. Sexual and reproductive health implementation research in humanitarian contexts: a scoping review. Reprod Health 21 , 64 (2024). https://doi.org/10.1186/s12978-024-01793-2

Download citation

Received : 06 November 2023

Accepted : 12 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1186/s12978-024-01793-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Humanitarian settings

Reproductive Health

ISSN: 1742-4755

• General enquiries: [email protected]