evidence in research paper meaning

Evidence-Based Research Series-Paper 1: What Evidence-Based Research is and why is it important?

Affiliations.

• 1 Johns Hopkins Evidence-based Practice Center, Division of General Internal Medicine, Department of Medicine, Johns Hopkins University, Baltimore, MD, USA.
• 2 Digital Content Services, Operations, Elsevier Ltd., 125 London Wall, London, EC2Y 5AS, UK.
• 3 School of Nursing, McMaster University, Health Sciences Centre, Room 2J20, 1280 Main Street West, Hamilton, Ontario, Canada, L8S 4K1; Section for Evidence-Based Practice, Western Norway University of Applied Sciences, Inndalsveien 28, Bergen, P.O.Box 7030 N-5020 Bergen, Norway.
• 4 Department of Sport Science and Clinical Biomechanics, University of Southern Denmark, Campusvej 55, 5230, Odense M, Denmark; Department of Physiotherapy and Occupational Therapy, University Hospital of Copenhagen, Herlev & Gentofte, Kildegaardsvej 28, 2900, Hellerup, Denmark.
• 5 Musculoskeletal Statistics Unit, the Parker Institute, Bispebjerg and Frederiksberg Hospital, Copenhagen, Nordre Fasanvej 57, 2000, Copenhagen F, Denmark; Department of Clinical Research, Research Unit of Rheumatology, University of Southern Denmark, Odense University Hospital, Denmark.
• 6 Section for Evidence-Based Practice, Western Norway University of Applied Sciences, Inndalsveien 28, Bergen, P.O.Box 7030 N-5020 Bergen, Norway. Electronic address: [email protected].
• PMID: 32979491
• DOI: 10.1016/j.jclinepi.2020.07.020

Objectives: There is considerable actual and potential waste in research. Evidence-based research ensures worthwhile and valuable research. The aim of this series, which this article introduces, is to describe the evidence-based research approach.

Study design and setting: In this first article of a three-article series, we introduce the evidence-based research approach. Evidence-based research is the use of prior research in a systematic and transparent way to inform a new study so that it is answering questions that matter in a valid, efficient, and accessible manner.

Results: We describe evidence-based research and provide an overview of the approach of systematically and transparently using previous research before starting a new study to justify and design the new study (article #2 in series) and-on study completion-place its results in the context with what is already known (article #3 in series).

Conclusion: This series introduces evidence-based research as an approach to minimize unnecessary and irrelevant clinical health research that is unscientific, wasteful, and unethical.

Keywords: Clinical health research; Clinical trials; Evidence synthesis; Evidence-based research; Medical ethics; Research ethics; Systematic review.

Copyright © 2020 Elsevier Inc. All rights reserved.

Publication types

• Research Support, Non-U.S. Gov't
• Biomedical Research* / methods
• Biomedical Research* / organization & administration
• Clinical Trials as Topic / ethics
• Clinical Trials as Topic / methods
• Clinical Trials as Topic / organization & administration
• Ethics, Research
• Evidence-Based Medicine / methods*
• Needs Assessment
• Reproducibility of Results
• Research Design* / standards
• Research Design* / trends
• Systematic Reviews as Topic
• Treatment Outcome

What this handout is about

This handout will provide a broad overview of gathering and using evidence. It will help you decide what counts as evidence, put evidence to work in your writing, and determine whether you have enough evidence. It will also offer links to additional resources.

Introduction

Many papers that you write in college will require you to make an argument ; this means that you must take a position on the subject you are discussing and support that position with evidence. It’s important that you use the right kind of evidence, that you use it effectively, and that you have an appropriate amount of it. If, for example, your philosophy professor didn’t like it that you used a survey of public opinion as your primary evidence in your ethics paper, you need to find out more about what philosophers count as good evidence. If your instructor has told you that you need more analysis, suggested that you’re “just listing” points or giving a “laundry list,” or asked you how certain points are related to your argument, it may mean that you can do more to fully incorporate your evidence into your argument. Comments like “for example?,” “proof?,” “go deeper,” or “expand” in the margins of your graded paper suggest that you may need more evidence. Let’s take a look at each of these issues—understanding what counts as evidence, using evidence in your argument, and deciding whether you need more evidence.

What counts as evidence?

Before you begin gathering information for possible use as evidence in your argument, you need to be sure that you understand the purpose of your assignment. If you are working on a project for a class, look carefully at the assignment prompt. It may give you clues about what sorts of evidence you will need. Does the instructor mention any particular books you should use in writing your paper or the names of any authors who have written about your topic? How long should your paper be (longer works may require more, or more varied, evidence)? What themes or topics come up in the text of the prompt? Our handout on understanding writing assignments can help you interpret your assignment. It’s also a good idea to think over what has been said about the assignment in class and to talk with your instructor if you need clarification or guidance.

What matters to instructors?

Instructors in different academic fields expect different kinds of arguments and evidence—your chemistry paper might include graphs, charts, statistics, and other quantitative data as evidence, whereas your English paper might include passages from a novel, examples of recurring symbols, or discussions of characterization in the novel. Consider what kinds of sources and evidence you have seen in course readings and lectures. You may wish to see whether the Writing Center has a handout regarding the specific academic field you’re working in—for example, literature , sociology , or history .

What are primary and secondary sources?

A note on terminology: many researchers distinguish between primary and secondary sources of evidence (in this case, “primary” means “first” or “original,” not “most important”). Primary sources include original documents, photographs, interviews, and so forth. Secondary sources present information that has already been processed or interpreted by someone else. For example, if you are writing a paper about the movie “The Matrix,” the movie itself, an interview with the director, and production photos could serve as primary sources of evidence. A movie review from a magazine or a collection of essays about the film would be secondary sources. Depending on the context, the same item could be either a primary or a secondary source: if I am writing about people’s relationships with animals, a collection of stories about animals might be a secondary source; if I am writing about how editors gather diverse stories into collections, the same book might now function as a primary source.

Where can I find evidence?

Here are some examples of sources of information and tips about how to use them in gathering evidence. Ask your instructor if you aren’t sure whether a certain source would be appropriate for your paper.

Print and electronic sources

Books, journals, websites, newspapers, magazines, and documentary films are some of the most common sources of evidence for academic writing. Our handout on evaluating print sources will help you choose your print sources wisely, and the library has a tutorial on evaluating both print sources and websites. A librarian can help you find sources that are appropriate for the type of assignment you are completing. Just visit the reference desk at Davis or the Undergraduate Library or chat with a librarian online (the library’s IM screen name is undergradref).

Observation

Sometimes you can directly observe the thing you are interested in, by watching, listening to, touching, tasting, or smelling it. For example, if you were asked to write about Mozart’s music, you could listen to it; if your topic was how businesses attract traffic, you might go and look at window displays at the mall.

An interview is a good way to collect information that you can’t find through any other type of research. An interview can provide an expert’s opinion, biographical or first-hand experiences, and suggestions for further research.

Surveys allow you to find out some of what a group of people thinks about a topic. Designing an effective survey and interpreting the data you get can be challenging, so it’s a good idea to check with your instructor before creating or administering a survey.

Experiments

Experimental data serve as the primary form of scientific evidence. For scientific experiments, you should follow the specific guidelines of the discipline you are studying. For writing in other fields, more informal experiments might be acceptable as evidence. For example, if you want to prove that food choices in a cafeteria are affected by gender norms, you might ask classmates to undermine those norms on purpose and observe how others react. What would happen if a football player were eating dinner with his teammates and he brought a small salad and diet drink to the table, all the while murmuring about his waistline and wondering how many fat grams the salad dressing contained?

Personal experience

Using your own experiences can be a powerful way to appeal to your readers. You should, however, use personal experience only when it is appropriate to your topic, your writing goals, and your audience. Personal experience should not be your only form of evidence in most papers, and some disciplines frown on using personal experience at all. For example, a story about the microscope you received as a Christmas gift when you were nine years old is probably not applicable to your biology lab report.

Using evidence in an argument

Does evidence speak for itself.

Absolutely not. After you introduce evidence into your writing, you must say why and how this evidence supports your argument. In other words, you have to explain the significance of the evidence and its function in your paper. What turns a fact or piece of information into evidence is the connection it has with a larger claim or argument: evidence is always evidence for or against something, and you have to make that link clear.

As writers, we sometimes assume that our readers already know what we are talking about; we may be wary of elaborating too much because we think the point is obvious. But readers can’t read our minds: although they may be familiar with many of the ideas we are discussing, they don’t know what we are trying to do with those ideas unless we indicate it through explanations, organization, transitions, and so forth. Try to spell out the connections that you were making in your mind when you chose your evidence, decided where to place it in your paper, and drew conclusions based on it. Remember, you can always cut prose from your paper later if you decide that you are stating the obvious.

Here are some questions you can ask yourself about a particular bit of evidence:

• OK, I’ve just stated this point, but so what? Why is it interesting? Why should anyone care?
• What does this information imply?
• What are the consequences of thinking this way or looking at a problem this way?
• I’ve just described what something is like or how I see it, but why is it like that?
• I’ve just said that something happens—so how does it happen? How does it come to be the way it is?
• Why is this information important? Why does it matter?
• How is this idea related to my thesis? What connections exist between them? Does it support my thesis? If so, how does it do that?
• Can I give an example to illustrate this point?

Answering these questions may help you explain how your evidence is related to your overall argument.

How can I incorporate evidence into my paper?

There are many ways to present your evidence. Often, your evidence will be included as text in the body of your paper, as a quotation, paraphrase, or summary. Sometimes you might include graphs, charts, or tables; excerpts from an interview; or photographs or illustrations with accompanying captions.

When you quote, you are reproducing another writer’s words exactly as they appear on the page. Here are some tips to help you decide when to use quotations:

• Quote if you can’t say it any better and the author’s words are particularly brilliant, witty, edgy, distinctive, a good illustration of a point you’re making, or otherwise interesting.
• Quote if you are using a particularly authoritative source and you need the author’s expertise to back up your point.
• Quote if you are analyzing diction, tone, or a writer’s use of a specific word or phrase.
• Quote if you are taking a position that relies on the reader’s understanding exactly what another writer says about the topic.

Be sure to introduce each quotation you use, and always cite your sources. See our handout on quotations for more details on when to quote and how to format quotations.

Like all pieces of evidence, a quotation can’t speak for itself. If you end a paragraph with a quotation, that may be a sign that you have neglected to discuss the importance of the quotation in terms of your argument. It’s important to avoid “plop quotations,” that is, quotations that are just dropped into your paper without any introduction, discussion, or follow-up.

Paraphrasing

When you paraphrase, you take a specific section of a text and put it into your own words. Putting it into your own words doesn’t mean just changing or rearranging a few of the author’s words: to paraphrase well and avoid plagiarism, try setting your source aside and restating the sentence or paragraph you have just read, as though you were describing it to another person. Paraphrasing is different than summary because a paraphrase focuses on a particular, fairly short bit of text (like a phrase, sentence, or paragraph). You’ll need to indicate when you are paraphrasing someone else’s text by citing your source correctly, just as you would with a quotation.

When might you want to paraphrase?

• Paraphrase when you want to introduce a writer’s position, but their original words aren’t special enough to quote.
• Paraphrase when you are supporting a particular point and need to draw on a certain place in a text that supports your point—for example, when one paragraph in a source is especially relevant.
• Paraphrase when you want to present a writer’s view on a topic that differs from your position or that of another writer; you can then refute writer’s specific points in your own words after you paraphrase.
• Paraphrase when you want to comment on a particular example that another writer uses.
• Paraphrase when you need to present information that’s unlikely to be questioned.

When you summarize, you are offering an overview of an entire text, or at least a lengthy section of a text. Summary is useful when you are providing background information, grounding your own argument, or mentioning a source as a counter-argument. A summary is less nuanced than paraphrased material. It can be the most effective way to incorporate a large number of sources when you don’t have a lot of space. When you are summarizing someone else’s argument or ideas, be sure this is clear to the reader and cite your source appropriately.

Statistics, data, charts, graphs, photographs, illustrations

Sometimes the best evidence for your argument is a hard fact or visual representation of a fact. This type of evidence can be a solid backbone for your argument, but you still need to create context for your reader and draw the connections you want them to make. Remember that statistics, data, charts, graph, photographs, and illustrations are all open to interpretation. Guide the reader through the interpretation process. Again, always, cite the origin of your evidence if you didn’t produce the material you are using yourself.

Do I need more evidence?

Let’s say that you’ve identified some appropriate sources, found some evidence, explained to the reader how it fits into your overall argument, incorporated it into your draft effectively, and cited your sources. How do you tell whether you’ve got enough evidence and whether it’s working well in the service of a strong argument or analysis? Here are some techniques you can use to review your draft and assess your use of evidence.

Make a reverse outline

A reverse outline is a great technique for helping you see how each paragraph contributes to proving your thesis. When you make a reverse outline, you record the main ideas in each paragraph in a shorter (outline-like) form so that you can see at a glance what is in your paper. The reverse outline is helpful in at least three ways. First, it lets you see where you have dealt with too many topics in one paragraph (in general, you should have one main idea per paragraph). Second, the reverse outline can help you see where you need more evidence to prove your point or more analysis of that evidence. Third, the reverse outline can help you write your topic sentences: once you have decided what you want each paragraph to be about, you can write topic sentences that explain the topics of the paragraphs and state the relationship of each topic to the overall thesis of the paper.

For tips on making a reverse outline, see our handout on organization .

Color code your paper

You will need three highlighters or colored pencils for this exercise. Use one color to highlight general assertions. These will typically be the topic sentences in your paper. Next, use another color to highlight the specific evidence you provide for each assertion (including quotations, paraphrased or summarized material, statistics, examples, and your own ideas). Lastly, use another color to highlight analysis of your evidence. Which assertions are key to your overall argument? Which ones are especially contestable? How much evidence do you have for each assertion? How much analysis? In general, you should have at least as much analysis as you do evidence, or your paper runs the risk of being more summary than argument. The more controversial an assertion is, the more evidence you may need to provide in order to persuade your reader.

Play devil’s advocate, act like a child, or doubt everything

This technique may be easiest to use with a partner. Ask your friend to take on one of the roles above, then read your paper aloud to them. After each section, pause and let your friend interrogate you. If your friend is playing devil’s advocate, they will always take the opposing viewpoint and force you to keep defending yourself. If your friend is acting like a child, they will question every sentence, even seemingly self-explanatory ones. If your friend is a doubter, they won’t believe anything you say. Justifying your position verbally or explaining yourself will force you to strengthen the evidence in your paper. If you already have enough evidence but haven’t connected it clearly enough to your main argument, explaining to your friend how the evidence is relevant or what it proves may help you to do so.

Common questions and additional resources

• I have a general topic in mind; how can I develop it so I’ll know what evidence I need? And how can I get ideas for more evidence? See our handout on brainstorming .
• Who can help me find evidence on my topic? Check out UNC Libraries .
• I’m writing for a specific purpose; how can I tell what kind of evidence my audience wants? See our handouts on audience , writing for specific disciplines , and particular writing assignments .
• How should I read materials to gather evidence? See our handout on reading to write .
• How can I make a good argument? Check out our handouts on argument and thesis statements .
• How do I tell if my paragraphs and my paper are well-organized? Review our handouts on paragraph development , transitions , and reorganizing drafts .
• How do I quote my sources and incorporate those quotes into my text? Our handouts on quotations and avoiding plagiarism offer useful tips.
• How do I cite my evidence? See the UNC Libraries citation tutorial .
• I think that I’m giving evidence, but my instructor says I’m using too much summary. How can I tell? Check out our handout on using summary wisely.
• I want to use personal experience as evidence, but can I say “I”? We have a handout on when to use “I.”

Works consulted

We consulted these works while writing this handout. This is not a comprehensive list of resources on the handout’s topic, and we encourage you to do your own research to find additional publications. Please do not use this list as a model for the format of your own reference list, as it may not match the citation style you are using. For guidance on formatting citations, please see the UNC Libraries citation tutorial . We revise these tips periodically and welcome feedback.

Lunsford, Andrea A., and John J. Ruszkiewicz. 2016. Everything’s an Argument , 7th ed. Boston: Bedford/St Martin’s.

Miller, Richard E., and Kurt Spellmeyer. 2016. The New Humanities Reader , 5th ed. Boston: Cengage.

University of Maryland. 2019. “Research Using Primary Sources.” Research Guides. Last updated October 28, 2019. https://lib.guides.umd.edu/researchusingprimarysources .

You may reproduce it for non-commercial use if you use the entire handout and attribute the source: The Writing Center, University of North Carolina at Chapel Hill

Make a Gift

A Guide to Evidence Synthesis: What is Evidence Synthesis?

• Meet Our Team
• Our Published Reviews and Protocols
• What is Evidence Synthesis?
• Types of Evidence Synthesis
• Evidence Synthesis Across Disciplines
• Finding and Appraising Existing Systematic Reviews
• 0. Develop a Protocol
• 1. Draft your Research Question
• 2. Select Databases
• 3. Select Grey Literature Sources
• 4. Write a Search Strategy
• 5. Register a Protocol
• 6. Translate Search Strategies
• 7. Citation Management
• 8. Article Screening
• 9. Risk of Bias Assessment
• 10. Data Extraction
• 11. Synthesize, Map, or Describe the Results
• Evidence Synthesis Institute for Librarians
• Open Access Evidence Synthesis Resources

What are Evidence Syntheses?

What are evidence syntheses.

According to the Royal Society, 'evidence synthesis' refers to the process of bringing together information from a range of sources and disciplines to inform debates and decisions on specific issues. They generally include a methodical and comprehensive literature synthesis focused on a well-formulated research question.  Their aim is to identify and synthesize all  of the scholarly research on a particular topic, including both published and unpublished studies. Evidence syntheses are conducted in an unbiased, reproducible way to provide evidence for practice and policy-making, as well as to identify gaps in the research. Evidence syntheses may also include a meta-analysis, a more quantitative process of synthesizing and visualizing data retrieved from various studies. 

Evidence syntheses are much more time-intensive than traditional literature reviews and require a multi-person research team. See this PredicTER tool to get a sense of a systematic review timeline (one type of evidence synthesis). Before embarking on an evidence synthesis, it's important to clearly identify your reasons for conducting one. For a list of types of evidence synthesis projects, see the next tab.

How Does a Traditional Literature Review Differ From an Evidence Synthesis?

How does a systematic review differ from a traditional literature review.

One commonly used form of evidence synthesis is a systematic review.  This table compares a traditional literature review with a systematic review.

Video: Reproducibility and transparent methods (Video 3:25)

Reporting Standards

There are some reporting standards for evidence syntheses. These can serve as guidelines for protocol and manuscript preparation and journals may require that these standards are followed for the review type that is being employed (e.g. systematic review, scoping review, etc). ​

• PRISMA checklist Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.
• PRISMA-P Standards An updated version of the original PRISMA standards for protocol development.
• PRISMA - ScR Reporting guidelines for scoping reviews and evidence maps
• PRISMA-IPD Standards Extension of the original PRISMA standards for systematic reviews and meta-analyses of individual participant data.
• EQUATOR Network The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network is an international initiative that seeks to improve the reliability and value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines. They provide a list of various standards for reporting in systematic reviews.

Video: Guidelines and reporting standards

PRISMA Flow Diagram

The  PRISMA  flow diagram depicts the flow of information through the different phases of an evidence synthesis. It maps the search (number of records identified), screening (number of records included and excluded), and selection (reasons for exclusion).  Many evidence syntheses include a PRISMA flow diagram in the published manuscript.

See below for resources to help you generate your own PRISMA flow diagram.

• PRISMA Flow Diagram Tool
• PRISMA Flow Diagram Word Template
• << Previous: Our Published Reviews and Protocols
• Next: Types of Evidence Synthesis >>
• Last Updated: May 6, 2024 12:12 PM
• URL: https://guides.library.cornell.edu/evidence-synthesis

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Using Research and Evidence

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

What type of evidence should I use?

There are two types of evidence.

First hand research is research you have conducted yourself such as interviews, experiments, surveys, or personal experience and anecdotes.

Second hand research is research you are getting from various texts that has been supplied and compiled by others such as books, periodicals, and Web sites.

Regardless of what type of sources you use, they must be credible. In other words, your sources must be reliable, accurate, and trustworthy.

How do I know if a source is credible?

You can ask the following questions to determine if a source is credible.

Who is the author? Credible sources are written by authors respected in their fields of study. Responsible, credible authors will cite their sources so that you can check the accuracy of and support for what they've written. (This is also a good way to find more sources for your own research.)

How recent is the source? The choice to seek recent sources depends on your topic. While sources on the American Civil War may be decades old and still contain accurate information, sources on information technologies, or other areas that are experiencing rapid changes, need to be much more current.

What is the author's purpose? When deciding which sources to use, you should take the purpose or point of view of the author into consideration. Is the author presenting a neutral, objective view of a topic? Or is the author advocating one specific view of a topic? Who is funding the research or writing of this source? A source written from a particular point of view may be credible; however, you need to be careful that your sources don't limit your coverage of a topic to one side of a debate.

What type of sources does your audience value? If you are writing for a professional or academic audience, they may value peer-reviewed journals as the most credible sources of information. If you are writing for a group of residents in your hometown, they might be more comfortable with mainstream sources, such as Time or Newsweek . A younger audience may be more accepting of information found on the Internet than an older audience might be.

Be especially careful when evaluating Internet sources! Never use Web sites where an author cannot be determined, unless the site is associated with a reputable institution such as a respected university, a credible media outlet, government program or department, or well-known non-governmental organizations. Beware of using sites like Wikipedia , which are collaboratively developed by users. Because anyone can add or change content, the validity of information on such sites may not meet the standards for academic research.

Write Clearly: Using Evidence Effectively

What is evidence, what are primary and secondary sources, how do i use evidence, should i cite my evidence.

• Selecting and Analyzing Evidence
• Incorporating Evidence

Ask Us: Chat, email, visit or call

More writing resources

• Check out our full list of online writing resources These guides, templates, and videos are designed to help academic writers at various stages of their writing process, including the pre-writing and revising stages.

Get assistance

The library offers a range of helpful services.  All of our appointments are free of charge and confidential.

• Book an appointment

Most of the assignments you will do in university will ask you to make an argument, to take a stance, or to prove a hypothesis. The best way to do this is to research the topic, develop a thesis statement, hypothesis, or claim and then use evidence to support this claim.

Evidence is the facts, examples, or sources used to support a claim. In the sciences, this might be data retrieved from an experiment or a scientific journal article. In the humanities, it may be a quotation from the text, published information from academic critics, or a theory that supports your claims.

Think of evidence as the supports that buttress your claim, making it more solid than it would be alone. In fact, if you make a claim or an argument without evidence, your paper could appear to be unsupported opinion or not particularly well-researched. Even when the assignment elicits opinion, your paper will be more convincing if you provide evidence and the instructor may still be looking for an argument. It is also very important to remember that to use evidence effectively means to incorporate it well and to analyse it in a way that makes its connection to your argument clear and logical.

Evidence can be separated into two categories, primary and secondary sources.

Primary Sources

Primary sources are first-hand experiences, accounts, observations, reports, or narratives. Primary sources could include diaries, letters, contemporary newspapers, or eyewitness accounts of events. Official documents (e.g. the Canadian Charter of Rights and Freedoms), data collected from surveys, and lab results are also primary sources. In the humanities, the text you are writing about is also considered your primary text. So, for example, if you are writing a paper on Macbeth, then the play is your primary source. In the sciences, primary sources are also the results of an experiment that have been peer-reviewed and published in an academic journal.

Secondary Sources

Secondary sources are critiques written by academics and scholars. These sources are considered secondary because they examine primary sources to present an argument or support a point of view; as such, they may be selective with their evidence or insert themselves in a debate happening among a number of scholars. In the sciences, reviews, which are surveys of articles that demonstrate an understanding of a field, are considered secondary. It is a good idea to be aware of the bias in secondary sources when employing them as evidence.

Frequently the assignment will specify whether you need to use primary or secondary sources; however, if you are unsure about what kind of sources you need, ask your professor for clarification.

Among the forms of evidence you might draw from are:

• Graphs, charts, tables, or figures
• Experiments or studies done by peer-reviewed sources
• Surveys conducted by reputable sources
• Quotes or paraphrases from primary sources
• Quotes or paraphrases from secondary sources

Note: In general, you should not use quotes in science papers.

Each discipline and each genre of writing will have standards against which it will gauge the academic merit and use of evidence. But some general rules apply (a detailed explanation of each rule follows this list):

• Make sure your evidence is appropriate to the paper you are writing
• Make sure the evidence does, in fact, support your argument or your claims
• Tell your reader why this evidence supports your argument/claims
• Make sure you have an appropriate amount of evidence
• Make sure to appropriately cite your evidence

NOTE: Though not a general rule, your paper will be strengthened by acknowledging competing evidence – evidence that challenges your argument. This demonstrates that you have fully researched your topic and can counter claims against your argument.

After integrating your evidence into your paper, it is very important that you properly cite your evidence. Each discipline has their preferred style (MLA, APA, Chicago, etc.); if you are unclear what citation style to use, ask your professor or teaching assistant for direction.

• Next: Selecting and Analyzing Evidence >>
• Last Updated: Jul 12, 2022 1:00 PM
• URL: https://guides.lib.uoguelph.ca/UseEvidenceEffectively

Suggest an edit to this guide

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

• Research Process

Levels of evidence in research

• 5 minute read
• 101.9K views

Table of Contents

Level of evidence hierarchy

When carrying out a project you might have noticed that while searching for information, there seems to be different levels of credibility given to different types of scientific results. For example, it is not the same to use a systematic review or an expert opinion as a basis for an argument. It’s almost common sense that the first will demonstrate more accurate results than the latter, which ultimately derives from a personal opinion.

In the medical and health care area, for example, it is very important that professionals not only have access to information but also have instruments to determine which evidence is stronger and more trustworthy, building up the confidence to diagnose and treat their patients.

5 levels of evidence

With the increasing need from physicians – as well as scientists of different fields of study-, to know from which kind of research they can expect the best clinical evidence, experts decided to rank this evidence to help them identify the best sources of information to answer their questions. The criteria for ranking evidence is based on the design, methodology, validity and applicability of the different types of studies. The outcome is called “levels of evidence” or “levels of evidence hierarchy”. By organizing a well-defined hierarchy of evidence, academia experts were aiming to help scientists feel confident in using findings from high-ranked evidence in their own work or practice. For Physicians, whose daily activity depends on available clinical evidence to support decision-making, this really helps them to know which evidence to trust the most.

So, by now you know that research can be graded according to the evidential strength determined by different study designs. But how many grades are there? Which evidence should be high-ranked and low-ranked?

There are five levels of evidence in the hierarchy of evidence – being 1 (or in some cases A) for strong and high-quality evidence and 5 (or E) for evidence with effectiveness not established, as you can see in the pyramidal scheme below:

Level 1: (higher quality of evidence) – High-quality randomized trial or prospective study; testing of previously developed diagnostic criteria on consecutive patients; sensible costs and alternatives; values obtained from many studies with multiway sensitivity analyses; systematic review of Level I RCTs and Level I studies.

Level 2: Lesser quality RCT; prospective comparative study; retrospective study; untreated controls from an RCT; lesser quality prospective study; development of diagnostic criteria on consecutive patients; sensible costs and alternatives; values obtained from limited stud- ies; with multiway sensitivity analyses; systematic review of Level II studies or Level I studies with inconsistent results.

Level 3: Case-control study (therapeutic and prognostic studies); retrospective comparative study; study of nonconsecutive patients without consistently applied reference “gold” standard; analyses based on limited alternatives and costs and poor estimates; systematic review of Level III studies.

Level 4: Case series; case-control study (diagnostic studies); poor reference standard; analyses with no sensitivity analyses.

Level 5: (lower quality of evidence) – Expert opinion.

By looking at the pyramid, you can roughly distinguish what type of research gives you the highest quality of evidence and which gives you the lowest. Basically, level 1 and level 2 are filtered information – that means an author has gathered evidence from well-designed studies, with credible results, and has produced findings and conclusions appraised by renowned experts, who consider them valid and strong enough to serve researchers and scientists. Levels 3, 4 and 5 include evidence coming from unfiltered information. Because this evidence hasn’t been appraised by experts, it might be questionable, but not necessarily false or wrong.

Examples of levels of evidence

As you move up the pyramid, you will surely find higher-quality evidence. However, you will notice there is also less research available. So, if there are no resources for you available at the top, you may have to start moving down in order to find the answers you are looking for.

• Systematic Reviews: -Exhaustive summaries of all the existent literature about a certain topic. When drafting a systematic review, authors are expected to deliver a critical assessment and evaluation of all this literature rather than a simple list. Researchers that produce systematic reviews have their own criteria to locate, assemble and evaluate a body of literature.
• Meta-Analysis: Uses quantitative methods to synthesize a combination of results from independent studies. Normally, they function as an overview of clinical trials. Read more: Systematic review vs meta-analysis .
• Critically Appraised Topic: Evaluation of several research studies.
• Critically Appraised Article: Evaluation of individual research studies.
• Randomized Controlled Trial: a clinical trial in which participants or subjects (people that agree to participate in the trial) are randomly divided into groups. Placebo (control) is given to one of the groups whereas the other is treated with medication. This kind of research is key to learning about a treatment’s effectiveness.
• Cohort studies: A longitudinal study design, in which one or more samples called cohorts (individuals sharing a defining characteristic, like a disease) are exposed to an event and monitored prospectively and evaluated in predefined time intervals. They are commonly used to correlate diseases with risk factors and health outcomes.
• Case-Control Study: Selects patients with an outcome of interest (cases) and looks for an exposure factor of interest.
• Background Information/Expert Opinion: Information you can find in encyclopedias, textbooks and handbooks. This kind of evidence just serves as a good foundation for further research – or clinical practice – for it is usually too generalized.

Of course, it is recommended to use level A and/or 1 evidence for more accurate results but that doesn’t mean that all other study designs are unhelpful or useless. It all depends on your research question. Focusing once more on the healthcare and medical field, see how different study designs fit into particular questions, that are not necessarily located at the tip of the pyramid:

• Questions concerning therapy: “Which is the most efficient treatment for my patient?” >> RCT | Cohort studies | Case-Control | Case Studies
• Questions concerning diagnosis: “Which diagnose method should I use?” >> Prospective blind comparison
• Questions concerning prognosis: “How will the patient’s disease will develop over time?” >> Cohort Studies | Case Studies
• Questions concerning etiology: “What are the causes for this disease?” >> RCT | Cohort Studies | Case Studies
• Questions concerning costs: “What is the most cost-effective but safe option for my patient?” >> Economic evaluation
• Questions concerning meaning/quality of life: “What’s the quality of life of my patient going to be like?” >> Qualitative study

Find more about Levels of evidence in research on Pinterest:

17 March 2021 – Elsevier’s Mini Program Launched on WeChat Brings Quality Editing Straight to your Smartphone

• Manuscript Review

Professor Anselmo Paiva: Using Computer Vision to Tackle Medical Issues with a Little Help from Elsevier Author Services

You may also like.

Descriptive Research Design and Its Myriad Uses

Five Common Mistakes to Avoid When Writing a Biomedical Research Paper

Making Technical Writing in Environmental Engineering Accessible

To Err is Not Human: The Dangers of AI-assisted Academic Writing

When Data Speak, Listen: Importance of Data Collection and Analysis Methods

Choosing the Right Research Methodology: A Guide for Researchers

Why is data validation important in research?

Writing a good review article

Input your search keywords and press Enter.

• U.S. Locations
• UMGC Europe
• Learn Online
• Find Answers
• 855-655-8682
• Current Students

Online Guide to Writing and Research

The research process, explore more of umgc.

• Online Guide to Writing

Planning and Writing a Research Paper

Collect Evidence

The evidence you collect will shape your research paper tremendously. You will have to decide what evidence is appropriate for your audience, purpose, and thesis. To help you make this decision, consider what kind of appeal you are making to your audience—logical, emotional, or ethical. Click on the tabs below for more information.

• LOGOS OR LOGICAL APPEAL
• PATHOS OR EMOTIONAL APPEAL
• ETHOS OR ETHICAL APPEAL

You appeal to the reader’s intellect through factual or objective evidence. 

You appeal to the reader’s feelings and their heart. 

You appeal to the reader’s sense of justice, fair play or trust.  The writer is seen to have authority regarding what they are writing about.  

EXAMPLE OF A RESEARCH QUESTION AND ARGUMENT

Here is an example that describes research evidence to support an observable trend; this collection of evidence appeals to the readers’ logic and intelligence.

Research Question:  What trends in research led to the computer industry segmentation that has occurred since the 1960s?

Argument Appealing to Logic and Intelligence:   You might learn from early research that the initial phase of the US space program generated much interest in robotics and in programmable machines. That interest led to government funding for research in these areas during the 1960s. This evidence might suggest to you that the government’s role in financing research was instrumental in nurturing the fledgling computer industry of the 1960s. At the same time, you might learn that the government came into conflict with proponents of the growing industry in the 1970s by attempting to curtail domination by a single manufacturer through enforcement of the Sherman Antitrust Act. This evidence, in turn, will help you write about and explain the industry segmentation that has occurred since the 1960s, with its attendant competitive emphasis on constant improvement and innovation, realized in paradigm shifts such as those that occur in object-oriented programming.  

Key Takeaways

• The evidence you collect will shape your research paper tremendously.
• To help you make this decision, consider what kind of appeal you are making to your audience—logical, emotional, or ethical. 

Mailing Address: 3501 University Blvd. East, Adelphi, MD 20783 This work is licensed under a  Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License . © 2022 UMGC. All links to external sites were verified at the time of publication. UMGC is not responsible for the validity or integrity of information located at external sites.

Table of Contents: Online Guide to Writing

Chapter 1: College Writing

How Does College Writing Differ from Workplace Writing?

What Is College Writing?

Why So Much Emphasis on Writing?

Chapter 2: The Writing Process

Doing Exploratory Research

Getting from Notes to Your Draft

Introduction

Prewriting - Techniques to Get Started - Mining Your Intuition

Prewriting: Targeting Your Audience

Prewriting: Techniques to Get Started

Prewriting: Understanding Your Assignment

Rewriting: Being Your Own Critic

Rewriting: Creating a Revision Strategy

Rewriting: Getting Feedback

Rewriting: The Final Draft

Techniques to Get Started - Outlining

Techniques to Get Started - Using Systematic Techniques

Thesis Statement and Controlling Idea

Writing: Getting from Notes to Your Draft - Freewriting

Writing: Getting from Notes to Your Draft - Summarizing Your Ideas

Writing: Outlining What You Will Write

Chapter 3: Thinking Strategies

A Word About Style, Voice, and Tone

A Word About Style, Voice, and Tone: Style Through Vocabulary and Diction

Critical Strategies and Writing

Critical Strategies and Writing: Analysis

Critical Strategies and Writing: Evaluation

Critical Strategies and Writing: Persuasion

Critical Strategies and Writing: Synthesis

Developing a Paper Using Strategies

Kinds of Assignments You Will Write

Patterns for Presenting Information

Patterns for Presenting Information: Critiques

Patterns for Presenting Information: Discussing Raw Data

Patterns for Presenting Information: General-to-Specific Pattern

Patterns for Presenting Information: Problem-Cause-Solution Pattern

Patterns for Presenting Information: Specific-to-General Pattern

Patterns for Presenting Information: Summaries and Abstracts

Supporting with Research and Examples

Writing Essay Examinations

Writing Essay Examinations: Make Your Answer Relevant and Complete

Writing Essay Examinations: Organize Thinking Before Writing

Writing Essay Examinations: Read and Understand the Question

Chapter 4: The Research Process

Planning and Writing a Research Paper: Ask a Research Question

Planning and Writing a Research Paper: Cite Sources

Planning and Writing a Research Paper: Collect Evidence

Planning and Writing a Research Paper: Decide Your Point of View, or Role, for Your Research

Planning and Writing a Research Paper: Draw Conclusions

Planning and Writing a Research Paper: Find a Topic and Get an Overview

Planning and Writing a Research Paper: Manage Your Resources

Planning and Writing a Research Paper: Outline

Planning and Writing a Research Paper: Survey the Literature

Planning and Writing a Research Paper: Work Your Sources into Your Research Writing

Research Resources: Where Are Research Resources Found? - Human Resources

Research Resources: What Are Research Resources?

Research Resources: Where Are Research Resources Found?

Research Resources: Where Are Research Resources Found? - Electronic Resources

Research Resources: Where Are Research Resources Found? - Print Resources

Structuring the Research Paper: Formal Research Structure

Structuring the Research Paper: Informal Research Structure

The Nature of Research

The Research Assignment: How Should Research Sources Be Evaluated?

The Research Assignment: When Is Research Needed?

The Research Assignment: Why Perform Research?

Chapter 5: Academic Integrity

Academic Integrity

Giving Credit to Sources

Giving Credit to Sources: Copyright Laws

Giving Credit to Sources: Documentation

Giving Credit to Sources: Style Guides

Integrating Sources

Practicing Academic Integrity

Practicing Academic Integrity: Keeping Accurate Records

Practicing Academic Integrity: Managing Source Material

Practicing Academic Integrity: Managing Source Material - Paraphrasing Your Source

Practicing Academic Integrity: Managing Source Material - Quoting Your Source

Practicing Academic Integrity: Managing Source Material - Summarizing Your Sources

Types of Documentation

Types of Documentation: Bibliographies and Source Lists

Types of Documentation: Citing World Wide Web Sources

Types of Documentation: In-Text or Parenthetical Citations

Types of Documentation: In-Text or Parenthetical Citations - APA Style

Types of Documentation: In-Text or Parenthetical Citations - CSE/CBE Style

Types of Documentation: In-Text or Parenthetical Citations - Chicago Style

Types of Documentation: In-Text or Parenthetical Citations - MLA Style

Types of Documentation: Note Citations

Chapter 6: Using Library Resources

Finding Library Resources

Chapter 7: Assessing Your Writing

How Is Writing Graded?

How Is Writing Graded?: A General Assessment Tool

The Draft Stage

The Draft Stage: The First Draft

The Draft Stage: The Revision Process and the Final Draft

The Draft Stage: Using Feedback

The Research Stage

Using Assessment to Improve Your Writing

Chapter 8: Other Frequently Assigned Papers

Reviews and Reaction Papers: Article and Book Reviews

Reviews and Reaction Papers: Reaction Papers

Writing Arguments

Writing Arguments: Adapting the Argument Structure

Writing Arguments: Purposes of Argument

Writing Arguments: References to Consult for Writing Arguments

Writing Arguments: Steps to Writing an Argument - Anticipate Active Opposition

Writing Arguments: Steps to Writing an Argument - Determine Your Organization

Writing Arguments: Steps to Writing an Argument - Develop Your Argument

Writing Arguments: Steps to Writing an Argument - Introduce Your Argument

Writing Arguments: Steps to Writing an Argument - State Your Thesis or Proposition

Writing Arguments: Steps to Writing an Argument - Write Your Conclusion

Writing Arguments: Types of Argument

Appendix A: Books to Help Improve Your Writing

Dictionaries

General Style Manuals

Researching on the Internet

Special Style Manuals

Writing Handbooks

Appendix B: Collaborative Writing and Peer Reviewing

Collaborative Writing: Assignments to Accompany the Group Project

Collaborative Writing: Informal Progress Report

Collaborative Writing: Issues to Resolve

Collaborative Writing: Methodology

Collaborative Writing: Peer Evaluation

Collaborative Writing: Tasks of Collaborative Writing Group Members

Collaborative Writing: Writing Plan

General Introduction

Peer Reviewing

Appendix C: Developing an Improvement Plan

Working with Your Instructor’s Comments and Grades

Appendix D: Writing Plan and Project Schedule

Devising a Writing Project Plan and Schedule

Reviewing Your Plan with Others

By using our website you agree to our use of cookies. Learn more about how we use cookies by reading our  Privacy Policy .

• Library databases
• Library website

Evidence-Based Research: Levels of Evidence Pyramid

Introduction.

One way to organize the different types of evidence involved in evidence-based practice research is the levels of evidence pyramid. The pyramid includes a variety of evidence types and levels.

• systematic reviews
• critically-appraised topics
• critically-appraised individual articles
• randomized controlled trials
• cohort studies
• case-controlled studies, case series, and case reports
• Background information, expert opinion

Levels of evidence pyramid

The levels of evidence pyramid provides a way to visualize both the quality of evidence and the amount of evidence available. For example, systematic reviews are at the top of the pyramid, meaning they are both the highest level of evidence and the least common. As you go down the pyramid, the amount of evidence will increase as the quality of the evidence decreases.

Text alternative for Levels of Evidence Pyramid diagram

EBM Pyramid and EBM Page Generator, copyright 2006 Trustees of Dartmouth College and Yale University. All Rights Reserved. Produced by Jan Glover, David Izzo, Karen Odato and Lei Wang.

Filtered Resources

Filtered resources appraise the quality of studies and often make recommendations for practice. The main types of filtered resources in evidence-based practice are:

Scroll down the page to the Systematic reviews , Critically-appraised topics , and Critically-appraised individual articles sections for links to resources where you can find each of these types of filtered information.

Systematic reviews

Authors of a systematic review ask a specific clinical question, perform a comprehensive literature review, eliminate the poorly done studies, and attempt to make practice recommendations based on the well-done studies. Systematic reviews include only experimental, or quantitative, studies, and often include only randomized controlled trials.

You can find systematic reviews in these filtered databases :

• Cochrane Database of Systematic Reviews Cochrane systematic reviews are considered the gold standard for systematic reviews. This database contains both systematic reviews and review protocols. To find only systematic reviews, select Cochrane Reviews in the Document Type box.
• JBI EBP Database (formerly Joanna Briggs Institute EBP Database) This database includes systematic reviews, evidence summaries, and best practice information sheets. To find only systematic reviews, click on Limits and then select Systematic Reviews in the Publication Types box. To see how to use the limit and find full text, please see our Joanna Briggs Institute Search Help page .

You can also find systematic reviews in this unfiltered database :

To learn more about finding systematic reviews, please see our guide:

• Filtered Resources: Systematic Reviews

Critically-appraised topics

Authors of critically-appraised topics evaluate and synthesize multiple research studies. Critically-appraised topics are like short systematic reviews focused on a particular topic.

You can find critically-appraised topics in these resources:

• Annual Reviews This collection offers comprehensive, timely collections of critical reviews written by leading scientists. To find reviews on your topic, use the search box in the upper-right corner.
• Guideline Central This free database offers quick-reference guideline summaries organized by a new non-profit initiative which will aim to fill the gap left by the sudden closure of AHRQ’s National Guideline Clearinghouse (NGC).
• JBI EBP Database (formerly Joanna Briggs Institute EBP Database) To find critically-appraised topics in JBI, click on Limits and then select Evidence Summaries from the Publication Types box. To see how to use the limit and find full text, please see our Joanna Briggs Institute Search Help page .
• National Institute for Health and Care Excellence (NICE) Evidence-based recommendations for health and care in England.
• Filtered Resources: Critically-Appraised Topics

Critically-appraised individual articles

Authors of critically-appraised individual articles evaluate and synopsize individual research studies.

You can find critically-appraised individual articles in these resources:

• EvidenceAlerts Quality articles from over 120 clinical journals are selected by research staff and then rated for clinical relevance and interest by an international group of physicians. Note: You must create a free account to search EvidenceAlerts.
• ACP Journal Club This journal publishes reviews of research on the care of adults and adolescents. You can either browse this journal or use the Search within this publication feature.
• Evidence-Based Nursing This journal reviews research studies that are relevant to best nursing practice. You can either browse individual issues or use the search box in the upper-right corner.

To learn more about finding critically-appraised individual articles, please see our guide:

• Filtered Resources: Critically-Appraised Individual Articles

Unfiltered resources

You may not always be able to find information on your topic in the filtered literature. When this happens, you'll need to search the primary or unfiltered literature. Keep in mind that with unfiltered resources, you take on the role of reviewing what you find to make sure it is valid and reliable.

Note: You can also find systematic reviews and other filtered resources in these unfiltered databases.

The Levels of Evidence Pyramid includes unfiltered study types in this order of evidence from higher to lower:

You can search for each of these types of evidence in the following databases:

TRIP database

Background information & expert opinion.

Background information and expert opinions are not necessarily backed by research studies. They include point-of-care resources, textbooks, conference proceedings, etc.

• Family Physicians Inquiries Network: Clinical Inquiries Provide the ideal answers to clinical questions using a structured search, critical appraisal, authoritative recommendations, clinical perspective, and rigorous peer review. Clinical Inquiries deliver best evidence for point-of-care use.
• Harrison, T. R., & Fauci, A. S. (2009). Harrison's Manual of Medicine . New York: McGraw-Hill Professional. Contains the clinical portions of Harrison's Principles of Internal Medicine .
• Lippincott manual of nursing practice (8th ed.). (2006). Philadelphia, PA: Lippincott Williams & Wilkins. Provides background information on clinical nursing practice.
• Medscape: Drugs & Diseases An open-access, point-of-care medical reference that includes clinical information from top physicians and pharmacists in the United States and worldwide.
• Virginia Henderson Global Nursing e-Repository An open-access repository that contains works by nurses and is sponsored by Sigma Theta Tau International, the Honor Society of Nursing. Note: This resource contains both expert opinion and evidence-based practice articles.
• Previous Page: Phrasing Research Questions
• Next Page: Evidence Types
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

• Skip to Content
• Skip to Main Navigation
• Skip to Search

Indiana University Bloomington Indiana University Bloomington IU Bloomington

• Mission, Vision, and Inclusive Language Statement
• Locations & Hours
• Undergraduate Employment
• Graduate Employment
• Frequently Asked Questions
• Newsletter Archive
• Support WTS
• Schedule an Appointment
• Online Tutoring
• Before your Appointment
• WTS Policies
• Group Tutoring
• Students Referred by Instructors
• Paid External Editing Services
• Writing Guides
• Scholarly Write-in
• Dissertation Writing Groups
• Journal Article Writing Groups
• Early Career Graduate Student Writing Workshop
• Workshops for Graduate Students
• Teaching Resources
• Syllabus Information
• Course-specific Tutoring
• Nominate a Peer Tutor
• Tutoring Feedback
• Schedule Appointment
• Campus Writing Program

Writing Tutorial Services

Using evidence.

Like a lawyer in a jury trial, a writer must convince her audience of the validity of her argument by using evidence effectively. As a writer, you must also use evidence to persuade your readers to accept your claims. But how do you use evidence to your advantage? By leading your reader through your reasoning.

The types of evidence you use change from discipline to discipline--you might use quotations from a poem or a literary critic, for example, in a literature paper; you might use data from an experiment in a lab report.

The process of putting together your argument is called analysis --it interprets evidence in order to support, test, and/or refine a claim . The chief claim in an analytical essay is called the thesis . A thesis provides the controlling idea for a paper and should be original (that is, not completely obvious), assertive, and arguable. A strong thesis also requires solid evidence to support and develop it because without evidence, a claim is merely an unsubstantiated idea or opinion.

This Web page will cover these basic issues (you can click or scroll down to a particular topic):

• Incorporating evidence effectively.
• Integrating quotations smoothly.
• Citing your sources.

Incorporating Evidence Into Your Essay

When should you incorporate evidence.

Once you have formulated your claim, your thesis (see the WTS pamphlet, " How to Write a Thesis Statement ," for ideas and tips), you should use evidence to help strengthen your thesis and any assertion you make that relates to your thesis. Here are some ways to work evidence into your writing:

• Offer evidence that agrees with your stance up to a point, then add to it with ideas of your own.
• Present evidence that contradicts your stance, and then argue against (refute) that evidence and therefore strengthen your position.
• Use sources against each other, as if they were experts on a panel discussing your proposition.
• Use quotations to support your assertion, not merely to state or restate your claim.

Weak and Strong Uses of Evidence

In order to use evidence effectively, you need to integrate it smoothly into your essay by following this pattern:

• State your claim.
• Give your evidence, remembering to relate it to the claim.
• Comment on the evidence to show how it supports the claim.

To see the differences between strong and weak uses of evidence, here are two paragraphs.

Weak use of evidence

Today, we are too self-centered. Most families no longer sit down to eat together, preferring instead to eat on the go while rushing to the next appointment (Gleick 148). Everything is about what we want.

This is a weak example of evidence because the evidence is not related to the claim. What does the claim about self-centeredness have to do with families eating together? The writer doesn't explain the connection.

The same evidence can be used to support the same claim, but only with the addition of a clear connection between claim and evidence, and some analysis of the evidence cited.

Stronger use of evidence

Today, Americans are too self-centered. Even our families don't matter as much anymore as they once did. Other people and activities take precedence. In fact, the evidence shows that most American families no longer eat together, preferring instead to eat on the go while rushing to the next appointment (Gleick 148). Sit-down meals are a time to share and connect with others; however, that connection has become less valued, as families begin to prize individual activities over shared time, promoting self-centeredness over group identity.

This is a far better example, as the evidence is more smoothly integrated into the text, the link between the claim and the evidence is strengthened, and the evidence itself is analyzed to provide support for the claim.

Using Quotations: A Special Type of Evidence

One effective way to support your claim is to use quotations. However, because quotations involve someone else's words, you need to take special care to integrate this kind of evidence into your essay. Here are two examples using quotations, one less effective and one more so.

Ineffective Use of Quotation

Today, we are too self-centered. "We are consumers-on-the-run . . . the very notion of the family meal as a sit-down occasion is vanishing. Adults and children alike eat . . . on the way to their next activity" (Gleick 148). Everything is about what we want.

This example is ineffective because the quotation is not integrated with the writer's ideas. Notice how the writer has dropped the quotation into the paragraph without making any connection between it and the claim. Furthermore, she has not discussed the quotation's significance, which makes it difficult for the reader to see the relationship between the evidence and the writer's point.

A More Effective Use of Quotation

Today, Americans are too self-centered. Even our families don't matter as much any more as they once did. Other people and activities take precedence, as James Gleick says in his book, Faster . "We are consumers-on-the-run . . . the very notion of the family meal as a sit-down occasion is vanishing. Adults and children alike eat . . . on the way to their next activity" (148). Sit-down meals are a time to share and connect with others; however, that connection has become less valued, as families begin to prize individual activities over shared time, promoting self-centeredness over group identity.

The second example is more effective because it follows the guidelines for incorporating evidence into an essay. Notice, too, that it uses a lead-in phrase (". . . as James Gleick says in his book, Faster ") to introduce the direct quotation. This lead-in phrase helps to integrate the quotation with the writer's ideas. Also notice that the writer discusses and comments upon the quotation immediately afterwards, which allows the reader to see the quotation's connection to the writer's point.

REMEMBER: Discussing the significance of your evidence develops and expands your paper!

Citing Your Sources

Evidence appears in essays in the form of quotations and paraphrasing. Both forms of evidence must be cited in your text. Citing evidence means distinguishing other writers' information from your own ideas and giving credit to your sources. There are plenty of general ways to do citations. Note both the lead-in phrases and the punctuation (except the brackets) in the following examples:

Quoting: According to Source X, "[direct quotation]" ([date or page #]).

Paraphrasing: Although Source Z argues that [his/her point in your own words], a better way to view the issue is [your own point] ([citation]).

Summarizing: In her book, Source P's main points are Q, R, and S [citation].

Your job during the course of your essay is to persuade your readers that your claims are feasible and are the most effective way of interpreting the evidence.

Questions to Ask Yourself When Revising Your Paper

• Have I offered my reader evidence to substantiate each assertion I make in my paper?
• Do I thoroughly explain why/how my evidence backs up my ideas?
• Do I avoid generalizing in my paper by specifically explaining how my evidence is representative?
• Do I provide evidence that not only confirms but also qualifies my paper's main claims?
• Do I use evidence to test and evolve my ideas, rather than to just confirm them?
• Do I cite my sources thoroughly and correctly?

Produced by Writing Tutorial Services, Indiana University, Bloomington, IN

Writing Tutorial Services social media channels

12.1 Introducing Research and Research Evidence

Learning outcomes.

By the end of this section, you will be able to:

• Articulate how research evidence and sources are key rhetorical concepts in presenting a position or an argument.
• Locate and distinguish between primary and secondary research materials.
• Implement methods and technologies commonly used for research and communication within various fields.

The writing tasks for this chapter and the next two chapters are based on argumentative research. However, not all researched evidence (data) is presented in the same genre. You may need to gather evidence for a poster, a performance, a story, an art exhibit, or even an architectural design. Although the genre may vary, you usually will be required to present a perspective , or viewpoint, about a debatable issue and persuade readers to support the “validity of your viewpoint,” as discussed in Position Argument: Practicing the Art of Rhetoric . Remember, too, that a debatable issue is one that has more than a single perspective and is subject to disagreement.

The Research Process

Although individual research processes are rhetorically situated, they share some common aspects:

• Interest. The researcher has a genuine interest in the topic. It may be difficult to fake curiosity, but it is possible to develop it. Some academic assignments will allow you to pursue issues that are personally important to you; others will require you to dive into the research first and generate interest as you go.
• Questions. The researcher asks questions. At first, these questions are general. However, as researchers gain more knowledge, the questions become more sharply focused. No matter what your research assignment is, begin by articulating questions, find out where the answers lead, and then ask still more questions.
• Answers. The researcher seeks answers from people as well as from print and other media. Research projects profit when you ask knowledgeable people, such as librarians and other professionals, to help you answer questions or point you in directions to find answers. Information about research is covered more extensively in Research Process: Accessing and Recording Information and Annotated Bibliography: Gathering, Evaluating, and Documenting Sources .
• Field research. The researcher conducts field research. Field research allows researchers not only to ask questions of experts but also to observe and experience directly. It allows researchers to generate original data. No matter how much other people tell you, your knowledge increases through personal observations. In some subject areas, field research is as important as library or database research. This information is covered more extensively in Research Process: Accessing and Recording Information .
• Examination of texts. The researcher examines texts. Consulting a broad range of texts—such as magazines, brochures, newspapers, archives, blogs, videos, documentaries, or peer-reviewed journals—is crucial in academic research.
• Evaluation of sources. The researcher evaluates sources. As your research progresses, you will double-check information to find out whether it is confirmed by more than one source. In informal research, researchers evaluate sources to ensure that the final decision is satisfactory. Similarly, in academic research, researchers evaluate sources to ensure that the final product is accurate and convincing. Previewed here, this information is covered more extensively in Research Process: Accessing and Recording Information .
• Writing. The researcher writes. The writing during the research process can take a range of forms: from notes during library, database, or field work; to journal reflections on the research process; to drafts of the final product. In practical research, writing helps researchers find, remember, and explore information. In academic research, writing is even more important because the results must be reported accurately and thoroughly.
• Testing and Experimentation. The researcher tests and experiments. Because opinions vary on debatable topics and because few research topics have correct or incorrect answers, it is important to test and conduct experiments on possible hypotheses or solutions.
• Synthesis. The researcher synthesizes. By combining information from various sources, researchers support claims or arrive at new conclusions. When synthesizing, researchers connect evidence and ideas, both original and borrowed. Accumulating, sorting, and synthesizing information enables researchers to consider what evidence to use in support of a thesis and in what ways.
• Presentation. The researcher presents findings in an interesting, focused, and well-documented product.

Types of Research Evidence

Research evidence usually consists of data, which comes from borrowed information that you use to develop your thesis and support your organizational structure and reasoning. This evidence can take a range of forms, depending on the type of research conducted, the audience, and the genre for reporting the research.

Primary Research Sources

Although precise definitions vary somewhat by discipline, primary data sources are generally defined as firsthand accounts, such as texts or other materials produced by someone drawing from direct experience or observation. Primary source documents include, but are not limited to, personal narratives and diaries; eyewitness accounts; interviews; original documents such as treaties, official certificates, and government documents detailing laws or acts; speeches; newspaper coverage of events at the time they occurred; observations; and experiments. Primary source data is, in other words, original and in some way conducted or collected primarily by the researcher. The Research Process: Where to Look for Existing Sources and Compiling Sources for an Annotated Bibliography contain more information on both primary and secondary sources.

Secondary Research Sources

Secondary sources , on the other hand, are considered at least one step removed from the experience. That is, they rely on sources other than direct observation or firsthand experience. Secondary sources include, but are not limited to, most books, articles online or in databases, and textbooks (which are sometimes classified as tertiary sources because, like encyclopedias and other reference works, their primary purpose might be to summarize or otherwise condense information). Secondary sources regularly cite and build upon primary sources to provide perspective and analysis. Effective use of researched evidence usually includes both primary and secondary sources. Works of history, for example, draw on a large range of primary and secondary sources, citing, analyzing, and synthesizing information to present as many perspectives of a past event in as rich and nuanced a way as possible.

It is important to note that the distinction between primary and secondary sources depends in part on their use: that is, the same document can be both a primary source and a secondary source. For example, if Scholar X wrote a biography about Artist Y, the biography would be a secondary source about the artist and, at the same time, a primary source about the scholar.

As an Amazon Associate we earn from qualifying purchases.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute OpenStax.

Access for free at https://openstax.org/books/writing-guide/pages/1-unit-introduction

• Authors: Michelle Bachelor Robinson, Maria Jerskey, featuring Toby Fulwiler
• Publisher/website: OpenStax
• Book title: Writing Guide with Handbook
• Publication date: Dec 21, 2021
• Location: Houston, Texas
• Book URL: https://openstax.org/books/writing-guide/pages/1-unit-introduction
• Section URL: https://openstax.org/books/writing-guide/pages/12-1-introducing-research-and-research-evidence

© Dec 19, 2023 OpenStax. Textbook content produced by OpenStax is licensed under a Creative Commons Attribution License . The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

• Privacy Policy

Home » Research Paper – Structure, Examples and Writing Guide

Research Paper – Structure, Examples and Writing Guide

Table of Contents

Research Paper

Definition:

Research Paper is a written document that presents the author’s original research, analysis, and interpretation of a specific topic or issue.

It is typically based on Empirical Evidence, and may involve qualitative or quantitative research methods, or a combination of both. The purpose of a research paper is to contribute new knowledge or insights to a particular field of study, and to demonstrate the author’s understanding of the existing literature and theories related to the topic.

Structure of Research Paper

The structure of a research paper typically follows a standard format, consisting of several sections that convey specific information about the research study. The following is a detailed explanation of the structure of a research paper:

The title page contains the title of the paper, the name(s) of the author(s), and the affiliation(s) of the author(s). It also includes the date of submission and possibly, the name of the journal or conference where the paper is to be published.

The abstract is a brief summary of the research paper, typically ranging from 100 to 250 words. It should include the research question, the methods used, the key findings, and the implications of the results. The abstract should be written in a concise and clear manner to allow readers to quickly grasp the essence of the research.

Introduction

The introduction section of a research paper provides background information about the research problem, the research question, and the research objectives. It also outlines the significance of the research, the research gap that it aims to fill, and the approach taken to address the research question. Finally, the introduction section ends with a clear statement of the research hypothesis or research question.

Literature Review

The literature review section of a research paper provides an overview of the existing literature on the topic of study. It includes a critical analysis and synthesis of the literature, highlighting the key concepts, themes, and debates. The literature review should also demonstrate the research gap and how the current study seeks to address it.

The methods section of a research paper describes the research design, the sample selection, the data collection and analysis procedures, and the statistical methods used to analyze the data. This section should provide sufficient detail for other researchers to replicate the study.

The results section presents the findings of the research, using tables, graphs, and figures to illustrate the data. The findings should be presented in a clear and concise manner, with reference to the research question and hypothesis.

The discussion section of a research paper interprets the findings and discusses their implications for the research question, the literature review, and the field of study. It should also address the limitations of the study and suggest future research directions.

The conclusion section summarizes the main findings of the study, restates the research question and hypothesis, and provides a final reflection on the significance of the research.

The references section provides a list of all the sources cited in the paper, following a specific citation style such as APA, MLA or Chicago.

How to Write Research Paper

You can write Research Paper by the following guide:

• Choose a Topic: The first step is to select a topic that interests you and is relevant to your field of study. Brainstorm ideas and narrow down to a research question that is specific and researchable.
• Conduct a Literature Review: The literature review helps you identify the gap in the existing research and provides a basis for your research question. It also helps you to develop a theoretical framework and research hypothesis.
• Develop a Thesis Statement : The thesis statement is the main argument of your research paper. It should be clear, concise and specific to your research question.
• Plan your Research: Develop a research plan that outlines the methods, data sources, and data analysis procedures. This will help you to collect and analyze data effectively.
• Collect and Analyze Data: Collect data using various methods such as surveys, interviews, observations, or experiments. Analyze data using statistical tools or other qualitative methods.
• Organize your Paper : Organize your paper into sections such as Introduction, Literature Review, Methods, Results, Discussion, and Conclusion. Ensure that each section is coherent and follows a logical flow.
• Write your Paper : Start by writing the introduction, followed by the literature review, methods, results, discussion, and conclusion. Ensure that your writing is clear, concise, and follows the required formatting and citation styles.
• Edit and Proofread your Paper: Review your paper for grammar and spelling errors, and ensure that it is well-structured and easy to read. Ask someone else to review your paper to get feedback and suggestions for improvement.
• Cite your Sources: Ensure that you properly cite all sources used in your research paper. This is essential for giving credit to the original authors and avoiding plagiarism.

Research Paper Example

Note : The below example research paper is for illustrative purposes only and is not an actual research paper. Actual research papers may have different structures, contents, and formats depending on the field of study, research question, data collection and analysis methods, and other factors. Students should always consult with their professors or supervisors for specific guidelines and expectations for their research papers.

Research Paper Example sample for Students:

Title: The Impact of Social Media on Mental Health among Young Adults

Abstract: This study aims to investigate the impact of social media use on the mental health of young adults. A literature review was conducted to examine the existing research on the topic. A survey was then administered to 200 university students to collect data on their social media use, mental health status, and perceived impact of social media on their mental health. The results showed that social media use is positively associated with depression, anxiety, and stress. The study also found that social comparison, cyberbullying, and FOMO (Fear of Missing Out) are significant predictors of mental health problems among young adults.

Introduction: Social media has become an integral part of modern life, particularly among young adults. While social media has many benefits, including increased communication and social connectivity, it has also been associated with negative outcomes, such as addiction, cyberbullying, and mental health problems. This study aims to investigate the impact of social media use on the mental health of young adults.

Literature Review: The literature review highlights the existing research on the impact of social media use on mental health. The review shows that social media use is associated with depression, anxiety, stress, and other mental health problems. The review also identifies the factors that contribute to the negative impact of social media, including social comparison, cyberbullying, and FOMO.

Methods : A survey was administered to 200 university students to collect data on their social media use, mental health status, and perceived impact of social media on their mental health. The survey included questions on social media use, mental health status (measured using the DASS-21), and perceived impact of social media on their mental health. Data were analyzed using descriptive statistics and regression analysis.

Results : The results showed that social media use is positively associated with depression, anxiety, and stress. The study also found that social comparison, cyberbullying, and FOMO are significant predictors of mental health problems among young adults.

Discussion : The study’s findings suggest that social media use has a negative impact on the mental health of young adults. The study highlights the need for interventions that address the factors contributing to the negative impact of social media, such as social comparison, cyberbullying, and FOMO.

Conclusion : In conclusion, social media use has a significant impact on the mental health of young adults. The study’s findings underscore the need for interventions that promote healthy social media use and address the negative outcomes associated with social media use. Future research can explore the effectiveness of interventions aimed at reducing the negative impact of social media on mental health. Additionally, longitudinal studies can investigate the long-term effects of social media use on mental health.

Limitations : The study has some limitations, including the use of self-report measures and a cross-sectional design. The use of self-report measures may result in biased responses, and a cross-sectional design limits the ability to establish causality.

Implications: The study’s findings have implications for mental health professionals, educators, and policymakers. Mental health professionals can use the findings to develop interventions that address the negative impact of social media use on mental health. Educators can incorporate social media literacy into their curriculum to promote healthy social media use among young adults. Policymakers can use the findings to develop policies that protect young adults from the negative outcomes associated with social media use.

References :

• Twenge, J. M., & Campbell, W. K. (2019). Associations between screen time and lower psychological well-being among children and adolescents: Evidence from a population-based study. Preventive medicine reports, 15, 100918.
• Primack, B. A., Shensa, A., Escobar-Viera, C. G., Barrett, E. L., Sidani, J. E., Colditz, J. B., … & James, A. E. (2017). Use of multiple social media platforms and symptoms of depression and anxiety: A nationally-representative study among US young adults. Computers in Human Behavior, 69, 1-9.
• Van der Meer, T. G., & Verhoeven, J. W. (2017). Social media and its impact on academic performance of students. Journal of Information Technology Education: Research, 16, 383-398.

Appendix : The survey used in this study is provided below.

Social Media and Mental Health Survey

• How often do you use social media per day?
• Less than 30 minutes
• 30 minutes to 1 hour
• 1 to 2 hours
• 2 to 4 hours
• More than 4 hours
• Which social media platforms do you use?
• Others (Please specify)
• How often do you experience the following on social media?
• Social comparison (comparing yourself to others)
• Cyberbullying
• Fear of Missing Out (FOMO)
• Have you ever experienced any of the following mental health problems in the past month?
• Do you think social media use has a positive or negative impact on your mental health?
• Very positive
• Somewhat positive
• Somewhat negative
• Very negative
• In your opinion, which factors contribute to the negative impact of social media on mental health?
• Social comparison
• In your opinion, what interventions could be effective in reducing the negative impact of social media on mental health?
• Education on healthy social media use
• Counseling for mental health problems caused by social media
• Social media detox programs
• Regulation of social media use

Thank you for your participation!

Applications of Research Paper

Research papers have several applications in various fields, including:

• Advancing knowledge: Research papers contribute to the advancement of knowledge by generating new insights, theories, and findings that can inform future research and practice. They help to answer important questions, clarify existing knowledge, and identify areas that require further investigation.
• Informing policy: Research papers can inform policy decisions by providing evidence-based recommendations for policymakers. They can help to identify gaps in current policies, evaluate the effectiveness of interventions, and inform the development of new policies and regulations.
• Improving practice: Research papers can improve practice by providing evidence-based guidance for professionals in various fields, including medicine, education, business, and psychology. They can inform the development of best practices, guidelines, and standards of care that can improve outcomes for individuals and organizations.
• Educating students : Research papers are often used as teaching tools in universities and colleges to educate students about research methods, data analysis, and academic writing. They help students to develop critical thinking skills, research skills, and communication skills that are essential for success in many careers.
• Fostering collaboration: Research papers can foster collaboration among researchers, practitioners, and policymakers by providing a platform for sharing knowledge and ideas. They can facilitate interdisciplinary collaborations and partnerships that can lead to innovative solutions to complex problems.

When to Write Research Paper

Research papers are typically written when a person has completed a research project or when they have conducted a study and have obtained data or findings that they want to share with the academic or professional community. Research papers are usually written in academic settings, such as universities, but they can also be written in professional settings, such as research organizations, government agencies, or private companies.

Here are some common situations where a person might need to write a research paper:

• For academic purposes: Students in universities and colleges are often required to write research papers as part of their coursework, particularly in the social sciences, natural sciences, and humanities. Writing research papers helps students to develop research skills, critical thinking skills, and academic writing skills.
• For publication: Researchers often write research papers to publish their findings in academic journals or to present their work at academic conferences. Publishing research papers is an important way to disseminate research findings to the academic community and to establish oneself as an expert in a particular field.
• To inform policy or practice : Researchers may write research papers to inform policy decisions or to improve practice in various fields. Research findings can be used to inform the development of policies, guidelines, and best practices that can improve outcomes for individuals and organizations.
• To share new insights or ideas: Researchers may write research papers to share new insights or ideas with the academic or professional community. They may present new theories, propose new research methods, or challenge existing paradigms in their field.

Purpose of Research Paper

The purpose of a research paper is to present the results of a study or investigation in a clear, concise, and structured manner. Research papers are written to communicate new knowledge, ideas, or findings to a specific audience, such as researchers, scholars, practitioners, or policymakers. The primary purposes of a research paper are:

• To contribute to the body of knowledge : Research papers aim to add new knowledge or insights to a particular field or discipline. They do this by reporting the results of empirical studies, reviewing and synthesizing existing literature, proposing new theories, or providing new perspectives on a topic.
• To inform or persuade: Research papers are written to inform or persuade the reader about a particular issue, topic, or phenomenon. They present evidence and arguments to support their claims and seek to persuade the reader of the validity of their findings or recommendations.
• To advance the field: Research papers seek to advance the field or discipline by identifying gaps in knowledge, proposing new research questions or approaches, or challenging existing assumptions or paradigms. They aim to contribute to ongoing debates and discussions within a field and to stimulate further research and inquiry.
• To demonstrate research skills: Research papers demonstrate the author’s research skills, including their ability to design and conduct a study, collect and analyze data, and interpret and communicate findings. They also demonstrate the author’s ability to critically evaluate existing literature, synthesize information from multiple sources, and write in a clear and structured manner.

Characteristics of Research Paper

Research papers have several characteristics that distinguish them from other forms of academic or professional writing. Here are some common characteristics of research papers:

• Evidence-based: Research papers are based on empirical evidence, which is collected through rigorous research methods such as experiments, surveys, observations, or interviews. They rely on objective data and facts to support their claims and conclusions.
• Structured and organized: Research papers have a clear and logical structure, with sections such as introduction, literature review, methods, results, discussion, and conclusion. They are organized in a way that helps the reader to follow the argument and understand the findings.
• Formal and objective: Research papers are written in a formal and objective tone, with an emphasis on clarity, precision, and accuracy. They avoid subjective language or personal opinions and instead rely on objective data and analysis to support their arguments.
• Citations and references: Research papers include citations and references to acknowledge the sources of information and ideas used in the paper. They use a specific citation style, such as APA, MLA, or Chicago, to ensure consistency and accuracy.
• Peer-reviewed: Research papers are often peer-reviewed, which means they are evaluated by other experts in the field before they are published. Peer-review ensures that the research is of high quality, meets ethical standards, and contributes to the advancement of knowledge in the field.
• Objective and unbiased: Research papers strive to be objective and unbiased in their presentation of the findings. They avoid personal biases or preconceptions and instead rely on the data and analysis to draw conclusions.

Advantages of Research Paper

Research papers have many advantages, both for the individual researcher and for the broader academic and professional community. Here are some advantages of research papers:

• Contribution to knowledge: Research papers contribute to the body of knowledge in a particular field or discipline. They add new information, insights, and perspectives to existing literature and help advance the understanding of a particular phenomenon or issue.
• Opportunity for intellectual growth: Research papers provide an opportunity for intellectual growth for the researcher. They require critical thinking, problem-solving, and creativity, which can help develop the researcher’s skills and knowledge.
• Career advancement: Research papers can help advance the researcher’s career by demonstrating their expertise and contributions to the field. They can also lead to new research opportunities, collaborations, and funding.
• Academic recognition: Research papers can lead to academic recognition in the form of awards, grants, or invitations to speak at conferences or events. They can also contribute to the researcher’s reputation and standing in the field.
• Impact on policy and practice: Research papers can have a significant impact on policy and practice. They can inform policy decisions, guide practice, and lead to changes in laws, regulations, or procedures.
• Advancement of society: Research papers can contribute to the advancement of society by addressing important issues, identifying solutions to problems, and promoting social justice and equality.

Limitations of Research Paper

Research papers also have some limitations that should be considered when interpreting their findings or implications. Here are some common limitations of research papers:

• Limited generalizability: Research findings may not be generalizable to other populations, settings, or contexts. Studies often use specific samples or conditions that may not reflect the broader population or real-world situations.
• Potential for bias : Research papers may be biased due to factors such as sample selection, measurement errors, or researcher biases. It is important to evaluate the quality of the research design and methods used to ensure that the findings are valid and reliable.
• Ethical concerns: Research papers may raise ethical concerns, such as the use of vulnerable populations or invasive procedures. Researchers must adhere to ethical guidelines and obtain informed consent from participants to ensure that the research is conducted in a responsible and respectful manner.
• Limitations of methodology: Research papers may be limited by the methodology used to collect and analyze data. For example, certain research methods may not capture the complexity or nuance of a particular phenomenon, or may not be appropriate for certain research questions.
• Publication bias: Research papers may be subject to publication bias, where positive or significant findings are more likely to be published than negative or non-significant findings. This can skew the overall findings of a particular area of research.
• Time and resource constraints: Research papers may be limited by time and resource constraints, which can affect the quality and scope of the research. Researchers may not have access to certain data or resources, or may be unable to conduct long-term studies due to practical limitations.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Paper Format – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 9. The Conclusion
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The conclusion is intended to help the reader understand why your research should matter to them after they have finished reading the paper. A conclusion is not merely a summary of the main topics covered or a re-statement of your research problem, but a synthesis of key points derived from the findings of your study and, if applicable, where you recommend new areas for future research. For most college-level research papers, two or three well-developed paragraphs is sufficient for a conclusion, although in some cases, more paragraphs may be required in describing the key findings and their significance.

Conclusions. The Writing Center. University of North Carolina; Conclusions. The Writing Lab and The OWL. Purdue University.

Importance of a Good Conclusion

A well-written conclusion provides you with important opportunities to demonstrate to the reader your understanding of the research problem. These include:

• Presenting the last word on the issues you raised in your paper . Just as the introduction gives a first impression to your reader, the conclusion offers a chance to leave a lasting impression. Do this, for example, by highlighting key findings in your analysis that advance new understanding about the research problem, that are unusual or unexpected, or that have important implications applied to practice.
• Summarizing your thoughts and conveying the larger significance of your study . The conclusion is an opportunity to succinctly re-emphasize  your answer to the "So What?" question by placing the study within the context of how your research advances past research about the topic.
• Identifying how a gap in the literature has been addressed . The conclusion can be where you describe how a previously identified gap in the literature [first identified in your literature review section] has been addressed by your research and why this contribution is significant.
• Demonstrating the importance of your ideas . Don't be shy. The conclusion offers an opportunity to elaborate on the impact and significance of your findings. This is particularly important if your study approached examining the research problem from an unusual or innovative perspective.
• Introducing possible new or expanded ways of thinking about the research problem . This does not refer to introducing new information [which should be avoided], but to offer new insight and creative approaches for framing or contextualizing the research problem based on the results of your study.

Bunton, David. “The Structure of PhD Conclusion Chapters.” Journal of English for Academic Purposes 4 (July 2005): 207–224; Conclusions. The Writing Center. University of North Carolina; Kretchmer, Paul. Twelve Steps to Writing an Effective Conclusion. San Francisco Edit, 2003-2008; Conclusions. The Writing Lab and The OWL. Purdue University; Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8.

Structure and Writing Style

I.  General Rules

The general function of your paper's conclusion is to restate the main argument . It reminds the reader of the strengths of your main argument(s) and reiterates the most important evidence supporting those argument(s). Do this by clearly summarizing the context, background, and necessity of pursuing the research problem you investigated in relation to an issue, controversy, or a gap found in the literature. However, make sure that your conclusion is not simply a repetitive summary of the findings. This reduces the impact of the argument(s) you have developed in your paper.

When writing the conclusion to your paper, follow these general rules:

• Present your conclusions in clear, concise language. Re-state the purpose of your study, then describe how your findings differ or support those of other studies and why [i.e., what were the unique, new, or crucial contributions your study made to the overall research about your topic?].
• Do not simply reiterate your findings or the discussion of your results. Provide a synthesis of arguments presented in the paper to show how these converge to address the research problem and the overall objectives of your study.
• Indicate opportunities for future research if you haven't already done so in the discussion section of your paper. Highlighting the need for further research provides the reader with evidence that you have an in-depth awareness of the research problem but that further investigations should take place beyond the scope of your investigation.

Consider the following points to help ensure your conclusion is presented well:

• If the argument or purpose of your paper is complex, you may need to summarize the argument for your reader.
• If, prior to your conclusion, you have not yet explained the significance of your findings or if you are proceeding inductively, use the end of your paper to describe your main points and explain their significance.
• Move from a detailed to a general level of consideration that returns the topic to the context provided by the introduction or within a new context that emerges from the data [this is opposite of the introduction, which begins with general discussion of the context and ends with a detailed description of the research problem]. 

The conclusion also provides a place for you to persuasively and succinctly restate the research problem, given that the reader has now been presented with all the information about the topic . Depending on the discipline you are writing in, the concluding paragraph may contain your reflections on the evidence presented. However, the nature of being introspective about the research you have conducted will depend on the topic and whether your professor wants you to express your observations in this way. If asked to think introspectively about the topics, do not delve into idle speculation. Being introspective means looking within yourself as an author to try and understand an issue more deeply, not to guess at possible outcomes or make up scenarios not supported by the evidence.

II.  Developing a Compelling Conclusion

Although an effective conclusion needs to be clear and succinct, it does not need to be written passively or lack a compelling narrative. Strategies to help you move beyond merely summarizing the key points of your research paper may include any of the following:

• If your essay deals with a critical, contemporary problem, warn readers of the possible consequences of not attending to the problem proactively.
• Recommend a specific course or courses of action that, if adopted, could address a specific problem in practice or in the development of new knowledge leading to positive change.
• Cite a relevant quotation or expert opinion already noted in your paper in order to lend authority and support to the conclusion(s) you have reached [a good source would be from your literature review].
• Explain the consequences of your research in a way that elicits action or demonstrates urgency in seeking change.
• Restate a key statistic, fact, or visual image to emphasize the most important finding of your paper.
• If your discipline encourages personal reflection, illustrate your concluding point by drawing from your own life experiences.
• Return to an anecdote, an example, or a quotation that you presented in your introduction, but add further insight derived from the findings of your study; use your interpretation of results from your study to recast it in new or important ways.
• Provide a "take-home" message in the form of a succinct, declarative statement that you want the reader to remember about your study.

III. Problems to Avoid

Failure to be concise Your conclusion section should be concise and to the point. Conclusions that are too lengthy often have unnecessary information in them. The conclusion is not the place for details about your methodology or results. Although you should give a summary of what was learned from your research, this summary should be relatively brief, since the emphasis in the conclusion is on the implications, evaluations, insights, and other forms of analysis that you make. Strategies for writing concisely can be found here .

Failure to comment on larger, more significant issues In the introduction, your task was to move from the general [the field of study] to the specific [the research problem]. However, in the conclusion, your task is to move from a specific discussion [your research problem] back to a general discussion framed around the implications and significance of your findings [i.e., how your research contributes new understanding or fills an important gap in the literature]. In short, the conclusion is where you should place your research within a larger context [visualize your paper as an hourglass--start with a broad introduction and review of the literature, move to the specific analysis and discussion, conclude with a broad summary of the study's implications and significance].

Failure to reveal problems and negative results Negative aspects of the research process should never be ignored. These are problems, deficiencies, or challenges encountered during your study. They should be summarized as a way of qualifying your overall conclusions. If you encountered negative or unintended results [i.e., findings that are validated outside the research context in which they were generated], you must report them in the results section and discuss their implications in the discussion section of your paper. In the conclusion, use negative results as an opportunity to explain their possible significance and/or how they may form the basis for future research.

Failure to provide a clear summary of what was learned In order to be able to discuss how your research fits within your field of study [and possibly the world at large], you need to summarize briefly and succinctly how it contributes to new knowledge or a new understanding about the research problem. This element of your conclusion may be only a few sentences long.

Failure to match the objectives of your research Often research objectives in the social and behavioral sciences change while the research is being carried out. This is not a problem unless you forget to go back and refine the original objectives in your introduction. As these changes emerge they must be documented so that they accurately reflect what you were trying to accomplish in your research [not what you thought you might accomplish when you began].

Resist the urge to apologize If you've immersed yourself in studying the research problem, you presumably should know a good deal about it [perhaps even more than your professor!]. Nevertheless, by the time you have finished writing, you may be having some doubts about what you have produced. Repress those doubts! Don't undermine your authority as a researcher by saying something like, "This is just one approach to examining this problem; there may be other, much better approaches that...." The overall tone of your conclusion should convey confidence to the reader about the study's validity and realiability.

Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8; Concluding Paragraphs. College Writing Center at Meramec. St. Louis Community College; Conclusions. The Writing Center. University of North Carolina; Conclusions. The Writing Lab and The OWL. Purdue University; Freedman, Leora  and Jerry Plotnick. Introductions and Conclusions. The Lab Report. University College Writing Centre. University of Toronto; Leibensperger, Summer. Draft Your Conclusion. Academic Center, the University of Houston-Victoria, 2003; Make Your Last Words Count. The Writer’s Handbook. Writing Center. University of Wisconsin Madison; Miquel, Fuster-Marquez and Carmen Gregori-Signes. “Chapter Six: ‘Last but Not Least:’ Writing the Conclusion of Your Paper.” In Writing an Applied Linguistics Thesis or Dissertation: A Guide to Presenting Empirical Research . John Bitchener, editor. (Basingstoke,UK: Palgrave Macmillan, 2010), pp. 93-105; Tips for Writing a Good Conclusion. Writing@CSU. Colorado State University; Kretchmer, Paul. Twelve Steps to Writing an Effective Conclusion. San Francisco Edit, 2003-2008; Writing Conclusions. Writing Tutorial Services, Center for Innovative Teaching and Learning. Indiana University; Writing: Considering Structure and Organization. Institute for Writing Rhetoric. Dartmouth College.

Writing Tip

Don't Belabor the Obvious!

Avoid phrases like "in conclusion...," "in summary...," or "in closing...." These phrases can be useful, even welcome, in oral presentations. But readers can see by the tell-tale section heading and number of pages remaining that they are reaching the end of your paper. You'll irritate your readers if you belabor the obvious.

Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8.

Another Writing Tip

New Insight, Not New Information!

Don't surprise the reader with new information in your conclusion that was never referenced anywhere else in the paper. This why the conclusion rarely has citations to sources. If you have new information to present, add it to the discussion or other appropriate section of the paper. Note that, although no new information is introduced, the conclusion, along with the discussion section, is where you offer your most "original" contributions in the paper; the conclusion is where you describe the value of your research, demonstrate that you understand the material that you’ve presented, and position your findings within the larger context of scholarship on the topic, including describing how your research contributes new insights to that scholarship.

Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8; Conclusions. The Writing Center. University of North Carolina.

• << Previous: Limitations of the Study
• Next: Appendices >>
• Last Updated: May 22, 2024 12:03 PM
• URL: https://libguides.usc.edu/writingguide

• Open access
• Published: 15 May 2024

Learning together for better health using an evidence-based Learning Health System framework: a case study in stroke

• Helena Teede 1 , 2   na1 ,
• Dominique A. Cadilhac 3 , 4   na1 ,
• Tara Purvis 3 ,
• Monique F. Kilkenny 3 , 4 ,
• Bruce C.V. Campbell 4 , 5 , 6 ,
• Coralie English 7 ,
• Alison Johnson 2 ,
• Emily Callander 1 ,
• Rohan S. Grimley 8 , 9 ,
• Christopher Levi 10 ,
• Sandy Middleton 11 , 12 ,
• Kelvin Hill 13 &
• Joanne Enticott   ORCID: orcid.org/0000-0002-4480-5690 1  

BMC Medicine volume  22 , Article number:  198 ( 2024 ) Cite this article

349 Accesses

2 Altmetric

Metrics details

In the context of expanding digital health tools, the health system is ready for Learning Health System (LHS) models. These models, with proper governance and stakeholder engagement, enable the integration of digital infrastructure to provide feedback to all relevant parties including clinicians and consumers on performance against best practice standards, as well as fostering innovation and aligning healthcare with patient needs. The LHS literature primarily includes opinion or consensus-based frameworks and lacks validation or evidence of benefit. Our aim was to outline a rigorously codesigned, evidence-based LHS framework and present a national case study of an LHS-aligned national stroke program that has delivered clinical benefit.

Current core components of a LHS involve capturing evidence from communities and stakeholders (quadrant 1), integrating evidence from research findings (quadrant 2), leveraging evidence from data and practice (quadrant 3), and generating evidence from implementation (quadrant 4) for iterative system-level improvement. The Australian Stroke program was selected as the case study as it provides an exemplar of how an iterative LHS works in practice at a national level encompassing and integrating evidence from all four LHS quadrants. Using this case study, we demonstrate how to apply evidence-based processes to healthcare improvement and embed real-world research for optimising healthcare improvement. We emphasize the transition from research as an endpoint, to research as an enabler and a solution for impact in healthcare improvement.

Conclusions

The Australian Stroke program has nationally improved stroke care since 2007, showcasing the value of integrated LHS-aligned approaches for tangible impact on outcomes. This LHS case study is a practical example for other health conditions and settings to follow suit.

Peer Review reports

Internationally, health systems are facing a crisis, driven by an ageing population, increasing complexity, multi-morbidity, rapidly advancing health technology and rising costs that threaten sustainability and mandate transformation and improvement [ 1 , 2 ]. Although research has generated solutions to healthcare challenges, and the advent of big data and digital health holds great promise, entrenched siloes and poor integration of knowledge generation, knowledge implementation and healthcare delivery between stakeholders, curtails momentum towards, and consistent attainment of, evidence-and value-based care [ 3 ]. This is compounded by the short supply of research and innovation leadership within the healthcare sector, and poorly integrated and often inaccessible health data systems, which have crippled the potential to deliver on digital-driven innovation [ 4 ]. Current approaches to healthcare improvement are also often isolated with limited sustainability, scale-up and impact [ 5 ].

Evidence suggests that integration and partnership across academic and healthcare delivery stakeholders are key to progress, including those with lived experience and their families (referred to here as consumers and community), diverse disciplines (both research and clinical), policy makers and funders. Utilization of evidence from research and evidence from practice including data from routine care, supported by implementation research, are key to sustainably embedding improvement and optimising health care and outcomes. A strategy to achieve this integration is through the Learning Health System (LHS) (Fig.  1 ) [ 2 , 6 , 7 , 8 ]. Although there are numerous publications on LHS approaches [ 9 , 10 , 11 , 12 ], many focus on research perspectives and data, most do not demonstrate tangible healthcare improvement or better health outcomes. [ 6 ]

Monash Learning Health System: The Learn Together for Better Health Framework developed by Monash Partners and Monash University (from Enticott et al. 2021 [ 7 ]). Four evidence quadrants: Q1 (orange) is evidence from stakeholders; Q2 (green) is evidence from research; Q3 (light blue) is evidence from data; and, Q4 (dark blue) is evidence from implementation and healthcare improvement

In developed nations, it has been estimated that 60% of care provided aligns with the evidence base, 30% is low value and 10% is potentially harmful [ 13 ]. In some areas, clinical advances have been rapid and research and evidence have paved the way for dramatic improvement in outcomes, mandating rapid implementation of evidence into healthcare (e.g. polio and COVID-19 vaccines). However, healthcare improvement is challenging and slow [ 5 ]. Health systems are highly complex in their design, networks and interacting components, and change is difficult to enact, sustain and scale up. [ 3 ] New effective strategies are needed to meet community needs and deliver evidence-based and value-based care, which reorients care from serving the provider, services and system, towards serving community needs, based on evidence and quality. It goes beyond cost to encompass patient and provider experience, quality care and outcomes, efficiency and sustainability [ 2 , 6 ].

The costs of stroke care are expected to rise rapidly in the next decades, unless improvements in stroke care to reduce the disabling effects of strokes can be successfully developed and implemented [ 14 ]. Here, we briefly describe the Monash LHS framework (Fig.  1 ) [ 2 , 6 , 7 ] and outline an exemplar case in order to demonstrate how to apply evidence-based processes to healthcare improvement and embed real-world research for optimising healthcare. The Australian LHS exemplar in stroke care has driven nationwide improvement in stroke care since 2007.

An evidence-based Learning Health System framework

In Australia, members of this author group (HT, AJ, JE) have rigorously co-developed an evidence-based LHS framework, known simply as the Monash LHS [ 7 ]. The Monash LHS was designed to support sustainable, iterative and continuous robust benefit of improved clinical outcomes. It was created with national engagement in order to be applicable to Australian settings. Through this rigorous approach, core LHS principles and components have been established (Fig.  1 ). Evidence shows that people/workforce, culture, standards, governance and resources were all key to an effective LHS [ 2 , 6 ]. Culture is vital including trust, transparency, partnership and co-design. Key processes include legally compliant data sharing, linkage and governance, resources, and infrastructure [ 4 ]. The Monash LHS integrates disparate and often siloed stakeholders, infrastructure and expertise to ‘Learn Together for Better Health’ [ 7 ] (Fig.  1 ). This integrates (i) evidence from community and stakeholders including priority areas and outcomes; (ii) evidence from research and guidelines; (iii) evidence from practice (from data) with advanced analytics and benchmarking; and (iv) evidence from implementation science and health economics. Importantly, it starts with the problem and priorities of key stakeholders including the community, health professionals and services and creates an iterative learning system to address these. The following case study was chosen as it is an exemplar of how a Monash LHS-aligned national stroke program has delivered clinical benefit.

Australian Stroke Learning Health System

Internationally, the application of LHS approaches in stroke has resulted in improved stroke care and outcomes [ 12 ]. For example, in Canada a sustained decrease in 30-day in-hospital mortality has been found commensurate with an increase in resources to establish the multifactorial stroke system intervention for stroke treatment and prevention [ 15 ]. Arguably, with rapid advances in evidence and in the context of an ageing population with high cost and care burden and substantive impacts on quality of life, stroke is an area with a need for rapid research translation into evidence-based and value-based healthcare improvement. However, a recent systematic review found that the existing literature had few comprehensive examples of LHS adoption [ 12 ]. Although healthcare improvement systems and approaches were described, less is known about patient-clinician and stakeholder engagement, governance and culture, or embedding of data informatics into everyday practice to inform and drive improvement [ 12 ]. For example, in a recent review of quality improvement collaborations, it was found that although clinical processes in stroke care are improved, their short-term nature means there is uncertainty about sustainability and impacts on patient outcomes [ 16 ]. Table  1 provides the main features of the Australian Stroke LHS based on the four core domains and eight elements of the Learning Together for Better Health Framework described in Fig.  1 . The features are further expanded on in the following sections.

Evidence from stakeholders (LHS quadrant 1, Fig.  1 )

Engagement, partners and priorities.

Within the stroke field, there have been various support mechanisms to facilitate an LHS approach including partnership and broad stakeholder engagement that includes clinical networks and policy makers from different jurisdictions. Since 2008, the Australian Stroke Coalition has been co-led by the Stroke Foundation, a charitable consumer advocacy organisation, and Stroke Society of Australasia a professional society with membership covering academics and multidisciplinary clinician networks, that are collectively working to improve stroke care ( https://australianstrokecoalition.org.au/ ). Surveys, focus groups and workshops have been used for identifying priorities from stakeholders. Recent agreed priorities have been to improve stroke care and strengthen the voice for stroke care at a national ( https://strokefoundation.org.au/ ) and international level ( https://www.world-stroke.org/news-and-blog/news/world-stroke-organization-tackle-gaps-in-access-to-quality-stroke-care ), as well as reduce duplication amongst stakeholders. This activity is built on a foundation and culture of research and innovation embedded within the stroke ‘community of practice’. Consumers, as people with lived experience of stroke are important members of the Australian Stroke Coalition, as well as representatives from different clinical colleges. Consumers also provide critical input to a range of LHS activities via the Stroke Foundation Consumer Council, Stroke Living Guidelines committees, and the Australian Stroke Clinical Registry (AuSCR) Steering Committee (described below).

Evidence from research (LHS quadrant 2, Fig.  1 )

Advancement of the evidence for stroke interventions and synthesis into clinical guidelines.

To implement best practice, it is crucial to distil the large volume of scientific and trial literature into actionable recommendations for clinicians to use in practice [ 24 ]. The first Australian clinical guidelines for acute stroke were produced in 2003 following the increasing evidence emerging for prevention interventions (e.g. carotid endarterectomy, blood pressure lowering), acute medical treatments (intravenous thrombolysis, aspirin within 48 h of ischemic stroke), and optimised hospital management (care in dedicated stroke units by a specialised and coordinated multidisciplinary team) [ 25 ]. Importantly, a number of the innovations were developed, researched and proven effective by key opinion leaders embedded in the Australian stroke care community. In 2005, the clinical guidelines for Stroke Rehabilitation and Recovery [ 26 ] were produced, with subsequent merged guidelines periodically updated. However, the traditional process of periodic guideline updates is challenging for end users when new research can render recommendations redundant and this lack of currency erodes stakeholder trust [ 27 ]. In response to this challenge the Stroke Foundation and Cochrane Australia entered a pioneering project to produce the first electronic ‘living’ guidelines globally [ 20 ]. Major shifts in the evidence for reperfusion therapies (e.g. extended time-window intravenous thrombolysis and endovascular clot retrieval), among other advances, were able to be converted into new recommendations, approved by the Australian National Health and Medical Research Council within a few months of publication. Feedback on this process confirmed the increased use and trust in the guidelines by clinicians. The process informed other living guidelines programs, including the successful COVID-19 clinical guidelines [ 28 ].

However, best practice clinical guideline recommendations are necessary but insufficient for healthcare improvement and nesting these within an LHS with stakeholder partnership, enables implementation via a range of proven methods, including audit and feedback strategies [ 29 ].

Evidence from data and practice (LHS quadrant 3, Fig.  1 )

Data systems and benchmarking : revealing the disparities in care between health services. A national system for standardized stroke data collection was established as the National Stroke Audit program in 2007 by the Stroke Foundation [ 30 ] following various state-level programs (e.g. New South Wales Audit) [ 31 ] to identify evidence-practice gaps and prioritise improvement efforts to increase access to stroke units and other acute treatments [ 32 ]. The Audit program alternates each year between acute (commencing in 2007) and rehabilitation in-patient services (commencing in 2008). The Audit program provides a ‘deep dive’ on the majority of recommendations in the clinical guidelines whereby participating hospitals provide audits of up to 40 consecutive patient medical records and respond to a survey about organizational resources to manage stroke. In 2009, the AuSCR was established to provide information on patients managed in acute hospitals based on a small subset of quality processes of care linked to benchmarked reports of performance (Fig.  2 ) [ 33 ]. In this way, the continuous collection of high-priority processes of stroke care could be regularly collected and reviewed to guide improvement to care [ 34 ]. Plus clinical quality registry programs within Australia have shown a meaningful return on investment attributed to enhanced survival, improvements in quality of life and avoided costs of treatment or hospital stay [ 35 ].

Example performance report from the Australian Stroke Clinical Registry: average door-to-needle time in providing intravenous thrombolysis by different hospitals in 2021 [ 36 ]. Each bar in the figure represents a single hospital

The Australian Stroke Coalition endorsed the creation of an integrated technological solution for collecting data through a single portal for multiple programs in 2013. In 2015, the Stroke Foundation, AuSCR consortium, and other relevant groups cooperated to design an integrated data management platform (the Australian Stroke Data Tool) to reduce duplication of effort for hospital staff in the collection of overlapping variables in the same patients [ 19 ]. Importantly, a national data dictionary then provided the common data definitions to facilitate standardized data capture. Another important feature of AuSCR is the collection of patient-reported outcome surveys between 90 and 180 days after stroke, and annual linkage with national death records to ascertain survival status [ 33 ]. To support a LHS approach, hospitals that participate in AuSCR have access to a range of real-time performance reports. In efforts to minimize the burden of data collection in the AuSCR, interoperability approaches to import data directly from hospital or state-level managed stroke databases have been established (Fig.  3 ); however, the application has been variable and 41% of hospitals still manually enter all their data.

Current status of automated data importing solutions in the Australian Stroke Clinical Registry, 2022, with ‘ n ’ representing the number of hospitals. AuSCR, Australian Stroke Clinical Registry; AuSDaT, Australian Stroke Data Tool; API, Application Programming Interface; ICD, International Classification of Diseases; RedCAP, Research Electronic Data Capture; eMR, electronic medical records

For acute stroke care, the Australian Commission on Quality and Safety in Health Care facilitated the co-design (clinicians, academics, consumers) and publication of the national Acute Stroke Clinical Care Standard in 2015 [ 17 ], and subsequent review [ 18 ]. The indicator set for the Acute Stroke Standard then informed the expansion of the minimum dataset for AuSCR so that hospitals could routinely track their performance. The national Audit program enabled hospitals not involved in the AuSCR to assess their performance every two years against the Acute Stroke Standard. Complementing these efforts, the Stroke Foundation, working with the sector, developed the Acute and Rehabilitation Stroke Services Frameworks to outline the principles, essential elements, models of care and staffing recommendations for stroke services ( https://informme.org.au/guidelines/national-stroke-services-frameworks ). The Frameworks are intended to guide where stroke services should be developed, and monitor their uptake with the organizational survey component of the Audit program.

Evidence from implementation and healthcare improvement (LHS quadrant 4, Fig.  1 )

Research to better utilize and augment data from registries through linkage [ 37 , 38 , 39 , 40 ] and to ensure presentation of hospital or service level data are understood by clinicians has ensured advancement in the field for the Australian Stroke LHS [ 41 ]. Importantly, greater insights into whole patient journeys, before and after a stroke, can now enable exploration of value-based care. The LHS and stroke data platform have enabled focused and time-limited projects to create a better understanding of the quality of care in acute or rehabilitation settings [ 22 , 42 , 43 ]. Within stroke, all the elements of an LHS culminate into the ready availability of benchmarked performance data and support for implementation of strategies to address gaps in care.

Implementation research to grow the evidence base for effective improvement interventions has also been a key pillar in the Australian context. These include multi-component implementation interventions to achieve behaviour change for particular aspects of stroke care, [ 22 , 23 , 44 , 45 ] and real-world approaches to augmenting access to hyperacute interventions in stroke through the use of technology and telehealth [ 46 , 47 , 48 , 49 ]. The evidence from these studies feeds into the living guidelines program and the data collection systems, such as the Audit program or AuSCR, which are then amended to ensure data aligns to recommended care. For example, the use of ‘hyperacute aspirin within the first 48 h of ischemic stroke’ was modified to be ‘hyperacute antiplatelet…’ to incorporate new evidence that other medications or combinations are appropriate to use. Additionally, new datasets have been developed to align with evidence such as the Fever, Sugar, and Swallow variables [ 42 ]. Evidence on improvements in access to best practice care from the acute Audit program [ 50 ] and AuSCR is emerging [ 36 ]. For example, between 2007 and 2017, the odds of receiving intravenous thrombolysis after ischemic stroke increased by 16% 9OR 1.06 95% CI 1.13–1.18) and being managed in a stroke unit by 18% (OR 1.18 95% CI 1.17–1.20). Over this period, the median length of hospital stay for all patients decreased from 6.3 days in 2007 to 5.0 days in 2017 [ 51 ]. When considering the number of additional patients who would receive treatment in 2017 in comparison to 2007 it was estimated that without this additional treatment, over 17,000 healthy years of life would be lost in 2017 (17,786 disability-adjusted life years) [ 51 ]. There is evidence on the cost-effectiveness of different system-focussed strategies to augment treatment access for acute ischemic stroke (e.g. Victorian Stroke Telemedicine program [ 52 ] and Melbourne Mobile Stroke Unit ambulance [ 53 ]). Reciprocally, evidence from the national Rehabilitation Audit, where the LHS approach has been less complete or embedded, has shown fewer areas of healthcare improvement over time [ 51 , 54 ].

Within the field of stroke in Australia, there is indirect evidence that the collective efforts that align to establishing the components of a LHS have had an impact. Overall, the age-standardised rate of stroke events has reduced by 27% between 2001 and 2020, from 169 to 124 events per 100,000 population. Substantial declines in mortality rates have been reported since 1980. Commensurate with national clinical guidelines being updated in 2007 and the first National Stroke Audit being undertaken in 2007, the mortality rates for men (37.4 deaths per 100,000) and women (36.1 deaths per 100,0000 has declined to 23.8 and 23.9 per 100,000, respectively in 2021 [ 55 ].

Underpinning the LHS with the integration of the four quadrants of evidence from stakeholders, research and guidelines, practice and implementation, and core LHS principles have been addressed. Leadership and governance have been important, and programs have been established to augment workforce training and capacity building in best practice professional development. Medical practitioners are able to undertake courses and mentoring through the Australasian Stroke Academy ( http://www.strokeacademy.com.au/ ) while nurses (and other health professionals) can access teaching modules in stroke care from the Acute Stroke Nurses Education Network ( https://asnen.org/ ). The Association of Neurovascular Clinicians offers distance-accessible education and certification to develop stroke expertise for interdisciplinary professionals, including advanced stroke co-ordinator certification ( www.anvc.org ). Consumer initiative interventions are also used in the design of the AuSCR Public Summary Annual reports (available at https://auscr.com.au/about/annual-reports/ ) and consumer-related resources related to the Living Guidelines ( https://enableme.org.au/resources ).

The important success factors and lessons from stroke as a national exemplar LHS in Australia include leadership, culture, workforce and resources integrated with (1) established and broad partnerships across the academic-clinical sector divide and stakeholder engagement; (2) the living guidelines program; (3) national data infrastructure, including a national data dictionary that provides the common data framework to support standardized data capture; (4) various implementation strategies including benchmarking and feedback as well as engagement strategies targeting different levels of the health system; and (5) implementation and improvement research to advance stroke systems of care and reduce unwarranted variation in practice (Fig.  1 ). Priority opportunities now include the advancement of interoperability with electronic medical records as an area all clinical quality registry’s programs needs to be addressed, as well as providing more dynamic and interactive data dashboards tailored to the need of clinicians and health service executives.

There is a clear mandate to optimise healthcare improvement with big data offering major opportunities for change. However, we have lacked the approaches to capture evidence from the community and stakeholders, to integrate evidence from research, to capture and leverage data or evidence from practice and to generate and build on evidence from implementation using iterative system-level improvement. The LHS provides this opportunity and is shown to deliver impact. Here, we have outlined the process applied to generate an evidence-based LHS and provide a leading exemplar in stroke care. This highlights the value of moving from single-focus isolated approaches/initiatives to healthcare improvement and the benefit of integration to deliver demonstrable outcomes for our funders and key stakeholders — our community. This work provides insight into strategies that can both apply evidence-based processes to healthcare improvement as well as implementing evidence-based practices into care, moving beyond research as an endpoint, to research as an enabler, underpinning delivery of better healthcare.

Availability of data and materials

Not applicable

Abbreviations

Australian Stroke Clinical Registry

Confidence interval

• Learning Health System

World Health Organization. Delivering quality health services . OECD Publishing; 2018.

Enticott J, Braaf S, Johnson A, Jones A, Teede HJ. Leaders’ perspectives on learning health systems: A qualitative study. BMC Health Serv Res. 2020;20:1087.

Article   PubMed   PubMed Central   Google Scholar  

Melder A, Robinson T, McLoughlin I, Iedema R, Teede H. An overview of healthcare improvement: Unpacking the complexity for clinicians and managers in a learning health system. Intern Med J. 2020;50:1174–84.

Article   PubMed   Google Scholar  

Alberto IRI, Alberto NRI, Ghosh AK, Jain B, Jayakumar S, Martinez-Martin N, et al. The impact of commercial health datasets on medical research and health-care algorithms. Lancet Digit Health. 2023;5:e288–94.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Dixon-Woods M. How to improve healthcare improvement—an essay by Mary Dixon-Woods. BMJ. 2019;367: l5514.

Enticott J, Johnson A, Teede H. Learning health systems using data to drive healthcare improvement and impact: A systematic review. BMC Health Serv Res. 2021;21:200.

Enticott JC, Melder A, Johnson A, Jones A, Shaw T, Keech W, et al. A learning health system framework to operationalize health data to improve quality care: An Australian perspective. Front Med (Lausanne). 2021;8:730021.

Dammery G, Ellis LA, Churruca K, Mahadeva J, Lopez F, Carrigan A, et al. The journey to a learning health system in primary care: A qualitative case study utilising an embedded research approach. BMC Prim Care. 2023;24:22.

Foley T, Horwitz L, Zahran R. The learning healthcare project: Realising the potential of learning health systems. 2021. Available from https://learninghealthcareproject.org/wp-content/uploads/2021/05/LHS2021report.pdf . Accessed Jan 2024.

Institute of Medicine. Best care at lower cost: The path to continuously learning health care in America. Washington: The National Academies Press; 2013.

Google Scholar  

Zurynski Y, Smith CL, Vedovi A, Ellis LA, Knaggs G, Meulenbroeks I, et al. Mapping the learning health system: A scoping review of current evidence - a white paper. 2020:63

Cadilhac DA, Bravata DM, Bettger J, Mikulik R, Norrving B, Uvere E, et al. Stroke learning health systems: A topical narrative review with case examples. Stroke. 2023;54:1148–59.

Braithwaite J, Glasziou P, Westbrook J. The three numbers you need to know about healthcare: The 60–30-10 challenge. BMC Med. 2020;18:1–8.

Article   Google Scholar  

King D, Wittenberg R, Patel A, Quayyum Z, Berdunov V, Knapp M. The future incidence, prevalence and costs of stroke in the UK. Age Ageing. 2020;49:277–82.

Ganesh A, Lindsay P, Fang J, Kapral MK, Cote R, Joiner I, et al. Integrated systems of stroke care and reduction in 30-day mortality: A retrospective analysis. Neurology. 2016;86:898–904.

Lowther HJ, Harrison J, Hill JE, Gaskins NJ, Lazo KC, Clegg AJ, et al. The effectiveness of quality improvement collaboratives in improving stroke care and the facilitators and barriers to their implementation: A systematic review. Implement Sci. 2021;16:16.

Australian Commission on Safety and Quality in Health Care. Acute stroke clinical care standard. 2015. Available from https://www.safetyandquality.gov.au/our-work/clinical-care-standards/acute-stroke-clinical-care-standard . Accessed Jan 2024.

Australian Commission on Safety and Quality in Health Care. Acute stroke clinical care standard. Sydney: ACSQHC; 2019. Available from https://www.safetyandquality.gov.au/publications-and-resources/resource-library/acute-stroke-clinical-care-standard-evidence-sources . Accessed Jan 2024.

Ryan O, Ghuliani J, Grabsch B, Hill K, G CC, Breen S, et al. Development, implementation, and evaluation of the Australian Stroke Data Tool (AuSDaT): Comprehensive data capturing for multiple uses. Health Inf Manag. 2022:18333583221117184.

English C, Bayley M, Hill K, Langhorne P, Molag M, Ranta A, et al. Bringing stroke clinical guidelines to life. Int J Stroke. 2019;14:337–9.

English C, Hill K, Cadilhac DA, Hackett ML, Lannin NA, Middleton S, et al. Living clinical guidelines for stroke: Updates, challenges and opportunities. Med J Aust. 2022;216:510–4.

Cadilhac DA, Grimley R, Kilkenny MF, Andrew NE, Lannin NA, Hill K, et al. Multicenter, prospective, controlled, before-and-after, quality improvement study (Stroke123) of acute stroke care. Stroke. 2019;50:1525–30.

Cadilhac DA, Marion V, Andrew NE, Breen SJ, Grabsch B, Purvis T, et al. A stepped-wedge cluster-randomized trial to improve adherence to evidence-based practices for acute stroke management. Jt Comm J Qual Patient Saf. 2022.

Elliott J, Lawrence R, Minx JC, Oladapo OT, Ravaud P, Jeppesen BT, et al. Decision makers need constantly updated evidence synthesis. Nature. 2021;600:383–5.

Article   CAS   PubMed   Google Scholar  

National Stroke Foundation. National guidelines for acute stroke management. Melbourne: National Stroke Foundation; 2003.

National Stroke Foundation. Clinical guidelines for stroke rehabilitation and recovery. Melbourne: National Stroke Foundation; 2005.

Phan TG, Thrift A, Cadilhac D, Srikanth V. A plea for the use of systematic review methodology when writing guidelines and timely publication of guidelines. Intern Med J . 2012;42:1369–1371; author reply 1371–1362

Tendal B, Vogel JP, McDonald S, Norris S, Cumpston M, White H, et al. Weekly updates of national living evidence-based guidelines: Methods for the Australian living guidelines for care of people with COVID-19. J Clin Epidemiol. 2021;131:11–21.

Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:50.

Harris D, Cadilhac D, Hankey GJ, Hillier S, Kilkenny M, Lalor E. National stroke audit: The Australian experience. Clin Audit. 2010;2:25–31.

Cadilhac DA, Purvis T, Kilkenny MF, Longworth M, Mohr K, Pollack M, et al. Evaluation of rural stroke services: Does implementation of coordinators and pathways improve care in rural hospitals? Stroke. 2013;44:2848–53.

Cadilhac DA, Moss KM, Price CJ, Lannin NA, Lim JY, Anderson CS. Pathways to enhancing the quality of stroke care through national data monitoring systems for hospitals. Med J Aust. 2013;199:650–1.

Cadilhac DA, Lannin NA, Anderson CS, Levi CR, Faux S, Price C, et al. Protocol and pilot data for establishing the Australian Stroke Clinical Registry. Int J Stroke. 2010;5:217–26.

Ivers N, Jamtvedt G, Flottorp S, Young J, Odgaard-Jensen J, French S, et al. Audit and feedback: Effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev . 2012

Australian Commission on Safety and Quality in Health Care. Economic evaluation of clinical quality registries. Final report. . 2016:79

Cadilhac DA, Dalli LL, Morrison J, Lester M, Paice K, Moss K, et al. The Australian Stroke Clinical Registry annual report 2021. Melbourne; 2022. Available from https://auscr.com.au/about/annual-reports/ . Accessed 6 May 2024.

Kilkenny MF, Kim J, Andrew NE, Sundararajan V, Thrift AG, Katzenellenbogen JM, et al. Maximising data value and avoiding data waste: A validation study in stroke research. Med J Aust. 2019;210:27–31.

Eliakundu AL, Smith K, Kilkenny MF, Kim J, Bagot KL, Andrew E, et al. Linking data from the Australian Stroke Clinical Registry with ambulance and emergency administrative data in Victoria. Inquiry. 2022;59:469580221102200.

PubMed   Google Scholar  

Andrew NE, Kim J, Cadilhac DA, Sundararajan V, Thrift AG, Churilov L, et al. Protocol for evaluation of enhanced models of primary care in the management of stroke and other chronic disease (PRECISE): A data linkage healthcare evaluation study. Int J Popul Data Sci. 2019;4:1097.

CAS   PubMed   PubMed Central   Google Scholar  

Mosalski S, Shiner CT, Lannin NA, Cadilhac DA, Faux SG, Kim J, et al. Increased relative functional gain and improved stroke outcomes: A linked registry study of the impact of rehabilitation. J Stroke Cerebrovasc Dis. 2021;30: 106015.

Ryan OF, Hancock SL, Marion V, Kelly P, Kilkenny MF, Clissold B, et al. Feedback of aggregate patient-reported outcomes (PROs) data to clinicians and hospital end users: Findings from an Australian codesign workshop process. BMJ Open. 2022;12:e055999.

Grimley RS, Rosbergen IC, Gustafsson L, Horton E, Green T, Cadigan G, et al. Dose and setting of rehabilitation received after stroke in Queensland, Australia: A prospective cohort study. Clin Rehabil. 2020;34:812–23.

Purvis T, Middleton S, Craig LE, Kilkenny MF, Dale S, Hill K, et al. Inclusion of a care bundle for fever, hyperglycaemia and swallow management in a national audit for acute stroke: Evidence of upscale and spread. Implement Sci. 2019;14:87.

Middleton S, McElduff P, Ward J, Grimshaw JM, Dale S, D’Este C, et al. Implementation of evidence-based treatment protocols to manage fever, hyperglycaemia, and swallowing dysfunction in acute stroke (QASC): A cluster randomised controlled trial. Lancet. 2011;378:1699–706.

Middleton S, Dale S, Cheung NW, Cadilhac DA, Grimshaw JM, Levi C, et al. Nurse-initiated acute stroke care in emergency departments. Stroke. 2019:STROKEAHA118020701.

Hood RJ, Maltby S, Keynes A, Kluge MG, Nalivaiko E, Ryan A, et al. Development and pilot implementation of TACTICS VR: A virtual reality-based stroke management workflow training application and training framework. Front Neurol. 2021;12:665808.

Bladin CF, Kim J, Bagot KL, Vu M, Moloczij N, Denisenko S, et al. Improving acute stroke care in regional hospitals: Clinical evaluation of the Victorian Stroke Telemedicine program. Med J Aust. 2020;212:371–7.

Bladin CF, Bagot KL, Vu M, Kim J, Bernard S, Smith K, et al. Real-world, feasibility study to investigate the use of a multidisciplinary app (Pulsara) to improve prehospital communication and timelines for acute stroke/STEMI care. BMJ Open. 2022;12:e052332.

Zhao H, Coote S, Easton D, Langenberg F, Stephenson M, Smith K, et al. Melbourne mobile stroke unit and reperfusion therapy: Greater clinical impact of thrombectomy than thrombolysis. Stroke. 2020;51:922–30.

Purvis T, Cadilhac DA, Hill K, Reyneke M, Olaiya MT, Dalli LL, et al. Twenty years of monitoring acute stroke care in Australia from the national stroke audit program (1999–2019): Achievements and areas of future focus. J Health Serv Res Policy. 2023.

Cadilhac DA, Purvis T, Reyneke M, Dalli LL, Kim J, Kilkenny MF. Evaluation of the national stroke audit program: 20-year report. Melbourne; 2019.

Kim J, Tan E, Gao L, Moodie M, Dewey HM, Bagot KL, et al. Cost-effectiveness of the Victorian Stroke Telemedicine program. Aust Health Rev. 2022;46:294–301.

Kim J, Easton D, Zhao H, Coote S, Sookram G, Smith K, et al. Economic evaluation of the Melbourne mobile stroke unit. Int J Stroke. 2021;16:466–75.

Stroke Foundation. National stroke audit – rehabilitation services report 2020. Melbourne; 2020.

Australian Institute of Health and Welfare. Heart, stroke and vascular disease: Australian facts. 2023. Webpage https://www.aihw.gov.au/reports/heart-stroke-vascular-diseases/hsvd-facts/contents/about (accessed Jan 2024).

Download references

Acknowledgements

The following authors hold National Health and Medical Research Council Research Fellowships: HT (#2009326), DAC (#1154273), SM (#1196352), MFK Future Leader Research Fellowship (National Heart Foundation #105737). The Funders of this work did not have any direct role in the design of the study, its execution, analyses, interpretation of the data, or decision to submit results for publication.

Author information

Helena Teede and Dominique A. Cadilhac contributed equally.

Authors and Affiliations

Monash Centre for Health Research and Implementation, 43-51 Kanooka Grove, Clayton, VIC, Australia

Helena Teede, Emily Callander & Joanne Enticott

Monash Partners Academic Health Science Centre, 43-51 Kanooka Grove, Clayton, VIC, Australia

Helena Teede & Alison Johnson

Stroke and Ageing Research, Department of Medicine, School of Clinical Sciences at Monash Health, Monash University, Level 2 Monash University Research, Victorian Heart Hospital, 631 Blackburn Rd, Clayton, VIC, Australia

Dominique A. Cadilhac, Tara Purvis & Monique F. Kilkenny

Stroke Theme, The Florey Institute of Neuroscience and Mental Health, University of Melbourne, Heidelberg, VIC, Australia

Dominique A. Cadilhac, Monique F. Kilkenny & Bruce C.V. Campbell

Department of Neurology, Melbourne Brain Centre, Royal Melbourne Hospital, Parkville, VIC, Australia

Bruce C.V. Campbell

Department of Medicine, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Victoria, Australia

School of Health Sciences, Heart and Stroke Program, University of Newcastle, Hunter Medical Research Institute, University Drive, Callaghan, NSW, Australia

Coralie English

School of Medicine and Dentistry, Griffith University, Birtinya, QLD, Australia

Rohan S. Grimley

Clinical Excellence Division, Queensland Health, Brisbane, Australia

John Hunter Hospital, Hunter New England Local Health District and University of Newcastle, Sydney, NSW, Australia

Christopher Levi

School of Nursing, Midwifery and Paramedicine, Australian Catholic University, Sydney, NSW, Australia

Sandy Middleton

Nursing Research Institute, St Vincent’s Health Network Sydney and and Australian Catholic University, Sydney, NSW, Australia

Stroke Foundation, Level 7, 461 Bourke St, Melbourne, VIC, Australia

Kelvin Hill

You can also search for this author in PubMed   Google Scholar

Contributions

HT: conception, design and initial draft, developed the theoretical formalism for learning health system framework, approved the submitted version. DAC: conception, design and initial draft, provided essential literature and case study examples, approved the submitted version. TP: revised the manuscript critically for important intellectual content, approved the submitted version. MFK: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. BC: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. CE: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. AJ: conception, design and initial draft, developed the theoretical formalism for learning health system framework, approved the submitted version. EC: revised the manuscript critically for important intellectual content, approved the submitted version. RSG: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. CL: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. SM: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. KH: revised the manuscript critically for important intellectual content, provided essential literature and case study examples, approved the submitted version. JE: conception, design and initial draft, developed the theoretical formalism for learning health system framework, approved the submitted version. All authors read and approved the final manuscript.

Authors’ Twitter handles

@HelenaTeede

@DominiqueCad

@Coralie_English

@EmilyCallander

@EnticottJo

Corresponding authors

Correspondence to Helena Teede or Dominique A. Cadilhac .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Teede, H., Cadilhac, D.A., Purvis, T. et al. Learning together for better health using an evidence-based Learning Health System framework: a case study in stroke. BMC Med 22 , 198 (2024). https://doi.org/10.1186/s12916-024-03416-w

Download citation

Received : 23 July 2023

Accepted : 30 April 2024

Published : 15 May 2024

DOI : https://doi.org/10.1186/s12916-024-03416-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Evidence-based medicine
• Person-centred care
• Models of care
• Healthcare improvement

BMC Medicine

ISSN: 1741-7015

• Submission enquiries: [email protected]
• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 22.5.2024 in Vol 26 (2024)

The Power of Rapid Reviews for Bridging the Knowledge-to-Action Gap in Evidence-Based Virtual Health Care

Authors of this article:

• Megan MacPherson, PhD   ; 
• Sarah Rourke, MSN  

Fraser Health, Surrey, BC, Canada

Corresponding Author:

Megan MacPherson, PhD

Fraser Health

400-13450 102nd Avenue

Surrey, BC, V3T 0H1

Phone: 1 6045616605

Email: [email protected]

Despite the surge in popularity of virtual health care services as a means of delivering health care through technology, the integration of research evidence into practice remains a challenge. Rapid reviews, a type of time-efficient evidence synthesis, offer a potential solution to bridge the gap between knowledge and action. This paper aims to highlight the experiences of the Fraser Health Authority’s Virtual Health team in conducting rapid reviews. This paper discusses the experiences of the Virtual Health team in conducting 15 rapid reviews over the course of 1.5 years and the benefit of involving diverse stakeholders including researchers, project and clinical leads, and students for the creation of user-friendly knowledge products to summarize results. The Virtual Health team found rapid reviews to be a valuable tool for evidence-informed decision-making in virtual health care. Involving stakeholders and focusing on implementation considerations are crucial for maximizing the impact of rapid reviews. Health care decision makers are encouraged to consider implementing rapid review processes to improve the translation of research evidence into practice, ultimately enhancing patient outcomes and promoting a culture of evidence-informed care.

Introduction

Virtual health care services, which involve the delivery of health care through information and communication technologies, have gained popularity among health care providers, patients, and organizations. In recent decades, several initiatives have been undertaken to implement virtual care and improve the access, quality, and safety of health care delivery in Canada [ 1 ]; however, technological advancement and a rapidly expanding evidence base make supporting virtual care with research evidence challenging. Specifically, to adequately support virtual care, health care decision makers are expected to keep up with available technologies, their applications, and evidence of their effectiveness among a variety of health conditions.

Despite decision makers recognizing the need to consider research evidence in the context of public health problems [ 2 , 3 ], there is still a knowledge-to-action (KTA) gap between what is known and what is put into practice clinically [ 4 - 6 ], with health care professionals worldwide demonstrating suboptimal use of research evidence within clinical practice [ 7 - 14 ]. Further, it has been estimated that one-third of patients do not receive treatments that have proven efficacious, one-quarter receive treatments that are potentially harmful, and up to three-quarters of patients and half of clinicians do not receive the information necessary for research-informed decision-making [ 15 ]. Clearly, there is a need to improve the translation of research evidence into practice, particularly in the case of virtual care where technological innovations and research evidence are rapidly expanding.

Knowledge Translation

The field of knowledge translation (KT) strives to enhance the usefulness of research evidence through the design and conduct of stakeholder-informed, patient-oriented studies as well as the dissemination and implementation of research findings into practice [ 16 ]. The Canadian Institutes for Health Research defines KT as the ethical exchange, synthesis, and application of knowledge among researchers and users to accelerate the benefits of research for Canadian people [ 17 ]. The ultimate goal of KT has been further described as the facilitation of evidence-informed decision-making [ 18 ] and the integration of various forms of evidence into public health practice and policy.

The Canadian Institutes for Health Research describes 2 “Death Valleys” on the continuum from research to action, which contributes to the KTA gap [ 19 ]. Valley 1 refers to the reduced ability to translate basic biomedical research discoveries from the laboratory to the bedside and to effectively commercialize health innovations. Valley 2 refers to the reduced ability to synthesize, disseminate, and integrate research findings more broadly into clinical practice and clinical decision-making. To improve the utility of biomedical and clinical research, enhance health outcomes, and ensure an evidence-based and sustainable health care system, strategic attempts to bridge these valleys must be made.

Rapid Reviews

One way to help overcome the second valley is through evidence syntheses such as systematic, scoping, and rapid reviews [ 20 ]. Evidence syntheses have emerged as valuable methods for KT as they can compile large bodies of evidence into a single knowledge product, making them an essential tool for decision makers to enhance evidence-informed decision-making [ 21 , 22 ]. Systematic reviews offer a comprehensive synthesis of available evidence on a particular topic, playing an ever-expanding role in informing policy making and practice [ 23 , 24 ]; however, the resource-intensive nature of conducting systematic reviews, in terms of both time and cost, presents a significant obstacle to facilitating prompt and efficient decision-making [ 25 ].

Given the time constraints health care practitioners and policy makers often face [ 26 ], rapid reviews provide a more resource- and time-efficient means to conduct evidence syntheses that offer actionable evidence in a more relevant manner compared to other types of evidence syntheses such as systematic or scoping reviews [ 20 , 26 - 34 ]. Specifically, rapid reviews are a form of evidence synthesis in which systematic review steps are streamlined to generate actionable evidence within a condensed time frame [ 35 ]. To expedite the review process, rapid reviews often compromise on the rigor typically associated with systematic reviews, resulting in a less precise and robust evaluation in comparison [ 32 ]. That being said, rapid reviews have gained traction in health systems’ policy making, health-related intervention development, and health technology assessment [ 34 - 36 ]. This paper outlines the experiences of the Fraser Health (FH) Authority Virtual Health team in rapidly producing and disseminating rapid review results to date. Rapid reviews were chosen as they are often highly driven by end-user demands [ 37 ] and have been highlighted as a viable tool to disseminate knowledge within the rapidly growing field of virtual health [ 33 ].

FH Authority Context

As the largest regional health authority in British Columbia, Canada, FH serves more than 1.9 million people in Canada [ 38 ]. In recent years, FH has prioritized the expansion of virtual care [ 39 ], conducting over 1.9 million virtual visits between January 2019 and 2023 (roughly 27% of all visits). Within the Virtual Health department at FH, the “research and evaluation team” aims to improve the translation of research into practice while engaging in ongoing collaborative evaluation of existing Virtual Health programming. During Virtual Health strategic planning, rapid reviews have emerged as a central tool for knowledge dissemination and have been used to inform the development of frameworks, services, and program scale-up. This paper highlights FH’s experience in conducting 15 rapid reviews over the course of 1.5 years. This paper is meant to serve as an overview on the utility and feasibility of rapid reviews within a health authority; for more information on rapid review methods to aid in conducting reviews within a team-based setting, see MacPherson et al [ 33 ].

Rapid reviews are used within the Virtual Health team to provide an overview of available evidence addressing a research question related to a single topic produced within a short time frame (typically 1 week to 4 months). From October 2022 until March 2024, the Virtual Health team conducted 15 rapid reviews following published recommendations [ 33 ]. Questions posed to date include the following:

• What are the perspectives on virtual care among immigrant, refugee, and Indigenous people in Canada [ 40 ]?
• What virtual care solutions exist for people with heart failure [ 41 ]?
• What virtual care solutions exist for people with diabetes [ 41 ]?
• What virtual care solutions exist for people with chronic obstructive pulmonary disease (COPD) [ 41 ]?
• What are currently used decision guides or algorithms to inform escalation within remote patient monitoring services for people with heart failure?
• What barriers, facilitators, and recommendations exist for remote patient monitoring services within the context of respiratory care [ 42 ]?
• What virtual care or digital innovations are used by physicians in acute care [ 43 ]?
• What barriers and facilitators exist for patient-to-provider virtual messaging (eg, SMS text messaging) [ 44 ]?
• What is the existing evidence for centralized remote patient monitoring services [ 45 ]?
• What domains are included within virtual care frameworks targeting appropriateness and safety?
• What are patient and provider barriers to virtual care [ 46 ]?
• What is the evidence for virtual hospital programs [ 47 ]?
• What KT strategies exist that could be used by the Virtual Health research and evaluation team in their efforts to translate research findings into practice?
• What is the available evidence on virtual decision-making and clinical judgment?
• What is the available evidence for, and are there existing validated assessment criteria for nursing assessment frameworks?

Team members assisting with the rapid reviews included researchers, project leads, clinical leads, and students previously unfamiliar with the review process. Knowledge users within the Virtual Health team (eg, clinical leads and clinical directors) were involved throughout the entirety of the review process from developing the research questions to the presentation of research findings in Virtual Health team meetings and the implementation of findings into Virtual Health practice.

Similar to other rapid reviews [ 20 ], results were collated and narratively or visually summarized (eg, through infographics) and presented to Virtual Health team members. The final knowledge products were created to offer a high-level overview of the evidence arranged in a user-friendly manner, aiming to provide VH team members with a high-level understanding of the available evidence [ 41 ].

Experiences and Lessons Learned

The Virtual Health team’s journey in conducting 15 rapid reviews over the course of 1.5 years has provided valuable insights into the feasibility and utility of rapid reviews within a health authority setting. These lessons learned are from the perspectives of the authors of this paper. MM is the research and KT lead of the Virtual Health department at the FH Authority. Prior to creating the rapid review program within the Virtual Health department, she has prior experience conducting systematic, scoping, and rapid reviews. SR is a clinical nurse specialist within the Virtual Health department at FH. As a system-level leader, SR leverages evidence to informed clinical and service model changes to optimize patient care and outcomes and support strategic priorities. Prior to her involvement in the Virtual Health rapid review program, SR had no previous experience with conducting evidence reviews.

Importance of Defining a Clear and Actionable Research Question

Throughout this journey, one of the key lessons learned was about the importance of the research question being actionable to ensure that the results of rapid reviews can be readily integrated into practice. Initially, our reviews had broader scopes aimed at informing future Virtual Health service implementations across various populations such as COPD, diabetes, and heart failure. While these reviews were informative, they did not lead to immediate changes in Virtual Health practice and required strategic efforts to disseminate findings and integrate results into practice. Subsequently, we learned that focusing on specific programs or initiatives within the Virtual Health setting yields more actionable results. For instance, a review focused on identifying patient and provider barriers to virtual care was conducted with the explicit purpose of informing the development of a framework to improve video visit uptake among primary care providers. This targeted approach enabled us to directly address the identified barriers through the development of a framework focused on the uptake of safe and appropriate video visits within primary care.

Benefits and Challenges Involving Knowledge Users

The involvement of knowledge users such as clinical leads and directors in the rapid review process proved to be invaluable. First, they helped focus the scope of reviews by providing insights into the practical needs and priorities within the FH context. For example, the reviews focusing on virtual care solutions for patients with heart failure, COPD, and diabetes were initiated by 1 of the directors within Virtual Health and included an occupational therapist and clinical nurse specialist on the review team. The diverse insights offered by clinician team members helped shape the review questions, search strategy, and analysis, ensuring it addressed the practical needs in delivering virtual care to this specific patient population.

Second, the engagement of nonresearchers, students, and health care professionals in the review process not only enhanced the quality and relevance of the rapid reviews but also provided an opportunity for experiential learning and professional development. By participating in the rapid review process, students and other team members developed essential skills such as critical appraisal, evidence synthesis, and scientific communication. This approach has the potential to bridge the gap between research and practice by building a generation of clinicians who are well versed in evidence-based practice and can effectively translate research findings into clinical decision-making. For example, a team of nursing students participated in a rapid review focused on algorithms for care escalation within remote patient monitoring services for patients with heart failure. While they lacked prior review experience, their fresh perspectives and familiarity with health care practice as it relates to heart failure brought unique insights helping to shape the clinician-oriented KT efforts.

While involving knowledge users throughout the review process offers numerous benefits, it can also extend the time required to complete a review. This is often due to the necessity for these individuals to familiarize themselves with new software while simultaneously mastering the intricacies of conducting reviews and adhering to all associated steps. For instance, several Virtual Health team members have observed that during their initial and subsequent reviews, they encountered difficulties in efficiently navigating the study screening phase. The abundance of potentially relevant literature posed a challenge, with concerns arising about potentially overlooking papers containing valuable insights or “hidden gems.” This underscores the importance of establishing clear eligibility criteria and providing comprehensive training from the outset to ensure reviewers feel empowered to exclude papers confidently, even those that may initially appear intriguing.

Resources and Staff Time Involved

Readers interested in starting a rapid review program in their own health systems may find it helpful to understand the resources and staff time involved in our process. As the research and KT lead within the Virtual Health team, MM has been responsible for building the rapid review program, training team members, and leading rapid reviews. Her full-time role allows for dedicated focus on these as well as other research and KT-related activities, ensuring the smooth operation of the rapid review process.

Additionally, strong leadership support within the Virtual Health team has been instrumental in fostering a culture of evidence-informed decision-making and facilitating the integration of research evidence into practice. While we do not have a core team with a dedicated full-time equivalent specifically for rapid reviews, a call is put out to the Virtual Health department at the beginning of a review to identify who has the capacity to assist in a review. A testament to the value of these reviews is that VH team members have begun autonomously conducting rapid reviews with the research and KT lead acting as an advisor, not a lead on the reviews. For example, a nurse who was tasked with creating a framework for a virtual nursing assessment requested assistance in running a search for her team to complete a rapid review, to ensure that the resulting framework did not miss any key components seen in the literature.

Rapid Review Process

The overall process map for our team (an adaptation of MacPherson et al [ 33 , 48 ]) can be found in Figure 1 . Our journey in conducting rapid reviews has been accompanied by several challenges and the implementation of quality assurance measures to ensure the integrity of our findings. The overall process of reviews within the Virtual Health team includes Virtual Health team members submitting a request or having an informal meeting with the research and KT lead outlining the scope and purpose of the review, which is then refined to ensure that it will result in actionable evidence relevant to the Virtual Health team and is in alignment with organizational priorities.

Challenges or obstacles encountered during the rapid review process have included resource constraints. When there are not enough people to assist with a review, either the time to complete the review needs to be extended or additional constraints must be placed on the review question. Time limitations have also been a factor, especially when there is an urgent request. Clear communication on how the results will be used is needed to refine the review topic and search strategy to quickly produce actionable evidence. Given the wealth of research, we have started all reviews by first exploring if our questions can be answered by conducting a review of reviews. This has allowed for the timely synthesis of evidence instead of relying on individual studies. We have also found that decision makers value the most up-to-date evidence (especially regarding virtual health care technologies); as such, many of our reviews have imposed limitations to the past 5-10 years to ensure their relevance to decision makers. Additionally, difficulties in accessing relevant literature have been noted, as health authorities often do not have access to the same resources as academic institutions. This results in increased time to secure papers through interlibrary loans, which can be overcome by collaborating with academics.

Another strength of the Virtual Health team’s rapid review approach was the development of easily digestible knowledge products highlighting key data synthesized in the review. Rather than providing end users with lengthy reports that often go unread, clinicians within the Virtual Health team helped to create brief summaries and infographics highlighting the main findings and recommendations. This approach was aimed at improving the uptake of research evidence into practice by presenting the information in a format that was easily accessible and understandable for clinicians and other stakeholders. By creating visually appealing and user-friendly knowledge products, the Virtual Health team was able to efficiently communicate key takeaways from the rapid reviews, thus facilitating their dissemination and implementation within the FH context. This approach also helped to overcome a common challenge of KT, where research evidence can be difficult to access, understand, and apply in practice. By presenting the information in a format that was relevant and easily digestible, the Virtual Health team was able to enhance the applicability of the rapid reviews, thereby building clinician capacity and increasing their potential impact on patient outcomes.

Leveraging Rapid Reviews for Clinically Based Tools

Our most recent reviews were focused on developing a virtual nursing assessment and virtual nursing decision-making framework. Unlike traditional KT efforts used within other reviews, where the focus often lies on creating user-friendly summaries and infographics, our approach took a slightly different path. We aimed to directly inform the development of clinical decision support tools (DSTs).

Rather than developing traditional KT products, the raw data extracted from these reviews served as a foundational resource for the development of the clinical DSTs. Each piece of information was carefully referenced and integrated into the tool, providing evidence-based support for specific components and functionalities. This direct integration of research evidence into the tool development process not only strengthened the validity and credibility of the tool but also facilitated the transparent communication of the evidence behind each recommendation or feature.

Within these reviews, the active participation of those who were responsible for the development of the DSTs proved invaluable. Their involvement was crucial in ensuring understanding and confidence in the information as well as in merging research evidence with their own clinical expertise. By involving end users in the review process, we could tailor the outcomes to their specific needs and preferences, ultimately enhancing the relevance and applicability of the extracted evidence. This collaborative approach ensured that the resulting DSTs were not only evidence based but also resonated effectively with the clinical context they were intended for.

Principal Findings

The Virtual Health team’s experience with conducting 15 rapid reviews over the course of 1.5 years highlights the potential of rapid reviews as a time-efficient tool for improving the translation and uptake of research evidence into Virtual Health programming. Compared to more traditional review types (eg, systematic or scoping), which can take more than a year to complete [ 49 ], rapid reviews provide a practical way of synthesizing available evidence to inform clinical decision-making. The ability to produce a high-quality evidence summary in a shorter time frame can be particularly valuable in rapidly evolving areas of health care, such as virtual health. While rapid reviews are not new, our program offers insights into their application in a dynamic and rapidly evolving field such as virtual health. The lessons learned from FH’s rapid review program have important implications for evidence-based decision-making and KT within health care settings.

One of our primary lessons learned underscores the importance of establishing clear and actionable research questions. By outlining precise objectives, rapid reviews can ensure the relevance and applicability of their results, thus facilitating their seamless integration into clinical practice. Moreover, our experiences highlight the transformative impact of involving knowledge users throughout the review process. This collaborative approach not only enhances the quality and relevance of the evidence synthesized but also fosters a culture of evidence-informed decision-making within the organization. This type of early and continued engagement of knowledge users in research endeavors has been increasingly recognized as pivotal for establishing research priorities and enhancing the utility of research findings in real-world health care contexts [ 50 , 51 ]. In line with this, the overarching goal of knowledge-user engagement in health research is to coproduce knowledge that directly addresses the needs of decision makers. By involving knowledge users from the outset, research priorities can be aligned with the practical requirements of health care delivery, thereby increasing the relevance and utility of research outputs [ 52 - 54 ].

Limitations of Rapid Reviews

Despite its benefits, the rapid review approach is not without limitations. Loss of rigor, as mentioned earlier in this paper, remains a concern. The rapid nature of the process may compromise the depth and comprehensiveness of the literature search and synthesis, potentially leading to oversights or biases in the evidence presented. Furthermore, within the context of virtual health, the rapid pace of technological advancements poses a challenge. New technologies may outpace the generation of peer-reviewed literature, resulting in a lag between their implementation and the availability of robust evidence.

In response to the challenge posed by rapidly evolving technologies, FH’s Virtual Health department has used creative solutions to capture relevant evidence. While peer-reviewed literature remains a primary source, we have also incorporated gray literature, such as news articles, trade publications, and reports, from other health care authorities or departments within the review processes when applicable. Additionally, to supplement reviews and provide more contextual evidence, additional research and evaluation methodologies are used (time permitting) to inform Virtual Health service development such as consulting Patient and Family Advisory Councils within FH, conducting interviews with patient and clinician partners, and conducting analyses on existing data within FH.

Next Steps for FH’s Rapid Review Program

We remain committed to advancing the rapid review program to meet the evolving needs of the Virtual Health department at FH. While we have heard anecdotally that knowledge users value the user-friendly knowledge products developed for rapid reviews, the next steps of this program include an evaluation of our knowledge dissemination to assess the reach and impact the reviews are having within the Virtual Health department.

Conclusions

Rapid reviews are a valuable tool for the timely synthesis of available research evidence to inform health care decision-making. The Virtual Health team’s experience with conducting rapid reviews highlights the importance of involving a diverse range of knowledge users in the review process and the need to focus on implementation considerations. By engaging knowledge users beyond designated researchers, and particularly by involving clinicians across the research process, rapid reviews become more robust, applicable, and aligned with the practical needs of health care providers and organizations, which can help to bridge the KTA gap.

Conflicts of Interest

None declared.

• Goodridge D, Marciniuk D. Rural and remote care: overcoming the challenges of distance. Chron Respir Dis. 2016;13(2):192-203. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bowen S, Zwi AB. Pathways to "evidence-informed" policy and practice: a framework for action. PLoS Med. 2005;2(7):e166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jacobs JA, Jones E, Gabella BA, Spring B, Brownson RC. Tools for implementing an evidence-based approach in public health practice. Prev Chronic Dis. 2012;9:E116. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davis D, Evans M, Jadad A, Perrier L, Rath D, Ryan D, et al. The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 2003;327(7405):33-35. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003;362(9391):1225-1230. [ CrossRef ] [ Medline ]
• Grol R, Jones R. Twenty years of implementation research. Fam Pract. 2000;17(Suppl 1):S32-S35. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Straus SE, McAlister FA. Evidence-based medicine: a commentary on common criticisms. CMAJ. 2000;163(7):837-841. [ FREE Full text ] [ Medline ]
• Mellis C. Evidence-based medicine: what has happened in the past 50 years? J Paediatr Child Health. 2015;51(1):65-68. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Villar J, Carroli G, Gülmezoglu AM. The gap between evidence and practice in maternal healthcare. Int J Gynaecol Obstet. 2001;75(Suppl 1):S47-S54. [ Medline ]
• Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care. 2001;39(8 Suppl 2):II46-II54. [ CrossRef ] [ Medline ]
• Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States? Milbank Q. 1998;76(4):517-563. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635-2645. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lauer MS, Skarlatos S. Translational research for cardiovascular diseases at the National Heart, Lung, and Blood Institute: moving from bench to bedside and from bedside to community. Circulation. 2010;121(7):929-933. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lang ES, Wyer PC, Haynes RB. Knowledge translation: closing the evidence-to-practice gap. Ann Emerg Med. 2007;49(3):355-363. [ CrossRef ] [ Medline ]
• Kitson AL, Straus SE. Identifying knowledge to action gaps. Knowledge Transl Health Care. 2013:97-109. [ FREE Full text ] [ CrossRef ]
• Graham ID, Logan JL, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13-24. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Knowledge translation strategy 2004-2009: innovation in action. Canadian Institutes of Health Research. 2004. URL: https://cihr-irsc.gc.ca/e/26574.html [accessed 2024-04-25]
• Ciliska D, Thomas H, Buffett C. A compendium of critical appraisal tools for public health practice. National Collaborating Centre for Methods and Tools. 2008. URL: https://www.nccmt.ca/uploads/media/media/0001/01/b331668f85bc6357f262944f0aca38c14c89c5a4.pdf [accessed 2024-04-25]
• Canada's strategy for patient-oriented research. Government of Canada. Canadian Institutes of Health Research. 2011. URL: https://cihr-irsc.gc.ca/e/44000.html#a1.1 [accessed 2023-04-06]
• Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1(1):10. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chambers D, Wilson PM, Thompson CA, Hanbury A, Farley K, Light K. Maximizing the impact of systematic reviews in health care decision making: a systematic scoping review of knowledge-translation resources. Milbank Q. 2011;89(1):131-156. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7(1):50. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen JA, Dröschel D, et al. Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9(3):e1001185. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP). Norwegian Knowledge Centre for the Health Services. 2010. URL: https://fhi.brage.unit.no/fhi-xmlui/bitstream/handle/11250/2378076/NOKCrapport4_2010.pdf?sequence=1 [accessed 2023-11-22]
• Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14(1):2. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci. 2010;5(1):56. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moore G, Redman S, Rudge S, Haynes A. Do policy-makers find commissioned rapid reviews useful? Health Res Policy Syst. 2018;16(1):17. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Flores EJ, Jue JJ, Giradi G, Schoelles K, Mull NK, Umscheid CA. AHRQ EPC Series on improving translation of evidence: use of a clinical pathway for C. Difficile treatment to facilitate the translation of research findings into practice. Jt Comm J Qual Patient Saf. 2019;45(12):822-828. [ CrossRef ] [ Medline ]
• Hartling L, Guise J, Kato E, Anderson J, Belinson S, Berliner E, et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. J Clin Epidemiol. 2015;68(12):1451-1462.e3. [ CrossRef ] [ Medline ]
• Hartling L, Guise JM, Kato E, Anderson J, Aronson N, Belinson S, et al. EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews. Rockville, MD. Agency for Healthcare Research and Quality; 2015.
• Hartling L, Guise JM, Hempel S, Featherstone R, Mitchell MD, Motu'apuaka ML, et al. Fit for purpose: perspectives on rapid reviews from end-user interviews. Syst Rev. 2017;6(1):32. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Featherstone RM, Dryden DM, Foisy M, Guise JM, Mitchell MD, Paynter RA, et al. Advancing knowledge of rapid reviews: an analysis of results, conclusions and recommendations from published review articles examining rapid reviews. Syst Rev. 2015;4(1):50. [ FREE Full text ] [ CrossRef ] [ Medline ]
• MacPherson MM, Wang RH, Smith EM, Sithamparanathan G, Sadiq CA, Braunizer AR. Rapid reviews to support practice: a guide for professional organization practice networks. Can J Occup Ther. 2023;90(3):269-279. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Watt A, Cameron A, Sturm L, Lathlean T, Babidge W, Blamey S, et al. Rapid reviews versus full systematic reviews: an inventory of current methods and practice in health technology assessment. Int J Technol Assess Health Care. 2008;24(2):133-139. [ CrossRef ] [ Medline ]
• Polisena J, Garritty C, Kamel C, Stevens A, Abou-Setta AM. Rapid review programs to support health care and policy decision making: a descriptive analysis of processes and methods. Syst Rev. 2015;4(1):26. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Harker J, Kleijnen J. What is a rapid review? A methodological exploration of rapid reviews in health technology assessments. Int J Evid Based Healthc. 2012;10(4):397-410. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, et al. Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. J Clin Epidemiol. 2021;130:13-22. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fraser Health. 2023. URL: https://www.fraserhealth.ca/ [accessed 2023-04-06]
• Virtual Health. Fraser Health. URL: https://www.fraserhealth.ca/patients-and-visitors/virtual-health [accessed 2023-04-06]
• MacPherson M. Immigrant, refugee, and Indigenous Canadians' experiences with virtual health care services: rapid review. JMIR Hum Factors. 09, 2023;10:e47288. [ FREE Full text ] [ CrossRef ] [ Medline ]
• MacPherson M. Virtual care in heart failure, chronic obstructive pulmonary disease, and diabetes: a rapid review protocol. OSF Registries. 2023. URL: https://osf.io/xn2pe [accessed 2023-09-11]
• MacPherson M. Barriers, facilitators, and recommendations to inform the expansion of remote patient monitoring services for respiratory care: a rapid review. OSF Registries. 2022. URL: https://osf.io/asf2v/ [accessed 2024-04-25]
• MacPherson M. Virtual health services in the context of acute care: a rapid review. OSF Registries. 2023. URL: https://osf.io/ub2d8/ [accessed 2024-04-25]
• MacPherson MM, Kapadia S. Barriers and facilitators to patient-to-provider messaging using the COM-B model and theoretical domains framework: a rapid umbrella review. BMC Digit Health. 2023;1(1):33. [ FREE Full text ] [ CrossRef ]
• Chan L, MacPherson M. Remote patient monitoring: an evidence synthesis. OSF Registries. 2023. URL: https://osf.io/7wqb8/ [accessed 2024-04-25]
• Montenegro M, MacPherson M. Barriers to virtual care experienced by patients and healthcare providers: a rapid umbrella review. OSF Registries. 2023. URL: https://osf.io/nufg4/ [accessed 2024-04-25]
• Montenegro M, MacPherson M. Virtual hospitals: a rapid review. OSF Registries. 2023. URL: https://osf.io/m3a4b/ [accessed 2024-04-25]
• Attribution 4.0 International (CC BY 4.0). Creative Commons. URL: https://creativecommons.org/licenses/by/4.0/ [accessed 2024-05-13]
• Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. 2012;1(2):181-194. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bragge P, Clavisi O, Turner T, Tavender E, Collie A, Gruen RL. The Global Evidence Mapping Initiative: scoping research in broad topic areas. BMC Med Res Methodol. 2011;11(1):92. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Langlois EV, Montekio VB, Young T, Song K, Alcalde-Rabanal J, Tran N. Enhancing evidence informed policymaking in complex health systems: lessons from multi-site collaborative approaches. Health Res Policy Syst. 2016;14(1):20. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vindrola-Padros C, Pape T, Utley M, Fulop NJ. The role of embedded research in quality improvement: a narrative review. BMJ Qual Saf. 2017;26(1):70-80. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ghaffar A, Langlois EV, Rasanathan K, Peterson S, Adedokun L, Tran NT. Strengthening health systems through embedded research. Bull World Health Organ. 2017;95(2):87. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by Z Yin; submitted 22.11.23; peer-reviewed by W LaMendola, M Willenbring, Y Zhang, P Blasi; comments to author 10.03.24; revised version received 15.03.24; accepted 13.04.24; published 22.05.24.

©Megan MacPherson, Sarah Rourke. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 22.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

How Much Research Is Being Written by Large Language Models?

New studies show a marked spike in LLM usage in academia, especially in computer science. What does this mean for researchers and reviewers?

In March of this year, a  tweet about an academic paper went viral for all the wrong reasons. The introduction section of the paper, published in  Elsevier’s  Surfaces and Interfaces , began with this line:  Certainly, here is a possible introduction for your topic. 

Look familiar? 

It should, if you are a user of ChatGPT and have applied its talents for the purpose of content generation. LLMs are being increasingly used to assist with writing tasks, but examples like this in academia are largely anecdotal and had not been quantified before now. 

“While this is an egregious example,” says  James Zou , associate professor of biomedical data science and, by courtesy, of computer science and of electrical engineering at Stanford, “in many cases, it’s less obvious, and that’s why we need to develop more granular and robust statistical methods to estimate the frequency and magnitude of LLM usage. At this particular moment, people want to know what content around us is written by AI. This is especially important in the context of research, for the papers we author and read and the reviews we get on our papers. That’s why we wanted to study how much of those have been written with the help of AI.”

In two papers looking at LLM use in scientific publishings, Zou and his team* found that 17.5% of computer science papers and 16.9% of peer review text had at least some content drafted by AI. The paper on LLM usage in peer reviews will be presented at the International Conference on Machine Learning.

Read  Mapping the Increasing Use of LLMs in Scientific Papers and  Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews  

Here Zou discusses the findings and implications of this work, which was supported through a Stanford HAI Hoffman Yee Research Grant . 

How did you determine whether AI wrote sections of a paper or a review?

We first saw that there are these specific worlds – like commendable, innovative, meticulous, pivotal, intricate, realm, and showcasing – whose frequency in reviews sharply spiked, coinciding with the release of ChatGPT. Additionally, we know that these words are much more likely to be used by LLMs than by humans. The reason we know this is that we actually did an experiment where we took many papers, used LLMs to write reviews of them, and compared those reviews to reviews written by human reviewers on the same papers. Then we quantified which words are more likely to be used by LLMs vs. humans, and those are exactly the words listed. The fact that they are more likely to be used by an LLM and that they have also seen a sharp spike coinciding with the release of LLMs is strong evidence.

Some journals permit the use of LLMs in academic writing, as long as it’s noted, while others, including  Science and the ICML conference, prohibit it. How are the ethics perceived in academia?

This is an important and timely topic because the policies of various journals are changing very quickly. For example,  Science said in the beginning that they would not allow authors to use language models in their submissions, but they later changed their policy and said that people could use language models, but authors have to explicitly note where the language model is being used. All the journals are struggling with how to define this and what’s the right way going forward.

You observed an increase in usage of LLMs in academic writing, particularly in computer science papers (up to 17.5%). Math and  Nature family papers, meanwhile, used AI text about 6.3% of the time. What do you think accounts for the discrepancy between these disciplines? 

Artificial intelligence and computer science disciplines have seen an explosion in the number of papers submitted to conferences like ICLR and NeurIPS. And I think that’s really caused a strong burden, in many ways, to reviewers and to authors. So now it’s increasingly difficult to find qualified reviewers who have time to review all these papers. And some authors may feel more competition that they need to keep up and keep writing more and faster. 

You analyzed close to a million papers on arXiv, bioRxiv, and  Nature from January 2020 to February 2024. Do any of these journals include humanities papers or anything in the social sciences?  

We mostly wanted to focus more on CS and engineering and biomedical areas and interdisciplinary areas, like  Nature family journals, which also publish some social science papers. Availability mattered in this case. So, it’s relatively easy for us to get data from arXiv, bioRxiv, and  Nature . A lot of AI conferences also make reviews publicly available. That’s not the case for humanities journals.

Did any results surprise you?

A few months after ChatGPT’s launch, we started to see a rapid, linear increase in the usage pattern in academic writing. This tells us how quickly these LLM technologies diffuse into the community and become adopted by researchers. The most surprising finding is the magnitude and speed of the increase in language model usage. Nearly a fifth of papers and peer review text use LLM modification. We also found that peer reviews submitted closer to the deadline and those less likely to engage with author rebuttal were more likely to use LLMs. 

This suggests a couple of things. Perhaps some of these reviewers are not as engaged with reviewing these papers, and that’s why they are offloading some of the work to AI to help. This could be problematic if reviewers are not fully involved. As one of the pillars of the scientific process, it is still necessary to have human experts providing objective and rigorous evaluations. If this is being diluted, that’s not great for the scientific community.

What do your findings mean for the broader research community?

LLMs are transforming how we do research. It’s clear from our work that many papers we read are written with the help of LLMs. There needs to be more transparency, and people should state explicitly how LLMs are used and if they are used substantially. I don’t think it’s always a bad thing for people to use LLMs. In many areas, this can be very useful. For someone who is not a native English speaker, having the model polish their writing can be helpful. There are constructive ways for people to use LLMs in the research process; for example, in earlier stages of their draft. You could get useful feedback from a LLM in real time instead of waiting weeks or months to get external feedback. 

But I think it’s still very important for the human researchers to be accountable for everything that is submitted and presented. They should be able to say, “Yes, I will stand behind the statements that are written in this paper.”

*Collaborators include:  Weixin Liang ,  Yaohui Zhang ,  Zhengxuan Wu ,  Haley Lepp ,  Wenlong Ji ,  Xuandong Zhao ,  Hancheng Cao ,  Sheng Liu ,  Siyu He ,  Zhi Huang ,  Diyi Yang ,  Christopher Potts ,  Christopher D. Manning ,  Zachary Izzo ,  Yaohui Zhang ,  Lingjiao Chen ,  Haotian Ye , and Daniel A. McFarland .

Stanford HAI’s mission is to advance AI research, education, policy and practice to improve the human condition.  Learn more . 

More News Topics

Impact of Retail CBDC on Digital Payments, and Bank Deposits: Evidence from India

Interest in central bank digital currencies (CBDCs) has been burgeoning with 134 countries now exploring its implementation. In December 2022, India started its CBDC pilot program to continue its transition towards a digitized payments economy. This paper presents the first empirical analysis utilizing detailed transaction data to explore the dynamics between CBDCs and existing digital payment methods, as well as the implications of increased CBDC usage on traditional bank deposits. Our findings reveal that policies which increase transaction costs for current digital payment methods catalyze a substitution effect, bolstering CBDC adoption. Furthermore, an uptick in CBDC usage is associated with a notable decline in bank, cash, and savings deposits, suggesting potential paths to bank disintermediation. This study contributes critical insights into the evolving competition between digital currencies and established financial infrastructures, highlighting the transformative potential of CBDCs on the broader economy.

The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.

MARC RIS BibTeΧ

Download Citation Data

Working Groups

More from nber.

In addition to working papers , the NBER disseminates affiliates’ latest findings through a range of free periodicals — the NBER Reporter , the NBER Digest , the Bulletin on Retirement and Disability , the Bulletin on Health , and the Bulletin on Entrepreneurship  — as well as online conference reports , video lectures , and interviews .

Child Tax Benefits and Labor Supply: Evidence from California

In the United States today, some of the largest social welfare programs focused on children – including the Earned Income Tax Credit (EITC) and Child Tax Credit (CTC) – require that parents earn income from work. While intended to encourage recipients to work, tax credit work requirements may also harm the lowest-income families. In this paper, the authors study whether eliminating child tax credit work requirements affects parents’ decision to work.

The authors study this question in the context of California’s Young Child Tax Credit (YCTC), a refundable state tax credit for low-income parents with children younger than six. When the YCTC was enacted in 2019 it was available to any taxpayer with income over $1. Then, beginning in 2022, California eliminated the work requirement altogether.   Using federal administrative tax data, the authors compare the labor force participation of mothers with children who just barely qualify for the YCTC to those with children just above the age cutoff, before and after the work requirement was eliminated. They find the following:

• Eliminating the YCTC work requirement did not cause a significant number of California mothers to exit the labor force. The authors estimate that working mothers’ labor force participation fell by no more than 0.4 percentage points with the elimination of the work requirement.

The results of the study suggest that eliminating the work requirement from the federal CTC would cause fewer exits from the labor force than prior studies suggest. The results also provide new evidence for states considering adopting or reforming their own child tax benefits, as a central issue in designing such policies is whether to condition benefits on work.

More on this topic

Tax Policy and Investment in a Global Economy

The Short-Term Labor Supply Response to the Expanded Child Tax Credit

The macroeconomics of the greek depression.