research results mean

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 7. The Results
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The results section is where you report the findings of your study based upon the methodology [or methodologies] you applied to gather information. The results section should state the findings of the research arranged in a logical sequence without bias or interpretation. A section describing results should be particularly detailed if your paper includes data generated from your own research.

Annesley, Thomas M. "Show Your Cards: The Results Section and the Poker Game." Clinical Chemistry 56 (July 2010): 1066-1070.

Importance of a Good Results Section

When formulating the results section, it's important to remember that the results of a study do not prove anything . Findings can only confirm or reject the hypothesis underpinning your study. However, the act of articulating the results helps you to understand the problem from within, to break it into pieces, and to view the research problem from various perspectives.

The page length of this section is set by the amount and types of data to be reported . Be concise. Use non-textual elements appropriately, such as figures and tables, to present findings more effectively. In deciding what data to describe in your results section, you must clearly distinguish information that would normally be included in a research paper from any raw data or other content that could be included as an appendix. In general, raw data that has not been summarized should not be included in the main text of your paper unless requested to do so by your professor.

Avoid providing data that is not critical to answering the research question . The background information you described in the introduction section should provide the reader with any additional context or explanation needed to understand the results. A good strategy is to always re-read the background section of your paper after you have written up your results to ensure that the reader has enough context to understand the results [and, later, how you interpreted the results in the discussion section of your paper that follows].

Bavdekar, Sandeep B. and Sneha Chandak. "Results: Unraveling the Findings." Journal of the Association of Physicians of India 63 (September 2015): 44-46; Brett, Paul. "A Genre Analysis of the Results Section of Sociology Articles." English for Specific Speakers 13 (1994): 47-59; Go to English for Specific Purposes on ScienceDirect;Burton, Neil et al. Doing Your Education Research Project . Los Angeles, CA: SAGE, 2008; Results. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College; Kretchmer, Paul. Twelve Steps to Writing an Effective Results Section. San Francisco Edit; "Reporting Findings." In Making Sense of Social Research Malcolm Williams, editor. (London;: SAGE Publications, 2003) pp. 188-207.

Structure and Writing Style

I.  Organization and Approach

For most research papers in the social and behavioral sciences, there are two possible ways of organizing the results . Both approaches are appropriate in how you report your findings, but use only one approach.

• Present a synopsis of the results followed by an explanation of key findings . This approach can be used to highlight important findings. For example, you may have noticed an unusual correlation between two variables during the analysis of your findings. It is appropriate to highlight this finding in the results section. However, speculating as to why this correlation exists and offering a hypothesis about what may be happening belongs in the discussion section of your paper.
• Present a result and then explain it, before presenting the next result then explaining it, and so on, then end with an overall synopsis . This is the preferred approach if you have multiple results of equal significance. It is more common in longer papers because it helps the reader to better understand each finding. In this model, it is helpful to provide a brief conclusion that ties each of the findings together and provides a narrative bridge to the discussion section of the your paper.

NOTE :   Just as the literature review should be arranged under conceptual categories rather than systematically describing each source, you should also organize your findings under key themes related to addressing the research problem. This can be done under either format noted above [i.e., a thorough explanation of the key results or a sequential, thematic description and explanation of each finding].

II.  Content

In general, the content of your results section should include the following:

• Introductory context for understanding the results by restating the research problem underpinning your study . This is useful in re-orientating the reader's focus back to the research problem after having read a review of the literature and your explanation of the methods used for gathering and analyzing information.
• Inclusion of non-textual elements, such as, figures, charts, photos, maps, tables, etc. to further illustrate key findings, if appropriate . Rather than relying entirely on descriptive text, consider how your findings can be presented visually. This is a helpful way of condensing a lot of data into one place that can then be referred to in the text. Consider referring to appendices if there is a lot of non-textual elements.
• A systematic description of your results, highlighting for the reader observations that are most relevant to the topic under investigation . Not all results that emerge from the methodology used to gather information may be related to answering the " So What? " question. Do not confuse observations with interpretations; observations in this context refers to highlighting important findings you discovered through a process of reviewing prior literature and gathering data.
• The page length of your results section is guided by the amount and types of data to be reported . However, focus on findings that are important and related to addressing the research problem. It is not uncommon to have unanticipated results that are not relevant to answering the research question. This is not to say that you don't acknowledge tangential findings and, in fact, can be referred to as areas for further research in the conclusion of your paper. However, spending time in the results section describing tangential findings clutters your overall results section and distracts the reader.
• A short paragraph that concludes the results section by synthesizing the key findings of the study . Highlight the most important findings you want readers to remember as they transition into the discussion section. This is particularly important if, for example, there are many results to report, the findings are complicated or unanticipated, or they are impactful or actionable in some way [i.e., able to be pursued in a feasible way applied to practice].

NOTE:   Always use the past tense when referring to your study's findings. Reference to findings should always be described as having already happened because the method used to gather the information has been completed.

III.  Problems to Avoid

When writing the results section, avoid doing the following :

• Discussing or interpreting your results . Save this for the discussion section of your paper, although where appropriate, you should compare or contrast specific results to those found in other studies [e.g., "Similar to the work of Smith [1990], one of the findings of this study is the strong correlation between motivation and academic achievement...."].
• Reporting background information or attempting to explain your findings. This should have been done in your introduction section, but don't panic! Often the results of a study point to the need for additional background information or to explain the topic further, so don't think you did something wrong. Writing up research is rarely a linear process. Always revise your introduction as needed.
• Ignoring negative results . A negative result generally refers to a finding that does not support the underlying assumptions of your study. Do not ignore them. Document these findings and then state in your discussion section why you believe a negative result emerged from your study. Note that negative results, and how you handle them, can give you an opportunity to write a more engaging discussion section, therefore, don't be hesitant to highlight them.
• Including raw data or intermediate calculations . Ask your professor if you need to include any raw data generated by your study, such as transcripts from interviews or data files. If raw data is to be included, place it in an appendix or set of appendices that are referred to in the text.
• Be as factual and concise as possible in reporting your findings . Do not use phrases that are vague or non-specific, such as, "appeared to be greater than other variables..." or "demonstrates promising trends that...." Subjective modifiers should be explained in the discussion section of the paper [i.e., why did one variable appear greater? Or, how does the finding demonstrate a promising trend?].
• Presenting the same data or repeating the same information more than once . If you want to highlight a particular finding, it is appropriate to do so in the results section. However, you should emphasize its significance in relation to addressing the research problem in the discussion section. Do not repeat it in your results section because you can do that in the conclusion of your paper.
• Confusing figures with tables . Be sure to properly label any non-textual elements in your paper. Don't call a chart an illustration or a figure a table. If you are not sure, go here .

Annesley, Thomas M. "Show Your Cards: The Results Section and the Poker Game." Clinical Chemistry 56 (July 2010): 1066-1070; Bavdekar, Sandeep B. and Sneha Chandak. "Results: Unraveling the Findings." Journal of the Association of Physicians of India 63 (September 2015): 44-46; Burton, Neil et al. Doing Your Education Research Project . Los Angeles, CA: SAGE, 2008;  Caprette, David R. Writing Research Papers. Experimental Biosciences Resources. Rice University; Hancock, Dawson R. and Bob Algozzine. Doing Case Study Research: A Practical Guide for Beginning Researchers . 2nd ed. New York: Teachers College Press, 2011; Introduction to Nursing Research: Reporting Research Findings. Nursing Research: Open Access Nursing Research and Review Articles. (January 4, 2012); Kretchmer, Paul. Twelve Steps to Writing an Effective Results Section. San Francisco Edit ; Ng, K. H. and W. C. Peh. "Writing the Results." Singapore Medical Journal 49 (2008): 967-968; Reporting Research Findings. Wilder Research, in partnership with the Minnesota Department of Human Services. (February 2009); Results. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College; Schafer, Mickey S. Writing the Results. Thesis Writing in the Sciences. Course Syllabus. University of Florida.

Writing Tip

Why Don't I Just Combine the Results Section with the Discussion Section?

It's not unusual to find articles in scholarly social science journals where the author(s) have combined a description of the findings with a discussion about their significance and implications. You could do this. However, if you are inexperienced writing research papers, consider creating two distinct sections for each section in your paper as a way to better organize your thoughts and, by extension, your paper. Think of the results section as the place where you report what your study found; think of the discussion section as the place where you interpret the information and answer the "So What?" question. As you become more skilled writing research papers, you can consider melding the results of your study with a discussion of its implications.

Driscoll, Dana Lynn and Aleksandra Kasztalska. Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

• << Previous: Insiderness
• Next: Using Non-Textual Elements >>
• Last Updated: Apr 11, 2024 1:27 PM
• URL: https://libguides.usc.edu/writingguide

• Affiliate Program

• UNITED STATES
• 台灣 (TAIWAN)
• TÜRKIYE (TURKEY)
• Academic Editing Services
• - Research Paper
• - Journal Manuscript
• - Dissertation
• - College & University Assignments
• Admissions Editing Services
• - Application Essay
• - Personal Statement
• - Recommendation Letter
• - Cover Letter
• - CV/Resume
• Business Editing Services
• - Business Documents
• - Report & Brochure
• - Website & Blog
• Writer Editing Services
• - Script & Screenplay
• Our Editors
• Client Reviews
• Editing & Proofreading Prices
• Wordvice Points
• Partner Discount
• Plagiarism Checker
• APA Citation Generator
• MLA Citation Generator
• Chicago Citation Generator
• Vancouver Citation Generator
• - APA Style
• - MLA Style
• - Chicago Style
• - Vancouver Style
• Writing & Editing Guide
• Academic Resources
• Admissions Resources

How to Write the Results/Findings Section in Research

What is the research paper Results section and what does it do?

The Results section of a scientific research paper represents the core findings of a study derived from the methods applied to gather and analyze information. It presents these findings in a logical sequence without bias or interpretation from the author, setting up the reader for later interpretation and evaluation in the Discussion section. A major purpose of the Results section is to break down the data into sentences that show its significance to the research question(s).

The Results section appears third in the section sequence in most scientific papers. It follows the presentation of the Methods and Materials and is presented before the Discussion section —although the Results and Discussion are presented together in many journals. This section answers the basic question “What did you find in your research?”

What is included in the Results section?

The Results section should include the findings of your study and ONLY the findings of your study. The findings include:

• Data presented in tables, charts, graphs, and other figures (may be placed into the text or on separate pages at the end of the manuscript)
• A contextual analysis of this data explaining its meaning in sentence form
• All data that corresponds to the central research question(s)
• All secondary findings (secondary outcomes, subgroup analyses, etc.)

If the scope of the study is broad, or if you studied a variety of variables, or if the methodology used yields a wide range of different results, the author should present only those results that are most relevant to the research question stated in the Introduction section .

As a general rule, any information that does not present the direct findings or outcome of the study should be left out of this section. Unless the journal requests that authors combine the Results and Discussion sections, explanations and interpretations should be omitted from the Results.

How are the results organized?

The best way to organize your Results section is “logically.” One logical and clear method of organizing research results is to provide them alongside the research questions—within each research question, present the type of data that addresses that research question.

Let’s look at an example. Your research question is based on a survey among patients who were treated at a hospital and received postoperative care. Let’s say your first research question is:

“What do hospital patients over age 55 think about postoperative care?”

This can actually be represented as a heading within your Results section, though it might be presented as a statement rather than a question:

Attitudes towards postoperative care in patients over the age of 55

Now present the results that address this specific research question first. In this case, perhaps a table illustrating data from a survey. Likert items can be included in this example. Tables can also present standard deviations, probabilities, correlation matrices, etc.

Following this, present a content analysis, in words, of one end of the spectrum of the survey or data table. In our example case, start with the POSITIVE survey responses regarding postoperative care, using descriptive phrases. For example:

“Sixty-five percent of patients over 55 responded positively to the question “ Are you satisfied with your hospital’s postoperative care ?” (Fig. 2)

Include other results such as subcategory analyses. The amount of textual description used will depend on how much interpretation of tables and figures is necessary and how many examples the reader needs in order to understand the significance of your research findings.

Next, present a content analysis of another part of the spectrum of the same research question, perhaps the NEGATIVE or NEUTRAL responses to the survey. For instance:

  “As Figure 1 shows, 15 out of 60 patients in Group A responded negatively to Question 2.”

After you have assessed the data in one figure and explained it sufficiently, move on to your next research question. For example:

  “How does patient satisfaction correspond to in-hospital improvements made to postoperative care?”

This kind of data may be presented through a figure or set of figures (for instance, a paired T-test table).

Explain the data you present, here in a table, with a concise content analysis:

“The p-value for the comparison between the before and after groups of patients was .03% (Fig. 2), indicating that the greater the dissatisfaction among patients, the more frequent the improvements that were made to postoperative care.”

Let’s examine another example of a Results section from a study on plant tolerance to heavy metal stress . In the Introduction section, the aims of the study are presented as “determining the physiological and morphological responses of Allium cepa L. towards increased cadmium toxicity” and “evaluating its potential to accumulate the metal and its associated environmental consequences.” The Results section presents data showing how these aims are achieved in tables alongside a content analysis, beginning with an overview of the findings:

“Cadmium caused inhibition of root and leave elongation, with increasing effects at higher exposure doses (Fig. 1a-c).”

The figure containing this data is cited in parentheses. Note that this author has combined three graphs into one single figure. Separating the data into separate graphs focusing on specific aspects makes it easier for the reader to assess the findings, and consolidating this information into one figure saves space and makes it easy to locate the most relevant results.

Following this overall summary, the relevant data in the tables is broken down into greater detail in text form in the Results section.

• “Results on the bio-accumulation of cadmium were found to be the highest (17.5 mg kgG1) in the bulb, when the concentration of cadmium in the solution was 1×10G2 M and lowest (0.11 mg kgG1) in the leaves when the concentration was 1×10G3 M.”

Captioning and Referencing Tables and Figures

Tables and figures are central components of your Results section and you need to carefully think about the most effective way to use graphs and tables to present your findings . Therefore, it is crucial to know how to write strong figure captions and to refer to them within the text of the Results section.

The most important advice one can give here as well as throughout the paper is to check the requirements and standards of the journal to which you are submitting your work. Every journal has its own design and layout standards, which you can find in the author instructions on the target journal’s website. Perusing a journal’s published articles will also give you an idea of the proper number, size, and complexity of your figures.

Regardless of which format you use, the figures should be placed in the order they are referenced in the Results section and be as clear and easy to understand as possible. If there are multiple variables being considered (within one or more research questions), it can be a good idea to split these up into separate figures. Subsequently, these can be referenced and analyzed under separate headings and paragraphs in the text.

To create a caption, consider the research question being asked and change it into a phrase. For instance, if one question is “Which color did participants choose?”, the caption might be “Color choice by participant group.” Or in our last research paper example, where the question was “What is the concentration of cadmium in different parts of the onion after 14 days?” the caption reads:

 “Fig. 1(a-c): Mean concentration of Cd determined in (a) bulbs, (b) leaves, and (c) roots of onions after a 14-day period.”

Steps for Composing the Results Section

Because each study is unique, there is no one-size-fits-all approach when it comes to designing a strategy for structuring and writing the section of a research paper where findings are presented. The content and layout of this section will be determined by the specific area of research, the design of the study and its particular methodologies, and the guidelines of the target journal and its editors. However, the following steps can be used to compose the results of most scientific research studies and are essential for researchers who are new to preparing a manuscript for publication or who need a reminder of how to construct the Results section.

Step 1 : Consult the guidelines or instructions that the target journal or publisher provides authors and read research papers it has published, especially those with similar topics, methods, or results to your study.

• The guidelines will generally outline specific requirements for the results or findings section, and the published articles will provide sound examples of successful approaches.
• Note length limitations on restrictions on content. For instance, while many journals require the Results and Discussion sections to be separate, others do not—qualitative research papers often include results and interpretations in the same section (“Results and Discussion”).
• Reading the aims and scope in the journal’s “ guide for authors ” section and understanding the interests of its readers will be invaluable in preparing to write the Results section.

Step 2 : Consider your research results in relation to the journal’s requirements and catalogue your results.

• Focus on experimental results and other findings that are especially relevant to your research questions and objectives and include them even if they are unexpected or do not support your ideas and hypotheses.
• Catalogue your findings—use subheadings to streamline and clarify your report. This will help you avoid excessive and peripheral details as you write and also help your reader understand and remember your findings. Create appendices that might interest specialists but prove too long or distracting for other readers.
• Decide how you will structure of your results. You might match the order of the research questions and hypotheses to your results, or you could arrange them according to the order presented in the Methods section. A chronological order or even a hierarchy of importance or meaningful grouping of main themes or categories might prove effective. Consider your audience, evidence, and most importantly, the objectives of your research when choosing a structure for presenting your findings.

Step 3 : Design figures and tables to present and illustrate your data.

• Tables and figures should be numbered according to the order in which they are mentioned in the main text of the paper.
• Information in figures should be relatively self-explanatory (with the aid of captions), and their design should include all definitions and other information necessary for readers to understand the findings without reading all of the text.
• Use tables and figures as a focal point to tell a clear and informative story about your research and avoid repeating information. But remember that while figures clarify and enhance the text, they cannot replace it.

Step 4 : Draft your Results section using the findings and figures you have organized.

• The goal is to communicate this complex information as clearly and precisely as possible; precise and compact phrases and sentences are most effective.
• In the opening paragraph of this section, restate your research questions or aims to focus the reader’s attention to what the results are trying to show. It is also a good idea to summarize key findings at the end of this section to create a logical transition to the interpretation and discussion that follows.
• Try to write in the past tense and the active voice to relay the findings since the research has already been done and the agent is usually clear. This will ensure that your explanations are also clear and logical.
• Make sure that any specialized terminology or abbreviation you have used here has been defined and clarified in the  Introduction section .

Step 5 : Review your draft; edit and revise until it reports results exactly as you would like to have them reported to your readers.

• Double-check the accuracy and consistency of all the data, as well as all of the visual elements included.
• Read your draft aloud to catch language errors (grammar, spelling, and mechanics), awkward phrases, and missing transitions.
• Ensure that your results are presented in the best order to focus on objectives and prepare readers for interpretations, valuations, and recommendations in the Discussion section . Look back over the paper’s Introduction and background while anticipating the Discussion and Conclusion sections to ensure that the presentation of your results is consistent and effective.
• Consider seeking additional guidance on your paper. Find additional readers to look over your Results section and see if it can be improved in any way. Peers, professors, or qualified experts can provide valuable insights.

One excellent option is to use a professional English proofreading and editing service  such as Wordvice, including our paper editing service . With hundreds of qualified editors from dozens of scientific fields, Wordvice has helped thousands of authors revise their manuscripts and get accepted into their target journals. Read more about the  proofreading and editing process  before proceeding with getting academic editing services and manuscript editing services for your manuscript.

As the representation of your study’s data output, the Results section presents the core information in your research paper. By writing with clarity and conciseness and by highlighting and explaining the crucial findings of their study, authors increase the impact and effectiveness of their research manuscripts.

For more articles and videos on writing your research manuscript, visit Wordvice’s Resources page.

Wordvice Resources

• How to Write a Research Paper Introduction 
• Which Verb Tenses to Use in a Research Paper
• How to Write an Abstract for a Research Paper
• How to Write a Research Paper Title
• Useful Phrases for Academic Writing
• Common Transition Terms in Academic Papers
• Active and Passive Voice in Research Papers
• 100+ Verbs That Will Make Your Research Writing Amazing
• Tips for Paraphrasing in Research Papers

• Langson Library
• Science Library
• Grunigen Medical Library
• Law Library
• Connect From Off-Campus
• Accessibility
• Gateway Study Center

Email this link

Writing a scientific paper.

• Writing a lab report
• INTRODUCTION

Writing a "good" results section

Figures and Captions in Lab Reports

"Results Checklist" from: How to Write a Good Scientific Paper. Chris A. Mack. SPIE. 2018.

Additional tips for results sections.

• LITERATURE CITED
• Bibliography of guides to scientific writing and presenting
• Peer Review
• Presentations
• Lab Report Writing Guides on the Web

This is the core of the paper. Don't start the results sections with methods you left out of the Materials and Methods section. You need to give an overall description of the experiments and present the data you found.

• Factual statements supported by evidence. Short and sweet without excess words
• Present representative data rather than endlessly repetitive data
• Discuss variables only if they had an effect (positive or negative)
• Use meaningful statistics
• Avoid redundancy. If it is in the tables or captions you may not need to repeat it

A short article by Dr. Brett Couch and Dr. Deena Wassenberg, Biology Program, University of Minnesota

• Present the results of the paper, in logical order, using tables and graphs as necessary.
• Explain the results and show how they help to answer the research questions posed in the Introduction. Evidence does not explain itself; the results must be presented and then explained. 
• Avoid: presenting results that are never discussed;  presenting results in chronological order rather than logical order; ignoring results that do not support the conclusions; 
• Number tables and figures separately beginning with 1 (i.e. Table 1, Table 2, Figure 1, etc.).
• Do not attempt to evaluate the results in this section. Report only what you found; hold all discussion of the significance of the results for the Discussion section.
• It is not necessary to describe every step of your statistical analyses. Scientists understand all about null hypotheses, rejection rules, and so forth and do not need to be reminded of them. Just say something like, "Honeybees did not use the flowers in proportion to their availability (X2 = 7.9, p<0.05, d.f.= 4, chi-square test)." Likewise, cite tables and figures without describing in detail how the data were manipulated. Explanations of this sort should appear in a legend or caption written on the same page as the figure or table.
• You must refer in the text to each figure or table you include in your paper.
• Tables generally should report summary-level data, such as means ± standard deviations, rather than all your raw data.  A long list of all your individual observations will mean much less than a few concise, easy-to-read tables or figures that bring out the main findings of your study.  
• Only use a figure (graph) when the data lend themselves to a good visual representation.  Avoid using figures that show too many variables or trends at once, because they can be hard to understand.

From:  https://writingcenter.gmu.edu/guides/imrad-results-discussion

• << Previous: METHODS
• Next: DISCUSSION >>
• Last Updated: Aug 4, 2023 9:33 AM
• URL: https://guides.lib.uci.edu/scientificwriting

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

12.3 Expressing Your Results

Learning objectives.

• Write out simple descriptive statistics in American Psychological Association (APA) style.
• Interpret and create simple APA-style graphs—including bar graphs, line graphs, and scatterplots.
• Interpret and create simple APA-style tables—including tables of group or condition means and correlation matrixes.

Once you have conducted your descriptive statistical analyses, you will need to present them to others. In this section, we focus on presenting descriptive statistical results in writing, in graphs, and in tables—following American Psychological Association (APA) guidelines for written research reports. These principles can be adapted easily to other presentation formats such as posters and slide show presentations.

Presenting Descriptive Statistics in Writing

When you have a small number of results to report, it is often most efficient to write them out. There are a few important APA style guidelines here. First, statistical results are always presented in the form of numerals rather than words and are usually rounded to two decimal places (e.g., “2.00” rather than “two” or “2”). They can be presented either in the narrative description of the results or parenthetically—much like reference citations. Here are some examples:

The mean age of the participants was 22.43 years with a standard deviation of 2.34.

Among the low self-esteem participants, those in a negative mood expressed stronger intentions to have unprotected sex ( M = 4.05, SD = 2.32) than those in a positive mood ( M = 2.15, SD = 2.27).

The treatment group had a mean of 23.40 ( SD = 9.33), while the control group had a mean of 20.87 ( SD = 8.45).

The test-retest correlation was .96.

There was a moderate negative correlation between the alphabetical position of respondents’ last names and their response time ( r = −.27).

Notice that when presented in the narrative, the terms mean and standard deviation are written out, but when presented parenthetically, the symbols M and SD are used instead. Notice also that it is especially important to use parallel construction to express similar or comparable results in similar ways. The third example is much better than the following nonparallel alternative:

The treatment group had a mean of 23.40 ( SD = 9.33), while 20.87 was the mean of the control group, which had a standard deviation of 8.45.

Presenting Descriptive Statistics in Graphs

When you have a large number of results to report, you can often do it more clearly and efficiently with a graph. When you prepare graphs for an APA-style research report, there are some general guidelines that you should keep in mind. First, the graph should always add important information rather than repeat information that already appears in the text or in a table. (If a graph presents information more clearly or efficiently, then you should keep the graph and eliminate the text or table.) Second, graphs should be as simple as possible. For example, the Publication Manual discourages the use of color unless it is absolutely necessary (although color can still be an effective element in posters, slide show presentations, or textbooks.) Third, graphs should be interpretable on their own. A reader should be able to understand the basic result based only on the graph and its caption and should not have to refer to the text for an explanation.

There are also several more technical guidelines for graphs that include the following:

• The graph should be slightly wider than it is tall.
• The independent variable should be plotted on the x- axis and the dependent variable on the y- axis.
• Values should increase from left to right on the x- axis and from bottom to top on the y- axis.

Axis Labels and Legends

• Axis labels should be clear and concise and include the units of measurement if they do not appear in the caption.
• Axis labels should be parallel to the axis.
• Legends should appear within the boundaries of the graph.
• Text should be in the same simple font throughout and differ by no more than four points.
• Captions should briefly describe the figure, explain any abbreviations, and include the units of measurement if they do not appear in the axis labels.
• Captions in an APA manuscript should be typed on a separate page that appears at the end of the manuscript. See Chapter 11 “Presenting Your Research” for more information.

As we have seen throughout this book, bar graphs are generally used to present and compare the mean scores for two or more groups or conditions. The bar graph in Figure 12.12 “Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues” is an APA-style version of Figure 12.5 “Bar Graph Showing Mean Clinician Phobia Ratings for Children in Two Treatment Conditions” . Notice that it conforms to all the guidelines listed. A new element in Figure 12.12 “Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues” is the smaller vertical bars that extend both upward and downward from the top of each main bar. These are error bars , and they represent the variability in each group or condition. Although they sometimes extend one standard deviation in each direction, they are more likely to extend one standard error in each direction (as in Figure 12.12 “Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues” ). The standard error is the standard deviation of the group divided by the square root of the sample size of the group. The standard error is used because, in general, a difference between group means that is greater than two standard errors is statistically significant. Thus one can “see” whether a difference is statistically significant based on a bar graph with error bars.

Figure 12.12 Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues

Line Graphs

Line graphs are used to present correlations between quantitative variables when the independent variable has, or is organized into, a relatively small number of distinct levels. Each point in a line graph represents the mean score on the dependent variable for participants at one level of the independent variable. Figure 12.13 “Sample APA-Style Line Graph Based on Research by Carlson and Conard” is an APA-style version of the results of Carlson and Conard. Notice that it includes error bars representing the standard error and conforms to all the stated guidelines.

Figure 12.13 Sample APA-Style Line Graph Based on Research by Carlson and Conard

In most cases, the information in a line graph could just as easily be presented in a bar graph. In Figure 12.13 “Sample APA-Style Line Graph Based on Research by Carlson and Conard” , for example, one could replace each point with a bar that reaches up to the same level and leave the error bars right where they are. This emphasizes the fundamental similarity of the two types of statistical relationship. Both are differences in the average score on one variable across levels of another. The convention followed by most researchers, however, is to use a bar graph when the variable plotted on the x- axis is categorical and a line graph when it is quantitative.

Scatterplots

Scatterplots are used to present relationships between quantitative variables when the variable on the x- axis (typically the independent variable) has a large number of levels. Each point in a scatterplot represents an individual rather than the mean for a group of individuals, and there are no lines connecting the points. The graph in Figure 12.14 “Sample APA-Style Scatterplot” is an APA-style version of Figure 12.8 “Statistical Relationship Between Several College Students’ Scores on the Rosenberg Self-Esteem Scale Given on Two Occasions a Week Apart” , which illustrates a few additional points. First, when the variables on the x- axis and y -axis are conceptually similar and measured on the same scale—as here, where they are measures of the same variable on two different occasions—this can be emphasized by making the axes the same length. Second, when two or more individuals fall at exactly the same point on the graph, one way this can be indicated is by offsetting the points slightly along the x- axis. Other ways are by displaying the number of individuals in parentheses next to the point or by making the point larger or darker in proportion to the number of individuals. Finally, the straight line that best fits the points in the scatterplot, which is called the regression line, can also be included.

Figure 12.14 Sample APA-Style Scatterplot

Expressing Descriptive Statistics in Tables

Like graphs, tables can be used to present large amounts of information clearly and efficiently. The same general principles apply to tables as apply to graphs. They should add important information to the presentation of your results, be as simple as possible, and be interpretable on their own. Again, we focus here on tables for an APA-style manuscript.

The most common use of tables is to present several means and standard deviations—usually for complex research designs with multiple independent and dependent variables. Figure 12.15 “Sample APA-Style Table Presenting Means and Standard Deviations” , for example, shows the results of a hypothetical study similar to the one by MacDonald and Martineau (2002) discussed in Chapter 5 “Psychological Measurement” . (The means in Figure 12.15 “Sample APA-Style Table Presenting Means and Standard Deviations” are the means reported by MacDonald and Martineau, but the standard errors are not). Recall that these researchers categorized participants as having low or high self-esteem, put them into a negative or positive mood, and measured their intentions to have unprotected sex. Although not mentioned in Chapter 5 “Psychological Measurement” , they also measured participants’ attitudes toward unprotected sex. Notice that the table includes horizontal lines spanning the entire table at the top and bottom, and just beneath the column headings. Furthermore, every column has a heading—including the leftmost column—and there are additional headings that span two or more columns that help to organize the information and present it more efficiently. Finally, notice that APA-style tables are numbered consecutively starting at 1 (Table 1, Table 2, and so on) and given a brief but clear and descriptive title.

Figure 12.15 Sample APA-Style Table Presenting Means and Standard Deviations

Another common use of tables is to present correlations—usually measured by Pearson’s r —among several variables. This is called a correlation matrix . Figure 12.16 “Sample APA-Style Table (Correlation Matrix) Based on Research by McCabe and Colleagues” is a correlation matrix based on a study by David McCabe and colleagues (McCabe, Roediger, McDaniel, Balota, & Hambrick, 2010). They were interested in the relationships between working memory and several other variables. We can see from the table that the correlation between working memory and executive function, for example, was an extremely strong .96, that the correlation between working memory and vocabulary was a medium .27, and that all the measures except vocabulary tend to decline with age. Notice here that only half the table is filled in because the other half would have identical values. For example, the Pearson’s r value in the upper right corner (working memory and age) would be the same as the one in the lower left corner (age and working memory). The correlation of a variable with itself is always 1.00, so these values are replaced by dashes to make the table easier to read.

Figure 12.16 Sample APA-Style Table (Correlation Matrix) Based on Research by McCabe and Colleagues

As with graphs, precise statistical results that appear in a table do not need to be repeated in the text. Instead, the writer can note major trends and alert the reader to details (e.g., specific correlations) that are of particular interest.

Key Takeaways

• In an APA-style article, simple results are most efficiently presented in the text, while more complex results are most efficiently presented in graphs or tables.
• APA style includes several rules for presenting numerical results in the text. These include using words only for numbers less than 10 that do not represent precise statistical results, and rounding results to two decimal places, using words (e.g., “mean”) in the text and symbols (e.g., “ M ”) in parentheses.
• APA style includes several rules for presenting results in graphs and tables. Graphs and tables should add information rather than repeating information, be as simple as possible, and be interpretable on their own with a descriptive caption (for graphs) or a descriptive title (for tables).
• Practice: In a classic study, men and women rated the importance of physical attractiveness in both a short-term mate and a long-term mate (Buss & Schmitt, 1993). The means and standard deviations are as follows. Men / Short Term: M = 5.67, SD = 2.34; Men / Long Term: M = 4.43, SD = 2.11; Women / Short Term: M = 5.67, SD = 2.48; Women / Long Term: M = 4.22, SD = 1.98. Present these results (a) in writing, (b) in a graph, and (c) in a table.

Buss, D. M., & Schmitt, D. P. (1993). Sexual strategies theory: A contextual evolutionary analysis of human mating. Psychological Review, 100 , 204–232.

MacDonald, T. K., & Martineau, A. M. (2002). Self-esteem, mood, and intentions to use condoms: When does low self-esteem lead to risky health behaviors? Journal of Experimental Social Psychology, 38 , 299–306.

McCabe, D. P., Roediger, H. L., McDaniel, M. A., Balota, D. A., & Hambrick, D. Z. (2010). The relationship between working memory capacity and executive functioning. Neuropsychology, 243 , 222–243.

Research Methods in Psychology Copyright © 2016 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Generate accurate APA citations for free

• Knowledge Base
• APA Style 7th edition
• Reporting Statistics in APA Style | Guidelines & Examples

Reporting Statistics in APA Style | Guidelines & Examples

Published on April 1, 2021 by Pritha Bhandari . Revised on January 17, 2024.

The APA Publication Manual is commonly used for reporting research results in the social and natural sciences. This article walks you through APA Style standards for reporting statistics in academic writing.

Statistical analysis involves gathering and testing quantitative data to make inferences about the world. A statistic is any number that describes a sample : it can be a proportion, a range , or a measurement, among other things.

When reporting statistics, use these formatting rules and suggestions from APA where relevant.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

Numbers and measurements, decimal places and leading zeros, formatting mathematical formulas, formatting statistical terms, reporting means and standard deviations, reporting chi-square tests, reporting z tests and t tests, reporting analysis of variance (anovas), reporting correlations, reporting regressions, reporting confidence intervals, other interesting articles, frequently asked questions about apa style statistics.

In general, APA advises using words for numbers under 10 and numerals for 10 and greater . However, always spell out a number that appears at the start of a sentence (or rephrase).

You should always use numerals for:

• Exact numbers before units of measurement or time
• Mathematical equations
• Percentages and percentiles
• Ratios, decimals, and uncommon fractions
• Scores and points on scales (e.g., 7-point scale)
• Exact amounts of money

Units of measurement and time

Report exact measurements using numerals, and use symbols or abbreviations for common units of measurement when they accompany exact measurements. Include a space between the number and the abbreviation.

When stating approximate figures, use words to express numbers under 10, and spell out the names of units of measurement.

• The ball weighed 7 kg.
• The ball weighed approximately seven kilograms.

Measurements should be reported in metric units. If you recorded measurements in non-metric units, include metric equivalents in your report as well as the original units.

Percentages

Use numerals for percentages along with the percent symbol (%). Don’t insert a space between the number and the symbol.

Words for “percent” or “percentage” should only be used in text when numbers aren’t used, or when a percentage appears at the start of a sentence.

• Of these respondents, 15% agreed with the statement.
• Fifteen percent of respondents agreed with the statement.
• The percentage was higher in 2020.

Prevent plagiarism. Run a free check.

The number of decimal places to report depends on what you’re reporting. Generally, you should aim to round numbers while retaining precision. It’s best to present fewer decimal digits to aid easy understanding.

The following guidelines are usually applicable.

Use two or three decimal places and report exact values for all p values greater than .001. For p values smaller than .001, report them as p < .001.

Leading zeros

A leading zero is zero before the decimal point for numbers less than one. In APA Style, it’s only used in some cases.

Use a leading zero only when the statistic you’re describing can be greater than one. If it can never exceed one, omit the leading zero.

• Consumers reported high satisfaction with the services ( M = 4.1, SD = 0.8).
• The correlation was medium-sized ( r = .35).
• Although significant results were obtained, the effect was relatively small ( p = .015, d = 0.11).

Provide formulas only when you use new or uncommon equations. For short equations, present them within one line in the main text whenever possible.

Make the order of operations as clear as possible by using parentheses (round brackets) for the first step, brackets [square brackets] for the second step, and braces {curly brackets} for the third step, where necessary.

More complex equations, or equations that take more than one line, should be displayed on their own lines. Equations should be displayed and numbered if you will reference them later on, regardless of their complexity. Number equations by placing the numbers in parentheses near the right edge of the page.

When reporting statistical results , present information in easily understandable ways. You can use a mix of text, tables, and figures to present data effectively when you have a lot of numbers to report.

In your main text, use helpful words like “respectively” or “in order”  to aid understanding when listing several statistics in a sequence.

The APA manual provides guidelines for dealing with statistical terms, symbols and abbreviations.

Symbols and abbreviations

Population parameters are often represented with Greek letters, while sample statistics are often represented with italicized Latin letters.

Use the population symbol ( N ) for the total number of elements in a sample, and use the sample symbol ( n ) for the number of elements in each subgroup of the full sample.

In general, abbreviations should be defined on first use, but this isn’t always the case for common statistical abbreviations.

Capitalization, italicization and hyphenation

Statistical terms such as t test, z test, and p value always begin with a lowercase, italicized letter. Never begin a sentence with lowercase statistical abbreviations.

These statistical terms should only be hyphenated when they modify a subsequent word (e.g., “ z -test results” versus results of “ z tests”).

You can form plurals of statistical symbols (e.g., M or p ) by adding a non-italicized “s” to the end with no apostrophe (e.g., M s or p s).

In general, the following guidelines apply.

Parentheses vs. brackets

Always aim to avoid nested parentheses and brackets when reporting statistics. Instead, you should use commas to separate related statistics.

• Scores improved between the pretest and posttest ( p < .001).
• Significant differences in test scores were recorded, F (1, 30) = 4.67, p = .003.
• (A previous meta-analysis highlighted low effect sizes [ d = 0.1] in the field).

Report descriptive statistics to summarize your data. Quantitative data is often reported using means and standard deviations, while categorical data (e.g., demographic variables) is reported using proportions.

Means and standard deviations can be presented in the main text and/or in parentheses. You don’t need to repeat the units of measurement (e.g., centimeters) for statistics relating to the same data.

• Average sample height was 136.4 cm ( SD = 15.1).
• The height of the initial sample was relatively low ( M = 125.9 cm, SD = 16.6).
• Height significantly varied between children aged 5–7, 8–10, and 11–13. The means were 115.3, 133.5, and 149.1 cm, respectively.

To report the results of a chi-square test , include the following:

• the degrees of freedom ( df ) in parentheses
• the chi-square (Χ 2 ) value (also referred to as the chi-square test statistic)
• the p value
• A chi-square test of independence revealed a significant association between gender and product preference, Χ 2 (8) = 19.7, p = .012.
• Based on a chi-square test of goodness of fit , Χ 2 (4) = 11.34, p = .023, the sample’s distribution of religious affiliations matched that of the population’s.

For z tests

To report the results of a z test, include the following:

• the z value (also referred to as the z statistic or z score)
• The participants’ scores were higher than the population average, z = 2.48, p = .013.
• Higher scores were obtained on the new 20-item scale compared to the previous 40-item scale, z = 2.67, p = .007.

For t tests

To report the results of a t test , include the following:

• the t value (also referred to as the t statistic)
• Older adults experienced significantly more loneliness than younger adults, t (32) = 2.94, p = .006.
• Reaction times were significantly faster for mice in the experimental condition, t (53) = 5.94, p < .001.

To report the results of an ANOVA , include the following:

• the degrees of freedom (between groups, within groups) in parentheses
• the F value (also referred to as the F statistic)
• A one-way ANOVA demonstrated that the effect of leadership style was significant for employee engagement, F (2, 78) = 4.58, p = .013.
• We found a statistically significant main effect of age group on social media use, F (3, 117) = 3.19, p = .026.

To report the results of a correlation, include the following:

• the degrees of freedom in parentheses
• the r value (the correlation coefficient)
• We found a strong correlation between average temperature and new daily cases of COVID-19, r (357) = .42, p < .001.

Results of regression analyses are often displayed in a table because the output includes many numbers.

To report the results of a regression analysis in the text, include the following:

• the R 2 value (the coefficient of determination)

The format is usually:

• SAT scores predicted college GPA, R 2 = .34, F (1, 416) = 6.71, p = .009.

You should report confidence intervals of effect sizes (e.g., Cohen’s d ) or point estimates where relevant.

To report a confidence interval, state the confidence level and use brackets to enclose the lower and upper limits of the confidence interval, separated by a comma.

• Older adults experienced significantly more loneliness than younger adults, t (32) = 2.94, p = .006, d = 0.81, 95% CI [0.6, 1.02].
• On average, the treatment resulted in a 30% reduction in migraine frequency, 99% CI [26.5, 33.5].

When presenting multiple confidence intervals with the same confidence levels in a sequence, don’t repeat the confidence level or the word “CI.”

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square test of independence
• Statistical power
• Descriptive statistics
• Degrees of freedom
• Pearson correlation
• Null hypothesis

Methodology

• Double-blind study
• Case-control study
• Research ethics
• Data collection
• Hypothesis testing
• Structured interviews

Research bias

• Hawthorne effect
• Unconscious bias
• Recall bias
• Halo effect
• Self-serving bias
• Information bias

According to the APA guidelines, you should report enough detail on inferential statistics so that your readers understand your analyses.

Report the following for each hypothesis test:

• the test statistic value
• the degrees of freedom
• the exact p value (unless it is less than 0.001)
• the magnitude and direction of the effect

You should also present confidence intervals and estimates of effect sizes where relevant.

Use one decimal place for:

• Standard deviations
• Descriptive statistics based on discrete data

Use two decimal places for:

• Correlation coefficients
• Proportions
• Inferential test statistics such as t values, F values, and chi-squares.

In APA style, statistics can be presented in the main text or as tables or figures . To decide how to present numbers, you can follow APA guidelines:

• To present three or fewer numbers, try a sentence,
• To present between 4 and 20 numbers, try a table,
• To present more than 20 numbers, try a figure.

Since these are general guidelines, use your own judgment and feedback from others for effective presentation of numbers.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2024, January 17). Reporting Statistics in APA Style | Guidelines & Examples. Scribbr. Retrieved April 9, 2024, from https://www.scribbr.com/apa-style/numbers-and-statistics/

Is this article helpful?

Pritha Bhandari

Other students also liked, how to write an apa results section, how to write an apa methods section, apa format for academic papers and essays, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

Organizing Academic Research Papers: 7. The Results

• Purpose of Guide
• Design Flaws to Avoid
• Glossary of Research Terms
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Executive Summary
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tertiary Sources
• What Is Scholarly vs. Popular?
• Qualitative Methods
• Quantitative Methods
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Annotated Bibliography
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• How to Manage Group Projects
• Multiple Book Review Essay
• Reviewing Collected Essays
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Research Proposal
• Acknowledgements

The results section of the research paper is where you report the findings of your study based upon the information gathered as a result of the methodology [or methodologies] you applied. The results section should simply state the findings, without bias or interpretation, and arranged in a logical sequence. The results section should always be written in the past tense. A section describing results [a.k.a., "findings"] is particularly necessary if your paper includes data generated from your own research.

Importance of a Good Results Section

When formulating the results section, it's important to remember that the results of a study do not prove anything . Research results can only confirm or reject the research problem underpinning your study. However, the act of articulating the results helps you to understand the problem from within, to break it into pieces, and to view the research problem from various perspectives.

The page length of this section is set by the amount and types of data to be reported . Be concise, using non-textual elements, such as figures and tables, if appropriate, to present results more effectively. In deciding what data to describe in your results section, you must clearly distinguish material that would normally be included in a research paper from any raw data or other material that could be included as an appendix. In general, raw data should not be included in the main text of your paper unless requested to do so by your professor.

Avoid providing data that is not critical to answering the research question . The background information you described in the introduction section should provide the reader with any additional context or explanation needed to understand the results. A good rule is to always re-read the background section of your paper after you have written up your results to ensure that the reader has enough context to understand the results [and, later, how you interpreted the results in the discussion section of your paper].

Bates College; Burton, Neil et al. Doing Your Education Research Project . Los Angeles, CA: SAGE, 2008; Results . The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Structure and Writing Style

I. Structure and Approach

For most research paper formats, there are two ways of presenting and organizing the results .

• Present the results followed by a short explanation of the findings . For example, you may have noticed an unusual correlation between two variables during the analysis of your findings. It is correct to point this out in the results section. However, speculating as to why this correlation exists, and offering a hypothesis about what may be happening, belongs in the discussion section of your paper.
• Present a section and then discuss it, before presenting the next section then discussing it, and so on . This is more common in longer papers because it helps the reader to better understand each finding. In this model, it can be helpful to provide a brief conclusion in the results section that ties each of the findings together and links to the discussion.

NOTE: The discussion section should generally follow the same format chosen in presenting and organizing the results.

II.  Content

In general, the content of your results section should include the following elements:

• An introductory context for understanding the results by restating the research problem that underpins the purpose of your study.
• A summary of your key findings arranged in a logical sequence that generally follows your methodology section.
• Inclusion of non-textual elements, such as, figures, charts, photos, maps, tables, etc. to further illustrate the findings, if appropriate.
• In the text, a systematic description of your results, highlighting for the reader observations that are most relevant to the topic under investigation [remember that not all results that emerge from the methodology that you used to gather the data may be relevant].
• Use of the past tense when refering to your results.
• The page length of your results section is guided by the amount and types of data to be reported. However, focus only on findings that are important and related to addressing the research problem.

Using Non-textual Elements

• Either place figures, tables, charts, etc. within the text of the result, or include them in the back of the report--do one or the other but never do both.
• In the text, refer to each non-textual element in numbered order [e.g.,  Table 1, Table 2; Chart 1, Chart 2; Map 1, Map 2].
• If you place non-textual elements at the end of the report, make sure they are clearly distinguished from any attached appendix materials, such as raw data.
• Regardless of placement, each non-textual element must be numbered consecutively and complete with caption [caption goes under the figure, table, chart, etc.]
• Each non-textual element must be titled, numbered consecutively, and complete with a heading [title with description goes above the figure, table, chart, etc.].
• In proofreading your results section, be sure that each non-textual element is sufficiently complete so that it could stand on its own, separate from the text.

III. Problems to Avoid

When writing the results section, avoid doing the following :

• Discussing or interpreting your results . Save all this for the next section of your paper, although where appropriate, you should compare or contrast specific results to those found in other studies [e.g., "Similar to Smith [1990], one of the findings of this study is the strong correlation between motivation and academic achievement...."].
• Reporting background information or attempting to explain your findings ; this should have been done in your Introduction section, but don't panic! Often the results of a study point to the need to provide additional background information or to explain the topic further, so don't think you did something wrong. Revise your introduction as needed.
• Ignoring negative results . If some of your results fail to support your hypothesis, do not ignore them. Document them, then state in your discussion section why you believe a negative result emerged from your study. Note that negative results, and how you handle them, often provides you with the opportunity to write a more engaging discussion section, therefore, don't be afraid to highlight them.
• Including raw data or intermediate calculations . Ask your professor if you need to include any raw data generated by your study, such as transcripts from interviews or data files. If raw data is to be included, place it in an appendix or set of appendices that are referred to in the text.
• Be as factual and concise as possible in reporting your findings . Do not use phrases that are vague or non-specific, such as, "appeared to be greater or lesser than..." or "demonstrates promising trends that...."
• Presenting the same data or repeating the same information more than once . If you feel the need to highlight something, you will have a chance to do that in the discussion section.
• Confusing figures with tables . Be sure to properly label any non-textual elements in your paper. If you are not sure, look up the term in a dictionary.

Burton, Neil et al. Doing Your Education Research Project . Los Angeles, CA: SAGE, 2008;  Caprette, David R. Writing Research Papers . Experimental Biosciences Resources. Rice University; Hancock, Dawson R. and Bob Algozzine. Doing Case Study Research: A Practical Guide for Beginning Researchers . 2nd ed. New York: Teachers College Press, 2011; Introduction to Nursing Research: Reporting Research Findings. Nursing Research: Open Access Nursing Research and Review Articles. (January 4, 2012); Reporting Research Findings. Wilder Research, in partnership with the Minnesota Department of Human Services. (February 2009); Results . The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College; Schafer, Mickey S. Writing the Results . Thesis Writing in the Sciences. Course Syllabus. University of Florida.

Writing Tip

Why Don't I Just Combine the Results Section with the Discussion Section?

It's not unusual to find articles in social science journals where the author(s) have combined a description of the findings from the study with a discussion about their implications. You could do this. However, if you are inexperienced writing research papers, consider creating two sections for each element in your paper as a way to better organize your thoughts and, by extension, your  paper. Think of the results section as the place where you report what your study found; think of the discussion section as the place where you interpret your data and answer the "so what?" question. As you become more skilled writing research papers, you may want to meld the results of your study with a discussion of its implications.

• << Previous: Quantitative Methods
• Next: Using Non-Textual Elements >>
• Last Updated: Jul 18, 2023 11:58 AM
• URL: https://library.sacredheart.edu/c.php?g=29803
• QuickSearch
• Library Catalog
• Databases A-Z
• Publication Finder
• Course Reserves
• Citation Linker
• Digital Commons
• Our Website

Research Support

• Ask a Librarian
• Appointments
• Interlibrary Loan (ILL)
• Research Guides
• Databases by Subject
• Citation Help

Using the Library

• Reserve a Group Study Room
• Renew Books
• Honors Study Rooms
• Off-Campus Access
• Library Policies
• Library Technology

User Information

• Grad Students
• Online Students
• COVID-19 Updates
• Staff Directory
• News & Announcements
• Library Newsletter

My Accounts

• Interlibrary Loan
• Staff Site Login

FIND US ON  

Understanding the Interpretation of Results in Research

Doing the interpretation of results in research is crucial to obtaining valuable findings. Learn how to achieve a good interpretation here!

Research is a powerful tool for gaining insights into the world around us. Whether in academia, industry, or the public sector, research studies can inform decision-making, drive innovation, and improve our understanding of complex phenomena. However, the value of research lies not only in the data collected but also in the interpretation of results. Properly interpreting research findings is critical to extracting meaningful insights, drawing accurate conclusions, and informing future research directions. 

In this Mind the Graph article, you’ll understand the basic concept of interpretation of results in research. The article will go over the right procedure for checking, cleaning, and editing your data as well as how to organize it effectively to aid interpretation.

What is the interpretation of results in research?

The process of interpreting and making meaning of data produced in a research study is known as research result interpretation. It entails studying the data’s patterns, trends, and correlations in order to develop reliable findings and make meaningful conclusions.  

Interpretation is a crucial step in the research process as it helps researchers to determine the relevance of their results, relate them to existing knowledge, and shape subsequent research goals. A thorough interpretation of results in research may assist guarantee that the findings are legitimate and trustworthy and that they contribute to the development of knowledge in an area of study. 

The interpretation of results in research requires multiple steps, including checking, cleaning, and editing data to ensure its accuracy, and properly organizing it in order to simplify interpretation. To examine data and derive reliable findings, researchers must employ suitable statistical methods. They must additionally consider the larger ramifications of their results and how they apply to everyday scenarios. 

It’s crucial to keep in mind that coming to precise conclusions while generating meaningful inferences is an iterative process that needs thorough investigation. 

The process of checking, cleaning, and editing data

The process of data checking, cleaning, and editing may be separated into three stages: screening, diagnostic, and treatment . Each step has a distinct goal and set of tasks to verify the data’s accuracy and reliability. 

Screening phase

The screening process consists of a first inspection of the data to find any errors or anomalies. Running basic descriptive statistics, reviewing data distributions, and discovering missing values may all be part of this. This phase’s goal is to discover any concerns with the data that need to be investigated further.

Diagnostic phase

The diagnostic phase entails a more extensive review of the data to identify particular concerns that must be addressed. Identifying outliers, investigating relationships between variables, and spotting abnormalities in the data are all examples of this. This phase’s goal is to identify any problems with the data and propose suitable treatment options.

Treatment phase

The treatment phase entails taking action to resolve any difficulties found during the diagnostic phase. This may involve eliminating outliers, filling in missing values, transforming data, and editing data. This phase’s goal is to guarantee that the data is reliable, precise, and in the appropriate format for analysis.

Researchers may guarantee that their data is high-quality and acceptable for analysis by using a structured approach to data checking, cleaning, and editing.

How to organize data display and description?

Organizing data display and description is another critical stage in the process of analyzing study results. The format in which data is presented has a significant influence on how quickly it may be comprehended and interpreted. The following are some best practices for data display and description organization.

Best practices for qualitative data include the following:

• Use quotes and anecdotes: Use quotes and anecdotes from participants to illustrate key themes and patterns in the data.
• Group similar responses: Similar replies should be grouped together to find major themes and patterns in the data.
• Use tables: Tables to arrange and summarize major themes, categories, or subcategories revealed by the data.
• Use figures: Figures, such as charts or graphs, may help you visualize data and spot patterns or trends.
• Provide context: Explain the research project’s topic or hypothesis being examined, as well as any important background information, before presenting the findings.
• Use simple and direct language: To describe the data being given, use clear and succinct language.

Best practices for quantitative data include the following:

• Use relevant charts and graphs: Select the right chart or graph for the data being presented. A bar chart, for example, could be ideal for categorical data, but a scatter plot might be appropriate for continuous data.
• Label the axes and include a legend: Label the axes of the chart or graph and include a legend to explain any symbols or colors used. This makes it easier for readers to comprehend the information offered.
• Provide context: Give context to the data that is being given. This may include a brief summary of the research issue or hypothesis under consideration, as well as any pertinent background information.
• Use clear and succinct language: To describe the data being given, use clear and concise language. Avoid using technical jargon or complex language that readers may find difficult to grasp.
• Highlight significant findings: Highlight noteworthy findings in the provided data. Identifying any trends, patterns, or substantial disparities across groups is one example.
• Create a summary table: Provide a summary table that explains the data being provided. Key data such as means, medians, and standard deviations may be included.

3 Tips for interpretation of results in research

Here are some key tips to keep in mind when interpreting research results:  

• Keep your research question in mind: The most important piece of advice for interpreting the results is to keep your research question in mind. Your interpretation should be centered on addressing your research question, and all of your analysis should be directed in that direction.
• Consider alternate explanations: It’s critical to think about alternative explanations for your results. Ask yourself whether any other circumstances might be impacting your findings, and carefully assess them. This can assist guarantee that your interpretation is based on the evidence and not on assumptions or biases. 
• Contextualize the results: Put the results into perspective by comparing them to past research in the topic at hand. This can assist in identifying trends, patterns, or discrepancies that you may have missed otherwise, as well as providing a foundation for subsequent research. 

By following these three tips, you may assist guarantee that your interpretation of data is correct, useful, and relevant to your research topic and the larger context of your field of research.

Professional and custom designs for your publications

Mind the Graph is a sophisticated tool that provides professional and customizable research publication designs. Enhance the visual impact of your research by using eye-catching visuals, charts, and graphs. With Mind the Graph, you can simply generate visually appealing and informative publications that captivate your audience and successfully explain the research’s findings.

Subscribe to our newsletter

Exclusive high quality content about effective visual communication in science.

About Jessica Abbadia

Jessica Abbadia is a lawyer that has been working in Digital Marketing since 2020, improving organic performance for apps and websites in various regions through ASO and SEO. Currently developing scientific and intellectual knowledge for the community's benefit. Jessica is an animal rights activist who enjoys reading and drinking strong coffee.

Content tags

• How It Works
• PhD thesis writing
• Master thesis writing
• Bachelor thesis writing
• Dissertation writing service
• Dissertation abstract writing
• Thesis proposal writing
• Thesis editing service
• Thesis proofreading service
• Thesis formatting service
• Coursework writing service
• Research paper writing service
• Architecture thesis writing
• Computer science thesis writing
• Engineering thesis writing
• History thesis writing
• MBA thesis writing
• Nursing dissertation writing
• Psychology dissertation writing
• Sociology thesis writing
• Statistics dissertation writing
• Buy dissertation online
• Write my dissertation
• Cheap thesis
• Cheap dissertation
• Custom dissertation
• Dissertation help
• Pay for thesis
• Pay for dissertation
• Senior thesis
• Write my thesis

Results Section Of A Research Paper: How To Write It Properly

The results section of a research paper refers to the part that represents the study’s core findings from the methods that the researcher used to collect and analyze data. This section presents the results logically without interpretation or bias from the author.

Thus, this part of a research paper sets up the read for evaluation and analysis of the findings in the discussion section. Essentially, this section breaks down the information into several sentences, showing its importance to the research question. Writing results section in a research paper entails summarizing the gathered data and the performed statistical analysis. That way, the author presents or reports the results without subjective interpretation.

What Is The Results Section Of A Research Paper?

In its simplest definition, a research paper results section is where the researcher reports the findings of a study based on the applied methodology for gathering information. It’s the part where the author states the research findings in a logical sequence without interpreting them. If the research paper has data from actual research, this section should feature a detailed description of the results.

When writing a dissertation, a thesis, or any other academic paper, the result section should come third in sections’ sequence. It should follow the Methods and Materials presentation and the Discussion section comes after it. But most scientific papers present the Results and Discussion sections together. However, the results section answers the question, “What did your research uncover?”

Ideally, this section allows you to report findings in research paper, creating the basis for sufficiently justified conclusions. After writing the study findings in the results section, you interpret them in the subsequent discussion part. Therefore, your results section should report information that will justify your claims. That way, you can look back on the results section when writing the discussion part to ensure that your report supports your conclusions.

What Goes in the Results Section of a Research Paper?

This section should present results in research paper. The findings part of a research paper can differ in structure depending on the study, discipline, and journal. Nevertheless, the results section presents a description of the experiment while presenting the research results. When writing this part of your research paper, you can use graphs and tables if necessary.

However, state the findings without interpreting them. For instance, you can find a correlation between variables when analyzing data. In that case, your results section can explain this correlation without speculating about the causes of this correlation.

Here’s what to include in the results section of research paper:

A brief introductory of the context, repeating the research questions to help the readers understand the results A report about information collection, participants, and recruitment: for instance, you can include a demographic summary with the participants’ characteristics A systematic findings’ description, with a logical presentation highlighting relevant and crucial results A contextual data analysis explaining the meaning in sentences Information corresponding to the primary research questions Secondary findings like subgroup analysis and secondary outcomes Visual elements like charts, figures, tables, and maps, illustrating and summarizing the findings

Ensure that your results section cites and numbers visual elements in an orderly manner. Every table or figure should stand alone without text. That means visual elements should have adequate non-textual content to enable the audiences to understand their meanings.

If your study has a broad scope, several variables, or used methodologies that yielded different results, state the most relevant results only based on the research question you presented in your Introduction section.

The general rule is to leave out any data that doesn’t present your study’s direct outcome or findings. Unless the professor, advisor, university faulty, or your target journal requests you to combine the Results and Discussion sections, omit the interpretations and explanations of the results in this section.

How Long Should A Results Section Be?

The findings section of a research paper ranges between two and three pages, with tables, text, and figures. In most cases, universities and journals insist that this section shouldn’t exceed 1,000 words over four to nine paragraphs, usually with no references.

But a good findings section occupies 5% of the entire paper. For instance, this section should have 500 words if a dissertation has 10,000 words. If the educator didn’t specify the number of words to include in this chapter, use the data you collect to determine its length. Nevertheless, be as concise as possible by featuring only relevant results that answer your research question.

How To Write Results Section Of Research Paper

Perhaps, you have completed researching and writing the preceding sections, and you’re now wondering how to write results. By the time you’re composing this section, you already have findings or answers to your research questions. However, you don’t even know how to start a results section. And your search for guidelines landed you on this page.

Well, every research project is different and unique. That’s why researchers use different strategies when writing this section of their research papers. The scientific or academic discipline, specialization field, target journal, and the author are factors influencing how you write this section. Nevertheless, there’s a general way of writing this section, although it might differ slightly between disciplines. Here’s how to write results section in a research paper.

Check the instructions or guidelines. Check their instructions or guidelines first, whether you’re writing the research paper as part of your coursework or for an academic journal. These guidelines outline the requirements for presenting results in research papers. Also, check the published articles to know how to approach this section. When reviewing the procedures, check content restrictions and length. Essentially, learn everything you can about this section from the instructions or guidelines before you start writing. Reflect on your research findings. With instructions and guidelines in mind, reflect on your research findings to determine how to present them in your research paper. Decide on the best way to show the results so that they can answer the research question. Also, strive to clarify and streamline your report, especially with a complex and lengthy results section. You can use subheadings to avoid peripheral and excessive details. Additionally, consider breaking down the content to make it easy for the readers to understand or remember. Your hypothesis, research question, or methodologies might influence the structure of the findings sections. Nevertheless, a hierarchy of importance, chronological order, or meaningful grouping of categories or themes can be an effective way of presenting your findings. Design your visual presentations. Visual presentations improve the textual report of the research findings. Therefore, decide on the figures and styles to use in your tables, graphs, photos, and maps. However, check the instructions and guidelines of your faculty or journal to determine the visual aids you can use. Also, check what the guidelines say about their formats and design elements. Ideally, number the figures and tables according to their mention in the text. Additionally, your figures and tables should be self-explanatory. Write your findings section. Writing the results section of a research paper entails communicating the information you gathered from your study. Ideally, be as objective and factual as possible. If you gathered complex information, try to simplify and present it accurately, precisely, and clearly. Therefore, use well-structured sentences instead of complex expressions and phrases. Also, use an active voice and past tense since you’ve already done the research. Additionally, use correct spelling, grammar, and punctuation. Take your time to present the findings in the best way possible to focus your readers on your study objectives while preparing them for the coming speculations, interpretations, and recommendations. Edit Your Findings Section. Once you’ve written the results part of your paper, please go through it to ensure that you’ve presented your study findings in the best way possible. Make sure that the content of this section is factual, accurate, and without errors. You’ve taken a considerable amount of time to compose the results scientific paper audiences will find interesting to read. Therefore, take a moment to go through the draft and eliminate all errors.

Practical Tips on How to Write a Results Section of a Research Paper

The results part of a research paper aims to present the key findings objectively in a logical and orderly sequence using text and illustrative materials. A common mistake that many authors make is confusing the information in the discussion and the results sections. To avoid this, focus on presenting your research findings without interpreting them or speculating about them.

The following tips on how to write a results section should make this task easier for you:

Summarize your study results: Instead of reporting the findings in full detail, summarize them. That way, you can develop an overview of the results. Present relevant findings only: Don’t report everything you found during your research. Instead, present pertinent information only. That means taking time to analyze your results to know what your audiences want to know. Report statistical findings: When writing this section, assume that the audiences understand statistical concepts. Therefore, don’t try to explain the nitty-gritty in this section. Remember that your work is to report your study’s findings in this section. Be objective and concise: You can interpret the findings in the discussion sections. Therefore, focus on presenting the results objectively and concisely in this section. Use the suitable format: Use the correct style to present the findings depending on your study field.

Get Professional Help with the Research Section

Maybe you’re pursuing your graduate or undergraduate studies but cannot write the results part of your paper. Perhaps, you’re done researching and analyzing information, but this section proves too tricky for you to write. Well, you’re not alone because many students across the world struggle to present their research findings.

Luckily, our highly educated, talented, and experienced writers are always ready to assist such learners. If you are stuck with the results part of your paper, our professionals can help you . We offer high-quality, custom writing help online. We’re a reliable team of experts with a sterling reputation for providing comprehensive assistance to college, high school, and university learners. We deliver highly informative academic papers after conducting extensive and in-depth research. Contact us saying something like, “please do my thesis” to get quality help with your paper!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Comment * Error message

Name * Error message

Email * Error message

Save my name, email, and website in this browser for the next time I comment.

As Putin continues killing civilians, bombing kindergartens, and threatening WWIII, Ukraine fights for the world's peaceful future.

Ukraine Live Updates

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7.1 Reading results in quantitative research

Learning objectives.

Learners will be able to…

• Describe how statistical significance and confidence intervals demonstrate which results are most important

Pre-awareness check (Knowledge)

What do you know about previously conducted research on your topic (e.g., statistical analyses, qualitative and quantitative results)?

If you recall, empirical journal articles are those that report the results of quantitative or qualitative data analyzed by the author. They follow a set structure—introduction, methods, results, discussion/conclusions. This chapter is about reading what is often the most challenging section: results.

Quantitative results

Quantitative articles often contain tables, and scanning them is a good way to begin reading the results. A table usually provides a quick, condensed summary of the report’s key findings. Tables are a concise way to report large amounts of data. Some tables present descriptive information about a researcher’s sample (often the first table in a results section). These tables will likely contain frequencies ( n ) and percentages (%). For example, if gender happened to be an important variable for the researcher’s analysis, a descriptive table would show how many and what percent of all study participants are of a particular gender. Frequencies or “how many” will probably be listed as n , while the percent symbol (%) might be used to indicate percentages. The symbol N is used for the entire sample size, and  n is used for the size of a portion of the entire sample.

In a table presenting a causal relationship, two sets of variables are represented. The independent variable , or cause, and the dependent variable , the effect. We’ll go into more detail on variables in Chapter 8. Independent variable attributes are typically presented in the table’s columns, while dependent variable attributes are presented in rows. This allows the reader to scan a table’s rows to see how values on the dependent variable change as the independent variable values change. Tables displaying results of quantitative analysis will also likely include some information about which relationships are significant or not. We will discuss the details of significance and p -values later in this section.

Let’s look at a specific example: Table 7.1 below.

Table 7.1 presents the association between gender and experiencing harassing behaviors at work. In this example, gender is the independent variable (the predictor) and the harassing behaviors listed are the dependent variables (the outcome). [1] Therefore, we place gender in the table’s columns and harassing behaviors in the table’s rows.

Reading across the table’s top row, we see that 2.9% of women in the sample reported experiencing subtle or obvious threats to their safety at work, while 4.7% of men in the sample reported the same. We can read across each of the rows of the table in this way. Reading across the bottom row, we see that 9.4% of women in the sample reported experiencing staring or invasion of their personal space at work while just 2.3% of men in the sample reported having the same experience. We’ll discuss  p- values later in this section.

While you can certainly scan tables for key results, they are often difficult to understand without reading the text of the article. The article and table were meant to complement each other, and the text should provide information on how the authors interpret their findings. The table is not redundant with the text of the results section. Additionally, the first table in most results sections is a summary of the study’s sample, which provides more background information on the study than information about hypotheses and findings. It is also a good idea to look back at the methods section of the article as the data analysis plan the authors outline should walk you through the steps they took to analyze their data which will inform how they report them in the results section.

Statistical significance

The statistics reported in Table 7.1 represent what the researchers found in their sample. The purpose of statistical analysis is usually to generalize from a the small number of people in a study’s sample to a larger population of people. Thus, the researchers intend to make causal arguments about harassing behaviors at workplaces beyond those covered in the sample.

Generalizing is key to understanding statistical significance . According to Cassidy et al. (2019), [2] 89% of research methods textbooks in psychology define statistical significance incorrectly. This includes an early draft of this textbook which defined statistical significance as “the likelihood that the relationships we observe could be caused by something other than chance.” If you have previously had a research methods class, this might sound familiar to you. It certainly did to me!

But statistical significance is less about “random chance” than more about the null hypothesis . Basically, at the beginning of a study a researcher develops a hypothesis about what they expect to find, usually that there is a statistical relationship between two or more variables . The null hypothesis is the opposite. It is the hypothesis that there is no relationship between the variables in a research study. Researchers then can hopefully reject the null hypothesis because they find a relationship between the variables.

For example, in Table 7.1 researchers were examining whether gender impacts harassment. Of course, researchers assumed that women were more likely to experience harassment than men. The null hypothesis, then, would be that gender has no impact on harassment. Once we conduct the study, our results will hopefully lead us to reject the null hypothesis because we find that gender impacts harassment. We would then generalize from our study’s sample to the larger population of people in the workplace.

Statistical significance is calculated using a p -value which is obtained by comparing the statistical results with a hypothetical set of results if the researchers re-ran their study a large number of times. Keeping with our example, imagine we re-ran our study with different men and women from different workplaces hundreds and hundred of times and we assume that the null hypothesis is true that gender has no impact on harassment. If results like ours come up pretty often when the null hypothesis is true, our results probably don’t mean much. “The smaller the p -value, the greater the statistical incompatibility with the null hypothesis” (Wasserstein & Lazar, 2016, p. 131). [3] Generally, researchers in the social sciences have set alpha at .05 for the value at which a result is significant ( p is less than or equal to .05) or not significant ( p is greater than .05). The p -value .05 refers to if less than 5% of those hypothetical results from re-running our study show the same or more extreme relationships when the null hypothesis is true. Researchers, however, may choose a stricter standard such as .01 in which 1% or less of those hypothetical results are more extreme or a more lenient standard like .1 in which 10% or less of those hypothetical results are more extreme than what was found in the study.

Let’s look back at Table 7.1. Which one of the relationships between gender and harassing behaviors is statistically significant? It’s the last one in the table, “staring or invasion of personal space,” whose p -value is .039 (under the p<.05 standard to establish statistical significance). Again, this indicates that if we re-ran our study over and over again and gender did not  impact staring/invasion of space (i.e., the null hypothesis was true), only 3.9% of the time would we find similar or more extreme differences between men and women than what we observed in our study. Thus, we conclude that for staring or invasion of space only , there is a statistically significant relationship.

For contrast, let’s look at “being pushed, hit, or grabbed” and run through the same analysis to see if it is statistically significant. If we re-ran our study over and over again and the null hypothesis was true, 48% of the time ( p =.48) we would find similar or more extreme differences between men and women. That means these results are not statistically significant.

This discussion should also highlight a point we discussed previously: that it is important to read the full results section, rather than simply relying on the summary in the abstract. If the abstract stated that most tests revealed no statistically significant relationships between gender and harassment, you would have missed the detail on which behaviors were and were not associated with gender. Read the full results section! And don’t be afraid to ask for help from a professor in understanding what you are reading, as results sections are often not written to be easily understood.

Statistical significance and p -values have been critiqued recently for a number of reasons, including that they are misused and misinterpreted (Wasserstein & Lazar, 2016) [4] , that researchers deliberately manipulate their analyses to have significant results (Head et al., 2015) [5] , and factor into the difficulty scientists have today in reproducing many of the results of previous social science studies (Peng, 2015). [6] For this reason, we share these principles, adapted from those put forth by the American Statistical Association, [7]  for understanding and using p -values in social science:

• p -values provide evidence against a null hypothesis.
• p -values do not indicate whether the results were produced by random chance alone or if the researcher’s hypothesis is true, though both are common misconceptions.
• Statistical significance can be detected in minuscule differences that have very little effect on the real world.
• Nuance is needed to interpret scientific findings, as a conclusion does not become true or false when the p -value passes from p =.051 to p =.049.
• Real-world decision-making must use more than reported p -values. It’s easy to run analyses of large datasets and only report the significant findings.
• Greater confidence can be placed in studies that pre-register their hypotheses and share their data and methods openly with the public.
• “By itself, a p -value does not provide a good measure of evidence regarding a model or hypothesis. For example, a p -value near .05 taken by itself offers only weak evidence against the null hypothesis. Likewise, a relatively large p -value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data” (Wasserstein & Lazar, 2016, p. 132).

Confidence intervals

Because of the limitations of p -values, scientists can use other methods to determine whether their models of the world are true. One common approach is to use a confidence interval , or a range of values in which the true value is likely to be found. Confidence intervals are helpful because, as principal #5 above points out, p -values do not measure the size of an effect (Greenland et al., 2016). [8] Remember, something that has very little impact on the world can be statistically significant, and the values in a confidence interval would be helpful. In our example from Table 7.1, imagine our analysis produced a confidence interval that women are 1.2-3.4 times more likely to experience “staring or invasion of personal space” than men. As with p -values, calculation for a confidence interval compares what was found in one study with a hypothetical set of results if we repeated the study over and over again. If we calculated 95% confidence intervals for all of the hypothetical set of hundreds and hundreds of studies, that would be our confidence interval. 

Confidence intervals are pretty intuitive. As of this writing, my wife and are expecting our second child. The doctor told us our due date was December 11th. But the doctor also told us that December 11th was only their best estimate. They were actually 95% sure our baby might be born any time in the 30-day period between November 27th and December 25th. Confidence intervals are often listed with a percentage, like 90% or 95%, and a range of values, such as between November 27th and December 25th. You can read that as: “we are 95% sure your baby will be born between November 27th and December 25th because we’ve studied hundreds of thousands of fetuses and mothers, and we’re 95% sure your baby will be within these two dates.”

Notice that we’re hedging our bets here by using words like “best estimate.” When testing hypotheses, social scientists generally phrase their findings in a tentative way, talking about what results “indicate” or “support,” rather than making bold statements about what their results “prove.” Social scientists have humility because they understand the limitations of their knowledge. In a literature review, using a single study or fact to “prove” an argument right or wrong is often a signal to the person reading your literature review (usually your professor) that you may not have appreciated the limitations of that study or its place in the broader literature on the topic. Strong arguments in a literature review include multiple facts and ideas that span across multiple studies.

You can learn more about creating tables, reading tables, and tests of statistical significance in a class focused exclusively on statistical analysis. We provide links to many free and openly licensed resources on statistics in Chapter 16. For now, we hope this brief introduction to reading tables will improve your confidence in reading and understanding the results sections in quantitative empirical articles.

Key Takeaways

• The results section of empirical articles are often the most difficult to understand.
• To understand a quantitative results section, look for results that were statistically significant and examine the confidence interval, if provided.

Post-awareness check (Emotional)

On a scale of 1-10 (10 being excellent), how would you rate your confidence level in your ability to understand a quantitative results section in empirical articles on your topic of interest?

TRACK 1 (IF YOU ARE CREATING A RESEARCH PROPOSAL FOR THIS CLASS)

Select a quantitative empirical article related to your topic.

• Write down the results the authors identify as statistically significant in the results section.
• How do the authors interpret their results in the discussion section?
• Do the authors provide enough information in the introduction for you to understand their results?

TRACK 2 (IF YOU  AREN’T CREATING A RESEARCH PROPOSAL FOR THIS CLASS)

You are interested in researching the effects of race-based stress and burnout among social workers.

Select a quantitative empirical article related to this topic.

• It wouldn’t make any sense to say that people’s workplace experiences predict their gender, so in this example, the question of which is the independent variable and which are the dependent variables has a pretty obvious answer. ↵
• Cassidy, S. A., Dimova, R., Giguère, B., Spence, J. R., & Stanley, D. J. (2019). Failing grade: 89% of introduction-to-psychology textbooks that define or explain statistical significance do so incorrectly. Advances in Methods and Practices in Psychological Science ,  2 (3), 233-239. ↵
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p -values: context, process, and purpose. The American Statistician, 70 , p. 129-133. ↵
• Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS biology, 13 (3). ↵
• Peng, R. (2015), The reproducibility crisis in science: A statistical counterattack. Significance , 12 , 30–32. ↵
• Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations.  European journal of epidemiology ,  31 (4), 337-350. ↵

report the results of a quantitative or qualitative data analysis conducted by the author

a quick, condensed summary of the report’s key findings arranged by row and column

causes a change in the dependent variable

a variable that depends on changes in the independent variable

(as in generalization) to make claims about a large population based on a smaller sample of people or items

"Assuming that the null hypothesis is true and the study is repeated an infinite number times by drawing random samples from the same populations(s), less than 5% of these results will be more extreme than the current result" (Cassidy et al., 2019, p. 233).

the assumption that no relationship exists between the variables in question

“a logical grouping of attributes that can be observed and measured and is expected to vary from person to person in a population” (Gillespie & Wagner, 2018, p. 9)

summarizes the incompatibility between a particular set of data and a proposed model for the data, usually the null hypothesis. The lower the p-value, the more inconsistent the data are with the null hypothesis, indicating that the relationship is statistically significant.

a range of values in which the true value is likely to be, to provide a more accurate description of their data

Doctoral Research Methods in Social Work Copyright © by Mavs Open Press. All Rights Reserved.

Share This Book

What it means when scientists say their results are ‘significant’

Post-Doctoral Researcher, South Australian Health & Medical Research Institute

Disclosure statement

Yazad Irani does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

View all partners

This article is part of the series This is research , where we ask academics to share and discuss open access articles that reveal important aspects of science. Today’s piece looks at statistics, and how to interpret them for meaning in the real world.

Let’s face it, scientific papers aren’t exactly page turners. They are written by scientists, for scientists, and often in a language that seems to only vaguely resemble English.

And perhaps one of the most daunting aspects of a scientific paper is the statistics (“stats”) section.

But what do stats really mean in the real world? Here’s an example from leukaemia research to help you break it down.

Read more: Our survey found 'questionable research practices' by ecologists and biologists – here's what that means

Strength of results

Stats are key to good research – they help researchers determine whether the results observed are strong enough to be due to an important scientific phenomenon.

As a research student I would always look for the magic number which indicates statistically significant differences in my experiments: most people agree this number to be 0.05 (you may see this in a paper written as p < 0.05).

When comparing two groups in a scientific study, statistical significance indicated by a p-value of less than 0.05 means that, in the case where there was no real difference between groups, there’s less than a 5% chance of the observed result arising.

But the focus on looking for statistically significant differences can blind us to the bigger picture. As I advanced through my scientific training I learnt to look for biologically significant differences.

Read more: The curious case of the missing workplace teaspoons

Biological significance

Biological significance addresses the question of whether the statistical difference actually means anything in terms of a real outcome, like a disease. Can the result explain how the disease is caused? Does it provide a new avenue to treat the disease? Basically, is it relevant?

A recent paper published in the journal Leukaemia will help explain my point. The paper looked at why some people are able to stop their treatment for chronic myeloid leukaemia without the cancer coming back, while in others the cancer relapsed.

The key finding of this study was that patients who did not relapse had a higher proportion of natural killer cells compared to patients that did relapse.

Natural killer cells are a type of immune cell that controls viral infections and tumours. So, the more cells there are to kill the cancer, the less likely the cancer was to relapse – makes sense!

This finding has the potential to guide doctors in seeing which patients are likely to remain cancer-free after stopping treatment. This is definitely biologically significant.

Read more: My cancer is in remission – does this mean I'm cured?

Not so relevant

Another result from the same paper (Figure 3a if you want to click through to the data) shows a statistically significant difference in a sub-type of natural killer cells (called adaptive natural killer cells). But is this difference biologically relevant?

At this stage there is little evidence of a role for adaptive natural killer cells in the context of leukaemia. Also, the difference between the groups is relatively small, with a large variation within the groups (there are large error bars on the graph).

These factors make it more likely that the differences may be due to the mathematics involved in the statistical test rather than a biological effect. As with any new finding, time and further studies will be vital in working out whether this result actually means anything.

Act like an expert

So how do you pick if the statistical differences have biological value? Being a highly trained expert in the field certainly helps.

Another way to determine if the findings in a paper have biological relevance is to look for other papers that show similar results. If a result is “real” it should be found by other scientists who will build on it and publish more papers.

This means there will be lots of papers for you to read and apply your new-found passion for statistics.

The open access research paper for this analysis is Increased proportion of mature NK cells is associated with successful imatinib discontinuation in chronic myeloid leukemia .

The definition of statistical significance has been edited since this article was first published.

• This is research

GRAINS RESEARCH AND DEVELOPMENT CORPORATION CHAIRPERSON

Project Officer, Student Program Development

Faculty of Law - Academic Appointment Opportunities

Operations Manager

Audience Development Coordinator (fixed-term maternity cover)

Jump to navigation

Cochrane Training

Chapter 15: interpreting results and drawing conclusions.

Holger J Schünemann, Gunn E Vist, Julian PT Higgins, Nancy Santesso, Jonathan J Deeks, Paul Glasziou, Elie A Akl, Gordon H Guyatt; on behalf of the Cochrane GRADEing Methods Group

Key Points:

• This chapter provides guidance on interpreting the results of synthesis in order to communicate the conclusions of the review effectively.
• Methods are presented for computing, presenting and interpreting relative and absolute effects for dichotomous outcome data, including the number needed to treat (NNT).
• For continuous outcome measures, review authors can present summary results for studies using natural units of measurement or as minimal important differences when all studies use the same scale. When studies measure the same construct but with different scales, review authors will need to find a way to interpret the standardized mean difference, or to use an alternative effect measure for the meta-analysis such as the ratio of means.
• Review authors should not describe results as ‘statistically significant’, ‘not statistically significant’ or ‘non-significant’ or unduly rely on thresholds for P values, but report the confidence interval together with the exact P value.
• Review authors should not make recommendations about healthcare decisions, but they can – after describing the certainty of evidence and the balance of benefits and harms – highlight different actions that might be consistent with particular patterns of values and preferences and other factors that determine a decision such as cost.

Cite this chapter as: Schünemann HJ, Vist GE, Higgins JPT, Santesso N, Deeks JJ, Glasziou P, Akl EA, Guyatt GH. Chapter 15: Interpreting results and drawing conclusions. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

15.1 Introduction

The purpose of Cochrane Reviews is to facilitate healthcare decisions by patients and the general public, clinicians, guideline developers, administrators and policy makers. They also inform future research. A clear statement of findings, a considered discussion and a clear presentation of the authors’ conclusions are, therefore, important parts of the review. In particular, the following issues can help people make better informed decisions and increase the usability of Cochrane Reviews:

• information on all important outcomes, including adverse outcomes;
• the certainty of the evidence for each of these outcomes, as it applies to specific populations and specific interventions; and
• clarification of the manner in which particular values and preferences may bear on the desirable and undesirable consequences of the intervention.

A ‘Summary of findings’ table, described in Chapter 14 , Section 14.1 , provides key pieces of information about health benefits and harms in a quick and accessible format. It is highly desirable that review authors include a ‘Summary of findings’ table in Cochrane Reviews alongside a sufficient description of the studies and meta-analyses to support its contents. This description includes the rating of the certainty of evidence, also called the quality of the evidence or confidence in the estimates of the effects, which is expected in all Cochrane Reviews.

‘Summary of findings’ tables are usually supported by full evidence profiles which include the detailed ratings of the evidence (Guyatt et al 2011a, Guyatt et al 2013a, Guyatt et al 2013b, Santesso et al 2016). The Discussion section of the text of the review provides space to reflect and consider the implications of these aspects of the review’s findings. Cochrane Reviews include five standard subheadings to ensure the Discussion section places the review in an appropriate context: ‘Summary of main results (benefits and harms)’; ‘Potential biases in the review process’; ‘Overall completeness and applicability of evidence’; ‘Certainty of the evidence’; and ‘Agreements and disagreements with other studies or reviews’. Following the Discussion, the Authors’ conclusions section is divided into two standard subsections: ‘Implications for practice’ and ‘Implications for research’. The assessment of the certainty of evidence facilitates a structured description of the implications for practice and research.

Because Cochrane Reviews have an international audience, the Discussion and Authors’ conclusions should, so far as possible, assume a broad international perspective and provide guidance for how the results could be applied in different settings, rather than being restricted to specific national or local circumstances. Cultural differences and economic differences may both play an important role in determining the best course of action based on the results of a Cochrane Review. Furthermore, individuals within societies have widely varying values and preferences regarding health states, and use of societal resources to achieve particular health states. For all these reasons, and because information that goes beyond that included in a Cochrane Review is required to make fully informed decisions, different people will often make different decisions based on the same evidence presented in a review.

Thus, review authors should avoid specific recommendations that inevitably depend on assumptions about available resources, values and preferences, and other factors such as equity considerations, feasibility and acceptability of an intervention. The purpose of the review should be to present information and aid interpretation rather than to offer recommendations. The discussion and conclusions should help people understand the implications of the evidence in relation to practical decisions and apply the results to their specific situation. Review authors can aid this understanding of the implications by laying out different scenarios that describe certain value structures.

In this chapter, we address first one of the key aspects of interpreting findings that is also fundamental in completing a ‘Summary of findings’ table: the certainty of evidence related to each of the outcomes. We then provide a more detailed consideration of issues around applicability and around interpretation of numerical results, and provide suggestions for presenting authors’ conclusions.

15.2 Issues of indirectness and applicability

15.2.1 the role of the review author.

“A leap of faith is always required when applying any study findings to the population at large” or to a specific person. “In making that jump, one must always strike a balance between making justifiable broad generalizations and being too conservative in one’s conclusions” (Friedman et al 1985). In addition to issues about risk of bias and other domains determining the certainty of evidence, this leap of faith is related to how well the identified body of evidence matches the posed PICO ( Population, Intervention, Comparator(s) and Outcome ) question. As to the population, no individual can be entirely matched to the population included in research studies. At the time of decision, there will always be differences between the study population and the person or population to whom the evidence is applied; sometimes these differences are slight, sometimes large.

The terms applicability, generalizability, external validity and transferability are related, sometimes used interchangeably and have in common that they lack a clear and consistent definition in the classic epidemiological literature (Schünemann et al 2013). However, all of the terms describe one overarching theme: whether or not available research evidence can be directly used to answer the health and healthcare question at hand, ideally supported by a judgement about the degree of confidence in this use (Schünemann et al 2013). GRADE’s certainty domains include a judgement about ‘indirectness’ to describe all of these aspects including the concept of direct versus indirect comparisons of different interventions (Atkins et al 2004, Guyatt et al 2008, Guyatt et al 2011b).

To address adequately the extent to which a review is relevant for the purpose to which it is being put, there are certain things the review author must do, and certain things the user of the review must do to assess the degree of indirectness. Cochrane and the GRADE Working Group suggest using a very structured framework to address indirectness. We discuss here and in Chapter 14 what the review author can do to help the user. Cochrane Review authors must be extremely clear on the population, intervention and outcomes that they intend to address. Chapter 14, Section 14.1.2 , also emphasizes a crucial step: the specification of all patient-important outcomes relevant to the intervention strategies under comparison.

In considering whether the effect of an intervention applies equally to all participants, and whether different variations on the intervention have similar effects, review authors need to make a priori hypotheses about possible effect modifiers, and then examine those hypotheses (see Chapter 10, Section 10.10 and Section 10.11 ). If they find apparent subgroup effects, they must ultimately decide whether or not these effects are credible (Sun et al 2012). Differences between subgroups, particularly those that correspond to differences between studies, should be interpreted cautiously. Some chance variation between subgroups is inevitable so, unless there is good reason to believe that there is an interaction, review authors should not assume that the subgroup effect exists. If, despite due caution, review authors judge subgroup effects in terms of relative effect estimates as credible (i.e. the effects differ credibly), they should conduct separate meta-analyses for the relevant subgroups, and produce separate ‘Summary of findings’ tables for those subgroups.

The user of the review will be challenged with ‘individualization’ of the findings, whether they seek to apply the findings to an individual patient or a policy decision in a specific context. For example, even if relative effects are similar across subgroups, absolute effects will differ according to baseline risk. Review authors can help provide this information by identifying identifiable groups of people with varying baseline risks in the ‘Summary of findings’ tables, as discussed in Chapter 14, Section 14.1.3 . Users can then identify their specific case or population as belonging to a particular risk group, if relevant, and assess their likely magnitude of benefit or harm accordingly. A description of the identifying prognostic or baseline risk factors in a brief scenario (e.g. age or gender) will help users of a review further.

Another decision users must make is whether their individual case or population of interest is so different from those included in the studies that they cannot use the results of the systematic review and meta-analysis at all. Rather than rigidly applying the inclusion and exclusion criteria of studies, it is better to ask whether or not there are compelling reasons why the evidence should not be applied to a particular patient. Review authors can sometimes help decision makers by identifying important variation where divergence might limit the applicability of results (Rothwell 2005, Schünemann et al 2006, Guyatt et al 2011b, Schünemann et al 2013), including biologic and cultural variation, and variation in adherence to an intervention.

In addressing these issues, review authors cannot be aware of, or address, the myriad of differences in circumstances around the world. They can, however, address differences of known importance to many people and, importantly, they should avoid assuming that other people’s circumstances are the same as their own in discussing the results and drawing conclusions.

15.2.2 Biological variation

Issues of biological variation that may affect the applicability of a result to a reader or population include divergence in pathophysiology (e.g. biological differences between women and men that may affect responsiveness to an intervention) and divergence in a causative agent (e.g. for infectious diseases such as malaria, which may be caused by several different parasites). The discussion of the results in the review should make clear whether the included studies addressed all or only some of these groups, and whether any important subgroup effects were found.

15.2.3 Variation in context

Some interventions, particularly non-pharmacological interventions, may work in some contexts but not in others; the situation has been described as program by context interaction (Hawe et al 2004). Contextual factors might pertain to the host organization in which an intervention is offered, such as the expertise, experience and morale of the staff expected to carry out the intervention, the competing priorities for the clinician’s or staff’s attention, the local resources such as service and facilities made available to the program and the status or importance given to the program by the host organization. Broader context issues might include aspects of the system within which the host organization operates, such as the fee or payment structure for healthcare providers and the local insurance system. Some interventions, in particular complex interventions (see Chapter 17 ), can be only partially implemented in some contexts, and this requires judgements about indirectness of the intervention and its components for readers in that context (Schünemann 2013).

Contextual factors may also pertain to the characteristics of the target group or population, such as cultural and linguistic diversity, socio-economic position, rural/urban setting. These factors may mean that a particular style of care or relationship evolves between service providers and consumers that may or may not match the values and technology of the program.

For many years these aspects have been acknowledged when decision makers have argued that results of evidence reviews from other countries do not apply in their own country or setting. Whilst some programmes/interventions have been successfully transferred from one context to another, others have not (Resnicow et al 1993, Lumley et al 2004, Coleman et al 2015). Review authors should be cautious when making generalizations from one context to another. They should report on the presence (or otherwise) of context-related information in intervention studies, where this information is available.

15.2.4 Variation in adherence

Variation in the adherence of the recipients and providers of care can limit the certainty in the applicability of results. Predictable differences in adherence can be due to divergence in how recipients of care perceive the intervention (e.g. the importance of side effects), economic conditions or attitudes that make some forms of care inaccessible in some settings, such as in low-income countries (Dans et al 2007). It should not be assumed that high levels of adherence in closely monitored randomized trials will translate into similar levels of adherence in normal practice.

15.2.5 Variation in values and preferences

Decisions about healthcare management strategies and options involve trading off health benefits and harms. The right choice may differ for people with different values and preferences (i.e. the importance people place on the outcomes and interventions), and it is important that decision makers ensure that decisions are consistent with a patient or population’s values and preferences. The importance placed on outcomes, together with other factors, will influence whether the recipients of care will or will not accept an option that is offered (Alonso-Coello et al 2016) and, thus, can be one factor influencing adherence. In Section 15.6 , we describe how the review author can help this process and the limits of supporting decision making based on intervention reviews.

15.3 Interpreting results of statistical analyses

15.3.1 confidence intervals.

Results for both individual studies and meta-analyses are reported with a point estimate together with an associated confidence interval. For example, ‘The odds ratio was 0.75 with a 95% confidence interval of 0.70 to 0.80’. The point estimate (0.75) is the best estimate of the magnitude and direction of the experimental intervention’s effect compared with the comparator intervention. The confidence interval describes the uncertainty inherent in any estimate, and describes a range of values within which we can be reasonably sure that the true effect actually lies. If the confidence interval is relatively narrow (e.g. 0.70 to 0.80), the effect size is known precisely. If the interval is wider (e.g. 0.60 to 0.93) the uncertainty is greater, although there may still be enough precision to make decisions about the utility of the intervention. Intervals that are very wide (e.g. 0.50 to 1.10) indicate that we have little knowledge about the effect and this imprecision affects our certainty in the evidence, and that further information would be needed before we could draw a more certain conclusion.

A 95% confidence interval is often interpreted as indicating a range within which we can be 95% certain that the true effect lies. This statement is a loose interpretation, but is useful as a rough guide. The strictly correct interpretation of a confidence interval is based on the hypothetical notion of considering the results that would be obtained if the study were repeated many times. If a study were repeated infinitely often, and on each occasion a 95% confidence interval calculated, then 95% of these intervals would contain the true effect (see Section 15.3.3 for further explanation).

The width of the confidence interval for an individual study depends to a large extent on the sample size. Larger studies tend to give more precise estimates of effects (and hence have narrower confidence intervals) than smaller studies. For continuous outcomes, precision depends also on the variability in the outcome measurements (i.e. how widely individual results vary between people in the study, measured as the standard deviation); for dichotomous outcomes it depends on the risk of the event (more frequent events allow more precision, and narrower confidence intervals), and for time-to-event outcomes it also depends on the number of events observed. All these quantities are used in computation of the standard errors of effect estimates from which the confidence interval is derived.

The width of a confidence interval for a meta-analysis depends on the precision of the individual study estimates and on the number of studies combined. In addition, for random-effects models, precision will decrease with increasing heterogeneity and confidence intervals will widen correspondingly (see Chapter 10, Section 10.10.4 ). As more studies are added to a meta-analysis the width of the confidence interval usually decreases. However, if the additional studies increase the heterogeneity in the meta-analysis and a random-effects model is used, it is possible that the confidence interval width will increase.

Confidence intervals and point estimates have different interpretations in fixed-effect and random-effects models. While the fixed-effect estimate and its confidence interval address the question ‘what is the best (single) estimate of the effect?’, the random-effects estimate assumes there to be a distribution of effects, and the estimate and its confidence interval address the question ‘what is the best estimate of the average effect?’ A confidence interval may be reported for any level of confidence (although they are most commonly reported for 95%, and sometimes 90% or 99%). For example, the odds ratio of 0.80 could be reported with an 80% confidence interval of 0.73 to 0.88; a 90% interval of 0.72 to 0.89; and a 95% interval of 0.70 to 0.92. As the confidence level increases, the confidence interval widens.

There is logical correspondence between the confidence interval and the P value (see Section 15.3.3 ). The 95% confidence interval for an effect will exclude the null value (such as an odds ratio of 1.0 or a risk difference of 0) if and only if the test of significance yields a P value of less than 0.05. If the P value is exactly 0.05, then either the upper or lower limit of the 95% confidence interval will be at the null value. Similarly, the 99% confidence interval will exclude the null if and only if the test of significance yields a P value of less than 0.01.

Together, the point estimate and confidence interval provide information to assess the effects of the intervention on the outcome. For example, suppose that we are evaluating an intervention that reduces the risk of an event and we decide that it would be useful only if it reduced the risk of an event from 30% by at least 5 percentage points to 25% (these values will depend on the specific clinical scenario and outcomes, including the anticipated harms). If the meta-analysis yielded an effect estimate of a reduction of 10 percentage points with a tight 95% confidence interval, say, from 7% to 13%, we would be able to conclude that the intervention was useful since both the point estimate and the entire range of the interval exceed our criterion of a reduction of 5% for net health benefit. However, if the meta-analysis reported the same risk reduction of 10% but with a wider interval, say, from 2% to 18%, although we would still conclude that our best estimate of the intervention effect is that it provides net benefit, we could not be so confident as we still entertain the possibility that the effect could be between 2% and 5%. If the confidence interval was wider still, and included the null value of a difference of 0%, we would still consider the possibility that the intervention has no effect on the outcome whatsoever, and would need to be even more sceptical in our conclusions.

Review authors may use the same general approach to conclude that an intervention is not useful. Continuing with the above example where the criterion for an important difference that should be achieved to provide more benefit than harm is a 5% risk difference, an effect estimate of 2% with a 95% confidence interval of 1% to 4% suggests that the intervention does not provide net health benefit.

15.3.2 P values and statistical significance

A P value is the standard result of a statistical test, and is the probability of obtaining the observed effect (or larger) under a ‘null hypothesis’. In the context of Cochrane Reviews there are two commonly used statistical tests. The first is a test of overall effect (a Z-test), and its null hypothesis is that there is no overall effect of the experimental intervention compared with the comparator on the outcome of interest. The second is the (Chi 2 ) test for heterogeneity, and its null hypothesis is that there are no differences in the intervention effects across studies.

A P value that is very small indicates that the observed effect is very unlikely to have arisen purely by chance, and therefore provides evidence against the null hypothesis. It has been common practice to interpret a P value by examining whether it is smaller than particular threshold values. In particular, P values less than 0.05 are often reported as ‘statistically significant’, and interpreted as being small enough to justify rejection of the null hypothesis. However, the 0.05 threshold is an arbitrary one that became commonly used in medical and psychological research largely because P values were determined by comparing the test statistic against tabulations of specific percentage points of statistical distributions. If review authors decide to present a P value with the results of a meta-analysis, they should report a precise P value (as calculated by most statistical software), together with the 95% confidence interval. Review authors should not describe results as ‘statistically significant’, ‘not statistically significant’ or ‘non-significant’ or unduly rely on thresholds for P values , but report the confidence interval together with the exact P value (see MECIR Box 15.3.a ).

We discuss interpretation of the test for heterogeneity in Chapter 10, Section 10.10.2 ; the remainder of this section refers mainly to tests for an overall effect. For tests of an overall effect, the computation of P involves both the effect estimate and precision of the effect estimate (driven largely by sample size). As precision increases, the range of plausible effects that could occur by chance is reduced. Correspondingly, the statistical significance of an effect of a particular magnitude will usually be greater (the P value will be smaller) in a larger study than in a smaller study.

P values are commonly misinterpreted in two ways. First, a moderate or large P value (e.g. greater than 0.05) may be misinterpreted as evidence that the intervention has no effect on the outcome. There is an important difference between this statement and the correct interpretation that there is a high probability that the observed effect on the outcome is due to chance alone. To avoid such a misinterpretation, review authors should always examine the effect estimate and its 95% confidence interval.

The second misinterpretation is to assume that a result with a small P value for the summary effect estimate implies that an experimental intervention has an important benefit. Such a misinterpretation is more likely to occur in large studies and meta-analyses that accumulate data over dozens of studies and thousands of participants. The P value addresses the question of whether the experimental intervention effect is precisely nil; it does not examine whether the effect is of a magnitude of importance to potential recipients of the intervention. In a large study, a small P value may represent the detection of a trivial effect that may not lead to net health benefit when compared with the potential harms (i.e. harmful effects on other important outcomes). Again, inspection of the point estimate and confidence interval helps correct interpretations (see Section 15.3.1 ).

MECIR Box 15.3.a Relevant expectations for conduct of intervention reviews

15.3.3 Relation between confidence intervals, statistical significance and certainty of evidence

The confidence interval (and imprecision) is only one domain that influences overall uncertainty about effect estimates. Uncertainty resulting from imprecision (i.e. statistical uncertainty) may be no less important than uncertainty from indirectness, or any other GRADE domain, in the context of decision making (Schünemann 2016). Thus, the extent to which interpretations of the confidence interval described in Sections 15.3.1 and 15.3.2 correspond to conclusions about overall certainty of the evidence for the outcome of interest depends on these other domains. If there are no concerns about other domains that determine the certainty of the evidence (i.e. risk of bias, inconsistency, indirectness or publication bias), then the interpretation in Sections 15.3.1 and 15.3.2 . about the relation of the confidence interval to the true effect may be carried forward to the overall certainty. However, if there are concerns about the other domains that affect the certainty of the evidence, the interpretation about the true effect needs to be seen in the context of further uncertainty resulting from those concerns.

For example, nine randomized controlled trials in almost 6000 cancer patients indicated that the administration of heparin reduces the risk of venous thromboembolism (VTE), with a risk ratio of 43% (95% CI 19% to 60%) (Akl et al 2011a). For patients with a plausible baseline risk of approximately 4.6% per year, this relative effect suggests that heparin leads to an absolute risk reduction of 20 fewer VTEs (95% CI 9 fewer to 27 fewer) per 1000 people per year (Akl et al 2011a). Now consider that the review authors or those applying the evidence in a guideline have lowered the certainty in the evidence as a result of indirectness. While the confidence intervals would remain unchanged, the certainty in that confidence interval and in the point estimate as reflecting the truth for the question of interest will be lowered. In fact, the certainty range will have unknown width so there will be unknown likelihood of a result within that range because of this indirectness. The lower the certainty in the evidence, the less we know about the width of the certainty range, although methods for quantifying risk of bias and understanding potential direction of bias may offer insight when lowered certainty is due to risk of bias. Nevertheless, decision makers must consider this uncertainty, and must do so in relation to the effect measure that is being evaluated (e.g. a relative or absolute measure). We will describe the impact on interpretations for dichotomous outcomes in Section 15.4 .

15.4 Interpreting results from dichotomous outcomes (including numbers needed to treat)

15.4.1 relative and absolute risk reductions.

Clinicians may be more inclined to prescribe an intervention that reduces the relative risk of death by 25% than one that reduces the risk of death by 1 percentage point, although both presentations of the evidence may relate to the same benefit (i.e. a reduction in risk from 4% to 3%). The former refers to the relative reduction in risk and the latter to the absolute reduction in risk. As described in Chapter 6, Section 6.4.1 , there are several measures for comparing dichotomous outcomes in two groups. Meta-analyses are usually undertaken using risk ratios (RR), odds ratios (OR) or risk differences (RD), but there are several alternative ways of expressing results.

Relative risk reduction (RRR) is a convenient way of re-expressing a risk ratio as a percentage reduction:

For example, a risk ratio of 0.75 translates to a relative risk reduction of 25%, as in the example above.

The risk difference is often referred to as the absolute risk reduction (ARR) or absolute risk increase (ARI), and may be presented as a percentage (e.g. 1%), as a decimal (e.g. 0.01), or as account (e.g. 10 out of 1000). We consider different choices for presenting absolute effects in Section 15.4.3 . We then describe computations for obtaining these numbers from the results of individual studies and of meta-analyses in Section 15.4.4 .

15.4.2 Number needed to treat (NNT)

The number needed to treat (NNT) is a common alternative way of presenting information on the effect of an intervention. The NNT is defined as the expected number of people who need to receive the experimental rather than the comparator intervention for one additional person to either incur or avoid an event (depending on the direction of the result) in a given time frame. Thus, for example, an NNT of 10 can be interpreted as ‘it is expected that one additional (or less) person will incur an event for every 10 participants receiving the experimental intervention rather than comparator over a given time frame’. It is important to be clear that:

• since the NNT is derived from the risk difference, it is still a comparative measure of effect (experimental versus a specific comparator) and not a general property of a single intervention; and
• the NNT gives an ‘expected value’. For example, NNT = 10 does not imply that one additional event will occur in each and every group of 10 people.

NNTs can be computed for both beneficial and detrimental events, and for interventions that cause both improvements and deteriorations in outcomes. In all instances NNTs are expressed as positive whole numbers. Some authors use the term ‘number needed to harm’ (NNH) when an intervention leads to an adverse outcome, or a decrease in a positive outcome, rather than improvement. However, this phrase can be misleading (most notably, it can easily be read to imply the number of people who will experience a harmful outcome if given the intervention), and it is strongly recommended that ‘number needed to harm’ and ‘NNH’ are avoided. The preferred alternative is to use phrases such as ‘number needed to treat for an additional beneficial outcome’ (NNTB) and ‘number needed to treat for an additional harmful outcome’ (NNTH) to indicate direction of effect.

As NNTs refer to events, their interpretation needs to be worded carefully when the binary outcome is a dichotomization of a scale-based outcome. For example, if the outcome is pain measured on a ‘none, mild, moderate or severe’ scale it may have been dichotomized as ‘none or mild’ versus ‘moderate or severe’. It would be inappropriate for an NNT from these data to be referred to as an ‘NNT for pain’. It is an ‘NNT for moderate or severe pain’.

We consider different choices for presenting absolute effects in Section 15.4.3 . We then describe computations for obtaining these numbers from the results of individual studies and of meta-analyses in Section 15.4.4 .

15.4.3 Expressing risk differences

Users of reviews are liable to be influenced by the choice of statistical presentations of the evidence. Hoffrage and colleagues suggest that physicians’ inferences about statistical outcomes are more appropriate when they deal with ‘natural frequencies’ – whole numbers of people, both treated and untreated (e.g. treatment results in a drop from 20 out of 1000 to 10 out of 1000 women having breast cancer) – than when effects are presented as percentages (e.g. 1% absolute reduction in breast cancer risk) (Hoffrage et al 2000). Probabilities may be more difficult to understand than frequencies, particularly when events are rare. While standardization may be important in improving the presentation of research evidence (and participation in healthcare decisions), current evidence suggests that the presentation of natural frequencies for expressing differences in absolute risk is best understood by consumers of healthcare information (Akl et al 2011b). This evidence provides the rationale for presenting absolute risks in ‘Summary of findings’ tables as numbers of people with events per 1000 people receiving the intervention (see Chapter 14 ).

RRs and RRRs remain crucial because relative effects tend to be substantially more stable across risk groups than absolute effects (see Chapter 10, Section 10.4.3 ). Review authors can use their own data to study this consistency (Cates 1999, Smeeth et al 1999). Risk differences from studies are least likely to be consistent across baseline event rates; thus, they are rarely appropriate for computing numbers needed to treat in systematic reviews. If a relative effect measure (OR or RR) is chosen for meta-analysis, then a comparator group risk needs to be specified as part of the calculation of an RD or NNT. In addition, if there are several different groups of participants with different levels of risk, it is crucial to express absolute benefit for each clinically identifiable risk group, clarifying the time period to which this applies. Studies in patients with differing severity of disease, or studies with different lengths of follow-up will almost certainly have different comparator group risks. In these cases, different comparator group risks lead to different RDs and NNTs (except when the intervention has no effect). A recommended approach is to re-express an odds ratio or a risk ratio as a variety of RD or NNTs across a range of assumed comparator risks (ACRs) (McQuay and Moore 1997, Smeeth et al 1999). Review authors should bear these considerations in mind not only when constructing their ‘Summary of findings’ table, but also in the text of their review.

For example, a review of oral anticoagulants to prevent stroke presented information to users by describing absolute benefits for various baseline risks (Aguilar and Hart 2005, Aguilar et al 2007). They presented their principal findings as “The inherent risk of stroke should be considered in the decision to use oral anticoagulants in atrial fibrillation patients, selecting those who stand to benefit most for this therapy” (Aguilar and Hart 2005). Among high-risk atrial fibrillation patients with prior stroke or transient ischaemic attack who have stroke rates of about 12% (120 per 1000) per year, warfarin prevents about 70 strokes yearly per 1000 patients, whereas for low-risk atrial fibrillation patients (with a stroke rate of about 2% per year or 20 per 1000), warfarin prevents only 12 strokes. This presentation helps users to understand the important impact that typical baseline risks have on the absolute benefit that they can expect.

15.4.4 Computations

Direct computation of risk difference (RD) or a number needed to treat (NNT) depends on the summary statistic (odds ratio, risk ratio or risk differences) available from the study or meta-analysis. When expressing results of meta-analyses, review authors should use, in the computations, whatever statistic they determined to be the most appropriate summary for meta-analysis (see Chapter 10, Section 10.4.3 ). Here we present calculations to obtain RD as a reduction in the number of participants per 1000. For example, a risk difference of –0.133 corresponds to 133 fewer participants with the event per 1000.

RDs and NNTs should not be computed from the aggregated total numbers of participants and events across the trials. This approach ignores the randomization within studies, and may produce seriously misleading results if there is unbalanced randomization in any of the studies. Using the pooled result of a meta-analysis is more appropriate. When computing NNTs, the values obtained are by convention always rounded up to the next whole number.

15.4.4.1 Computing NNT from a risk difference (RD)

A NNT may be computed from a risk difference as

where the vertical bars (‘absolute value of’) in the denominator indicate that any minus sign should be ignored. It is convention to round the NNT up to the nearest whole number. For example, if the risk difference is –0.12 the NNT is 9; if the risk difference is –0.22 the NNT is 5. Cochrane Review authors should qualify the NNT as referring to benefit (improvement) or harm by denoting the NNT as NNTB or NNTH. Note that this approach, although feasible, should be used only for the results of a meta-analysis of risk differences. In most cases meta-analyses will be undertaken using a relative measure of effect (RR or OR), and those statistics should be used to calculate the NNT (see Section 15.4.4.2 and 15.4.4.3 ).

15.4.4.2 Computing risk differences or NNT from a risk ratio

To aid interpretation of the results of a meta-analysis of risk ratios, review authors may compute an absolute risk reduction or NNT. In order to do this, an assumed comparator risk (ACR) (otherwise known as a baseline risk, or risk that the outcome of interest would occur with the comparator intervention) is required. It will usually be appropriate to do this for a range of different ACRs. The computation proceeds as follows:

As an example, suppose the risk ratio is RR = 0.92, and an ACR = 0.3 (300 per 1000) is assumed. Then the effect on risk is 24 fewer per 1000:

The NNT is 42:

15.4.4.3 Computing risk differences or NNT from an odds ratio

Review authors may wish to compute a risk difference or NNT from the results of a meta-analysis of odds ratios. In order to do this, an ACR is required. It will usually be appropriate to do this for a range of different ACRs. The computation proceeds as follows:

As an example, suppose the odds ratio is OR = 0.73, and a comparator risk of ACR = 0.3 is assumed. Then the effect on risk is 62 fewer per 1000:

The NNT is 17:

15.4.4.4 Computing risk ratio from an odds ratio

Because risk ratios are easier to interpret than odds ratios, but odds ratios have favourable mathematical properties, a review author may decide to undertake a meta-analysis based on odds ratios, but to express the result as a summary risk ratio (or relative risk reduction). This requires an ACR. Then

It will often be reasonable to perform this transformation using the median comparator group risk from the studies in the meta-analysis.

15.4.4.5 Computing confidence limits

Confidence limits for RDs and NNTs may be calculated by applying the above formulae to the upper and lower confidence limits for the summary statistic (RD, RR or OR) (Altman 1998). Note that this confidence interval does not incorporate uncertainty around the ACR.

If the 95% confidence interval of OR or RR includes the value 1, one of the confidence limits will indicate benefit and the other harm. Thus, appropriate use of the words ‘fewer’ and ‘more’ is required for each limit when presenting results in terms of events. For NNTs, the two confidence limits should be labelled as NNTB and NNTH to indicate the direction of effect in each case. The confidence interval for the NNT will include a ‘discontinuity’, because increasingly smaller risk differences that approach zero will lead to NNTs approaching infinity. Thus, the confidence interval will include both an infinitely large NNTB and an infinitely large NNTH.

15.5 Interpreting results from continuous outcomes (including standardized mean differences)

15.5.1 meta-analyses with continuous outcomes.

Review authors should describe in the study protocol how they plan to interpret results for continuous outcomes. When outcomes are continuous, review authors have a number of options to present summary results. These options differ if studies report the same measure that is familiar to the target audiences, studies report the same or very similar measures that are less familiar to the target audiences, or studies report different measures.

15.5.2 Meta-analyses with continuous outcomes using the same measure

If all studies have used the same familiar units, for instance, results are expressed as durations of events, such as symptoms for conditions including diarrhoea, sore throat, otitis media, influenza or duration of hospitalization, a meta-analysis may generate a summary estimate in those units, as a difference in mean response (see, for instance, the row summarizing results for duration of diarrhoea in Chapter 14, Figure 14.1.b and the row summarizing oedema in Chapter 14, Figure 14.1.a ). For such outcomes, the ‘Summary of findings’ table should include a difference of means between the two interventions. However, when units of such outcomes may be difficult to interpret, particularly when they relate to rating scales (again, see the oedema row of Chapter 14, Figure 14.1.a ). ‘Summary of findings’ tables should include the minimum and maximum of the scale of measurement, and the direction. Knowledge of the smallest change in instrument score that patients perceive is important – the minimal important difference (MID) – and can greatly facilitate the interpretation of results (Guyatt et al 1998, Schünemann and Guyatt 2005). Knowing the MID allows review authors and users to place results in context. Review authors should state the MID – if known – in the Comments column of their ‘Summary of findings’ table. For example, the chronic respiratory questionnaire has possible scores in health-related quality of life ranging from 1 to 7 and 0.5 represents a well-established MID (Jaeschke et al 1989, Schünemann et al 2005).

15.5.3 Meta-analyses with continuous outcomes using different measures

When studies have used different instruments to measure the same construct, a standardized mean difference (SMD) may be used in meta-analysis for combining continuous data. Without guidance, clinicians and patients may have little idea how to interpret results presented as SMDs. Review authors should therefore consider issues of interpretability when planning their analysis at the protocol stage and should consider whether there will be suitable ways to re-express the SMD or whether alternative effect measures, such as a ratio of means, or possibly as minimal important difference units (Guyatt et al 2013b) should be used. Table 15.5.a and the following sections describe these options.

Table 15.5.a Approaches and their implications to presenting results of continuous variables when primary studies have used different instruments to measure the same construct. Adapted from Guyatt et al (2013b)

15.5.3.1 Presenting and interpreting SMDs using generic effect size estimates

The SMD expresses the intervention effect in standard units rather than the original units of measurement. The SMD is the difference in mean effects between the experimental and comparator groups divided by the pooled standard deviation of participants’ outcomes, or external SDs when studies are very small (see Chapter 6, Section 6.5.1.2 ). The value of a SMD thus depends on both the size of the effect (the difference between means) and the standard deviation of the outcomes (the inherent variability among participants or based on an external SD).

If review authors use the SMD, they might choose to present the results directly as SMDs (row 1a, Table 15.5.a and Table 15.5.b ). However, absolute values of the intervention and comparison groups are typically not useful because studies have used different measurement instruments with different units. Guiding rules for interpreting SMDs (or ‘Cohen’s effect sizes’) exist, and have arisen mainly from researchers in the social sciences (Cohen 1988). One example is as follows: 0.2 represents a small effect, 0.5 a moderate effect and 0.8 a large effect (Cohen 1988). Variations exist (e.g. <0.40=small, 0.40 to 0.70=moderate, >0.70=large). Review authors might consider including such a guiding rule in interpreting the SMD in the text of the review, and in summary versions such as the Comments column of a ‘Summary of findings’ table. However, some methodologists believe that such interpretations are problematic because patient importance of a finding is context-dependent and not amenable to generic statements.

15.5.3.2 Re-expressing SMDs using a familiar instrument

The second possibility for interpreting the SMD is to express it in the units of one or more of the specific measurement instruments used by the included studies (row 1b, Table 15.5.a and Table 15.5.b ). The approach is to calculate an absolute difference in means by multiplying the SMD by an estimate of the SD associated with the most familiar instrument. To obtain this SD, a reasonable option is to calculate a weighted average across all intervention groups of all studies that used the selected instrument (preferably a pre-intervention or post-intervention SD as discussed in Chapter 10, Section 10.5.2 ). To better reflect among-person variation in practice, or to use an instrument not represented in the meta-analysis, it may be preferable to use a standard deviation from a representative observational study. The summary effect is thus re-expressed in the original units of that particular instrument and the clinical relevance and impact of the intervention effect can be interpreted using that familiar instrument.

The same approach of re-expressing the results for a familiar instrument can also be used for other standardized effect measures such as when standardizing by MIDs (Guyatt et al 2013b): see Section 15.5.3.5 .

Table 15.5.b Application of approaches when studies have used different measures: effects of dexamethasone for pain after laparoscopic cholecystectomy (Karanicolas et al 2008). Reproduced with permission of Wolters Kluwer

1 Certainty rated according to GRADE from very low to high certainty. 2 Substantial unexplained heterogeneity in study results. 3 Imprecision due to wide confidence intervals. 4 The 20% comes from the proportion in the control group requiring rescue analgesia. 5 Crude (arithmetic) means of the post-operative pain mean responses across all five trials when transformed to a 100-point scale.

15.5.3.3 Re-expressing SMDs through dichotomization and transformation to relative and absolute measures

A third approach (row 1c, Table 15.5.a and Table 15.5.b ) relies on converting the continuous measure into a dichotomy and thus allows calculation of relative and absolute effects on a binary scale. A transformation of a SMD to a (log) odds ratio is available, based on the assumption that an underlying continuous variable has a logistic distribution with equal standard deviation in the two intervention groups, as discussed in Chapter 10, Section 10.6  (Furukawa 1999, Guyatt et al 2013b). The assumption is unlikely to hold exactly and the results must be regarded as an approximation. The log odds ratio is estimated as

(or approximately 1.81✕SMD). The resulting odds ratio can then be presented as normal, and in a ‘Summary of findings’ table, combined with an assumed comparator group risk to be expressed as an absolute risk difference. The comparator group risk in this case would refer to the proportion of people who have achieved a specific value of the continuous outcome. In randomized trials this can be interpreted as the proportion who have improved by some (specified) amount (responders), for instance by 5 points on a 0 to 100 scale. Table 15.5.c shows some illustrative results from this method. The risk differences can then be converted to NNTs or to people per thousand using methods described in Section 15.4.4 .

Table 15.5.c Risk difference derived for specific SMDs for various given ‘proportions improved’ in the comparator group (Furukawa 1999, Guyatt et al 2013b). Reproduced with permission of Elsevier 

15.5.3.4 Ratio of means

A more frequently used approach is based on calculation of a ratio of means between the intervention and comparator groups (Friedrich et al 2008) as discussed in Chapter 6, Section 6.5.1.3 . Interpretational advantages of this approach include the ability to pool studies with outcomes expressed in different units directly, to avoid the vulnerability of heterogeneous populations that limits approaches that rely on SD units, and for ease of clinical interpretation (row 2, Table 15.5.a and Table 15.5.b ). This method is currently designed for post-intervention scores only. However, it is possible to calculate a ratio of change scores if both intervention and comparator groups change in the same direction in each relevant study, and this ratio may sometimes be informative.

Limitations to this approach include its limited applicability to change scores (since it is unlikely that both intervention and comparator group changes are in the same direction in all studies) and the possibility of misleading results if the comparator group mean is very small, in which case even a modest difference from the intervention group will yield a large and therefore misleading ratio of means. It also requires that separate ratios of means be calculated for each included study, and then entered into a generic inverse variance meta-analysis (see Chapter 10, Section 10.3 ).

The ratio of means approach illustrated in Table 15.5.b suggests a relative reduction in pain of only 13%, meaning that those receiving steroids have a pain severity 87% of those in the comparator group, an effect that might be considered modest.

15.5.3.5 Presenting continuous results as minimally important difference units

To express results in MID units, review authors have two options. First, they can be combined across studies in the same way as the SMD, but instead of dividing the mean difference of each study by its SD, review authors divide by the MID associated with that outcome (Johnston et al 2010, Guyatt et al 2013b). Instead of SD units, the pooled results represent MID units (row 3, Table 15.5.a and Table 15.5.b ), and may be more easily interpretable. This approach avoids the problem of varying SDs across studies that may distort estimates of effect in approaches that rely on the SMD. The approach, however, relies on having well-established MIDs. The approach is also risky in that a difference less than the MID may be interpreted as trivial when a substantial proportion of patients may have achieved an important benefit.

The other approach makes a simple conversion (not shown in Table 15.5.b ), before undertaking the meta-analysis, of the means and SDs from each study to means and SDs on the scale of a particular familiar instrument whose MID is known. For example, one can rescale the mean and SD of other chronic respiratory disease instruments (e.g. rescaling a 0 to 100 score of an instrument) to a the 1 to 7 score in Chronic Respiratory Disease Questionnaire (CRQ) units (by assuming 0 equals 1 and 100 equals 7 on the CRQ). Given the MID of the CRQ of 0.5, a mean difference in change of 0.71 after rescaling of all studies suggests a substantial effect of the intervention (Guyatt et al 2013b). This approach, presenting in units of the most familiar instrument, may be the most desirable when the target audiences have extensive experience with that instrument, particularly if the MID is well established.

15.6 Drawing conclusions

15.6.1 conclusions sections of a cochrane review.

Authors’ conclusions in a Cochrane Review are divided into implications for practice and implications for research. While Cochrane Reviews about interventions can provide meaningful information and guidance for practice, decisions about the desirable and undesirable consequences of healthcare options require evidence and judgements for criteria that most Cochrane Reviews do not provide (Alonso-Coello et al 2016). In describing the implications for practice and the development of recommendations, however, review authors may consider the certainty of the evidence, the balance of benefits and harms, and assumed values and preferences.

15.6.2 Implications for practice

Drawing conclusions about the practical usefulness of an intervention entails making trade-offs, either implicitly or explicitly, between the estimated benefits, harms and the values and preferences. Making such trade-offs, and thus making specific recommendations for an action in a specific context, goes beyond a Cochrane Review and requires additional evidence and informed judgements that most Cochrane Reviews do not provide (Alonso-Coello et al 2016). Such judgements are typically the domain of clinical practice guideline developers for which Cochrane Reviews will provide crucial information (Graham et al 2011, Schünemann et al 2014, Zhang et al 2018a). Thus, authors of Cochrane Reviews should not make recommendations.

If review authors feel compelled to lay out actions that clinicians and patients could take, they should – after describing the certainty of evidence and the balance of benefits and harms – highlight different actions that might be consistent with particular patterns of values and preferences. Other factors that might influence a decision should also be highlighted, including any known factors that would be expected to modify the effects of the intervention, the baseline risk or status of the patient, costs and who bears those costs, and the availability of resources. Review authors should ensure they consider all patient-important outcomes, including those for which limited data may be available. In the context of public health reviews the focus may be on population-important outcomes as the target may be an entire (non-diseased) population and include outcomes that are not measured in the population receiving an intervention (e.g. a reduction of transmission of infections from those receiving an intervention). This process implies a high level of explicitness in judgements about values or preferences attached to different outcomes and the certainty of the related evidence (Zhang et al 2018b, Zhang et al 2018c); this and a full cost-effectiveness analysis is beyond the scope of most Cochrane Reviews (although they might well be used for such analyses; see Chapter 20 ).

A review on the use of anticoagulation in cancer patients to increase survival (Akl et al 2011a) provides an example for laying out clinical implications for situations where there are important trade-offs between desirable and undesirable effects of the intervention: “The decision for a patient with cancer to start heparin therapy for survival benefit should balance the benefits and downsides and integrate the patient’s values and preferences. Patients with a high preference for a potential survival prolongation, limited aversion to potential bleeding, and who do not consider heparin (both UFH or LMWH) therapy a burden may opt to use heparin, while those with aversion to bleeding may not.”

15.6.3 Implications for research

The second category for authors’ conclusions in a Cochrane Review is implications for research. To help people make well-informed decisions about future healthcare research, the ‘Implications for research’ section should comment on the need for further research, and the nature of the further research that would be most desirable. It is helpful to consider the population, intervention, comparison and outcomes that could be addressed, or addressed more effectively in the future, in the context of the certainty of the evidence in the current review (Brown et al 2006):

• P (Population): diagnosis, disease stage, comorbidity, risk factor, sex, age, ethnic group, specific inclusion or exclusion criteria, clinical setting;
• I (Intervention): type, frequency, dose, duration, prognostic factor;
• C (Comparison): placebo, routine care, alternative treatment/management;
• O (Outcome): which clinical or patient-related outcomes will the researcher need to measure, improve, influence or accomplish? Which methods of measurement should be used?

While Cochrane Review authors will find the PICO domains helpful, the domains of the GRADE certainty framework further support understanding and describing what additional research will improve the certainty in the available evidence. Note that as the certainty of the evidence is likely to vary by outcome, these implications will be specific to certain outcomes in the review. Table 15.6.a shows how review authors may be aided in their interpretation of the body of evidence and drawing conclusions about future research and practice.

Table 15.6.a Implications for research and practice suggested by individual GRADE domains

The review of compression stockings for prevention of deep vein thrombosis (DVT) in airline passengers described in Chapter 14 provides an example where there is some convincing evidence of a benefit of the intervention: “This review shows that the question of the effects on symptomless DVT of wearing versus not wearing compression stockings in the types of people studied in these trials should now be regarded as answered. Further research may be justified to investigate the relative effects of different strengths of stockings or of stockings compared to other preventative strategies. Further randomised trials to address the remaining uncertainty about the effects of wearing versus not wearing compression stockings on outcomes such as death, pulmonary embolism and symptomatic DVT would need to be large.” (Clarke et al 2016).

A review of therapeutic touch for anxiety disorder provides an example of the implications for research when no eligible studies had been found: “This review highlights the need for randomized controlled trials to evaluate the effectiveness of therapeutic touch in reducing anxiety symptoms in people diagnosed with anxiety disorders. Future trials need to be rigorous in design and delivery, with subsequent reporting to include high quality descriptions of all aspects of methodology to enable appraisal and interpretation of results.” (Robinson et al 2007).

15.6.4 Reaching conclusions

A common mistake is to confuse ‘no evidence of an effect’ with ‘evidence of no effect’. When the confidence intervals are too wide (e.g. including no effect), it is wrong to claim that the experimental intervention has ‘no effect’ or is ‘no different’ from the comparator intervention. Review authors may also incorrectly ‘positively’ frame results for some effects but not others. For example, when the effect estimate is positive for a beneficial outcome but confidence intervals are wide, review authors may describe the effect as promising. However, when the effect estimate is negative for an outcome that is considered harmful but the confidence intervals include no effect, review authors report no effect. Another mistake is to frame the conclusion in wishful terms. For example, review authors might write, “there were too few people in the analysis to detect a reduction in mortality” when the included studies showed a reduction or even increase in mortality that was not ‘statistically significant’. One way of avoiding errors such as these is to consider the results blinded; that is, consider how the results would be presented and framed in the conclusions if the direction of the results was reversed. If the confidence interval for the estimate of the difference in the effects of the interventions overlaps with no effect, the analysis is compatible with both a true beneficial effect and a true harmful effect. If one of the possibilities is mentioned in the conclusion, the other possibility should be mentioned as well. Table 15.6.b suggests narrative statements for drawing conclusions based on the effect estimate from the meta-analysis and the certainty of the evidence.

Table 15.6.b Suggested narrative statements for phrasing conclusions

Another common mistake is to reach conclusions that go beyond the evidence. Often this is done implicitly, without referring to the additional information or judgements that are used in reaching conclusions about the implications of a review for practice. Even when additional information and explicit judgements support conclusions about the implications of a review for practice, review authors rarely conduct systematic reviews of the additional information. Furthermore, implications for practice are often dependent on specific circumstances and values that must be taken into consideration. As we have noted, review authors should always be cautious when drawing conclusions about implications for practice and they should not make recommendations.

15.7 Chapter information

Authors: Holger J Schünemann, Gunn E Vist, Julian PT Higgins, Nancy Santesso, Jonathan J Deeks, Paul Glasziou, Elie Akl, Gordon H Guyatt; on behalf of the Cochrane GRADEing Methods Group

Acknowledgements: Andrew Oxman, Jonathan Sterne, Michael Borenstein and Rob Scholten contributed text to earlier versions of this chapter.

Funding: This work was in part supported by funding from the Michael G DeGroote Cochrane Canada Centre and the Ontario Ministry of Health. JJD receives support from the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. JPTH receives support from the NIHR Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

15.8 References

Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. Cochrane Database of Systematic Reviews 2005; 3 : CD001927.

Aguilar MI, Hart R, Pearce LA. Oral anticoagulants versus antiplatelet therapy for preventing stroke in patients with non-valvular atrial fibrillation and no history of stroke or transient ischemic attacks. Cochrane Database of Systematic Reviews 2007; 3 : CD006186.

Akl EA, Gunukula S, Barba M, Yosuico VE, van Doormaal FF, Kuipers S, Middeldorp S, Dickinson HO, Bryant A, Schünemann H. Parenteral anticoagulation in patients with cancer who have no therapeutic or prophylactic indication for anticoagulation. Cochrane Database of Systematic Reviews 2011a; 1 : CD006652.

Akl EA, Oxman AD, Herrin J, Vist GE, Terrenato I, Sperati F, Costiniuk C, Blank D, Schünemann H. Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database of Systematic Reviews 2011b; 3 : CD006776.

Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, Mustafa RA, Rada G, Rosenbaum S, Morelli A, Guyatt GH, Oxman AD, Group GW. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ 2016; 353 : i2016.

Altman DG. Confidence intervals for the number needed to treat. BMJ 1998; 317 : 1309-1312.

Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW, Jr., Zaza S. Grading quality of evidence and strength of recommendations. BMJ 2004; 328 : 1490.

Brown P, Brunnhuber K, Chalkidou K, Chalmers I, Clarke M, Fenton M, Forbes C, Glanville J, Hicks NJ, Moody J, Twaddle S, Timimi H, Young P. How to formulate research recommendations. BMJ 2006; 333 : 804-806.

Cates C. Confidence intervals for the number needed to treat: Pooling numbers needed to treat may not be reliable. BMJ 1999; 318 : 1764-1765.

Clarke MJ, Broderick C, Hopewell S, Juszczak E, Eisinga A. Compression stockings for preventing deep vein thrombosis in airline passengers. Cochrane Database of Systematic Reviews 2016; 9 : CD004002.

Cohen J. Statistical Power Analysis in the Behavioral Sciences . 2nd edition ed. Hillsdale (NJ): Lawrence Erlbaum Associates, Inc.; 1988.

Coleman T, Chamberlain C, Davey MA, Cooper SE, Leonardi-Bee J. Pharmacological interventions for promoting smoking cessation during pregnancy. Cochrane Database of Systematic Reviews 2015; 12 : CD010078.

Dans AM, Dans L, Oxman AD, Robinson V, Acuin J, Tugwell P, Dennis R, Kang D. Assessing equity in clinical practice guidelines. Journal of Clinical Epidemiology 2007; 60 : 540-546.

Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials . 2nd edition ed. Littleton (MA): John Wright PSG, Inc.; 1985.

Friedrich JO, Adhikari NK, Beyene J. The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: a simulation study. BMC Medical Research Methodology 2008; 8 : 32.

Furukawa T. From effect size into number needed to treat. Lancet 1999; 353 : 1680.

Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E. Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services: Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011.

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schünemann HJ. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology 2011a; 64 : 383-394.

Guyatt GH, Juniper EF, Walter SD, Griffith LE, Goldstein RS. Interpreting treatment effects in randomised trials. BMJ 1998; 316 : 690-693.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336 : 924-926.

Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, Alonso-Coello P, Falck-Ytter Y, Jaeschke R, Vist G, Akl EA, Post PN, Norris S, Meerpohl J, Shukla VK, Nasser M, Schünemann HJ. GRADE guidelines: 8. Rating the quality of evidence--indirectness. Journal of Clinical Epidemiology 2011b; 64 : 1303-1310.

Guyatt GH, Oxman AD, Santesso N, Helfand M, Vist G, Kunz R, Brozek J, Norris S, Meerpohl J, Djulbegovic B, Alonso-Coello P, Post PN, Busse JW, Glasziou P, Christensen R, Schünemann HJ. GRADE guidelines: 12. Preparing summary of findings tables-binary outcomes. Journal of Clinical Epidemiology 2013a; 66 : 158-172.

Guyatt GH, Thorlund K, Oxman AD, Walter SD, Patrick D, Furukawa TA, Johnston BC, Karanicolas P, Akl EA, Vist G, Kunz R, Brozek J, Kupper LL, Martin SL, Meerpohl JJ, Alonso-Coello P, Christensen R, Schünemann HJ. GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes. Journal of Clinical Epidemiology 2013b; 66 : 173-183.

Hawe P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. Journal of Epidemiology and Community Health 2004; 58 : 788-793.

Hoffrage U, Lindsey S, Hertwig R, Gigerenzer G. Medicine. Communicating statistical information. Science 2000; 290 : 2261-2262.

Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Controlled Clinical Trials 1989; 10 : 407-415.

Johnston B, Thorlund K, Schünemann H, Xie F, Murad M, Montori V, Guyatt G. Improving the interpretation of health-related quality of life evidence in meta-analysis: The application of minimal important difference units. . Health Outcomes and Qualithy of Life 2010; 11 : 116.

Karanicolas PJ, Smith SE, Kanbur B, Davies E, Guyatt GH. The impact of prophylactic dexamethasone on nausea and vomiting after laparoscopic cholecystectomy: a systematic review and meta-analysis. Annals of Surgery 2008; 248 : 751-762.

Lumley J, Oliver SS, Chamberlain C, Oakley L. Interventions for promoting smoking cessation during pregnancy. Cochrane Database of Systematic Reviews 2004; 4 : CD001055.

McQuay HJ, Moore RA. Using numerical results from systematic reviews in clinical practice. Annals of Internal Medicine 1997; 126 : 712-720.

Resnicow K, Cross D, Wynder E. The Know Your Body program: a review of evaluation studies. Bulletin of the New York Academy of Medicine 1993; 70 : 188-207.

Robinson J, Biley FC, Dolk H. Therapeutic touch for anxiety disorders. Cochrane Database of Systematic Reviews 2007; 3 : CD006240.

Rothwell PM. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet 2005; 365 : 82-93.

Santesso N, Carrasco-Labra A, Langendam M, Brignardello-Petersen R, Mustafa RA, Heus P, Lasserson T, Opiyo N, Kunnamo I, Sinclair D, Garner P, Treweek S, Tovey D, Akl EA, Tugwell P, Brozek JL, Guyatt G, Schünemann HJ. Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. Journal of Clinical Epidemiology 2016; 74 : 28-39.

Schünemann HJ, Puhan M, Goldstein R, Jaeschke R, Guyatt GH. Measurement properties and interpretability of the Chronic respiratory disease questionnaire (CRQ). COPD: Journal of Chronic Obstructive Pulmonary Disease 2005; 2 : 81-89.

Schünemann HJ, Guyatt GH. Commentary--goodbye M(C)ID! Hello MID, where do you come from? Health Services Research 2005; 40 : 593-597.

Schünemann HJ, Fretheim A, Oxman AD. Improving the use of research evidence in guideline development: 13. Applicability, transferability and adaptation. Health Research Policy and Systems 2006; 4 : 25.

Schünemann HJ. Methodological idiosyncracies, frameworks and challenges of non-pharmaceutical and non-technical treatment interventions. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 2013; 107 : 214-220.

Schünemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, Shea B, Wells G, Helfand M. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Research Synthesis Methods 2013; 4 : 49-62.

Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, Ventresca M, Brignardello-Petersen R, Laisaar KT, Kowalski S, Baldeh T, Zhang Y, Raid U, Neumann I, Norris SL, Thornton J, Harbour R, Treweek S, Guyatt G, Alonso-Coello P, Reinap M, Brozek J, Oxman A, Akl EA. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ: Canadian Medical Association Journal 2014; 186 : E123-142.

Schünemann HJ. Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision? Journal of Clinical Epidemiology 2016; 75 : 6-15.

Smeeth L, Haines A, Ebrahim S. Numbers needed to treat derived from meta-analyses--sometimes informative, usually misleading. BMJ 1999; 318 : 1548-1551.

Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, Diaz-Granados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, Alonso-Coello P, Hassouneh B, Walter SD, Heels-Ansdell D, Bhatnagar N, Altman DG, Guyatt GH. Credibility of claims of subgroup effects in randomised controlled trials: systematic review. BMJ 2012; 344 : e1553.

Zhang Y, Akl EA, Schünemann HJ. Using systematic reviews in guideline development: the GRADE approach. Research Synthesis Methods 2018a: doi: 10.1002/jrsm.1313.

Zhang Y, Alonso-Coello P, Guyatt GH, Yepes-Nunez JJ, Akl EA, Hazlewood G, Pardo-Hernandez H, Etxeandia-Ikobaltzeta I, Qaseem A, Williams JW, Jr., Tugwell P, Flottorp S, Chang Y, Zhang Y, Mustafa RA, Rojas MX, Schünemann HJ. GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-Risk of bias and indirectness. Journal of Clinical Epidemiology 2018b: doi: 10.1016/j.jclinepi.2018.1001.1013.

Zhang Y, Alonso Coello P, Guyatt G, Yepes-Nunez JJ, Akl EA, Hazlewood G, Pardo-Hernandez H, Etxeandia-Ikobaltzeta I, Qaseem A, Williams JW, Jr., Tugwell P, Flottorp S, Chang Y, Zhang Y, Mustafa RA, Rojas MX, Xie F, Schünemann HJ. GRADE Guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences - Inconsistency, Imprecision, and other Domains. Journal of Clinical Epidemiology 2018c: doi: 10.1016/j.jclinepi.2018.1005.1011.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

Communicating the Results

• First Online: 02 November 2017

Cite this chapter

• Erik Mooi 4 ,
• Marko Sarstedt 5 &
• Irma Mooi-Reci 6  

Part of the book series: Springer Texts in Business and Economics ((STBE))

157k Accesses

Communicating the results of your study, project, or business case is crucial in market research. We discuss the core elements of a written research report, provide guidelines on how to structure its core elements, and how you can communicate the research findings to your audience in terms of their characteristics and needs. We show you how to organize and simplify complex and dense information in an efficient and reader-friendly way. Using Stata, and drawing on a case study, we show how you can combine and present several graphs and (regression) tables concisely and clearly. We also provide guidelines for oral presentations, suggestions for visual aids that facilitate the communication of difficult ideas, and ideas on how to best structure the oral presentation of results. Finally, we discuss ethical issues that may arise when communicating report findings to your client.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Armstrong, J. S. (2010). Persuasive advertising: Evidence-based principles . New York: Palgrave Macmillan.

Book   Google Scholar  

Churchill, G. A., Jr., & Iacobucci, D. (2009). Marketing research: Methodological foundations (10 th ed.). Mason: South-Western College Publishers.

Google Scholar  

Cox, N. J. (2008). Speaking Stata: Between tables and graphs. Stata Journal, 8 (2), 269–289.

European Society for Opinion and Marketing Research (ESOMAR). (2007). ICC/ESOMAR International Code On Market And Social Research. http://www.esomar.org/uploads/public/knowledge-and-standards/codes-and-guidelines/ICCESOMAR_Code_English_.pdf

Huff, D. (1993). How to lie with statistics . New York: W. W. Norton & Company.

Jann, B. (2014). Plotting regression coefficients and other estimates. Stata Journal, 14 (4), 708–737.

Minto, B. (2009). The pyramid principle: Logic in writing and thinking (3 rd ed.). Harlow: Pearson.

Tufte, E. R. (2001). The visual display of quantitative information (2 nd ed.). Cheshire: Graphics Press.

Download references

Author information

Authors and affiliations.

Department of Management and Marketing, University of Melbourne, Parkville, Victoria, Australia

Chair of Marketing, Otto-von-Guericke-University, Magdeburg, Sachsen-Anhalt, Germany

Marko Sarstedt

School of Social and Political Sciences, University of Melbourne, Parkville, Victoria, Australia

Irma Mooi-Reci

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Mooi, E., Sarstedt, M., Mooi-Reci, I. (2018). Communicating the Results. In: Market Research. Springer Texts in Business and Economics. Springer, Singapore. https://doi.org/10.1007/978-981-10-5218-7_10

Download citation

DOI : https://doi.org/10.1007/978-981-10-5218-7_10

Published : 02 November 2017

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5217-0

Online ISBN : 978-981-10-5218-7

eBook Packages : Business and Management Business and Management (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Saudi J Anaesth
• v.15(4); Oct-Dec 2021

Statistical significance or clinical significance? A researcher's dilemma for appropriate interpretation of research results

Hunny sharma.

Department of Community and Family Medicine, All India Institute of Medical Sciences, Raipur, Chhattisgarh, India

It is incredibly essential that the current clinicians and researchers remain updated with findings of current biomedical literature for evidence-based medicine. However, they come across many types of research that are nonreproducible and are even difficult to interpret clinically. Statistical and clinical significance is one such difficulty that clinicians and researchers face across many instances. In simpler terms, the P value tests all hypothesis about how the data were produced (model as whole), and not just the targeted hypothesis that it is intended to test (such as a null hypothesis) keeping in mind how reliable are the of the research results. Most of the times it is misinterpreted and misunderstood as a measure to judge the results as clinically significant. Hence this review aims to impart knowledge about “P” value and its importance in biostatistics, also highlights the importance of difference between statistical and clinical significance for appropriate interpretation of research results.

Introduction

Currently, in the publish or perish era where most of the researches are judged based on their statistically significant findings, it is often difficult for young researchers to interpret the correct findings of the research. The recent development of high-speed and more sophisticated computing power, utilizing high-end computers and statistical software packages, has resulted in a significant increase in the use of statistical methods, tests for hypothesis testing and reporting to the health literature. Unluckily, the appropriate interpretation of research results from the clinical point of not received similar interest.[ 1 ] This imbalance from decades to determine the actual importance of statistical and clinical significance and publication of such results in reputed indexed journals had led researchers to consider statistically significant results also as a clinically important one. It is essential to understand that publications in reputed indexed journals do not indicate that appropriate study design or statistics methods were used. This dilemma of the young researchers creates obstacles in their clinical decision-making and ultimately affects their role in Evidence-based practice.[ 2 ]

Researchers must realize that a clinical study is valuable and is of importance to clinical practice when the results are appropriately interpreted. Every year hundreds of studies and clinical trials are conducted to test different hypothesis. These trials are entirely dependent on appropriate statistical tests to assess whether new therapies or treatment protocol are better in clinical practice as compared to the usual approach or methods. Researchers should understand what is the importance of both statistical and clinical significance.[ 3 ]

When looking from a clinical point of view, the statistically significant difference among groups is not of prime importance. If a well-conducted study shows a difference in treatment options within two groups, it is of prime importance to know whether that difference is of clinically importance or not.[ 4 ] Since sample size and measurement variability can easily influence the statistical results, a nonsignificant outcome does not imply that the new therapy or treatment protocol is not clinically useful.[ 5 , 6 ]

Hence this review aims to impart knowledge about “ P ” value and its importance in biostatistics, also highlights the importance of difference between statistical and clinical significance for appropriate interpretation of research results.

What does P value infer?

In simpler terms, the P value tests all hypothesis about how the data were produced (the whole model), not just the targeted hypothesis that it is intended to test (such as a null hypothesis).[ 7 ]

The P value is the likelihood that if every model assumption, including the test hypothesis, were correct, the chosen test statistic would have been at least as large as its observed value.[ 7 ]

The most common threshold value for the “ P ” we find in biomedical literature is 0.05 (or 5%), and most often the P value is distorted into a dichotomous number where results are considered “statistically significant” when P falls on or below a cut-off (usually 0.05) and otherwise its declared “nonsignificant”.[ 7 ]

Why are “ P ” values not enough?

According to Ron Wasserstein, ASA's executive director, the P value was never meant to substitute the scientific reasoning, which is of greater interest. P value, which is a number whose value can range from zero to one in relation to a threshold value, represents the probability that the difference between the groups is not by chance. A well-reasoned and scientifically driven explanation will always remain the basis of reporting scientific outcomes.[ 8 ]

On what factors does the “ P ” value depend?

It should be borne in mind that the “ P ” value only represents that to what extent the data are inconsistent or incompatible with a given specific statistical model (i.e., null hypothesis). Hence it only aims to accept or reject the null hypothesis rather than focusing on the research hypothesis. From a statistical point of view, it measures the strength of evidence against the null hypothesis.[ 9 ]

With the advancement in biostatistics, it is now clear that the “ P ” value can easily be affected by various factors like sample size, the magnitude of the relationship and error. Each of these factors can work independently or in combination to distort the study findings based on “ P ” values.[ 10 ]

(1) Effect of error on “ P ” values

In general, two types of errors that is, systematic and random error effects the “ P ” value.

“Systematic errors,” that is, “Non-random errors” of certain significant magnitude distorts the research results towards a specific direction or can result in altered observed association in either direction. This type of error generally occurs when a single examiner takes the measurement leading to an unintended bias of deviating the research results to his/her expectations or may also result from modification of the measuring technique. Hence, Systematic error is a systematic flaw in the measurement of a variable due to methodological error leading to underestimation or overestimation of measurements. The extent of systematic errors can be determined by re-examination and re-measurement of a certain sufficient number (i.e., 20%, not always applicable) of individuals again by material and method used in the agreement. Some statistical tests like paired t -test, the intraclass correlation and the Bland-Altman method can also help in the determination of systematic errors.[ 10 , 11 , 12 ]

A “random error” is defined as a variability of the data which cannot be explained. Random errors of high magnitude means trouble in reproducibility of the measurements, which may result in questionable methodology and questionable examiners’ ability. This occurs randomly across the population, ultimately distorting the results. Random errors can be minimized by taking a large number of samples or measurements. Let us understand this by taking an example of measuring Mid-Upper Arm Circumference (MUAC) of the population. While measuring the MUAC of each individual in the population, random error may exhibit itself in the form of random MUAC among individuals that is, less or more MUAC measured as compared to the actual measurement. This may be a result of how the tape was held while taking the measurement, at what position it was measured (ideally midway between the olecranon process and the acromion), and who was the researcher who took the measurement. Random error can be reduced by incorporating a large number of samples or measurements that is, the more study participants are included in these measurements, the smaller the effect of random error will become.[ 10 ]

(2) Effect of sample size on “ P ” values

It is well known that the P value depends on the sample size to a vast extent. More the sample size less will be the variability of the measurement or data, and more precise will be the measurement for the study population. With an increase in sample size, the magnitude of random error decreases and the study is more likely to find a significant relationship if it exists.[ 10 ]

(3) Effect of magnitude of relationship between groups on “ P ” values

P -value also relies on the magnitude of difference or relationship between the groups compared. In simpler terms, if the magnitude of difference between the two groups is more substantial, then it will be easy to detect and will have a small P value.[ 10 ]

What are the American Statistical Association (ASA) principal statements on statistical significance and P values?

ASA on 8 th March 2016, in the event of the growing concern of misuse and misinterpretation of P values, gave six principal statements to improve conduct and interpretation of quantitative research and increase research reproducibility. The six principal statements issued regarding significance and P value which are as follows:

• P -value shows the extent of incompatibility of the data with the stated statistical model.[ 8 ]
• P -value is neither the measure of the probability of the studied hypothesis being true nor is the representation of the probability that study data were produced by random chance alone.[ 8 ]
• It is extremely important to note that any business model, policy decision, or conclusion related to any scientific study or experiment should not be based on the P value and merely on the fact whether it passes a specific threshold or not.[ 8 ]
• It is the moral duty of the authors and researchers to report the research or experimental findings to its full extent and with transparency.[ 8 ]
• A P value is neither represents the importance of research results nor is the representation of the effect size of the study.[ 8 ]
• P -value does not give a sufficient measure of evidence regarding a model or “hypothesis”.[ 8 ]

What are clinically significant outcomes?

The term “clinically significant” can be used for the researches in which clinically relevant results or outcomes are used to assess the effectiveness or efficacy of a treatment modality. When used the term “clinically significant” findings are those who make the patient improves the quality of life and makes him/her feel, function well.[ 13 ]

Clinically significant findings are those which improve medical care resulting in the improvement of individual's physical function, his/her mental status, and ability to engage in social life. The term improvement of quality of life in medical care deals with both subjective and objective terms. Here the term objective deals with improvement in performance status, duration of remission of disease, and prolongation of life-span, while subjective improvement in quality-of-life deals with improved mood, attitude, physical and social activity, feeling of general wellbeing, and the alleviation of distressing symptoms like pain, weakness, and discomfort.

Since statistical significance results do not necessarily mean that the results are clinically relevant and lead to improvement in the quality of life of the individuals. Hence, many outcomes can be statistically significant but not clinically relevant in a clinical point of view. Hence, clinicians and researchers should give importance to both statistical and clinical significance.[ 13 ]

A clinically relevant intervention justifies the effects which over-benefits the associated costs, harm, and the inconveniences caused to the individuals for whom it is targeted. The main difference between statistical and clinical significance is that the clinical significance observes dissimilarity between the two groups or the two treatment modalities, while statistical significance implies whether there is any mathematical significance to the carried analysis of the results or not.

Different studies can have a similar statistical significance but may differ significantly in clinical significance. Let's consider an example of two different chemotherapy agents for cancer. The first study estimates to increase the survival of treated patient with Drug A (Less Expensive than usual chemotherapeutic agents) by five years ( P = 0.01) and alpha being 0.005, similarly a Second study utilizing Drug B (Expensive than usual chemotherapeutic agents) estimates to increase the survival of treated patient by mere five months ( P = 0.01) and alpha being 0.005. In both cases, the statistical test is significant, but Drug B only increases the survival by only five months which is not clinically significant as compared to Drug A which increases survival by five years, nor useful in terms of cost-effectiveness and superiority when compared to already available chemotherapeutic agents.[ 14 , 15 ]

Hence from the above description of statistically significant and clinically significant results, it is clear that both the notations have the importance of their own. The statistically significant results may not of clinical importance, vice versa the results which are of clinical importance may not be statistically significant. It is high time now that the researchers, journal editors, and readers should take a keen interest in looking beyond the threshold “ P ” value and also consider the results from a clinical point of view rather than just assessing the worth of research by considering the “ P ” values. All the researchers should also take into account the design, sample size, effect size of the study, bias incorporated, and reproducibility of the study while analyzing the study results. An aware researcher with a logically and critically thinking mind is in the best position to evaluate research results and thereby applying them to practice evidence-based medicine. Logically, discussion of the clinically significant research results will increase discussion and understanding of the new treatment modalities and will help in the upliftment of evidence-based practice.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

• Open access
• Published: 14 November 2023

Employment of patients with rheumatoid arthritis - a systematic review and meta-analysis

• Lilli Kirkeskov 1 , 2 &
• Katerina Bray 1 , 3  

BMC Rheumatology volume  7 , Article number:  41 ( 2023 ) Cite this article

1678 Accesses

1 Citations

7 Altmetric

Metrics details

Patients with rheumatoid arthritis (RA) have difficulties maintaining employment due to the impact of the disease on their work ability. This review aims to investigate the employment rates at different stages of disease and to identify predictors of employment among individuals with RA.

The study was carried out according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines focusing on studies reporting employment rate in adults with diagnosed RA. The literature review included cross-sectional and cohort studies published in the English language between January 1966 and January 2023 in the PubMed, Embase and Cochrane Library databases. Data encompassing employment rates, study demographics (age, gender, educational level), disease-related parameters (disease activity, disease duration, treatment), occupational factors, and comorbidities were extracted. Quality assessment was performed employing Newcastle–Ottawa Scale. Meta-analysis was conducted to ascertain predictors for employment with odds ratios and confidence intervals, and test for heterogeneity, using chi-square and I 2 -statistics were calculated. This review was registered with PROSPERO (CRD42020189057).

Ninety-one studies, comprising of a total of 101,831 participants, were included in the analyses. The mean age of participants was 51 years and 75.9% were women. Disease duration varied between less than one year to more than 18 years on average. Employment rates were 78.8% (weighted mean, range 45.4–100) at disease onset; 47.0% (range 18.5–100) at study entry, and 40.0% (range 4–88.2) at follow-up. Employment rates showed limited variations across continents and over time. Predictors for sustained employment included younger age, male gender, higher education, low disease activity, shorter disease duration, absence of medical treatment, and the absence of comorbidities.

Notably, only some of the studies in this review met the requirements for high quality studies. Both older and newer studies had methodological deficiencies in the study design, analysis, and results reporting.

Conclusions

The findings in this review highlight the prevalence of low employment rates among patients with RA, which increases with prolonged disease duration and higher disease activity. A comprehensive approach combining clinical and social interventions is imperative, particularly in early stages of the disease, to facilitate sustained employment among this patient cohort.

Peer Review reports

Rheumatoid arthritis (RA) is a chronic, inflammatory joint disease that can lead to joint destruction. RA particularly attacks peripheral joints and joint tissue, gradually resulting in bone erosion, destruction of cartilage, and, ultimately, loss of joint integrity. The prevalence of RA varies globally, ranging from 0.1- 2.0% of the population worldwide [ 1 , 2 ]. RA significantly reduces functional capacity, quality of life, and results in an increase in sick leave, unemployment, and early retirement [ 3 , 4 , 5 ]. The loss of productivity due to RA is substantial [ 2 , 5 , 6 , 7 ]. A 2015 American study estimated the cost of over $250 million annually from RA-related absenteeism in United States alone [ 8 ].

Research has highlighted the importance of maintaining a connection to the labour market [ 3 , 9 ], Even a short cessation from work entails a pronounced risk of enduring work exclusion [ 10 ]. In Denmark merely 55% on sick leave for 13 weeks succeeded in re-joining the workforce within one year. Among those on sick leave for 26 weeks, only 40% returned to work within the same timeframe [ 11 ]. Sustained employment is associated with an improved health-related quality of life [ 12 , 13 ]. Early and aggressive treatment of RA is crucial for importance in achieving remission and a favourable prognosis reducing the impact of the disease [ 2 , 14 , 15 , 16 ]. Therefore, initiating treatment in a timely manner and supporting patients with RA in maintaining their jobs with inclusive and flexible workplaces if needed is critical [ 3 , 17 ].

International studies have indicated, that many patients with RA are not employed [ 18 ]. In 2020, the average employment rate across Organization for Economic Co-operation and Development (OECD) countries was 69% in the general population (15 to 64 years of age), exhibiting variations among countries, ranging from 46–47% in South Africa and India to 85% in Iceland [ 19 ]. Employment rates were lower for individuals with educational levels below upper secondary level compared to those with upper secondary level or higher education [ 19 ]. For individuals suffering with chronic diseases, the employment rates tend to be lower. Prognostic determinants for employment in the context of other chronic diseases encompasses the disease’s severity, employment status prior to getting a chronic disease, and baseline educational level [ 20 , 21 , 22 ]. These somatic and social factors may similarly influence employment status of patients with RA. Several factors, including the type of job (especially physically demanding occupations), support from employers and co-workers, social safety net, and disease factors such as duration and severity, could have an impact on whether patients with RA are employed [ 17 , 23 , 24 ]. Over the years, politicians and social welfare systems have tried to improve the employment rates for patients with chronic diseases. In some countries, rehabilitation clinics have been instrumental in supporting patients to remain in paid work. Healthcare professionals who care for patients with RA occupy a pivotal role in preventing work-related disability and support the patients to remain in work. Consequently, knowledge of the factors that contribute to retention of patients with RA at work is imperative [ 17 , 25 ].

The aim of this study is therefore to conduct a systematic review, with a primary focus on examining employment rates among patients with RA at the onset of the disease, at study entry, and throughout follow-up. Additionally, this study intends to identify predictors of employment. The predefined predictors, informed by the author’s comprehensive understanding of the field and specific to RA, encompass socioeconomic factors such as age, gender, level of education, employment status prior to the disease, disease stage and duration, treatment modalities, and comorbidities, including depression, which are relevant both to RA and other chronic conditions [ 26 ].

This systematic review was carried out according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) for studies that included employment rate in patients with rheumatoid arthritis [ 27 ]. PROSPERO registration number: CRD42020189057.

Selection criteria and search strategies

A comprehensive literature search was conducted, covering the period from January 1966 to January 2023 across the PubMed, Embase, and Cochrane Library databases using the following search terms: (Rheumatoid arthritis OR RA) AND (employment OR return to work). Only studies featuring a minimum cohort size of thirty patients and articles in the English language were deemed eligible for inclusion.

The initial screening of articles was based on the titles and abstracts. Studies comprising a working-age population, with current or former employment status, and with no limitations to gender, demographics, or ethnicity were included in this review. Articles addressing topics of employment, work ability or disability, return to work or disability pension were encompassed within the scope of this review. Full-time and part-time employment, but not ‘working as housewives’ was included in this review’s definition of employment. Studies involving other inflammatory diseases than RA were excluded. Reference lists in the selected articles were reviewed, and more articles were included if relevant. A review of the reference lists in the initially selected articles was conducted, with additional articles incorporated if they proved relevant to the research objectives. The eligible study designs encompassed cohort studies, case–control studies, and cross-sectional studies. All other study designs, including reviews, case series/case reports, in vitro studies, qualitative studies, and studies based on health economics were systematically excluded from the review.

Data extraction, quality assessment and risk-of-bias

The data extraction from the selected articles included author names, year of publication, study design, date for data collection, employment rate, study population, age, gender, educational level, ethnicity, disease duration, and pharmacological treatment. To ensure comprehensive evaluation of study quality and potential bias, quality assessment was independently assessed by two reviewers (LK and KB) using the Newcastle–Ottawa Scale (NOS) for cross-sectional and cohort studies [ 28 ]. Any disparities in the assessment were resolved by discussion until consensus was reached. For cross-sectional studies the quality assessment included: 1) Selection (maximum 5 points): representativeness of the sample, sample size, non-respondents, ascertainment of the risk factor; 2) Comparability (maximum 2 points); study controls for the most important, and any additional factor; 3) Outcome (maximum 3 points): assessment of outcome, and statistical testing. For cohort studies the assessment included: 1) Selection (maximum 4 points): representativeness of the exposed cohort, selection of the non-exposed cohort, ascertainment of exposure, demonstration that the outcome of interest was not present at start of study; 2) Comparability (maximum 2 points): comparability of cohorts on the basis of the design or analysis; 3) Outcome (maximum 3 points): assessment of outcome, was the follow-up long enough for outcomes to occur, and adequacy of follow up of cohorts. The rating scale was based on 9–10 items dividing the studies into high (7–9/10), moderate (4–6) or low (0–3) quality. A low NOS score (range 0–3) indicated a high risk of bias, and a high NOS score (range 7–9/10) indicated a lower risk of bias.

Analytical approach

For outcomes reported in numerical values or percentages, the odds ratio along with their 95% confidence intervals (CI) were calculated, whenever feasible. Weighted means were calculated, and comparisons between these were conducted using t-test for unpaired data. Furthermore, meta-analysis concerning the pre-determined and potentially pivotal predictors for employment status, both at disease onset, study entry, and follow-up was undertaken. The predictors included age, gender, ethnicity, level of education, duration of disease, treatment, and the presence of comorbities, contingent upon the availability of the adequate data. Additionally, attempts have been made to find information regarding on job categorizations, disease activity (quantified through DAS28; disease activity score for number of swollen joints), and quality of life (SF-36 scores ranging from 0 (worst) to 100 (best)). Age was defined as (< = 50/ > 50 years), gender (male/female), educational level college education or more/no college education), race (Caucasian/not Caucasian), job type (non-manual/manual), comorbidities (not present/present), MTX ever (no/yes), biological treatment ever (no/yes), prednisolone ever (no/yes), disease duration, HAQ score (from 0–3)), joint pain (VAS from 1–10), and DAS28 score. Age, disease duration, HAQ score, VAS score, SF36 and DAS28 were in the studies reported by mean values and standard deviations (SD). Challenges were encountered during attempts to find data which could be used for analysing predictors of employment status before disease onset, and at follow-up, as well as factors related to treatments beyond MTX, prednisolone, and biological as predictors for being employed after disease onset. Test for heterogeneity was done using Chi-squared statistics and I 2 , where I 2 below 40% might not be important; 30–60% may represent moderate heterogeneity; 50–90% substantial heterogeneity; and 75–100% considerable heterogeneity. Meta-analysis for predictors for employment and odds ratio; confidence intervals; and test for heterogeneity were calculated using the software Review Manager (RevMan, version 5.3. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2014).

General description of included studies

The search yielded a total of 2277 references addressing RA its association with employment. Following the initial title screen, 199 studies were considered relevant for further evaluation. Of those, 91 studies ultimately met the inclusion criteria. Figure  1 shows the results of the systematic search strategy.

Flow chart illustrating the systematic search for studies examining employment outcome in patients with rheumatoid arthritis

Table 1 summarizes the general characteristics of the included studies. The publication year of the included studies ranged from 1971 to 2022. Among the studies, 60 (66%) adopted a cross-sectional research design [ 13 , 18 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 129 ] with a total of 41,857 participants analysing data at a specific point in time. Concurrently, 31 studies (34%) adopted a cohort design [ 89 , 90 , 91 , 92 , 93 , 94 , 95 , 96 , 97 , 98 , 99 , 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 , 120 , 121 , 122 , 130 ] with a total of 59,974 participants. Most of these studies exhibited a small to moderate sample size, with a median of 652 participants. Additionally, single centre studies and studies from high-income countries were predominant. Study details are shown in Table 1 .

General description of study participants

On average, patients with RA were 51 years old, with an age range spanning from 42 to 64 years. Furthermore, the female population accounted for 75.9% of the patient cohort, with a range from 41 to 92%. The duration of the disease at study entry exhibited significant variability, ranging from less than one year up to more than 18 years on average.

• Employment rate

At disease onset, the employment rate was 78.8% (weighted mean, range 45.4–100), at study entry 47.0% (range 18.5–100), and during the follow-up period 40.0% (range 4–88.2), as shown in Table 2 . Notably, a comparative analysis of the employment rates between Europe and North America indicated no substantial difference ( p  = 0.93). However, the comparison between Europe, North America and ‘other continents’ did yield significant differences (or nearly differences) with p -values of 0.003 and 0.08, respectively.

The employment rate exhibited no change, when comparing studies from the 1980s through to 2022. Specifically, the weighted mean for the years 1981–2000 was 49.2%, aligning closely with the corresponding figures for the years 2001–2010 (49.2%) and 2011–2022 43.6%. These findings were statistically non-significant, with p -values of 0.80 for comparison between year 1981–2000 and 2001–2010; 0.66 for 2001–2010 and 2011–2022, and 0.94 for 1981–2000 and 2011–2022, shown in Figure S 1 , see Additional file.

Among the studies included in the analysis, nineteen studies included data of employment at follow-up, with durations ranging from 1 to 20 years, Table 2 . For instance, Jäntti, 1999 [ 97 ] reported an employment rate 69% one year after disease onset, which gradually declined to 50% after 15 years and further to 20% after 20 years. Similarly, Mäkisara, 1982 [ 63 ] demonstrated that 60% of the patients were employed 5 years after disease onset, 50% after 10 years, and 33% after 15 years. Nikiphorou, 2012 [ 101 ] reported an employment rate of 67% at study entry, which decreased to 43% after 10 years.

In addition, seven studies included data of employment rate among patients comparing different medical treatments [ 18 , 44 , 56 , 91 , 105 , 110 , 119 ]. These studies indicated that, on average, 55.0% (weighted mean) of the patients were employed after receiving treatment with MTX, while 42.8% after undergoing treatment with a combination of MTX + Adalimumab (all patients were employed at disease onset in these specific studies).

Predictors for employment

Information of normative comparison data to use for meta-analysis of predictors for employment at study entry was available for age, gender, educational level, race, job type, comorbidities, MTX at any time, biological treatment at any time, prednisolone at any time, disease duration, HAQ score, joint pain (VAS-score), and disease activity (DAS28 score). Predictors for employment at study entry was being younger /age below 50 years, being a male, higher educational level (college or more), non-manual work, having no comorbidities, no medical treatment, short disease duration, and low HAQ score, VAS-score, or DAS28 score. Heterogeneity was small for age, gender, medical treatment, and moderate for educational level, and job type as indicted by the I 2 values, Table  3 , and shown in detail in Figures S 2 , S 3 , S 4 , S 5 , S 6 , S 7 , S 8 , S 9 , S 10 , S 11 , S 12 , S 13 , S 14 , S 15 and S 16 , see Additional file.

Assessment of quality of included studies

All studies were subject to rigorous quality assessment. These assessments resulted in categorisation of either medium quality ( n  = 64; 70%) or high-quality studies ( n  = 27; 30%), with no studies falling into the low-quality category. The quality assessment is shown in Tables  4 and 5 .

Notably, many studies were characterised by several common attributes, including cross-sectional study design, single-centre-settings, relatively small sample sizes, and the reliance on self-reported patient data. When including only the high-quality studies in the analyses, the employment rates at study entry changed from 47% (weighted mean, all studies) to 50% (weighted mean, high quality studies).

Key findings

This systematic review has identified a decline in the employment rate among patients with RA, with a notable decrease from disease onset during the study entry to follow-up, where only half of the patients were employed. These findings corroborate earlier research that indicated a substantial decline in employment rates among patients with RA over time. Notably, previous studies have reported that approximately one third of patients with RA stopped working within 2 to 3 years after disease onset, and more than half was unable to work after 10 to 15 years [ 23 , 63 , 93 , 97 , 101 ]. Only few studies have included data from the general population, comparing the employment rates with the rates for patients with RA [ 89 , 90 ]. Comparisons with the general population further underscored the challenges faced by RA patients, as their employment rates were consistently lower.

Despite changes in medical treatment, social security systems, and societal norms over the past decades, there was no significant improvement in the employment for patients with RA. This pattern aligns with data from the Global Burden of Disease studies, highlighting the persistent need for novel approaches and dedicated efforts to support patients with RA in sustaining employment [ 2 , 123 ]. Recent recommendations from EULAR (European Alliance of Associations for Rheumatology) and ACR (American College of Rheumatology) have emphasized the importance of enabling individuals with rheumatic and musculoskeletal diseases to engage in healthy and sustainable work [ 17 , 124 , 125 ].

While different countries possess different social laws and health care systems for supporting patients with chronic diseases, the variations in the weighted mean of employment rates across countries were relatively minor.

In the meta-analysis, one of the strongest predictors for maintaining employment was younger age at disease onset [ 43 , 51 , 101 , 116 ]. Verstappen, 2004 found that older patients with RA had an increased risk of becoming work disabled, potentially caused by the cumulative effects of long-standing RA, joint damage, and diminished coping mechanisms, compared to younger patients [ 23 ].

More women than men develop RA, however this study showed that a higher proportion of men managed to remain employed compared to women [ 18 , 36 , 42 , 43 , 46 , 62 , 71 , 89 , 101 , 116 ]. Previous studies have shown inconsistent results in this regard. Eberhart, 2007 found that a significantly higher number of men with RA worked even though there was no difference in any disease state between the sexes [ 93 ]. De Roos,1999 showed that work-disabled women were less likely to be well-educated and more likely to be in a nonprofessional occupation than working women. Interestingly, there was no association of these variables among men. Type of work and disease activity may influence work capacity more in women than in men [ 46 ]. Sokka, 2010 demonstrated a lower DAS28 and HAQ-score in men compared to women among the still working patients with RA, which indicated that women continued working at higher disability and disease activity levels compared with men [ 18 ].

Disease duration also played a significant role as a predictor of employment outcomes [ 33 , 36 , 45 , 71 , 77 , 86 , 102 , 111 ]. Longer disease duration correlate with decreased employment likelihood, which could be attributed to older age and increased joint damage and disability in patients with longer-standing RA.

Higher educational levels were associated with a greater possibility of employment [ 30 , 43 , 45 , 46 , 51 , 62 , 86 ]. This is probably due to enhanced job opportunities, flexibility, lower physical workload, better insurance coverage, and improved health care for well-educated individuals. This is further supported by the fact that having a manual work was a predictor for not being employed [ 30 , 39 , 43 , 44 , 45 ].

Furthermore, health-related quality of life, as measured by SF 36, lower disease activity (DAS28 scores), reduced joint pain (VAS-score), and lower disability (HAQ score) were additionally predictors for being employed [ 33 , 35 , 36 , 45 , 71 , 86 ]. This support the statement that the fewer symptoms from RA, the greater the possibility of being able to work.

The results showed that the presence of comorbidity was a predictor for not being employed, aligning with findings from previous studies that chronic diseases such as cardiovascular disease, lung disease, diabetes, cancer, and depression reduced the chances of being employed [ 126 ]. Moreover, the risk of exiting paid work increased with multimorbidity [ 127 ].

While limited data were available for assessing the impact of treatment on employment, indications suggested that patients with RA were receiving medical treatments, such as MTX or biological medicine, were more likely to be unemployed. One possible explanation for this phenomenon could be that patients with RA, who were receiving medical treatment, had a more severe and a longer duration of RA compared to those, who had never been on medical treatment. However, the scarcity of relevant studies necessitates caution when drawing definitive conclusions in this regard.

Therefore, the predictors for employment found in this review were being younger, being a male, having higher education, low disease activity, low disease duration, and being without comorbidities. This is supported by previous studies [ 93 , 116 ]

In summary, this review underscores the importance of managing disease activity, offering early support to patients upon diagnosis, and reducing physically demanding work to maintain employment among patients with RA. Achieving success in this endeavour requires close cooperation among healthcare professionals, rehabilitation institutions, companies, and employers. Furthermore, it is important that these efforts are underpinned by robust social policies that ensure favourable working conditions and provide financial support for individuals with physical disabilities, enabling them to remain active in the labour market.

Strengths and limitations

The strength of this review and meta-analysis lies in the inclusion of a large number of articles originating from various countries. Furthermore, the data showed a consistent employment rate in high quality studies compared to all studies. However, there are some limitations to this review. No librarian was used to define search terms and only three databases were searched. Furthermore, the initial search, selection of articles, data extraction, and analysis was undertaken only by one author, potentially leading to the omission of relevant literature and data. The review also extended back to 1966, with some articles from the 1970s and 1980s included. Given the significant changes in medical treatment, social security systems, and society over the past decades, the generalizability of the findings may be limited.

Moreover, the majority of studies did not include a control group from the general population, which limited the ability to compare employment rates with the general population in the respective countries. Many studies were cross-sectional in design, which limits the evidence of causality between employment rate and having RA. However, the employment rate was approximately the same in high quality studies compared to all studies, which supports an association. A substantial number of studies relied on self-reported employment rates, introducing the potential for recall bias. Additionally, many studies did not account for all relevant risk factors for unemployment failing to control for all relevant confounders.

EULAR have made recommendation for point to consider when designing, analysing, and reporting of studies with work participation as an outcome domain in patients with inflammatory arthritis. These recommendations include study design, study duration, and the choice of work participation outcome domains (e.g., job type, social security system) and measurement instruments, the power to detect meaningful effects, interdependence among different work participation outcome domains (e.g., between absenteeism and presentism), the populations included in the analysis of each work participation outcome domain and relevant characteristics should be described. In longitudinal studies work-status should be regularly assessed and changes reported, and both aggregated results and proportions of predefined meaningful categories should be considered [ 128 ]. Only some of the studies in this review met the requirements for high quality studies. In both older and newer studies methodological deficiencies persisted in study design, analysis, and reporting of results, as recommended by EULAR.

Perspectives for future studies

Future research in this area should focus on developing and evaluating new strategies to address the ongoing challenges faced by patients with RA in maintaining employment. Despite many initiatives over the years, there has been no success in increasing employment rates for patients with RA in many countries. Therefore, there is a pressing need for controlled studies that investigated the effectiveness of interventions such as education, social support, and workplace adaptations in improving employment outcomes for these individuals.

This systematic review underscores the low employment rate among patients with RA. Key predictors of sustained employment include being younger, having higher educational level, short disease duration, and lower disease activity, along with fewer comorbidities. Importantly, the review reveals that the employment rate has not changed significantly across different time periods. To support patients with RA in maintaining their employment, a comprehensive approach that combines early clinical treatment with social support is crucial. This approach can play a pivotal role in helping patients with RA stay connected to the labour market.

Availability of data and materials

The datasets used and/or analyzed during the current study are available in the supplementary file.

Abbreviations

• Rheumatoid arthritis

Methotrexate

Newcastle Ottawa Quality Assessment Scale

Standard deviation

Not analyzed

Not relevant

Disease activity

Health Assessment Questionnaire

Visual analog scale for pain

European Alliance of Associations for Rheumatology

American College of Rheumatology

Almutairi K, Nossent J, Preen D, Keen H, Inderjeeth C. The global prevalence of rheumatoid arthritis: a meta-analysis based on a systematic review. Rheumatol Int. 2021;41:863–77.

Article   PubMed   Google Scholar  

Safiri S, Kolahi AA, Hoy D, Smith E, Bettampadi D, Mansournia MA, et al. Global, regional and national burden of rheumatoid arthritis 1990–2017: a systematic analysis of the Global Burden of Disease study 2017. Ann Rheum Dis. 2019;78:1463–71.

Verstappen SMM. Rheumatoid arthritis and work: The impact of rheumatoid arthritis on absenteeism and presenteeism. Best Pract Res Clin Rheumatol. 2015;29:495–511.

Madsen CMT, Bisgaard SK, Primdahl J, Christensen JR, von Bülow C. A systematic review of job loss prevention interventions for persons with inflammatory arthritis. J Occup Rehabil. 2021;4:866–85.

Kessler RC, Maclean JR, Petukhova M, Sarawate CA, Short L, Li TT, et al. The effects of rheumatoid arthritis on labor force participation, work performance, and healthcare costs in two workplace samples. J Occup Environ Med. 2008;50:88–98.

Filipovic I, Walker D, Forster F, Curry AS. Quantifying the economic burden of productivity loss in rheumatoid arthritis. Rheumatology. 2011;50:1083–90.

Burton W, Morrison A, Maclean R, Ruderman E. Systematic review of studies of productivity loss due to rheumatoid arthritis. Occup Med. 2006;56:18–27.

Article   Google Scholar  

Gunnarsson C, Chen J, Rizzo JA, Ladapo JA, Naim A, Lofland JH. The employee absenteeism costs of reumatoid arthritis. Evidence from US National Survey Data. J Occup Environ Med. 2015;57:635–42.

van der Noordt M, Ijzelenberg H, Droomers M, Proper KI. Health effects of employment: a systematic review of prospective studies. Occup Environ Health. 2014;71:730–6.

Google Scholar  

Virtanen M, Kivimäki M, Vahtera J, Elovainio M, Sund R, Virtanen P, et al. Sickness absence as a risk factor for job termination, unemployment, and disability pension among temporary and permanent employees. Occup Environ Med. 2006;63:212–7.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Vilhelmsen J. Længerevarende sygefravær øger risikoen for udstødning [long-term sick-leave increase the risk of job termination]. 2007.  https://www.ae.dk/analyse/2007-10-laengerevarende-sygefravaer-oeger-risikoen-for-udstoedning .

Grønning K, Rødevand E, Steinsbekk A. Paid work is associated with improved health-related quality of life in patients with rheumatoid arthritis. Clin Rheumatol. 2010;29:1317–22.

Article   PubMed   PubMed Central   Google Scholar  

Chorus AMJ, Miedema HS, Boonen A, van der Linden S. Quality of life and work in patients with rheumatoid arthritis and ankylosing spondylitis of working age. Ann Rheum Dis. 2003;62:7.

Ma MHY, Kingsley GH, Scott DL. A systematic comparison of combination DMARD therapy and tumour necrosis inhibitor therapy with methotrexate in patients with early rheumatoid arthritis. Rheumatology (Oxford). 2010;49:91–8.

Article   CAS   PubMed   Google Scholar  

Vermeer M, Kuper HH, Hoekstra M, Haagsma CJ, Posthumus MD, Brus HL, et al. Implementation of a treat-to-target strategy in very early rheumatoid arthritis. Results of the Dutch arthritis monitoring remission induction cohort study. Arthritis Rheum. 2011;63:2865–72.

Vermeer M, Kuper HH, Bernelot Moens HJ, Drossaers-Bakker KW, van der Bijl AE, van Riel PL, et al. Sustained beneficial effects of a protocolized treat-to-target strategy in very early rheumatoid arthritis: three-year results of the Dutch rheumatoid arthritis monitoring remission induction cohort. Arthritis Care Res. 2013;65:1219–26.

Article   CAS   Google Scholar  

Boonen A, Webers C, Butink M, Barten B, Betteridge N, Black DC, et al. 2021 EULAR points to consider to support people with rheumatic and musculoskeletal diseases to participate in healthy and sustainable paid work. Ann Rheum Dis. 2023;82:57–64.

Sokka T, Kautianen H, Pincus T, Verstappen SMM, Aggarwai A, Alten R, et al. Work disability remains a major problem in rheumatoid arthritis in the 2000s: data from 32 countries in the QUEST-RA study. Arthritis Res Ther. 2010;1(R42):1–10.

OECD. Employment rate (indicator). 2020. https://dataoecd.org/emp/employment-rate.htm . Assessed on 11 May.

Hannerz H, Pedersen BH, Poulsen OM, Humle F, Andersen LL. A nationwide prospective cohort study on return to gainful occupation after stroke in Denmark 1996–2006. BMJ Open. 2011;1:1–5.

Tumin D, Chou H, Hayes D Jr, Tobias JD, Galantowicz M, McConnell PI. Employment after hearth transplantation among adults with congenital heart disease. Congenit Heart Dis. 2017;12:794–9.

Islam T, Dahlui M, Majid HA, Nahar AM, MohdTaib NA, Su TT, MyBCC study group. Factors associated with return to work of breast cancer survivors: a systematic review. BMC Public Health. 2014;14:1–13.

Verstappen SMM, Bijlsma JWJ, Verkleij H, Buskens E, Blaauw AAM, Borg EJ, Jacobs JWG. Overview of work disability in rheumatoid arthritis patients as observed in cross-sectional and longitudinal surveys. Arthritis Rheum. 2004;51:488–97.

Wilkie R, Bjork M, Costa-Black KM, Parker M, Pransky G. Managing work participation for people with rheumatic and musculoskeletal diseases. Best Pract Res. 2020;34:1–16.

Varekamp I, Haafkens JA, Detaille SI, Tak PP, van Dijk FJH. Preventing work disability among employees with rheumatoid arthritis: what medical preofessionals can learn form patients’ perspective. Arthritis Rheum. 2005;53:965–72.

Kirkeskov L, Carlsen RK, Lund T, Buus NH. Emloyment of patients with kidney failure treated with dialysis or kidney transplantation - a systematic review and metaanalysis. BMC Nephrol. 2021;22–348:1–17.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2020;2021:372. https://doi.org/10.1136/bmj.n71 .

Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. Newcastle-Ottawa Scale (NOS) for assessing the quality if nonrandomized studies in meta-analyses. 2009. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp .

Al-Jabi SW, Seleit DI, Badran A, Koni A, Zyoud SH. Impact of socio-demographic and clinical characteristics on functional disability and health-related quality of life in patients with rheumatoid arthritis: a cross-sectional study from Palestine. Health Qual Life Outcomes. 2021;19:241.

Allaire SH, Anderson JJ, Meenan RF. Reducing work disability associated with rheumatoid arthritis: Identifiction of additional risk factors and persons likely to benefit from intervention. Arthritis Care Res. 1996;9(5):9.

Allaire S, Wolfe F, Niu J, Lavalley MP. Comtemporary prevalence and incidence of work disability associated with rheumatoid arthritis in the US. Arthritis Rheum. 2008;59(4):7.

Anno S, Sugioka Y, Inui K, Tada M, Okano T, Mamoto K. Evaluation of work disability in Japanese patients with rheumatoid arthritis: from the TOMORROW study. Clin Rheumatol. 2018;37:9.

Azevedo ABC, Ferraz MB, Ciconelli RM. Indirect costs of rheumatoid arthritis in Brazil. Value Health. 2008;11:869–77.

Backman CL, Kennedy SM, Chalmers A, Singer J. Participation in paid and unpaid work by adults with rheumatoid arthritis. J Rheumatol. 2004;31:47–57.

PubMed   Google Scholar  

Berner C, Haider S, Grabovac I, Lamprecht T, Fenzl KH, Erlacher L, et al. Work ability and employment in rheumatoid arthritis: a cross-sectional study on the role of muscle strength and lower extremity function. Int J Rheumatol. 2018;2018:11.

Bertin P, Fagnani F, Duburcq A, Woronoff AS, Chauvin P, Cukierman G, et al. Impact of rheumatoid arthritis on career progression, productivity, and employability: the PRET Study. Joint Bone Spine. 2016;83:6.

Bodur H, Borman P, Alper B, Keskin D. Work status and related variables in patients with rheumatoid arthitis and ankylosing spondylitis. Turk J Rheumatol. 2011;26(2):19.

Cadena J, Vinaccia S, Perez A, Rico MI, Hinojosa R, Anaya JM. The impact of disease activity on the quality of life, mental health status, and family dysfunction in Colombian patients with rheumatoid arthritis. Clin Rheumatol. 2003;9:142–50.

Callahan LF, Bloch DA, Pincus T. Identification of work disability in rheumatoid arthritis : physical, radiographic and laboratory variables do not add explanatory power to demographic and functional variables. J Clin Epidemiol. 1992;45(2):12.

Camilleri JP, Jessop AM, Davis S, Jessop JD, Hall M. A survey of factors affecting the capacity to work in patients with rheumatoid arthritis in South Wales. Clin Rehabil. 1995;9:4.

Chen MH, Lee MH, Liao HT, Chen WS, Lai CC, Tsai CY. Health-related quality of life outcomes in patients with rehumatoid arthritis and ankylosing spondylitis after tapering biologic treatment. Clin Rheumatol. 2018;37:429–38.

Chorus AMJ, Miedema HS, Wevers CJ, van der Linden S. Labour force participation among patients with rheumtoid arthritis. Ann Rheum Dis. 2000;59:6.

Chorus AMJ, Miedema HS, Wevers CJ, van der Linden S. Work factors and behavioural coping in relation to withdrawal from the labour force in patients with rheumatoid arthritis. Ann Rheum Dis. 2001;60:8.

Chung CP, Sokka T, Arbogast PG, Pincus T. Work disability in early rheumatoid arthritis: higher rates but better clinical status in Finland compared with the US. Ann Rheum Dis. 2006;65:5.

Dadoniene J, Stropuviene S, Venalis A, Boonen A. High work disability rate among rheumatoid arthritis patients in Lithuania. Arthritis Rheum. 2004;51:433–9.

De Roos AJ, Callahan LF. Differences by sex in correlates of work status in rheumatoid arthritis patients. Arthritis Care Res. 1999;12:381–91.

Dejaco C, Mueller T, Zamani O, Kurtz U, Egger S, Resch-Passini J, et al. A prospective study to evaluate the impact of Golimumab therapy on work productivity and activity, and quality of life in patients with rheumatoid arthritis, psoriatic arthritis and axil spondylarthritis in a real life setting in AUSTRIA. The Go-ACTIVE Study. Front Med. 2022;9:1–9.

Doeglas D, Suurmeijer T, Krol B, Sanderman R, van Leeuwen M, van Rijswijk M. Work disability in early rheumatoid arthritis. Ann Rheum Dis. 1995;54:6.

Fara N, Recchia O, Sequeira G, Sanchez K. Disability due to rheumatic diseases in the city of Junín, Argentina. Rheumatol Int. 2019;39:729–33.

Fifield J, Reisine S, Sheehan TJ, McQuillan J. Gender, paid work, and symptoms of emotional distress in rheumatoid arthritis patients. Arthritis Rheum. 1996;39:427–35.

Gomes RKS, Schreiner LC, Vieira MO, Machado PH, Nobre MRC. Staying in the labor force among patients with rheumatoid arthritis and associated factors in Southern Brazil. Adv Rheumatol. 2018;58(14):1–9.

Hamdeh HA, Al-Jabi SW, Koni A, Zyoud SH. Health-related quality of life and treatment satisfaction in Palestinians with rheumatoid arthritis: a cross-sectional study. BMC Rheumatol. 2022;6(19):1–12.

Hazes JM, Taylor P, Strand V, Purcaru O, Coteur G, Mease P. Physical function improvements and relief from fatigue and pain are associated with incresed productivity at work and at home in rheumatoid arthritis patients treated with certolizumab pegol. Rheumatology. 2010;49:1900–10.

Hulander E, Lindqvist HM, Wadell AT, Gjertsson I, Winkvist A, Bärebring L. Improvements in body composition after a proposed anti-inflammatory diet are modified by employment status in weight-stable patients with rheumatoid arthritis, a randomized controlled crossover trial. Nutrients. 2022;14:1058.

Intriago M, Maldonado G, Guerrero R, Moreno M, Moreno L, Rios C. Functional disability and its determinants in Ecudorian patients with rheumatoid arthritis. Open Access Rheumatol. 2020;12:97–104.

Kavanaugh A, Smolen JS, Emery P, Purcaru O, Keystone E, Richard L, et al. Effect of certolizumab pegol with ethotrexate on home and work place productivity and social activities in patients with active rheumatoid arthritis. Arthritis Rheum. 2009;61:1592–600.

Kwon JM, Rhee J, Ku H, Lee EK. Socioeconomic and employment status of patients with rheumatoid arthritis in Korea. Epidemiol Health. 2003;34:1–7.

Lacaille D, Sheps S, Spinelli JJ, Chalmers A, Esdaile JM. Identification of modifiable work-related factors that influence the risk of work disability in rheumatoid arthritis. Arthritis Rheum. 2004;51:843–52.

Lahiri M, Cheung PPM, Dhanasekaran P, Wong SR, Yap A, Tan DSH, et al. Evaluation of a multidisciplinary care model to improve quality of life in rheumatoid arthritis: a randomised controlled trial. Qual Life Res. 2022;31:1749–59.

Lapcevic M, Vukovic M, Gvozdenovic BS, Mioljevic V, Marjanovic S. Socioeconomic and therapy factor influence on self-reported fatigue, anxiety and depression in rheumatoid arthritis patients. Rev Bras Reumatol. 2017;57(6):12.

Mattila K, Buttgereit F, Tuominen R. Impact of morning stiffness on working behaviour and performance in people with rhematoid arthritis. Rheumatol Int. 2014;34:1751–8.

McQuillan J, Andersen JA, Berdahl TA, Willett J. Associations of rheumatoid arthritis and depressive symptoms over time: Are there differences by education, race/ethnicity, and gender? Arthritis Care Res. 2022;0:1–9.

Mäkisara GL, Mäkisara P. Prognosis of funcrional capacity and work capacity in rheumatoid arthritis. Clin Rheumatol. 1982;1(2):9.

Meenan RF, Yelin EH, Nevitt M, Epstein WV. The impact of chronic disease. A sociomedical profile of rheumatoid arthritis. Arthritis Rheum. 1981;24:544–9.

Morf H, Castelar-Pinheiro GR, Vargas-Santos AB, Baerwald C, Seifert O. Impact of clinical and psychological factors associated with depression in patients with rheumatoid arthritis: comparative study between Germany and Brazil. Clin Rheumatol. 2021;40:1779–87.

Newhall-Perry K, Law NJ, Ramos B, Sterz M, Wong WK, Bulpitt KJ, et al. Direct and indirect costs associated with the onset of seropositive rheumatoid arthritis. J Rheumatol. 2000;27:1156–63.

CAS   PubMed   Google Scholar  

Osterhaus JT, Purcaru O, Richard L. Discriminant validity, responsiveness and reliability of the rheumatoid arthritis-specific Work Productivity Survey (WPS-RA). Arthritis Res Ther. 2009;11(R73):1–12.

Pieringer H, Puchner R, Pohanka E, Danninger K. Power of national economy, disease control and employment status in patients with RA - an analytical multi-site ecological study. Clin Rheumatol. 2016;35:5.

Rosa-Gocalves D, Bernardes M, Costa L. Quality of life and functional capacity in patients with rheumatoid arthritis - Cross-sectional study. Reumatol Clin. 2018;14:360–6.

Sacilotto NC, Giorgi RDN, Vargas-Santos AB, Albuquerque CP, Radominski SC, Pereira IA, et al. Real - rheumatoid arthritis in real life - study cohort: a sociodemographic profile of rheumatoid arthritis in Brazil. Adv Rheumatol. 2020;60:20.

Shanahan EM, Smith M, Roberts-Thomson L, Esterman A, Ahern M. Influence of rheumatoid arthritis on work participation in Australia. Intern Med J. 2008;38:166–73.

Smolen JS, van der Heijde DM, Keystone EC, van Vollenhoven RF, Golding MB, Guérette B, et al. Association of joint space narrowing with impairment of physical function and work ability in patients with early rheumatoid arthritis: protection beyond disease control by adalimumab plus methotrexate. Ann Rheum Dis. 2012;72:1156–62.

Syngle D, Singh A, Verma A. Impact of rheumatoid arthritis on work capacity impairment and its predictors. Clin Rheumatol. 2020;39:1101–9.

Tamborenea MN, Pisoni C, Toloza S, Mysler E, Tate G. Pereira D et al Work instability in rheumatoid arthritis patients from Argentina: prevalence and associated factors. Rheumatol Int. 2015;35:107–14.

Tanaka Y, Kameda H, Saito K, Kanedo Y, Tanaka E, Yasuda S, et al. Response to tocilizumab and work productivity in patients with rheumatoid arthritis: 2-year follow-up of FIRST ACT-SC study. Mod Rheumatol. 2021;21:42–52.

van der Zee-Neuen A, Putrik P, Ramiro S, Keszei AP, Hmamouchi I, Dougados M, Boonen A. Large country differences in work outcomes in patients with RA - an analysis in the multinational study COMORA. Arthritis Res Ther. 2017;19:216.

van Jaarsveld CHM, Jacobs JWG, Schrijvers AJP, van Albada-Kuipers GA, Hofman DM, Bijlsma JWJ. Effects of rheumatoid arthritis on employment and social participation during the first years of disease in the Netherlands. Br J Rheumatol. 1998;37:848–53.

Verstappen SMM, Boonen A, Bijlsma JWJ, Buskens E, Verkleij H, Schenk Y, et al. Working status among Dutch patients with rheumatoid arthritis: work disability and working conditions. Rheumatology. 2005;44:202–6.

Vliet Vlieland TPM, Buitenhuis NA, van Zeben D, Vandenbroucke JP, Breedveld FC, Hazes JMW. Sociodemographic factors and the outcome of rheumatoid arthritis in young women. Ann Rheum Dis. 1994;53:803–6.

Li F, Ai W, Ye J, Wang C, Yuan S, Xie Y, et al. Inflammatory markers and risk factors of RA patients with depression and application of different scales in judging depression. Clin Rheumatol. 2022;41:2309–17.

Wan SW, He HG, Mak A, Lahiri M, Luo N, Cheung PP, et al. Health-related quality of life and its predictors among patients with rheumatoid arthritis. Appl Nurs Res. 2016;30:176–83.

Xavier RM, Zerbini CAF, Pollak DF, Morales-Torres JLA, Chalem P, Restrepo JFM, et al. Burden of rheumatoid arthritis on patients’ work productivity and quality of life. Adv Rheumatol. 2019;59:47.

Yajima N, Kawaguchi T, Takahashi R, Nishiwaki H, Toyoshima Y, Oh K, et al. Adherence to methotrexate and associated factors considering social desirability in patients with rheumatoid arthritis: a multicenter cross-sectional study. BMC Rheumatol. 2022;6(75):1–8.

Yates M, Ledingham JM, Hatcher PA, Adas M, Hewitt S, Bartlett-Pestell S, et al. Disease activity and its predictors in early inflammatory arthritis: findings from a national cohort. Rheumatology. 2021;60:4811–20.

Yelin E, Henke C, Epstein W. The work dynamics of the person with rheumatoid arthritis. Arthritis Rheum. 1987;30:507–12.

Zhang W, Bansback N, Guh D, Li X, Nosyk B, Marra CA, et al. Short-term influence of adalimumab on work productivity outcomes in patients with rheumatoid arthritis. J Rheumatol. 2008;35:1729–36.

Żołnierczyk-Zreda D, Jędryka-Góral A, Bugajska J, Bedyńska S, Brzosko M, Pazdur J. The relationship between work, mental health, physical health, and fatigue in patients with rheumatoid arthritis: a cross-sectional study. J Health Psychol. 2020;25:665–73.

da Rocha Castellar Pinheiro G, Khandker RK, Sato R, Rose A, Piercy J. Impact of rheumatoid arthritis on quality of life, work productivity and resource utilisation: an observational, cross-sectional study in Brazil. Clin Exp Rheumatol. 2013;31:334–40.

Albers JMC, Kuper HH, van Riel PLCM, Prevoo MLL, Van’t Hof MA, van Gestel AM, et al. Socio-economic consequences of rheumatoid arthritis in the first year of the disease. Rheumatology. 1999;38:423–30.

Barrett EM, Scott DGI, Wiles NJ. The impact of rheumatoid arthritis on employment status in the early years of disease: a UK community-based study. Rheumatology. 2000;39:7.

Bejano V, Quinn M, Conaghan PG, Reece R, Keenan AM, Walker D, et al. Effect of the early use of the anti–tumor necrosis factor Adalimumab on the prevention of job loss in patients with early rheumatoid arthritis. Arthritis Care Res. 2008;59:1467–74.

Eberhardt K, Larsson BM, Nived K. Early rheumatoid arthritis – some social, economical, and psychological aspects. Scand J Rheum. 1993;22:119–23.

Eberhardt K, Larsson BM, Nived K, Lindqvist E. Work disability in rheumatoid arthritis- development over 15 years and evaluation of predictive factors over time. J Rheumatol. 2007;34:481–7.

Halpern MT, Cifaldi MA, Kvien TK. Impact of adalimumab on work participation in rheumatoid arthritis: comparison of an open-label extension study and a registry-based control group. Ann Rheum Dis. 2009;68:930–7.

Herenius MMJ, Hoving JI, Sluiter JK, Raterman HG, Lems WF, Dijkmans BAC, et al. Improvement of work ability, quality of life, and fatique in patients with rheumatoid arthritis treated with adalimumab. J Occup Environ Health. 2010;52:618–21.

Hoving JL, Bartelds GM, Sluiter JK, Sadiraj K, Groot I, Lems WF, et al. Perceived work ability, quality of life, and fatigue in patients with rheumatoid arthritis after a 6-month course of TNF inhibitors: prospective intervention study and partial economic evaluation. Scand J Rheumatol. 2009;38:246–50.

Jäntti J, Aho K, Kaarela K, Kautiainen H. Work disability in an inception cohort of patients with seropositive rheumatoid arthritis: a 20 year study. Rheumatology. 1999;38:4.

Kaarela K, Lehtinen K, Luukkainen R. Work capacity of patients with inflammatory joint diseases: an eight-year follow-up study. Scand J Rheumatol. 1987;16:403–6.

McWilliams DF, Varughese S, Young A, Kiely PD, Walsh DA. Work disability and state benefit claims in early rheumatoid arthritis: the ERAN cohort. Rheumatology. 2014;53:9.

Mau W, Bornmann M, Weber H, Weidemann HF, Hecker H, Raspe HH. Prediction of permanent work disability in a follow-up study of early rheumatoid arthritis: results of a tree structured analysis using RECPAM. Br J Rheumatol. 1996;35:652–9.

Nikiphorou E, Guh D, Bansback N, Zhang W, Dixey J, Williams P, et al. Work disability rates in RA. Results from an inception cohort with 24 years follow-up. Rheumatology. 2012;51:8.

Nordmark B, Blomqvist P, Andersson B, Hägerström M, Nordh-Grate K, Rönnqvist R, et al. A two-year follow-up of work capacity in early rheumatoid arthritis: a study of multidisciplinary team care with emphasis on vocational support. Scand J Rheumatol. 2006;35:7–14.

Pincus T, Callahan LF, Sale WG, Brooks AL, Payne LE, Vaughn WK. Severe functional declines, work disability, and increased mortality in seventy-five rheumatoid arthritis patients studied over mine years. Arthritis Rheum. 1984;27:864–72.

Puolakka K, Kautiainen H, Möttönen T, Hannonen P, Hakala M, Korpela M, et al. Predictors of productivity loss in early rheumatoid arthritis: a 5 year follow up study. Ann Rheum Dis. 2005;64:130–3.

Puolakka K, Kautiainen H, Möttönen T, Hannonen P, Korpela M, Julkunen H, et al. Impact of initial aggressive drug treatment with a combination of disease-modifying antirheumatic drugs on the development of work disability in early rheumatoid arthritis. Arthritis Rheum. 2004;50:55–62.

Reisine S, Fifield J, Walsh S, Feinn R. Factors associated with continued employment among patients with rheumatoid arthritis: a survival model. J Rheumatol. 2001;28:2400–8.

Reisine S, Fifield J, Walsh S, Dauser D. Work disability among two cohorts of women with recent onset rheumatoid arthritis: a survival analysis. Arthritis Rheum. 2007;57:372–80.

Robinson HS, Walters K. Return to work after treatment of rheumatoid arthritis. Can Med Assoc J. 1971;105:166–9.

CAS   PubMed   PubMed Central   Google Scholar  

Smolen JS, Han C, van der Heijde D, Emery P, Bathon JM, Keystone E, et al. Infliximab treatment maintains employability in patients with early rheumatoid arthritis. Arthritis Rheum. 2006;54:716–22.

van Vollenhoven RF, Cifaldi MA, Ray S, Chen N, Weisman MH. Improvement in work place and household productivity for patients with early rheumatoid arthritis treated with adalimumab plus methotrexate: work outcomes and their correlations with clinical and radiographic measures from a randomized controlled trial companion study. Arthritis Care Res. 2010;62:226–34.

Vazquez-Villegas ML, Gamez-Nava JI, Celis A, Sanchez-Mosco D, de la Cerda-Trujillo LF, Murillo-Vazquez JD, et al. Prognostic factors for permanent work disability in patients with rheumatoid arthritis who received combination therapy of conventional synthetic disease-modifying antirheumatic drugs. A retrospective cohort study. J Clin Rheumatol. 2017;23:376–82.

Verstappen SMM, Jacobs JWG, Kruize AA, Erlich JC, van Albada-Kuipers GA, Verkleij H, et al. Trends in economic consequences of rheumatoid arthritis over two subsequent years. Rheumatology. 2007;46:968–74.

Vlak T, Eldar R. Disability in rheumatoid arthritis after monotherapy with DMARDs. Int J Rehabil Res. 2003;26:207–12.

Yelin E, Trupin L, Katz P, Lubeck D, Rush S, Wanke L. Association between etanercept use and employment outcomes among patients with rheumatoid arthritis. Arthritis & Rheum. 2003;48:3046–54.

Young A, Dixey J, Cox N, Davies P, Devlin J, Emery P, et al. How does functional disability in early rheumatoid arthritis (RA) affect patients and their lifes? Results of 5 years of follow-up in 732 patients from the Early RA Study (ERAS). Rheumatology. 2000;39:603–11.

Young A, Dixey J, Kulinskaya E, Cox N, Davies P, Devlin J, et al. Which patients stop working because of rheumatoid arthritis? Results of five years’ follow up in 732 patients from the Early RA Study (ERAS). Ann Rheum Dis. 2002;61:335–40.

Zirkzee EJM, Sneep AC, de Buck PDM, Allaart CF, Peeters AJ, Ronday HK, et al. Sick leave and work disability in patients with early arthritis. Clin Rheumatol. 2008;27:9.

Reisine S, McQuillan J, Fifield J. Predictors of work disability in rheumatoid arthritis patients. Arthritis Rheum. 2005;38:1630–7.

Verstappen SMM, Watson KD, Lunt M, McGrother K, Symmons PM, Hyrich KL. Working status in patients with rheumatoid arthritis, ankylosing spondylitis and psoriatic arthritis: results from the British Society for Rheumatology Biogics Register. Rheumatology. 2010;49:1570–7.

Nissilä M, Isomäki H, Kaarela K, Kiviniemi P, Martio J, Sarna S. Prognosis of inflammatory joint diseases. A three-year follow-up study. Scand J Rheumatol. 1983;12:33–8.

Han C, Smolen J, Kavanaugh A, St.Clair EW, Baker D, Bala M. Comparison of employability outcomes among patients with early or long-standing rheumatoid arthritis. Arthritis Rheum. 2008;59:510–4.

Gwinnutt JM, Leggett S, Lunt M, Barton A, Hyrich KL, Walker-Bone K, et al. Predictors of presenteeism, absenteeism and job loss in patients commencing methotrexate or biologic therapy for rheumatoid arthritis. Rheumatology. 2020;59:2908–19.

Cieza A, Causey K, Kamenov K, Hanson SW, Chatterji S, Vos T. Global estimates of the need for rehabilitation based on the Global Burden of Disease study 2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020. https://doi.org/10.1016/S0140-6736(20)32340-0:1-12 .

Gwinnutt JM, Wieczorek M, Balanescu A, Bischoff-Ferrari HA, Boonen A, Cavalli G, et al. 2021 EULAR recommendations regarding lifestyle behaviours and work participation to prevent progression of rheumatic and musculoskeletal diseases. Ann Rheum Dis. 2023;82:48–56.

England BR, Smith BJ, Baker NA, Barton JL, Oatis CA, Guyatt G, et al. 2022 American College of Rheumatology Guideline for exercise, rehabilitation, diet, and additional integrative interventions for rheumatoid arthritis. Arthritis Care Res. 2023;75:1603–15.

Vu M, Carvalho N, Clarke PM, Buchbinder R, Tran-Duy A. Impact of comorbid conditions on healtcare expenditure and work-related outcomes in patients with rheumatoid arthritis. J Rheumatol. 2021;48:1221–9.

Amaral GSG, Ots P, Brouwer S, van Zon SKR. Multimorbidity and exit from paid employment: the effect of specific combinations of chronic health conditions. Eur J Public Health. 2022;32:392–7.

Boonen A, Putrik P, Marques ML, Alunno A, Abasolo L, Beaton D, et al. EULAR Points to Consider (PtC) for designing, analysing and reporting of studies with work participation as an outcome domain in patients with inflammatory arthritis. Ann Rheum Dis. 2021;80:1116–23.

Lajas C, Abasolo L, Bellajdel B, Hernandez-Garcia C, Carmona L, Vargas E, et al. Costs and predictors of costs in rheumatoid arthritis: A prevalence-based study. Arthritis Care Res. 2003;49:64–70.

Reisine S, McQuillan J, Fifield J. Predictors of work disability in rheumatoid arthritis patients. Arthritis Rheum. 1995;38:1630–7.

Download references

Acknowledgements

Open access funding provided by Royal Library, Copenhagen University Library

Author information

Authors and affiliations.

Department of Social Medicine, University Hospital Bispebjerg-Frederiksberg, Copenhagen, Denmark

Lilli Kirkeskov & Katerina Bray

Department of Social Medicine, University Hospital Bispebjerg-Frederiksberg, Nordre Fasanvej 57, Vej 8, Opgang 2.2., 2000, Frederiksberg, Denmark

Lilli Kirkeskov

Department of Occupational and Social Medicine, Holbaek Hospital, Holbaek, Denmark

Katerina Bray

You can also search for this author in PubMed   Google Scholar

Contributions

LK performed the systematic research, including reading articles, performed the blinded quality assessment and the meta-analysis, and drafted and revised the article. KM performed the blinded quality assessment and the discussion afterwards of articles to be included in the research and the scores, and drafted and revised the article.

Corresponding author

Correspondence to Lilli Kirkeskov .

Ethics declarations

Ethics approval and consent to participate.

Not applicable as this is a systematic review. All the studies that are included have obtained ethical approval and consent as appreciated by the journal in which they have been published.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: figure s1..

Employment; year of investigation.

Additional file 2: Figure S2.

Forest Plot of Comparison: Predictors for employment. Outcome: Younger or older age.

Additional file 3: Figure S3.

Forest Plot of Comparison: Predictors for employment. Outcome: >50 yr or <50 yr of age.

Additional file 4: Figure S4.

Forest Plot of Comparison: Predictors for employment. Outcome: Gender: Male or Female.

Additional file 5: Figure S5.

Forest Plot of Comparison: Predictors for employment. Outcome: Educational level: no college education or college education or higher.

Additional file 6: Figure S6.

Forest Plot of Comparison: Predictors for employment. Outcome: no comorbidities present or one or more comorbidities present.

Additional file 7: Figure S7.

Forest Plot of Comparison: Predictors for employment. Outcome: Ethnicity: Caucasian or other than Caucasian.

Additional file 8: Figure S8.

Forest Plot of Comparison: Predictors for employment. Outcome: Short or long disease duration.

Additional file 9: Figure S9.

Forest Plot of Comparison: Predictors for employment. Outcome: Low or high Health Assessment Questionnaire, HAQ-score.

Additional file 10: Figure S10.

Forest Plot of Comparison: Predictors for employment. Outcome: Low or high VAS-score.

Additional file 11: Figure S11.

Forest Plot of Comparison: Predictors for employment. Outcome: Job type: blue collar workers or other job types.

Additional file 12: Figure S12.

Forest Plot of Comparison: Predictors for employment. Outcome: No MTX or MTX.

Additional file 13: Figure S13.

Forest Plot of Comparison: Predictors for employment. Outcome: No biological or biological.

Additional file 14: Figure S14.

Forest Plot of Comparison: Predictors for employment. Outcome: No prednisolone or prednisolone.

Additional file 15: Figure S15.

Forest Plot of Comparison: Predictors for employment. Outcome: Low or high DAS score.

Additional file 16: Figure S16.

Forest Plot of Comparison: Predictors for employment. Outcome: Low or high SF 36-score.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kirkeskov, L., Bray, K. Employment of patients with rheumatoid arthritis - a systematic review and meta-analysis. BMC Rheumatol 7 , 41 (2023). https://doi.org/10.1186/s41927-023-00365-4

Download citation

Received : 07 June 2023

Accepted : 20 October 2023

Published : 14 November 2023

DOI : https://doi.org/10.1186/s41927-023-00365-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Return to work
• Unemployment

BMC Rheumatology

ISSN: 2520-1026

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Chapter 12: Descriptive Statistics

12.3 expressing your results, learning objectives.

• Write out simple descriptive statistics in American Psychological Association (APA) style.
• Interpret and create simple APA-style graphs—including bar graphs, line graphs, and scatterplots.
• Interpret and create simple APA-style tables—including tables of group or condition means and correlation matrixes.

Once you have conducted your descriptive statistical analyses, you will need to present them to others. In this section, we focus on presenting descriptive statistical results in writing, in graphs, and in tables—following American Psychological Association (APA) guidelines for written research reports. These principles can be adapted easily to other presentation formats such as posters and slide show presentations.

Presenting Descriptive Statistics in Writing

When you have a small number of results to report, it is often most efficient to write them out. There are a few important APA style guidelines here. First, statistical results are always presented in the form of numerals rather than words and are usually rounded to two decimal places (e.g., “2.00” rather than “two” or “2”). They can be presented either in the narrative description of the results or parenthetically—much like reference citations. Here are some examples:

The mean age of the participants was 22.43 years with a standard deviation of 2.34. Among the low self-esteem participants, those in a negative mood expressed stronger intentions to have unprotected sex ( M = 4.05, SD = 2.32) than those in a positive mood ( M = 2.15, SD = 2.27). The treatment group had a mean of 23.40 ( SD = 9.33), while the control group had a mean of 20.87 ( SD = 8.45). The test-retest correlation was .96. There was a moderate negative correlation between the alphabetical position of respondents’ last names and their response time ( r = −.27).

Notice that when presented in the narrative, the terms mean and standard deviation are written out, but when presented parenthetically, the symbols M and SD are used instead. Notice also that it is especially important to use parallel construction to express similar or comparable results in similar ways. The third example is much better than the following nonparallel alternative:

Presenting Descriptive Statistics in Graphs

When you have a large number of results to report, you can often do it more clearly and efficiently with a graph. When you prepare graphs for an APA-style research report, there are some general guidelines that you should keep in mind. First, the graph should always add important information rather than repeat information that already appears in the text or in a table. (If a graph presents information more clearly or efficiently, then you should keep the graph and eliminate the text or table.) Second, graphs should be as simple as possible. For example, the Publication Manual discourages the use of color unless it is absolutely necessary (although color can still be an effective element in posters, slide show presentations, or textbooks.) Third, graphs should be interpretable on their own. A reader should be able to understand the basic result based only on the graph and its caption and should not have to refer to the text for an explanation.

There are also several more technical guidelines for graphs that include the following:

• The graph should be slightly wider than it is tall.
• The independent variable should be plotted on the x- axis and the dependent variable on the y- axis.
• Values should increase from left to right on the x- axis and from bottom to top on the y- axis.

Axis Labels and Legends

• Axis labels should be clear and concise and include the units of measurement if they do not appear in the caption.
• Axis labels should be parallel to the axis.
• Legends should appear within the boundaries of the graph.
• Text should be in the same simple font throughout and differ by no more than four points.
• Captions should briefly describe the figure, explain any abbreviations, and include the units of measurement if they do not appear in the axis labels.
• Captions in an APA manuscript should be typed on a separate page that appears at the end of the manuscript. See Chapter 11 “Presenting Your Research” for more information.

As we have seen throughout this book, bar graphs are generally used to present and compare the mean scores for two or more groups or conditions. The bar graph in Figure 12.12 “Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues” is an APA-style version of Figure 12.5 “Bar Graph Showing Mean Clinician Phobia Ratings for Children in Two Treatment Conditions” . Notice that it conforms to all the guidelines listed. A new element in Figure 12.12 “Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues” is the smaller vertical bars that extend both upward and downward from the top of each main bar. These are error bars , and they represent the variability in each group or condition. Although they sometimes extend one standard deviation in each direction, they are more likely to extend one standard error in each direction (as in Figure 12.12 “Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues” ). The standard error is the standard deviation of the group divided by the square root of the sample size of the group. The standard error is used because, in general, a difference between group means that is greater than two standard errors is statistically significant. Thus one can “see” whether a difference is statistically significant based on a bar graph with error bars.

Figure 12.12 Sample APA-Style Bar Graph, With Error Bars Representing the Standard Errors, Based on Research by Ollendick and Colleagues

Line Graphs

Line graphs are used to present correlations between quantitative variables when the independent variable has, or is organized into, a relatively small number of distinct levels. Each point in a line graph represents the mean score on the dependent variable for participants at one level of the independent variable. Figure 12.13 “Sample APA-Style Line Graph Based on Research by Carlson and Conard” is an APA-style version of the results of Carlson and Conard. Notice that it includes error bars representing the standard error and conforms to all the stated guidelines.

Figure 12.13 Sample APA-Style Line Graph Based on Research by Carlson and Conard

In most cases, the information in a line graph could just as easily be presented in a bar graph. In Figure 12.13 “Sample APA-Style Line Graph Based on Research by Carlson and Conard” , for example, one could replace each point with a bar that reaches up to the same level and leave the error bars right where they are. This emphasizes the fundamental similarity of the two types of statistical relationship. Both are differences in the average score on one variable across levels of another. The convention followed by most researchers, however, is to use a bar graph when the variable plotted on the x- axis is categorical and a line graph when it is quantitative.

Scatterplots

Scatterplots are used to present relationships between quantitative variables when the variable on the x- axis (typically the independent variable) has a large number of levels. Each point in a scatterplot represents an individual rather than the mean for a group of individuals, and there are no lines connecting the points. The graph in Figure 12.14 “Sample APA-Style Scatterplot” is an APA-style version of Figure 12.8 “Statistical Relationship Between Several College Students’ Scores on the Rosenberg Self-Esteem Scale Given on Two Occasions a Week Apart” , which illustrates a few additional points. First, when the variables on the x- axis and y -axis are conceptually similar and measured on the same scale—as here, where they are measures of the same variable on two different occasions—this can be emphasized by making the axes the same length. Second, when two or more individuals fall at exactly the same point on the graph, one way this can be indicated is by offsetting the points slightly along the x- axis. Other ways are by displaying the number of individuals in parentheses next to the point or by making the point larger or darker in proportion to the number of individuals. Finally, the straight line that best fits the points in the scatterplot, which is called the regression line, can also be included.

Figure 12.14 Sample APA-Style Scatterplot

Expressing Descriptive Statistics in Tables

Like graphs, tables can be used to present large amounts of information clearly and efficiently. The same general principles apply to tables as apply to graphs. They should add important information to the presentation of your results, be as simple as possible, and be interpretable on their own. Again, we focus here on tables for an APA-style manuscript.

The most common use of tables is to present several means and standard deviations—usually for complex research designs with multiple independent and dependent variables. Figure 12.15 “Sample APA-Style Table Presenting Means and Standard Deviations” , for example, shows the results of a hypothetical study similar to the one by MacDonald and Martineau (2002) discussed in Chapter 5 “Psychological Measurement” . (The means in Figure 12.15 “Sample APA-Style Table Presenting Means and Standard Deviations” are the means reported by MacDonald and Martineau, but the standard errors are not). Recall that these researchers categorized participants as having low or high self-esteem, put them into a negative or positive mood, and measured their intentions to have unprotected sex. Although not mentioned in Chapter 5 “Psychological Measurement” , they also measured participants’ attitudes toward unprotected sex. Notice that the table includes horizontal lines spanning the entire table at the top and bottom, and just beneath the column headings. Furthermore, every column has a heading—including the leftmost column—and there are additional headings that span two or more columns that help to organize the information and present it more efficiently. Finally, notice that APA-style tables are numbered consecutively starting at 1 (Table 1, Table 2, and so on) and given a brief but clear and descriptive title.

Figure 12.15 Sample APA-Style Table Presenting Means and Standard Deviations

Another common use of tables is to present correlations—usually measured by Pearson’s r —among several variables. This is called a correlation matrix . Figure 12.16 “Sample APA-Style Table (Correlation Matrix) Based on Research by McCabe and Colleagues” is a correlation matrix based on a study by David McCabe and colleagues (McCabe, Roediger, McDaniel, Balota, & Hambrick, 2010). They were interested in the relationships between working memory and several other variables. We can see from the table that the correlation between working memory and executive function, for example, was an extremely strong .96, that the correlation between working memory and vocabulary was a medium .27, and that all the measures except vocabulary tend to decline with age. Notice here that only half the table is filled in because the other half would have identical values. For example, the Pearson’s r value in the upper right corner (working memory and age) would be the same as the one in the lower left corner (age and working memory). The correlation of a variable with itself is always 1.00, so these values are replaced by dashes to make the table easier to read.

Figure 12.16 Sample APA-Style Table (Correlation Matrix) Based on Research by McCabe and Colleagues

As with graphs, precise statistical results that appear in a table do not need to be repeated in the text. Instead, the writer can note major trends and alert the reader to details (e.g., specific correlations) that are of particular interest.

Key Takeaways

• In an APA-style article, simple results are most efficiently presented in the text, while more complex results are most efficiently presented in graphs or tables.
• APA style includes several rules for presenting numerical results in the text. These include using words only for numbers less than 10 that do not represent precise statistical results, and rounding results to two decimal places, using words (e.g., “mean”) in the text and symbols (e.g., “ M ”) in parentheses.
• APA style includes several rules for presenting results in graphs and tables. Graphs and tables should add information rather than repeating information, be as simple as possible, and be interpretable on their own with a descriptive caption (for graphs) or a descriptive title (for tables).
• Practice: In a classic study, men and women rated the importance of physical attractiveness in both a short-term mate and a long-term mate (Buss & Schmitt, 1993). The means and standard deviations are as follows. Men / Short Term: M = 5.67, SD = 2.34; Men / Long Term: M = 4.43, SD = 2.11; Women / Short Term: M = 5.67, SD = 2.48; Women / Long Term: M = 4.22, SD = 1.98. Present these results (a) in writing, (b) in a graph, and (c) in a table.

Buss, D. M., & Schmitt, D. P. (1993). Sexual strategies theory: A contextual evolutionary analysis of human mating. Psychological Review, 100 , 204–232.

MacDonald, T. K., & Martineau, A. M. (2002). Self-esteem, mood, and intentions to use condoms: When does low self-esteem lead to risky health behaviors? Journal of Experimental Social Psychology, 38 , 299–306.

McCabe, D. P., Roediger, H. L., McDaniel, M. A., Balota, D. A., & Hambrick, D. Z. (2010). The relationship between working memory capacity and executive functioning. Neuropsychology, 243 , 222–243.

• Research Methods in Psychology. Provided by : University of Minnesota Libraries Publishing. Located at : http://open.lib.umn.edu/psychologyresearchmethods/ . License : CC BY-NC-SA: Attribution-NonCommercial-ShareAlike

Privacy Policy

This paper is in the following e-collection/theme issue:

Published on 12.4.2024 in Vol 26 (2024)

The Effectiveness of a Digital App for Reduction of Clinical Symptoms in Individuals With Panic Disorder: Randomized Controlled Trial

Authors of this article:

Original Paper

• KunJung Kim, MD   ; 
• Hyunchan Hwang, MD, PhD   ; 
• Sujin Bae, PhD   ; 
• Sun Mi Kim, MD, PhD   ; 
• Doug Hyun Han, MD, PhD  

Chung Ang University Hospital, Seoul, Republic of Korea

Corresponding Author:

Doug Hyun Han, MD, PhD

Chung Ang University Hospital

102 Heucsock ro

Seoul, 06973

Republic of Korea

Phone: 82 2 6299 3132

Fax:82 2 6299 3100

Email: [email protected]

Background: Panic disorder is a common and important disease in clinical practice that decreases individual productivity and increases health care use. Treatments comprise medication and cognitive behavioral therapy. However, adverse medication effects and poor treatment compliance mean new therapeutic models are needed.

Objective: We hypothesized that digital therapy for panic disorder may improve panic disorder symptoms and that treatment response would be associated with brain activity changes assessed with functional near-infrared spectroscopy (fNIRS).

Methods: Individuals (n=50) with a history of panic attacks were recruited. Symptoms were assessed before and after the use of an app for panic disorder, which in this study was a smartphone-based app for treating the clinical symptoms of panic disorder, panic symptoms, depressive symptoms, and anxiety. The hemodynamics in the frontal cortex during the resting state were measured via fNIRS. The app had 4 parts: diary, education, quest, and serious games. The study trial was approved by the institutional review board of Chung-Ang University Hospital (1041078-202112-HR-349-01) and written informed consent was obtained from all participants.

Results: The number of participants with improved panic symptoms in the app use group (20/25, 80%) was greater than that in the control group (6/21, 29%; χ 2 1 =12.3; P =.005). During treatment, the improvement in the Panic Disorder Severity Scale (PDSS) score in the app use group was greater than that in the control group ( F 1,44 =7.03; P =.01). In the app use group, the total PDSS score declined by 42.5% (mean score 14.3, SD 6.5 at baseline and mean score 7.2, SD 3.6 after the intervention), whereas the PDSS score declined by 14.6% in the control group (mean score 12.4, SD 5.2 at baseline and mean score 9.8, SD 7.9 after the intervention). There were no significant differences in accumulated oxygenated hemoglobin (accHbO 2 ) at baseline between the app use and control groups. During treatment, the reduction in accHbO 2 in the right ventrolateral prefrontal cortex (VLPFC; F 1,44 =8.22; P =.006) and the right orbitofrontal cortex (OFC; F 1,44 =8.88; P =.005) was greater in the app use than the control group.

Conclusions: Apps for panic disorder should effectively reduce symptoms and VLPFC and OFC brain activity in patients with panic disorder. The improvement of panic disorder symptoms was positively correlated with decreased VLPFC and OFC brain activity in the resting state.

Trial Registration: Clinical Research Information Service KCT0007280; https://cris.nih.go.kr/cris/search/detailSearch.do?seq=21448

Introduction

Panic disorder is a common and important disease in clinical practice that leads to a reduction of individual productivity and increased use of health care [ 1 ]. The lifetime prevalence of panic disorder in the general population is 4.8%, and 22.7% of people experience panic attacks [ 2 ]. The most common symptoms of panic disorder include palpitations, shortness of breath, chest pain, numbness of the hands and feet, and cardiorespiratory-type symptoms, in addition to fear of dying, sweating, tremors, dizziness, nausea, and chills [ 3 ]. The US Food and Drug Administration has currently only approved selective serotonin reuptake inhibitors (SSRIs) for the treatment of panic disorder [ 4 ]. However, it is clinically difficult to expect an improvement in symptoms using SSRIs alone in the acute phase; thus we treat patients with benzodiazepine, which can lead to dependence and withdrawal symptoms [ 5 , 6 ]. The most common side effects of SSRIs reported by patients are reduced sexual function, drowsiness, and weight gain [ 7 ], and clinicians may hesitate to use benzodiazepines due to dependence and withdrawal symptoms [ 8 ]. Cognitive behavioral therapy (CBT) is the most widely used nonpharmaceutical treatment for anxiety disorders [ 9 ]. Additional nonpharmaceutical treatments, such as group therapy and supportive psychotherapy, are also available for patients with panic disorder [ 10 , 11 ]. However, these treatments have the disadvantage of requiring face-to-face contact; therefore, other therapeutic alternatives should be offered to patients during pandemics such as COVID-19.

The definition of a digital therapeutic (DTx) is a therapeutic that delivers evidence-based interventions to prevent, manage, or treat a medical disorder or disease; DTxs are currently used in many areas [ 12 ]. This kind of medical and public health use of smartphones and digital technologies is also known as mobile health (mHealth). DTxs related to mental health medicine are actively used in various psychiatric disorders, such as insomnia, substance abuse, attention-deficit/hyperactivity disorder, and anxiety and depression, among others [ 13 ]. In particular, the use of Freespira, a panic disorder DTx, reduced panic symptoms, avoidance behaviors, and treatment costs in patients with panic disorder [ 14 ].

As brain imaging technology advances, a great deal of functional mapping information on the human brain has been accumulated from positron emission tomography (PET), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). Among these technologies, fNIRS can measure brain activity in a noninvasive and safe manner through measuring changes in the hemoglobin oxygenation state of the human brain [ 15 ]. Various studies have been conducted using fNIRS and fMRI to reveal correlations between panic disorder and brain regions. For example, patients with panic disorder show increased activity in the inferior frontal cortex, hippocampus, cingulate (both anterior and posterior), and orbitofrontal cortex (OFC) [ 16 ]. Previously, we confirmed that patients with panic disorder during rest periods showed increased activity in the OFC [ 17 ].

In this study, we determined whether an app for panic disorder would improve panic disorder symptoms. In addition, we used fNIRS to confirm the association between changes in panic disorder symptoms and changes in activity in specific brain regions.

Participants

Patients who had experiences of panic attacks were recruited between March 1 and July 30, 2022, through billboard advertisements at our hospital. The inclusion criteria for the study were as follows: (1) age between 20 and 65 years, (2) diagnosis of panic disorder based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition and (3) ability to use apps without problems. The exclusion criteria were as follows: (1) a history of other psychiatric disorders, except for anxiety disorder, or substance dependence, except for habitual alcohol and tobacco use; and (2) a history of head trauma and chronic medical conditions. The research clinician assessed whether patients fulfilled the inclusion or exclusion criteria. Written informed consent was acquired from all participants at the first visit. This study has been registered with the Clinical Research Information Service (KCT0007280).

Assessment Scales for Anxiety Symptoms

The severity of panic symptoms was assessed using the Panic Disorder Severity Scale (PDSS). The PDSS was developed by Shear et al [ 18 ] in 1997. It is a 7-item instrument used to rate the overall severity of panic disorder and was validated in Korea by Lim et al [ 19 ] in 2001.

The anxiety symptoms of all participants were assessed using the clinician-based Hamilton Anxiety Scale (HAM-A) questionnaire and the participant-based Generalized Anxiety Disorder-7 (GAD-7) questionnaire. The HAM-A was developed by Hamilton in 1969 [ 20 ]. The 14-item version remains the most used outcome measure in clinical trials of treatments for anxiety disorders and was validated in Korea by Kim [ 21 ] in 2001.

The GAD-7 questionnaire, developed by Spitzer et al [ 22 ], is a 7-item self-report anxiety questionnaire designed to assess the patient’s health status during the previous 2 weeks. The GAD-7 was translated into the Korean language and is freely downloadable on the Patient Health Questionnaire website [ 23 ].

Hemodynamic Response of the Prefrontal Cortex

The hemodynamics in the frontal cortex during the resting state were measured using the fNIRS device (NIRSIT; OBELAB Inc). The NIRSIT has 24 laser diodes (sources) emitting light at 2 wavelengths (780 nm and 850 nm) and 32 photodetectors with a sampling rate of 8.138 Hz [ 24 ]. The distance between the source and photodetector is 15 mm. Based on the suggested suitable sensor-detector separation distance for measuring cortical hemodynamic changes, only 30-mm channels were analyzed in this study [ 25 ].

For our study, we used the 48-channel configuration ( Figure 1 ). The detected light signals in each wavelength were filtered with a band-pass filter (0.00 Hz-0.1 Hz) to reduce the effect of environmental noise–related light and body movements. In addition, channels with low-quality information (signal-to-noise ratio <30 dB) were removed from the hemodynamic analysis. The accumulated oxygenated hemoglobin (accHbO 2 ) values in the resting state represent the activation of the prefrontal cortex. In accordance with the theory that oxygenated hemoglobin has superior sensitivity and signal-to-noise ratio compared to deoxygenated hemoglobin data, only oxygenated hemoglobin were used for this analysis [ 26 - 28 ].

The means and SDs for accHbO 2 were calculated from regions of interest (ROIs) in the right and left dorsolateral prefrontal cortices (DLPFCs), right and left ventrolateral prefrontal cortices (VLPFCs), right and left frontopolar cortices (FPCs), and right and left orbitofrontal cortices (OFCs), based on Brodmann area 46. The right and left DLPFCs comprise channels 1, 2, 3, 5, 6, 11, 17, and 18 and channels 19, 20, 33, 34, 35, 38, 39, and 43, respectively. The right and left VLPFCs comprise channels 4, 9, and 10 and channels 40, 44, and 45, respectively. The right and left FPCs comprise channels 7, 8, 12, 13, 21, 22, 25, and 26 and channels 23, 24, 27, 28, 36, 37, 41, and 42, respectively. The right and left OFCs comprise channels 14, 15, 16, 29, and 30 and channels 31, 32, 46, 47, and 48, respectively ( Figure 1 ).

Digital App for Panic Disorder

The app for panic disorder is a smartphone-based app for treatment of clinical symptoms of panic disorder. The mobile app has 4 categories: diary, education, quest, and serious games. The diary category has three items: (1) assessment of daily psychological status, including mood and anxiety; (2) assessment of panic symptoms, including frequency and severity; and (3) consumption of medication, including regular medication and pro re nata medications. The education category has three items: (1) knowledge about panic disorders, (2) knowledge about medications for panic disorder, and (3) knowledge about panic disorder treatment, including CBT, breathing therapy, and positive thinking therapy. The quests include two treatments: (1) eye movement desensitization and reprocessing therapy and (2) positive thinking therapy. The serious games include two games: (1) a breathing game and (2) an exposure therapy game.

The diary, education, and serious games (ie, the breathing game and exposure therapy game) are important parts of CBT for panic disorder [ 29 - 32 ]. The efficacy of CBT for panic disorder has been examined in various randomized controlled trials [ 33 , 34 ]. Eye movement desensitization and reprocessing therapy are also known to help reduce panic symptoms [ 35 , 36 ]. We confirmed that the replacement of worry with different forms of positive ideation shows beneficial effects [ 37 ], so a similar type of positive thinking therapy can also be expected to show benefits. Multimedia Appendix 1 provides additional information on the app.

Ethical Considerations

The study trial was approved by the institutional review board of Chung-Ang University Hospital (1041078-202112-HR-349-01) and written informed consent was obtained from all participants. Participants received an explanation from the researchers that included an overview of the study and a description of the methodology and purpose before deciding to participate. Additionally, they were informed that participation was voluntary, informed about our confidentiality measures, given the option to withdraw, and informed about potential side effects and compensation. Participants in this study received ₩100,000 (US $75.50) as transportation reimbursement. Additionally, the various scales and fNIRS assessments were offered at no cost to the participants. The participants received the results of the tests in the form of a report via postal mail or email after the conclusion of the study. They also receive an explanatory document and consent form from the researchers that included contact information for any inquiries. If the participant agreed to take part in the study after understanding the consent form, the research proceeded. The participants’ personal information was not collected. Instead, a unique identifier was assigned to the collected data for the sole purpose of research management.

Study Procedure

A randomized and treatment-as-usual–controlled design was applied in this study. After screening, all participants with panic disorder were randomly assigned to the app use group or the control group. The randomization sequence in our design was generated using SPSS (version 24.0; IBM Corp), with a 1:1 allocation between groups. At baseline and after intervention, all patients with panic disorder were assessed with the PDSS for panic symptoms, the HAM-A for objective anxiety symptoms, and the GAD-7 for subjective anxiety symptoms. At baseline and after intervention, the hemodynamic response in all patients with panic disorder was assessed using NIRSIT. The app use group was asked to use the app for panic disorder 20 minutes per day, 5 times per week, for 4 weeks. The control group was asked to read short educational letters that were delivered via a social network service 5 times per week for 4 weeks. The short letters contained information about panic disorder and its treatment.

Demographic and Clinical Characteristics

After recruitment, 56 patients underwent eligibility assessments. A total of 6 individuals were excluded because they did not meet the inclusion criteria. The remaining patients were divided into 2 groups: 25 were assigned to the app use group and 21 to the control group, as 4 patients were excluded; contact was suddenly lost with 1 patient contact and 1 dropped out for personal reasons. In addition, 2 patients in the control group quit the study after reporting poor benefits from the short educational letters. Therefore, 25 people in the app use group and 21 people in the control group were analyzed. Figure 2 shows the Consolidated Standards of Reporting Trials (CONSORT) flowchart for participant flow through the trial.

There were no significant differences in age, sex ratio, years of education, marital status, employment status, or substance habits, including smoking and alcohol use, between the app use group and the control group ( Table 1 ).

b Chi-square.

There were no significant differences in HAM-A score, GAD-7 score, or PDSS score at baseline between the app use group and control group ( Table 1 ).

Comparison of Changes in Clinical Scales Between App Use Group and Control Group

The number of participants with improved panic symptoms in the app use group (20/25, 80%) was greater than in the control group (6/21, 29%; χ 2 1 =12.3; P =.005).

During the treatment period, the app use group showed greater improvement in PDSS score than the control group ( F 1,44 =7.03; P =.01). In the app use group, the PDSS score decreased by 42.5% (mean score 14.3, SD 6.5 at baseline and mean score 7.2, SD 3.6 after the intervention), while the score decreased by 14.6% in the control group (mean score 12.4, SD 5.2 at baseline and mean score 9.8, SD 7.9 after intervention) ( Figure 3 ).

During the treatment period, there were no significant differences in the change in HAM-A scores ( F 1,44 =2.83; P =.09) and GAD-7 scores ( F 1,44 =0.22; P =.64) between the app use group and control group ( Figure 3 ).

Comparison of Changes in accHbO 2 Values Between App Use Group and Control Group

There were no significant differences in accHbO 2 in the right (t 45 =0.84; P =.40) or left (t 45 =0.73; P =.46) DLPFCs, right (t 45 =1.04; P =.31) or left (t 45 =0.88; P =.39) VLPFCs, right (t 45 =-0.18; P =.86) or left (t 45 =1.85; P =.07) FPCs, or right (t 45 =0.33; P =.74) or left (t 45 =1.89; P =.07) OFCs in the app use and control groups at baseline.

During the treatment period, the app use group showed a greater reduction in accHbO 2 in the right VLPFC ( F 1,44 =8.22; P =.006) and right OFC ( F 1,44 =8.88; P =.005) compared to the control group ( Figure 1 ). During the treatment period, there were no significant differences in the change in accHbO 2 in the other ROIs between the app use and control groups.

Correlations Between the Changes in PDSS Scores and the Changes in accHbO 2

In all participants (ie, the app use group plus the control group), there was a positive correlation between the change in PDSS score and the change in accHbO 2 in the right VLPFC ( r =0.44; P =.002). In the app use group, there was a positive correlation between the change in PDSS score and the changes in accHbO 2 in the right VLPFC ( r =0.42; P =.04). However, in the control group, there was no significant correlation between the change in PDSS score and the change in accHbO 2 in the right VLPFC ( r =0.22; P =.16).

In all participants, there was a positive correlation between the change in PDSS score and the change in accHbO 2 in the right OFC ( r =0.44; P =.002). In both the app use group ( r =0.34; P =.09) and control group ( r =0.33; P =.13), there was no significant correlation between the change in PDSS score and the change in accHbO 2 in the right OFC ( Figure 4 ).

Principal Findings

This study showed that a digital app was effective for symptom reduction, as well as decreasing brain activity in the VLPFCs and OFCs, in patients with panic disorder. In addition, the panic disorder symptom improvement was positively correlated with decreased brain activity in the VLPFCs and OFCs in the resting state.

The digital app used in this trial proved to be effective in reducing panic symptoms when compared to the control group, as demonstrated by the reduction in the PDSS score. We believe that this is due to the combined effect of the 4 parts of the program, namely the diary, education, quest, and serious games. The diary component helps identify and correct faulty perceptions and enables cognitive reconstruction. The education component provides information about the nature and physiology of panic disorder. The breathing game helps the participant return to a relaxed condition, while the exposure therapy game allows the participant to experience agoraphobic situations in a safe environment, which helps cognitive restructuring. These are the important parts of CBT for panic disorder and have shown efficacy, as reported earlier [ 29 - 32 ]. The control group also received educational data, including the importance of keeping a diary of one’s panic symptoms and how to do it, as well as self-guided direction on breathing exercises, but failed to show a significant reduction of symptoms compared to the app use group. We think this is due to lack of proper feedback in the control group. The app shows real-time feedback on breathing exercises using breathing sounds, and a message was sent if the user of the program failed to use the program for more than 2 days. We know that the therapeutic effect is better when immediate feedback is provided to patients undergoing CBT treatment [ 38 ]. Therefore, we think that the decrease in PDSS score was smaller because the control group did not receive feedback from the app.

The control group also received educational data on diary recording, panic disorder information, and how to execute breathing therapy and exposure therapy. We measured their reduction in the PDSS score, but we found it was less than in the app use group due to a lack of proper daily management.

However, the app failed to lead to a difference in the reduction in anxiety, as defined by the HAM-A and GAD-7 scales, between the 2 groups. This is most likely due to a lack of power, as the trial was conducted as a pilot study. Other studies using CBT techniques or serious games have demonstrated reductions in anxiety symptoms in patients with panic disorder [ 14 ]. Likewise, this study showed a trend toward a reduction in anxiety symptoms, although this was not statistically significant, and future research with more participants may show that these kinds of programs are also effective in controlling anxiety.

Two major changes in brain activity were noted in the app use group, namely reductions in VLPFC and OFC activation. The functions of the OFC are varied and include control of inappropriate behavior and emotional responses, decision-making, and solving problems [ 39 , 40 ]. Abnormalities in the function of the OFC can cause problems in dealing with anxiety and show that it is deeply involved in the increasing the sense of fear in the fear response [ 17 ]. The results of this study confirm that OFC activity decreases as treatment progresses. This reinforces the results of a previous study, which showed that patients with panic disorder had increased OFC activity and that when the panic disorder was treated, the activity of the OFC was reduced, as indicated by decreased cerebral glucose metabolic rates [ 17 , 41 ].

The VLPFC is known to be associated with the amygdala and to maintain flexible attention and responses to environmental threats [ 42 , 43 ]. The amygdala is the backbone of the fear network, and the VLPFC is also known to be deeply involved in the processing of fear [ 43 - 45 ]. Several studies have shown increased activity in patients with panic disorder in the inferior frontal gyrus, which envelops the VLPFC, and other related regions, including the prefrontal cortex, hippocampus, and OFC [ 16 , 46 , 47 ]. After panic disorder treatment, such as with CBT, decreased amygdala and inferior frontal gyrus activation in fear situations was confirmed [ 48 , 49 ]. Through panic disorder treatment, inferior frontal gyrus activation decreased to a normal level; this happened because the treatment reduced fear cognition related to harm expectancy or attention to threats [ 49 - 51 ]. We consider that VLPFC activation increases to modulate the amygdala and decreases with treatment for panic disorder.

We believe that these reductions of brain activity in the VLPFC and OFC reflect how the app affected the patients. We know that overprediction of fear or panic is an important feature of anxiety disorders [ 52 ]. The app for panic disorder, including diary, education, quest, and serious game components, allowed users to correct their faulty perceptions about fear. As mentioned earlier, the VLPFC and OFC are related to fear management, so we can expect that activity of the VLPFC and OFC will be reduced through repeated app use as users learn how to deal with fear.

Limitations

This study has the following limitations: Most of the patients were effectively treated with alprazolam or other anxiolytics, such as SSRIs. Thus, treatment with antianxiety drugs may have influenced our results. Moreover, this study assessed changes immediately after app use. A long-term follow-up to evaluate the sustainability of the observed improvements would provide valuable insights into the effectiveness of the intervention over time. App use time could be easily tracked for the app use group; however, it was challenging to independently monitor the time the control group spent reading educational materials. Due to the limitations of available research tools, no investigation has been conducted on deep brain structures such as the amygdala, which is most closely related to panic disorders.

Conclusions

We believe that this app for panic disorder effectively reduces symptoms and noticeably impacts brain activity in specific areas. We observed a positive link between improvement in panic symptoms and decreased brain activity in the VLPFCs and OFCs in a resting state. These findings support the use of targeted interventions to determine the brain’s contribution to symptom relief. Further research should explore the duration of these positive effects and make digital therapy accessible to more individuals, thus unlocking its full potential in mental health care.

Data Availability

The data sets generated and analyzed during this study are not publicly available as they contain information that could compromise the privacy and consent of the research participants. However, the transformed data are available upon reasonable request from the authors.

Conflicts of Interest

None declared.

Digital app for panic disorder.

CONSORT-eHEALTH checklist (V 1.6.1).

• Roy-Byrne PP, Craske MG, Stein MB. Panic disorder. Lancet. Sep 16, 2006;368(9540):1023-1032. [ CrossRef ] [ Medline ]
• Kessler RC, Chiu WT, Jin R, Ruscio AM, Shear K, Walters EE. The epidemiology of panic attacks, panic disorder, and agoraphobia in the National Comorbidity Survey Replication. Arch Gen Psychiatry. Apr 2006;63(4):415-424. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Meuret AE, White KS, Ritz T, Roth WT, Hofmann SG, Brown TA. Panic attack symptom dimensions and their relationship to illness characteristics in panic disorder. J Psychiatr Res. Sep 2006;40(6):520-527. [ CrossRef ] [ Medline ]
• Locke A, Kirst N, Shultz CG. Diagnosis and management of generalized anxiety disorder and panic disorder in adults. Am Fam Physician. May 01, 2015;91(9):617-624. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bruce SE, Vasile RG, Goisman RM, Salzman C, Spencer M, Machan JT, et al. Are benzodiazepines still the medication of choice for patients with panic disorder with or without agoraphobia? Am J Psychiatry. Aug 2003;160(8):1432-1438. [ CrossRef ] [ Medline ]
• Owen R, Tyrer P. Benzodiazepine dependence. A review of the evidence. Drugs. Apr 1983;25(4):385-398. [ CrossRef ] [ Medline ]
• Cascade E, Kalali AH, Kennedy SH. Real-world data on SSRI antidepressant side effects. Psychiatry (Edgmont). Feb 2009;6(2):16-18. [ FREE Full text ] [ Medline ]
• Breilmann J, Girlanda F, Guaiana G, Barbui C, Cipriani A, Castellazzi M, et al. Benzodiazepines versus placebo for panic disorder in adults. Cochrane Database Syst Rev. Mar 28, 2019;3(3):CD010677. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schuurmans J, Comijs H, Emmelkamp PMG, Weijnen IJC, van den Hout M, van Dyck R. Long-term effectiveness and prediction of treatment outcome in cognitive behavioral therapy and sertraline for late-life anxiety disorders. Int Psychogeriatr. Dec 2009;21(6):1148-1159. [ CrossRef ] [ Medline ]
• Rufer M, Albrecht R, Schmidt O, Zaum J, Schnyder U, Hand I, et al. Changes in quality of life following cognitive-behavioral group therapy for panic disorder. Eur Psychiatry. Jan 2010;25(1):8-14. [ CrossRef ] [ Medline ]
• Shear MK, Houck P, Greeno C, Masters S. Emotion-focused psychotherapy for patients with panic disorder. Am J Psychiatry. Dec 2001;158(12):1993-1998. [ CrossRef ] [ Medline ]
• Digital health, digital medicine, digital therapeutics (DTx): What’s the difference. HealthXL. Oct 23, 2023. URL: https:/​/www.​healthxl.com/​blog/​digital-health-digital-medicine-digital-therapeutics-dtx-whats-the-difference [accessed 2023-03-19]
• Cho C, Lee H. Could digital therapeutics be a game changer in psychiatry? Psychiatry Investig. Feb 2019;16(2):97-98. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kaplan A, Mannarino AP, Nickell PV. Evaluating the impact of Freespira on panic disorder patients' health outcomes and healthcare costs within the Allegheny Health Network. Appl Psychophysiol Biofeedback. Sep 2020;45(3):175-181. [ CrossRef ] [ Medline ]
• Hoshi Y. Functional near-infrared optical imaging: utility and limitations in human brain mapping. Psychophysiology. Jul 2003;40(4):511-520. [ CrossRef ] [ Medline ]
• Bystritsky A, Pontillo D, Powers M, Sabb FW, Craske MG, Bookheimer SY. Functional MRI changes during panic anticipation and imagery exposure. Neuroreport. Dec 21, 2001;12(18):3953-3957. [ CrossRef ] [ Medline ]
• Milad MR, Rauch SL. The role of the orbitofrontal cortex in anxiety disorders. Ann N Y Acad Sci. Dec 2007;1121(1):546-561. [ CrossRef ] [ Medline ]
• Shear MK, Brown TA, Barlow DH, Money R, Sholomskas DE, Woods SW, et al. Multicenter collaborative Panic Disorder Severity Scale. Am J Psychiatry. Nov 1997;154(11):1571-1575. [ CrossRef ] [ Medline ]
• Lim YJ, Yu BH, Kim JH. Korean Panic Disorder Severity Scale: construct validity by confirmatory factor analysis. Depress Anxiety. 2007;24(2):95-102. [ CrossRef ] [ Medline ]
• Maier W, Buller R, Philipp M, Heuser I. The Hamilton Anxiety Scale: reliability, validity and sensitivity to change in anxiety and depressive disorders. J Affect Disord. 1988;14(1):61-68. [ CrossRef ] [ Medline ]
• Kim CY. Psychiatric Assessment Instruments. Seoul, South Korea. Hana Psychiatric Publishing; 2001;99-101.
• Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [ CrossRef ] [ Medline ]
• Screener overview. PHQ Screeners. URL: https://www.phqscreeners.com/select-screener [accessed 2024-03-27]
• Choi J, Kim J, Hwang G, Yang J, Choi M, Bae H. Time-divided spread-spectrum code-based 400 fw-detectable multichannel fNIRS IC for portable functional brain imaging. IEEE J Solid-State Circuits. Feb 2016;51(2):484-495. [ FREE Full text ] [ CrossRef ]
• Yamamoto T, Maki A, Kadoya T, Tanikawa Y, Yamad Y, Okada E, et al. Arranging optical fibres for the spatial resolution improvement of topographical images. Phys Med Biol. Sep 21, 2002;47(18):3429-3440. [ CrossRef ] [ Medline ]
• Cui X, Bray S, Bryant DM, Glover GH, Reiss AL. A quantitative comparison of NIRS and fMRI across multiple cognitive tasks. Neuroimage. Feb 14, 2011;54(4):2808-2821. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Miyai I, Tanabe HC, Sase I, Eda H, Oda I, Konishi I, et al. Cortical mapping of gait in humans: a near-infrared spectroscopic topography study. Neuroimage. Nov 2001;14(5):1186-1192. [ CrossRef ] [ Medline ]
• Strangman G, Culver JP, Thompson JH, Boas DA. A quantitative comparison of simultaneous BOLD fMRI and NIRS recordings during functional brain activation. Neuroimage. Oct 2002;17(2):719-731. [ Medline ]
• Barlow DH, Craske MG, Cerny JA, Klosko JS. Behavioral treatment of panic disorder. Behav Ther. 1989;20(2):261-282. [ CrossRef ]
• Clark D, Salkovskis PM, Hackmann A, Wells A, Ludgate J, Gelder M. Brief cognitive therapy for panic disorder: a randomized controlled trial. J Consult Clin Psychol. Aug 1999;67(4):583-589. [ CrossRef ] [ Medline ]
• Telch MJ, Lucas JA, Schmidt NB, Hanna HH, LaNae Jaimez T, Lucas RA. Group cognitive-behavioral treatment of panic disorder. Behav Res Ther. Mar 1993;31(3):279-287. [ CrossRef ] [ Medline ]
• Bouchard S, Gauthier J, Laberge B, French D, Pelletier M, Godbout C. Exposure versus cognitive restructuring in the treatment of panic disorder with agoraphobia. Behav Res Ther. Mar 1996;34(3):213-224. [ CrossRef ] [ Medline ]
• Barlow D, Gorman JM, Shear MK, Woods SW. Cognitive-behavioral therapy, imipramine, or their combination for panic disorder: A randomized controlled trial. JAMA. May 17, 2000;283(19):2529-2536. [ CrossRef ] [ Medline ]
• Gould RA, Ott MW, Pollack MH. A meta-analysis of treatment outcome for panic disorder. Clin Psychol Rev. Jan 1995;15(8):819-844. [ CrossRef ]
• Faretta E. EMDR and cognitive behavioral therapy in the treatment of panic disorder: a comparison. J EMDR Prac Res. Jan 01, 2013;7(3):121-133. [ CrossRef ]
• Goldstein AJ, Feske U. Eye movement desensitization and reprocessing for panic disorder: a case series. J Anxiety Disord. 1994;8(4):351-362. [ CrossRef ]
• Eagleson C, Hayes S, Mathews A, Perman G, Hirsch CR. The power of positive thinking: pathological worry is reduced by thought replacement in generalized anxiety disorder. Behav Res Ther. Mar 2016;78:13-18. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shapiro JR, Bauer S. Use of short message service (SMS)-based interventions to enhance low intensity CBT. Oxford guide to low intensity CBT interventions. In: Bennett-Levy J, editor. Oxford Guide to Low Intensity CBT Interventions. Oxford, UK. Oxford University Press; 2010;281-286.
• Schoenbaum G, Roesch M. Orbitofrontal cortex, associative learning, and expectancies. Neuron. Sep 01, 2005;47(5):633-636. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Doherty J. Can't learn without you: predictive value coding in orbitofrontal cortex requires the basolateral amygdala. Neuron. Aug 28, 2003;39(5):731-733. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nordahl TE, Stein MB, Benkelfat C, Semple WE, Andreason P, Zametkin A, et al. Regional cerebral metabolic asymmetries replicated in an independent group of patients with panic disorders. Biol Psychiatry. Nov 15, 1998;44(10):998-1006. [ CrossRef ] [ Medline ]
• Monk CS, Nelson EE, McClure EB, Mogg K, Bradley BP, Leibenluft E, et al. Ventrolateral prefrontal cortex activation and attentional bias in response to angry faces in adolescents with generalized anxiety disorder. Am J Psychiatry. Jun 2006;163(6):1091-1097. [ CrossRef ] [ Medline ]
• Hariri AR, Mattay VS, Tessitore A, Fera F, Weinberger DR. Neocortical modulation of the amygdala response to fearful stimuli. Biol Psychiatry. Mar 15, 2003;53(6):494-501. [ CrossRef ] [ Medline ]
• Gorman J, Kent JM, Sullivan GM, Coplan JD. Neuroanatomical hypothesis of panic disorder, revised. Am J Psychiatry. Apr 2000;157(4):493-505. [ CrossRef ] [ Medline ]
• Tian S, Huang F, Gao J, Li P, Ouyang X, Zhou S, et al. Ventrolateral prefrontal cortex is required for fear extinction in a modified delay conditioning paradigm in rats. Neuroscience. Aug 25, 2011;189:258-268. [ CrossRef ] [ Medline ]
• Ball TM, Ramsawh HJ, Campbell-Sills L, Paulus MP, Stein MB. Prefrontal dysfunction during emotion regulation in generalized anxiety and panic disorders. Psychol Med. Jul 2013;43(7):1475-1486. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dresler T, Hindi Attar C, Spitzer C, Löwe B, Deckert J, Büchel C, et al. Neural correlates of the emotional Stroop task in panic disorder patients: an event-related fMRI study. J Psychiatr Res. Dec 2012;46(12):1627-1634. [ CrossRef ] [ Medline ]
• Beutel M, Stark R, Pan H, Silbersweig D, Dietrich S. Changes of brain activation pre-post short-term psychodynamic inpatient psychotherapy: an fMRI study of panic disorder patients. Psychiatry Res. Nov 30, 2010;184(2):96-104. [ CrossRef ] [ Medline ]
• Kircher T, Arolt V, Jansen A, Pyka M, Reinhardt I, Kellermann T, et al. Effect of cognitive-behavioral therapy on neural correlates of fear conditioning in panic disorder. Biol Psychiatry. Jan 01, 2013;73(1):93-101. [ CrossRef ] [ Medline ]
• Hofmann S. Cognitive processes during fear acquisition and extinction in animals and humans: implications for exposure therapy of anxiety disorders. Clin Psychol Rev. Feb 2008;28(2):199-210. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bishop SJ. Neural mechanisms underlying selective attention to threat. Ann N Y Acad Sci. 2008;1129(1):141-152. [ CrossRef ] [ Medline ]
• Rachman S. The overprediction of fear: a review. Behav Res Ther. Sep 1994;32(7):683-690. [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 03.08.23; peer-reviewed by M Aksoy; comments to author 01.09.23; revised version received 11.09.23; accepted 08.03.24; published 12.04.24.

©KunJung Kim, Hyunchan Hwang, Sujin Bae, Sun Mi Kim, Doug Hyun Han. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 12.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Read our research on: Gun Policy | International Conflict | Election 2024

Regions & Countries

Political typology quiz.

Notice: Beginning April 18th community groups will be temporarily unavailable for extended maintenance. Thank you for your understanding and cooperation.

Where do you fit in the political typology?

Are you a faith and flag conservative progressive left or somewhere in between.

Take our quiz to find out which one of our nine political typology groups is your best match, compared with a nationally representative survey of more than 10,000 U.S. adults by Pew Research Center. You may find some of these questions are difficult to answer. That’s OK. In those cases, pick the answer that comes closest to your view, even if it isn’t exactly right.

About Pew Research Center Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of The Pew Charitable Trusts .