critical appraisal of quantitative research

• Teesside University Student & Library Services
• Learning Hub Group

Critical Appraisal for Health Students

• Critical Appraisal of a quantitative paper
• Critical Appraisal: Help
• Critical Appraisal of a qualitative paper
• Useful resources

Appraisal of a Quantitative paper: Top tips

• Introduction

Critical appraisal of a quantitative paper (RCT)

This guide, aimed at health students, provides basic level support for appraising quantitative research papers. It's designed for students who have already attended lectures on critical appraisal. One framework for appraising quantitative research (based on reliability, internal and external validity) is provided and there is an opportunity to practise the technique on a sample article.

Please note this framework is for appraising one particular type of quantitative research a Randomised Controlled Trial (RCT) which is defined as 

a trial in which participants are randomly assigned to one of two or more groups: the experimental group or groups receive the intervention or interventions being tested; the comparison group (control group) receive usual care or no treatment or a placebo.  The groups are then followed up to see if there are any differences between the results.  This helps in assessing the effectiveness of the intervention.(CASP, 2020)

Support materials

• Framework for reading quantitative papers (RCTs)
• Critical appraisal of a quantitative paper PowerPoint

To practise following this framework for critically appraising a quantitative article, please look at the following article:

Marrero, D.G.  et al  (2016) 'Comparison of commercial and self-initiated weight loss programs in people with prediabetes: a randomized control trial',  AJPH Research , 106(5), pp. 949-956.

Critical Appraisal of a quantitative paper (RCT): practical example

• Internal Validity
• External Validity
• Reliability Measurement Tool

How to use this practical example 

Using the framework, you can have a go at appraising a quantitative paper - we are going to look at the following article:

Marrero, d.g.  et al  (2016) 'comparison of commercial and self-initiated weight loss programs in people with prediabetes: a randomized control trial',  ajph research , 106(5), pp. 949-956.,            step 1.  take a quick look at the article, step 2.  click on the internal validity tab above - there are questions to help you appraise the article, read the questions and look for the answers in the article. , step 3.   click on each question and our answers will appear., step 4.    repeat with the other aspects of external validity and reliability. , questioning the internal validity:, randomisation : how were participants allocated to each group did a randomisation process taken place, comparability of groups: how similar were the groups eg age, sex, ethnicity – is this made clear, blinding (none, single, double or triple): who was not aware of which group a patient was in (eg nobody, only patient, patient and clinician, patient, clinician and researcher) was it feasible for more blinding to have taken place , equal treatment of groups: were both groups treated in the same way , attrition : what percentage of participants dropped out did this adversely affect one group has this been evaluated, overall internal validity: does the research measure what it is supposed to be measuring, questioning the external validity:, attrition: was everyone accounted for at the end of the study was any attempt made to contact drop-outs, sampling approach: how was the sample selected was it based on probability or non-probability what was the approach (eg simple random, convenience) was this an appropriate approach, sample size (power calculation): how many participants was a sample size calculation performed did the study pass, exclusion/ inclusion criteria: were the criteria set out clearly were they based on recognised diagnostic criteria, what is the overall external validity can the results be applied to the wider population, questioning the reliability (measurement tool) internal validity:, internal consistency reliability (cronbach’s alpha). has a cronbach’s alpha score of 0.7 or above been included, test re-test reliability correlation. was the test repeated more than once were the same results received has a correlation coefficient been reported is it above 0.7 , validity of measurement tool. is it an established tool if not what has been done to check if it is reliable pilot study expert panel literature review criterion validity (test against other tools): has a criterion validity comparison been carried out was the score above 0.7, what is the overall reliability how consistent are the measurements , overall validity and reliability:, overall how valid and reliable is the paper.

• << Previous: Critical Appraisal of a qualitative paper
• Next: Useful resources >>
• Last Updated: Aug 25, 2023 2:48 PM
• URL: https://libguides.tees.ac.uk/critical_appraisal

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 21, Issue 4
• How to appraise quantitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

This article has a correction. Please see:

• Correction: How to appraise quantitative research - April 01, 2019

• Xabi Cathala 1 ,
• Calvin Moorley 2
• 1 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
• 2 Nursing Research and Diversity in Care , School of Health and Social Care, London South Bank University , London , UK
• Correspondence to Mr Xabi Cathala, Institute of Vocational Learning, School of Health and Social Care, London South Bank University London UK ; cathalax{at}lsbu.ac.uk and Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/eb-2018-102996

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Some nurses feel that they lack the necessary skills to read a research paper and to then decide if they should implement the findings into their practice. This is particularly the case when considering the results of quantitative research, which often contains the results of statistical testing. However, nurses have a professional responsibility to critique research to improve their practice, care and patient safety. 1  This article provides a step by step guide on how to critically appraise a quantitative paper.

Title, keywords and the authors

The authors’ names may not mean much, but knowing the following will be helpful:

Their position, for example, academic, researcher or healthcare practitioner.

Their qualification, both professional, for example, a nurse or physiotherapist and academic (eg, degree, masters, doctorate).

This can indicate how the research has been conducted and the authors’ competence on the subject. Basically, do you want to read a paper on quantum physics written by a plumber?

The abstract is a resume of the article and should contain:

Introduction.

Research question/hypothesis.

Methods including sample design, tests used and the statistical analysis (of course! Remember we love numbers).

Main findings.

Conclusion.

The subheadings in the abstract will vary depending on the journal. An abstract should not usually be more than 300 words but this varies depending on specific journal requirements. If the above information is contained in the abstract, it can give you an idea about whether the study is relevant to your area of practice. However, before deciding if the results of a research paper are relevant to your practice, it is important to review the overall quality of the article. This can only be done by reading and critically appraising the entire article.

The introduction

Example: the effect of paracetamol on levels of pain.

My hypothesis is that A has an effect on B, for example, paracetamol has an effect on levels of pain.

My null hypothesis is that A has no effect on B, for example, paracetamol has no effect on pain.

My study will test the null hypothesis and if the null hypothesis is validated then the hypothesis is false (A has no effect on B). This means paracetamol has no effect on the level of pain. If the null hypothesis is rejected then the hypothesis is true (A has an effect on B). This means that paracetamol has an effect on the level of pain.

Background/literature review

The literature review should include reference to recent and relevant research in the area. It should summarise what is already known about the topic and why the research study is needed and state what the study will contribute to new knowledge. 5 The literature review should be up to date, usually 5–8 years, but it will depend on the topic and sometimes it is acceptable to include older (seminal) studies.

Methodology

In quantitative studies, the data analysis varies between studies depending on the type of design used. For example, descriptive, correlative or experimental studies all vary. A descriptive study will describe the pattern of a topic related to one or more variable. 6 A correlational study examines the link (correlation) between two variables 7  and focuses on how a variable will react to a change of another variable. In experimental studies, the researchers manipulate variables looking at outcomes 8  and the sample is commonly assigned into different groups (known as randomisation) to determine the effect (causal) of a condition (independent variable) on a certain outcome. This is a common method used in clinical trials.

There should be sufficient detail provided in the methods section for you to replicate the study (should you want to). To enable you to do this, the following sections are normally included:

Overview and rationale for the methodology.

Participants or sample.

Data collection tools.

Methods of data analysis.

Ethical issues.

Data collection should be clearly explained and the article should discuss how this process was undertaken. Data collection should be systematic, objective, precise, repeatable, valid and reliable. Any tool (eg, a questionnaire) used for data collection should have been piloted (or pretested and/or adjusted) to ensure the quality, validity and reliability of the tool. 9 The participants (the sample) and any randomisation technique used should be identified. The sample size is central in quantitative research, as the findings should be able to be generalised for the wider population. 10 The data analysis can be done manually or more complex analyses performed using computer software sometimes with advice of a statistician. From this analysis, results like mode, mean, median, p value, CI and so on are always presented in a numerical format.

The author(s) should present the results clearly. These may be presented in graphs, charts or tables alongside some text. You should perform your own critique of the data analysis process; just because a paper has been published, it does not mean it is perfect. Your findings may be different from the author’s. Through critical analysis the reader may find an error in the study process that authors have not seen or highlighted. These errors can change the study result or change a study you thought was strong to weak. To help you critique a quantitative research paper, some guidance on understanding statistical terminology is provided in  table 1 .

• View inline

Some basic guidance for understanding statistics

Quantitative studies examine the relationship between variables, and the p value illustrates this objectively.  11  If the p value is less than 0.05, the null hypothesis is rejected and the hypothesis is accepted and the study will say there is a significant difference. If the p value is more than 0.05, the null hypothesis is accepted then the hypothesis is rejected. The study will say there is no significant difference. As a general rule, a p value of less than 0.05 means, the hypothesis is accepted and if it is more than 0.05 the hypothesis is rejected.

The CI is a number between 0 and 1 or is written as a per cent, demonstrating the level of confidence the reader can have in the result. 12  The CI is calculated by subtracting the p value to 1 (1–p). If there is a p value of 0.05, the CI will be 1–0.05=0.95=95%. A CI over 95% means, we can be confident the result is statistically significant. A CI below 95% means, the result is not statistically significant. The p values and CI highlight the confidence and robustness of a result.

Discussion, recommendations and conclusion

The final section of the paper is where the authors discuss their results and link them to other literature in the area (some of which may have been included in the literature review at the start of the paper). This reminds the reader of what is already known, what the study has found and what new information it adds. The discussion should demonstrate how the authors interpreted their results and how they contribute to new knowledge in the area. Implications for practice and future research should also be highlighted in this section of the paper.

A few other areas you may find helpful are:

Limitations of the study.

Conflicts of interest.

Table 2 provides a useful tool to help you apply the learning in this paper to the critiquing of quantitative research papers.

Quantitative paper appraisal checklist

• 1. ↵ Nursing and Midwifery Council , 2015 . The code: standard of conduct, performance and ethics for nurses and midwives https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21.8.18 ).
• Gerrish K ,
• Moorley C ,
• Tunariu A , et al
• Shorten A ,

Competing interests None declared.

Patient consent Not required.

Provenance and peer review Commissioned; internally peer reviewed.

Correction notice This article has been updated since its original publication to update p values from 0.5 to 0.05 throughout.

Linked Articles

• Miscellaneous Correction: How to appraise quantitative research BMJ Publishing Group Ltd and RCN Publishing Company Ltd Evidence-Based Nursing 2019; 22 62-62 Published Online First: 31 Jan 2019. doi: 10.1136/eb-2018-102996corr1

Read the full text or download the PDF:

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Mayo Clinic Library Manuals & Materials
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: Mar 25, 2024 8:25 AM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

Library Services

UCL LIBRARY SERVICES

• Guides and databases
• Library skills

LibrarySkills@UCL for NHS staff

• Critical Appraisal of a quantitative study (RCT)
• Library skills for NHS staff
• Accessing resources
• Evidence-based resources
• Bibliographic databases
• Developing your search
• Reviewing and refining your search
• Health management information
• Critical appraisal of a qualitative study
• Critical appraisal of a systematic review
• Referencing basics
• Referencing software
• Referencing styles
• Publishing and sharing research outputs
• Publishing a protocol for your systematic or scoping review
• Communicating and disseminating research
• Help and training

Critical appraisal of a quantitative study (RCT)

The following video (5 mins, 36 secs.) helps to clarify the process of critical appraisal, how to systematically examine research, e.g. using checklists; the variety of tools /checklists available, and guidance on identifying the type of research you are faced with (so you can select the most appropriate appraisal tool).

Critical appraisal of an RCT: introduction to use of CASP checklists

The following video (4 min. 58 sec.) introduces the use of CASP checklists, specifically for critical appraisal of a randomised controlled trial (RCT) study paper; how the checklist is structured, and how to effectively use it.

Webinar recording of critical appraisal of an RCT

The following video is a recording of a webinar, with facilitator and participants using a CASP checklist, to critically appraise a randomised controlled trial paper, and determine whether it constitutes good practice.

'Focus on' videos

The following videos (all approx. 2-7 mins.) focus on a particular aspects of critical appraisal methodology for quantitative studies. 

• << Previous: Evaluating information & critical appraisal
• Next: Critical appraisal of a qualitative study >>
• Last Updated: Feb 28, 2024 12:08 PM
• URL: https://library-guides.ucl.ac.uk/nhs

Critical Appraisal Tools

Jbi’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers..

These tools have been revised. Recently published articles detail the revision.

"Assessing the risk of bias of quantitative analytical studies: introducing the vision for critical appraisal within JBI systematic reviews"

"revising the jbi quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process".

End to end support for developing systematic reviews

Analytical Cross Sectional Studies  

Checklist for analytical cross sectional studies, how to cite, associated publication(s), case control studies  , checklist for case control studies, case reports  , checklist for case reports, case series  , checklist for case series.

Munn Z, Barker TH, Moola S, Tufanaru C, Stern C, McArthur A, Stephenson M, Aromataris E. Methodological quality of case series studies: an introduction to the JBI critical appraisal tool. JBI Evidence Synthesis. 2020;18(10):2127-2133

Methodological quality of case series studies: an introduction to the JBI critical appraisal tool

Cohort studies  , checklist for cohort studies, diagnostic test accuracy studies  , checklist for diagnostic test accuracy studies.

Campbell JM, Klugar M, Ding S, Carmody DP, Hakonsen SJ, Jadotte YT, White S, Munn Z. Chapter 9: Diagnostic test accuracy systematic reviews. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020

JBI Manual for Evidence Synthesis

Chapter 9: Diagnostic test accuracy systematic reviews

Economic Evaluations  

Checklist for economic evaluations, prevalence studies  , checklist for prevalence studies.

Munn Z, Moola S, Lisy K, Riitano D, Tufanaru C. Chapter 5: Systematic reviews of prevalence and incidence. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020

Chapter 5: Systematic reviews of prevalence and incidence

Qualitative Research  

Checklist for qualitative research.

Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc. 2015;13(3):179–187

Chapter 2: Systematic reviews of qualitative evidence

Qualitative research synthesis

Methodological guidance for systematic reviewers utilizing meta-aggregation

Quasi-Experimental Studies  

Checklist for quasi-experimental studies, randomized controlled trials  , randomized controlled trials.

Barker TH, Stone JC, Sears K, Klugar M, Tufanaru C, Leonardi-Bee J, Aromataris E, Munn Z. The revised JBI critical appraisal tool for the assessment of risk of bias for randomized controlled trials. JBI Evidence Synthesis. 2023;21(3):494-506

The revised JBI critical appraisal tool for the assessment of risk of bias for randomized controlled trials

Randomized controlled trials checklist (archive), systematic reviews  , checklist for systematic reviews.

Aromataris E, Fernandez R, Godfrey C, Holly C, Kahlil H, Tungpunkom P. Summarizing systematic reviews: methodological development, conduct and reporting of an Umbrella review approach. Int J Evid Based Healthc. 2015;13(3):132-40.

Chapter 10: Umbrella Reviews

Textual Evidence: Expert Opinion  

Checklist for textual evidence: expert opinion.

McArthur A, Klugarova J, Yan H, Florescu S. Chapter 4: Systematic reviews of text and opinion. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020

Chapter 4: Systematic reviews of text and opinion

Textual Evidence: Narrative  

Checklist for textual evidence: narrative, textual evidence: policy  , checklist for textual evidence: policy.

Critical Appraisal of Quantitative Research

• Living reference work entry
• Later version available View entry history
• First Online: 27 February 2018
• Cite this living reference work entry

• Rocco Cavaleri 2 ,
• Sameer Bhole 3 &
• Amit Arora 2 , 4 , 5  

234 Accesses

Critical appraisal skills are important for anyone wishing to make informed decisions or improve the quality of healthcare delivery. A good critical appraisal provides information regarding the believability and usefulness of a particular study. However, the appraisal process is often overlooked, and critically appraising quantitative research can be daunting for both researchers and clinicians. This chapter introduces the concept of critical appraisal and highlights its importance in evidence-based practice. Readers are then introduced to the most common quantitative study designs and key questions to ask when appraising each type of study. These studies include systematic reviews, experimental studies (randomized controlled trials and non-randomized controlled trials), and observational studies (cohort, case-control, and cross-sectional studies). This chapter also provides the tools most commonly used to appraise the methodological and reporting quality of quantitative studies. Overall, this chapter serves as a step-by-step guide to appraising quantitative research in healthcare settings.

• Critical appraisal
• Quantitative research
• Methodological quality
• Reporting quality

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Altman DG, Bland JM. Treatment allocation in controlled trials: why randomise? BMJ. 1999;318(7192):1209.

Article   Google Scholar  

Arora A, Scott JA, Bhole S, Do L, Schwarz E, Blinkhorn AS. Early childhood feeding practices and dental caries in preschool children: a multi-centre birth cohort study. BMC Public Health. 2011;11(1):28.

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, … Lijmer JG. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med. 2003;138(1):W1–12.

Google Scholar  

Cavaleri R, Schabrun S, Te M, Chipchase L. Hand therapy versus corticosteroid injections in the treatment of de quervain’s disease: a systematic review and meta-analysis. J Hand Ther. 2016;29(1):3–11. https://doi.org/10.1016/j.jht.2015.10.004 .

Centre for Evidence-based Management. Critical appraisal tools. 2017. Retrieved 20 Dec 2017, from https://www.cebma.org/resources-and-tools/what-is-critical-appraisal/ .

Centre for Evidence-based Medicine. Critical appraisal worksheets. 2017. Retrieved 3 Dec 2017, from http://www.cebm.net/blog/2014/06/10/critical-appraisal/ .

Clark HD, Wells GA, Huët C, McAlister FA, Salmi LR, Fergusson D, Laupacis A. Assessing the quality of randomized trials: reliability of the jadad scale. Control Clin Trials. 1999;20(5):448–52. https://doi.org/10.1016/S0197-2456(99)00026-4 .

Critical Appraisal Skills Program. Casp checklists. 2017. Retrieved 5 Dec 2017, from http://www.casp-uk.net/casp-tools-checklists .

Dawes M, Davies P, Gray A, Mant J, Seers K, Snowball R. Evidence-based practice: a primer for health care professionals. London: Elsevier; 2005.

Dumville JC, Torgerson DJ, Hewitt CE. Research methods: reporting attrition in randomised controlled trials. BMJ. 2006;332(7547):969.

Greenhalgh T, Donald A. Evidence-based health care workbook: understanding research for individual and group learning. London: BMJ Publishing Group; 2000.

Guyatt GH, Sackett DL, Cook DJ, Guyatt G, Bass E, Brill-Edwards P, … Gerstein H. Users’ guides to the medical literature: II. How to use an article about therapy or prevention. JAMA. 1993;270(21):2598–601.

Guyatt GH, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, … Jaeschke R. GRADE guidelines: 1. Introduction – GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4), 383–94.

Herbert R, Jamtvedt G, Mead J, Birger Hagen K. Practical evidence-based physiotherapy. London: Elsevier Health Sciences; 2005.

Hewitt CE, Torgerson DJ. Is restricted randomisation necessary? BMJ. 2006;332(7556):1506–8.

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.0.2. The cochrane collaboration. 2009. Retrieved 3 Dec 2017, from http://www.cochrane-handbook.org .

Hoffmann T, Bennett S, Del Mar C. Evidence-based practice across the health professions. Chatswood: Elsevier Health Sciences; 2013.

Hoffmann T, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, … Johnston M. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ, 2014;348: g1687.

Joanna Briggs Institute. Critical appraisal tools. 2017. Retrieved 4 Dec 2017, from http:// joannabriggs.org/research/critical-appraisal-tools.html .

Mhaskar R, Emmanuel P, Mishra S, Patel S, Naik E, Kumar A. Critical appraisal skills are essential to informed decision-making. Indian J Sex Transm Dis. 2009;30(2):112–9. https://doi.org/10.4103/0253-7184.62770 .

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med Res Methodol. 2001;1(1):2. https://doi.org/10.1186/1471-2288-1-2 .

Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. PLoS Med. 2009;6(7):e1000097.

National Health and Medical Research Council. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. Canberra: NHMRC; 2009. Retrieved from https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf .

National Heart Lung and Blood Institute. Study quality assessment tools. 2017. Retrieved 17 Dec 2017, from https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools .

Physiotherapy Evidence Database. PEDro scale. 2017. Retrieved 10 Dec 2017, from https://www.pedro.org.au/english/downloads/pedro-scale/ .

Portney L, Watkins M. Foundations of clinical research: application to practice. 2nd ed. Upper Saddle River: F.A. Davis Company/Publishers; 2009.

Roberts C, Torgerson DJ. Understanding controlled trials: baseline imbalance in randomised controlled trials. BMJ. 1999;319(7203):185.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, … Kristjansson E. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. https://doi.org/10.1136/bmj.j4008 .

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, … Boutron I. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, … Thacker SB. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, Initiative S. The strengthening the reporting of observational studies in epidemiology (strobe) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9.

Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, … Bossuyt PM. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155(8):529–36.

Download references

Author information

Authors and affiliations.

School of Science and Health, Western Sydney University, Sydney, NSW, Australia

Rocco Cavaleri & Amit Arora

Faculty of Dentistry, The University of Sydney, Surry Hills, NSW, Australia

Sameer Bhole

Discipline of Child and Adolescent Health, Sydney Medical School, Sydney, NSW, Australia

Oral Health Services, Sydney Local Health District and Sydney Dental Hospital, NSW Health, Sydney, NSW, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rocco Cavaleri .

Editor information

Editors and affiliations.

Health, Locked Bag 1797, CA.02.35, Western Sydney Univ, School of Science & Health, Locked Bag 1797, CA.02.35, Penrith, New South Wales, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

Cavaleri, R., Bhole, S., Arora, A. (2018). Critical Appraisal of Quantitative Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences . Springer, Singapore. https://doi.org/10.1007/978-981-10-2779-6_120-1

Download citation

DOI : https://doi.org/10.1007/978-981-10-2779-6_120-1

Received : 20 January 2018

Accepted : 12 February 2018

Published : 27 February 2018

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-2779-6

Online ISBN : 978-981-10-2779-6

eBook Packages : Springer Reference Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

• Publish with us

Policies and ethics

Chapter history

DOI: https://doi.org/10.1007/978-981-10-2779-6_120-2

DOI: https://doi.org/10.1007/978-981-10-2779-6_120-1

• Find a journal
• Track your research

• Critical Appraisal Tools
• Introduction
• Related Guides
• Getting Help

Critical Appraisal of Studies

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value/relevance in a particular context by providing a framework to evaluate the research. During the critical appraisal process, researchers can:

• Decide whether studies have been undertaken in a way that makes their findings reliable as well as valid and unbiased
• Make sense of the results
• Know what these results mean in the context of the decision they are making
• Determine if the results are relevant to their patients/schoolwork/research

Burls, A. (2009). What is critical appraisal? In What Is This Series: Evidence-based medicine. Available online at  What is Critical Appraisal?

Critical appraisal is included in the process of writing high quality reviews, like systematic and integrative reviews and for evaluating evidence from RCTs and other study designs. For more information on systematic reviews, check out our  Systematic Review  guide.

• Next: Critical Appraisal Tools >>
• Last Updated: Nov 16, 2023 1:27 PM
• URL: https://guides.library.duq.edu/critappraise

Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process

Affiliations.

• 1 JBI, Faculty of Health and Medical Sciences, The University of Adelaide, Adelaide, SA, Australia.
• 2 Queen's Collaboration for Health Care Quality, Queen's University, Kingston, ON, Canada.
• 3 Czech National Centre for Evidence-Based Healthcare and Knowledge Translation (Cochrane Czech Republic; The Czech Republic [Middle European] Centre for Evidence-Based Healthcare: A JBI Centre of Excellence; Masaryk University GRADE Centre), Faculty of Medicine, Institute of Biostatistics and Analyses, Masaryk University, Brno, Czech Republic.
• 4 The Nottingham Centre for Evidence-Based Healthcare: A JBI Centre of Excellence, School of Medicine, University of Nottingham, Nottingham, UK.
• 5 Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia.
• PMID: 36121230
• DOI: 10.11124/JBIES-22-00125

JBI offers a suite of critical appraisal instruments that are freely available to systematic reviewers and researchers investigating the methodological limitations of primary research studies. The JBI instruments are designed to be study-specific and are presented as questions in a checklist. The JBI instruments have existed in a checklist-style format for approximately 20 years; however, as the field of research synthesis expands, many of the tools offered by JBI have become outdated. The JBI critical appraisal tools for quantitative studies (eg, randomized controlled trials, quasi-experimental studies) must be updated to reflect the current methodologies in this field. Cognizant of this and the recent developments in risk-of-bias science, the JBI Effectiveness Methodology Group was tasked with updating the current quantitative critical appraisal instruments. This paper details the methods and rationale that the JBI Effectiveness Methodology Group followed when updating the JBI critical appraisal instruments for quantitative study designs. We detail the key changes made to the tools and highlight how these changes reflect current methodological developments in this field.

Copyright © 2023 JBI.

Publication types

• Research Support, Non-U.S. Gov't
• Research Design*

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PMC Canada Author Manuscripts

A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies

The practice of mixed-methods research has increased considerably over the last 10 years. While these studies have been criticized for violating quantitative and qualitative paradigmatic assumptions, the methodological quality of mixed-method studies has not been addressed. The purpose of this paper is to identify criteria to critically appraise the quality of mixed-method studies in the health literature. Criteria for critically appraising quantitative and qualitative studies were generated from a review of the literature. These criteria were organized according to a cross-paradigm framework. We recommend that these criteria be applied to a sample of mixed-method studies which are judged to be exemplary. With the consultation of critical appraisal experts and experienced qualitative, quantitative, and mixed-method researchers, further efforts are required to revise and prioritize the criteria according to importance.

1. Introduction

The practice of mixed-methods research has increased considerably over the last 10 years, as seen in numerous articles, chapters, and books published 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 . Journal series have been devoted to this topic as well (see Volume 19 of Health Education Quarterly 10 ; Number 74 of New Directions for Evaluation 11 ; Volume 34 of Health Services Research 12 ). Despite being criticized for violating quantitative and qualitative paradigmatic assumptions, the methodological quality of mixed-method studies has not been examined. While one could argue that the quality of mixed-method studies cannot be assessed until the quantitative-qualitative philosophical debate is resolved, it will be demonstrated that under the assumption that methods are linked to paradigms, it is possible to identify criteria to critically appraise mixed-method studies.

There are difficulties with developing criteria to critically appraise mixed-method studies. The definitions for qualitative and quantitative methods as well as the paradigms linked with these methods vary considerably. The critical appraisal criteria proposed for the methods often overlap depending on the dominant paradigm view taken. Finally, the language of criteria is often vague, requiring the inexperienced reader to make judgement calls 13 . This paper will address the above difficulties. It will attempt to clarify the language applied to critical appraisal and the assumptions of the authors will be presented.

The purpose of this paper is to identify criteria to critically appraise the methods of primary studies in the health literature which employ mixed-methods. A mixed-methods study is defined as one in which quantitative and qualitative methods are combined in a single study. A primary study is defined as one that contains original data on a topic 14 . The methods refer to the procedures of the study. Before conducting the literature review, the authors will first outline their assumptions concerning methods linked to paradigms, criteria linked to paradigms, and the quantitative and qualitative method. Second, the concept of critical appraisal as it applies to quantitative and qualitative methods will be reviewed. Third, the theoretical framework upon which the authors rely will be described.

1.1 Assumptions

1.1.1 methods linked to paradigms.

A paradigm is a patterned set of assumptions concerning reality (ontology), knowledge of that reality (epistemology), and the particular ways of knowing that reality (methodology) 15 . The assumption of linking methods to a paradigm or paradigms is not a novel idea. It has been argued that different paradigms should be applied to different research questions within the same study and that the paradigms should remain clearly identified and separated from each other 16 . In qualitative studies, the methods are most often linked with the paradigms of interpretivism 16 , 17 , 18 and constructivism 32 . However, other paradigms for qualitative methods have been proposed such as positivism 19 , post-positivism 13 , 19 , postmodernism 20 , and critical theory 20 . Historically, quantitative methods have been linked with the paradigm of positivism. In recent years, quantitative methods have been viewed as evolving into the practice of post-positivism 21 , 22 . In relation to critical appraisal, we view positivism and post-positivism as compatible. For the remainder of the text, the term positivism will apply to both positivism and post-positivism.

This paper assumes that quantitative methods are linked with the paradigm of positivism and that qualitative methods are linked with the paradigms of constructivism and interpretivism. This is an initial proposal for paradigms. We realize that critical appraisal criteria may change based on other paradigmatic assumptions.

1.1.2 Criteria linked to paradigms

Under the assumption that methods are linked to paradigms, it follows that the criteria to critically appraise these methods should also be linked to paradigms. The application of separate criteria for each method and paradigm is supported by many researchers 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 . Method-specific criteria provides an alternative to blending methodological assumptions 27 . Method-specific criteria also implies that each method is valued equally. While it is assumed that criteria are linked to paradigms, it is acknowledged that not all researchers 17 , 31 maintain this perspective.

1.1.3 The quantitative method

Under the assumptions that the quantitative methods are based on the paradigm of positivism, the underlying view is that all phenomenon can be reduced to empirical indicators which represent the truth. The ontological position of this paradigm is that there is only one truth, an objective reality that exists independent of human perception. Epistemologically, the investigator and investigated are independent entities. Therefore, an investigator is capable of studying a phenomenon without influencing it or being influenced by it; “inquiry takes place as through a one way mirror” 32 .

1.1.4 The qualitative method

Under the assumption that qualitative methods are based on the paradigm of interpretivism and constructivism, multiple realities or multiple truths exist based on one’s construction of reality. Reality is socially constructed 33 and so is constantly changing. On an epistemological level, there is no access to reality independent of our minds, no external referent by which to compare claims of truth 34 . The investigator and the object of study are interactively linked so that findings are mutually created within the context of the situation which shapes the inquiry 35 , 36 . This suggests that reality has no existence prior to the activity of investigation, and reality ceases to exist when we lose interest in it 34 . The emphasis of qualitative research is on process and meanings 36 .

1.2 The Concept of Critical Appraisal for Quantitative and Qualitative Methods

Guidelines for critically appraising quantitative methods exist in many peer-reviewed journals. For example, BMJ has published criteria for the critical appraisal of manuscripts which are submitted to it. More recently, qualitative journals have been publishing their own set of criteria. For example, the Canadian Journal of Public Health now has published guidelines for qualitative research papers. However, it has been argued that the idea of static criteria may violate the assumptions of the paradigms to which qualitative methods belong 37 , 38 , 39 . In this paper, we posit that there are static elements of qualitative methods which should be subject to criticism.

1.3 Theoretical Framework – Trustworthiness and Rigor proposed by Lincoln and Guba 23 , 24

Our paper relies on Lincoln and Guba’s 23 , 24 framework of trustworthiness and rigor. This framework which was selected due to its cross-paradigm appeal, guided the organization of the criteria for critically appraising mixed-method studies generated by the literature review. Although Lincoln and Guba intended the framework to reflect conventional (positivistic) versus naturalistic paradigms, we feel the framework encompasses constructivism and interpretivism under that which is “naturalistic”. (Guba and Lincoln themselves have used the term constructivism to refer to naturalistic inquiry 40 .) The term “trustworthiness”, referred to as “goodness criteria” 35 , parallels the term “rigor” used in quantitative methods 24 . Trustworthiness and rigor encompass four goals which apply to the respective methods and paradigms:

• Truth value – internal validity for quantitative methods versus credibility for qualitative methods;
• Applicability – external validity for quantitative methods versus transferability or fittingness for qualitative methods;
• Consistency – reliability for quantitative methods versus dependability for qualitative methods;
• Neutrality – objectivity for quantitative methods versus confirmability for qualitative methods.

Consistent with these goals, we propose to extend Lincoln and Guba’s theoretical framework to include parallel quantitative and qualitative critical appraisal criteria to which mixed-method studies can be applied.

The purpose of the literature review was to generate criteria which have been proposed for critically appraising the methods of quantitative, qualitative, and mixed-method studies. Although the purpose of this review was not to develop a measure of critical appraisal, the initial search for criteria was based on the principles of item generation 41 used in measurement theory. Following the principles of item generation, inclusion and exclusion rules were applied to the criteria considered.

2.1 Search Strategy

Articles judged to be exemplary by the authors were extracted from personal files and the files of colleagues, and were reviewed for MESH headings and keywords. The Subject Headings Indices for Medline, Cinahl, and PsychINFO were searched for terms referring to quantitative methods, qualitative methods, mixed-methods research, and critical appraisal. Textwords (words that appear in the titles or abstracts) were specified where there were no subject headings for quantitative, qualitative, and mixed-method studies. Keywords/textwords were assigned to a particular block; blocks were constructed using the Boolean ‘OR’ operator. Critical appraisal terms were then combined with the blocks for qualitative, quantitative, or mixed-methods using the Boolean ‘AND’ operator. Table 1 outlines the search strategy which included all available years of the selected databases.

Search strategy for critically appraising mixed-method studies

The databases Medline (1966–2000), HealthStar (1975–1999), CINAHL (1982–1999), PsycINFO (1967–2000/2001), Sociological Abstracts (1963–2000), Social Sciences Index (1983–1999), ERIC (1966–1999/09), and Current Contents (Jan 3, 1999–Feb 14, 2000) were searched. A “words anywhere” search was conducted in WebSPIRS as there are limited subject headings in the included databases. The terms for the critical appraisal block were limited in WebSPIRS as other terms such as “guidelines”, “criteria”, “assessment”, and “quality” produced a high volume of hits which were not relevant. Articles were limited to those in the English language. To complement the literature search strategy, reference lists of articles were also perused for relevant articles and the journal Qualitative Health Research for the years 1997–2001 was hand-searched. A total of four articles could not be located and therefore were not retrieved.

2.2 Inclusion and Exclusion Criteria

The articles retrieved were reviewed and the criteria for critical appraisal were subjected to inclusion and exclusion criteria. Criteria appearing in a list or scale format were extracted from this format and screened independently for inclusion and exclusion. The criteria had to refer to the methods or methods sections of the research. Finally, the criteria had to be operational in that a reader would be able to review an article and determine whether criteria were met or not. (The authors operationalized the language typically applied in the critical appraisal literature; this document is available upon request). Criteria which required judgement calls on the part of the reader such as “is the purpose of the study one of discovery and description, conceptualization, illustration, or sensitization?” 42 and “was the design of the study sensible?” 43 were excluded. Other exclusion criteria are noted in Table 2 . Where appropriate, examples of the exclusions are given.

Exclusion Criteria

As we suspected, no criteria for critically appraising mixed-method studies were found. Table 3 is a review of the criteria for critically appraising quantitative and qualitative methods separately. Because this paper assumes that methods and criteria are linked to paradigms, the criteria appear separately. As we have proposed, the organization of the table is based on the framework of trustworthiness and rigor proposed by Lincoln and Guba 23 , 24 . The criteria have been grouped accordingly but it is acknowledged that there may be some overlap of criteria across the four goals.

Review of Critical Appraisal Criteria for Quantitative and Qualitative Methods

4. Discussion

This paper has attempted to propose criteria for the purpose of critically appraising primary mixed-method studies. It is acknowledged that this is not a simple task; rather than a final product, this paper is an initial step in the process of identifying such criteria. It is necessary that the “final product” is manageable and realistic to achieve in the length of a peer-reviewed journal article. We therefore envision the final product to be a reduced version of Table 3 combined with criteria specific to mixed-methods such as acknowledgement of paradigm assumptions/differences and identification of the mixed-methods design.

An alternative to this recommendation might be the criteria shared by both methods such as the description of participants/study population and the description of the study context or setting. However, it is the authors’ opinion that the methodological evaluation of mixed-method studies should not be limited to those criteria which are shared. Paradigmatic differences between quantitative and qualitative methods imply that criteria shared by both methods may not be equally important to each method.

It is interesting to note that most of the criteria generated by the literature review fit into the first 2 categories of Lincoln and Guba’s framework. That is, most of these criteria address the goals of truth value (credibility vs. internal validity) and applicability (transferability vs. external validity). It is possible that the remaining two goals are not as relevant for critical appraisal or that it is more difficult to operationalize criteria which should be included under these goals. It is also possible that the criteria overlap and that those which are suitable for these goals are more suitable under one of the first 2 goals.

While this paper has attempted to identify criteria which are generic to mixed-method studies, future guidelines for criteria might be more specific depending upon the mixed-methods design. As mixed-methods research becomes more sophisticated, there may be varying combinations of criteria for the specific designs. For example, the criteria to critically appraise an ethnographic-randomized controlled trial design may differ from those for a phenomenological-case-control design.

The overall goal of proposing criteria for critically appraising mixed-method studies is to promote standards for guiding and assessing the methodological quality of studies of this nature. It is anticipated that, as with quantitative and qualitative research, there will eventually be guidelines for reporting and writing about mixed-method studies. It is hoped that this paper will stimulate future dialogue on the critical appraisal of such studies.

5. Future Considerations in the Pursuit of Critical Appraisal for Mixed-method Studies

The application of criteria to mixed-method studies will rely upon reported criteria. It is possible that criteria will have been met but not reported due to editorial revisions or journal guidelines concerning the length of articles. However, one purpose of this paper is to improve the reporting of methods in mixed-method studies; it can be argued, therefore, that is appropriate to make recommendations based on criteria which have been reported.

This paper is limited by the judgements made by the authors concerning exclusion and inclusion of criteria. We believed that the proposed criteria for each method should be generic and applicable to other studies within that method. We also believed that criteria had to be operationally defined. Having said this, “description of study population” does not specify what elements of a study population should be described. For the purpose of this paper, any description of the sample allowed this criteria to be met. All exclusions have been identified in Table 1 ; it is possible that these exclusions may be challenged during the revision of criteria. The authors welcome such debate. In the spirit of furthering this debate, we recommend the following considerations:

The criteria in Table 3 should be further refined and then applied to a sample of mixed-method studies in the health literature. It is also recommended that experts in mixed-methods research be asked to identify mixed-method studies which they judge to be exemplary so that the criteria met by these studies can be assessed as well.

It is noted that there are few subject headings in the selected databases which reflect the key concepts of this paper. An attempt was made to devise a comprehensive search strategy which often relied on the use of textwords. The inclusiveness of this strategy is unknown. The search strategy also focused on the peer-reviewed literature; relevant books, conference proceedings, and unpublished manuscripts may have been missed.

This paper assumed that all criteria generated by the literature were equally important. However, it is possible that the criteria met by mixed-method studies might be those which are easiest to meet rather than those which are important. The revision of criteria should involve consultation with critical appraisal experts and experienced qualitative, quantitative, and mixed-method researchers. It is proposed that this panel of experts and experienced researchers rate the criteria according to importance and that this rating be taken into account during the revision process.

Fastest Nurse Insight Engine

• MEDICAL ASSISSTANT
• Abdominal Key
• Anesthesia Key
• Basicmedical Key
• Otolaryngology & Ophthalmology
• Musculoskeletal Key
• Obstetric, Gynecology and Pediatric
• Oncology & Hematology
• Plastic Surgery & Dermatology
• Clinical Dentistry
• Radiology Key
• Thoracic Key
• Veterinary Medicine
• Gold Membership

Critical Appraisal of Quantitative and Qualitative Research for Nursing Practice

Chapter 12 Critical Appraisal of Quantitative and Qualitative Research for Nursing Practice Chapter Overview When Are Critical Appraisals of Studies Implemented in Nursing? Students’ Critical Appraisal of Studies Critical Appraisal of Studies by Practicing Nurses, Nurse Educators, and Researchers Critical Appraisal of Research Following Presentation and Publication Critical Appraisal of Research for Presentation and Publication Critical Appraisal of Research Proposals What Are the Key Principles for Conducting Intellectual Critical Appraisals of Quantitative and Qualitative Studies? Understanding the Quantitative Research Critical Appraisal Process Step 1: Identifying the Steps of the Research Process in Studies Step 2: Determining the Strengths and Weaknesses in Studies Step 3: Evaluating the Credibility and Meaning of Study Findings Example of a Critical Appraisal of a Quantitative Study Understanding the Qualitative Research Critical Appraisal Process Step 1: Identifying the Components of the Qualitative Research Process in Studies Step 2: Determining the Strengths and Weaknesses in Studies Step 3: Evaluating the Trustworthiness and Meaning of Study Finding Example of a Critical Appraisal of a Qualitative Study Key Concepts References Learning Outcomes After completing this chapter, you should be able to: 1.  Describe when intellectual critical appraisals of studies are conducted in nursing. 2.  Implement key principles in critically appraising quantitative and qualitative studies. 3.  Describe the three steps for critically appraising a study: (1) identifying the steps of the research process in the study; (2) determining study strengths and weaknesses; and (3) evaluating the credibility and meaning of the study findings. 4.  Conduct a critical appraisal of a quantitative research report. 5.  Conduct a critical appraisal of a qualitative research report. Key Terms Confirmability, p. 392 Credibility, p. 392 Critical appraisal, p. 362 Critical appraisal of qualitative studies, p. 389 Critical appraisal of quantitative studies, p. 362 Dependability, p. 392 Determining strengths and weaknesses in the studies, p. 370 Evaluating the credibility and meaning of study findings, p. 374 Identifying the steps of the research process in studies, p. 366 Intellectual critical appraisal of a study, p. 365 Qualitative research critical appraisal process, p. 389 Quantitative research critical appraisal process, p. 366 Referred journals, p. 364 Transferable, p. 392 Trustworthiness, p. 392 The nursing profession continually strives for evidence-based practice (EBP), which includes critically appraising studies, synthesizing the findings, applying the scientific evidence in practice, and determining the practice outcomes ( Brown, 2014 ; Doran, 2011 ; Melnyk & Fineout-Overholt, 2011 ). Critically appraising studies is an essential step toward basing your practice on current research findings. The term critical appraisal or critique is an examination of the quality of a study to determine the credibility and meaning of the findings for nursing. Critique is often associated with criticize, a word that is frequently viewed as negative. In the arts and sciences, however, critique is associated with critical thinking and evaluation—tasks requiring carefully developed intellectual skills. This type of critique is referred to as an intellectual critical appraisal. An intellectual critical appraisal is directed at the element that is created, such as a study, rather than at the creator, and involves the evaluation of the quality of that element. For example, it is possible to conduct an intellectual critical appraisal of a work of art, an essay, and a study. The idea of the intellectual critical appraisal of research was introduced earlier in this text and has been woven throughout the chapters. As each step of the research process was introduced, guidelines were provided to direct the critical appraisal of that aspect of a research report. This chapter summarizes and builds on previous critical appraisal content and provides direction for conducting critical appraisals of quantitative and qualitative studies. The background provided by this chapter serves as a foundation for the critical appraisal of research syntheses (systematic reviews, meta-analyses, meta-syntheses, and mixed-methods systematic reviews) presented in Chapter 13. This chapter discusses the implementation of critical appraisals in nursing by students, practicing nurses, nurse educators, and researchers. The key principles for implementing intellectual critical appraisals of quantitative and qualitative studies are described to provide an overview of the critical appraisal process. The steps for critical appraisal of quantitative studies , focused on rigor, design validity, quality, and meaning of findings, are detailed, and an example of a critical appraisal of a published quantitative study is provided. The chapter concludes with the critical appraisal process for qualitative studies and an example of a critical appraisal of a qualitative study. When are Critical Appraisals of Studies Implemented in Nursing? In general, studies are critically appraised to broaden understanding, summarize knowledge for practice, and provide a knowledge base for future research. Studies are critically appraised for class projects and to determine the research evidence ready for use in practice. In addition, critical appraisals are often conducted after verbal presentations of studies, after a published research report, for selection of abstracts when studies are presented at conferences, for article selection for publication, and for evaluation of research proposals for implementation or funding. Therefore nursing students, practicing nurses, nurse educators, and nurse researchers are all involved in the critical appraisal of studies. Students’ Critical Appraisal of Studies One aspect of learning the research process is being able to read and comprehend published research reports. However, conducting a critical appraisal of a study is not a basic skill, and the content presented in previous chapters is essential for implementing this process. Students usually acquire basic knowledge of the research process and critical appraisal process in their baccalaureate program. More advanced analysis skills are often taught at the master’s and doctoral levels. Performing a critical appraisal of a study involves the following three steps, which are detailed in this chapter: (1) identifying the steps or elements of the study; (2) determining the study strengths and limitations; and (3) evaluating the credibility and meaning of the study findings. By critically appraising studies, you will expand your analysis skills, strengthen your knowledge base, and increase your use of research evidence in practice. Striving for EBP is one of the competencies identified for associate degree and baccalaureate degree (prelicensure) students by the Quality and Safety Education for Nurses ( QSEN, 2013 ) project, and EBP requires critical appraisal and synthesis of study findings for practice ( Sherwood & Barnsteiner, 2012 ). Therefore critical appraisal of studies is an important part of your education and your practice as a nurse. Critical Appraisal of Studies by Practicing Nurses, Nurse Educators, and Researchers Practicing nurses need to appraise studies critically so that their practice is based on current research evidence and not on tradition or trial and error ( Brown, 2014 ; Craig & Smyth, 2012 ). Nursing actions need to be updated in response to current evidence that is generated through research and theory development. It is important for practicing nurses to design methods for remaining current in their practice areas. Reading research journals and posting or e-mailing current studies at work can increase nurses’ awareness of study findings but are not sufficient for critical appraisal to occur. Nurses need to question the quality of the studies, credibility of the findings, and meaning of the findings for practice. For example, nurses might form a research journal club in which studies are presented and critically appraised by members of the group ( Gloeckner & Robinson, 2010 ). Skills in critical appraisal of research enable practicing nurses to synthesize the most credible, significant, and appropriate evidence for use in their practice. EBP is essential in agencies that are seeking or maintaining Magnet status. The Magnet Recognition Program was developed by the American Nurses Credentialing Center ( ANCC, 2013 ) to “recognize healthcare organizations for quality patient care, nursing excellence, and innovations in professional nursing,” which requires implementing the most current research evidence in practice (see http://www.nursecredentialing.org/Magnet/ProgramOverview.aspx ). Your faculty members critically appraise research to expand their clinical knowledge base and to develop and refine the nursing educational process. The careful analysis of current nursing studies provides a basis for updating curriculum content for use in clinical and classroom settings. Faculty serve as role models for their students by examining new studies, evaluating the information obtained from research, and indicating which research evidence to use in practice. For example, nursing instructors might critically appraise and present the most current evidence about caring for people with hypertension in class and role-model the management of patients with hypertension in practice. Nurse researchers critically appraise previous research to plan and implement their next study. Many researchers have a program of research in a selected area, and they update their knowledge base by critically appraising new studies in this area. For example, selected nurse researchers have a program of research to identify effective interventions for assisting patients in managing their hypertension and reducing their cardiovascular risk factors. Critical Appraisal of Research Following Presentation and Publication When nurses attend research conferences, they note that critical appraisals and questions often follow presentations of studies. These critical appraisals assist researchers in identifying the strengths and weaknesses of their studies and generating ideas for further research. Participants listening to study critiques might gain insight into the conduct of research. In addition, experiencing the critical appraisal process can increase the conference participants’ ability to evaluate studies and judge the usefulness of the research evidence for practice. Critical appraisals have been published following some studies in research journals. For example, the research journals Scholarly Inquiry for Nursing Practice: An International Journal and Western Journal of Nursing Research include commentaries after the research articles. In these commentaries, other researchers critically appraise the authors’ studies, and the authors have a chance to respond to these comments. Published research critical appraisals often increase the reader’s understanding of the study and the quality of the study findings ( American Psychological Association [APA], 2010 ). A more informal critical appraisal of a published study might appear in a letter to the editor. Readers have the opportunity to comment on the strengths and weaknesses of published studies by writing to the journal editor. Critical Appraisal of Research for Presentation and Publication Planners of professional conferences often invite researchers to submit an abstract of a study they are conducting or have completed for potential presentation at the conference. The amount of information available is usually limited, because many abstracts are restricted to 100 to 250 words. Nevertheless, reviewers must select the best-designed studies with the most significant outcomes for presentation at nursing conferences. This process requires an experienced researcher who needs few cues to determine the quality of a study. Critical appraisal of an abstract usually addresses the following criteria: (1) appropriateness of the study for the conference program; (2) completeness of the research project; (3) overall quality of the study problem, purpose, methodology, and results; (4) contribution of the study to nursing’s knowledge base; (5) contribution of the study to nursing theory; (6) originality of the work (not previously published); (7) implication of the study findings for practice; and (8) clarity, conciseness, and completeness of the abstract ( APA, 2010 ; Grove, Burns, & Gray, 2013 ). Some nurse researchers serve as peer reviewers for professional journals to evaluate the quality of research papers submitted for publication. The role of these scientists is to ensure that the studies accepted for publication are well designed and contribute to the body of knowledge. Journals that have their articles critically appraised by expert peer reviews are called peer-reviewed journals or referred journals ( Pyrczak, 2008 ). The reviewers’ comments or summaries of their comments are sent to the researchers to direct their revision of the manuscripts for publication. Referred journals usually have studies and articles of higher quality and provide excellent studies for your review for practice. Critical Appraisal of Research Proposals Critical appraisals of research proposals are conducted to approve student research projects, permit data collection in an institution, and select the best studies for funding by local, state, national, and international organizations and agencies. You might be involved in a proposal review if you are participating in collecting data as part of a class project or studies done in your clinical agency. More details on proposal development and approval can be found in Grove et al. (2013, Chapter 28) . Research proposals are reviewed for funding from selected government agencies and corporations. Private corporations develop their own format for reviewing and funding research projects ( Grove et al., 2013 ). The peer review process in federal funding agencies involves an extremely complex critical appraisal. Nurses are involved in this level of research review through national funding agencies, such as the National Institute of Nursing Research ( NINR, 2013 ) and the Agency for Healthcare Research and Quality ( AHRQ, 2013 ). What are the Key Principles for Conducting Intellectual Critical Appraisals of Quantitative and Qualitative Studies? An intellectual critical appraisal of a study involves a careful and complete examination of a study to judge its strengths, weaknesses, credibility, meaning, and significance for practice. A high-quality study focuses on a significant problem, demonstrates sound methodology, produces credible findings, indicates implications for practice, and provides a basis for additional studies ( Grove et al., 2013 ; Hoare & Hoe, 2013 ; Hoe & Hoare, 2012 ). Ultimately, the findings from several quality studies can be synthesized to provide empirical evidence for use in practice ( O’Mathuna, Fineout-Overholt, & Johnston, 2011 ). The major focus of this chapter is conducting critical appraisals of quantitative and qualitative studies. These critical appraisals involve implementing some key principles or guidelines, outlined in Box 12-1 . These guidelines stress the importance of examining the expertise of the authors, reviewing the entire study, addressing the study’s strengths and weaknesses, and evaluating the credibility of the study findings ( Fawcett & Garity, 2009 ; Hoare & Hoe, 2013 ; Hoe & Hoare, 2012 ; Munhall, 2012 ). All studies have weaknesses or flaws; if every flawed study were discarded, no scientific evidence would be available for use in practice. In fact, science itself is flawed. Science does not completely or perfectly describe, explain, predict, or control reality. However, improved understanding and increased ability to predict and control phenomena depend on recognizing the flaws in studies and science. Additional studies can then be planned to minimize the weaknesses of earlier studies. You also need to recognize a study’s strengths to determine the quality of a study and credibility of its findings. When identifying a study’s strengths and weaknesses, you need to provide examples and rationale for your judgments that are documented with current literature. Box 12-1    Key Principles for Critically Appraising Quantitative and Qualitative Studies 1.  Read and critically appraise the entire study. A research critical appraisal involves examining the quality of all aspects of the research report. 2.  Examine the organization and presentation of the research report. A well-prepared report is complete, concise, clearly presented, and logically organized. It does not include excessive jargon that is difficult for you to read. The references need to be current, complete, and presented in a consistent format. 3.  Examine the significance of the problem studied for nursing practice. The focus of nursing studies needs to be on significant practice problems if a sound knowledge base is to be developed for evidence-based nursing practice. 4.  Indicate the type of study conducted and identify the steps or elements of the study. This might be done as an initial critical appraisal of a study; it indicates your knowledge of the different types of quantitative and qualitative studies and the steps or elements included in these studies. 5.  Identify the strengths and weaknesses of a study. All studies have strengths and weaknesses, so attention must be given to all aspects of the study. 6.  Be objective and realistic in identifying the study’s strengths and weaknesses. Be balanced in your critical appraisal of a study. Try not to be overly critical in identifying a study’s weaknesses or overly flattering in identifying the strengths. 7.  Provide specific examples of the strengths and weaknesses of a study. Examples provide evidence for your critical appraisal of the strengths and weaknesses of a study. 8.  Provide a rationale for your critical appraisal comments. Include justifications for your critical appraisal, and document your ideas with sources from the current literature. This strengthens the quality of your critical appraisal and documents the use of critical thinking skills. 9.  Evaluate the quality of the study. Describe the credibility of the findings, consistency of the findings with those from other studies, and quality of the study conclusions. 10.  Discuss the usefulness of the findings for practice. The findings from the study need to be linked to the findings of previous studies and examined for use in clinical practice. Critical appraisal of quantitative and qualitative studies involves a final evaluation to determine the credibility of the study findings and any implications for practice and further research (see Box 12-1 ). Adding together the strong points from multiple studies slowly builds a solid base of evidence for practice. These guidelines provide a basis for the critical appraisal process for quantitative research discussed in the next section and the critical appraisal process for qualitative research (see later). Understanding the Quantitative Research Critical Appraisal Process The quantitative research critical appraisal process includes three steps: (1) identifying the steps of the research process in studies; (2) determining study strengths and weaknesses; and (3) evaluating the credibility and meaning of study findings. These steps occur in sequence, vary in depth, and presume accomplishment of the preceding steps. However, an individual with critical appraisal experience frequently performs two or three steps of this process simultaneously. This section includes the three steps of the quantitative research critical appraisal process and provides relevant questions for each step. These questions have been selected as a means for stimulating the logical reasoning and analysis necessary for conducting a critical appraisal of a study. Those experienced in the critical appraisal process often formulate additional questions as part of their reasoning processes. We will identify the steps of the research process separately because those new to critical appraisal start with this step. The questions for determining the study strengths and weaknesses are covered together because this process occurs simultaneously in the mind of the person conducting the critical appraisal. Evaluation is covered separately because of the increased expertise needed to perform this step. Step 1: Identifying the Steps of the Research Process in Studies Initial attempts to comprehend research articles are often frustrating because the terminology and stylized manner of the report are unfamiliar. Identifying the steps of the research process in a quantitative study is the first step in critical appraisal. It involves understanding the terms and concepts in the report, as well as identifying study elements and grasping the nature, significance, and meaning of these elements. The following guidelines will direct you in identifying a study’s elements or steps. Guidelines for Identifying the Steps of the Research Process in Studies The first step involves reviewing the abstract and reading the study from beginning to end. As you read, think about the following questions regarding the presentation of the study: •  Was the study title clear? •  Was the abstract clearly presented? •  Was the writing style of the report clear and concise? •  Were relevant terms defined? You might underline the terms you do not understand and determine their meaning from the glossary at the end of this text. •  Were the following parts of the research report plainly identified ( APA, 2010 )? •   Introduction section, with the problem, purpose, literature review, framework, study variables, and objectives, questions, or hypotheses •   Methods section, with the design, sample, intervention (if applicable), measurement methods, and data collection or procedures •   Results section, with the specific results presented in tables, figures, and narrative •   Discussion section, with the findings, conclusions, limitations, generalizations, implications for practice, and suggestions for future research ( Fawcett & Garity, 2009 ; Grove et al., 2013 ) We recommend reading the research article a second time and highlighting or underlining the steps of the quantitative research process that were identified previously. An overview of these steps is presented in Chapter 2 . After reading and comprehending the content of the study, you are ready to write your initial critical appraisal of the study. To write a critical appraisal identifying the study steps, you need to identify each step of the research process concisely and respond briefly to the following guidelines and questions. 1.  Introduction a.  Describe the qualifications of the authors to conduct the study (e.g., research expertise conducting previous studies, clinical experience indicated by job, national certification, and years in practice, and educational preparation that includes conducting research [PhD]). b.  Discuss the clarity of the article title. Is the title clearly focused and does it include key study variables and population? Does the title indicate the type of study conducted—descriptive, correlational, quasi-experimental, or experimental—and the variables ( Fawcett & Garity, 2009 ; Hoe & Hoare, 2012 ; Shadish, Cook, & Campbell, 2002 )? c.  Discuss the quality of the abstract (includes purpose, highlights design, sample, and intervention [if applicable], and presents key results; APA, 2010 ). 2.  State the problem. a.  Significance of the problem b.  Background of the problem c.  Problem statement 3.  State the purpose. 4.  Examine the literature review. a.  Are relevant previous studies and theories described? b.  Are the references current (number and percentage of sources in the last 5 and 10 years)? c.  Are the studies described, critically appraised, and synthesized ( Brown, 2014 ; Fawcett & Garity, 2009 )? Are the studies from referred journals? d.  Is a summary provided of the current knowledge (what is known and not known) about the research problem? 5.  Examine the study framework or theoretical perspective. a.  Is the framework explicitly expressed, or must you extract the framework from statements in the introduction or literature review of the study? b.  Is the framework based on tentative, substantive, or scientific theory? Provide a rationale for your answer. c.  Does the framework identify, define, and describe the relationships among the concepts of interest? Provide examples of this. d.  Is a map of the framework provided for clarity? If a map is not presented, develop a map that represents the study’s framework and describe the map. e.  Link the study variables to the relevant concepts in the map. f.  How is the framework related to nursing’s body of knowledge ( Alligood, 2010 ; Fawcett & Garity, 2009 ; Smith & Liehr, 2008 )? 6.  List any research objectives, questions, or hypotheses. 7.  Identify and define (conceptually and operationally) the study variables or concepts that were identified in the objectives, questions, or hypotheses. If objectives, questions, or hypotheses are not stated, identify and define the variables in the study purpose and results section of the study. If conceptual definitions are not found, identify possible definitions for each major study variable. Indicate which of the following types of variables were included in the study. A study usually includes independent and dependent variables or research variables, but not all three types of variables. a.  Independent variables: Identify and define conceptually and operationally. b.  Dependent variables: Identify and define conceptually and operationally. c.  Research variables or concepts: Identify and define conceptually and operationally. 8.  Identify attribute or demographic variables and other relevant terms. 9.  Identify the research design. a.  Identify the specific design of the study (see Chapter 8 ). b.  Does the study include a treatment or intervention? If so, is the treatment clearly described with a protocol and consistently implemented? c.  If the study has more than one group, how were subjects assigned to groups? d.  Are extraneous variables identified and controlled? Extraneous variables are usually discussed as a part of quasi-experimental and experimental studies. e.  Were pilot study findings used to design this study? If yes, briefly discuss the pilot and the changes made in this study based on the pilot ( Grove et al., 2013 ; Shadish et al., 2002 ). 10.  Describe the sample and setting. a.  Identify inclusion and exclusion sample or eligibility criteria. b.  Identify the specific type of probability or nonprobability sampling method that was used to obtain the sample. Did the researchers identify the sampling frame for the study? c.  Identify the sample size. Discuss the refusal number and percentage, and include the rationale for refusal if presented in the article. Discuss the power analysis if this process was used to determine sample size ( Aberson, 2010 ). d.  Identify the sample attrition (number and percentage) for the study. e.  Identify the characteristics of the sample. f.  Discuss the institutional review board (IRB) approval. Describe the informed consent process used in the study. g.  Identify the study setting and indicate whether it is appropriate for the study purpose. 11.  Identify and describe each measurement strategy used in the study. The following table includes the critical information about two measurement methods, the Beck Likert scale and a physiological instrument to measure blood pressure. Completing this table will allow you to cover essential measurement content for a study ( Waltz, Strickland, & Lenz, 2010 ). a.  Identify each study variable that was measured. b.  Identify the name and author of each measurement strategy. c.  Identify the type of each measurement strategy (e.g., Likert scale, visual analog scale, physiological measure, or existing database). d.  Identify the level of measurement (nominal, ordinal, interval, or ratio) achieved by each measurement method used in the study ( Grove, 2007 ). e.  Describe the reliability of each scale for previous studies and this study. Identify the precision of each physiological measure ( Bialocerkowski, Klupp, & Bragge, 2010 ; DeVon et al., 2007 ). f.  Identify the validity of each scale and the accuracy of physiological measures ( DeVon et al., 2007 ; Ryan-Wenger, 2010 ). Variable Measured Name of Measurement Method (Author) Type of Measurement Method Level of Measurement Reliability or Precision Validity or Accuracy Depression Beck Depression Inventory (Beck) Likert scale Interval Cronbach alphas of 0.82-0.92 from previous studies and 0.84 for this study; reading level at 6th grade. Construct validity—content validity from concept analysis, literature review, and reviews of experts; convergent validity of 0.04 with Zung Depression Scale; predictive validity of patients’ future depression episodes; successive use validity with the conduct of previous studies and this study. Blood pressure Omron blood pressure (BP) equipment (equipment manufacturer) Physiological measurement method Ratio Test-retest values of BPs in previous studies; BP equipment new and recalibrated every 50 BP readings in this study; average three BP readings to determine BP. Documented accuracy of systolic and diastolic BPs to 1 mm Hg by company developing Omron BP cuff; designated protocol for taking BP average three BP readings to determine BP. 12.  Describe the procedures for data collection. 13.  Describe the statistical analyses used. a.  List the statistical procedures used to describe the sample ( Grove, 2007 ). b.  Was the level of significance or alpha identified? If so, indicate what it was (0.05, 0.01, or 0.001). c.  Complete the following table with the analysis techniques conducted in the study: (1) identify the focus (description, relationships, or differences) for each analysis technique; (2) list the statistical analysis technique performed; (3) list the statistic; (4) provide the specific results; and (5) identify the probability ( p ) of the statistical significance achieved by the result ( Grove, 2007 ; Grove et al., 2013 ; Hoare & Hoe, 2013 ; Plichta & Kelvin, 2013 ). Purpose of Analysis Analysis Technique Statistic Results Probability (p) Description of subjects’ pulse rate Mean Standard deviation Range M SD range 71.52 5.62 58-97   Difference between adult males and females on blood pressure t -Test t 3.75 p  = 0.001 Differences of diet group, exercise group, and comparison group for pounds lost in adolescents Analysis of variance F 4.27 p  = 0.04 Relationship of depression and anxiety in older adults Pearson correlation r 0.46 p  = 0.03 14.  Describe the researcher’s interpretation of findings. a.  Are the findings related back to the study framework? If so, do the findings support the study framework? b.  Which findings are consistent with those expected? c.  Which findings were not expected? d.  Are the findings consistent with previous research findings? ( Fawcett & Garity, 2009 ; Grove et al., 2013 ; Hoare & Hoe, 2013 ) 15.  What study limitations did the researcher identify? 16.  What conclusions did the researchers identify based on their interpretation of the study findings? 17.  How did the researcher generalize the findings? 18.  What were the implications of the findings for nursing practice? 19.  What suggestions for further study were identified? 20.  Is the description of the study sufficiently clear for replication? Step 2: Determining the Strengths and Weaknesses in Studies The second step in critically appraising studies requires determining strengths and weaknesses in the studies . To do this, you must have knowledge of what each step of the research process should be like from expert sources such as this text and other research sources ( Aberson, 2010 ; Bialocerkowski et al., 2010 ; Brown, 2014 ; Creswell, 2014 ; DeVon et al., 2007 ; Doran, 2011 ; Fawcett & Garity, 2009 ; Grove, 2007 ; Grove et al., 2013 ; Hoare & Hoe, 2013 ; Hoe & Hoare, 2012 ; Morrison, Hoppe, Gillmore, Kluver, Higa, & Wells, 2009 ; O’Mathuna et al., 2011 ; Ryan-Wenger, 2010 ; Santacroce, Maccarelli, & Grey, 2004 ; Shadish et al., 2002 ; Waltz et al., 2010 ). The ideal ways to conduct the steps of the research process are then compared with the actual study steps. During this comparison, you examine the extent to which the researcher followed the rules for an ideal study, and the study elements are examined for strengths and weaknesses. You also need to examine the logical links or flow of the steps in the study being appraised. For example, the problem needs to provide background and direction for the statement of the purpose. The variables identified in the study purpose need to be consistent with the variables identified in the research objectives, questions, or hypotheses. The variables identified in the research objectives, questions, or hypotheses need to be conceptually defined in light of the study framework. The conceptual definitions should provide the basis for the development of operational definitions. The study design and analyses need to be appropriate for the investigation of the study purpose, as well as for the specific objectives, questions, or hypotheses. Examining the quality and logical links among the study steps will enable you to determine which steps are strengths and which steps are weaknesses. Guidelines for Determining the Strengths and Weaknesses in Studies The following questions were developed to help you examine the different steps of a study and determine its strengths and weaknesses. The intent is not for you to answer each of these questions but to read the questions and then make judgments about the steps in the study. You need to provide a rationale for your decisions and document from relevant research sources, such as those listed previously in this section and in the references at the end of this chapter. For example, you might decide that the study purpose is a strength because it addresses the study problem, clarifies the focus of the study, and is feasible to investigate ( Brown, 2014 ; Fawcett & Garity, 2009 ; Hoe & Hoare, 2012 ). 1.  Research problem and purpose a.  Is the problem significant to nursing and clinical practice ( Brown, 2014 )? b.  Does the purpose narrow and clarify the focus of the study ( Creswell, 2014 ; Fawcett & Garity, 2009 )? c.  Was this study feasible to conduct in terms of money commitment, the researchers’ expertise; availability of subjects, facilities, and equipment; and ethical considerations? 2.  Review of literature a.  Is the literature review organized to demonstrate the progressive development of evidence from previous research ( Brown, 2014 ; Creswell, 2014 ; Hoe & Hoare, 2012 )? b.  Is a clear and concise summary presented of the current empirical and theoretical knowledge in the area of the study ( O’Mathuna et al., 2011 )? c.  Does the literature review summary identify what is known and not known about the research problem and provide direction for the formation of the research purpose? 3.  Study framework a.  Is the framework presented with clarity? If a model or conceptual map of the framework is present, is it adequate to explain the phenomenon of concern ( Grove et al., 2013 )? b.  Is the framework related to the body of knowledge in nursing and clinical practice? c.  If a proposition from a theory is to be tested, is the proposition clearly identified and linked to the study hypotheses ( Alligood, 2010 ; Fawcett & Garity, 2009 ; Smith & Liehr, 2008 )? 4.  Research objectives, questions, or hypotheses a.  Are the objectives, questions, or hypotheses expressed clearly? b.  Are the objectives, questions, or hypotheses logically linked to the research purpose? c.  Are hypotheses stated to direct the conduct of quasi-experimental and experimental research ( Creswell, 2014 ; Shadish et al., 2002 )? d.  Are the objectives, questions, or hypotheses logically linked to the concepts and relationships (propositions) in the framework ( Chinn & Kramer, 2011 ; Fawcett & Garity, 2009 ; Smith & Liehr, 2008 )? 5.  Variables a.  Are the variables reflective of the concepts identified in the framework? b.  Are the variables clearly defined (conceptually and operationally) and based on previous research or theories ( Chinn & Kramer, 2011 ; Grove et al., 2013 ; Smith & Liehr, 2008 )? c.  Is the conceptual definition of a variable consistent with the operational definition? 6.  Design a.  Is the design used in the study the most appropriate design to obtain the needed data ( Creswell, 2014 ; Grove et al., 2013 ; Hoe & Hoare, 2012 )? b.  Does the design provide a means to examine all the objectives, questions, or hypotheses? c.  Is the treatment clearly described ( Brown, 2002 )? Is the treatment appropriate for examining the study purpose and hypotheses? Does the study framework explain the links between the treatment (independent variable) and the proposed outcomes (dependent variables)? Was a protocol developed to promote consistent implementation of the treatment to ensure intervention fidelity ( Morrison et al., 2009 )? Did the researcher monitor implementation of the treatment to ensure consistency ( Santacroce et al., 2004 )? If the treatment was not consistently implemented, what might be the impact on the findings? d.  Did the researcher identify the threats to design validity (statistical conclusion validity, internal validity, construct validity, and external validity [see Chapter 8 ]) and minimize them as much as possible ( Grove et al., 2013 ; Shadish et al., 2002 )? e.  If more than one group was used, did the groups appear equivalent? f.  If a treatment was implemented, were the subjects randomly assigned to the treatment group or were the treatment and comparison groups matched? Were the treatment and comparison group assignments appropriate for the purpose of the study? 7.  Sample, population, and setting a.  Is the sampling method adequate to produce a representative sample? Are any subjects excluded from the study because of age, socioeconomic status, or ethnicity, without a sound rationale? b.  Did the sample include an understudied population, such as young people, older adults, or minority group? c.  Were the sampling criteria (inclusion and exclusion) appropriate for the type of study conducted ( O’Mathuna et al., 2011 )? d.  Was a power analysis conducted to determine sample size? If a power analysis was conducted, were the results of the analysis clearly described and used to determine the final sample size? Was the attrition rate projected in determining the final sample size ( Aberson, 2010 )? e.  Are the rights of human subjects protected ( Creswell, 2014 ; Grove et al., 2013 )? f.  Is the setting used in the study typical of clinical settings? g.  Was the rate of potential subjects’ refusal to participate in the study a problem? If so, how might this weakness influence the findings? h.  Was sample attrition a problem? If so, how might this weakness influence the final sample and the study results and findings ( Aberson, 2010 ; Fawcett & Garity, 2009 ; Hoe & Hoare, 2012 )? 8.  Measurements a.  Do the measurement methods selected for the study adequately measure the study variables? Should additional measurement methods have been used to improve the quality of the study outcomes ( Waltz et al., 2010 )? b.  Do the measurement methods used in the study have adequate validity and reliability? What additional reliability or validity testing is needed to improve the quality of the measurement methods ( Bialocerkowski et al., 2010 ; DeVon et al., 2007 ; Roberts & Stone, 2003 )? c.  Respond to the following questions, which are relevant to the measurement approaches used in the study: 1)  Scales and questionnaires (a)  Are the instruments clearly described? (b)  Are techniques to complete and score the instruments provided? (c)  Are the validity and reliability of the instruments described ( DeVon et al., 2007 )? (d)  Did the researcher reexamine the validity and reliability of the instruments for the present sample? (e)  If the instrument was developed for the study, is the instrument development process described ( Grove et al., 2013 ; Waltz et al., 2010 )? 2)  Observation (a)  Is what is to be observed clearly identified and defined? (b)  Is interrater reliability described? (c)  Are the techniques for recording observations described ( Waltz et al., 2010 )? 3)  Interviews (a)  Do the interview questions address concerns expressed in the research problem? (b)  Are the interview questions relevant for the research purpose and objectives, questions, or hypotheses ( Grove et al., 2013 ; Waltz et al., 2010 )? 4)  Physiological measures (a)  Are the physiological measures or instruments clearly described ( Ryan-Wenger, 2010 )? If appropriate, are the brand names of the instruments identified, such as Space Labs or Hewlett-Packard? (b)  Are the accuracy, precision, and error of the physiological instruments discussed ( Ryan-Wenger, 2010 )? (c)  Are the physiological measures appropriate for the research purpose and objectives, questions, or hypotheses? (d)  Are the methods for recording data from the physiological measures clearly described? Is the recording of data consistent? 9.  Data collection a.  Is the data collection process clearly described ( Fawcett & Garity, 2009 ; Grove et al., 2013 )? b.  Are the forms used to collect data organized to facilitate computerizing the data? c.  Is the training of data collectors clearly described and adequate? d.  Is the data collection process conducted in a consistent manner? e.  Are the data collection methods ethical? f.  Do the data collected address the research objectives, questions, or hypotheses? g.  Did any adverse events occur during data collection, and were these appropriately managed? 10.  Data analysis a.  Are data analysis procedures appropriate for the type of data collected ( Grove, 2007 ; Hoare & Hoe, 2013 ; Plichta & Kelvin, 2013 )? b.  Are data analysis procedures clearly described? Did the researcher address any problem with missing data, and explain how this problem was managed? c.  Do the data analysis techniques address the study purpose and the research objectives, questions, or hypotheses ( Fawcett & Garity, 2009 ; Grove et al., 2013 ; Hoare & Hoe, 2013 )? d.  Are the results presented in an understandable way by narrative, tables, or figures or a combination of methods ( APA, 2010 )? e.  Is the sample size sufficient to detect significant differences, if they are present? f.  Was a power analysis conducted for nonsignificant results (Aberson, 2010) ? g.  Are the results interpreted appropriately?

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)

Related posts:

• Outcomes Research
• Understanding Statistics in Research
• Research Problems, Purposes, and Hypotheses
• Clarifying Quantitative Research Designs

Stay updated, free articles. Join our Telegram channel

Comments are closed for this page.

Full access? Get Clinical Tree